Skip to content

Improve output of tests for frameworks which have no <IT::> #262

@hobovsky

Description

@hobovsky

Initially asked on Discord:

Codewars runner supports some languages and testing frameworks which do not fit well into the BDD pattern and do not have constructs which map well into <DESCRIBE::> and <IT::> markers used by test output panel. As a consequence, some reporters emit artificially created <IT::> tags, with titles derived from some additional context data because authors cannot assign any title to them. This often results in an awkward output with confusing or misplaced titles (see #239 , #241 , codewars/FactCheck.jl#3 ), or which spam <IT::> beyond reason.

My proposal would be to get rid of artificial <IT::>'s from all reporters for frameworks which do not have a corresponding construct, and add support for assertions which are not in scope of any <IT::>. Maybe it would be good enough to allow for assertions inside of <DESCRIBE::>, or maybe a better solution would be to provide some new marker, which would be used for frameworks which do not map well into BDD and use simpler concept of a test and a context, for exampel something like:

<TEST::>
  <TEST::>                     (if nesting is allowed)
    <ASSERTIONS_CONTEXT::>     (optional)
      <ASSERTIONS_CONTEXT::>   (if nesting is possible)
        <ASSERTION::>          (probably <PASSED::> or <FAILED::>)
        <ASSERTION::>          (if not stop on first failure)
      <COMPLETEDIN::>
    <COMPLETEDIN::>
  <COMPLETEDIN::>
<COMPLETEDIN::>

With such setup, Julia reporter would emit <TEST::> for @facts (which is optional), <ASSERTIONS_CONTEXT::> for @context (which is optional), and <PASSED::> or <FAILED::> for @fact. For Clojure, we'd get <TEST::> for deftest (whici is mandatory), <ASSERTIONS_CONTEXT::> for testing (which is optional), and <PASSED::> or <FAILED::> for is.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions