Support for Cucumber Engine

[Leon0402]

I tried this sbt plugin with cucumber for JUnit5. It uses the JUnit Suite Engine for configuration.

I use these libraries:

libraryDependencies += "io.cucumber" %% "cucumber-scala" % "8.26.1" % Test
libraryDependencies += "io.cucumber" % "cucumber-junit-platform-engine" % "7.21.1" % Test
libraryDependencies += "org.junit.platform" % "junit-platform-suite" % "1.11.4" % Test
libraryDependencies += "org.junit.jupiter" % "junit-jupiter-api" % "5.11.4" % Test
libraryDependencies += "com.github.sbt.junit" % "jupiter-interface" % "0.13.3" % Test
libraryDependencies += "org.scalatest" %% "scalatest" % "3.2.19" % Test

and this configuration:

package steps

import org.junit.platform.suite.api.{SelectPackages, Suite}
import org.junit.platform.suite.api.SelectClasspathResource

@Suite
@SelectClasspathResource("features/example.feature")
class RunCucumberTest

The output is:

org.junit.platform.commons.JUnitException: TestEngine with ID 'cucumber' failed to discover tests
Caused by: org.junit.platform.commons.PreconditionViolationException: Could not load class with name: Example feature
....
[info] Test run started (JUnit Platform Suite)
[info] Test classpath:features/example.feature 3 started
[info] Given step executed
[info] When step executed
[info] Then step executed
[info] Test run finished: 0 failed, 0 ignored, 1 total, 0.146s

I suspect that this plugin somehow executes the TestEngine twice. Once without my configuration (which fails) and once with (which succeeds). Although, I am not deep enough into the topic to say with certainty. Is this a bug / missing feature of this plugin, or did I misconfigure something?

I have created a minimal repo for reproduction:
https://github.com/Leon0402/Cucumber-Scala-with-JUnit5

Thank you!

[mpkorstanje]

I was looking just at the same thing (cucumber/cucumber-jvm-scala#399).

The problem is two fold

1. The JUnit Platform supports tests in arbitrary files. Not just classes.

2. The SBT Jupiter interface assumes that all tests are class based.

Specifically the problem sits on the last line of this method:

sbt-jupiter-interface/src/library/src/main/java/com/github/sbt/junit/jupiter/api/JupiterTestCollector.java

Lines 260 to 282 in 67f99db

	private String fullyQualifiedName(TestIdentifier testIdentifier) {
	TestSource testSource = testIdentifier.getSource().orElse(null);
	if (testSource instanceof ClassSource) {
	ClassSource classSource = (ClassSource) testSource;
	return classSource.getClassName();
	}
	if (testSource instanceof MethodSource) {
	MethodSource methodSource = (MethodSource) testSource;
	return methodSource.getClassName()
	+ '#'
	+ methodSource.getMethodName()
	+ '('
	+ methodSource.getMethodParameterTypes()
	+ ')';
	}
	return testIdentifier.getLegacyReportingName();
	}

It is not guaranteed that testIdentifier.getLegacyReportingName() returns a class name. As the Javadoc notes:

https://docs.junit.org/5.0.1/api/org/junit/platform/engine/TestDescriptor.html#getLegacyReportingName--

Get the name of this descriptor in a format that is suitable for legacy reporting infrastructure — for example, for reporting systems built on the Ant-based XML reporting format for JUnit 4.

The default implementation simply delegates to getDisplayName().

And for getDisplayName it says:

A display name is a human-readable name for a test or container that is typically used for test reporting in IDEs and build tools. Display names may contain spaces, special characters, and emoji, and the format may be customized by TestEngines or potentially by end users as well. Consequently, display names should never be parsed; rather, they should be used for display purposes only.

So the fix is presumably to filter out tests without a class or method sources.

This won't affect Cucumber as the recommended way to run Cucumber with build tools that that only support class based tests is through the JUnit Platform Suite engine which is class based.

[mpkorstanje]

Also worth noting:

sbt-jupiter-interface/src/library/src/main/java/com/github/sbt/junit/jupiter/api/JupiterTestCollector.java

Lines 228 to 235 in 67f99db

	Set<Path> classPathRoots = new HashSet<>();
	classPathRoots.add(Paths.get(classDirectory.getAbsolutePath()));
	LauncherDiscoveryRequest request =
	LauncherDiscoveryRequestBuilder.request()
	.selectors(selectClasspathRoots(classPathRoots))
	.selectors(selectDirectory(classDirectory))
	.build();

By doing a discovery request with both directory and class path root selectors Cucumber will discover all features twice. In this instance I think the selectDirectory can be safely omitted as this is not an arbitrary directory but the classpath root.

[mpkorstanje]

Since cucumber/cucumber-jvm#3023 you can work around this problem by upgrading to Cucumber v7.26 and setting cucumber.junit-platform.discovery.as-root-engine to false in junit-platform.properties. This should ensure Cucumber only discovers tests when called by another engine (e.g. the JUnit Platform Suite Engine) and bypass SBTs discovery issues.