Add test_dependency_versions, update test_runner #1729
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@simuons @liucijus Note that this adds three new CI jobs. Of the three, the macOS version is dog slow for some reason. So if you want to only run a Linux job, Linux and Windows, or leave Also note that I updated the PR to prevent the |
This was referenced Apr 29, 2025
Adds `test_dependency_versions.sh` to validate the minimum Bazel and
dependency versions required by `rules_scala`, with and without the
precompiled protocol compiler toolchain. Extracts setup, run, and
teardown helpers from `test/shell/test_bzlmod_helpers.sh` into
`test_runner.sh` for reuse by `test_dependency_versions.sh`.
Also:
- Adds test files supporting `test_dependency_versions.sh` in
`deps/test`.
- Adds a mechanism to skip tests by prefixing their name with `_` to
`run_test_local` and `run_test_ci`.
- Adds the `RULES_SCALA_TEST_REGEX` environment variable to
`test_runner.sh`.
- Adds documentation for `RULES_SCALA_TEST_{ONLY,REGEX,VERBOSE}` to the
header comment of `test_runner.sh`.
- Adds `./test_dependency_versions` jobs to `.bazelci/presubmit.yml`.
---
Continuation of the previous change to ensure we don't force users to
upgrade their dependencies beyond the minimum versions supported by
`rules_scala`. Only builds using Bzlmod, as `WORKSPACE` is considered
legacy.
Inspired by a thread in the #bzlmod channel of the Bazel Slack workspace
on 2025-01-01 indicating that rules should require the minumum versions
possible:
- https://bazelbuild.slack.com/archives/C014RARENH0/p1743597941149639
Copied the setup for the Windows `test_rules_scala` job to the Windows
`test_dependency_versions` job. Replaced `cp "${test_files[@]}" .` with
copying the list of files directly instead of keeping them in an array.
The Linux job passed. The Windows job didn't run the script at all. The
macOS build broke in a way I've not seen while developing locally:
```txt
cp:
(/Users/buildkite/builds/bk-macos-intel-m3q2/bazel/rules-scala-scala/deps/test/*.bzl
/Users/buildkite/builds/bk-macos-intel-m3q2/bazel/rules-scala-scala/examples/testing/multi_frameworks_toolchain/example/*.scala
/Users/buildkite/builds/bk-macos-intel-m3q2/bazel/rules-scala-scala/test/jmh/data.txt
/Users/buildkite/builds/bk-macos-intel-m3q2/bazel/rules-scala-scala/test/proto/standalone.proto
/Users/buildkite/builds/bk-macos-intel-m3q2/bazel/rules-scala-scala/test/src/main/scala/scalarules/test/twitter_scrooge/thrift/thrift2/thrift3/Thrift3.thrift):
No such file or directory
```
The script expects a directory path at the beginning of
`${BASH_SOURCE[0]}`, else the following error occurs:
```txt
test_dependency_versions.sh: line 18: cd: test_dependency_versions.sh:
Not a directory
```
This is because `${BASH_SOURCE[0]%/*}` will return the original
`${BASH_SOURCE[0]}` without a leading path.
This failure in the Linux `test_dependency_versions` job looks like a
fluke, since the previous run succeeded, as did other current runs:
```
WARNING: Download from
https://github.com/bazelbuild/bazel-skylib/releases/download/1.6.0/bazel-skylib-1.6.0.tar.gz
failed: class java.io.IOException GET returned 618 jwt:jwt-not-provided
``
Works around the following `test_dependency_version.sh` failure:
```txt
FATAL:
ExecuteProgram(C:\tools\msys64\home\b\_bazel_b\...\ScalafmtTest.format-test)
failed: ERROR: src/main/native/windows/process.cc(202):
CreateProcessW("C:\tools\msys64\home\b\_bazel_b\...\ScalafmtTest.format-test"):
%1 is not a valid Win32 application.
(error: 193)
Test "test_precompiled_protoc_rules_java_7" failed (49 sec)
```
mbland
force-pushed
the
test-dependency-versions
branch
from
cefbe8d to
56903c0
Compare
simuons
reviewed
May 7, 2025
There was a problem hiding this comment.
Thanks, @mbland
Note that this adds three new CI jobs. Of the three, the macOS version is dog slow for some reason. So if you want to only run a Linux job, Linux and Windows, or leave test_dependency_versions.sh out of the CI build altogether (like dt_patches/dt_patch_test.sh) let me know.
Let's keep all ci jobs for now.
|
For some reason github doesn't allow me to merge and suggest to try later. I will :) |
simuons
pushed a commit
to simuons/rules_scala
that referenced
this pull request
May 7, 2025
* Add test_dependency_versions, update test_runner
Adds `test_dependency_versions.sh` to validate the minimum Bazel and
dependency versions required by `rules_scala`, with and without the
precompiled protocol compiler toolchain. Extracts setup, run, and
teardown helpers from `test/shell/test_bzlmod_helpers.sh` into
`test_runner.sh` for reuse by `test_dependency_versions.sh`.
Also:
- Adds test files supporting `test_dependency_versions.sh` in
`deps/test`.
- Adds a mechanism to skip tests by prefixing their name with `_` to
`run_test_local` and `run_test_ci`.
- Adds the `RULES_SCALA_TEST_REGEX` environment variable to
`test_runner.sh`.
- Adds documentation for `RULES_SCALA_TEST_{ONLY,REGEX,VERBOSE}` to the
header comment of `test_runner.sh`.
- Adds `./test_dependency_versions` jobs to `.bazelci/presubmit.yml`.
---
Continuation of the previous change to ensure we don't force users to
upgrade their dependencies beyond the minimum versions supported by
`rules_scala`. Only builds using Bzlmod, as `WORKSPACE` is considered
legacy.
Inspired by a thread in the #bzlmod channel of the Bazel Slack workspace
on 2025-01-01 indicating that rules should require the minumum versions
possible:
- https://bazelbuild.slack.com/archives/C014RARENH0/p1743597941149639
* Fix test_dependency_versions.sh on macOS, Windows
Copied the setup for the Windows `test_rules_scala` job to the Windows
`test_dependency_versions` job. Replaced `cp "${test_files[@]}" .` with
copying the list of files directly instead of keeping them in an array.
The Linux job passed. The Windows job didn't run the script at all. The
macOS build broke in a way I've not seen while developing locally:
```txt
cp:
(/Users/buildkite/builds/bk-macos-intel-m3q2/bazel/rules-scala-scala/deps/test/*.bzl
/Users/buildkite/builds/bk-macos-intel-m3q2/bazel/rules-scala-scala/examples/testing/multi_frameworks_toolchain/example/*.scala
/Users/buildkite/builds/bk-macos-intel-m3q2/bazel/rules-scala-scala/test/jmh/data.txt
/Users/buildkite/builds/bk-macos-intel-m3q2/bazel/rules-scala-scala/test/proto/standalone.proto
/Users/buildkite/builds/bk-macos-intel-m3q2/bazel/rules-scala-scala/test/src/main/scala/scalarules/test/twitter_scrooge/thrift/thrift2/thrift3/Thrift3.thrift):
No such file or directory
```
* Fix ./test_dependency_versions.sh path on Windows
The script expects a directory path at the beginning of
`${BASH_SOURCE[0]}`, else the following error occurs:
```txt
test_dependency_versions.sh: line 18: cd: test_dependency_versions.sh:
Not a directory
```
This is because `${BASH_SOURCE[0]%/*}` will return the original
`${BASH_SOURCE[0]}` without a leading path.
This failure in the Linux `test_dependency_versions` job looks like a
fluke, since the previous run succeeded, as did other current runs:
```
WARNING: Download from
https://github.com/bazelbuild/bazel-skylib/releases/download/1.6.0/bazel-skylib-1.6.0.tar.gz
failed: class java.io.IOException GET returned 618 jwt:jwt-not-provided
``
* Disable //:ScalafmtTest.format-test on Windows
Works around the following `test_dependency_version.sh` failure:
```txt
FATAL:
ExecuteProgram(C:\tools\msys64\home\b\_bazel_b\...\ScalafmtTest.format-test)
failed: ERROR: src/main/native/windows/process.cc(202):
CreateProcessW("C:\tools\msys64\home\b\_bazel_b\...\ScalafmtTest.format-test"):
%1 is not a valid Win32 application.
(error: 193)
Test "test_precompiled_protoc_rules_java_7" failed (49 sec)
```
mbland
added a commit
to mbland/rules_scala
that referenced
this pull request
Aug 3, 2025
Removes almost all of the copied `deps/test` sources and targets in favor of invoking `@rules_scala` and `@multi_frameworks_toolchain` tests directly. Moves targets depending on `@rules_python` and `@rules_shell` to new packages so test packages don't break when `@rules_scala` isn't the main module. Introduces `RULES_SCALA_TARGETS`, `MULTI_FRAMEWORKS_TOOLCHAIN_TARGETS`, and `ALL_TARGETS` arrays to easily specify compatibility test targets instead of copying them. Only keeps a single `ScalafmtTest` target within `deps/test/BUILD.bazel.test`, which is a special case (described below) and executes successfully on Windows. Though these are new packages, they're comprised of existing files and targets from their parent packages: - //test/jmh/runtime - //test/sh_tests - //test/src/main/scala/scalarules/test/twitter_scrooge/strings Also: - Marks every `bazel_dep` for `@latest_dependencies` with `dev_dependency = True`. - Replaces `$(location ...)` instances in moved targets with `$(rootpath ...)` (and in one case, with `$(execpath ...)`) per bazelbuild/bazel#25198. --- While drafting a blog post describing `test_dependency_versions.sh`, I revisited the decision to copy targets and files in bazel-contrib#1729 and bazel-contrib#1738. Executing test targets from `@rules_scala` directly from the test module actually works, making the test far less complex, and making for better testing advice. This required moving targets that referenced dev dependencies to their own packages, fixing `@rules_scala` test package breakages. Making the `@latest_dependencies` module a dev dependency eliminated module version warnings from `@multi_frameworks_toolchain`, since the test sets older dependency versions. Then making `@latest_dependencies` a dev dependency everywhere, instead of only in `@multi_frameworks_toolchain`, seemed more consistent. The test retains the `//:ScalafmtTest` target because `rules_scala` doesn't actually provide Scalafmt rules, but only provides `ext_scalafmt` to create custom rules, per: - docs/phase_scalafmt.md - docs/customizable_phase.md I discovered this after going down a rabbit hole regarding the implicitly defined (and ostensibly deprecated) `.format` and `.format-test` predeclared outputs for Scalafmt rules. Long story short, implicitly defined predeclared outputs have been "deprecated" since 2018-03-05, predating Bazel 0.28.0: - https://bazel.build/versions/8.3.0/extending/rules#requesting_output_files - https://bazel.build/versions/8.3.0/extending/rules#deprecated_predeclared_outputs - https://bazel.build/versions/8.3.0/rules/lib/globals/bzl#rule.outputs - 2018-03-05: Output Map Madness https://docs.google.com/document/d/1ic9lJPn-0VqgKcqSbclVWwYDW2eiV-9k6ZUK_xE6H5E/edit - 2019-04-08: bazelbuild/bazel#7977: incompatible_no_rule_outputs_param: disable outputs param of rule function bazelbuild/bazel#7977 - 2019-06-07: Rollforward "Disable outputs param of rule function" with fix (introduced --incompatible_no_rule_outputs_param in Bazel 0.28.0) bazelbuild/bazel@36c70a6 - 2019-09-04: Document the replacements for the `outputs` parameter to the `rule()` function. (Bazel 1.0.0) bazelbuild/bazel@e29ddda However, I learned that the `.format` and `.format-test` targets are Bash scripts run _after_ the original rule executes Scalafmt. The generated scripts don't depend on the `//scala/scalafmt:scalafmt` binary at all. Plus, they only work when invoked within the main module, since they reference relative paths within `BUILD_WORKSPACE_DIRECTORY`. And finally, `bazel test @rules_scala//test/scalafmt:formatted-test` fails because it tries to declare output files in a nonexistent directory. ```txt ERROR: .../external/rules_scala~/test/scalafmt/BUILD:43:20: in scalafmt_scala_test rule @@rules_scala~//test/scalafmt:formatted-test: Traceback (most recent call last): File ".../external/rules_scala~/scala/private/rules/scala_test.bzl", line 38, column 22, in _scala_test_impl return run_phases( File ".../external/rules_scala~/scala/private/phases/api.bzl", line 45, column 23, in run_phases return _run_phases(ctx, builtin_customizable_phases, target = None) File ".../external/rules_scala~/scala/private/phases/api.bzl", line 77, column 32, in _run_phases new_provider = function(ctx, current_provider) File ".../external/rules_scala~/scala/private/phases/phase_scalafmt.bzl", line 10, column 46, in phase_scalafmt manifest, files, srcs = _build_format(ctx) File ".../external/rules_scala~/scala/private/phases/phase_scalafmt.bzl", line 30, column 44, in _build_format file = ctx.actions.declare_file("{}.fmt.output".format(src.short_path)) Error in declare_file: the output artifact 'external/rules_scala~/test/rules_scala~/test/scalafmt/formatted/formatted-test.scala.fmt.output' is not under package directory 'external/rules_scala~/test/scalafmt' for target '@@rules_scala~//test/scalafmt:formatted-test' ERROR: .../external/rules_scala~/test/scalafmt/BUILD:43:20: Analysis of target '@@rules_scala~//test/scalafmt:formatted-test' failed ``` So invoking `bazel test //:ScalafmtTest` (via `/...` from `ALL_TARGETS`) is sufficient, since we're only validating that the toolchains execute properly. (`test_scalafmt.sh` tests the behavior of the `.format` and `.format-test` scripts, _including_ on Windows via `--run_under=bash`.) This also means we can invoke this target on Windows, instead of having special case logic to avoid invoking the previous `bazel run` command.
simuons
pushed a commit
that referenced
this pull request
Aug 5, 2025
Removes almost all of the copied `deps/test` sources and targets in favor of invoking `@rules_scala` and `@multi_frameworks_toolchain` tests directly. Moves targets depending on `@rules_python` and `@rules_shell` to new packages so test packages don't break when `@rules_scala` isn't the main module. Introduces `RULES_SCALA_TARGETS`, `MULTI_FRAMEWORKS_TOOLCHAIN_TARGETS`, and `ALL_TARGETS` arrays to easily specify compatibility test targets instead of copying them. Only keeps a single `ScalafmtTest` target within `deps/test/BUILD.bazel.test`, which is a special case (described below) and executes successfully on Windows. Though these are new packages, they're comprised of existing files and targets from their parent packages: - //test/jmh/runtime - //test/sh_tests - //test/src/main/scala/scalarules/test/twitter_scrooge/strings Also: - Marks every `bazel_dep` for `@latest_dependencies` with `dev_dependency = True`. - Replaces `$(location ...)` instances in moved targets with `$(rootpath ...)` (and in one case, with `$(execpath ...)`) per bazelbuild/bazel#25198. --- While drafting a blog post describing `test_dependency_versions.sh`, I revisited the decision to copy targets and files in #1729 and #1738. Executing test targets from `@rules_scala` directly from the test module actually works, making the test far less complex, and making for better testing advice. This required moving targets that referenced dev dependencies to their own packages, fixing `@rules_scala` test package breakages. Making the `@latest_dependencies` module a dev dependency eliminated module version warnings from `@multi_frameworks_toolchain`, since the test sets older dependency versions. Then making `@latest_dependencies` a dev dependency everywhere, instead of only in `@multi_frameworks_toolchain`, seemed more consistent. The test retains the `//:ScalafmtTest` target because `rules_scala` doesn't actually provide Scalafmt rules, but only provides `ext_scalafmt` to create custom rules, per: - docs/phase_scalafmt.md - docs/customizable_phase.md I discovered this after going down a rabbit hole regarding the implicitly defined (and ostensibly deprecated) `.format` and `.format-test` predeclared outputs for Scalafmt rules. Long story short, implicitly defined predeclared outputs have been "deprecated" since 2018-03-05, predating Bazel 0.28.0: - https://bazel.build/versions/8.3.0/extending/rules#requesting_output_files - https://bazel.build/versions/8.3.0/extending/rules#deprecated_predeclared_outputs - https://bazel.build/versions/8.3.0/rules/lib/globals/bzl#rule.outputs - 2018-03-05: Output Map Madness https://docs.google.com/document/d/1ic9lJPn-0VqgKcqSbclVWwYDW2eiV-9k6ZUK_xE6H5E/edit - 2019-04-08: bazelbuild/bazel#7977: incompatible_no_rule_outputs_param: disable outputs param of rule function bazelbuild/bazel#7977 - 2019-06-07: Rollforward "Disable outputs param of rule function" with fix (introduced --incompatible_no_rule_outputs_param in Bazel 0.28.0) bazelbuild/bazel@36c70a6 - 2019-09-04: Document the replacements for the `outputs` parameter to the `rule()` function. (Bazel 1.0.0) bazelbuild/bazel@e29ddda However, I learned that the `.format` and `.format-test` targets are Bash scripts run _after_ the original rule executes Scalafmt. The generated scripts don't depend on the `//scala/scalafmt:scalafmt` binary at all. Plus, they only work when invoked within the main module, since they reference relative paths within `BUILD_WORKSPACE_DIRECTORY`. And finally, `bazel test @rules_scala//test/scalafmt:formatted-test` fails because it tries to declare output files in a nonexistent directory. ```txt ERROR: .../external/rules_scala~/test/scalafmt/BUILD:43:20: in scalafmt_scala_test rule @@rules_scala~//test/scalafmt:formatted-test: Traceback (most recent call last): File ".../external/rules_scala~/scala/private/rules/scala_test.bzl", line 38, column 22, in _scala_test_impl return run_phases( File ".../external/rules_scala~/scala/private/phases/api.bzl", line 45, column 23, in run_phases return _run_phases(ctx, builtin_customizable_phases, target = None) File ".../external/rules_scala~/scala/private/phases/api.bzl", line 77, column 32, in _run_phases new_provider = function(ctx, current_provider) File ".../external/rules_scala~/scala/private/phases/phase_scalafmt.bzl", line 10, column 46, in phase_scalafmt manifest, files, srcs = _build_format(ctx) File ".../external/rules_scala~/scala/private/phases/phase_scalafmt.bzl", line 30, column 44, in _build_format file = ctx.actions.declare_file("{}.fmt.output".format(src.short_path)) Error in declare_file: the output artifact 'external/rules_scala~/test/rules_scala~/test/scalafmt/formatted/formatted-test.scala.fmt.output' is not under package directory 'external/rules_scala~/test/scalafmt' for target '@@rules_scala~//test/scalafmt:formatted-test' ERROR: .../external/rules_scala~/test/scalafmt/BUILD:43:20: Analysis of target '@@rules_scala~//test/scalafmt:formatted-test' failed ``` So invoking `bazel test //:ScalafmtTest` (via `/...` from `ALL_TARGETS`) is sufficient, since we're only validating that the toolchains execute properly. (`test_scalafmt.sh` tests the behavior of the `.format` and `.format-test` scripts, _including_ on Windows via `--run_under=bash`.) This also means we can invoke this target on Windows, instead of having special case logic to avoid invoking the previous `bazel run` command.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Adds
test_dependency_versions.shto validate the minimum Bazel and dependency versions required byrules_scala, with and without the precompiled protocol compiler toolchain. Extracts setup, run, and teardown helpers fromtest/shell/test_bzlmod_helpers.shintotest_runner.shfor reuse bytest_dependency_versions.sh. Part of #1482.Also:
Adds test files supporting
test_dependency_versions.shindeps/test.Adds a mechanism to skip tests by prefixing their name with
_torun_test_localandrun_test_ci.Adds the
RULES_SCALA_TEST_REGEXenvironment variable totest_runner.sh.Adds documentation for
RULES_SCALA_TEST_{ONLY,REGEX,VERBOSE}to the header comment oftest_runner.sh.Adds
./test_dependency_versionsjobs to.bazelci/presubmit.yml.Motivation
Continuation of the previous change to ensure we don't force users to upgrade their dependencies beyond the minimum versions supported by
rules_scala. Only builds using Bzlmod, asWORKSPACEis considered legacy.Inspired by a thread in the #bzlmod channel of the Bazel Slack workspace on 2025-01-01 indicating that rules should require the minumum versions possible: