Proposal: testing strategy using the Dart stable SDK

[elliette]

Follow up to dart-lang/sdk#53459

We currently run Webdev and DWDS tests on the main and dev channels of the Dart SDK, but not the stable channel. This is because feature development in DWDS/Webdev often requires never versions of the SDK than what's on the stable channel. However, this means that breakages when using the stable channel might not be caught (see dart-lang/sdk#53459).

We could add testing for our latest stable releases against the latest stable Dart SDK by:

1. Tagging the latest release of DWDS/Webdev as latest-dev and the latest release using a stable SDK constraint as latest-stable (Note: we could also use pre-release versions for versions depending on unstable Dart SDK versions)
2. Add a daily cron test that:

• makes sure we have a release version tagged with latest-stable
• makes sure that the latest Dart SDK stable version is allowed by the latest-stable SDK constraints
• makes sure that the latest Dart SDK stable version is not allowed by the latest-dev SDK constraints

3. Add daily testing that uses the latest Dart SDK stable version, and runs test against the latest-stable DWDS/Webdev versions

[jakemac53]

I don't think you need the latest-stable tags necessarily? I don't know that overwriting tags all the time like that would be a good idea (typically they are fixed).

You could have a job which runs on stable, and just does a pub global activate webdev? That would give you the latest version that anybody can actually get (on stable). It might not actually be the latest version of webdev that claims compatibility, if one of its deps is restricting it, but it does most accurately reflect what real users would get? And then you could run some integration tests using that webdev?

[elliette]

You could have a job which runs on stable, and just does a pub global activate webdev? That would give you the latest version that anybody can actually get (on stable). It might not actually be the latest version of webdev that claims compatibility, if one of its deps is restricting it, but it does most accurately reflect what real users would get? And then you could run some integration tests using that webdev?

That makes sense for testing that Webdev works as expected. The actual problem in this case was the Dart stable SDK was incompatible with the DWDS version Webdev was using. We have DWDS tests that would have caught the issue (see https://github.com/dart-lang/webdev/actions/runs/6149799798/job/16686493467?pr=2228) but not any Webdev tests. So ideally we could leverage our existing DWDS tests, instead of adding a Webdev integration test that would require interacting with DevTools to set a breakpoint, etc.

We could potentially use pub global activate to version solve for Webdev, and then use that to detect the DWDS version to run the tests against. Another option (less overhead than tags) is to use release branches (currently we release from the main branch). We could run daily stable tests against the release branches. Of course, this doesn't guarantee we are testing the DWDS version that's in the latest Webdev version. However, since we always release Webdev along with DWDS (with the latest DWDS pinned), this should catch any breakages.

[annagrin]

I think we can just have a cron job using latest stable SDK and pulling the webdev using pub global activate, and running the webdev e2e tests (they include some evaluation so i think the bug would have been detected in this scenario, of if not we can easily add a test for it). All we need to know if we have a working webdev for the latest sdk, am I right?

[elliette]

I think we can just have a cron job using latest stable SDK and pulling the webdev using pub global activate, and running the webdev e2e tests (they include some evaluation so i think the bug would have been detected in this scenario, of if not we can easily add a test for it). All we need to know if we have a working webdev for the latest sdk, am I right?

Ah interesting, we do have evaluation tests in webdev (https://github.com/dart-lang/webdev/blob/master/webdev/test/e2e_test.dart#L472). It doesn't look like they detected this particular regression: https://github.com/dart-lang/webdev/actions/runs/6149799798/job/16686493467?pr=2228

I think running the webdev e2e tests on a cron job would be a good first step. However, I also think this regression is a good reason to add more comprehensive stable testing for both our Webdev and DWDS releases

[elliette]

I think we can just have a cron job using latest stable SDK and pulling the webdev using pub global activate, and running the webdev e2e tests (they include some evaluation so i think the bug would have been detected in this scenario, of if not we can easily add a test for it). All we need to know if we have a working webdev for the latest sdk, am I right?

Ah, I've realized the problem with this approach. Because we are almost developing Webdev with an SDK constraint that includes the current stable (for example, right now Dart stable is at 3.1.3 but our SDK constraint is ^3.2.0-36.0.dev), we can't run any tests in Webdev if we are on Dart stable:

dart test [path to test]
Resolving dependencies in /Users/elliottbrooks/dev/webdev/webdev...
The current Dart SDK version is 3.1.3.

Because webdev requires SDK version ^3.2.0-36.0.dev, version solving failed.

[jakemac53]

Because we are almost developing Webdev with an SDK constraint that includes the current stable (for example, right now Dart stable is at 3.1.3 but our SDK constraint is ^3.2.0-36.0.dev), we can't run any tests in Webdev if we are on Dart stable:

You should be able to pub global activate webdev still though - the key is using whatever pub will give you and not the current version.

This would only really work for true integration tests (tests which are actually separate packages where you can just run any version of webdev from within that package, and not necessarily the local version).

[elliette]

This would only really work for true integration tests (tests which are actually separate packages where you can just run any version of webdev from within that package, and not necessarily the local version).

Unfortunately we don't have true integration tests in Webdev. I think I've figured out a different way to get Stable test coverage, I've opened up #2251.

It adds a daily Github workflow that will:

• Set up the Dart stable SDK
• Run dart pub global activate webdev to get the latest Webdev compatible with the stable SDK
• Get the activated Webdev version
• Check out the Webdev repo at the commit for that version tag
• Run all the Webdev tests at that commit

I think we should be able to expand it to add stable testing for DWDS as well (by parsing the DWDS version in the checked-out Webdev pubspec).