Automate creating issues for failing tests on main

[mx-psi]

Component(s)

No response

Describe the issue you're reporting

We have a bunch of tests that fail on main and are not getting enough attention. We should automate creating an issue (see e.g. here) on failing tests on main. Ideally it should contain:

• What test is failing
• On what commit it is failing
• (Optional) Ideally logs/a link to the logs?
• (Optional) A diff for skipping the test, or even a PR?

The workflow should check if an issue already exists (and leave a comment on the existing issue if not?)

Information about failing unit tests can be retrieved via the JSON output of gotestsum.

[mowies]

This could get pretty spammy. If there's good way to prevent that, I would be for something like that.

[mx-psi]

This could get pretty spammy. If there's good way to prevent that, I would be for something like that.

Is the spamminess about the total number of issues (i.e. the total number of tests failing)? Or about each issue getting a lot of comments?

Instead of comments we could just edit the issue body and add a new line for each commit in which a test fails (that would avoid some notifications).

One thing I like is that the spam is proportional to the number of things that are broken, so if somebody wants less spam, they know what to do 😅 but I agree we don't want too much spam

[mowies]

One thing I like is that the spam is proportional to the number of things that are broken, so if somebody wants less spam, they know what to do 😅 but I agree we don't want too much spam

That's one way to look at it, yeah 😄

No, I mean I just expressed my concern. There's definitely a good way of solving this without being too spammy.

[pratik-mahalle]

Is this issue still open?

[mx-psi]

@pratik-mahalle Yes! @ArthurSens was working on this (and, indirectly, @ankitpatel96 as well). You are welcome to talk to them if you want to help :)

[ArthurSens]

Yeah, I've been working on this before the end-of-the-year holidays but I still haven't caught up after coming back.

A WIP branch is here, but I haven't tested yet

[pratik-mahalle]

@ArthurSens Has issue resolved or should I start working on it?

[ArthurSens]

@ArthurSens Has issue resolved or should I start working on it?

It's not resolved yet! But I'm not sure how one could help here, do you want to continue the WIP branch? Were you thinking about starting with another strategy?