Epic: More validation at release time to make managing a Quarkiverse extension easier

[holly-cummins]

This is a draft; I need a place to write down various enhancements I/we want to do. The high-level goal is that releasing a 'correct' Quarkiverse extension should have as few manual steps as possible, and as few opportunities for error/missed steps as possible.

At the moment, it can take many attempts to do a successful release, and even when the release is done, publication may not be 'complete':

Common problems

• Authors forget to add extensions to the registry - see, for example, easy rag and jlama
• Authors do it wrong when they add extensions to quarkiverse docs - see Automatically alphabetise source file in an async job quarkiverse-docs#161
• Authors forget to sign up to the ecosystem CI
• Authors publish extensions with dead guide links
• Authors do not fill in the extension description, and so we show a parent description (see Some Quarkiverse extensions descriptions using parent description quarkusio/quarkusio.github.io#1993 and Maven extension plugin should not fill in description from parent pom quarkusio/quarkus#31634, or https://github.com/quarkusio/quarkus/pull/41471/files#diff-e979fd661e259c4bf161e3506834ed9699b80c60e16411c39897dbb610185c86R13 for a recent example of it happening). This isn't fatal, but it looks stupid.
• Other problems with extension metadata (although I'm hazy on what, and any fix here should probably be in the ExtensionDescriptorMojo and associated tooling)
• Releases take multiple attempts, because validations happen in the main release script, not the pre-release script (it took 11 attempts to do the first release the rockstar extension)
• Categories: https://quarkus.io/guides/extension-metadata says this for the category field: "Categories the extension should appear under on code.quarkus.io. This field can be omitted, the extension will still be listed on code.quarkus.io but won’t be categorized." Note we don't actually give people the list of acceptable values. We see that many extension authors choose a category which isn't one of the 'allowed' ones, because the allowed list is something they'd have to hunt for, and we don't validate it.
• Statuses: As with categories, we have an informal expectation about allowed statuses, but don't validate. See https://quarkusio.zulipchat.com/#narrow/stream/187038-dev/topic/draft.20status.20.3F/near/452366025 and https://quarkusio.zulipchat.com/#narrow/channel/187038-dev/topic/Extension.20status/near/526535105
• An extension sets an icon-url, but it doesn't point to an image. Ok, this isn't all that common, but it happened once and broke extensions site updates until we debugged it and coded in some mitigations.
• Making a quickstart is hard (see quarkus-cxf quickstart quarkus-cxf#399)

Possible solutions

• Create draft PRs to add an extension to the registry + docs catalog at the same time as creating the initial repo
• On release, automatically make a draft PR to add an extension to the registry if it's missing
• On releasing multi-extension repos, if some extensions are already in the registry, prompt to add others/automatically make a draft PR/warn/fail the CI
• Have the pre-release script just to call maven release plugin with a dry run flag; this would catch some problems, but not issues like using io.quarkiverse as the group ID
• Better documentation of allowed categories.
• Build-time validation of categories; if it's not an allowed one, suggest people choose one of the allowed ones (and show them the list), and also suggest they put the 'extra' stuff in as a keyword.
• Harmonisation of allowed categories. We should also add 'Testing' to the allowed categories, since there's six extensions using it. We should also make sure that code.quarkus and the build tooling share a common source of truth.
• The release (or build tooling) should check that if an extension sets an icon-url, it points to an image. Doing it at build time is maybe too heavy since then builds wouldn't work offline, and it wouldn't give people a window in which to sort out image publication.
• The pre-release (or build) tooling should enforce that an extension sets its own description
• Quarkiverse action to generate quickstarts (Create a Github Action to generate quickstarts for Quarkiverse extensions quarkusio/quarkus#24870)
• Better documentation for extensions, in general

Implementation mechanics for release scripts

• A custom action has the advantage people outside the quarkiverse could use it, and it's easier to get new code out to people ... it's also maybe a bit heavy for this use case, but Guillaume says it's very easy to do with it with https://github.com/quarkiverse/quarkus-github-action/.
• Alternatively, we enhance the scripts - and then possibly also push changes out to existing repos

*Related discussions: *

• General discussion of pre-release validation, plus advantages + disadvantages of GitHub actions: zulip
• Discussion of how to help extension owners realise they forgot to add themselves to the registry: zulip
• Discussion of category validation/non-validation: zulip

In general, we have to accept that external resources may not always be available, but when there's a clear mismatch between intent ("I think this url is live!" or "I think this url has an image on the other end of it") and reality ("uh, that's not an image, that's an html page"), then we're helping people by letting them know early. The aim here is to reduce friction and shorten feedback cycles.

[holly-cummins]

Related idea, but not part of this work item - automated checking against the Extension Maturity Model (eg having a dev ui)

[ppalaga]

Authors publish extensions with dead guide links

If you mean broken links in the given extension's Antora docs, then Quarkus Antora covers that need for local links at least. It has to be added as a dependency to the docs module and a simple test class has to be created. The Antora site served by Quarkus is then started before running the test. If there are broken local links, the app simply does not start.

I considered to implement also checking of external links some time ago and filed an issue for that right now. quarkiverse/quarkus-antora#23

[ppalaga]

Note we don't actually give people the list of acceptable values. We see that many extension authors choose a category which isn't one of the 'allowed' ones, because the allowed list is something they'd have to hunt for, and we don't validate it.

Where is that list?

[holly-cummins]

Authors publish extensions with dead guide links

If you mean broken links in the given extension's Antora docs, then Quarkus Antora covers that need for local links at least. It has to be added as a dependency to the docs module and a simple test class has to be created. The Antora site served by Quarkus is then started before running the test. If there are broken local links, the app simply does not start.

I considered to implement also checking of external links some time ago and filed an issue for that right now. quarkiverse/quarkus-antora#23

The problem I was referring to was dead links in the guide: element of the extension metadata itself (see, for example, https://quarkus.io/extensions/io.quarkus.qson/quarkus-qson/ - if you click on the 'Guide' link, you get a 404). We run a link checker and automatically detect those dead links when they hit the extensions page, but by then it's too late, because the bad data is already in a release. A tighter feedback cycle would be better.

So the thing I meant is is orthogonal to dead links in an extensions docs, but helping to avoid those is also a good thing.

[holly-cummins]

Note we don't actually give people the list of acceptable values. We see that many extension authors choose a category which isn't one of the 'allowed' ones, because the allowed list is something they'd have to hunt for, and we don't validate it.

Where is that list?

I believe it's https://github.com/quarkusio/quarkus/blob/main/devtools/bom-descriptor-json/src/main/resources/catalog-overrides.json.

[cescoffier]

@holly-cummins https://github.com/cescoffier/quarkus-extension-linter

[ppalaga]

Authors forget to sign up to the ecosystem CI

Is it this one for Quarkus CXF? quarkiverse/quarkiverse#14
We have our job that we maintain and look at. The ecosystem CI does not work for us. Could please Quarkus CXF get unlisted from ecosystem CI and could you please use our job for the overall reporting? Here are our jobs for main and 3.20