v3.1.2: New percent-encoding section (partial fix)

[handrews]

This is the starting point for fixing issue #4813. It has taken me over a week of research and multiple write-up attempts to distill this down and figure out how to give guidance that is both readable and sufficiently concrete.

What is not apparent from this PR is that the specs contradict each other- see #4813 for some of the details (although I did not put everything even there). I realized that if we talk about percent-decoding compliance, everything gets much easier. This make the perecent-encoding requirements a bit ambiguous, but that is inherent in the mess of conflicting requirements, and the way that different specs can add reserved behavior or scheme-specific rules.

The rest of the guidance in the spec won't need too much changing, but I'd like to see if this new section works for folks before tweaking all of the appendixes and examples to align with it. A bit of this is duplicated in Appendix E, but I'll take care of that when updating Appendix E once this is approved.

My goal here is to get the essential guidance up in the early part of the spec, and leave the really fiddly details in the appendix.

• schema changes are included in this pull request
• schema changes are needed for this pull request but not done yet
• no schema changes are needed for this pull request

[mikekistler]

@handrews I appreciate your diligence on this and I think everything that you have written is clear and (to the best of my knowledge) accurate and helpful to implementors of HTTP protocol libraries. But it feels a little out of place up at the top of the OpenAPI spec, which is not about describing how to correctly implement HTTP RFCs but how to describe HTTP APIs. I looked at the issue and the main complaint there (as best I understood it) is that some examples might not comply with HTTP RFCs. And I think the purpose of this section and maybe additional content in the appendices is to document our interpretation of the HTTP RFCs and support our contention that the examples do indeed conform, and perhaps to establish a "style guide" for examples going forward. This is all laudable. I'm just not sure we need it anywhere except in an appendix.

Apologies if I have misunderstood the purpose here ... just trying to help.

[handrews]

@mikekistler I'm not really sure where this belongs- perhaps in the Parameter Object although that is not the only Object the requires it. But it is the main one and the Encoding Object already references the Paramter Object so that would be OK.

which is not about describing how to correctly implement HTTP RFCs but how to describe HTTP APIs.

I think you are seriously mischaracterizing what I'm doing here.

If you think this jis just about "how to correctly implement HTTP RFCs" (almost none of this is about HTTP, btw), you are welcome to go read the RFCs and tell me how to correctly implement them, in the context of the OAS, given the normative citations that we already make. [EDIT: A more direct "why do we care?" answer can be found below] In order to do so, you need to understand the history and reconcile the discrepancies among:

• RFC1738 Uniform Resource Locators (URL) Dec. 1994, obsolete but cited by RFC1866
• RFC1866 Hypertext Markup Language - 2.0 Nov. 1995, obsolete but normatively cited by us
• RFC3986 Uniform Resource Identifier (URI): Generic Syntax Jan. 2005, obsoleted RFC1738 by way of RFC2396
• RFC6570 URI Template Mar. 2012
• WHATWG URL "Living Standard", obsoleted RFC1866 by way of W3C HTML 4.01; also claims to obsolete RFC3986 but excludes all non-browser use cases; is sometimes aligned with RFC3986, sometimes with RFC1738, and sometimes doing its own thing, but it is the actual current form-urlencoded spec we should cite in 3.2

Note that all of them contradict each other to some degree (except RFC6570 which is fully compatible with RFC3986- pick any other pair and they have contradictions), some of them paraphrase each other inaccurately, and some of them are actively hostile to each other to the point of putting petty snipes criticizing past specs in the newer specs. Plus there are many ambiguous requirements where things may or may not be allowed, or may or may not need encoding, plus certain frameworks out there do things that outright contradict some of these specifications and yet are essential to the ongoing functioning of large APIs.

I looked at the issue and the main complaint there (as best I understood it) is that some examples might not comply with HTTP RFCs

Yes, and what does compliance mean in this context?

Figuring out how to write this section is one of the most technically challenging tasks I have ever done for the OAS or JSON Schema. IIRC, I spent a solid month on Appendix E last year and still got parts of it wrong. I went through a whole bunch of attempts to write up something more clear in the past week or so. I think this was the sixth or seventh attempt. It was hard to figure out what guidance could be given without contradicting anything that we're citing normatively.

And now I will have to go through every example and make sure that it aligns with this clarified guidance. And edit Appendix E, although honestly not much of it will change AFAICT.

[handrews]

Oh, also, I wanted to write this in a way that we could replace RFC1866 with WHATWG URL in 3.2 without breaking compatibility. Despite all of the contradictions.

[handrews]

@mikekistler I've updated the URI-> API URLs clarification, but I'm still considering where to perhaps move this section.

[handrews]

@mikekistler I realize I forgot to directly answer the "why do we care?" aspect: Because we cited those contradictory specifications in ways that don't quite work. So it's not just that you have to apply them one at a time, which would be easy and we could just point to the RFCs. It's that you have to make sense of the problems caused by the way in which we combined different specifications in ways that they were not intended to work.

[handrews]

@mikekistler (with apologies for the flood of notifications) — I have moved the section under the Parameter Object.

[mikekistler]

Looks good. 👍