Formally define 3.x URL templating behavior

[handrews]

As noted in Slack discussions, there is no formal specification for where variables can appear in server URL templates and path templates. There does not seem to be any reason to place restrictions on it.

Ideally, we could leverage the ABNF of RFC 6570 URI Templates even though OAS 3.x templating is not the same as RFC 6570 in general. (@darrelmiller @webron @MikeRalphson any concerns here?)

[handrews]

Issue #2119 brings up a lot of points relevant to this.

[char0n]

@handrews I created a implementation with formal ANBF grammar in https://github.com/char0n/openapi-path-templating.

I did a couple of assumption, because of the fact that the templating is not defined very precisely. Hope it helps a bit.

There does not seem to be any reason to place restrictions on it.

IMHO the restriction would be to now allow template expressions in URL query (searchParams) and URL fragment

[handrews]

@char0n thank you! I don't know if we're going to get to this in the current patch releases or not (I'm unassigning it for onw as I have too many things in my backlog), but this will help when we do get there!

[handrews]

@char0n I took a look at your ABNF- it looks like it restricts path template variables to whole segments. This issue got filed because in a discussion with TSC folks on slack, it was decided that template variables are not restricted to whole segments. You can have a template like /foo{bar}/whatever. I don't know how many API descriptions actually do that, but I figured you might want to know since the initial comment in this issue was not sufficiently clear.

[char0n]

@handrews I've created implementation with formal ABNF grammar for Server URL Templating in https://github.com/char0n/openapi-server-url-templating.

It's distinct from https://github.com/char0n/openapi-path-templating, as the Server URL Templating is based on RFC 6570, but Path Templating is more based on RFC 3986.

[char0n]

@char0n I took a look at your ABNF- it looks like it restricts path template variables to whole segments. This issue got filed because in a discussion with TSC folks on slack, it was decided that template variables are not restricted to whole segments. You can have a template like /foo{bar}/whatever. I don't know how many API descriptions actually do that, but I figured you might want to know since the initial comment in this issue was not sufficiently clear.

The grammar is not restricted to the whole segments:

path-segment                   = 1*( path-literal / template-expression )

path-segment is defined as non-terminal rule containing one or more path-literal or template-expression.

This path template /foo{bar}/whatever' is parsed into following AST:

[
  [ 'path-template', '/foo{bar}/whatever' ],
  [ 'path', '/foo{bar}/whatever' ],
  [ 'slash', '/' ],
  [ 'path-literal', 'foo' ],
  [ 'template-expression', '{bar}' ],
  [ 'template-expression-param-name', 'bar' ],
  [ 'slash', '/' ],
  [ 'path-literal', 'whatever' ]
]

NOTE: there is a test that guarantee this works as expected in https://github.com/char0n/openapi-path-templating/blob/bea2682e889a6bc06da95596b303dcc77304402f/test/parse.js#L45

[char0n]

@handrews,

There is one un-clarity related to template-expression-param-name rule from openapi-path-templating. OpenAPI spec doesn't technically constrain Parameter Object.name fixed field in any way. It just says it's a string. This means that people can use "parameter name" as a value.

This also means that the path template will need to be defined as /{parameter name}

It's just another example how OpenAPI Path Templating differs from RFC 6570.

---

@handrews taking into account the above, template-expression-param-name should be less constrained and in theory should allow any non empty string value. What do you think?

[LasneF]

The key point here is that RFC is defining a way to represent template
according to this specification it defines that template are a string , with basic rules about concatenation and one about size. no more.

in OAS leveraging parameter with a type and a format allows munch more complex definition but the ambigious pattern policy is not fully defined, especially this one /pets/{petId} vs /pets/{name} mentioned as invalid ; where there is nothing defined toward type of format

in the other hands some large framework has implemented that path templating can support either type , like in Python on Flask

• /Hello/int:post_id
• /hello/string:you

or in Java where you URI are always considered as a string BUT allowing regular expression for matching

• "/users/{id: [0-9]*}
• or even more complex by defining a "binder"

changing to the RFC 'like' meaning path as string with low format capability , could be seen as a breaking change , and would requires to a significant change in OAS spec to be precise (like not leveraging parameter object for instance)

using the full OAS capability as defines here need just more sample about ambiguous path
and setting the limit as
do we consider the json type only , or the type + the format (but that s more tricky about as for pattern can overlap)

notice that if /a/{b} vs /a/c is declared as non ambigious , looks on URI evaluation to me /{b}/a and /a/{b} should be same level taking the same principle but applying segment per segment from left to right

[handrews]

@LasneF currently, parameter type is not considered as part of request matching, so that should be a new issue (or, given its open-ended nature, probably better as a new enhancement discussion as we're kind of trying to make issues actionable things now).

This issue is just for defining what is already there.

[handrews]

@char0n I addressed this some in PR #3818 by adding the following section to the new Appendix C on RFC6570 equivalences. Since this example is a query string, I did not show the OpenAPI path template. But if it were in: path would it look like /whatever/{❤️}/etc or /whatever/{%E2%9D%A4%EF%B8%8F}/etc? 🤔

---

Illegal Variable Names as Parameter Names

In this example, the heart emoji is not legal in URI Template names (or URIs):

parameters:
- name: ❤️
  in: query
  schema:
    type: string

We can't just pass ❤️: love! to an RFC6570 implementation.
Instead, we have to pre-percent-encode the name (which is a six-octet UTF-8 sequence) in both the data and the URI Template:

"%E2%9D%A4%EF%B8%8F": love!

{?%E2%9D%A4%EF%B8%8F}

This will expand to the result:

?%E2%9D%A4%EF%B8%8F=love%21

---

[char0n]

@handrews,

So is it safe to assume that {name} part must abide to RFC6570 and is defined by following ABNF non-terminal?

name       =  varchar *( ["."] varchar )
varchar    =  ALPHA / DIGIT / "_" / pct-encoded

My current implementation assume (probably incorrectly):

name                   = unreserved / pct-encoded / sub-delims / ":" / "@"
unreserved          = ALPHA / DIGIT / "-" / "." / "_" / "~"
pct-encoded         = "%" HEXDIG HEXDIG
sub-delims          = "!" / "$" / "&" / "'" / "(" / ")"
                                / "*" / "+" / "," / ";" / "="

[LasneF]

@handrews

@LasneF currently, parameter type is not considered as part of request matching

here i do not want to open a pandora box , may be better to add a sample mentioning it

@char0n , i am not even sure that is safe , as i am not sure of the support of the complexity of such pattern by most of framework

{/list*,path:4} or
{+path,x}/here with sample like this /foo/bar,1024/here

i am wondering if this should not be limited to something simplistic (that still need to be defined)

[handrews]

@char0n I don't think there's anything safe to assume here. I have no idea what tools do in such cases, and we do not place syntactic restrictions on our URL Template variables. I think it would be equally valid (based on the current text) to say you can use whatever name you want, but before you pass it to an RFC6570 library you need to encode it. That's essentially how the new Appendix C says to treat in: query params.

[handrews]

@OAI/tsc review request: Does anyone have the bandwidth to sort this out for 3.0.4 and 3.1.1? If not, let's change the milestone to 3.2.0. It does not seem to be something that is causing problems in practice and I need to focus more on Moonwalk now and don't have the time to research this.