STRING i32 pairs for UTF-16LE

[dcodeIO]

Regarding

JavaScript hosts might additionally provide:

STRING | Converts the next two arguments from a pair of i32s to a utf8 string. It treats the first as an address in linear memory of the string bytes, and the second as a length.

Any chance that there'll be support for JS-style strings (UTF-16LE) as well? I know this doesn't really fit into the C/C++ world, but languages approaching things the other way around will most likely benefit when not having to convert back and forth on every host binding call.

[dcodeIO]

Since this hasn't received any comments yet, allow me to bump this: I am still curious if UTF-16 strings can be supported. In AssemblyScript's case all strings are UTF-16LE already, so only having the option to re-encode (potentially twice if the bound API wants UTF-16) does seem like it should be taken into account.

[fgmccabe]

This specific operator has not been discussed. In part that is because we are still at a more 'skeletal' level in the effort. However, although definitely possible, the bar for coercion operators is going to look something like this: a. Is the operator consistent with the requirements of the majority of hosts (i.e., mostly browsers at this point but with a definite leaning towards non-browser implementations)? b. Is the operator consistent with the majority use case for representing (in this case) string values? If most hosts would require copying unicode16 into urf8 anyway, you may have trouble with (a). But, essentially, it's a bit early to consider committing to any operators at the moment.

…

On Sun, Jun 30, 2019 at 1:56 AM Daniel Wirtz ***@***.***> wrote: Since this hasn't received any comments yet, allow me to bump this: I am still curious if UTF-16 strings can be supported. In AssemblyScript's case all strings are UTF-16LE already, so only having the option to re-encode (potentially twice if the bound API wants UTF-16) does seem like it should be taken into account. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#13>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAQAXUEQUF23TJ7APNWB5ODP5BYNVANCNFSM4EJGLDYA> .

-- Francis McCabe SWE

[dcodeIO]

If most hosts would require copying unicode16 into urf8 anyway, you may
have trouble with (a).

To me it looks like having such an operator can lead to significantly less work where UTF-16 is already present on both sides of the equation, while implementing the APIs in any case where either side is UTF-8 can easily be implemented by reencoding conditionally. Hence, the module would chose the operator that fits its internal string layout ideally, and the host would do whatever is necessary to make it fit into theirs. This leaves us with those cases:

1. Both UTF-8: Essentially memcpy
2. Both UTF-16: Essentially memcpy
3. One UTF-8, the other UTF-16: Reencoding once

while avoiding the very unfortunate case of

• Module UTF-16, host UTF-16: Reencode twice because UTF-8 is all the bindings understand

But, essentially, it's a bit early to consider committing to any operators
at the moment.

I see, yet I thought it might make sense to raise this early so once committing to operators, this case is well thought through :)

[jgravelle-google]

while avoiding the very unfortunate case of

• Module UTF-16, host UTF-16: Reencode twice because UTF-8 is all the bindings understand

Yeah that would be unfortunate.

Any chance that there'll be support for JS-style strings (UTF-16LE)

The real question is, can we skip JS entirely? At which point, what does the host API use internally? For example a similarly bad outcome would be

• Source UTF-8 -> JS UTF-16 -> Web API UTF-8

So I think the sanest way to handle that is with a declarative API on the bindings layer. Which is what you said earlier:

Hence, the module would chose the operator that fits its internal string layout ideally, and the host would do whatever is necessary to make it fit into theirs.

So the higher-level point is that we should be able to adequately describe the most common/reasonable ways to encode strings, so that we can minimize the number of encodings in the best case.

---

But, essentially, it's a bit early to consider committing to any operators
at the moment.

Agree + disagree. On the one, it's all in the sketch stage at the moment, where we're feeling out the rough edges. So from a managing-expectations point of view, this makes sense to say.

On the other hand, it's kind of incongruous to say "everything's up in the air, so don't raise any design issues." I don't think that was the intent, but that's kind of how it sounded. A more real translation of how I heard it was "don't worry about this now, we'll figure it out later." To which I would say, as a general principle, that yes we'll figure it out later, but we should raise it now to figure out if we should worry about it now. Especially because multiple people can think about different bits of the spec asynchronously.

[jgravelle-google]

Also something I should mention explicitly:

I find it incredibly likely that we will default to 1 binding expression per ⛄-type per wasm representation (e.g. 1 for linear memory and 1 for gc), which is to say the MVP of ⛄-bindings will have one binding expression per type, because gc will probably not be shipped yet. On that basis, we will probably start with only UTF-8-encoding (I imagine we will drop the utf8-cstr binding too, for similar reasons).

My general mental model here is that we can always add bindings in the future as we find a need for them. And it may be the case that in practice, the re-encoding from UTF-16 isn't enough of a bottleneck to be worth it. Unless it is, at which point we can add that binding, and it will be more obviously useful because we'll have much more real-world data.

---

Also for AssemblyScript specifically, would it be reasonable to change the internal string representation from UTF-16 to UTF-8 in the presence of ⛄-bindings? It is, after all, "Definitely not a TypeScript to WebAssembly compiler" 😄

[dcodeIO]

And it may be the case that in practice, the re-encoding from UTF-16 isn't enough of a bottleneck to be worth it. Unless it is, at which point we can add that binding, and it will be more obviously useful because we'll have much more real-world data.

At the end of the day we are building just tools here and one can't know the use case of everyone. Like, any use case extensively calling bound functions with string arguments would hit this and my expectation would be that this'll happen anyway (in certain use cases). Like, if we'd wait, this'll surface sooner or later, so it can as well be addressed from the start, instead of having to tell everyone running into this that their use case is currently not well-supported even through we did see it coming. Especially since specification and implementation of new operators can take a long time again.

Also for AssemblyScript specifically, would it be reasonable to change the internal string representation from UTF-16 to UTF-8 in the presence of ⛄️-bindings? It is, after all, "Definitely not a TypeScript to WebAssembly compiler" 😄

I'm sorry, the "⛄️-bindings" term is new to me. Would you point me into the right direction where I can learn about it? :)

Regarding UTF-8: In fact we have been thinking about this but it doesn't seem feasible, because we are re-implemeting String after the JS-API (with other stdlib components relying on it) and going with something else than UCS-2 representation seems suboptimal since the API is so deeply rooted into the language that mimicking UCS-2 would cost too much perf-wise. After all we are trying to stay as close to TS as reasonable to make picking up AssemblyScript a smooth experience. Also would like to note that this isn't exclusively an AssemblyScript thing, as other languages are using UTF-16LE as well, like everything in the .NET/Mono space.

[jgravelle-google]

Like, if we'd wait, this'll surface sooner or later, so it can as well be addressed from the start, instead of having to tell everyone running into this that their use case is currently not well-supported even through we did see it coming. Especially since specification and implementation of new operators can take a long time again.

It's ultimately a tradeoff. My thoughts here are that it will be strictly easier to spec and implement a bindings proposal that defines 8 operators, as opposed to one that defines 40. So we could just add UTF-16, but we could also just add C-strings and we could just add Scheme cons-list strings and we could just add Haskell lazy cons thunks, and so on. So for MVP I think we need to be really strict as to what exactly is "minimal", and in this context Minimal means "we can reason about strings at all".

We also need to balance the "viable" portion. Originally I was thinking we should avoid reasoning about strings and allocators at all, due to the complexity they add. Further discussion on this (see: #25) made me realize that not having an answer for allocators would compromise the viability of the proposal entirely. On that basis, not having UTF-16 support from day 1 is unlikely to leave the bindings proposal dead in the water.

By means of analogy, I would rather we ship anyref without waiting for the full gc proposal, because anyref on its own is a very enabling feature. It is in many ways suboptimal, but it it is more useful than what we had before. On that basis, I want to be very cautious about adding scope to the bindings MVP, especially when that scope is separable to a v2 that describes an expanded set of binding expressions.

I'm sorry, the "snowman-bindings" term is new to me. Would you point me into the right direction where I can learn about it? :)

Sure, @lukewagner presented at the June CG meeting, and here's the slide deck: https://docs.google.com/presentation/d/1wtAknL-UJWDoIgSbyF5paTBSpVVj-fKU4tiHMxJbSzE/edit

tl;dr does this wasm binding layer we're describing need to reason about WebIDL at its core, or is WebIDL another target with a produce/consume pair? If the latter, and we suspect that is the case, then we're free to design an IDL that better matches what we're trying to do, rather than try to retrofit that on top of WebIDL.

Full notes of the accompanying discussion here: https://github.com/WebAssembly/meetings/blob/master/2019/CG-06.md#webidl-bindings-1-2-hrs

Also would like to note that this isn't exclusively an AssemblyScript thing

Didn't mean to sound like I was saying it was :x, sorry. I was thinking that if AssemblyScript was using UTF-16 for easier FFI with JS, then in the presence of something-bindings it would be possible to decouple that ABI. And also that AssemblyScript would probably have an easier time of making that ABI switch than a more-ossified target like .NET, on account of it's a younger platform.

[dcodeIO]

My thoughts here are that it will be strictly easier to spec and implement a bindings proposal that defines 8 operators, as opposed to one that defines 40

Makes sense, yeah. Though, to me it seems not overly complex to have a (potentially extensible) immediate operand on str (/ alloc-str) that indicates a well-known encoding. I'd consider UTF-8, UTF-16LE and maybe ASCII here (not sure), with length always provided by the caller (even if null-terminated), but I'm certainly not an expert in this regard.

By means of analogy, I would rather we ship anyref without waiting for the full gc proposal, because anyref on its own is a very enabling feature. It is in many ways suboptimal, but it it is more useful than what we had before.

I totally agree with the anyref mention, but don't entirely agree on the comparison with encodings. anyref is a useful feature on its own with everything else building upon it, while not addressing encoding challenges on introduction of the feature that would need to deal with it leads to half a feature that unnecessarily limits what certain ecosystems with (imo) perfectly legit use case scenarios like UTF-16 can do efficiently.

Sure, @lukewagner presented at the June CG meeting, and here's the slide deck: https://docs.google.com/presentation/d/1wtAknL-UJWDoIgSbyF5paTBSpVVj-fKU4tiHMxJbSzE/edit

Thanks! :)

So, looking at this slide it mentions utf8 exclusively similar to what we have with WebIDL. Not quite sure how it would solve the underlying issue, that is making a compatible string from raw bytes, if it moves the problem from "directly allocating a string compatible with WebIDL bindings" to "creating a DOMString/anyref compatible with ⛄️-bindings" (if I understood this correctly?). For instance, TextEncoder doesn't support UTF-16LE (anymore), but TextDecoder does.

I'd expect that at some point in either implementation "making a compatible string from raw bytes" will be necessary anyway if the primary string implementation is provided by the module, which is likely. Please correct me if I'm missing something here. Ultimately, the issue doesn't have to be solved in the WebIDL spec, but any other spec solving it would be perfectly fine as well - as long as it is solved.

Didn't mean to sound like I was saying it was :x, sorry. I was thinking that if AssemblyScript was using UTF-16 for easier FFI with JS, then in the presence of something-bindings it would be possible to decouple that ABI. And also that AssemblyScript would probably have an easier time of making that ABI switch than a more-ossified target like .NET, on account of it's a younger platform.

All good, your point makes perfect sense. Just wanted to emphasize that, even if AssemblyScript would make this change, this is a broader problem than what it might look like from this issue alone :)

[MaxGraey]

I thought if WebAssembly implement WebIDL binding it should follow WebIDL spec which support three types of strings: DOMString, ByteString and USVString. Most of WebIDL which relate to WebApi mostly using DOMString which commonly interpreted as UTF-16 encoded strings [RFC2781]. ByteString is actually ASCII and at last USVString which not require concrete encoding format. Addisianal note about USVString from WebIDL spec:

Specifications should only use USVString for APIs that perform text processing and need a string of Unicode scalar values to operate on. Most APIs that use strings should instead be using DOMString, which does not make any interpretations of the code units in the string. When in doubt, use DOMString.

[Pauan]

@dcodeIO I'd consider UTF-8, UTF-16LE and maybe ASCII here (not sure)

UTF-8 was intentionally designed as a strict super-set of ASCII, therefore UTF-8 can be used to efficiently transfer ASCII text.

[dcodeIO]

UTF-8 was intentionally designed as a strict super-set of ASCII, therefore UTF-8 can be used to efficiently transfer ASCII text.

Yeah, tried to be careful there (in regards to C-strings) but the more I think about it the less I believe that this distinction is necessary, especially since any API being bound will very likely be reasonably modern anyway. So that'd leave us with UTF-8 and UTF-16LE. Anything else you could imagine would fit there in terms of "well-known encodings" (in context of modern programming languages)?

[MaxGraey]

@Pauan WebIDL (except ByteString) and Javascript not using ASCII at all. Strings in javascript represented as UTF-16LE by default but v8 for example can represent strings in different ways and encodings internally. For example during concatenation strings can represent as rope structure which flattened to "normal" string before serialization / father conversion or before passing to Web Api. But that doesn't mean we should use rope structure as default structure for string for example. The same with UTF8

[dcodeIO]

Side note: USVString looks like it can be described in terms of UTF-32 (not sure if that makes sense as I don't know anything using it for its internal representation). But maybe the least common denominator is UTF here?