How to distribute `THREAD_MODEL=posix` builds?

[abrown]

When attempting to use wasi-sdk to compile a program that might use threads (or shared memories, etc.), the program will fail to link because the libc used has not been compiled with all of the right features (i.e., --shared-memory which relies on --features='atomics,bulk-memory'):

$ cat << EOF > test.c
#include <stdio.h>
int main() {
  printf("...");
  return 0;
}
EOF

$ /opt/wasi-sdk/wasi-sdk-16.0/bin/clang -O3 --target=wasm32-wasi \
    --sysroot=/opt/wasi-sdk/wasi-sdk-16.0/share/wasi-sysroot \
    -mthread-model posix -Xlinker --features='atomics,bulk-memory' -Xlinker --shared-memory \
    test.c -o test.wasm -v
...
 "/opt/wasi-sdk/wasi-sdk-16.0/bin/wasm-ld" -m wasm32 -L/opt/wasi-sdk/wasi-sdk-16.0/share/wasi-sysroot/lib/wasm32-wasi /opt/wasi-sdk/wasi-sdk-16.0/share/wasi-sysroot/lib/wasm32-wasi/crt1-command.o --features=atomics,bulk-memory --shared-memory /tmp/test-86de01.o -lc /opt/wasi-sdk/wasi-sdk-16.0/lib/clang/14.0.4/lib/wasi/libclang_rt.builtins-wasm32.a -o test.wasm
/opt/wasi-sdk/wasi-sdk-16.0/bin/wasm-ld: /lib64/libtinfo.so.6: no version information available (required by /opt/wasi-sdk/wasi-sdk-16.0/bin/wasm-ld)
wasm-ld: error: --shared-memory is disallowed by errno.o because it was not compiled with 'atomics' or 'bulk-memory' features.
clang-14: error: linker command failed with exit code 1 (use -v to see invocation)

There should be some way to package up the output of a make THREAD_MODEL=posix build in the released binaries and helpfully link in the right things when, e.g., -pthreads or -mthread-model=posix are used. Any thoughts on how to move forward on this?

[abrown]

cc: @mingqiusun, @penzn

[sbc100]

I would suggest that, at least initially you just build a completely separate sysroot. e.g. wasi-sysroot-mt?

In he long run we could come up with some kind of scheme where clang you select the library subdirectory within the sysroot differently, just like it does for architecture. e.g. -L<sysroot>/lib/wasm32-wasi/ would become something like -L<sysroot>/lib/wasm32-wasi/mt/. That why we could have single sysroot (and indeed a single architecture within that sysroot) that contains both a the mutlithreaded and singlethreaded libraries.

But I think two different sysroots should for a the time being.

[abrown]

Here's what we discussed today: wasi-libc would get distributed with a new subdirectory mt containing a whole new sysroot, the output of make THREAD_MODEL=posix. Then, we would need to change clang to append mt to the sysroot path when it detects flags that require a reentrant build (e.g., -pthread). @sunfishcode, @sbc100, does that sound right? (And maybe it's just a restatement of what @sbc100 just said above?)

[sbc100]

I think we discussed not a separate sysroot just a new directory inside the existing sysroot (e.g. lib/was32-unknown-unknown/mt)? This will require a clang patch.

The completely separate sysroot is what we could use to try this out without first landing the clang patch.

[abrown]

Oh, ok, then I was misunderstanding what was said. So the contents of mt would instead contain the threads-compatible versions of all of the lib artifacts (e.g., crt1.o, libc.a, libm.a, libwasi-emulated-*.a, etc.)?

[sbc100]

Basically yes, although crt1.o tends to be handled separately ... it might be special handling. Take a look a the current clang driver code for now it picks the lib directly along with how it picks crt1.o.

[penzn]

I think we discussed not a separate sysroot just a new directory inside the existing sysroot (e.g. lib/was32-unknown-unknown/mt)?

It can also be treated as a 'feature' of the target, and maybe triple can become wasm32-unknown-unknown-mt or wasm32-unknown-mt. The upside of that, in my opinion, is that it better reflects that 'mt' and 'non-mt' object files are more or less incompatible (mixing them leads to link errors, in all or some cases).

[sbc100]

Sure, if you think we can shoehorn it into the tipple that would great!

[sbc100]

That might be a bit tricky tough... what happens if you build with -triple=wasm32-unknown + -pthread .. should that fail because you didn't specify a triple that supports threading?

[sbc100]

I guess when the triple gets expanded from what the user specified on the command line and the missing components are added we could inject the -mt suffiix if the -pthread flag is present.

[penzn]

I guess when the triple gets expanded from what the user specified on the command line and the missing components are added we could inject the -mt suffiix if the -pthread flag is present.

Probably, I'll look at the driver and see what would be easier.

[alexcrichton]

One thing that may be worth pointing out here from a Rust-perspective is that Rust can probably only really support this if the string is a brand new target. Rust additionally has precompiled artifacts (the Rust standard library) and I don't think that infrastructurally it's really possible to change the Rust compiler and distribution infrastructure for just the wasm target to have two sysroots in the similar manner that's being proposed for C here.

Personally I have always hoped to avoid having a threaded and non-threaded target for all wasm targets but at the current point in time I think it's the best solution at least for Rust. I'm sure, though, that whatever naming scheme is used for a C target (if that's settled on) will want to be mirrored in Rust as well.

[sunfishcode]

To add to what Alex said here, adding a new triple ends up having a lot of administrative overhead.

I'd like to second the above ideas of having threading-enabled artifacts to be available in the same sysroot, under the same target triple, and teaching the clang and rust drivers how to find them, so that we don't need a whole separate target.

On a tangential note, I expect I'll have a hard time remembering the name "mt" stands for; could we name the artifacts something with "thread" in the name?