emulated-tls ignored by clang and lld when using LTO

[Rokkubro]

Hello, I've just run into this issue when cross-compiling with LLVM 16.0.5(ubuntu-mvcrt) from the releases page; with stdlib set to libstdc++.
Without LTO enabled, passing -femulated-tls to clang generates the correct symbols in the object files. However, when LTO is enabled(-flto=full or -flto=thin), lld throws an undefined symbol error looking for _ZSt15__once_callable and friends instead of __imp____emutls_v._ZSt15__once_callable. I thought it might be related to issue 62077 in the llvm-project, but passing -Wl,-plugin-opt=-emulated-tls=1 to clang or -plugin-opt=-emulated-tls=1 to lld during linking still has the problem. Has this issue come up before? Am I missing something?

Here's the exact commands passed to clang if it helps.

When compiling one of the files that references std::__once_callable:

"/usr/bin/clang" -cc1 -triple x86_64-w64-windows-gnu -emit-llvm-bc -flto=thin -flto-unit -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name web_backend.cpp -mrelocation-model pic -pic-level 2 -mframe-pointer=none -fmath-errno -ffp-contract=on -fno-rounding-math -mconstructor-aliases -mms-bitfields -funwind-tables=2 -target-cpu x86-64 -tune-cpu generic -mllvm -treat-scalable-fixed-error-as-warning -debugger-tuning=gdb -v -fcoverage-compilation-dir=/__w/citra/citra/build -resource-dir /usr/lib/clang/16 -dependency-file src/web_service/CMakeFiles/web_service.dir/web_backend.cpp.obj.d -MT src/web_service/CMakeFiles/web_service.dir/web_backend.cpp.obj -sys-header-deps -isystem /__w/citra/citra/externals/boost -D BOOST_ALL_NO_LIB -D BOOST_ASIO_DISABLE_CONCEPTS -D BOOST_DATE_TIME_NO_LIB -D BOOST_ERROR_CODE_HEADER_ONLY -D BOOST_REGEX_NO_LIB -D BOOST_SYSTEM_NO_LIB -D CPPHTTPLIB_OPENSSL_SUPPORT -D CPP_JWT_USE_VENDORED_NLOHMANN_JSON -D "CRYPTOPP_DATA_DIR=\"/__w/citra/citra/externals/cryptopp\"" -D ENABLE_FFMPEG_VIDEO_DUMPER -D MINGW_HAS_SECURE_API -D NDEBUG -D _FILE_OFFSET_BITS=64 -I /__w/citra/citra/src/. -I /__w/citra/citra/externals/fmt/include -I /__w/citra/citra/externals/./microprofile -I /__w/citra/citra/externals/./json -I /__w/citra/citra/externals/./httplib -I /__w/citra/citra/externals/./cpp-jwt/include -D NDEBUG -D CPPHTTPLIB_OPENSSL_SUPPORT -internal-isystem /usr/x86_64-w64-mingw32/include/c++ -internal-isystem /usr/x86_64-w64-mingw32/include/c++/x86_64-w64-mingw32 -internal-isystem /usr/x86_64-w64-mingw32/include/c++/backward -internal-isystem /usr/x86_64-w64-mingw32/include/c++/12.2.0 -internal-isystem /usr/x86_64-w64-mingw32/include/c++/12.2.0/x86_64-w64-mingw32 -internal-isystem /usr/x86_64-w64-mingw32/include/c++/12.2.0/backward -internal-isystem /usr/include/c++/12.2.0 -internal-isystem /usr/include/c++/12.2.0/x86_64-w64-mingw32 -internal-isystem /usr/include/c++/12.2.0/backward -internal-isystem /usr/lib/gcc/x86_64-w64-mingw32/12.2.0/include/c++ -internal-isystem /usr/lib/gcc/x86_64-w64-mingw32/12.2.0/include/c++/x86_64-w64-mingw32 -internal-isystem /usr/lib/gcc/x86_64-w64-mingw32/12.2.0/include/c++/backward -internal-isystem /usr/lib/gcc/x86_64-w64-mingw32/12.2.0/include/g++-v12.2.0 -internal-isystem /usr/lib/gcc/x86_64-w64-mingw32/12.2.0/include/g++-v12.2.0/x86_64-w64-mingw32 -internal-isystem /usr/lib/gcc/x86_64-w64-mingw32/12.2.0/include/g++-v12.2.0/backward -internal-isystem /usr/lib/gcc/x86_64-w64-mingw32/12.2.0/include/g++-v12.2 -internal-isystem /usr/lib/gcc/x86_64-w64-mingw32/12.2.0/include/g++-v12.2/x86_64-w64-mingw32 -internal-isystem /usr/lib/gcc/x86_64-w64-mingw32/12.2.0/include/g++-v12.2/backward -internal-isystem /usr/lib/gcc/x86_64-w64-mingw32/12.2.0/include/g++-v12 -internal-isystem /usr/lib/gcc/x86_64-w64-mingw32/12.2.0/include/g++-v12/x86_64-w64-mingw32 -internal-isystem /usr/lib/gcc/x86_64-w64-mingw32/12.2.0/include/g++-v12/backward -internal-isystem /usr/lib/clang/16/include -internal-isystem /usr/x86_64-w64-mingw32/include -internal-isystem /usr/x86_64-w64-mingw32/usr/include -O3 -Wall -Wno-attributes -Winvalid-pch -std=gnu++20 -fdeprecated-macro -fdebug-compilation-dir=/__w/citra/citra/build -ferror-limit 19 -femulated-tls -pthread -fno-use-cxa-atexit -fgnuc-version=4.2.1 -fcxx-exceptions -fexceptions -exception-model=seh -vectorize-loops -vectorize-slp -include-pch /__w/citra/citra/build/src/web_service/CMakeFiles/web_service.dir/cmake_pch.hxx.pch -include /__w/citra/citra/build/src/web_service/CMakeFiles/web_service.dir/cmake_pch.hxx -faddrsig -o src/web_service/CMakeFiles/web_service.dir/web_backend.cpp.obj -x c++ /__w/citra/citra/src/web_service/web_backend.cpp

When linking:

"/usr/sbin/ld.lld" -m i386pep -Bdynamic -o bin/Release/citra-room.exe /usr/x86_64-w64-mingw32/lib/crt2.o /usr/lib/gcc/x86_64-w64-mingw32/12.2.0/crtbegin.o -L/usr/lib/gcc/x86_64-w64-mingw32/12.2.0 -L/usr/x86_64-w64-mingw32/lib -L/usr/x86_64-w64-mingw32/mingw/lib -L/usr/lib/clang/16/lib/windows -no-demangle -plugin-opt=-emulated-tls=1 src/dedicated_room/CMakeFiles/citra-room.dir/citra-room.cpp.obj src/dedicated_room/CMakeFiles/citra-room.dir/citra-room.rc.res --out-implib src/dedicated_room/libcitra-room.dll.a --major-image-version 0 --minor-image-version 0 src/common/libcitra_common.a src/network/libnetwork.a src/web_service/libweb_service.a externals/cryptopp-cmake/cryptopp/libcryptopp.a -lwinmm -lws2_32 -lpsapi -limm32 -lversion src/network/libnetwork.a src/web_service/libweb_service.a externals/enet/libenet.a -lwinmm -lws2_32 src/common/libcitra_common.a externals/fmt/libfmt.a externals/libboost_serialization.a externals/libboost_iostreams.a externals/zstd/build/cmake/lib/libzstd.a /usr/x86_64-w64-mingw32/lib/libssl.dll.a /usr/x86_64-w64-mingw32/lib/libcrypto.dll.a -lcrypt32 -lkernel32 -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 -lstdc++ -lmingw32 /usr/lib/clang/16/lib/windows/libclang_rt.builtins-x86_64.a -lunwind -lmoldname -lmingwex -lmsvcrt -lpthread -ladvapi32 -lshell32 -luser32 -lkernel32 -lmingw32 /usr/lib/clang/16/lib/windows/libclang_rt.builtins-x86_64.a -lunwind -lmoldname -lmingwex -lmsvcrt -lkernel32 /usr/lib/gcc/x86_64-w64-mingw32/12.2.0/crtend.o

[mstorsjo]

Just an acknowledgement, that I've seen your issue, but haven't had time to look into it.

You're definitely on the right track in any case - if enabling -femulated-tls in a setting where that's not the default, that also needs to be propagated into the LTO compilation. I'm not very familiar at all with how parameters are passed to that (you seem to know more about it than me). So I guess one would have to sit down with this and dig in and see where the parameter does get propagated and why that doesn't have the desired effect.

[lat9nq]

I made a comment saying a commit might have fixed it, but failed to notice the original Issue also mentions the same commit (llvm/llvm-project#62077) -- apologies on that.

I was interested in this Issue as I am running into the same error, but through using https://github.com/tpoechtrager/wclang which as far as I can tell is not directly related to this repository? All this to say I think this may still be an upstream problem.

[mstorsjo]

I had a quick look at this now, and the reason seems to be that LLD/mingw currently entirely ignores all the -plugin and -plugin-opt options (GCC passes a bunch of them).

The referenced commit llvm/llvm-project@a78816a does pass -plugin-opt=-emulated-tls=1 automatically if -femulated-tls is passed to Clang together with -flto when linking - however the MinGW driver doesn't call addLTOOptions so it doesn't get included in that case.

And even if it was passed, and when you're passing it manually, the LLD MinGW driver just ignores the option instead of doing something about it. I've got local patches now that hooks this up, but I'll need to test with GCC to make sure that LLD doesn't end up choking on options that GCC passes either.

In practice I noticed that the LLD MinGW driver does support handling similar options already, with the -Wl,-mllvm= syntax. So currently by passing -Wl,-mllvm=-emulated-tls I believe you should get the right result, even if this isn't the ideal way the option should be passed.

I'll look into cleaning up those patches for handling LTO options properly for the mingw case.

[mstorsjo]

FWIW, the in-progress diff against llvm-project that enables these things, looks like this:

diff --git a/clang/lib/Driver/ToolChains/MinGW.cpp b/clang/lib/Driver/ToolChains/MinGW.cpp
index b47041dcca70..4674e7fa268f 100644
--- a/clang/lib/Driver/ToolChains/MinGW.cpp
+++ b/clang/lib/Driver/ToolChains/MinGW.cpp
@@ -238,6 +238,12 @@ void tools::MinGW::Linker::ConstructJob(Compilation &C, const JobAction &JA,
 
   AddLinkerInputs(TC, Inputs, Args, CmdArgs, JA);
 
+  if (D.isUsingLTO()) {
+    assert(!Inputs.empty() && "Must have at least one input.");
+    addLTOOptions(TC, Args, CmdArgs, Output, Inputs[0],
+                  D.getLTOMode() == LTOK_Thin);
+  }
+
   if (C.getDriver().IsFlangMode()) {
     addFortranRuntimeLibraryPath(TC, Args, CmdArgs);
     addFortranRuntimeLibs(TC, CmdArgs);
diff --git a/lld/MinGW/Driver.cpp b/lld/MinGW/Driver.cpp
index a5b83d2f12ca..c378d666fe8d 100644
--- a/lld/MinGW/Driver.cpp
+++ b/lld/MinGW/Driver.cpp
@@ -413,6 +413,9 @@ bool link(ArrayRef<const char *> argsArr, llvm::raw_ostream &stdoutOS,
   for (auto *a : args.filtered(OPT_mllvm))
     add("-mllvm:" + StringRef(a->getValue()));
 
+  for (auto *a : args.filtered(OPT_plugin_opt_eq_minus))
+    add("-mllvm:-" + StringRef(a->getValue()));
+
   for (auto *a : args.filtered(OPT_Xlink))
     add(a->getValue());
 
diff --git a/lld/MinGW/Options.td b/lld/MinGW/Options.td
index c9c60638a4e5..a73432df602c 100644
--- a/lld/MinGW/Options.td
+++ b/lld/MinGW/Options.td
@@ -131,6 +131,9 @@ def version: F<"version">, HelpText<"Display the version number and exit">;
 defm wrap: Eq<"wrap", "Use wrapper functions for symbol">,
      MetaVarName<"<symbol>">;
 
+def plugin_opt_eq_minus: J<"plugin-opt=-">,
+  HelpText<"Specify an LLVM option for compatibility with LLVMgold.so">;
+
 // LLD specific options
 def _HASH_HASH_HASH : Flag<["-"], "###">,
     HelpText<"Print (but do not run) the commands to run for this compilation">;

[mstorsjo]

Fixes for this are pushed now, in llvm/llvm-project@a23bf17 and llvm/llvm-project@5c92c9f. They should show up in the next successful nightly builds at https://github.com/mstorsjo/llvm-mingw/releases/nightly.

[mstorsjo]

The new handling of LTO options caused broken builds whenever Clang passed any other LTO flags to the linker, but that should have been fixed by llvm/llvm-project@0b51e64 now as well (included in the latest nightly).