Statically linked nifs for mobile OTP targets

[adiibanez]

Hi

https://github.com/elixir-desktop/exqlite Works just fine as statically linked nif.
In the very same OTP-27.3 build a statically linked rustler lib fails to load resp. crashes whenever I call the functions.

I posted some infos about my issue and about the attempts at replicating what exqlite does: #686

The relevant parts about the static nif are somewhere here https://github.com/elixir-desktop/exqlite/blob/main/c_src/sqlite3_nif.c
but I couldn't find much official documentation about the procedure. Any insights would be very valuable.

I totally agree with @filmor that this is not the standard approach of loading a nif. But there aren't many options on mobile targets.

Any ideas / recommendations?

My activities are based on the elixir-desktop project, there are OTP-27 builds for macos, ios. Android is available too but currently disabled as I was focusing on ios. Can provide infos and example projects if anyone want's to take it for a spin.
The ci pipeline needs some cleanup but additional libraries can be included easily in the package.
I plan to integrate wasmex if possible. But my current focus is on ble resp. getting the basics up and running. And enabling rustler is kind of essential for that.
Having rustler nifs available would be really, really nice and enable some interesting usecases and optimizations.

[filmor]

Well, having now looked at the code it looks less involved than I'd have expected. Effectively, we have to

• generate the nif_init named <lib>_nif_init (easy enough, we can do this unconditionally and just generate the nif_init if we are compiling a dynamic library)
• potentially adjust the Windows code path to use the symbols from the running executable instead of the dyn-callbacks (I think both ways will work, but the C header opts for the former)

This is just for the Rust side of things. I don't have a good idea how to handle the use Rustler part. Simply skipping the compilation might be enough.

[filmor]

Could you please link me the actual build steps instead of the logs? If you do a normal cargo build or cargo rustc, the package name should be the top-level name of the package. The name it sees in your case is that of a dependency. If you want, you can adjust the file I edited to use some environment variable instead: https://github.com/rusterlium/rustler/pull/688/files#diff-77c1b552325c542ce68c2b866d4b9a0d6c9bbd5fa6d49ecc0ab74649c5c1a6a7R140

[adiibanez]

The current state of the workflow is a bit wild, sorry. But works reliably.
I'm in the process of cleaning up and splitting it into multiple files but haven't reached a stable state.

https://github.com/adiibanez/elixir-desktop-runtimes/blob/refactoring/.github/workflows/Matrix.yml

rust toolchain setup: https://github.com/adiibanez/elixir-desktop-runtimes/blob/refactoring/.github/workflows/Matrix.yml#L514
cargo build: https://github.com/adiibanez/elixir-desktop-runtimes/blob/refactoring/.github/workflows/Matrix.yml#L593

static nif lookup: https://github.com/adiibanez/elixir-desktop-runtimes/blob/refactoring/.github/workflows/Matrix.yml#L658

otp build step with nifs: https://github.com/adiibanez/elixir-desktop-runtimes/blob/refactoring/.github/workflows/Matrix.yml#L709

I'll have a look at your suggestion with an env variable

[filmor]

Just FYI, it works here locally (on Linux, though) with wasmex. The load_nif has to match as well, so the name we use for the <pkg>_nif_init should be the same as the filename we produce in the dynamic case.

[adiibanez]

That sounds great with wasmex! What OTP version did you try?

Did I understand correctly that I can use CARGO_PKG_NAME to define the libname?
I tried to implement that but it still uses libbtleplug_client-6a166d2feed33387.a
Where is that hash coming from? Is that something rustler generates? Or do I have another configuration problem. As far as I remember it has always been generated without any hashes.

[adiibanez]

I think I see the problem. I'm picking the wrong lib

./target/aarch64-apple-darwin/debug/deps/libbtleplug_client-95dc70775c4d89ee.a
./target/aarch64-apple-darwin/debug/libbtleplug_client.a

Haven't understood why both libs are created; they seem identical.
before both didn't have a hash. now one of them is build with a hash

[adiibanez]

I fixed the lib selection but still see an error

Undefined symbols for architecture arm64:
"_libbtleplug_client_nif_init", referenced from:
_erts_static_nif_tab in driver_tab.o
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[4]: *** [/Users/runner/work/elixir-desktop-runtimes/elixir-desktop-runtimes/_build/otp_builder/bin/aarch64-apple-darwin/beam.emu] Error 1
make[3]: *** [opt] Error 2
make[2]: *** [opt] Error 2
make[1]: *** [emu] Error 2
make: *** [emulator] Error 2

https://github.com/adiibanez/elixir-desktop-runtimes/actions/runs/13874490305/job/38825099507

[filmor]

Check the current version of the branch

[adiibanez]

static rustler nif running on ios. at last ... what a ride. Looking forward to make good use of rustler nifs!

@filmor Thanks a LOT, a LOT for your quick engagement in the topic. Really enabled some weeks of painful work to finally manifest in something "real".

[adiibanez]

@filmor I can build and link rustler_btleplug but not wasmex

I used the very same setup for both libraries, in particular rustler crate dependency on the static branch

https://github.com/adiibanez/rustler_btleplug/tree/static-no-precompiled
https://github.com/adiibanez/wasmex/tree/static-no-precompiled

also the cargo command is the same

CARGO_CMD="cargo rustc --crate-type staticlib --features rustler/staticlib --target=${{matrix.platform.rust_target}}"

/Applications/Xcode_15.4.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/15.0.0/lib/darwin/libclang_rt.osx.a
Undefined symbols for architecture arm64:
"_libwasmex_nif_init", referenced from:
_erts_static_nif_tab in driver_tab.o
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[4]: *** [/Users/runner/work/elixir-desktop-runtimes/elixir-desktop-runtimes/_build/otp_builder/bin/aarch64-apple-darwin/beam.emu] Error 1
make[3]: *** [opt] Error 2

with wasmex:
https://github.com/adiibanez/elixir-desktop-runtimes/actions/runs/13884036527/job/38846312789

without wasmex:
https://github.com/adiibanez/elixir-desktop-runtimes/actions/runs/13884102572/job/38846475824

[filmor]

Either use the previous state or wait a bit. I'm working on this. My plan going forward is that we drop the lib prefix everywhere.

Erlang's logic is quite simple: If you pass an object archive named some_obj.a to the enable-static-nifs parameter, it expects it to export a function some_obj_nif_init. Rust will always prepend lib for libraries built for non-Windows systems, which interferes with this.

My plan is to provide a new mix task rustler.compile_static that builds the static library and puts it into some well-defined directory (rustler_out or something like that) for your sub-sequent task to pick up.

[adiibanez]

I hit another road block regarding using more than one static rustler nif.
Not sure if rustler could do anything about. Sounds more like a general problem with rust staticlibs and the std-lib.
The builds worked fine for a few times ( didn't get to test the wasmex nif yet ). but since deleting the otp / openssl caches that might have been masking the issue I get following error:

ld: warning: object file (/Users/runner/work/elixir-desktop-runtimes/elixir-desktop-runtimes/_build/nifs/libwasmex.a54) was built for newer 'macOS' version (14.5) than being linked (14.0)
duplicate symbol '__ZN3std9panicking11EMPTY_PANIC17h38cb21ffaeece953E' in:
/Users/runner/work/elixir-desktop-runtimes/elixir-desktop-runtimes/_build/nifs/libwasmex.a2
/Users/runner/work/elixir-desktop-runtimes/elixir-desktop-runtimes/_build/nifs/libbtleplug_client.a2
duplicate symbol '_rust_eh_personality' in:
/Users/runner/work/elixir-desktop-runtimes/elixir-desktop-runtimes/_build/nifs/libwasmex.a2
/Users/runner/work/elixir-desktop-runtimes/elixir-desktop-runtimes/_build/nifs/libbtleplug_client.a2
ld: 2 duplicate symbols
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[4]: *** [/Users/runner/work/elixir-desktop-runtimes/elixir-desktop-runtimes/_build/otp_builder/bin/aarch64-apple-darwin/beam.emu] Error 1
make[3]: *** [opt] Error 2
make[2]: *** [opt] Error 2
make[1]: *** [emu] Error 2
make: *** [emulator] Error 2

Are you aware of any cargo build flags or other approaches that would allow 2 rustler nifs to be included in a otp build?
I tried quite a few options but no luck so far.
Was also thinking about using a temporary "workspace" crate with dependencies to multiple crates but that would most probably interfere with the OTP nif loading mechanism.

CARGO_CMD="cargo rustc --crate-type staticlib --features rustler/staticlib --release --target=${{matrix.platform.rust_target}} -- -C opt-level=z -C codegen-units=1 -v"

https://www.amyspark.me/blog/posts/2024/01/10/stripping-rust-libraries.html
rust-lang/rust#111594

[adiibanez]

I think I found the key difference that breaks the build process. It's the cargo --release flag.
Then it's guaranteed to lead to duplicate symbols during the OTP build step. That was a bit unexpected.

This is what I do in the build and it seems to work well.

CARGO_CMD="cargo rustc --crate-type staticlib --target=${{matrix.platform.rust_target}} --features rustler/staticlib -- -C opt-level=z -C codegen-units=1"

The first rust nif get's built with eg.
RUSTFLAGS="-C target-feature=-crt-static"
$CARGO_CMD

any "additional" rust nif:
RUSTFLAGS="-C target-feature=-crt-static -C link-arg=-nostdlib -C panic=abort"
$CARGO_CMD

[filmor]

Compiling with panic=abort could bite you hard. I think the only correct way to handle this is to make a common library that depends on both NIF crates, compile that as staticlib and ensure that the respective *_nif_init exist. OTP building has to be adjusted for that, though.

[filmor]

According to rust-lang/rust#44322, you can also try to disable lto (-C lto=false).

[adiibanez]

I'm takling some of these issues with "blissful ignorance". Basically trying to please the static lib linking step in OTP in whatever way required / possible. More than happy to avoid panic=abort ( didn't fully understand the consequences but it never sounded like a rust pattern to follow )

Would be more than interested in picking your brain about a feasible and robust "common library" approach. Thought about / hoped for the possibility but assumed that it would interfere with the lib per nif structure required by otp.

Could I dynamically generate a Cargo.toml that includes any relevant rustler nifs as dependency and only --enable-static-nifs="$STATIC_NIFS" that single common library?

I actually started using lto=fat and some other flags to reduce the size of the libraries from eg. wasmex 150MB to ~20MB. Not sure how -C lto=false would help avoiding the duplicate symbols issue?

TARGET=aarch64-apple-ios-sim
RUSTFLAGS="-C embed-bitcode=yes -C lto=fat"
cargo +nightly rustc --crate-type staticlib --release
--target=$TARGET
--
-C opt-level=z
-C strip=symbols
-C panic=abort
-C link-arg=-Wl,-dead_strip
-C link-arg=-Wl,--gc-sections

[filmor]

Check the linked thread, apparently LTO leads to exactly the issue you were observing. The size of the .a files is in the end irrelevant, you can strip/LTO/optimize the actual OTP build.

Regarding panic=abort: I would have actually hoped for the compiler shouting at you for this as rustler should enforce "proper" panics. If you panic!() in a NIF with the default settings, the panic will be caught at the "boundary" and translated into an Erlang exception. With panic=abort, the whole VM will crash without being able to clean up.

Regarding the static NIFs: There is a Perl script that generates a C table with the actual NIF init calls from the enable-static-nif parameter. If lto=false works, I think that's preferable.

[adiibanez]

@filmor lto=off works. I could also remove any panic=aborts and only use -C link-arg=-nostdlib to include more than one rustler lib.

using --release in the cargo command breaks OTP builds

CARGO_OPTIMIZATION_FLAGS="-C lto=off -C embed-bitcode=no -C opt-level=z -C codegen-units=1 -C strip=symbols -C link-arg=-Wl,-dead_strip -C link-arg=-Wl,--gc-sections"

before stripping:
271M Mar 20 13:00 _build/nifs/libwasmex.a
2.0M Mar 20 13:01 _build/nifs/sqlite3_nif.a
52M Mar 20 12:55 _build/nifs/libbtleplug_client.a

after stripping:

60M _build/nifs/libwasmex.a
2.1M _build/nifs/sqlite3_nif.a
16M _build/nifs/libbtleplug_client.a

thats approx 3x the size I saw for the rustler libs compared to using lto=fat

Would be interested in hearing how you would "you can strip/LTO/optimize the actual OTP build". I did some attempts with
-flto" in the xcomp config file but it breaks the packaging as xcframework with unable to find any architecture information in the binary at 'temp_lipo/liberlang_macos.a': Unknown header: 0xb17c0de

Regarding the perl script I'm not sure I understand what role that would cover.

[adiibanez]

the suggestion with sidestepping "double standard library" issues in the first place sounds great!

@filmor do you have an understanding why lto=fat/thin breaks the otp build?

there is quite a difference in library sizes eg. wasmex:

lto=off 73MB
lto=fat 30MB

[filmor]

Again, the library size is irrelevant. What counts is the final beam.jit or beam.smp size. Check the linked issue (rust-lang/rust#44322). This describes exactly the situation you have: Multiple statically linked Rust libraries (wasmex.a and ble_client.a), linked together into a common executable (beam.jit). And apparently it is not at all clear why this only shows up in lto'd builds.

[adiibanez]

Ok, I had another look at the link and found the infos describing the problem. sorry for bringing it up again.

Regarding library size I don't understand why you think it's irrelevant.
I observe a 1:1 correlation between input static lib size and the resulting beam size. Whatever goes in comes out in the OTP build step. eg. stripping that final file doesn't make any difference. Nor have I identified any ways in OTP to achieve similar results than lto on rust side.

Anyway, achieving reproducible OTP builds is already quite an improvement. Thanks for all the infos.
Looking forward to further improve the process.
As soon as I clean up the ci pipeline and split it up into smaller more manageable sub workflows I will start exploring other rustler libraries like explorer or maybe create some sample projects that can be compiled as quick as possible. Let me know if there is anything particular you want me to test. Very interested in combined rustler library approach but I might need some hints in the right direction and it also might be to early for that anyway.

[garazdawi]

Would you be open to a slight extension of this process that would allow us to pass something like --enable-static-nifs=/path/to/library_a.a:nif_mod1:nif_mod2,/path/to/library_b.a to "decouple" the linking and loading slightly? That way one could "pre-link" all Rustler-based components into a single .a or .lib, side-stepping the whole "double standard library" class of issues.

I don't think that should be any problem.

[filmor]

@adiibanez I created a PR for this: erlang/otp#9625

[filmor]

If erlang/otp#9625 (or something like it) is merged, maybe the best option becomes to

• Create a new crate with lib target staticlib
• Make the rustler::init! macro only generate the nif_init (without prefix, the one necessary for the usual dynamic library case) if it is the primary crate
• Generate the prefix for nif_init from the module name, exposing this generation with a new macro for use in the "bundling" lib

I'll test this approach.

[adiibanez]

@filmor Not trying to rush, but could it make sense if I would test some of those otp / primary crate features in parallel? In particular if there is any chance of reenabling lto=thin or lto=fat? Already have a slightly modified otp branch in place for some of the apple targets.

[filmor]

I'll look into this next week. The upstream changes have now been merged, so I'll go forward with the other two steps. The approach taken here should allow you to do as much LTO as you like as long as you bundle everything into a single library and then use the new --configure-static-nifs=$(realpath my_nif_bundle.a):wasmex:btleplug_client option to configure OTP.

[adiibanez]

Awesome! great news regarding OTP merge. Thanks for making this possible, very much appreciated. Looking forward to less "dodgy" no-std CI pipeline hacks resp. a solid approach to the problem.

[filmor]

I haven't tried it, yet, but the newest version of the static branch now follows this logic:

• It always generates a {crate_name}_nif_init
• It generates a nif_init (without prefix) only if the currently built package is a primary package (meaning it was selected as the package to compile)

You should now be able to achieve what you want by creating an (empty) library with a staticlib target that depends on all NIF libraries you are interested in and then building a current OTP from master (the changes don't seem to have made it into the OTP28 RCs so far) with the corresponding ./configure option.

[adiibanez]

Hi. Nice, thanks for looking into it and letting me know. I don't have the pipeline in place to build OTP master branch but will look into a way to test it. Looking forward to reenable lto optimization and see the difference in beam binary size.

[filmor]

This is now merged in rustler. Still lacks documentation, will follow up on that in the next weeks before cutting a release.