`cargo miri test` does not run doc-tests

[RalfJung]

I am not sure what it would take to also make it build and execute doc-tests. Probably just some tweaks in how we wrap cargo rustc?

[bjorn3]

Doc tests need rustdoc as driver, while miri needs itself as driver.

[SimonSapin]

@rust-lang/rustdoc, can rustdoc be used as a library (not a driver) to extract the doctests without running them?

[RalfJung]

Or, alternatively, can we set some environment variable to make rustdoc call miri instead of calling rustc and running the binary on each test after extraction?

[QuietMisdreavus]

@rust-lang/rustdoc, can rustdoc be used as a library (not a driver) to extract the doctests without running them?

Unfortunately, librustdoc was taken out of the sysroot a while back, and i don't think we auto-publish the source for that as a crate either, so there's not really an easy solution for that. Rustdoc itself doesn't really provide functionality to only save off the extracted/converted doctest, either.

Or, alternatively, can we set some environment variable to make rustdoc call miri instead of calling rustc and running the binary on each test after extraction?

Same response - unless the test is manually marked as ignore (or given some non-rust language string) in the source, rustdoc will always run the converted doctest through rustc via the librustc_interface interface. There's an open PR to provide an option to run the compiled binaries through a "test runner", but it doesn't sound like that will help either.

[RalfJung]

How open would the rustdoc team be to provide a way for Miri to run the extracted tests, and what would be the preferred way of achieving that (librustdoc vs controlling what happens after extraction)?

For me it seems like the second option is the simpler one.

[bjorn3]

The compilation of tests seems to be performed at https://github.com/rust-lang/rust/blob/17e62f77f954bed97aae839624bfd6dd68342daf/src/librustdoc/test.rs#L181.

[RalfJung]

Yeah, now I see what @QuietMisdreavus meant by librustc_interface -- it does not call the rustc binary, it calls the rustc library.

The "easiest" way to get Miri into this would be to provide a way to call some binary instead, but I am not sure if that is feasible.

[QuietMisdreavus]

If there were some flag we could provide to just save off all the doctests as standalone rust files, would that be enough to allow miri to attempt to build them? They're meant to be compiled as binaries and link against the regular crate and its dependency tree.

[RalfJung]

If there were some flag we could provide to just save off all the doctests as standalone rust files, would that be enough to allow miri to attempt to build them?

I guess we could make cargo miri test pick them up and run them all through Miri. The hard part would be to figure out the right command-line flags to get all the dependencies to work, but I am not sure if rustdoc can even help there?

[QuietMisdreavus]

Cargo hands rustdoc all the --extern flags it needs when running rustdoc --test from cargo test, so presumably that information would be present in the crate's metadata.

[RalfJung]

Hm, that raises the interesting question how cargo miri test would invoke rustdoc. Probably through cargo test? Can we do that targeting just the doctests of a particular binary?

Currently I wouldn't know how cargo miri test can get those flags; for the binaries we run we get them from cargo as well (cargo miri sets itself into the RUSTC env var so that cargo calls it with all the right flags and it just has to forward them).