Add a rustdoc flag to show the source code externally

[pietroalbini]

One of the features of docs.rs is to show the source code in its UI, but rustdoc provides its own view of the source as well. This duplication is both confusing for the user, but also duplicates the amount of storage docs.rs uses to store source code.

It'd be nice to have a rustdoc flag (unstable is fine) like --source-code-base, that prevents rustdoc from emitting the source pages on its own and replaces the [src] links with links to the source code base. For example, with --source-code-base ../../crate/lazy_static/1.4.0/source, the [src] for lib.rs would be ../../crate/lazy_static/1.4.0/source/src/lib.rs, and rustdoc's own src page would be missing.

cc @GuillaumeGomez @jyn514

[Mark-Simulacrum]

Have we discussed doing the reverse? That is, pointing docs.rs generated sources into rustdoc [src] sources?

[pietroalbini]

Well, as with everything it's a tradeoff. While reducing the amount of code in docs.rs is tempting, I think the advantages of serving it from docs.rs are worth more. @rust-lang/docs-rs, what do you think?

Pros for serving the source code from rustdoc:

• Less code for the docs.rs team to maintain.
• The UX is the same as local docs.

Pros for serving the source code from docs.rs:

• Less storage space used, as docs.rs stores the raw source and renders it on the fly.
• We can update the interface without having to rebuild the world.
• We can guarantee the source code was not changed by a build script.

[Mark-Simulacrum]

If we're not storing rendered source code anyway then there's not really duplicate storage implications? (Just wasted storage from rustdoc, which we could plausibly redirect ourselves, without rustdoc changes?)

We could probably decompress crates.io tarballs on the fly and pull out the appropriate file we need.

[pietroalbini]

At the moment, for each source file, we're storing:

• rustdoc's html file with the pre-rendered highlighted source code
• the original source file (in s3://rust-docs-rs/sources/...), smaller as it doesn't have HTML in it

We should stop storing one of the twos.

We could probably decompress crates.io tarballs on the fly and pull out the appropriate file we need.

That could slow down the server a lot, as we have a few crates that go over 10MB.

[Mark-Simulacrum]

It seems like linking to rustdoc's pre-rendered sources is the best (and plausibly easiest) bet, personally.

That could slow down the server a lot, as we have a few crates that go over 10MB.

Yeah, that's true. More of a loose thought -- maybe e.g. it makes sense to do so for small crates.

[jyn514]

At the moment, for each source file, we're storing rustdoc's html file with the pre-rendered highlighted source code

It's actually even worse than this, we store a separate copy of rustdoc's source code for each platform. See e.g. https://docs.rs/regex/1.3.1/i686-pc-windows-msvc/src/regex/lib.rs.html and https://docs.rs/regex/1.3.1/i686-unknown-linux-gnu/src/regex/lib.rs.html.

[jyn514]

We can update the interface without having to rebuild the world.

This is a very convincing argument (to me) for not using the rustdoc sources. I think I've linked more to rust-lang/docs.rs#464 than to any other issue.

[jyn514]

Btw, I don't think it would be too hard to implement this on the docs.rs side without adding a new flag to rustdoc. We could just redirect /:crate/:version[/:platform]/src/:crate-name to /crate/:crate/:version/source/src/

[GuillaumeGomez]

Adding an option to rustdoc to link to a given location would be easy and seem to be what we want to do in here. That'd remove the duplicated files. When everyone agrees on a solution, I'll implement it.

Note for myself: if this option is used, no need to generate source code files.

[jyn514]

Adding an option to rustdoc to link to a given location would be easy.

I'd be happy with this, but it only works with future docs. What if we did both: rustdoc adds this option which links directly and then docs.rs still redirects past crates to the version of the source code we host? That would allow getting rid of a lot of storage costs for all those crates and also fix issues like rust-lang/docs.rs#484 and rust-lang/docs.rs#359