`cargo build` panics with include_bytes!() of the file itself

This code looks meaningless, but it's still valid code. However it can cause cargo to panic.

// This file is: main.rs
fn main() {
    let mut bytes = vec![];
    bytes.append(&mut include_bytes!("main.rs").to_vec());
    println!("Byte count: {}", bytes.len());
}

First build is ok. Then call shell touch main.rs and build again, cargo will crash with:

thread 'main' panicked at 'index out of bounds: the len is 0 but the index is 1',
/checkout/src/libcore/slice/mod.rs:2079:10

Version: cargo 1.28.0 (96a2c7d16 2018-07-13)

[Havvy]

[havvy@rust-nightly:~/scratchspace]$ cargo new --bin include_self
     Created binary (application) `include_self` project

[havvy@rust-nightly:~/scratchspace]$ cd include_self/

[havvy@rust-nightly:~/scratchspace/include_self]$ ls
Cargo.toml  src

[havvy@rust-nightly:~/scratchspace/include_self]$ sublime .

// Replace main.rs with the file in the opening comment.

[havvy@rust-nightly:~/scratchspace/include_self]$ cargo run
   Compiling include_self v0.1.0 (file:///home/havvy/scratchspace/include_self)                                                                                                                                      
    Finished dev [unoptimized + debuginfo] target(s) in 1.50s
     Running `target/debug/include_self`
Byte count: 170

[havvy@rust-nightly:~/scratchspace/include_self]$ cd src

[havvy@rust-nightly:~/scratchspace/include_self/src]$ touch main.rs

[havvy@rust-nightly:~/scratchspace/include_self/src]$ cargo run
   Compiling include_self v0.1.0 (file:///home/havvy/scratchspace/include_self)                                                                                                                                      
thread 'main' panicked at 'index out of bounds: the len is 0 but the index is 2', /checkout/src/libcore/slice/mod.rs:2046:10
note: Run with `RUST_BACKTRACE=1` for a backtrace.

error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports

note: rustc 1.30.0-nightly (9395f0af7 2018-09-02) running on x86_64-unknown-linux-gnu

note: compiler flags: -C debuginfo=2 -C incremental --crate-type bin

note: some of the compiler flags provided by cargo are hidden

error: Could not compile `include_self`.

To learn more, run the command again with --verbose.

[Havvy]

Backtrace:

stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
   1: std::sys_common::backtrace::print
   2: std::panicking::default_hook::{{closure}}
   3: std::panicking::default_hook
   4: rustc::util::common::panic_hook
   5: std::panicking::rust_panic_with_hook
   6: std::panicking::continue_panic_fmt
   7: rust_begin_unwind
   8: core::panicking::panic_fmt
   9: core::panicking::panic_bounds_check
  10: <rustc::ty::query::on_disk_cache::CacheDecoder<'a, 'tcx, 'x> as serialize::serialize::SpecializedDecoder<syntax_pos::span_encoding::Span>>::specialized_decode
  11: serialize::serialize::Decoder::read_struct
  12: serialize::serialize::Decoder::read_seq
  13: serialize::serialize::Decoder::read_struct
  14: serialize::serialize::Decoder::read_seq
  15: <rustc::mir::Mir<'tcx> as serialize::serialize::Decodable>::decode::{{closure}}
  16: rustc::ty::query::on_disk_cache::OnDiskCache::try_load_query_result
  17: <rustc::ty::query::queries::optimized_mir<'tcx> as rustc::ty::query::config::QueryDescription<'tcx>>::try_load_from_disk
  18: rustc::ty::query::plumbing::<impl rustc::ty::context::TyCtxt<'a, 'gcx, 'tcx>>::try_get_query
  19: rustc::ty::<impl rustc::ty::context::TyCtxt<'a, 'gcx, 'tcx>>::instance_mir
  20: rustc_mir::monomorphize::collector::collect_items_rec
  21: rustc_mir::monomorphize::collector::collect_crate_mono_items::{{closure}}
  22: rustc_mir::monomorphize::collector::collect_crate_mono_items
  23: rustc::util::common::time
  24: rustc_codegen_llvm::base::collect_and_partition_mono_items
  25: rustc::ty::context::tls::with_context
  26: rustc::ty::context::tls::with_related_context
  27: rustc::ty::query::plumbing::<impl rustc::ty::context::TyCtxt<'a, 'gcx, 'tcx>>::get_query
  28: <rustc_codegen_llvm::LlvmCodegenBackend as rustc_codegen_utils::codegen_backend::CodegenBackend>::codegen_crate
  29: rustc::util::common::time
  30: rustc_driver::driver::phase_4_codegen
  31: rustc_driver::driver::compile_input::{{closure}}
  32: rustc::ty::context::tls::enter_context
  33: <std::thread::local::LocalKey<T>>::with
  34: rustc::ty::context::TyCtxt::create_and_enter
  35: rustc_driver::driver::compile_input
  36: rustc_driver::run_compiler_with_pool
  37: <scoped_tls::ScopedKey<T>>::set
  38: rustc_driver::run_compiler
  39: <scoped_tls::ScopedKey<T>>::set
  40: <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
  41: __rust_maybe_catch_panic
  42: rustc_driver::run
  43: rustc_driver::main
  44: std::rt::lang_start::{{closure}}
  45: std::panicking::try::do_call
  46: __rust_maybe_catch_panic
  47: std::rt::lang_start_internal
  48: main
  49: __libc_start_main
  50: <unknown>
query stack during panic:
#0 [collect_and_partition_mono_items] collect_and_partition_mono_items
end of query stack

[matthew-russo]

So I've been debugging this quite a bit and have boiled it down to the (semi) root cause but don't really know how continue with it and I'd love some guidance or at least to follow closely with whoever takes the reigns.

This actually has has to do generally with checking contents of any source file being used and how files are cached. It is also reproducible as follows:

file src/main.rs

mod external;

fn main() {
    external::hello();
    let mut bytes = vec![];
    bytes.append(&mut include_bytes!("external.rs").to_vec());
    println!("Byte count: {}", bytes.len());
}

file src/external.rs

pub fn hello() {
    println!("hello");
}

cargo build once and its all good, touch src/main.rs and cargo build again and error happens.

Basically whats happening is that the source code for the files is getting put into the SourceMap then the include_bytes! macro expansion overwrites entry for the given file but with NO source content:

src/libsyntax/ext/source_util.rs, line 187 -- pub fn expand_include_bytes

// Add this input file to the code map to make it available as
// dependency information, but don't enter it's contents
cx.source_map().new_source_file(file.into(), String::new());

Then when it finds the file in the source map on subsequent compilations, expecting to find source code in it, it finds an empty file with no source, no lines, etc and panics.

I'm pretty new to working on the compiler and don't have a strong grasp on the query system or where a fix would fit into this. Not sure if its something where the expand_include_bytes function should check if the file is already in the source map and if so, use the contents of whatever is already there, or if it should be a change made elsewhere in the code. Any guidance, suggestions would be greatly appreciated.

[eddyb]

cc @michaelwoerister @nikomatsakis