Adding tests has a high friction

[kornelski]

Contributing anything to Cargo needs tests, but writing tests for Cargo is not easy.

Unit tests are usually quick to write, cheap to run. Unfortunately, a lot of code in Cargo needs a GlobalContext, Workspace, etc. and I don't see any helpers that would create mocked or temporary ones, and doing so seems non-trivial to do from scratch for a single test.

This means that majority of things need to use integration tests in testsuite, but these tests are slow.

These tests have such a high overhead, that I feel bad for adding them, because testing small things like a new error message is just not worth the cost of spawning an entire cargo process. It also needs several files, and unreasonable amount of boilerplate.

e.g.

use cargo_test_support::compare::assert_ui;
use cargo_test_support::current_dir;
use cargo_test_support::file;
use cargo_test_support::prelude::*;
use cargo_test_support::str;
use cargo_test_support::Project;

#[cargo_test]
fn case() {
    cargo_test_support::registry::init();
    let project = Project::from_template(current_dir!().join("in"));
    let project_root = project.root();
    let cwd = project_root.join("primary");

    snapbox::cmd::Command::cargo_ui()
        .arg("add")
        .arg_line("cargo-list-test-fixture-dependency --path ../dependency --base my_base")
        .current_dir(&cwd)
        .masquerade_as_nightly_cargo(&["path-base"])
        .assert()
        .success()
        .stdout_eq(str![""])
        .stderr_eq(file!["stderr.term.svg"]);

    assert_ui().subset_matches(current_dir!().join("out"), &project_root);
}

use cargo_test_support::compare::assert_ui;
use cargo_test_support::current_dir;
use cargo_test_support::file;
use cargo_test_support::prelude::*;
use cargo_test_support::str;
use cargo_test_support::Project;

#[cargo_test]
fn case() {
    cargo_test_support::registry::init();
    let project = Project::from_template(current_dir!().join("in"));
    let project_root = project.root();
    let cwd = project_root.join("primary");

    snapbox::cmd::Command::cargo_ui()
        .arg("add")
        .arg_line("cargo-list-test-fixture-dependency --path ../dependency --base my_base")
        .current_dir(&cwd)
        .masquerade_as_nightly_cargo(&["path-base"])
        .assert()
        .success()
        .stdout_eq(str![""])
        .stderr_eq(file!["stderr.term.svg"]);

    assert_ui().subset_matches(current_dir!().join("out"), &project_root);
}

These differ by a single line! But each test requires several files itself, and some SVG!? I don't see any documentation for how to make the SVG required by these tests.

[epage]

Yes, we prioritize end-to-end tests over unit tests and you shouldn't feel bad about adding them.

Our UI tests have extra overhead but its much lower in our end-to-end tests.

• We do use snapshot testing so the tests can create the SVG for you.
• Would love it if we didn't need mod foo; to mount each test but auto mod is unclear if we'll ever have it
• filesystem-fixtures-by-example was meant to make it easier to create a test from a reproduction case and live inside the test but the reality of needing to use filesystem registries spoils this and I'm unsure about it
• filesystem-assertions-by examples was meant for easy comparison and snapshotting. We have more snapshotting now that we can re-evaluate this. Directory snapshotting might still be a weak area.

[kornelski]

Unfortunately, I don't have any decent Linux-compatible hardware, and Cargo's performance issues on macOS and Windows are heavily amplified by the test suite.

Launching of command-line executables has a high overhead on macOS and Windows.

On macOS, cargo's tests are heavily throttled by the XProtectService, Apple's built-in "anti-virus" subsystem, that frantically tries to check executables for malware and verify their code signatures (burning CPU on creating ad-hoc signatures for unsigned executables). Disabling this protection against Epic Games having Mac customers requires booting from a recovery OS image, which isn't stored on disk, but downloaded from the Internet every time it's used.

I can't even run Cargo's tests quicker in a Linux VM, because such VMs share their file system with the host via network/virtual file systems that are either slow and have race conditions that make stat and open disagree, which randomly breaks all sorts of things, or have buggy timestamps that make Cargo maddeningly infuriatingly unusably slow due to thinking libc and proc-macro2 are out of date 50% of the time.

Creation of files is also generally slow outside of Linux, and CACHEDIR.TAG is a Linux-only talisman.

On macOS, when you create files in any user-accessible directory, except ~/Library/Caches, within a second it triggers indexing of files for search. The mds daemon loads every file, consults a bunch of codecs and plugins that try to find any additional keywords in it, and then updates the global search index with the file's metadata and whatever the plugins hallucinate. It obsessively does this for all 80000+ files that Cargo writes to target/tmp/cit, hogging the CPU and disk I/O. Additionally, file system watching on macOS has only per-directory granularity, so whenever anything in .fingerprint or incremental directories is touched, they get rescanned entirely. The search index is designed for user saving a PDF to the desktop once in a while, not for cargo/rustc dumping thousands of files per second to a directory that the system thinks is just as important as files on the desktop.

Cargo's design breaks Apple's assumptions. Apple wants executables with code signing, using sqlite db instead of many files, and apps that only write to hidden directories specified by the OS. Writing to user-visible directories inside $HOME is only for the "Save As…" dialog.

[epage]

I believe we have people regularly developing Cargo on a Mac; I wonder if they have suggestions from how they do it that improves the experience.

Cargo's design breaks Apple's assumptions. Apple wants executables with code signing, using sqlite db instead of many files, and apps that only write to hidden directories specified by the OS. Writing to user-visible directories inside $HOME is only for the "Save As…" dialog.

We are working towards stabilizing build-dir (#14125) which should allow people to workaround this. The next steps for completely solving that are (1) changing the defaults for build-dir and (2) providing an override or changing the default for a cache home

We also are looking at re-organizing the target directory layout (#15010) which should reduce the number of files per directory.