How Rust Tests Itself: Inside compiletest and the Rustc Test Suite
There’s a running joke inside the Rust community: “Rust doesn’t have users. It has testers.” Because every time you type cargo build, you’re benefiting from tens of thousands of tests that run before every Rust release — from parser edge cases to weird macro expansions to borrow checker nightmares no mortal would think of. But what powers that? How does a language like Rust, with a compiler so complex it can compile itself, actually test itself? The answer lies deep inside something called compiletest — the framework that keeps rustc honest. Today, we’re diving into one of Rust’s most underappreciated achievements: how it tests its own compiler. Why Rust Needs a Custom Test Framework When you’re testing normal software, you use frameworks like JUnit, pytest, or cargo test. When you’re testing a compiler, none of those will do. Why? Because compilers don’t just run code — they:
- Parse it,
- Optimize it,
- Emit warnings or errors,
- And sometimes crash spectacularly.
So Rust needed a way to test what happens when compilation fails — and verify that the error message, the exit code, and the generated output are all exactly what they should be. That’s where compiletest comes in. It’s the tool that tests the compiler’s behavior itself — not the user’s code. The Architecture: Where compiletest Fits Here’s a high-level diagram showing how compiletest integrates with the Rust compiler: ┌──────────────────────┐ │ rustc source code │ │ (parser, MIR, etc.) │ └─────────┬────────────┘
│
[build via x.py]
│
┌─────────▼───────────┐ │ compiletest driver │ │ - reads test files │ │ - invokes rustc │ │ - checks output │ └─────────┬───────────┘
│
[test outputs]
│
┌─────────▼────────────┐ │ compare with .stderr│ │ expected behavior │ └───────────────────────┘ In short: compiletest compiles a test file → captures the output → compares it to the “expected” output stored in the repo. If there’s a mismatch, the test fails. If everything matches, the compiler passes that test. How a Typical Rust Compiler Test Looks If you open the Rust repo (in src/test/ui/), you’ll find thousands of .rs files that look deceptively simple. Here’s one example: // compile-flags: --edition=2021 // check-pass
fn main() {
let x = 42;
let y = &x;
println!("{}", y);
} And another one: // compile-flags: --crate-type=lib // error-pattern: cannot move out of `*p` which is behind a shared reference
fn main() {
let p = &String::from("hello");
let _x = *p; //~ ERROR cannot move out of `*p` which is behind a shared reference
} These comments aren’t random — they’re directives that compiletest understands. Here’s what they mean: | Directive | Description | | ------------------- | -------------------------------------------- | | `// compile-flags:` | Pass extra flags to rustc | | `// check-pass` | Test should compile successfully | | `// error-pattern:` | Expect this error message in compiler output | | `//~ ERROR` | Inline marker for expected errors | This is how the compiler team encodes expected behavior inside the test file itself. How compiletest Runs Tests Under the hood, compiletest is a Rust program that orchestrates test execution. It’s written in the same repo, in src/tools/compiletest. Here’s the simplified code flow: fn main() {
let config = Config::parse_args(); run_tests(&config);
}
fn run_tests(config: &Config) {
let test_files = collect_tests(&config);
for file in test_files {
run_test(&config, &file);
}
} fn run_test(config: &Config, file: &Path) {
let output = invoke_rustc(file, config.flags()); let expected = read_expected_output(file); compare_outputs(&output, &expected);
} And that’s the essence of how it works:
- Collect all test files.
- Run rustc on each file.
- Capture output (stdout + stderr).
- Compare with expected output.
- Report differences.
The Types of Rust Tests Rust’s test suite isn’t monolithic — it’s made up of different kinds of tests: | Directive | Description | | ------------------- | -------------------------------------------- | | `// compile-flags:` | Pass extra flags to rustc | | `// check-pass` | Test should compile successfully | | `// error-pattern:` | Expect this error message in compiler output | | `//~ ERROR` | Inline marker for expected errors | Each directory is a test category with its own configuration — all handled by compiletest. Example: Testing a Borrow Checker Error Let’s look at one of the most important test categories: borrow checker tests. Here’s an example test (src/test/ui/borrowck/borrowck_move.rs): fn main() {
let s = String::from("hi");
let t = s;
println!("{}", s); //~ ERROR borrow of moved value: `s`
} When compiletest runs this file:
- It invokes rustc on it.
- Captures stderr.
- Checks if “borrow of moved value” appears in the output.
If the compiler forgets to emit that message, the test fails. That means a regression in the borrow checker got caught. This is why Rust’s safety guarantees are so reliable — every single behavior is locked in by tests. The Role of x.py and Test Orchestration You might’ve seen commands like this in Rust contributor docs: $ ./x.py test That command does a lot under the hood. It’s the main build & test orchestrator written in Python that:
- Builds the compiler.
- Runs compiletest for each suite.
- Summarizes all results.
Here’s a simplified view: x.py test
│ ├── build rustc ├── run compiletest (ui tests) ├── run compiletest (run-pass tests) ├── run tidy checks └── show summary
And yes — x.py is slow because it’s doing thousands of runs of rustc. But that’s the price of reliability at compiler scale. Diagram: The Testing Stack ┌──────────────────────────┐ │ Developer writes test.rs │ │ with directives │ └──────────┬───────────────┘
│
▼
┌──────────────────────────┐ │ compiletest driver │ │ - reads test file │ │ - calls rustc │ │ - captures stderr/stdout │ └──────────┬───────────────┘
│
▼
┌──────────────────────────┐ │ compares to expected .stderr │ │ or inline error markers │ └──────────┬───────────────┘
│
▼
┌──────────────────────────┐ │ test pass/fail report │ └──────────────────────────┘ compiletest in Code: Minimal Example Here’s a tiny toy version of compiletest you could build yourself: use std::process::Command; use std::fs;
fn main() {
let tests = fs::read_dir("tests").unwrap();
for test in tests {
let test_path = test.unwrap().path();
println!("Running {:?}", test_path);
let output = Command::new("rustc")
.arg("--emit=metadata")
.arg(&test_path)
.output()
.unwrap();
let stderr = String::from_utf8_lossy(&output.stderr);
if stderr.contains("error:") {
println!("❌ Test failed: {}", test_path.display());
} else {
println!("✅ Passed!");
}
}
} That’s essentially the idea behind compiletest, scaled up 100x with smarter parsing, directives, and comparison tools. Why This Matters Rust doesn’t just test if the compiler runs. It tests if the compiler behaves exactly the same across versions. That’s how it prevents regressions like:
- Borrow checker becoming too permissive.
- Error messages changing silently.
- Optimizations miscompiling user code.
These tests are what make Rust’s guarantees stick — they’re living documentation of the language’s semantics. The Emotional Core The compiler’s test suite is where Rust’s soul lives. Every tiny .rs file under src/test/ represents a real bug, a fixed soundness hole, or a subtle human mistake turned into a permanent safeguard. Every //~ ERROR comment is a developer’s whisper from years ago saying, “Let’s never make this mistake again.” When you cargo build your project today and it just works, that’s not luck. That’s thousands of these little promises being kept, line by line. Closing Thoughts Rust’s testing system isn’t flashy — there are no dashboards or CI animations to show it off. But it’s one of the main reasons why the language is trusted to run inside kernels, browsers, and rockets. compiletest doesn’t just test the compiler — it teaches it what correctness means. And as Rust grows — into async, into const evaluation, into GPU space — it’s this machinery that will quietly keep the language safe for everyone.