Some context first: I'm one person building an open-source research tool, and most of the code is written by AI coding agents (Claude Code and Codex) running in parallel. They write code quickly. What slowed everything down was the test suite: on one recent change the merge check selected 406 test files and took just over four hours, and the agents sat idle the whole time.
So I pulled together every timing the suite had already recorded (old JUnit XML and --durations output) and summed it per test. That covers about 9,000 test IDs seen across past runs, around 6,800 of which still exist.
243 of them, under 3%, took 77% of the summed time. Most of those build a whole workspace and run an end-to-end day of the app. Also, not a single test qualified as "small" in the Google sense (no disk, no network), because one autouse fixture asks for tmp_path on every test.
So I'm moving to Google's small/medium/large split (the 2010 Testing Blog post). Merge checks will run small and medium only, and the large ones go to release candidates and a nightly run. After that I want per-size timeouts with pytest-timeout, so a "medium" test that slowly grows into a large one fails instead of quietly eating everyone's time.
Two mistakes I made along the way:
The first labelling pass went by names. Anything with "golden" or "first_use" in it got marked large. When someone actually read the bodies, 28 of the 54 groups were fast in-memory checks. They'd have dropped out of merge runs entirely. Label by fixtures and what the test does, not the name.
The other one was worse. I select tests for a merge by following imports from the changed files. My script followed imports through the package but never through shared test helpers or conftest. For one change to a common helper it selected 6 test files; a selector that follows those edges selected 405. I'd been running a fast gate that was mostly blind to helper changes.
Next I'm trying pytest-testmon for coverage-based selection, running it next to the import-based one on real diffs before I trust it on its own.
Curious how others handle this. Do you split by size, or just throw more CI machines at it?