Often when developers say their builds are slow, they really mean their tests are slow or they believe their builds are fine, but their tests are slow. Either way test time is often the key driver of build times. There are many factors that contribute to test time including the growing number of integration and functional tests to cover a wide range of inputs, running tests sequentially, and dependencies on expensive external services.
Test Distribution is an approach that accelerates test execution and complements build caching while addressing many of the limitations of single-machine parallelism and CI fanout. And it gets around limitations in the number of parallel forks dictated by a single machine’s CPU and memory resources. It does this by extending test parallelism by fanning out test execution to remote executors.
The Develocity extension for the Maven & Gradle build tool coordinates partitioning of the test set and collection of results, while Develocity requests and caches test inputs, executes tests, and reports results back to the build tool for storage, all that with a low overhead that allows for fine-grained distribution.
Limitations of Existing Test Acceleration Techniques
- Build Caching
is great in many cases but doesn’t help when test inputs have changed.
- Single machine parallelism
is limited by that machine’s resources.
- CI fanout
does not help during local development, is inefficient (in particular on ephemeral CI agents or without build cache), requires manual setup and test partitioning, and causes
disjoint test reporting.