The Java code includes unit tests, integration tests and system tests. We use JUnit 5, with AssertJ for assertions.
The Rust code includes unit tests and integration tests. We use the built-in test framework.
The Python code includes unit tests and integration tests with Pytest.
Our system tests are an acceptance test suite that includes end to end testing of the system, and tests for performance, scaling, throughput. See the system tests guide.
We have a manual testing setup that combines system test tools with a deployed instance of Sleeper, documented in the system tests guide.
We have a separate article on test design techniques that we try to apply uniformly across the codebase. These should be followed as part of the test strategy.
We'll establish definitions of the different types of test, and talk about when and how we use them.
A unit test is any test that runs entirely in-memory without any I/O operations (e.g. file system or network calls). If you configure your IDE to run all Sleeper's unit tests, they should finish in less than a minute. The unit of a test should be a particular behaviour or scenario, rather than e.g. a specific method.
A system test is a test that works with an instance of Sleeper deployed on AWS. These are JUnit tests found in the
module system-test/system-test-suite. They use the class SleeperSystemTest as the entry point to work with an
instance of Sleeper. This is the acceptance test suite we use to define releasability of the system. This is documented
in the system tests guide.
An integration test is any test which does not meet the definition of a unit test or a system test. Usually it uses external dependencies with Testcontainers, tests network calls with WireMock, or uses the local file system.
We always prefer to provide test coverage with in-memory unit tests. We use integration tests when it is not possible to test a behaviour with unit tests, or when faking or mocking I/O operations would either couple the tests too directly to the I/O technology, or would exclude all logic from the test.
All code should be covered with unit tests or integration tests, except for experimental features that were implemented without TDD. TDD is preferred in all cases, including for experimental features, and test coverage should be improved as soon as is practical.
We write system tests for all features that are not experimental and that we consider stable. We can also use them pre-emptively during a process of stabilisation, to find problems and monitor stability. This is implemented in JUnit. We keep the number of system tests per feature low, as these tests are relatively slow to run. They should verify that each feature is functional when deployed in AWS, and test properties such as performance and throughput.
When we add a new feature, we add some simple system test coverage to verify it still works when deployed. This is a complement to more detailed unit testing, at most one or two cases, or an adjustment to existing system tests. This should cover things like problems with permissions or networking configuration, that would only show up when deployed.
We consider performance testing when we want to provide guarantees about the scalability or performance of the system. We also use this to try to discover any differences in behaviour at scale, or bugs that only show up with a large amount of data or processes. This is included in our JUnit system test suite.
We try to avoid needing manual testing. If any behaviour in the system breaks, we want to know about it from one of the tests described above. Manual testing should usually be purely exploratory.
Unit tests should be in a class ending with Test, like MyFeatureTest. Integration tests should be in a class ending with IT, like MyFeatureIT. Classes named this way will be picked up by Maven's Surefire plugin for unit tests, and Failsafe for integration tests.
System tests should be in a class ending with ST, like CompactionPerformanceST, and must be tagged with the annotation
SystemTest. This means they will only be run as part of a system test suite, or directly. They may also be tagged
with an annotation for which test suite they should run in, which will default to the quick test suite. See
the system tests guide.
For assertions we use AssertJ instead of JUnit's built-in assertions. We also have some approval tests with ApprovalTests.Java.
For system tests we use a three tier design of tests, domain specific language (DSL), and drivers. The tests interact with the system through the DSL, and the DSL interacts with the system through drivers. The drivers are the only part that knows how to interact with a deployment of the system.
The drivers can be swapped out for an alternative implementation backed by in-memory test fakes or by a local alternative to AWS like LocalStack. That lets us work on the DSL or the drivers without needing to run against real instances of Sleeper.