Clear Test Boundaries
Badly defined test boundaries can lead to tests that are too broad, too narrow, or too fragile. This can result in tests that are difficult to maintain, provide little value, or break easily. Determining the appropriate scope and size of a “unit” in unit testing can be challenging, as it significantly impacts the effectiveness and maintainability of tests.
Define the scope of a “unit under test” based on the responsibility and functionality it provides rather than strictly adhering to structural
boundaries such as layers or modules. Adjust the size of the unit according to the specific functionality being tested, ensuring that tests focus on
relevant components while minimizing dependencies on external elements.
In short: Based on the functionality you are looking to validate, adjust your test boundary to contain only the components that are
responsible for supplying said functionality.
The following factors support effective application of the practice:
The following factors prevent effective application of the practice:
In general, the goal of testing is to validate the system’s functionality and ensure that it behaves as expected. By defining test boundaries based on functional slices, we ensure the functionality offered to outside components is tested thoroughly.
Testing components based on their responsibility ensures that tests are meaningful and directly aligned with the system’s functional requirements. Most of the time, the system’s functionality is not strictly aligned with structural boundaries such as layers or modules. By defining the testing scope based on the functionality being tested, we ensure that tests are focused on asserting the behaviour of the system, rather than its implementation details. This approach reduces the likelihood of brittle tests and makes the test suite more maintainable and effective in catching issues. It also allows for more flexibility in refactoring and changing the system’s structure without breaking the tests.
The primary considerations relate to the potential difficulty in team adoption and the initial complexity of setting up functional slicing tests. While the practice is highly beneficial in the long run, it requires a commitment to understanding the system’s functionality and adapting testing strategies accordingly.
This case study explores the transition from a layered testing approach to a functional slicing approach in software testing. Initially, the system’s architecture followed a traditional layered model, but challenges in maintaining effective tests led to the adoption of a more dynamic, functionality-focused slicing approach.
The system under examination is an online retail platform. Initially, its architecture was divided into distinct layers:
Testing was initially structured around these layers. The team employed a mix of unit, integration, and end-to-end tests to validate the system’s functionality. They had unit test suites for each layer, integration tests to validate interactions between layers, and end-to-end tests that simulated user journeys through the system. However, several challenges arose with this approach:
In order to test their “User Registration” functionality, the team wrote tests at each implementation layer, from the presentation layer down to the database access layer. They implemented these tests by mocking out the lower layers and validating that the expected data was passed downwards. The code below shows and example of such a test, at the API-level:
class UserRegistrationAPITest implements WithAssertions {
@Mock private UserBusinessService userService;
@Mock private EmailValidationService emailValidationService;
private UserRegistrationAPI api;
@Before
public void setUp() {
api = new UserRegistrationAPI(userService, emailValidationService);
}
@Test
void whenUserSuppliesValidDetails_anAccountIsCreated() {
when(userService.createUser(any(CreateUserCommand.class))).thenReturn(fakeResponse);
when(emailValidationService.validate(anyString())).thenReturn(Validation.success());
api.registerUser(new User("John", "Doe", "someEmail@host.org", "JohnnyBoy69"));
verify(emailValidationService, times(1)).validate(anyString());
verify(userService, times(1)).createUser(any(CreateUserCommand.class));
}
@Test
void whenUserSuppliesInvalidEmail_noAccountIsCreated() {
when(emailValidationService.validate(anyString())).thenReturn(Validation.failed());
api.registerUser(new User("John", "Doe", "someEmail@host.org", "JohnnyBoy69"));
verify(emailValidationService, times(1)).validate(anyString());
verify(userService, never()).createUser(any(CreateUserCommand.class));
}
// More tests...
}
The team found that these tests were difficult to maintain and often broke when changes were made to the system. A prime example of this was the need to add a Captcha validation step to the registration process. This required changes to the API layer, which in turn broke most of the tests that were written at the API level. These tests needed to be updated to account for the new functionality, by adding additional stubbing, and making sure the correct methods were called. This process was time-consuming and error-prone, leading to a decrease in the team’s confidence in the system’s validity. In order to ensure a stable release, after updating the tests, the team spent weeks manually testing the user registration system to ensure that the new functionality was working as expected.
To address these issues, the team decided to transition to a functional slicing approach, focusing on vertical slices of functionality rather than horizontal layers. They redefined their test boundaries based on the system’s functional responsibilities, rather than its architectural layers, and ended up with the following test structure:
The team rewrote their tests to focus on the functionality of the system, rather than its implementation details. They rewrote the “User Registration” tests to cover the entire functionality, from the presentation layer down to the database access layer, within a single test.
class UserRegistrationTest implements WithAssertions {
@Mock private UserDatabaseProxy database;
private UserRegistrationAPI api;
@Before
public void setUp() {
api = new UserRegistrationAPI(new UserBusinessService(database));
}
@Test
void whenUserSuppliesValidDetails_anAccountIsCreated() {
var userToRegister = new User("John", "Doe", "someEmail@host.org");
var response = api.registerUser(userToRegister);
assertThat(response).isNotEmpty()
.extracting(RegistrationResponse::status)
.isEqualTo(RegistrationStatus.CREATED);
assertThat(api.detailsFor(response::userName))
.extracting(UserDetails::displayName)
.isEqualTo("JohnnyBoy69");
}
@Test
void whenUserSuppliesInvalidEmail_noAccountIsCreated() {
var userToRegister = new User("John", "Doe", "notAnEmail", "JohnnyBoy69");
var response = api.registerUser(userToRegister);
assertThat(response).isNotEmpty()
.extracting(RegistrationResponse::status)
.isEqualTo(RegistrationStatus.INVALID_EMAIL);
verify(database, never()).save(any(Account.class));
}
// More tests...
}
The transition from a layered approach to a functional slicing strategy significantly improved the testing process. By focusing on vertical slices of functionality, the team achieved more robust and maintainable tests, faster feedback loops, and a more resilient testing framework. This case study underscores the importance of adapting test strategies to align with system architecture and project needs, ensuring effective validation and continuous delivery of high-quality software.
