Software developers should care at least as much about the quality of their test code as of their productive code. Ever been handed a project, told to make some changes, seen tests break, and pull your hair out in frustration trying to understand what's going on? Often the tests don't communicate their intention well because they were just written to provide a safety net and nothing more. They capture the existing behaviour of the application and help to keep it constant, but don't allow for changing requirements. Good developers should do better. When writing tests, think about the poor soul who will be staring at your code in 5 years after you are gone. You've already been there -- don't make their life hell.
In this post I'm going to focus on one aspect of clean test code: the assertions. They serve more than the obvious purpose of determining whether the test passes or fails. Good assertions also document the meaning of the test, communicate the nature of the failure, and help to diagnose the reason for a failure. Good assertions share three characteristics in particular:
- They are written in a way which communicates the intention.
- They are focused, testing only what is relevant to the test at hand.
- The message which is generated on a failure carries all relevant information.
The last point in particular deserves some further explanation. What features should the failure message have?
- It should show both the actual and expected behaviours so that the reader can compare them.
- It should show sufficient context to diagnose the problem. Supposing, for example, that an object has five fields containing strings and one of them is wrong, the message should identify which one was at fault.
- It should be relevant to the test. For example, if an object returned from the system under test does not have the desired property, the message should not be that some exception was thrown during execution of the test.
Let's consider an example: we are writing a unit test for code which maps between two different representations of a collection of customers with addresses, say for translation between two external interfaces. The two address classes have a similar structure:
class Address {
private String houseNumber;
private String street;
private String postalCode;
private String city;
private String country;
private String phone;
// Getters for the above fields
}
class Customer {
private String name;
private Address address;
private CustomerType type;
}
enum CustomerType {
PRIVATE_CUSTOMER, BUSINESS_CUSTOMER, UNKNOWN
}
class ExternalCustomer {
private ExternalAddress address;
private ExternalCustomerType type;
}
class ExternalAddress {
private String name;
private String addressLine1;
private String addressLine2;
private String zip;
private String city;
private String state;
private String country;
private String phone;
// Getters and setters for the above fields
}
enum ExternalCustomerType {
PRIV, BUS, UNKNOWN
}
Suppose the method under test converts a collection of customers of the first representation into the second:
List<ExternalCustomer> convertCustomers(List<Customer> customers)
How do we test this? Let's look at some possibilities and see what patterns and antipatterns we find.
Asserts on single fields
In the first example, we see a lot of asserts which involve reaching into the output, pulling out individual fields, and comparing them with expected values.
@Test
public void convertCustomersShouldConvertListOfOneCustomer() {
// Create list with one customer and some relevant data
List result =
converter.convertCustomers(customers);
assertThat(result.get(0).getType(),
is(ExternalCustomerType.PRIV));
assertThat(result.get(0).getAddress().getName(),
is("Max Mustermann"));
assertThat(result.get(0).getAddress().getAddressLine1(),
is("Hauptstraße 62"));
assertThat(result.get(0).getAddress().getAddressLine2(),
is(""));
assertThat(result.get(0).getAddress().getZip(), is("12345");
assertThat(result.get(0).getAddress().getCountry(), is("DE");
assertThat(result.get(0).getAddress().getPhone(),
is("+49 89 23 98 50 87"));
}
What's to be criticised here? First, consider what happens if, as a result of some bug, the street and street number are concatenated into the second line of the address and not the first line. What kind of failure message does one get? Something like the following:
Expected "Hauptstraße 62" but was "".
This does indeed clearly tell the reader the expected and actual values. However, it doesn't indicate in which field the problem was, nor in which element of the collection the problem occurred. That is, the entire context of the failure is missing. The reader must then hunt for the line in the source code to learn more about the failure.
Now consider what happens if, due to some bug, the converter returns an empty list. Then the first field access throws an IndexOutOfBoundsException due to the call to get(0). The message will just be a stacktrace of that exception with the corresponding line in the test code buried somewhere within. It provides no context at all about the nature of the failure. In fact, because it is triggered by an exception rather than a failure of comparison, it is precisely the same message as one would get if one used assertTrue rather than assertThat:
assertTrue(result.get(0).getType() == ExternalCustomerType.PRIV);
Finally, consider what happens when just one but multiple fields are incorrect. In this case, the test stops with the first assertion failure and does not check the remaining fields. This hides the scope of the problem from the reader.
Asserting equality on full objects
One simple attempt to improve on this is just to assert equality on the full ExternalCustomer object:
// Create expectedCustomer with expected data assertThat(result.get(0), is(expectedCustomer));
This improves on some, but not all, of the points mentioned above: one sees which fields are incorrect and just the values in those fields, and all fields will be compared so that the reader sees all fields which were incorrect rather than just one at a time. However, this does not solve the problem of reaching through a container. It also brings new problems:
- This requires that the equals method of ExternalCustomer be implemented in a way to compare the objects field by field, including a recursive comparison by subobjects. Thus the implementation of the test is bound to that of the object in a way which is not relevant to the functionality being tested. It could be that ExternalCustomer otherwise would not need an equals method or (worse yet) that its equals method should be defined differently -- say, by comparing just the name field. In the worst case, the equals method might be changed in response to a new requirement so that it checks fewer fields, in which case the test might succeed even when some fields are incorrect.
- What about the enumeration CustomerType? It's important to test the mapping of each value of the enumeration. It's convenient to do this in a separate test. But then there are two tests testing the mapping of the enumeration -- albeit one test only tests the mapping of one value.
Matchers
An alternative is to write a custom matcher. Here there are again a few variants, but we'll start by focusing on the comparison of just one property.
assertThat(result,
contains(externalCustomerWithType(
ExternalCustomerType.PRIV)));
Here we use the standard Hamcrest matcher contains, which takes a variable number of matchers as arguments and matches the result if and only if every matcher matches an element in the collection and the size of the collection equals the number of given matchers.
The function externalCustomerWithType is defined (in Java versions 6 and 7) as follows:
public static
FeatureMatcher<ExternalCustomer, ExternalCustomerType>
externalCustomerWithType(ExternalCustomerType type) {
return new FeatureMatcher<>(is(type), "type", "type") {
@Overrides
public ExternalCustomerType featureValueOf(
ExternalCustomer actual) {
return actual.getType();
}
}
}
Here we are using Hamcrests FeatureMatcher, which allows extracting an arbitrary datum out of a value and matching it according to a given matcher. It is convenient for comparing fields or small sets of fields in objects.
This approach has indeed a lot of boilerplate -- the lack of lambda expressions in Java versions up through 7 is to blame -- but it has some important advantages. Consider what happens if the test fails due to the type being wrong:
Expected <Collection of <type is PRIV>> but was [ExternalCustomer<type=BUS,...>]
Here you get the full context: not just the expected and actual values of that one field, but the name of the field, given as the two parameters "type" in the constructor to FeatureMatcher, as well as the full collection returned.
Suppose instead that an empty collection is returned. Then the result will appear as follows:
Expected <Collection of <type is PRIV>> but was: []
Now an empty collection shows up as a normal test failure and not a random exception.
Consider further the question of readability. Which of the two variants -- using a custom matcher and comparing equality on the field -- better expresses the intention of the test:
assertThat(result,
contains(externalCustomerWithType(
ExternalCustomerType.PRIV)));
or
assertThat(result.get(0).getType(),
is(ExternalCustomerType.PRIV));
I find the first to be more natural: we are asserting that the collection contains an ExternalCustomer with the property that its type is ExternalCustomerType.PRIV. It is irrelevant for the test, that the type be obtained by calling a getter on the first element of a list. The second variant fails to hide these details from the test, making the test more brittle (imagine what must be changed if the List were replaced by a Collection) and subtly harder to read.
What about multiple properties? Here we can collect several such matchers together with the help of Hamcrest's matcher allOf, which takes a set of matchers and matches its input if and only if all of the given matchers match that input. I find it helpful for readability to create an alias externalCustomerWithProperties for this purpose and to correspondingly rename the matchers:
assertThat(result, contains(externalCustomerWithProperties(
type(ExternalCustomerType.PRIV),
name("Max Mustermann"),
address("Hauptstraße 62"),
...);
This has an important advantage compared to the set of single assertions: all of the incorrect fields will be shown in the test output, so one immediately knows the scope of the failure. Unlike, however, the approach of testing equality of the whole object, we keep the test focused on only those fields which are relevant for it. We have the flexibility to test as few or as many fields as we deem appropriate in one test.
Matchers in Java 8
In Java 8 one can use lambda expressions to eliminate nearly all the boilerplate. First we define a kind of factory for FeatureMatcher.
public class PropertyMatcher<T, U> {
private final String propertyName;
private final Function<T, U> accessor;
public static <T, U> PropertMatcher<T, U>
property(String propertyName, Function<T, U> accessor) {
return new PropertyMatcher<>(propertyName, accessor);
}
public PropertyMatcher(String propertyName,
Function<T, U> accessor) {
this.propertyName = propertyName;
this.accessor = accessor;
}
public FeatureMatcher<T, U> matches(
Matcher<? super U> innerMatcher) {
return new FeatureMatcher<>(innerMatcher, propertyName,
propertyName) {
@Override
public U propertyValueOf(T actual) {
return accessor.apply(actual);
}
}
}
}
With this we can redefine our function externalCustomerWithType as follows:
public FeatureMatcher<ExternalCustomer, ExternalCustomerType>
externalCustomerWithType(ExternalCustomerType type) {
return property("type", ExteralCustomer::getType)
.matches(is(type));
}
We can even inline this method in the test itself without losing much readability, this eliminating nearly all bootstrap without losing any of the advantages of matchers.
Fest, JAssert, and Truth
The last few years have seen a wave of new assertion libraries which take a different approach: rather than relying on matchers, they provide a fluent interface for testing. For example, equality can be tested with the following:
assertThat(result.get(0).getType())
.isEqualTo(ExternalCustomerType.PRIV);
This has one major advantage vis-a-vis matchers: one can take advantage of autocompletion in modern IDEs such as Eclipse and IntelliJ so that one does not have to look up the name of the matcher one needs. When what one wants to do is within the scope of what the library provides, that's certainly an attractive advantage.
There is, however, a downside: it's harder to extend such frameworks with domain-specific matchers. Each one offers a slightly different approach to doing so, but they all involve a bit more work (and more bootstrap) than writing one's own matchers and using the standard JUnit assertThat.
My general view is that these libraries are a fine choice as long as one's requirements are within the bounds of what they provide, e.g. comparing equality on two relatively simple objects, assertions on collections of simple objects, and so on. As soon as one needs more complex assertions, such as "every element in this collection should satisfy the following property," one should switch to matchers. That said, there is no problem with using both approaches in the same program or even the same test class. Just use the best tool for the job.
Conclusion
Clean assertions are an integral part of clean, solid tests. Whatever approach you decide to use, you should take some time to consider the issues I brought up here and to choose the best tool for the job.