When it came to defining tests, JUnit 4 had a tremendous weakness: It had to happen at compile time. Now, JUnit 5 will fix this oversight! Milestone 1 just got released and it comes with the brand-new dynamic tests, which allow the creation of tests at run time.
Overview
Other posts in this series about JUnit 5:
This series is based on the pre-release version Milestone 1, which is of course subject to change. The posts will get updated when a new milestone or the general availability release gets published.
Most of what you will read here and more can be found in the emerging JUnit 5 user guide (that link went to the Milestone 1 version – you can find the most current version here). The code samples I show here can be found on GitHub.
Static Tests
JUnit 3 identified tests by parsing method names and checking whether they started with test. JUnit 4 took advantage of the (then new) annotations and introduced @Test, which gave us much more freedom. Both of these techniques share the same approach: Tests are defined at compile time.
This can turn out to be quite limiting, though. Consider, for example, the common scenario that the same test is supposed to be executed for a variety of input data, in this case for many different points:
void testDistanceComputation(Point p1, Point p2, double distance) {
assertEquals(distance, p1.distanceTo(p2));
}What are our options? The most straight forward is to create a number of interesting points then just call our test method in a loop:
@Test
void testDistanceComputations() {
List<PointPointDistance> testData = createTestData();
for (PointPointDistance datum : testData) {
testDistanceComputation(
datum.point1(), datum.point2(), datum.distance());
}
}If we do that, though, JUnit will see our loop as a single tests. This means that tests are only executed until the first fails, reporting will suffer, and tool support is generally subpar.
There are a couple of JUnit 4 features and extensions that address this issue. They all more or less work but are often limited to a specific use case (Theories), are awkward to use (Parameterized), and usually require a runner (like the commendable JUnitParams). The reason is that they all suffer from the same limitation: JUnit 4 does not really support creating tests at run time.
The same applies to creating tests with lambdas. Some would like to define tests like this:
class PointTest {
"Distance To Origin" -> {
Point origin = Point.create(0,0);
Point p = Point.create(3,4);
assertEquals(5, origin.distanceTo(p));
}
}This is of course just an ideal – it does not even compile in Java. Nevertheless, it would be interesting to see how close we can get. Alas, individual lambdas can not be statically identified, either, so the same limitation applies here.
But I wouldn’t be writing all of this if JUnit 5 did not propose a solution: Dynamic tests to the rescue!
Dynamic Tests
Since very recently the JUnit 5 code base sports a new type and a new annotation and together they address our problem.
First, there is DynamicTest, a simple wrapper for a test. It has a name and holds the code that makes up the test’s body. The latter happens in the form of an Executable, which is like a Runnable but can throw any Throwable (formidable naming). It is created with a static factory method:
public static DynamicTest dynamicTest(String name, Executable test);
Then there is @TestFactory, which can annotate methods. Those methods must return an Iterator, Iterable, or Stream of dynamic tests. (This can of course not be enforced at compile time so JUnit will barf at run time if we return something else.)
It is easy to see how they cooperate:
- When looking for @Test methods, JUnit will also discover @TestFactory methods.
- While building the test tree, it will execute these methods and add the generated tests to the tree.
- Eventually, the tests will be executed.
We are hence able to dynamically create tests at run time:
@TestFactory
List<DynamicTest> createPointTests() {
return Arrays.asList(
DynamicTest.dynamicTest(
"A Great Test For Point",
() -> {
// test code
}),
DynamicTest.dynamicTest(
"Another Great Test For Point",
() -> {
// test code
})
);
}Let’s see how we can use it to solve the problems we described above.
Parameterized Tests
To create parameterized tests, we do something very similar to before:
@TestFactory
Stream<DynamicTest> testDistanceComputations() {
List<PointPointDistance> testData = createTestData();
return testData.stream()
.map(datum -> DynamicTest.dynamicTest(
"Testing " + datum,
() -> testDistanceComputation(
datum.point1(), datum.point2(), datum.distance()
)));
}The critical difference to what we did above is that we do not directly execute testDistanceComputation anymore. Instead we create a dynamic test for each datum, which means that JUnit will know that these are many tests and not just one.
In cases like this we might use a different method to generate the dynamic tests:
@TestFactory
Stream<DynamicTest> testDistanceComputations() {
return DynamicTest.stream(
createTestData().iterator(),
datum -> "Testing " + datum,
datum -> testDistanceComputation(
datum.point1(), datum.point2(), datum.distance()));
}Here we hand our test data to stream and then tell it how to create names and tests from that.
So what do you think? Maybe something along the lines of “It’s cool that JUnit 5 treats these as individual tests but syntactically it’s still cumbersome”? Well, at least that’s what I think. The feature is nice but somewhat ungainly.
But this is only Milestone 1 so there is enough time for improvement. Maybe extensions can provide a more comfortable way to create dynamic tests but I don’t quite see how. I guess, a new extension point would help.
Lambda Tests
Ok, let’s see how close we can get to the much-coveted lambda tests. Now, dynamic tests were not explicitly created for this so we have to tinker a bit. (This tinkering is, err, “heavily inspired” by one of Jens Schauder‘spresentations about JUnit 5. Thanks Jens!)
A dynamic test needs a name and an executable and it sounds reasonable to create the latter with a lambda. To be able to do do this, though, we need a target, i.e. something the lambda is assigned to. A method parameter comes to mind…
But what would that method do? Obviously it should create a dynamic test but then what? Maybe we can dump that test somewhere and have JUnit pick it up later?
public class LambdaTest {
private final List<DynamicTest> tests = new ArrayList<>();
// use lambda to create the 'Executable'
public void registerTest(String name, Executable test) {
tests.add(DynamicTest.dynamicTest(name, test));
}
@TestFactory
void List<DynamicTest> tests() {
return tests;
}
}Ok, that looks promising. But where do we get an instance of LambdaTest? The easiest solution would be for our test class to simply extend it and then repeatedly call registerTest. If we do so, we might prefer a shorter name, though; and we can also make it protected:
// don't do this at home!
protected void λ(String name, Executable test) {
tests.add(DynamicTest.dynamicTest(name, test));
}Looks like we’re getting there. All that’s left is to call λ and the only apparent way to do this is from inside our test class’ constructor:
class PointTest extends LambdaTest {
public PointTest() {
λ("A Great Test For Point", () -> {
// test code
})
}
}We’re done tinkering. To get further, we have to start hacking. Ever heard of double brace initialization? This is a somewhat strange feature that creates an anonymous subclass and executes the given code in the new class’s constructor. With it, we can go further:
class PointTest extends LambdaTest {{
λ("A Great Test For Point", () -> {
// test code
});
}}If we’re really eager we can shave off another couple of symbols. With this one weird trick (we’re now being inspired by Benji Weber), we can determine a lambda’s parameter name via reflection and use that as the test’s name. To take advantage of that we need a new interface and have to change LambdaTest::λ a bit:
@FunctionalInterface
// the interface we are extending here allows us
// to retrieve the parameter name via 'prettyName()'
// (the black magic is hidden inside that method;
// look at 'MethodFinder' and 'NamedValue' in Benji's post)
public interface NamedTest extends ParameterNameFinder {
void execute(String name);
}
protected void λ(NamedTest namedTest) {
String name = namedTest.prettyName();
Executable test = () -> namedTest.execute(name);
tests.add(DynamicTest.dynamicTest(name, test));
}Putting it all together we can create tests as follows:
class PointTest extends LambdaTest {{
λ(A_Great_Test_For_Point -> {
// test code
});
}}What do you think? Is it worth all that hacking? To be honest, I don’t mind having my IDE generate test method boilerplate so my answer would be “No”. But it was a fun experiment. :)
Lifecycle
The current implementation of dynamic tests is deliberately “raw”. One of the ways this shows is that they are not integrated into the lifecycle. From the user guide:
This means that @BeforeEach and @AfterEach methods and their corresponding extension callbacks are not executed for dynamic tests. In other words, if you access fields from the test instance within a lambda expression for a dynamic test, those fields will not be reset by callback methods or extensions between the execution of dynamic tests generated by the same @TestFactory method.
There is already an issue to address this, though.
Reflection
So what have we seen? Up to now JUnit only knew about tests that were declared at compile time. JUnit 5 has a concept of dynamic tests, which are created at run time and consist of a name and an executable that holds the test code. With that we have seen how we can create parameterized tests and use lambdas to define tests in a more modern style.
What do you think? Eager to try it out?
| Reference: | JUnit 5 – Dynamic Tests from our JCG partner Nicolai Parlog at the CodeFx blog. |
