About Me

My Photo
A Sun certified Java professional with time proven experience in architecture and designing of mid/large scale Java/JEE based applications. Creator of the EasyTest Framework.A lot of experience with technologies such as Spring framework, Hibernate , JPA, Java4-6, REST, SOA , YUI , JUnit , Cloud Foundry PaaS and other technologies.

Monday, August 13, 2012

JUnit Theories and Data Driven Testing

EasyTest : Write simple and intuitive Data Driven Tests with Junit

In the previous posts, we have seen the way to write traditional tests and also how we can use JUnit's Parameterized class to write tests that are close to being data driven. We saw the drawbacks of both the approaches and in this post we will try to overcome those drawbacks using yet another feature from JUnit called Theories.

Theories in Junit are similar to Parameter driven testing in that they both allow you to specify test data outside of the test case. But then the similarity ends there. Whereasa Parameterized runner is capable of identifying @Test annotations, Theories runner has its own special annotation @Theory.

I have my own opinions on JUnit's Theories but I will keep them seaparate from this this current post and I will talk about it in a later post. In case you want to know my views, you can have a look here.

So how does Theories help us in writing data driven tests?

JUnit theories have the following Annotations :

@Theory - This annotation identifies the Theory test to run.
@DataPoint - This annotation identifies a single set of test data(but interpreted in a totally differnt way by JUnit, as we will see very shortly)
@DataPoints - This annotation identifies multiple sets of test data(again  interpreted  in a totally different way)
@ParametersSuppliedBy - This annotation is responsible for providing the parameters to the test cases.

It also has the following important classes:

Theories.java - This is the JUnit Runner for the Theory based test cases and extends org.junit.runners.BlockJUnit4ClassRunner

ParameterSupplier.java - This is the abstract class that gives us the handle on the parameters that we can supply to the test case. We will see in detail later how we can leverage this to our benefit.

Right. The basics out of the way, lets see Theories in action.

We will continue with our same example that we had identified in part 1 and extended in part 2 of this series.

Writing tests using Theories Runner

Here is the example Service once again for easy reference:

public interface ItemService {

    public List getItems(@NotNull Id libraryId , String searchText , String itemType);

    public Item findItem(@NotNull Id libraryId , @NotNull Id itemId);
SIDE NOTE: JUnit came up with the idea of Theories for testing cases that are universal truths. Now a universal truth is confined to the universe that the situation is being tested in. So given a clear definition of what a universe means to someone, he/she should be able to use Theories to test different conditions. And that's my take on Theories. Thus, in my closed technical universe that consists of my project within my company, I can have a theory that is completely valid for my closed world. I normally do not care about the hypothesis that could occur outside of my software, because thats not relevant to me or to my software. Therefore I consider that I can use Theories to test conditions that I consider universal truth in my own universe. (I sincerely hope I don't annoy JUnit creators by making this assumption. I just want to bring focus on a different perspective of interpreting Theories. BTW, in my next post I will also talk about Junit Theories and testing universally accepted mathematical theorems, so stay tuned :))

So lets again try to test the above Service, this time by writing Tests that are run by Theories.java JUnit Runner.

public class ItemServiceTheoriesTest {

private ItemService testSubject;

    public void setUp(){
        testSubject = new RealItemService();

    @DataPoint public static ItemId itemId = new ItemId(2L);

    @DataPoint public static LibraryId libraryId = new LibraryId(2L);

    public void testFindItem() {
        Item item = testSubject.findItem(libraryId, itemId);

Lets go through this class quickly.

  • The first thing to notice is that we are now using Theories.class to run our JUnit tests. This class is an extension of BlockJUnit4ClassRunner.
  • Next we are setting up our testService class to run our tests against. This is like before.
  • Next we have used @DataPoint annotation that comes as part of JUnit Theories feature. This annotation tells the Theories Runner that the annotated static field(s) is the data point that needs to be provided to the test case. In our case it is the method testFindItem.
  • Finally we have annotated our method with @Theory instead of @Test. This is the annotation that Theories Runner looks for to run the test cases.
Ok, the basics are now out of the way. Lets see what happens when we run this test. 

When the test case is run, Theories runner looks for methods annotated with @Theory and feeds this method with the test data instance. So in our case our test method will run exactly one time and the values passed to the test method will be the values we defined with @DataPoint annotation.

The above was a simple case where we had just one set of test data for a given test. Lets extend this test case and try to test the following theory : Test that ItemService returns an Item for all the passed set of ItemIds.

public class ItemServiceTheoriesTest {

private ItemService testSubject;

    public void setUp(){
        testSubject = new RealItemService();

    @DataPoints public static ItemId[] data(){
        return new ItemId[] {
            new ItemId(1L),
            new ItemId(2L),
            new ItemId(3L)

    @DataPoint public static InstitutionId libraryId = new InstitutionId(2L);

    public void testFindItem(ItemId itemId) {
        Item item = testSubject.findItem(libraryId, itemId);
There are two major things to notice here:
  • I am now using @DataPoints instead of @DataPoint to let the Theories Runner know that i am passing a set of test data instead of a single instance.
  • I am now declaring a parameter to testFindItem called itemId of type ItemId. This is something that JUnit test writers have never done before, passing a parameter to a test method. This is new and very powerful. We will discuss in more detail later.
So now my test method testFindItem will run 3 times with the following combination :
library Id : 2 and item id : 1
library Id : 2 and item id : 2
library Id : 2 and item id : 3
As we see, Theories is already turning out to be a real winner. Theories has given me the power to separate my test data per parameter instead of per set of parameters. Also I now have the ability to pass on parameters to my test method which is a first in JUnit. finally The runner takes care of passing in the right arguments to the test method without any interference or any extra code from my side. Also we can use the combination of DataPoint with DataPoints annotation and Theories runner will take care of injecting the right data.

So far so good. Lets define another theory to test for my above test class :

I want to search for all the items with the following combination for Library 1 and Library 2: 

[itemType = ebook and a list of searchText = {"potter" , "poppins" , "superman"}]
[itemType = book and a list of searchText = {"potter" , "spiderman" , "batman"}]
[itemType = journal and a list of searchText = {"java" , "junit" , "nasa"}]

I can do the above using Parameterized runner, but the drawbacks discussed in my previous post applies here as well(no per method test data for example, define constructor etc.) and so I effectively would not like to use Parameterized Runner.

As it turns out, I do not have a straight forward way of achieving my above goal. Not with using @DataPoint(s) alone. I cannot write my test like this :

public class ItemServiceTheoriesTest {

private ItemService testSubject;

    public void setUp(){
        testSubject = new RealItemService();

    @DataPoints public static LibraryId[] libraryData(){
        return new LibraryId[] {
            new  LibraryId (1L),
            new  LibraryId(2L)      

    @DataPoints public static String[] itemType(){
        return new String[] {

    @DataPoints public static String[][] searchText(){
        return new String[][] {
            {"potter" , "poppins" , "superman"},
            {"potter" , "spiderman" , "batman"},
            {"java" , "junit" , "nasa"}

    public void testGetItems(LibraryId libraryId , String itemType , String[] searchText ) {
        List items = testSubject.getItems(libraryId, searchText[0] , itemType);
The reason is as follows:

What JUnit is doing behind the scene is it is creating a List of PotentialAssignment instances (List) from ALL the @DataPoints annotated methods. So in the above case, JUnit Theories will create the following List:

[[LibraryId(1)], [LibraryId(2)], [ebook], [book], [journal], [[potter, poppins, superman], [[potter, spiderman, batman], [[java, junit, nasa]]

It will then try to invoke the test method with ALL the possible combinations of test data from the above list. Thus JUnit invokes my test method with the following combination of test data :libraryId : LibraryId[1], itemType : LibraryId[1], searchText :Library[1] which will obviously fail with the error :

org.junit.experimental.theories.internal.ParameterizedAssertionError: testgetItems(libraryData[0], libraryData[0], libraryData[0])
at org.junit.experimental.theories.Theories$TheoryAnchor.reportParameterizedError(Theories.java:183)
at org.junit.experimental.theories.Theories$TheoryAnchor$1$1.evaluate(Theories.java:138)
at org.junit.experimental.theories.Theories$TheoryAnchor.runWithCompleteAssignment(Theories.java:119)
at org.junit.experimental.theories.Theories$TheoryAnchor.runWithAssignment(Theories.java:103)
at org.junit.experimental.theories.Theories$TheoryAnchor.runWithIncompleteAssignment(Theories.java:112)
at org.junit.experimental.theories.Theories$TheoryAnchor.runWithAssignment(Theories.java:101)
at org.junit.experimental.theories.Theories$TheoryAnchor.runWithIncompleteAssignment(Theories.java:112)
at org.junit.experimental.theories.Theories$TheoryAnchor.runWithAssignment(Theories.java:101)
at org.junit.experimental.theories.Theories$TheoryAnchor.runWithIncompleteAssignment(Theories.java:112)
at org.junit.experimental.theories.Theories$TheoryAnchor.runWithAssignment(Theories.java:101)
at org.junit.experimental.theories.Theories$TheoryAnchor.evaluate(Theories.java:89)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
Caused by: java.lang.IllegalArgumentException: argument type mismatch
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at org.junit.experimental.theories.Theories$TheoryAnchor$2.evaluate(Theories.java:167)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at org.junit.experimental.theories.Theories$TheoryAnchor$1$1.evaluate(Theories.java:133)

Thus we have seen in this post that even though JUnit has a nice feature in Theories, it is difficult to write complex tests with complex test data set. In the next post. we will see how we can overcome the above problem by using ParameterSuppliedBy annotation extension.

SideNote: You can find the code base that improves JUnit's support for Data Driven Testing here : https://github.com/EaseTech/easytest-core

I am looking for feedback to further improve the project.

Next Post : JUnit Theories and ParameterSuppliedBy annotation

No comments: