About Me

My Photo
A Sun certified Java professional with time proven experience in architecture and designing of mid/large scale Java/JEE based applications. Creator of the EasyTest Framework.A lot of experience with technologies such as Spring framework, Hibernate , JPA, Java4-6, REST, SOA , YUI , JUnit , Cloud Foundry PaaS and other technologies.

Wednesday, August 8, 2012

Junit's Parameterized Runner and Data Driven Testing Strategy

EasyTest : Write simple and intuitive Data Driven Tests with Junit


This is the PART 2 in the series that talks about Data Driven Testing in the Java world using Junit. You can find PART 1 of this series here:  Data Driven Testing with Junit PART 1 : Basic understanding

In the previous series, we saw an example of a traditional Junit test case which had a lot of duplicated code and data coupling. We looked at the major drawbacks of writing test cases in a traditional way. You can find more details here.

Next in the line is to look at Junit's Parameterized Runner that can help eliminate some of the drawbacks identified in the previous blog post. We will also look at the reasons why even Parameterized runner can not be used for writing effective Data Driven tests.

Using JUnit's Parameterized Runner:


Keeping the same example for consistency, Lets write the test case for the following ItemService class, but this time using Parameterized Runner of Junit :


public interface ItemService {
 
    public List getItems(LibraryId id , String searchText , String itemType);
}

 Here is the test class that tests all the four scenarios that were identified previously


@RunWith(Parameterized.class)
public class ItemServiceParameterizedTest {
 
    private String searchText;
 
    private String itemType;
 
    private Id libraryId;
 
    private Integer expectedRecords;
 
    private ItemService testSubject;
 
    @Before
    public void setUp(){
        testSubject = new RealItemService();
        //testSubject.setItemDao(new ItemRecordDao());
    }
 
    public ItemServiceParameterizedTest(String searchText , String itemType ,
                    Id libraryId , Integer expectedRecords) {
       this.searchText = searchText;
       this.itemType = itemType;
       this.libraryId = libraryId;
    }
    @Parameters
    public static Collection data() {
      Object[][] data = new Object[][] {
             { "harry Potter" , "ebook" , new LibraryId(1L) , 2 },
             { "Peter Pan" , null , new LibraryId(2L) , 3 },
             {null , "CD" , new LibraryId(3L) , 10  },
             {null , null , new LibraryId(3L) , 20 } };
        return Arrays.asList(data);
    }
    @Test
    public void testGetItems() {
        List items = testSubject.getItems(libraryId, searchText, itemType);    
        Assert.assertNotNull(items);
        Assert.assertEquals(expectedRecords, new Integer(items.size()));
    }
}
This looks much better than our previous example. So what exactly did we do.
We did the following:
  1. Introduced @RunWith(Parameterized.class) which is a way of telling Junit that run this test class using the supplied runner. The supplied runner in our case is the Parameterized Java class that understands @Parameters annotation and that calls our testGetItems method for each set of test data. Thus in our case testGetItems method will be called 4 times since we are specifying 4 sets of data inside the data() method.
The main benefits that are straight away clear from the above are :
  1. The actual test code has reduced to 1/4th of the previous code. 
  2. The test data now resides outside of the test case and therefore is not coupled to the test case.
There are also some drawbacks to this. I will discuss one of those drawbacks using an example.

Drawbacks of Parameterized Runner


Now in the real world a Business Service doesnt have a single method, instead it has many. So lets add one more method to our existing ItemService class.

public interface ItemService {
 
    public List getItems(@NotNull Id id , String searchText , String itemType);
 
    public Item findItem(@NotNull Id libraryId , @NotNull Id itemId);
}
Here we have a new method findItem that takes a libraryId and an itemId and returns an instance of Item.

Lets try to write the test case for this Service method using Parameterized runner of Junit.


@RunWith(Parameterized.class)
public class ItemServiceParameterizedTest {
 
    private String searchText;
 
    private String itemType;
 
    private Id libraryId;
 
    private Integer expectedRecords;
 
    private Id itemId;
 
    private ItemService testSubject;
 
    @Before
    public void setUp(){
        testSubject = new RealItemService();
        //testSubject.setItemDao(new ItemRecordDao());
    }
 
    public ItemServiceParameterizedTest(String searchText , String itemType ,  Id libraryId , Integer expectedRecords , Id itemId) {
       this.searchText = searchText;
       this.itemType = itemType;
       this.libraryId = libraryId;
       this.itemId = itemId;
    }
    @Parameters
    public static Collection data() {
      Object[][] data = new Object[][] {
          { "harry Potter" , "ebook" , new LibraryId(1L) , 2 , new ItemId("2") },
          { "Peter Pan" , null , new LibraryId(2L) , 3 , new ItemId("2")  },
          {null , "CD" , new LibraryId(3L) , 10 , new ItemId("2")  },
          {null , null , new LibraryId(3L) , 20 , new ItemId("2") } };
      return Arrays.asList(data);
    }
    @Test
    public void testGetItems() {
        List items = testSubject.getItems(libraryId, searchText, itemType);    
        Assert.assertNotNull(items);
        Assert.assertEquals(expectedRecords, new Integer(items.size()));
    }
 
    @Test
    public void testFindItem() {
        Item item = testSubject.findItem(libraryId, itemId);    
        Assert.assertNotNull(item);
    }
}
Do you find something weird about the above test class? HINT: The two test methods have a strong data coupling.

If you look closely, we have a big problem now in our hands. We have no way of specifying the test data for a single test method. The test data has to be specified using @Parameters annotation for ALL the test cases. Also we have no way now to specify that "testGetItems" method should run for 4 sets of test data and "testFindItem" should run for only 2 sets of test data. All our test cases are bound with the number of test data set that we define using @Parameters annotation. This is clearly undesirable. For startes this will unnecessary run our test cases multiple times, therby increasing the time taken by the tests to run and also bringing in redundancy to the entire test suite.

Besides this there are other not so visible side effects(or shall I say drawbacks) of using Parameterized annotations :
  • We are forced to follow Junit's way of declaring the test data which may work in some scenarios but does not work in all. 
  • We do not have any straightaway mechanism to externalize the test data from the test class. What this means is that if the test case needs to be tested with more data, the Test class file needs to be modified. I feel this is one of the biggest restrictions in adopting Parameterized test strategy for data driven testing.
  • All the parameters have to be declared as member variables adding unnecessary complexity to the system
  • All the parameters have to be passed into the single constructor in order for JUnit to initialize the members variables for use in test cases.
  • The return type of parameter class is an Array of List, which may not be straight away a drawback but some people may have problems with that.
So based on the above discussion, we could clearly say that whilst Parameterized runner promised to be a step in the right direction, it lacks a lot of features that would help us in achieving our goal of writing Data Driven Tests that are decoupled from the test data, where each Test can be supplied its own test data and where writing tests does not involve unwanted things like writing a custom constructor, introducing member variables etc.

In the next post I will talk about how we can overcome the difficulties we have identified in this post using another experimental feature of JUnit, called Junit Theories. 


SideNote: You can find the code base that improves JUnit's support for Data Driven Testing here : https://github.com/EaseTech/easytest-core

I am looking for feedback to further improve the project.

Next Post : JUnit Theories and Data Driven Testing

3 comments:

braghome said...

wondering how do your data model LibraryId looks like, their is not mooch light on how exactly your models are described.

braghome said...

also class's listed below would be nice to see the implementation

RealItemService

ItemId

Item

InstitutionId

Anuj said...

Hi braghome,
Have a look at examples of EasyTest here https://github.com/EaseTech/easytest-samples/tree/master/src/main/java/org/easetech/easytest/samples

You canfind the implementations as well as some working examples.