Data-Driven Automation

Data-Driven Automation

Data-driven testing is the idea of building a generic testing framework and letting the data define and drive the tests - you can change your tests simply by changing the data either in content or order. 

I’m personally torn on this subject. 

The developer in me says “yes, this is amazing, absolutely do this!” Separating data from code is a core tenet of best practices. 

The QA in me says “test cases are more nuanced - too nuanced to be purely data driven in any practical sense with existing technology.” Designing a testing framework that is truly data-driven would require far too much work in development and maintenance then would be practical (that hasn’t stopped me from trying - and failing). 

Objectifying Your Data: The First (Achievable) Step

If we look back at the post on ACS, you’ll see code like this:


This is explicitly coupling the metadata - “xXx_bob_XxX” - to the test step setUsername. This isn't a bad thing - it makes is very easy to read your test case and see exactly what data is being set line-by-line. This helps with the Watchman Problem. But on the other hand, it’s difficult to maintain. We use that username several times through that test - if we wanted change the username at a later date, we’d have to remember to updated it multiple locations. This problem could be exacerbated if we use the same data across multiple tests - such as parent accounts or SKUs. 

Instead, we can remove the hard coded data. The obvious baby step is to define your data in variables at the start of the test case:

String username = “xXx_bob_XxX”

But we can do better. Most testing frameworks are Object-Oriented (Java, Python, C#), so lets leverage that power. If we define our test data in JSON (or a database and pull it in via JSON):


... build out a Class User to match...

class user{
    private String username

... we can use a library to parse in the JSON data into our classes (GSON, Jackson)...

JsonArray array = json.getAsJsonArray(jsonKey);
List<Object> objects = new ArrayList<Object>();
for (JsonElement e : array){
Object object = gson.fromJson(e, blankObject.getClass());
this.users = (List<User>) objects

... and now we can access out data in our test without ever having to reference it directly:

login_ui.setUsername() // uses the current ‘test.user’ reference

This gives us a huge amount of power and makes our tests far more maintainable. Our test code is now truly only Commands, our UI code is purely Actions and the Data is isolated into a single well-organized collection. This gain is magnified when testing REST APIs, as you can literally just write the JSON payload directly in your Data, package and send - no more escaping character literals. This does come at the drawback of readability - you now need to have at least three files open at any time to really understand a test case - data, test, and ui. But I feel the pros outweigh that drawback. 

True Data Driven Testing: A Potential Pipe Dream

All the above is nice, and it’s a huge step forward for a great many frameworks - but it’s not true data-driven testing. True data driven testing is when your data defines your tests: you don't ‘write’ tests, you write a framework that is capable of testing, you then write ‘test definitions’ in plain English that contain both the data and the instructions for the test. The final piece is a powerful parser that interprets what you wrote in your definition and calls the appropriate methods in the framework in proper order. 

The problem I have with this is that tests tend to be too nuanced for this to work at any large scale. Your definition language would need to be extremely strict and strictly regulated to the point that you’re basically just writing test code. Your framework would need to be so generic, so open to handling every possible request that might be asked of it, that it would be practically useless and possibly unreadable. Essentially, you’d come full circle back to writing tests that contain everything they need to do within the test class itself, which defeats the purpose. 

In my humble opinion, this is why adoption of Cucumber has been fairly slow going: it’s difficult to write and maintain properly. Not impossible, just impractical for many projects. I’m still a big fan of Cucumber, and would love to see it implemented more, but I have to acknowledge the difficulty of doing it right. 

This has been the dream of computers since computers were a thing: a computer that can do anything just by asking it to. The classic Star Trek computer that can simply be asked things and have them done without being told any specific instructions on how to do them. I don't believe this is impossible, Artificial Intelligence is making huge gains, but it’s not here yet, and it’s certainly not going to be solved by your AQA engineer. 

Data Generators: The Wise Fool

Data Generators: The Wise Fool

Action-Command Scripting

Action-Command Scripting