Integration Testing: The Foundation of AQA

Integration Testing: The Foundation of AQA

In my humble opinion, integration testing is the foundation of AQA. The tests are simple to write, fast to run, and will find you the most bugs for the least amount of effort. It also tests a part of the system that is not only difficult to test manually but also deterministic and objective  - these are areas where AQA shines and help underscore the usefulness of implementing AQA as part of your organization.

Integration testing tests multiple units in concert and typically also tests external dependencies, such as APIs and databases. The units here are small enough to be easy to write and maintain but large enough to provide value to the project. Most integration testing these days is done through REST APIs, which is where I’m going to focus most of this post.

Integration testing can really help get buy-in around AQA. It’s easy for people to wrap their head around. When you can go to a manager and say, ‘Hey, I wrote 50 test cases around 20 APIs in 2 weeks’ that sounds really impressive - and if your testing has primarily been manual and front-end based, it is very impressive.

How do API tests work? Well, fairly simply - you send a request and verify that the server responses with what you expected. You can verify the status code, status description, and parse the body of the response to ensure the data you requested actually exists. When it comes to any request that modifies data, such as POST requests, you’ll need to chain requests by performing a POST and then immediately doing a GET to ensure the data was actually created.

Building a Foundation for AQA

Integration tests should build the foundation for your AQA efforts. The speed and reliability of the tests will allow you to build dozens of them very quickly and maintain them with little effort. Further they allow you to use the best of Behavior Driven Development (BDD) and Test Driven Development (TDD) (see below) to get your AQA off the ground right.

First, identify if you can do integration testing. This sounds a bit odd, but a lot of legacy projects don't have APIs or any external-facing integration layers. If this is your project, this kind of testing isn't for you (legacy projects are hard to add AQA to anyway, and that deserves it’s own post).

Second, choose a tool. If you’re new to AQA, choose a simple tool like Postman. If your process is more mature or has dedicated resources, go all in with RESTAssured and Cucumber (or any other tool that fits your fancy).

Third, build a framework. With Postman, this is as simple as learning how to structure your Collections. With more code-based tools you’ll need to figure out your package structure and start building our some common library functions. The trick here is to fail faster: don't freeze up in analysis. Get an idea and try it. Try anything. Don't try to account for every conceivable test. Build for the tests you need right now and adapt as you go (and be prepared to do some major refactors down the line).  

Finally, build some tests and run them! This step shouldn't be paralyzing. Developers writing code will compile and run their code dozens of times before checking it in, each time fixing little bugs. Your tests don't need to be any different. Run them, they will break, it will be some stupid logic error. Fix them. Run again. Fix another logic error. Repeat until they work (note: not pass - If they fail because there is a bug in the project, let them keep failing until the bug is fixed).

Congratulations, you have AQA off the ground, now just keep it running! Use the lessons you learn from building integration tests to improve your process and build better AQA moving forward. Much of the same principles apply to front-end automation, event-driven testing,  performance testing and beyond; but integration tests are so simple they allow you to learn these lessons faster and can adapt with more agility.

Start reading your test logs and making adjustments. Build larger and more complex suites. Learn the maintenance cost curve of your tests. Improve your framework to better suit your needs. Improve your process to better streamline your AQA. Improve your reporting for easier debugging of your tests and for better bug reports.

Tools

Tools for integration tests are many and various. The only real requirement you need is the ability to query an API or database, which basically everything can do these days (including your light bulbs). That said, not everything that can use an API makes a good testing tool.

Below are all ones I have personal experience with, but if they don't meet your personal flavor do some searching and find one that does. Generally speaking I advocate for choosing a test framework that is of a similar programming paradigm as your project - EG if your project is written in Java or C#, use a Java or C#-based testing tool.

Postman (GUI)

Postman is a favorite of mine for getting AQA off the ground at any organization. It’s a very simple GUI based tool that can make simple REST requests and do simple response verification. With a bit of structuring and the Newman extension, it can even be run headless.

Postman is fantastic for answering that first, basic question, ‘Is AQA worth it for us?’ With its simple GUI, it shouldn't take much more than a day for any Developer or QA resource to get some basic tests up and humming. It’s easy to then build on that momentum and realize how quickly you can build suites of tests that can be run auto-magically on-demand - and that kind of power garners attention - and a budget.

RESTAssured (Driver)

Back in mid-2015, I did an analysis on RESTAssured for a company project. It was... not good. It’s come a long way since those days, and I now very much recommend it.

RESTAssured is a Java library that allows simple ‘given-when-then’ syntax to be applied to API tests. It provides a wealth of methods for interacting with APIs that streamline the process immensely. It’s commonly paired with a Java testing framework, typically TestNG or Cucumber, to drive the actual tests.

TestNG (Harness)

TestNG is a common test harness for any Java-based testing. It uses Java Reflection library to run your tests in a tidy order. The only drawback here is you’ll need to learn how to structure your tests in a TestNG compatible fashion. It’s not structured the same as a standard Java program and there are a lot of special functions and overrides to change how your tests execute. It works, however, and it’s internally consistent once you dig into the documentation a bit.

Robot (Driver and Harness)

Robot is a relatively new kid on the block compared to the Java-based frameworks. It is written in Python and uses a Gherkin-style syntax to drive your tests. Python being Python, your tests are quite succinct, well structured, and easy on the eyes. At the time of this writing I don't have any professional experience, but I’ve done some tutorials and it’s pretty easy to get into. The only drawback is there are far more Java developers in the world then Python - and odds are your project code is writing in Java or something more similar to Java (C#) then Python, meaning it’ll be harder to steal or cross-train existing resources.

SoapUI (GUI)

As the name suggests, SoapUI specializes in SOAP APIs. SoapUI can do REST APIs as well, but you really don't want it to. SoapUI is a rather clunky UI based testing tool (which is slightly less clunky in the paid pro version). In all honesty, avoid it as much as possible. It’s a pretty terrible tool. But, it's basically the only tool that can handle SOAP APIs, so if that’s what you need, then here you are.

Fiddler (GUI)

I’m going to call this out here real quick: DON'T USE FIDDLER FOR API TESTING. Back in the days before Postman and other widely available API testing tools, Fiddler seemed like a good GUI-based option. However, Fiddler is NOT an API testing tool. It is a network analyzer. It gives you information about your network requests. That fact that it can make API requests and read responses is a side-effect, not a feature. There are far, far better tools to use in modern times. Please, update your tests. Your team will thank you.

Contract Driven Testing

Behavior Driven Development (BDD) teaches to build your application based on the behavior you want it to exhibit. This is the given-when-then syntax and Gherkin style syntax that is slowly becoming the norm: your user stories should state what the software should do under certain circumstances. Test Driven Development (TDD) fits well into this mold: once you know what your software should do, you can build tests in parallel with development. Your tests will start passing once development reaches a certain maturity (assuming there are no bugs). While you can (and should) apply BDD and TDD to just about any automated test, they are particularly strong paradigms when building API tests.

With API testing we can take them one step further. I like to call this paradigm Contract Driven Testing. With BDD, there’s always room for ambiguity. Language is tricky like that, two people (or a whole development team) can have a completely different interpretation of a given story. With APIs, we can be more precise. We build out contracts before development starts: this is the exact data this API will surface, this is the exact format of data this API will accept, etc. These contracts allow us to build extremely precise test cases before development even starts. In a mature environment, the automated API tests can be finished weeks in advance of the actual APIs.

Data Driven Testing

DDT is another excellent paradigm to apply to integration testing. This happens when you seperate your data from your implementation.

Given [data set]
When I call [API]
Then I expect [response data]

[data set] and [response data] here can be abstracted outside the test case. You can put it in metadata: external JSON files. The trick is having a way to read in the metadata dynamically.

Enter, GSON (or other similar libraries, or Python that can do it dynamically). GSON allows you to parse JSON data directly into Java Objects. All you need to do is build out classes that mirror your APIs. Each field in the API JSON becomes a property in the Java class. This allows you to read in your metadata into Java classes, parse it to JSON to send it to the API, then take the API response and parse it into a second Java Object. Now you have two objects that allow you to easily compare them.

You can take this idea one step further by chaining APIs. An example would be a User API that has an Address where Address is a separate API:

{
     "id":1,
     "firstName":"john",
     "lastName":"smith",
     "address": {
          "id": 12345
     }
}

GSON is capable of reading JSON Objects as separate Java Object classes. If you structure your classes properly, you can have your test automatically call the Address API with the given ID and GSON will construct a proper Java Object reference:

public class User {
     private Integer id;
     private String firstName;
     private String lastName;
     private Address address;
     ...
}

Pitfall: Not Having Data Contracts or API Documentation

“With Scrum/Agile, we no longer need to write documentation, we can just write code all the time! Weeee~” Please, for the love of all that is AQA, stop this mentality! APIs work on data contracts. Each system needs to know exactly what the other accepts and surfaces. This is doubly the case when different teams (or companies!) are handling the different integrated systems. AQA needs those same contracts to work efficiently.

Without those data contracts, AQA will have to wait until the API is actually built to build the test - which means they cannot work in parallel which is a key strength of API testing. Further, they’ll have to refer back to the Developers every time they make a request. If I had a nickel for every time I had to send out email to the development team asking ‘is THIS the right response? Does THIS field look right? When I POST with THIS data I get THIS response, but when I POST with THIS data I get THIS response...” I could retire on a beach somewhere. If I got a dime for every “Hmm, I’m not sure, I’ll get back to you tomorrow” reply I got, I could buy my own island.

Pitfall: System of Record - Verifying Your Data

There is a common thought when doing API testing: the API returned x dataset, but is that what it was supposed to return? I asked it for all users and it returned 47 - is that actually all the users? I asked it for John’s address, and it returned an address - how do I know it’s John’s?

My first thought here is to not worry about this problem. You are far more likely to find issues with the contract itself then with the data it returns. Typos, fields missing, extra fields, incorrect pagination. These are going to be the primary bugs you’ll find.

My second thought is when you reach the point of maturity where you have the ability to worry about this, design tests that explicitly test for it. Not every test needs to verify the dataset is correct, a handful that do is enough. Design datasets that you know are structured a certain way. How do you know if it’s giving you John’s data? First go make John (either with a POST or a SQL query) then query GET to retrieve John - is it the same data you gave it? Don't rely on data that already happens to be in the system, design your own data.

If you absolutely have to, you can query the database directly and use that to verify your API data. Depending on the complexity of your database, this can be easier said than done and may not be viable for all APIs. But, if you trust the database as the system of record (and most would argue this correct), then this is a viable option.

Finally, don't use static data to verify your results. Don't do the query once, copy the entire response, and use that to verify future responses. This will make your tests exceedingly fragile and prone to error, not to mention tied to a very specific dataset.

Pitfall: Tying Integration to Front End

This typically takes two forms: either using the front end to verify the API data (see above for other solutions) or using the front end to retrieve some dataset dynamically to use in an API request (usually as a certain piece of data is unobtainable via API). Don't do this. This creates a needless slowdown and adds infinite complexity to your integration tests for very little payoff. The key part of integration tests is they are simple and fast. Tying your tests to the front end removes both of those benefits and muddles your tests.

If you need a piece of dynamic data that can't be obtained from an API, don't build a test that requires it. This goes into the realm of Quality Advocate: it should be considered a quality defect in the APIs. APIs should be self-sustaining and headless: they should not require input via the front-end. Request a review of your API contracts. Request that the data be surfaced by an API (or if it’s already slated to be built, don't build the test until it’s ready). This is almost less about good test design and more about good system design.  

The reverse is sometimes also attempted: using APIs to ‘speed up’ front-end automated tests. I have mixed feelings on this. On one hand, it can give a notable speed improvement to front-end tests, especially if you have a lot of data setup. On the other, you’re now no longer testing certain portions of a user workflow, which defeats the purpose. I do feel like this should be avoided by instead better structuring your front-end tests to focus on user workflows (and simplifying them).

Front-End Testing - Quagmire of Automation

Front-End Testing - Quagmire of Automation

Unit Testing: Minimum Viable Test

Unit Testing: Minimum Viable Test