Provide Value: The Absent-Minded Professor

Provide Value: The Absent-Minded Professor

A question for you: why did Doc Brown build a time machine?
Yes, we know why he used a Delorean. Yes, we know why he powered with nuclear energy. But why did he build it in the first place - what was the point?

I’ve inherited old AQA codebases and been assigned the task of getting them ‘working’, meaning of course ‘passing’. After a month or so of working through the test cases, I could not figure out their value. I couldn't figure out what they tested or how. I walked into my managers office and asked them what value these tests were providing. I came to find out that they didn't even know what value the tests had. They just knew that once a couple years ago someone proposed AQA, they approved it, and now there were test reports that they knew needed to be passing in some abstract sense.

I find this is most prevalent in front-end automation, where people try to emulate manual QA processes using code. Don't do that.

The Four Questions

First, we need to understand what makes a good test in general. Ask yourself these questions:

  • What does it test?
  • How does it test it?
  • How do we know when it passes?
  • What does it mean when it fails?

If you can't answer those questions, don't write the test.
 
Any assignment that doesn't answer these questions should spawn a slurry of questions from you, the test writer:

  • What value does this functionality have to the end user?
  • Where are the specification and requirements documents?
  • What is our technology stack for this functionality?
  • What’s a full end-to-end run of this functionality look like?
  • What other levels of testing are already being done and may I review them?
  • Is this functionality going to be changed or altered in the near future (if not, a manual test may be sufficient)?

The Four Answers

For many tests I’ve seen, just answering those questions at all is an improvement as it gets people thinking. However, to write truly good tests, you need to be specific in your answers. Beware vague answers or circular reasoning as these can be too open to interpretation to get meaningful results. Equally don't mix your scope: tests do a specific thing in a specific place, lower level tests are more valuable than higher level, ensure that you use the right test in the right place.

A form validation example:

Bad Form Validation Test (vague and circular answers):

  • What does it test?
    • The form on the page.
  • How does it test it?
    • Selenium to enter values.
  • How do we know when it passes?
    • Selenium will show passing tests.
  • What does it mean when it fails?
    • The form is not functioning correctly.

Notes: I really wish this was just a bad example that doesn't exist in the real world, but sadly it’s probably the most prevalent test definition out there.

The first answer is far too vague - a form on a page is far more than a form on a page: there’s the front-end, the UI, the API (if any?), the validator (single source? Custom? External service?) What’s even the point of the form?
The second answer doesn't really tell you what you’re testing at all - what values? What are you looking for?
The third answer is literally circular - what specifically is in your Assertion logic? How does Selenium know when it passes? Does it just Assert(True) on line 1 and call it a day?
And finally we can refer back to question three, forms are more than ‘just forms’.

Bad Form Validation Test (mixing scope):

  • What does it test?
    • The form on the page.
  • How does it test it?
    • Selenium enters many combinations of values on the front-end, then Java queries the database to ensure the proper tables are filled. RESTAssured is triggered on the next build to query the queue and ensure the data is waiting to be polled. Finally, we’ll look for performance at each step and measure it against the last builds results.
  • How do we know when it passes?
    • When each test has comparable results to the previous.
  • What does it mean when it fails?
    • The form is not functioning correctly.

Notes: obviously this has several of the same trappings of the first bad answers, but specifically the second answer has been fleshed out to include lots of disparate tests. It’s not wrong that those tests probably need to be run, but they shouldn't be run as part of a singular test or inside a singular test class. Break these out and break them down.

For another example of mixing scope, consider when you have lots of tests but they all overlap. Consider a front-end test that checks the database and API, an API test that goes to a front-end form, and a Unit test that also checks the Alert message logic. If you do each of those individually, there’s no need to mix them together.  

Good Form Validation Test (part 1: Unit test):

  • What does it test?
    • Unit test. Ensures the form validator function accepts and rejects appropriate data sets. This function will be used by all higher-level functions posting to this table. This function ensures the database stays clean. Bad data can result in any number of issues at higher levels in the system.
  • How does it test it?
    • JUnit. Runs a series of data sets (as many combinations as possible) through the form validator function and ensures it returns per the requirements specifications.
  • How do we know when it passes?
    • The appropriate response (true or false, if false the fields that are invalid) is returned.
  • What does it mean when it fails?
    • The validator is incorrect. Check the regex function for any flaws.

Notes: Unit tests are simple, and thus so will your answers be. That doesn't mean they don't have value: because we do our input combination tests at this level, we won't need to do so again at higher levels. Here we are strictly concerned with the validation logic: does it accept and reject the appropriate inputs?

Good Form Validation Test (part 2: API test):

  • What does it test?
    • API test. Checks the queue to ensure it contains proper form validation data. The data in the queue should conform to the standards in the requirements specifications. This queue feeds into our other integrated systems to keep them up-to-date. If the data fails to make it up stream, those systems will be out of sync and the end user may be using bad data.
  • How does it test it?
    • The POST API is first used to inject data into the queue (POST API is tested as part of an earlier suite including negative tests to kick out bad data, if that suite fails this test is invalid). A GET is then performed on the queue to ensure the data propagates appropriately.  
  • How do we know when it passes?
    • 200 status code along with the proper data. Since this is a shared queue, there may be other ‘aberrant’ data that will be ignored, we only look for the data we inject.
  • What does it mean when it fails?
    • It could mean several things:
      • The API may be down (consider checking a Status API first and throwing a special error)
      • The data is correct, but the status code is wrong. There may be an issue with the server.
      • The status code is correct, but no or incorrect data is returned. Something could have failed with the injection (check data set and POST API) or the queue is not picking up on changes.

Notes: For this example we’re assuming the form is backed by an API and that the data, once submitted, will be pushed to a queue - this is very good specifications to gather before building your tests. The only thing we care about here is that the data appears in the queue. We already know the validator is working in the Unit test. We also know the POST API is working via another test in a different suite, thus it’s noted that this test is dependent upon that one.

Good Form Validation Test (part 3: front-end test):

  • What does it test?
    • Front-end test. Form gives an Success alert message to the user with valid data, else gives an Error alert message with invalid data. The Error message lists which field failed validation. This feedback is important to the user as confirmation of successful data submission.
  • How does it test it?
    • Selenium, from an end-user perspective. Enter in various combinations of data (as defined in the specifications) and ensure the appropriate messages are displayed (one success, one failure per field, and two combination errors).
  • How do we know when it passes?
    • We are able to see the Success and Error messages accordingly. The alert Divs will not be visible until displayed and will reset with each attempt. We will check each iteration to ensure that you won't receive a Success AND Error message (as that should be impossible). The combination ensures that it picks up multiple failing fields in one alert.
  • What does it mean when it fails?
    • It could mean several things:
      • Selenium may have issues accessing the DOM (automation failure)
      • The DOM changed and even though the Alert appeared Selenium could not find it (automation failure)
      • The alerts did not appear, but the data was appropriately validated (front-end failure). This can be manually verified by checking the data elsewhere in the system.
      • The data was not appropriately validated (back-end failure). Check the unit tests. If those pass, check that the form is using the correct validator function.

Notes: Notice we only care about the front-end, which in this case is the Alert message. We make no mention of trying all possible combinations of values, because we don't need to - that was handled in the unit test. We’re not checking the queue because the API test does that. We also have several things that we may want to check if this test fails, starting with the possibility that Selenium itself is broken - alas this is just the reality of front-end tests.

The Fifth Question

The first four questions shows what value the test provides - what we are testing and why. With AQA we need to dig deeper. Automation is a step beyond regular testing; it requires a considerable amount more effort plus maintenance costs down the road - you need to be able to justify that or else you’ll just find yourself with sunk costs and no real benefit.

Thus, ask:
“What benefit does Automation provide to this test?”

The first initial response is, ‘yes, because after 50 runs we’ll be saving time vs manually!’ That is, most often, an incorrect answer - don't judge the value of an automation test by how many runs it will take to save you time. Very rarely do you run the same test manually 50 times - or even 10 times - if we did, suicide rates for QA would be far higher.  

The second rebound response is, ‘Well, surely it’ll increase the reliability of our tests, it’s more accurate and reliable then human eyes!’, in which case I implore your manual QA to seek employment elsewhere as you clearly do not give them enough credit for the work they do. QA is so much more then comparing actual vs expected results, and it’s quite rare that QA is comparing lines in a spreadsheet all day.

Instead, consider what Automation brings to the table that manual QA cannot. Certain things in your system would be difficult (APIs) or impossible (performance, units) to test manually. There may be certain areas of your project that do, in fact, need a higher degree of confidence (logins, user creation) that are worth automating so you can use larger and more varied data sets. And in fact there may be parts of the system that are highly intertwined and may inadvertently be affected by a change elsewhere in the system (yes, this is bad system design, but it happens often), in which case having a static automation test that’s run every build may indeed pay off.

Don't build automated tests because you can. Build them because they provide you with some value that other methods cannot.

Make Tests, Not Tasks

Make Tests, Not Tasks

Story Time: Dont Merge the Conflux

Story Time: Dont Merge the Conflux