Manual vs Automated Testing

Manual vs Automated Testing

Think to yourself for a moment:

  • What does Automated Quality Assurance mean to you?
  • What is quality?
  • How is quality automated?
  • What does an automated test case look like?
  • How do these things differ for manual QA?

This is where I always like to start when discussing AQA. Asking someone what they think the difference between manual and automated testing is can give you a good idea of their knowledge of AQA. It can also reveal the maturity of their QA processes in general.

I find for many people there is a strong correlation in the belief of these sentiments:

  • Manual QA is fallible - they’re only human, they can miss things.
  • AQA is simply automating manual QA - take what QA is currently doing and compile it into code.
  • Automation is the ultimate solution to this chart - it increases the ‘good’ while decreasing the ‘bad’ in some abstract sense.
  • With AQA, we can eventually eliminate our QA entirely.

The thought process is easy to follow:

  • QA is expensive.
  • Even our best QA don't 'catch all the bugs.'
  • With AQA performing testing, our QA expense will go down and we'll catch more bugs.
  • Therefore, our cost and bug count will reach zero as everything will be automated.

Therein lies a fallacy however: AQA is not automating manual QA.

QA vs QA

I believe this stems from calling both fields ‘QA’ - manual QA and automated QA - as if implying that they are the same thing only one involves robots. More realistically, we should think of them as Quality Advocates and Quality Assurance. Same abbreviation, but now with more delineation.

Manual QA are your Quality Advocates. They Advocate for a Quality product. They work closely with the designers, developers, and other stakeholders to ensure a quality product is built. This is called Validation: that the right product is built.

Automated QA are your Quality Assurance. They Assure a Quality product. They closely read the requirements of the product and work with the developers to code tests that ensures the product meets those requirements. This is called Verification: that the product that was built works.

Consider this example. You’re building an eCommerce application. A key feature of this application is the ability for people to purchase things by clicking a Checkout button, thus earning you money. You build a product that looks like the example below.

Side Note: do not allow me to design your UI.

Side Note: do not allow me to design your UI.

Quality Assurance can verify that the checkout button exists and that clicking it takes you to the payment page - the product that was built works. You write a test case that looks something like:
#navigate id=checkout
# div?
#Click div
# page=next?
You run your automated tests - and everything passes. It verified that the checkout button exists and that clicking it takes you to the next page.

But of course, there is something wrong here. Where is the checkout button? Quality Advocates can tell you that this menu fails a human element - you can't see the button. Plus, burying your checkout button under three sub-menus is a bad idea. This product will fail many tests to a human eye that a robot would never find.

“AQA is not automating manual QA”

Until we develop sufficiently advanced AI, AQA cannot tell you if someone is willing to use your product - or if it’s even usable at all. If you tell a program, an automated test, that a checkout button is buried three layers deep in your application, as long as that button exists your test will pass. It won't raise a single objection because, to it, there is nothing wrong.

Automated QA can only verify what you tell it to look for, and some things robots are really terrible at noticing. Manual QA can use your product as a person would - clicking buttons and reading menus in a distinctly human way (which includes clicking buttons in nonsensical orders). This is called Exploratory Testing and is vital to ensure a high quality product.

AQA is not magic. It is not the ‘magic mallet of quality’, James Bach attests no such thing exists. It is not the silver bullet providing orders of magnitude efficiency increases, as Fredrick Brooks would agree. AQA is a tool that can increase the quality of the end product - it’s up to us to use it correctly.

Don't think in terms of replacing manual QA with robots. Think instead of turning them into cyborgs. AQA can augment your existing QA allowing them to perform tasks that would be impossible (performance, security, or extensive regression testing) or extremely monotonous (bulk data imports, basic integration data verification) to do manually thus freeing them up to perform tasks and find bugs a robot never could.


AQA Character Reference (D&D Version)

AQA Character Reference (D&D Version)

The Milliways Defect

The Milliways Defect