Thursday 2 May 2013

Principles of painless test automation #1 - LSP

Test automation causes massive headaches. Period. I'm pretty sure that every one of us who ever tried it knows what I mean. We end up with way too many full-stack tests, way too much reliance on Selenium/Webdriver and way too many business people thinking that "everything is equally important".
The number of ways to tackle the pain depends on the number of people involved with the solution. Unfortunately there is no ultimate solution to test automation. There are however principles that we can follow. Just as with programming. If we keep our code clean, decoupled and readable, we can eventually focus on the less technical aspects of our work.

In the Principles of painless test automation series I will share my experience about building automation frameworks. I will cover programming language agnostic thoughts, principles and solutions.


The most common pitfall we tend to fall in is replicating business logic in our test suite. Let that be unit tests that calculate the correct result of something, or integration test code that is limited by what it represents.
A trivial example would be testing addition with a simple unit test.

assertEquals(4, add(2,2))

Simple, right?
Of course we like to be sure that the add method does not simply return 4, so we triangulate and add more examples. After some more assertions like above, sometimes we get drawn into writing a small loop where for example we go from 1 to 10, and check that adding the numbers 1+1, 2+2, 3+3, etc. always results in what's expected of it. But we're lazy. Most of us will definitely not bother writing the solution into the examples. Instead, we will create a function that will calculate the sums for us and return it, so that we could use it as reference.
Extracting methods results in less and readable code. Sometimes duplicating so isolated and small pieces of functionality could be the right thing to do. The pain begins when the mirroring does not stop at the bottom of the testing pyramid. Note, that the previous example refers only to unit testing.

Imagine we're testing an application that has some guest users, registered users and admin users. Each group has different permissions on our website. The guests can view the items, but cannot purchase them. The registered users have a shopping cart and payment methods. The admins have the permission to upload new items into the store, but cannot purchase.
The number of ways to create a test automation framework to verify such product's correctness depends on the number of teams implementing them. Each solution will be unique to the domain and to the people creating them. I've seen one thing in common though. Each framework had the notion of guests, registered users and admins. Most had implemented the page object model quite successfully to an extent, and everything was working perfectly.

One day, a new member joined on of these teams. She spent a massive time getting familiar with the product and the company. She got assigned to one of the delivery teams as a tester, so on her second week she started playing around with the product. Her job as a tester was to advise the programmers in charge of the test automation on what tests to write. She spent days clicking about, doing some exploratory testing and examining the existing tests.
On the third week of her new job, she asked something quite naive.
"So I see that you don't have any tests that would show what the admin interface does when a normal user tries to use it. Can we add some tests around that?"The developers nodded in appreciation, because they indeed did not think of that scenario. They even agreed that it is a very important thing, and they should check that an unauthorized personnel cannot carry out an admin action. So they went back to code-land and started implementing the tests, which led them to a very unusual realization.
Their implementation of the admin page objects did only allow the use of the AdminUser, as their RegisterdUser could only use pages that were accessible by a logged in user.
Within minutes, they realized that their whole model of the product is constraining them in performing a seemingly trivial test.

So what did they get wrong? What caused their pain?
  • The first smell was the appearance of business roles within the testing framework.
  • The second indicator might be that the permissions of each role was replicated by the way the objects were composed.
Let's just stop here for a moment and realize something.
One of the key elements of the S.O.L.I.D. principles in code is the Liskov Substituion Principle. It seems tricky to grasp at first, but Uncle Bob has a brilliant analogy that helps us with understanding it.
He starts his story by explaining a situation when programmers have a Rectangle object. When they are asked to create a Square object, they extend the Square from the Rectangle because we know from geometry that a square is a type of rectangle. His words follow:

The problem is this is not a Rectangle. It’s a piece of code! It’s a representation of a Rectangle. Representatives do not share the relationships of the things they represent. For example, imagine 2 people who are getting divorced. Each one of them has a lawyer who represents them. Its very unlikely those 2 lawyers themselves are getting divorces because the representatives of things do not share the relationships of the things they represent!

So what can we learn from this?

The testing framework acts as a representative of a user of the product, and to some extent it represents the product towards those users by the page object model.
Therefore the users and the pages should not adhere to the same relationship as they do within the product. 

But there is a difference between a guest, a registered and an admin users!
Yes, indeed! We are so used to modelling objects in code, that sometimes we just create them for the sake of object creation. We found that the only difference between users of our system is the credentials they use. Nothing less, nothing more. We could still create named instances of the User object in our test code though, but from this point they will be dead simple named value objects which only differ in their names and credentials they hold.
The business action representations like the page objects will have no notion of the distinction, they will operate with any User implementation.

The ability for a user to carry out an action should be decided by the product, and not the testing framework.

I've assembled a little framework that helps me to achieve what the article above is about. Check it out at:

1 comment :

  1. Hi,

    Nice well thought out article on the pains of Test Automation. Looking forward to further articles


Note: only a member of this blog may post a comment.