Why automation isn’t King!

Automation should be King in development but it’s not and that’s generally because the standard of a lot of the suites out there is poor and that is due to unrealistic expectations of the tool by those performing the automation.

Some companies have a thousand tests, but the tests are flaky and they perform really badly, they fail now and again and no-one knows why and no-one really cares and as a result the test suites are useless and no-one really believes that automation works.

Tools like Watir and Capybara are breeding grounds for these flaky tests. They provide a level of abstraction that insulates the new automation engineer from the level of subtlety that stable automation requires.

Their level of abstraction makes people think that they can just define some elements in a page class, write a scenario in a feature file and then use nothing but a bit of watir and rspec direct in the step_defintions like this:

site.my_area.my_page.my_checkbox.when_present.click
site.my_area.my_page.my_button.when_present.click
site.my_area.my_other_page.my_field_1.when_present.set('my_value')
site.my_area.my_other_page.my_button.when_present.click
expect(site.my_area.my_important_page.h2.text_,to_eq(some_value)

But I’ve got news for you, it doesn’t work like that! Not in real life. In real life you need to do some magic behind the scenes, because maybe you’ve got some javascript still loading, or maybe this page is really slow to load (and the performance is not going to get fixed) and maybe on your crap test environment the button even needs to be pressed twice to get it to load? So you need belt and braces for pretty much most things.

 For some elements you might find you need to fall back into Selenium commands; you’ll be doing everything you can to avoid using x-path as a locator; you might even need to fall back directly into javascript to interact with the DOM. You’ll probably need to add some robust error handling for when a page hasn’t loaded due to network issues; you’ll want to do set up and tear down of data; you’ll need to be thinking about what you’re verifying and why and what you should be verifying stuff against. You might be reading in copy text files for site verification or grabbing data from the CMS API; there is a whole world of stuff that you should be doing…you should be making your tests independant; flexible; extensible; reusable; performant; robust; and self-verifying.

So why are your tests failing?

  • Using brittle selectors such as x-path
  • Using fixed sleeps instead of enhanced waits
  • Using the same test data while running your tests concurrently
  • Not having error handling in your API calls
  • Badly written automation code – not taking care with each element
  • Making your tests too long
  • Not making your suite robust enough to handle network issues
  • Failing to version your tests causing the wrong tests to run against the wrong branches

If automation was as simple as banging out straight forward lines of Watir or Capybara then everyone would do it; they’d do it quickly and it would be cheap; but it’s not that simple. It requires consideration and design and most importantly it requires TESTING!

“Flaky test suites are sloppy. They are worse than no tests at all!!”

So next time you come across flaky test suites, disable those suites immediately and fix them so that they are no longer failing intermittently. If we all take care and we all do it right then people will see that automation does work and automation will be King.

Advertisements

Please leave a comment - any questions or feedback welcome.

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s