Automation should be King in development but it’s not and that’s generally because the standard of a lot of the suites out there is poor and that is due to unrealistic expectations of the tool by those performing the automation.
Some companies have a thousand tests, but the tests are flaky and they perform really badly, they fail now and again and no-one knows why and no-one really cares and as a result the test suites are useless and no-one really believes that automation works.
Tools like Watir and Capybara are breeding grounds for these flaky tests. They provide a level of abstraction that insulates the new automation engineer from the level of subtlety that stable automation requires.
Their level of abstraction makes people think that they can just define some elements in a page class, write a scenario in a feature file and then use nothing but a bit of watir and rspec direct in the step_defintions like this:
site.my_area.my_page.my_checkbox.when_present.click site.my_area.my_page.my_button.when_present.click site.my_area.my_other_page.my_field_1.when_present.set('my_value') site.my_area.my_other_page.my_button.when_present.click expect(site.my_area.my_important_page.h2.text_,to_eq(some_value)
So why are your tests failing?
- Using brittle selectors such as x-path
- Using fixed sleeps instead of enhanced waits
- Using the same test data while running your tests concurrently
- Not having error handling in your API calls
- Badly written automation code – not taking care with each element
- Making your tests too long
- Not making your suite robust enough to handle network issues
- Failing to version your tests causing the wrong tests to run against the wrong branches
If automation was as simple as banging out straight forward lines of Watir or Capybara then everyone would do it; they’d do it quickly and it would be cheap; but it’s not that simple. It requires consideration and design and most importantly it requires TESTING!
“Flaky test suites are sloppy. They are worse than no tests at all!!”
So next time you come across flaky test suites, disable those suites immediately and fix them so that they are no longer failing intermittently. If we all take care and we all do it right then people will see that automation does work and automation will be King.