Scenario Outline Madness

One of my pets hates is seeing test suites which perform really badly because of the overuse of the Scenario Outline construct.

Seriously, you need to think about why you are using them? Do you really need to run this entire scenario including the Background Steps (if you have them) all over again?

Here’s an example of where it is not required and how to change it:

Feature: Personal Details

Scenario Outline: Verify that a user is prevented from entering an invalid phone number

Given I am logged in

When I goto my personal details

And I enter an invalid  <telephone_number>

Then I see an <error_message>

Examples:

|telephone_number| error_message|

|abd|Please enter only numbers|

|12|Please enter a valid telephone number|

|44++ 22|Please enter a valid telephone number|

——————————

Now why would you want to re-login every time here? Why would you not do this:

Feature: Personal Detail

Scenario: Verify a user is prevented from entering an invalid phone number.

Given I am logged in

When I goto my personal details

Then on entering an invalid telephone number I see an error message :

|telephone_number| error_message|

|abd|Please enter only numbers|

|12|Please enter a valid telephone number|

|44++ 22|Please enter a valid telephone number|

——————————

I mean ok, I accept that two actions are included within one step i.e. edit the telephone number and verify the error message, but we have multiple actions in one step all the time …for example, ‘Given I am logged in’ often covers a whole host of actions e.g. visit the home page; click on login; visit the login page; enter your login details; submit your login details; verify that you’re logged in, so it’s fine – honestly it is!!

I can understand why people are reluctant to this, it down to semantics and the constraints people feel when using Gherkin, it’s those WHEN and THEN keywords, because naturally you’d want to say: “When I enter an invalid telephone number then I see an error message” and people are shit scared of that “then” right there in the middle and become convinced that there is no other way to write this test than as a Scenario Outline.

Just making this simple change to this test will make it run 3 times faster – remember, you’re not testing login here,  you are testing that the correct error message is displayed – therefore that is the only bit you need to re-run!!

 

So, when do you use a Scenario Outline?

So, when do you use Scenario Outline? Well, I believe it should only be used when you actually need to rerun the whole scenario. So, for example, let’s say that you want to check that a users details have persisted after logout, then you do actually need to re-run everything, so you do need to use a Scenario Outline.

Scenario Outline: User details have persisted after logout

Given I have logged in

And I have set my personal details <name>, <telephone_number>, <date_of_birth>

And then logged out

When I log back in

Then my personal details are still set to <name>, <telephone_number>, <date_of_birth>

Examples:

|name|telephone_number|date_of_birth|

|John|0772219999|18/01/1990|

|Jo|077221991|19/02/1991|

——————————

 

Sublime Text 2 with Cucumber and Gherkin formatting

I normally use RubyMine for my calabash automation code but it’s pretty memory intensive so I thought I’d try something a bit more lightweight (and free)..I choose Sublime Text 2 and installed the cucumber-sublime bundle from drewada on Github.

It works a treat – the only annoying this is that when you hit tab it always tries to autocomplete – this is particularly annoying when creating tables in your feature files.

The cucumber sublime bundle does everything that you’d expect to do i.e. it makes both feature files and step files easy to read; it prompts existing steps and autocompletes variables etc.

Note: it’s also a nifty ruby editor, bash editor and YAML editor. I find myself only opening up RubyMine now when I need to debug something or do something like find other usages of methods etc.

Here’s a little screen shot of how it formats a Gherkin in a Feature file:

cucumber_text_editor

Installing Appium with Ruby bindings – moving from Calabash – Part 1

If you read my blog you will know that my test automation tool of choice for mobile applications is Calabash using Ruby. However there is definitely a shift in the market towards Appium as the preferred tool of choice for mobile app testing and towards Java as the preferred language of choice for a Software Developer In Test (SDET).

Until now I have avoided Appium as my initial results with it a couple of years ago showed it to be “buggy” and somewhat unreliable – for example a test would execute successfully 9 times and then fail on the 10th attempt for no known reason…but maybe it’s time to give it a go again.

This post shows what I needed to do to get Appium with Ruby Bindings set up and running on my mac

 To Install Appium with Ruby Bindings

  1. Install the appium console
gem install --no-rdoc --no-ri appium_console

2.  Install the flaky gem (f you don’t already have it)

gem install --no-rdoc --no-ri flaky

3.  Do  a brew update (I’m assuming here that you already have brew installed, if not take a look at the calabash set up post https://webdevjourney.wordpress.com/2013/09/30/calasba-sh-for-testing-ios-apps-part-1/ to see how to install homebrew

brew update

[Note – if you get a HOMEBREW_CORE_FORMULA_REGEX error here – just run brew update again (it’s a known brew issue)

4. Upgrade nodesj as follows

brew upgrade node

5. Install grunt

npm install -g grunt grunt-cli

6. I am assuming that you have ant and maven already installed – you may want to upgrade them.

7. Clone the appium git repo

git clone git://github.com/appium/appium.git

8. Ensure you have an Xcode version greater than 4.6.3 installed. See this post for a quick way to check your version of Xcode from the command line: https://webdevjourney.wordpress.com/2015/10/15/find-your-xcode-version-using-terminal/

9. cd into your appium folder (from the repo clone that you did in step 7 and execute

./reset.sh

Note – I encountered a problem here – i had the following output:

* Configuring Android bootstrap
---- FAILURE: reset.sh exited with status 1 ----

Please see the following post on how to resolve this issue:

10. Finally run:

 grunt authorize

11. You are then ready to launch Appium by running:

node .

You will then see a message like this:

info: Welcome to Appium v1.4.13 (REV 96cfaef4aafa2693b4f0630b9d44ca7f0f6ed369)

In the next article I look at running the ruby console to query your app.