Coping with changing object id’s

Sometimes you might be running automation on more than one branch on code. For example, it’s pretty common to be running some smoke tests on the ‘Development’ or ‘Integration’ branch and also be running tests on ‘Master’. So how do you cope when an object’s id is set to one thing on one branch and another thing on another branch. Well it’s easy – here’s an example using calabash and also one using watir

If I have an input field with an id of “important_x01” and my tests reference the field in a method as follows:

 

//calabash example
def important_info_input
   query("* marked: 'important_x01'")
end

//watir example
def important_info_input
  @browser.text_field(id: 'important_x01')
end

 

Then let’s say that there is a change to that object id on the integration/development branch and it is now called “i_info”. So, until this change is promoted to Master this object will be known on different branches as two different things. Well the easy way to handle this is to allow the code to return whichever of these objects exists:

//calabash example
def import_info_input
 if element_exists("* marked: 'important_x01'")
    query("* marked: 'important_x01'")
 else
    query("* marked:' i_info'")
 end
end  

// watir example
def important_info_input
 if @browser.text_field(id: 'important_x01').exist?
   @browser.text_field(id: 'important_x01')
 else
   @browser.text_field(id: 'i_info')
 end
end

 

Advertisements

Why automation isn’t King!

Automation should be King in development but it’s not and that’s generally because the standard of a lot of the suites out there is poor and that is due to unrealistic expectations of the tool by those performing the automation.

Some companies have a thousand tests, but the tests are flaky and they perform really badly, they fail now and again and no-one knows why and no-one really cares and as a result the test suites are useless and no-one really believes that automation works.

Tools like Watir and Capybara are breeding grounds for these flaky tests. They provide a level of abstraction that insulates the new automation engineer from the level of subtlety that stable automation requires.

Their level of abstraction makes people think that they can just define some elements in a page class, write a scenario in a feature file and then use nothing but a bit of watir and rspec direct in the step_defintions like this:

site.my_area.my_page.my_checkbox.when_present.click
site.my_area.my_page.my_button.when_present.click
site.my_area.my_other_page.my_field_1.when_present.set('my_value')
site.my_area.my_other_page.my_button.when_present.click
expect(site.my_area.my_important_page.h2.text_,to_eq(some_value)

But I’ve got news for you, it doesn’t work like that! Not in real life. In real life you need to do some magic behind the scenes, because maybe you’ve got some javascript still loading, or maybe this page is really slow to load (and the performance is not going to get fixed) and maybe on your crap test environment the button even needs to be pressed twice to get it to load? So you need belt and braces for pretty much most things.

 For some elements you might find you need to fall back into Selenium commands; you’ll be doing everything you can to avoid using x-path as a locator; you might even need to fall back directly into javascript to interact with the DOM. You’ll probably need to add some robust error handling for when a page hasn’t loaded due to network issues; you’ll want to do set up and tear down of data; you’ll need to be thinking about what you’re verifying and why and what you should be verifying stuff against. You might be reading in copy text files for site verification or grabbing data from the CMS API; there is a whole world of stuff that you should be doing…you should be making your tests independant; flexible; extensible; reusable; performant; robust; and self-verifying.

So why are your tests failing?

  • Using brittle selectors such as x-path
  • Using fixed sleeps instead of enhanced waits
  • Using the same test data while running your tests concurrently
  • Not having error handling in your API calls
  • Badly written automation code – not taking care with each element
  • Making your tests too long
  • Not making your suite robust enough to handle network issues
  • Failing to version your tests causing the wrong tests to run against the wrong branches

If automation was as simple as banging out straight forward lines of Watir or Capybara then everyone would do it; they’d do it quickly and it would be cheap; but it’s not that simple. It requires consideration and design and most importantly it requires TESTING!

“Flaky test suites are sloppy. They are worse than no tests at all!!”

So next time you come across flaky test suites, disable those suites immediately and fix them so that they are no longer failing intermittently. If we all take care and we all do it right then people will see that automation does work and automation will be King.

Scenario Outline Madness

One of my pets hates is seeing test suites which perform really badly because of the overuse of the Scenario Outline construct.

Seriously, you need to think about why you are using them? Do you really need to run this entire scenario including the Background Steps (if you have them) all over again?

Here’s an example of where it is not required and how to change it:

Feature: Personal Details

Scenario Outline: Verify that a user is prevented from entering an invalid phone number

Given I am logged in

When I goto my personal details

And I enter an invalid  <telephone_number>

Then I see an <error_message>

Examples:

|telephone_number| error_message|

|abd|Please enter only numbers|

|12|Please enter a valid telephone number|

|44++ 22|Please enter a valid telephone number|

——————————

Now why would you want to re-login every time here? Why would you not do this:

Feature: Personal Detail

Scenario: Verify a user is prevented from entering an invalid phone number.

Given I am logged in

When I goto my personal details

Then on entering an invalid telephone number I see an error message :

|telephone_number| error_message|

|abd|Please enter only numbers|

|12|Please enter a valid telephone number|

|44++ 22|Please enter a valid telephone number|

——————————

I mean ok, I accept that two actions are included within one step i.e. edit the telephone number and verify the error message, but we have multiple actions in one step all the time …for example, ‘Given I am logged in’ often covers a whole host of actions e.g. visit the home page; click on login; visit the login page; enter your login details; submit your login details; verify that you’re logged in, so it’s fine – honestly it is!!

I can understand why people are reluctant to this, it down to semantics and the constraints people feel when using Gherkin, it’s those WHEN and THEN keywords, because naturally you’d want to say: “When I enter an invalid telephone number then I see an error message” and people are shit scared of that “then” right there in the middle and become convinced that there is no other way to write this test than as a Scenario Outline.

Just making this simple change to this test will make it run 3 times faster – remember, you’re not testing login here,  you are testing that the correct error message is displayed – therefore that is the only bit you need to re-run!!

 

So, when do you use a Scenario Outline?

So, when do you use Scenario Outline? Well, I believe it should only be used when you actually need to rerun the whole scenario. So, for example, let’s say that you want to check that a users details have persisted after logout, then you do actually need to re-run everything, so you do need to use a Scenario Outline.

Scenario Outline: User details have persisted after logout

Given I have logged in

And I have set my personal details <name>, <telephone_number>, <date_of_birth>

And then logged out

When I log back in

Then my personal details are still set to <name>, <telephone_number>, <date_of_birth>

Examples:

|name|telephone_number|date_of_birth|

|John|0772219999|18/01/1990|

|Jo|077221991|19/02/1991|

——————————

 

Mobile App Automation and Localisation

A common pitfall when beginning automation is to use the text for objects instead of their id’s.

For example, you have a button which says “Go to Dashbord”, so you have something in your class which looks
a bit like this:

def go_to_dashboard_btn
 "widget.type marked:'Go to Dashboard'"
end

But you also have a screen which asks the user which language they would like to use the app in and you select
a language which is not English. Suddenly your “Go to Dashboard” button “does not exist”. Why? Well because it’s the
text string has been localised and the button now says something like: “Vamos a Dashboard” (forgive my spelling).
You can see the different text values if oyu query the object while the app is using the different localised string
So if you query this object (query(“widget.type.button”) when using the app as an English user you will see:

 
 [0] {
 "id" => "goto_dashboard_btn",
 "enabled" => true,
 "contentDescription" => nil,
 "text" => "Go to Dashboard",
 "visible" => true,
 "tag" => nil,
 "description" => "widget.type.button{426535e0 V.ED...",
 "class" => "widget.type.button",
 "rect" => {
 "center_y" => 707,
 "center_x" => 186,
 "height" => 64,
 "y" => 675,
 "width" => 118,
 "x" => 127
 }

And if you query this object (query(“widget.type.button”) when using the app as an Spanish user you will see:

[0] {
 "id" => "goto_dashboard_btn",
 "enabled" => true,
 "contentDescription" => nil,
 "text" => "Vamos a Dashboard",
 "visible" => true,
 "tag" => nil,
 "description" => "widget.type.button{426535e0 V.ED...",
 "class" => "widget.type.button",
 "rect" => {
 "center_y" => 707,
 "center_x" => 186,
 "height" => 64,
 "y" => 675,
 "width" => 118,
 "x" => 127
 } 
 
 
As you can see the text string changes, so you need to make sure that you use the accessibility id.
So now your methods look like this:
def go_to_dashboard_btn
 "widget.type marked:'goto_dashboard_btn'"
end

If you make sure that the developers provide accessibility ids for everything then localisation won’t affect your
automated tests.

Note this general approach applies to both Calabash and Appium (and other tools)

Appium and Cucumber – Managing different iOS and device configurations

Usually you will want to run your cucumber appium test suite on a few different configurations.

For example, you may want to run it on an iPhone 6 9.0 simulator and you also may want to run it on an iPad Air 8.4 simulator. So how do you do this?
Well it’s easy. First you use your cucumber.yml file to list the different configurations. So if we use our example above
we could create two profiles, lets call them iphonesim and ipadsim.
They would look like this in the cucumber.yml file:

ipadsim: IDEVICENAME='ipad simulator'
iphonesim: IDEVICENAME='iphone simulator'

We then need to create two different appium.txt files. One for the ipadsim configuration and one for ipadsim configuration. For example, the iphonesim appium.txt one would look like this.

[caps]
platformName = "ios"
deviceName = "iPhone 6"
platformVersion = "9.0"
app = "../../../apps/TestApp/build/release-iphonesimulator/myApp.app"

[appium_lib]
sauce_username = false
sauce_access_key = false

Save theiphonesim appium.txt file in a folder under Features/Support called “iphonesim” and theipadsim appium.txt under in a folder under Features/Support called “ipadsim”.

Then in env.rb tell Appium where to go to pick up the correct appium.txt based on which profile you have started your tests with.
Do this by adding something like this to your env.rb file

if ENV['IDEVICENAME']=='ipad simulator'
caps = Appium.load_appium_txt file: File.expand_path("./../ipadsim/appium.txt", __FILE__), verbose: true
elsif ENV['IDEVICENAME']=='iphone simulator'
caps = Appium.load_appium_txt file: File.expand_path("./../iphonesim/appium.txt", __FILE__), verbose: true
else
caps = Appium.load_appium_txt file: File.expand_path('./', __FILE__), verbose: true
end

Lastly, to run with your selected profile, cd into your cucumber folder and do the following:
To run the iPhone configuration execute:

cucumber -p iphonesim

To run the iPad configuration execute:

cucumber -p ipadsim

What does this actually do?

Well, if we use “cucumber -p iphonesim” as an example, firstly, the -p flag indicates to cucumber that you want to use a specific profile, so it looks up the profile in the cucumber.yml and sees IDEVICENAME=’ipad simulator’.
Cucumber then looks at the env.rb file and sees that when IDEVICENAME=’ipad simulator’ it should use the appium.txt file in the phonesim folder.
Appium then launches the simulator with the capabilites defined in the Features/Support/iphonesim/appium.txt.

This allows us to switch around configurations without changing anything other than our initial cucumber -p command.

Neat eh?

Installing Appium with Ruby bindings – moving from Calabash – Part 3

Building an IOS app for Appium

For those coming across from calabash, you will be used to having to create a duplicate -cal scheme to embed the calabash.framework into your app. Well, thankfully with Appium there is no need to do this at all. Simply open your Xcode project and build the app. That’s it – nothing special required.

Benefit of Appium over Calabash

The major benefit of Appium here is that the app that you are testing is exactly the same app as you release into production – there is no linked automation framework. This is a definite advantage over calabash.

Running Appium

All you need to do now is open 2 tabs terminal tabs.

In the first one run :

node .

Once this has started up, in the second tab run:

 arc

You now have a fully functioning Appium environment.

Important note: I strongly advise that  you use bundler with gemfiles if you plan to have appium with ruby bindings installed on the same machine as calabash. If you don’t you can end up with horrible things happening.

If you are experiencing gem issues I whole heartedly recommend this post  Installing ruby the Correct Way by cbednarski. It also has a great Appendix on cleaning your environment before re-installing your gems.

Installing Appium with Ruby bindings – moving from Calabash – Part 2

In part 1 we looked at the set up of Appium when moving from Calabash. In the next articles we look at getting the ruby console (ARC) working to query your app, but first we have to generate an appium.txt file. This article will help you resolve the annoying App doesn’t exist error message relating to appium.txt when running the arc command.

Create the appium.txt file

The next step is to create the appium.txt file – simple right? Well when I first tried to this i hit a couple of problems.

The appium.txt tells appium what platform you are automating and what simulator you wish to use  and where your app is (amongst other things). Although it is nicely explained in the Appium tutuorial it fails to point out three things

  1.   You can create a sample empty appium.txt using the command
 arc setup ios

2.   Not everyone likes using nano to create files but if you use TextEdit to create / edit the appium.txt file you will end up with the wrong kind of double quotes and later on when you come to run the ARC (ruby console) you’ll get appium.txt errors. If this happens copy the correct double quoted from the file generated by the arc setup ios command.

3. When adding the path to you app, appium will automatically prefix it with the path to the folder where are running the arc command from i.e. it is a relative path, not an absolute path. if you use the absolute path to your app you will get an error something like this:

 App doesn't exist

My appium.txt looks like this – you can use it as an example:

[caps]
platformName = "ios"
platformVersion = "9.0"
deviceName ="iPhone 5s"
app = "./myAppName.app"
autoAcceptAlerts = "true"
waitForAppScript = "true"

[appium_lib]
sauce_username = ""
sauce_access_key = ""

I’ll explain the autoAcceptAlerts and waitForAppScript values in another article – running the Appium Ruby Console (arc).

In the next article i’ll explain the difference between building an app for calabash and building an app for Appium.

Installing Appium with Ruby bindings – moving from Calabash – Part 1

If you read my blog you will know that my test automation tool of choice for mobile applications is Calabash using Ruby. However there is definitely a shift in the market towards Appium as the preferred tool of choice for mobile app testing and towards Java as the preferred language of choice for a Software Developer In Test (SDET).

Until now I have avoided Appium as my initial results with it a couple of years ago showed it to be “buggy” and somewhat unreliable – for example a test would execute successfully 9 times and then fail on the 10th attempt for no known reason…but maybe it’s time to give it a go again.

This post shows what I needed to do to get Appium with Ruby Bindings set up and running on my mac

 To Install Appium with Ruby Bindings

  1. Install the appium console
gem install --no-rdoc --no-ri appium_console

2.  Install the flaky gem (f you don’t already have it)

gem install --no-rdoc --no-ri flaky

3.  Do  a brew update (I’m assuming here that you already have brew installed, if not take a look at the calabash set up post https://webdevjourney.wordpress.com/2013/09/30/calasba-sh-for-testing-ios-apps-part-1/ to see how to install homebrew

brew update

[Note – if you get a HOMEBREW_CORE_FORMULA_REGEX error here – just run brew update again (it’s a known brew issue)

4. Upgrade nodesj as follows

brew upgrade node

5. Install grunt

npm install -g grunt grunt-cli

6. I am assuming that you have ant and maven already installed – you may want to upgrade them.

7. Clone the appium git repo

git clone git://github.com/appium/appium.git

8. Ensure you have an Xcode version greater than 4.6.3 installed. See this post for a quick way to check your version of Xcode from the command line: https://webdevjourney.wordpress.com/2015/10/15/find-your-xcode-version-using-terminal/

9. cd into your appium folder (from the repo clone that you did in step 7 and execute

./reset.sh

Note – I encountered a problem here – i had the following output:

* Configuring Android bootstrap
---- FAILURE: reset.sh exited with status 1 ----

Please see the following post on how to resolve this issue:

10. Finally run:

 grunt authorize

11. You are then ready to launch Appium by running:

node .

You will then see a message like this:

info: Welcome to Appium v1.4.13 (REV 96cfaef4aafa2693b4f0630b9d44ca7f0f6ed369)

In the next article I look at running the ruby console to query your app.

Appium install failure – reset.sh exited with status 1

PROBLEM

When trying to complete the install of Appium by running

./reset.sh

I was seeing an error which said

 reset.sh exited with status 1

I re-ran the script for ios only and it completed successfully…see below:

 ./reset.sh --ios
---- reset.sh completed successfully ----

I then tried running the script for android only and turned on verbose logging – it failed as follows:

 ./reset.sh --android --verbose

Error: Target id is not valid. Use 'android list targets' to get the target ids.
Warning: Setup cmd /Users/user/Android/sdk/tools/android failed with code 1 Use --force to continue.

SOLUTION

This error occurred because the reset.sh script has a hard coded reference to the Android SDK Level 19 API and I didn’t have that version of the API installed.
To rectify this I installed the SDK 19 API and then re-ran the script for android. It completed successfully..see below:

 ./reset.sh --android --verbose

---- reset.sh completed successfully ----