Coping with changing object id’s

Sometimes you might be running automation on more than one branch on code. For example, it’s pretty common to be running some smoke tests on the ‘Development’ or ‘Integration’ branch and also be running tests on ‘Master’. So how do you cope when an object’s id is set to one thing on one branch and another thing on another branch. Well it’s easy – here’s an example using calabash and also one using watir

If I have an input field with an id of “important_x01” and my tests reference the field in a method as follows:

 

//calabash example
def important_info_input
   query("* marked: 'important_x01'")
end

//watir example
def important_info_input
  @browser.text_field(id: 'important_x01')
end

 

Then let’s say that there is a change to that object id on the integration/development branch and it is now called “i_info”. So, until this change is promoted to Master this object will be known on different branches as two different things. Well the easy way to handle this is to allow the code to return whichever of these objects exists:

//calabash example
def import_info_input
 if element_exists("* marked: 'important_x01'")
    query("* marked: 'important_x01'")
 else
    query("* marked:' i_info'")
 end
end  

// watir example
def important_info_input
 if @browser.text_field(id: 'important_x01').exist?
   @browser.text_field(id: 'important_x01')
 else
   @browser.text_field(id: 'i_info')
 end
end

 

Why automation isn’t King!

Automation should be King in development but it’s not and that’s generally because the standard of a lot of the suites out there is poor and that is due to unrealistic expectations of the tool by those performing the automation.

Some companies have a thousand tests, but the tests are flaky and they perform really badly, they fail now and again and no-one knows why and no-one really cares and as a result the test suites are useless and no-one really believes that automation works.

Tools like Watir and Capybara are breeding grounds for these flaky tests. They provide a level of abstraction that insulates the new automation engineer from the level of subtlety that stable automation requires.

Their level of abstraction makes people think that they can just define some elements in a page class, write a scenario in a feature file and then use nothing but a bit of watir and rspec direct in the step_defintions like this:

site.my_area.my_page.my_checkbox.when_present.click
site.my_area.my_page.my_button.when_present.click
site.my_area.my_other_page.my_field_1.when_present.set('my_value')
site.my_area.my_other_page.my_button.when_present.click
expect(site.my_area.my_important_page.h2.text_,to_eq(some_value)

But I’ve got news for you, it doesn’t work like that! Not in real life. In real life you need to do some magic behind the scenes, because maybe you’ve got some javascript still loading, or maybe this page is really slow to load (and the performance is not going to get fixed) and maybe on your crap test environment the button even needs to be pressed twice to get it to load? So you need belt and braces for pretty much most things.

 For some elements you might find you need to fall back into Selenium commands; you’ll be doing everything you can to avoid using x-path as a locator; you might even need to fall back directly into javascript to interact with the DOM. You’ll probably need to add some robust error handling for when a page hasn’t loaded due to network issues; you’ll want to do set up and tear down of data; you’ll need to be thinking about what you’re verifying and why and what you should be verifying stuff against. You might be reading in copy text files for site verification or grabbing data from the CMS API; there is a whole world of stuff that you should be doing…you should be making your tests independant; flexible; extensible; reusable; performant; robust; and self-verifying.

So why are your tests failing?

• Using brittle selectors such as x-path
• Using fixed sleeps instead of enhanced waits
• Using the same test data while running your tests concurrently
• Not having error handling in your API calls
• Badly written automation code – not taking care with each element
• Making your tests too long
• Not making your suite robust enough to handle network issues
• Failing to version your tests causing the wrong tests to run against the wrong branches

If automation was as simple as banging out straight forward lines of Watir or Capybara then everyone would do it; they’d do it quickly and it would be cheap; but it’s not that simple. It requires consideration and design and most importantly it requires TESTING!

“Flaky test suites are sloppy. They are worse than no tests at all!!”

So next time you come across flaky test suites, disable those suites immediately and fix them so that they are no longer failing intermittently. If we all take care and we all do it right then people will see that automation does work and automation will be King.

Cannot see Android device via ADB

So, the plug came out of the back of the CI server…and at the same time 2 of the Samsung phones upgraded themselves and I had switched from one usb hub to another and replaced the usb cables….and suddenly I couldn’t see the Samsung S4 mini or the Samsung S5 via adb and all my calabash automation tests in jenkins were failing and the world was collapsing around my ears…for at least an hour! Was it the hub, the cables, the restart, the upgrade?

After a good 15 minutes stopping and restarting adb and plugging and unplugging cables I decided that it must be something to do with the new ports I was using…so I ran:

$ lsusb

and could see the devices connected – so the usb ports were working.

So then I ran :

$ ls -l dev/bus/usb/001

and for some reason plugdev was only showing for one of the devices through adb write permissions were missing:

crw-rw-r--  1 root root    189, 128 Apr  8 11:46 001

crw-rw-r--  1 root root    189, 129 Apr  8 11:46 002

crw-rw-rw-+ 1 root plugdev 189, 135 Apr  8 16:59 008

crw-rw----+ 1 root audio   189, 137 Apr  8 16:16 010

 

So, what had changed…I checked my android.rules file; checked that ubuntu hadn’t upgraded after the restart and then double checked the phones where I discovered a new informational message…’connected as a media device’!!

Well, somehow both samsungs had connected as Media Devices when they restarted and they need to be connected as Camera devices – so I changed both phone to connect as a Camera device and as soon as I made the change I could see them when i ran

$ adb devices

and all my jenkins jobs were able to run again.

 

Calabash (RunLoop::Xcrun::TimeoutError)

While I remember, for those of you with older macs who keep encountering the dreaded timeout error when running calabash ios against the simulator:

 Xcrun timed out after 30.27 seconds executing

   xcrun instruments -s templates

  with a timeout of 30

   (RunLoop::Xcrun::TimeoutError)

  ./features/ios/support/01_launch.rb:27:in `Before'

You can change the default RunLoop timeout by adding a override in your support/env.rb file. For example to increase the timeout to 150 seconds, add this:

RunLoop::Xcrun::DEFAULT_OPTIONS[:timeout] = 150

Give it a try!!!

 

 

—–UPDATE—–

Note: RunLoop errors using calabash-ios are the most frustrating errors to debug when using physical devices. Sometimes there are other issues which cause this error, such as

A) Forgetting to put the calabash ios server in the ipa;

B) Having a typo in the one of the environment variables (BUNDLE_ID, DEVICE_TARGET, DEVICE_ENDPOINT).

C) Not having the device enabled for development

If you encounter RunLoop errors and you can’t understand why …and you are sure that it’s not one of the first 3 reasons then:

1. Check UIAutomation is set to ON on the device and that you don’t have an iOS update alert on your screen
2. If the Developer row is missing or UIAutomation is already turned on, try the remaining steps. If there is an IOS update alert, do the update or delete it (note it will download again and you will have the same issue unless you block apple updates in your router – if you want to do this then google it)
3. Unplug your device
4. Plug it into a different computer
5. Open Xcode and view Devices
6. Ensure there is no yellow warning triangle, if there is then allow xcode to copy the symbol files across to your device
7. Open Instruments and try to record a session using the device
8. Unplug the device
9. Restart the device
10. Ensure the Developer row is present in settings and that UI automation is turned on
11. Plug back into your original machine
12. Open Xcode – check for yellow triangle and repeat symbol copying (or you might need to update xcode – xcode will tell you)
13. Open instruments and try to record
14. Set UIAutomation to ON again
15. Retry your tests.

Note that this is process that you may find yourself going through everytime you either update xcode, your device iOS version, switch machines, plug a new device into your CI…get used to going through this ritual if you are in charge of an iOS device library!!

See cucumber scenarios by tag – solution

I was complaining the other day that I couldn’t find a little utility which would allow me to see my cucumber scenarios by tag so I wrote one.

I’m just about to put it up on GitHub and I’ll package it as a gem as well then add the link to it from this site.

Basically, it’s as simple as providing the directory for your feature files; a filename for the output and the tagname that you want to search for:

dir = '/Users/user/GitHub/my-calabash/features/'
filename = 'tag_info_out.html'
tagname = '@manual'
generate_report(dir, filename, tagname)

and then it presents the results in an html file:

Ok- so the formatting leaves a lot to be desired and I will work on improving that, but it’s so handy to be able to just input a tagname and then get a report of all the scenarios that would be called using that tagname.

The next incarnation will produce a report for multiple / concatenated tagnames.

Mobile App Automation and Localisation

A common pitfall when beginning automation is to use the text for objects instead of their id’s.

For example, you have a button which says “Go to Dashbord”, so you have something in your class which looks
a bit like this:

def go_to_dashboard_btn
 "widget.type marked:'Go to Dashboard'"
end

But you also have a screen which asks the user which language they would like to use the app in and you select
a language which is not English. Suddenly your “Go to Dashboard” button “does not exist”. Why? Well because it’s the
text string has been localised and the button now says something like: “Vamos a Dashboard” (forgive my spelling).
You can see the different text values if oyu query the object while the app is using the different localised string
So if you query this object (query(“widget.type.button”) when using the app as an English user you will see:

 
 [0] {
 "id" => "goto_dashboard_btn",
 "enabled" => true,
 "contentDescription" => nil,
 "text" => "Go to Dashboard",
 "visible" => true,
 "tag" => nil,
 "description" => "widget.type.button{426535e0 V.ED...",
 "class" => "widget.type.button",
 "rect" => {
 "center_y" => 707,
 "center_x" => 186,
 "height" => 64,
 "y" => 675,
 "width" => 118,
 "x" => 127
 }

And if you query this object (query(“widget.type.button”) when using the app as an Spanish user you will see:

[0] {
 "id" => "goto_dashboard_btn",
 "enabled" => true,
 "contentDescription" => nil,
 "text" => "Vamos a Dashboard",
 "visible" => true,
 "tag" => nil,
 "description" => "widget.type.button{426535e0 V.ED...",
 "class" => "widget.type.button",
 "rect" => {
 "center_y" => 707,
 "center_x" => 186,
 "height" => 64,
 "y" => 675,
 "width" => 118,
 "x" => 127
 } 
 
 
As you can see the text string changes, so you need to make sure that you use the accessibility id.
So now your methods look like this:

def go_to_dashboard_btn
 "widget.type marked:'goto_dashboard_btn'"
end

If you make sure that the developers provide accessibility ids for everything then localisation won’t affect your
automated tests.

Note this general approach applies to both Calabash and Appium (and other tools)

Jenkins can’t find android device

Sometimes, we see the problem where the jenkins is no longer listing the android device when trying to run our automated tests.

We try adb devices and there is nothing coming back. We unplug and replug and nothing happens.

Well…I worked out what was causing so this so that I no longer see it…and I’ll tell you about that later, but in order to get things running again, here’s what you generally need to do.

You need to kill and restart the adb as root on the jenkins server and THEN you need to unplug and replug the device(s). Do the following (on an ubuntu server)

sudo su
enter password
cd Android/Sdk/platform-tools/
./adb kill-server 
./adb start-server 
./adb devices 
exit

So, what’s causing this? Generally it’s two calabash jobs running at the same time that causes adb to get confused as hell. To prevent this either build a rake task or install a build blocker plug-in.

Searching cucumber feature files by tag

You know when you’re working on project and you end up with hundreds of cucumber feature files and your working environment doesn’t use JIRA very well and you just have all these feature files not linked to anything to help you determine the status of them…even though you’ve tagged them all nicely?…And you think to yourself, all I want to be able to do is to find all the feature files that I have tagged as @to_do so that I can work through them. Or someone wants a list of all the feature files where the scenarios are tagged @manual? Or you want to see what feature files you have tagged for @android_only or for @ios_only…and so on.

Well, since you’re in an environment which does’t want you messing up JIRA with all your test stuff!!??!!!???? …then you have to take control yourself…so what’s the answer here?

Update: The answer is to create the utility yourself!!
See this post..get files listed by tagname

 

Appium and Cucumber – Managing different iOS and device configurations

Usually you will want to run your cucumber appium test suite on a few different configurations.

For example, you may want to run it on an iPhone 6 9.0 simulator and you also may want to run it on an iPad Air 8.4 simulator. So how do you do this?
Well it’s easy. First you use your cucumber.yml file to list the different configurations. So if we use our example above
we could create two profiles, lets call them iphonesim and ipadsim.
They would look like this in the cucumber.yml file:

ipadsim: IDEVICENAME='ipad simulator'
iphonesim: IDEVICENAME='iphone simulator'

We then need to create two different appium.txt files. One for the ipadsim configuration and one for ipadsim configuration. For example, the iphonesim appium.txt one would look like this.

[caps]
platformName = "ios"
deviceName = "iPhone 6"
platformVersion = "9.0"
app = "../../../apps/TestApp/build/release-iphonesimulator/myApp.app"

[appium_lib]
sauce_username = false
sauce_access_key = false

Save theiphonesim appium.txt file in a folder under Features/Support called “iphonesim” and theipadsim appium.txt under in a folder under Features/Support called “ipadsim”.

Then in env.rb tell Appium where to go to pick up the correct appium.txt based on which profile you have started your tests with.
Do this by adding something like this to your env.rb file

if ENV['IDEVICENAME']=='ipad simulator'
caps = Appium.load_appium_txt file: File.expand_path("./../ipadsim/appium.txt", __FILE__), verbose: true
elsif ENV['IDEVICENAME']=='iphone simulator'
caps = Appium.load_appium_txt file: File.expand_path("./../iphonesim/appium.txt", __FILE__), verbose: true
else
caps = Appium.load_appium_txt file: File.expand_path('./', __FILE__), verbose: true
end

Lastly, to run with your selected profile, cd into your cucumber folder and do the following:
To run the iPhone configuration execute:

cucumber -p iphonesim

To run the iPad configuration execute:

cucumber -p ipadsim

What does this actually do?

Well, if we use “cucumber -p iphonesim” as an example, firstly, the -p flag indicates to cucumber that you want to use a specific profile, so it looks up the profile in the cucumber.yml and sees IDEVICENAME=’ipad simulator’.
Cucumber then looks at the env.rb file and sees that when IDEVICENAME=’ipad simulator’ it should use the appium.txt file in the phonesim folder.
Appium then launches the simulator with the capabilites defined in the Features/Support/iphonesim/appium.txt.

This allows us to switch around configurations without changing anything other than our initial cucumber -p command.

Neat eh?

Installing Appium with Ruby bindings – moving from Calabash – Part 3

Building an IOS app for Appium

For those coming across from calabash, you will be used to having to create a duplicate -cal scheme to embed the calabash.framework into your app. Well, thankfully with Appium there is no need to do this at all. Simply open your Xcode project and build the app. That’s it – nothing special required.

Benefit of Appium over Calabash

The major benefit of Appium here is that the app that you are testing is exactly the same app as you release into production – there is no linked automation framework. This is a definite advantage over calabash.

Running Appium

All you need to do now is open 2 tabs terminal tabs.

In the first one run :

node .

Once this has started up, in the second tab run:

 arc

You now have a fully functioning Appium environment.

Important note: I strongly advise that  you use bundler with gemfiles if you plan to have appium with ruby bindings installed on the same machine as calabash. If you don’t you can end up with horrible things happening.

If you are experiencing gem issues I whole heartedly recommend this post  Installing ruby the Correct Way by cbednarski. It also has a great Appendix on cleaning your environment before re-installing your gems.