Creating filters can be one of the trickiest things in QC OTA API usage. The first thing you might run into when using Ruby is how you set a filter. In Visual Basic you set a filter like this:
testsetFilter.Filter("CY_CYCLE_ID") = "> 0"
However, in Ruby you have to use the setproperty method e.g.
testsetFilter.setproperty('Filter', 'CY_CYCLE_ID', '> 0')
There were a couple of filters that I was struggling with, but finally got them working. The first one was when I wanted to filter only test cases for some specific test sets in the Test Lab. The working Ruby code looks like this:
testsetf = qc.TestSetFactory
testsetFilter = testsetf.Filter
testsetFilter.setproperty('Filter', 'CY_CYCLE_ID', '> 0')
testsetl = testsetf.NewList(testsetFilter.Text)
testf = qc.TestFactory
testFilter = testf.Filter
testFilter.setproperty('XFilter','TEST-TESTSET',testsetFilter.Text)
testl = testFilter.NewList()
... and I have a list with the filtered tests.
The second problem was with filtering tests for only a specific folder path in Test Plan. The working Ruby code looks like this:
aFilter.setproperty('Filter', 'TS_SUBJECT', "^Subject\\release1^")
... using the TS_PATH property only returned empty value and the trick with TS_SUBJECT property was to use "^ and ^" before and after the value. To get the test path from a list of tests e.g.
thisTest.Field('TS_SUBJECT').Path
Friday, September 14, 2012
Friday, August 31, 2012
Cucumber more deeply ...
Cucumber's strength is the ability to write the feature descriptions in plain text and also possible in your native language. Feature is usually a user story. When all scenarios in a feature file is passing, it tells that the story is done. The format of the feature file is:
feature: <short description>
<full description>
background:
scenario:
...
scenario outline:
examples:
...
The feature file starts with a description of the feature. The format of the description is free, but usually e.g. following template is used:
In order to ...
as a ...
I need to ...
For example:
Feature: Application sign-in
In order to sign in
as an end user
I need to enter valid credentials
After the description comes the background. The background contains the preconditions for all scenarios. The format of the background is similar to the format of the scenarios, but the only difference is that the steps in the background will be executed before each scenario. It could for example, be like this:
Background:
Given a registered user
Then comes the scenarios or scenario outlines. A scenario is like one test case. A scenario is made up of three steps: Given, When and Then.
Given is the precondition for the test case.
When is the action to be taken.
Then is the verification / what should happen.
For example:
Scenario: Successful sign-in
Given sign-in page to the application
When I enter valid credentials
And click on the sign-in button
Then I'm successfully logged in
But no error text should be displayed
Notice that And or But can be given in any of these three steps.
The other possibility is to use a scenario outline. It could look like this:
Scenario outline: Application sign-in
Given sign-in page to the application
When I enter username <username> and password <password>
And click on the sign-in button
Then I'm <loginstatus> logged in
Examples:
| username | password | loginstatus |
| validuser | validpwd | successfully |
| invaliduser | validpwd | not |
| validuser | invalidpwd | not |
When running the feature, the scenario will be repeated with the values in each row in the examples table. So in this case the scenario will be repeated four times.
You could also have many examples tables for one scenario outline e.g.
Examples: Successful logins
| username | password | loginstatus |
| validuser | validpwd | successfully |
Examples: Failed logins
| username | password | loginstatus |
| invaliduser | validpwd | not |
| validuser | invalidpwd | not |
Now if you save your feature file and run it, you will get an output similar to this:
1 scenario (1 undefined)
5 steps (5 undefined)
You can implement step definitions for undefined steps with these snippets:
Given /^sign-in page to the application$/ do
pending
end
When /^I enter username validuser and password validpwd$/ do
pending
end
When /^I enter username invaliduser and password validpwd$/ do
pending
end
...
This tells that there are five missing step definitions i.e. you need to have matching step definitions for each line in the feature file in your step definition file. To get started with your step definitions file, copy the above pending step definitions to your step definitions file. The next thing I would do is to combine / group step definitions. As you can notice the step definitions:
When /^I enter username validuser and password validpwd$/ do
pending
end
When /^I enter username invaliduser and password validpwd$/ do
pending
end
... are very close to each other. They could be combined to one step definition:
When /^I enter username (.*) and password (.*)$/ do |user,pwd|
end
After this you need to insert the actual test script code in each step definition. It is also possible to call another step definition from within a step definition e.g.
When /^I enter username (.*) and password (.*)$/ do |user,pwd|
step "I enter valid credentials"
end
There are still a couple of things that you need to / is good to know - tags and hooks!
Tags
In your feature file you can add tags. Tags are strings starting with @ e.g.
@highpriority
Scenario: ...
If you now wan't to run all scenarios tagged with @highpriority, you can do it like this:
cucumber --tags @highpriority
If you wan't to run all scenarios, except those ones tagged with @highpriority:
cucumber --tags ~@highpriority
Hooks
With the help of hooks you can run some specific script code in different places during the test runs. There are many predefined hooks available in Cucumber. Hooks are defined in files in the support folder, usually env.rb or hooks.rb.
To do something before starting to run any scenario, put that code at the top of the file. To do something when the run ends, use the at_exit hook. To run some code before or after a scenario, use the Before or After hooks. You can also execute some specific code after each step with the AfterStep hook. It is also possible to run different code if scenario passes or fails:
Then there is also possibility to use tagged hooks i.e. just execute some code for tagged scenarios. For example.
feature: <short description>
<full description>
background:
scenario:
...
scenario outline:
examples:
...
The feature file starts with a description of the feature. The format of the description is free, but usually e.g. following template is used:
In order to ...
as a ...
I need to ...
For example:
Feature: Application sign-in
In order to sign in
as an end user
I need to enter valid credentials
After the description comes the background. The background contains the preconditions for all scenarios. The format of the background is similar to the format of the scenarios, but the only difference is that the steps in the background will be executed before each scenario. It could for example, be like this:
Background:
Given a registered user
Then comes the scenarios or scenario outlines. A scenario is like one test case. A scenario is made up of three steps: Given, When and Then.
Given is the precondition for the test case.
When is the action to be taken.
Then is the verification / what should happen.
For example:
Scenario: Successful sign-in
Given sign-in page to the application
When I enter valid credentials
And click on the sign-in button
Then I'm successfully logged in
But no error text should be displayed
Notice that And or But can be given in any of these three steps.
The other possibility is to use a scenario outline. It could look like this:
Scenario outline: Application sign-in
Given sign-in page to the application
When I enter username <username> and password <password>
And click on the sign-in button
Then I'm <loginstatus> logged in
Examples:
| username | password | loginstatus |
| validuser | validpwd | successfully |
| invaliduser | validpwd | not |
| validuser | invalidpwd | not |
When running the feature, the scenario will be repeated with the values in each row in the examples table. So in this case the scenario will be repeated four times.
You could also have many examples tables for one scenario outline e.g.
Examples: Successful logins
| username | password | loginstatus |
| validuser | validpwd | successfully |
Examples: Failed logins
| username | password | loginstatus |
| invaliduser | validpwd | not |
| validuser | invalidpwd | not |
Now if you save your feature file and run it, you will get an output similar to this:
1 scenario (1 undefined)
5 steps (5 undefined)
You can implement step definitions for undefined steps with these snippets:
Given /^sign-in page to the application$/ do
pending
end
When /^I enter username validuser and password validpwd$/ do
pending
end
When /^I enter username invaliduser and password validpwd$/ do
pending
end
...
This tells that there are five missing step definitions i.e. you need to have matching step definitions for each line in the feature file in your step definition file. To get started with your step definitions file, copy the above pending step definitions to your step definitions file. The next thing I would do is to combine / group step definitions. As you can notice the step definitions:
When /^I enter username validuser and password validpwd$/ do
pending
end
When /^I enter username invaliduser and password validpwd$/ do
pending
end
... are very close to each other. They could be combined to one step definition:
When /^I enter username (.*) and password (.*)$/ do |user,pwd|
end
After this you need to insert the actual test script code in each step definition. It is also possible to call another step definition from within a step definition e.g.
When /^I enter username (.*) and password (.*)$/ do |user,pwd|
step "I enter valid credentials"
end
There are still a couple of things that you need to / is good to know - tags and hooks!
Tags
In your feature file you can add tags. Tags are strings starting with @ e.g.
@highpriority
Scenario: ...
If you now wan't to run all scenarios tagged with @highpriority, you can do it like this:
cucumber --tags @highpriority
If you wan't to run all scenarios, except those ones tagged with @highpriority:
cucumber --tags ~@highpriority
Hooks
With the help of hooks you can run some specific script code in different places during the test runs. There are many predefined hooks available in Cucumber. Hooks are defined in files in the support folder, usually env.rb or hooks.rb.
To do something before starting to run any scenario, put that code at the top of the file. To do something when the run ends, use the at_exit hook. To run some code before or after a scenario, use the Before or After hooks. You can also execute some specific code after each step with the AfterStep hook. It is also possible to run different code if scenario passes or fails:
Before do
# Do something before each scenario
end
After do |scenario|
if(scenario.failed?)
# Do something if scenario failed
end
if(scenario.passed?)
# Do something if scenario passed
end
end
Then there is also possibility to use tagged hooks i.e. just execute some code for tagged scenarios. For example.
After('@highpriority', '@mediumpriority') do
# Do something after scenarios tagged with @highpriority OR @mediumpriority
end
Before('@highpriority, @mediumpriority') do
# Do something before scenarios tagged with @highpriority AND @mediumpriority
end
Thursday, August 30, 2012
Ruby charts
Managers love charts and statistics. Few open-source test automation tools are having any built-in tools for it. If you are using Ruby for scripting, then one of the easiest way to create charts is to use the googlecharts gem. It is using Google Charts API for creating charts.
You can install googlecharts gem with following command:
gem install googlecharts
Here is an example on how to use it:
require 'gchart'
g = Gchart.bar(:data => [5, 10, 20, 15],
:bar_colors => 'DDFFAA,5BFFA5,FFCDCD,FFADAD',
:stacked => true,
:size => '666x450',
:bg => 'f8f8ff',
:bar_width_and_spacing => '50,12,24',
:legend => ['1-Low','2-Medium','3-High'],
:max_value => 50,
:axis_with_labels => 'x,y',
:axis_labels => [['New','Open','Closed'],[5,10,15,20,25],
:title => "Defects Summary Graph - by Status group by Severity" )
gcharturl = g.split('?')[0]
gchartpostdata = g.split('?')[1]
url = URI.parse(gcharturl)
ht = Net::HTTP.new(url.host, url.port)
ht.start
res = ht.request_post(gcharturl, gchartpostdata)
ht.finish
open("summary.png", "wb") { |file|
file.write(res.body)
}
... and it would create an image close to this one:
With Ruby you can easily access MySQL database. So test results can be easily saved to database and the data can be used to create nice graphs for the management to love!
You can install googlecharts gem with following command:
gem install googlecharts
Here is an example on how to use it:
require 'gchart'
g = Gchart.bar(:data => [5, 10, 20, 15],
:bar_colors => 'DDFFAA,5BFFA5,FFCDCD,FFADAD',
:stacked => true,
:size => '666x450',
:bg => 'f8f8ff',
:bar_width_and_spacing => '50,12,24',
:legend => ['1-Low','2-Medium','3-High'],
:max_value => 50,
:axis_with_labels => 'x,y',
:axis_labels => [['New','Open','Closed'],[5,10,15,20,25],
:title => "Defects Summary Graph - by Status group by Severity" )
gcharturl = g.split('?')[0]
gchartpostdata = g.split('?')[1]
url = URI.parse(gcharturl)
ht = Net::HTTP.new(url.host, url.port)
ht.start
res = ht.request_post(gcharturl, gchartpostdata)
ht.finish
open("summary.png", "wb") { |file|
file.write(res.body)
}
... and it would create an image close to this one:
A couple of things to notice.
- The Gchart function call will return the URL to the graph. You can access it using GET, but it won't work if the URL is longer than 1024 characters . That's why I used POST to get the image data.
- The chart can contain max. 300000 pixels, so if you specify a size larger than that, then creating the image will fail.
With Ruby you can easily access MySQL database. So test results can be easily saved to database and the data can be used to create nice graphs for the management to love!
If you haven't used Ruby with MySQL, here's simple instructions:
- Install mysql gem with command: gem install mysql
- Make sure that libmysql.dll exist on your PC and that it is accessible (in a folder in PATH). Also the version need to match with the gem version. More information about it can be found from here.
How to use it e.g.
require 'mysql'
con = Mysql.new(myserver.com, 'myuser', 'mypwd', 'mydb')
con.query("UPDATE xxx SET yyy=1 WHERE 1=1")
con.close
More information about googlecharts:
Tuesday, August 28, 2012
ALM / QC integration using OTA API
To integrate your test automation tools with HP Quality Center / Application Lifecycle Management, the easiest way is to use QC's Open Test Architecture (OTA) API.
OTA API is a COM library that enables you to integrate external applications with QC. Through this interface you can do most of the same tasks as you can do through the GUI.
To be able to access OTA API from your PC, you have to install the HP Quality Center Connectivity Add-in. To do so, go to Help -> Add-ins page and select HP Quality Center Connectivity. It will download the OTAClient.dll to your PC.
Here is an example script in Ruby, which counts the amount of open defects in QC:
require 'win32ole'
opendefects = 0
# Create the TDConnection object
qc = WIN32OLE.new('TDApiOle80.TDConnection')
qcserver = 'http://localhost:8080/qcbin/'
# Initiate connection to the QC server
qc.InitConnectionEx(qcserver)
puts "Connection status: #{qc.Connected}"
# Authenticates to QC
qc.Login("myuser", "mypassword")
puts "Logged in status: #{qc.LoggedIn}"
# Connect to the QC Domain and Project
qc.Connect("MYDOMAIN", "myproject")
puts "Project Name: #{qc.ProjectName}"
# Get defect list and count open defects
bugf = qc.BugFactory
bugl = bugf.NewList("")
for thisBug in bugl
if thisBug.Status == "New" or thisBug.Status == "Open" or thisBug.Status == "Reopen"
opendefects = opendefects+1
end
end
# Disconnect and Logout from QC
qc.Disconnect
qc.Logout
puts "Logged in status: #{qc.LoggedIn}"
# Release connection from the QC server
qc.ReleaseConnection
puts "Connection status: #{qc.Connected}"
# Print results
puts "Total number of defects: " + bugl.count.to_s
puts "Total number of open defects: " + opendefects.to_s
... and the output should be something like:
Connection status: true
Logged in status: true
Project Name: myproject
Logged in status: false
Connection status: false
Total number of defects: 52
Total number of open defects: 5
OTA API is a COM library that enables you to integrate external applications with QC. Through this interface you can do most of the same tasks as you can do through the GUI.
To be able to access OTA API from your PC, you have to install the HP Quality Center Connectivity Add-in. To do so, go to Help -> Add-ins page and select HP Quality Center Connectivity. It will download the OTAClient.dll to your PC.
Here is an example script in Ruby, which counts the amount of open defects in QC:
require 'win32ole'
opendefects = 0
# Create the TDConnection object
qc = WIN32OLE.new('TDApiOle80.TDConnection')
qcserver = 'http://localhost:8080/qcbin/'
# Initiate connection to the QC server
qc.InitConnectionEx(qcserver)
puts "Connection status: #{qc.Connected}"
# Authenticates to QC
qc.Login("myuser", "mypassword")
puts "Logged in status: #{qc.LoggedIn}"
# Connect to the QC Domain and Project
qc.Connect("MYDOMAIN", "myproject")
puts "Project Name: #{qc.ProjectName}"
# Get defect list and count open defects
bugf = qc.BugFactory
bugl = bugf.NewList("")
for thisBug in bugl
if thisBug.Status == "New" or thisBug.Status == "Open" or thisBug.Status == "Reopen"
opendefects = opendefects+1
end
end
# Disconnect and Logout from QC
qc.Disconnect
qc.Logout
puts "Logged in status: #{qc.LoggedIn}"
# Release connection from the QC server
qc.ReleaseConnection
puts "Connection status: #{qc.Connected}"
# Print results
puts "Total number of defects: " + bugl.count.to_s
puts "Total number of open defects: " + opendefects.to_s
... and the output should be something like:
Connection status: true
Logged in status: true
Project Name: myproject
Logged in status: false
Connection status: false
Total number of defects: 52
Total number of open defects: 5
Best practices
Here is my list of best practices for a successful test automation project:
Maintainability
This is the most important thing to keep in mind - always think about maintainability of your test scripts. If you don't keep this in mind from the beginning of your project, then you will run in to major problems and maintenance of test scripts will take a lot of time when your test suite is getting bigger.
Modularity and reusability
By creating modules / functions will help maintaining your scripts and keep the scripts more simple and understandable.
Readable and simple
Write your tests so that they are readable and as simply as possible. This makes it easy for other persons than only you to understand what the script does and why, and it also helps maintaining the scripts.
Easy
Tests should be easy to understand, modify and extend.
Addressing components
Never address any components directly in your scripts i.e. don't hardcode any values in your test scripts. Keep variables and parameters in a separate place. This is very important for the maintainability of your tests.
Independent
Tests should be independent i.e. not depending on any other script.
Write "Test to Fail"
When writing tests you should always keep in mind that they might / will fail and they should always be able to handle the failure and do necessary things so that the failure won't affect any following scripts e.g log out, close application.
Use GUI element IDs
In web applications use GUI element IDs to locate a component on a page. It will make your scripts more tolerant to changes e.g .changing page layout won't affect your scripts. If components are missing ID's - ask developers to add them.
Use facades
Don't access APIs directly from your scripts, use some sort of facade. In case API changes, it is easier to make the changes in the facade, than in all the hundreds of test scripts.
Easy debugging
Make it easy to debug failures e.g take screenshots, HTML captures, save system logs in case of failure.
Efficient scripts
Tests should be efficient i.e. run in a reasonable
amount of time. Don't make scripts that will take half an hour to run.
Traceability
It should be traceable what part of code and what requirement each test script is testing, otherwise you will later run in to this question - what is this script testing and why?
Robot Framework
Robot Framework is a generic open-source test automation framework for ATDD (Acceptance Test-Driven Development). It utilizes keyword driven testing approach, so even without any scripting knowledge you are able to write test scripts.
There are many different testing libraries / plugins available for Robot Framework e.g. Selenium / Watir / Watin libraries, Java GUI library, AutoIt library for testing Windows applications, Database libraries for testing databases, HTTP and SSH libraries, Jenkins plugin .. You can also create your own libraries in Java or Python.
Robot Framework has very good built-in logging and reporting functionalities with automatic screenshots.
Test scripts can be created using keyword-driven, data-driven or behavior-driven approach. The format of the test scripts can be HTML tables, tab separated values (TSV), plain text ... You can also create your own keywords, by using existing keywords. There is a test script editor available for Robot Framework called Ride.
Robot Framework development is supported by Nokia Siemens Networks.
Below I will describe how to install and use Robot Framework Selenium2Library with plain text scripts.
Installation
https://github.com/robotframework/RIDE
There are many different testing libraries / plugins available for Robot Framework e.g. Selenium / Watir / Watin libraries, Java GUI library, AutoIt library for testing Windows applications, Database libraries for testing databases, HTTP and SSH libraries, Jenkins plugin .. You can also create your own libraries in Java or Python.
Robot Framework has very good built-in logging and reporting functionalities with automatic screenshots.
Test scripts can be created using keyword-driven, data-driven or behavior-driven approach. The format of the test scripts can be HTML tables, tab separated values (TSV), plain text ... You can also create your own keywords, by using existing keywords. There is a test script editor available for Robot Framework called Ride.
Robot Framework development is supported by Nokia Siemens Networks.
Below I will describe how to install and use Robot Framework Selenium2Library with plain text scripts.
Installation
- Download Python 2.7 installer from http://www.python.org and run it.
- Add Python root and Scripts folders to PATH.
- Download Robot Framework installer from http://www.robotframework.org and run it. If you have problems with the installer (as I did), then download Robot Framework source and install it using command: python setup.py install
- Download Robot Framework Selenium2Library installer or source from https://github.com/rtomac/robotframework-selenium2library and install it in the same way as Robot Framework.
How to run scripts
By running following command:
pybot <scriptname>.html|txt|tsv
Example script - robottest.txt
*** Settings ***
Documentation A test suite for Google search.
Resource robotresource.txt
*** Test Cases ***
Valid Search
Open Browser to Google Page
Input Query Robot Framework
Submit Query
Check link to Robot Framework
[Teardown] Close Browser
Example script - robotresource.txt
*** Settings ***
application.Library Selenium2Library
*** Variables ***
${BROWSER} firefox
${DELAY} 0
*** Keywords ***
Open Browser to
Google Page
Open Browser
http://www.google.com ${BROWSER}
Maximize Browser Window
Set Selenium Speed ${DELAY}
Input Query
[Arguments] ${query}
Input Text
q ${query}
Submit Query
Click Button
gbqfb
Check link to Robot Framework
Wait Until Page Contains Robot Framework 10
More information
Monday, August 27, 2012
Cucumber
Cucumber is an open-source test automation framework for BDD (Behaviour-Driven Development). Cucumber lets software development teams describe how software should behave in plain text. The text is written in business-readable domain-specific language (Gherkin) and serves as documentation, automated tests and development-aid. It supports test scripting in Ruby, Java, C#, Python, PHP, Perl ..
Cucumber is an excellent test automation framework / tool together with e.g. Watir. Cucumber scripts serves as documentation for the whole team and they also helps developer to quickly see that they have implemented everything and it works as expected. Cucumber also has good logging / reporting functionality.
How to run scripts
cucumber features (--out <filename>.html --format html)
http://cukes.info
Cucumber is an excellent test automation framework / tool together with e.g. Watir. Cucumber scripts serves as documentation for the whole team and they also helps developer to quickly see that they have implemented everything and it works as expected. Cucumber also has good logging / reporting functionality.
Installation
- Download Ruby installer from http://www.rubyinstaller.org and run it. During installation select Ruby to be added to PATH and to associate .rb files to this Ruby installation.
- Download Ruby Devkit from http://www.rubyinstaller.org and extract it. In the extracted folder, run commands ruby dk.rb init and ruby dk.rb install to setup and install it.
- Install Cucumber gem with command: gem install cucumber
- Install Syntax gem with command: gem install syntax
Cucumber scripts usually consists of three files - feature, step definitions and support files. Feature file contains the feature description. Step definitions contains the actual test script code. For each line in the feature file there should be a matching step definition in the step definitions file. The support files can contain code for ramp-up, tear-down and hooks for something to be done before / after steps / scenarios.
How to run scripts
By running following command:
cucumber features (--out <filename>.html --format html)
Example script - feature file - cucumber.feature
Feature: Google
search
In order to find Cucumber homepage using
Google search
As an end user
I need to enter a correct keyword in the
search field
Background:
Given Google search page
Scenario: Search for Cucumber
Given keywords ‘Cucumber’ in search
field
When I submit form
Then a link to Cucumbers’s homepage
is found
Example script - step definitions file - cucumber.rb
Given /^Google search
page$/ do
$browser.goto
‘http://www.google.com’
end
Given /^keyword ‘Cucumber’ in search
field$/ do
$browser.text_field(:name,
"q").set 'Cucumber'
end
When /^I submit
form$/ do
$browser.button(:id, ‘gbqfb’).click
end
Then /^a link to Cucumber’s homepage
is found$/ do
Watir::Wait.until { $browser.text.include? 'Making BDD fun‘ }
end
Example script - support file - env.rb
require 'rubygems'
require 'watir-webdriver'
$browser = Watir::Browser.new(:firefox)
at_exit do
$browser.close
end
More information
http://cukes.info
Subscribe to:
Posts (Atom)