Learning Software Testing with Test Studio

By Rawane Madi

Learning Software Testing with Test Studio is a practical, hands-on guide that will help you get started with Test Studio to design your automated solution and tests. All through the book, there are best practices and tips and tricks inside Test Studio which can be employed to improve your solution just like an experienced QA.If you are a beginner or a professional QA who is seeking a fast, clear, and direct to the point start in automated software testing inside Test Studio, this book is for you. You should be familiar with the .NET framework, mainly Visual Studio, C#, and SQL, as the book's examples rely on them. Prior testing knowledge will also be helpful.

• Language: English
• Publisher: Packt Publishing
• Release date: Sep 24, 2013
• ISBN: 9781849688918

Book preview

Table of Contents

Learning Software Testing with Test Studio

Credits

About the Author

About the Reviewers

www.PacktPub.com

Support files, eBooks, discount offers, and more

Why Subscribe?

Free Access for Packt account holders

Instant Updates on New Packt Books

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Downloading the example code

Errata

Piracy

Questions

1. Introduction

Testing concepts

Test case

Executing a test case

The exit criteria

The pesticide paradox

Element recognition

Testing phases

Testing techniques

Test Studio uncovered through the chapters

Approach

Setting up your environment

The File Comparer application

Reports

Additional files

Other reference sources

2. Automating Functional Tests

Getting started with automation

Where does automation fit best?

Test strategies

Capture and playback

Data-driven architecture

Keyword-driven architecture

About Test Studio

Supported technologies

Architecture

Functional test automation

Recording an automated test

Record and playback

Editing tests

Translators and verification steps

Inserting a verification step

Test Studio's built-in log

Custom logging

Code refactoring

Automated test case design

Using tests as steps

Logical branching

Extracting values to variables

Additional automation tweaks in the code

Functional random keyword-driven tests

Test execution settings and debugging

Syntax errors

Execution errors

Execution settings

Debugging

Integration with Visual Studio

Summary

3. Data-driven Tests

Data-driven testing architecture

Binding tests to data sources

Excel data source binding

Importing an Excel data source

Binding columns to Excel columns

Data-driven verification

Data-driven verification binding

CSV data binding

Editing a data source

Logging the test metadata

XML data source binding

Importing an XML data source

Binding columns to XML attributes

Database binding

A non data-driven test comprising a data-driven test

Creating parent and child tests

A data-driven test comprising a non data-driven test

Creating child tests

A data-driven test comprising another data-driven test

Creating parent and child tests

Creating parent and child tests

Summary

4. Maintaining Test Elements

Automation element recognition

The element repository

Adding elements

The recorder toolbar DOM Explorer

Forming the element repository

Filtering options

3D Viewer Element Adding and Verification

Finding locators

Using the chained expression

The Find class

The Element factory

Summary

5. Manual Testing

Manual Testing

Manual tests

Excel integration

Adding existing tests

Hybrid tests

Refactor tests

Repository maintenance

Test execution

Data binding

Transitioning to automated tests

Versioning

Integration with Visual Studio

Tailor tests in Test Studio

Decision tables

State transition diagrams

Summary

6. Test Lists and Reports

Test suites

Creating test lists

Lists

Dynamic lists

Executing test lists

Dynamic list automatic update

Test list settings

Specialized test list settings

Reporting

Result integration

Visual Studio report integration

SQL integration

Submitting to SQL

Excel Charts

Execution metrics for last run

Test case defect density

Execution metrics over time

Feature Coverage

Test Execution Duration

List result comparison

Summary

7. Performance Testing

Performance testing

Performance time measurements

Performance testing in Test Studio

How it works

Performance test planning and execution

The planning phase

Defining the objectives and expectations

Baseline test environment

BugNet user classification

BugNet user activities

Performance metrics selection

Baseline tests

The implementation phase

The execution and analysis phase

The overview mode

The details view

The compare view

The history view

Summary

8. Load Testing

Load testing

Measurements

Load testing in Test Studio

How it works

Load test planning and execution

The Planning and design phase

Defining goals

Designing workloads

Defining test cases

The implementation phase

Configuring services

Designing tests

The Execution and analysis phase

Database reporting

Creating reports

Summary

9. Mobile Testing

iOS testing inside Test Studio

Requirements

Deployment

Simulator

Automating tests

Creating tests

Execution

Testing on the device

Debugging tests

Test elements

Wait on elements

Locating elements

Web portal

Registering an application

Testing

Contributors

Feedback and crash reports

Configuring settings

The Feedback option

Crash reports

Data management

Configuring settings

Database backup actions

Database restore actions

Summary

10. Tips and Tricks

Introduction

Maintainability

Making a test work for both data-driven and hardcoded input

Extension libraries

Data-driven design

Parameterizing the find element's expressions

Variables

Passing variables between tests

Summary

A. Configuring BugNet

Creating user accounts

Creating a project

Index

Learning Software Testing with Test Studio

---

Learning Software Testing with Test Studio

Copyright © 2013 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

First published: September 2013

Production Reference: 1170913

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham B3 2PB, UK.

ISBN 978-1-84968-890-1

www.packtpub.com

Cover Image by Artie Ng (<[e-mail]>)

Credits

Author

Rawane Madi

Reviewers

Jim Holmes

Dimo Mitev

Angel Tsvetkov

Acquisition Editor

Amarabha Banerjee

Lead Technical Editor

Susmita Panda

Technical Editors

Krishnaveni Haridas

Shali Sasidharan

Copy Editors

Mradula Hedge

Laxmi Subramanian

Gladson Monterio

Aditya Nair

Kirti Pai

Adithi Shetty

Project Coordinator

Shiksha Chaturvedi

Proofreader

Lesley Harison

Indexer

Monica Ajmera Mehta

Graphics

Abhinash Sahu

Yuvraj Mannari

Ronak Dhruv

Production Coordinator

Conidon Miranda

Cover Work

Conidon Miranda

About the Author

Rawane Madi has a Bachelor’s degree in Computer Science and around five years of experience in software testing and automation. She started her career as a Quality Control Engineer at a multinational company that provides both web and Windows applications’ solutions. She is a holder of the ISTQB Certified Tester Foundation Level certification and the author for article reviews on automated testing tools published online on DevPro.

No one can whistle a symphony. It takes a whole orchestra to play it. by Halford E. Luccok

With this, I would like to start by thanking Packt Publishing for offering me the generous opportunity of sharing my experience in this book. I would also like to thank Telerik’s team support, and particularly Jim Holmes for providing his important contribution and critical review.

A lot of gratitude goes to my employer and the management who were always ready to offer their kind support and technical help. These include Raed Gharzeddine, Imad Koussa, and Bilal Haidar.

Finally, special thanks to my family and friends, especially, Liliane Madi, Nadim Mohsen, Layal Abi Farraj, and Jad Al Deeb for their continuous encouragement.

About the Reviewers

Jim Holmes is the Director of Engineering for Test Studio at Telerik. He has over 25 years of experience in the IT field in positions including PC Technician, WAN Manager, Customer Relations Manager, Developer, and Tester. Jim has held jobs in the US Air Force, the Department of Defense (DOD) sector, the software consulting domain, and commercial software product sectors. He has been a longtime advocate of test automation and has delivered software on a wide range of platforms. He coauthored the book, Windows Developer Power Tools, and blogs frequently at http://FrazzledDad.com. Jim is also the President of the Board of Directors for the CodeMash conference held in the middle of winter at an indoor waterpark in Sandusky, Ohio.

Dimo Mitev has almost 10 years of experience working in the IT industry in various international companies, and roles such as Test Consultant, QA Engineer, QA Lead, and Team Lead. Dimo has gained variety of experience in different software testing types, including functional, web service, performance testing, and automation testing. He is currently working as a QA Architect for Telerik Corp., a leading market provider of end-to-end solutions for application development, automated testing, Agile project management, and reporting and content management across all major Microsoft development platforms. Dimo always tries to share his knowledge and at the moment, he is a lecturer in one of Telerik’s academies, specialized in teaching young people on software testing discipline. His favorite credo is that quality is more than the lack of bugs. Dimo is currently working on his first book about software testing and is expecting it to be released very soon.

Angel Tsvetkov is an experienced, goal-oriented Quality Assurance Engineer with proven ability in test automation. He has an exceptional ability to enter new environments and produce immediate results through the use of flexible test techniques with excellent communication skills. Driven by challenge, his excellent interpersonal skills provide the ability to operate effectively at all levels and across all test activities. Angel established a QA process in one of the teams and helped in the improvement of automation testing across the Telerik company. While working with IBM, he set up the automation testing as a main approach for testing and spread the knowledge even to the customers to increase their confidence. He worked with Musala Soft and took part in the establishment of service, performance, and functional testing across different teams dealing with the development of web solutions as well as standalone and mobile solutions. Also, he was involved in one of the biggest company projects such as Johnson Controls for a huge client. His main responsibility was the development of automated test scripts for testing of the devices integrated in vehicles. His strong knowledge of electronics was beneficial for the success of the project.

To know more about Angel, visit his blog at http://qaagent.com.

www.PacktPub.com

Support files, eBooks, discount offers, and more

You might want to visit www.PacktPub.com for support files and downloads related to your book.

Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at for more details.

At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks.

http://PacktLib.PacktPub.com

Do you need instant solutions to your IT questions? PacktLib is Packt’s online digital book library. Here, you can access, read and search across Packt’s entire library of books.

Why Subscribe?

Fully searchable across every book published by Packt

Copy and paste, print, and bookmark content

On demand and accessible via web browser

Free Access for Packt account holders

If you have an account with Packt at www.PacktPub.com, you can use this to access PacktLib today and view nine entirely free books. Simply use your login credentials for immediate access.

Instant Updates on New Packt Books

Get notified! Find out when new books are published by following @PacktEnterprise on Twitter, or the Packt Enterprise Facebook page.

Preface

Test Studio is a tool that offers a variety of features to build custom automated solutions for desktop and web applications. It hosts and simplifies the testing process by supporting manual test creation, automation, and execution. Additionally, its reporting capabilities help in conveying to managers the quality status of the application under test. Test Studio can be extended with bug tracking tools, thus bringing the developers into the loop by seamlessly combining bug reporting within the testing process. This tool is tightly integrated with Team Foundation Server and Visual Studio, which provide the user with a wider set of benefits related to version controlling, management, development, and other powerful software test development characteristics.

Test Studio has two editions for testing desktop and web applications, the standalone and the Visual Studio plugin. Along with these editions, it supports an extended version intended for mobile testing.

Throughout this book, testers will learn how to use Test Studio features in order to create automated tests using interface recording and how to customize these tests by adding functionalities from the underlying test framework library. The different chapters cover the available built-in templates, such as manual, WPF, web, performance, and load test templates, where each chapter elucidates the underlying features that constitute the testing type implied by the template. Therefore, this book helps in learning manual test creation, automation, and execution against a WPF application, in addition to performance testing, benchmarking, workload designing, load testing, result analysis, and reporting against a web application. Furthermore, it contains an overview of Test Studio integration with Team Foundation Server and Visual Studio. A section is also dedicated to revealing the usage of Test Studio extension for automating, executing, reporting, and managing the tests for iOS applications in the cloud.

This book presents hands-on examples, code snippets, and snapshots to conduct the implementation, execution, and reporting phases pertaining to the software testing process inside Test Studio. It makes use of the available tools to present solutions for testing the functional and nonfunctional aspects of an application.

What this book covers

Chapter 1, Introduction, explains what software testing is, introduces some important concepts, and the comprised phases. The introduction also provides a summary of Test Studio features and the chapters in which they are reflected. This chapter also introduces the applications that will be used in the examples and how to properly set up their environment.

Chapter 2, Automating Functional Tests, introduces Test Studio, its architecture, and the supported technologies. It explains how to use its features to automate functional tests using simple recording, coding, execution, debugging, and logging. This chapter also provides an overview of test integration with Visual Studio.

Chapter 3, Data-driven Tests, demonstrates the usage of the data source features in order to create and execute data-driven automated tests. It covers the entire list of supported file system and database management sources to craft independent and embedded data-driven tests.

Chapter 4, Maintaining Test Elements, introduces Test Studio concepts on test element creation, recognition, finding, and editing using the tool IDE and framework library. This chapter also explains some techniques to help in sustaining the test repository and referencing it from within the automated tests.

Chapter 5, Manual Testing, demonstrates manual test creation, integration with MS Excel, execution, and transitioning to automated tests. Test Studio provides a managed process for converting manual testing into a semiautomated version through hybrid tests and then to a fully automated version. Furthermore, this chapter covers source and version controlling functions in Team Foundation Server, managed from Test Studio and Visual Studio IDEs.

Chapter 6, Test Lists and Reports, presents Test Studio's support for test suites through its test lists. It explains and demonstrates the creation and grouping of tests under static and dynamic lists. This chapter also dwells on the reporting side in this tool, its integration with Visual Studio, and customization with MS SQL server.

Chapter 7, Performance Testing, covers the nonfunctional aspects of an application's performance. It explains Test Studio's mechanism for carrying out performance testing by using the specific test template. The built-in capabilities are revealed through simulating real user requests, measuring performance counters, calculating metrics, comparing executions for the same test, and benchmarking the desired instance.

Chapter 8, Load Testing, demonstrates the steps to replicate real-life user loads. Using the load test template, virtual users can be employed to simulate any number of users, automated tests can be executed via different browsers, agents can be installed to initiate requests in a sparse environment, performance counters can be profiled on target machines as well as the networks, and user activity can be designed and tuned. Test Studio also offers a built-in mechanism to chart and compare the results collected for the multiple test runs.

Chapter 9, Mobile Testing, uses Test Studio extensions for iOS to demonstrate functional testing for a mobile application. The chapter explains UI test recording, execution, failure debugging, and accessing of test elements in order to automate the desired tests. Other Test Studio collaboration and reporting features are also exemplified. These features are supported in the feedback and crash reports modules that are viewable from the registered application web portal, which hosts a dashboard enabling other management and tracking testing functionalities related to test execution progress, quality monitoring, and users' collaboration.

Chapter 10, Tips and Tricks, presents some ideas resulting from the combination of different Test Studio features in order to provide useful solutions that enhance test reusability, maintainability, and flexibility in test functions.

Appendix A, Configuring BugNet, contains step-by-step instructions to configure the BugNet project and underlying objects for testing purposes.

What you need for this book

This book assumes basic knowledge in C# in order to comprehensively follow the examples pertaining calls to the test framework library. Test Studio integration with the Team Foundation Server and Visual Studio requires familiarity with these tools' IDEs and features. This book contains material related to iOS testing, which also requires familiarity with the XCode development IDE. The reader must have prior knowledge of software testing concepts. The following is a list of the required tools in order to execute the examples in the book.

For functional and automated web performance and load testing:

Test Studio

MS SQL Server 2008

Visual Studio Ultimate 2010

Team Foundation Server 2010

A File Comparer WPF application

The BugNet Issue Tracker open source application

For mobile testing:

Mac OS X 10.7

iOS SDK 6

XCode 4.6

Test Studio extension for iOS

Test Studio bundle

The Switchy open source application

Who this book is for

This book is for any person motivated by software testing who wishes to exploit and apply Test Studio features in order to manage an automated testing environment from test case creation to reporting.

It assumes a prior knowledge with the testing concepts in addition to having a basic C# knowledge and familiarity with Visual Studio IDE.

Conventions

In this book, you will find a number of styles of text that distinguish between different kinds of information. Here are some examples of these styles, and an explanation of their meaning.

Code words in text are shown as follows: Clicking on the FileComparer.Test node it will display its properties in this pane.

A block of code is set as follows:

[default]

exten => s,1,Dial(Zap/1|30)

exten => s,2,Voicemail(u100)

exten => s,102,Voicemail(b100)

exten => i,1,Voicemail(s0)

When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold:

[default]

exten => s,1,Dial(Zap/1|30)

exten => s,2,Voicemail(u100)

exten => s,102,Voicemail(b100)

 

exten => i,1,Voicemail(s0)

Any command-line input or output is written as follows:

# cp /usr/src/asterisk-addons/configs/cdr_mysql.conf.sample

    /etc/asterisk/cdr_mysql.conf

New terms and important words are shown in bold. Words that you see on the screen, in menus or dialog boxes for example, appear in the text like this: On the startup window, click on Create New Project.

Note

Warnings or important notes appear in a box like this.

Tip

Tips and tricks appear like this.

Reader feedback

Feedback from our readers is always welcome. Let us know what you think about this book—what you liked or may have disliked. Reader feedback is important for us to develop titles that you really get the most out of.

To send us general feedback, simply send an e-mail to <[e-mail]>, and mention the book title via the subject of your message.

If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, see our author guide on www.packtpub.com/authors.

Customer support

Now that you are the proud owner of a Packt book, we have a number of things to help you to get the most from your purchase.

Downloading the example code

You can download the example code files for all Packt books you have purchased from your account at http://www.packtpub.com. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you.

Errata

Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you find a mistake in one of our books—maybe a mistake in the text or the code—we would be grateful if you would report this to us. By doing so, you can save other readers from frustration and help us improve subsequent versions of this book. If you find any errata, please report them by visiting http://www.packtpub.com/submit-errata, selecting your book, clicking on the errata submission form link, and entering the details of your errata. Once your errata are verified, your submission will be accepted and the errata will be uploaded on our website, or added to any list of existing errata, under the Errata section of that title. Any existing errata can be viewed by selecting your title from http://www.packtpub.com/support.

Piracy

Piracy of copyright material on the Internet is an ongoing problem across all media. At Packt, we take the protection of our copyright and licenses very seriously. If you come across any illegal copies of our works, in any form, on the Internet, please provide us with the location address or website name immediately so that we can pursue a remedy.

Please contact us at <[e-mail]> with a link to the suspected pirated material.

We appreciate your help in protecting our authors, and our ability to bring you valuable content.

Questions

You can contact us at <[e-mail]> if you are having a problem with any aspect of the book, and we will do our best to address it.

Chapter 1. Introduction

Some of the reasons why you would want to achieve software quality are to make sure that the system does what it's supposed to do; uncover errors and/or to provide assurance for your software user. This chapter will explore some general terminologies and processes in software testing to shed light over some concepts used in this book, and briefly introduce the tool automation features that are covered in the next chapters.

If you are already familiar with the following testing concepts, you can jump to Test Studio uncovered in the later chapters of this book.

Testing concepts

The following is a conceptual overview of some fundamental testing terminologies and principles. These are used in day-to-day testing activities and will be directly referred to in the chapters when explaining the business case for our examples.

Test case

A test case is a scenario that will be executed by the tester or by an automation tool, such as the Test Studio for any of the software testing purposes, such as uncovering potential errors in the system. It contains:

Test case identifier: This identifier uniquely distinguishes a test case.

Priority: The priority holds a value to indicate the importance of a test case so that the most important ones are executed first and so on.

Preconditions: The preconditions describe the initial application state in which the test case is to be executed. It includes actions that need to be completed before starting the execution of the test case, such as performing certain configurations on the application, or other details about the application's state that are found relevant.

Procedure: The procedure of a test case is the set of steps that the tester or automated testing tool needs to follow.

Expected behavior: It is important to set an expected behavior resulting from the procedure. How else would you verify the functionality you are testing? The expected behavior of a test case is specified before running a test, and it describes a logical and friendly response to your input from the system. When you compare the actual response of the system to the preset expected behavior, you determine whether the test case was a success or a failure.

Executing a test case

When executing a test case, you would add at least one field to your test case description. It is called the actual behavior and it logs the response of the system to the procedure. If the actual behavior deviates from the expected behavior, an incident report is created. This incident report is further analyzed and in case a flaw is identified in the system, a fix is provided to solve the issue. The information