Software Testing at Scale

By Jonathan Opperman

Software Testing at Scale: Navigating the Challenges of Agile and DevOps is a comprehensive guide for software testers, developers, and leaders who are looking to improve their approach to quality assurance in an agile and devops-driven world. Written for testers already familiar with the basics, as well as developers who want to understand the impact of quality assurance on their work and leaders who need to adapt their culture and attitude towards testing, this book offers real-world examples and thought-provoking insights on the current expectations of software testers and their role in an era that is all about constant change.

We will revisit traditional approaches that are no longer sufficient, but still applicable, and provide practical strategies for organizations that have adopted agile and devops practices but are still struggling with quality assurance. This book challenges the status quo and poses important questions about why we do things a certain way, making it a valuable handbook for software testing and a reference for software developers.

Whether you're a seasoned tester looking to improve your skills, a developer looking to contribute to the overall quality of the software you build, or a leader looking to adapt your organization's approach to testing, this book offers valuable insights and actionable strategies for achieving success in today's fast-paced development environments.
 

• Language: English
• Publisher: Jonathan Opperman
• Release date: Feb 23, 2023
• ISBN: 9798215988114

Jonathan Opperman

Jonathan is a software quality assurance professional with over 20 years of experience in the field. Born and raised in Pretoria, South Africa, he developed a passion for computers and technology at a young age, with his first computer being a Sinclair ZX81. He has witnessed the evolution of the industry throughout his career, particularly in the areas of DevOps and Agile. Currently residing in Cape Town, South Africa, Jonathan continues to work in the DevOps space, utilizing his expertise and experience to drive innovation and improve software development processes.

Book preview

ACKNOWLEDGMENTS

ABOUT THE AUTHOR

INTRODUCTION

ABOUT THIS BOOK

SOFTWARE TESTING IS DEAD

THE HISTORY OF SOFTWARE TESTING

THE FUNDAMENTALS

THE MARK OF THE CRAFTSMAN

THE TEST PYRAMID

SPECIFICATIONS

TEST PLANS AND TEST CASES

WHAT IS WATERFALL

WHAT IS AGILE

WHAT IS DEVOPS

SOFTWARE REQUIREMENTS

NON FUNCTIONAL REQUIREMENTS

FUNCTIONAL REQUIREMENTS

AGILE & DEVOPS

DESK CHECKS

DEFINITION OF DONE

ESTIMATIONS

AGILE MATURITY SCORECARD

MINIMUM VIABLE PRODUCT

PROOF OF CONCEPT

ARCHITECTURE DECISION RECORDS

SHIFT TESTING LEFT

CROSS-FUNCTIONAL TEAMS

BUILDING CROSS-FUNCTIONAL TEAMS

GUILDS AND CHAPTERS

BLAMELESS POST-MORTEMS

TESTING TECHNIQUES

COMPATIBILITY TESTING

USER EXPERIENCE TESTING

USER ACCESSIBILITY TESTING

MOBILE TESTING

COMPLIANCE TESTING

UI AUTOMATION TESTING

UI AUTOMATION PATTERNS

API TESTING

CONTRACT TESTING

SECURITY TESTING

COMBINATORIAL TESTING

END-TO-END TESTING

MODEL-BASED TESTING

TEST AUTOMATION

TEST AUTOMATION FRAMEWORKS

DRY

MOCKING AND STUBBING

DISTRIBUTED SYSTEMS

CONTAINERIZATION

DATABASES & STORAGE

TEST DATA

GENERATING TEST DATA

IMPORTS AND EXPORTS

DATA MASKING

CI/CD

SOURCE CONTROL

CONTINUOUS DEPLOYMENTS

BLUE-GREEN DEPLOYMENTS

ROLLING DEPLOYMENTS

CANARY DEPLOYMENTS

FEATURE TOGGLES

METRICS AND MONITORING

INFRASTRUCTURE AS CODE

SITE RELIABILITY ENGINEERING

DISASTER RECOVERY

DEPLOYMENT ROLLBACKS

RISK MITIGATION

ERROR HANDLING

CHAOS ENGINEERING

CHAOS GAME DAYS

AI ASSISTED TESTING

SOFTWARE BILL OF MATERIALS

ENCRYPTION

TEST STRATEGY

STRATEGY MATRIX

DEFECT TRACKING

COST-BENEFIT ANALYSIS

RISK ASSESSMENT

TEST AS CODE

TEST REPORTING

TEST DASHBOARDS

RESOURCES

THE INTERNET OF THINGS

TESTING IN IOT

TESTOPS

TEST SPECIALISTS

TIPS & TRICKS

SANDBOX ENVIRONMENTS

PRACTICE

NETWORKING

REFERENCES

ACKNOWLEDGMENTS

We don't accomplish anything in this world alone... and whatever happens is the result of the whole tapestry of one's life and all the weavings of individual threads from one to another that creates something. - Sandra Day O'Connor

To Celeste: my best friend, life partner and Snowdrop. Without your encouragement this book would be nothing but an echo chamber.

To Devon: mentor and friend whose quiet sensibility and reasoning tempered my rashness.

ABOUT THE AUTHOR

Jonathan is a software quality assurance professional with over 20 years of experience in the field. Born and raised in Pretoria, South Africa, he developed a passion for computers and technology at a young age, with his first computer being a Sinclair ZX81. He has witnessed the evolution of the industry throughout his career, particularly in the areas of DevOps and Agile. Currently residing in Cape Town, South Africa, Jonathan continues to work in the DevOps space, utilizing his expertise and experience to drive innovation and improve software development processes.

INTRODUCTION

If you had asked me if I'd ever write a book on software testing I would have choked on my coffee. I may even have retorted with rhetorical sarcasm and asked if you'd write a book on making your bed. Such was my boorishness and jaded attitude toward software testing and probably software development at the time.

I was in a bit of a rut in my career and I was totally burnt out. It wasn't just the hours and weekends of work, I felt like a washout, out of touch and severely depressed. After just hanging on long enough to make ends meet I finally decided to take a career break. I didn't completely cease to work, I just found creative ways to keep myself busy, more or less on my own terms, while I worked on my wellbeing.

This enabled me to take just enough of a step back to observe and reflect and finally I decided to start writing. At the time, not specifically this book, just random fleeting thoughts and ideas. I'd have discussions with friends and colleagues in the software industry, I would bounce ideas off them and most importantly: Listen.

I started coding when I was about six or seven years old (though my parent’s claim it was earlier than that but I trust neither their memory nor mine). At first, I would create batch files that would run on boot up, back in the MS-DOS days. I had created a simple wizard for each member of the family that would load a sub menu for their profile, for example; recipes for my mother or launch WordStar (a really old word processor) and Jack Nicklaus Turbo Golf for my dad. I had my own profile too with games and software I used, like QBasic and Prince of Persia. This continued throughout my school years as I enjoyed video games (it was the nineties, who didn’t?) and I still wanted to learn more about code so that I can make my own video games.

There were, of course, other distractions that are common in one's youth but I never strayed too far away from a computer. In the holidays I’d often help my dad fix mainframes or run network cables in factories (which I hated) and got my first taste of software bugs and disasters when once a month, without fail, one of my dad’s client’s payroll system would fall over and we’d be called in to help get the system back up and running as soon as possible. I also got into electronics as my dad has a background in electronics being a radar technician in the South African Air Defence Force, knowledge he would use to repair old PLC’s in factories that could not afford to replace them with new, expensive (and imported) equipment and needed to keep manufacturing running 24/7. During this time I learned a bit about electronics too and so my software and hardware knowledge became established as a teenager.

Fast forward a few years and I’m in a corporate office working as a Test Analyst as a contractor. 

A few years later I was retrenched and found work at other companies, getting more and more exposure but somehow staying in the software testing space, which soon got tiresome. I then used what I had self-taught in programming and started automating my testing and just as test automation started becoming increasingly popular I dove in head first. Then came Agile and DevOps (which had already been around for years but remember that most companies were slow to adapt at first) and software delivery became more competitive and rapid. Stakeholders demanded speed in time to market. Compliance became more tricky as new legislations were introduced to keep up with the impact software development had on society and became more observable.

Curiosity and the insatiable desire to innovate constantly drove me on and soon I was in leadership roles and trying to get to grips with Docker and Kubernetes and then Infrastructure as Code and Site Reliability Engineering, Scaled SCRUM, performance reviews, one-on-ones, estimations, planning, people management, managing expectations, spending weekends learning new technology and...I fell. I just completely let go as it felt like I was in the ocean and the current was pulling me further and farther out to sea. The land, distant and blurred. Whispers of doubt, a self-esteem floating on a rickety raft, a hollow, empty conch, neither awake or asleep. I was burned out.

I decided to start my own company, hoping that I could avoid the noise of the software industry and focus on my unique set of skills but ultimately failed. I wasn’t ready. I was still too broken.

I wasn't sure what my next step would be but I had to get back to work, boots on the ground, deployed. This time though, I was going to write about my work. I was going to reboot myself. I was going to write about the next generation of software testing. I suppose, in a way, this book is partially about how I would have done it all differently if I could do it over again.

ABOUT THIS BOOK

Software Testing at Scale: Navigating the Challenges of Agile and DevOps is a comprehensive guide for software testers, developers, and leaders who are looking to improve their approach to quality assurance in an agile and devops-driven world. Written for testers already familiar with the basics, as well as developers who want to understand the impact of quality assurance on their work and leaders who need to adapt their culture and attitude towards testing, this book offers real-world examples and thought-provoking insights on the current expectations of software testers and their role in an era that is all about constant change.

We will revisit traditional approaches that are no longer sufficient, but still applicable, and provide practical strategies for organizations that have adopted agile and devops practices but are still struggling with quality assurance. This book challenges the status quo and poses important questions about why we do things a certain way, making it a valuable handbook for software testing and a reference for software developers.

Whether you're a seasoned tester looking to improve your skills, a developer looking to contribute to the overall quality of the software you build, or a leader looking to adapt your organization's approach to testing, this book offers valuable insights and actionable strategies for achieving success in today's fast-paced development environments.

SOFTWARE TESTING IS DEAD

This is a myth that has been widely circulated and as is the nature of myths they emerge from a culmination of half-truths, opinions and other kinds of misinformation. They also often lack the context of their origins and are mostly exaggerated, usually intended to justify a point of view.

Before going forward lets just consider a few things around the nature of software development itself and why the software testing landscape may have changed.

There are several common developer mistakes that are now avoided before QA would test a feature, thanks to the advanced tooling available to developers.

Here are a few examples:

●  Syntax errors: IDEs and text editors now often include real-time syntax checking, which can help developers to catch syntax errors as they type. This can help to avoid issues that would otherwise have been found during testing.

●  Code style and quality issues: Linting and static analysis tools can automatically check code for issues related to code style and quality. This can help developers to identify and fix issues such as inconsistent indentation, missing comments, and potential bugs before they make it to testing.

●  Memory leaks and other performance issues: Many IDEs and development tools now include built-in tools for monitoring and debugging memory usage and performance, which can help developers to identify and fix issues related to memory management and performance before they lead to problems in testing.

●  Security vulnerabilities: Many development tools now include built-in security checks and analysis, which can help developers to identify and fix common security vulnerabilities, such as SQL injection, cross-site scripting, and other issues, before they are discovered in testing.

As for testing exercises, QA would use to find defects that are now redundant, some examples are:

●  Unit testing: With the rise of automated testing, developers can now use unit testing to test individual code components in isolation. This can help to catch and fix issues early on, before they lead to problems in integration or system testing.

●  Manual testing: With the advent of automated testing, manual testing has become less important as many common test cases can be automated. This has led to a shift towards more targeted manual testing, focusing on more complex or edge cases.

●  Functional testing: With the rise of tools that can automatically check code for issues related to functionality and usability, functional testing has become less important. This has led to a shift towards more targeted functional testing, focusing on more complex or edge cases.

Those are just examples on a practical level but the landscape has also changed due to the advent of DevOps and Agile practices (covered in later chapters).

Overall, these changes in software development have led to a shift towards more proactive and automated testing, which can be more efficient and effective than manual testing alone.

Traditional software testing methods are (mostly) obsolete and a generally negative opinion of software testing may have stemmed from this. As I mentioned, the landscape has indeed changed. Yet there is still a high demand for Software Testers or QA's all over LinkedIn and recruitment agencies all over the world, so what’s up with that?

Software testing in the industry, is in a state of confusion and misguidance, and has in some ways been unfair on employees in these roles, confusing to employers as to what proficiencies these roles require and of course what their market related salaries are, and sadly, some employers are exploiting this (by having impossibly demanding broad skill set for a single individual or underpaying skilled QA engineers under the veneer of their job title, for example; an experienced manual testers that has constantly learned new skills and competencies that are in demand but is utilized like a swiss-army knife. From my experience, this ultimately leads to burnout or silent quitting. However, this is evident in the industry:

●  According to the World Quality Report 2020-2021, there has been an increasing trend towards using automation for software testing, with more than 60% of organizations using automation for functional and non-functional testing. (Fullen et al.)

●  The report also shows that there is a growing need for skilled professionals in the areas of Agile, DevOps and continuous testing, which are becoming more prevalent in organizations worldwide, and that there is a high demand for professionals who can work in these environments.

●  The same report found that there is a shortage of skilled professionals in these areas, which can lead to difficulties in finding the right personnel and can increase the cost of hiring.

●  According to the report, there is a growing trend towards cross-functional teams, where testing is integrated with development, which can lead to confusion over role titles and responsibilities.

●  A survey by Gartner in 2020, found that 42% of organizations believe that the role of QA is changing, and that more than half of them said they are shifting to test automation, as well as shifting QA responsibilities to development teams.

●  According to the same survey, 34% of organizations say they are facing difficulty in finding the right talent for QA, which is indicative of a shortage of skilled professionals in this area.

This is a wake up call for both QA engineers and organizations but here is something to think about: What happened to other roles that DevOps affected?

The shift to DevOps has indeed had an impact on not just manual QA engineers but also on system administrators, database administrators, operations teams, and IT managers. DevOps is a cultural and technical movement that emphasizes collaboration, automation, and integration between development and operations teams. This shift has led to changes in roles and responsibilities for these individuals.

System administrators, for example, have had to adapt to an increased focus on automation and the use of automated tools to manage infrastructure. This has required them to learn new skills and technologies, such as containerization, infrastructure as code, and cloud computing.

Database administrators have had to adapt to new technologies such as NoSQL databases and learn how to use them in a DevOps environment. They also have had to learn how to use automated tools for database management and how to integrate them into the software development life cycle (SDLC).

Operations teams have had to adapt to new methodologies such as Agile and Lean, which emphasize continuous delivery and integration, and require them to work more closely with development teams. This has also required them to learn new tools and technologies such as automation, monitoring, and incident management.

IT managers have had to lead the change and adaptation to DevOps by providing the necessary resources and support for their teams. They have had to align their teams with the DevOps culture and methodologies, and to make sure the teams have the necessary skills and tools to implement DevOps successfully.

In general, it's not an easy transition, but it's not impossible either. The key for individuals in these roles is to stay current with the latest technologies, methodologies and best practices, and to be open to learning new skills and adapting to new roles. They also need to be flexible and willing to take on new responsibilities and to collaborate with other teams. Furthermore, it's important for them to be proactive in seeking out training and development opportunities and to network with other professionals in their field to stay informed about the latest trends and developments.

On the flip side, organizations should provide clear incentives and support for them to learn new skills and adapt to the increased workloads that come with the shift to DevOps. This can be achieved through a variety of means:

●  Provide training and development opportunities: Organizations should invest in training and development opportunities for individuals in these roles. This can include providing access to online courses, workshops, and certifications, as well as on-the-job training and mentoring.

●  Encourage collaboration and knowledge sharing: Organizations should create a culture of collaboration and knowledge sharing, where individuals in these roles can learn from one another and share best practices. This can be achieved through team-building activities, cross-functional teams, and regular meetings or forums where team members can share their experiences and learn from one another.

●  Recognize and reward employees for their efforts: Organizations should recognize and reward employees for their efforts in learning new skills and adapting to new roles. This can include bonuses, promotions, and other forms of recognition such as public praise, awards, and other forms of incentives.

●  Provide competitive compensation and benefits: Organizations should provide competitive compensation and benefits for individuals in these roles, to ensure that they are fairly compensated for the increased workloads and new skills they are acquiring. This can include providing competitive salaries, flexible working arrangements, health benefits, and other forms of compensation.

●  Provide opportunities for career advancement: Organizations should provide opportunities for career advancement for individuals in these roles. This can include providing opportunities for employees to take on new responsibilities, lead projects, and take on more senior roles within the organization.

●  Overall, organizations should be clear and transparent in communicating their expectations and support the development and growth of their employees. Additionally, they should be willing to invest in their employees and provide them with the necessary resources to adapt to the changes. This can help create a positive work environment, increase employee satisfaction and retention, and ultimately lead to better performance, and results for the organization.

DevOps emphasizes the importance of continuous delivery and testing, which also requires a dedicated team of QA engineers and software testers to ensure that