The key to optimizing development while increasing delivery, agility & quality

White paper

Continuous Testing: Digital Demands Continuous Improvement

A Research Report by Apexon

OPTIMIZING SOFTWARE DEVELOPMENT with Continuous testing

The key to optimizing software development while increasing delivery agility and quality

Continuous Testing enables organizations to accelerate cycle times while delivering real business outcomes – higher quality products and services, enhanced customer experience, competitive differentiation, and operational efficiency. This white paper looks at some of the key challenges and requirements involved in pursuing a Continuous Testing strategy and lays out a path for successful deployment and execution.

Software Failure

More than three billion people were affected by software failures in one way or another in 2017. No industry was spared.

Every major sector had several dozen software horror stories – with Public Service/Healthcare and Consumer Tech/Retail industries making up over 63% of the failures. The impact ranged from “best-case” scenarios of user dissatisfaction, poor usage rates, and customer loss – to worst-case scenarios of serious security breaches, shattered brand reputations, severe revenue loss, expensive legal complications, and crashing stock prices.

Tricentis unearthed the scope of these failures in its annual Software Fail Watch report. The 2017 report identified 606 recorded software failures, impacting half of the world’s population (3.7 billion people), $1.7 trillion in assets, and 314 companies. And that’s just the incidents that made their way into the news.

The Need for Speed, But No Seatbelt

A constant stream of new digital offerings have flooded the market. From start-ups to improved offerings and upgrades from incumbents, they threaten competitive differentiators and market share every day.

continuous testing challenges and requirements

software testing and automation processes

The ability to get new functionality to market ahead of the competition – like Bluetooth connected devices, Apple Pay, and facial recognition – can often be the difference between winning and losing.

But in the race to meet customer’s digital demands, many companies are unable to assure optimal performance, functionality, security and quality.

Many of these companies are simply not as mature in their software testing and automation processes as they know they should be; for example, only automating a fraction of their regression testing. We surveyed 100+ customers during last year and the most common reasons we found for this lack of evolution were: An inability to put the latest best practices into action, difficulties putting the right team and skill sets together, and a lack of expertise with new technologies and tools such as Qmetry, Perfecto, Headspin, Tricentis Tosca, Applitools, and many more. Even opensource tools in the market like Selenium, Appium which are widely used, haven’t even been explored, much less taken full advantage by many of these companies. And the faster delivery paces being driven by DevOps and Agile initiatives only exposes those shortcomings further.

In a recent Apexon survey of test/QA professionals and executives in the US and UK, participants reported their teams being very challenged by the increasing complexity of software testing and growing pressure to ensure seamless processes and integrations. This includes different components of Continuous Integration such as Version Control tools, Distributed Build Systems, and Artifactory; Continuous Testing elements such as Execution Grids, Containers, Test Case Analytics, and Build Analytics; Continuous Delivery aspects like Cloud, On-Premise, and Appstore deployments; and Logging and Monitoring layers like Log Aggregators, Log Indexers, and Log Analyzers. 50% of the respondents felt that this high degree of complexity caused by increased technological combinations was a major challenge. An equal number said that they did not have sufficient time to test new apps and services before pushing them into production.

While there is a strong understanding that automation can help speed test cycle time and is critical to advancing their DevOps and Agile goals, the majority of organizations polled by Apexon had automated less than 25% of their testing. In the latest World Quality Report, the average level of overall automation was only 16%. Both surveys highlight a huge opportunity for improvement. But as mentioned earlier, delivery paces are becoming faster, and that will mean more test cases. If organizations don’t figure out, and quickly, that they must automate their existing backlog of manual test cases, they will be stuck with more test cases, less time, and no feasible way to determine the correct gap analysis for future releases – leaving them in the digital dust with low quality product and inferior user experience.

If real progress is to be made, and higher levels of automation are to be achieved in a material way, enterprises must find innovative ways to optimize the complete testing cycles.

Continuous Improvement Required

Without continuous improvement in software delivery, digital projects are destined to fall short of consumer expectations.

This can happen in user experience, functionality, quality, security, performance – or a combination of them all. And when it does, it’s inevitable that declines in market share, revenue and customer loyalty will follow. It might be a slow slide, or a sharp decline, but it will happen, especially where a high-profile software error occurs. Remember Equifax?

The point is companies have to continue to innovate in software testing just as they do in software development. To deliver better products, delivery teams need to be able to test better. Most people are entirely oblivious to how many versions and updates of their most-used apps hit the market weekly. Many auto-update daily, integrate with multiple systems and accounts in the digital and physical world, and all have to operate seamlessly without anyone knowing it. This is just one example of how test and QA organizations need to keep pace in this digital age or app economy. They must look at the entire architecture of their testing cycle, testing tools, testing process, resource skills and how they are organized to enable the best business outcome. There has to be a cultural shift and commitment to continuously improve their processes for faster cycle time and greater operational efficiency Successful organizations achieve Continuous Testing by taking this approach, and not simply looking at speeding up and automating regression cycles to get the desired outcome.

While this all may seem concerning and disheartening to those leading the transformation that many organizations are making, in fact it is meant to do the opposite. While there are some organizations thriving at implementing continuous environments, the fact is that those still making the transformation aren’t alone. And most importantly, there are experts who know how to make this transformation as seamless as possible.

A few of the common misconceptions in continuous testing are what it means, what it encompasses, where to start, and how to make it successful.

assure performance, functionality and quality with continuous testing

The ability to get new software functionality to market ahead of the competition can be the difference between winning and losing. But many companies are unable to assure performance, functionality and quality.

user experience, functionality, and performance with continuous testing

If user experience, functionality, and performance suffer, it’s inevitable that declines in market share, revenue and customer loyalty will follow.

The Role of Continuous Testing

What it means

First off, many people perceive continuous testing as the next evolution of testing from test automation; and thus simply increasing your test automation will result in a continuous testing environment. In fact, continuous testing ins much more than automation.

Automation by definition is the technique, method, or system of operation or controlling g process by highly automatic means, as by electronic devices, reduction human intervention to a minimum. The main words here are ‘a process’, which is a series of actions or steps.

Continuous Testing on the other hand is a methodology, which is a system of ways of doing something. So within Continuous Testing we must look at how we do everything from a holistic stance, not just a tactical one. From a testing perspective, that includes code quality, test coverage, performance testing, and what components of the entire testing environment can be done in parallel or need to be further optimized. This means Continuous Testing needs to enhance the testing of software on a broader level through an organization’s approach to testing.

Through the traditional waterfall approach in development, it was easier to know exactly what you had to test and when to find the respective bugs and defects. However, that approach put a higher importance on UI testing, but all layers need to be accounted for in today’s digital landscape – API, component, and unit testing all need equal consideration in the adoption process. This is why there is a need to look at the testing activities from a holistic perspective which originally wasn’t there. Additionally, now development isn’t done in waterfall. It is done to increase speed and success within an Agile methodology. Again, this isn’t a direct process, but a way of reprioritizing items based on ability, importance and potential success. Those fundamentals of agile development are often what lead to the success of Continuous Testing: reprioritizing testing activities to test what you can as soon as you can by testing in the early stages of development.

While this testing early in the stages of development may seem like an easy adoption – this tends to be an issue with how traditional organizations are set up due to the defacto cultural implications that waterfall provided. Gaining the trust of developers in the initial phases of projects is pivotal to understanding not only their development methodology, but also why they are prioritizing certain aspects over others. This in turn gives testers the ability to provision the correct environments based off what could potentially be delivered in the latest sprint. Once this understanding occurs, testing teams will be able to leverage their expertise to enhance the performance and code quality of an application based on that same feature prioritization. The ability for testing to provide actionable insights earlier in the process creates an invaluable benefit to both development time and consumer experience.

That is why we mentioned earlier there needs to be continuous improvement. By accelerating the feedback cycle of this enhanced environment, we can simultaneously limit business risks and increase the business values associated with any given release. Continuous Testing extends beyond traditional test automation strategies to address the faster pace and increased complexity of current application development and delivery.

The Role of Continuous Testing

What Does It Encompass

Continuous Testing is not necessarily new, but very few organizations have truly embraced it in its real spirit.

Essentially, Continuous Testing demands holistic testing optimization for higher rates of effective testing and that can’t be done without automation. This can only be enabled by making some fundamental changes in the test and QA process, including:

Implementing DevOps lifecycle with CI to leverage existing automated testcases to execute for every change being made as part of build process

Using Robotic Process Automation (RPA) to eliminate duplicates and enhance efficiencies of existing test cases

Implementing strict automation guidelines with self-healing protocols to confirm adherence

Leveraging data to prioritize test execution by risk, objective, and pass/fail prediction

Using test data management and service virtualization to conduct valid end-to-end tests in representative user and operating environments

Aligning AUT components with quality milestones and risk analysis that can be factored into release decisions for the best possible CX

Many businesses focus on automating just one or two aspects of the testing ecosystem. There are a lot of tools on the market – many good ones like Applitools, Hexawise, and Mabl– that do just that. But these tools can only do so much on their own. Continuous Testing requires tight integration and coordination between every stage of the test/QA process starting with the definition of Requirements & Expectations and optimizing future test with the added Insights, Analytics and User Feedback.

As a result of this broader, more integrated view, Continuous Testing enables organizations to understand where they can and should automate for the best results, as well as where they shouldn’t. This translates into big cycle-time efficiencies and effectiveness to get desired test coverage while also truly enhancing the end product.

Where to start

This approach has helped countless personas within the testing and development lifecycles, but their adoptions often come with varying degrees of challenges and hurdles at the beginning of the process.

minimizing existing operational costs by optimizing testing repositories

With so many tools readily available in the market to help different testing phase, it is important to align your testing evolution with your business strategy.

For example, running your entire operation as Cloud Native and having automatic triggers to spin up or down testing environments with absolutely no manual intervention. A great example of what wants to be technically achieved from a testing standpoint, but the business initiative may be more geared to minimizing existing operational costs by optimizing testing repositories increasing UAT automation tools.

Companies and organizations often want “the shiniest and newest thing” first, and to make testing and enabler to your company’s success, understanding that prioritization of what is needed and what is wanted must occur. Yes, there are major advantages that mature organizations can take advantage, but more often than not – addressing the maturity of a testing organization has to be done first. These aren’t always the shiniest tools, but they often are the most pivotal.

In situations where the applications are already in market and departments need to adopt the agile approach, the best approach is to implement complementing tools in parallel. This can be done in multiple phases with multiple tools, or in one phase with multiple tools. For example in phase of test identification and creation, an organization can both optimize the existing test repository, implement model based testing for new application functionalities, and integrate auto test case generation and changes to existing functionalities.

In other situations, the implementation and optimization of testing fundamentals like service virtualization and microservices designed to leverage today’s cloud technology are applicable. However, this all depends on the complexity and dependencies that exist within an organization’s digital ecosystem. But regardless of the maturity of an organization’s digital ecosystem, infrastructure, or testing department – there are small steps everyone can take to move towards an more efficient and effecting methodology. The benefits from the increase at an as exponential rate due to these tools being able to generate mass amounts of data that create a feedback loop for better decision making.

How to make it successful

There is no doubt within the industry that Continuous Testing can benefit every organization. They all agree, yet follow it up with ‘when can you have it done by?’.

They all agree, yet follow it up with ‘when can you have it done by?’. This is when we have to reiterate the fundamentals we established earlier: process versus methodology.

Continuous Testing it isn’t a plug-and-play process, it is a methodology that needs to be pruned, tuned, and enhanced along the longevity of an organization. The most effective way to get the point across is by proclaiming “Well, when do you want to stop ensuring the confidence levels of your coverage? When do you want to stop optimizing your resources and become more cost and effort efficient?”

While this is a blunt approach, the bottom line is while we are moving at the speed of light, the lines between testing being accomplished from a syntaxial context versus and a semantical one blur more and more each day. The only way to incorporate the evolving parameters of what is done correctly from a structural execution standpoint (syntaxial) as well as what is done correctly by the desired end result (semantically), is to implement a Continuous Testing methodology that can adapt in parallel to your application development.

The most impactful way to make those decision is by investing in dynamic, real-time dashboards. Just like it was previously mentioned, all of these tools give us data to make better decisions. But there are so many tools today integrated into your organization and environment – not just in testing. They are all spitting out vast amounts of logs which all need to be aggregated to make those best decisions. For example, once a build gets checked in you want to start the unit testing, and follow-up stages with the right quality gates at each level. That way when a build fails, you will be able to and want to know why so you can take the right. Action.

Continuous Testing

Advantages

It’s hard to argue with all the benefits of Continuous Testing, but how do you get there?

Test/QA Benefits

End-to-end coverage in testing setup, management, control, and visibility

Automated test case generation and backlog clearance for optimal coverage

Intelligent test result analysis to increase response time

Intelligent test automation execution and infrastructure optimizations

Accelerated release cycle

Effective test data management for repeated execution

Incorporation of next-gen technology for automating new challenges like AR/VR/Chatbots…

Business Impact

Enhance customer experience with better quality products

Deliver high-quality products faster

Adjust quickly to increase customer/employee retention, satisfaction, and usability

Reduce cost while improving operational efficiency

Continuous Testing

AI: ENABLING

For organizations to take advantage of these benefits, they often must address their automation, or lack there of. Continuous Testing can’t happen without a high percentage of the test footprint being

Artificial intelligence (AI), including machine learning and BOTs, has emerged as a major disruptive force in the current digital landscape. 43% of the respondents to the Apexon survey mentioned above see the increased use of AI as inevitable in new applications and devices, as well as software testing processes. These AI-powered test automation solutions, also known as intelligent testing, can help accelerate test automation in multiple ways by:

Test Case Management Optimizing test case management for eliminating duplicates and achieve optimum test coverage

Test Result Analysis Automating test result analysis to identify, categorize, and prioritize defect patterns for regression engineers

Adaptive Learning Using adaptive learning to identify high risk defects and vulnerabilities to focus on Risk Based Testing first

Self-Healing Test Automation Using self-healing test automation technologies to automatically align with changes in UI for testing resulting in optimized maintenance efforts

By incorporating AI into their test automation efforts, businesses can increase their ability to leverage Continuous Testing to overcome time-to-market pressures while also meeting the stringent quality standards needed to achieve desired business outcomes.

Many believe AI will be the core enabler for organizations trying to achieve widespread test automation to fuel Continuous Testing. Nearly 60% of respondents to the Apexon survey see AI and self-learning technologies as a priority in their QA efforts, and there is a high demand for talent with intelligence-driven QA skills.

APEXON

Continuous Testing Services

At Apexon we have created a Continuous Testing framework to optimize every aspect of the test/QA process. Our approach is designed to enhance the success of software development and its impact on businesses and their end users.

Apexon provides the systems, processes, engineering, and automation needed to enhance the agility, efficiency and quality at each testing phase of the software development lifecycle – resulting in accelerated development cycles, faster time to market and increased customer satisfaction and loyalty.

We leverage our extensive expertise in Quality Engineering and integrate the latest technologies to incorporate continuous feedback and analytics into testing activities to ensure business objectives and user requirements are being met even as they evolve.

Our approach focuses on:

Continuous testing can’t happen without a high percentage of your test footprint being automated.

Continuous Testing

Addressing the Key Challenges

By looking at all phases of the QA process, Apexon can help organizations identify the key challenges they face in their software testing, pinpoint the best places to start making improvements, and understand how that will impact and improve other areas.

Requirements & Expectations	Lack of traceability – requirements to test cases	Expert assessment of current test environment, framework and tools Migration roadmap
Test Case Definition & Maintenance	Lack of categorization of tests Delay in understanding health of the product	Categorisation of tests based on priority and risk; eg. Basic acceptance, feature sanity, smoke etc. Better/faster understanding of product health with fewer tests Model based test case generation and maintenance Remove duplicates and refactor to increase reusability
Test Automation	Insufficient automation delays QA cycle time	Test Automation services Acceptance Bots for code check and compliance Automation tool selection Test Automation framework for consistency and reusability
Test Data & Regression (VP of engineering/CTO)	Lack of skillset/ enablement of existing team members; difficulty recruiting and scaling	Regression Testing services Functional Testing Compatibility Testing Visual Testing UI/UX Testing Scale and support with qualified external resources
Root Cause Analysis	Lack of real-time reporting on build and test process	Intelligent Test Automation with real-time dashboard
Insights & Analytics Feedback	Lack of centralized dashboard – for quality metrics, build metrics	Intelligent Testing with real-time dashboard

apexon

Continuous Testing Services

Assessment of Current Frameworks/Tools
Assistance with Tool/Framework Selection
Migration Roadmap
Bot Deployment (for migration)
Test Execution
Automated Testing
Root Cause Analysis
Test Data Management
Test Environment Management
Service Virtualization

Regression Testing
SOR/SOE/SOC Predictability
Test Case Maintenance
Test Case Definition
Business Objective Assurance
User Expectation Assurance
Multi-Dimensional Result
Analysis & Feedback
Maintenance & Support

apexon

Continuous Testing Services

Once organizations have properly deployed Continuous Testing, they are typically utilizing microservices and service virtualization, provisioning test infrastructures, automating parallel test executions, and gaining valuable diagnostic feedback with predictive and prescriptive analytics.

This data can be used to take your Continuous Testing initiative to the next level.

Combining our expertise in quality engineering and test automation with the latest AI technologies, Apexon helps organizations leverage these analytics to optimize their testing processes even further.

These capabilities include:

Expert, certified
resources

to help delivery organizations to optimize their test/QA quickly and efficiently.

Proven methodologies

which accelerate problem-solving and ensure clients get it right the first time.

A week-long assessment

to identify additional tools for further automation and provide a roadmap for implementation in the organization with defined success criteria.

SaaS & on-premise
solution scenarios

which provide multi-app, multi-branch release planning and management pipelines.

Actionable release
pipeline analytics

leveraging integration tools across the release pipeline lifecycle for real-time, data-driven decision-making about release management and the most affective DevOps pipeline.

AI accelerators

to optimize deployment and reduce risk.

Continuous Testing
roadmap & services

to enhance organizational maturity and increase self-sufficiency.

Analytics accelerator
solutions

that can ingest, and aggregate the varied data generated from disparate data sources and provide effective custom visualizations for different personas; e.g., Development Managers, QA Managers, DevOps Managers — to help them to make more informed decisions to improve overall pipeline performance.

Predictive &
prescriptive
recommendations

to enterprise test/QA teams; e.g., prompting users of systems about possible delays in pipeline executions based on prior data experience of changes to specific code areas; or regression recommendations to QA managers to select more effective test plans.

Test Automation Factory

Outcome-Based Outsourced Services

For companies that do not want to deal with the significant skill and resource requirements of Continuous Testing, Apexon offers Test Automation Factory. This is a full or partial- outsourced offering that delivers specific and measurable business outcomes.

Test Case Management

Big backlogs of test cases causing organizations to be less agile than the business requires
Increasing cost, complexity and cycle time for software testing
Inability to get quality new digital services to market ahead of competition
Lack of visibility in testing issues
Disappointment with results achieved with Test Automation to date

Apexon Test Automation Factory Outcome-Based Model

Business case definition with ROI/Operation/Outcome-based metrics
Factory requirements including vendors and tools need to get started
Factory requirements including infrastructure, process, vendors and tools need to get started
Accelerated Agile initiatives
Bigger ROI from Test Automation efforts

Apexon Approach

Putting Continuous Testing to Work in Your Organization

Continuous Testing enables organizations to put in place the systems, processes, engineering, and automation needed to truly accelerate cycle time and deliver real business outcomes – higher quality products and services, enhanced customer experience, competitive differentiation, and operational efficiency.

Continuous Testing increases the flexibility of their delivery

Furthermore, it increases the flexibility of their delivery organization to adjust to changes in the business, an increasingly important capability – no matter what business you are in.

Apexon can help streamline your path to Continuous Testing and take advantage of all the functional and business benefits that come with it

Continuous Testing: Digital Demands Continuous Improvement

The key to optimizing software development while increasing delivery agility and quality

Software Failure

Continuous Improvement Required

What it means

Continuous Testing Services

Addressing the Key Challenges

Outcome-Based Outsourced Services

Putting Continuous Testing to Work in Your Organization

Download Resource