Share this page with a friend.

Identifying and implementing the optimal mobile QA strategy

Mobile Application Quality

Implementing the optimal mobile testing strategy

The Mobile Market is Booming

The mobile market is booming, the app economy is exploding and the need for mobile testing continues to grow. But mobile testing is very different from traditional enterprise, web and packaged application testing. There are various devices with numerous form factors, fast changing mobile OSs, and growing demand to support mobile applications globally.

Each phase of mobile quality brings its own set of unique challenges to be addressed. For example, in Planning, there needs to be a focus on device selection. In Definition, specific types of tests and processes need to be well defined. In Execution, remote access and conditioning issues for devices need to be covered. With Automation, users have to account for ROI, tools, types of test cases, etc. And in the last phase of mobile quality, Optimization, post release support and upgrades need to be measured and addressed.

To do all this, the overall testing strategy needs to be well defined with the right tools, framework and technology. This white paper discusses the different considerations for mobile testing and lays out a blueprint for creating an optimal mobile testing strategy with risk mitigation plans and streamlined processes.

And Continues to Evolve

The mobile landscape is changing real time – new devices, new versions of operating systems, new carriers and of course, new applications. The challenge is to make your application work and work well across this constantly evolving ecosystem.

This means quality mobile app development needs

Global testing
coverage
Cross-platform testing
capabilities
Ongoing support and user
experience management

The Mobile Boom by the Numbers

mobile-stats
  • There are 1.6 million apps in Google Play and 1.5 million apps in Apple’s App Store
    (Source: Statistica, June 2015)
  • $15 billion has been paid to developers by Google and Apple in 12 months (in the 12 months up to June 2015)
    (Source: Benedict Evans, June 2015)
  • Apps account for 77% of time spent consuming digital media
    (Source: comScore, 2015 Mobile App Report)
  • By 2020, around three-fifths of the global population will have a mobile subscription
    (Source: GSMA Global Mobile Economy Report 2015)
  • By 2020, there will be 4.6 billion unique mobile subscribers
    (Source: GSMA Global Mobile Economy Report 2015)
  • Mobile app revenues will reach $76.52 billion by 2017
    (Source: Statistica, June 2015)

Mobile Testing is Different

Mobile testing is different from traditional enterprise QA. With a zillion device / OS combinations challenging testing coverage, security issues, and varied device connection speeds impacting application performance, mobile testing quickly becomes the largest elephant in the room.
Apart from adding device provisioning and device level validations, there are issues related to memory, network bandwidth, cross platform, screen size and resolution compatibility. There are functional issues related to swipes, UI and localization. There are security concerns such as caching, uploading personal information to servers, virus scanning, memory corruption and more. All this makes mobile QA a very different animal.
mobile-testing-phases

Mobile Testing Challenges – At a Glance

Industry challenges

  • BYOD Mobile Device Management
  • Enterprise Level Security
  • Testing on the Cloud
  • Consumer and Enterprise Mobility
  • Native vs. Hybrid vs. mWeb vs. HTML5
  • Mobile Test Automation, Security Tools

Network Service Providers

  • Network Delay and Jitter
  • Network Disruption Testing
  • Network Performance Testing
  • Network Congestion Testing
  • Multi-network/Trace Testing
  • Enterprise Carrier Network Switching

Networking Connectivity/Varying Speeds

  • 2G, 2.73G (GPRS, EDGE), 3G, 4G
  • WiFi (Secured, Unsecured)
  • WiMax
  • VPN, Hotspot, Tethering
  • Cellular Internet, Telecom Carriers
  • Data Loss Over Network
  • Bandwidth, Signal Loss

Device Diversity

  • Device Makers, Models
  • Smart Phones, Phablets, Tablets
  • Device Density (ldpi, mdpi, hdpi, xhdpi)
  • Device Screen Sizes (Small, Normal, Larger and Extra Large Screens)
  • Device Orientation Challenges
  • Device Browsers, Rendering Differences

Device OS Fragmentation

  • iOS (3, 3GS, 4, 4GS, 5.x, 6.x)
  • Android (2.x, 3.x, 4.x)
  • Blackberry (4.x, 5.x, 6.x, 7.x)
  • Windows Phone (2003, 5.x, 6.x, 7.x)
  • Symbian, BREW
  • J2ME, BADA, others

Device Hardware

  • Limited Processing Capability / Power
  • Locations and Languages
  • Limited Memory Size
  • External Memory SD Card Availability
  • Device Comm Protocols (WAP, HTTP)
  • Device Configuration

Device Data Input Methods

  • Touch Screen Keypad
  • Non Touch, Soft/Physical Keypad
  • Track Balls, Qwerty Keypad
  • Voice (SIRI, Skyvi)
  • Split Keypad
  • Copy and Paste
  • Blink of an Eye

Device Capability & App Functionality

  • Bandwidth Connectivity
  • Camera
  • Phone / SMS / MMS
  • Contact List

Others

  • Rapid Rollouts, Deployment Velocity
  • Mobile Context, Usability, Performance
  • Energy Consumption Monitoring
  • Testing with Emulators / Simulators
  • Device Interruption Scenarios
  • Field Testing, Test Execution Challenges

Mobile Demands a Comprehensive Test Strategy

Mobile demands a comprehensive testing strategy. This means:

Interaction with Device/OS native capabilities

Optimal combination of devices and OSs

Effective combination of manual and automated testing

Need for faster agile adoption

On-time and off-line connectivity

Mobile Native Apps vs. Mobile Web vs. Hybrid

The best strategy to test a mobile app depends on the type of app. If it is a native app, then there are certain conditions which need to be tested. The tester needs to ensure that the app can be successfully downloaded, executes on the device and interacts with the backend infrastructure. Apps render differently across devices, so each app needs to be thoroughly tested on all relevant devices. We must also ensure backward compatibility with selected older generation devices and OSs. Creating test plans for manual testing on each of these devices can be a cumbersome process.

Mobile web apps pose a different set of challenges. These apps are viewed by users across the globe, presenting unique challenges in terms of networks, scripting and devices. No matter what type of app it is – native, hybrid or web – manual testing of these apps has become more rampant. Testing on simulators does not accurately replicate real user experience. And, testing on all device/OS combination is not practical .The other issue that we have already touched on is the lack of a standardized mobile SDLC, which results in an inconsistent testing methodology.

The Right Testing Approach for Your Enterprise

Selecting the right mobile testing methodology is critical to the success of any mobile initiative. You need a partner who understands the intricate details and brings the required knowledge as well as the right infrastructure and enabling technology.

enterprise-mobile-testing-approach
enterprise-mobile-testing-dimensions

It’s imperative to link business goals and quality goals, leveraging key quality dimensions specific to mobile testing.

This means:

  • Understanding the quality dimensions (see graphic)
  • Aligning quality dimensions with a tactical testing plan
  • Selecting the right service partner to deliver these dimensions

Key Activities and Outputs from Mobile Testing

Test Strategy

Define key policies, components, review criteria, approach, testing types, scope, communication and resource planning necessary to control and manage risks throughout the Software Testing Program.

Test Plan

Address key areas such as test approach, scope, test types, review criteria, test data, incident management, resources and supplementary materials.

Test Case/Specification

Identify and outline key objectives of the test validation, related requirements and testable items. All required test pre-conditions and processing in detail with associated supplementary materials.

Test Summary Report

Provide dynamic snapshot of progress throughout the testing life cycle, providing visibility to management and stakeholders. The overall summary, scope deviation, requirements coverage, analysis/design progress, execution progress, review criteria and outstanding defects are areas that will be addressed throughout the life of the project for review, approval and sign-off.

Test Automation

Define repeatable process and help you streamline your testing to support on-demand releases and critical software changes. A hybrid automation framework is deployed for immediate use, eliminating development ramp-up time.

Traceability Matrix

Provide a relational model of your software testing validation framework. Requirements and User Stories are mapped to Test Cases and Test Plan and are executed against target software build releases, configurations and environments.

Test Results

Each of the test result is well supported with tester comments and screenshots along with the required logs for the developer to understand the issue, and be able to reproduce the same without any trouble.

Test Assessment

Provide an overall review of the efficiency and effectiveness of testing activities. Assessment methodology uses interviews with key testing leaders and an analysis of your testing deliverables in a short time frame to provide recommendations.

Different Testing Types: What You Need to Know

There are different types of testing that need to be addressed – independent of the platform – to ensure mobile app quality. Following is a snapshot of the various testing types and test cases.

Functional Testing

Scenarios Description
Sign-up & Login Make sure the app is easily accessible to the users.
Menu Options Ensure menu items like Help, About, etc. are easy to find and navigate.
Keys Make sure key functionality is clear and consistent. Assure the app will run with a keyboard and/or touch-screen.
Data Handling Managing data handling, data deletion and unintended data deletion (important for games in particular) in the testing scope.
Connection Speed / Carrier Incorporate most popular and likely carriers in most basic app tests to ensure performance.
Operating System Ensure app performance on different mobile OSs.
Screen Size Testing for screen size discrepancies.
Interruptions Test mobile app behavior against device status variations such as battery strength (full, medium, low), incoming calls, SMS, MMS and video calls, etc.
Error Messages Ensure error messages are clear, concise and actionable.

Usability Testing

Scenarios Description
Connectivity Check for slow and unreliable wireless network connections with low bandwidth taking into account various network conditions.
Small Screen Size Check for physical constraints of mobile devices, especially small screen sizes which might impact app usability.
Different Display Resolution Check for low resolution that can degrade the quality of the multimedia information.

Layout & Design Testing

Test Case Description
Page Layout Evaluate the overall effectiveness of the page layout.
Color Schemas Evaluate use of colors in the background, text, links, icons and buttons.
Findability Evaluate visibility and accessibility of key items like Help, About, Instructions, search.
Localization Evaluate app consistency in terms of text, message and symbols in all supported languages.
Touchscreens Test for different touchscreen scenarios: multi-touch vs. single touch; long touch vs. short touch; button size, workflow and any other supported input methods.
Trackballs, Track Wheels and Touchpads Confirm user ability to move from object to object regardless of navigation tool.
Soft Keyboards Check user interaction with, and accessibility of, the soft keyboard.
Hard Keys Check the available device hard keys; e.g., Start, Home, Menu, Back – interact with the application in similar way.
Short Cuts Check for expected shortcuts for the device and test their use within the application.

Localization Testing

Test Case Description
General Check user interface, content files, keyboard, text filters, hot keys, spelling rules, sorting rules, upper and lower case conversion rules.
Format Testing Check for date formats, currencies per country.
Local Market Compliance Check app compliance with local laws and regulations.
Content Testing Check static and dynamic content like video, catalogs, search results, metadata.
Characters Check different languages and character sets.
Postal Codes Verify if the target countries support only number or alphanumeric.
Language Direction Verify the direction for the way language is written – left to right or right to left.
Currency Conversion Test currency conversion for Internet retail applications.
Tax Calculation Verification Test rules around the calculation of sales tax and VAT.
Translation Validation Test for on-demand translation validation and accuracy in including accessibility from any market location.

Security Testing

Web Applications Test with common web security and testing tools such as Firefox web developer. Ensure that web proxy is used to intercept all mobile device traffic to monitor data and test for security issues.
Format Testing Test for password/usernames being left unencrypted on device. Ensure footprint and fingerprint analysis can be used to make hash files.
Local Market Compliance Research permissions configuration on the device; test to ensure that app can get access to the device areas it needs in order to function properly, and it provides useful error messages directing the user to set the permissions properly, if it cannot. Verify that the app can be successfully installed at all expected device security level settings. After installation, ensure it is possible to change the security settings of device or application permissions. Include some test cases to cover these scenarios as well.

Load & Stress Testing

Test Infrastructure Management & Reporting Ensure there is enough research on app to decide upon the test infrastructure on live or simulated loads. Feed the mobile app under test with the largest tasks it can handle. Make sure to increase the load on the system via automated tools to simulate real time scenario with virtual users. Set up report management using internal and external bug reporting and test management tools.

Interruption Testing

Incoming Calls and SMS, Battery and Cable Removals

Must be tested for:

  • Incoming and Outgoing SMS and MMS
  • Incoming and Outgoing Calls
  • Battery Removal
  • Cable Insertion and Removal
  • Network Outage and Recovery
  • Media Player On/Off
  • Device Power Cycle

Test Automation

Because mobile testing must be performed on so many different combinations of devices, browsers, and operating systems, doing it manually quickly becomes impractical in terms of both time and expense. Test automation can bring significant efficiencies to the process – dramatically accelerating go to market plans and reducing costs.

Why Automate?

The Device OS Maze

The virtually endless combination of devices, OS models, and device sizes. Test automation enables QA teams to test the same functionality on different devices, OSs and form factors.

Changing Industry Trends

Constant OS updates, upgrades, new devices, and new manufacturers. In order to ensure that the application is up-to-date and supported by new trends, testing it in a timely manner becomes important.

Rapid Upgrades to Apps

Apps are regularly updated with new features in order to keep up with market demand. In order to introduce upgrades into the market quickly, automated testing is crucial.

The Infostretch Approach to Automation

Brower Plug-IN

  • Simulate device request using browser proxy plug-in leveraging user-agent string and other device profiles
  • Ability to run manual/automated tests for functional testing

Native API / Tools / Simulators

  • Leverage iOS and Android APIs and tools
  • Ability to run automated tests for functional and regression testing

end-to-end automation

  • Define a data driven test automation framework using remote device access tools, native tools and traditional test automation tools

Real device automation

  • Setup private cloud with handsets
  • Created virtual function library to increase reuse and reduce script maintenance

Picking the Right Mobile QA Partner

Mobile testing can be complex but app quality is critical to success in today’s business world. The traditional rules of testing do not apply to mobile due to the fast changing landscape, unique features of apps and devices, limited bandwidth and unreliability of wireless networks.
There is a need to develop mobile specific test methodologies and guidelines and therefore a need to rely on a testing partner. This partner should be able to provide insights to modern approaches in testing and suggest the right strategy and coverage based on specific business needs.
After defining the strategy, your QA partner should provide a well-defined plan on what types of testing and what attributes need to be tested, validated and verified. Ideally, they would have executed testing strategies for enterprises across different verticals – retail, travel, banking and insurance, and digital entertainment – so they understand all the different possibilities and use case scenarios.

What to Look For When Selecting a Vendor

  • The ability to use the right combination of emulator and real device testing to ensure predictability; emulators with scripting help to test many aspects at an effective cost
  • The ability to provide intelligence behind use of select devices for your application
  • The ability to provide automation and agile solutions for faster time-to-market by building the right user stories, increasing use of test sprints and creating script reusability
  • The ability to have access to the real devices remotely when it’s not possible to run physical test on a slew of mobile devices
  • The ability to increase the app store star ratings of the app by closely monitoring it and providing reactive and proactive remediation
  • The ability to optimize the adoption and monetization of the application post-launch

Your Requirements

We would love to hear more about your project.

Even a short phone call can help us explain how our solutions can accelerate your mobility, jump start your continuous delivery and help reduce costs.
And that’s just for starters, understanding more about your project will enable us to build a solution that fits your objectives, infrastructure and aspirations!

Contact us
Our team is available to chat through your project.   Contact us

Latest News, Events, and Thought Leadership

Hear us Speak on “How to Make Use of Serverless Compute Solutions for IoT Device Simulation”
Learn More
Deven Samant, Director
Nov 1, 2017 @ 4:20pm to 4:55pm
Santa Clara Convention Center
Nov 02, 2017
See More Events
Auto-Convert Freestyle Jenkins Jobs to Coded DevOps Pipelines Webinar
Register Now
December 7, 2017
11AM PT / 2PM ET
Online
December 8, 2017
See More News
Re-tool Your CI/CD Pipeline for Infrastructure and Configuration as Code
Read More

The triple effect of cloud adoption, DevOps and CI/CD has dealt a blow to the old tried-and-tested means of...

See More Blogs