Implementing the optimal mobile testing strategy
The mobile market is booming, the app economy is exploding and the need for mobile testing continues to grow. But mobile testing is very different from traditional enterprise, web and packaged application testing. There are various devices with numerous form factors, fast changing mobile OSs, and growing demand to support mobile applications globally.
Each phase of mobile quality brings its own set of unique challenges to be addressed. For example, in Planning, there needs to be a focus on device selection. In Definition, specific types of tests and processes need to be well defined. In Execution, remote access and conditioning issues for devices need to be covered. With Automation, users have to account for ROI, tools, types of test cases, etc. And in the last phase of mobile quality, Optimization, post release support and upgrades need to be measured and addressed.
To do all this, the overall testing strategy needs to be well defined with the right tools, framework and technology. This white paper discusses the different considerations for mobile testing and lays out a blueprint for creating an optimal mobile testing strategy with risk mitigation plans and streamlined processes.
Interaction with Device/OS native capabilities
Optimal combination of devices and OSs
Effective combination of manual and automated testing
Need for faster agile adoption
On-time and off-line connectivity
The best strategy to test a mobile app depends on the type of app. If it is a native app, then there are certain conditions which need to be tested. The tester needs to ensure that the app can be successfully downloaded, executes on the device and interacts with the backend infrastructure. Apps render differently across devices, so each app needs to be thoroughly tested on all relevant devices. We must also ensure backward compatibility with selected older generation devices and OSs. Creating test plans for manual testing on each of these devices can be a cumbersome process.
Mobile web apps pose a different set of challenges. These apps are viewed by users across the globe, presenting unique challenges in terms of networks, scripting and devices. No matter what type of app it is – native, hybrid or web – manual testing of these apps has become more rampant. Testing on simulators does not accurately replicate real user experience. And, testing on all device/OS combination is not practical .The other issue that we have already touched on is the lack of a standardized mobile SDLC, which results in an inconsistent testing methodology.
Selecting the right mobile testing methodology is critical to the success of any mobile initiative. You need a partner who understands the intricate details and brings the required knowledge as well as the right infrastructure and enabling technology.
It’s imperative to link business goals and quality goals, leveraging key quality dimensions specific to mobile testing.
Define key policies, components, review criteria, approach, testing types, scope, communication and resource planning necessary to control and manage risks throughout the Software Testing Program.
Address key areas such as test approach, scope, test types, review criteria, test data, incident management, resources and supplementary materials.
Identify and outline key objectives of the test validation, related requirements and testable items. All required test pre-conditions and processing in detail with associated supplementary materials.
Provide dynamic snapshot of progress throughout the testing life cycle, providing visibility to management and stakeholders. The overall summary, scope deviation, requirements coverage, analysis/design progress, execution progress, review criteria and outstanding defects are areas that will be addressed throughout the life of the project for review, approval and sign-off.
Define repeatable process and help you streamline your testing to support on-demand releases and critical software changes. A hybrid automation framework is deployed for immediate use, eliminating development ramp-up time.
Provide a relational model of your software testing validation framework. Requirements and User Stories are mapped to Test Cases and Test Plan and are executed against target software build releases, configurations and environments.
Each of the test result is well supported with tester comments and screenshots along with the required logs for the developer to understand the issue, and be able to reproduce the same without any trouble.
Provide an overall review of the efficiency and effectiveness of testing activities. Assessment methodology uses interviews with key testing leaders and an analysis of your testing deliverables in a short time frame to provide recommendations.
|Sign-up & Login||Make sure the app is easily accessible to the users.|
|Menu Options||Ensure menu items like Help, About, etc. are easy to find and navigate.|
|Keys||Make sure key functionality is clear and consistent. Assure the app will run with a keyboard and/or touch-screen.|
|Data Handling||Managing data handling, data deletion and unintended data deletion (important for games in particular) in the testing scope.|
|Connection Speed / Carrier||Incorporate most popular and likely carriers in most basic app tests to ensure performance.|
|Operating System||Ensure app performance on different mobile OSs.|
|Screen Size||Testing for screen size discrepancies.|
|Interruptions||Test mobile app behavior against device status variations such as battery strength (full, medium, low), incoming calls, SMS, MMS and video calls, etc.|
|Error Messages||Ensure error messages are clear, concise and actionable.|
|Connectivity||Check for slow and unreliable wireless network connections with low bandwidth taking into account various network conditions.|
|Small Screen Size||Check for physical constraints of mobile devices, especially small screen sizes which might impact app usability.|
|Different Display Resolution||Check for low resolution that can degrade the quality of the multimedia information.|
|Page Layout||Evaluate the overall effectiveness of the page layout.|
|Color Schemas||Evaluate use of colors in the background, text, links, icons and buttons.|
|Findability||Evaluate visibility and accessibility of key items like Help, About, Instructions, search.|
|Localization||Evaluate app consistency in terms of text, message and symbols in all supported languages.|
|Touchscreens||Test for different touchscreen scenarios: multi-touch vs. single touch; long touch vs. short touch; button size, workflow and any other supported input methods.|
|Trackballs, Track Wheels and Touchpads||Confirm user ability to move from object to object regardless of navigation tool.|
|Soft Keyboards||Check user interaction with, and accessibility of, the soft keyboard.|
|Hard Keys||Check the available device hard keys; e.g., Start, Home, Menu, Back – interact with the application in similar way.|
|Short Cuts||Check for expected shortcuts for the device and test their use within the application.|
|General||Check user interface, content files, keyboard, text filters, hot keys, spelling rules, sorting rules, upper and lower case conversion rules.|
|Format Testing||Check for date formats, currencies per country.|
|Local Market Compliance||Check app compliance with local laws and regulations.|
|Content Testing||Check static and dynamic content like video, catalogs, search results, metadata.|
|Characters||Check different languages and character sets.|
|Postal Codes||Verify if the target countries support only number or alphanumeric.|
|Language Direction||Verify the direction for the way language is written – left to right or right to left.|
|Currency Conversion||Test currency conversion for Internet retail applications.|
|Tax Calculation Verification||Test rules around the calculation of sales tax and VAT.|
|Translation Validation||Test for on-demand translation validation and accuracy in including accessibility from any market location.|
|Web Applications||Test with common web security and testing tools such as Firefox web developer. Ensure that web proxy is used to intercept all mobile device traffic to monitor data and test for security issues.|
|Format Testing||Test for password/usernames being left unencrypted on device. Ensure footprint and fingerprint analysis can be used to make hash files.|
|Local Market Compliance||Research permissions configuration on the device; test to ensure that app can get access to the device areas it needs in order to function properly, and it provides useful error messages directing the user to set the permissions properly, if it cannot. Verify that the app can be successfully installed at all expected device security level settings. After installation, ensure it is possible to change the security settings of device or application permissions. Include some test cases to cover these scenarios as well.|
|Test Infrastructure Management & Reporting||Ensure there is enough research on app to decide upon the test infrastructure on live or simulated loads. Feed the mobile app under test with the largest tasks it can handle. Make sure to increase the load on the system via automated tools to simulate real time scenario with virtual users. Set up report management using internal and external bug reporting and test management tools.|
|Incoming Calls and SMS, Battery and Cable Removals||
Must be tested for:
The virtually endless combination of devices, OS models, and device sizes. Test automation enables QA teams to test the same functionality on different devices, OSs and form factors.
Constant OS updates, upgrades, new devices, and new manufacturers. In order to ensure that the application is up-to-date and supported by new trends, testing it in a timely manner becomes important.
Apps are regularly updated with new features in order to keep up with market demand. In order to introduce upgrades into the market quickly, automated testing is crucial.
We would love to hear more about your project.
Even a short phone call can help us explain how our solutions can accelerate your mobility, jump start your continuous delivery and help reduce costs.
And that’s just for starters, understanding more about your project will enable us to build a solution that fits your objectives, infrastructure and aspirations!
“It is a capital mistake to theorize before one has data.” – Arthur Conan Doyle, author of Sherlock Holmes...