Emulators – How Good?

IDC predicts that the volume of smartphones is estimated to reach over 500 million in shipments by 2014. These smartphones will run millions of applications embedded in them. A lot of these applications will communicate with complex ERP and CRM systems which will require rigorous security testing and verification. And, all these applications will be used on many OS’s which then will need a good look at their functionality and integration capabilities.  Unlike the PC based environment, the mobile world comprises of a zillion devices with diverse hardware and software configurations presenting a unique challenge in QA and designing testing strategies.

Testing is and will be quintessential for mobile applications and testing spend will exponentially grow along with the volume of smartphones — almost to about 20billion by 2015 per IDC. The first question that then comes to mind — do we need to test an application on all these devices or can we use emulators/simulators to simulate the real device experience. Diversity in this environment with multiple device types, browsers, hardware configurations and network related challenges might make actual device testing a little difficult, but some key attributes of applications have to be tested on real devices only. Emulators can only be used to test some device independent features.

Which route should you go? Well, it depends — on the type of application, network and firewall configurations, on the QA strategy, current and immediate needs, cost and resource availability. But let’s take a look at the comparison matrix to analyze the emulator vs. the real device experience to arrive at a more definitive conclusion


Real devices

The application can be installed through apk file backup which is a time consuming process and it requires a real device The application can be directly installed in the device.
Real time applications (i.e. GPS based, server based, motion, multitouch or tilt control based) cannot be tested on emulators All types of application can be very accurately tested on a real device.
Emulator is slower than a real device. So, the app performance cannot be measured accurately and slower speed of emulators effects productivity A real device is always more accurate than an emulator
Communication with the emulated device may be blocked by a firewall program running on computer because the emulator can act like a normal application on workstation. There are no firewall restrictions on a real devices
There is no support for USB connections, SD card insert/eject, battery charge level, device-attached headphones etc. Real devices supports all of this without any restriction

To firmly establish this premise, we conducted an experiment by testing top 75 apps in the market on emulators as well as real devices. Here’s what we found, emulators are suitable for initial stages or only to test some device independent features. But to ensure the release of a zero defect application, real devices are important and a very critical part of the testing cycle.


Other stories you may enjoy...

The Secrets to Five Star App Reviews

Let's face it. Every app developer wants five star reviews. But the process of generating five star reviews needs to be a little more sophisticated than releasing it and crossing...

Is the ALM Reboot Resulting in the Downfall of QA?

Everyone will become developers during sprint — and quality will be everyone’s responsibility Lately we have all been hearing terms like Agile lifecycle, Continuous...

Everything You Need to Know About Shift Left Testing

It’s becoming increasingly important for organizations to “Shift Left.” Shift Left refers to the practice of starting testing earlier in the software development lifecycle,...