A Mobile Testing Strategy That Works

Statista reported that in 2015 there were over 226 billion app downloads across the world. By 2017 this number is expected to cross 270 billion. With the number of smartphone users increasing every day, the world is getting increasingly ‘appified’ as an increasing number of people begin to prefer mobile apps over mobile websites. An infographic released by Go-Globe shows that on average, smartphone users spent 89% of their mobile time on mobile apps. Mobile apps have also evolved considerably. With great user interfaces and features and enhanced functionality, mobile apps today demand technological innovation to deliver hyper-personalized experiences. In many ways, the success of an app has a little less to do with how ‘cool’ it looks and more on how it performs. No one likes to interact with slow, buggy apps. In fact, according to a report from Swrve’s App Metrics Report 35% of mobile app engagements last less than a minute. It does not take long for people to give their judgement so it becomes all the more essential to make sure that the app performs as it should. Mobile apps can fail for many reasons but improper testing is one of the key reasons by apps fail. Having said this, mobile app testing is quite different from web app testing. Considering the device diversity, a variety of network connectivity options, short release cycles, etc. mobile app testing demands a well-thought-out and structured strategy. As compared to the web app or desktop app testing, mobile app testing is often more intensive, complex and time-consuming. While desktop applications need to be tested on a single dominant platform such as Windows, mobile apps need to be tested on iOS, Android, and maybe even Windows and other such platforms. Apart from this, the device diversity adds a layer of complexity to mobile app testing since there is no standardized reference hardware. In mobile apps, there are well over 60 devices, screen sizes, form factors and resolutions to choose from for Android devices alone. Add to this the fact that mobile apps need a unique environment to operate in as changed in network behaviour can affect application performance. Mobile apps also demand frequent upgrades and releases to make sure that the app is performing optimally at all times. It is because of all these reasons that mobile app testing is more intensive and time consuming since testers have to make sure that the mobile app performs perfectly and flawlessly in real world conditions. The complexities of mobile app testing do not end here as along with all this we need to understand the various aspects of the mobile app that need to be tested considering there is a different kind of mobile apps. You have native apps, mobile web apps and hybrid apps. Hence it is essential that before taking the mobile app testing deep dive, we first determine a testing strategy to ensure that all technical aspects of app testing are covered. A mobile app testing strategy should cover the following:

Selection of target devices :

Selection of devices is perhaps one of the greatest challenges for mobile app testing. Considering there are thousands of devices that could access your application, it becomes essential to consider almost all the options. However, along with the volume of devices, testers also have to consider the operating systems and their different versions and their permutations which can get quite out-of-hand. Analysing the market and choosing the devices that are most widely used and identifying the devices that the target audience uses is the first part of starting the testing process. Testers can use real devices or emulators for this process. While real devices have the advantage of displaying the limitations of application in the actual environment, it can be an expensive proposition and even access can be an issue as device release cycles are very fast. Mobile emulators, on the other hand, can be used effectively for device testing. You can load the device profile on the emulator and test your native/mobile web/hybrid application in the same way as you would on a real device. Emulators are capable of handling the testing needs for mobile apps as they are designed with testing in mind and are efficient in capturing app diagnostics. The three kinds of mobile emulators used in mobile app testing are:

1. Device emulators
2. Browser emulators
3. OS emulators

Testing on mobile emulators in the initial stages of development is cost effective and efficient. However, since emulators lack the faults and characteristics of the actual devices, in the later stages mobile apps needs to be tested on a real device. An emulated device is also not sensitive to ambient conditions which impact the device behaviour and eventually app performance. So for effective mobile app testing, testers need to start testing in an emulated environment and then move to an actual device for real-life testing scenarios.

Cloud-based testing :

Cloud-based mobile testing is emerging as a viable option for mobile app testing as it provides a scalable, asset light and on-demand pay-per-use models. You can test, update and maintain complex mobile applications on the cloud in a convenient manner. A cloud environment gives testers to utilize the cloud infrastructure to test the apps virtually in a real-world environment and provide infinite scenario combinations easier to manage. The tests in the cloud environment are usually pre-standardized and have pre-configured architecture and give testing teams the freedom from spending time and resources to configure test environments.

Network Connectivity :

Since mobile applications rely heavily on network connectivity for functionality, testing mobile apps in actual network environments becomes essential. This can be accomplished using network simulation tools and assess how the application performs in different network speeds, connection dropouts and bandwidth limitations etc. In order to be a winning application, it needs to perform consistently across different network environments.

Mobile v/s Automated :

Which should you choose – manual or automated testing? Mobile app test automation requires investments in test tools and scripting which can seem expensive. However, given the fact that these automated tests need to be run multiple times especially during development and regression, the ROI for the same can be realized quite fast. However, instead of automating everything, testers need to judiciously choose test automation cases to achieve the maximum benefits of automation. Selecting a great test automation tool also helps in the testing process. Keeping the following criteria’s in mind when selecting a test automation tool helps in making the selection process easier:

• Multi-platform support
• Script reusability
• Jailbreak requirement
• Source Code changes
• Lead Time for New OS version / Device
• Test workflows
• Results and reports archives
• Seamless integration capabilities
• Deployment options
• Services and support
• Cost effectiveness

The points discussed in this blog cover the basic starting point of mobile app testing. In our consequent blogs we will cover more details about testing mobile apps and discuss the dimensions and other aspects of testing them.