Aligning MTA with DevOps

 




With number of mobile users reaching up to a mark of 4.92 billion, it is important for the enterprises to select a right mobile test automation strategy, which can seamlessly integrate with the DevOps, to help achieve maximum test coverage earlier in the cycle.

Executive Summary:


In today’s world, smartphone is a growth enabler, positively influencing various spheres of life of an individual and therefore impacting the evolving knowledge economy. While the shift from feature phones to smartphones is occurring in most markets worldwide, the rate at which it is happening differs from region to region. Recent analytics show that more than half of the world’s web traffic now comes from mobile phones. In such a context, smartphone’s inability of managing the growing needs of a consumer can be a major impediment to smartphone adoption.


In their annual report “Digital in 2017: Global Overview” WeareSocialᴬ have mentioned that the number of global mobile users in 2017 has reached up to a mark of 4.92 billion. In addition to which recent statistics published by eMarketerᴮ shows that Apple’s App Store is home to over 2 million mobile apps, and the number of apps available on the Google Play Store is even higher—at 3.6 million. Mobile applications are one of the fastest and most continuously updated software for end users; hence they need constant testing and deployment. Hence, in order to test the applications continuously mobile test automation is important, because of which it has is broadened and bolstered by practices such as risk-based test case design and test-driven service virtualization, test data management and seamless integration into the DevOps toolchain. With highly fragmented market on the basis of – OEMs, OS, OS versions, network bandwidths, dimensions, etc. it has increasingly become important to test each application for their quality for long term success Moreover the applications need to be tested, not just from functional point of view, but also from non-functional aspects: user experience, UI, usability, security and performance


This whitepaper will help you select the correct test automation strategy along with industry best practices w.r.t. mobile test automation. With a well-symphonized mobile test automation strategy the development team as well as the testing team can reduce the overall time to market for the application, while at the same time providing a seamless experience to its application’s users.

Expectations from Mobile Test Automation (MTA):

Mobile Test Automation (MTA) is a very holistic term which covers all aspects of mobile testing (excluding manual testing and test case authoring). Which are further supported by multiple factors and expectations that drive an organization’s decision to automate the testing process of their mobile application or responsive website (on mobile interface).

1


Below are the major influencing factors:

  • DevOps Ready: The framework implemented should be readily integrated with various CI-CD tools like JIRA, Jenkins, etc. so that the test automation activity is automatically inclined with development cycles.
  • End-to-end QA Automation: After this activity all QA related activities –functional & non-functional, are equally addressed.
  • Increase Business Participation: With new ready to use tools/frameworks coming into picture, it has become possible to involve SMEs/BAs in testing activity.
  • Parallel Executions: Parallel executions on multiple platforms/devices should be possible with the implementation of test automation.
  • Maximum Device/Platform coverage: With test automation in place the testers should be able to test on all target devices/platforms.
  • Reusability: The scripts created during this activity should be durable and stable enough to be executed in different environment with same/different data sets

Key Mobile Test Automation Challenges:

  • Selecting the right set of devices:Massive market of mobile phones with multiple OEMs, dimensions, features, OS, etc. making it more and more difficult while creating the correct device matrix for testing
  • Network challenges:The accessibility and stability of an application across multiple networks (2G, 3G, 4G, Wi-fi) and then different bandwidths by different operators
  • Limited Hardware:Even after creating the right Device Matrix it is impossible to possess all the physical devices required for test coverage
  • Technology behind application development:Each application (native / Hybrid / responsive web) is developed using different technologies. With constant vacillation in technology it is quite a challenge to cope up at the testing front.
  • Selecting the right tool for automation:For each testing requirement (functional/non-functional) there is a different tool in the market.

MTA and DevOps:


With DevOps approach integrated with MTA the apparent benefits for business are: full adoption into agile methodology, getting things done faster & with focused mentality on development, testing, delivery, support and services.
An important aspect of DevOps is Continuous Testing (CT) process. And to empower Continuous testing process, an end-to-end mobile test automation strategy is required to perform the tests at higher pace. This helps QA Teams to balance the speed of DevOps, enabling developers to deliver new software features quicker.

Types of Mobile Applications: 

  • Native Mobile Application: The application is built to be OS (iOS / Android / Blackberry / Windows) specific and won’t work if opened on any OS except their own. But an advantage of these applications is that they are faster and provide a better user experience.
  • Hybrid mobile application: These applications are basically the web applications built with a native app’s wrapper. Hybrid applications are built to be run any device with any OS 
  • Web Applications: The responsive version of any application which enables a user to access a website on desktop’s web browser or on mobile browsers. Web applications require minimum of device memory, as a rule.

What is Mobile Test Automation?


Mobile test automation is basically an approach in itself that helps an enterprise test more and more often if implemented correctly and at right time during the SDLC. Once automated the same scripts can be executed ‘n’ no. of times on ‘n’ no. of devices, in parallel or sequentially (on the basis of requirement), resulting in timely identification of bugs, hence reducing the overall time to-market.

How can we do it right?


After understanding the challenges, expectation and finally what Mobile Test Automation is, it is important that it be done correctly. For which it is crucial to follow the right approach and carefully design the test automation strategy, keeping in mind all the aspect related to mobile applications.

Step 1) Select the appropriate testing type/s:

  • Platform Compatibility Testing: Testing to be performed on multiple devices for an application’s compatibility across different OS, OEM, device dimensions, etc.
  • Network compatibility Testing: Application is tested for its compatibility, when executed on different networks (2G, 3G, 4G, Wi-Fi)
  • Bandwidth simulation Testing: Testing to be done to observe an application’s behaviour when there is a change in bandwidth within the same network
  • Functional Testing: Testing of an application from functional stability point of view
  • UI/UX Testing: Testing of an application from end-user’s stand point for
  • Localized Testing: Testing needs to be performed outside premise at actual locations from where the application will be accessed 
  • Interrupt Testing: Testing: There might be certain interruptions (like call, messages, notifications etc.) while accessing the application. So it is important that the application be tested for such scenarios
  • Performance Testing: Testing an application for response time, battery consumption, CPU Usage, etc. are important parameters for the users accessing the application


Step 2) Select the right set of tools for test automation:

  • Open Source Tools : There are plenty of good tool options available in open source tools, but the only challenge is leveraging additional programming skills to use these tools effectively
  • Commercial tools: Commercial tools are more reliable with better customer support. Also most of these tools are easy to use and DevOps ready


Step 3) Select suitable execution platform:

  • Cloud based device lab: 100s of real devices, are placed in a device centre and then given access to all its users over cloud. Users can book any of these devices (as per use). Such platforms are also helpful during parallel executions. Also all latest devices with updated OS of different OEMs are accessible here. Such platforms are very useful when testing needs to be done on diverse set of devices.
  • Physical device lab: For this the enterprise needs to possess all physical devices required for test coverage. With the current diversity in devices and new, frequent updates, it gets very expensive to procure all devices required. Hence, it does neither generate the desired RoI (Return on Investment), nor is it cost effective.
  • Simulator/Emulator: These provide a similar UI when compared to a real device. With its help testers can test their applications for different configurations (but still real device experience is missing) which leaves a lot of gaps (from end user’s standpoint).

 

Mobile test automation’s success: What to do & how to do…


Below are the 11 steps to follow in order to achieve best results from mobile test automation:

  1. Identify whether the application is a right candidate for test automation or not, before the team begins with this activity
  2. Identify multiple platforms and version support for the target audience
  3. Create a device matrix which would cover different OS, OEM, OS versions, and screen sizes for script execution
  4. Assess the size of the application to be automated and the main challenging areas
  5. All test cases (functional / non-functional) should be authored manually or written in any test management tool (should be well defined in a logical flow/business logic/testing type) for easy mapping with the scripts to be automated.
  6. Since mobile app testing is mostly Agile, hence prioritize the test cases to be automated
  7. Identify the tool to be used for test automation (that can be possibly integrated with other CI-CD tools for Agile)
  8. Look for data driven testing frameworks, as that would help execute same test cases with multiple data sets
  9. Perform multiple Proof of Concept (PoCs) activity to ensure that the identified tool is capable of automating the application to be tested and is also able to execute scripts on varied devices and browsers
  10. Ensure the tool can be integrated for execution on different execution platforms to optimize runs varied mobile platforms
  11. Define realistic milestones and calculate the RoI at each milestone


Footnotes
ᴬ “Digital in 2017 : Global Overview” WeareSocial
https://wearesocial.com/special-reports/digital-in-2017-global-overview

ᴮ ”How to improve your app’s visibility” eMarketer; May22,2018
https://www.emarketer.com/content/how-to-improve-your-app-s-visibility-organically

About the Author:
Nikita Sood is a Solution Architect in CresTech Software Systems’ Pre-Sales Team. Nikita has good experience in the information technology industry, with experience spanning test automation, mobility, cloud, solution designing and technology consulting. Nikita can be reached at nikita.sood@crestechglobal.com

About CresTech:
CresTech is a market leader in providing Software Quality Management Solutions & Services. CresTech’s solutions & services have helped organizations meet their project time lines, budget and quality goals. With a commitment to offer the best and experience in delivering quality solutions & services across industries, the company has 250+ test specialists with global delivery centres across Noida, Bengaluru, Gurugram and USA

 Download: Aligning-MTA-with-DevOps.pdf (Pdf , 1,022.90 kB )