Continuous Integration for Mobile QA and Testing

Continuous integration for mobile QA and testing

Beth Rominic: Hello, and welcome to today’s web seminar, Continuous Integration for Mobile App QA and Testing, presented by Apexon. I’m your host and moderator, Beth Rominic. Thanks for joining us today. Before I hand it off to the speaker, let me explain the controls for this console. The console opens with three panels on your screen. All of these can be moved around and resized to your liking. At the bottom of your console is a widget toolbar. Hover your cursor over the icons to see labels for each widget. Through the additional resources widget, you can download a PDF of the speaker’s slides and share this presentation.

If you experience any technical issues, please send a note through the question field beneath the panels, and we’ll be on hand to help. Our responses can be read in the Q&A panel to the left of the slides. This is also how you’ll submit questions. Feel free to ask questions throughout the presentation, and we’ll answer as many as possible during the Q&A session at the end. Finally, to optimize your bandwidth for the best viewing experience possible, please close any unnecessary applications you have running in the background. This event is being recorded and will be made available for on-demand viewing.

Once the recording is ready, you’ll receive an email with instructions about how to access the presentation and slides on demand. And with that, I’d like to hand it over to our speaker, Harshal Vora.

Harshal Vora: Thank you, Beth. Good afternoon and good morning to everyone. Thank you again for joining us. Today we are going to talk about continuous integration for mobile app testing, which is a very challenging and upcoming challenge in the world of mobile testing. So about Apexon, who we are and what we do. We are based in Santa Clara; we’re quartered in Santa Clara. We have about 700 people. We’ve been in the business for about ten years. We develop mobile testing and mobile application development, including dev ops, since dev ops is going to be an integral part of the application type of management now. So in today’s webinar what we are going to talk about is, what is CI.

What are different tool sets from the mobile perspective are very important to achieve CI. Why CI and its value to business. CI implementation and architecture for mobile applications. And then we’ll talk about mobile cloud solutions and CI, how we can marry the continuous integration with mobile cloud. And we’ll do a small case study, where we have already implemented continuous integration for QA at one of our private healthcare customers. So let’s start with talking about what is CI.

So CI is basically a software practice in which multiple code bases or isolated code bases are immediately tested and reported on as soon as they are added to the code base. So say if I have my developers working out of three or four different offices, they all have a continuous challenge, how I merge those code bases into one single code base so that I don’t have multiple code bases or multiple instances of the same code? And that’s where the whole concept of CI into the picture. And let’s go over the key principles of CI. And we’ll talk about each of the principles.

There are seven or eight principles which are very key to achieve CI or continuous integration. And I am going to use the word continuous integration and continuous deployment interchangeably with you because CI and CD go hand in hand together in terms of mobile applications. So let’s start with implementation of the source control system or our revision control system. This is a very important aspect of achieving CI because there are multiple code bases which need to be aggregated together into a single system. And to achieve this, there are different routes available. As you can see, it depends what kind of system can be used to achieve CI, like Git, Subversion, CVS, Stash.

We also. Now what are to do? It’s basically create different versions and revisions of that code so that we know at what instance of time what level of code was checked in, and revisions of those systems with the CI. Now moving forward, the second principle, which is basically beginning to build automation. So now CI is all about getting automation or achieving automation so that we get – all the many interventions can be reduced, and the big automation is. So once a developer checks in a code or Stash, it needs to for IOS or Android needs to, and if that process is not automated, then. So for example, big automation like Maven, Gradle, and then Spring IO, which has an open source too.

automation by configuring some shell scripting. And it does not require a lot of expertise. It’s just shell scripting, and you can use this straight out of the box. Now once this build is ready to go, you need to get into a CI build tool, like say Jenkins or Hudson, Circle CI, Bamboo, TeamCity, which are all integrated of all the different tools that are going to be used for the whole software development life cycle. As you can see, Jenkins is an open source tool. Circle CI, it is a very up and coming tool, which is being used in the industry for continuous integration software.

And then Bamboo is part of, again, Atlassian Suite to manage the jobs, to create the job that will my build automation, that will getting the code checking and get automation of the deployment. So now comes the automation of deployment, which is, it has been in the industry, has been a big pain point, and we’ll talk more about that as we go farther. But let’s see what kind of tools can be used to do a deployment of mobile applications automatically. In terms of Android applications, it was pretty easy. But since this has been a mobile application world, it has been a tricky problem to solve.

There have been automation deployment tools like TestFlight, HockeyApp, IBM Urban{code}, which are already available. And there are cloud-based tools which already have the integration for automation of the deployment process. To just name one, Perfecto and Sauce Labs, they already have integration of those APIs available, which will enable us to, from the CI software, deploy right into my mobile application, sorry, right onto my mobile devices, whether it’s a physical device or emulator or simulator, either or. But it’s very important to achieve CI, but whole automation from end to end once the code is checked in, deployment and the testing.

Another key principle is self-testing the builds. So once a build has been built and deployed, there has to be done some amount of the unit testing as well as some amount of code quality tests, which can be integrated using SonarQube or Fortify kind of tools, which will give us the code quality as well as the security perspective of my code, as well as the aspect end of tools, which will do my reports back to my continuous integration software, and this all is automated. Then the last and the most important part of the CI to achieve CI for mobile apps testing is testing in a clone to our production environment, automate the user acceptance test, and getting a very important reporting structure of dashboards.

All of this might be a tool. So to ensure that we are testing through a clone in the production environment, there are a lot of tools available, like which will ensure that the server is of a test environment that will build automated rather than a manual or process, where are already built. And I have to be reliant on IT teams to get those. In the CI, what we want to do is have an automated process with environments that build as soon as my CI job initiates to tell my test to be run, regression test or the functional type or a user acceptance type. And both tests are built on the fly.

Builds integrations and the requirements defined during my scripts. And enable us to create those virtualization servers for my test environment. Now another aspect of my testing in a clone to production is going to be a load testing or performance testing. What we have seen is it’s very important to use these performance testing tools and integrate them with the app so that at least we can get an idea of how a mobile application is going to perform, in terms of a very load. There may be 200 or 400 users, and tools like Soasta helps us to do that, where we can load up the servers, we can do that in the test environment, and then create a certain kind of load on my mobile application and see how does my new build behave for that.

Now we are going to run through a small. I’m sure you’re all able to see the question on the screen. The first question I would like everyone to get answered is what level of CI is already implemented for your organization? And feel free to check all that apply for your organization. The options here are version control. Great, so we already have an answer. Version control, build automation, automated deployment, unit functional test, regression test. As you can see, most of the people are already having version control, build automation. It’s still doing that. It’s just we are figuring the out.

As you can see now, so most of the people have a version control system and a build automation system. And as I pointed out, the pain point is automated deployment and the functional test and regression test. And as we progress further, we’ll talk more about automated deployment, the regression test, the functional test, how we use those to achieve CI. And the next full question I would like to get answered is what are the main pain points in your current implementation? Automated deployment, minimal automation, false positive results, longer execution times, or the reporting is not structured, is not well-defined.

If you all can please – we are starting to get some results for that. Let’s see. I see minimal automation is going to be the area. Most of the people say automation is very minimal. And for CI, automation is a key aspect of CI. To achieve a strong continuous integration system, we need to have at least 90 percent of automation from all aspects of that. So now let’s see the results. As you can see the results are 56 percent say minimal automation. And I do agree with that. We’ll talk more about it, how we can achieve maximum automation, how we can reduce the execution times, and how we can get accurate reporting out of my continuous integration tools.

The last question I have is what kind of tools are used to achieve CI for mobile at your organization? Do you use Appium and Sauce combination? Do you use Perfecto/Selenium combination? Or it’s some other which you are already using? And there are a lot of other tools available in the market that can help us to achieve CI on the mobile front. Appium/Sauce combination is a very strong combination. So is the Perfecto/Selenium combination. And we have even tested both of those combinations successfully. In terms of others, there are a lot of open source tools that can be used to achieve automation.

And then we need to make a choice, whether we want to go to open source tools or we want to go with cloud-based tools, and what are the end goals of my continuous integration system. So now let’s move on. Let’s talk about how the CI is beneficial to an organization. Why CI? We all know that mobile apps live and die by the rating. So say tomorrow my Bank of America application started getting some bad reviews, that’s going to impact a lot in terms of my user experience and all of a sudden I receive – all of my ratings are going to infect my mobile apps.

So to continuously improve and get my apps the market, to get certain has a great mobile application where you can pick and choose your ingredients on the fly. Whereas on the other hand, kind of thing, they don’t have that kind of great application. So they try to come up with some basic functionality, but my mobile apps will get those reviews faster and my users are going to ask for certain kinds of features. And to get my apps with those new features out to the market, continuous integration is going to help us to get there because you’ll be testing at every aspect, developing faster, getting the feedback faster to my engineering team, and getting my apps faster.

We are going to be, as I said. Right initially, it’s going to be a commando force behind multiple developers, developing at the same time, and if we don’t have a source code management system, then I’ve got. There will be multiple levels of my code, which will be integrated together and might not be product integration, and so CI is going to be very important for that. And development control aspect of CI helps us to roll back to my previous version if my new version has some issues with it. Then we are not deploying manually between different environments. The test environment, production environment, the manual deployment as we assign as well.

The deployment process is an issue for most of the organizations to achieve CI. And with CI, if we get the automation done with the deployment process, then that is going to be a great success for all the organization. Again, the test environment, as I said, most of the places it’s static, but CI helps us to create those test environments on the fly with my automation so that I don’t have to rely on a specific test environment or a specific resource to get my up and running.

Again, we are stuck in a hot mess of hot fixes, so a lot of them are waiting until too late to get my issues fixed. It’s very essential for an engineering team to know my issues up front and reduce my defect leakage into the market. Moving further, what’s the value of CI to the business? The key benefits are basically we identify the defects earlier, so early defect detection. Then we know the overall health of a software early enough by establishing continuous testing into automated integration process. We reduce assumptions because CI gives us a chance to replace those assumptions with knowledge, thereby eliminating all development stage.

We reduce overhead, finding the bug at development stage itself so that I don’t have to invest time and money at a later stage, if my bugs are found in the QA environment. It brings that quality assurance to the team. And this is the biggest benefit from a customer perspective, right? CI will enable you to release the global software at any point in time. In absence of CI, projects are going to be delayed and because of some deployment issues. It enables the project visibility, right? Because since the integration, it provides the opportunity to notice the trends and make informed decisions.

Noticing the trends in build success or failure, for example, say I am creating 500 builds, and out of those 500 builds, only 100 builds are qualified to go through my user acceptance testing, then that’s a great achievement, which CI will get to you. Because of your automation testing, your smoke testing, unit testing, will give feedback faster to the engineering team and let them – and bring them on the floor early enough so that they can fix those issues. And get that project visibility into the CI software. Let’s see what a CI/CD process would look like. As you can see, there are six different verticals according to CI/CD process. To achieve this CI continuous deployment process, a delivery team is very important for any aspect.

Then there is version control system, build and unit tests. Then we have automated acceptance tests, as well as user acceptance tests and releases. Now as you can see, once the delivery team stops checking the code in a version control system like, my CI/CD check and do the follow-up and based on those check-ins, I create a job to my build automation, so I create those build automations and to create a build and feedback right away. But yeah, it’s not passing this initial criteria of my build and unit test. So I basically save time by – save time on QA that they are not testing a non-qualified build, an unqualified build, and they get more time to basically automate for their feature engagement, and it gives more time to the delivery team to fix that issue early enough and get their code check-ins faster.

Usually we have seen is the code check-ins are every four hours or maybe at every code change they do code check-in, so it gives a confidence to the engineering team as well that any change that they make early enough are good to go and make a build and pass the unit test. And that’s a very important aspect of build and unit tests that you see here. Once my build and unit test are passed, I have a smoke or acceptance test that needs to be, and which is automated from the CI perspective. And the QA team, the software quality team, is going to own that aspect of automation, and like certain smoke test case scenarios that will be taken care of.

If my certain smoke tests are going to fail, then it will again get back to my engineering team and give them that report that these are some issues that are not working properly, or this cannot go to the production environment because of this issue. So again, as you can see, this is really early on in the game, that I get my issues identified, at least early detection. And it gives a better possibility. That you get acceptance on the regression test, can also be part of – before it goes to the go-live production.

Now how do I get to the CI/CD process, right? What are the different stages to get to the CI/CD process? So the first that we have taken here at Apexon, to have a jumpstart program where we then take two to three months. We get to the CI/CD process and help an organization review the execution times, guide the applications faster to the market. So we have a – basically this jumpstart program is developed in three phases. The initial phase is about reviewing what are the requirements, understanding the expectations from different stakeholders like, the engineering team, QA team, really the automation team, and understanding the different strategies.

What is the strategy for qualifying the build, production and maintaining the release cycle? Understanding the different dependencies. So the whole review cycle is to strategize and come up with some kind of policies or some kind of feedback that can be implemented here in the definition and the planning stage. So the definition and the planning stage, usually it spans from four to six weeks. And here we define what kind of different CI tool sets earlier that are built in the organizational structure, how we can leverage those existing CI tools. Do we want to go with cloud-based devices? Do we want to go with real devices?

Do we want to go with? Those kind of definitions we are defining here. How often do my regression tests run? How often does my user acceptance test run? Do I run? Those kind of things are defined. The whole CI flow is designed during this definition stage. We define the automation framework, say do we want to go with Appium? Do we want to go with Selenium? Do we want to go with Robotium? Those kind of automation frameworks are defined. And then during the deployment phase we start executing based on those frameworks. We start creating certain smoke tests that can be included as part of this whole CI process.

At the end of the three-month period, we get a good CI process implemented, which can be taken to my production environment. Now how would a CI/CD architecture using the mobile cloud look like? So system like developers and that will be – that will be initiating a job into my CI software like Jenkins, which will create some very, for me, push code from my Git or Stash, and we on the push. So for this initial phase, for unit testing, we can just test on emulators or IOS and/or simulators, right? And once those unit tests or my build tests start, I can take it to my Android acceptance test and use my cloud-based tools like say Perfecto or SYNCHRO or Sauce Labs or Mobile Labs, and test on real developers in real environments.

And I get my results back to my CI server and then build reporting, which will give me an exact idea of how my tests have done. How is my new build? Does it have all the special features? Do my new features work fine? Everything will be taken care of in this dashboard and reporting structure. We define the reporting structure based on the different tests and requirements defined by. So how we can use this CI tool and mobile cloud, why we should go with mobile cloud integration? What we have here is these are some of the benefits of harnessing CI and cloud. Mobile-ready plugins, most of these are cloud-based tools like Perfecto and Sauce.

There already have plugins for Jenkins kind of tools. It’s just easy for me to plug and play this tool into my existing infrastructure. It gives us unlimited scalability and elasticity to test on various devices, on IOS as well as Android. So I don’t have to limit things to real devices. I can use this cloud-based tool which are hosting the devices for me, and I deploy my applications using my CI jobs on these cloud-based tools. So I get a whole lot of idea what’s going on on my application from multiple devices, rather than just on one or two devices. It helps us to leverage existing infrastructure and I have already an established Atlassian Suite that I’m using planning.

But I choose Bamboo and Stash, which will help me in integrate the whole CI with the mobile cloud, like Perfecto. And then I have a strong integration with. So I don’t have to reinvest into getting new tools for my CI system, because the mobile cloud already has integration with existing CI-based tools. I enhance my mobile workflow, or CI workflow using the real user conditions. Like, I want to plan my application to test on say AT&T network. What happens on AT&T network? What happens on T-Mobile network? What kind of real user conditions can be created, as well as some non-functional testing can be created. But what happens on the application when a phone call comes in?

Those kind of things can be done using the cloud as devices. Another good factor is on-demand availability, that I can test around the clock. Once my eight-hour day is over, I can the job to run on the cloud as devices. And we don’t have to worry about those devices are on or off. We schedule those things and they will be available on demand. And faster issue resolution because these cloud-based tools have a strong reporting structure with screen shots, video capture, as well as detailed console logs. So what are the different types which are important to select those cloud-based tools? As I said, on-demand cloud, automation support and integration with the support systems, which is basically the existing system.

So as you can see, the automation support is a very important aspect. Perfecto supports Selenium automation support, as well as they are coming out with Appium support. support, Appium, Selenium, and this gives me the flexibility of writing my automation framework only for one platform, and that can be multiple platforms, for IOS as well as Android. And to achieve CI automation is a very important aspect. So let’s go to the case study that we did for a leading healthcare provider in the Bay Area. What are the basic challenges that they were facing? As you saw as well, slow automation execution time. It was about 22 minutes per test case. Sequential execution on the devices rather than the parallel execution.

So longer execution times again. Lots of test data dependencies. And the scripts automated were already utilizing regression cycles. And no framework. There was no framework established for IOS or Android. I didn’t have a single framework which would enable me to forward my scripts to both the platforms. What we did is we recommended best practices for automating native as well as web app using Perfecto cloud. We collaborated with teams. We integrated the whole Perfecto cloud automation with like HP ALM. And the benefits were, as you can see, we optimized automation execution time to 8 to 10 minutes, which is we reduced it by 50 percent.

So faster execution time. Parallel executions, rather than sequential execution. We utilized the automation script in different release cycles. So we two, three times per week average each cycle. We didn’t do, we did about 400 test cases, which is a big number if you look at it. And it gives a confidence of engineering team as a product management team to me, that yes, my product looks valid, even if I have some issues, that those issues can be resolved or figured out. And this was the whole architecture that was used for, as you can see, the whole CI system.

Jenkins is my CI server, which is controlling different aspects of my tool chain, which is quality centered, which is management, automation management, and Selenium tested, creating VMs, virtual machines for my test environment, creating – or getting access to my on-demand devices from the Perfecto cloud. And all this is automated using shell scripts, as well as already-established scripting tools. Thank you for your time, and now I will come for Q&A. So please go ahead and type in your questions in the Q&A module available.

Beth Rominic: Okay, thanks very much. We’ll get through as many questions as we can right now. The first question from the audience is are CI and dev ops synonymous? How is CI different from an agile sprint where dev and QA co-exist?

Harshal Vora: Sure. Thanks. CI and dev ops. Dev ops is. CI is also, right? And CI is part of dev ops. In the mobile world, I would say it’s an interchangeable word, and CI is part of dev ops. development and operation, CI is very essential and automation is going to take us through the dev op process, as well as the CI processes.

Beth Rominic: Okay, thanks. Next question, how much effort was required to integrate Jenkins with Quality Center?

Harshal Vora: That’s a good question. Since we are already running with Jenkins, as well as Quality Center for a long period of time, it was a very easy process for us to integrate Jenkins and Quality Center, I would say. Within our definition phase, initial phase itself, we were able to integrate Jenkins and Quality Center based on the requirements within one to two weeks.

Beth Rominic: Okay. What is the right level of test automation for CI?

Harshal Vora: That’s the burning question that everyone wants to answer and wants to know. CI, what you see are there are different levels of automation. Automation is required at every aspect of CI, from build automation to deploying the build, to testing, smoke testing. And what we recommend usually is to at least do the build automation, deploying the build automatically, and getting to a certain degree of smoke automation, smoke test automation. And if you have a level, then the regression test automation is very by the test automation.

Beth Rominic: Great, thank you. Next question, what are the main challenges for implementing CI for mobile apps?

Harshal Vora: As you see, as you saw in my as well, the main challenges are minimal automation. And because a lack of automation framework, deployment challenges, test environment challenges, and the reporting challenges. Since there are multiple tools that are involved in achieving CI for mobile devices, it’s very hard to integrate all those tool sets, or the whole tool chain together for mobile devices. And I see those three aspects of deployment, the automation and lack of automation framework, the main challenges to achieving CI for mobile apps.

Beth Rominic: All right, thanks for that. All right, this person says, in my company, we have static test environments. Is it advisable for us to build test environments as part of CI?

Harshal Vora: Yes, that is very important, and we, right? It is very important that, instead of – you can use those static environments, but if you use the dynamic environment beneficial to the whole organization. Because this environments or test environments, will be built on the fly, based on the need, and will do my activity as well for my CI, as well as for my mobile app operation.

Beth Rominic: Okay, thanks. This person wants to know how many devices or simulators do you recommend for the different test phases, such as build acceptance testing, unit testing, smoke, functional, regression testing, etc.?

Harshal Vora: Sure, sure. And that’s a good question. So build and unit tests, we usually recommend to test it on emulators or simulators for each. I take IOS one simulator or Android I take one simulator for a unit test and my build test. Once we move to the smoke test, I would like to make sure that my application is running successfully for at least multiple OEMs for Android. So I would test on real devices, on multiple OEMs and multiple. So for a smoke test, I would say at least three Android devices and two IOS devices.

And going further, for the functional test and the regression test, so the regression test is a very important part. That can be run on – so right now, we are. We have implemented the regression suite for 20 Android devices and 10 IOS devices.

Beth Rominic: Okay.

Harshal Vora: I hope that answers the question.

Beth Rominic: Okay, thanks. Next question, per your experience is QTP a good option for CI for mobile apps?

Harshal Vora: It’s a yes and a no. So now and UFC is strongly supported. So if you already have a tool chain established for quality center and along with say Perfecto mobile apps kind of qual solutions, it’s a good option. And if you don’t have any other framework already established, then it can give us a very good exposure, a good automation framework, which can be leveraged for both the platforms.

Beth Rominic: Okay, thanks. This person says, we use Appium plus Jenkins or Travis to executive mobile automation. Do you have any recommendations for creating valuable reporting from Appium test executions?

Harshal Vora: Yes. So that’s a very good question. So that’s for reporting from the Appium test execution, right? It is dependent on what – so what kind of reports I would like to see from my test execution is the module execution, as well as all the tests and device-based execution. So say if I’m running my application on multiple devices, using my Appium framework, and I would like to see how my application fared on IOS devices, how did it fare on Android devices? And what modules or what functionality within the application were leveraged? So basically I structure my reporting based on my modules and devices.

Beth Rominic: Okay. We’ve got another question about Quick Test Professional. This person asks, is QTP better to use than TFS or Team Foundation Server for CI?

Harshal Vora: Well, that’s again, a very specific tool requirement. So QTP as well as, they both are pretty good and have very good integration with existing tool sets available. So it’s very specific to the requirement and application, what tools to go with, TFS or QTP.

Beth Rominic: Okay, thanks. Well it looks like those are all the questions then from the audience. So that will end our event for today. I’d like to thank our speaker, Harshal Vora for his time, and I’d like to thank Apexon for sponsoring this web seminar. And a special thank you goes out to you, the audience, for spending this time with us. Have a great day, and we hope to see you at a future event.

Harshal Vora: All right. Thank you everyone.