How To Do Mobile Apps Data Extraction On Scale?

By Author: Mobile Apps Data Extraction Services
Total Articles: 24
Comment this article

Reverse Engineering

Now, the question is how can we do that? Assume you need to scrape data from the mobile application, let’s assume that we have got APK of an Android app and we wish to scrape 500,000 data points (UI screens) every day, how can you do that and what will be the cost?

It’s important to find how a client communicates with the servers, which protocol he is using, how they are transferring messages to each other.

As this might look like the finest scalable and affordable solution, it may only provide solutions to one application, so to do if we want to repeat a process again and again with other applications? What if an API gets changed? So, you can see that it’s hard to guess the efforts it should make.

After that, we have used Android Emulator, made installation of the APK, connected that to the proxy and observed the data.

The entire communication was done using HTTPS, after a few hours, we could able to monitor traffic from the clients to the server as well as even capable to simulate calls to a server.

Outcome

Reverse engineering is very easy to start ...
... and looks the most affordable and most scalable way of doing it. However, it may take some long days as well as the development costs are random and you don’t always get the end results.

Selendroid or Appium

With tools like Selendroid or Appium, the scenario is completely different. You can easily write the scenario that you need to test and automatically run that test script again and again. We have decided to use Appium with Android Emulator.

Android emulators are known as impossible tools to deal with for mobile development, though with the release of x86 emulators, things have started to work easily and it feels like working the applications inside laptops run quicker than the physical devices themselves.

Later, we created a Docker container using Ubuntu 16.04, Appium and Android x86 emulator to start the test about how many of those we could run at the same time.

So, assuming that we can utilize 1 CPU for 1 emulator, we will require 700 CPU’s for 700 emulators! It is a huge requirement and very expensive too!

Outcome

Physical hardware always brings good performance however, it’s very hard to deal with on a large scale.

So what to do to avoid the physical hardware management?

Well. We can use the public cloud like AWS. However, when we took this approach to the cloud, things worked completely different. Actually, Linux, Docker, AWS, and Android have worked really well together, however, with an emulator, they’re not. AWS EC2 gives you a Virtual Machine and Android Emulator is a Virtual Machine on top of that. To take benefits from the hardware acceleration while using x86 Android emulator, the host machine needs to reveal this competence, however Amazon, as well as any public clouds, don’t expose this, rather they utilize it for themselves for serving us with the virtual machines, therefore, we were unable to even start an Android x86 emulator!

So How We Have Done That? Well, We have used Ravello.

Ravello Cloud
The Ravello solution has provided nested virtualization or Kernel-based Virtual Machine support on host machine when running on a public cloud.

It has given us the capability to run the x86 Android emulators on the cloud. We have tried it and it worked also, however, in terms of performance, the scrapping has taken 3 times more time compared to physical machines and with the use of more emulators, things have got worse.