Guides/Monitoring network traffic from an Android app

From Rhythm Game Wiki
Jump to navigation Jump to search

To monitor and view network traffic in an Android application, you can use a proxy server. This guide will help set up your device and view network traffic from a specific Android app.

Prerequisites

To follow this guide, you have to have:

  • an Android device (you do not need root access),
  • a computer (with any operating system),
  • a USB cable to connect your mobile device to the computer.
  • access to a Wi-Fi network that you can connect to with the computer and the mobile device.

Install a proxy server

In this guide, we will use the Charles proxy server. There may be other, better alternatives that I have not tried using. Feel free to edit this guide if you feel like you have a better proxy server.

Note: The free trial version of Charles only allows for 30-minute sessions. After that time passes, you can reopen Charles to continue using it.

  1. Go to https://www.charlesproxy.com/download/ on your computer, and download the version of Charles for your operating system.
  2. Launch Charles.
  3. At the top bar, select the option Help > SSL Proxying > Install Charles Root Certificate on a Mobile Device or Remote Browser.

    Screenshot of Charles Proxy, selecting the option Help > SSL Proxying > Install Charles Root Certificate on a Mobile Device or Remote Browser.

    This will show a message box with your computer's local IP address and a port. In the next step, we will use this information to connect the mobile device to the proxy server.

    Pop-up window, showing the local IP address and port of the proxy server. In this case: 192.168.3.57:8888.
    In this example, the IP shown on the pop-up is: 192.168.3.57, and the port is: 8888.

Set up proxy settings on the mobile device

A screenshot showing proxy settings on Android.
In this example, we set the Host Name to 192.168.3.57, and the port to 8888 - the same values as shown in the Charles pop-up earlier.

To make network traffic go from the device to your proxy server, you have to set proxy settings on your mobile device.

  1. On your mobile device, go to Settings > Wi-Fi and select your Wi-Fi network.
  2. Under Advanced settings, change Proxy to Manual.
  3. Set the Host Name and Port settings to the IP address and port displayed in the Charles pop-up window in step 1.
  4. Save the settings.
  5. If everything was done correctly, a pop-up should appear in Charles, informing of a new connection. Click on "Allow" to allow the proxy connection. If you click "Deny" on accident, you can simply restart Charles and try to connect to the proxy again.

    A pop-up informing the user of a new connection to the proxy.
    This pop-up window should appear on your computer after saving proxy settings on the mobile device.

  6. After clicking allow, you can see some network traffic going through. You can try opening apps on your phone, and new entries will appear.

    A screenshot of Charles, showing some network traffic as a structured tree-like view.
    Network traffic from your mobile device, shown on your computer.

    You can also enable Sequence view by clicking the Sequence tab on the top. This will show all of the network traffic in chronological order, with the most recent requests on the bottom.

A screenshot of Charles, showing some network traffic as a list of requests.
Sequence view might be more helpful in some cases.

Now we have set-up our proxy server. However, even though we can see some host names and parts of requests, we cannot actually see the contents of most of the requests. This is because all modern applications use the HTTPS protocol to communicate; this protocol encrypts the data to prevent man-in-the-middle attacks.

This current setup will still work for HTTP requests. If you try to access a website via HTTP on the mobile device, you will be able to see the contents of the request, as well as the response in Charles. This is because, unlike HTTPS, HTTP is not encrypted, and the proxy server can read requests and responses sent via HTTP. You can try this yourself by opening an HTTP website (such as http://example.com/) in a web browser on your mobile device. You should see something similar to this:

(Note: You may need to scroll down to see the newest requests in Sequence view.)

A screenshot showing response data in Charles.
Response to a GET request to http://example.com/ shown in Charles.

In the next steps, we will install a custom certificate and modify an application, which will allow us to intercept HTTPS requests as well.

Install a certificate to the mobile device

To look inside HTTPS requests, we need to install a certificate to our mobile device.

After navigating to chls.pro/ssl, click on the "Download" button to download the certificate.
Screenshot of a safety warning that appears when installing a custom certificate on Android.
A screen warning you of the potential dangers of using a certificate that you don't trust.
  1. On your mobile device, open your browser and type chls.pro/ssl into the URL bar, while connected to the proxy. This will allow you to download the certificate for your instance of Charles. Tap on the "Download" button to confirm that you want to download the certificate file.
    This URL is also shown in the Help pop-up window in Charles - it's in the same pop-up that we used to get the IP and port for the proxy settings.
  2. Go to Settings and search for options about certificates. On my device, it is under Settings > Password & security > System security > Credential storage. From here, you should be able to view, add, and remove certificates installed on the system.
    Screenshot of the "Credential storage" menu in Android Settings
    Here, you can view, install and remove certificates from your device.

  3. Select Install certificates from storage, and then CA certificate.
    Screenshot of the "Install a certificate" menu in Android Settings
    Choose the "CA certificate" option on this screen.

  4. A pop-up may show up, warning you of the safety concerns related to adding custom certificates. Read it, and if you are sure you want to do this, tap on Install anyway.
    Remember! After you are done exploring network traffic from an app, you can remove the certificate by going into this settings menu, selecting Trusted certificates, switching to the User tab, selecting the certificate and tapping on Uninstall.
  5. You may be asked to verify your identity. After confirming your identity, you will be asked to choose a file. Select the file you downloaded before - it should be named downloadfile.crt and it should be located in your Downloads folder.
    Screenshot of an Android file picker, with a downloadfile.crt file visible.
    Choose the downloaded certificate file (usually called downloadfile.crt)
  6. After selecting the file, the certificate should be installed. You can verify that, by going into the Trusted certificates menu, and selecting the User tab - your certificate should be there. You can tap on it to view detailed information about the certificate or uninstall it if you don't need it anymore.
Screenshot of the "Trusted certificates" menu in the Android settings, showing a Charles proxy certificate.
The certificate was installed successfully.

The certificate should now be installed properly on the mobile device, however we still will not be able to see HTTPS traffic in Charles.

Enable SSL Proxying in Charles

To enable SSL proxying in Charles, do the following steps:

  1. From the menu bar at the top, select Proxy > SSL Proxying Settings....
    Screenshot of Charles Proxy, selecting the option Proxy > SSL Proxying Settings...
  2. Make sure the Enable SSL Proxying checkbox is enabled.
  3. Below the Include table, click on Add.
  4. In the pop-up, for the Host field, enter * and leave the Port field empty. This will make it so all hosts' HTTPS requests will be captured and attempted to be read by Charles.
  5. Click on OK on the pop-up, and OK on the "SSL Proxying Settings" window.

If you now try to open an application on your mobile device, you will see that it will not be able to connect to the servers. This is because all traffic is now captured by Charles. Most apps do not accept user-installed certificates as valid, which means even though our certificate is installed, the applications are acting as if it is an unknown certificate, and therefore cannot connect to the server at all.

Charles now shows many errors - applications do not trust the user-installed certificate authority.

If you want to allow some hostnames to use SSL as normal, you can add them to the Exclude table in the "SSL Proxying Settings" window. This setting makes it so Charles just acts as a pass-through for that specific host, and doesn't attempt to read any data inside of the requests. Alternatively, you can add hosts to the Include table instead of the * wildcard. This will make it so only the hosts in the Include table are captured by Charles.

Get APK files of the application that we will modify

We will modify the application slightly so that it will accept user-installed certificates. In this guide, we will be using Rizline as an example.

Install tools

First we will need to download some tools:

If everything is installed correctly, you should be able to open a terminal and type adb, apktool, zipalign and apksigner commands without errors.

Get APK files from the device

To modify the application files, we first need to get them onto our computer.

  1. Enable USB Debugging on your mobile device.
    • To do that, go to Settings > Additional settings > Developer options > USB debugging (found in the Debugging section).
    • If you cannot find the "Developer options" button, you might need to enable Developer Mode by going into Settings > About device > Version and tapping on the Build number option seven times. You should get a message saying "You are now a developer!"
    • If you have any trouble with this, try following the official instructions from developer.android.com.
  2. Connect your Android device to your computer using a USB cable. Choose "Android Auto" on your device if prompted.
  3. On your mobile device, a pop-up should appear, asking for permission to allow USB debugging. Tap on allow; and if you plan on doing this often, you might want to tap the checkbox to always allow USB debugging from this computer. If the pop-up doesn't appear immediately, don't worry; it might appear after typing the command in the next step.
  4. On your computer, open up a terminal and type: 
    adb shell

    You should now have access to your mobile device on your computer via the terminal.

    A screenshot showing the command "adb shell" typed in a terminal.
    *hacking noises* we're in.
  5. Now we need to find the package ID of the app that we are interested in. One way of doing that is finding the ID in the list of installed packages. In adb, type the following command to show the list of packages installed on the device: 
    pm list packages
    A screenshot showing a few of the 443 package IDs.
    On my device, this command shows 443 packages. Here is a few of them.

    This command will only show the package IDs, so you cannot search by name. If you have a lot of applications installed, you might want to try piping the command to grep. grep is a command line utility that will filter down results by showing only the lines that have a specified string of text inside of them. For example the following command will show applications that contain the word "Pigeon" in their ID: 

    pm list packages | grep Pigeon
    A screenshot showing the filtered results with only two package IDs.
    This time, only applications made by Pigeon Games will be shown.



    Another way of getting the ID of an application, is just searching the app on the Google Play Store website. When you go to an application page on Google Play, the ID of the package will be embedded in the URL. For Rizline, the link is https://play.google.com/store/apps/details?id=com.PigeonGames.Rizline, and so the package ID is com.PigeonGames.Rizline.

  6. The next step is to find the path of the APK files on the mobile device. Note that it is common for one application to have multiple APK files. Usually, there is one base.apk file, that is common for all devices, and several split APK files, that are device specific. For example, if your mobile device has a larger screen, it may download a split apk with extra high-quality assets. To find the paths to the APK files, you can use the pm path <package_id> command. In our case, the command will be: 
    pm path com.PigeonGames.Rizline
    A screenshot showing two paths to APK files.
    Rizline has two APK files - a base.apk file and a split_config.arm64_v8a.apk file.

    Now we know that the paths to our two APK files are:

    • /data/app/~~tMHzIziIkD3B_oH7sbOphw==/com.PigeonGames.Rizline-P6NKIGPvbzqSbdVySK53MQ==/base.apk, and
    • /data/app/~~tMHzIziIkD3B_oH7sbOphw==/com.PigeonGames.Rizline-P6NKIGPvbzqSbdVySK53MQ==/split_config.arm64_v8a.apk. We can use that information to download the files from the device to our computer.

  7. Exit the adb shell: 
    exit
  8. Create a new directory for our APK files. To do that, you can either create it using a GUI file explorer, or by typing the mkdir command. In my case, the directory name is rizline_mod 
    mkdir rizline_mod
  9. Navigate to the new directory using the cd command. 
    cd rizline_mod
  10. Now we can use the adb pull command to copy the files to the computer. 
    adb pull "/data/app/~~tMHzIziIkD3B_oH7sbOphw==/com.PigeonGames.Rizline-P6NKIGPvbzqSbdVySK53MQ==/base.apk" "base.apk"
adb pull "/data/app/~~tMHzIziIkD3B_oH7sbOphw==/com.PigeonGames.Rizline-P6NKIGPvbzqSbdVySK53MQ==/split_config.arm64_v8a.apk" "split_config.arm64_v8a.apk"

    The adb pull command takes two arguments: the first one is the path on the mobile device, and the second one is the destination path, on your computer. In this case, we just name the resulting files base.apk and split_config.arm64_v8a.apk respectively.

Screenshot of a terminal with all commands used to get APK files from a mobile device.
In summary, here are all of the commands used to copy APK files from a mobile device to a computer

Decompile the APK files using apktool

A screenshot of an AndroidManifest.xml file, when extracted from base.zip -- this doesn't look like an .xml file at all!

To modify the files of an APK package, we will first need to extract individual files from the downloaded APK files.

APK files are essentially just ZIP files in disguise. If you change the .apk file extension to .zip, you can open the file in any archiving software, just like a normal ZIP file. However, some files inside the APK file (such as AndroidManifest.xml) are in a weird, binary format. To read all of the files properly, we will use Apktool.

A screenshot of an AndroidManifest.xml file, when extracted using apktool -- this looks much more familiar!
  1. Open a terminal in the directory with your copied APK files.
  2. Type the following commands to decompile the APK files using Apktool: 
    apktool d base.apk
apktool d split_config.arm64_v8a.apk

    This will create new directories - base and split_config.arm64_v8a with the contents of the APK files.

You can now explore the application files freely using a file manager.

Modify the application to allow user-installed certificates

  1. Firstly, we will add a new file to our application. This file will allow the app to trust user-added certificate authorities. Create the file at base/res/xml/network_security_config.xml with the following contents:
    base/res/xml/network_security_config.xml: 
    <network-security-config>
    <debug-overrides>
        <trust-anchors>
            <!-- Trust user added CAs while debuggable only -->
            <certificates src="user" />
        </trust-anchors>
    </debug-overrides>
</network-security-config>
  2. The next few changes we will make will occur in the AndroidManifest.xml file. Open base/AndroidManifest.xml with a text editor of your choice.
  3. Find the <application> tag. It is located within the main <manifest> tag. We will insert two new attributes into this tag:
    android:debuggable="true" android:networkSecurityConfig="@xml/network_security_config"

    This will make the app "debuggable", and it will load the settings from the network_security_config.xml file we created before.

    Screenshot highlighting where to add the two attributes.
    We added two new attributes: android:debuggable="true" and android:networkSecurityConfig="@xml/network_security_config" to the <application> tag.

  4. Find an attribute named package="..." in the top-most <manifest> tag, and replace the contents of this attribute with your own app ID. In this example, we will replace com.PigeonGames.Rizline with rgwiki.rizline.example_mod. This change will allow us to have both the original, unmodified version of Rizline as well as our modified version at the same time.
    Screenshot of the AndroidManifest.xml file with the package attribute highlighted.
    We will change the com.PigeonGames.Rizline ID, to our own, made-up one: rgwiki.rizline.example_mod.

  5. Find other instances of the original package ID and change them to our new package ID as well. In the case of Rizline, the com.PigeonGames.Rizline ID also appears later in the file, as part of another ID:
    Screenshot highlighting the package ID found in a different place in the file
    We need to replace this part of an ID with our new ID as well.

    This varies from application to application. In this case, the app cannot be installed unless this other ID is changed to a new one. Other apps may have more IDs that you have to replace; and in some cases, you don't have to replace every ID with a new one, only some.

  6. There is one more ID we have to change. Open the split_config.arm64_v8a/AndroidManifest.xml file. This file also has a package ID, and it should match with the package ID from the AndroidManifest.xml in the base directory.
    A screenshot showing a short AndroidManifest.xml file for a split APK
    This is the last ID we have to change in Rizline. In other apps, there may be more. Try to search the directory for the original ID and find them all!

  7. The last change we will make, is a change of the application name, to differentiate between the two applications on the device more easily. Open base/res/values/strings.xml in a text editor. Find an entry that has the attribute: name="app_name", and modify the value to whatever you want. In this case we will change "Rizline" to "Rizline Modded".

We have modified all the files we need. Now we have to build the modified app and install it on our device.

Build and sign the modified APK

Install the modified APK to the device

This page has some work to do. Help the wiki by editing this page!

  • Add download link to adb, zipalign and apksigner
  • Finish the guide
Retrieved from "https://rgwiki.stary.pc.pl/w/index.php?title=Guides/Monitoring_network_traffic_from_an_Android_app&oldid=2972"