Editing Guides/Monitoring network traffic from an Android app/editcopy

Many Android apps use web (HTTP) requests to communicate with their servers. This guide will help you set up a proxy and view web traffic going between your Android device and the internet.

== Prerequisites ==

To follow this guide, you need:
* an Android device (root access optional, but recommended),
* a computer (with Linux, macOS or Windows - other operating systems might work, but might require additional steps),
* a USB cable to connect your mobile device to the computer,
* access to a network (eg. via Wi-Fi) that you can connect to with the computer and the mobile device.

== Set up a proxy ==

First, we will install and setup mitmproxy, a proxy that will intercept web requests going between the Android device and the internet.

mitmproxy has two interfaces: a web interface (<code>mitmweb</code>) and a terminal interface (<code>mitmproxy</code>). We will use the web interface, as it's simpler, but the terminal interface offers more features for advanced users.

<ol>
<li>Go to https://mitmproxy.org on your computer, and download ''mitmproxy'' for your operating system.</li>
<li>
Run the <code>mitmweb</code> executable. The mitmproxy web interface should open in your default browser. If you need to open it manually, go to http://localhost:8081.
[[File:mitmproxy web interface.png|none|frame|The mitmproxy web interface.]]
</li>
<li>
Note the IP address of your computer on the network.
<tabber class="rgwiki-main-os-tabber">
|-|Linux=
In a terminal, run <code>ip a</code>. This will list your network interfaces. Look for those with names starting with <code>enp</code> and <code>wlp</code>, they are your wired and Wi-Fi interfaces. Within those, look for a line starting with <code>inet</code>, which is followed by the IP address and mask, separated by a slash. The mask is not needed here.
[[File:ip a command output.png|none|frame|<code>ip a</code> output. There are three interfaces, <code>lo</code>, <code>enp4s0</code> and <code>virbr0</code>. <code>enp4s0</code> is the wired interface. The IP address is highlighted.]]
|-|macOS=
In a terminal, run <code>ifconfig</code>. This will list your network interfaces.
|-|Windows=
In a terminal, run <code>ipconfig</code>. This will list your network interfaces.
</tabber>
</li>
</ol>

'''Note:''' You might need to change firewall settings for the proxy to be accessible on the network.

== Set up proxy settings on the mobile device ==

[[File:Proxy Settings Android Screenshot.jpg|thumb|x500px|right|alt=A screenshot showing proxy settings on Android.|In this example, we set the host name to 192.168.3.57, and the port to 8080 - the same values as shown in the terminal output above.]]

To make web traffic from the device go through the proxy server, you have to change its proxy settings.

'''Note:''' Due to differences between different Android vendors, these steps might slightly differ.

# On your Android device, go to ''Settings'' → ''Network & internet'' → ''Wi-Fi'' and select your Wi-Fi network.
# Tap the pencil icon to edit the connection, then in ''Advanced options'', change ''Proxy'' to ''Manual''.
# Set ''Proxy hostname'' to the IP address of your computer, and ''Proxy port'' to 8080, which is what mitmproxy uses by default.
# Save the settings. You might need to disconnect from the network and connect to it again.

You should be able to see some traffic going through on the mitmproxy web interface, in chronological order.
[[File:mitmproxy web interface after connection.png|800px|none|frame|mitmproxy web interface after connection.]]

The proxy is now set up. However, if you try to access a website on your Android device, you might notice that it's either marked as insecure or doesn't work at all. This is because most websites and software nowadays are secured using the HTTPS protocol, which prevents bad actors from doing what we are trying to do - intercepting your web traffic.

HTTPS utilizes certificates, which allow clients to verify the identity of servers. When mitmproxy intercepts traffic, it uses it's own certificate, which is (rightfully) not trusted by your Android device, thus the identity verification fails, and an error is reported.

== Install the certificate on the Android device ==

To intercept HTTPS requests, you need to install the mitmproxy certificate on your Android device.

Sadly, Android is a closed down operating system, and it's not possible to easily install a certificate in a way that it's trusted by all apps.

You have two options:

* '''Install the certificate to the system store (requires root access).''' This is trusted by (almost) all apps.
* '''Install the certificate to the user store (doesn't require root access).''' This usually requires modifying the app to work, with the process having to be repeated for each app you want to monitor and every time you want to update.

=== Installing to the system store (root access) ===

'''Note:''' This assumes the device is rooted with Magisk. Steps might differ for other solutions.

# On your Android device, go to http://mitm.it while connected to the proxy. This website is hosted by mitmproxy, and allows you to download a Magisk module containing the certificate. Scroll down to ''Android'' and tap on ''Show instructions''. Scroll down to the instructions for Magisk, and click on the ''this Magisk module'' link to download the file.
# Open ''Magisk'' and go to the ''Modules'' tab.
# Select ''Install from storage'', and select the module file. Confirm installation of the module, and wait until it completes.
# Reboot the device by tapping on ''Reboot''.

Apps should now trust the mitmproxy certificate, and HTTPS traffic should appear in the mitmproxy web interface.

=== Installing to the user store (no root access) ===

# On your Android device, go to http://mitm.it while connected to the proxy. This website is hosted by mitmproxy, and allows you to download its certificate. Scroll down to ''Android'' and tap on ''Get mitmproxy-ca-cert.cer'' to download the file.
# Go to ''Settings'' and search for options about certificates. On my device, it is under ''Settings > Password & security > System security > Credential storage''. From here, you should be able to view, add, and remove certificates installed on the system.<br>[[File:Credential storage Android Screenshot.jpg|x300px|thumb|center|alt=Screenshot of the "Credential storage" menu in Android Settings|Here, you can view, install and remove certificates from your device.]]<br>
# Select ''Install certificates from storage'', and then ''CA certificate''.<br>[[File:Install a certificate menu Android Screenshot.jpg|x300px|thumb|center|alt=Screenshot of the "Install a certificate" menu in Android Settings|Choose the "CA certificate" option on this screen.]]<br>
# A pop-up may show up, warning you of the safety concerns related to adding custom certificates. Read it, and if you are sure you want to do this, tap on ''Install anyway''.<br>'''Remember!''' After you are done exploring network traffic from an app, you can remove the certificate by going into this settings menu, selecting ''Trusted certificates'', switching to the ''User'' tab, selecting the certificate and tapping on ''Uninstall''.
# You may be asked to verify your identity. After confirming your identity, you will be asked to choose a file. Select the file you downloaded before - it should be named <code>downloadfile.crt</code> and it should be located in your <code>Downloads</code> folder. [[File:Downloadfile.crt screenshot.jpg|x300px|thumb|center|alt=Screenshot of an Android file picker, with a downloadfile.crt file visible. |Choose the downloaded certificate file (usually called <code>downloadfile.crt</code>)]]
# After selecting the file, the certificate should be installed. You can verify that, by going into the ''Trusted certificates'' menu, and selecting the ''User'' tab - your certificate should be there. You can tap on it to view detailed information about the certificate or uninstall it if you don't need it anymore.

[[File:Charles proxy certificate installed on Android screenshot.jpg|x300px|thumb|center|alt=Screenshot of the "Trusted certificates" menu in the Android settings, showing a Charles proxy certificate.|The certificate was installed successfully.]]

The certificate should now be installed properly on the mobile device, however we still will not be able to see HTTPS traffic in ''Charles''.

[[File:YouTube Android app cannot connect to servers Screenshot.jpg|x500px|thumb|right|alt=Screenshot of the Android app for YouTube trying to connect to the servers, but failing, because it doesn't trust the certificate signed by an user-installed certificate authority.|Now, the YouTube app does not load.]]
If you now try to open an application on your mobile device, you will see that it will not be able to connect to the servers. This is because all traffic is now captured by ''Charles''. Most apps do not accept user-installed certificates as valid, which means even though our certificate is installed, the applications are acting as if it is an unknown certificate, and therefore cannot connect to the server at all.

==== Get APK files of the application that we will modify ====

We will modify the application slightly so that it will accept user-installed certificates. In this guide, we will be using ''[[Rizline:Rizline|Rizline]]'' as an example.

===== Install tools =====

First we will need to download some tools:
* To connect to the device via a USB cable, we will use [https://developer.android.com/tools/adb Android Debug Bridge (adb)]
* To decompile and build Android applications, we will use [https://apktool.org/ Apktool]. To download and install Apktool, follow the instructions on this page: https://apktool.org/docs/install.
* To optimize the built APK, we will use [https://developer.android.com/tools/zipalign zipalign].
* To sign the modified APK, we will use [https://developer.android.com/tools/apksigner apksigner].

If everything is installed correctly, you should be able to open a terminal and type <code>adb</code>, <code>apktool</code>, <code>zipalign</code> and <code>apksigner</code> commands without errors.

===== Get APK files from the device =====

[[File:Android Allow USB debugging popup screenshot.jpg|x500px|thumb|right|alt=Screenshot of a pop-up asking the user to allow or deny USB debugging on Android.|A pop-up that appears after connecting the mobile device to your computer, with <code>adb</code> installed. If the pop-up doesn't appear, try typing <code>adb</code> in the terminal, or try using a different USB cable.]]
To modify the application files, we first need to get them onto our computer.

<ol>
<li>Enable USB Debugging on your mobile device.</li>
* To do that, go to ''Settings > Additional settings > Developer options > USB debugging'' (found in the ''Debugging'' section).
* If you cannot find the "Developer options" button, you might need to enable Developer Mode by going into ''Settings > About device > Version'' and tapping on the ''Build number'' option seven times. You should get a message saying "You are now a developer!"
* If you have any trouble with this, try following [https://developer.android.com/studio/debug/dev-options the official instructions from developer.android.com].
<li>Connect your Android device to your computer using a USB cable. Choose "Android Auto" on your device if prompted.</li>
<li>
On your mobile device, a pop-up should appear, asking for permission to allow USB debugging. Tap on allow; and if you plan on doing this often, you might want to tap the checkbox to always allow USB debugging from this computer. If the pop-up doesn't appear immediately, don't worry; it might appear after typing the command in the next step.</li>
<li>On your computer, open up a terminal and type:
<pre>
adb shell
</pre>
You should now have access to your mobile device on your computer via the terminal.
[[File:Adb shell Screenshot.png|x40px|thumb|center|alt=A screenshot showing the command "adb shell" typed in a terminal.|''*hacking noises*'' we're in.]]
</li>
<li>
Now we need to find the package ID of the app that we are interested in. One way of doing that is finding the ID in the list of installed packages. In <code>adb</code>, type the following command to show the list of packages installed on the device:
<pre>
pm list packages
</pre>
[[File:Pm list packages Screenshot.png.png|400px|thumb|center|alt=A screenshot showing a few of the 443 package IDs.|On my device, this command shows 443 packages. Here is a few of them.]]
This command will only show the package IDs, so you cannot search by name. If you have a lot of applications installed, you might want to try piping the command to <code>grep</code>. <code>grep</code> is a command line utility that will filter down results by showing only the lines that have a specified string of text inside of them. For example the following command will show applications that contain the word "Pigeon" in their ID:
<pre>
pm list packages | grep Pigeon
</pre>
[[File:Pm list packages with grep example Screenshot.png|x80px|thumb|center|alt=A screenshot showing the filtered results with only two package IDs.|This time, only applications made by Pigeon Games will be shown.]]
<br><br>
Another way of getting the ID of an application, is just searching the app on the Google Play Store website. When you go to an application page on Google Play, the ID of the package will be embedded in the URL. For ''Rizline'', the link is https://play.google.com/store/apps/details?id=com.PigeonGames.Rizline, and so the package ID is <code>com.PigeonGames.Rizline</code>.
<br><br>
</li>
<li>
The next step is to find the path of the APK files on the mobile device. Note that it is common for one application to have multiple APK files. Usually, there is one <code>base.apk</code> file, that is common for all devices, and several <code>split</code> APK files, that are device specific. For example, if your mobile device has a larger screen, it may download a <code>split</code> apk with extra high-quality assets.

To find the paths to the APK files, you can use the <code>pm path <package_id></code> command. In our case, the command will be:
<pre>
pm path com.PigeonGames.Rizline
</pre>
[[File:Pm path example Screenshot.png|x80px|thumb|center|alt=A screenshot showing two paths to APK files.|''Rizline'' has two APK files - a <code>base.apk</code> file and a <code>split_config.arm64_v8a.apk</code> file.]]
Now we know that the paths to our two APK files are:
* <code>/data/app/~~tMHzIziIkD3B_oH7sbOphw==/com.PigeonGames.Rizline-P6NKIGPvbzqSbdVySK53MQ==/base.apk</code>, and
* <code>/data/app/~~tMHzIziIkD3B_oH7sbOphw==/com.PigeonGames.Rizline-P6NKIGPvbzqSbdVySK53MQ==/split_config.arm64_v8a.apk</code>. We can use that information to download the files from the device to our computer. <br><br>
</li>
<li>
Exit the <code>adb shell</code>:
<pre>
exit
</pre>
</li>
<li>
Create a new directory for our APK files. To do that, you can either create it using a GUI file explorer, or by typing the <code>mkdir</code> command. In my case, the directory name is <code>rizline_mod</code>
<pre>
mkdir rizline_mod
</pre>
</li>
<li>
Navigate to the new directory using the <code>cd</code> command.
<pre>
cd rizline_mod
</pre>
</li>
<li>
Now we can use the <code>adb pull</code> command to copy the files to the computer.
<pre>
adb pull "/data/app/~~tMHzIziIkD3B_oH7sbOphw==/com.PigeonGames.Rizline-P6NKIGPvbzqSbdVySK53MQ==/base.apk" "base.apk"
adb pull "/data/app/~~tMHzIziIkD3B_oH7sbOphw==/com.PigeonGames.Rizline-P6NKIGPvbzqSbdVySK53MQ==/split_config.arm64_v8a.apk" "split_config.arm64_v8a.apk"
</pre>
The <code>adb pull</code> command takes two arguments: the first one is the path on the mobile device, and the second one is the destination path, on your computer. In this case, we just name the resulting files <code>base.apk</code> and <code>split_config.arm64_v8a.apk</code> respectively.

<!--[[File:Adb pull example Screenshot.png|500px|thumb|center|Copying APK files from an Android device to a computer.]]-->
</li>
</ol>

[[File:Adb get apk files full example screenshot.png|1200px|thumb|center|alt=Screenshot of a terminal with all commands used to get APK files from a mobile device.|In summary, here are all of the commands used to copy APK files from a mobile device to a computer]]

==== Decompile the APK files using apktool ====

[[File:Compressed AndroidManifest.xml screenshot.png|thumb|right|A screenshot of an [[AndroidManifest.xml]] file, when extracted from base.zip -- this doesn't look like an .xml file at all!]]

To modify the files of an APK package, we will first need to extract individual files from the downloaded APK files.

APK files are essentially just ZIP files in disguise. If you change the <code>.apk</code> file extension to <code>.zip</code>, you can open the file in any archiving software, just like a normal ZIP file. However, some files inside the APK file (such as [[AndroidManifest.xml]]) are in a weird, binary format. To read all of the files properly, we will use [https://apktool.org/ Apktool].

[[File:Uncompressed AndroidManifest.xml screenshot.png|thumb|right|A screenshot of an [[AndroidManifest.xml]] file, when extracted using apktool -- this looks much more familiar!]]
<ol>
<li>Open a terminal in the directory with your copied APK files.</li>
<li>
Type the following commands to decompile the APK files using Apktool:
<pre>
apktool d base.apk
apktool d split_config.arm64_v8a.apk
</pre>
This will create new directories - <code>base</code> and <code>split_config.arm64_v8a</code> with the contents of the APK files.
</li>
</ol>

You can now explore the application files freely using a file manager.

==== Modify the application to allow user-installed certificates ====

<ol>
<li>
Firstly, we will add a new file to our application. This file will allow the app to trust user-added certificate authorities. Create the file at <code>base/res/xml/network_security_config.xml</code> with the following contents:
<br>
'''<code>base/res/xml/network_security_config.xml</code>''':
<pre><nowiki>
<network-security-config>
    <debug-overrides>
        <trust-anchors>
            <!-- Trust user added CAs while debuggable only -->
            <certificates src="user" />
        </trust-anchors>
    </debug-overrides>
</network-security-config>
</nowiki></pre>
</li>
<li>
The next few changes we will make will occur in the <code>[[AndroidManifest.xml]]</code> file. Open <code>base/AndroidManifest.xml</code> with a text editor of your choice.
</li>
<li>
Find the <code><nowiki><application></nowiki></code> tag. It is located within the main <code><nowiki><manifest></nowiki></code> tag. We will insert two new attributes into this tag:<br>
<pre>
android:debuggable="true" android:networkSecurityConfig="@xml/network_security_config"
</pre>
This will make the app "debuggable", and it will load the settings from the <code>network_security_config.xml</code> file we created before.

[[File:AndroidManifest.xml adding android-networkSecurityConfig Screenshot.png|900px|thumb|center|alt=Screenshot highlighting where to add the two attributes.|We added two new attributes: <code>android:debuggable="true"</code> and <code>android:networkSecurityConfig="@xml/network_security_config"</code> to the <code><nowiki><application></nowiki></code> tag.]]<br>
</li>
<li>
Find an attribute named <code>package="..."</code> in the top-most <code><nowiki><manifest></nowiki></code> tag, and replace the contents of this attribute with your own app ID. In this example, we will replace <code>com.PigeonGames.Rizline</code> with <code>rgwiki.rizline.example_mod</code>. This change will allow us to have both the original, unmodified version of ''Rizline'' as well as our modified version at the same time.

[[File:AndroidManifest.xml package ID Screenshot.png|700px|thumb|center|alt=Screenshot of the AndroidManifest.xml file with the package attribute highlighted.|We will change the <code>com.PigeonGames.Rizline</code> ID, to our own, made-up one: <code>rgwiki.rizline.example_mod</code>.]]<br>
</li>
<li>
Find other instances of the original package ID and change them to our new package ID as well. In the case of ''Rizline'', the <code>com.PigeonGames.Rizline</code> ID also appears later in the file, as part of another ID:
[[File:AndroidManifest.xml package ID in a provider ID Screenshot.png|900px|thumb|center|alt=Screenshot highlighting the package ID found in a different place in the file|We need to replace this part of an ID with our new ID as well.]]

This varies from application to application. In this case, the app cannot be installed unless this other ID is changed to a new one. Other apps may have more IDs that you have to replace; and in some cases, you don't have to replace every ID with a new one, only some.<br><br>
</li>
<li>
There is one more ID we have to change. Open the <code>split_config.arm64_v8a/AndroidManifest.xml</code> file. This file also has a package ID, and it should match with the package ID from the <code>AndroidManifest.xml</code> in the <code>base</code> directory.

[[File:AndroidManifest.xml split package ID Screenshot.png.png|900px|thumb|center|alt=A screenshot showing a short AndroidManifest.xml file for a split APK|This is the last ID we have to change in ''Rizline''. In other apps, there may be more. Try to search the directory for the original ID and find them all!]]<br>
</li>
<li>
The last change we will make, is a change of the application name, to differentiate between the two applications on the device more easily. Open <code>base/res/values/strings.xml</code> in a text editor. Find an entry that has the attribute: <code>name="app_name"</code>, and modify the value to whatever you want. In this case we will change "Rizline" to "Rizline Modded".

[[File:Values.xml app name Rizline Modded.png|500px|thumb|center]]
</li>
</ol>

We have modified all the files we need. Now we have to build the modified app and install it on our device.

==== Build and sign the modified APKs ====

Building the app from a directory of files is simple. Unfortunately, creating an APK file is not enough to install it to a device. To actually install the application, we will need to ''sign'' the APK files as well.

<ol>
<li>
To build the APKs from files, we will use [https://apktool.org/ Apktool] again. Open the terminal in the <code>rizline_mod</code> directory and type the following commands:
<pre>
apktool b "base"
apktool b "split_config.arm64_v8a"
</pre>
This will combine all of the files in the <code>base</code> directory into an APK file, and separately, combine all of the files in the <code>split_config.arm64_v8a</code> directory into a second APK file. Those new APK files are stored in <code>base/dist/base.apk</code> and <code>split_config.arm64_v8a/dist/split_config.arm64_v8a.apk</code> respectively.
</li>
<li>
Now we will use <code>zipalign</code> to optimize the APK.
<pre>
zipalign -P 16 -f -v 4 "base/dist/base.apk" "base/dist/base_signed.apk"
zipalign -P 16 -f -v 4 "split_config.arm64_v8a/dist/split_config.arm64_v8a.apk" "split_config.arm64_v8a/dist/split_config.arm64_v8a_signed.apk"
</pre>
This will create new files named <code>base_signed.apk</code> and <code>split_config.arm64_v8a_signed.apk</code> in the same directory as the original files.
</li>
<li>
Before signing the APK, we need to create a keystore. A keystore is a file that stores our key pair used to sign APKs. We only need to create a keystore file once, and then we can use it repeatedly. We will use <code>keytool</code> to create a new keystore. The command below will create a new keystore with the filename <code>keystore.jks</code>, and it will be valid for 99999 days.
<pre>
keytool -genkeypair -keystore "keystore.jks" -keyalg RSA -validity 99999
</pre>
After entering this comand, you will be asked to set a password. In this guide, we will simply use the word "password" as the password. Don't forget the password you set - you will need it later.

Then, you will be prompted to provide various pieces of information. You can skip all of the prompts by just pressing Enter - apart from the last one, where you have to type <code>yes</code>.
[[File:Keytool create keystore example screenshot.png|800px|thumb|center|alt=Screenshot of a terminal with a basic keytool command.|You can just skip most of the prompts, and type "yes" at the end.]]
</li>
<li>
Now it's time to sign the APK files. Use the commands below and type in the keystore password you set at the previous step to sign the APKs.
<pre>
apksigner sign --ks "keystore.jks" "base/dist/base_signed.apk"
apksigner sign --ks "keystore.jks" "split_config.arm64_v8a/dist/split_config.arm64_v8a_signed.apk"
</pre>
</li>
<li>
We can verify that we've signed the files properly using <code>apksigner verify</code>:
<pre>
apksigner verify "base/dist/base_signed.apk"
apksigner verify "split_config.arm64_v8a/dist/split_config.arm64_v8a_signed.apk"
</pre>
If there is no output, that means that we've done everything correctly! If there are some warnings, chances are you can probably ignore them. In the case of ''Rizline'', there are some warnings, but everything still works. If there are errors, the app will likely not work.

These are the steps that you have to do every time you make a change to the app files. If you want to speed up the process of building in the future, you might want to consider creating a shell script to execute all of these commands for you.
[[File:Apktool building and signing an APK file terminal screenshot.png|900px|thumb|center|alt=A screenshot of commands used to create an APK file and sign it.|All commands used to build and sign an APK file. In this example, we omitted the <code>-v</code> flag in <code>zipalign</code> to hide unnecesary output.]]
</li>
</ol>

==== Install the modified APKs to the device ====

To install the files to your device, open the terminal and type the following command:
<pre>
adb install-multiple "base/dist/base_signed.apk" "split_config.arm64_v8a/dist/split_config.arm64_v8a_signed.apk"
</pre>

You can now launch the app and observe the HTTPS traffic in ''Charles''!

=== Certificate pinning ===

If you are sure you set up the certificate correctly and intercepting secure traffic still doesn't work, this might mean that the app uses ''certificate pinning''. This means the app will only trust certain certificates. Bypassing this issue can be complicated and is out of scope for this guide.

=== Monitoring web traffic on Android ===

If you cannot use a separate computer with Linux, macOS or Windows, you should still be able to monitor web traffic with Android alone. Android is just a really weird Linux distribution, and you can use [https://f-droid.org/en/packages/com.termux/ Termux] to create a more reasonable Linux environment. mitmproxy runs in Termux as well, alternatively you could also use [https://f-droid.org/en/packages/com.emanuelef.remote_capture/ PCAPdroid]. However, this is out of scope for this guide.

{{Todo|
* Add download link to adb, zipalign and apksigner}}

[[Category:Guides]]