stut-it Martin Stut - Information Technology Tailored to You
By Martin Stut, 2017-02-06
I have heard of too many stories of browser add-ons reporting innocent user’s browsing history to data vendors, disguised as “anonymous usage statistics”, which later turns out to be not very anonymous, but easily traceable to individuals. I have set up an environment to check what really happens. This includes intercepting SSL traffic, which in theory should be hard, but in practice is easy, by using Mitmproxy as a web proxy.
Mitmproxy’s home page is https://mitmproxy.org/ . It is a suite consisting of the interactive program Mitmproxy and the stream dumper (think tcpdump) mitmdump . The name is a combination of the acronym Man In The Middle (the type of attack performed here) and proxy, a type of server widely use especially in company’s internal networks.
You absolutely need a 64-bit OS to run Mitmproxy on. Attempting it on a 32-bit system fails with odd error messages you won’t at first glance relate to 64-vs-32-bit issues.
Mitmproxy is at home on Linux, but there is also a Windows version available. I did it on a desktop PC running Ubuntu Mate 16.04 LTS, 64-bit.
Working along http://docs.mitmproxy.org/en/latest/install.html :
mitmproxy-1.0.2-linux.tar.gzfrom https://github.com/mitmproxy/mitmproxy/releases .
tar xvzf mitmproxy-1.0.2-linux.tar.gzresults in three binaries:
./mitmdump --helpshows a long list of options, I won’t repeat here. I found needing only very few of them.
./mitmproxy. This is an interactive program (character based, curses-like user interface), initially showing one line per flow (HTTP request-response pair).
192.168.127.44:8080as a proxy for all protocols. In my first test this was Firefox, and it could well have been on a different computer, as long as it can talk TCP on port 8080 to the Mitmproxy machine. 192.168.127.44 happened to be the IP address of the Mitmproxy machine in my home network at the time of testing.
./mitmproxy -w test2.stream
--anticache(modify the request to make the server re-send content possibly cached by the browser) and
--anticomp(prevent the server from sending compressed data) options can be very relevant, so my recommended command line is
./mitmdump --anticache --anticomp -a log.stream.
-ameans append instead of
-wfor write/overwrite. I put this line into a little shell script
/home/martin/bin/catch-as-catch-can, to be able to start catching without having to remember all these options.
./mitmdump -n -r infile -w outfile "~m post" reads
infile and creates
outfile, containing only streams using the
post method (
For a full list of command line options see http://docs.mitmproxy.org/en/latest/mitmdump.html
Filter expressions are described in detail in http://docs.mitmproxy.org/en/latest/features/filters.html
You can also do filtering interactively by calling
./mitmproxy -n -r infile and then using keyboard commands:
As a testing environment, I’m using a Windows 10 preview VM inside VirtualBox.
In the Windows VM, I have set the proxy to
192.168.127.44:8080, both in Internet settings and in admin cmd >
netsh winhttp import proxy source=ie
Installing Mitmproxy’s certificate the standard way produced no error message, but both the Edge browser and Windows Update kept complaining about certificate issues.
What helped was the hint from http://docs.mitmproxy.org/en/stable/certinstall.html , Windows (automated) section:
certutil.exe -importpfx Root mitmproxy-ca-cert.p12 . It asked for a “PFX Password”, but Just hitting enter, specifying an empty password, made it go through without error message.
After closing and reopening Edge I could call HTTPS websites which were happily displayed (and logged by Mitmproxy).
When starting Windows Update, Mitmproxy throws certificate verification errors.
Also the Windows 10 telemetry seems to do some non-standard things. Mitmproxy complains
192.168.127.28:54054: CONNECT watson.telemetry.microsoft.com:443 << HTTP protocol error in client request: Unexpected EOF
After performing the steps above, I have an environment capable of logging and analyzing the data flow of apps and browser add-ons, even if they encrypt network traffic by HTTPS. The observation results of certain products are likely to be the topic of more blog posts.