Introduction and problem statement
I was recently working on setting up an Azure IoT hub based system for MQTT messaging and had to explore some low level stuff while debugging communication issues. It was then that I came across the fact that Azure does TLS re-negotiation while authenticating clients that use X.509 certificates.
To re-create the scenario, let us use openssl’s
s_client to post a message to the event hub using HTTP post. The command to use would be:
1 openssl s_client -host vysakh-testHub1.azure-devices.net -port 443 -cert selfsigned.crt -key selfsigned.key
This command will connect to Azure IoT hub over a secure channel, do a full ssl handshake and derive a session specific
Master secret. This key will be printed in the console and it can be used to decrypt application logs using wireshark. At this point, device certificate is not requested.
When the actual request is posted, if authentication mode of the requesting device is X.509, Azure IoT hub will perform a TLS re-negotiation by sending
hello request TLS handshake. This triggers a full handshake between the device and server but this time, it would include a
certificate request handshake. At the end of this re-negotiation, a new
Master secret is derived and it would be used for further application data transfer. However, s_client does not print this re-negotiated key and using default setups, it would be impossible to decrypt further application data exchange between the device and client.
Another issue is that while using a standard MQTT client like
mosquitto_pub over secure channels there is no way to observe application level data once secure channel is established.
I will be describing how to decrypt TLS traffic using wireshark in upcoming sections
overview of the hack
Even without detailed understanding of openssl, it is easy to understand that the sure-shot way to hijack/print keys from the stack is to modify
SSL_write(). These functions will definitely be called by all code written on top of openssl when it has to send or receive data over a TLS socket.
These API takes an argument (
s) of type
struct ssl_st. On digging a bit through the code, it was clear that this can be used to print the master key using the member:
modifying and compiling libssl
Latest release version of openssl can be downloaded from its website. To compile, use following commands after unpacking the archives. At the time of writing, latest release version was
1 2 ./config make
Before compiling, we will modify the read function to print
master_key of the calling session with the following patch in
We can now run the freshly compiled openssl app using the modified library by running the following command:
1 LD_LIBRARY_PATH=. apps/openssl s_client -connect google.com:443
Once the session is established, send a dummy
GET / request to see the patch in action
using the hack with re-negotiating servers
Now that we have the capability to print out session master_key using our modified version of openssl, we can use it with Azure to see the re-negotiated key.
I have already created a device that uses X.509 certificates for authentication. Issue the following command to connect to Azure IoT hub and POST an event request as this device:
1 LD_LIBRARY_PATH=. apps/openssl s_client -connect vysakh-TestHub1.azure-devices.net:443 -cert ../selfsigned.crt -key ../selfsigned.key
Once a connection is established, post an event using the below HTTP request:
1 2 3 4 5 6 7 POST /devices/testDevice2/messages/events?api-version=2016-02-03 HTTP/1.1 Host: vysakh-TestHub1.azure-devices.net User-Agent: testDevice Content-Length: 8 Accept: application/json testData
From the console output of our patch, you can see that a new key has been negotiated
decrypting re-negotiated session using wireshark
Wireshark comes with an inbuilt capability to decrypt TLS traffic if master_key can be provided. Master key is matched to the sessions
random send as part of
Client hello using a “master-secret log file”
You can tell Wireshark where to find the key file via
Edit→Preferences→Protocols→SSL→(Pre)-Master-Secret log filename. Key log file format is :
1 CLIENT_RANDOM <space> <64 bytes of hex encoded client_random> <space> <96 bytes of hex encoded master secret>
when there are more than one CLIENT_RANDOMs involved, add more similar lines to the file and Wireshark will select the appropriate master key for decryption.
Initial client_random can be obtained form the logs itself. In this case, we will first log the entire transaction using wireshark, fetch the initial client_random from the logs and then match them to the master_keys printed by our patch.
In the encrypted logs we will be able to see only the first
Client Hello since the re-negotiated hello will be under the encryption of the first session. Copy the random as a “hex stream” by right clicking
random as shown in the image. Paste it along with the first master secret printed by our patch into a file in the format described above.
After saving the file, point wireshark to it and reload the file by clicking
Upon reloading, the re-negotiation client hello will be available in clear.
Now, copy the new
random, append it to the key log file along with the re-negotiated master key from our patch and re-load the file to see the entire log in clear.
Decryption of re-negotiated traffic was recently fixed in wireshark. I was not able to see it in action in release
Version 2.2.6 (v2.2.6-0-g32dac6a). Instead, I used automated build version
In case you are sniffing WiFi traffic from a secure AP, the packets in air themselves will be encrypted.
In case of WPA authenticated network, first generate the PSK by entering SSID and passphrase in the online Wireshark Raw key geenrator tool here
Next, open wireshark and go to
Edit>Preferences>IEEE 802.11 and click “Edit” next to Decryption keys”. Add a new key under
wpa-psk and enter the generated PSK there.
decrypting secure MQTT traffic
Now that we have a modified TLS library that can provide us required information, we can use it to have more fun. For instance, I used it to decrypt MQTT traffic that is going through a secure channel. An alternate option would have been to modify the MQTT client source to print the session keys. But where is the fun in that??
However, I faced some issues in doing this using the library that we just compiled. At first,
ldd command showed that default mosquitto_pub in my system was linked against v1.0.0 of libssl. So, I downloaded sources of openssl 1.0.0k adn compiled it with the patch. However, upon running the tool against the newly compiled library, I faced another issue where it complaints about missing version information
1 2 3 mosquitto_pub: ./libcrypto.so.1.0.0: no version information available (required by /usr/local/lib/libmosquitto.so.1) mosquitto_pub: ./libssl.so.1.0.0: no version information available (required by /usr/local/lib/libmosquitto.so.1) mosquitto_pub: ./libssl.so.1.0.0: no version information available (required by /usr/local/lib/libmosquitto.so.1)
I figured out that this is because debian patches for openssl that is expected by my mosquitto_pub client is missing in the official sources. So, I downloaded the official debian version sources and patches and compiled it.
applying debian patches and compiling sources
quilt for applying patches to official sources. For applying these patches and compiling, download the following files from ubuntu repository:
- original sources: openssl_1.0.2g.orig.tar.gz
- patches and rules: openssl_1.0.2g-1ubuntu11.debian.tar.xz
unpackage both the archives and copy
debian folder from the patch package into original souces folder. Now, to apply the patch, issue the following commands:
1 2 export QUILT_PATCHES=debian/patches quilt push -a
instlall debhelper package using
sudo apt install debhelper before building.
Now, compile the package using the following command.
Once compiled, make the same changes to SSL_read() as we did before and re-compile the library by issuing
debian\rules will cause tests that are run as part of the script to fail since we are adding additional prints. This way, we just re-compile the files effected by
ssl_lib.c changes and re-package the lobrary.
Once compilation is through, run
mosquitto_pub using the following command to see the patch in action. Wireshark based decryption uses the same method above
1 mosquitto_pub -h iot.eclipse.org -p 8883 -t "testTopic" -m "testMessage" --capath /etc/ssl/certs/ -d
In case MQTT server uses a non-standard port (e.g. 8884 in case of
test.mosquitto.org for mutual authentication), the captured logs will not be identified as TLS. To get this detected as TLS, add the port into a comma separated list in
The above step would enable us to get
CLIENT_RANDOM out of the dissected handshake. However, application data after TLS negotiation is not detected as MQTT packets. For this, right click on the application data packet and click
Decode As... and enter
MQTT in the