SSL Handshake Capture with tcpdump

SSL is by far the most common protocol protocol that is used for encrypted data transmission over TCP connections. During this procedure, both parties must exchange different kinds of information, including public keys, encryption algorithms, protocol versions, and many others.

Sometimes, however, the exchange doesn’t go as it is supposed to and there is a need to detect issues and fix them. A common way to do it is by capturing SSL handshake with tchpdump. By doing this, you’ll be able to analyze in detail, what’s going on with your data transmission and take steps to solving the issues.

The tcpdump Command Overview

tcpdump is a powerful command-line network analysis tool that is used for capturing and displaying the packets transmitted and received over a network interface, available on most Unix-like operating systems, including Linux and macOS. If your point is to capture SSL handshake, you should know that the tool by itself can capture vast volumes of data, which you won’t be able to analyze in a productive way. With this in mind, you’ll have to apply a series of filters that will single out the packages that have directly to do with your task. The filtering criteria include source and destination IP addresses, port numbers, and protocols. 

Introduction to SSL Handshake

The SSL (Secure Sockets Layer) handshake is an essential part of the SSL/TLS (Transport Layer Security) protocol that is widely used for securing communication over computer networks. Although SSL has been largely superseded by TLS the term "SSL handshake" is still widely used to describe the initial negotiation phase of a TLS connection.

An SSL handshake consists of a series of messages that are exchanged between the client and the server, and these messages are the stages of establishing a secure and encrypted communication channel. Let’s have a look at them.

1. ClientHello: The handshake begins with the client sending a ClientHello message to the server. This message consists of the client's SSL/TLS version number, a list of supported cipher suites (algorithms for encryption, key exchange, and message authentication), a list of supported compression methods, and possibly a list of supported extensions.
2. ServerHello: To give a response to the previous message, the server sends a ServerHello message to the client, containing the server's SSL/TLS version number, the cipher suite and compression method selected from the client's list, and possibly server-selected extensions.
3. Server Certificate: For the sake of authentication, the server sends its certificate to the client, which contains the server’s public key, later used for key exchange by the client.
4. Certificate Request (optional): In case the server requires authentication from the side of the client, like with two-way SSL, the server sends the message to the client.
5. Key exchange: The client and the server exchange key information. This can take place in different ways. For example, the client may send a pre-master secret encrypted with the server's public key (like in the case of RSA key exchange). Alternatively, both client and server may generate keys together (like in the case of Diffie-Hellman key exchange). Based on the chosen cipher suite, this step may vary.
6. Server Hello Done: After the server finished sending messages to support the key exchange and server authentication, it sends the ServerHelloDone message to indicate it.
7. Client Key Exchange: To respond to the previous message the client sends a key exchange message, including a pre-master secret that depends on the key exchange method.
8. Certificate Verify (optional): In case client authentication is needed, the client may send a digitally-signed certificate verify message, allowing the server to verify the client's certificate.
9. Finished: Finally, both the client and server exchange encrypted Finished messages, which verify that the key exchange and authentication processes were successful.

SSL Handshake Message Filtering in tcpdump

Since tcpdump filters information on the packet level and cannot deal with such constructus as SSL/TLS protocols directly, you’ll have to apply filtres relying on general characteristics of handshake packets, like port numbers and packer sizes, to capture only the handshake process. Let’s have a look at approaches that you can use for this task.

• Filtering by port:

SSL/TLS handshakes typically occur over port 443 for HTTPS, although other ports used for securing protocols may be involved as well. In particular, to capture all traffic on the default HTTPS port 443, use:

tcpdump -i any 'port 443'

Keep in mind, that this will capture all the traffic on the port and not only the handshake messages.

• Capturing the beginning of the connection:

Since a handshake takes place at the beginning of an SSL/TLS connection, capturing the initial packet exchanges can help single out the handshake messages. For example, you can capture SYN and SYN-ACK packets so you can see the start of TCP connections taking place before the SSL/TLS handshake.

tcpdump -i any 'port 443 and (tcp[tcpflags] & (tcp-syn|tcp-ack) == tcp-syn)'

• Length filters:

Since handshake messages, ClientHello and ServerHello, in particular, tend to have a distinctive range in sizes, this fact can theoretically be taken advantage of to filter packets based on their TCP payload size.  Nevertheless, since there is a great degree of randomness to such variables, as size, this method is rather imprecise and unreliable.

• Capturing and analyzing with Wireshark:

Wireshark is a special tool that can provide you with a more detailed analysis of the traffic you’ve captured previously with tcpdump, filtering for specific handshake messages. First, capture the traffic from the specific port, writing it to a file:

tcpdump -i any 'port 443' -w capture_file.pcap

Open the capture_file.pcap file with Wireshark and use its SSL/TLS filters, like ssl.handshake.type for further analysis.

TLS Version-Specific PackeIt Capture

Although most modern applications use TLSv1.3 protocol for message exchange. Nevertheless, to preserve compatibility with older versions, TLSv1.0, TLSv1.1, and TLSv1.2 are supported. Each version is assigned a numerical code:

SSLv3 – 0x300

TLSv1.0 – 0x0301

TLSv1.1 – 0x0302

TLSv1.2 – 0x0303

TLSv1.3 – 0x0304

Given that this information is contained in the 10th and 11th bytes of the data in the SSL handshake message, the following filter can be used:

tcpdump "tcp port 8081 and (tcp[((tcp[12] & 0xf0) >>2)] = 0x16) \\
  && (tcp[((tcp[12] & 0xf0) >>2)+9] = 0x03) \\
  && (tcp[((tcp[12] & 0xf0) >>2)+10] = 0x03)"

Application Data Packet Capture over TLS

Handshake messages are followed by application data, which also contain the TLS version in the 2nd and the 3rd data byte. To capture it:

tcpdump "tcp port 8081 and (tcp[((tcp[12] & 0xf0) >>2)] = 0x17) \\
  && (tcp[((tcp[12] & 0xf0) >>2)+1] = 0x03) \\
  && (tcp[((tcp[12] & 0xf0) >>2)+2] = 0x03)" -w appdata.pcap

This way we can capture packets inholding 17 in the 1st and 03 in the 2nd and 3rd bytes.

SSL Connection Failure Tracing

For filtering errors, you should check the 1st byte containing either 15 or 21, numbers corresponding to specific failures:

tcpdump "tcp port 8081 and (tcp[((tcp[12] & 0xf0) >>2)] = 0x15) || (tcp[((tcp[12] & 0xf0) >>2)] = 0x21)" -w error.pcap

Concluding Remarks

Using tcpdump for capturing SSL handshake is an efficient way of analyzing traffic and finding bugs and failures if anything goes wrong. In this article, we’ve faced some approaches that can help you filter the captured traffic in proper way to be able to detect any issues. We hope that this guide was helpful, take care!