Title: Essential HTTP/2 Troubleshooting: Resolving Curl Access Failures Due to Header Formatting Issues

After launching HTTP/2 for a specific service, accessing a particular interface through curl consistently failed, which prompted the need for HTTP/2 troubleshooting.

The developers suspected that the operational team’s HTTP/2 configuration was incorrect, causing access failures. However, other domains configured with HTTP/2 were functioning normally, so we decided to take a closer look at the issue together.

Step one of troubleshooting: When in doubt, capture packets. In the absence of any preliminary information, first capture packets to see what was being transmitted. Since HTTP/2 requires communication over HTTPS, capturing packets here involves using some Wireshark techniques for capturing HTTPS packets. I’ve discussed this in a previous Bilibili share, which you can check out if you’re interested. Here’s the link: “Ways to Capture HTTPS Traffic with Wireshark” [https://www.bilibili.com/video/BV1ur4y1Y7NB]

In short, it involves exporting the premaster-secret to help Wireshark decrypt data. The format of the Wireshark SSL keylog looks like this, specifically defined at https://github.com/boundary/wireshark/blob/master/epan/dissectors/packet-ssl-utils.c#L4183

Wireshark SSL keylog format

What curl needs to do is print the key to a file, and the source code for this part can be found at: https://github.com/curl/lib/vtls/keylog.c, as shown below.

curl source code

For curl, we only need to specify an environment variable, and then we can decrypt the captured packets.

Wireshark decryption results are shown below.

Wireshark decryption result

It seemed that an incorrect packet was sent by the HTTP/2 server, causing the client to respond with an RST frame.

Let’s continue to see what the HTTP/2 server replied. By examining the packets, we indeed found something interesting.

Wireshark packet result

After the expires header, there was an extra space, while the other headers didn’t have it. It was suspected that this caused the issue. We discovered that by adding a Cache-Control request header, no expires header was returned in the response, and the request succeeded, further confirming this was the problem.

The returned result is shown below.

After confirming with the business team, it was indeed the case that there was an extra space after expires. Once removed, access returned to normal immediately.

Expires header

In the era of HTTP/1.1, curl was legal without issues, but in HTTP/2, this presented a problem.

Of course, this still doesn’t prove it conclusively unless curl tells me directly.

Why does the space cause an issue? It requires analysis from curl’s underlying layers. curl’s HTTP/2 is implemented using the nghttp library, which can also initiate requests directly via the command line.

Using nghttp to access it confirmed our suspicions.

nghttp access result

nghttp is an open-source project, where you can conveniently clone and compile the source code locally for debugging. It checks the legality of headers during header processing.

The legality of ASCII characters is defined here.

As shown in the image below, the space (SPC) has an ASCII value of 32 (0x20), and its VALID is 0, indicating that space is an illegal header character.

This was confirmed synchronously via GDB.

GDB debugging

At this point, we understand why curl fails to process headers with spaces. Chrome and Safari have similar issues, which might interest those of you who want to explore further.