Nginx Reverse Proxy Configuration: Fixing Prometheus 404

1. Problem Description

When setting up an Nginx reverse proxy configuration for a multi-path backend Prometheus application, Prometheus automatically redirects to /graph for normal access. If you configure the Nginx reverse proxy with just a basic http_proxy to the backend Prometheus port, it results in a 404 error. The following describes the specific configuration and error details:

    location /prometheus {
     proxy_pass http://10.10.10.227:9090;
     }

2. Use tcpdump to capture packets

tcpdump -i ens192 tcp port 80 or tcp port 9090 -w nginx.pcap

3. Use wireshark to analyze

From the above screenshots, we can see that client 189, which is the browser, requests 103, and 103 requests 227. From the 404 returned by 227, we can see that the actual requested address is http://10.10.10.227:9090/prometheus, and this path does not exist in 227. The real address is http://10.10.10.227/graph.

Because nginx is a multi-path location /prometheus, the path he requested has /prometheus, which makes it inaccessible. If location / is used, the real address he requested is http://10.10.10.227:9090. At this time, 227 will automatically jump to http://10.10.10.227:9090/graph, so nginx here needs to be modified.

4. Modify nginx configuration

Change to the following:

    location /prometheus {
     rewrite   ^/prometheus/?(.*)$ /$1 break;
     proxy_pass http://10.10.10.227:9090;
     }

The rewrite part of the above code means that the path starting with /promethues will be removed from /promethues and only the content after it will be retained. However, if the rewrite process is just continued, the access will become like this:

It still does not meet our expectations, so we have to modify it and use proxy_redirectit. Modify it to the following:

    location /prometheus {
     rewrite   ^/prometheus/?(.*)$ /$1 break;
     proxy_pass http://10.10.10.227:9090;
     proxy_redirect $uri /prometheus$uri;
     }

The proxy_redirect field means adding a /promethues in front of the access path, so that the path combination /promethus/graph specifically accesses the backend prometheus, and the front /promethues represents the content of the nginx proxy part, and the back /graph is the real Prometheus address.

5. Multiple nginx scenarios

In reality, there may be another nginx in front of nginx as the entry nginx, or there may be a load device such as F5 in front of nginx. At this time, the above configuration has problems. Because it is a customer environment, the VPN has expired and I can’t take a screenshot. Here I will directly write out the solution. I still need to modify the nginx configuration and change it to the following

  location /prometheus {
    rewrite   ^/prometheus/?(.*)$ /$1 break;
    proxy_pass http://10.10.10.227:9090;
    proxy_redirect / https://xxx.xxx.com/prometheus/;
}

The last proxy_redirect simply redirects any link to the final address to be visited.

6. Conclusion

This guide addresses issues related to Nginx Reverse Proxy Configuration, specifically resolving 404 errors when using a reverse proxy for Prometheus. It explains how Prometheus’s automatic redirection can cause conflicts with Nginx configurations and provides step-by-step solutions using rewrite and proxy_redirect directives. Additionally, it covers packet analysis using tcpdump and Wireshark to troubleshoot and optimize your reverse proxy setup, ensuring seamless integration even in multi-layered Nginx environments.