Behind a reverse proxy

The server speaks plaintext HTTP on port 3000 and does not terminate TLS. Anywhere other than a laptop, something in front of it almost always does.

Four things to get right

TLS terminates at the proxy. Bind the server’s port to loopback (or to a private network) so nothing reaches 3000 from outside, and let the proxy hold the certificate.

ISSUER is the public https URL. Not the internal address, not http://. It is the iss claim of every token, the base of every endpoint in the discovery document, and the redirect target of the console client. A client library that fetches discovery will follow whatever it says, so a wrong ISSUER produces a deployment that half-works, in ways that tend to be tedious to diagnose. See the environment reference.

Pass the original Host through. The server reads that header, and only that header, to work out which user bucket a request is addressed to when buckets are given hostnames of their own. It deliberately does not read X-Forwarded-Host: that header is set by whatever spoke to your proxy, so a proxy which does not strip an inbound copy would let a caller choose which population answers them — the highest-leverage thing a request could influence here. Host is what the client actually connected to, and what your proxy’s own routing already matched on.

The nginx block below sets proxy_set_header Host $host; for this reason, and Caddy’s reverse_proxy preserves it by default. A proxy that rewrites Host to the upstream’s internal name leaves every host-addressed bucket unreachable; path-addressed buckets are unaffected, which is what makes this worth stating rather than leaving to be discovered.

Choose rateLimit.trustedProxy on purpose. The next section covers it.

HSTS is emitted unconditionally

Strict-Transport-Security: max-age=63072000; includeSubDomains is sent on every response, including plaintext ones, and there is no branch on the scheme; that is intentional. In production the hop the RFC governs, server to user agent, is HTTPS, because TLS terminated at your proxy. The plaintext this process sees on its internal port is an artefact of termination. On http://localhost the header is inert, since a user agent must ignore it when it arrives over plaintext.

Two consequences for a proxy: do not add a second Strict-Transport-Security header (you will send two), and preload is left out on purpose. Adding it is a decision for whoever controls the apex domain, at their own edge.

rateLimit.trustedProxy has a wrong answer in each direction

From the project’s README:

rateLimit.trustedProxy is the one setting with a wrong answer in each direction. Leave it on when anything sits in front of this server (the shipped fly.toml does): with it off, every caller arrives as the proxy’s own address, so the whole internet shares one allowance and all traffic is refused within seconds. Turn it off when the server is directly exposed: with it on, any caller can set Fly-Client-IP to a fresh value per request and is never limited.

So: behind a proxy, leave rateLimit.trustedProxy at true, and make sure your proxy actually sets a forwarded header and overwrites any the client supplied instead of appending to it.

Exactly which header the resolver reads

With rateLimit.trustedProxy set to true, the origin resolver takes the first non-empty value of, in this order:

1. Fly-Client-IP
2. the first comma-separated entry of X-Forwarded-For
3. X-Real-IP

The value is trimmed and truncated to 64 characters. If none of the three is present, the request falls into a single shared unknown bucket instead of going uncounted. Omitting the header must not be a way to bypass the limiter.

With rateLimit.trustedProxy set to false, none of those headers is read at all: the transport peer address is used, and a request with no identifiable peer also lands in the shared bucket.

Forwarded (RFC 7239) is not read. If your proxy only emits that, add one of the three above.

Sample configuration

nginx

server {
  listen 443 ssl http2;
  server_name auth.example.com;

  ssl_certificate     /etc/letsencrypt/live/auth.example.com/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/auth.example.com/privkey.pem;

  location / {
    proxy_pass http://127.0.0.1:3000;
    proxy_http_version 1.1;

    proxy_set_header Host              $host;
    # $remote_addr, not $proxy_add_x_forwarded_for: this overwrites any header the client sent
    # instead of appending to it, and the resolver reads the FIRST entry.
    proxy_set_header X-Real-IP         $remote_addr;
    proxy_set_header X-Forwarded-For   $remote_addr;
    proxy_set_header X-Forwarded-Proto $scheme;
  }

  # Never rate-limit or auth-gate the liveness probe.
  location = /health {
    proxy_pass http://127.0.0.1:3000;
    access_log off;
  }
}

server {
  listen 80;
  server_name auth.example.com;
  return 301 https://$host$request_uri;
}

Caddy

auth.example.com {
  reverse_proxy 127.0.0.1:3000 {
    # header_up REPLACES the header rather than appending, which is the point: Caddy's default
    # X-Forwarded-For handling appends the peer to whatever the client sent, and this server
    # reads the FIRST entry — so an appended header would let a caller choose its own origin.
    header_up X-Forwarded-For {remote_host}
    header_up X-Real-IP {remote_host}
  }
}

Caddy obtains and renews the certificate itself, which is most of why its block is so much shorter.

Client certificates (mTLS)

Only for RFC 8705 — clients authenticating with a TLS certificate, or certificate-bound access tokens. TLS ends at the proxy, so the server never sees the handshake: the proxy has to verify the client certificate and forward it in a header, and the server has to be told to believe that header.

The header is RFC 9440’s Client-Cert: the DER certificate as a byte sequence, base64 between colons. A bare base64 X-Client-Cert is read when Client-Cert is absent, for proxies configured before RFC 9440. The server reads neither until mTLS.trustProxyCertificateHeader is on, and that setting has a wrong answer in each direction: off, no certificate is ever read; on behind a proxy that passes an incoming header through, anyone can send a certificate — which is public — and be believed.

So the proxy must set the header on every request, overwriting whatever the client sent, and remove the other one. In Caddy:

auth.example.com {
  tls {
    client_auth {
      mode verify_if_given
      trust_pool file /etc/caddy/client-ca.pem
    }
  }
  reverse_proxy 127.0.0.1:3000 {
    header_up Client-Cert ":{http.request.tls.client.certificate_der_base64}:"
    header_up -X-Client-Cert
  }
}

verify_if_given keeps every other client working without a certificate. nginx has no variable holding the DER certificate as base64 — $ssl_client_escaped_cert is URL-encoded PEM, which this server does not read — so behind nginx either put a proxy that can produce Client-Cert in front of the server, or override features.mTLS.getCertificate to read nginx’s form. Whichever you choose, clear both headers on requests that carry no certificate, then switch the setting on.

Checking it worked

curl -sI https://auth.example.com/health
curl -s https://auth.example.com/.well-known/openid-configuration | jq -r .issuer

The second command must print the public https URL you set as ISSUER. If it prints an internal address or an http:// URL, fix that before anything else — every client will believe it.

To confirm the forwarded header is reaching the server, watch what happens when you exceed a limit. A 429 with Retry-After that arrives for your address alone, while other callers carry on, is the resolver working. No RateLimit-* headers are emitted, and that is intentional: they would tell a caller probing for the threshold exactly where it is.