your rate limiter trusts x-forwarded-for. mine did too, until it was measured.

last time i told you about the elo race that ate 79% of every vote — the anchor finding from a pre-launch hardening pass on a little pairwise-voting side project. i promised, at the bottom of that post, that the other critical finding was a vote rate-limiter that a rotated header walked straight through. this is that post.

here’s the setup. the side project is a nougat-ranking toy: you get shown two flavours, you pick one, an elo rating moves, a leaderboard falls out. because the leaderboard is the entire point, the interesting attack is obvious — cast a pile of votes for your favourite and bend the ranking. so i’d put a rate limiter in front of it. two, actually. i felt responsible.

then, during the audit, i sat down and attacked them instead of admiring them. both fell over. one to a header i let the open internet write. one to a cookie i simply didn’t send. neither took more than a few lines of curl.

a limiter you have never fired a spoofed header at is not a limiter. it’s a comment.

the setup: two limiters, both felt responsible

the front door is a blanket per-ip limit. every request through the whole app passes it — a hundred a minute, keyed by ip:

internal/http/server.go

// Rate limiting middleware (100 requests per minute per IP)
r.Use(httprate.Limit(
	100,                                     // requests
	1*time.Minute,                           // per duration
	httprate.WithKeyFuncs(httprate.KeyByIP), // per IP address
	httprate.WithLimitHandler(func(w http.ResponseWriter, r *http.Request) {
		logger.Warn("[HTTP Server] Rate limit exceeded for IP: %s", r.RemoteAddr)
		http.Error(w, "Rate limit exceeded. Please try again later.", http.StatusTooManyRequests)
	}),
))

httprate ↗ is go-chi’s sliding-window limiter, and KeyByIP does what it says: it derives the bucket key from the request’s ip. behind it, scoped to just the vote routes, sits a second, tighter limiter keyed on who you are rather than where you are — twenty votes a minute per user:

internal/http/server.go

voteRateLimiter := httprate.Limit(
	20,
	1*time.Minute,
	httprate.WithKeyFuncs(func(r *http.Request) (string, error) {
		return GetUserIDFromContext(r.Context()), nil
	}),
	// ... limit handler ...
)

internal/http/server.go

r.With(voteRateLimiter).Post("/pairings/{id}/vote", srv.handler.result)

two layers. the outer one caps raw request volume from an address; the inner one caps how fast a single identity can vote. defence in depth, on paper. the trouble is that both keys are things the client gets to choose, and i hadn’t noticed.

bypass #1: the ip your limiter believes

KeyByIP keys off r.RemoteAddr. so the only question that matters is: who sets RemoteAddr? on a raw tcp connection it’s the real peer and you can’t lie about it. but the app runs behind a reverse proxy, so the real client ip arrives in a header, and something has to promote that header into RemoteAddr. that something was this, fourth line down the chain:

internal/http/server.go

r.Use(middleware.Logger)
r.Use(middleware.Recoverer)
r.Use(middleware.RequestID)
r.Use(middleware.RealIP)
r.Use(middleware.URLFormat)

chi’s middleware.RealIP ↗ reads X-Forwarded-For, splits on commas, and overwrites r.RemoteAddr with the left-most entry. then KeyByIP, a few lines later, dutifully buckets on that value. the entire per-ip limiter now keys on a header i copied straight off the wire.

and the left-most X-Forwarded-For entry is the one value in that header a client fully controls. proxies append right-to-left — each hop tacks the peer it saw onto the end — so the right-most entries are written by infrastructure you own and the left-most is whatever the original caller claimed. MDN is blunt about it ↗ : “any security-related use of X-Forwarded-For (such as for rate limiting or IP-based access control) must only use IP addresses added by a trusted proxy. using untrustworthy values can result in rate-limiter avoidance, access-control bypass, memory exhaustion, or other negative security or availability consequences.” the standardized replacement, RFC 7239’s Forwarded header ↗ , carries the same warning in its own words — the forwarding information “cannot be relied upon to be correct, as it may be modified, whether mistakenly or for malicious reasons, by every node on the way to the server, including the client making the request.” OWASP files the whole thing under IP spoofing via http headers ↗ . i had done the exact thing three separate docs tell you not to.

measuring it took one loop. hold the header fixed and the limiter works — a hundred requests pass, then 429. rotate it, one made-up ip per request, and:

test/e2e/test_security.sh

_blocked=0
for _i in $(seq 1 140); do
	_c=$(curl -s -o /dev/null -w '%{http_code}' \
		-H "X-Forwarded-For: 10.10.$((_i/256)).$((_i%256))" \
		"http://localhost:${_sec_port}/api/campaign/countdown")
	[[ "$_c" == "429" ]] && _blocked=$((_blocked+1))
done

against the original code: 140 requests, 140 × 200, zero 429. every request was a brand-new ip as far as the limiter could tell, so every request got its own fresh bucket of one. “100/min per ip” was really “100/min per value the attacker picks,” which is no limit at all. a three-character increment in a shell loop walked straight through it.

bypass #2: no cookie, no limit

the per-user limiter looked sturdier — it keys on a user id, not an ip, and the user id comes from a cookie the server sets. except: what happens the first time you show up, before you have a cookie? you have to get one somehow. here’s the middleware that hands them out:

internal/http/middleware.go

// Try to get existing user ID from cookie
cookie, err := r.Cookie(cookieName)
if err == nil && cookie.Value != "" {
	// ... validate the user exists in the DB, use it ...
}

// Create new user if no valid userId found
if userId == "" {
	userId = uuid.NewString()

	newUser := &domain.User{
		Id:        userId,
		VoteCount: 0,
	}

	createdUser, err := h.userRepo.Create(r.Context(), newUser)
	// ...
	userId = createdUser.Id
	logger.Info("[UserMiddleware] Created new user: %s", userId)
}

no cookie means userId == "" means mint a brand-new user, right now, with a fresh uuid. that’s the correct, friendly behaviour for a real first-time visitor — no signup, you just start voting. but read it as an attacker. this middleware runs globally, before the vote limiter. so a request that sends no cookie reaches the per-user limiter already carrying a user id that was invented microseconds ago and has cast exactly zero votes. its bucket is empty. it always will be, because the next cookieless request gets a different fresh id with its own empty bucket.

so the per-user limiter’s key, like the per-ip one’s, is attacker-chosen — you choose it by choosing to send nothing. i fired fifty concurrent votes with no cookie. fifty × 200. the limiter keyed on identity never saw the same identity twice.

these two are independent, which is the part that stings. even if you’d hardened one — pin the ip, say — the other still walks. and combined they’re worse than either: rotate the forwarded header and drop the cookie and you are, from the server’s point of view, an infinite supply of brand-new, single-request strangers, each one perfectly within every limit. for a leaderboard whose entire value is that the numbers mean something, that’s the whole ballgame.

the fix: resolve the ip you’re allowed to trust

the bug in #1 isn’t “we read X-Forwarded-For.” behind a proxy you have to. the bug is reading it from everyone. the fix is to honor forwarding headers only when the immediate tcp peer is a proxy we actually control, and to take the client from the part of the header that peer wrote — not the part the client did.

that means replacing middleware.RealIP with a resolver that knows which peers are trusted:

internal/http/middleware.go

func (t *trustedProxyResolver) clientIP(r *http.Request) string {
	host := r.RemoteAddr
	if h, _, err := net.SplitHostPort(r.RemoteAddr); err == nil {
		host = h
	}
	peer := net.ParseIP(host)
	if peer == nil || !t.isTrusted(peer) {
		return host
	}
	if xff := r.Header.Get("X-Forwarded-For"); xff != "" {
		parts := strings.Split(xff, ",")
		for i := len(parts) - 1; i >= 0; i-- {
			p := strings.TrimSpace(parts[i])
			ip := net.ParseIP(p)
			if ip == nil || t.isTrusted(ip) {
				continue
			}
			return p
		}
	}
	if xr := strings.TrimSpace(r.Header.Get("X-Real-IP")); xr != "" && net.ParseIP(xr) != nil {
		return xr
	}
	return host
}

two rules do all the work. one: if the direct peer isn’t in the trusted set, ignore forwarding headers entirely and key on the real tcp address — a public client can’t spoof that, it’s the socket. two: if the peer is trusted, walk X-Forwarded-For from the right and return the first entry that isn’t itself a trusted proxy. the right-most entries were written by my own proxies; the first non-trusted one going leftward is the client as my outermost proxy actually saw it. anything further left is the client’s own claim, and i never reach it. it’s exactly the right-to-left, skip-trusted walk the newer guidance settled on — httprate has since deprecated its own KeyByRealIP ↗ for precisely this class of bug, noting “there is no safe default IP source, so you state your trust model explicitly.”

“trusted” is a config value — a list of cidrs, defaulting to loopback and the private ranges, which is right for a proxy sitting on the same host or private network:

internal/config/config.go

var DefaultTrustedProxies = []string{
	"127.0.0.0/8",    // IPv4 loopback
	"::1/128",        // IPv6 loopback
	"10.0.0.0/8",     // RFC1918 private
	"172.16.0.0/12",  // RFC1918 private
	"192.168.0.0/16", // RFC1918 private
	"169.254.0.0/16", // IPv4 link-local
	"fc00::/7",       // IPv6 unique-local
	"fe80::/10",      // IPv6 link-local
}

the resolver mutates r.RemoteAddr just like RealIP did, so nothing downstream changes — KeyByIP still keys on RemoteAddr, it just finally keys on a value the client can’t forge. wire it in ahead of the limiter and the chain looks almost identical:

internal/http/server.go

realIP := newTrustedProxyResolver(srv.trustedProxies)

r.Use(middleware.Logger)
r.Use(middleware.Recoverer)
r.Use(middleware.RequestID)
r.Use(realIP.middleware)
r.Use(middleware.URLFormat)

same rotation test, now against an untrusted peer: the spoofed X-Forwarded-For is ignored, every request keys on the real socket, and the loop trips 429 right on schedule. bypass #1 closed — and because the now-unspoofable per-ip cap sits in front of everything, it also backstops bypass #2: the cookieless-stranger trick still mints fresh identities, but they all now share one very real ip, and that ip runs out of its hundred a minute.

the honest part: what this fix is and isn’t

it’s a real fix, and it moved the two measured bypasses from “trivial” to “closed.” but there are edges worth naming out loud.

the allowlist is a footgun with two triggers. the whole scheme rests on that cidr list being exactly right for the deployment. list too much — trust a range your proxy doesn’t actually own, or leave a permissive default in front of a topology that changed — and you’re back to honoring forged headers: fail-open, the bypass reopens silently. list too little — forget the real proxy’s subnet — and every genuine client collapses onto the proxy’s own ip, one shared bucket, and legitimate users start eating each other’s 429s: fail-closed, a self-inflicted outage. the code fails safe on a bad entry (an unparseable cidr is logged and skipped rather than crashing startup, and an unknown peer degrades to the real socket), but it cannot save you from a plausible-but-wrong list. this is config you have to get right per environment, and no amount of go makes that not true.

closing #2 for real is a product decision i punted. the per-ip cap now bounds the cookieless-stranger abuse, but it doesn’t end it — an attacker with a spread of real ips still gets a fresh identity per request. and there’s a quieter cost i left standing: every one of those cookieless requests still writes a brand-new user row. that’s fine for real first-time visitors and grim under a spray of cookieless traffic — an unauthenticated, unbounded way to grow my database, one insert per request, no vote required. the honest closures are the ones that don’t rely on the client’s key at all: dedup votes in the database with a real uniqueness constraint, or defer minting an identity until someone actually does something, or put a cheap challenge in front of the first vote. i wrote down that i’m trading a slightly abusable limiter today for not reworking the identity model the week before launch. that’s a defensible trade only because i measured the exposure instead of guessing at it.

lessons learned

• a limiter you haven’t attacked is decoration. both of mine returned tidy 429s in the happy path and looked, in review, exactly like protection. the entire distance between “looks like a limiter” and “is a limiter” is one curl loop with a rotating header and one with no cookie. run it — against your own staging, before someone else runs it against prod.
• every rate-limit key is an input; treat it like one. the bucket key is chosen by the caller unless you can prove otherwise — an ip is forgeable through a forwarding header, a “user id” is forgeable by not having one yet. ask of any key: what stops the client from picking a fresh value per request? if the answer isn’t “the tcp socket” or “a signature,” you don’t have a limit, you have a suggestion.
• only trust the parts of a forwarding header your own infra wrote. the client owns the left of X-Forwarded-For; you own the right. take the right-most entry that isn’t one of your proxies, gate the whole thing behind a trusted-peer check, and don’t read the header at all from a peer you don’t recognise.
• fail safe, and make the safe direction the boring one. unparseable cidr → skip it. unknown peer → use the real socket. no configured proxies → trust nothing. the defaults should make a misconfiguration quiet and closed, not quiet and open.

next time, staying in the same audit: a public stats page that ran three full-table scans on every request and fell over the first time i pointed real concurrency at it — the difference between a page that’s merely slow and a page that’s a denial-of-service you host yourself.

the two bypasses took an afternoon to find, because i finally sat down and measured instead of admiring the middleware chain. the MDN notes on X-Forwarded-For ↗ say everything i learned the slow way, and i should’ve read them before i wrote the limiter, not after i broke it.