I'm stress-testing a small service that sends roughly 1,000 requests per second through Cloudflared and then Traefik on k3s. The machine has two OCPUs and 12 GB of RAM, and the tunnel is configured with a 1,000-connection limit. Cloudflared and Traefik together consume more than 80% of the CPU, while the Rust API and SQLite workload use only about 20%. The application performs database queries, parsing, JSON serialization, routing, batching, and disk writes, yet it uses far less CPU than the two proxy layers. With around 100 workers, CPU usage can spike near 90%. I've checked the behavior with htop, btop, and perf, and perf suggests that a significant portion of Traefik's time is spent in system calls. Is this level of overhead expected for a double-proxy setup, or could a configuration issue be causing excessive CPU usage? A previous setup using Caddy for TLS and routing achieved over 10,000 requests per second on weaker hardware with lower resource usage and steadier latency.
3 Answers
At 1,000 requests per second, two virtual CPUs can become a bottleneck surprisingly quickly, especially when every request passes through two user-space proxies, a tunnel, connection handling, encryption, and container networking. The CPU percentage alone doesn’t prove something is broken, but the comparison with your earlier Caddy setup suggests you should benchmark each layer separately rather than treating the proxies as one component.
Run controlled tests with the API directly, through Traefik without the tunnel, and through Cloudflared without the extra proxy if possible. Keep TLS, compression, access logging, tracing, retries, middleware, and connection settings consistent. Also verify whether you’re measuring requests or connections, because short-lived connections create much more accept, handshake, and system-call overhead than keep-alive traffic.
The system-time result from perf is worth investigating. Check container and Kubernetes networking, conntrack pressure, MTU or fragmentation problems, TLS mode, logging, and whether the tunnel is opening too many connections or retransmitting traffic. Compare CPU per request and latency under long-lived keep-alive connections. If a direct Nginx or Caddy test is dramatically cheaper, that points toward the tunnel, Traefik middleware, or the networking path rather than the API or database.
That’s what I’m trying to isolate. The application can handle several thousand requests per second while using less than half a core, so the proxy overhead looks unusually high compared with the previous setup.