How can I monitor container disk usage when no ephemeral-storage limit is set?
Asked by
canvas__primer
We have Windows containers running on AKS with containerd, and the pods currently define CPU and memory resources but no ephemeral-storage requests or limits. Our monitoring reports fsUsedBytes, which support explained is the host-side writable layer plus container logs—not simply the free space shown inside the container with PowerShell. The reported fsCapacityBytes also appears to represent the node's disk capacity rather than a per-container limit. In production, these pods become unresponsive and restart when the reported usage reaches about 20 GB, even though Get-PSDrive still shows 8–9 GB free inside the container. Is there a reliable way to calculate a meaningful usage percentage without an explicit ephemeral-storage limit? Should we add ephemeral-storage requests and limits, and are there other safeguards or metrics we should use for the immediate disk-pressure issue?
Answers
The 20 GB behavior on Windows is likely a container scratch-disk or sandbox size rather than the free space reported by the guest filesystem. The host-side metric includes the writable layer and logs, while Get-PSDrive shows the container’s own volume view, so the numbers are not expected to match. Ask the platform team to confirm the Windows container storage configuration and whether the scratch size can be overridden, for example through the relevant container runtime storage settings.
Reply to this answer
Use the node agent’s summary metrics to separate each container’s root filesystem usage from its log usage, such as rootfs.usedBytes and logs.usedBytes. Also monitor nodefs.available and imagefs.available, with alerts before the kubelet’s eviction thresholds. A growth-rate alert on fsUsedBytes can help identify a runaway workload before node-wide disk pressure causes evictions or affects neighboring pods.
Reply to this answer
Without a configured limit, there isn’t a trustworthy per-container percentage. The container can consume space from the node’s shared filesystem, while other pods and images consume it too. Treat the node’s available space and filesystem-pressure thresholds as the real capacity indicators, and alert on absolute container usage or rapid growth instead of inventing a per-container denominator. Adding ephemeral-storage requests and limits is the cleanest way to establish an expected ceiling and prevent one workload from consuming the node’s storage.
Reply to this answer
For a lasting fix, define ephemeral-storage requests and limits directly or use a policy such as a LimitRange to provide defaults. Then alert against the configured limit while continuing to monitor node-level disk pressure. The limit should be sized for the application’s writable data and logs; it is not a replacement for log rotation or node filesystem alerts.
Reply to this answer
Add your answer
Answers are reviewed before publication.
MAYBE__LATER: This also helps distinguish a growing writable layer from stdout/stderr logs. Log rotation is worth configuring regardless, since excessive container logs can consume the same local storage.