I'm running a KEDA-triggered Container Apps Job from a Service Bus queue, with one execution per message. Each execution runs a GPU-heavy Python worker that writes dozens of log lines per second to stdout.
Some executions have logs that suddenly stop in the middle of processing. There's no exception, OOM message, non-zero exit code, or timeout indication. At first it looks like the worker crashed, but the output files from later processing stages are present in Blob Storage, even though those stages would have logged after the point where the stream went silent. The execution status also reports Succeeded.
I've ruled out an OOM kill and the job timeout. Has anyone encountered Container Apps logs being truncated while the job itself continues and finishes successfully? What's the best way to diagnose or work around this?
3 Answers
The logging volume may be part of the problem. Container Apps log collection can be unreliable when an execution produces a very high rate of stdout messages, even if the workload itself completes normally. Try reducing the per-stage verbosity, batching progress messages, or writing important milestones and errors to a durable sink instead of treating stdout as the authoritative record.
For critical diagnostics, include stage start and completion markers in Blob Storage or another external telemetry system.
A useful first test is to send the same application logs to another destination, such as a Storage Account or Event Hub, instead of relying only on Log Analytics. If the alternate destination contains the complete output while Log Analytics still has a gap, that points to an ingestion, buffering, or throttling issue in the Log Analytics path rather than the job or Python process.
Also compare the application log stream with the system logs and verify that the worker continues writing to stdout after the apparent cutoff.
Application Insights is another practical alternative for job telemetry. Sending structured logs there can make it easier to tell whether the application stopped logging or whether only the Container Apps-to-Log Analytics pipeline lost records. It also gives you a separate place to check timestamps, stage completion, and exceptions.
If the alternate sinks show the full run but the built-in logs remain incomplete, collect an execution ID, timestamps, system logs, and a minimal reproduction, then open an Azure support case because it is likely an observability pipeline limitation or service issue rather than a failed job.
The application does need fairly verbose logging, so reducing it significantly may not be practical. I’ll compare an external log destination and Application Insights with the Log Analytics output first, then escalate it with the execution details if only the built-in stream is affected.