I'm running a KEDA-triggered Container Apps Job that reads one message at a time from a Service Bus queue. Each execution runs a GPU workload, and the Python worker writes a large amount of output to stdout—sometimes dozens of lines per second.
For some executions, the Log Analytics stream stops suddenly in the middle of processing. There's no exception, out-of-memory message, non-zero exit code, or timeout indication. It initially looks like the worker crashed, but the job execution is reported as Succeeded.
The generated output files confirm that processing continued: files from later stages, including stages that should have logged after the apparent cutoff, are present in blob storage. I've also ruled out an OOM kill and the job timeout because both have different, identifiable symptoms in this workload.
What could cause the logs to stop arriving while the job continues successfully, and what's the best way to diagnose or work around it?
3 Answers
Application Insights is another option worth testing for this workload. If the same execution produces complete telemetry there, it provides a more reliable record than depending solely on the container stdout stream. I’d also add explicit stage-start and stage-complete events, along with execution IDs, so missing log sections can be compared against the files written to storage.
The volume of stdout might be contributing to the problem. Container logging pipelines can behave poorly when an application emits a very large number of lines, even if the process itself is healthy. Try reducing the log rate, batching progress messages, or moving high-volume diagnostic output to files or a dedicated telemetry sink. Keep concise milestones and errors in stdout so the platform logs remain useful.
Since the execution succeeds and the actual output proves the worker kept running, this sounds more like a logging or ingestion problem than an application failure. As a test, send the container logs to another destination such as a storage account or Event Hubs. If those logs are complete while Log Analytics still has gaps, you’ve isolated the issue to the Log Analytics ingestion path. It’s also worth checking system logs and confirming that the application really continues writing to stdout after the cutoff.
The application does need to produce a lot of detail, so reducing the output may not be practical. I’ll test another destination and consider routing the verbose diagnostics away from the standard container log stream.