Hey everyone, I'm using OpenTelemetry's auto-instrumentors to monitor my Python application. They successfully instrument certain frameworks like FastAPI and OpenAI, generating spans for specific events like requests and API calls, with attributes giving context on inputs and outputs. My question is, why isn't there a Python auto-instrumentor that creates spans for each function, adding arguments and return values as attributes? Is it just a bad idea, or is it technically unfeasible? Also, I've coded my own auto-instrumentor that creates spans for functions in user code – check it out in my repo!
6 Answers
Instrumenting every function creates noise and inconsistent attribute names, which isn't very useful for debugging or monitoring. Auto instrumentation is meant to cover common functions, and then you can go deeper with your own important spans as needed. Also, OpenTelemetry is developing a profiling signal, which might give you the detailed insights you're looking for!
There are some commercial solutions that do this, but they aren't free. For example, Pydantic Logfire works great up to a certain limit before costs kick in. But if you're exploring options, just know those comprehensive solutions come with a price.
Any chance you can share a few examples? I'm curious about what's out there!
Pydantic Logfire sounds interesting; I'll have to check that out.
Creating spans for every function might sound useful, but it could lead to a ton of overwhelming trace data that doesn't really inform you much. The existing auto-instrumentors focus on meaningful parts of your apps, like handling requests, and that makes them much more effective. Instead of covering every single function, try instrumenting significant areas of your application, like file operations or processing steps, to get more useful insights.
I get that, but having spans for every function could help identify bottlenecks in data processing or file management. It seems valuable for understanding overall execution time.
You're right, it could add noise, but wouldn't it give a full picture if done right?
Well, maybe it's because the tools that are available already fulfill the needs without overwhelming the system. Developers usually wrap specific functions manually with decorators that they care about. The quality of these integrations matters, so perhaps just focusing on the key functions is sufficient!
Yeah, custom integrations provide more meaningful data than blanket measures would.
That makes sense, having tailored integrations could lead to better overall performance and insights.
If your goal is to understand where time is spent in your code, using a profiler might be a good option. It’s less noisy and can be effective for spotting performance bottlenecks, especially if used on replicas instead of in production. Writing middleware to publish profiling info isn't too complex, but just keep in mind the performance hit it might cause.
OpenTelemetry is still rolling out its core features, focusing on metrics and traces first. They might revisit the idea of broader auto instrumentation, but for now, they recommend using specific libraries for each framework. You can also look into Kubernetes-based solutions for some server containers, but those are quite enterprise-focused and don't cover all languages.
I didn't know about the separate profiling signal! That could be a game changer, thanks for the heads up!