Hey everyone! I've been tasked with implementing observability for my company, and right now I'm focusing on AWS Lambda functions. I'm new to this, so please bear with me.
Here's what I'm trying to accomplish:
- Push logging, metrics, and traces from an AWS Python Lambda function to my LGTM Grafana setup, as detailed in [this guide](https://grafana.com/docs/opentelemetry/docker-lgtm/).
- For now, I'd like to avoid manual instrumentation and just apply auto instrumentation to our existing Lambda function as a proof of concept. Developers can add manual instrumentation later if needed.
Here's what I've done so far:
1. The AWS native services like X-Ray and CloudWatch work right out of the box.
2. I've set up the ADOT Lambda layer for Python.
3. Following AI suggestions, I created a simple function that successfully works locally when I run `opentelemetry-instrument python test_telemetry.py`. It sends data straight to the OpenTelemetry collector in my LGTM stack.
I also configured an AWS Lambda function with the ADOT layer and necessary environment variables:
- AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-instrument
- OPENTELEMETRY_COLLECTOR_CONFIG_URI: /var/task/collector.yaml
- OTEL_PYTHON_DISABLED_INSTRUMENTATIONS: none
- OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED: true
- OTEL_LOG_LEVEL: debug
In my collector.yaml file, I've set up protocols and endpoints for traces, metrics, and logs, but I'm not seeing anything come through. I've ensured the network settings for my LGTM stack are open to the public internet with no authentication issues.
Does anyone have experience with this setup? How should I proceed from here?
1 Answer
First, I would check the internet connectivity of your Lambda function to make sure it's connecting properly. If that's fine, try deploying your `collector.yaml` again. Here's a revised version that you might want to test out:
```
receivers:
otlp:
protocols:
grpc:
endpoint: localhost:4317
http:
endpoint: localhost:4318
processors:
batch:
exporters:
debug:
verbosity: detailed
otlphttp:
endpoint: "http://3.106.242.96:4318"
compression: gzip
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [debug, otlphttp]
metrics:
receivers: [otlp]
processors: [batch]
exporters: [debug, otlphttp]
logs:
receivers: [otlp]
processors: [batch]
exporters: [debug, otlphttp]
telemetry:
logs:
level: debug
```
This should help you get some debug logs working to troubleshoot the issue.
Yeah, the internet connection seems fine since my Lambda can make other requests. I tried your revised setup, but still no debug logs were generated. I suspect the `collector.yaml` isn't doing anything.