I'm tackling a pretty exciting automation challenge and wanted to gather some insights from the community.
**The Goal:** I'm trying to set up a system that automatically resolves certain production errors reported through error trackers like Airbrake, Sentry, or Rollbar without any human intervention. Here's the flow I envision:
1. An error happens in production.
2. Airbrake logs this and creates a GitHub Issue with the error details, like the error message, the file path (e.g., app/models/some_model.rb:45), and the backtrace.
3. A GitHub Action kicks off upon the creation of a new issue labeled 'exception'.
4. This Action grabs the issue content, identifies the error, and extracts what file and line are involved.
5. The Action then sends this context to an AI tool (currently using Aider Chat powered by Claude 3.7 Sonnet).
6. The AI figures out the necessary code alterations to address the specific error.
7. Finally, the Action commits the changes in a new branch and opens a Pull Request for review.
**The Challenge:** I'm struggling with refining this into a completely no-human-involvement setup (except for the PR review). The major issue is ensuring the AI receives just the right amount of context and detail necessary for it to make accurate, minimal changes based solely on the error log and issue description.
**What I've Tried:**
- Setting up a GitHub Actions workflow to trigger on labeled issues.
- Parsing the issue's content in the Action to get error specifics.
- Creating a comprehensive prompt for Aider/Claude that includes the error info and guidelines.
- Using git ls-files in the Action to provide the AI with a list of relevant files for context, which actually helped.
- Applying Aider's --no-web-browse flag to keep the AI focused.
**Current Challenges:**
1. Even though I've crafted a detailed PROMPT.txt file containing all backtrace information, Aider + Claude still struggle to modify the right file accurately.
2. I noticed that if I use an identical prompt in other AI tools like Cline, it manages to fix the problem.
I'd appreciate any strategies, tool recommendations, prompt engineering tips, or general feedback on my approach. How would you approach building a system like this?
1 Answer
This sounds like a really cool project! It might be a context issue with your AI not having enough information when it tries to make the changes. Have you thought about third-party services like Context7 or Repomix? They might help clarify the context for the AI. Also, I think you might be a little early in trying to eliminate human involvement completely; maybe a hybrid approach would be more effective to start?
I hear you! Just adding detected files might not cover all bases since there could be other dependencies involved in fixing the error. This is definitely more challenging than I thought it would be. But thanks for the suggestions, I’ll check them out!