Putting AI in Charge of GitHub Issue Triage
Managing a GitHub repo is hard. New issues often arrive with vague titles or no steps to reproduce. You spend your day labeling, checking for duplicates, and asking for more details. This work eats the time you want to spend coding.
A 2024 survey showed that 60% of open-source maintainers have thought about quitting. Managing issues is a heavy burden.
I built a bot to handle the first pass. It runs in GitHub Actions and uses a language model. It performs these tasks:
- Applies labels based on your rules.
- Flags potential duplicates.
- Asks for missing details in thin descriptions.
The bot is an agent, not a rigid script. I give the model a set of tools. It decides which tools to use and in what order. A clean issue might need one tool. A messy issue might need three.
The first version was too chatty. It posted comments on every issue. This created noise. Noise makes people ignore bots.
I fixed this with one sentence in the system prompt. I gave the model permission to do nothing. I told it that silence is often the best answer. This changed its behavior more than any code change.
I used LiteLLM so the bot works with Claude, GPT, or Gemini. The bot relies on function calling to act on issues.
Real-world testing revealed three main challenges:
- GitHub security prevents bots from seeing secrets in pull requests.
- Prompt injection is a risk when you feed untrusted text into a model.
- Permissions and API errors can be confusing to debug.
I also made a rule for the bot: it only checks for security risks. It does not grade code quality. A bot judging design drives contributors away. Leave those decisions to humans.
If issue paperwork wears you down, this setup can help. It takes away the repetitive work and leaves the real decisions to you.
Source: https://dev.to/jescalada/putting-an-ai-agent-in-charge-of-github-issue-triage-3085
Optional learning community: https://t.me/GyaanSetuAi