Can an LLM Review My Own Pull Requests? (Sample)
A weekend experiment pointing a language model at my own diffs to see whether the feedback is useful or just confident.
- Python
- LLMs
- Git
Sample content. This entry is placeholder material included to demonstrate the layout, not a record of real work.
The question
If I feed a model my own diffs, does it find real problems, or does it produce plausible-sounding notes that would waste a reviewer's time?
Method
Take a set of commits where I already know what the bug was, strip the commit messages, and ask for a review of the diff alone. Score each response on whether it names the actual defect, not on whether it sounds reasonable.
What I want to learn
Mostly whether the useful signal survives when the model has no context beyond the diff. My guess is that it catches mechanical mistakes and misses everything that depends on knowing what the code is for.
Notes
The scoring is the hard part. Judging output by whether it feels insightful is exactly the trap this experiment is meant to avoid.