How Do You Grade an Open-Ended Answer? How I Validated an AI Grader
From factual consistency and self-preference bias to evaluation noise, how I designed and validated an AI grading system for postmortem notes.
Tag
engineering
Notes collected under the engineering tag.
Reviewing the Wrong Answers: How I Refined the Prompt for Follow-up Improvement Areas
An analysis of overclassification and missed labels in follow-up improvement areas, and how scoring and error attribution helped clarify the prompt's decision boundaries.
Building a Golden Dataset from 100 Production Issues
How I clarified the evaluation criteria for a Postmortem Agent through human review and AI-assisted consistency checks.
Why I Let AI Make Suggestions, Not Decisions
A note on the design tradeoffs behind an operations postmortem agent.