How Do You Grade an Open-Ended Answer? How I Validated an AI Grader

One day, I casually asked AI a question: “Which model does the grader use to score the postmortem notes?” I almost wish I had not asked. It turned out that the grader and the postmortem agent were using the same model. Wasn’t that like letting the same person be both player and referee? Could I trust the resulting scores?

Before deciding how serious it was for the agent and the judge—the AI grader—to share a model, I first needed to understand the judge’s rubric. Only then could I tell how much it mattered that the grader was evaluating work produced by the same model. After all, even the same person would probably struggle to write exactly the same answer twice on an open-ended exam.

The Difficulty of Grading Open-Ended Answers

What would go wrong if I compared the note generated by the agent with the note in the golden dataset and used textual similarity as the score? A response could receive a low score even when the wording differed but the facts were the same. Another response could receive a higher score because the wording was similar, even though it introduced details that never appeared in the discussion.

Suppose the reference answer is, “Alex took yesterday off because of a cold and is back at work today,” and the agent produces one of the following:

• Alex was not feeling well yesterday and did not come into the office. He has returned to work today.
• Alex took yesterday off because of a cold and is back at work today. He handed his work over to Jamie yesterday.

From a human perspective, the first answer is the one that remains consistent with the facts. A simple similarity comparison, however, would probably favor the second—even though it invents a handoff that never happened.

To avoid this problem, I included the following instruction in the judge prompt:

Evaluate factual consistency only. Do not consider writing quality, tone, or length. If the note in the golden dataset is empty, it means a human reviewer determined that the discussion contained insufficient information to reach a conclusion.

The grader also receives more than just the golden dataset as a reference answer. It receives three inputs in total.

Three Inputs, Four Yes-or-No Questions

In addition to the note generated by the agent—the answer being graded—and the note in the golden dataset—the reference answer—the grader receives the original team discussion, which serves as the primary source of truth. The judge scores the candidate note using the discussion as its main evidence and the golden dataset as a secondary reference.

Simply handing over the three inputs and asking, “Is this note correct?” would still be too vague. Instead, I asked the judge to answer four yes-or-no questions:

• Q1: Is the problem described in the note consistent with the discussion?
• Q2: Is the root cause described in the note consistent with the root cause that was actually confirmed in the discussion? Treating a symptom as the root cause also counts as inconsistent—for example, writing “the page displayed an error” when the confirmed root cause was “the upstream system did not send the data.”
• Q3: Is the solution described in the note consistent with the discussion? If the discussion genuinely contains no conclusion and the note says that the information is insufficient, count it as consistent.
• Q4: Does the note contain any concrete detail that does not appear in the discussion, such as a person’s name, a number, a Pull Request ID, or a causal relationship? If so, answer yes. If the discussion contains no conclusion and the note says that the information is insufficient, that is a correct judgment.

The direction of the answers is not identical across all four questions. A “yes” on Q1 through Q3 means the note passes that check, while a “yes” on Q4 means the note contains something that does not exist in the discussion.

I also required the judge to quote the original discussion as evidence for every answer:

Every yes-or-no judgment must include supporting evidence. Quote the relevant sentence from the discussion. If the answer to Q4 is yes, list the content that appears in the candidate note without support. If no supporting sentence can be found, write, “This information does not appear in the discussion.”

The four binary questions and mandatory citations leave the judge as little room to improvise as possible. “Is the root cause correct?” is more likely to produce a consistent answer than “Is this well written?” Imagine two plates of mapo tofu in front of you. “Do these dishes use the same ingredients?” is more objective than “Does this one taste better than the other?”

This scoring method did, in fact, catch errors in the notes that would have been easy to overlook. In my second article, Building a Golden Dataset from 100 Production Issues, I mentioned a record whose note in the golden dataset said, “Root cause not confirmed.” During a later evaluation, the postmortem agent wrote a plausible-looking root cause for that issue. When the judge checked the discussion, it found that nobody had ever confirmed that explanation. The agent had filled in the missing root cause on its own, and the judge gave it a zero on Q2.

In another note, the agent wrote that an event happened “in the early hours of the morning.” The judge caught that too, because the discussion contained no specific time. “Time” was not explicitly listed among the examples in Q4, but the broader category of unsupported concrete details was enough to catch the hallucination.

Did the Note Scores Move Across the Three Versions?

In my previous article, Reviewing the Wrong Answers: How I Refined the Prompt for Follow-up Improvement Areas, I described three prompt versions and reported only the scores for follow-up improvement areas. I did not include postmortem note scores because the evaluation pipeline for the notes did not exist when I ran those first experiments. I built the note-grading mechanism about a week after completing the three prompt versions.

Fortunately, each prompt version and the golden dataset had been preserved in commit history. Once the judge was ready, I went back and tested whether the three prompt versions affected the postmortem note scores. The results barely moved across the four questions: the changes in Q2, which evaluated the root cause, and Q4, which detected hallucinations, were both within one record.

That was exactly what I hoped to see. All three versions changed the definitions of the follow-up improvement areas, while the prompt for the postmortem note remained untouched. The scores moved only after I changed the corresponding part of the prompt. It was not until a later version actually modified the postmortem note prompt that the note scores showed their first substantial change.

Q1 and Q3, however, were nearly perfect. When everyone gets a question right, does that mean the concept is thoroughly understood—or was the question simply too easy?

Self-Preference Bias, and How I Validated the Judge

Now that the grading method is clear, let’s return to the question from the beginning: the judge scoring the note used the same model as the agent that wrote it. Research has found that models can systematically prefer text generated by themselves. How could I address this self-preference bias?

The first safeguard was the one described above: four yes-or-no questions with mandatory citations, which constrained the judge’s freedom to improvise. The second was to manually review at least 20 issues. I would accept the judge’s scores only when its judgments agreed with mine more than 90% of the time. Otherwise, I would continue revising the judge prompt.

What Does Human Review Actually Check?

What did that manual review look like? In one issue, someone wrote in the discussion, “My guess is that the browser cache may have caused this.” The candidate note, however, said, “The root cause was the browser cache.” The judge gave Q2 a zero and cited the following reasoning: “This was speculation, not a confirmed root cause. The candidate note incorrectly turned a hypothesis into a definite conclusion.” Human review meant checking whether judgments and citations like this actually held up.

Earlier, I asked whether the perfect Q1 and Q3 scores were genuinely deserved or whether the questions were simply too easy. After the manual review, I concluded that the perfect scores were reasonable—at least for the first 27 issues.

Unfortunately, I did not preserve the actual number from this review. My work log said only, “Agreement threshold met; judge officially accepted,” without recording the number of matching judgments or the final percentage. I required the judge to provide evidence for every answer, yet my own review left behind only the word “met.” When I tried to cite the number while writing this article, there was nothing to cite. A reminder to my future self: preserve the exact figures next time.

More importantly, this manual review validated one specific combination of judge prompt and model. If either one changes, the review needs to be repeated.

Does that mean the self-preference problem was not so serious after all? For my current use case, it did not appear to be. But when I asked two AIs about the problem, they gave me opposite advice.

Two AIs Gave Me Opposite Advice

I asked both AIs the same question: “Should I switch the judge to a stronger, different model to eliminate the self-preference concern?”

AI A answered, “Switch. Using a different model can reduce concerns about self-preference bias.”

AI B answered, “Not yet,” for four reasons. First, all current comparisons used the same model with different prompts. With the judge and grading method held constant, the effect of the bias was likely to be similar across versions, so the setup could still support relative comparisons. Second, changing the judge model would reset the baseline and require another round of human review. Third, a stronger model would cost more and might face practical usage limits. Finally, the largest problem at this stage was noise, not bias.

Both answers were reasonable. The difference was that the two AIs were making different assumptions about my plan. If I was continuing to refine the prompt, AI B’s answer was more useful. If I eventually wanted to compare different models using the same prompt, then reducing self-preference bias—as AI A suggested—would become much more important.

At this stage, I was not trying to determine which model produced the best output, so I followed AI B’s advice and wrote this in my work log:

For now, the consistency of the judge matters more than its capability—until I need to compare models or cite absolute numbers.

Once again, AI provided the advice, but the final decision remained with a human. This echoed a line from my first article, Why I Let AI Make Suggestions, Not Decisions:

It is about how we introduced AI into the process without allowing it to become the final decision-maker.

More Important Than Self-Preference Bias: Noise

On the day I completed the note evaluation, I ran the v3 prompt with the judge and got a hallucination rate of 26.9%. The next day, while retrospectively running v1 through v3 to check whether changes to the follow-up improvement area prompt affected the note scores, the hallucination rate for v3 suddenly became 14.8%. The prompt was the same. The model was the same. The golden dataset was the same. I went to sleep and woke up to a difference of roughly 12 percentage points. I had assumed that the same prompt and model would produce similar scores. They did not.

When I discussed this with AI, it pointed out that the difference could come from randomness in the agent’s note generation, noise in the judge, or both. To isolate the judge’s behavior first, I asked it to evaluate the same batch of notes three times.

The surprising discovery was that the grader’s own scores moved up and down. The pattern became clearer after the golden dataset grew to 100 issues and I ran every evaluation three times. Across three evaluations of the same batch of notes, the overall hallucination rate ranged from 2.1% to 6.3%. The highest value was three times the lowest, even though the absolute difference was only 4.2 percentage points. The notes generated by the agent were nondeterministic, but even after I constrained the judge with yes-or-no questions, its scores still fluctuated.

Once the golden dataset reached 100 issues, I decided to run each evaluation three times—but the follow-up improvement areas and the notes were handled differently. For follow-up improvement areas, every label on every issue received its own majority vote, smoothing out fluctuations in the agent’s labeling. For postmortem notes, the pipeline kept only the note from the first successful run. The judge then evaluated that same note three times, with each of the four questions receiving its own majority vote. In other words, the note score smoothed out instability in the judge, but not randomness in the agent’s note generation. The current note metrics therefore represent only one sample from the agent. This pipeline does not yet measure how much the generated notes themselves fluctuate.

I also learned that when the sample size is small, a difference of one record can produce a noticeable swing in percentage points. In those cases, the raw count makes the practical difference easier to understand than the percentage alone. A change within one record can usually be treated as noise rather than something that needs to be optimized immediately.

This experience taught me that noise needed attention before self-preference bias. After all, if the ruler itself is shaking, there is little meaning in any length you measure with it.

The Five Coordinates Behind Every Evaluation Run

Every score is determined by five coordinates: the version of the golden dataset, the version of the postmortem agent prompt, the model used by the postmortem agent, the version of the judge prompt, and the model used by the judge.

To make a score change more attributable to a single adjustment, I try to change only one coordinate at a time and use repeated runs to reduce random noise. In the v1-through-v3 comparison from the previous article, only the prompt for follow-up improvement areas changed. The other coordinates remained fixed, so I could be more confident that score changes came from the prompt revision. For the note metrics, however, I still had to account for the limitation of measuring only one generated note per issue.

Imagine brewing pour-over coffee while changing the grind size, the filter, and the pouring technique all at once. How could you know which variable made the coffee taste better?

The position of all five coordinates therefore needs to be recorded for every evaluation run. Only then can I later answer the question, “What changed between these two scores?”

Two of my earliest three runs, however, did not match the prompt versions in the commit history. At the time, I ran the evaluation first and committed the prompt afterward, leaving room for the prompt used during the run to differ from the version that was eventually committed. That mistake led to a new rule: commit the prompt version before running the evaluation.

What Matters When Grading Open-Ended Answers

After this very long tour, here is the short version of what matters when grading postmortem notes:

• The goal is to determine whether the facts are correct, not whether the note resembles the golden dataset.
• A score from the AI grader is trustworthy only for a specific combination of judge prompt and model. If either changes, the combination needs to be validated again.
• Before worrying about bias, check whether the result is actually noise.

Now that the evaluation foundations for follow-up improvement areas and postmortem notes are finally in place, can I start seriously refining the prompt?

Next, I want to talk about a stubborn problem I encountered after that work began: one follow-up improvement area that refused to improve no matter how I changed the prompt. Its score was always a disaster.