Reviewing the Wrong Answers: How I Refined the Prompt for Follow-up Improvement Areas

After a long stretch of preparation, it was finally time to run the evaluation.
On the first evaluation of the follow-up improvement areas, the exact-match score was only 14.81%. A record counted as an exact match only when the complete set of areas selected by the agent was identical to the golden dataset. Adding one extra label or missing a single label made the entire record incorrect.
Of the 27 issues in that run, only four matched exactly. My first thought was, “Is the prompt I gave the AI really that far from the criteria I had in mind?”
Once I took a step back, however, I realized that the exact-match score did not tell me very much. What should I change? Which improvement area was causing the problem?

In my previous article, Building a Golden Dataset from 100 Production Issues, I asked how we can tell what is wrong with the current prompt, what needs to be fixed, and what “good” even means. In this article, I will explain how I designed the scoring system, identified what needed to change, and improved exact match from 14.81% to 48.15%. After all, when we took tests in school, teachers expected us to review our incorrect answers. A score alone cannot tell us how to improve if we do not know what we got wrong.

A Layered Evaluation Architecture

First, let’s briefly revisit what the postmortem agent does.
The agent analyzes each historical issue and generates a set of follow-up improvement areas and a postmortem note. There are six possible follow-up improvement areas, while the postmortem note summarizes the retrospective in three sentences.

The evaluation system for the postmortem agent needs to handle two very different kinds of questions.
The follow-up improvement areas are a multiple-select question: the agent chooses one or more options from six categories. This is like filling in an answer sheet and feeding it into a scanner. For each issue, code can determine whether the complete answer is correct and whether each label was added unnecessarily or omitted. No LLM call is required to calculate the score.
A postmortem note, however, is an open-ended question. The same idea can be expressed in a hundred different ways, so string matching cannot tell us whether the agent’s output is equivalent to the reference answer in the golden dataset. Instead, another AI acts as the grader. It receives the complete team discussion and uses the golden dataset as a reference to check each part of the answer: Is the problem described correctly? Is the root cause accurate? Does the proposed solution match the facts? Has the agent invented anything that does not appear in the discussion?

Grading open-ended answers takes time, and this AI grader is expensive. Before asking it to read and score an answer, I use code as a teaching assistant to filter out responses that do not meet the required format. For example, before a note reaches the grading stage, code checks whether it is empty, contains more than the three sentences allowed by the prompt, or fails to acknowledge insufficient information when the corresponding issue is marked that way in the golden dataset. In any of these cases, the pipeline skips the judge stage and avoids calling the expensive grader—in other words, it saves tokens.

This follows the same cost-allocation principle as the rest of the system: deterministic work goes to code, like a teaching assistant; semantic judgment goes to AI, like a grader; and the most expensive decision—whether a label truly belongs—is left to a human.

How to Score a Multiple-Select Question

At the beginning of this article, I mentioned that the first evaluation produced an exact-match score of only 14.81%. That number did not reveal how each individual improvement area was performing, so I added two more metrics: macro precision and macro recall.

What Are P and R?

Macro precision and macro recall are based on precision and recall, respectively. The two metrics reflect different kinds of errors: precision primarily captures extra labels, while recall primarily captures missed labels.

• Precision: Of the labels the agent applied, how many were correct?
  • Suppose the agent assigns “Monitoring” to 10 issues, and six of those issues are also labeled “Monitoring” in the golden dataset. Precision is 6/10, or 60%.
  • The numerator is the number of issues for which both the agent and the golden dataset assign the category. The denominator is the number of issues to which the agent assigned it. Low precision means the agent tends to over-apply that category.
• Recall: Of the labels that should have been applied, how many did the agent identify?
  • Suppose eight issues in the golden dataset are labeled “Add to Playbook,” but the agent identifies only six of them. Recall is 6/8, or 75%.
  • The numerator is again the number of issues for which both the agent and the golden dataset assign the category. This time, the denominator is the number of issues assigned that category in the golden dataset. Low recall means the agent tends to miss that category.

These two metrics pull in opposite directions. They share the same numerator but use different denominators: precision divides by the number of issues labeled by the agent, while recall divides by the number labeled in the golden dataset. If the prompt defines an improvement area too broadly, the agent may assign it to almost every issue. Recall will then be very high, but precision will be low.
If the prompt defines an improvement area too conservatively, the agent may avoid assigning it. Precision will rise, but recall will fall.
In multiple-choice terms, the difference is roughly between “select it whenever it seems vaguely plausible” and “leave it unselected whenever there is any doubt.”

What Does “Macro” Mean?

There are six follow-up improvement areas in total. The macro score is the arithmetic mean of P or R across all six. For example, if the precision scores are 90%, 80%, 85%, 70%, 60%, and 20%, macro precision is the average of those six values, or approximately 67.5%.

The alternative is a micro average, which pools the results from all categories before calculating a single overall metric. For the postmortem agent, that would mean putting every correct, missed, and extra label into one pool and calculating P and R only once. Categories that account for most of the samples would therefore dominate the score.
With a macro average, every follow-up improvement area has equal weight, which matches the needs of this evaluation. Some categories, such as “Add Logging” and “User and Operational Experience,” are naturally rare, but they are no less important than the others.

The trade-off is that a drop in the score of one rare category can have a noticeable effect on macro precision and macro recall. The first evaluation contained only 27 issues, and a rare category might appear in just three or four of them. One mistake could therefore move the overall score significantly. This is why continuing to expand the golden dataset matters. As the title of the previous article suggests, I gradually increased the dataset to 100 issues while deliberately adding more examples of specific categories. This reduced the degree to which rare categories made the scores fluctuate.

Error Attribution: One Error, Two Possible Causes

After the first evaluation, I reviewed the issues the agent had mislabeled. Many were assigned “Product and Technical Improvements” even though the golden dataset did not include that label. The problem had already been fixed, and I did not think completed fixes should be classified as follow-up improvements.

I realized that an incorrect follow-up improvement label could have two different causes:

• Missing input: The agent never saw the part of the discussion that described the fix, so it assumed the problem was still unresolved. Technical problems in the discussion-fetching process can result in an incomplete input. In this case, the reading stage needs to be fixed—for example, by checking the discussion retrieval pipeline and verifying that the input is complete.
• Incorrect judgment: The agent saw the fix and even wrote “resolved” in the note, but still applied the label. The first version of the “Product and Technical Improvements” definition never said to omit the label once the problem had been fixed, so the agent did not know that rule. In this case, the judgment needs to be fixed by adding the labeling rule to the prompt.

Both failing to read the entire question and understanding the question but applying the wrong formula will cost points on a test, yet the remedies are completely different.
How did I distinguish between the two? At first, I used the most manual approach possible: I pulled out each mislabeled issue and compared it with the note generated by the agent. If the note said the problem had been fixed but the agent still assigned the label, it was a judgment error. If the problem had been fixed but the note did not mention it, the agent had missed part of the input.
This attribution process is not automated yet. At this stage, the golden dataset was still small enough that reviewing each record did not take much time. At a larger scale, however, key facts from each issue could be extracted into structured fields—whether the problem was resolved, whether the discussion explicitly named a follow-up improvement, and so on—so that these causes could be aggregated automatically.

The greatest value of an evaluation is not the score itself, but the attributable errors it reveals. A score tells you how often the system is wrong and whether the prompt is maintaining the same level of performance. Attribution tells you where it is wrong and how to fix it.

Three Versions in One Day

The first evaluation produced an exact-match score of only 14.81%. That same day, I officially began my prompt-refinement journey.

v1—Label Everything

Across the 27 issues in the first run, the agent missed seven labels and added 36 extra ones. The problem was obvious: the prompt was too permissive, so the agent tended to over-label every issue.
Looking back at the v1 prompt, each follow-up improvement area had only a one-sentence definition. For “Product and Technical Improvements,” v1 contained just this short description:

Problems with opportunities for systematic or structural improvement.

There was no additional guidance explaining when the agent should or should not apply each label.

v2—More Rules, but an Overcorrection

If v1 was too permissive, the obvious next step was to add rules. In v2, I introduced more explicit guidance. For “Product and Technical Improvements,” the definition became:

Problems with opportunities for systematic or structural improvement—for example, design flaws, technical debt, missing safeguards such as retries, fallbacks, or requeuing, and compatibility risks during dependency upgrades. This also includes improvements explicitly scheduled for follow-up in the discussion. It does not include one-off bugs that have already been fixed when the same problem cannot recur.

In addition to revising the definitions, I added general labeling principles:

• A label represents a direction still worth investing in. Do not apply one when the discussion confirms that the problem has been fixed and there is no remaining opportunity to prevent similar issues, or when the required mechanism—such as monitoring or logging—already exists.
• Prefer precision over coverage. Select only the most central options that have explicit support in the discussion, usually zero to two. Generic arguments that apply to most issues, such as “more logs always help with debugging” or “any investigation can become an SOP,” are not enough to justify a label.
• An empty list is a common and valid answer. Use no labels for cases such as a customer misunderstanding or mistake that is resolved through clarification, a one-off human error, or an isolated incident that has been fully eliminated with no follow-up work required.

The second evaluation was encouraging: exact match rose to 44.44%, and precision improved by nearly 22 percentage points. Recall, however, was lower than in v1.
When I investigated the decline in recall, I found that the “do not label resolved issues” rule was suppressing valid cases. One issue, for example, involved an external data specification. Although the data provider had corrected the immediate problem, it could still happen again unless we made a corresponding improvement. The agent applied the “resolved means no label” rule, causing recall to fall.

The second evaluation showed why no single metric is sufficient. Exact match and precision both improved, but recall declined. Looking at any one of these metrics in isolation would have pointed the prompt revision in the wrong direction.

v3—Fix the Boundaries, Not the General Rule

In v3, I left most category definitions unchanged and instead added explicit boundaries:

External Data Specifications: These problems cannot be fixed unilaterally by our team. External data anomalies must still be reported and corrected by the external party. Apply this label even when the immediate incident has been resolved, because the label captures the ongoing risk in the external data specification.

User and Operational Experience: Apply this label only when there is room for improvement on the system side, such as clearer guidance or guardrails against user error. Do not apply it when the case is simply a customer mistake or misunderstanding that is resolved through clarification and requires no system change.

All three metrics improved in the v3 evaluation. The increases may look substantial, but in terms of exact match, v3 correctly classified only one more issue than v2. Later repeated runs of the same version showed that a single run naturally fluctuates by two to four percentage points. So although the v3 scores looked better, it was difficult to determine the true size of the improvement. Still, both cases that v2 had incorrectly suppressed were fixed in v3, providing concrete evidence that refining the boundaries recovered those specific cases.

Metric	v1	v2	v3
exact match	14.81% (4/27)	44.44% (12/27)	48.15% (13/27)
macro precision	32.86%	54.37%	59.72%
macro recall	72.22%	65.48%	73.41%
missed / extra labels	7 / 36	8 / 11	6 / 11

Note: All scores in this article come from retrospective runs using the same version of the golden dataset under identical conditions. They differ slightly from the original scores I saw on the day of each experiment; rerunning them this way makes the three versions directly comparable.

Multiple-Choice Questions Have Answer Scanners. What About Open-Ended Questions?

After reaching v3, I concluded that the sample size was still too small. Continuing to tune the prompt for follow-up improvement areas was unlikely to produce meaningful gains, so I kept v3 and focused on adding more samples to the golden dataset. The real benefit of running the evaluation was not making the numbers look better; it was making prompt revisions evidence-based.
This article has focused mostly on the multiple-select question: evaluating follow-up improvement areas. But what about the open-ended question? I will save that for the next article, where I will discuss the grader’s rubric—and whether the grader itself can be trusted.