The Prompt Wasn't the Problem. The Answer Wasn't in the Input

In the previous articles, I explained how I evaluated the follow-up improvement areas and postmortem notes. With the evaluation system finally in place, I could begin the real work of revising the prompt. I thought the hardest part was behind me and everything would be smoother from there. It was not. One follow-up improvement area refused to improve, no matter how I rewrote its prompt. Was the label definition too vague? Or did the agent need more information?

I tried five different approaches, switched models, and even replaced the architecture, but none of them solved the underlying problem.
Eventually, I realized that the prompt itself might not be the problem. The agent answered some questions well because the answers were present in the text it could read. It struggled with others because the answers depended on team context it had never seen.
Before getting into what I could not teach the agent, however, let me start with something I could.

What I Could Fix

In the third article, Reviewing the Wrong Answers: How I Refined the Prompt for Follow-up Improvement Areas, I described the first three versions of the follow-up improvement area prompt.
I continued revising that part of the prompt afterward, but throughout those iterations, I did not change the postmortem note instructions at all. When progress on the labels stalled, I decided to work on the note instead.

In the fourth article, How Do You Grade an Open-Ended Answer? How I Validated an AI Grader, the note scores already looked fairly good. Why revise them?
The notes were usually acceptable, but they would occasionally hallucinate details or present an unconfirmed claim as established fact. Once the number of issues exceeds 1,000, even rare failures accumulate into a meaningful quality problem. My goal was simple: make the notes more honest.
Across 55 records and three runs, root-cause accuracy increased from 87.3% to 94.4%. The hallucination rate—for example, inventing a time of day that never appeared in the discussion—fell from 10.9% to 7.4%. Dangerous hallucinations, such as fabricated root causes, fell to zero.

During this work, however, something even stranger happened. I had changed only the note instructions, yet recall for the follow-up improvement areas suddenly collapsed.

Isolating the Cause

At first, I wondered whether repeated revisions had created some unexpected interaction between the label prompt and the note prompt.
To isolate the variable, I restored the label section until everything except one playbook instruction matched v3. I left the note prompt unchanged. Even with the label section almost identical to v3, its score was still far below the original v3 result.

The remaining suspect was the more cautious guidance I had added to make the notes honest:

Insufficient-information rule: “If the root cause is only a hypothesis, no cause was found before the customer was asked to handle it themselves, the issue recovered on its own without follow-up, or the investigation simply ended without a conclusion, write only: ‘Insufficient information for a complete analysis.’”
Trigger-condition trap: “Someone identifying a trigger and the person involved confirming it does not mean the root cause has been confirmed.”
Grounding requirement: “Pull Request IDs, module or command names, error messages, and key numbers must appear verbatim in the thread.”

I suspected that this section was affecting label recall, so I added a firewall immediately after it:

Label decisions are independent of the note: The caution rules above—requiring a confirmed root cause and explicitly stating when information is insufficient—apply only to the postmortem note. They do not apply to labels. Even when the note says “insufficient information,” evaluate the labels according to the definitions below. Labels are based on reasonable signals in the thread and do not require the same level of evidence as a confirmed root cause.

And just like that, recall returned.
Even though the prompt clearly separated follow-up improvement areas from the postmortem note, the agent had allowed the caution rules for the note to affect the other section.
At the same time, label exact match declined from 41.8% to 38.2%. The difference was close to the range of random variation, so I could not determine whether the prompt change had caused it.
After the revision, however, the notes themselves were more trustworthy. I also learned that this kind of honesty could be taught. Whether the root cause had been confirmed and whether a Pull Request ID appeared were facts already present in the discussion.

What I Could Not Fix

With the successful note revision out of the way, it was time to return to the part I could not fix.
One follow-up improvement area was called “Product and Technical Improvements.” Applying the label depended on one question: Had the problem already been fixed, or had someone taken ownership of it? If someone was already handling it, the label should not be applied. It belonged only on improvements that remained unresolved and unowned. This was the only one of the six areas that depended entirely on that distinction, and it was responsible for most of the extra labels: 32 of 49.
For the first attempt, I wrote the rule out in prose and told the agent not to apply the label when the work was complete or already owned.

…and the improvement must still be unresolved and unowned at the end of the thread. Two common cases where the label should not be applied are: (1) the fix was completed within the thread—a hotfix or Pull Request was merged and deployed, data was corrected, or a configuration was changed—and no remaining improvement was raised; a completed fix is not a follow-up; or (2) someone has taken ownership of the improvement—it was explicitly assigned and accepted, a Pull Request was opened, or a separate tracking ticket was created. Apply the label only when an improvement has been identified but remains unowned.

The result was a loss on both fronts. Recall for “Product and Technical Improvements” fell from 76% to 40%. Worse, the prose rule also affected “Monitoring,” whose recall fell from 86% to 43% even though I had not changed its definition. I had tried to teach the agent to hold back in one category, and it became hesitant across the entire answer sheet.

For the second attempt, I made the rule more precise and added a safeguard in the opposite direction:

Note: Completion of the primary fix does not automatically mean the label should be omitted. If the thread identifies a separate structural improvement that remains unowned, apply the label.

I added this rule because some issues required an urgent workaround before the underlying problem could be addressed properly. Part of the dramatic recall drop in the first attempt came from these cases: the immediate incident had been handled, but follow-up work remained.
The second result was a partial win. “Product and Technical Improvements” recovered, but the instruction to hold back once something was fixed still leaked into other categories.

For the third attempt, I stopped explaining the logic and gave the agent four examples instead. Two of them were:

The fix Pull Request has been merged and deployed, the result has been confirmed, the issue is closed, and no further improvement was raised → Do not apply the label; a completed fix is not a follow-up.
The primary fix is complete, but someone identified a separate structural problem that no one has taken ownership of → Apply the label.

This time, even “Add Logging” was dragged down.

At the beginning of the article, I said I had made five attempts. Three of them targeted “Product and Technical Improvements.” The other two tried to teach the agent when not to apply “Add to Playbook.”
The model clearly understood what to look for: recall rose from 30% to 80%. But the number of extra labels also jumped from one to 61.
In the next version, I added three brakes and tested 15 issues. Of the eight that had been incorrectly labeled in the previous version, seven were still labeled incorrectly.
It saw the brakes, but it did not use them.

The five revisions appeared to teach different rules, but they all failed for the same reason: the model could not hold back selectively. Sometimes the suppression leaked into other labels. Other times, the brakes were present in the prompt but went unused.

What About a Different Model?

If this model could not answer the question, perhaps the question was simply too difficult for it. What if I tried a newer model?
I returned to the second version of the “Product and Technical Improvements” experiment and asked the newer model to assign the labels. The result was worse: extra labels rose from 49 to 68, and exact match fell to 27%.

This experiment showed that switching to a newer model would not necessarily help.
The new model might have responded better to a different style of prompt, but that would have meant restarting the prompt-tuning process. For the time being, I stayed with the original model.

If It Would Not Listen, Could I Ask a Different Question?

No matter how I revised the prompt, the agent’s judgments still differed from mine. What if I changed the question entirely?

I split the process into two stages instead of asking the agent to decide whether to apply a label directly:
First, the agent would extract structured facts from the discussion: Had the problem been fixed? Had someone taken ownership? Was there a Pull Request?
Then Python rules would decide whether to apply the label. Once this second-stage decision was written as code, the same input would always produce the same result, and the rules could be tested.

I had good reason to trust the rules: each of the 23 test cases came from a real decision I had made while reviewing the golden dataset. In one issue, for example, the team agreed to “discuss it later.” I had treated that as unresolved work, which meant the issue should receive the “Product and Technical Improvements” label.
As I changed the architecture, that decision became a pytest case. Whenever the criteria changed, I could run the tests to see whether my current judgment still matched my earlier one. For the first time, the boundaries of my judgment had an executable, regression-testable form.

The resulting score was still disappointing.
The failure was not in the Python rules, but in the first-stage fact extraction. The golden dataset contained only 10 issues labeled “Add to Playbook,” yet the model claimed to find a “reusable procedure” in 54 of 101 issues. For “Product and Technical Improvements,” it misclassified both unresolved and already-owned work. The second-stage rules were stable, but the facts they received were not. A newer model performed even worse as the extractor.

When I reviewed the agent’s answers, I realized that the two labels—“Product and Technical Improvements” and “Add to Playbook”—depended on information such as whether someone had taken ownership and whether something was worth turning into a guide. That information lived in the team’s workflow, another internal tracking system, and offline conversations.

Why could the note be taught? Because it was a summary, and its answers were present in the text. The follow-up improvement areas required additional context: Was the issue worth fixing for the team? Had the team discussed it and decided whether to act? None of that necessarily appeared in a single discussion thread.

A postmortem note is a grounded restatement. A label is an ungrounded judgment.

In the end, I did not put the two-stage architecture into production. I returned to the original v9 prompt, but kept the rules and 23 test cases. Perhaps they will have a chance to return someday.

An Unexpected Contribution: Two Architectures Found Problems in the Answer Key

After struggling with the prompt, I began to wonder whether the answer key itself had been wrong from the start.
While reviewing the two-stage results, I noticed something strange: on some questions, the v9 prompt and the two-stage architecture produced the same answer, yet both disagreed with the golden dataset.
That made me even more suspicious of the answer key.

Reviewing the Labels Again

I selected roughly 10 suspicious issues, hid the existing labels, and reviewed them blind. The review revealed two different problems.
The first was in the golden dataset. For five of the 10 issues, my blind-review labels differed from my original ones. After comparing the reasoning behind the decisions, I updated the dataset. Exact match increased by about three percentage points. Changing only five records was enough to move the overall score noticeably, which showed how dependent label-level metrics were on the quality of the golden dataset.

That still was not enough to explain the experiments’ overall failure.

The second problem was in the wording of the rule. Another five of the 10 issues involved “User and Operational Experience.” In the blind review, I concluded that a customer using our interface incorrectly was itself a signal worth tracking, so the label should be applied. When I returned to the reasoning I had recorded while building the golden dataset a month earlier, I found that my judgment had been the same then.
In other words, this was not another instance of the problem from Building a Golden Dataset from 100 Production Issues, where I had been less consistent than I believed. My decision criterion had remained consistent. The conflict came from the wording in the prompt, which failed to describe what I actually meant.
I changed the “User and Operational Experience” rule to:

Apply the label when an issue is confirmed to result from a customer’s incorrect use or misunderstanding of our interface, API, or configuration. Apply it even if no one proposes an improvement in the thread and the customer corrects the issue themselves. The mismatch itself is a system-side improvement signal, such as an opportunity for better documentation, clearer guidance, or stronger guardrails.

Afterward, I decided to record the reasoning behind every label decision. A label contains only the answer; the reasoning preserves the decision boundary that produced it.

If It Could Not Be Fixed, Should I Remove It?

After reviewing the results, I came to a realization: the answer was not in the discussion thread. If the agent could not reliably decide whether to apply “Product and Technical Improvements,” perhaps I should remove that label from the task entirely.
I first ran an offline calculation using the existing outputs and ignored “Product and Technical Improvements” during scoring. Exact match jumped from 43.6% to 71.3%.

The Most Obvious Approach: Remove It

In the first implementation, I took the most obvious approach and removed the improvement area from the prompt.
The result was not nearly as good as the offline calculation. The tendency to over-apply “Product and Technical Improvements” did not disappear. It simply moved elsewhere.
The offline calculation predicted only 16 extra labels, but the actual run produced 47. Extra “Monitoring” labels rose from five to 15, and even “Add to Playbook” accumulated 11 extra labels.
Apparently, the agent’s urge to apply labels did not disappear when one option was removed. It simply found another outlet.

The final solution may sound strange. I left the definition of “Product and Technical Improvements” in the prompt and let the agent apply it whenever it wanted. But before writing the results back to the internal record system, the program silently discarded that label. If I could not change the agent’s behavior, I could handle its output at the system level.
This reduced extra labels from 47 to 22 and produced an exact-match score of 66.3%. I decided to ship that version. Its configuration matched the one measured by the offline simulation, and 66.3% fell within the predicted range of 65% to 70%. The offline simulation, the counterexample created by removing the option, and the recovery after restoring the definition all supported the same hypothesis: labeling pressure was conserved.

Final Evaluation Results

The final evaluation used 101 records from the golden dataset and three runs. Average exact match was 67.3%, with run-to-run variation of approximately plus or minus 4.7 percentage points.
This 67.3% cannot be compared directly with the 48% from the third article. The golden dataset had grown from 27 to 101 issues, and one improvement area had been removed. The only directly comparable result under identical conditions was the offline calculation: 43.6% versus 71.3%.

The final scores for each follow-up improvement area and the postmortem note were:

Follow-up improvement area	P	R
Add to Playbook	56%	45%
Add Logging	29%	50%
Monitoring	79%	85%
External Data Specifications	72%	86%
User and Operational Experience	60%	38%

Postmortem note metric	Result
Format compliance	100%
Q1: Problem-description accuracy	99%
Q2: Root-cause accuracy	94.9%
Q3: Solution-description accuracy	99%
Q4: Hallucination rate	6.1%
Honest handling of insufficient information	97%

Note: Hallucination rate is the only metric where lower is better. Higher is better for every other metric.

The system still had several known limitations:

• Recall for “User and Operational Experience” was only 38%. I tried broadening its definition, but when its recall rose, recall for “External Data Specifications” fell from 81% to 57%. Two consecutive versions showed similar competition between the categories, so I reverted the changes and accepted the limitation for now.
• The golden dataset contained only four “Add Logging” records, so its score was not stable.

The workflow was designed around “AI makes suggestions; humans decide.” Even with these limitations, the results were useful enough to surface cases worth reviewing. After 17 prompt versions, I decided to put the system into production and run it on real data.

Back to the Beginning

After five articles, the title of the first one—Why I Let AI Make Suggestions, Not Decisions—has taken on another meaning for me. Humans must decide not only the direction of the project, but also which questions should be given to AI in the first place.
AI continues to evolve even as I write this article. Perhaps in six months or a year, the problems I encountered—or the tasks I once considered difficult—will be straightforward.
When I built this agent, however, AI was still better suited to grounded work: restating information, extracting facts supported by explicit evidence, and making preliminary judgments whose evidence could be found in the input. Until humans document organizational context, decisions that depend on it are still better left to humans.
A prompt is not a magic cure. It cannot give AI information that was never present in its context.

At this point, I could finally run the agent on the backlog of historical issues. This series about the postmortem agent is also approaching its end.
In the next article, I plan to share what we did after putting the agent into production.