Building a Golden Dataset from 100 Production Issues

Hi everyone, I’m back—slightly sooner than I expected.
At the end of my previous article, Why I Let AI Make Suggestions, Not Decisions, I wrote:

Human review may provide a relatively stable safeguard, but its throughput does not scale, and different reviewers may reach different conclusions. A more sustainable path would be to gradually move from “a human reviews every output” toward “the system measures output quality systematically.”

But how can I tell whether each prompt change actually produces better results than the previous version? New models seem to enter the market faster than I can work my way through my reading list—although, to be fair, I am a very slow reader.
Ignoring cost for a moment, does switching to a newer model always produce more accurate results with fewer mistakes? In practice, however, it is almost impossible to ignore budget constraints.
Once these questions arise, it is time to start building evaluations.
Before comparing prompts or models, however, I first needed a reusable baseline for comparison: a golden dataset.

What Are LLM Evaluations?

There is already more material about LLM evaluations online than I could possibly cover, so I will focus only on the problems I encountered:

• The sample size is too small: If I update a prompt and inspect a few cases, can I assume future outputs will maintain the same quality? Or did I simply happen to choose easier cases?
• Memory bias: We tend to remember obvious failures clearly, but it is much harder to notice when an AI system is quietly getting worse. For example, during the investigation of a production issue, the team may mention something as a possible cause. In the AI-generated postmortem note, however, that possibility might be rewritten as the confirmed root cause.
• No regression testing: Suppose the description of one follow-up improvement area is not precise enough, so I tweak it slightly. How do I know that the change has not affected the classification of other improvement areas?

I think of LLM evaluations as similar to the test cases we write during software development. The main difference is that LLM outputs are nondeterministic: the same input does not necessarily produce exactly the same output.
The goal of these evaluations is also not to answer a broad question like, “Is this model good?” Instead, I want to measure things such as the accuracy of follow-up improvement classifications and the consistency of postmortem-note quality. To be honest, I cannot always clearly perceive the differences between models during everyday AI use.

Building a Golden Dataset from Real Production Issues

To build LLM evaluations, I first needed to give the AI a definition of what “correct” looks like.
For the postmortem agent, each record in the golden dataset includes a human-reviewed set of follow-up improvement areas and a postmortem note for a specific production issue.

While building the dataset, AI pointed out that I had been mixing two different classification standards. For one issue, related monitoring was already in place, but I still labeled it as “Monitoring” because monitoring was part of the domain involved in the issue. For another issue, the fix had already been completed, so I did not label it as “Product or Technical Improvement,” because I was instead applying the standard of whether further action was still required. I had labeled both records myself, less than a week apart.
If I could not apply the criteria consistently, how could I expect the model to make the correct judgment? This section is about how, while building the golden dataset, I discovered that the least reliable component was actually me—and what I did about it.

How I Built the Golden Dataset

My process was to first ask AI to generate an initial set of follow-up improvement areas and a postmortem note for each issue. I then reviewed every record manually, corrected anything that seemed unreasonable, and documented the reasoning behind my decisions. After reviewing the full dataset, I asked AI to scan the golden dataset and flag potentially contradictory labels. The two records mentioned earlier were identified during this step.

Of course, using AI to review a dataset that I had already reviewed, and then using that dataset as the evaluation standard, may sound like letting the same person be both player and referee.
My safeguard was to separate responsibilities. AI was allowed to identify possible inconsistencies, but I made the final decision. Since this dataset would later become the benchmark used to evaluate models, AI could help surface problems, but it could not decide what the correct answer should be.
For the two records mentioned earlier, I established the following principle: follow-up improvement areas should represent work that still needs to be done, not simply the domains involved in the issue. If monitoring already exists, the issue should not be labeled “Monitoring.” If the problem has already been fixed and eliminated at the root, it should not be labeled “Product or Technical Improvement.”
Once the rule was established, I went back through the earlier records to make sure the entire dataset followed the same standard. This seems obvious in hindsight—it is called a follow-up improvement area, after all. This seems obvious in hindsight—it is called a follow-up improvement area, after all. A gentle knock on my own head.

To be honest, this safeguard is not complete. If AI fails to identify an inconsistency, I am unlikely to notice it either. The best I can do is document my reasoning throughout the labeling process, so that AI—or my future self—has a chance to reconstruct why a particular decision was made. This is a trade-off under limited resources. Ideally, the team would establish a shared and consistently applied review process.

I Don’t Have Two Annotators, but I Do Have Myself at Different Points in Time

Speaking of cost, the ideal approach for my use case would be to ask two independent annotators to assess the follow-up improvement areas and postmortem notes, then measure their level of agreement. This would help me understand how much ambiguity the task contains and provide an important reference point when interpreting model performance. After all, humans can reach different judgments, so it is unrealistic to expect AI to agree perfectly with every human reviewer.
However, each issue requires reading the team’s entire discussion thread before making a decision. Some threads contain more than 100 messages. Asking two engineers to spend time on this work—and squeezing it into the gaps between existing development tasks—would be extremely expensive.
My alternative was to review the follow-up improvement areas myself at different points in time, while asking the engineer who originally handled each issue to verify the corresponding postmortem note. This cannot fully replace two independent annotators, but it adds another layer of review at a manageable cost. The inconsistencies described earlier are a good example of why this matters.
When asking engineers to review the postmortem notes, I made one small but important design choice. Rather than asking, “Do you agree with what the AI wrote?” I asked more specific questions, such as:

• Has the root cause described in the note been confirmed as the primary cause of this issue?
• Does the solution described in the note match what you actually did?

Simply asking, “Do you agree with the AI-generated content?” may make it harder for reviewers to actively detect problems. I wanted to reduce the anchoring effect caused by presenting AI-generated content first.

How Did I Select the Issues?

How did I decide which samples to include in the golden dataset?
The selection was not random. I initially selected issues whose follow-up improvement areas had already been manually classified and whose postmortem notes had been manually written—around 30 records in total. Later, however, I noticed that some improvement categories were extremely underrepresented, such as “Add Logging” and “Operational Experience.” With so few examples, a single missed or incorrectly assigned label could cause the category’s score to fluctuate dramatically.
To address this, I deliberately searched for issues that might belong to these rare follow-up improvement categories and added them to the golden dataset. Finding possible examples among a large volume of historical issues was another task where I used AI. I asked it to identify issue titles that appeared likely to match these rare categories, and then used those issues as candidate samples for the golden dataset.

The Role of the Golden Dataset

The name golden dataset may suggest a collection of ideal, perfectly written reference examples. In practice, however, it is a record of the boundaries of human judgment.

Consider one record from my dataset. It contains a fairly long team discussion in which someone reports that a number in a report looks incorrect. After an extended investigation, the final message says that the report value has returned to normal. However, nobody in the thread identifies a confirmed root cause.

For this sample, I wrote the following in the postmortem note:

Root cause not confirmed in the discussion thread.

That answer is itself the value of the golden dataset. Not every issue has a known root cause or a clearly documented resolution. In such cases, we need to make an honest judgment: an inference made by AI cannot be written as the root cause unless it was actually confirmed.

A Golden Dataset Is a Living Asset

Finally, a golden dataset is not an asset that is finished once 100 issues have been labeled and the dataset has been frozen.
New edge cases will continue to appear. Every new issue that makes me hesitate is a candidate for inclusion in the golden dataset. If the evaluation criteria evolve again—for example, if a follow-up improvement category is added or removed—the existing records must be reviewed again. Records based on outdated criteria may even need to be removed. Otherwise, the dataset will quietly accumulate decisions made under several incompatible standards. The 100 records I have today are simply a cache of how I define the decision criteria at this particular point in time.

A trustworthy golden dataset only answers the question, “What should we use as the reference?”
How to score the results—and how those scores can reveal what needs to be fixed in the current prompt—is a topic for the next article.