A short piece of fiction, written by API Evangelist. The reader is imagined; the post, and every number and quotation from it, is real: "Consistent But Wrong," Kin Lane, November 9, 2026, https://apievangelist.com/2026/11/09/consistent-but-wrong/
Marisol builds the intake tooling for a claims team, and she read the post on a Monday morning with her coffee going cold. She had come for the decision-model material, because her team had been talking about a Jev-style pick-one step for routing. She stayed for the table.
The healthcare row bothered her first, though nothing in her work touches drugs. Every answer passed every check, and more than half were the wrong strength. The part she reread was why. The model had been handed bare codes and nothing saying which was which, so it guessed, and the guess was valid. She thought about her own schemas, which were full of field names that made sense only to the three people who had written them.
Then she reached the insurance section, which was close enough to her own work to make her put the cup down. The email said Last Thursday, it arrived on a Sunday, and the model's date came back two days early. The date was real and inside the policy period, and it was on the right policy. The model had even cited the email word for word. The post said every rule passed it. Twenty-four percent of the answers were wrong that way, even with the fields described.
She disagreed with some of it. The post says to keep a person on the judgments the rules can't see, and she knew how that went. Her reviewers already skimmed anything that looked tidy, and the post said as much itself: the careful, well-shaped answer is the kind a reviewer skims past because everything around it is right. A person on the loop was not a fix if the person was trained by the tidiness to stop looking. She wrote that in the margin of her notes and underlined it. She also noted that the author admitted these were templates, with accuracy to be read as an upper bound, so her real emails would probably do worse. That did not comfort her.
What she kept returning to was the sentence that the checks can prove an answer is consistent, but they can’t prove it’s right. Her dashboard had a green pass rate that she had shown to her manager twice. She had been proud of it. Now it looked like a measure of something narrower than she had claimed.
She did not rewrite anything that day. She did not propose a new review stage either, since she doubted she could win that argument. She opened a blank spreadsheet and pulled ten old claim emails from last quarter, the ones with relative dates in them. For each she worked out the correct date herself, from the day the email was sent, and typed it in a column beside the email. It took her most of an hour, and twice she caught herself unsure which day she meant, which told her something too.
The post had told her to measure the answers that pass, not just the ones that fail. Ten rows was a very small start. She saved the file as the first of her known-answer set and added a note at the top to run it whenever the shape, the model or the engine changed. Then she went to find out who owned the schema descriptions.