Field Tests

Everything in this book was tested in the wrong order.

The creed came first. The systems were built under it, at work, with real money and real people, over several years. The reading came last, and the reading is what narrowed the rules. Which means the cases in this chapter are not demonstrations of the reconciled creed working. They are a record of what the unreconciled version produced, run back through the seven gates to see what a better instrument would have caught.

That is the only honest test available to me, and it is a harsher one than a demonstration. A worked example chosen after the fact proves nothing. A completed project scored against a standard it was not built to meet is at least capable of failing.

Five cases, four of them built. I am describing them at the level of the design problem rather than the implementation, because the implementation details belong to a company and the design problems belong to anybody.

The problem. Work requests move through states. Which state a request is in determines who touches it, how fast it moves, whether anyone is watching. Three people decided this from judgment, using criteria nobody had written down, and they did not agree with one another, which nobody knew.

What the old creed said. Define the variables. Standardize them. Reduce the decision to a formula. All three of those were correct and the project was worth doing.

What was built. A named set of states with written entry conditions. The state space went from something nobody could enumerate to something countable. Two people looking at the same request produced the same answer.

What actually happened. Three things, and I have used all three elsewhere in this book because they were the most instructive failures I have had.

Requests that fit no state got assigned the nearest one, which made them look like the other requests in that state, which is precisely how they stopped being noticed. Under the old judgment-based arrangement they had been handled by somebody finding them odd.

States began to be chosen for their downstream effects rather than their accuracy, once people understood what each one triggered.

And the people who had been making the judgment stopped reading the requests. That was the intent. What nobody had noticed is that reading the request whole was how they knew which accounts were about to become a problem, and within a year that knowledge was gone, and the reason was not that anyone got worse.

What the gates would have caught. Gate three would have caught the second failure before the build. Not manufacturable asks who benefits from a condition being true and whether they can bring it about, and the answer for a state that stops a clock is immediate and obvious once the question is asked. The fix is not a better definition. It is that the state and the clock must not be designed by people who are not looking at each other.

Gate five would have caught the third, which is the expensive one. What does the current work make somebody look at? The answer was: the whole request. The remedy is not to abandon the classification. It is to keep the reading, deliberately, as a step, after the engine proposes, which costs a few minutes a day and preserves the capability the whole operation depended on without anyone having named it.

Gate four would have surfaced the first. The inventory asks what the person knows that the rule will not, and some of these are weird and I can tell is exactly the kind of answer that comes out of that conversation and gets dismissed as unhelpful. It is not unhelpful. It is a specification for the exception path that the standard needed and did not have.

Score. Three gate failures, all catchable in advance, none of them caught. The project still improved things. It improved them by less than it appeared to, and the difference went into a place the reporting could not see.

The reporting is the part of this case I find hardest. Every number the project was measured on improved and stayed improved. Classification consistency, time in state, the rate of requests sitting past a threshold: all better, all still better today, none of them wrong. There was no moment at which anything looked like it was going badly, and there is still no report that would show what was lost, because the lost thing was a capability that had never been named and therefore never had a baseline.

An organization running on those numbers would conclude the project was a success and would be correct, on the numbers. Which is the argument for gate five stated as harshly as I can state it: if you do not identify the unmeasured thing before you remove its cause, you will never find out you removed it. Not because anybody is careless. Because every instrument that exists is pointed at the measured thing, and the unmeasured thing has no instrument by definition.

The problem. A customer calls with something wrong. Before anyone can help, a form needs to know who they are, which building, which unit, when it was installed, what is wrong, and who to call back. Six questions asked of somebody whose only interest is getting off the phone.

What the old creed said. Minimize user input, maximize automated output. Derive what can be derived.

What was built. Phone number resolves to account, account to building, building to a list of units. Two of six questions disappear and two more become selections from a real list rather than free text.

What actually happened. It worked, and then it failed in a way that was more interesting than if it had failed outright. A caller's number matched the wrong account, because the number belonged to a general contractor who works for several customers. The form arrived pre-filled with a building nobody on the call was discussing. The person taking the call did not notice, because filled fields are not read. The request went to the wrong place and sat for nine days.

Nobody made an error. Somebody accepted a default.

What the gates would have caught. Gate three, on observable. A phone number resolving to an account is observable in the easy case and ambiguous in a specific population, and the design had no representation for the difference. The repair is the three-state derivation: certain, probable, ambiguous, with the third one handed back to a person even though handing back is an input the original rule argues against.

Gate six is the one this case makes vivid, though it is not where the failure happened. Could the person taking the call read the rule that filled the field? No. It presented a conclusion with no provenance, which is an assertion. A pre-filled value showing unit 4B, because you called from the number on file for that building is evidence, and people check evidence, and the whole failure hinges on that one line not being there.

Score. Two gate failures, both fixed after the fact at a cost of one incident. This is the case I would point to if somebody asked what the reconciled creed is actually for, because every repair here came from a rule that did not exist in the original twelve.

One further consequence of this case is worth carrying, because it reaches past intake forms. The nine days happened because a derived value and a supplied value were rendered identically, and once that is true, a person cannot triage what they are looking at and will stop trying. The repair is not care. It is that the two kinds of fact have to look different on the screen, permanently, at a small cost in visual tidiness.

That principle now applies to everything I build and it is the single cheapest thing in this book. Anything the system asserted looks one way. Anything a person supplied looks another. Anything the system guessed looks a third way and says what it guessed from. Three renderings, one afternoon of work, and it converts a class of silent failure into a class of visible one.

The problem. A job needs installing somewhere. Which partners can do it? The answer lived in a spreadsheet with a state abbreviation in it.

What the old creed said. Define every variable to its deepest identifiable condition. A state abbreviation is nowhere near deep enough.

What was built. Geography decomposed into a home base, a radius, and what happens past the radius. Licensure decomposed by jurisdiction and work type, with expirations. Demonstrated scale. Capacity was deliberately left out.

What actually happened. The parts that went deep on slow-moving verifiable facts held up. The parts that would have gone deep on fast-moving facts were correctly refused, and the refusal was the good decision in the project. Capacity changes weekly, cannot be verified without asking, and a stale precise answer is worse than an acknowledged gap, because a stale precise answer gets trusted.

What the gates say. This is the one case where the old creed and the new one mostly agree, and the reason is instructive: the project was constrained by somebody's judgment about maintenance, which is the cost condition the reconciled second ism now states explicitly. The judgment was correct and it was not derivable from the creed as it stood. It came from having been burned before.

Gate seven is where it still falls short. The matrix went in with no maintenance allocation. Licensure expirations carry dates and so the expiry instrument applies for free, which covers one dimension. Nothing samples the rest. Nothing records the case where a scheduler called a partner the matrix did not list, which is the override signal and which would have been one field.

Score. One gate failure, still open. This case is a reminder that the creed's worst habit is not building the wrong thing. It is building the right thing and walking away from it.

There is a second thing here that took me a while to see, and it changes how I read the whole case. The decision to stop short on capacity looked at the time like a limitation being accepted. It was actually the most valuable feature in the design, because a matrix that says call to confirm capacity sends somebody to find out, and a matrix that reports a number from last quarter stops the search.

That generalizes past this project and it belongs with the seventh gate rather than filed as a quirk of one build. A system that admits what it does not know is more useful than one that guesses, because the first one routes a person to the answer and the second one convinces them they have it. Every design I have built since has a place where the honest output is unknown, and it is always the field somebody argues about, because a blank looks like an incomplete system and a stale number looks like a working one.

The problem. People in the company know things that exist nowhere else. Asking them to document their work had been tried and had failed the way it always fails, which is not refusal but compliance producing nothing usable.

What the old creed said. If it lives in your head, it should live in the system.

What was built. Video rather than documents, because somebody who cannot write down how they read a drawing set can be recorded reading one. An intake form cut from eleven questions to eight, because every required field is a reason not to submit. An explicit ask of five to fifteen minutes, because the alternative to a short video is not a thorough document, it is nothing, and then the person leaves.

What actually happened. Too early to say, and no adoption figure would mean much yet. The design is in place and adoption is the open question.

What the gates say. This is the only one of the four designed after most of the reading, and it shows.

Gate four, the inventory, is what produced the video decision. The estimators' knowledge of how a given architecture firm lays out a drawing set is in the third pile: cannot be captured as text by its holder. The design response is to capture a demonstration rather than a statement, which is the reconciled seventh ism operating rather than the original.

Gate one is the one I am least sure about. Does this repeat? Each recording is about one firm, which makes each a one-off wearing a shared name. What repeats is the act of recording rather than the content, and I think that clears the gate. I am aware that is the kind of reasoning somebody produces when they want a project to pass.

Gate five is where the design took its real decision. What does the current arrangement do for the people doing it? It makes them the only source, which is a form of standing, and a capture program is a request that they give it up. The three-question test from the fourteenth ism applies to the participants rather than to a rule here, and the honest answer is that the counter-offer is thin: faster onboarding of people who would otherwise interrupt them, and visible credit. Whether that is enough varies by person and I would not claim otherwise.

Score. The gates were used, two of them changed the design materially, and one of them names a problem that is not solved and may not be.

Gate seven has an unusual shape on this project, because the thing that decays is not a rule. It is the relevance of the recordings. A demonstration of how a particular architecture firm laid out drawings in 2026 is a document about 2026, firms change their conventions, and the recording will not know. A library of confidently outdated demonstrations is a specific and nasty failure, because a newcomer watching one has no way to tell that the convention moved.

The instrument for that is cheap and it is not the manual sample. It is a visible date on every recording and a convention that anything past some age gets a one-line note from somebody current saying whether it still holds. That is closer to the expiry instrument than to sampling, and it works here for the reason expiry works anywhere: the thing being recorded has a natural clock attached, which most knowledge does not.

This is the most useful case in the chapter and it is the hardest to write, because there is no artifact to point at and nothing happened.

The problem. Somebody wanted a system to decide which jobs to bid on. There is data: size, location, customer, architect, competitor presence, historical margin. There is a decision made repeatedly. It looks exactly like the others in this chapter.

What the old creed said. Define the variables. Reduce it to a formula. It is a recurring decision with observable inputs and a scoreable outcome, which is the profile the creed is built for, and I was enthusiastic.

What the gates said. Gate one passed. Perhaps sixty of these a year, each taking real deliberation. Volume is marginal and the per-instance cost is high enough to carry it.

Gate two is where it stopped. Does anyone predict this in advance and turn out to be right? People explain bid outcomes fluently afterward and nobody predicts them beforehand at better than chance, and once that question was asked out loud, everybody in the room knew the answer and nobody had ever said it. Does the same intervention produce the same result twice? No, because bidding is played against other people who adjust. Can an experienced person explain it to a newcomer well enough to work? Not really, and the explanations offered were post-hoc narratives.

Three for three toward complex. The situation had felt complicated for years because it involved a lot of factors, and a lot of factors is what complicated feels like, and the distinguishing property is not the number of factors but whether cause and effect can be known in advance.

What happened instead. Nothing was built. The data got collected anyway, without a rule attached to it, because collecting is cheap and the complex domain calls for probing rather than ordering: run small variations, keep the record, let a pattern show itself if there is one. Three years later there is a pattern in one narrow segment and there is no formula, and the narrow segment is where a formula might now pass gate two.

Score. The only case here where the instrument's main function, which is stopping projects, actually operated. It saved perhaps four months of work and it produced no evidence of having done so, which is the permanent problem with this gate and the reason it will always be undersold.

I include it because four success stories and no refusals would be a dishonest chapter. The instrument's most common correct output is no, and the entire difficulty of using it is that no is invisible and yes ships something.

Five cases, and the pattern across them is more useful than any of them individually.

Every failure was a gate that nobody thought to ask. Not one of the built projects failed on execution. The status engine was built correctly. The intake derivation worked. The matrix was well designed. The failures were all at the level of a question that was never on anybody's list, and they were all cheap to ask and expensive to discover.

The failures clustered in gates five, six, and seven. The technical gates, one through four, are the ones the old creed already covered, and the projects mostly passed them. Gate five is the social cost, gate six is the standing of the person governed, gate seven is the maintenance. Those three are the ones that came out of the objections rather than out of the original creed, and they are where all the unrecovered damage is.

That is a stronger result than I expected when I started scoring these, and it is the closest thing to a validation this book contains. The rules I had were not wrong. They covered one part of the problem completely and another part not at all, and every case above failed in the part they did not cover.

And the maintenance gate is the one that is still open on three of the four built. Not because anyone disagreed with it. Because it has no completion date, produces nothing anybody can point at, and has to survive a budget conversation every year against line items that do. I have argued for it, in this book, at length, with better arguments than I had before, and I have not implemented it on three of my own projects.

Which is the most honest thing I can put in a chapter about field tests. The hardest part of this creed is not the part that requires insight. It is the part that requires somebody to keep paying attention to a thing that appears to be working.

The reason that is hard is not laziness and it is worth being precise about, because the wrong diagnosis produces the wrong remedy. Maintenance on a working system is unrewarded in a specific structural way: the best possible outcome of doing it is that nothing happens, and the worst possible outcome of skipping it is that something happens in two years and gets attributed to whatever was nearest at the time. The incentives are not merely weak. They point the other way, at every level, for everybody.

Which means the remedy cannot be exhortation and cannot be individual discipline, because both are asking a person to do unrewarded work indefinitely. It has to be structural: the allocation named in the original build, carried as a line item with the system rather than with a person, reviewed when the system is reviewed. That is the only form I have seen survive a year, and I have seen it survive a year exactly once.

Two things I would tell somebody about to run this on their own work, both of which come from getting them wrong here.

Run the gates with the people who do the work, in the room, out loud. Every useful finding in these five cases came out of a conversation rather than out of analysis. The dispatcher's four facts. The fact that nobody predicts bid outcomes, which everyone knew and nobody had said. The reading of the whole request, which nobody identified as valuable until it was gone. None of those is discoverable from a desk, because the people who hold them do not experience them as knowledge. They experience them as the work being normal.

The failure mode on the other side is running the gates as an interview, where somebody with a list asks questions and writes answers down. That produces the same polite, plentiful, generic material that every documentation effort produces. What works is running the gates while looking at actual instances: pull ten real cases from last month and go through them one at a time, asking what happened and why. The abstractions come out of the instances, never the reverse.

And write down the gate failures, including the ones you decided to proceed past. A gate failure that gets discussed and overruled in a meeting leaves no record, and six months later when the predicted thing happens, nobody remembers that it was predicted. The status engine's manufacturability problem was raised, by somebody, in a room, before the build. I am fairly sure of this and I cannot prove it, because there is no record, and the absence of the record is the lesson.

A gate failure recorded with its date and the reason for proceeding anyway is worth more than the same failure prevented, because it converts a category of surprise into a category of known risk, and known risks get watched.

A last observation, and it is about the difference between these four built cases and the fifth.

The four that shipped were all proposed by somebody who wanted them, myself included, and the gates were applied retroactively in this chapter by the same person. That is a weak test and I would rather say so than let the scoring stand as though it were independent. Somebody else running these gates on my projects would find failures I did not, because the gate a person misses is the one whose answer they would not like, and I am not exempt from that.

The fifth case is the only one where the gates ran before the decision, with people in the room who had no stake in the outcome, and it is the only one that produced a no. I do not think that ordering is a coincidence.

Which suggests something about how the instrument should be used that is not in Chapter 18 and probably should be: the person running the gates should not be the person who wants the project. That is inconvenient in a small organization, where the person who wants it is usually the only one who understands it well enough to ask. The partial answer is that the questions in gates one, two, five and six are deliberately answerable by somebody without domain expertise, which was a design goal and which I now think is the most important property they have. They can be asked by a skeptic.

One case I cannot report on, and the absence is worth recording rather than papering over.

The outline for this chapter called for two more field tests drawn from my own published writing, reading a public argument through the creed's own method. I was not able to retrieve those pieces to work from, and reconstructing what I think I argued in an essay I could not reread would be exactly the kind of confident vagueness don't make the specific unnecessarily vague is against. They are missing on purpose, and the chapter is shorter for it.