Reviewing AI Rules
Your AI setup got better, and a required step now looks like overhead. Someone on the team asks whether it can go. The answer depends on what that step is doing.
What reviewing a rule means
Reviewing an AI rule means examining what purpose it serves, whether that purpose still matters, and how well its current method serves it.
A rule is an instruction, a required step, a check or a limit on action. A review starts with a trigger: a better model, a new tool, a recurring failure, a change in workload, or evidence that a required step costs more than it returns. It ends in a decision about one rule, and a record of why.
Find the purpose before you touch the rule
A sales team, constructed for this example, uses AI to draft customer proposals. Its earlier tool kept leaving out customer requirements. So the team added a rule: the AI first outlines where each requirement will be answered, and a colleague checks the outline before drafting. Now a newer model writes drafts that look complete. Does the outline still earn its cost?
In coding-agent benchmarks, adding a context file of instructions gave no significant overall improvement in task success. Guidance written by developers did better than generated guidance (Evaluating AGENTS.md). But that does not show that instructions are unnecessary or that shorter files are better. So examine each rule on its own, in context.
Four questions recover a rule’s purpose:
- What result does the rule help someone achieve?
- What failure or unacceptable action does it prevent?
- Who depends on it, including people further along the delivery path?
- What evidence led to it, and do those conditions still hold?
A rule added after an error can hold a real lesson, but its wording can reach past the case that caused it. When nobody remembers why a rule was added, recover the reason from the people, records and systems that depend on the rule. If the reason stays unclear, record that. Then find who can authorize an investigation or a limited test. Missing history is not permission to remove a protection.
Review the rules whose cost or consequences justify the effort, not every sentence in every prompt.
Separate the purpose from the method
The sales team’s workflow has two more rules. One makes the team reconcile conflicting prices before a proposal goes out. The other requires approval for any commitment beyond delegated authority.
Each rule has a purpose, what the rule is for, and a method, the way it serves that purpose. The outline’s purpose is to help the AI cover every requirement. Its method is a fixed sequence: outline, check, then draft. A better model makes another method possible: state the required content clearly, check the draft against the customer’s requirements, and skip the outline.
But a better model does not by itself show that the purpose is gone. And better drafts alone are no reason to change the other two rules. A complete draft does not settle which price is right, and permission to commit does not come from the model.
The table below is broader: six purposes worth checking in any rule.
| What the rule serves | Question to ask |
|---|---|
| Purpose and acceptance | Does it make clear what a useful, acceptable result must accomplish? |
| Authority and obligations | Does it limit what may be accessed, changed or promised? |
| Coordination and shared state | Does it keep contributions compatible and prevent conflicting actions? |
| Quality and evidence | Does it establish an adequate basis for accepting the work? |
| Method support | Does it prescribe an approach that helps this AI setup perform the task? |
| Recovery and learning | Does it help people detect, correct or learn from a failure? |
The outline falls under method support, because it prescribes how the AI drafts. But the six purposes overlap, and one rule can serve several: a required handoff format can support coordination, evidence and recovery at once. So check each purpose before you change the wording or remove the step.
Choose what to do with the rule
The choice is not just keep or delete:
| Response | What would justify it |
|---|---|
| Retain it | The purpose still matters, and the rule serves it at a proportionate cost. |
| Revise it | The purpose is sound, but the wording, scope or trigger creates avoidable problems. |
| Let something else serve its purpose | Evidence shows that a tool permission, interface, automated check or other arrangement serves it more effectively. |
| Test a relaxation | There is a credible alternative, and a contained comparison can reveal relevant losses before broader use. |
| Retire it | Its purpose no longer matters, or another verified arrangement already serves it adequately. |
Take a requirement out of a prompt, and something else has to do what the requirement did: a permission, a tool or a person. Count that work when you compare effort, cost and reliability.
The sales team will test a relaxation of the outline, but it will retain the approval requirement.
Test the change where you can contain it
The sales team does not relax the outline in live work. Its trial is draft-only, because a draft can be examined before it reaches a customer or an external system. The tools cannot send a proposal or change an external record. The access and commitment limits stay in place.
Pick one change of method whose consequences you can contain, in work you are authorized to change. If someone else owns the rule, take your findings to that owner before you test a relaxation.
1. Describe the purpose and one alternative
The outline’s purpose does not change. The sales team tests one alternative method: the one that skips the outline.
The team also names who the work serves. The recipient is the colleague who reviews and assembles the proposal, and the customer needs a clear, accurate basis for deciding whether the offer meets their needs.
2. Decide what counts as better or worse before you look
The sales team writes its conditions before it sees a result. Acceptance conditions define an adequate result. For this team, an adequate draft answers every customer requirement accurately and takes the recipient no longer to repair than a draft made with the outline. An adequate draft also does more than sound convincing. It preserves real uncertainty and lets the recipient explain or revise the recommendation.
Stop conditions say when to end the trial early. This team stops at the first draft that presents an estimate as a commitment.
Choose checks that would catch the failure the rule was meant to prevent. Anthropic’s guide to agent evaluations separates a task’s final state from the actions that produced it. Both can need assessment, so don’t let a good final output hide an unacceptable action. And overly rigid grading can reject valid approaches, so don’t require the old sequence because it is familiar.
3. Compare under the same conditions
The sales team runs the same cases with the outline and with the alternative. The model, settings, tools and information stay the same. The team records that setup, with the prompts and the review process, because setup alone can move a result. Anthropic traced an apparent drop in Claude Code’s quality to interacting product settings and context-handling changes, not to a change in the underlying model.
Include the hard cases the rule exists for, not only routine successes. Repeat cases where variation could change the conclusion. Where practical, the recipient assesses outputs without knowing which method produced them. If the format gives the method away, record that and apply the same criteria to both methods. Record the work it took to create, review and repair each result.
A small comparison can reveal a problem or support a limited next step. But it cannot establish protection against rare, consequential failures, or show that the arrangement will hold for future models. So scale the comparison to the consequences of the decision.
4. Follow the effect past the step you changed
The AI writes drafts faster, but the recipient spends longer checking them. Or the recipient saves time, but the customer still waits on an unrelated decision. In a randomized study of Microsoft 365 Copilot, knowledge workers saved time individually, but no change was detected in the quantity or mix of their tasks.
So keep a real local gain, but keep it separate from a demonstrated improvement for the customer. Ask whether the change reduced ambiguity, waiting or rework, helped someone make a better decision, or moved the work to someone else.
A draft-only trial shows whether the drafts are acceptable and what they cost the recipient. But what the change does for the customer becomes clear only after an authorized change reaches real work. So record what is still uncertain about the customer and when to look again. When you look, judge both the output’s usefulness and what happens when the customer acts on it.
Record the decision and when to look again
Retiring a rule is a conclusion about its purpose and its replacement, not a reward for a shorter instruction file.
If the comparison is inconclusive, record what it could not establish and keep the required protections. No detected difference does not prove equivalence. Then decide whether better checks or different cases would justify another test.
Name who can authorize the change. Name who responds if the new arrangement stops working. Where recovery is possible, keep a baseline you can return to. But where an action cannot be undone, the protection has to come before it.
Then choose what triggers the next review: a change in model, tool, workload or authority, or an observed failure. Pick the ones that bear on the rule’s purpose, because maintaining a rule should cost less than the problem it addresses.
For the next person, write down why. That record keeps today’s decision from becoming the next unexplained rule.
A short record to copy
- Rule and purposes: What does it require? Which results, protections or dependencies does each part serve?
- People served: Who receives this output? How should it contribute to the customer’s outcome?
- Ownership and authority: Who can decide on this change, and who will respond to problems?
- Proposed alternative: What will change, if anything, and what conditions will remain the same?
- Acceptance and stop conditions: What evidence would support the alternative or reveal a loss?
- Observed result: What did the comparison or other evidence show, locally and downstream? What remains uncertain?
- Decision and follow-up: What will be retained or changed, how can problems be contained or corrected, and what will trigger another review?
Sources and lineage
Reviewing AI Rules is part of Applied Enterprise AI, a body of work developed by Curtis Hibbs and Joshua Barnes. It is a way of reasoning the authors propose. The six purposes and the four-step comparison have not been validated as a general method across enterprises.
- Evaluating AGENTS.md, June 2026 revision.
- Shifting Work Patterns with Generative AI, Microsoft Research and Harvard Business School, November 2025 revision.
- Anthropic, quality investigation of Claude Code.
- Anthropic, guide to agent evaluations.