Table of contents
Two reviewers can look at the same condition and assign different codes. One calls it a documentation gap, the other calls it a guideline exception. The difference matters because the code drives the queue, the owner, and the borrower message. A team that never measures agreement cannot tell whether its codes mean the same thing to everyone. This study measures how consistently reviewers code conditions on the same file. It uses a codebook, a blind second review, and a disagreement log. The result shows where the definitions are ambiguous and which codes need an example.
Question and scope
The study asks how often two reviewers assign the same condition code and severity to the same file. Scope names the team, the condition types, the codebook version, and the period. A team might study income and asset conditions during a four-week window. The result describes that setting and should not be presented as an industry average.
Define the unit as the condition, not the file. A file with five conditions gives five coding opportunities. Report the file-level agreement beside the condition-level agreement. The two views answer different questions.
State the codebook version before the sample. A study that mixes versions cannot separate a definition problem from a training problem. Keep the codebook unchanged during the sample. If a change is necessary, start a new segment.
Building the codebook
The codebook should list each condition code with a short definition, two examples, and a rule for the boundary cases. A code without an example is a label, not a standard. The examples should come from real files with identifying details removed. Include a severity scale with clear anchors. For example, a low severity condition may be a missing page that the borrower can supply quickly, while a high severity condition may require a lender decision. The anchors should describe the consequence, not the reviewer's feeling.
Keep the codebook short enough to use in a review. A twenty-page guide will not be consulted during a busy day. A one-page card with the top codes and the boundary rules is more likely to be applied. The full guide can hold the examples.
Selecting the sample
Use consecutive weeks and include conditions from more than one reviewer. A sample drawn from one reviewer's files cannot measure agreement across reviewers. Record the reviewer pair and the condition type for each item. Include conditions that were easy and conditions that were hard. A sample of only hard cases will overstate disagreement. A sample of only easy cases will understate it. Use the normal mix and note the difficulty band. If the volume is small, use all conditions in the window. A small sample can still reveal a code that nobody applies consistently. Report the count and the confidence range so the reader can judge the precision.
Running the blind second review
The second reviewer should code the condition without seeing the first reviewer's choice. Use a separate form or a hidden field. A second review that sees the first code will anchor on it. The blind condition is the core of the method.
Give the second reviewer the same file evidence and the same codebook. Do not give extra context that the first reviewer did not have. A difference in evidence is a different study. Record any missing evidence as a separate finding.
Set a time limit for the second review. A review that takes an hour per condition will not be repeated. A short review with the same evidence is enough to test the code. The goal is agreement, not a second full audit.
Coding and comparing
Compare the code, the severity, and the boundary rule used. Report agreement separately for the code and the severity. A high code agreement with low severity agreement means the reviewers see the same issue but disagree about its cost. Use a simple agreement percentage and a chance-corrected measure if the sample is large enough. Report the raw counts so the reader can see the basis. A percentage without the counts can hide a small sample. Keep a disagreement log with the condition, the two codes, and the reason each reviewer gave. The log is the most useful output because it shows the exact ambiguity. A summary number without the examples is hard to act on.
Reading the disagreements
Group the disagreements by code and by cause. A code that is confused with another code needs a boundary example. A severity disagreement needs a clearer anchor. A missing-evidence disagreement needs an intake rule.
Look for patterns across reviewers. One reviewer may apply a stricter reading of the same rule. That is not necessarily an error, but it is a training or codebook issue. The goal is a shared standard, not a forced consensus.
Do not use the result to rank reviewers. A low agreement rate is a property of the codebook and the training, not a verdict on a person. A study that feels punitive will produce careful but dishonest coding. Keep the focus on the definitions.
Reporting the result
Report the code agreement, the severity agreement, and the number of conditions reviewed. Break the result down by condition type and by reviewer pair. Show the most common disagreement and the codebook change it suggests. A high agreement rate does not prove the codes are correct. Reviewers can share the same misunderstanding. Pair the agreement study with a quality sample that checks the codes against the guideline. The combination shows consistency and accuracy. The mortgage operations support lead can use the result to prioritize codebook changes. The loan processing support team can test the revised definitions on a new sample. A short retest after the change shows whether the agreement improved.
What the result can change
A low agreement on one code may justify a boundary example. A low agreement on severity may justify a new anchor. A low agreement across many codes may justify a simpler codebook. The study should list the candidate changes before recommending one.
Test one change and rerun the study. A revised definition is easy to test because the team can compare agreement before and after. Use the same sample size and the same blind method. A change that only improves agreement on easy cases is not enough. If the agreement does not improve, the training may not have reached the reviewers. The mortgage operations support lead can check the training record and the codebook distribution. The result should distinguish a definition problem from a training problem.
Limits and sources of error
The study measures agreement, not correctness. Two reviewers can agree on a wrong code. The quality sample is the check on accuracy. Report the limitation so the reader does not treat agreement as truth.
The sample may be too small to support a stable estimate. Report the count and the range. A small team should use the disagreement log as the primary output and treat the percentage as a guide. The reviewers may change their coding after the study begins. A mid-study change can inflate or deflate agreement. Freeze the codebook and the evidence rules for the sample. If a change is necessary, start a new segment.
Review cadence
Repeat the study after a codebook change, a new reviewer, or a process change. Use the same sample method and the same blind review. A stable agreement rate over time is a sign that the standard is holding. A drop is a sign that the work or the team has changed.
Share the result with the reviewers and ask which codes were hardest to apply. The discussion can reveal an example that should be added. The next study should test one of those findings. End with a sentence that states the code agreement, the severity agreement, and the period. The Consumer Financial Protection Bureau's mortgage rules and the Fannie Mae Selling Guide use categories that a team codebook should reflect. The goal is a condition code that means the same thing to every reviewer.