A Missed Requirement Leaves Nothing to Review
What to measure before you trust an AI-built compliance matrix
A requirement that never makes it into the compliance matrix can't be checked, assigned or answered. Every later safeguard (the claim check, the colour teams, the final sign-off) works on the rows the matrix holds. The buyer's evaluators score against the solicitation, so they find the gap even when nobody on the bid team did.
This post is about the step that builds the matrix: pulling the requirements out of the solicitation. It covers where requirements hide, how to measure whether a tool finds them, what we measured, and the questions to ask any tool that builds the matrix for you.
1. The terms, once
- The solicitation is the buyer's set of documents: instructions, statement of work, forms, questionnaires, attachments.
- A requirement is anything the bidder must or should do, provide, meet or comply with, in the proposal or during the contract. An obligation on the buyer ("The Government will evaluate…") is not one, and neither is a heading or a definition.
- Mandatory means binding: shall, must, will, required, or an instruction like "Provide…". Advisory means should, may or preferred.
- The compliance matrix is the list of requirements a bid team works from, one row each.
- Extraction is building that list from the solicitation.
- The answer key is a hand-checked list of every requirement in a set of documents, used to score extraction.
- Recall is the share of the answer key's requirements that the tool found. It measures what the tool missed.
- Precision is the share of the tool's rows that are real requirements. It measures how much noise the tool added.
- Held-back documents are documents set aside to score a tool and never looked at while adjusting it.
Everything below uses these words and only these.
2. Where requirements hide
The traditional way to extract is a set of keyword rules: find each sentence with "shall", "must" or "required" and make it a row. Those rules find the easy requirements. The rest are written in shapes the rules can't see.
EXAMPLE 1: the obligation is only in the lead-in
"The proposal shall describe: (a) the staffing plan; (b) the transition approach; (c) a complaint feedback system."
Three requirements, but "shall" appears once, before the list. A keyword rule makes one row for the lead-in, or a row for each item with the obligation stripped off. A reader who sees "(c) a complaint feedback system" on its own can't tell it is mandatory.
EXAMPLE 2: no trigger word at all
"Provide a draft Quality Control Plan within 15 days of award."
An instruction with no "shall" in it. Table rows and questionnaire cells work the same way: a security questionnaire with 200 must-answer questions may contain the word "must" once, in its cover note.
EXAMPLE 3: the binding words are in a column
A specification table ranks each line R (required), I (important) or N (nice to have).
Every line reads alike. Only the letter in the column says which lines are mandatory, and a rule that reads sentences never looks at it.
On our test documents, keyword rules missed 367 mandatory requirements with no trigger word in them and 166 whose trigger was only in a lead-in. Most rows a tool misses follow one of these three patterns.
3. How to measure extraction
A demo on a clean solicitation shows that a tool can find requirements. It doesn't show what share it finds. To know that, you need four things.
An answer key made without the tool's help. Our key covers 26 real solicitation documents: US federal, state and local, non-US, scanned PDFs, spreadsheet questionnaires and one SAP Ariba event. Two labellers worked from the same written guide without seeing each other's work, and without seeing any tool's output. They came from different AI model families. Every disagreement was resolved against the guide, and a person checked a random sample.
A noise floor. Two careful labellers don't agree perfectly, and a difference between tools smaller than the labellers' own disagreement is not a finding. Ours agreed on about nine requirements in ten. The person checking the sample judged 53 of 60 key items right. All seven wrong ones were section titles or list lead-ins, not obligations.
Held-back documents. 19 of the 26 documents were never looked at while adjusting anything. A tool tuned on the documents it is scored on looks better than it is.
A pass mark set before the results. Decide what counts as good enough before reading any score. Otherwise the pass mark drifts towards whatever was measured.
Then score four things:
- Recall on mandatory requirements, overall and by type of document. An average can hide a type the tool barely reads.
- Precision. A matrix padded with headings and buyer statements costs reading time.
- Mandatory flag accuracy. A must-answer question marked advisory is a miss that looks like a find.
- Word-for-word text. Every row should quote the solicitation exactly, and code should confirm the quote is in the document. A tool that rewords a requirement can change what it says.
4. What we measured
On the 19 held-back documents (805 mandatory requirements, three runs each):
| Method | Mandatory recall | Precision |
|---|---|---|
| Keyword rules alone | 50% | 73% |
| Keyword rules, then an AI pass | 80% | 82% |
The AI pass reads the document alongside the rules' rows. It keeps or drops each row, repairs rows that were split or merged by mistake, and adds the requirements the rules missed. Code then keeps only the rows whose text appears word for word in the document.
The gain is largest where the rules are weakest. These document types had the same requirements in both readings:
| Document type | Keyword rules | With the AI pass |
|---|---|---|
| Tables | 39% | 89% |
| Scanned documents | 40% | 81% |
| Statements of work | 51% | 80% |
| Combined federal solicitations | 78% | 90% |
The mandatory flag was right on 98% of the requirements found. Every row shown quoted the document exactly. The median cost was about 23 cents of AI use per opportunity.
5. Four findings that apply to any tool
A hidden row counts as a miss. A common design scores each row and hides the low scorers to keep the matrix tidy. When an AI scored and hid rows from our keyword rules, about 31 real mandatory requirements per run disappeared, and precision rose 6.5 points. We tried other cut-offs, and none kept the precision while bringing back the lost requirements. A scorer should count a hidden row as missed, and a tool that hides rows should say how many and show them on request. Marking a row as uncertain is better than hiding it.
Ask the AI to quote, and have code check every quote. When the AI extracted requirements on its own, without the keyword rules, both models we measured reworded some of them even though they were told to quote, and some rewordings included amounts that are not in the document. With the rules' rows in front of it, a model still sometimes returned a sentence with a page header or footer spliced into it. A check in code that replaces a broken quote with the exact source text, and drops any quote it can't find, made every shown row word for word. A dropped row counts against recall, so the check can't hide a miss.
A cheaper model can return fewer rows that are all correct. We tested a newer, cheaper model against the one then in use, on the same documents in the same run. Its precision and flag accuracy held, and the cost per opportunity fell from 22 cents to 4. Its mandatory recall fell from 77% to 59%, because it added about half as many requirements of its own. Its rows were as accurate as the other model's, so reading its output would not show the problem. Only recall shows it. For the same model, a prompt change that scored 87% on our seven practice documents scored 70% on the 19 held-back ones. The practice documents didn't predict the unseen ones.
How the answer is sent can change the answer. Two versions with identical instructions differed only in the format the model returned. One format marked the must-answer questions in one questionnaire as advisory in all three runs, and the other didn't. If a tool changes anything about how it calls the model, the change needs a new measurement.
6. What this doesn't settle
80% is not 100%. One mandatory requirement in five is still missed. An AI-built matrix is a starting point that a person checks against the solicitation, especially the instructions and evaluation sections. In BidAngel every extracted row stays a suggestion until a person approves it.
Some documents stay hard. Non-US documents scored about a third, with or without the AI pass. Unusual layouts and legal drafting styles read badly to both methods.
19 documents settle large gaps, not small ones. A 30-point difference between methods is a finding. A few points between two document types is not.
The labellers and the tools are related. Most labelling passes used Claude models, and the AI pass runs on Claude Haiku. That could favour it. The second labeller, from a different model family, and the human check are the guards against that bias. Removing every key item of the kind the human check found wrong moved the gain by less than two points.
The held-back set has now been read many times. Each reading is a chance to tune towards it. Before any further prompt changes, we will add new held-back documents that nobody has read.
7. How BidAngel handles it
Keyword rules run on every document. When someone starts qualifying an opportunity, or asks for more requirements, the AI pass described above runs on top of them. Rows the AI pass adds are marked as found by AI. Rows from the rules that it drops are set aside, not deleted: the page says how many there are, and one click shows them. When the AI pass doesn't run, rows the system is unsure of are listed and marked low confidence, not hidden. Every row quotes the solicitation, and a person approves each one before it counts. When a workspace runs the AI pass on a model we haven't measured, the requirements page says so.
The same rules apply to us when anything changes. A new model or a change to the prompt is measured on the held-back documents against the version in use, and it ships only if it meets a set of thresholds written down before the reading.
8. Questions for any tool that builds your matrix
Ask these of any tool before its matrix reaches your bid team, ours included:
- What share of mandatory requirements does it find, measured on documents it wasn't tuned on?
- Against what answer key, and how well did two independent labellers agree on it?
- Does it find requirements with no "shall" in them: instructions, table rows, questionnaire cells, items under a lead-in?
- How does recall break down by type of document? An average hides the type the tool can't read.
- Does it hide rows it is unsure of, or show and mark them?
- Is every row word for word from the solicitation, and does code check that?
- When the vendor changes model or prompt, what is measured again, and what result would stop the change?
A tool that answers all seven well can still miss requirements. You'll know roughly how many, and where they're likely to be.
See how BidAngel builds a compliance matrix you can check against the solicitation.
Request Demo