Lesson 2: Reasoning Reviewer vs Pattern Scanner, and the Limits

Two ways to read code

A rule-based scanner works like a spell checker. It holds a list of known-bad shapes, such as string-built SQL, a call to eval or a hard-coded key, and it flags every place the code matches one. It is fast, cheap and repeatable: same code, same rules, same result.

A reasoning reviewer works more like a colleague doing code review. It reads the function, opens the helpers it calls, checks who calls it, and asks whether anything on the path makes the dangerous thing safe. That is what Claude does when you ask it to review code, and it is why it can find bugs that have no recognisable shape.

The cleanest way to see the difference is a bug made of code that is missing.

The practice bug: a check that isn't there

Add these two files to your practice repository:

# app/routes.py
@app.get('/invoices/<int:invoice_id>')
@login_required                      # proves WHO you are, nothing more
def get_invoice(invoice_id):
    invoice = store.load_invoice(invoice_id)        # helper lives in another file
    return jsonify(invoice)

@app.get('/invoices/<int:invoice_id>/pdf')
@login_required
def get_invoice_pdf(invoice_id):
    invoice = store.load_invoice_for(current_user.id, invoice_id)
    return render_pdf(invoice)

# app/store.py
def load_invoice(invoice_id):
    # parameterised query: no SQL injection here
    return db.one('SELECT * FROM invoices WHERE id = ?', (invoice_id,))

def load_invoice_for(user_id, invoice_id):
    # ownership is enforced inside the query
    return db.one('SELECT * FROM invoices WHERE id = ? AND owner_id = ?',
                  (invoice_id, user_id))

get_invoice is broken. Any signed-in user can read any invoice by changing the number in the URL. This is CWE-639, authorization bypass through a user-controlled key. The minimal reproduction is to sign in as user A and request an invoice id that belongs to user B: you get a 200 where you should get a 404.

Why a regex rule misses it

A typical injection rule looks for SQL assembled from strings:

# flag execute()/one() calls that build SQL with an f-string or concatenation
grep -nE "(execute|one)\((f['\"]|.*\+)" app/*.py
# (no output)

No output is the correct answer for that rule, because the query is parameterised. The bug is a line that was never written, and a pattern cannot match an absence. You could write a rule that says every route must call load_invoice_for, but that rule only works for this one codebase and only after someone already knows the bug class is there.

Fair's fair: modern static analysers do more than regex, and many track tainted data across functions. They still need a rule that describes the flaw. "The ownership check your app's design requires is missing" is very hard to put in a rule.

What a reasoning reviewer does instead

1. Reads get_invoice and sees that invoice_id comes straight from the URL.
2. Notices that @login_required is authentication, not authorization.
3. Opens store.py to see whether load_invoice filters by owner. It does not.
4. Compares the sibling route, which uses load_invoice_for. The app clearly intends to check ownership, so one route forgot.

Steps 3 and 4 are what matter. The reviewer followed the data into another file and looked for the thing that would have made it safe.

The four limits to plan around

The flexibility that finds this bug has costs. Plan for them from the start.

• Results vary between runs. The same code reviewed twice can give different findings, wording or severity. Anthropic's documentation for its deep-scan plugin says plainly that scans are nondeterministic. A finding's absence in one run proves nothing.
• False positives. If the reviewer stops at routes.py and never opens store.py, it may flag get_invoice_pdf too, even though the helper enforces ownership. It sounds confident and it is wrong.
• Misses. A clean answer means "nothing found this time", not "nothing there". The reviewer may not have read the file that mattered.
• No substitute for tests or sign-off. A review is an opinion about code. A test is evidence about behaviour, and a named human approving a change is accountability. Anthropic's guidance positions these tools as a complement to existing scanners and manual review, not a replacement.

Rule of thumb: treat every finding as a hypothesis with a file and line attached. Your job is to confirm or kill it.

Exercise: review one file twice

You will run the same review twice in separate conversations, so the second run cannot see the first.

# 1. Work on a throwaway branch of the practice repo (code you are allowed to scan)
git switch -c l2-review-twice

# 2. Run one: -p runs Claude Code non-interactively and prints the answer
claude -p "Review app/routes.py for security vulnerabilities. Follow calls into other files before deciding. For each finding give file, line, CWE and the evidence you read. Do not change any files." > run1.md

# 3. Run two: the identical prompt, as a new invocation
claude -p "Review app/routes.py for security vulnerabilities. Follow calls into other files before deciding. For each finding give file, line, CWE and the evidence you read. Do not change any files." > run2.md

# 4. Compare
diff run1.md run2.md

Gotcha: do not add --continue to the second command. That flag resumes the previous conversation, so run two would simply agree with run one.

Record what you see. The rows below are illustrative, and yours will differ:

# notes/l2-compare.md
| Item                         | Run 1          | Run 2           | Verified by me                                 |
|------------------------------|----------------|-----------------|------------------------------------------------|
| get_invoice missing owner    | found, High    | found, Medium   | Real: store.load_invoice has no owner filter   |
| get_invoice_pdf same issue   | not reported   | reported        | False positive: load_invoice_for checks owner  |
| CWE cited                    | CWE-639        | CWE-639         | n/a                                            |

Fill in the last column yourself by opening the code. Then write the proof: a test that logs in as one user, requests another user's invoice and expects a 404. It should fail today, and that failing test is your evidence.

Key takeaways

• A pattern scanner matches shapes. A reasoning reviewer follows data through callers, helpers and sanitizers.
• A missing check has no shape, which is why a regex rule reports nothing and a reviewer that reads store.py can catch it.
• Plan for four limits: run-to-run variation, false positives, misses, and no replacement for tests or human sign-off.
• Two fresh runs of the same prompt are a cheap way to see the variation for yourself. Keep the comparison on record.
• A finding is a hypothesis until you have read the code and, ideally, written a failing test.