Repository navigation
366 lines (336 loc) · 18.2 KB
/
Copy pathreview.yml
File metadata and controls
366 lines (336 loc) · 18.2 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
name: Review
# A second pair of eyes, enforced rather than hoped for.
#
# WHY THIS EXISTS. Phase 2 of the lean transformation removed ~24,000 lines
# straight to `main` with nothing reviewing it. That was recorded all along as an
# accepted risk — "one developer, app is stopped" — but the recording was wrong
# about the cause: branch protection existed the whole time with the right
# required checks, and `enforce_admins` was simply false, so every push walked
# past it. The risk was a misconfiguration nobody checked, not a decision.
#
# WHAT IT MUST CATCH, and why a generic code review will not. The failure mode of
# a deletion program is not a bug; it is deleting a control while it looks like
# deleting ceremony. Phase 2 deleted, correctly, a signed approval bundle, two
# cache-generation comparisons and an approval-binding FK — and kept, correctly,
# a live kill switch, a permit state machine and a spend cap. Those decisions
# look identical in a diff. Telling them apart means reading the tree to find
# what still enforces the rule, which is why this uses Claude Code rather than a
# diff-only reviewer: it can grep.
#
# PRECISION OVER RECALL ON THE BLOCKING PATH. A CONTROL_REMOVAL or UNEVIDENCED
# finding fails the job. NOTE findings do not. A reviewer that flags everything
# trains you to click past it, which is a rubber stamp with extra steps — the
# same reasoning as the Fallow audit's `gate: new-only`.
#
# NOT A BOT APPROVAL. GitHub counts a GitHub App review toward
# `required_approving_review_count`. This deliberately does not do that: it
# reports findings and exits non-zero. A red required check is honest; a bot
# approving a bot's work satisfies the rule and protects nothing.
#
# CREDENTIALS. Either works, and the jobs accept whichever is present:
# - `CLAUDE_CODE_OAUTH_TOKEN` — what `/install-github-app` provisions. In use.
# - `ANTHROPIC_API_KEY` — a static API key.
# Best option if you have Anthropic org admin: Workload Identity Federation
# (`anthropic_federation_rule_id` + `anthropic_organization_id`, with the
# `id-token: write` already granted below), which exchanges the workflow's
# GitHub OIDC token for a short-lived credential and leaves no static secret.
#
# THIS FILE CANNOT REVIEW ITS OWN CHANGES. claude-code-action refuses to run
# when the workflow file differs from the copy on the default branch:
#
# Skipping action due to workflow validation: the workflow file must exist
# and have identical content to the version on the repository's default branch
#
# That is a real control, not a bug — otherwise a pull request could rewrite its
# own reviewer to print secrets. The consequences are worth knowing:
# - Until this file is merged to `main`, both reviewers skip on every run.
# - Any future PR that edits this file will skip them again, and must be
# reviewed by a human.
# - A skipped action exits SUCCESS and emits nothing. Both jobs therefore fail
# when their verdict file is absent. That is not defensive programming: the
# security job actually reported green through four runs in which nothing
# was reviewed, which is how the requirement was found.
#
# NOT A REQUIRED CHECK, BY DECISION (2026-09-28): `security` and
# `control-vs-ceremony` stay informational (docs/operations/security-ci-policy.md,
# "The AI review workflow is not a gate"). A bot approving a bot's work protects
# nothing, and a lapsed credential must never block every merge.
on:
pull_request:
types: [opened, synchronize, reopened]
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
permissions:
contents: read
jobs:
# Fail explicitly when no credential is present, rather than letting the
# review jobs skip and leave a green-looking PR that nothing examined.
credentials:
runs-on: ubuntu-latest
steps:
- name: Require review credentials
env:
OAUTH: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
run: |
if [ -z "${OAUTH}" ] && [ -z "${API_KEY}" ]; then
echo "::error::No review credential is set. The review gate cannot run."
echo "::error::Add CLAUDE_CODE_OAUTH_TOKEN (what /install-github-app"
echo "::error::provisions) or ANTHROPIC_API_KEY under Settings >"
echo "::error::Secrets and variables > Actions."
exit 1
fi
control-vs-ceremony:
needs: [credentials]
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: write
# The action exchanges the workflow's GitHub OIDC token for its App token;
# without this it fails with "Failed to get OIDC token" before it ever
# reaches the model. Also what Workload Identity Federation needs.
id-token: write
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
# Full history so the reviewer can diff against the base and read what
# a deleted line used to do, not merely that it is gone.
fetch-depth: 0
- name: Adversarial control-vs-ceremony review
id: review
uses: anthropics/claude-code-action@db77d4d3c98f3f6380e7cd45ffba2342a3dbde3c # v1
with:
claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
prompt: |
You are an adversarial reviewer on a codebase mid-way through a
deliberate deletion program. Roughly 24,000 lines have already been
removed on purpose. Your job is NOT to object to deletion.
Your job is to find the ONE deletion that removed a CONTROL while
looking like it removed CEREMONY.
The distinction this codebase uses:
- CEREMONY: machinery whose only job is to prove a document, a
catalogue, a counter or a snapshot is consistent with itself.
Deleting it is the point of this program.
- CONTROL: anything that changes what the software will actually
refuse to do. A live authorization check, a spend cap, a kill
switch, a state machine, a tenant predicate, a deadline, an input
bound, a fail-closed branch.
For EVERY removed or weakened check in this diff, answer all four:
1. What did this gate before?
2. What gates it now? Name the file and the line.
3. Is that replacement evaluated LIVE per request, or is it a
stored snapshot compared later? A snapshot is not equivalent
to a live read.
4. If it was a test: did it lose its SUBJECT (the behaviour no
longer exists, so deletion is correct), or did it lose
COVERAGE (the behaviour still exists and is now unasserted,
which is a finding)?
Read the tree to answer 2 and 3. Do not answer from the diff alone;
a diff cannot show you what still enforces a rule. Use grep.
Traps this codebase has already sprung, so look for them:
- An edit reported as applied that silently did not apply. Verify
claims in the commit message against actual file contents.
- A test that passes vacuously because its input was mangled before
assertion. If a test searches text, check the text survived.
- A required field dropped from a type while a SQL function, a
regprocedure cast, a GRANT or a placeholder list still names it.
- A deny code removed from a union while a downstream mapper still
classifies it.
- A guard whose parameter is now unused, which usually means the
guard stopped guarding before it stopped existing.
- A workflow or config whose subject was deleted, leaving a required
check that cannot parse its own input.
OUTPUT. Write your findings to `verdict.json` in the repository
root, and write nothing else. Shape:
{"verdict":"BLOCK"|"PASS",
"findings":[{"classification":"CONTROL_REMOVAL"|"UNEVIDENCED"|"NOTE",
"summary":"...","evidence":"path:line"}]}
Set `verdict` to BLOCK if and only if there is at least one
CONTROL_REMOVAL or UNEVIDENCED finding; NOTE findings alone must
produce PASS. Every finding needs `evidence` naming a file and line
— a finding you cannot point at is a NOTE at most.
Do not hedge. An uncertain BLOCK is more useful than a confident
PASS, but a reflexive BLOCK on every pull request makes this gate
worthless, so justify each one.
# A FILE, not `--json-schema`. The schema route failed with "Claude
# did not return structured_output", and the run behind that message
# had num_turns 1, total_cost_usd 0 and empty modelUsage after 1.9s —
# the CLI died before reaching the model, and the action reports the
# missing output rather than the cause. A file the model writes with a
# tool it demonstrably has is one less moving part, and the dump step
# below now surfaces the underlying error either way.
claude_args: |
--allowed-tools Read,Grep,Glob,Write,Bash(git diff:*),Bash(git log:*),Bash(git show:*)
# Prints why the reviewer failed instead of leaving it in a file on a
# discarded runner. Added because two full cycles were spent guessing at a
# message that described a symptom.
- name: Dump the reviewer execution log
if: always()
env:
EXEC: ${{ steps.review.outputs.execution_file }}
run: |
echo "conclusion: ${{ steps.review.outputs.conclusion }}"
if [ -n "${EXEC:-}" ] && [ -f "${EXEC}" ]; then
echo "--- execution log ---"
tail -c 4000 "${EXEC}"
else
echo "no execution file at '${EXEC:-<unset>}'"
fi
- name: Enforce the verdict
env:
GH_TOKEN: ${{ github.token }}
PR: ${{ github.event.pull_request.number }}
REPO: ${{ github.repository }}
run: |
set -euo pipefail
# Read from the file the reviewer wrote. Two earlier designs failed
# here for reasons worth keeping:
# 1. Scraping "REVIEW-VERDICT:" off the last line of a PR comment.
# The action posts a pull-request REVIEW, not an issue comment,
# so the scraper found nothing. A gate that parses prose fails on
# formatting.
# 2. `--json-schema` + `structured_output`. The action reported the
# output missing while the real failure was upstream of the model
# (one turn, zero cost, 1.9s).
# Absent or unparseable file is still a failure: a review gate with no
# verdict has not reviewed anything.
if [ ! -f verdict.json ]; then
echo "::error::The reviewer wrote no verdict.json."
echo "::error::See the execution log dumped by the previous step."
exit 1
fi
if ! jq -e '.verdict and (.findings | type == "array")' verdict.json >/dev/null 2>&1; then
echo "::error::verdict.json is not the required shape:"
head -c 2000 verdict.json
exit 1
fi
verdict=$(jq -r '.verdict' verdict.json)
echo "verdict: ${verdict}"
# Post the findings so the reasoning lands where a reader sees it,
# not only in a job log.
{
echo "## Control-vs-ceremony review"
echo
echo "**Verdict: ${verdict}**"
echo
jq -r 'if (.findings | length) == 0 then "No findings." else (.findings[] | "- **\(.classification)** — \(.summary)\n - evidence: \(.evidence)") end' verdict.json
} > review-comment.md
gh pr comment "${PR}" --repo "${REPO}" --body-file review-comment.md
# Belt and braces: block on the verdict OR on any blocking finding, so
# a model that mislabels its verdict while correctly classifying a
# finding still stops the merge.
blocking=$(jq '[.findings[] | select(.classification == "CONTROL_REMOVAL" or .classification == "UNEVIDENCED")] | length' verdict.json)
if [ "${verdict}" = "BLOCK" ] || [ "${blocking}" -gt 0 ]; then
echo "::error::Review found ${blocking} blocking finding(s). See the PR comment."
exit 1
fi
security:
needs: [credentials]
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: write
id-token: write
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
fetch-depth: 0
# Kept separate from the reviewer above on purpose: that one asks "did a
# control disappear", this one asks "is there a vulnerability class here".
# Merging them dilutes both prompts.
#
# Uses the same pinned action rather than
# `anthropics/claude-code-security-review`, which publishes no tags.
# `check:action-pins` requires a SHA plus a `# v…` comment and there is no
# version to name — so the options were to widen a security policy for one
# convenience, or not depend on an unversioned action. This program exists
# to stop making the first trade.
- name: Security review
id: review
uses: anthropics/claude-code-action@db77d4d3c98f3f6380e7cd45ffba2342a3dbde3c # v1
with:
claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
prompt: |
Review this pull request for security vulnerabilities only. Ignore
style, naming and architecture — another reviewer covers those.
This codebase handles merchant OAuth credentials for Google Business
Profile and calls a paid AI provider, so weight these classes:
- credential exposure: a token, client secret or encryption key
reaching a log, an error body, a URL, an event payload, a
persisted column or the browser
- authorization bypass: a path reaching a provider or a tenant's
data without re-proving organization, property and capability
- tenant isolation: a query missing its organization predicate
- spend: an AI provider call that can be made without passing the
per-route cost ceiling
- injection: SQL built by concatenation, a shell argument from
request data, a regex from user input
- fail-open: a catch that swallows a refusal, a default that
permits, an optional dependency whose absence skips a check
Report only findings you can evidence with a file and line. For
each, state the input that reaches it and what an attacker gains.
Do not report theoretical issues with no reachable path — a false
positive here costs more than it saves, because it teaches the next
reader to skim.
OUTPUT. Write your findings to `security.json` in the repository
root, and write nothing else. Shape:
{"findings":[{"severity":"HIGH"|"MEDIUM"|"LOW","summary":"...",
"evidence":"path:line","attacker_gains":"..."}]}
Write the file even when you find nothing — an empty `findings`
array is the clean result, and its presence is how this gate tells
"found nothing" apart from "did not run".
claude_args: |
--allowed-tools Read,Grep,Glob,Write,Bash(git diff:*),Bash(git log:*),Bash(git show:*)
- name: Dump the reviewer execution log
if: always()
env:
EXEC: ${{ steps.review.outputs.execution_file }}
run: |
echo "conclusion: ${{ steps.review.outputs.conclusion }}"
if [ -n "${EXEC:-}" ] && [ -f "${EXEC}" ]; then
echo "--- execution log ---"
tail -c 4000 "${EXEC}"
else
echo "no execution file at '${EXEC:-<unset>}'"
fi
# Without this the job is a FALSE GREEN, and it already was one. When
# `review.yml` differs from the copy on the default branch the action
# skips itself by design (see the header), exits success, and produces no
# output — so this job reported green through four runs in which no
# security review happened at all. A gate that cannot tell "found nothing"
# from "did not run" is not a gate.
- name: Enforce the security findings
env:
GH_TOKEN: ${{ github.token }}
PR: ${{ github.event.pull_request.number }}
REPO: ${{ github.repository }}
run: |
set -euo pipefail
if [ ! -f security.json ]; then
echo "::error::The security reviewer wrote no security.json."
echo "::error::Either it was skipped by workflow validation (does"
echo "::error::this PR change .github/workflows/review.yml?) or it"
echo "::error::failed — see the execution log above. Not treating"
echo "::error::silence as a clean bill."
exit 1
fi
if ! jq -e '.findings | type == "array"' security.json >/dev/null 2>&1; then
echo "::error::security.json is not the required shape:"
head -c 2000 security.json
exit 1
fi
{
echo "## Security review"
echo
jq -r 'if (.findings | length) == 0 then "No findings." else (.findings[] | "- **\(.severity)** — \(.summary)\n - evidence: \(.evidence)\n - attacker gains: \(.attacker_gains)") end' security.json
} > security-comment.md
gh pr comment "${PR}" --repo "${REPO}" --body-file security-comment.md
# HIGH and MEDIUM block. LOW is reported and does not.
blocking=$(jq '[.findings[] | select(.severity == "HIGH" or .severity == "MEDIUM")] | length' security.json)
if [ "${blocking}" -gt 0 ]; then
echo "::error::Security review found ${blocking} HIGH/MEDIUM finding(s)."
exit 1
fi