Skip to content

Add verification suite - #124

Open
kuhe wants to merge 1 commit into
developfrom
kuhe/chore/more-static-analysis
Open

kuhe wants to merge 1 commit into
developfrom
kuhe/chore/more-static-analysis

Conversation

@kuhe

@kuhe kuhe commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

As with smithy-lang/smithy-python#898, this adds a static (loosely defined) analysis verification suite.

FAQ: this is a lot of duplication of code. Won't it drift?

Yes, but it's ok. It's not runtime code, and it allows this code to differ intentionally from smithy-python's verification suite when needed. It avoids the creation of a package containing this code to be publicly maintained and accept slight variations between the two repositories just for DRY.

It allows the code to be edited in the same PR that changes any of the assumptions.


Note: the PR does not fix the violations revealed by the suite. That's for later.
Approving the PR doesn't even necessarily mean you accept all the verifications up front. Their enforcement PRs will be reviewed separately and also present opportunities to tweak the rules.

@kuhe
kuhe marked this pull request as ready for review November 10, 2026 16:37
@kuhe
kuhe requested a review from a team as a code owner November 10, 2026 16:37
@kuhe
kuhe force-pushed the kuhe/chore/more-static-analysis branch 3 times, most recently from 85e5ab9 to 7c6dfa3 Compare November 10, 2026 20:05
@kuhe
kuhe force-pushed the kuhe/chore/more-static-analysis branch from 7c6dfa3 to f73d639 Compare November 10, 2026 20:10
},
)
lines = checked.stdout.splitlines()
return [line for line in lines if " - error: " in line]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If pyright ever fails with exit code 2–4, the failure goes unnoticed with the current check. Should we also verify the exit code?

roots = [str(p.src) for p in packages()]

def one(name: str) -> tuple[str, dict[str, Any]]:
proc = run([python, str(PROBE), name, *roots], timeout=300)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens if a probe runs for more than 300 seconds? Would that crash the whole check?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think individual checks should be considered in error (either code is wrong or the check is wrong) if they run for that long.

For legitimately slow things like typechecking a huge client, exceptions should be made.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants