Skip to content

Fail the build when tests don't run or assertions are off - #945

Merged
oryan-block merged 4 commits into
agent-friendlyfrom
agent-friendly-p2-test-trust
Nov 4, 2026
Merged

oryan-block merged 4 commits into
agent-friendlyfrom
agent-friendly-p2-test-trust

Conversation

@oryan-block

@oryan-block oryan-block commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Why

Agents working in graphql-java-tools keep repeating the same mistakes, and the repo should change so a change that looks right from one file is right for the whole repo. Two of those mistakes hide in the test setup. A test class that isn't named *Test never runs and the build still passes. And most of our assertions are Kotlin assert(), including assertEquals in TestUtils, which does nothing when the JVM runs without -ea.

Description

  • surefire now includes every class and fails when no tests run. A class named FooTests or FooSpec runs, and an include that matches nothing fails with No tests were executed! instead of passing with zero tests. No helper or fixture class gets picked up, it's still 440 tests in 46 classes
  • failIfNoTests is set as a property instead of in the plugin config, so the -DfailIfNoTests=false that surefire's error message suggests still turns the check off
  • the Kotlin test compile gets -Xassertions=always-enable, so assert() in the tests runs whatever the runner does with -ea. Test compile only, so the published jar is unchanged
  • removed the surefire-junit4 pin. Its comment said TestNG from reactive-streams-tck makes surefire run none of the JUnit 4 tests, which doesn't happen anymore. Surefire 3.6 runs JUnit 4 on the JUnit Platform and stopped publishing surefire-junit4 after 3.5.6, so the pin kept a 3.5.6 provider inside the 3.6.1 plugin. Without it all 440 still run, and failIfNoTests covers the case the pin was there for

Surefire and IntelliJ both pass -ea by default, so the assert part is cheap insurance more than a fix for something we hit. I didn't move TestUtils onto JUnit's Assert, since about 100 bare assert() calls in the tests would still need the flag.

No release note, nothing changes for library users.

Checklist

  • Pull requests follows the contribution guide
  • New or modified functionality is covered by tests

No new tests since it's build config. Each check was run against the mistake it's for, with mvn -B -ntp clean verify on JDK 25:

  • a ProbeMisnamedTests class with a failing test passes on agent-friendly (440 tests) and fails here (Tests run: 441, Failures: 1)
  • with the include changed to **/*Tests.*, agent-friendly passes with no tests run, this branch fails with No tests were executed!
  • a broken assertEquals and a broken bare assert in DeprecatedDirectiveTest, run with -DenableAssertions=false, both pass without the flag and both fail with it

mvn clean verify passes on JDK 17 and 25 with 440 tests, same as agent-friendly. Build time didn't move noticeably (~11s, single runs).

oryan-block and others added 4 commits November 4, 2026 13:57
A class named FooTests or FooSpec never ran, and an include that matched nothing passed with zero tests.
Without -ea a Kotlin assert() does nothing, so a broken assertEquals from TestUtils or a bare assert passed. The flag is on the test compile only, so the published jar is unchanged.
Surefire 3.6 runs JUnit 4 tests on the JUnit Platform and no longer publishes surefire-junit4, so the pin kept a 3.5.6 provider inside a 3.6.1 plugin. TestNG from reactive-streams-tck no longer hides the JUnit 4 tests: all 440 run without the pin, and failIfNoTests fails the build if a provider ever runs none.
Surefire only reads the failIfNoTests user property when the plugin config leaves the parameter unset, so the literal value ignored the flag its own error message suggests.

Co-Authored-By: Claude <noreply@anthropic.com>
@sonarqubecloud

sonarqubecloud Bot commented Nov 4, 2026

Copy link
Copy Markdown

@oryan-block
oryan-block merged commit 851ffae into agent-friendly Nov 4, 2026
5 checks passed
@oryan-block
oryan-block deleted the agent-friendly-p2-test-trust branch November 4, 2026 19:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant