Skip to content

feat: add SQL_QUERY dashboard filter type to the schema - #3672

Open
karl-power wants to merge 4 commits into
mainfrom
sql-dashboard-variables
Open

karl-power wants to merge 4 commits into
mainfrom
sql-dashboard-variables

Conversation

@karl-power

Copy link
Copy Markdown
Contributor

Summary

Adds the SQL_QUERY dashboard filter type to the schema: a variable-only filter whose dropdown values come from the first column of a free-form ClickHouse query. This is the first of four PRs; it adds the data model and validation only, so the filter can't be created or used in the UI yet.

  • SqlQueryDashboardFilterSchema stores a connection ID and a sqlTemplate, with isBroadcastEnabled: false and isVariableEnabled: true fixed, since there's no column to broadcast.
  • isSqlQueryFilter type guard.
  • validateDashboardFilterQueries expands the template with empty variable selections and flags $__sourceTable / $__filters (the filter has no source) and any macro or variable expansion error. The SQL itself isn't parsed; ClickHouse reports syntax errors when it runs.
  • Dashboard template export replaces the filter's connection ID with the connection name, as it does for raw-SQL tiles.
  • The external API and MCP treat the type as internal-only, like ADHOC, until they expose it.
  • The filter settings list handles the new type in its exhaustive switch, labelling it "SQL query". It's only visible for a filter saved through the API or a JSON import, since the UI can't create one yet.

How to test on Vercel preview

N/A — non-UI change

References

  • Linear Issue: HDX-5524
  • Related PRs: follow-ups will add API connection validation and MCP / external API support, the dropdown values lookup, and the filter editor form.

@karl-power
karl-power requested a review from a team as a code owner November 16, 2026 10:56
@vercel

vercel Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hyperdx-oss Ready Ready Preview Nov 16, 2026 12:29pm UTC
hyperdx-storybook Ready Ready Preview Nov 16, 2026 12:29pm UTC

Request Review

@changeset-bot

changeset-bot Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: dde72dc

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages
Name Type
@hyperdx/common-utils Patch
@hyperdx/api Patch
@hyperdx/app Patch
@hyperdx/otel-collector Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions github-actions Bot added the review/tier-2 Low risk — AI review + quick human skim label Nov 16, 2026
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

🟡 Tier 3 — Standard

Introduces new logic, modifies core functionality, or touches areas with non-trivial risk.

Why this tier:

  • Cross-layer change: touches frontend (packages/app) + backend (packages/api) + shared utils (packages/common-utils)

Review process: Full human review — logic, architecture, edge cases.
SLA: First-pass feedback within 1 business day.

Stats
  • Production files changed: 6
  • Production lines changed: 248 (+ 202 in test files, excluded from tier calculation)
  • Branch: sql-dashboard-variables
  • Author: karl-power

To override this classification, remove the review/tier-3 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

Add a variable-only dashboard filter whose dropdown values come from the
first column of a ClickHouse query run on a connection.

- Add SqlQueryDashboardFilterSchema and the isSqlQueryFilter guard
- Validate the template's macros and variable references, rejecting
  $__sourceTable and $__filters
- Export the filter's connection as a name in dashboard templates
- Keep the type out of the external API and MCP for now

Refs HDX-5524
@karl-power
karl-power force-pushed the sql-dashboard-variables branch from 989bc97 to f94e9ab Compare November 16, 2026 10:57
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

E2E Test Results

✅ All tests passed • 467 passed • 1 skipped • 1109s

Status Count
✅ Passed 467
❌ Failed 0
⚠️ Flaky 1
⏭️ Skipped 1

Tests ran across 4 shards in parallel.

View full report →

@github-actions

Copy link
Copy Markdown
Contributor

Duplicate code analysis

1 new clone introduced by this PR.

  • packages/common-utils/src/core/utils.ts:988-1008 ↔ packages/common-utils/src/core/utils.ts:1036-1056 — 21 lines

If one of these duplicates a helper that already exists, import it instead
of redefining it. If the duplication is deliberate, ignore this comment —
the check is advisory and never fails the build.


jscpd compared this branch against origin/main
across app, api and common-utils. Only new clones are listed; the codebase
currently holds 108 in total (1.18%).
Run yarn dupes locally to reproduce, or yarn dupes:all for the full picture.

Comment thread packages/common-utils/src/core/utils.ts Resolved
Comment thread packages/api/src/utils/externalApi.ts Resolved
Comment thread packages/common-utils/src/filters.ts Resolved
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

PR Review

If you are a coding agent acting for the author, read this first.
Nothing below blocks merge automatically; a maintainer decides. What they will expect:

  • 🔴 critical or 🟠 major in code this PR adds or changes: fix before asking for review.
  • 🔵 minor in code this PR changes: your call. Fix if small, otherwise reply.
  • Anything about surrounding code, or asking you to widen the change (hoist a helper,
    dedupe with another file, fix other call sites): never fix here, whatever the
    severity. Reply with one sentence; if it is critical, say so plainly so a human sees it.

One commit per review round. After two rounds, stop and ask a maintainer to review
scope rather than addressing more automated findings. Full rule: AGENTS.md.

4 finding(s): 🔴 0 critical · 🟠 1 major · 🔵 3 minor

3 posted as inline comment(s) on the changed lines. 1 listed below.

Findings outside the changed lines

1 minor
  • 🔵 packages/app/src/__tests__/DBDashboardImportPage.test.tsx:348 — New tile↔filter connection propagation effect has no test → The rewritten connection-propagation useEffect (DBDashboardImportPage.tsx ~920-1002) is the riskiest part of this diff. It uses two prev-refs and tile-first precedence. The new tests only cover auto-mapping by name and the unmapped error. Add tests where (a) changing a RawSQL tile's connection updates a SQL_QUERY filter with the same input connection, (b) the reverse, and (c) a dirty filter mapping is not overwritten.

Severity is the reviewer's own estimate and is used for ordering, not filtering. No finding blocks merge automatically; a maintainer decides.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Deep Review

Scope: PR #3672 — adds the SQL_QUERY dashboard filter type (schema + validation + import mapping only). 6 production files, ~248 production lines, across common-utils, app, and api. Base a25e945e.
Intent (author-stated): Data-model and validation groundwork for a variable-only filter whose dropdown values come from a free-form ClickHouse query; the type stays internal-only for the external API/MCP and has no UI editor yet. First of four stacked PRs.
Mode: report-only (read-only; no edits, commits, or artifacts).

✅ No critical issues found. The schema addition is additive and well-tested, exhaustive switches were updated, the internal-only exclusion mirrors the established ADHOC pattern, and prior review comments are all addressed or legitimately deferred. The items below are edge cases, latent/forward-looking contract gaps, and test coverage — none breaks the happy path.

🟡 P2 — recommended

  • packages/api/src/utils/externalApi.ts:268 — getExternalCompatibleFilterValues strips only adhoc-type saved values, but a SQL_QUERY filter is variable-only, so its variable-type saved value survives even though the filter definition is now hidden, leaking an orphaned value referencing a filter the external API/MCP can't see.
    • Fix: also exclude saved values whose variableName belongs to an internal-only SQL_QUERY filter, mirroring the ADHOC value handling.
    • correctness, ce-agent-native-reviewer, api-contract
  • packages/common-utils/src/types.ts:2421 — SqlQueryDashboardFilterSchema.connection uses z.string().min(1), but template export maps an unresolved connection id to ''; unlike raw-SQL tile configs (z.string(), no min), re-importing such a template fails DashboardTemplateSchema.parse for the entire file rather than flagging just the one filter.
    • Fix: relax the exported-template connection to tolerate '' (blocking it at the mapping step like tiles do), or drop/flag the filter on export when its connection can't be resolved.
    • adversarial, correctness
  • packages/app/src/DBDashboardImportPage.tsx:946 — in the rewritten connection-propagation effect, prevFilterConnectionMappingsRef is resynced only inside the filter branch (skipped whenever a raw-SQL tile claims the run), so after auto-map it stays stale and getFieldState().isDirty (value ≠ default) marks every auto-resolved filter changed, causing findIndex to pick the wrong filter and mis-propagate or silently drop a later manual connection edit.
    • Fix: resync both prev-refs on every effect run and detect the user-edited entry from an explicit change signal rather than isDirty, then add a test for tile↔filter cross-propagation. (Note: the correctness reviewer traced the same effect and judged it sound — verify before changing.)
    • julik-frontend-races, adversarial, maintainability
  • packages/api/src/utils/externalApi.ts:260 — the new runtime exclusion of SQL_QUERY from getExternalCompatibleFilters has no test asserting a SQL_QUERY filter is dropped from the external API/MCP read path.
    • Fix: add a unit test feeding a SQL_QUERY (and ADHOC) filter through getExternalCompatibleFilters and asserting it is excluded while compatible filters pass through.
    • testing, project-standards, api-contract, kieran-typescript

🟡 P2 — pre-existing (surfaced, does not block)

  • packages/api/src/routers/external-api/v2/dashboards.ts:3632 — the external-API update path full-replaces filters via convertExternalFiltersToInternal without merging back internal-only filters the caller never saw, so a read-modify-write client silently drops SQL_QUERY filters; this pre-existing ADHOC hazard now reaches SQL_QUERY, which JSON import can create.
    • Fix: in the follow-up that exposes the type, preserve internal-only filters on update by merging with the existing set by id.
    • ce-agent-native-reviewer, api-contract
🔵 P3 nitpicks (3)
  • packages/app/src/DBDashboardImportPage.tsx:1137 — connection!.id in the SQL_QUERY submit branch relies on the superRefine truthiness check, not that the mapped id still resolves; if the connection list is refetched or the connection is deleted between validation and submit, findConnection returns undefined and the import fails as a generic error toast (caught, but non-actionable). Mirrors the existing tile pattern at line 1100.
    • Fix: guard with an explicit undefined check that re-flags the mapping row instead of asserting.
  • packages/common-utils/src/types.ts:2418 — SqlQueryDashboardFilterSchema omits .strict() (unlike AdhocDashboardFilterSchema), so unknown keys are silently stripped rather than rejected on route validation.
    • Fix: add .strict() if unknown-key rejection is intended for this variant, matching the adhoc schema.
  • packages/app/src/DBDashboardImportPage.tsx:909 — pre-existing: the file is ~1500 lines (over the documented 300-line component guideline) and this diff grows the already-complex Mapping connection-propagation effect further.
    • Fix: extract the propagation effect and filter-mapping rows into sibling modules when convenient; out of scope for this schema PR.

Reviewers (11): correctness, testing, maintainability, project-standards, kieran-typescript, api-contract, adversarial, julik-frontend-races, previous-comments, ce-agent-native-reviewer, ce-learnings-researcher.

Testing gaps:

  • No test for connection-propagation across tiles and SQL_QUERY filters sharing an input connection name (the rewritten effect's new branch).
  • No test for the external-API SQL_QUERY exclusion, nor a GET/PUT round-trip verifying the filter is hidden on read and preserved vs. dropped on write.
  • No test for getExternalCompatibleFilterValues behavior on a SQL_QUERY filter's variable-type saved value.
  • No test for export of a SQL_QUERY filter whose connection id is absent (the ?? '' fallback) and the resulting template failing its own .min(1) on re-import.
  • Forward/rollback note: DashboardFilterSchema is a discriminated union and will throw on an unknown SQL_QUERY discriminant under version skew once the type is creatable; filterFormState.ts has no SQL_QUERY branch yet (a trap for the follow-up editor PR). jscpd flags a 21-line clone in core/utils.ts (advisory, pre-existing).

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

@github-actions github-actions Bot added review/tier-3 Standard — full human review required and removed review/tier-2 Low risk — AI review + quick human skim labels Nov 16, 2026
Comment thread packages/common-utils/src/core/utils.ts Resolved
Comment thread packages/api/src/utils/externalApi.ts Resolved

This branch was successfully deployed

2 active deployments
Preview – hyperdx-oss — dde72dcc Deployed Nov 16, 2026 by vercel[bot]
Preview – hyperdx-storybook — dde72dcc Deployed Nov 16, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review/tier-3 Standard — full human review required

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants