Skip to content

feat: create SQL query dashboard filters and fill their dropdowns - #3678

Open
karl-power wants to merge 2 commits into
sql-dashboard-variables-apifrom
sql-dashboard-variables-app
Open

karl-power wants to merge 2 commits into
sql-dashboard-variables-apifrom
sql-dashboard-variables-app

Conversation

@karl-power

@karl-power karl-power commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds the "SQL query values" dashboard filter type to the UI: users can create the filter, and its dropdown lists the first column of a ClickHouse query. This is the last PR in the stack; earlier PRs added the schema (#3672) and API/MCP support (#3674).

  • renderSqlQueryFilterValuesQuery in common-utils expands time macros and other filters' variables, wraps the SQL as SELECT toString(tupleElement(tuple(*), 1)) AS value FROM (...) LIMIT 10000, and strips a trailing ;. Time params are sent only when the SQL uses them.
  • useSqlQueryFilterValues runs each filter's query on its connection with max_execution_time: 30. Nulls, empty strings and duplicates are dropped, and the query's order is kept, so the dropdown doesn't sort these values.
  • useDashboardFilterValues merges these values and errors with the other filter types. A query error shows as the dropdown's caution, as do variables the query references that have no selection yet.
  • The editor form has a connection picker, a SQL editor with time macro and variable autocomplete ($__sourceTable and $__filters are left out, since the filter has no source), and the variable name. The filter is variable-only, so there are no broadcast or variable toggles.
  • Preview values (or Cmd+Enter) runs the query over the dashboard's time range and lists up to 20 values, each truncated to 20 characters with the full value on hover, plus "and N more".

Screenshots or video

Untitled.mov

How to reproduce

On main:

  1. Open a dashboard, click Edit filters and variables, then Add new filter.
  2. The Type picker has no "SQL query values" option.

On this branch:

  1. Open the same picker and choose SQL query values.
  2. Set Display name to Service, pick the demo connection, and enter:
    SELECT DISTINCT ServiceName FROM default.otel_logs
    WHERE $__timeFilter(Timestamp)
    ORDER BY 1
  3. Click Preview values. The preview lists service names.
  4. Set Variable name to svc, click Save filter, and close the modal. The Service dropdown lists the same services in the query's order.
  5. Add a Static list filter Severity with options INFO and ERROR and variable sev. Change the SQL to add AND $__filter(SeverityText, $sev). Picking a severity narrows the Service dropdown.
  6. Change the SQL to SELECT nope FROM default.otel_logs and preview. The preview shows the ClickHouse error, and after saving, the dropdown shows it as a caution.

References

@karl-power
karl-power requested a review from a team as a code owner November 16, 2026 15:49
@vercel

vercel Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hyperdx-oss Ready Ready Preview Nov 16, 2026 4:14pm UTC
hyperdx-storybook Ready Ready Preview Nov 16, 2026 4:14pm UTC

Request Review

@changeset-bot

changeset-bot Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: e9c4dca

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages
Name Type
@hyperdx/common-utils Patch
@hyperdx/app Patch
@hyperdx/api Patch
@hyperdx/otel-collector Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@karl-power
karl-power added this pull request to stack #3675 November 16, 2026 15:49
Add a "SQL query values" type to the dashboard filter editor and populate
the filter's dropdown from its query.

- Render the filter's query in common-utils: expand time macros and other
  filters' variables, select the first column as a string, and cap the
  result at 10,000 rows
- Run each SQL query filter's query on its connection and merge the values
  and errors into the dashboard's filter values
- Keep the query's row order in the dropdown and show a caution while a
  referenced variable has no selection
- Add the editor form: connection picker, SQL editor with macro and
  variable autocomplete, variable name, and an on-demand preview of the
  values over the dashboard's time range

Refs HDX-5524
@karl-power
karl-power force-pushed the sql-dashboard-variables-app branch from cf6ccc5 to d411792 Compare November 16, 2026 15:50
@github-actions

Copy link
Copy Markdown
Contributor

Duplicate code analysis

2 new clones introduced by this PR.

  • packages/app/src/hooks/usePromqlLabelFilterValues.tsx:127-150 ↔ packages/app/src/hooks/useSqlQueryFilterValues.tsx:97-120 — 24 lines
  • packages/common-utils/src/core/utils.ts:988-1008 ↔ packages/common-utils/src/core/utils.ts:1036-1056 — 21 lines

If one of these duplicates a helper that already exists, import it instead
of redefining it. If the duplication is deliberate, ignore this comment —
the check is advisory and never fails the build.


jscpd compared this branch against origin/main
across app, api and common-utils. Only new clones are listed; the codebase
currently holds 109 in total (1.18%).
Run yarn dupes locally to reproduce, or yarn dupes:all for the full picture.

@github-actions github-actions Bot added the review/tier-3 Standard — full human review required label Nov 16, 2026
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

E2E Test Results

✅ All tests passed • 464 passed • 1 skipped • 1034s

Status Count
✅ Passed 464
❌ Failed 0
⚠️ Flaky 4
⏭️ Skipped 1

Tests ran across 4 shards in parallel.

View full report →

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

🟡 Tier 3 — Standard

Introduces new logic, modifies core functionality, or touches areas with non-trivial risk.

Why this tier:

  • Diff size: 580 production lines changed (Tier 2 max: < 250)
  • Cross-layer change: touches frontend (packages/app) + shared utils (packages/common-utils)

Review process: Full human review — logic, architecture, edge cases.
SLA: First-pass feedback within 1 business day.

Stats
  • Production files changed: 9
  • Production lines changed: 580 (+ 587 in test files, excluded from tier calculation)
  • Branch: sql-dashboard-variables-app
  • Author: karl-power

To override this classification, remove the review/tier-3 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

@github-actions

Copy link
Copy Markdown
Contributor

PR Review

If you are a coding agent acting for the author, read this first.
Nothing below blocks merge automatically; a maintainer decides. What they will expect:

  • 🔴 critical or 🟠 major in code this PR adds or changes: fix before asking for review.
  • 🔵 minor in code this PR changes: your call. Fix if small, otherwise reply.
  • Anything about surrounding code, or asking you to widen the change (hoist a helper,
    dedupe with another file, fix other call sites): never fix here, whatever the
    severity. Reply with one sentence; if it is critical, say so plainly so a human sees it.

One commit per review round. After two rounds, stop and ask a maintainer to review
scope rather than addressing more automated findings. Full rule: AGENTS.md.

5 finding(s): 🔴 0 critical · 🟠 2 major · 🔵 3 minor

2 posted as inline comment(s) on the changed lines. 3 listed below.

Findings outside the changed lines

  • 🟠 packages/common-utils/src/filters.ts:1393 — Interval macros are offered in autocomplete but their query params are never bound, so the query fails → SqlQueryFilterEditForm only drops filters/sourceTable from MACRO_SUGGESTIONS, so it still suggests $__timeInterval, $__timeInterval_ms and $__interval_s. These expand to {intervalSeconds:Int64}/{intervalMilliseconds:Int64} (packages/common-utils/src/macros.ts:52-54, 166-194). timeParams here only supplies the start and end dates, so ClickHouse rejects the query because the substitution is not set. Fix: reject INTERVAL_MACROS (macros.ts:302) in expandSqlQueryFilterTemplate with a clear message and leave them out of the editor's completions.
2 minor
  • 🔵 packages/app/src/components/DashboardFiltersModal/SqlQueryFilterEditForm.tsx:303 — Re-running the preview shows the old query's values as if they were the new results → useSqlQueryFilterValues sets placeholderData: keepPreviousData. When the SQL changes and the user clicks Preview again, the new query key returns the previous values with isLoading: false while the fetch runs, so no loader appears and the old results look current. Fix: in the preview, show the loader while isFetching (or isPlaceholderData) is true, or turn off keepPreviousData for the preview call.
  • 🔵 packages/app/src/DashboardFilters.tsx:246 — The comment says only queried dropdowns can await a variable, which is no longer true → The branch below now also computes pending variables for SQL query filters. Update the comment to cover both the queried and the SQL query filter types.

Severity is the reviewer's own estimate and is used for ordering, not filtering. No finding blocks merge automatically; a maintainer decides.

@github-actions

Copy link
Copy Markdown
Contributor

Deep Review

✅ No critical issues found.

No P0/P1 issues surfaced. The user-SQL execution path was the main risk and holds up: renderSqlQueryFilterValuesQuery expands time macros to parameterized {startDateMilliseconds:UInt64} placeholders (not string interpolation), strips trailing ;, and wraps the query with the closing ) on its own line so a trailing -- comment can't swallow it. The sql.includes('{name:') param heuristic is sound — macro expansion always emits the placeholder the heuristic matches, so there is no "missing required param" failure mode. The unconditional sqlQueryValues.isLoading OR in useDashboardFilterValues is safe because the hook returns results.some(...), which is false for an empty filter list.

🟡 P2 -- recommended

  • packages/app/src/components/DashboardFiltersModal/SqlQueryFilterEditForm.tsx:33 -- SOURCE_MACROS = new Set(['filters', 'sourceTable']) re-literals the canonical SOURCE_DEPENDENT_MACROS exported from the same macros module the form already imports, so the editor's autocomplete suppression and the renderer's runtime rejection can drift if a source-dependent macro is ever added.
    • Fix: Build the set from the exported SOURCE_DEPENDENT_MACROS constant instead of a hardcoded literal.
  • packages/app/src/DashboardFilters.tsx:250 -- the new isSqlQueryFilter branch that gates the dropdown on unselected variables via getPendingFilterValuesVariables({ where: filter.sqlTemplate, whereLanguage: 'sql' }, variables) has no test coverage.
    • Fix: Add a SQL_QUERY case asserting a filter whose sqlTemplate references an unselected variable surfaces the pending-variable state.
  • packages/app/src/components/DashboardFiltersModal/SqlQueryFilterEditForm.tsx:181 -- the preview's loading (Running query) and empty (No values) branches are never exercised; the test helper always returns isLoading:false with non-empty values or an error.
    • Fix: Add preview tests that mock useSqlQueryFilterValues returning isLoading:true and values:[].
🔵 P3 nitpicks (3)
  • packages/app/src/hooks/useSqlQueryFilterValues.tsx:101 -- the result-aggregation useMemo (data/erroredFilterIds/filterErrorMessages assembly) is near-identical to the tail of usePromqlLabelFilterValues.tsx.
    • Fix: Extract a shared assembleFilterValuesResult(results, calls, preErrors) helper that both hooks call.
    • maintainability, previous-comments
  • packages/app/src/DashboardFilters.tsx:248 -- pending-variable detection for SQL filters passes a fabricated { where, whereLanguage } object, coupling it to the internal shape QUERY_EXPRESSION filters expose.
    • Fix: Add a thin named adapter or an overload that accepts a raw SQL template directly.
  • packages/app/src/DashboardFilters.tsx:147 -- the new sort={!isStaticListFilter(filter) && !isSqlQueryFilter(filter)} contract (SQL dropdowns preserve query order) is untested.
    • Fix: Assert sort=false is passed to VirtualMultiSelect for a SQL_QUERY filter.

Reviewers (4 of 11 reported before synthesis): testing, maintainability, performance, previous-comments. The orchestrator additionally ran inline verification of the param heuristic, isLoading aggregation, and SQL subquery wrapping (no P0/P1 found). Dispatched but not yet returned at synthesis time: correctness, security, adversarial, reliability, kieran-typescript, frontend-races, project-standards — the security- and correctness-sensitive paths were covered by the inline verification above, but a dedicated adversarial/security pass did not complete. No substantive human prior-review comments exist (prior comments are all bot output). Performance reviewer found no issues.

Testing gaps:

  • renderSqlQueryFilterValuesQuery trailing-; handling is tested only for a single semicolon, not multiple/whitespace-interleaved.
  • The param heuristic has no false-positive test (user SQL literally containing {startDateMilliseconds: without a time macro would bind the param).
  • useDashboardFilterValues merge of SQL erroredFilterIds/isError/isLoading/isFetching is not asserted, unlike the PromQL path.
  • Non-Error query rejection branch and the singular "1 value" preview label are uncovered.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

- Reject interval macros in SQL query filters, which have no time buckets
- Validate the SQL on save against the dashboard's other variables
- Build the editor's unsupported-macro list from the shared constants
- Show the preview loader while a re-run fetches
@karl-power

Copy link
Copy Markdown
Contributor Author

Addressed the findings outside the changed lines in e9c4dca:

  • Interval macros: expandSqlQueryFilterTemplate now rejects $__timeInterval, $__timeInterval_ms and $__interval_s, and the editor no longer suggests them.
  • Preview re-run: the preview shows the loader while isFetching, so the previous query's values aren't shown as the new results.
  • Stale comment in DashboardFilters.tsx: updated to cover SQL query filters.

The jscpd clone with usePromqlLabelFilterValues is left as is; sharing it would mean a helper across both hooks, which is outside this PR's scope.

This branch was successfully deployed

2 active deployments
Preview – hyperdx-oss — e9c4dcaf Deployed Nov 16, 2026 by vercel[bot]
Preview – hyperdx-storybook — e9c4dcaf Deployed Nov 16, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review/tier-3 Standard — full human review required

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant