Skip to content

feat: accept SQL_QUERY dashboard filters in the API, external API, and MCP - #3674

Open
karl-power wants to merge 3 commits into
sql-dashboard-variablesfrom
sql-dashboard-variables-api
Open

karl-power wants to merge 3 commits into
sql-dashboard-variablesfrom
sql-dashboard-variables-api

Conversation

@karl-power

Copy link
Copy Markdown
Contributor

Summary

Lets the API, external API v2, and MCP save and return SQL_QUERY dashboard filters, and checks their connection on save. This is the second of four PRs; the dropdown still has no values and the UI can't create the filter yet.

  • The internal dashboard route rejects a SQL_QUERY filter whose connection doesn't belong to the team. A connection the stored filter with the same ID already used is skipped, matching how filter sources are checked.
  • The external API and MCP expose the filter with a connectionId field, translated to and from the internal connection. The PR 1 stopgap that hid the type is removed; ADHOC stays hidden.
  • The external API's connection check now includes filter connections, so an unknown one fails with "Could not find the following connection IDs". The source check only reads filters that have a sourceId.
  • MCP returns non-blocking warnings when the filter's SQL uses $__sourceTable or $__filters (the filter has no source), leaves a macro unclosed, or references an unknown variable.
  • MCP tool descriptions and the dashboard prompt describe the filter, including $__filter(<column>, $<variable>) for reading another filter's selection. openapi.json is regenerated.

How to reproduce

Starting point: yarn dev, with the MCP server connected.

On the base branch (#3672):

  1. Call clickstack_save_dashboard with a filter { "type": "SQL_QUERY", "name": "Service", "connectionId": "<connection ID>", "sqlTemplate": "SELECT DISTINCT ServiceName FROM default.otel_logs WHERE $__timeFilter(Timestamp) ORDER BY 1", "variableName": "svc" }.
  2. The call fails schema validation, since the type isn't exposed.

On this branch:

  1. Repeat the call. The dashboard saves and the response includes the filter with connectionId.
  2. Add a raw-SQL tile containing $__filter(ServiceName, $svc) and run clickstack_query_tile with and without variableValues: [{ "name": "svc", "values": ["<service>"] }]. The tile narrows to that service only when a value is selected.
  3. Save with connectionId: "000000000000000000000000". The call fails with "Could not find the following connection IDs".
  4. Save with sqlTemplate: "SELECT 1 FROM $__sourceTable". The dashboard saves and returns a warning that the filter has no source.

How to test on Vercel preview

N/A — non-UI change

References

…d MCP

Validate a SQL query filter's connection on save, and let the external API
and MCP read and write the filter.

- Reject dashboard saves whose SQL_QUERY filter names a connection outside
  the team, skipping a connection the stored filter already used
- Map the filter's connection to connectionId in the external API and MCP,
  and stop hiding the type there
- Check filter connections in the external API's connection validation
- Warn in MCP about source-only macros, unclosed macros, and unknown
  variables in the filter's SQL
- Document the filter in the MCP tool descriptions, prompts, and OpenAPI spec

Refs HDX-5524
@karl-power
karl-power requested a review from a team as a code owner November 16, 2026 12:09
@vercel

vercel Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hyperdx-oss Ready Ready Preview Nov 16, 2026 12:30pm UTC
hyperdx-storybook Ready Ready Preview Nov 16, 2026 12:30pm UTC

Request Review

@changeset-bot

changeset-bot Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: e5c0930

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages
Name Type
@hyperdx/api Patch
@hyperdx/app Patch
@hyperdx/otel-collector Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@karl-power
karl-power added this pull request to stack #3675 November 16, 2026 12:10
@github-actions

Copy link
Copy Markdown
Contributor

Duplicate code analysis

1 new clone introduced by this PR.

  • packages/common-utils/src/core/utils.ts:988-1008 ↔ packages/common-utils/src/core/utils.ts:1036-1056 — 21 lines

If one of these duplicates a helper that already exists, import it instead
of redefining it. If the duplication is deliberate, ignore this comment —
the check is advisory and never fails the build.


jscpd compared this branch against origin/main
across app, api and common-utils. Only new clones are listed; the codebase
currently holds 108 in total (1.18%).
Run yarn dupes locally to reproduce, or yarn dupes:all for the full picture.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

PR Review

If you are a coding agent acting for the author, read this first.
Nothing below blocks merge automatically; a maintainer decides. What they will expect:

  • 🔴 critical or 🟠 major in code this PR adds or changes: fix before asking for review.
  • 🔵 minor in code this PR changes: your call. Fix if small, otherwise reply.
  • Anything about surrounding code, or asking you to widen the change (hoist a helper,
    dedupe with another file, fix other call sites): never fix here, whatever the
    severity. Reply with one sentence; if it is critical, say so plainly so a human sees it.

One commit per review round. After two rounds, stop and ask a maintainer to review
scope rather than addressing more automated findings. Full rule: AGENTS.md.

1 finding(s): 🔴 0 critical · 🟠 0 major · 🔵 1 minor

None could be anchored to changed lines; they are listed below.

Findings outside the changed lines

1 minor
  • 🔵 packages/api/src/routers/api/dashboards.ts:603 — Team-connection lookup now written twice: validateFilterConnections copies getMissingConnections → validateFilterConnections calls getConnectionsByTeam, builds a set of _id.toString() and returns the IDs not in it. getMissingConnections in packages/api/src/routers/external-api/v2/utils/dashboards.ts:1589 does the same thing. Move that step into one shared helper, for example getMissingTeamConnectionIds(teamId, ids) in packages/api/src/controllers/connection.ts, and call it from both places. Each caller keeps its own rule for which IDs to check (the internal route skips connections the stored filter already used).

Severity is the reviewer's own estimate and is used for ordering, not filtering. No finding blocks merge automatically; a maintainer decides.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Deep Review

The change adds SQL_QUERY dashboard filters to the internal API, external API v2, and MCP, with a save-time team-ownership check on the filter's ClickHouse connection and symmetric connection ↔ connectionId translation. I verified the team-isolation path end to end: validateFilterConnections (packages/api/src/routers/api/dashboards.ts) mirrors the existing validateFilterSources skip-existing semantics, existingFilters is sourced from the already-team-scoped stored dashboard, and getConnectionsByTeam(teamId) scopes the lookup — no cross-team bypass is reachable, since no SQL_QUERY filter could have been persisted before this PR. The MCP and external paths re-parse through externalDashboardFilterSchema, which supplies the required isBroadcastEnabled:false / isVariableEnabled:true literals, so persisted internal filters stay schema-valid. Prior review feedback (connection-swap PATCH test, validateSqlQueryFilterTemplate wiring, the 'sourceId' in filter decision) is addressed in the diff.

✅ No critical issues found.

🟡 P2 -- recommended

  • packages/api/src/routers/external-api/__tests__/dashboards.int.test.ts:848 -- The external-API SQL_QUERY suite covers create and GET round-trip but never exercises an update that introduces or swaps a filter connectionId, so the external re-validation path for filter connections is untested even though the internal route has explicit PATCH coverage.
    • Fix: Add an external-API update test that submits a SQL_QUERY filter with a connection outside the team and asserts a 400, plus one that moves it to another team-owned connection and asserts success.
🔵 P3 nitpicks (2)
  • packages/api/src/routers/api/dashboards.ts:124 -- validateFilterConnections and the external getMissingConnections (packages/api/src/routers/external-api/v2/utils/dashboards.ts:1110) implement the same team-ownership check with divergent shapes and messages, a drift risk as filter-connection rules evolve.
    • Fix: Factor the connection-membership check into one shared helper reused by both routes, or track the divergence as a follow-up.
  • packages/api/src/mcp/tools/dashboards/validation.ts:470 -- getFilterVariableWarnings overloads the local where variable to also hold a sqlTemplate, which obscures that two structurally different filter inputs flow through the same validation branch.
    • Fix: Rename the variable to reflect that it is the dropdown-values query regardless of source field, or branch the SQL_QUERY case explicitly.

Reviewers (9): correctness, security, adversarial, api-contract, kieran-typescript, testing, maintainability, previous-comments, project-standards.

Testing gaps: External-API update path for SQL_QUERY filter connections is unverified; internal PATCH create/move/delete-tolerance cases are covered.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

E2E Test Results

✅ All tests passed • 467 passed • 1 skipped • 1025s

Status Count
✅ Passed 467
❌ Failed 0
⚠️ Flaky 1
⏭️ Skipped 1

Tests ran across 4 shards in parallel.

View full report →

@karl-power

Copy link
Copy Markdown
Contributor Author

Review follow-up:

  • Deep Review P2 (connection swap test): added in e5c0930. PATCH to another team-owned connection returns 200.
  • Deep Review P3 ('sourceId' in filter): keeping it. The old !isStaticListFilter check assumed every other filter type has a source, which is what broke when SQL_QUERY was added. The property check doesn't break that way for new types.
  • clickstack_list_sources nextStep text: valid, but it was already wrong for raw SQL tiles and this PR doesn't touch that file. Will fix in a follow-up.
  • Duplicate-code report (common-utils/src/core/utils.ts): not part of this PR's diff against its base (feat: add SQL_QUERY dashboard filter type to the schema #3672); it shows up because the check compares against main.

@github-actions github-actions Bot added the review/tier-4 Critical — deep review + domain expert sign-off label Nov 16, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🔴 Tier 4 — Critical

Touches authentication, tenancy data models, the public API or shipped database config — or substantially changes the query rendering engine, background tasks, the OTel pipeline, image build, or release CI.

Why this tier:

  • Critical-path files (2) — tenancy, public API, or shipped database config:
    • packages/api/src/routers/external-api/v2/dashboards.ts
    • packages/api/src/routers/external-api/v2/utils/dashboards.ts

Review process: Deep review from a domain expert. Synchronous walkthrough may be required.
SLA: Schedule synchronous review within 2 business days.

Stats
  • Production files changed: 9
  • Production lines changed: 376 (+ 237 in test files, excluded from tier calculation)
  • Critical-path lines changed: 97
  • Branch: sql-dashboard-variables-api
  • Author: karl-power

To override this classification, remove the review/tier-4 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

This branch was successfully deployed

2 active deployments
Preview – hyperdx-oss — e5c0930b Deployed Nov 16, 2026 by vercel[bot]
Preview – hyperdx-storybook — e5c0930b Deployed Nov 16, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review/tier-4 Critical — deep review + domain expert sign-off

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant