Skip to content

feat(api): support ad hoc dashboard filters in the external API - #3659

Open
knudtty wants to merge 7 commits into
aaron/adhoc-follow-upfrom
aaron/adhoc-external-api
Open

knudtty wants to merge 7 commits into
aaron/adhoc-follow-upfrom
aaron/adhoc-external-api

Conversation

@knudtty

@knudtty knudtty commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #3658.

Summary

The external v2 dashboards API (/api/v2/dashboards) can now create, read, and update dashboards with ADHOC filters. Until now it hid them on GET and rejected them on POST/PUT. PromQL external API support is already on main, so PromQL ad hoc filters (sourceType: "promql") are included alongside SQL ones.

  • Schema: ADHOC is a new variant in the external filter schemas, with fields named as stored (sources, not sourceIds) so no translation is needed. Source IDs must be valid ObjectIds. As with STATIC_LIST, isBroadcastEnabled and isVariableEnabled each have one allowed value (true), so callers can omit them.
  • Saved values: savedFilterValues accepts and returns adhoc entries, which hold a filter's conditions.
  • Validation: every ID in sources and appliesToSourceIds must exist and be a kind the filter's sourceType supports: Log, Trace, or Session for sql, PromQL for promql. The internal and external APIs now share one helper (utils/filterSources.ts) for this check, for both ADHOC and PROMETHEUS_LABEL filters. On PUT, references a stored filter with the same ID already made are skipped, so a source that was later deleted or changed kind does not block a GET → PUT round trip. PROMETHEUS_LABEL filters with a missing or non-PromQL source now get the shared error message, "The following filter source IDs do not exist or are not a kind of source the filter supports: …", instead of "PROMETHEUS_LABEL filters require a PromQL source…" or "Could not find the following source IDs…".
  • MCP: unchanged. Its read paths pass a new hideAdhoc option to convertToExternalDashboard, and its input schemas are separate. MCP support is the next PR in this stack.
  • OpenAPI: adds the AdhocFilterInput, AdhocSavedFilterValue, and AdhocFilterCondition schemas and regenerates openapi.json.

How to reproduce

Starting point: yarn dev, with a personal API key from Team Settings and a Logs source ID.

On main:

  1. POST /api/v2/dashboards with filters: [{ "type": "ADHOC", "name": "Conditions", "sourceType": "sql", "sources": ["<logs-source-id>"] }] returns 400.
  2. GET on a dashboard that already has an ad hoc filter, created in the UI, returns filters: [].

On this branch:

  1. The same POST returns 200, and the filter comes back with isBroadcastEnabled: true and isVariableEnabled: true.
  2. Adding savedFilterValues: [{ "type": "adhoc", "name": "Conditions", "conditions": [{ "key": "ServiceName", "operator": "IN", "value": "api" }] }] stores and returns the conditions. Opening the dashboard shows them in the filter input.
  3. Pointing sources at a PromQL source while keeping sourceType: "sql" returns 400 with that source ID in the message.
  4. GET followed by PUT of the response body round-trips unchanged.

How to test on Vercel preview

N/A. API-only change.

Testing

  • make dev-int FILE=src/routers/external-api/__tests__/dashboards.int.test.ts passes, and covers GET, POST for SQL and PromQL, a GET → PUT round-trip, schema rejections, and source-kind rejections.
  • make dev-int FILE=src/mcp/__tests__/dashboards passes, so MCP behavior is unchanged.
  • New unit test for convertToExternalDashboard({ hideAdhoc }).

References

The v2 dashboards API now creates, reads, and updates ADHOC filters (SQL and
PromQL source types) and their adhoc saved conditions. Sources and
appliesToSourceIds must exist and match the filter's sourceType. MCP keeps
hiding ad hoc filters via convertToExternalDashboard's hideAdhoc option.
@knudtty
knudtty requested a review from a team as a code owner November 13, 2026 21:28
@vercel

vercel Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hyperdx-oss Ready Ready Preview Nov 16, 2026 3:29pm UTC
hyperdx-storybook Ready Ready Preview Nov 16, 2026 3:29pm UTC

Request Review

@changeset-bot

changeset-bot Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 6953c12

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages
Name Type
@hyperdx/api Minor
@hyperdx/app Minor
@hyperdx/otel-collector Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

* Returns an error message if an ad hoc filter reads from or applies to a
* source of a kind its `sourceType` can't query, or null if none does.
*/
function getAdhocFilterSourceKindError(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — Ad hoc source-kind check copies the internal API's and drops its skip for unchanged references, so GET → PUT can fail

getFilterSourceIds and getAdhocFilterSourceKindError repeat getFilterSourceRefs/validateFilterSources in packages/api/src/routers/api/dashboards.ts:68-109. The internal copy skips references the stored filter with the same ID already had, so a source that was later deleted or changed kind doesn't block unrelated saves. This copy checks every ADHOC filter on each PUT (dashboards.ts:3738). So if a UI-created ad hoc filter points at a source that has since been deleted or changed kind, a verbatim GET → PUT now returns 400. That contradicts the PR's claim that "the internal API enforces the same rule". Fix: move the ref-building and existing-ref skip into one shared helper, for example in common-utils filters or packages/api/src/utils/, and pass existingDashboard.filters from the PUT path, as the heatmap and formula gates already do with existingTiles.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

PR Review

If you are a coding agent acting for the author, read this first.
Nothing below blocks merge automatically; a maintainer decides. What they will expect:

  • 🔴 critical or 🟠 major in code this PR adds or changes: fix before asking for review.
  • 🔵 minor in code this PR changes: your call. Fix if small, otherwise reply.
  • Anything about surrounding code, or asking you to widen the change (hoist a helper,
    dedupe with another file, fix other call sites): never fix here, whatever the
    severity. Reply with one sentence; if it is critical, say so plainly so a human sees it.

One commit per review round. After two rounds, stop and ask a maintainer to review
scope rather than addressing more automated findings. Full rule: AGENTS.md.

3 finding(s): 🔴 0 critical · 🟠 0 major · 🔵 3 minor

2 posted as inline comment(s) on the changed lines. 1 listed below.

Findings outside the changed lines

1 minor
  • 🔵 packages/api/src/utils/filterSources.ts:1366 — The filter.id === undefined branch only runs in the unit test, because the external caller passes an id: '' sentinel instead → Use one way to say "no stored filter". Either keep the FilterWithOptionalId path and stop injecting { id: '', ...filter } in validateDashboardTiles (routers/external-api/v2/utils/dashboards.ts:1845), or drop FilterWithOptionalId and the undefined check. The internal API always sends filters with an ID, since DashboardFilterSchema requires id, and the external API always sends '' or a real ID. So the branch can't be reached in production, and the test 'checks a filter without an ID' covers code no caller uses.

Severity is the reviewer's own estimate and is used for ordering, not filtering. No finding blocks merge automatically; a maintainer decides.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Deep Review

✅ No critical issues found. The change is well-scoped and well-tested. The ADHOC external-filter support reuses the stored field names (no translation), the source-kind validation is correctly consolidated into one shared helper (utils/filterSources.ts) backing both the internal and external APIs, and the GET→PUT round-trip invariant holds: AdhocDashboardFilterSchema omits minSelections/maxSelections/isGlobalRequirement and is .strict(), so the verbatim translateFilterToExternalFilter passthrough cannot leak fields the strict external schema would later reject. Source validation is correctly tenant-scoped via getSources(teamId), and the "skip references the stored filter already made" logic re-checks any changed ref (the refKey includes the required kinds), so a reused filter id cannot smuggle a missing or wrong-kind source past validation.

🟡 P2 — recommended

  • packages/api/src/routers/external-api/v2/utils/dashboards.ts:809 — GET responses now include ADHOC filters and adhoc savedFilterValues that were previously stripped, widening the FilterInput/SavedFilterValue response unions; a client pinned to the old generated schema could fail to parse a dashboard where a UI user added an ad hoc filter.
    • Fix: Confirm the response-shape change is called out in the release notes so consumers on strict generated clients regenerate; no code change needed if intended.
    • api-contract
🔵 P3 nitpicks (4)
  • packages/api/src/utils/__tests__/filterSources.test.ts:1 — Unit tests exercise only the id === undefined and real/new-id conventions, not the empty-string id the external path actually injects via translateExternalFilterToFilter({ id: '', ...filter }); that convention is covered only end-to-end.
    • Fix: Add a getChangedFilterSourceRefs unit case asserting an empty-string id matches no stored filter and is treated as a new reference.
    • maintainability, testing
  • packages/api/src/utils/filterSources.ts:35 — getChangedFilterSourceRefs supports two equivalent "no stored counterpart" conventions (id === undefined and the empty-string id passed by the external caller), and only the former is covered by direct unit tests.
    • Fix: Pick one convention — have the external caller omit id and rely on the undefined branch, or drop that branch and document the empty-string contract.
  • packages/api/src/routers/external-api/v2/utils/dashboards.ts:808 — The two parallel hideAdhoc ? ... : (x ?? []) ternaries feeding directly into the .filter().map() chain are slightly hard to scan.
    • Fix: Hoist to named locals (const visibleFilters = ..., const visibleSavedValues = ...) before the chain.
  • packages/common-utils/src/filters.ts:1028 — An ADHOC condition key is interpolated verbatim into ClickHouse SQL (toString(${key})) with no escaping; this is parity with the raw-SQL tiles the external API already accepts and executes only against the caller's own team sources, so it is not a new trust boundary (pre-existing).
    • Fix: No change for this PR; re-gate this expression surface (and raw SQL tiles) if a lower-privilege/read-only external key scope is ever introduced.

Reviewers (7): security, api-contract, maintainability, project-standards, agent-native, previous-comments, learnings-researcher. Core correctness and adversarial checks (source-validation refactor, reused-id bypass attempts, empty-string-id create path, GET→PUT round-trip fidelity) were additionally verified directly against the code.

Testing gaps:

  • No test asserts a GET→PUT round-trip of a PromQL-sourceType ad hoc filter (only the SQL round-trip is covered end-to-end).
  • No test asserts an MCP saveDashboard/patchDashboard round-trip preserves a REST-created ADHOC filter rather than dropping it (MCP tests only confirm adhoc is hidden from output); worth adding with the staged MCP follow-up.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

@github-actions

Copy link
Copy Markdown
Contributor

Duplicate code analysis

2 new clones introduced by this PR.

  • packages/api/src/mcp/tools/dashboards/getDashboardTile.ts:44-59 ↔ packages/api/src/mcp/tools/dashboards/queryTile.ts:71-86 — 16 lines
  • packages/api/src/mcp/tools/dashboards/queryTile.ts:64-83 ↔ packages/api/src/mcp/tools/dashboards/queryTiles.ts:170-189 — 20 lines

If one of these duplicates a helper that already exists, import it instead
of redefining it. If the duplication is deliberate, ignore this comment —
the check is advisory and never fails the build.


jscpd compared this branch against origin/main
across app, api and common-utils. Only new clones are listed; the codebase
currently holds 108 in total (1.18%).
Run yarn dupes locally to reproduce, or yarn dupes:all for the full picture.

@github-actions github-actions Bot added the review/tier-4 Critical — deep review + domain expert sign-off label Nov 13, 2026
@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

🔴 Tier 4 — Critical

Touches authentication, tenancy data models, the public API or shipped database config — or substantially changes the query rendering engine, background tasks, the OTel pipeline, image build, or release CI.

Why this tier:

  • Critical-path files (2) — tenancy, public API, or shipped database config:
    • packages/api/src/routers/external-api/v2/dashboards.ts
    • packages/api/src/routers/external-api/v2/utils/dashboards.ts

Review process: Deep review from a domain expert. Synchronous walkthrough may be required.
SLA: Schedule synchronous review within 2 business days.

Stats
  • Production files changed: 14
  • Production lines changed: 573 (+ 382 in test files, excluded from tier calculation)
  • Critical-path lines changed: 204
  • Branch: aaron/adhoc-external-api
  • Author: knudtty

To override this classification, remove the review/tier-4 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

E2E Test Results

✅ All tests passed • 467 passed • 1 skipped • 1314s

Status Count
✅ Passed 467
❌ Failed 0
⚠️ Flaky 1
⏭️ Skipped 1

Tests ran across 4 shards in parallel.

View full report →

… on PUT

The internal and external dashboard APIs now build filter source references
and report bad ones through one helper. The external PUT passes the stored
filters, so an ad hoc filter whose source was later deleted or changed kind
no longer blocks a GET -> PUT round trip.
…r on PUT

Label filters now go through the same filter source check as ad hoc filters,
so references the stored filter already made are skipped and a GET -> PUT
round trip survives a deleted or changed PromQL source. The wrong-kind
appliesToSourceIds test now asserts the source-kind check caused the 400.
/**
* An error naming the referenced sources that are missing or of the wrong
* kind, or null when there are none.
*/

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — getFilterSourceRefError repeats getNonPromqlSourceError, which now has a single subject value

getNonPromqlSourceError (routers/external-api/v2/utils/dashboards.ts:1370) does the same work: map sources by ID, then flag IDs that are missing or the wrong kind. After this PR its only caller passes 'PromQL tiles', so subject never varies. Run the PromQL tile check through getFilterSourceRefError, or a shared getInvalidSourceRefs(refs, sources) with each caller writing its own message. If that's not worth it, at least drop the subject parameter, which no longer varies.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

});

// Field names match the stored filter, so it needs no translation.
const externalAdhocFilterShape = AdhocDashboardFilterSchema.extend({

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — External ADHOC schema accepts where and whereLanguage on sourceType: 'promql' filters, though the docs say they are sql-only

The OpenAPI for AdhocFilterInput.where says 'sql filters only', but externalAdhocFilterShape and validateDashboardFilterFieldGating (common-utils/src/dashboardValidation.ts:242) accept a PromQL filter with a where. That filter is then stored and silently ignored. Reject it with a refinement, or split the schema on sourceType so where and whereLanguage exist only on the sql variant. This is now a public contract, so old payloads will stay valid for good.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

This branch was successfully deployed

2 active deployments
Preview – hyperdx-storybook — 55f2a0b6 Deployed Nov 16, 2026 by vercel[bot]
Preview – hyperdx-oss — 55f2a0b6 Deployed Nov 16, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review/tier-4 Critical — deep review + domain expert sign-off

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant