Skip to content

feat: add per-source row key override for the side-panel row lookup - #3673

Merged
karl-power merged 1 commit into
mainfrom
agent/row-key-expression
Nov 16, 2026
Merged

karl-power merged 1 commit into
mainfrom
agent/row-key-expression

Conversation

@karl-power

@karl-power karl-power commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds an optional "Row key expression" to log and trace sources. When set, the side panel's row lookup selects and filters on only those columns.

For tables like Distributed → Distributed → Merge → SharedMergeTree, the top table reports empty keys, so the lookup falls back to a WHERE on every selected column, including Body, sent to every region.

  • rowKeyExpression added to the log and trace source schemas and the Mongoose model.
  • When set, useConfigWithAdditionalSelect appends the expressions to the select as hidden columns and builds the row WHERE from them and their column references. It skips the table metadata, columns, and MergeTree settings queries.
  • When unset, behaviour is unchanged.
  • New "Row key expression" field in the log and trace source forms.
  • Exposed in MCP clickstack_save_source / clickstack_describe_source and in the external v2 OpenAPI docs.

Backport of the EE change. Only difference: the form fields pass tableConnection/control inline, since OSS has no commonProps helper.

How to reproduce

On main:

  1. Open a row's side panel. The row query's WHERE is built from the detected keys, or from every selected column.

On this branch:

  1. Open the source menu next to the source selector, click "Edit source", then "Configure Optional Fields".
  2. Set "Row key expression" to Timestamp, ServiceName and save.
  3. Open a row's side panel. The row loads, and the row query's WHERE uses only those columns.

How to test on Vercel preview

Preview routes: /search

Steps:

  1. Open the source actions menu next to the source selector and click "Edit source".
  2. Click "Configure Optional Fields".
  3. Verify a "Row key expression" field is shown below "Known Columns List".
  4. Enter Timestamp, ServiceName in "Row key expression" and save the source.
  5. Close the modal and click any row in the results table.
  6. Verify the row side panel opens and shows the row's details.

References

  • Linear Issue: HDX-6001
  • Related PRs: DeploySentinel/hyperdx-ee#4086

@karl-power
karl-power requested a review from a team as a code owner November 16, 2026 11:00
@vercel

vercel Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hyperdx-oss Ready Ready Preview Nov 16, 2026 11:05am UTC
hyperdx-storybook Ready Ready Preview Nov 16, 2026 11:05am UTC

Request Review

@changeset-bot

changeset-bot Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 3bf0c20

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages
Name Type
@hyperdx/common-utils Minor
@hyperdx/api Minor
@hyperdx/app Minor
@hyperdx/otel-collector Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions github-actions Bot added the review/tier-4 Critical — deep review + domain expert sign-off label Nov 16, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🔴 Tier 4 — Critical

Touches authentication, tenancy data models, the public API or shipped database config — or substantially changes the query rendering engine, background tasks, the OTel pipeline, image build, or release CI.

Why this tier:

  • Critical-path files (1) — tenancy, public API, or shipped database config:
    • packages/api/src/routers/external-api/v2/sources.ts
  • Cross-layer change: touches frontend (packages/app) + backend (packages/api) + shared utils (packages/common-utils)

Review process: Deep review from a domain expert. Synchronous walkthrough may be required.
SLA: Schedule synchronous review within 2 business days.

Stats
  • Production files changed: 9
  • Production lines changed: 126 (+ 89 in test files, excluded from tier calculation)
  • Critical-path lines changed: 10
  • Branch: agent/row-key-expression
  • Author: karl-power

To override this classification, remove the review/tier-4 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

@github-actions

Copy link
Copy Markdown
Contributor

Duplicate code analysis

No new duplicate code introduced by this PR.


jscpd compared this branch against origin/main
across app, api and common-utils. Only new clones are listed; the codebase
currently holds 108 in total (1.18%).
Run yarn dupes locally to reproduce, or yarn dupes:all for the full picture.

Add an optional rowKeyExpression to log and trace sources. When set, the row lookup selects and filters on only those expressions (plus their column references) and skips the table metadata, columns, and MergeTree settings queries. This covers sources whose table metadata hides the keys, e.g. Distributed over Merge over SharedMergeTree, where the lookup otherwise falls back to a full-row WHERE including Body. Also exposed in MCP and the external API docs.

Unset sources behave as before.
@karl-power
karl-power force-pushed the agent/row-key-expression branch from 9a87746 to 3bf0c20 Compare November 16, 2026 11:01
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

E2E Test Results

✅ All tests passed • 465 passed • 1 skipped • 990s

Status Count
✅ Passed 465
❌ Failed 0
⚠️ Flaky 3
⏭️ Skipped 1

Tests ran across 4 shards in parallel.

View full report →

@github-actions

Copy link
Copy Markdown
Contributor

PR Review

If you are a coding agent acting for the author, read this first.
Nothing below blocks merge automatically; a maintainer decides. What they will expect:

  • 🔴 critical or 🟠 major in code this PR adds or changes: fix before asking for review.
  • 🔵 minor in code this PR changes: your call. Fix if small, otherwise reply.
  • Anything about surrounding code, or asking you to widen the change (hoist a helper,
    dedupe with another file, fix other call sites): never fix here, whatever the
    severity. Reply with one sentence; if it is critical, say so plainly so a human sees it.

One commit per review round. After two rounds, stop and ask a maintainer to review
scope rather than addressing more automated findings. Full rule: AGENTS.md.

5 finding(s): 🔴 0 critical · 🟠 0 major · 🔵 5 minor

4 posted as inline comment(s) on the changed lines. 1 listed below.

Findings outside the changed lines

1 minor
  • 🔵 packages/app/src/components/__tests__/DBRowTable.test.tsx:1351 — No test checks that the table reads the override from the source → The new tests call useConfigWithAdditionalSelect with a hard-coded string. Nothing covers getRowKeyExpression: reading the field from log or trace sources, ignoring other source kinds, or treating a whitespace-only value as unset. Nothing covers DBSqlRowTableComponent either. Wiring it to the wrong field, or dropping the .trim(), would still pass. Fix: add unit tests for the accessor, plus a test that renders the table with a mocked source that has rowKeyExpression set and checks the query config.

Severity is the reviewer's own estimate and is used for ordering, not filtering. No finding blocks merge automatically; a maintainer decides.

@github-actions

Copy link
Copy Markdown
Contributor

Deep Review

✅ No critical issues found.

The change is well-scoped and additive: when rowKeyExpression is unset the row-lookup path is byte-for-byte unchanged, and when set it short-circuits the metadata/columns/MergeTree-settings queries cleanly. Verified independently:

  • API round-trip is real, not doc-only. rowKeyExpression is a known field in LogSourceSchema/TraceSourceSchema, so toExternalSource (via SourceSchema.safeParse) preserves it on reads, and the external v2 create/update handlers validate input through SourceSchemaNoId — the field flows through both directions, matching the OpenAPI/swagger additions.
  • Hook move is safe. useSource({ id: sourceId }) is relocated earlier in DBSqlRowTableComponent and called exactly once; the later duplicate is removed, hook order stays unconditional, and source remains available to useDenoisedRows.
  • No new injection surface. The expression is raw SQL via SqlString.raw, identical to the sibling source expressions (knownColumnsListExpression, primary/partition keys); source config is admin-level, so this adds no attacker-reachable surface.
  • Type safety is intact — getRowKeyExpression narrows via isLogSource/isTraceSource with no cast, and rowKeyColumns: Set<string> flows typed into useRowWhere's primaryKeyColumns.
🔵 P3 nitpicks (4)
  • packages/app/src/components/DBRowTable.tsx:1630 — the override predicate is expressed twice: hasRowKeyOverride gates the data-fetching hooks while the useMemo re-derives sourceId && rowKeyExpression inline to pick the return branch, so a future change to the predicate must touch both sites or the hooks and the consuming branch silently diverge.
    • Fix: reuse hasRowKeyOverride inside the useMemo branch and add it to the dependency array.
    • maintainability
  • packages/app/src/components/DBRowTable.tsx:1636 — the new Set([...splitAndTrimWithBracket(expr), ...extractColumnReferencesFromKey(expr)]) construction and the { ...config, select, additionalKeysLength, rowKeyColumns } return shape are duplicated between the override branch and the existing metadata branch at line 1684.
    • Fix: extract a small columnNamesFromKeyExpr(expr) helper and share the return assembly.
    • maintainability
  • packages/app/src/components/DBRowTable.tsx:1956 — useRowWhere's primaryKeyColumns prop is now fed rowKeyColumns sourced from a free-form admin expression that need not be the table primary key, widening the gap between the prop name and what it carries.
    • Fix: rename the prop and the rowKeyColumns field to a key-neutral name (e.g. rowIdentifierColumns) in a follow-up.
    • maintainability
  • packages/app/src/components/DBRowTable.tsx:1583 — getRowKeyExpression's whitespace/empty-string → undefined coalescing and the type-guard branch are only exercised indirectly; the new useConfigWithAdditionalSelect tests pass rowKeyExpression explicitly and never drive the helper itself.
    • Fix: add a direct unit test for getRowKeyExpression covering the trim-to-undefined and non-log/trace-source cases.
    • kieran-typescript, testing

Reviewers (2): maintainability, kieran-typescript — plus orchestrator verification of the correctness, API-contract, and adversarial paths (empty-WHERE robustness, useSource hook relocation, raw-SQL expression surface, external-v2 round-trip).

Testing gaps:

  • No end-to-end assertion that the override rowKeyColumns actually produce a WHERE that excludes Body through useRowWhere — tests verify the rowKeyColumns set contents (including Body absence) but not the resulting clause.
  • A misconfigured rowKeyExpression (columns that do not uniquely identify a row, or whose rendered result-column name diverges from the raw expression for map/function forms) is unvalidated and can surface the wrong row in the side panel; this is admin-triggered with no concrete reproducing path in the diff, so it is a residual robustness risk rather than a finding.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

@karl-power
karl-power merged commit 743c3c1 into main Nov 16, 2026
29 checks passed

This branch was successfully deployed

2 active deployments
Preview – hyperdx-oss — 3bf0c20e Deployed Nov 16, 2026 by vercel[bot]
Preview – hyperdx-storybook — 3bf0c20e Deployed Nov 16, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review/tier-4 Critical — deep review + domain expert sign-off

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants