Repository navigation
feat: add SQL_QUERY dashboard filter type to the schema #3672
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
9796df2
c83b764
0b0a950
369bd81
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| --- | ||
| '@hyperdx/common-utils': patch | ||
| '@hyperdx/api': patch | ||
| '@hyperdx/app': patch | ||
| --- | ||
|
|
||
| feat: add the SQL query dashboard filter type to the dashboard schema | ||
|
|
||
| A `SQL_QUERY` filter stores a ClickHouse connection and a SQL template whose first column fills the filter's dropdown. It is variable-only. Validation flags templates that use `$__sourceTable` or `$__filters`, or whose macros and variable references can't expand. The filter is not yet exposed through the external API or MCP, and has no editor or values lookup yet. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -5,6 +5,7 @@ import { | |
| isFilterBroadcastEnabled, | ||
| isFilterRequired, | ||
| isFilterVariableEnabled, | ||
| isSqlQueryFilter, | ||
| } from '@hyperdx/common-utils/dist/filters'; | ||
| import { | ||
| AlertErrorType, | ||
|
|
@@ -241,7 +242,10 @@ export function translateExternalChartToTileConfig( | |
| } | ||
|
|
||
| /** Filter types the external API and MCP don't expose yet. */ | ||
| type InternalOnlyDashboardFilter = Extract<DashboardFilter, { type: 'ADHOC' }>; | ||
| type InternalOnlyDashboardFilter = Extract< | ||
| DashboardFilter, | ||
| { type: 'ADHOC' | 'SQL_QUERY' } | ||
| >; | ||
| export type ExternalCompatibleDashboardFilter = Exclude< | ||
| DashboardFilter, | ||
| InternalOnlyDashboardFilter | ||
|
|
@@ -253,7 +257,7 @@ export function getExternalCompatibleFilters( | |
| ): ExternalCompatibleDashboardFilter[] { | ||
| return (filters ?? []).filter( | ||
| (filter): filter is ExternalCompatibleDashboardFilter => | ||
| !isAdhocFilter(filter), | ||
| !isAdhocFilter(filter) && !isSqlQueryFilter(filter), | ||
|
karl-power marked this conversation as resolved.
Resolved
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟠 major — MCP and external-API saves silently delete SQL_QUERY filters, breaking tiles that use their variable These reads now hide SQL_QUERY filters, but the writes replace the whole filter array: MCP Nothing blocks merge automatically, but a maintainer will expect this fixed if it is a real defect in code this PR changes. If it is about surrounding code, reply and say so instead of patching. Do not widen the PR. How to respond |
||
| ); | ||
| } | ||
|
|
||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -22,6 +22,7 @@ import { | |
| isAdhocFilter, | ||
| isPrometheusLabelFilter, | ||
| isQueryExpressionFilter, | ||
| isSqlQueryFilter, | ||
| QUERY_EXPRESSION_FILTER_SOURCE_KINDS, | ||
| type SavedFilterValueIssue, | ||
| type SavedQueryIssue, | ||
|
|
@@ -337,10 +338,17 @@ function FileSelection({ | |
| details={ | ||
| <Stack gap={0}> | ||
| {filterQueryWarnings.map(warning => ( | ||
| <Text key={warning.filterId} c="yellow"> | ||
| {warning.filterName}: invalid {warning.language} query:{' '} | ||
| {warning.where} | ||
| </Text> | ||
| <Stack key={warning.filterId} gap={0}> | ||
|
karl-power marked this conversation as resolved.
Resolved
|
||
| <Text c="yellow"> | ||
| {warning.filterName}: invalid {warning.language} query:{' '} | ||
| {warning.where} | ||
| </Text> | ||
| {warning.detail && ( | ||
|
karl-power marked this conversation as resolved.
Resolved
|
||
| <Text c="yellow" size="sm"> | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 minor — New warning detail text uses raw palette color The conventions say to use Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond |
||
| {warning.detail} | ||
| </Text> | ||
| )} | ||
| </Stack> | ||
| ))} | ||
| </Stack> | ||
| } | ||
|
|
@@ -458,6 +466,8 @@ export function buildMappingFormSchema( | |
| connectionMappings: z.array(MappingValueSchema), | ||
| /** A list of filter source mappings, ordered by input filter index */ | ||
| filterSourceMappings: z.array(MappingValueSchema).optional(), | ||
| /** A list of filter connection mappings, ordered by input filter index. Only applicable for SQL query filters */ | ||
| filterConnectionMappings: z.array(MappingValueSchema).optional(), | ||
| /** | ||
| * Per-filter applies-to source mappings. Each entry is an array of mapped | ||
| * source IDs in the same order as the filter's `appliesToSourceIds` names | ||
|
|
@@ -546,6 +556,17 @@ export function buildMappingFormSchema( | |
| return; | ||
| } | ||
|
|
||
| if (isSqlQueryFilter(filter)) { | ||
| if (!data.filterConnectionMappings?.[idx]) { | ||
| ctx.addIssue({ | ||
| code: z.ZodIssueCode.custom, | ||
| path: ['filterConnectionMappings', idx], | ||
| message: 'Select a connection for this filter', | ||
| }); | ||
| } | ||
| return; | ||
| } | ||
|
|
||
| if (!isSourceBackedTemplateFilter(filter)) return; | ||
| const mappedSourceId = data.filterSourceMappings?.[idx]; | ||
| if (!mappedSourceId) { | ||
|
|
@@ -601,6 +622,7 @@ export function Mapping({ input }: { input: DashboardTemplate }) { | |
| tileSourceMappings: input.tiles.map(() => ''), | ||
| connectionMappings: input.tiles.map(() => ''), | ||
| filterSourceMappings: input.filters?.map(() => '') ?? [], | ||
| filterConnectionMappings: input.filters?.map(() => '') ?? [], | ||
| filterAppliesToSourceMappings: input.filters?.map(() => []) ?? [], | ||
| filterAdhocSourceMappings: input.filters?.map(() => []) ?? [], | ||
| onClickSourceMappings: input.tiles.map(() => ''), | ||
|
|
@@ -631,15 +653,20 @@ export function Mapping({ input }: { input: DashboardTemplate }) { | |
| return match?.id || ''; | ||
| }); | ||
|
|
||
| const findConnectionIdByName = (name: string) => | ||
| connections.find(conn => conn.name.toLowerCase() === name.toLowerCase()) | ||
| ?.id || ''; | ||
|
|
||
| const connectionMappings = input.tiles.map(tile => { | ||
| const config = tile.config as SavedChartConfig; | ||
| if (!isRawSqlSavedChartConfig(config)) return ''; | ||
| const match = connections.find( | ||
| conn => conn.name.toLowerCase() === config.connection.toLowerCase(), | ||
| ); | ||
| return match?.id || ''; | ||
| return findConnectionIdByName(config.connection); | ||
| }); | ||
|
|
||
| const filterConnectionMappings = input.filters?.map(filter => | ||
| isSqlQueryFilter(filter) ? findConnectionIdByName(filter.connection) : '', | ||
| ); | ||
|
|
||
| const filterSourceMappings = input.filters?.map(filter => { | ||
| const sourceName = sourceBackedTemplateFilter(filter)?.source; | ||
| if (sourceName == null) return ''; | ||
|
|
@@ -698,6 +725,7 @@ export function Mapping({ input }: { input: DashboardTemplate }) { | |
| setValue('tileSourceMappings', tileSourceMappings); | ||
| setValue('connectionMappings', connectionMappings); | ||
| setValue('filterSourceMappings', filterSourceMappings); | ||
| setValue('filterConnectionMappings', filterConnectionMappings); | ||
| multiMappingSlotsRef.current.clear(); | ||
| filterAppliesToSourceMappings?.forEach((mapping, idx) => { | ||
| setValue(`filterAppliesToSourceMappings.${idx}`, mapping); | ||
|
|
@@ -720,6 +748,10 @@ export function Mapping({ input }: { input: DashboardTemplate }) { | |
| name: 'onClickSourceMappings', | ||
| }); | ||
| const connectionMappings = useWatch({ control, name: 'connectionMappings' }); | ||
| const filterConnectionMappings = useWatch({ | ||
| control, | ||
| name: 'filterConnectionMappings', | ||
| }); | ||
| const onClickDashboardMappings = useWatch({ | ||
| control, | ||
| name: 'onClickDashboardMappings', | ||
|
|
@@ -728,6 +760,7 @@ export function Mapping({ input }: { input: DashboardTemplate }) { | |
| const prevFilterSourceMappingsRef = useRef(filterSourceMappings); | ||
| const prevOnClickSourceMappingsRef = useRef(onClickSourceMappings); | ||
| const prevConnectionMappingsRef = useRef(connectionMappings); | ||
| const prevFilterConnectionMappingsRef = useRef(filterConnectionMappings); | ||
| const prevOnClickDashboardMappingsRef = useRef(onClickDashboardMappings); | ||
|
|
||
| // Propagate source mapping changes to other tiles/filters/onClicks with the | ||
|
|
@@ -883,26 +916,49 @@ export function Mapping({ input }: { input: DashboardTemplate }) { | |
| sources, | ||
| ]); | ||
|
|
||
| // Propagate connection mapping changes to other RawSQL tiles with the same input connection | ||
| // Propagate connection mapping changes to other RawSQL tiles and SQL query | ||
|
karl-power marked this conversation as resolved.
Resolved
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 minor — The rewritten connection-propagation effect has no test This effect now propagates connection picks in both directions, tile→filter and filter→tile, using two separate prev-refs. The new tests only cover auto-mapping by name and the unmapped-connection error. Add a test with a raw-SQL tile and a SQL_QUERY filter that share a template connection: pick a connection on one row and assert the other row follows, and that a row the user already changed (dirty) is left alone. Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond |
||
| // filters with the same input connection | ||
| useEffect(() => { | ||
| if (isUpdatingRef.current) return; | ||
| if (!connectionMappings || !input.tiles) return; | ||
|
|
||
| const changedIdx = connectionMappings.findIndex( | ||
| (mapping, idx) => | ||
| mapping !== prevConnectionMappingsRef.current?.[idx] && | ||
| getFieldState(`connectionMappings.${idx}`).isDirty, | ||
| ); | ||
| if (changedIdx === -1) return; | ||
| if (!input.tiles) return; | ||
|
|
||
| prevConnectionMappingsRef.current = connectionMappings; | ||
| let inputConnectionName: string | undefined; | ||
| let connectionId = ''; | ||
|
|
||
| const tileIdx = | ||
| connectionMappings?.findIndex( | ||
| (mapping, idx) => | ||
| mapping !== prevConnectionMappingsRef.current?.[idx] && | ||
| getFieldState(`connectionMappings.${idx}`).isDirty, | ||
| ) ?? -1; | ||
| if (connectionMappings && tileIdx !== -1) { | ||
| prevConnectionMappingsRef.current = connectionMappings; | ||
| const config = input.tiles[tileIdx]?.config as | ||
| | SavedChartConfig | ||
| | undefined; | ||
| if (config && isRawSqlSavedChartConfig(config)) { | ||
| inputConnectionName = config.connection; | ||
| connectionId = connectionMappings[tileIdx] ?? ''; | ||
| } | ||
| } | ||
|
|
||
| const inputTile = input.tiles[changedIdx]; | ||
| const inputTileConfig = inputTile?.config as SavedChartConfig | undefined; | ||
| if (!inputTileConfig || !isRawSqlSavedChartConfig(inputTileConfig)) return; | ||
| if (inputConnectionName == null && filterConnectionMappings) { | ||
| const filterIdx = filterConnectionMappings.findIndex( | ||
| (mapping, idx) => | ||
| mapping !== prevFilterConnectionMappingsRef.current?.[idx] && | ||
| getFieldState(`filterConnectionMappings.${idx}`).isDirty, | ||
| ); | ||
| if (filterIdx !== -1) { | ||
| prevFilterConnectionMappingsRef.current = filterConnectionMappings; | ||
| const filter = input.filters?.[filterIdx]; | ||
| if (filter && isSqlQueryFilter(filter)) { | ||
| inputConnectionName = filter.connection; | ||
| connectionId = filterConnectionMappings[filterIdx] ?? ''; | ||
| } | ||
| } | ||
| } | ||
|
|
||
| const connectionId = connectionMappings[changedIdx] ?? ''; | ||
| const inputTileConnection = inputTileConfig.connection; | ||
| if (inputConnectionName == null) return; | ||
|
|
||
| const keysForTilesWithMatchingConnection = input.tiles | ||
| .map((tile, index) => ({ | ||
|
|
@@ -912,18 +968,38 @@ export function Mapping({ input }: { input: DashboardTemplate }) { | |
| .filter( | ||
| tile => | ||
| isRawSqlSavedChartConfig(tile.config) && | ||
| tile.config.connection === inputTileConnection, | ||
| tile.config.connection === inputConnectionName, | ||
| ) | ||
| .map(({ index }) => `connectionMappings.${index}` as const); | ||
|
|
||
| const keysForFiltersWithMatchingConnection = | ||
| input.filters | ||
| ?.map((filter, index) => ({ filter, index })) | ||
| .filter( | ||
| ({ filter }) => | ||
| isSqlQueryFilter(filter) && | ||
| filter.connection === inputConnectionName, | ||
| ) | ||
| .map(({ index }) => `filterConnectionMappings.${index}` as const) ?? []; | ||
|
|
||
| isUpdatingRef.current = true; | ||
| for (const key of keysForTilesWithMatchingConnection) { | ||
| for (const key of [ | ||
| ...keysForTilesWithMatchingConnection, | ||
| ...keysForFiltersWithMatchingConnection, | ||
| ]) { | ||
| if (!getFieldState(key).isDirty) { | ||
| setValue(key, connectionId, { shouldValidate: true }); | ||
| } | ||
| } | ||
| isUpdatingRef.current = false; | ||
| }, [connectionMappings, input.tiles, getFieldState, setValue]); | ||
| }, [ | ||
| connectionMappings, | ||
| filterConnectionMappings, | ||
| input.tiles, | ||
| input.filters, | ||
| getFieldState, | ||
| setValue, | ||
| ]); | ||
|
|
||
| // Propagate dashboard mapping changes to other tiles whose onClick targets | ||
| // the same input dashboard. Dashboard-type onClicks in a template store the | ||
|
|
@@ -1054,6 +1130,12 @@ export function Mapping({ input }: { input: DashboardTemplate }) { | |
| appliesToSourceIds, | ||
| }; | ||
| } | ||
| if (isSqlQueryFilter(filter)) { | ||
| const connection = findConnection( | ||
| data.filterConnectionMappings?.[idx], | ||
| ); | ||
| return { ...filter, connection: connection!.id }; | ||
| } | ||
| const source = findSource(data.filterSourceMappings?.[idx]); | ||
| return { | ||
| ...filter, | ||
|
|
@@ -1256,6 +1338,24 @@ export function Mapping({ input }: { input: DashboardTemplate }) { | |
| <Table.Td /> | ||
| </Table.Tr> | ||
| )} | ||
| {isSqlQueryFilter(filter) && ( | ||
| <Table.Tr> | ||
| <Table.Td>{filter.name} (Filter)</Table.Td> | ||
| <Table.Td>Data connection</Table.Td> | ||
| <Table.Td>{filter.connection}</Table.Td> | ||
| <Table.Td> | ||
| <SelectControlled | ||
| control={control} | ||
| name={`filterConnectionMappings.${i}`} | ||
| data={connections?.map(conn => ({ | ||
| value: conn.id, | ||
| label: conn.name, | ||
| }))} | ||
| placeholder="Select a connection" | ||
| /> | ||
| </Table.Td> | ||
| </Table.Tr> | ||
| )} | ||
| {isAdhocFilter(filter) && ( | ||
| <Table.Tr> | ||
| <Table.Td>{filter.name} (Filter)</Table.Td> | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -23,6 +23,7 @@ import { | |
| IconPencil, | ||
| IconRefresh, | ||
| IconSearch, | ||
| IconSql, | ||
| IconTrash, | ||
| } from '@tabler/icons-react'; | ||
|
|
||
|
|
@@ -91,6 +92,12 @@ function getValuesAttribute( | |
| tooltip: 'Sources the condition keys and values come from', | ||
| label: getSourceNames(filter.sources, sources), | ||
| }; | ||
| case 'SQL_QUERY': | ||
|
karl-power marked this conversation as resolved.
Resolved
karl-power marked this conversation as resolved.
Resolved
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟠 major — An imported SQL_QUERY filter can be edited, but the edit form drops its connection and SQL and can't save it Imports can now create SQL_QUERY filters, and the filter list shows them with the usual edit pencil. Clicking it runs Nothing blocks merge automatically, but a maintainer will expect this fixed if it is a real defect in code this PR changes. If it is about surrounding code, reply and say so instead of patching. Do not widen the PR. How to respond |
||
| return { | ||
| icon: <IconSql size={14} />, | ||
| tooltip: 'The dropdown values come from a SQL query', | ||
| label: 'SQL query', | ||
| }; | ||
| default: | ||
| filter satisfies never; // exhaustive check | ||
| return { icon: <IconSearch size={14} />, label: '' }; | ||
|
|
||
Uh oh!
There was an error while loading. Please reload this page.