Repository navigation
Store large node input values as delta-compressed blobs - #4926
TrueDoctor wants to merge 4 commits into
Conversation
There was a problem hiding this comment.
3 issues found across 66 files
Confidence score: 3/5
packet.rsputs all blob records in oneSyncPayload, which can exceedMAX_PACKET_BYTES; thenencodefails and the joining peer can stall. Split or chunk blob records across packets.blob.rsleaves largeSetNetworkExportvalues and network export slots embedded in registry and history data. Blobify them alongside node inputs so large exports do not bypass blob handling.network.rsstores only the blob when a hash also names a regular resource, leaving that resource missing. Handle both resource kinds for the shared hash.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="document/peer-transport/src/packet.rs">
<violation number="1" location="document/peer-transport/src/packet.rs:87">
P2: The new `blobs` field puts every blob record in one `SyncPayload`, so a large document can exceed `MAX_PACKET_BYTES`. Then `encode` fails with `TooLarge`, the host never sends the `Sync`, and the joining peer stalls with only a logged error. Consider sending blob records in bounded chunks or fetching them through the existing resource path.</violation>
</file>
<file name="document/graph-storage/src/blob.rs">
<violation number="1" location="document/graph-storage/src/blob.rs:76">
P2: Large network export values bypass blobification and remain embedded in registry and history data. Blobify `SetNetworkExport` values and network export slots alongside node inputs.</violation>
</file>
<file name="document/format/src/network.rs">
<violation number="1" location="document/format/src/network.rs:387">
P2: When a hash names both a blob and a regular resource, this branch stores only the blob and returns. Because `missing_resources` merges both kinds into one hash set, the regular resource stays missing and the peer can request the hash forever; store the bytes in both stores.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
| pub metadata: HistoryMetadata, | ||
| /// Every blob that the synced state names. A base comes before the blobs that use it. | ||
| #[serde(default)] | ||
| pub blobs: Vec<BlobRecord>, |
There was a problem hiding this comment.
P2: The new blobs field puts every blob record in one SyncPayload, so a large document can exceed MAX_PACKET_BYTES. Then encode fails with TooLarge, the host never sends the Sync, and the joining peer stalls with only a logged error. Consider sending blob records in bounded chunks or fetching them through the existing resource path.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At document/peer-transport/src/packet.rs, line 87:
<comment>The new `blobs` field puts every blob record in one `SyncPayload`, so a large document can exceed `MAX_PACKET_BYTES`. Then `encode` fails with `TooLarge`, the host never sends the `Sync`, and the joining peer stalls with only a logged error. Consider sending blob records in bounded chunks or fetching them through the existing resource path.</comment>
<file context>
@@ -82,6 +82,9 @@ pub struct SyncPayload {
pub metadata: HistoryMetadata,
+ /// Every blob that the synced state names. A base comes before the blobs that use it.
+ #[serde(default)]
+ pub blobs: Vec<BlobRecord>,
}
</file context>
| return; | ||
| } | ||
| RegistryDelta::AddNode { node, .. } => &mut node.inputs, | ||
| RegistryDelta::SetNodeInputs { inputs, .. } => inputs, |
There was a problem hiding this comment.
P2: Large network export values bypass blobification and remain embedded in registry and history data. Blobify SetNetworkExport values and network export slots alongside node inputs.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At document/graph-storage/src/blob.rs, line 76:
<comment>Large network export values bypass blobification and remain embedded in registry and history data. Blobify `SetNetworkExport` values and network export slots alongside node inputs.</comment>
<file context>
@@ -0,0 +1,206 @@
+ return;
+ }
+ RegistryDelta::AddNode { node, .. } => &mut node.inputs,
+ RegistryDelta::SetNodeInputs { inputs, .. } => inputs,
+ _ => return,
+ };
</file context>
|
|
||
| fn store_resource(&mut self, hash: ResourceHash, bytes: &[u8]) -> Result<(), TargetError> { | ||
| let blob = BlobHash(hash); | ||
| if self.session.names_blob(blob) { |
There was a problem hiding this comment.
P2: When a hash names both a blob and a regular resource, this branch stores only the blob and returns. Because missing_resources merges both kinds into one hash set, the regular resource stays missing and the peer can request the hash forever; store the bytes in both stores.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At document/format/src/network.rs, line 387:
<comment>When a hash names both a blob and a regular resource, this branch stores only the blob and returns. Because `missing_resources` merges both kinds into one hash set, the regular resource stays missing and the peer can request the hash forever; store the bytes in both stores.</comment>
<file context>
@@ -370,12 +372,36 @@ impl<L: Layout> SyncTarget for Gdd<L> {
+
+ fn store_resource(&mut self, hash: ResourceHash, bytes: &[u8]) -> Result<(), TargetError> {
+ let blob = BlobHash(hash);
+ if self.session.names_blob(blob) {
+ let bytes = BlobBytes::from([(blob, bytes.to_vec())]);
+ self.keep_blobs(NewBlobs { bytes, ..Default::default() })?;
</file context>
There was a problem hiding this comment.
1 issue found across 1 file (changes from recent commits).
Confidence score: 3/5
- In
document/format/src/blob_store.rs,loadcan reject a delta blob larger than 256 MiB even thoughsavewrote it successfully, leaving that record unavailable after reopening. Apply the peer packet limit only to peer packets, not local disk records.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="document/format/src/blob_store.rs">
<violation number="1" location="document/format/src/blob_store.rs:248">
P2: This cap comes from the peer packet limit, but it also applies to local records read from disk in `load`. A delta blob that decodes past 256 MiB is written on save and then fails to load on reopen. Apply the limit only on the peer `absorb` path, or skip it in `load`.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
|
|
||
| /// A delta that decodes to more bytes is rejected. A small delta from a peer could otherwise copy the same base range | ||
| /// until it fills memory. This matches the peer transport's packet limit, so no larger blob could be sent in full. | ||
| const MAX_BLOB_BYTES: usize = 256 * 1024 * 1024; |
There was a problem hiding this comment.
P2: This cap comes from the peer packet limit, but it also applies to local records read from disk in load. A delta blob that decodes past 256 MiB is written on save and then fails to load on reopen. Apply the limit only on the peer absorb path, or skip it in load.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At document/format/src/blob_store.rs, line 248:
<comment>This cap comes from the peer packet limit, but it also applies to local records read from disk in `load`. A delta blob that decodes past 256 MiB is written on save and then fails to load on reopen. Apply the limit only on the peer `absorb` path, or skip it in `load`.</comment>
<file context>
@@ -243,6 +243,10 @@ const COPY_BLOCK: usize = 16;
+/// A delta that decodes to more bytes is rejected. A small delta from a peer could otherwise copy the same base range
+/// until it fills memory. This matches the peer transport's packet limit, so no larger blob could be sent in full.
+const MAX_BLOB_BYTES: usize = 256 * 1024 * 1024;
+
/// Expected O(base + target) time. Each base block is indexed once and each unmatched target byte is looked up once.
</file context>
No description provided.