Skip to content

Store large node input values as delta-compressed blobs - #4926

Open
TrueDoctor wants to merge 4 commits into
masterfrom
blob-compression
Open

TrueDoctor wants to merge 4 commits into
masterfrom
blob-compression

Conversation

@TrueDoctor

Copy link
Copy Markdown
Member

No description provided.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

3 issues found across 66 files

Confidence score: 3/5

  • packet.rs puts all blob records in one SyncPayload, which can exceed MAX_PACKET_BYTES; then encode fails and the joining peer can stall. Split or chunk blob records across packets.
  • blob.rs leaves large SetNetworkExport values and network export slots embedded in registry and history data. Blobify them alongside node inputs so large exports do not bypass blob handling.
  • network.rs stores only the blob when a hash also names a regular resource, leaving that resource missing. Handle both resource kinds for the shared hash.
Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="document/peer-transport/src/packet.rs">

<violation number="1" location="document/peer-transport/src/packet.rs:87">
P2: The new `blobs` field puts every blob record in one `SyncPayload`, so a large document can exceed `MAX_PACKET_BYTES`. Then `encode` fails with `TooLarge`, the host never sends the `Sync`, and the joining peer stalls with only a logged error. Consider sending blob records in bounded chunks or fetching them through the existing resource path.</violation>
</file>

<file name="document/graph-storage/src/blob.rs">

<violation number="1" location="document/graph-storage/src/blob.rs:76">
P2: Large network export values bypass blobification and remain embedded in registry and history data. Blobify `SetNetworkExport` values and network export slots alongside node inputs.</violation>
</file>

<file name="document/format/src/network.rs">

<violation number="1" location="document/format/src/network.rs:387">
P2: When a hash names both a blob and a regular resource, this branch stores only the blob and returns. Because `missing_resources` merges both kinds into one hash set, the regular resource stays missing and the peer can request the hash forever; store the bytes in both stores.</violation>
</file>

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

pub metadata: HistoryMetadata,
/// Every blob that the synced state names. A base comes before the blobs that use it.
#[serde(default)]
pub blobs: Vec<BlobRecord>,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: The new blobs field puts every blob record in one SyncPayload, so a large document can exceed MAX_PACKET_BYTES. Then encode fails with TooLarge, the host never sends the Sync, and the joining peer stalls with only a logged error. Consider sending blob records in bounded chunks or fetching them through the existing resource path.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At document/peer-transport/src/packet.rs, line 87:

<comment>The new `blobs` field puts every blob record in one `SyncPayload`, so a large document can exceed `MAX_PACKET_BYTES`. Then `encode` fails with `TooLarge`, the host never sends the `Sync`, and the joining peer stalls with only a logged error. Consider sending blob records in bounded chunks or fetching them through the existing resource path.</comment>

<file context>
@@ -82,6 +82,9 @@ pub struct SyncPayload {
 	pub metadata: HistoryMetadata,
+	/// Every blob that the synced state names. A base comes before the blobs that use it.
+	#[serde(default)]
+	pub blobs: Vec<BlobRecord>,
 }
 
</file context>

Comment thread document/format/src/export.rs Resolved
Comment thread document/format/src/blob_store.rs Resolved
Comment thread document/format/src/blob_store.rs Resolved
return;
}
RegistryDelta::AddNode { node, .. } => &mut node.inputs,
RegistryDelta::SetNodeInputs { inputs, .. } => inputs,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Large network export values bypass blobification and remain embedded in registry and history data. Blobify SetNetworkExport values and network export slots alongside node inputs.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At document/graph-storage/src/blob.rs, line 76:

<comment>Large network export values bypass blobification and remain embedded in registry and history data. Blobify `SetNetworkExport` values and network export slots alongside node inputs.</comment>

<file context>
@@ -0,0 +1,206 @@
+				return;
+			}
+			RegistryDelta::AddNode { node, .. } => &mut node.inputs,
+			RegistryDelta::SetNodeInputs { inputs, .. } => inputs,
+			_ => return,
+		};
</file context>


fn store_resource(&mut self, hash: ResourceHash, bytes: &[u8]) -> Result<(), TargetError> {
let blob = BlobHash(hash);
if self.session.names_blob(blob) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: When a hash names both a blob and a regular resource, this branch stores only the blob and returns. Because missing_resources merges both kinds into one hash set, the regular resource stays missing and the peer can request the hash forever; store the bytes in both stores.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At document/format/src/network.rs, line 387:

<comment>When a hash names both a blob and a regular resource, this branch stores only the blob and returns. Because `missing_resources` merges both kinds into one hash set, the regular resource stays missing and the peer can request the hash forever; store the bytes in both stores.</comment>

<file context>
@@ -370,12 +372,36 @@ impl<L: Layout> SyncTarget for Gdd<L> {
+
+	fn store_resource(&mut self, hash: ResourceHash, bytes: &[u8]) -> Result<(), TargetError> {
+		let blob = BlobHash(hash);
+		if self.session.names_blob(blob) {
+			let bytes = BlobBytes::from([(blob, bytes.to_vec())]);
+			self.keep_blobs(NewBlobs { bytes, ..Default::default() })?;
</file context>

Comment thread document/format/src/network.rs Resolved

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 1 file (changes from recent commits).

Confidence score: 3/5

  • In document/format/src/blob_store.rs, load can reject a delta blob larger than 256 MiB even though save wrote it successfully, leaving that record unavailable after reopening. Apply the peer packet limit only to peer packets, not local disk records.
Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="document/format/src/blob_store.rs">

<violation number="1" location="document/format/src/blob_store.rs:248">
P2: This cap comes from the peer packet limit, but it also applies to local records read from disk in `load`. A delta blob that decodes past 256 MiB is written on save and then fails to load on reopen. Apply the limit only on the peer `absorb` path, or skip it in `load`.</violation>
</file>

Reply with feedback, questions, or to request a fix.

Re-trigger cubic


/// A delta that decodes to more bytes is rejected. A small delta from a peer could otherwise copy the same base range
/// until it fills memory. This matches the peer transport's packet limit, so no larger blob could be sent in full.
const MAX_BLOB_BYTES: usize = 256 * 1024 * 1024;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: This cap comes from the peer packet limit, but it also applies to local records read from disk in load. A delta blob that decodes past 256 MiB is written on save and then fails to load on reopen. Apply the limit only on the peer absorb path, or skip it in load.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At document/format/src/blob_store.rs, line 248:

<comment>This cap comes from the peer packet limit, but it also applies to local records read from disk in `load`. A delta blob that decodes past 256 MiB is written on save and then fails to load on reopen. Apply the limit only on the peer `absorb` path, or skip it in `load`.</comment>

<file context>
@@ -243,6 +243,10 @@ const COPY_BLOCK: usize = 16;
 
+/// A delta that decodes to more bytes is rejected. A small delta from a peer could otherwise copy the same base range
+/// until it fills memory. This matches the peer transport's packet limit, so no larger blob could be sent in full.
+const MAX_BLOB_BYTES: usize = 256 * 1024 * 1024;
+
 /// Expected O(base + target) time. Each base block is indexed once and each unmatched target byte is looked up once.
</file context>

This branch was successfully deployed

1 active deployment
graphite-dev (Preview) — 09e7b04b Deployed Nov 6, 2026 by github-actions[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant