Track compressed resource state sizes in deploy telemetry (direct engine)#5608
Open
shreyas-goenka wants to merge 1 commit into
Open
Track compressed resource state sizes in deploy telemetry (direct engine)#5608shreyas-goenka wants to merge 1 commit into
shreyas-goenka wants to merge 1 commit into
Conversation
Collaborator
Integration test reportCommit: c9ac30f
27 interesting tests: 15 SKIP, 7 KNOWN, 5 flaky
Top 24 slowest tests (at least 2 minutes):
|
1aacd8b to
8c625fa
Compare
Contributor
Approval status: pending
|
505c536 to
b0b017d
Compare
b0b017d to
ca58e77
Compare
pietern
reviewed
Jun 17, 2026
|
|
||
| result := make(map[string]int, len(keys)) | ||
| for i, key := range keys { | ||
| result[key] = sizes[i] |
Contributor
There was a problem hiding this comment.
The canonical pattern here is to run a goroutine for each element in the map and have it return a {key, int} on a channel. The main loop then drains that channel to collect the results and stores them in a map. There is no need to deal with GOMAXPROCS or "workers". The Go runtime takes care of scheduling.
Contributor
Author
There was a problem hiding this comment.
Done — switched to the canonical pattern: one goroutine per resource sending {key, size} on a buffered channel, drained into the map. Dropped the GOMAXPROCS/worker-pool machinery entirely.
| } | ||
| return buf.Len() | ||
| } | ||
|
|
Contributor
There was a problem hiding this comment.
This code belongs in a separate file where we have all the compression related stuff.
Contributor
Author
There was a problem hiding this comment.
Done — moved the compression code (compressedStateSize + compressStateSizes) into bundle/direct/dstate/compress.go.
| } | ||
| }) | ||
| } | ||
| } |
Contributor
There was a problem hiding this comment.
This doesn't have anything to do with state, only with compression.
Contributor
Author
There was a problem hiding this comment.
Done — the compression test and benchmarks now live in compress_test.go alongside the compression code.
…ine)
Deploy telemetry already reports per-resource-type raw state-size statistics
(state_size_{max,mean,median}_bytes). The deployment metadata service stores
that same per-resource state compressed, so this adds compressed-size
counterparts to gauge how much resource state shrinks under compression rather
than just the raw sizes:
- state_compressed_size_max_bytes
- state_compressed_size_mean_bytes
- state_compressed_size_median_bytes
The compressed length is computed per resource at state-export time (alongside
the existing raw length) using the standard library's compress/flate -- a
deliberately rough proxy for the server side (which uses zstd) that keeps the
dependency/supply-chain surface small while still giving useful signal on
compressibility. Since the largest resource states (~1 MB, ~20 ms to compress)
dominate the cost, the per-resource compression is fanned out across workers,
keeping multi-resource bundles cheap. Only the direct engine is measured,
matching the existing raw-size behavior.
Co-authored-by: Isaac
ca58e77 to
c9ac30f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Bundle deploy telemetry already reports per-resource-type raw state-size statistics (
state_size_{max,mean,median}_bytesinResourceMetadata). The same per-resource state is stored compressed downstream, so this adds the compressed-size counterparts to gauge how much resource state shrinks under compression, not just the raw sizes:state_compressed_size_max_bytesstate_compressed_size_mean_bytesstate_compressed_size_median_bytesPerformance
flateruns at state-export time over individually small resource states (each well under the server's per-resource limit), not in a tight loop, so even large bundles compress in a few milliseconds — negligible next to a deploy's network I/O. No background goroutine is warranted.Flate vs Zstd