-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Block splitter #4136
Block splitter #4136
Commits on Oct 23, 2024
-
instead of ingesting only full blocks, make an analysis of data, and infer where to split.
Configuration menu - View commit details
-
Copy full SHA for a5bce4a - Browse repository at this point
Copy the full SHA a5bce4aView commit details -
Configuration menu - View commit details
-
Copy full SHA for 9e52789 - Browse repository at this point
Copy the full SHA 9e52789View commit details -
Configuration menu - View commit details
-
Copy full SHA for 586ca96 - Browse repository at this point
Copy the full SHA 586ca96View commit details -
Configuration menu - View commit details
-
Copy full SHA for e2d7d08 - Browse repository at this point
Copy the full SHA e2d7d08View commit details -
though I really wonder if this is a property worth maintaining.
Configuration menu - View commit details
-
Copy full SHA for 6021b66 - Browse repository at this point
Copy the full SHA 6021b66View commit details -
Configuration menu - View commit details
-
Copy full SHA for fa147cb - Browse repository at this point
Copy the full SHA fa147cbView commit details -
Configuration menu - View commit details
-
Copy full SHA for 83a3402 - Browse repository at this point
Copy the full SHA 83a3402View commit details -
Configuration menu - View commit details
-
Copy full SHA for f83ed08 - Browse repository at this point
Copy the full SHA f83ed08View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8b3887f - Browse repository at this point
Copy the full SHA 8b3887fView commit details -
Configuration menu - View commit details
-
Copy full SHA for dd38c67 - Browse repository at this point
Copy the full SHA dd38c67View commit details -
Configuration menu - View commit details
-
Copy full SHA for 0d4b520 - Browse repository at this point
Copy the full SHA 0d4b520View commit details -
Configuration menu - View commit details
-
Copy full SHA for 20c3d17 - Browse repository at this point
Copy the full SHA 20c3d17View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6dc5212 - Browse repository at this point
Copy the full SHA 6dc5212View commit details -
Configuration menu - View commit details
-
Copy full SHA for 80a912d - Browse repository at this point
Copy the full SHA 80a912dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 6939235 - Browse repository at this point
Copy the full SHA 6939235View commit details -
Configuration menu - View commit details
-
Copy full SHA for cdddcaa - Browse repository at this point
Copy the full SHA cdddcaaView commit details -
Configuration menu - View commit details
-
Copy full SHA for 76ad1d6 - Browse repository at this point
Copy the full SHA 76ad1d6View commit details -
Configuration menu - View commit details
-
Copy full SHA for 31d48e9 - Browse repository at this point
Copy the full SHA 31d48e9View commit details -
Configuration menu - View commit details
-
Copy full SHA for 7f015c2 - Browse repository at this point
Copy the full SHA 7f015c2View commit details -
ZSTD_splitBlock_4k() uses externally provided workspace
ideally, this workspace would be provided from the ZSTD_CCtx* state
Configuration menu - View commit details
-
Copy full SHA for 73a6653 - Browse repository at this point
Copy the full SHA 73a6653View commit details -
Configuration menu - View commit details
-
Copy full SHA for 433f459 - Browse repository at this point
Copy the full SHA 433f459View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4685eaf - Browse repository at this point
Copy the full SHA 4685eafView commit details -
Configuration menu - View commit details
-
Copy full SHA for cae8d13 - Browse repository at this point
Copy the full SHA cae8d13View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4ce91cb - Browse repository at this point
Copy the full SHA 4ce91cbView commit details -
Configuration menu - View commit details
-
Copy full SHA for dac26ea - Browse repository at this point
Copy the full SHA dac26eaView commit details -
let's fill the initial stats directly into target fingerprint
Configuration menu - View commit details
-
Copy full SHA for 1c62e71 - Browse repository at this point
Copy the full SHA 1c62e71View commit details -
added a faster block splitter variant
that samples 1 in 5 positions. This variant is fast enough for lazy2 and btlazy2, but it's less good in combination with post-splitter at higher levels (>= btopt).
Configuration menu - View commit details
-
Copy full SHA for a167571 - Browse repository at this point
Copy the full SHA a167571View commit details -
Configuration menu - View commit details
-
Copy full SHA for 7bad787 - Browse repository at this point
Copy the full SHA 7bad787View commit details -
Configuration menu - View commit details
-
Copy full SHA for 5ae34e4 - Browse repository at this point
Copy the full SHA 5ae34e4View commit details -
conservatively estimate over-splitting in presence of incompressible …
…loss ensure data can never be expanded by more than 3 bytes per full block.
Configuration menu - View commit details
-
Copy full SHA for ea85dc7 - Browse repository at this point
Copy the full SHA ea85dc7View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4662f6e - Browse repository at this point
Copy the full SHA 4662f6eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 1ec5f9f - Browse repository at this point
Copy the full SHA 1ec5f9fView commit details -
Configuration menu - View commit details
-
Copy full SHA for 16450d0 - Browse repository at this point
Copy the full SHA 16450d0View commit details -
rewrote ZSTD_cwksp_initialAllocStart() to be easier to read
following a discussion with @felixhandte
Configuration menu - View commit details
-
Copy full SHA for 06b7cfa - Browse repository at this point
Copy the full SHA 06b7cfaView commit details -
Configuration menu - View commit details
-
Copy full SHA for 0be334d - Browse repository at this point
Copy the full SHA 0be334dView commit details -
due to integration of `sample5` strategy, leading to better compression ratios on a range of levels
Configuration menu - View commit details
-
Copy full SHA for d2eeed5 - Browse repository at this point
Copy the full SHA d2eeed5View commit details -
Configuration menu - View commit details
-
Copy full SHA for 18b1e67 - Browse repository at this point
Copy the full SHA 18b1e67View commit details -
Configuration menu - View commit details
-
Copy full SHA for 57239c4 - Browse repository at this point
Copy the full SHA 57239c4View commit details -
rewrite fingerprint storage to no longer need 64-bit members
so that it can be stored using standard alignment requirement (sizeof(void*)). Distance function still requires 64-bit signed multiplication though, so it won't change the issue regarding the bug in ubsan for clang 32-bit on github ci.
Configuration menu - View commit details
-
Copy full SHA for b68ddce - Browse repository at this point
Copy the full SHA b68ddceView commit details -
this helps make the streaming behavior more consistent, since it does no longer depend on having more data presented on the input. suggested by @terrelln
Configuration menu - View commit details
-
Copy full SHA for 7d3e5e3 - Browse repository at this point
Copy the full SHA 7d3e5e3View commit details -
Configuration menu - View commit details
-
Copy full SHA for c80645a - Browse repository at this point
Copy the full SHA c80645aView commit details -
Configuration menu - View commit details
-
Copy full SHA for bbda1ac - Browse repository at this point
Copy the full SHA bbda1acView commit details
Commits on Oct 24, 2024
-
apply limit conditions for all splitting strategies
instead of just for blind split. This is in anticipation of adversarial input, that would intentionally target the sampling pattern of the split detector. Note that, even without this protection, splitting can never expand beyond ZSTD_COMPRESSBOUND(), because this upper limit uses a 1KB block size worst case scenario, and splitting never creates blocks thath small. The protection is more to ensure that data is not expanded by more than 3-bytes per 128 KB full block, which is a much stricter limit.
Configuration menu - View commit details
-
Copy full SHA for 90095f0 - Browse repository at this point
Copy the full SHA 90095f0View commit details -
first block is no longer splitted since adding the @Savings over-split protection
Configuration menu - View commit details
-
Copy full SHA for 70c77d2 - Browse repository at this point
Copy the full SHA 70c77d2View commit details