dilim/containerd
1
0 Fork 0
mirror of https://github.com/containerd/containerd.git synced 2026-11-06 05:17:02 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #14471 from thaJeztah/2.4_bump_compress

Browse Source 
[release/2.4 backport] build(deps): bump github.com/klauspost/compress to 1.20.3
This commit is contained in:
Phil Estes authored and GitHub committed 2026-10-27 11:55:46 -04:00
2 parents 1a459edb24 72453b868c
commit 4f7e58a1f0
64 files changed
+9135 -4218
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

go.mod
+1 -1
View File
@@ -44,7 +44,7 @@ require (
github.com/google/uuid v1.6.0
github.com/grpc-ecosystem/go-grpc-middleware/providers/prometheus v1.1.0
github.com/intel/goresctrl v0.13.0
github.com/klauspost/compress v1.20.0
github.com/klauspost/compress v1.20.3
github.com/mdlayher/vsock v1.3.0
github.com/moby/locker v1.0.1
github.com/moby/sys/mountinfo v0.7.2
go.sum
+2 -2
View File
@@ -185,8 +185,8 @@ github.com/jsimonetti/rtnetlink/v2 v2.0.1 h1:xda7qaHDSVOsADNouv7ukSuicKZO7GgVUCX
github.com/jsimonetti/rtnetlink/v2 v2.0.1/go.mod h1:7MoNYNbb3UaDHtF8udiJo/RH6VsTKP1pqKLUTVCvToE=
github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM=
github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo=
github.com/klauspost/compress v1.20.0 h1:a3C1ke2ohxFymNlb2HWAHjDeKCI90scRskErZkR0ezA=
github.com/klauspost/compress v1.20.0/go.mod h1:LUdAzn7YLVvxLpc7y3V1m40wESHTgc1422pwwBSKYuI=
github.com/klauspost/compress v1.20.3 h1:zWNoFqq/lSuYQRim3dUXlXeA0IqwuZGicdG5ZTfY5j4=
github.com/klauspost/compress v1.20.3/go.mod h1:LUdAzn7YLVvxLpc7y3V1m40wESHTgc1422pwwBSKYuI=
github.com/knqyf263/go-plugin v0.9.0 h1:CQs2+lOPIlkZVtcb835ZYDEoyyWJWLbSTWeCs0EwTwI=
github.com/knqyf263/go-plugin v0.9.0/go.mod h1:2z5lCO1/pez6qGo8CvCxSlBFSEat4MEp1DrnA+f7w8Q=
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
vendor/github.com/klauspost/compress/.gitignore
Generated
+3
View File
@@ -30,3 +30,6 @@ perf.data.old
# gdb history
.gdb_history
# Silesia corpus, fetched by testdata/fetch_silesia.sh
**/testdata/silesia.tar*
vendor/github.com/klauspost/compress/AGENTS.md
Generated Vendored
+104
View File
@@ -0,0 +1,104 @@
# AGENTS.md
Guidance for people and coding agents changing this repository. It records
conventions and traps learned while working on the decoders and encoders; the
README describes the packages themselves.
## Checks before sending a change
- `gofmt`, `go vet ./...` and `go fix -diff ./...` (which should report
nothing). Code written for specific platforms must be vetted for them too:
run `go vet` with `GOOS`/`GOARCH` set to each platform it targets. `go vet`
caches results per package and does not notice edits to files excluded by
build tags; use a fresh `GOCACHE` to confirm a stale-looking error.
- `go test ./...` with and without `-tags=noasm` (and `nounsafe` if you
touched `unsafe` code), on every architecture whose assembly changed.
- Decoder changes: run the fuzzers CI runs (see `.github/workflows/go.yml`),
also with `-tags=noasm`, `-tags=nounsafe` and `-asan`, for longer than CI
does, and for zstd both `FuzzDecodeAll` and the `NoBMI2` variants.
- Raising the Go version in the root `go.mod` breaks the nested modules
(`*/_generate`, `s2/cmd/_s2sx`) until they are updated too. In a `go`
directive, `1.25` sorts before `1.25.0`, so a dependency that needs
`1.25.0` forces the full form.
## Tests
- Put a new test in the existing test file for the code it exercises
(`decoder_test.go` for Reader/DecodeAll behaviour, `seqdec_test.go` for
sequence execution, and so on), and reuse its helpers. Use table tests for
three or more similar cases.
- Extend the existing fuzzers (`func Fuzz…`) rather than adding parallel
harnesses; if a new harness supersedes an old one, remove the old one in
the same change.
- Prove which path a test or benchmark exercises. A benchmark once ran the
pure-Go sequence decoder instead of the assembly because of how its buffers
were set up. A `panic` planted in the path is a quick check; `println`
output is easy to miss.
- `sync.Pool` drops about a quarter of `Put`s under `-race` by design; never
assume `Get` returns the last `Put`.
- Don't add tests whose only purpose is to catch someone deliberately undoing
a performance pin (an alignment, say); a comment explaining it is enough.
## Performance claims
- Use real data. The maintainer's corpora are at
https://klauspost.com/files/compress/ (logs, JSON, CSV, serialized data,
tar files); put `.zst` files in `zstd/testdata/benchmark-custom/` and run
`BenchmarkDecoderWithCustomFiles`. Silesia, enwik and generated data are
fine for correctness but are not accepted as evidence for a speed-up.
- When a result depends on how the data was compressed, report the levels
separately (for example `SpeedDefault` and `SpeedBestCompression`).
- Anchor `-bench` patterns at every level with `$`: `-bench 'BenchmarkFoo$'`
also runs `BenchmarkFooBar` otherwise, which has produced fake gains and
regressions.
- Code placement alone moves results by about ±2%. For an A/B, insert
`PCALIGN $64` before every `TEXT` in the `.s` files of *both* builds
(measurement only, not committed).
- Build both variants as test binaries up front, then run them in turns: one
short `-count 1` run of each, repeated for ten or more rounds, swapping
which goes first every round (AB, BA, AB, …) so neither build always gets
the warmer or colder slot. Never run all of A and then all of B. Clock
speed, temperature, cache and page-cache state, and background load drift
over a session; alternating spreads that drift over both builds instead of
turning it into a difference between them. Pin to one core, use an
otherwise idle machine, and compare the pooled results with `benchstat`.
- Report the number of regressed cases next to the geomean; a small win with
no regressions is a different result from a larger one with some.
- No CPU-model-specific code paths or thresholds. A gate for arm64 must be
measured on several Neoverse generations (N1, V1, V2, V3), not one core.
## Pull requests
- One commit per independent change, with fixups squashed in. Number the
changes in the description as "(1) … (sha)" and name the test file for
each test.
- Mention each `#NNN` once in prose; refer back in words.
- Don't open many PRs against the same area at once; reviewer time is the
scarce resource.
- Keep Go comments short; generator and assembly comments can explain in
detail.
- Keep private or company-internal references (internal service names,
private links, chat or agent-session URLs) out of code, commit messages and
PR text.
## Assembly is generated
- zstd, huff0 and s2 assembly is generated with [avo](https://github.com/mmcloughlin/avo)
from `zstd/_generate`, `huff0/_generate` and `s2/_generate`. Never edit the
`.s` files by hand: change the generator, run `go generate -v -x` in the
`_generate` directory, and commit the regenerated output. CI's `generate` job
fails if the committed files differ from the generator's output.
- The arm64 files are lowered from the same amd64 program by the
`honeycombio/avo` fork that the `_generate/go.mod` files `replace` avo with.
A change to the generator changes both architectures; build and test both.
- When moving the avo pin, regenerate and check the output is byte-identical
unless a change is intended, so unrelated printer changes don't slip into a
review.
- Never put a label directly on a `PCALIGN` (emit the `PCALIGN` before the
label). A jump to it makes the amd64 assembler jump to the wrong place
(Go ≤ 1.25, golang/go#74648) or loop forever when it has to widen a branch
(Go ≥ 1.26, golang/go#81792). Current versions of the avo fork reject it.
- Build tags: `noasm` selects the pure-Go code and `nounsafe` the code without
`unsafe`. Every assembly or `unsafe` path needs a portable fallback, and CI
runs the default, `noasm`, `nounsafe` and `nounsafe,noasm` builds. The
library doesn't use cgo.
vendor/github.com/klauspost/compress/README.md
Generated Vendored
+52 -1
View File
@@ -29,8 +29,41 @@ Use the links above for more information on each.
# changelog
* Sep 25th, 2026 [1.20.1](https://github.com/klauspost/compress/releases/tag/v1.20.1)
* zstd: Apply reset options to reused frame decoders by @pellared in https://github.com/klauspost/compress/pull/1226
* flate: Report unexpected EOF for truncated Huffman extra bits by @rupayon123 in https://github.com/klauspost/compress/pull/1229
* flate: Avoid FMA in EstimatedBits for portable rounding by @malt3 in https://github.com/klauspost/compress/pull/1224
* gzhttp: Preserve caller request headers in Transport by @jakezwang in https://github.com/klauspost/compress/pull/1225
* gzhttp: Only decompress a request when gzip is the outermost coding by @youdie006 in https://github.com/klauspost/compress/pull/1223
* zstd/huff0: Optimizations by @lizthegrey in https://github.com/klauspost/compress/pull/1203 https://github.com/klauspost/compress/pull/1205 https://github.com/klauspost/compress/pull/1207 https://github.com/klauspost/compress/pull/1204 https://github.com/klauspost/compress/pull/1208 https://github.com/klauspost/compress/pull/1211 https://github.com/klauspost/compress/pull/1209 https://github.com/klauspost/compress/pull/1210
* flate: Speed improvements by @alex in https://github.com/klauspost/compress/pull/1220 https://github.com/klauspost/compress/pull/1218 https://github.com/klauspost/compress/pull/1216 https://github.com/klauspost/compress/pull/1215 https://github.com/klauspost/compress/pull/1214 https://github.com/klauspost/compress/pull/1217
* Sep 2nd, 2026 [1.20.0](https://github.com/klauspost/compress/releases/tag/v1.20.0)
* lzw: Add lzw reader/writer in https://github.com/klauspost/compress/pull/1189
* xpress: Add Microsoft XPRESS (MS-XCA) decompression by @MP-GOWTHAM in https://github.com/klauspost/compress/pull/1195
* s2: arm64 assembly encoders by @lizthegrey in https://github.com/klauspost/compress/pull/1187
* zstd, huff0: Improved arm64 assembly by @lizthegrey in https://github.com/klauspost/compress/pull/1175 https://github.com/klauspost/compress/pull/1199 https://github.com/klauspost/compress/pull/1197
* flate: Writer L7-9 with dictionary can emit the dictionary to stream in https://github.com/klauspost/compress/pull/1201
* gzhttp: Limit jitter hash to jitterBuffer bytes by @VedantMadane in https://github.com/klauspost/compress/pull/1192
* s2: Clamp the Go EncodeBetter skip to match the assembly by @lizthegrey in https://github.com/klauspost/compress/pull/1190
* xxhash: Bound input of non-preemptive assembly by @AskAlexSharov in https://github.com/klauspost/compress/pull/1196
* Aug 6th, 2026 [1.19.2](https://github.com/klauspost/compress/releases/tag/v1.19.2)
* huff0: Add arm64 assembly for Decompress4X/1X by @lizthegrey in https://github.com/klauspost/compress/pull/1172
* zstd: Re-enable unsafe decodeSync memory copies by @lizthegrey in https://github.com/klauspost/compress/pull/1171
* zstd: Fix arm64 asm frame offsets placing locals on the saved LR slot by @lizthegrey in https://github.com/klauspost/compress/pull/1176
* zstd: Avoid racing MaxDecodedSize write on shared dict litEnc by @zanarellidev in https://github.com/klauspost/compress/pull/1182
* zstd: Keep BuildDict recent-offsets positive and loadable by @zanarellidev in https://github.com/klauspost/compress/pull/1184
* zstd: Handle zero-literal BuildDict corpus by @cyphercodes in https://github.com/klauspost/compress/pull/1178
* zstd: Don't clear the registered dictionary when decoding past the window by @sueun-dev in https://github.com/klauspost/compress/pull/1177
* Jul 20th, 2026 [1.19.1](https://github.com/klauspost/compress/releases/tag/v1.19.1)
* zstd: Validate SnappyConverter literal copies in https://github.com/klauspost/compress/pull/1170
* flate: Use `Peek` instead of `ReadByte` for the `bufio.Reader` decode path by @joechenrh in https://github.com/klauspost/compress/pull/1169
* zstd: Bump avo pin, regenerate arm64 asm by @lizthegrey in https://github.com/klauspost/compress/pull/1167
* Jul 1st, 2026 [1.19.0](https://github.com/klauspost/compress/releases/tag/v1.19.0)
* zstd: Add true concurrent stream encodingin https://github.com/klauspost/compress/pull/1136
* zstd: Add true concurrent stream encoding in https://github.com/klauspost/compress/pull/1136
* zstd: arm64 decoder asm by @lizthegrey in https://github.com/klauspost/compress/pull/1160
* flate: Add inflate checkpoints in https://github.com/klauspost/compress/pull/1154
* zstd: avoid unused BuildDict encoder allocation by @snissn in https://github.com/klauspost/compress/pull/1147
@@ -41,6 +74,22 @@ Use the links above for more information on each.
* huff0: Allow building table from oversized histogram in https://github.com/klauspost/compress/pull/1156
* s2sx: Clean symlink targets in https://github.com/klauspost/compress/pull/1163
<details>
<summary>See changes to v1.18.x</summary>
* Jun 30th, 2026 [1.18.7](https://github.com/klauspost/compress/releases/tag/v1.18.7)
* s2: Fix `NewDict` out-of-bounds read. Security release without other changes.
* Apr 30th, 2026 [1.18.6](https://github.com/klauspost/compress/releases/tag/v1.18.6)
* s2: Fix amd64 stack frame corruption in https://github.com/klauspost/compress/pull/1145
* gzhttp: Canonicalize ETag header by @justinmayhew in https://github.com/klauspost/compress/pull/1139
* Pool hash tables in Go encode paths to reduce allocations by @huynhanx03 in https://github.com/klauspost/compress/pull/1143
* Mar 20th, 2026 [1.18.5](https://github.com/klauspost/compress/releases/tag/v1.18.5)
* zstd: Fix crash when changing encoder dictionary with same ID in https://github.com/klauspost/compress/pull/1135
* zstd: Default to full zero frames in https://github.com/klauspost/compress/pull/1134
* flate: Clean up histogram order in https://github.com/klauspost/compress/pull/1133
* Feb 9th, 2026 [1.18.4](https://github.com/klauspost/compress/releases/tag/v1.18.4)
* gzhttp: Add zstandard to server handler wrapper https://github.com/klauspost/compress/pull/1121
* zstd: Add ResetWithOptions to encoder/decoder https://github.com/klauspost/compress/pull/1122
@@ -72,6 +121,8 @@ Use the links above for more information on each.
* flate: Fix matchlen L5+L6 https://github.com/klauspost/compress/pull/1049
* flate: Cleanup & reduce casts https://github.com/klauspost/compress/pull/1050
</details>
<details>
<summary>See changes to v1.17.x</summary>
vendor/github.com/klauspost/compress/flate/deflate.go
Generated Vendored
+30 -5
View File
@@ -95,6 +95,11 @@ type advancedState struct {
index int
hashMatch [maxMatchLength + minMatchLength]uint32
// litBits[i] is the number of bits needed to encode window[index:i]
// as literals, where index is the window index at the start of the
// current deflateLazy call. Only allocated when d.chain > 100 (level 9).
litBits []int32
// Input hash chains
// hashHead[hashValue] contains the largest inputIndex with the specified hash value
// If hashHead[hashValue] is within the current window, then
@@ -284,7 +289,9 @@ func (d *compressor) findMatch(pos int, prevHead int, lookahead int) (length, of
minIndex := max(pos-windowSize, 0)
offset = 0
if d.chain < 100 {
// The gain estimate below needs the tables that deflateLazy only
// generates when d.chain > 100, so the conditions must match.
if d.chain <= 100 {
for i := prevHead; tries > 0; tries-- {
if wEnd == win[i+length] {
n := matchLen(win[i:i+minMatchLook], wPos)
@@ -312,21 +319,25 @@ func (d *compressor) findMatch(pos int, prevHead int, lookahead int) (length, of
}
// Minimum gain to accept a match.
cGain := 4
cGain := int32(4)
// Some like it higher (CSV), some like it lower (JSON)
const baseCost = 3
// Base is 4 bytes at with an additional cost.
// Matches must be better than this.
s := d.state
hashPrev := &s.hashPrev
hashOffset := s.hashOffset
litBits := s.litBits[pos : pos+minMatchLook+1]
for i := prevHead; tries > 0; tries-- {
if wEnd == win[i+length] {
n := matchLen(win[i:i+minMatchLook], wPos)
if n > length {
// Calculate gain. Estimate
newGain := d.h.bitLengthRaw(wPos[:n]) - int(offsetExtraBits[offsetCode(uint32(pos-i))]) - baseCost - int(lengthExtraBits[lengthCodes[(n-3)&255]])
newGain := litBits[n] - litBits[0] - int32(offsetExtraBits[offsetCode(uint32(pos-i))]) - baseCost - int32(lengthExtraBits[lengthCodes[(n-3)&255]])
//fmt.Println("gain:", newGain, "prev:", cGain, "raw:", d.h.bitLengthRaw(wPos[:n]), "this-len:", n, "prev-len:", length)
if newGain > cGain {
length = n
offset = pos - i
@@ -344,7 +355,7 @@ func (d *compressor) findMatch(pos int, prevHead int, lookahead int) (length, of
// hashPrev[i & windowMask] has already been overwritten, so stop now.
break
}
i = int(d.state.hashPrev[i&windowMask]) - d.state.hashOffset
i = int(hashPrev[i&windowMask]) - hashOffset
if i < minIndex {
break
}
@@ -428,6 +439,20 @@ func (d *compressor) deflateLazy() {
tmp[v]++
}
d.h.generate(tmp[:], 15)
// Compute the cumulative cost of encoding the window as literals,
// so findMatch can look up the cost of any range in constant time.
if s.litBits == nil {
s.litBits = make([]int32, 2*windowSize+1)
}
codes := d.h.codes[:256]
litBits := s.litBits[s.index : d.windowEnd+1]
total := int32(0)
litBits[0] = 0
for i, v := range d.window[s.index:d.windowEnd] {
total += int32(codes[v].len())
litBits[i+1] = total
}
}
s.maxInsertIndex = d.windowEnd - (minMatchLength - 1)
vendor/github.com/klauspost/compress/flate/huffman_bit_writer.go
Generated Vendored
+94 -128
View File
@@ -32,14 +32,12 @@ const (
// bufferFlushSize indicates the buffer size
// after which bytes are flushed to the writer.
// Should preferably be a multiple of 6, since
// we accumulate 6 bytes between writes to the buffer.
// Between checks, at most 7 bytes are added to the buffer,
// and writes are done as unconditional 8-byte stores, so the
// buffer must have at least bufferFlushSize+7+8 bytes.
bufferFlushSize = 246
)
// Minimum length code that emits bits.
const lengthExtraBitsMinCode = 8
// The number of extra bits needed by length code X - LENGTH_CODES_START.
var lengthExtraBits = [32]uint8{
/* 257 */ 0, 0, 0,
@@ -55,9 +53,6 @@ var lengthBase = [32]uint8{
64, 80, 96, 112, 128, 160, 192, 224, 255,
}
// Minimum offset code that emits bits.
const offsetExtraBitsMinCode = 4
// offset code word extra bits.
var offsetExtraBits = [32]int8{
0, 0, 0, 0, 1, 1, 2, 2, 3, 3,
@@ -67,8 +62,24 @@ var offsetExtraBits = [32]int8{
14, 14,
}
// offsetCombined combines the number of extra bits and the base offset
// of each offset code in a single table: extra bits in the low 8 bits,
// and the base offset (already reduced by baseMatchOffset) in the upper bits.
// Entries for the extended window codes 30 and 31 are unused.
var offsetCombined = [32]uint32{}
// lengthCombined combines, for each match length (reduced by
// baseMatchLength), the length code (relative to lengthCodesStart) in
// bits 0-4, the number of extra bits in bits 5-7 and the value of the
// extra bits in bits 8-12.
var lengthCombined = func() (t [256]uint32) {
for i := range t {
code := lengthCodes[i]
t[i] = uint32(code) | uint32(lengthExtraBits[code])<<5 | uint32(uint8(i)-lengthBase[code])<<8
}
return t
}()
func init() {
var offsetBase = [32]uint32{
/* normal deflate */
@@ -85,7 +96,7 @@ func init() {
for i := range offsetCombined[:] {
// Don't use extended window values...
if offsetExtraBits[i] == 0 || offsetBase[i] > 0x006000 {
if offsetBase[i] > 0x006000 {
continue
}
offsetCombined[i] = uint32(offsetExtraBits[i]) | (offsetBase[i] << 8)
@@ -115,10 +126,14 @@ type huffmanBitWriter struct {
lastHeader int
// Set between 0 (reused block can be up to 2x the size)
logNewTablePenalty uint
bytes [256 + 8]byte
literalFreq [lengthCodesStart + 32]uint16
offsetFreq [32]uint16
codegenFreq [codegenCodeCount]uint16
// bytes must hold at least bufferFlushSize+7+8 bytes (see bufferFlushSize).
// Its size is rounded up to a multiple of 8 to keep the following
// fields aligned, which is measurably faster in writeBlockHuff.
bytes [(bufferFlushSize + 7 + 8 + 7) &^ 7]byte
literalFreq [lengthCodesStart + 32]uint16
offsetFreq [32]uint16
codegenFreq [codegenCodeCount]uint16
// codegen must have an extra space for the final symbol.
codegen [literalCount + offsetCodeCount + 1]uint8
@@ -842,124 +857,75 @@ func (w *huffmanBitWriter) writeTokens(tokens []token, leCodes, oeCodes []hcode)
lengths := leCodes[lengthCodesStart:]
lengths = lengths[:32]
// Go 1.16 LOVES having these on stack.
// Keeping these on the stack instead of in w is significantly faster.
bits, nbits, nbytes := w.bits, w.nbits, w.nbytes
for _, t := range tokens {
// Flush whole bytes, so that nbits <= 7 when entering the loop below.
le.Store64(w.bytes[:], nbytes, bits)
nbytes += nbits >> 3
bits >>= nbits & 56
nbits &= 7
if nbytes >= bufferFlushSize {
_, w.err = w.writer.Write(w.bytes[:nbytes])
nbytes = 0
if w.err != nil {
return
}
}
// The loop below flushes whole bytes at the end of every iteration,
// so that nbits <= 7 at the top of every iteration.
// A literal adds at most 15 bits and a match at most
// 15+5+15+13 = 48 bits, so up to three literals or a single match
// can be added to the 64-bit accumulator without an intermediate flush.
// Unconditionally storing 8 bytes and advancing by the number of
// whole bytes avoids a poorly predicted branch per token.
for i := 0; i < len(tokens); i++ {
t := tokens[i]
if t < 256 {
//w.writeCode(lits[t.literal()])
c := lits[t]
bits |= c.code64() << (nbits & 63)
nbits += c.len()
if nbits >= 48 {
le.Store64(w.bytes[:], nbytes, bits)
bits >>= 48
nbits -= 48
nbytes += 6
if nbytes >= bufferFlushSize {
if w.err != nil {
nbytes = 0
return
}
_, w.err = w.writer.Write(w.bytes[:nbytes])
nbytes = 0
// Add up to two more literals before flushing.
if i+1 < len(tokens) && tokens[i+1] < 256 {
i++
c := lits[tokens[i]]
bits |= c.code64() << (nbits & 63)
nbits += c.len()
if i+1 < len(tokens) && tokens[i+1] < 256 {
i++
c := lits[tokens[i]]
bits |= c.code64() << (nbits & 63)
nbits += c.len()
}
}
continue
}
// Write the length
length := t.length()
lengthCode := lengthCode(length) & 31
if false {
w.writeCode(lengths[lengthCode])
} else {
// inlined
c := lengths[lengthCode]
bits |= c.code64() << (nbits & 63)
nbits += c.len()
if nbits >= 48 {
le.Store64(w.bytes[:], nbytes, bits)
bits >>= 48
nbits -= 48
nbytes += 6
if nbytes >= bufferFlushSize {
if w.err != nil {
nbytes = 0
return
}
_, w.err = w.writer.Write(w.bytes[:nbytes])
nbytes = 0
}
}
}
// Write the length code and its extra bits as one unit.
lc := lengthCombined[t.length()]
c := lengths[lc&31]
bits |= (c.code64() | uint64(lc>>8)<<(c.len()&63)) << (nbits & 63)
nbits += c.len() + uint8(lc>>5)&7
if lengthCode >= lengthExtraBitsMinCode {
extraLengthBits := lengthExtraBits[lengthCode]
//w.writeBits(extraLength, extraLengthBits)
extraLength := int32(length - lengthBase[lengthCode])
bits |= uint64(extraLength) << (nbits & 63)
nbits += extraLengthBits
if nbits >= 48 {
le.Store64(w.bytes[:], nbytes, bits)
bits >>= 48
nbits -= 48
nbytes += 6
if nbytes >= bufferFlushSize {
if w.err != nil {
nbytes = 0
return
}
_, w.err = w.writer.Write(w.bytes[:nbytes])
nbytes = 0
}
}
// Write the offset code and its extra bits as one unit.
offset := t.offset()
offCode := (offset >> 16) & 31
c = offs[offCode]
offComb := offsetCombined[offCode]
extra := (offset - (offComb >> 8)) & matchOffsetOnlyMask
bits |= (c.code64() | uint64(extra)<<(c.len()&63)) << (nbits & 63)
nbits += c.len() + uint8(offComb)
}
// Write the offset
offset := t.offset()
offsetCode := (offset >> 16) & 31
if false {
w.writeCode(offs[offsetCode])
} else {
// inlined
c := offs[offsetCode]
bits |= c.code64() << (nbits & 63)
nbits += c.len()
if nbits >= 48 {
le.Store64(w.bytes[:], nbytes, bits)
bits >>= 48
nbits -= 48
nbytes += 6
if nbytes >= bufferFlushSize {
if w.err != nil {
nbytes = 0
return
}
_, w.err = w.writer.Write(w.bytes[:nbytes])
nbytes = 0
}
}
}
if offsetCode >= offsetExtraBitsMinCode {
offsetComb := offsetCombined[offsetCode]
//w.writeBits(extraOffset, extraOffsetBits)
bits |= uint64((offset-(offsetComb>>8))&matchOffsetOnlyMask) << (nbits & 63)
nbits += uint8(offsetComb)
if nbits >= 48 {
le.Store64(w.bytes[:], nbytes, bits)
bits >>= 48
nbits -= 48
nbytes += 6
if nbytes >= bufferFlushSize {
if w.err != nil {
nbytes = 0
return
}
_, w.err = w.writer.Write(w.bytes[:nbytes])
nbytes = 0
}
le.Store64(w.bytes[:], nbytes, bits)
nbytes += nbits >> 3
bits >>= nbits & 56
nbits &= 7
if nbytes >= bufferFlushSize {
if w.err != nil {
nbytes = 0
return
}
_, w.err = w.writer.Write(w.bytes[:nbytes])
nbytes = 0
}
}
// Restore...
@@ -1094,8 +1060,8 @@ func (w *huffmanBitWriter) writeBlockHuff(eof bool, input []byte, sync bool) {
if debugDeflate {
count -= int(nbytes)*8 + int(nbits)
}
// Unroll, write 3 codes/loop.
// Fastest number of unrolls.
// Write 3 codes per iteration. The three codes are combined first,
// independently of the accumulator, and merged with a single shift.
for len(input) > 3 {
// We must have at least 48 bits free.
if nbits >= 8 {
@@ -1116,13 +1082,13 @@ func (w *huffmanBitWriter) writeBlockHuff(eof bool, input []byte, sync bool) {
_, w.err = w.writer.Write(w.bytes[:nbytes])
nbytes = 0
}
a, b := encoding[input[0]], encoding[input[1]]
bits |= a.code64() << (nbits & 63)
bits |= b.code64() << ((nbits + a.len()) & 63)
c := encoding[input[2]]
nbits += b.len() + a.len()
bits |= c.code64() << (nbits & 63)
nbits += c.len()
a, b, c := encoding[input[0]], encoding[input[1]], encoding[input[2]]
v := a.code64() | b.code64()<<(a.len()&63)
n := a.len() + b.len()
v |= c.code64() << (n & 63)
n += c.len()
bits |= v << (nbits & 63)
nbits += n
input = input[3:]
}
vendor/github.com/klauspost/compress/flate/huffman_code.go
Generated Vendored
+102 -36
View File
@@ -38,6 +38,9 @@ type huffmanEncoder struct {
// Possible lengths are codegenCodeCount, offsetCodeCount and literalCount.
// The largest of these is literalCount, so we allocate for that case.
freqcache [literalCount + 1]literalNode
// sortcache is scratch space for sortByFreq.
sortcache [literalCount + 1]literalNode
}
type literalNode struct {
@@ -139,14 +142,6 @@ func (h *huffmanEncoder) bitLength(freq []uint16) int {
return total
}
func (h *huffmanEncoder) bitLengthRaw(b []byte) int {
var total int
for _, f := range b {
total += int(h.codes[f].len())
}
return total
}
// canReuseBits returns the number of bits or math.MaxInt32 if the encoder cannot be reused.
func (h *huffmanEncoder) canReuseBits(freq []uint16) int {
var total int
@@ -309,27 +304,93 @@ func (h *huffmanEncoder) bitCounts(list []literalNode, maxBits int32) []int32 {
}
// Look at the leaves and assign them a bit count and an encoding as specified
// in RFC 1951 3.2.2
func (h *huffmanEncoder) assignEncodingAndSize(bitCount []int32, list []literalNode) {
code := uint16(0)
// in RFC 1951 3.2.2.
// list must be sorted by increasing frequency and codes must have
// all entries for literals not in list set to zero.
func (h *huffmanEncoder) assignEncodingAndSize(bitCount []int32, list []literalNode, codes []hcode) {
// The literals list[len(list)-bits] .. list[len(list)-1]
// are encoded using "bits" bits. Record the length of each literal
// temporarily in codes.
for n, bits := range bitCount {
code <<= 1
if n == 0 || bits == 0 {
continue
}
// The literals list[len(list)-bits] .. list[len(list)-bits]
// are encoded using "bits" bits, and get the values
// code, code + 1, .... The code values are
// assigned in literal order (not frequency order).
chunk := list[len(list)-int(bits):]
sortByLiteral(chunk)
for _, node := range chunk {
h.codes[node.literal] = newhcode(reverseBits(code, uint8(n)), uint8(n))
code++
for _, node := range list[len(list)-int(bits):] {
codes[node.literal] = hcode(n)
}
list = list[0 : len(list)-int(bits)]
}
// Code values are assigned in literal order (not frequency order)
// within each code length, starting from the first code of each length.
var nextCode [maxBitsLimit]uint16
code := uint16(0)
for n := 1; n < len(bitCount); n++ {
code = (code + uint16(bitCount[n-1])) << 1
nextCode[n] = code
}
for i, c := range codes {
if c == 0 {
continue
}
// n is at most maxBitsLimit-1; the mask only elides bounds checks.
n := uint8(c)
codes[i] = newhcode(reverseBits(nextCode[n&15], n), n)
nextCode[n&15]++
}
}
// sortByFreq sorts list by increasing frequency.
// The sort is stable, so nodes with equal frequencies keep their order.
func (h *huffmanEncoder) sortByFreq(list []literalNode) {
if len(list) <= 32 {
// Insertion sort is faster for small lists.
for i := 1; i < len(list); i++ {
n := list[i]
j := i
for ; j > 0 && list[j-1].freq > n.freq; j-- {
list[j] = list[j-1]
}
list[j] = n
}
return
}
// Counting sort on the low byte and then on the high byte
// of the frequency. Both passes are stable.
tmp := h.sortcache[:len(list)]
var count [256]uint16
var maxFreq uint16
for _, n := range list {
count[n.freq&0xff]++
maxFreq |= n.freq
}
var pos uint16
for i, c := range count {
count[i] = pos
pos += c
}
for _, n := range list {
tmp[count[n.freq&0xff]] = n
count[n.freq&0xff]++
}
if maxFreq < 256 {
copy(list, tmp)
return
}
count = [256]uint16{}
for _, n := range tmp {
count[n.freq>>8]++
}
pos = 0
for i, c := range count {
count[i] = pos
pos += c
}
for _, n := range tmp {
list[count[n.freq>>8]] = n
count[n.freq>>8]++
}
}
// Update this Huffman Code object to be the minimum code for the specified frequency count.
@@ -362,12 +423,14 @@ func (h *huffmanEncoder) generate(freq []uint16, maxBits int32) {
}
return
}
sortByFreq(list)
// list is in order of increasing literal value, so a stable sort
// by frequency orders it by (frequency, literal).
h.sortByFreq(list)
// Get the number of literals for each bit count
bitCount := h.bitCounts(list, maxBits)
// And do the assignment
h.assignEncodingAndSize(bitCount, list)
h.assignEncodingAndSize(bitCount, list, codes)
}
// atLeastOne clamps the result between 1 and 15.
@@ -394,24 +457,27 @@ func histogram(b []byte, h []uint16) {
}
func histogramSplit(b []byte, h []uint16) {
// Tested, and slightly faster than 2-way.
// Writing to separate arrays and combining is also slightly slower.
h = h[:256]
// Walk four quarters in parallel, each into its own table.
// Tested to be faster than walking halves. Using separate tables
// avoids the load/store dependencies that arise when two of the
// streams hit the same byte value, which is common for text.
var h0, h1, h2, h3 [256]uint16
// Make size divisible by 4
for len(b)&3 != 0 {
h[b[0]]++
h0[b[0]]++
b = b[1:]
}
n := len(b) / 4
x, y, z, w := b[:n], b[n:], b[n+n:], b[n+n+n:]
y, z, w = y[:len(x)], z[:len(x)], w[:len(x)]
for i, t := range x {
v0 := &h[t]
v1 := &h[y[i]]
v2 := &h[z[i]]
v3 := &h[w[i]]
*v0++
*v1++
*v2++
*v3++
h0[t]++
h1[y[i]]++
h2[z[i]]++
h3[w[i]]++
}
h = h[:256]
for i := range h {
h[i] += h0[i] + h1[i] + h2[i] + h3[i]
}
}
vendor/github.com/klauspost/compress/flate/huffman_sortByFreq.go
Generated Vendored
-159
View File
@@ -1,159 +0,0 @@
// Copyright 2009 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package flate
// Sort sorts data.
// It makes one call to data.Len to determine n, and O(n*log(n)) calls to
// data.Less and data.Swap. The sort is not guaranteed to be stable.
func sortByFreq(data []literalNode) {
n := len(data)
quickSortByFreq(data, 0, n, maxDepth(n))
}
func quickSortByFreq(data []literalNode, a, b, maxDepth int) {
for b-a > 12 { // Use ShellSort for slices <= 12 elements
if maxDepth == 0 {
heapSort(data, a, b)
return
}
maxDepth--
mlo, mhi := doPivotByFreq(data, a, b)
// Avoiding recursion on the larger subproblem guarantees
// a stack depth of at most lg(b-a).
if mlo-a < b-mhi {
quickSortByFreq(data, a, mlo, maxDepth)
a = mhi // i.e., quickSortByFreq(data, mhi, b)
} else {
quickSortByFreq(data, mhi, b, maxDepth)
b = mlo // i.e., quickSortByFreq(data, a, mlo)
}
}
if b-a > 1 {
// Do ShellSort pass with gap 6
// It could be written in this simplified form cause b-a <= 12
for i := a + 6; i < b; i++ {
if data[i].freq == data[i-6].freq && data[i].literal < data[i-6].literal || data[i].freq < data[i-6].freq {
data[i], data[i-6] = data[i-6], data[i]
}
}
insertionSortByFreq(data, a, b)
}
}
func doPivotByFreq(data []literalNode, lo, hi int) (midlo, midhi int) {
m := int(uint(lo+hi) >> 1) // Written like this to avoid integer overflow.
if hi-lo > 40 {
// Tukey's ``Ninther,'' median of three medians of three.
s := (hi - lo) / 8
medianOfThreeSortByFreq(data, lo, lo+s, lo+2*s)
medianOfThreeSortByFreq(data, m, m-s, m+s)
medianOfThreeSortByFreq(data, hi-1, hi-1-s, hi-1-2*s)
}
medianOfThreeSortByFreq(data, lo, m, hi-1)
// Invariants are:
// data[lo] = pivot (set up by ChoosePivot)
// data[lo < i < a] < pivot
// data[a <= i < b] <= pivot
// data[b <= i < c] unexamined
// data[c <= i < hi-1] > pivot
// data[hi-1] >= pivot
pivot := lo
a, c := lo+1, hi-1
for ; a < c && (data[a].freq == data[pivot].freq && data[a].literal < data[pivot].literal || data[a].freq < data[pivot].freq); a++ {
}
b := a
for {
for ; b < c && (data[pivot].freq == data[b].freq && data[pivot].literal > data[b].literal || data[pivot].freq > data[b].freq); b++ { // data[b] <= pivot
}
for ; b < c && (data[pivot].freq == data[c-1].freq && data[pivot].literal < data[c-1].literal || data[pivot].freq < data[c-1].freq); c-- { // data[c-1] > pivot
}
if b >= c {
break
}
// data[b] > pivot; data[c-1] <= pivot
data[b], data[c-1] = data[c-1], data[b]
b++
c--
}
// If hi-c<3 then there are duplicates (by property of median of nine).
// Let's be a bit more conservative, and set border to 5.
protect := hi-c < 5
if !protect && hi-c < (hi-lo)/4 {
// Lets test some points for equality to pivot
dups := 0
if data[pivot].freq == data[hi-1].freq && data[pivot].literal > data[hi-1].literal || data[pivot].freq > data[hi-1].freq { // data[hi-1] = pivot
data[c], data[hi-1] = data[hi-1], data[c]
c++
dups++
}
if data[b-1].freq == data[pivot].freq && data[b-1].literal > data[pivot].literal || data[b-1].freq > data[pivot].freq { // data[b-1] = pivot
b--
dups++
}
// m-lo = (hi-lo)/2 > 6
// b-lo > (hi-lo)*3/4-1 > 8
// ==> m < b ==> data[m] <= pivot
if data[m].freq == data[pivot].freq && data[m].literal > data[pivot].literal || data[m].freq > data[pivot].freq { // data[m] = pivot
data[m], data[b-1] = data[b-1], data[m]
b--
dups++
}
// if at least 2 points are equal to pivot, assume skewed distribution
protect = dups > 1
}
if protect {
// Protect against a lot of duplicates
// Add invariant:
// data[a <= i < b] unexamined
// data[b <= i < c] = pivot
for {
for ; a < b && (data[b-1].freq == data[pivot].freq && data[b-1].literal > data[pivot].literal || data[b-1].freq > data[pivot].freq); b-- { // data[b] == pivot
}
for ; a < b && (data[a].freq == data[pivot].freq && data[a].literal < data[pivot].literal || data[a].freq < data[pivot].freq); a++ { // data[a] < pivot
}
if a >= b {
break
}
// data[a] == pivot; data[b-1] < pivot
data[a], data[b-1] = data[b-1], data[a]
a++
b--
}
}
// Swap pivot into middle
data[pivot], data[b-1] = data[b-1], data[pivot]
return b - 1, c
}
// Insertion sort
func insertionSortByFreq(data []literalNode, a, b int) {
for i := a + 1; i < b; i++ {
for j := i; j > a && (data[j].freq == data[j-1].freq && data[j].literal < data[j-1].literal || data[j].freq < data[j-1].freq); j-- {
data[j], data[j-1] = data[j-1], data[j]
}
}
}
// quickSortByFreq, loosely following Bentley and McIlroy,
// ``Engineering a Sort Function,'' SP&E November 1993.
// medianOfThreeSortByFreq moves the median of the three values data[m0], data[m1], data[m2] into data[m1].
func medianOfThreeSortByFreq(data []literalNode, m1, m0, m2 int) {
// sort 3 elements
if data[m1].freq == data[m0].freq && data[m1].literal < data[m0].literal || data[m1].freq < data[m0].freq {
data[m1], data[m0] = data[m0], data[m1]
}
// data[m0] <= data[m1]
if data[m2].freq == data[m1].freq && data[m2].literal < data[m1].literal || data[m2].freq < data[m1].freq {
data[m2], data[m1] = data[m1], data[m2]
// data[m0] <= data[m2] && data[m1] < data[m2]
if data[m1].freq == data[m0].freq && data[m1].literal < data[m0].literal || data[m1].freq < data[m0].freq {
data[m1], data[m0] = data[m0], data[m1]
}
}
// now data[m0] <= data[m1] <= data[m2]
}
vendor/github.com/klauspost/compress/flate/huffman_sortByLiteral.go
Generated Vendored
-201
View File
@@ -1,201 +0,0 @@
// Copyright 2009 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package flate
// Sort sorts data.
// It makes one call to data.Len to determine n, and O(n*log(n)) calls to
// data.Less and data.Swap. The sort is not guaranteed to be stable.
func sortByLiteral(data []literalNode) {
n := len(data)
quickSort(data, 0, n, maxDepth(n))
}
func quickSort(data []literalNode, a, b, maxDepth int) {
for b-a > 12 { // Use ShellSort for slices <= 12 elements
if maxDepth == 0 {
heapSort(data, a, b)
return
}
maxDepth--
mlo, mhi := doPivot(data, a, b)
// Avoiding recursion on the larger subproblem guarantees
// a stack depth of at most lg(b-a).
if mlo-a < b-mhi {
quickSort(data, a, mlo, maxDepth)
a = mhi // i.e., quickSort(data, mhi, b)
} else {
quickSort(data, mhi, b, maxDepth)
b = mlo // i.e., quickSort(data, a, mlo)
}
}
if b-a > 1 {
// Do ShellSort pass with gap 6
// It could be written in this simplified form cause b-a <= 12
for i := a + 6; i < b; i++ {
if data[i].literal < data[i-6].literal {
data[i], data[i-6] = data[i-6], data[i]
}
}
insertionSort(data, a, b)
}
}
func heapSort(data []literalNode, a, b int) {
first := a
lo := 0
hi := b - a
// Build heap with greatest element at top.
for i := (hi - 1) / 2; i >= 0; i-- {
siftDown(data, i, hi, first)
}
// Pop elements, largest first, into end of data.
for i := hi - 1; i >= 0; i-- {
data[first], data[first+i] = data[first+i], data[first]
siftDown(data, lo, i, first)
}
}
// siftDown implements the heap property on data[lo, hi).
// first is an offset into the array where the root of the heap lies.
func siftDown(data []literalNode, lo, hi, first int) {
root := lo
for {
child := 2*root + 1
if child >= hi {
break
}
if child+1 < hi && data[first+child].literal < data[first+child+1].literal {
child++
}
if data[first+root].literal > data[first+child].literal {
return
}
data[first+root], data[first+child] = data[first+child], data[first+root]
root = child
}
}
func doPivot(data []literalNode, lo, hi int) (midlo, midhi int) {
m := int(uint(lo+hi) >> 1) // Written like this to avoid integer overflow.
if hi-lo > 40 {
// Tukey's ``Ninther,'' median of three medians of three.
s := (hi - lo) / 8
medianOfThree(data, lo, lo+s, lo+2*s)
medianOfThree(data, m, m-s, m+s)
medianOfThree(data, hi-1, hi-1-s, hi-1-2*s)
}
medianOfThree(data, lo, m, hi-1)
// Invariants are:
// data[lo] = pivot (set up by ChoosePivot)
// data[lo < i < a] < pivot
// data[a <= i < b] <= pivot
// data[b <= i < c] unexamined
// data[c <= i < hi-1] > pivot
// data[hi-1] >= pivot
pivot := lo
a, c := lo+1, hi-1
for ; a < c && data[a].literal < data[pivot].literal; a++ {
}
b := a
for {
for ; b < c && data[pivot].literal > data[b].literal; b++ { // data[b] <= pivot
}
for ; b < c && data[pivot].literal < data[c-1].literal; c-- { // data[c-1] > pivot
}
if b >= c {
break
}
// data[b] > pivot; data[c-1] <= pivot
data[b], data[c-1] = data[c-1], data[b]
b++
c--
}
// If hi-c<3 then there are duplicates (by property of median of nine).
// Let's be a bit more conservative, and set border to 5.
protect := hi-c < 5
if !protect && hi-c < (hi-lo)/4 {
// Lets test some points for equality to pivot
dups := 0
if data[pivot].literal > data[hi-1].literal { // data[hi-1] = pivot
data[c], data[hi-1] = data[hi-1], data[c]
c++
dups++
}
if data[b-1].literal > data[pivot].literal { // data[b-1] = pivot
b--
dups++
}
// m-lo = (hi-lo)/2 > 6
// b-lo > (hi-lo)*3/4-1 > 8
// ==> m < b ==> data[m] <= pivot
if data[m].literal > data[pivot].literal { // data[m] = pivot
data[m], data[b-1] = data[b-1], data[m]
b--
dups++
}
// if at least 2 points are equal to pivot, assume skewed distribution
protect = dups > 1
}
if protect {
// Protect against a lot of duplicates
// Add invariant:
// data[a <= i < b] unexamined
// data[b <= i < c] = pivot
for {
for ; a < b && data[b-1].literal > data[pivot].literal; b-- { // data[b] == pivot
}
for ; a < b && data[a].literal < data[pivot].literal; a++ { // data[a] < pivot
}
if a >= b {
break
}
// data[a] == pivot; data[b-1] < pivot
data[a], data[b-1] = data[b-1], data[a]
a++
b--
}
}
// Swap pivot into middle
data[pivot], data[b-1] = data[b-1], data[pivot]
return b - 1, c
}
// Insertion sort
func insertionSort(data []literalNode, a, b int) {
for i := a + 1; i < b; i++ {
for j := i; j > a && data[j].literal < data[j-1].literal; j-- {
data[j], data[j-1] = data[j-1], data[j]
}
}
}
// maxDepth returns a threshold at which quicksort should switch
// to heapsort. It returns 2*ceil(lg(n+1)).
func maxDepth(n int) int {
var depth int
for i := n; i > 0; i >>= 1 {
depth++
}
return depth * 2
}
// medianOfThree moves the median of the three values data[m0], data[m1], data[m2] into data[m1].
func medianOfThree(data []literalNode, m1, m0, m2 int) {
// sort 3 elements
if data[m1].literal < data[m0].literal {
data[m1], data[m0] = data[m0], data[m1]
}
// data[m0] <= data[m1]
if data[m2].literal < data[m1].literal {
data[m2], data[m1] = data[m1], data[m2]
// data[m0] <= data[m2] && data[m1] < data[m2]
if data[m1].literal < data[m0].literal {
data[m1], data[m0] = data[m0], data[m1]
}
}
// now data[m0] <= data[m1] <= data[m2]
}
vendor/github.com/klauspost/compress/flate/inflate_gen.go
Generated Vendored
+15 -15
View File
@@ -120,7 +120,7 @@ readLiteral:
if debugDecode {
fmt.Println("morebits n>0:", err)
}
f.err = err
f.err = noEOF(err)
return
}
f.roffset++
@@ -148,7 +148,7 @@ readLiteral:
if debugDecode {
fmt.Println("morebits f.nb<5:", err)
}
f.err = err
f.err = noEOF(err)
return
}
f.roffset++
@@ -216,7 +216,7 @@ readLiteral:
if debugDecode {
fmt.Println("morebits f.nb<nb:", err)
}
f.err = err
f.err = noEOF(err)
return
}
f.roffset++
@@ -371,7 +371,7 @@ readLiteral:
if debugDecode {
fmt.Println("morebits n>0:", err)
}
f.err = err
f.err = noEOF(err)
return
}
f.roffset++
@@ -399,7 +399,7 @@ readLiteral:
if debugDecode {
fmt.Println("morebits f.nb<5:", err)
}
f.err = err
f.err = noEOF(err)
return
}
f.roffset++
@@ -467,7 +467,7 @@ readLiteral:
if debugDecode {
fmt.Println("morebits f.nb<nb:", err)
}
f.err = err
f.err = noEOF(err)
return
}
f.roffset++
@@ -638,7 +638,7 @@ readLiteral:
if debugDecode {
fmt.Println("morebits n>0:", err)
}
f.err = err
f.err = noEOF(err)
return
}
}
@@ -674,7 +674,7 @@ readLiteral:
if debugDecode {
fmt.Println("morebits f.nb<5:", err)
}
f.err = err
f.err = noEOF(err)
return
}
}
@@ -757,7 +757,7 @@ readLiteral:
if debugDecode {
fmt.Println("morebits f.nb<nb:", err)
}
f.err = err
f.err = noEOF(err)
return
}
}
@@ -918,7 +918,7 @@ readLiteral:
if debugDecode {
fmt.Println("morebits n>0:", err)
}
f.err = err
f.err = noEOF(err)
return
}
f.roffset++
@@ -946,7 +946,7 @@ readLiteral:
if debugDecode {
fmt.Println("morebits f.nb<5:", err)
}
f.err = err
f.err = noEOF(err)
return
}
f.roffset++
@@ -1014,7 +1014,7 @@ readLiteral:
if debugDecode {
fmt.Println("morebits f.nb<nb:", err)
}
f.err = err
f.err = noEOF(err)
return
}
f.roffset++
@@ -1169,7 +1169,7 @@ readLiteral:
if debugDecode {
fmt.Println("morebits n>0:", err)
}
f.err = err
f.err = noEOF(err)
return
}
f.roffset++
@@ -1197,7 +1197,7 @@ readLiteral:
if debugDecode {
fmt.Println("morebits f.nb<5:", err)
}
f.err = err
f.err = noEOF(err)
return
}
f.roffset++
@@ -1265,7 +1265,7 @@ readLiteral:
if debugDecode {
fmt.Println("morebits f.nb<nb:", err)
}
f.err = err
f.err = noEOF(err)
return
}
f.roffset++
vendor/github.com/klauspost/compress/flate/matchlen_generic.go
Generated Vendored
+6 -7
View File
@@ -12,18 +12,17 @@ import (
// matchLen returns the maximum common prefix length of a and b.
// a must be the shortest of the two.
func matchLen(a, b []byte) (n int) {
left := len(a)
for left >= 8 {
diff := le.Load64(a, n) ^ le.Load64(b, n)
// Keeping a and b the same length and reslicing them as we go
// lets the compiler prove that no bounds checks are needed.
b = b[:len(a)]
for len(a) >= 8 {
diff := le.Load64(a, 0) ^ le.Load64(b, 0)
if diff != 0 {
return n + bits.TrailingZeros64(diff)>>3
}
a, b = a[8:], b[8:]
n += 8
left -= 8
}
a = a[n:]
b = b[n:]
for i := range a {
if a[i] != b[i] {
break
vendor/github.com/klauspost/compress/flate/token.go
Generated Vendored
+11 -7
View File
@@ -215,7 +215,11 @@ func mFastLog2(val float32) float32 {
ux &= -0x7f800001
ux += 127 << 23
uval := math.Float32frombits(uint32(ux))
log2 += ((-0.34484843)*uval+2.02466578)*uval - 0.67487759
p := float32(-0.34484843*uval) + 2.02466578
q := float32(p*uval) - 0.67487759
log2 += q
return log2
}
@@ -232,7 +236,8 @@ func (t *tokens) EstimatedBits() int {
for _, v := range t.litHist[:] {
if v > 0 {
n := float32(v)
shannon += atLeastOne(-mFastLog2(n*invTotal)) * n
w := float32(atLeastOne(-mFastLog2(n*invTotal)) * n)
shannon += w
}
}
// Just add 15 for EOB
@@ -240,7 +245,8 @@ func (t *tokens) EstimatedBits() int {
for i, v := range t.extraHist[1 : literalCount-256] {
if v > 0 {
n := float32(v)
shannon += atLeastOne(-mFastLog2(n*invTotal)) * n
w := float32(atLeastOne(-mFastLog2(n*invTotal)) * n)
shannon += w
bits += int(lengthExtraBits[i&31]) * int(v)
nMatches += int(v)
}
@@ -251,7 +257,8 @@ func (t *tokens) EstimatedBits() int {
for i, v := range t.offHist[:offsetCodeCount] {
if v > 0 {
n := float32(v)
shannon += atLeastOne(-mFastLog2(n*invTotal)) * n
w := float32(atLeastOne(-mFastLog2(n*invTotal)) * n)
shannon += w
bits += int(offsetExtraBits[i&31]) * int(v)
}
}
@@ -358,9 +365,6 @@ func (t token) offset() uint32 { return uint32(t) & offsetMask }
func (t token) length() uint8 { return uint8(t >> lengthShift) }
// Convert length to code.
func lengthCode(len uint8) uint8 { return lengthCodes[len] }
// Returns the offset code corresponding to a specific offset
func offsetCode(off uint32) uint32 {
if false {
vendor/github.com/klauspost/compress/fse/compress.go
Generated Vendored
+2 -1
View File
@@ -659,7 +659,8 @@ func (s *Scratch) validateNorm() (err error) {
if v >= 0 {
total += int(v)
} else {
total -= int(v)
// -1 takes one cell. Adding -v would let a wrapped -32768 pass.
total++
}
}
defer func() {
vendor/github.com/klauspost/compress/fse/decompress.go
Generated Vendored
+6
View File
@@ -3,6 +3,7 @@ package fse
import (
"errors"
"fmt"
"math"
)
const (
@@ -123,6 +124,11 @@ func (s *Scratch) readNCount() error {
remaining -= count
gotTotal += count
}
if count > math.MaxInt16 {
// Only one symbol filling a tableLog 15 table, which cannot be decoded.
// Truncated, it would leave the decoding table unwritten.
return fmt.Errorf("corruption detected (count %d > %d)", count, math.MaxInt16)
}
s.norm[charnum&0xff] = int16(count)
charnum++
previous0 = count == 0
vendor/github.com/klauspost/compress/huff0/bitreader.go
Generated Vendored
+63
View File
@@ -9,6 +9,7 @@ import (
"errors"
"fmt"
"io"
"math/bits"
"github.com/klauspost/compress/internal/le"
)
@@ -210,6 +211,68 @@ func (b *bitReaderShifted) remaining() uint {
return b.off*8 + uint(64-b.bitsRead)
}
// canUseAsm reports whether the reader has a full 8-byte window ahead of
// its read pointer, which the Decompress4X asm loops need to take over.
func (b *bitReaderShifted) canUseAsm() bool {
return b.off >= 8
}
// prepareForAsm establishes the invariant the Decompress4X asm loops rely
// on: value == load64(in[off:off+8]) << bitsRead with bitsRead <= 7, so
// that a full group of symbols can never shift their sentinel bit out of
// the container. init leaves bitsRead == 8 when the final byte of the
// stream is exactly 0x01; that whole byte is consumed, so the same position
// is the window one byte lower with nothing consumed. Requires canUseAsm.
func (b *bitReaderShifted) prepareForAsm() {
if b.bitsRead >= 8 {
b.off--
b.value = le.Load64(b.in, b.off)
b.bitsRead -= 8
}
}
// restoreFromAsm converts the state left behind by the Decompress4X asm
// loops back to the invariant the Go code relies on: value holds the 8
// bytes at in[off:off+8] shifted left by bitsRead, with the low bitsRead
// bits zero.
//
// The asm keeps a sentinel bit in value just below the unread bits, so its
// trailing zero count is the number of consumed bits. The sentinel is ORed
// over the lowest bit of the window, which the loop never consumes before
// re-reading memory, so the window is re-read here too rather than taken
// from value. The asm reports off as the signed distance from the start of
// the stream, and it can be negative: a reload always reads a whole 8-byte
// window, so a stream that is nearly drained ends with up to 7 bytes of the
// previous stream (or the jump table) below its start inside the window.
// Those bytes sit below the stream's own bits and count as consumed.
// Anything further below, or more bits consumed than the stream holds, is
// corruption.
func (b *bitReaderShifted) restoreFromAsm() error {
off := int(b.off)
consumed := uint(bits.TrailingZeros64(b.value))
if off < 0 {
if off < -7 {
return errors.New("corruption detected: stream underrun")
}
consumed += uint(-off) * 8
if consumed > 64 {
return errors.New("corruption detected: stream underrun")
}
off = 0
}
if off+8 > len(b.in) {
return errors.New("corruption detected: stream overrun")
}
b.off = uint(off)
b.bitsRead = uint8(consumed)
if consumed >= 64 {
b.value = 0
} else {
b.value = le.Load64(b.in, b.off) << consumed
}
return nil
}
// close the bitstream and returns an error if out-of-buffer reads occurred.
func (b *bitReaderShifted) close() error {
// Release reference.
vendor/github.com/klauspost/compress/huff0/bitwriter.go
Generated Vendored
+44 -49
View File
@@ -5,6 +5,50 @@
package huff0
import (
"github.com/klauspost/compress/internal/le"
"github.com/klauspost/compress/internal/regmask"
)
// bitState is a bit writer that stays in registers when passed and
// returned by value. It writes into a buffer owned by the caller.
type bitState struct {
c uint64
nb uint
pos int
}
// flush writes the whole bytes with one 8-byte store at pos, leaving
// at most 7 bits. buf must have 8 bytes of room at pos.
func (s bitState) flush(buf []byte) bitState {
le.Store64(buf, s.pos, s.c)
s.pos += int(s.nb >> 3)
s.c >>= (s.nb &^ 7) & regmask.Shift64ByUint
s.nb &= 7
return s
}
// add adds n bits from v, which must have no bits set above them.
// There must be room for them.
func (s bitState) add(v uint64, n uint) bitState {
s.c |= v << (s.nb & regmask.Shift64ByUint)
s.nb += n
return s
}
// fourSymbols returns the codes for a, b, c and d packed in that order
// from the LSB, and their total length.
func fourSymbols(a, b, c, d cTableEntry) (uint64, uint) {
bitsA := uint(a.nBits)
bitsB := bitsA + uint(b.nBits)
bitsC := bitsB + uint(c.nBits)
v := uint64(a.val) |
uint64(b.val)<<(bitsA&regmask.Shift64ByUint) |
uint64(c.val)<<(bitsB&regmask.Shift64ByUint) |
uint64(d.val)<<(bitsC&regmask.Shift64ByUint)
return v, bitsC + uint(d.nBits)
}
// bitWriter will write bits.
// First bit will be LSB of the first byte of output.
type bitWriter struct {
@@ -33,55 +77,6 @@ func (b *bitWriter) encSymbol(ct cTable, symbol byte) {
b.nBits += enc.nBits
}
// encTwoSymbols will add up to 32 bits. value may not contain more set bits than indicated.
// It will not check if there is space for them, so the caller must ensure that it has flushed recently.
func (b *bitWriter) encTwoSymbols(ct cTable, av, bv byte) {
encA := ct[av]
encB := ct[bv]
sh := b.nBits & 63
combined := uint64(encA.val) | (uint64(encB.val) << (encA.nBits & 63))
b.bitContainer |= combined << sh
if false {
if encA.nBits == 0 {
panic("nbitsA 0")
}
if encB.nBits == 0 {
panic("nbitsB 0")
}
}
b.nBits += encA.nBits + encB.nBits
}
// encFourSymbols adds up to 32 bits from four symbols.
// It will not check if there is space for them,
// so the caller must ensure that b has been flushed recently.
func (b *bitWriter) encFourSymbols(encA, encB, encC, encD cTableEntry) {
bitsA := encA.nBits
bitsB := bitsA + encB.nBits
bitsC := bitsB + encC.nBits
bitsD := bitsC + encD.nBits
combined := uint64(encA.val) |
(uint64(encB.val) << (bitsA & 63)) |
(uint64(encC.val) << (bitsB & 63)) |
(uint64(encD.val) << (bitsC & 63))
b.bitContainer |= combined << (b.nBits & 63)
b.nBits += bitsD
}
// flush32 will flush out, so there are at least 32 bits available for writing.
func (b *bitWriter) flush32() {
if b.nBits < 32 {
return
}
b.out = append(b.out,
byte(b.bitContainer),
byte(b.bitContainer>>8),
byte(b.bitContainer>>16),
byte(b.bitContainer>>24))
b.nBits -= 32
b.bitContainer >>= 32
}
// flushAlign will flush remaining full bytes and align to next byte boundary.
func (b *bitWriter) flushAlign() {
nbBytes := (b.nBits + 7) >> 3
vendor/github.com/klauspost/compress/huff0/compress.go
Generated Vendored
+18 -13
View File
@@ -4,6 +4,7 @@ import (
"fmt"
"math"
"runtime"
"slices"
"sync"
)
@@ -243,23 +244,27 @@ func (s *Scratch) compress1xDo(dst, src []byte) []byte {
bw.encSymbol(cTable, src[n+i-1])
}
n -= 4
if s.actualTableLog <= 8 {
for ; n >= 0; n -= 4 {
tmp := src[n : n+4]
// tmp should be len 4
bw.flush32()
bw.encFourSymbols(cTable[tmp[3]], cTable[tmp[2]], cTable[tmp[1]], cTable[tmp[0]])
// flush leaves at most 7 bits, so four symbols (at most 4*tableLogMax
// bits) always fit. Each chunk checks room for bytesPer4 per four
// symbols plus the 8-byte store.
const bytesPer4 = (7 + 4*tableLogMax) / 8
w := bitState{c: bw.bitContainer, nb: uint(bw.nBits), pos: len(bw.out)}
buf := bw.out[:cap(bw.out)]
for n >= 0 {
k := (len(buf) - w.pos - 8) / bytesPer4
if k <= 0 {
buf = slices.Grow(buf[:w.pos], (n/4+1)*bytesPer4+8)
buf = buf[:cap(buf)]
continue
}
} else {
for ; n >= 0; n -= 4 {
for stop := max(n-4*k, -4); n > stop; n -= 4 {
w = w.flush(buf)
tmp := src[n : n+4]
// tmp should be len 4
bw.flush32()
bw.encTwoSymbols(cTable, tmp[3], tmp[2])
bw.flush32()
bw.encTwoSymbols(cTable, tmp[1], tmp[0])
w = w.add(fourSymbols(cTable[tmp[3]], cTable[tmp[2]], cTable[tmp[1]], cTable[tmp[0]]))
}
}
bw.bitContainer, bw.nBits, bw.out = w.c, uint8(w.nb), buf[:w.pos]
bw.close()
return bw.out
}
vendor/github.com/klauspost/compress/huff0/decompress.go
Generated Vendored
+38 -11
View File
@@ -54,6 +54,10 @@ func ReadTable(in []byte, s *Scratch) (s2 *Scratch, remain []byte, err error) {
if len(in) < int(iSize) {
return s, nil, fmt.Errorf("input too small for table, want %d bytes, have %d", iSize, len(in))
}
// The low nibble is the weights' FSE tableLog - 5, which the format limits to 6.
if in[0]&15+5 > 6 {
return s, nil, errors.New("corrupt input: weights tableLog too big")
}
// FSE compressed weights
s.fse.DecompressLimit = 255
hw := s.huffWeight[:]
@@ -70,17 +74,32 @@ func ReadTable(in []byte, s *Scratch) (s2 *Scratch, remain []byte, err error) {
in = in[iSize:]
}
// collect weight stats
// collect weight stats. Weights repeat, so four histograms keep each
// increment from waiting on the previous store to the same counter.
var hist [4][16]uint32
var seen uint8
w := s.huffWeight[:s.symbolLen]
for ; len(w) >= 4; w = w[4:] {
hist[0][w[0]&15]++
hist[1][w[1]&15]++
hist[2][w[2]&15]++
hist[3][w[3]&15]++
seen |= w[0] | w[1] | w[2] | w[3]
}
for _, v := range w {
hist[0][v&15]++
seen |= v
}
var rankStats [16]uint32
weightTotal := uint32(0)
for _, v := range s.huffWeight[:s.symbolLen] {
if v > tableLogMax {
return s, nil, errors.New("corrupt input: weight too large")
}
v2 := v & 15
rankStats[v2]++
// (1 << (v2-1)) is slower since the compiler cannot prove that v2 isn't 0.
weightTotal += (1 << v2) >> 1
for i := range rankStats {
rankStats[i] = hist[0][i] + hist[1][i] + hist[2][i] + hist[3][i]
}
for i := 1; i < len(rankStats); i++ {
weightTotal += rankStats[i] << (i - 1)
}
if seen > 15 || rankStats[tableLogMax+1]|rankStats[tableLogMax+2]|rankStats[tableLogMax+3]|rankStats[tableLogMax+4] != 0 {
return s, nil, errors.New("corrupt input: weight too large")
}
if weightTotal == 0 {
return s, nil, errors.New("corrupt input: weights zero")
@@ -158,9 +177,17 @@ func ReadTable(in []byte, s *Scratch) (s2 *Scratch, remain []byte, err error) {
nBits: uint8(d.entry),
}
// length is a power of two; runs of 4 or more are filled 4 at a time.
single := s.dt.single[*rank : *rank+length]
for i := range single {
single[i] = d
if length >= 4 {
for i := 0; i < len(single); i += 4 {
s4 := single[i : i+4 : i+4]
s4[0], s4[1], s4[2], s4[3] = d, d, d, d
}
} else {
for i := range single {
single[i] = d
}
}
*rank += length
}
vendor/github.com/klauspost/compress/huff0/decompress_amd64.go
Generated Vendored
+88 -9
View File
@@ -1,4 +1,4 @@
//go:build amd64 && !appengine && !noasm && gc
//go:build amd64 && !noasm && gc
// amd64 stubs and dispatch for the asm loops used by decompress_asm.go.
package huff0
@@ -8,36 +8,99 @@ import (
)
// decompress4x_main_loop_amd64 is an x86 assembler implementation
// of Decompress4X when tablelog > 8.
// of Decompress4X when tablelog > 8, decoding fastSymbols symbols per
// stream between bit container reloads.
//
//go:noescape
func decompress4x_main_loop_amd64(ctx *decompress4xContext)
// decompress4x_8b_main_loop_amd64 is an x86 assembler implementation
// of Decompress4X when tablelog <= 8 which decodes 4 entries
// per loop.
// of Decompress4X when tablelog <= 8, decoding fast8bSymbols symbols
// per stream between bit container reloads.
//
//go:noescape
func decompress4x_8b_main_loop_amd64(ctx *decompress4xContext)
// decompress4x_4b_main_loop_amd64 is an x86 assembler implementation
// of Decompress4X when tablelog <= 4, decoding fast4bSymbols symbols
// per stream between bit container reloads.
//
//go:noescape
func decompress4x_4b_main_loop_amd64(ctx *decompress4xContext)
// decompress4x_4b_main_loop_bmi2 is the BMI2 twin of decompress4x_4b_main_loop_amd64.
//
//go:noescape
func decompress4x_4b_main_loop_bmi2(ctx *decompress4xContext)
// decompress4x_main_loop_bmi2 is the BMI2 twin of decompress4x_main_loop_amd64.
//
//go:noescape
func decompress4x_main_loop_bmi2(ctx *decompress4xContext)
// decompress4x_8b_main_loop_bmi2 is the BMI2 twin of decompress4x_8b_main_loop_amd64.
//
//go:noescape
func decompress4x_8b_main_loop_bmi2(ctx *decompress4xContext)
// decompress1x_main_loop_amd64 is an x86 assembler implementation
// of Decompress1X when tablelog > 8.
// of Decompress1X when tablelog > 8, decoding fastSymbols symbols
// between bit container reloads.
//
//go:noescape
func decompress1x_main_loop_amd64(ctx *decompress1xContext)
// decompress1x_main_loop_bmi2 is an x86 with BMI2 assembler implementation
// of Decompress1X when tablelog > 8.
// decompress1x_8b_main_loop_amd64 is an x86 assembler implementation
// of Decompress1X when tablelog <= 8, decoding fast8bSymbols symbols
// between bit container reloads.
//
//go:noescape
func decompress1x_8b_main_loop_amd64(ctx *decompress1xContext)
// decompress1x_4b_main_loop_amd64 is an x86 assembler implementation
// of Decompress1X when tablelog <= 4, decoding fast4bSymbols symbols
// between bit container reloads.
//
//go:noescape
func decompress1x_4b_main_loop_amd64(ctx *decompress1xContext)
// decompress1x_main_loop_bmi2 is the BMI2 twin of decompress1x_main_loop_amd64.
//
//go:noescape
func decompress1x_main_loop_bmi2(ctx *decompress1xContext)
// decompress1x_8b_main_loop_bmi2 is the BMI2 twin of decompress1x_8b_main_loop_amd64.
//
//go:noescape
func decompress1x_8b_main_loop_bmi2(ctx *decompress1xContext)
// decompress1x_4b_main_loop_bmi2 is the BMI2 twin of decompress1x_4b_main_loop_amd64.
//
//go:noescape
func decompress1x_4b_main_loop_bmi2(ctx *decompress1xContext)
func decompress4x_main_loop_asm(ctx *decompress4xContext) {
decompress4x_main_loop_amd64(ctx)
if cpuinfo.HasBMI2() {
decompress4x_main_loop_bmi2(ctx)
} else {
decompress4x_main_loop_amd64(ctx)
}
}
func decompress4x_8b_main_loop_asm(ctx *decompress4xContext) {
decompress4x_8b_main_loop_amd64(ctx)
if cpuinfo.HasBMI2() {
decompress4x_8b_main_loop_bmi2(ctx)
} else {
decompress4x_8b_main_loop_amd64(ctx)
}
}
func decompress4x_4b_main_loop_asm(ctx *decompress4xContext) {
if cpuinfo.HasBMI2() {
decompress4x_4b_main_loop_bmi2(ctx)
} else {
decompress4x_4b_main_loop_amd64(ctx)
}
}
func decompress1x_main_loop_asm(ctx *decompress1xContext) {
@@ -47,3 +110,19 @@ func decompress1x_main_loop_asm(ctx *decompress1xContext) {
decompress1x_main_loop_amd64(ctx)
}
}
func decompress1x_8b_main_loop_asm(ctx *decompress1xContext) {
if cpuinfo.HasBMI2() {
decompress1x_8b_main_loop_bmi2(ctx)
} else {
decompress1x_8b_main_loop_amd64(ctx)
}
}
func decompress1x_4b_main_loop_asm(ctx *decompress1xContext) {
if cpuinfo.HasBMI2() {
decompress1x_4b_main_loop_bmi2(ctx)
} else {
decompress1x_4b_main_loop_amd64(ctx)
}
}
vendor/github.com/klauspost/compress/huff0/decompress_amd64.s
Generated Vendored
+3342 -774
View File
File diff suppressed because it is too large. Load diff
vendor/github.com/klauspost/compress/huff0/decompress_arm64.go
Generated Vendored
+40 -5
View File
@@ -1,4 +1,4 @@
//go:build arm64 && !appengine && !noasm && gc
//go:build arm64 && !noasm && gc
// arm64 stubs and dispatch for the asm loops used by decompress_asm.go.
// The asm (decompress_arm64.s) is generated by the avo arm64 lowering
@@ -6,24 +6,47 @@
package huff0
// decompress4x_main_loop_arm64 is an arm64 assembler implementation
// of Decompress4X when tablelog > 8.
// of Decompress4X when tablelog > 8, decoding fastSymbols symbols per
// stream between bit container reloads.
//
//go:noescape
func decompress4x_main_loop_arm64(ctx *decompress4xContext)
// decompress4x_8b_main_loop_arm64 is an arm64 assembler implementation
// of Decompress4X when tablelog <= 8 which decodes 4 entries
// per loop.
// of Decompress4X when tablelog <= 8, decoding fast8bSymbols symbols
// per stream between bit container reloads.
//
//go:noescape
func decompress4x_8b_main_loop_arm64(ctx *decompress4xContext)
// decompress4x_4b_main_loop_arm64 is an arm64 assembler implementation
// of Decompress4X when tablelog <= 4, decoding fast4bSymbols symbols
// per stream between bit container reloads.
//
//go:noescape
func decompress4x_4b_main_loop_arm64(ctx *decompress4xContext)
// decompress1x_main_loop_arm64 is an arm64 assembler implementation
// of Decompress1X when tablelog > 8.
// of Decompress1X when tablelog > 8, decoding fastSymbols symbols
// between bit container reloads.
//
//go:noescape
func decompress1x_main_loop_arm64(ctx *decompress1xContext)
// decompress1x_8b_main_loop_arm64 is an arm64 assembler implementation
// of Decompress1X when tablelog <= 8, decoding fast8bSymbols symbols
// between bit container reloads.
//
//go:noescape
func decompress1x_8b_main_loop_arm64(ctx *decompress1xContext)
// decompress1x_4b_main_loop_arm64 is an arm64 assembler implementation
// of Decompress1X when tablelog <= 4, decoding fast4bSymbols symbols
// between bit container reloads.
//
//go:noescape
func decompress1x_4b_main_loop_arm64(ctx *decompress1xContext)
func decompress4x_main_loop_asm(ctx *decompress4xContext) {
decompress4x_main_loop_arm64(ctx)
}
@@ -32,6 +55,18 @@ func decompress4x_8b_main_loop_asm(ctx *decompress4xContext) {
decompress4x_8b_main_loop_arm64(ctx)
}
func decompress4x_4b_main_loop_asm(ctx *decompress4xContext) {
decompress4x_4b_main_loop_arm64(ctx)
}
func decompress1x_main_loop_asm(ctx *decompress1xContext) {
decompress1x_main_loop_arm64(ctx)
}
func decompress1x_8b_main_loop_asm(ctx *decompress1xContext) {
decompress1x_8b_main_loop_arm64(ctx)
}
func decompress1x_4b_main_loop_asm(ctx *decompress1xContext) {
decompress1x_4b_main_loop_arm64(ctx)
}
vendor/github.com/klauspost/compress/huff0/decompress_arm64.s
Generated Vendored
+1637 -770
View File
File diff suppressed because it is too large. Load diff
vendor/github.com/klauspost/compress/huff0/decompress_asm.go
Generated Vendored
+80 -20
View File
@@ -1,4 +1,4 @@
//go:build (amd64 || arm64) && !appengine && !noasm && gc
//go:build (amd64 || arm64) && !noasm && gc
// This file contains the specialisation of Decoder.Decompress4X
// and Decoder.Decompress1X that use an asm implementation of their main loops.
@@ -14,6 +14,10 @@ import (
// fallback8BitSize is the size where using Go version is faster.
const fallback8BitSize = 800
// decompress4xContext is the argument block of the Decompress4X asm loops.
// Go fills every field but decoded and inner; the asm advances ip, and on
// return leaves each bit reader in the form bitReaderShifted.restoreFromAsm
// expects.
type decompress4xContext struct {
pbr *[4]bitReaderShifted
peekBits uint8
@@ -21,9 +25,20 @@ type decompress4xContext struct {
dstEvery int
tbl *dEntrySingle
decoded int
limit *byte
limit *byte // stream 0's output pointer must stay below this
ilowest *byte // start of the input block; no read goes below it
ip [4]*byte // each stream's 8-byte input window, advanced by the asm
inner *byte // scratch for the asm: the current inner-loop limit
}
// Symbols decoded per stream between reloads by the 4X asm loops; must
// match the constants of the same names in _generate/gen.go.
const (
fastSymbols = 5 // tablelog 9..11
fast8bSymbols = 7 // tablelog 5..8
fast4bSymbols = 14 // tablelog <= 4
)
// Decompress4X will decompress a 4X encoded stream.
// The length of the supplied input must match the end of a block exactly.
// The *capacity* of the dst slice must match the destination size of
@@ -72,23 +87,55 @@ func (d *Decoder) Decompress4X(dst, src []byte) ([]byte, error) {
var decoded int
if len(out) > 4*4 && !(br[0].off < 4 || br[1].off < 4 || br[2].off < 4 || br[3].off < 4) {
nSyms := fastSymbols
if d.actualTableLog <= 4 {
nSyms = fast4bSymbols
} else if use8BitTables {
nSyms = fast8bSymbols
}
// The asm writes nSyms bytes per stream per iteration and only re-checks
// its bounds between batches of iterations (the batch size is derived
// from limit in _generate/gen.go), so stream 0 must stop early enough
// that stream 3 (which may be up to 3 bytes shorter than dstEvery)
// never writes past the end of out. Every stream needs a full 8-byte
// window ahead of its read pointer to enter the loop.
if limit := dstEvery - nSyms - 2; limit > 0 && br[0].canUseAsm() && br[1].canUseAsm() && br[2].canUseAsm() && br[3].canUseAsm() {
for i := range br {
br[i].prepareForAsm()
}
ctx := decompress4xContext{
pbr: &br,
peekBits: uint8((64 - d.actualTableLog) & 63), // see: bitReaderShifted.peekBitsFast()
out: &out[0],
dstEvery: dstEvery,
tbl: &single[0],
limit: &out[dstEvery-4], // Always stop decoding when first buffer gets here to avoid writing OOB on last.
limit: &out[limit],
// The 6-byte jump table sits below stream 0 inside src, so the
// lowest window may reach into it; restoreFromAsm accounts for
// bytes below a stream's start. Bounding by src rather than by
// stream 0 keeps the asm running about six bytes longer, which
// matters for streams with very short codes.
ilowest: &src[0],
}
if use8BitTables {
for i := range br {
ctx.ip[i] = &br[i].in[br[i].off]
}
switch nSyms {
case fast4bSymbols:
decompress4x_4b_main_loop_asm(&ctx)
case fast8bSymbols:
decompress4x_8b_main_loop_asm(&ctx)
} else {
default:
decompress4x_main_loop_asm(&ctx)
}
decoded = ctx.decoded
out = out[decoded/4:]
for i := range br {
if err := br[i].restoreFromAsm(); err != nil {
return nil, err
}
}
}
// Decode remaining.
@@ -128,18 +175,20 @@ func (d *Decoder) Decompress4X(dst, src []byte) ([]byte, error) {
return dst, nil
}
// decompress1xContext is the argument block of the Decompress1X asm loops.
// Go fills every field but decoded; the asm advances ip, and on return
// leaves the bit reader in the form bitReaderShifted.restoreFromAsm expects.
type decompress1xContext struct {
pbr *bitReaderShifted
peekBits uint8
out *byte
outCap int
outCap int // no write reaches out[outCap]
tbl *dEntrySingle
decoded int
ilowest *byte // start of the stream; no read goes below it
ip *byte // the 8-byte input window, advanced by the asm
}
// Error reported by asm implementations
const error_max_decoded_size_exeeded = -1
// Decompress1X will decompress a 1X encoded stream.
// The cap of the output buffer will be the maximum decompressed size.
// The length of the supplied input must match the end of a block exactly.
@@ -158,25 +207,36 @@ func (d *Decoder) Decompress1X(dst, src []byte) ([]byte, error) {
const tlSize = 1 << tableLogMax
const tlMask = tlSize - 1
if maxDecodedSize >= 4 {
// The asm decodes whole batches of symbols and only re-checks its
// bounds between them, so it needs at least one batch of room (the
// output bound rounds nSyms up to 16, see _generate/gen.go) and a full
// 8-byte window ahead of the read pointer to enter the loop. It also
// needs at most 7 bits consumed on entry so that a batch cannot run
// the container dry, which prepareForAsm establishes.
decoded := 0
if maxDecodedSize >= 16 && br.canUseAsm() {
br.prepareForAsm()
ctx := decompress1xContext{
pbr: &br,
out: &dst[0],
outCap: maxDecodedSize,
peekBits: uint8((64 - d.actualTableLog) & 63), // see: bitReaderShifted.peekBitsFast()
tbl: &d.dt.single[0],
ilowest: &src[0],
ip: &br.in[br.off],
}
decompress1x_main_loop_asm(&ctx)
if ctx.decoded == error_max_decoded_size_exeeded {
return nil, ErrMaxDecodedSizeExceeded
if d.actualTableLog <= 4 {
decompress1x_4b_main_loop_asm(&ctx)
} else if d.actualTableLog <= 8 {
decompress1x_8b_main_loop_asm(&ctx)
} else {
decompress1x_main_loop_asm(&ctx)
}
dst = dst[:ctx.decoded]
decoded = ctx.decoded
}
dst = dst[:decoded]
// br < 8, so uint8 is fine
bitsLeft := uint8(br.off)*8 + 64 - br.bitsRead
bitsLeft := br.remaining()
for bitsLeft > 0 {
br.fill()
if len(dst) >= maxDecodedSize {
@@ -186,7 +246,7 @@ func (d *Decoder) Decompress1X(dst, src []byte) ([]byte, error) {
v := d.dt.single[br.peekBitsFast(d.actualTableLog)&tlMask]
nBits := uint8(v.entry)
br.advance(nBits)
bitsLeft -= nBits
bitsLeft -= uint(nBits)
dst = append(dst, uint8(v.entry>>8))
}
return dst, br.close()
vendor/github.com/klauspost/compress/huff0/decompress_generic.go
Generated Vendored
+1 -1
View File
@@ -1,4 +1,4 @@
//go:build (!amd64 && !arm64) || appengine || !gc || noasm
//go:build (!amd64 && !arm64) || !gc || noasm
// This file contains a generic implementation of Decoder.Decompress4X.
package huff0
vendor/github.com/klauspost/compress/internal/cpuinfo/cpuinfo_amd64.go
Generated Vendored
+1 -1
View File
@@ -1,4 +1,4 @@
//go:build amd64 && !appengine && !noasm && gc
//go:build amd64 && !noasm && gc
package cpuinfo
vendor/github.com/klauspost/compress/internal/cpuinfo/cpuinfo_amd64.s
Generated Vendored
+1 -3
View File
@@ -1,6 +1,4 @@
// +build !appengine
// +build gc
// +build !noasm
//go:build !noasm && gc
#include "textflag.h"
#include "funcdata.h"
vendor/github.com/klauspost/compress/internal/le/unsafe_disabled.go
Generated Vendored
+1 -1
View File
@@ -1,4 +1,4 @@
//go:build !(amd64 || arm64 || ppc64le || riscv64) || nounsafe || purego || appengine
//go:build !(amd64 || arm64 || ppc64le || riscv64) || nounsafe || purego
package le
vendor/github.com/klauspost/compress/internal/le/unsafe_enabled.go
Generated Vendored
+1 -1
View File
@@ -1,6 +1,6 @@
// We enable 64 bit LE platforms:
//go:build (amd64 || arm64 || ppc64le || riscv64) && !nounsafe && !purego && !appengine
//go:build (amd64 || arm64 || ppc64le || riscv64) && !nounsafe && !purego
package le
vendor/github.com/klauspost/compress/zstd/bitwriter.go
Generated Vendored
+39
View File
@@ -5,6 +5,45 @@
package zstd
import (
"github.com/klauspost/compress/internal/le"
"github.com/klauspost/compress/internal/regmask"
)
// bitState is a bit writer that stays in registers when passed and
// returned by value. It writes into a buffer owned by the caller.
type bitState struct {
c uint64
nb uint
pos int
}
// flush writes the whole bytes with one 8-byte store at pos, leaving
// at most 7 bits. buf must have 8 bytes of room at pos.
func (s bitState) flush(buf []byte) bitState {
le.Store64(buf, s.pos, s.c)
s.pos += int(s.nb >> 3)
s.c >>= (s.nb &^ 7) & regmask.Shift64ByUint
s.nb &= 7
return s
}
// add adds n bits from v, which must have no bits set above them.
// There must be room for them.
func (s bitState) add(v uint64, n uint) bitState {
s.c |= v << (s.nb & regmask.Shift64ByUint)
s.nb += n
return s
}
// addState adds the bits that FSE-encode tt from state, and returns
// the next state from tab.
func (s bitState) addState(state uint16, tab []uint16, tt symbolTransform) (bitState, uint16) {
nbBitsOut := (uint32(state) + tt.deltaNbBits) >> 16
s = s.add(uint64(state&bitMask16[nbBitsOut&31]), uint(nbBitsOut))
return s, tab[int32(state>>(nbBitsOut&regmask.Shift16ByUint32))+int32(tt.deltaFindState)]
}
// bitWriter will write bits.
// First bit will be LSB of the first byte of output.
type bitWriter struct {
vendor/github.com/klauspost/compress/zstd/blockdec.go
Generated Vendored
+39 -23
View File
@@ -240,9 +240,11 @@ func (b *blockDec) decodeBuf(hist *history) error {
b.dst[i] = v
}
hist.appendKeep(b.dst)
hist.decoders.consumeSyncLen(len(b.dst))
return nil
case blockTypeRaw:
hist.appendKeep(b.data)
hist.decoders.consumeSyncLen(len(b.data))
return nil
case blockTypeCompressed:
saved := b.dst
@@ -416,6 +418,8 @@ func (b *blockDec) decodeLiterals(in []byte, hist *history) (remain []byte, err
if len(literals) != litRegenSize {
return in, fmt.Errorf("literal output size mismatch want %d, got %d", litRegenSize, len(literals))
}
// Re-cap to get extra size.
literals = b.literalBuf[:len(literals)]
case literalsBlockCompressed:
if len(in) < litCompSize {
@@ -487,10 +491,24 @@ func (b *blockDec) decodeCompressed(hist *history) error {
}
if hist.decoders.nSeqs == 0 {
b.dst = append(b.dst, hist.decoders.literals...)
hist.decoders.consumeSyncLen(len(hist.decoders.literals))
return nil
}
before := len(hist.decoders.out)
err = hist.decoders.decodeSync(hist.b[hist.ignoreBuffer:])
h := hist.b[hist.ignoreBuffer:]
if hist.ext != nil {
h = hist.ext
}
if hist.dict == nil && hist.decoders.useTwoPass() {
// Two passes, as the concurrent decoder does: knowing the offsets
// up front lets executeSimple prefetch match sources.
if err := b.decodeSequences(hist); err != nil {
return err
}
err = hist.decoders.executeSimple(b.sequence, h)
} else {
err = hist.decoders.decodeSync(h)
}
if err != nil {
return err
}
@@ -601,16 +619,11 @@ func (b *blockDec) prepareSequences(in []byte, hist *history) (err error) {
if seq.fse == nil || seq.fse.preDefined {
seq.fse = fseDecoderPool.Get().(*fseDecoder)
}
err := seq.fse.readNCount(&br, uint16(maxTableSymbol[i]))
err := seq.fse.readNCount(&br, tableIndex(i))
if err != nil {
println("Read table error:", err)
return err
}
err = seq.fse.transform(symbolTableX[i])
if err != nil {
println("Transform table error:", err)
return err
}
if debugDecoder {
println("Read table ok", "symbolLen:", seq.fse.symbolLen)
}
@@ -670,38 +683,41 @@ func (b *blockDec) decodeSequences(hist *history) error {
return err
}
// executeSequences executes the block's sequences straight into the history
// ring, as decodeCompressed does, and copies the result to b.dst for output.
func (b *blockDec) executeSequences(hist *history) error {
hbytes := hist.b
if len(hbytes) > hist.windowSize {
hbytes = hbytes[len(hbytes)-hist.windowSize:]
}
hist.ensureBlockRing()
start := len(hist.b)
hist.decoders.windowSize = hist.windowSize
hist.decoders.out = b.dst[:0]
err := hist.decoders.execute(b.sequence, hbytes)
hist.decoders.out = hist.b
err := hist.decoders.execute(b.sequence, hist.ext)
if err != nil {
return err
}
return b.updateHistory(hist)
return b.updateHistory(hist, start)
}
func (b *blockDec) updateHistory(hist *history) error {
// updateHistory copies the block's output, which starts at out[start], to
// b.dst and keeps it as history.
func (b *blockDec) updateHistory(hist *history, start int) error {
if len(b.data) > maxCompressedBlockSize {
return fmt.Errorf("compressed block size too large (%d)", len(b.data))
}
// Set output and release references.
b.dst = hist.decoders.out
out := hist.decoders.out
b.dst = append(b.dst[:0], out[start:]...)
hist.recentOffsets = hist.decoders.prevOffset
if b.Last {
// if last block we don't care about history.
println("Last block, no history returned")
if debugDecoder {
println("Last block, no history returned")
}
hist.b = hist.b[:0]
return nil
} else {
hist.append(b.dst)
if debugDecoder {
println("Finished block with ", len(b.sequence), "sequences. Added", len(b.dst), "to history, now length", len(hist.b))
}
}
hist.b = out
if debugDecoder {
println("Finished block with ", len(b.sequence), "sequences. Added", len(b.dst), "to history, now length", len(hist.b))
}
hist.decoders.out, hist.decoders.literals = nil, nil
vendor/github.com/klauspost/compress/zstd/blockenc.go
Generated Vendored
+98 -79
View File
@@ -628,38 +628,6 @@ func (b *blockEnc) encode(org []byte, raw, rawAllLits bool) error {
return err
}
// Choose the best compression mode for each type.
// Will evaluate the new vs predefined and previous.
chooseComp := func(cur, prev, preDef *fseEncoder) (*fseEncoder, seqCompMode) {
// See if predefined/previous is better
hist := cur.count[:cur.symbolLen]
nSize := cur.approxSize(hist) + cur.maxHeaderSize()
predefSize := preDef.approxSize(hist)
prevSize := prev.approxSize(hist)
// Add a small penalty for new encoders.
// Don't bother with extremely small (<2 byte gains).
nSize = nSize + (nSize+2*8*16)>>4
switch {
case predefSize <= prevSize && predefSize <= nSize || forcePreDef:
if debugEncoder {
println("Using predefined", predefSize>>3, "<=", nSize>>3)
}
return preDef, compModePredefined
case prevSize <= nSize:
if debugEncoder {
println("Using previous", prevSize>>3, "<=", nSize>>3)
}
return prev, compModeRepeat
default:
if debugEncoder {
println("Using new, predef", predefSize>>3, ". previous:", prevSize>>3, ">", nSize>>3, "header max:", cur.maxHeaderSize()>>3, "bytes")
println("tl:", cur.actualTableLog, "symbolLen:", cur.symbolLen, "norm:", cur.norm[:cur.symbolLen], "hist", cur.count[:cur.symbolLen])
}
return cur, compModeFSE
}
}
// Write compression mode
var mode uint8
if llEnc.useRLE {
@@ -670,7 +638,7 @@ func (b *blockEnc) encode(org []byte, raw, rawAllLits bool) error {
}
} else {
var m seqCompMode
llEnc, m = chooseComp(llEnc, b.coders.llPrev, &fsePredefEnc[tableLiteralLengths])
llEnc, m = b.chooseComp(llEnc, b.coders.llPrev, &fsePredefEnc[tableLiteralLengths])
mode |= uint8(m) << 6
}
if ofEnc.useRLE {
@@ -681,7 +649,7 @@ func (b *blockEnc) encode(org []byte, raw, rawAllLits bool) error {
}
} else {
var m seqCompMode
ofEnc, m = chooseComp(ofEnc, b.coders.ofPrev, &fsePredefEnc[tableOffsets])
ofEnc, m = b.chooseComp(ofEnc, b.coders.ofPrev, &fsePredefEnc[tableOffsets])
mode |= uint8(m) << 4
}
@@ -693,7 +661,7 @@ func (b *blockEnc) encode(org []byte, raw, rawAllLits bool) error {
}
} else {
var m seqCompMode
mlEnc, m = chooseComp(mlEnc, b.coders.mlPrev, &fsePredefEnc[tableMatchLengths])
mlEnc, m = b.chooseComp(mlEnc, b.coders.mlPrev, &fsePredefEnc[tableMatchLengths])
mode |= uint8(m) << 2
}
b.output = append(b.output, mode)
@@ -753,54 +721,57 @@ func (b *blockEnc) encode(org []byte, raw, rawAllLits bool) error {
println("Encoded seq", seq, s, "codes:", s.llCode, s.mlCode, s.ofCode, "states:", ll.state, ml.state, of.state, "bits:", llB, mlB, ofB)
}
seq--
// Store sequences in reverse...
wr.flush32()
// Store sequences in reverse, with the bit writer and states in locals.
// flush stores 8 bytes and advances by whole bytes, so each chunk
// checks room for seqMaxBytes per sequence plus the store's slack.
const seqMaxBytes = 12 // 26 state bits + 63 extra bits.
w := bitState{c: wr.bitContainer, nb: uint(wr.nBits), pos: len(wr.out)}
buf := wr.out[:cap(wr.out)]
ofState, mlState, llState := of.state, ml.state, ll.state
ofTab, mlTab, llTab := of.stateTable, ml.stateTable, ll.stateTable
seqs := b.sequences
for seq >= 0 {
s = b.sequences[seq]
ofB := ofTT[s.ofCode]
wr.flush32() // tablelog max is below 8 for each, so it will fill max 24 bits.
//of.encode(ofB)
nbBitsOut := (uint32(of.state) + ofB.deltaNbBits) >> 16
dstState := int32(of.state>>(nbBitsOut&15)) + int32(ofB.deltaFindState)
wr.addBits16NC(of.state, uint8(nbBitsOut))
of.state = of.stateTable[dstState]
// Accumulate extra bits.
outBits := ofB.outBits & 31
extraBits := uint64(s.offset & bitMask32[outBits])
extraBitsN := outBits
mlB := mlTT[s.mlCode]
//ml.encode(mlB)
nbBitsOut = (uint32(ml.state) + mlB.deltaNbBits) >> 16
dstState = int32(ml.state>>(nbBitsOut&15)) + int32(mlB.deltaFindState)
wr.addBits16NC(ml.state, uint8(nbBitsOut))
ml.state = ml.stateTable[dstState]
outBits = mlB.outBits & 31
extraBits = extraBits<<outBits | uint64(s.matchLen&bitMask32[outBits])
extraBitsN += outBits
llB := llTT[s.llCode]
//ll.encode(llB)
nbBitsOut = (uint32(ll.state) + llB.deltaNbBits) >> 16
dstState = int32(ll.state>>(nbBitsOut&15)) + int32(llB.deltaFindState)
wr.addBits16NC(ll.state, uint8(nbBitsOut))
ll.state = ll.stateTable[dstState]
outBits = llB.outBits & 31
extraBits = extraBits<<outBits | uint64(s.litLen&bitMask32[outBits])
extraBitsN += outBits
wr.flush32()
wr.addBits64NC(extraBits, extraBitsN)
if debugSequences {
println("Encoded seq", seq, s)
n := (len(buf) - w.pos - 16) / seqMaxBytes
if n <= 0 {
buf = slices.Grow(buf[:w.pos], (seq+1)*seqMaxBytes+16)
buf = buf[:cap(buf)]
continue
}
for stop := max(seq-n, -1); seq > stop; seq-- {
s := seqs[seq]
ofB, mlB, llB := ofTT[s.ofCode], mlTT[s.mlCode], llTT[s.llCode]
seq--
// w.nb <= 37 here, and the states add at most 26 bits.
w, ofState = w.addState(ofState, ofTab, ofB)
w, mlState = w.addState(mlState, mlTab, mlB)
w, llState = w.addState(llState, llTab, llB)
ofBits, mlBits, llBits := ofB.outBits&31, mlB.outBits&31, llB.outBits&31
extra := uint64(s.offset&bitMask32[ofBits])<<(mlBits+llBits) |
uint64(s.matchLen&bitMask32[mlBits])<<llBits |
uint64(s.litLen&bitMask32[llBits])
extraN := uint(ofBits + mlBits + llBits)
if w.nb+extraN > 63 {
w = w.flush(buf)
if extraN > 56 {
// Only offsets over 24 bits get here.
low := uint(mlBits + llBits)
w = w.add(extra&(1<<low-1), low).flush(buf)
extra >>= low
extraN -= low
}
}
w = w.add(extra, extraN).flush(buf)
if debugSequences {
println("Encoded seq", seq, s)
}
}
}
wr.bitContainer, wr.nBits, wr.out = w.c, uint8(w.nb), buf[:w.pos]
of.state, ml.state, ll.state = ofState, mlState, llState
ml.flush(mlEnc.actualTableLog)
of.flush(ofEnc.actualTableLog)
ll.flush(llEnc.actualTableLog)
@@ -826,6 +797,54 @@ func (b *blockEnc) encode(org []byte, raw, rawAllLits bool) error {
return nil
}
// chooseComp chooses the cheapest of the new, predefined and previous table.
func (b *blockEnc) chooseComp(cur, prev, preDef *fseEncoder) (*fseEncoder, seqCompMode) {
hist := cur.count[:cur.symbolLen]
nSize := cur.approxSize(hist) + cur.maxHeaderSize()
predefSize := preDef.approxSize(hist)
prevSize := prev.approxSize(hist)
// Add a small penalty for new encoders.
// Don't bother with extremely small (<2 byte gains).
nSize = nSize + (nSize+2*8*16)>>4
// The previous table may come from a dictionary and fit poorly,
// so it is compared against the exact header size, without penalty.
nSizePrev := nSize
if prevSize != math.MaxUint32 {
// Written to spare capacity; the real header goes there later.
if hdr, err := cur.writeCount(b.output[len(b.output):]); err == nil {
nSizePrev = cur.approxSize(hist) + uint32(len(hdr))*8
}
}
// Each stream ends with a flush of tableLog bits.
nSize += uint32(cur.actualTableLog)
nSizePrev += uint32(cur.actualTableLog)
if predefSize != math.MaxUint32 {
predefSize += uint32(preDef.actualTableLog)
}
if prevSize != math.MaxUint32 {
prevSize += uint32(prev.actualTableLog)
}
switch {
case predefSize <= prevSize && predefSize <= nSize || forcePreDef:
if debugEncoder {
println("Using predefined", predefSize>>3, "<=", nSize>>3)
}
return preDef, compModePredefined
case prevSize <= nSizePrev:
if debugEncoder {
println("Using previous", prevSize>>3, "<=", nSizePrev>>3)
}
return prev, compModeRepeat
default:
if debugEncoder {
println("Using new, predef", predefSize>>3, ". previous:", prevSize>>3, ">", nSize>>3, "header max:", cur.maxHeaderSize()>>3, "bytes")
println("tl:", cur.actualTableLog, "symbolLen:", cur.symbolLen, "norm:", cur.norm[:cur.symbolLen], "hist", cur.count[:cur.symbolLen])
}
return cur, compModeFSE
}
}
var errIncompressible = errors.New("incompressible")
func (b *blockEnc) genCodes() {
vendor/github.com/klauspost/compress/zstd/decoder.go
Generated Vendored
+92 -63
View File
@@ -7,8 +7,11 @@ package zstd
import (
"context"
"encoding/binary"
"errors"
"io"
"maps"
"sync"
"sync/atomic"
"github.com/klauspost/compress/zstd/internal/xxhash"
)
@@ -20,7 +23,8 @@ import (
// A decoder can safely be re-used even if the previous stream failed.
// To release the resources, you must call the Close() function on a decoder.
type Decoder struct {
o decoderOptions
// publishedOptions contains the current immutable decoder options.
publishedOptions atomic.Pointer[decoderOptions]
// Unreferenced decoders, ready for use.
decoders chan *blockDec
@@ -84,9 +88,10 @@ var (
func NewReader(r io.Reader, opts ...DOption) (*Decoder, error) {
initPredefined()
var d Decoder
d.o.setDefault()
for _, o := range opts {
err := o(&d.o)
var o decoderOptions
o.setDefault()
for _, opt := range opts {
err := opt(&o)
if err != nil {
return nil, err
}
@@ -99,17 +104,19 @@ func NewReader(r io.Reader, opts ...DOption) (*Decoder, error) {
}
// Initialize dict map if needed.
if d.o.dicts == nil {
d.o.dicts = make(map[uint32]*dict)
if o.dicts == nil {
o.dicts = make(map[uint32]*dict)
}
// Create decoders
d.decoders = make(chan *blockDec, d.o.concurrent)
for i := 0; i < d.o.concurrent; i++ {
dec := newBlockDec(d.o.lowMem)
dec.localFrame = newFrameDec(d.o)
d.decoders = make(chan *blockDec, o.concurrent)
for i := 0; i < o.concurrent; i++ {
dec := newBlockDec(o.lowMem)
dec.localFrame = newFrameDec(o)
dec.localFrame.o.dicts = nil
d.decoders <- dec
}
d.publishedOptions.Store(&o)
if r == nil {
return &d, nil
@@ -171,6 +178,9 @@ func (d *Decoder) Reset(r io.Reader) error {
d.drainOutput()
d.syncStream.br.r = nil
if d.frame != nil {
d.frame.o.dicts = nil
}
if r == nil {
d.current.err = ErrDecoderNilInput
if len(d.current.b) > 0 {
@@ -180,8 +190,10 @@ func (d *Decoder) Reset(r io.Reader) error {
return nil
}
o := *d.publishedOptions.Load()
// If bytes buffer and < 5MB, do sync decoding anyway.
if bb, ok := r.(byter); ok && bb.Len() < d.o.decodeBufsBelow && !d.o.limitToCap {
if bb, ok := r.(byter); ok && bb.Len() < o.decodeBufsBelow && !o.limitToCap {
bb2 := bb
if debugDecoder {
println("*bytes.Buffer detected, doing sync decode, len:", bb.Len())
@@ -193,18 +205,30 @@ func (d *Decoder) Reset(r io.Reader) error {
}
dst, err := d.DecodeAll(b, dst)
if err == nil {
err = io.EOF
if !errors.Is(err, ErrDecoderSizeExceeded) {
if err == nil {
err = io.EOF
}
// Save output buffer
d.syncStream.dstBuf = dst
d.current.b = dst
d.current.err = err
d.current.flushed = true
if debugDecoder {
println("sync decode to", len(dst), "bytes, err:", err)
}
return nil
}
// Save output buffer
d.syncStream.dstBuf = dst
d.current.b = dst
d.current.err = err
d.current.flushed = true
// The output does not fit WithDecoderMaxMemory, which caps the total
// decoded size when decoding in memory. A Reader follows the
// streaming contract, where the option caps the window instead, so
// decode this input as a stream after all; the shortcut has not
// consumed the reader. Drop the oversized buffer rather than keep it
// for reuse.
if debugDecoder {
println("sync decode to", len(dst), "bytes, err:", err)
println("sync decode exceeded max memory after", len(dst), "bytes; streaming instead")
}
return nil
d.syncStream.dstBuf = nil
}
// Remove current block.
d.stashDecoder()
@@ -217,14 +241,16 @@ func (d *Decoder) Reset(r io.Reader) error {
// Ensure no-one else is still running...
d.streamWg.Wait()
if d.frame == nil {
d.frame = newFrameDec(d.o)
d.frame = newFrameDec(o)
} else {
d.frame.setOptions(o)
}
if d.o.concurrent == 1 {
if o.concurrent == 1 {
return d.startSyncDecoder(r)
}
d.current.output = make(chan decodeOutput, d.o.concurrent)
d.current.output = make(chan decodeOutput, o.concurrent)
ctx, cancel := context.WithCancel(context.Background())
d.current.cancel = cancel
d.streamWg.Add(1)
@@ -238,13 +264,21 @@ func (d *Decoder) Reset(r io.Reader) error {
// Options are applied on top of the existing options.
// Some options cannot be changed on reset and will return an error.
func (d *Decoder) ResetWithOptions(r io.Reader, opts ...DOption) error {
d.o.resetOpt = true
defer func() { d.o.resetOpt = false }()
if d.current.err == ErrDecoderClosed {
return d.current.err
}
next := *d.publishedOptions.Load()
next.dicts = maps.Clone(next.dicts)
next.resetOpt = true
for _, o := range opts {
if err := o(&d.o); err != nil {
if err := o(&next); err != nil {
next.resetOpt = false
d.publishedOptions.Store(&next)
return err
}
}
next.resetOpt = false
d.publishedOptions.Store(&next)
return d.Reset(r)
}
@@ -324,6 +358,8 @@ func (d *Decoder) DecodeAll(input, dst []byte) ([]byte, error) {
// Grab a block decoder and frame decoder.
block := <-d.decoders
frame := block.localFrame
o := *d.publishedOptions.Load()
frame.setOptions(o)
initialSize := len(dst)
defer func() {
if debugDecoder {
@@ -335,6 +371,7 @@ func (d *Decoder) DecodeAll(input, dst []byte) ([]byte, error) {
frame.history.decoders.br.in = nil
frame.history.decoders.br.cursor = 0
}
frame.o.dicts = nil
d.decoders <- block
}()
frame.bBuf = input
@@ -354,20 +391,20 @@ func (d *Decoder) DecodeAll(input, dst []byte) ([]byte, error) {
if err = d.setDict(frame); err != nil {
return nil, err
}
if frame.WindowSize > d.o.maxWindowSize {
if frame.WindowSize > frame.o.maxWindowSize {
if debugDecoder {
println("window size exceeded:", frame.WindowSize, ">", d.o.maxWindowSize)
println("window size exceeded:", frame.WindowSize, ">", frame.o.maxWindowSize)
}
return dst, ErrWindowSizeExceeded
}
if frame.FrameContentSize != fcsUnknown {
if frame.FrameContentSize > d.o.maxDecodedSize-uint64(len(dst)-initialSize) {
if frame.FrameContentSize > frame.o.maxDecodedSize-uint64(len(dst)-initialSize) {
if debugDecoder {
println("decoder size exceeded; fcs:", frame.FrameContentSize, "> mcs:", d.o.maxDecodedSize-uint64(len(dst)-initialSize), "len:", len(dst))
println("decoder size exceeded; fcs:", frame.FrameContentSize, "> mcs:", frame.o.maxDecodedSize-uint64(len(dst)-initialSize), "len:", len(dst))
}
return dst, ErrDecoderSizeExceeded
}
if d.o.limitToCap && frame.FrameContentSize > uint64(cap(dst)-len(dst)) {
if frame.o.limitToCap && frame.FrameContentSize > uint64(cap(dst)-len(dst)) {
if debugDecoder {
println("decoder size exceeded; fcs:", frame.FrameContentSize, "> (cap-len)", cap(dst)-len(dst))
}
@@ -380,14 +417,14 @@ func (d *Decoder) DecodeAll(input, dst []byte) ([]byte, error) {
}
}
if cap(dst) == 0 && !d.o.limitToCap {
if cap(dst) == 0 && !frame.o.limitToCap {
// Allocate len(input) * 2 by default if nothing is provided
// and we didn't get frame content size.
size := min(
// Cap to 1 MB.
len(input)*2, 1<<20)
if uint64(size) > d.o.maxDecodedSize {
size = int(d.o.maxDecodedSize)
if uint64(size) > frame.o.maxDecodedSize {
size = int(frame.o.maxDecodedSize)
}
dst = make([]byte, 0, size)
}
@@ -396,7 +433,7 @@ func (d *Decoder) DecodeAll(input, dst []byte) ([]byte, error) {
if err != nil {
return dst, err
}
if uint64(len(dst)-initialSize) > d.o.maxDecodedSize {
if uint64(len(dst)-initialSize) > frame.o.maxDecodedSize {
return dst, ErrDecoderSizeExceeded
}
if len(frame.bBuf) == 0 {
@@ -459,7 +496,7 @@ func (d *Decoder) nextBlock(blocking bool) (ok bool) {
println("got", len(d.current.b), "bytes, error:", d.current.err, "data crc:", tmp)
}
if d.o.ignoreChecksum {
if d.frame.o.ignoreChecksum {
return true
}
@@ -497,7 +534,7 @@ func (d *Decoder) nextBlockSync() (ok bool) {
if d.current.err != nil {
return false
}
if d.frame.WindowSize > d.o.maxDecodedSize || d.frame.WindowSize > d.o.maxWindowSize {
if d.frame.WindowSize > d.frame.o.maxDecodedSize || d.frame.WindowSize > d.frame.o.maxWindowSize {
d.current.err = ErrDecoderSizeExceeded
return false
}
@@ -509,7 +546,7 @@ func (d *Decoder) nextBlockSync() (ok bool) {
if d.current.err != nil {
return false
}
d.frame.history.ensureBlock()
d.frame.history.ensureBlockRing()
if debugDecoder {
println("History trimmed:", len(d.frame.history.b), "decoded already:", d.syncStream.decodedFrame)
}
@@ -546,11 +583,11 @@ func (d *Decoder) nextBlockSync() (ok bool) {
// Update/Check CRC
if d.frame.HasCheckSum {
if !d.o.ignoreChecksum {
if !d.frame.o.ignoreChecksum {
d.frame.crc.Write(d.current.b)
}
if d.current.d.Last {
if !d.o.ignoreChecksum {
if !d.frame.o.ignoreChecksum {
d.current.err = d.frame.checkCRC()
} else {
d.current.err = d.frame.consumeCRC()
@@ -656,8 +693,8 @@ func (d *Decoder) startStreamDecoder(ctx context.Context, r io.Reader, output ch
defer d.streamWg.Done()
br := readerWrapper{r: r}
var seqDecode = make(chan *blockDec, d.o.concurrent)
var seqExecute = make(chan *blockDec, d.o.concurrent)
var seqDecode = make(chan *blockDec, d.frame.o.concurrent)
var seqExecute = make(chan *blockDec, d.frame.o.concurrent)
// Async 1: Decode sequences...
go func() {
@@ -714,7 +751,8 @@ func (d *Decoder) startStreamDecoder(ctx context.Context, r io.Reader, output ch
wg.Add(1)
// Async 3: Execute sequences...
frameHistCache := d.frame.history.b
// Reuse the history ring across streams.
frameRing := d.frame.history.ring
go func() {
var hist history
var decodedFrame uint64
@@ -733,20 +771,12 @@ func (d *Decoder) startStreamDecoder(ctx context.Context, r io.Reader, output ch
}
hist.reset()
hist.windowSize = block.async.newHist.windowSize
hist.allocFrameBuffer = block.async.newHist.allocFrameBuffer
if block.async.newHist.dict != nil {
hist.setDict(block.async.newHist.dict)
}
if cap(hist.b) < hist.allocFrameBuffer {
if cap(frameHistCache) >= hist.allocFrameBuffer {
hist.b = frameHistCache
} else {
hist.b = make([]byte, 0, hist.allocFrameBuffer)
println("Alloc history sized", hist.allocFrameBuffer)
}
if hist.ring == nil {
hist.ring = frameRing
}
hist.b = hist.b[:0]
fcs = block.async.fcs
decodedFrame = 0
}
@@ -769,13 +799,15 @@ func (d *Decoder) startStreamDecoder(ctx context.Context, r io.Reader, output ch
for i := range block.dst {
block.dst[i] = v
}
hist.append(block.dst)
hist.ensureBlockRing()
hist.appendKeep(block.dst)
do.b = block.dst
case blockTypeRaw:
if debugDecoder {
println("add raw block length:", len(block.data))
}
hist.append(block.data)
hist.ensureBlockRing()
hist.appendKeep(block.data)
do.b = block.data
case blockTypeCompressed:
if debugDecoder {
@@ -808,7 +840,7 @@ func (d *Decoder) startStreamDecoder(ctx context.Context, r io.Reader, output ch
output <- do
}
close(output)
frameHistCache = hist.b
frameRing = hist.ring
wg.Done()
if debugDecoder {
println("decoder goroutines finished")
@@ -858,9 +890,9 @@ decodeStream:
if err == nil {
err = d.setDict(frame)
}
if err == nil && d.frame.WindowSize > d.o.maxWindowSize {
if err == nil && d.frame.WindowSize > d.frame.o.maxWindowSize {
if debugDecoder {
println("decoder size exceeded, fws:", d.frame.WindowSize, "> mws:", d.o.maxWindowSize)
println("decoder size exceeded, fws:", d.frame.WindowSize, "> mws:", d.frame.o.maxWindowSize)
}
err = ErrDecoderSizeExceeded
@@ -887,9 +919,6 @@ decodeStream:
err := frame.next(dec)
if !historySent {
h := frame.history
if debugDecoder {
println("Alloc History:", h.allocFrameBuffer)
}
hist.reset()
if h.dict != nil {
hist.setDict(h.dict)
@@ -936,11 +965,11 @@ decodeStream:
close(seqDecode)
wg.Wait()
hist.reset()
d.frame.history.b = frameHistCache
d.frame.history.ring = frameRing
}
func (d *Decoder) setDict(frame *frameDec) (err error) {
dict, ok := d.o.dicts[frame.DictionaryID]
dict, ok := frame.o.dicts[frame.DictionaryID]
if ok {
if debugDecoder {
println("setting dict", frame.DictionaryID)
vendor/github.com/klauspost/compress/zstd/dict.go
Generated Vendored
+8 -6
View File
@@ -17,6 +17,7 @@ type dict struct {
litEnc *huff0.Scratch
llDec, ofDec, mlDec sequenceDec
llEnc, ofEnc, mlEnc *fseEncoder
offsets [3]int
content []byte
}
@@ -99,17 +100,12 @@ func loadDict(b []byte) (*dict, error) {
off: 0,
}
readDec := func(i tableIndex, dec *fseDecoder) error {
if err := dec.readNCount(&br, uint16(maxTableSymbol[i])); err != nil {
if err := dec.readNCount(&br, i); err != nil {
return err
}
if br.overread() {
return io.ErrUnexpectedEOF
}
err = dec.transform(symbolTableX[i])
if err != nil {
println("Transform table error:", err)
return err
}
if debugDecoder || debugEncoder {
println("Read table ok", "symbolLen:", dec.symbolLen)
}
@@ -145,6 +141,9 @@ func loadDict(b []byte) (*dict, error) {
if d.offsets[0] > len(d.content) || d.offsets[1] > len(d.content) || d.offsets[2] > len(d.content) {
return nil, fmt.Errorf("initial offset bigger than dictionary content size %d, offsets: %v", len(d.content), d.offsets)
}
d.llEnc = newDictSeqEncoder(d.llDec.fse, tableLiteralLengths)
d.ofEnc = newDictSeqEncoder(d.ofDec.fse, tableOffsets)
d.mlEnc = newDictSeqEncoder(d.mlDec.fse, tableMatchLengths)
return &d, nil
}
@@ -219,6 +218,9 @@ func BuildDict(o BuildDictOptions) ([]byte, error) {
if len(contents) == 0 {
return nil, errors.New("no content provided")
}
if o.Level < speedNotSet || o.Level >= speedLast {
return nil, fmt.Errorf("unknown encoder level %d", o.Level)
}
d := dict{
id: o.ID,
litEnc: nil,
vendor/github.com/klauspost/compress/zstd/enc_base.go
Generated Vendored
+1
View File
@@ -194,6 +194,7 @@ func (e *fastBase) resetBase(d *dict, singleBlock bool) {
}
// Transfer litenc.
e.blk.dictLitEnc = d.litEnc
e.blk.coders.seedPrevFromDict(d)
e.hist = append(e.hist, d.content...)
}
}
vendor/github.com/klauspost/compress/zstd/enc_better.go
Generated Vendored
+48 -42
View File
@@ -26,6 +26,16 @@ const (
betterShortTableShardSize = betterShortTableSize / betterShortTableShardCnt // Size of an individual shard
)
// betterHashL is hashLen(u, betterLongTableBits, betterLongLen) with prime = prime8bytes.
func betterHashL(u, prime uint64) uint32 {
return uint32((u * prime) >> (64 - betterLongTableBits))
}
// betterHashS is hashLen(u, betterShortTableBits, betterShortLen) with prime = prime5bytes.
func betterHashS(u, prime uint64) uint32 {
return uint32(((u << (64 - 40)) * prime) >> (64 - betterShortTableBits))
}
type prevEntry struct {
offset int32
prev int32
@@ -126,6 +136,7 @@ func (e *betterFastEncoder) Encode(blk *blockEnc, src []byte) {
// Override src
src = e.hist
sLimit := int32(len(src)) - inputMargin
primeL, primeS := hashPrimes8and5()
// stepSize is the number of bytes to skip on every main loop iteration.
// It should be >= 1.
const stepSize = 1
@@ -163,8 +174,8 @@ encodeLoop:
panic("offset0 was 0")
}
nextHashL := hashLen(cv, betterLongTableBits, betterLongLen)
nextHashS := hashLen(cv, betterShortTableBits, betterShortLen)
nextHashL := betterHashL(cv, primeL)
nextHashS := betterHashS(cv, primeS)
candidateL := e.longTable[nextHashL]
candidateS := e.table[nextHashS]
@@ -218,14 +229,13 @@ encodeLoop:
break encodeLoop
}
// Index skipped...
for index0 < s-1 {
for end := s - 1; index0 < end; index0 += 2 {
cv0 := load6432(src, index0)
cv1 := cv0 >> 8
h0 := hashLen(cv0, betterLongTableBits, betterLongLen)
h0 := betterHashL(cv0, primeL)
off := index0 + e.cur
e.table[betterHashS(cv1, primeS)] = tableEntry{offset: off + 1, val: uint32(cv1)}
e.longTable[h0] = prevEntry{offset: off, prev: e.longTable[h0].offset}
e.table[hashLen(cv1, betterShortTableBits, betterShortLen)] = tableEntry{offset: off + 1, val: uint32(cv1)}
index0 += 2
}
cv = load6432(src, s)
continue
@@ -275,14 +285,13 @@ encodeLoop:
}
// Index skipped...
for index0 < s-1 {
for end := s - 1; index0 < end; index0 += 2 {
cv0 := load6432(src, index0)
cv1 := cv0 >> 8
h0 := hashLen(cv0, betterLongTableBits, betterLongLen)
h0 := betterHashL(cv0, primeL)
off := index0 + e.cur
e.table[betterHashS(cv1, primeS)] = tableEntry{offset: off + 1, val: uint32(cv1)}
e.longTable[h0] = prevEntry{offset: off, prev: e.longTable[h0].offset}
e.table[hashLen(cv1, betterShortTableBits, betterShortLen)] = tableEntry{offset: off + 1, val: uint32(cv1)}
index0 += 2
}
cv = load6432(src, s)
// Swap offsets
@@ -356,7 +365,7 @@ encodeLoop:
// See if we can find a long match at s+1
const checkAt = 1
cv := load6432(src, s+checkAt)
nextHashL = hashLen(cv, betterLongTableBits, betterLongLen)
nextHashL = betterHashL(cv, primeL)
candidateL = e.longTable[nextHashL]
coffsetL = candidateL.offset - e.cur
@@ -423,7 +432,7 @@ encodeLoop:
// and still picked up as part of the match if they do.
const skipBeginning = 3
nextHashL := hashLen(load6432(src, s+matched), betterLongTableBits, betterLongLen)
nextHashL := betterHashL(load6432(src, s+matched), primeL)
s2 := s + skipBeginning
cv := load3232(src, s2)
candidateL := e.longTable[nextHashL]
@@ -501,13 +510,12 @@ encodeLoop:
// Index match start+1 (long) -> s - 1
off := index0 + e.cur
for index0 < s-1 {
for end := s - 1; index0 < end; index0 += 2 {
cv0 := load6432(src, index0)
cv1 := cv0 >> 8
h0 := hashLen(cv0, betterLongTableBits, betterLongLen)
e.table[betterHashS(cv1, primeS)] = tableEntry{offset: off + 1, val: uint32(cv1)}
h0 := betterHashL(cv0, primeL)
e.longTable[h0] = prevEntry{offset: off, prev: e.longTable[h0].offset}
e.table[hashLen(cv1, betterShortTableBits, betterShortLen)] = tableEntry{offset: off + 1, val: uint32(cv1)}
index0 += 2
off += 2
}
@@ -525,8 +533,8 @@ encodeLoop:
}
// Store this, since we have it.
nextHashL := hashLen(cv, betterLongTableBits, betterLongLen)
nextHashS := hashLen(cv, betterShortTableBits, betterShortLen)
nextHashL := betterHashL(cv, primeL)
nextHashS := betterHashS(cv, primeS)
// We have at least 4 byte match.
// No need to check backwards. We come straight from a match
@@ -644,6 +652,7 @@ func (e *betterFastEncoderDict) Encode(blk *blockEnc, src []byte) {
// Override src
src = e.hist
sLimit := int32(len(src)) - inputMargin
primeL, primeS := hashPrimes8and5()
// stepSize is the number of bytes to skip on every main loop iteration.
// It should be >= 1.
const stepSize = 1
@@ -681,8 +690,8 @@ encodeLoop:
panic("offset0 was 0")
}
nextHashL := hashLen(cv, betterLongTableBits, betterLongLen)
nextHashS := hashLen(cv, betterShortTableBits, betterShortLen)
nextHashL := betterHashL(cv, primeL)
nextHashS := betterHashS(cv, primeS)
candidateL := e.longTable[nextHashL]
candidateS := e.table[nextHashS]
@@ -737,17 +746,16 @@ encodeLoop:
break encodeLoop
}
// Index skipped...
for index0 < s-1 {
for end := s - 1; index0 < end; index0 += 2 {
cv0 := load6432(src, index0)
cv1 := cv0 >> 8
h0 := hashLen(cv0, betterLongTableBits, betterLongLen)
h0 := betterHashL(cv0, primeL)
off := index0 + e.cur
e.longTable[h0] = prevEntry{offset: off, prev: e.longTable[h0].offset}
e.markLongShardDirty(h0)
h1 := hashLen(cv1, betterShortTableBits, betterShortLen)
h1 := betterHashS(cv1, primeS)
e.table[h1] = tableEntry{offset: off + 1, val: uint32(cv1)}
e.markShortShardDirty(h1)
index0 += 2
e.longTable[h0] = prevEntry{offset: off, prev: e.longTable[h0].offset}
e.markLongShardDirty(h0)
}
cv = load6432(src, s)
continue
@@ -797,17 +805,16 @@ encodeLoop:
}
// Index skipped...
for index0 < s-1 {
for end := s - 1; index0 < end; index0 += 2 {
cv0 := load6432(src, index0)
cv1 := cv0 >> 8
h0 := hashLen(cv0, betterLongTableBits, betterLongLen)
h0 := betterHashL(cv0, primeL)
off := index0 + e.cur
e.longTable[h0] = prevEntry{offset: off, prev: e.longTable[h0].offset}
e.markLongShardDirty(h0)
h1 := hashLen(cv1, betterShortTableBits, betterShortLen)
h1 := betterHashS(cv1, primeS)
e.table[h1] = tableEntry{offset: off + 1, val: uint32(cv1)}
e.markShortShardDirty(h1)
index0 += 2
e.longTable[h0] = prevEntry{offset: off, prev: e.longTable[h0].offset}
e.markLongShardDirty(h0)
}
cv = load6432(src, s)
// Swap offsets
@@ -881,7 +888,7 @@ encodeLoop:
// See if we can find a long match at s+1
const checkAt = 1
cv := load6432(src, s+checkAt)
nextHashL = hashLen(cv, betterLongTableBits, betterLongLen)
nextHashL = betterHashL(cv, primeL)
candidateL = e.longTable[nextHashL]
coffsetL = candidateL.offset - e.cur
@@ -942,7 +949,7 @@ encodeLoop:
}
// Try to find a better match by searching for a long match at the end of the current best match
if s+matched < sLimit {
nextHashL := hashLen(load6432(src, s+matched), betterLongTableBits, betterLongLen)
nextHashL := betterHashL(load6432(src, s+matched), primeL)
cv := load3232(src, s)
candidateL := e.longTable[nextHashL]
coffsetL := candidateL.offset - e.cur - matched
@@ -1017,16 +1024,15 @@ encodeLoop:
// Index match start+1 (long) -> s - 1
off := index0 + e.cur
for index0 < s-1 {
for end := s - 1; index0 < end; index0 += 2 {
cv0 := load6432(src, index0)
cv1 := cv0 >> 8
h0 := hashLen(cv0, betterLongTableBits, betterLongLen)
e.longTable[h0] = prevEntry{offset: off, prev: e.longTable[h0].offset}
e.markLongShardDirty(h0)
h1 := hashLen(cv1, betterShortTableBits, betterShortLen)
h1 := betterHashS(cv1, primeS)
e.table[h1] = tableEntry{offset: off + 1, val: uint32(cv1)}
e.markShortShardDirty(h1)
index0 += 2
h0 := betterHashL(cv0, primeL)
e.longTable[h0] = prevEntry{offset: off, prev: e.longTable[h0].offset}
e.markLongShardDirty(h0)
off += 2
}
@@ -1044,8 +1050,8 @@ encodeLoop:
}
// Store this, since we have it.
nextHashL := hashLen(cv, betterLongTableBits, betterLongLen)
nextHashS := hashLen(cv, betterShortTableBits, betterShortLen)
nextHashL := betterHashL(cv, primeL)
nextHashS := betterHashS(cv, primeS)
// We have at least 4 byte match.
// No need to check backwards. We come straight from a match
vendor/github.com/klauspost/compress/zstd/enc_dfast.go
Generated Vendored
+40 -27
View File
@@ -22,6 +22,16 @@ const (
)
// dFastHashL is hashLen(u, dFastLongTableBits, dFastLongLen) with prime = prime8bytes.
func dFastHashL(u, prime uint64) uint32 {
return uint32((u * prime) >> (64 - dFastLongTableBits))
}
// dFastHashS is hashLen(u, dFastShortTableBits, dFastShortLen) with prime = prime5bytes.
func dFastHashS(u, prime uint64) uint32 {
return uint32(((u << (64 - 40)) * prime) >> (64 - dFastShortTableBits))
}
type doubleFastEncoder struct {
fastEncoder
longTable [dFastLongTableSize]tableEntry
@@ -87,6 +97,7 @@ func (e *doubleFastEncoder) Encode(blk *blockEnc, src []byte) {
// Override src
src = e.hist
sLimit := int32(len(src)) - inputMargin
primeL, primeS := hashPrimes8and5()
// stepSize is the number of bytes to skip on every main loop iteration.
// It should be >= 1.
const stepSize = 1
@@ -123,8 +134,8 @@ encodeLoop:
panic("offset0 was 0")
}
nextHashL := hashLen(cv, dFastLongTableBits, dFastLongLen)
nextHashS := hashLen(cv, dFastShortTableBits, dFastShortLen)
nextHashL := dFastHashL(cv, primeL)
nextHashS := dFastHashS(cv, primeS)
candidateL := e.longTable[nextHashL]
candidateS := e.table[nextHashS]
@@ -204,7 +215,7 @@ encodeLoop:
// See if we can find a long match at s+1
const checkAt = 1
cv := load6432(src, s+checkAt)
nextHashL = hashLen(cv, dFastLongTableBits, dFastLongLen)
nextHashL = dFastHashL(cv, primeL)
candidateL = e.longTable[nextHashL]
coffsetL = s - (candidateL.offset - e.cur) + checkAt
@@ -297,16 +308,16 @@ encodeLoop:
cv1 := load6432(src, index1)
te0 := tableEntry{offset: index0 + e.cur, val: uint32(cv0)}
te1 := tableEntry{offset: index1 + e.cur, val: uint32(cv1)}
e.longTable[hashLen(cv0, dFastLongTableBits, dFastLongLen)] = te0
e.longTable[hashLen(cv1, dFastLongTableBits, dFastLongLen)] = te1
e.longTable[dFastHashL(cv0, primeL)] = te0
e.longTable[dFastHashL(cv1, primeL)] = te1
cv0 >>= 8
cv1 >>= 8
te0.offset++
te1.offset++
te0.val = uint32(cv0)
te1.val = uint32(cv1)
e.table[hashLen(cv0, dFastShortTableBits, dFastShortLen)] = te0
e.table[hashLen(cv1, dFastShortTableBits, dFastShortLen)] = te1
e.table[dFastHashS(cv0, primeS)] = te0
e.table[dFastHashS(cv1, primeS)] = te1
cv = load6432(src, s)
@@ -323,8 +334,8 @@ encodeLoop:
}
// Store this, since we have it.
nextHashS := hashLen(cv, dFastShortTableBits, dFastShortLen)
nextHashL := hashLen(cv, dFastLongTableBits, dFastLongLen)
nextHashS := dFastHashS(cv, primeS)
nextHashL := dFastHashL(cv, primeL)
// We have at least 4 byte match.
// No need to check backwards. We come straight from a match
@@ -399,6 +410,7 @@ func (e *doubleFastEncoder) EncodeNoHist(blk *blockEnc, src []byte) {
// Override src
sLimit := int32(len(src)) - inputMargin
primeL, primeS := hashPrimes8and5()
// stepSize is the number of bytes to skip on every main loop iteration.
// It should be >= 1.
const stepSize = 1
@@ -429,8 +441,8 @@ encodeLoop:
var t int32
for {
nextHashL := hashLen(cv, dFastLongTableBits, dFastLongLen)
nextHashS := hashLen(cv, dFastShortTableBits, dFastShortLen)
nextHashL := dFastHashL(cv, primeL)
nextHashS := dFastHashS(cv, primeS)
candidateL := e.longTable[nextHashL]
candidateS := e.table[nextHashS]
@@ -511,7 +523,7 @@ encodeLoop:
// See if we can find a long match at s+1
const checkAt = 1
cv := load6432(src, s+checkAt)
nextHashL = hashLen(cv, dFastLongTableBits, dFastLongLen)
nextHashL = dFastHashL(cv, primeL)
candidateL = e.longTable[nextHashL]
coffsetL = s - (candidateL.offset - e.cur) + checkAt
@@ -601,16 +613,16 @@ encodeLoop:
cv1 := load6432(src, index1)
te0 := tableEntry{offset: index0 + e.cur, val: uint32(cv0)}
te1 := tableEntry{offset: index1 + e.cur, val: uint32(cv1)}
e.longTable[hashLen(cv0, dFastLongTableBits, dFastLongLen)] = te0
e.longTable[hashLen(cv1, dFastLongTableBits, dFastLongLen)] = te1
e.longTable[dFastHashL(cv0, primeL)] = te0
e.longTable[dFastHashL(cv1, primeL)] = te1
cv0 >>= 8
cv1 >>= 8
te0.offset++
te1.offset++
te0.val = uint32(cv0)
te1.val = uint32(cv1)
e.table[hashLen(cv0, dFastShortTableBits, dFastShortLen)] = te0
e.table[hashLen(cv1, dFastShortTableBits, dFastShortLen)] = te1
e.table[dFastHashS(cv0, primeS)] = te0
e.table[dFastHashS(cv1, primeS)] = te1
cv = load6432(src, s)
@@ -627,8 +639,8 @@ encodeLoop:
}
// Store this, since we have it.
nextHashS := hashLen(cv1>>8, dFastShortTableBits, dFastShortLen)
nextHashL := hashLen(cv, dFastLongTableBits, dFastLongLen)
nextHashS := dFastHashS(cv1>>8, primeS)
nextHashL := dFastHashL(cv, primeL)
// We have at least 4 byte match.
// No need to check backwards. We come straight from a match
@@ -733,6 +745,7 @@ func (e *doubleFastEncoderDict) Encode(blk *blockEnc, src []byte) {
// Override src
src = e.hist
sLimit := int32(len(src)) - inputMargin
primeL, primeS := hashPrimes8and5()
// stepSize is the number of bytes to skip on every main loop iteration.
// It should be >= 1.
const stepSize = 1
@@ -769,8 +782,8 @@ encodeLoop:
panic("offset0 was 0")
}
nextHashL := hashLen(cv, dFastLongTableBits, dFastLongLen)
nextHashS := hashLen(cv, dFastShortTableBits, dFastShortLen)
nextHashL := dFastHashL(cv, primeL)
nextHashS := dFastHashS(cv, primeS)
candidateL := e.longTable[nextHashL]
candidateS := e.table[nextHashS]
@@ -852,7 +865,7 @@ encodeLoop:
// See if we can find a long match at s+1
const checkAt = 1
cv := load6432(src, s+checkAt)
nextHashL = hashLen(cv, dFastLongTableBits, dFastLongLen)
nextHashL = dFastHashL(cv, primeL)
candidateL = e.longTable[nextHashL]
coffsetL = s - (candidateL.offset - e.cur) + checkAt
@@ -946,8 +959,8 @@ encodeLoop:
cv1 := load6432(src, index1)
te0 := tableEntry{offset: index0 + e.cur, val: uint32(cv0)}
te1 := tableEntry{offset: index1 + e.cur, val: uint32(cv1)}
longHash1 := hashLen(cv0, dFastLongTableBits, dFastLongLen)
longHash2 := hashLen(cv1, dFastLongTableBits, dFastLongLen)
longHash1 := dFastHashL(cv0, primeL)
longHash2 := dFastHashL(cv1, primeL)
e.longTable[longHash1] = te0
e.longTable[longHash2] = te1
e.markLongShardDirty(longHash1)
@@ -958,8 +971,8 @@ encodeLoop:
te1.offset++
te0.val = uint32(cv0)
te1.val = uint32(cv1)
hashVal1 := hashLen(cv0, dFastShortTableBits, dFastShortLen)
hashVal2 := hashLen(cv1, dFastShortTableBits, dFastShortLen)
hashVal1 := dFastHashS(cv0, primeS)
hashVal2 := dFastHashS(cv1, primeS)
e.table[hashVal1] = te0
e.markShardDirty(hashVal1)
e.table[hashVal2] = te1
@@ -980,8 +993,8 @@ encodeLoop:
}
// Store this, since we have it.
nextHashL := hashLen(cv, dFastLongTableBits, dFastLongLen)
nextHashS := hashLen(cv, dFastShortTableBits, dFastShortLen)
nextHashL := dFastHashL(cv, primeL)
nextHashS := dFastHashS(cv, primeS)
// We have at least 4 byte match.
// No need to check backwards. We come straight from a match
vendor/github.com/klauspost/compress/zstd/framedec.go
Generated Vendored
+15 -23
View File
@@ -49,13 +49,16 @@ const (
)
func newFrameDec(o decoderOptions) *frameDec {
d := frameDec{}
d.setOptions(o)
return &d
}
func (d *frameDec) setOptions(o decoderOptions) {
if o.maxWindowSize > o.maxDecodedSize {
o.maxWindowSize = o.maxDecodedSize
}
d := frameDec{
o: o,
}
return &d
d.o = o
}
// reset will read the frame header and prepare for block decoding.
@@ -255,18 +258,6 @@ func (d *frameDec) reset(br byteBuffer) error {
return ErrWindowSizeTooSmall
}
d.history.windowSize = int(d.WindowSize)
if !d.o.lowMem || d.history.windowSize < maxBlockSize {
// Alloc 2x window size if not low-mem, or window size below 2MB.
d.history.allocFrameBuffer = d.history.windowSize * 2
} else {
if d.o.lowMem {
// Alloc with 1MB extra.
d.history.allocFrameBuffer = d.history.windowSize + maxBlockSize/2
} else {
// Alloc with 2MB extra.
d.history.allocFrameBuffer = d.history.windowSize + maxBlockSize
}
}
if debugDecoder {
println("Frame: Dict:", d.DictionaryID, "FrameContentSize:", d.FrameContentSize, "singleseg:", d.SingleSegment, "window:", d.WindowSize, "crc:", d.HasCheckSum)
@@ -352,14 +343,12 @@ func (d *frameDec) runDecoder(dst []byte, dec *blockDec) ([]byte, error) {
if debugDecoder {
println("maxSyncLen:", d.history.decoders.maxSyncLen)
}
if !d.o.limitToCap && uint64(cap(dst)) < d.history.decoders.maxSyncLen {
// Alloc for output
dst2 := make([]byte, len(dst), d.history.decoders.maxSyncLen+compressedBlockOverAlloc)
copy(dst2, dst)
dst = dst2
}
}
var err error
// Hash each block while it is still in cache rather than the whole
// frame at the end.
hashCRC := d.HasCheckSum && !d.o.ignoreChecksum
crcDone := crcStart
for {
err = dec.reset(d.rawInput, d.WindowSize)
if err != nil {
@@ -372,6 +361,10 @@ func (d *frameDec) runDecoder(dst []byte, dec *blockDec) ([]byte, error) {
if err != nil {
break
}
if hashCRC {
d.crc.Write(d.history.b[crcDone:])
crcDone = len(d.history.b)
}
if uint64(len(d.history.b)-crcStart) > d.o.maxDecodedSize {
println("runDecoder: maxDecodedSize exceeded", uint64(len(d.history.b)-crcStart), ">", d.o.maxDecodedSize)
err = ErrDecoderSizeExceeded
@@ -402,7 +395,6 @@ func (d *frameDec) runDecoder(dst []byte, dec *blockDec) ([]byte, error) {
if d.o.ignoreChecksum {
err = d.consumeCRC()
} else {
d.crc.Write(dst[crcStart:])
err = d.checkCRC()
}
}
vendor/github.com/klauspost/compress/zstd/fse_decoder.go
Generated Vendored
+51 -25
View File
@@ -43,13 +43,36 @@ type fseDecoder struct {
preDefined bool
}
// codeShare returns, in 1/256ths, the table's probability mass on symbols
// minCode and above: for a table read from the block, the share of its
// sequences with those codes. The predefined tables describe the format's
// defaults, not the block, and RLE tables have no distribution; both
// report -1.
func (s *fseDecoder) codeShare(minCode int) int {
if s == nil || s.preDefined || s.actualTableLog == 0 {
return -1
}
n := 0
for c := minCode; c < int(s.symbolLen); c++ {
switch v := int(s.norm[c]); {
case v > 0:
n += v
case v < 0:
n++ // -1 marks a probability below one slot; it occupies one
}
}
return n * 256 >> s.actualTableLog
}
// tableStep returns the next table index.
func tableStep(tableSize uint32) uint32 {
return (tableSize >> 1) + (tableSize >> 3) + 3
}
// readNCount will read the symbol distribution so decoding tables can be constructed.
func (s *fseDecoder) readNCount(b *byteReader, maxSymbol uint16) error {
// readNCount will read the symbol distribution for table t and build its
// decoding table.
func (s *fseDecoder) readNCount(b *byteReader, t tableIndex) error {
maxSymbol := uint16(maxTableSymbol[t])
var (
charnum uint16
previous0 bool
@@ -167,7 +190,8 @@ func (s *fseDecoder) readNCount(b *byteReader, maxSymbol uint16) error {
if s.symbolLen <= 1 {
return fmt.Errorf("symbolLen (%d) too small", s.symbolLen)
}
if s.symbolLen > maxSymbolValue+1 {
// A run of zero counts can step past maxSymbol, so check the total.
if s.symbolLen > maxSymbol+1 {
return fmt.Errorf("symbolLen (%d) too big", s.symbolLen)
}
if remaining != 1 {
@@ -180,7 +204,30 @@ func (s *fseDecoder) readNCount(b *byteReader, maxSymbol uint16) error {
return fmt.Errorf("corruption detected (total %d != %d)", gotTotal, 1<<s.actualTableLog)
}
b.advance((bitCount + 7) >> 3)
return s.buildDtable()
return s.buildDtable(&symbolExtX[t])
}
// symbolExt holds, per symbol, a decSymbol with only addBits and baseline set.
type symbolExt [maxMatchLengthSymbol + 1]decSymbol
// buildDtable builds the decoding table, taking each state's addBits and
// baseline from its symbol's entry in ext.
func (s *fseDecoder) buildDtable(ext *symbolExt) error {
if int(s.symbolLen) > len(ext) {
return fmt.Errorf("symbolLen (%d) too big", s.symbolLen)
}
if err := s.fillDtable(ext); err != nil {
return err
}
// addBits never decreases with the symbol (checked in initPredefined),
// so the last symbol has the most. It may exceed the table's actual
// maximum only if the last symbol has no states, which just costs an
// extra bitreader fill.
s.maxBits = 0
if s.symbolLen > 0 {
s.maxBits = ext[s.symbolLen-1].addBits()
}
return nil
}
func (s *fseDecoder) mustReadFrom(r io.Reader) {
@@ -268,27 +315,6 @@ func (s *fseDecoder) setRLE(symbol decSymbol) {
s.dt[0] = symbol
}
// transform will transform the decoder table into a table usable for
// decoding without having to apply the transformation while decoding.
// The state will contain the base value and the number of bits to read.
func (s *fseDecoder) transform(t []baseOffset) error {
tableSize := uint16(1 << s.actualTableLog)
s.maxBits = 0
for i, v := range s.dt[:tableSize] {
add := v.addBits()
if int(add) >= len(t) {
return fmt.Errorf("invalid decoding table entry %d, symbol %d >= max (%d)", i, v.addBits(), len(t))
}
lu := t[add]
if lu.addBits > s.maxBits {
s.maxBits = lu.addBits
}
v.setExt(lu.addBits, lu.baseLine)
s.dt[i] = v
}
return nil
}
type fseState struct {
dt []decSymbol
state decSymbol
vendor/github.com/klauspost/compress/zstd/fse_decoder_amd64.s
Generated Vendored
+58 -52
View File
@@ -1,126 +1,132 @@
// Code generated by command: go run gen_fse.go -out ../fse_decoder.s -arch amd64,arm64 -pkg=zstd. DO NOT EDIT.
//go:build !appengine && !noasm && gc && !noasm
//go:build !noasm && gc
// func buildDtable_asm(s *fseDecoder, ctx *buildDtableAsmContext) int
TEXT ·buildDtable_asm(SB), $0-24
MOVQ ctx+8(FP), CX
MOVQ s+0(FP), DI
MOVQ s+0(FP), R8
// Load values
MOVBQZX 4098(DI), DX
MOVBQZX 4098(R8), DX
XORQ AX, AX
BTSQ DX, AX
MOVQ (CX), BX
MOVQ 16(CX), SI
LEAQ -1(AX), R8
MOVQ 24(CX), DI
LEAQ -1(AX), R9
MOVQ 8(CX), CX
MOVWQZX 4096(DI), DI
MOVWQZX 4096(R8), R8
// End load values
// Init, lay down lowprob symbols
XORQ R9, R9
XORQ R10, R10
JMP init_main_loop_condition
init_main_loop:
MOVWQSX (CX)(R9*2), R10
CMPW R10, $-1
MOVWQSX (CX)(R10*2), R11
CMPW R11, $-1
JNE do_not_update_high_threshold
MOVB R9, 1(SI)(R8*8)
DECQ R8
MOVQ $0x0000000000000001, R10
MOVB R10, 1(SI)(R9*8)
DECQ R9
MOVQ $0x0000000000000001, R11
do_not_update_high_threshold:
MOVW R10, (BX)(R9*2)
INCQ R9
MOVW R11, (BX)(R10*2)
INCQ R10
init_main_loop_condition:
CMPQ R9, DI
CMPQ R10, R8
JL init_main_loop
// Spread symbols
// Calculate table step
MOVQ AX, R9
SHRQ $0x01, R9
MOVQ AX, R10
SHRQ $0x03, R10
LEAQ 3(R9)(R10*1), R9
SHRQ $0x01, R10
MOVQ AX, R11
SHRQ $0x03, R11
LEAQ 3(R10)(R11*1), R10
// Fill add bits values
LEAQ -1(AX), R10
XORQ R11, R11
LEAQ -1(AX), R11
XORQ R12, R12
XORQ R13, R13
JMP spread_main_loop_condition
spread_main_loop:
XORQ R13, R13
MOVWQSX (CX)(R12*2), R14
XORQ R14, R14
MOVWQSX (CX)(R13*2), R15
JMP spread_inner_loop_condition
spread_inner_loop:
MOVB R12, 1(SI)(R11*8)
MOVB R13, 1(SI)(R12*8)
adjust_position:
ADDQ R9, R11
ANDQ R10, R11
CMPQ R11, R8
ADDQ R10, R12
ANDQ R11, R12
CMPQ R12, R9
JG adjust_position
INCQ R13
INCQ R14
spread_inner_loop_condition:
CMPQ R13, R14
CMPQ R14, R15
JL spread_inner_loop
INCQ R12
INCQ R13
spread_main_loop_condition:
CMPQ R12, DI
CMPQ R13, R8
JL spread_main_loop
TESTQ R11, R11
TESTQ R12, R12
JZ spread_check_ok
MOVQ ctx+8(FP), AX
MOVQ R11, 24(AX)
MOVQ R12, 32(AX)
MOVQ $+1, ret+16(FP)
RET
spread_check_ok:
// Build Decoding table
XORQ DI, DI
XORQ R8, R8
build_table_main_table:
MOVBQZX 1(SI)(DI*8), CX
MOVWQZX (BX)(CX*2), R8
LEAQ 1(R8), R9
MOVW R9, (BX)(CX*2)
MOVQ R8, R9
BSRQ R9, R9
MOVBQZX 1(SI)(R8*8), R9
MOVWQZX (BX)(R9*2), R10
LEAQ 1(R10), CX
MOVW CX, (BX)(R9*2)
MOVQ R10, R11
BSRQ R11, R11
MOVQ DX, CX
SUBQ R9, CX
SHLQ CL, R8
SUBQ AX, R8
MOVB CL, (SI)(DI*8)
MOVW R8, 2(SI)(DI*8)
CMPQ R8, AX
SUBQ R11, CX
SHLQ CL, R10
SUBQ AX, R10
MOVQ (DI)(R9*8), R9
ORQ CX, R9
MOVQ R10, R11
ANDQ $0x0000ffff, R11
SHLQ $0x10, R11
ORQ R11, R9
MOVQ R9, (SI)(R8*8)
CMPQ R10, AX
JLE build_table_check1_ok
MOVQ ctx+8(FP), CX
MOVQ R8, 24(CX)
MOVQ AX, 32(CX)
MOVQ R10, 32(CX)
MOVQ AX, 40(CX)
MOVQ $+2, ret+16(FP)
RET
build_table_check1_ok:
TESTB CL, CL
JNZ build_table_check2_ok
CMPW R8, DI
CMPW R10, R8
JNE build_table_check2_ok
MOVQ ctx+8(FP), AX
MOVQ R8, 24(AX)
MOVQ DI, 32(AX)
MOVQ R10, 32(AX)
MOVQ R8, 40(AX)
MOVQ $+3, ret+16(FP)
RET
build_table_check2_ok:
INCQ DI
CMPQ DI, AX
INCQ R8
CMPQ R8, AX
JL build_table_main_table
MOVQ $+0, ret+16(FP)
RET
vendor/github.com/klauspost/compress/zstd/fse_decoder_arm64.s
Generated Vendored
+64 -62
View File
@@ -1,124 +1,126 @@
// Code generated by command: go run gen_fse.go -out ../fse_decoder.s -arch amd64,arm64 -pkg=zstd. DO NOT EDIT.
// EXPERIMENTAL arm64 output lowered from an amd64 avo program.
//go:build arm64 && (!appengine && !noasm && gc && !noasm)
//go:build arm64 && (!noasm && gc)
// func buildDtable_asm(s *fseDecoder, ctx *buildDtableAsmContext) int
TEXT ·buildDtable_asm(SB), $0-24
MOVD ctx+8(FP), R1
MOVD s+0(FP), R6
MOVD s+0(FP), R7
// Load values
MOVBU 4098(R6), R2
MOVBU 4098(R7), R2
MOVD $0, R0
MOVD $1, R16
LSL R2, R16, R16
ORR R16, R0, R0
MOVD (R1), R3
MOVD 16(R1), R5
SUB $1, R0, R7
MOVD 24(R1), R6
SUB $1, R0, R8
MOVD 8(R1), R1
MOVHU 4096(R6), R6
MOVHU 4096(R7), R7
// End load values
// Init, lay down lowprob symbols
MOVD $0, R8
MOVD $0, R9
JMP init_main_loop_condition
init_main_loop:
MOVH (R1)(R8<<1), R9
MOVH (R1)(R9<<1), R10
MOVD $-1, R16
AND $0xffff, R16, R16
AND $0xffff, R9, R15
AND $0xffff, R10, R15
CMP R16, R15
BNE do_not_update_high_threshold
ADD R7<<3, R5, R15
MOVB R8, 1(R15)
SUB $1, R7, R7
MOVD $0x0000000000000001, R9
ADD R8<<3, R5, R15
MOVB R9, 1(R15)
SUB $1, R8, R8
MOVD $0x0000000000000001, R10
do_not_update_high_threshold:
MOVH R9, (R3)(R8<<1)
ADD $1, R8, R8
MOVH R10, (R3)(R9<<1)
ADD $1, R9, R9
init_main_loop_condition:
CMP R6, R8
CMP R7, R9
BLT init_main_loop
// Spread symbols
// Calculate table step
MOVD R0, R8
LSR $0x01, R8, R8
MOVD R0, R9
LSR $0x03, R9, R9
ADD R9, R8, R8
ADD $3, R8, R8
LSR $0x01, R0, R9
LSR $0x03, R0, R10
ADD R10, R9, R9
ADD $3, R9, R9
// Fill add bits values
SUB $1, R0, R9
MOVD $0, R10
SUB $1, R0, R10
MOVD $0, R11
MOVD $0, R12
JMP spread_main_loop_condition
spread_main_loop:
MOVD $0, R12
MOVH (R1)(R11<<1), R13
MOVD $0, R13
MOVH (R1)(R12<<1), R14
JMP spread_inner_loop_condition
spread_inner_loop:
ADD R10<<3, R5, R15
MOVB R11, 1(R15)
ADD R11<<3, R5, R15
MOVB R12, 1(R15)
adjust_position:
ADD R8, R10, R10
AND R9, R10, R10
CMP R7, R10
ADD R9, R11, R11
AND R10, R11, R11
CMP R8, R11
BGT adjust_position
ADD $1, R12, R12
ADD $1, R13, R13
spread_inner_loop_condition:
CMP R13, R12
CMP R14, R13
BLT spread_inner_loop
ADD $1, R11, R11
ADD $1, R12, R12
spread_main_loop_condition:
CMP R6, R11
CMP R7, R12
BLT spread_main_loop
TST R10, R10
TST R11, R11
BEQ spread_check_ok
MOVD ctx+8(FP), R0
MOVD R10, 24(R0)
MOVD R11, 32(R0)
MOVD $+1, R16
MOVD R16, ret+16(FP)
RET
spread_check_ok:
// Build Decoding table
MOVD $0, R6
MOVD $0, R7
build_table_main_table:
ADD R6<<3, R5, R15
MOVBU 1(R15), R1
MOVHU (R3)(R1<<1), R7
ADD $1, R7, R8
MOVH R8, (R3)(R1<<1)
MOVD R7, R8
CLZ R8, R16
MOVD $63, R8
SUB R16, R8, R8
ADD R7<<3, R5, R15
MOVBU 1(R15), R8
MOVHU (R3)(R8<<1), R9
ADD $1, R9, R1
MOVH R1, (R3)(R8<<1)
MOVD R9, R10
CLZ R10, R16
MOVD $63, R10
SUB R16, R10, R10
MOVD R2, R1
SUB R8, R1, R1
LSL R1, R7, R7
SUB R0, R7, R7
ADD R6<<3, R5, R15
MOVB R1, (R15)
ADD R6<<3, R5, R15
MOVH R7, 2(R15)
CMP R0, R7
SUB R10, R1, R1
LSL R1, R9, R9
SUB R0, R9, R9
MOVD (R6)(R8<<3), R8
ORR R1, R8, R8
MOVD R9, R10
AND $0x0000ffff, R10, R10
LSL $0x10, R10, R10
ORR R10, R8, R8
MOVD R8, (R5)(R7<<3)
CMP R0, R9
BLE build_table_check1_ok
MOVD ctx+8(FP), R1
MOVD R7, 24(R1)
MOVD R0, 32(R1)
MOVD R9, 32(R1)
MOVD R0, 40(R1)
MOVD $+2, R16
MOVD R16, ret+16(FP)
RET
@@ -126,20 +128,20 @@ build_table_main_table:
build_table_check1_ok:
TST $0xff, R1
BNE build_table_check2_ok
AND $0xffff, R6, R16
AND $0xffff, R7, R15
AND $0xffff, R7, R16
AND $0xffff, R9, R15
CMP R16, R15
BNE build_table_check2_ok
MOVD ctx+8(FP), R0
MOVD R7, 24(R0)
MOVD R6, 32(R0)
MOVD R9, 32(R0)
MOVD R7, 40(R0)
MOVD $+3, R16
MOVD R16, ret+16(FP)
RET
build_table_check2_ok:
ADD $1, R6, R6
CMP R0, R6
ADD $1, R7, R7
CMP R0, R7
BLT build_table_main_table
MOVD $+0, R16
MOVD R16, ret+16(FP)
vendor/github.com/klauspost/compress/zstd/fse_decoder_asm.go
Generated Vendored
+5 -3
View File
@@ -1,4 +1,4 @@
//go:build (amd64 || arm64) && !appengine && !noasm && gc
//go:build (amd64 || arm64) && !noasm && gc
package zstd
@@ -15,6 +15,7 @@ type buildDtableAsmContext struct {
stateTable *uint16
norm *int16
dt *uint64
ext *uint64
// outputs --- set by the procedure in the case of error;
// for interpretation please see the error handling part below
@@ -35,12 +36,13 @@ const (
errorNewStateNoBits = 3
)
// buildDtable will build the decoding table.
func (s *fseDecoder) buildDtable() error {
// fillDtable fills the decoding table; see buildDtable.
func (s *fseDecoder) fillDtable(ext *symbolExt) error {
ctx := buildDtableAsmContext{
stateTable: &s.stateTable[0],
norm: &s.norm[0],
dt: (*uint64)(&s.dt[0]),
ext: (*uint64)(&ext[0]),
}
code := buildDtable_asm(s, &ctx)
vendor/github.com/klauspost/compress/zstd/fse_decoder_generic.go
Generated Vendored
+4 -5
View File
@@ -1,4 +1,4 @@
//go:build (!amd64 && !arm64) || appengine || !gc || noasm
//go:build (!amd64 && !arm64) || !gc || noasm
package zstd
@@ -7,8 +7,8 @@ import (
"fmt"
)
// buildDtable will build the decoding table.
func (s *fseDecoder) buildDtable() error {
// fillDtable fills the decoding table; see buildDtable.
func (s *fseDecoder) fillDtable(ext *symbolExt) error {
tableSize := uint32(1 << s.actualTableLog)
highThreshold := tableSize - 1
symbolNext := s.stateTable[:256]
@@ -56,7 +56,6 @@ func (s *fseDecoder) buildDtable() error {
nextState := symbolNext[symbol]
symbolNext[symbol] = nextState + 1
nBits := s.actualTableLog - byte(highBits(uint32(nextState)))
s.dt[u&maxTableMask].setNBits(nBits)
newState := (nextState << nBits) - tableSize
if newState > tableSize {
return fmt.Errorf("newState (%d) outside table size (%d)", newState, tableSize)
@@ -65,7 +64,7 @@ func (s *fseDecoder) buildDtable() error {
// Seems weird that this is possible with nbits > 0.
return fmt.Errorf("newState (%d) == oldState (%d) and no bits", newState, u)
}
s.dt[u&maxTableMask].setNewState(newState)
s.dt[u&maxTableMask] = ext[symbol] | decSymbol(nBits) | decSymbol(newState)<<16
}
}
return nil
vendor/github.com/klauspost/compress/zstd/fse_predefined.go
Generated Vendored
+14 -4
View File
@@ -25,6 +25,10 @@ var (
// https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md#the-codes-for-literals-lengths-match-lengths-and-offsets
symbolTableX [3][]baseOffset
// symbolExtX holds symbolTableX as decSymbols carrying only addBits and
// baseline, which buildDtable merges into each state.
symbolExtX [3]symbolExt
// maxTableSymbol is the biggest supported symbol for each table type
// https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md#the-codes-for-literals-lengths-match-lengths-and-offsets
maxTableSymbol = [3]uint8{tableLiteralLengths: maxLiteralLengthSymbol, tableOffsets: maxOffsetLengthSymbol, tableMatchLengths: maxMatchLengthSymbol}
@@ -105,6 +109,15 @@ func initPredefined() {
}
fillBase(tmp[2:], 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30)
symbolTableX[tableOffsets] = tmp
for i, t := range symbolTableX {
for j, lu := range t {
// fseDecoder.buildDtable takes maxBits from the last symbol.
if j > 0 && lu.addBits < t[j-1].addBits {
panic(fmt.Sprintf("table %v: addBits decreases at symbol %d", tableIndex(i), j))
}
symbolExtX[i][j] = newDecSymbol(0, lu.addBits, 0, lu.baseLine)
}
}
// Fill predefined tables and transform them.
// https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md#default-distributions
@@ -135,10 +148,7 @@ func initPredefined() {
-1, -1, -1, -1, -1})
f.symbolLen = 53
}
if err := f.buildDtable(); err != nil {
panic(fmt.Errorf("building table %v: %v", tableIndex(i), err))
}
if err := f.transform(symbolTableX[i]); err != nil {
if err := f.buildDtable(&symbolExtX[i]); err != nil {
panic(fmt.Errorf("building table %v: %v", tableIndex(i), err))
}
f.preDefined = true
vendor/github.com/klauspost/compress/zstd/hash.go
Generated Vendored
+22
View File
@@ -4,6 +4,8 @@
package zstd
import "runtime"
const (
prime3bytes = 506832829
prime4bytes = 2654435761
@@ -13,6 +15,12 @@ const (
prime8bytes = 0xcf1bbcdcb7a56463
)
// hashPrimesInRegs reports whether encoders should keep the 64-bit hash primes
// in registers rather than as constants. The compiler rebuilds a constant at
// every use: one instruction on amd64, but up to 4 on arm64, which also has
// the registers to spare. See hashPrimes8and5.
const hashPrimesInRegs = runtime.GOARCH == "arm64"
// hashLen returns a hash of the lowest mls bytes of with length output bits.
// mls must be >=3 and <=8. Any other value will return hash for 4 bytes.
// length should always be < 32.
@@ -33,3 +41,17 @@ func hashLen(u uint64, length, mls uint8) uint32 {
return (uint32(u) * prime4bytes) >> (32 - length)
}
}
// hashPrimes85 is a var so arm64 keeps the primes in registers instead of
// rebuilding each 64-bit prime with up to 4 instructions per use.
var hashPrimes85 = [2]uint64{prime8bytes, prime5bytes}
// hashPrimes8and5 returns prime8bytes and prime5bytes for encoders that hash
// 8 and 5 bytes. Where registers are too scarce to hold them, it returns the
// constants.
func hashPrimes8and5() (prime8, prime5 uint64) {
if hashPrimesInRegs {
return hashPrimes85[0], hashPrimes85[1]
}
return prime8bytes, prime5bytes
}
vendor/github.com/klauspost/compress/zstd/history.go
Generated Vendored
+46 -43
View File
@@ -24,16 +24,26 @@ type history struct {
// when checking for matches in history
ignoreBuffer int
windowSize int
allocFrameBuffer int // needed?
error bool
dict *dict
windowSize int
error bool
dict *dict
// ring backs b when streaming (see ensureBlockRing).
// ext holds the part of the previous lap of ring still in the window,
// which precedes b[0] in the stream; nil when b holds all history.
// The previous lap is ring[extStart:extEnd].
ring []byte
ext []byte
extStart int
extEnd int
}
// reset will reset the history to initial state of a frame.
// The history must already have been initialized to the desired size.
func (h *history) reset() {
h.b = h.b[:0]
h.ext = nil
h.extStart, h.extEnd = 0, 0
h.ignoreBuffer = 0
h.error = false
h.recentOffsets = [3]int{1, 4, 8}
@@ -67,47 +77,40 @@ func (h *history) setDict(dict *dict) {
h.huffTree = dict.litEnc
}
// append bytes to history.
// This function will make sure there is space for it,
// if the buffer has been allocated with enough extra space.
func (h *history) append(b []byte) {
if len(b) >= h.windowSize {
// Discard all history by simply overwriting
h.b = h.b[:h.windowSize]
copy(h.b, b[len(b)-h.windowSize:])
return
// ensureBlockRing makes room for the next block without moving history,
// as the reference decoder does. b is a prefix of ring capped to one block
// of space. When a block no longer fits, the data written so far becomes
// ext and writing restarts at the front of ring. ext is trimmed so the
// block space ahead of b never overlaps it; with ring sized to window plus
// two blocks of space, b and ext together always hold a full window.
func (h *history) ensureBlockRing() {
// Room for the largest block plus the overrun of the fast copies.
ringBlockSpace := min(h.windowSize, maxCompressedBlockSize) + compressedBlockOverAlloc
size := h.windowSize + 2*ringBlockSpace
pos := len(h.b)
if cap(h.ring) < size || (pos > 0 && &h.b[0] != &h.ring[0]) {
// No ring yet, or b no longer lives in it: start one, keeping up
// to a window of what was decoded so far as the previous lap.
keep := min(pos, h.windowSize)
if cap(h.ring) < size {
h.ring = make([]byte, size)
}
h.ring = h.ring[:size]
copy(h.ring[size-keep:], h.b[pos-keep:])
h.extStart, h.extEnd = size-keep, size
pos = 0
} else if pos+ringBlockSpace > size {
// Wrap: everything written this lap becomes the previous lap.
h.extStart, h.extEnd = 0, pos
pos = 0
}
// If there is space, append it.
if len(b) < cap(h.b)-len(h.b) {
h.b = append(h.b, b...)
return
h.ring = h.ring[:size]
end := pos + ringBlockSpace
h.b = h.ring[:pos:end]
h.ext = nil
if start := max(end, h.extStart); start < h.extEnd {
h.ext = h.ring[start:h.extEnd]
}
// Move data down so we only have window size left.
// We know we have less than window size in b at this point.
discard := len(b) + len(h.b) - h.windowSize
copy(h.b, h.b[discard:])
h.b = h.b[:h.windowSize]
copy(h.b[h.windowSize-len(b):], b)
}
// ensureBlock will ensure there is space for at least one block...
func (h *history) ensureBlock() {
if cap(h.b) < h.allocFrameBuffer {
h.b = make([]byte, 0, h.allocFrameBuffer)
return
}
avail := cap(h.b) - len(h.b)
if avail >= h.windowSize || avail > maxCompressedBlockSize {
return
}
// Move data down so we only have window size left.
// We know we have less than window size in b at this point.
discard := len(h.b) - h.windowSize
copy(h.b, h.b[discard:])
h.b = h.b[:h.windowSize]
}
// append bytes to history without ever discarding anything.
vendor/github.com/klauspost/compress/zstd/internal/xxhash/xxhash_amd64.s
Generated Vendored
+1 -5
View File
@@ -1,8 +1,4 @@
//go:build !appengine && gc && !purego && !noasm
// +build !appengine
// +build gc
// +build !purego
// +build !noasm
//go:build gc && !purego && !noasm
#include "textflag.h"
vendor/github.com/klauspost/compress/zstd/internal/xxhash/xxhash_arm64.s
Generated Vendored
+1 -5
View File
@@ -1,8 +1,4 @@
//go:build !appengine && gc && !purego && !noasm
// +build !appengine
// +build gc
// +build !purego
// +build !noasm
//go:build gc && !purego && !noasm
#include "textflag.h"
vendor/github.com/klauspost/compress/zstd/internal/xxhash/xxhash_asm.go
Generated Vendored
+1 -6
View File
@@ -1,9 +1,4 @@
//go:build (amd64 || arm64) && !appengine && gc && !purego && !noasm
// +build amd64 arm64
// +build !appengine
// +build gc
// +build !purego
// +build !noasm
//go:build (amd64 || arm64) && gc && !purego && !noasm
package xxhash
vendor/github.com/klauspost/compress/zstd/internal/xxhash/xxhash_other.go
Generated Vendored
+1 -1
View File
@@ -1,4 +1,4 @@
//go:build (!amd64 && !arm64) || appengine || !gc || purego || noasm
//go:build (!amd64 && !arm64) || !gc || purego || noasm
package xxhash
vendor/github.com/klauspost/compress/zstd/matchlen_amd64.go
Generated Vendored
+1 -1
View File
@@ -1,4 +1,4 @@
//go:build amd64 && !appengine && !noasm && gc
//go:build amd64 && !noasm && gc
// Copyright 2019+ Klaus Post. All rights reserved.
// License information can be found in the LICENSE file.
vendor/github.com/klauspost/compress/zstd/matchlen_amd64.s
Generated Vendored
+1 -1
View File
@@ -1,6 +1,6 @@
// Copied from S2 implementation.
//go:build !appengine && !noasm && gc && !noasm
//go:build !noasm && gc
#include "textflag.h"
vendor/github.com/klauspost/compress/zstd/matchlen_generic.go
Generated Vendored
+1 -1
View File
@@ -1,4 +1,4 @@
//go:build !amd64 || appengine || !gc || noasm
//go:build !amd64 || !gc || noasm
// Copyright 2019+ Klaus Post. All rights reserved.
// License information can be found in the LICENSE file.
vendor/github.com/klauspost/compress/zstd/seqdec.go
Generated Vendored
+59 -5
View File
@@ -76,6 +76,31 @@ type sequenceDecs struct {
maxSyncLen uint64
}
// twoPassFarCode is the offset code from which a match counts as far: 20 is
// 1 MiB, the distance every measured arm64 core gains from. A Neoverse N1
// (Graviton2, Ampere Altra) gains from 128 KiB too, but that needs a
// per-core threshold override; not worth the CPU-specific special case for
// a core class already being phased out of the fleet.
const twoPassFarCode = 20
// useTwoPass reports whether the block is decoded in two passes with the
// match-source prefetch (seqdec_arm64.go): a window of at least
// decodeTwoPassMinWindow and, unless the offset table is predefined or
// RLE, a share of at least twoPassMinFarShare of codes twoPassFarCode+.
func (s *sequenceDecs) useTwoPass() bool {
if s.windowSize < decodeTwoPassMinWindow {
return false
}
fse := s.offsets.fse
if fse != nil && !fse.preDefined && fse.actualTableLog != 0 && int(fse.symbolLen) <= twoPassFarCode {
// No symbol reaches twoPassFarCode, so codeShare can only return 0:
// skip finding it.
return false
}
share := fse.codeShare(twoPassFarCode)
return share < 0 || share >= twoPassMinFarShare
}
// initialize all 3 decoders from the stream input.
func (s *sequenceDecs) initialize(br *bitReader, hist *history, out []byte) error {
if err := s.litLengths.init(br); err != nil {
@@ -99,6 +124,29 @@ func (s *sequenceDecs) initialize(br *bitReader, hist *history, out []byte) erro
return nil
}
// consumeSyncLen accounts for n bytes of frame output produced outside
// decodeSync. maxSyncLen bounds how much the frame may still produce and
// decodeSyncSimple compares it with the output buffer's slack to decide
// whether the extended 16-byte copies are safe. Blocks with sequences
// subtracted their output; raw blocks, RLE blocks and compressed blocks
// holding only literals did not, so a frame that opened with a raw block
// overstated its remaining size for every block after it and ran the
// bounds-exact copies for the rest of the frame.
//
// Zero means "no bound" and selects the conservative paths, so a block that
// meets or exceeds the bound lands there; the frame-size checks in
// runDecoder reject the excess afterwards.
func (s *sequenceDecs) consumeSyncLen(n int) {
if s.maxSyncLen == 0 {
return
}
if uint64(n) >= s.maxSyncLen {
s.maxSyncLen = 0
return
}
s.maxSyncLen -= uint64(n)
}
func (s *sequenceDecs) freeDecoders() {
if f := s.litLengths.fse; f != nil && !f.preDefined {
fseDecoderPool.Put(f)
@@ -217,11 +265,17 @@ func (s *sequenceDecs) execute(seqs []seqVals, hist []byte) error {
return nil
}
// decodeSyncGoOnly makes decodeSync skip the assembly decoder, so tests can
// compare it with the Go one in the same binary.
var decodeSyncGoOnly bool
// decode sequences from the stream with the provided history.
func (s *sequenceDecs) decodeSync(hist []byte) error {
supported, err := s.decodeSyncSimple(hist)
if supported {
return err
if !decodeSyncGoOnly {
supported, err := s.decodeSyncSimple(hist)
if supported {
return err
}
}
br := s.br
@@ -344,13 +398,13 @@ func (s *sequenceDecs) decodeSync(hist []byte) error {
if mo > len(out)+len(hist) || mo > s.windowSize {
if len(s.dict) == 0 {
return fmt.Errorf("match offset (%d) bigger than current history (%d)", mo, len(out)+len(hist)-startSize)
return fmt.Errorf("match offset (%d) bigger than current history (%d)", mo, len(out)-startSize)
}
// we may be in dictionary.
dictO := len(s.dict) - (mo - (len(out) + len(hist)))
if dictO < 0 || dictO >= len(s.dict) {
return fmt.Errorf("match offset (%d) bigger than current history (%d)", mo, len(out)+len(hist)-startSize)
return fmt.Errorf("match offset (%d) bigger than current history (%d)", mo, len(out)-startSize)
}
end := dictO + ml
if end > len(s.dict) {
vendor/github.com/klauspost/compress/zstd/seqdec_amd64.go
Generated Vendored
+11 -1
View File
@@ -1,8 +1,10 @@
//go:build amd64 && !appengine && !noasm && gc
//go:build amd64 && !noasm && gc
package zstd
import (
"math"
"github.com/klauspost/compress/internal/cpuinfo"
)
@@ -10,6 +12,14 @@ import (
// in seqdec_asm.go; this file only declares the amd64 asm routines and the
// dispatch helpers that pick the BMI2 / non-BMI2 (and 56-bit / safe) variant.
// See seqdec_arm64.go. Disabled here: on Sapphire Rapids, Granite Rapids
// and Zen 4 the two-pass decode costs 7-12% on real data; their caches
// already cover the offsets involved.
var (
twoPassMinFarShare = 257 // never: the share is at most 256
decodeTwoPassMinWindow = math.MaxInt
)
// sequenceDecs_decode implements the main loop of sequenceDecs in x86 asm.
//
// Please refer to seqdec_generic.go for the reference implementation.
vendor/github.com/klauspost/compress/zstd/seqdec_amd64.s
Generated Vendored
+1439 -840
View File
File diff suppressed because it is too large. Load diff
vendor/github.com/klauspost/compress/zstd/seqdec_arm64.go
Generated Vendored
+12 -1
View File
@@ -1,4 +1,4 @@
//go:build arm64 && !appengine && !noasm && gc
//go:build arm64 && !noasm && gc
package zstd
@@ -7,6 +7,17 @@ package zstd
// by the avo arm64 lowering printer) and the dispatch helpers. arm64 has no
// BMI2, so each helper selects only between the 56-bit / safe variants.
// The synchronous decoder takes the two-pass path, and executeSimple
// prefetches, when the window is at least decodeTwoPassMinWindow and the
// block's offset table puts at least twoPassMinFarShare/256 of its
// sequences 128 KiB or more back (sequenceDecs.useTwoPass). Thresholds
// from the per-file N1 measurement in #1212; variables so tests can force
// either path.
var (
twoPassMinFarShare = 46
decodeTwoPassMinWindow = 1 << 20
)
// sequenceDecs_decode_arm64 implements the main loop of sequenceDecs in arm64 asm.
//
// Please refer to seqdec_generic.go for the reference implementation.
vendor/github.com/klauspost/compress/zstd/seqdec_arm64.s
Generated Vendored
+1085 -654
View File
File diff suppressed because it is too large. Load diff
vendor/github.com/klauspost/compress/zstd/seqdec_asm.go
Generated Vendored
+27 -15
View File
@@ -1,4 +1,4 @@
//go:build (amd64 || arm64) && !appengine && !noasm && gc
//go:build (amd64 || arm64) && !noasm && gc
package zstd
@@ -55,6 +55,9 @@ type executeAsmContext struct {
outPosition int
litPosition int
windowSize int
// prefetch enables the match-source prefetch in the execute loop; see
// sequenceDecs.useTwoPass.
prefetch bool
}
const noError = 0
@@ -83,9 +86,10 @@ const errorOverread = 6
// When the output and literal buffers have compressedBlockOverAlloc (16)
// bytes of slack past their logical use, the assembly may use extended
// memory copies that read and write in 16-byte blocks, overrunning the end
// of a literal run or match by up to 15 bytes. Otherwise it must use the
// bounds-exact ("safe") copies. This mirrors the analogous, always-dynamic
// selection in executeSimple below.
// of a literal run or match by up to 15 bytes; an empty literal run still
// copies one block, reading and writing the full 16 bytes of slack past
// its position. Otherwise it must use the bounds-exact ("safe") copies.
// This mirrors the analogous, always-dynamic selection in executeSimple below.
//
// The unsafe copies were disabled in #644 (2022) as a mitigation for a
// crash, but that crash's root cause — an unguarded bitReader overread that
@@ -94,10 +98,15 @@ const errorOverread = 6
// See #1168. An asan-instrumented fuzz job (see .github/workflows/go.yml)
// covers the extended-copy path, which -race and plain fuzzing cannot.
func (s *sequenceDecs) useSafeDecodeSync() bool {
if s.maxSyncLen == 0 && cap(s.out)-len(s.out) < maxCompressedBlockSizeAlloc {
// A block never produces more than maxBlockSize bytes.
maxBlockSize := min(s.windowSize, maxCompressedBlockSize)
if s.maxSyncLen == 0 && cap(s.out)-len(s.out) < maxBlockSize+compressedBlockOverAlloc {
return true
}
if s.maxSyncLen > 0 && cap(s.out)-len(s.out)-compressedBlockOverAlloc < int(s.maxSyncLen) {
// The block, not the rest of the frame, must fit with the slack: a
// buffer sized exactly to the frame needs the safe copies only for the
// final block(s).
if s.maxSyncLen > 0 && cap(s.out)-len(s.out)-compressedBlockOverAlloc < min(int(s.maxSyncLen), maxBlockSize) {
return true
}
if cap(s.literals) < len(s.literals)+compressedBlockOverAlloc {
@@ -111,7 +120,8 @@ func (s *sequenceDecs) decodeSyncSimple(hist []byte) (bool, error) {
if len(s.dict) > 0 {
return false, nil
}
if s.maxSyncLen == 0 && cap(s.out)-len(s.out) < maxCompressedBlockSize {
maxBlockSize := min(s.windowSize, maxCompressedBlockSize)
if s.maxSyncLen == 0 && cap(s.out)-len(s.out) < maxBlockSize {
return false, nil
}
@@ -119,8 +129,6 @@ func (s *sequenceDecs) decodeSyncSimple(hist []byte) (bool, error) {
br := s.br
maxBlockSize := min(s.windowSize, maxCompressedBlockSize)
ctx := decodeSyncAsmContext{
llTable: s.litLengths.fse.dt[:maxTablesize],
mlTable: s.matchLengths.fse.dt[:maxTablesize],
@@ -153,7 +161,7 @@ func (s *sequenceDecs) decodeSyncSimple(hist []byte) (bool, error) {
case errorMatchOffTooBig:
return true, fmt.Errorf("match offset (%d) bigger than current history (%d)",
ctx.mo, ctx.outPosition+len(hist)-startSize)
ctx.mo, ctx.outPosition-startSize)
case errorNotEnoughLiterals:
return true, fmt.Errorf("unexpected literal count, want %d bytes, but only %d is available",
@@ -266,8 +274,10 @@ func (s *sequenceDecs) decode(seqs []seqVals) error {
// executeSimple handles cases when dictionary is not used.
func (s *sequenceDecs) executeSimple(seqs []seqVals, hist []byte) error {
// Ensure we have enough output size...
if len(s.out)+s.seqSize+compressedBlockOverAlloc > cap(s.out) {
// Grow only if the block does not fit. Lacking just the slack is
// handled by the safe copies below, so a buffer sized exactly to the
// frame is not reallocated and copied on its last block.
if len(s.out)+s.seqSize > cap(s.out) {
addBytes := s.seqSize + len(s.out) + compressedBlockOverAlloc
s.out = append(s.out, make([]byte, addBytes)...)
s.out = s.out[:len(s.out)-addBytes]
@@ -289,10 +299,12 @@ func (s *sequenceDecs) executeSimple(seqs []seqVals, hist []byte) error {
litPosition: 0,
literals: s.literals,
windowSize: s.windowSize,
prefetch: s.useTwoPass(),
}
// useSafe avoids overwriting the output buffer when the literals slice has
// not been allocated with the required over-allocation slack.
useSafe := cap(s.literals) < len(s.literals)+compressedBlockOverAlloc
// useSafe avoids writing past either buffer when the literals or the
// output lack the extended copies' over-allocation slack.
useSafe := cap(s.literals) < len(s.literals)+compressedBlockOverAlloc ||
cap(out) < len(out)+compressedBlockOverAlloc
ok := executeSimpleAsm(&ctx, useSafe)
if !ok {
vendor/github.com/klauspost/compress/zstd/seqdec_generic.go
Generated Vendored
+8 -1
View File
@@ -1,10 +1,17 @@
//go:build (!amd64 && !arm64) || appengine || !gc || noasm
//go:build (!amd64 && !arm64) || !gc || noasm
package zstd
import (
"fmt"
"io"
"math"
)
// See seqdec_arm64.go; the pure Go decoder has neither path.
var (
twoPassMinFarShare = 257 // never: the share is at most 256
decodeTwoPassMinWindow = math.MaxInt
)
// decode sequences from the stream with the provided history but without dictionary.
vendor/github.com/klauspost/compress/zstd/seqenc.go
Generated Vendored
+47
View File
@@ -110,3 +110,50 @@ func ofCode(offset uint32) uint8 {
// A valid offset will always be > 0.
return uint8(bits.Len32(offset) - 1)
}
// newDictSeqEncoder builds an encoder for a dictionary sequence table.
// Returns nil if the table is missing or unusable.
func newDictSeqEncoder(src *fseDecoder, t tableIndex) *fseEncoder {
if src == nil || src.symbolLen == 0 {
return nil
}
var e fseEncoder
e.norm = src.norm
e.symbolLen = src.symbolLen
e.actualTableLog = src.actualTableLog
if e.buildCTable() != nil {
return nil
}
e.setBits(bitTables[t])
return &e
}
// seedPrevFromDict loads the dictionary sequence tables as the "previous"
// encoders, so the first block can use repeat mode.
func (s *seqCoders) seedPrevFromDict(d *dict) {
seed := func(dst, src *fseEncoder) {
if src == nil {
return
}
ct := dst.ct
dst.symbolLen = src.symbolLen
dst.actualTableLog = src.actualTableLog
dst.maxBits = src.maxBits
dst.zeroBits = src.zeroBits
dst.useRLE = false
dst.preDefined = false
dst.reUsed = true
dst.norm = src.norm
// tableSymbol is only used while building, so it is not copied.
ct.stateTable = append(ct.stateTable[:0], src.ct.stateTable...)
if cap(ct.symbolTT) < len(src.ct.symbolTT) {
ct.symbolTT = make([]symbolTransform, len(src.ct.symbolTT))
}
ct.symbolTT = ct.symbolTT[:len(src.ct.symbolTT)]
copy(ct.symbolTT, src.ct.symbolTT[:src.symbolLen])
dst.ct = ct
}
seed(s.llPrev, d.llEnc)
seed(s.mlPrev, d.mlEnc)
seed(s.ofPrev, d.ofEnc)
}
vendor/github.com/klauspost/compress/zstd/simple_go124.go → vendor/github.com/klauspost/compress/zstd/simple.go
Generated Vendored
-2
View File
@@ -1,8 +1,6 @@
// Copyright 2025+ Klaus Post. All rights reserved.
// License information can be found in the LICENSE file.
//go:build go1.24
package zstd
import (
vendor/github.com/klauspost/compress/zstd/snappy.go
Generated Vendored
+12
View File
@@ -69,6 +69,10 @@ var (
// No CRC value is being generated and not all CRC values of the Snappy stream are checked.
// However, it provides really fast recompression of Snappy streams.
// The converter can be reused to avoid allocations, even after errors.
//
// Deprecated: Decode the stream with s2.NewReader, which reads Snappy streams,
// and compress the output with an Encoder. This validates the input and
// compresses much better.
type SnappyConverter struct {
r io.Reader
err error
@@ -373,6 +377,14 @@ func decodeSnappy(blk *blockEnc, src []byte) error {
offset = uint32(src[s-4]) | uint32(src[s-3])<<8 | uint32(src[s-2])<<16 | uint32(src[s-1])<<24
}
// A zstd match is at least zstdMinMatch bytes. A shorter copy
// cannot be represented as a sequence and would underflow the
// matchLen computation below, producing an out-of-range match
// length code.
if length < zstdMinMatch {
return ErrSnappyCorrupt
}
if offset <= 0 || blk.size+lits < int(offset) /*|| length > len(blk)-d */ {
println("offset <= 0 || blk.size+lits < int(offset)", offset, blk.size+lits, int(offset), blk.size, lits)
vendor/modules.txt
Vendored
+1 -1
View File
@@ -416,7 +416,7 @@ github.com/intel/goresctrl/pkg/utils
# github.com/json-iterator/go v1.1.12
## explicit; go 1.12
github.com/json-iterator/go
# github.com/klauspost/compress v1.20.0
# github.com/klauspost/compress v1.20.3
## explicit; go 1.25
github.com/klauspost/compress
github.com/klauspost/compress/flate