Add a NEON implementation of the fixed-point DCA LFE FIR interpolation
filter.
Each vector lane holds one of four consecutive LFE samples, so the
filter coefficients are used as by-element multipliers and can be
loaded contiguously without any permutes. One load of c[8j..8j+7] and
c[248-8j..255-8j] covers outputs j, 32+j, 31-j and 63-j. The 8-tap
sums are accumulated in 64 bits with SMULL/SMLAL, rounded with SQRSHRN,
clipped to 23 bits with SQSHL/SSHR and written with ST2 lane stores.
Recent clang versions auto-vectorize the C version, so the speedup
depends a lot on the compiler. On Neoverse-N1 (64 LFE samples):
C compiler speedup
clang 14 4.84x
gcc 11 1.75x
clang 20 1.21x
clang 22 1.21x
Signed-off-by: Jeongkeun Kim <variety0724@gmail.com>
Like the x86 versions, these may read and write past the end, here by
up to 15 bytes. The main loops process eight vectors per iteration;
with four, the in-order Cortex-A510 stalls on the latency of the prefix
sums, and add_left_pred_int16 was only 1.25x faster than C there.
checkasm --bench (CPU cycles via perf), seed 12345 (width 983), on a
Dimensity 9000:
Cortex-X2 Cortex-A710 Cortex-A510
add_gradient_pred_c: 2138.4 2183.6 4450.3
add_gradient_pred_neon: 225.7 (9.48x) 380.3 (5.74x) 711.6 (6.25x)
add_left_pred_int16_c: 1926.4 1941.4 2112.0
add_left_pred_int16_neon: 370.6 (5.20x) 630.6 (3.08x) 1058.6 (2.00x)
add_left_pred_rnd_acc_c: 1056.5 1029.6 1743.4
add_left_pred_rnd_acc_neon: 201.9 (5.23x) 347.5 (2.96x) 585.2 (2.98x)
add_left_pred_zero_c: 1056.0 1029.2 1742.3
add_left_pred_zero_neon: 201.8 (5.23x) 347.5 (2.96x) 585.2 (2.98x)
and on a Google Tensor (Pixel 6a):
Cortex-X1 Cortex-A76 Cortex-A55
add_gradient_pred_c: 2137.4 2848.2 6892.2
add_gradient_pred_neon: 238.2 (8.97x) 408.2 (6.98x) 1117.5 (6.17x)
add_left_pred_int16_c: 1935.0 2177.2 3950.8
add_left_pred_int16_neon: 398.0 (4.86x) 664.4 (3.28x) 1690.0 (2.34x)
add_left_pred_rnd_acc_c: 1059.7 1533.9 2967.1
add_left_pred_rnd_acc_neon: 214.5 (4.94x) 378.2 (4.06x) 925.4 (3.21x)
add_left_pred_zero_c: 1059.8 1533.9 2967.1
add_left_pred_zero_neon: 214.5 (4.94x) 378.2 (4.06x) 925.4 (3.21x)
Signed-off-by: Gyula Sebok <gyula.sbk@gmail.com>
The original implementation was slower than C on all tested cores.
The new version computes 8 outputs per iteration. The caller always
passes even start and end, which the x86 version also relies on, so
the tail only handles 4 and 2 outputs.
checkasm --bench (cycles, median of 7 runs). A55/A76/X1: Pixel 6a,
A510/A710/X2: Vivo Pad 2.
clang c neon_original neon_new
A55 900 2303 (0.39x) 576 (1.56x)
A76 302 550 (0.55x) 204 (1.48x)
X1 173 322 (0.54x) 109 (1.59x)
A510 952 2398 (0.40x) 459 (2.07x)
A710 262 498 (0.53x) 194 (1.35x)
X2 169 298 (0.57x) 106 (1.58x)
gcc c neon_original neon_new
A55 990 2301 (0.43x) 575 (1.72x)
A76 382 549 (0.69x) 205 (1.87x)
X1 208 322 (0.65x) 108 (1.91x)
A510 971 2395 (0.41x) 449 (2.16x)
A710 338 498 (0.68x) 194 (1.74x)
X2 202 298 (0.68x) 107 (1.89x)
5.1 HE-AAC decoding with -threads 1 is about 1-2.5% faster.
Signed-off-by: Gyula Sebok <gyula.sbk@gmail.com>
vector_clip_int32 is based on the 32-bit ARM NEON version. Unrolling to
16 or 32 elements per iteration gave no measurable improvement over 8.
vector_clipf applies fmin before fmaxnm, so any NaN input, including a
signaling one, yields min as in C and as documented for av_clipf.
checkasm --bench on Apple M4:
vector_clip_int32_c: 1029.6
vector_clip_int32_neon: 180.7 ( 5.38x)
vector_clipf_c: 682.9
vector_clipf_neon: 190.8 ( 3.34x)
Signed-off-by: Patryk Darmofalski <darpat039@gmail.com>
Adds a new dsp function to compute a constants buffer in the form most
suitable to the specific implementation. This gets called once per
tile component to prepare the constants buffer, and then the transquant
functions get a pointer to that buffer to use.
CPU: Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz
Before:
decode_transquant_10_c: 333.2
decode_transquant_10_avx2: 65.5 ( 5.09x)
After:
decode_transquant_10_c: 330.9
decode_transquant_10_avx2: 57.3 ( 5.78x)
No significant difference on ARM Cortex-A53/A73.
The eight positions of the eight-row window are the same walk for every
vertical tail, and calc_all already carries it. The wider weighted tails stay
as they are: calc_all2 fixes its window to v16..v31, so handing them over means
moving their accumulators and weight vectors out of that range first.
No functional changes.
The unweighted step macros in this file let a post-indexed load walk the
intermediate plane, while these weighted steps loaded a row and then advanced
sp with a separate add. Fold the advance into the load.
The step macro now names its row register without a width letter, which is
what the next change needs in order to drive these tails from calc_all.
The weighted narrowing after the horizontal filter was repeated at every
width. Sharing one stage at a time rather than a whole block at a time
keeps the serial sqxtn/sqxtn2/sqxtun chain of a block from stalling an
in-order core.
pngdsp.c calls ff_pngdsp_init_aarch64 on this architecture, and both
decoders link pngdsp.o. The init and NEON objects were selected only
for the PNG decoder, so a build with APNG or LSCR and without PNG
failed to link.
Signed-off-by: Jun Zhao <barryjzhao@tencent.com>
Unpack packed 10-bit YUV 4:2:2 in 16-pixel NEON blocks and unroll the
main loop to 32 pixels. Keep exact-width source loads and use the scalar
implementation for decoder tails smaller than eight pixels.
Enable the optimized implementation on little-endian AArch64. Retain the
C fallback on big-endian AArch64, where the byte layout expected by this
implementation does not apply.
The implementation passed 300 consecutive checkasm repetitions and
produced bit-exact output in a 100-frame UHD decode.
Thanks to Martin Storsjö for feedback and suggestions for improvement.
Correctness command:
./tests/checkasm/checkasm --test=bitpackeddec --repeat=300 66000
Benchmark command:
./tests/checkasm/checkasm --bench --test=bitpackeddec \
--function=bitpacked_unpack_yuv422p10 --duration=1000000 95000
Apple M4 Mac mini, Apple clang 17.0.0:
bitpacked_unpack_yuv422p10_c: 876.8 ns
bitpacked_unpack_yuv422p10_neon: 227.2 ns (3.86x)
Clang auto-vectorizes the C reference with NEON, so the result measures
the hand-written implementation against compiler-generated SIMD rather than
against purely scalar machine code.
calc_epelb and calc_epelb2 open with a widening multiply, which writes its
destination instead of accumulating into it, so the movi that cleared the
accumulator ahead of them had no effect. The zeroing that feeds a usdot or
an umlsl stays, since both add into their destination.
calc_qpelb and calc_qpelb2 open with a widening multiply, which writes its
destination instead of accumulating into it, so the movi that cleared the
accumulator ahead of them had no effect. The plain vertical steps at widths
4 to 64 and the bi steps at widths 4 to 32 carried it.
The position tables are indexed by sao_eo_class: index 2 is SAO_EO_135D,
with neighbours at (-1,-1) and (1,1), and index 3 is SAO_EO_45D.
Signed-off-by: Zhao Zhili <zhilizhao@tencent.com>
S16 saturating sum must be computed dot-product-first. The NEON code
accumulated the offset into the dot product with wrapping
multiply-accumate, so large-magnitude sums wrapped instead of saturating.
Accumulate the dot product from zero (wrapping is exact: the dot product
cannot overflow S16 for 8-bit with log2_denom < 7), then add the offset
with a saturating add.
This fixes bi-weight checkasm test for NEON.
Signed-off-by: Zuxy Meng <zuxy.meng@gmail.com>
Add/sub with saturation is used only on log2_denom > 1 path, but with
log2_denom == 1 (block[x]*weight + offset << 1) can fall below S16_MIN.
Use saturation in all paths for safety.
This fixes the weight checkasm test for arm and aarch64
Signed-off-by: Zuxy Meng <zuxy.meng@gmail.com>
The plain hv4 and hv6 vertical tails walk the intermediate plane using
a row pitch in x7 that their entry glue loads before the horizontal
kernel call. The neon kernels reuse x7 as their destination stride and
return with it set to 128, and the i8mm ones happen not to touch it, so
the value survives the call by accident even though x7 is caller-saved.
Load the pitch in the tails, as hv8, hv12, hv16 and hv32 already do.
Implement the fixed-point LFE x96 interpolation filter in AArch64
NEON. Four input samples are processed per iteration. The history
sample is kept in a vector lane, and ext constructs the delayed input.
Benchmarks by Martin Storsjö using checkasm:
a53 a55 a520 a72 a76 a720
1.86x 1.88x 1.24x 1.99x 1.62x 2.30x
Author's measurement on Neoverse-N1 (Oracle Cloud, clang 14): 1.29x.
Signed-off-by: Jeongkeun Kim <variety0724@gmail.com>
armasm64 warns "A4228: Alignment value exceeds AREA alignment; alignment
not guaranteed" on the .Lcoeffs literal pool. gas-preprocessor.pl emits
the text section as "AREA |.text|, CODE, READONLY, ALIGN=4", i.e. a
16-byte section alignment, so a 32-byte ALIGN inside it cannot be
guaranteed by the assembler.
16 is enough here: the pool holds .quad values that are read with
"ldr d0, .Lcoeffs" literal loads, which only require 8-byte alignment.
It also matches the rest of the aarch64 asm, where the function and
const macros both default to .align 4.
Signed-off-by: Kacper Michajłow <kasper93@gmail.com>
So callers can pass a named step macro instead of relying on a locally
redefined `calc` followed by `.purgem calc`.
Signed-off-by: Zhao Zhili <zhilizhao@tencent.com>
Only the sum of the offsets is ever used
(see equation 8-279 in the 2019 version of the H.265 spec).
Tested-by: Martin Storsjö <martin@martin.st>
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>
The aarch64 VP9 loopfilters actually violate aarch64 GCS
(Guarded Control Stack), even though we marked the code as GCS
compliant in 846746be4b.
This means that builds with GCS enabled, after that commit,
will crash when decoding VP9, on future hardware (or current
QEMU) that supports GCS. This also goes for ffmpeg version 8.1.1
where the GCS enabling was backported.
This matches the fix that was done for hevcdsp in
1f7ed8a78d.
This issue wasn't observed if running checkasm in QEMU - therefore,
I thought all GCS issues had been fixed by
846746be4b. (If I would have
tested the full "make fate" with QEMU, the issue would
have appeared though.)
However with the new checkasm, some of the GCS violations
do appear even in checkasm.
The reason is that the checkasm vp9 test intentionally craft
input pixels that attempt to trigger all the individual
separate cases in each input buffer (in
randomize_loopfilter_buffers). This means that the checkasm
tests actually never test or exercise the early exit cases,
which are the ones that violate GCS.
With the new checkasm, the call to "bench_new" always test
running the code at least once, even if not benchmarking.
As the input buffers weren't reinitialized between the test
and "bench_new", the pixel differences now differ from the
initial setup, so that the code now some times (often) would
end up hitting the early exit cases.
Ideally, the vp9 checkasm test would be repeated to cover all
cases of input buffers that allow early exits, in addition to
covering the case with all different cases in one block.
Add NEON-optimized implementations for HEVC angular intra prediction
modes 10 (pure horizontal) and 26 (pure vertical) at 8-bit depth.
Mode 10 (Horizontal):
- Broadcasts left[y] to fill each row using ld2r/ld4r for efficiency
- Applies edge smoothing for luma blocks smaller than 32x32
Mode 26 (Vertical):
- Copies top reference row to all output rows
- Applies edge smoothing for luma blocks smaller than 32x32
Edge smoothing uses uhsub+usqadd to compute the filtered result
directly in 8-bit, avoiding widening to 16-bit intermediates.
The C pred_angular wrappers are made non-static with ff_ prefix to
allow the NEON dispatch to fall back to C for modes not yet optimized.
This will be reverted once all angular modes are implemented.
Note: since pred_angular[] is a per-size function pointer (not
per-mode), checkasm benchmarks will show '_neon' for all 33 modes
even though only modes 10/26 are truly accelerated; unoptimized
modes show ~1.0x speedup as they pass through the NEON wrapper to
the C fallback with negligible overhead.
Speedup over C on Apple M4 (checkasm --bench, 15-run average):
Mode 10 (Horizontal):
4x4: 4.66x 8x8: 5.80x 16x16: 16.86x 32x32: 24.89x
Mode 26 (Vertical):
4x4: 1.16x 8x8: 1.83x 16x16: 2.45x 32x32: 4.50x
Signed-off-by: Jun Zhao <barryjzhao@tencent.com>
The C version is faster than the previous asm with clang and gcc > 12 on
rpi5, since compiler basically does the same unroll.
sum64x5_neon: before after
Cortex-A76 (gcc 12.4): 72.3 (3.63x) 47.4 (5.56x)
Cortex-A76 (gcc 14.2): 72.3 (0.69x) 47.4 (1.05x)
Apple M1 (clang 16): 0.2 (0.98x) 0.2 (0.99x)
Signed-off-by: Zhao Zhili <quinkblack@foxmail.com>
Port lfe_fir0_float and lfe_fir1_float to AArch64 NEON. These polyphase
FIR interpolation filters have an x86 SSE/AVX path but no AArch64
equivalent, falling back to scalar C.
The inner loop computes two dot products per output pair. Precomputing a
reversed LFE sample vector before the inner loop avoids per-iteration
shuffle overhead.
Benchmarks on AWS Graviton3 (Neoverse V1, c7g.xlarge):
lfe_fir0_float: C 5902.0 cycles -> NEON 2135.0 cycles (2.77x)
lfe_fir1_float: C 2836.3 cycles -> NEON 1527.8 cycles (1.86x)
Measured with: taskset -c 0 ./tests/checkasm/checkasm --test=dcadsp --bench,
3-run average, Ubuntu 22.04 (kernel 6.8.0-1052-aws), perf_event_paranoid=0.
Signed-off-by: Jeongkeun Kim <variety0724@gmail.com>
3-tap [1,2,1]>>2: shared implementation body across size-specialized
entry points (8x8/16x16/32x32) to reduce code size. Fold the 3-tap
kernel into uhadd + urhadd: uhadd gives floor((prev+next)/2), then
urhadd rounds with curr to produce (prev + 2*curr + next + 2) >> 2
on 16 bytes in-place (no widen/narrow needed). Overlap-last technique
for tail avoids partial stores. Caller pads input arrays by 16 bytes
to guarantee safe over-read.
Strong smoothing (32x32): preloaded weight tables, interleaved
umull/umlal pairs (two 16-byte blocks at a time) to hide
rshrn-to-store latency, with paired st1 for 32-byte writes.
checkasm --bench --runs=15 (Apple M4, average of 3 trials):
ref_filter_3tap_8x8_8_neon: 4.1x
ref_filter_3tap_16x16_8_neon: 3.3x
ref_filter_3tap_32x32_8_neon: 2.5x
ref_filter_strong_8_neon: 1.9x
Signed-off-by: Jun Zhao <barryjzhao@tencent.com>
Add NEON-optimized implementation for HEVC intra Planar prediction at
8-bit depth, supporting all block sizes (4x4 to 32x32).
Planar prediction implements bilinear interpolation using an incremental
base update: base_{y+1}[x] = base_y[x] - (top[x] - left[N]), reducing
per-row computation from 4 multiply-adds to 1 subtract + 1 multiply.
Uses rshrn for rounded narrowing shifts, eliminating manual rounding
bias. All left[y] values are broadcast in the NEON domain, avoiding
GP-to-NEON transfers.
4x4 interleaves row computations across 4 rows to break dependencies.
16x16 uses v19-v22 for persistent base/decrement vectors, avoiding
callee-saved register spills. 32x32 processes 8 rows per loop iteration
(4 iterations total) to reduce code size while maintaining full NEON
utilization.
Speedup over C on Apple M4 (checkasm --bench):
4x4: 2.25x 8x8: 6.40x 16x16: 9.72x 32x32: 3.21x
Signed-off-by: Jun Zhao <barryjzhao@tencent.com>
Add NEON-optimized implementation for HEVC intra DC prediction at 8-bit
depth, supporting all block sizes (4x4 to 32x32).
DC prediction computes the average of top and left reference samples
using uaddlv, with urshr for rounded division. For luma blocks smaller
than 32x32, edge smoothing is applied: the first row and column are
blended toward the reference using (ref[i] + 3*dc + 2) >> 2 computed
entirely in the NEON domain. Fill stores use pre-computed address
patterns to break dependency chains.
Also adds the aarch64 initialization framework (Makefile, pred.c/pred.h
hooks, hevcpred_init_aarch64.c).
Speedup over C on Apple M4 (checkasm --bench):
4x4: 2.28x 8x8: 3.14x 16x16: 3.29x 32x32: 3.02x
Signed-off-by: Jun Zhao <barryjzhao@tencent.com>
Whenever the link register is stored on the stack, sign it
before storing it and validate at a symmetrical point (with the
stack at the same level as when it was signed).
These macros only have an effect if built with PAC enabled (e.g.
through -mbranch-protection=standard), otherwise they don't
generate any extra instructions.
None of these cases were present when PAC support was added
in 248986a0db in 2022.
Without these changes, PAC still had an effect in the compiler
generated code and in the existing cases where we these macros were
used - but make it apply to the remaining cases of link register
on the stack.
The sme_entry/sme_exit macros already take care of backing up/restoring
these registers. Additionally, as long as no function calls are
made within the function, x30 doesn't need to be backed up at all.
For cases when returning early without updating any pixels, we
previously returned to return address in the caller's scope,
bypassing one function entirely. While this may seem like a neat
optimization, it makes the return stack predictor mispredict
the returns - which potentially can cost more performance than
it gains.
Secondly, if the armv9.3 feature GCS (Guarded Control Stack) is
enabled, then returns _must_ match the expected value; this feature
is being enabled across linux distributions, and by fixing the
hevc assembly, we can enable the security feature on ffmpeg as well.
Group assignments by filter family (qpel, epel), variant
(base, uni, bi, uni_w, bi_w) and direction (pixels, h, v, hv).
Add NEON8_FNASSIGN_QPEL_H macro to replace repeated manual
qpel horizontal assignments.
No functional change.
Signed-off-by: Jun Zhao <barryjzhao@tencent.com>
Add NEON-optimized implementations for HEVC QPEL uni-directional
weighted HV interpolation (put_hevc_qpel_uni_w_hv) at 8-bit depth,
for block widths 6, 12, 24, and 48.
These functions perform horizontal then vertical 8-tap QPEL filtering
with weighting (wx, ox, denom) and output to uint8_t. Previously
only widths 4, 8, 16, 32, 64 were implemented; this completes
coverage for all standard HEVC block widths.
Performance results on Apple M4:
./tests/checkasm/checkasm --test=hevc_pel --bench
put_hevc_qpel_uni_w_hv6_8_neon: 3.11x
put_hevc_qpel_uni_w_hv12_8_neon: 3.19x
put_hevc_qpel_uni_w_hv24_8_neon: 2.26x
put_hevc_qpel_uni_w_hv48_8_neon: 1.80x
Signed-off-by: Jun Zhao <barryjzhao@tencent.com>
Jeongkeun Kim