The code loads 32bit so we can at maximum use 32bit
the return type is also changed to uint16_t (was requested in review),
no path is known where a return value above 32767 is produced, but that was not exhaustively checked
Fixes: runtime error: shift exponent -9 is negative
Fixes: 439483046/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_PRORES_RAW_DEC_fuzzer-6649466540326912
Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
the values contain 3 4 bit values, thus using hex is more natural
and shows more information
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
Sometimes functions for bit depth 9..16 are the same (because they
actually only depend on the underlying pixel type). The macros added
here allow to support this usecase.
Reviewed-by: Kacper Michajłow <kasper93@gmail.com>
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>
For mv-hevc, the second layer of IDR frame can be a P slice.
long_term_rps wasn't been reset before the patch, which leading to
ff_hevc_frame_nb_refs return incorrect result.
This fix decoding failure for samples from Pico VR.
Fixes: signed integer overflow: 2147483565 + 128 cannot be represented in type 'int'
Fixes: 428055715/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_OSQ_fuzzer-6358069900804096
Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
The qScale syntax element has a maximum value of 512, which would overflow the 16-bit store from the VLD shader in extreme cases.
This fixes that edge case by forwarding the element in a storage buffer, and applying the inverse quantization fully in the IDCT shader.
It is no longer necessary now that the IDCTDSP is always ABI-compliant
(and free of MMX).
Reviewed-by: Lynne <dev@lynne.ee>
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>
Before this commit, the (32-bit only) simple idct came in three
versions: A pure MMX IDCT and idct-put and idct-add versions
which use SSE2 at the put and add stage, but still use pure MMX
for the actual IDCT.
This commit ports said IDCT to SSE2; this was entirely trivial
for the IDCT1-5 and IDCT7 parts (where one can directly use
the full register width) and was easy for IDCT6 and IDCT8
(involving a few movhps and pshufds). Unfortunately, DC_COND_INIT
and Z_COND_INIT still use only the lower half of the registers.
This saved 4658B here; the benchmarking option of the dct test tool
showed a 15% speedup.
Reviewed-by: Lynne <dev@lynne.ee>
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>
Unnecessary since the Xvid IDCT no longer uses MMX registers at all.
(Notice that the simple MMX IDCT issues emms and is therefore ABI
compliant.)
Reviewed-by: Lynne <dev@lynne.ee>
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>
It is higly surprising and would necessitate emms in order to be ABI
compliant; but it is better just not to use them in the first place.
Reviewed-by: Lynne <dev@lynne.ee>
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>
The test has been removed in bfb28b5ce8
when MMX idctdsp functions overridden by SSE2 were removed;
ff_simple_idct_mmx() has been completely disabled in this patch
for x64 and so the test should have been disabled on x64 instead
of removing it.
Reviewed-by: Lynne <dev@lynne.ee>
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>
bfb28b5ce8 removed the MMX idct_put
and idct_add functions, because they were overridden by SSE2 versions
(which use SSE2 only for the put/add part, not the actual IDCT).
This meant that for MMX, the idct functions are not set in unison,
so that the permutation which is meant to apply to all three
is incorrect on 32bit systems if SSE2 is unavailable/disabled.
Fix this by setting the MMX version only if SSE2 is enabled.
(No one complained, so apparently no one uses a new FFmpeg
with non-SSE2 capable systems.)
Reviewed-by: Lynne <dev@lynne.ee>
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>
The quantization parameter (qp) can exceed 63 when the base value
from frame header (0-63) is combined with the offset from slice data
(up to +2), resulting in qp=65. This causes out-of-bounds access to
the rv60_qp_to_idx[64] array in decode_cbp8(), decode_cbp16(), and
get_c4x4_set().
Fixes: Out-of-bounds read
Signed-off-by: oblivionsage <cookieandcream560@gmail.com>
No testsample is available
This is related to 61cbcaf93f and clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_RV60_fuzzer-5160167345291264
which fixed rv60_qp_to_idx[qp + 32] out of array access
These 2 checks are not redundant and neither covers the cases of the other
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
update the switch statement for EXR_UINT in pxr24_uncompress to
correctly use the tile width td->xsize instead of using the full window
width s->xdelta. s->delta is larger than td->xsize which lead to two
buffer overflows when interacting with the ptr variable in the same
switch statement.
Fixes: out of bounds read and write
Found-by: veygax's insomnia network (INSOMNIA-1)
Signed-off-by: veygax <veyga@veygax.dev>
The existing is_luma check is fragile as depending on the order
of channels it can be set or reset
No testcase
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
There is no need to check the whole AVChannelLayout; checking
its nb_channels is enough.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>
E.g. check that the ordinary entries are valid (e.g. no negative sample
rates, must have pix fmt descriptor) and also that the sentinels
are valid where they contain redundant information.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>
For decoders, the pixel format is negotiated via the get_format callback
(if there is a choice at all), so decoders should not set pix_fmts.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>
It is not supposed to be set by decoders (where format negotiation
happens via the get_format callback).
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>
Right now, FFCodec contains a dedicated pointer for
alpha_mode; this is wasteful, as there is only a very limited
range of options for this value, namely four.
So store it as a two-bit bitfield like color_ranges.
This reduces sizeof(FFCodec) by 16 here (the placement
of alpha_mode entailed unnecessary padding) and
saves 11328B of .data.rel.ro here (on a standard build with
no external libraries).
(If it were not for effective-type violations, one could
share the code and the table with AV_CODEC_CONFIG_COLOR_RANGE.)
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>
avcodec_get_supported_config() always sets a dummy for
the pointer to the number of entries in case it is not set.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>
And rename it to ff_encode_add_stats_side_data() and move its
declaration to encode.h. Also constify the error pointee.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>
pict_type is always used since 5e0eac3d4566839598f6d6fe5d77770023d6713a;
ctx in set_pix_fmt() seems to have been always unused.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>
The optimised version does not work if the stride is not a multiple 8,
which can occur as reproduce by vsynth3-asv1 and vsynth3-asv2 tests.
This reverts commit 02594c8c01.
Conflicts:
libavcodec/riscv/pixblockdsp_init.c
libavcodec/riscv/pixblockdsp_rvv.S
