set_string_number() parses the operand of a "+x" or "-x" flags update as
a double and casts it to int64_t to OR it into, or mask it out of, the
current flags. Only the combined value then reaches the range check in
write_number(), so the operand itself was never validated:
- A value outside the int64_t range, or inf, makes the conversion
undefined behaviour (C17 6.3.1.4). UBSan reports it for
av_opt_set(ctx, "flags", "+77777777777777777777", 0), and what the
cast produces depends on the target: measured on AArch64, where it
saturates, "+1e20" happened to be rejected downstream while "-1e20"
was accepted.
- A fractional operand is silently truncated, so "+1.5" and "-1.5" are
accepted although "1.5" is refused as not being a valid set of flags.
Run the existing flags range check on the operand before the cast, so an
invalid operand returns AVERROR(ERANGE) exactly like an invalid absolute
value, as av_opt_set() documents, and add the four cases to fate-opt.
Reported by FuzzAnything-ORG with a UBSan trace.
Fixes#22389