avformat/rtmpproto: validate compressed SWF header length

Reject truncated compressed SWF input before attempting to read the
8-byte header in rtmp_calc_swfhash().

Compressed SWF data identified by the "CWS" signature must be at least
8 bytes long to contain the fixed header. Bail out early when the input
is shorter to avoid operating on malformed data.

*Vulnerability reported by Zhenpeng (Leo) Lin at depthfirst*
*Patch validated by Zheng Yu at depthfirst*

Fixes: DFVULN-612
(cherry picked from commit 52b78cd3fe)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

libavformat/rtmpproto.c

@@ -1164,6 +1164,10 @@ static int rtmp_calc_swfhash(URLContext *s)
     if (!memcmp(in_data, "CWS", 3)) {
 #if CONFIG_ZLIB
         int64_t out_size;
+        if (in_size < 8) {
+            ret = AVERROR_INVALIDDATA;
+            goto fail;
+        }
         /* Decompress the SWF player file using Zlib. */
         if (!(out_data = av_malloc(8))) {
             ret = AVERROR(ENOMEM);