itx: cancel 16 out of '(x * 2896) >> 12' to avoid integer overflows (!494) · Merge requests · VideoLAN / dav1d

Janne Grunau requested to merge janne/dav1d:simplify_itx into master Dec 12, 2018

Fixes an integer overflow in inv_dct4_1d with clusterfuzz-testcase-dav1d_fuzzer-5634807321591808 and in inv_adst16_1d with clusterfuzz-testcase-dav1d_fuzzer-5761827623927808. Credits to oss-fuzz.

itx: cancel 16 out of '(x * 2896) >> 12' to avoid integer overflows

Merge request reports