x264 memory leaks error at __interceptor_malloc

I created a related confidential issue 5 days ago (since I cannot modify its visibility, I'm submitting a new public issue).

Summary of the bug:

Hello, dear developers:,

I encountered a memory leak issue while using FFmpeg with version git master. When I execute configure, if --enable-libx264 and --enable-gpl are enabled, it will trigger a problem. If I don't enable these two options, there won't be any problem. I tried compiling the latest master branch x264 on my own instead of downloading it from Ubuntu, and I was able to reproduce the issue.

The FFmpeg team confirmed that this issue is related to a memory leak in the x264 library, not FFmpeg itself. As suggested by the FFmpeg developers, I am reporting this issue to the x264 team.

Please confirm, I have also contacted the developers of FFmpeg, and here is their response:

by James, 10 days ago

Keywords:	bugs removed
Priority:	important → normal
Resolution:	→ invalid
Status:	new → closed

This looks like a memleak in libx264, not ffmpeg. x264_encoder_open() fails and apparently doesn't do a proper clean up. You should report it to them.

If this issue is confirmed and a patch is released, I would like to request a CVE ID for this vulnerability, in accordance with the usual procedure for security issues.

How to reproduce:

$unzip FFmpeg-master.zip
$cd FFmpeg-master-dev
$./configure --enable-libx264 --enable-gpl --cc=clang --cxx=clang++ --toolchain=clang-asan --enable-cross-compile

$make -j 30
$./ffmpeg -i poc test.mp4

poc

crash log:

=================================================================
==1590113==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 136 byte(s) in 1 object(s) allocated from:

#0 0x5e7e1e17e67e in interceptor_malloc (/home/swift/workstation/github/FFmpeg-master/ffmpeg+0x98067e) (BuildId: 2e3adc0cc0a1ce7cb06b6662338d6fd012a2a61b)
#1 0x731353c0c3d1 (/usr/lib/x86_64-linux-gnu/libx264.so.163+0xc3d1) (BuildId: b2e7160a74448d05d96f3305d73fc2316edbedaf)

Indirect leak of 30 byte(s) in 2 object(s) allocated from:

#0 0x5e7e1e169083 in interceptor_strdup (/home/swift/workstation/github/FFmpeg-master/ffmpeg+0x96b083) (BuildId: 2e3adc0cc0a1ce7cb06b6662338d6fd012a2a61b)
#1 0x731353c0c353 (/usr/lib/x86_64-linux-gnu/libx264.so.163+0xc353) (BuildId: b2e7160a74448d05d96f3305d73fc2316edbedaf)

SUMMARY: AddressSanitizer: 166 byte(s) leaked in 3 allocation(s).

ffmpeg version

ffmpeg version 7.1.git Copyright (c) 2000-2024 the FFmpeg developers
built with Ubuntu clang version 14.0.0-1ubuntu1.1
configuration: --enable-libx264 --cc=clang --cxx=clang++ --enable-gpl --toolchain=clang-asan
libavutil 59. 51.100 / 59. 51.100
libavcodec 61. 27.101 / 61. 27.101
libavformat 61. 9.101 / 61. 9.101
libavdevice 61. 4.100 / 61. 4.100
libavfilter 10. 6.101 / 10. 6.101
libswscale 8. 12.100 / 8. 12.100
libswresample 5. 4.100 / 5. 4.100
libpostproc 58. 4.100 / 58. 4.100

ubuntu version

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.3 LTS
Release: 22.04
Codename: jammy
$ uname -r
6.8.0-49-generic

libx264 version

$ dpkg -s libx264-dev
Package: libx264-dev
Status: install ok installed
Priority: optional
Section: libdevel
Installed-Size: 2761
Maintainer: Ubuntu Developers <​ubuntu-devel-discuss@lists.ubuntu.com>
Architecture: amd64
Multi-Arch: same
Source: x264
Version: 2:0.163.3060+git5db6aa6-2build1
Depends: libx264-163 (= 2:0.163.3060+git5db6aa6-2build1)
Description: development files for libx264

libx264 is an advanced encoding library for creating H.264 (MPEG-4 AVC)
video streams.
.
This package contains the static library and headers used to build programs
that use libx264.

Original-Maintainer: Debian Multimedia Maintainers <​debian-multimedia@lists.debian.org>
Homepage: ​https://www.videolan.org/developers/x264.html

Thanks!