As proposed in !4137 (comment 407083)
It returns ENOMEM and clean the format. I've added failure everywhere the only exported functions are
int SetupVideoES( demux_t *p_demux, const mp4_track_t *p_track,
const MP4_Box_t *p_sample, es_format_t *, track_config_t *);
int SetupAudioES( demux_t *p_demux, const mp4_track_t *p_track,
const MP4_Box_t *p_sample, es_format_t *, track_config_t * );
int SetupSpuES( demux_t *p_demux, const mp4_track_t *p_track,
const MP4_Box_t *p_sample, es_format_t *, track_config_t * );They are used only in TrackFillConfig() which will fail if they return 0
It is used in TrackCreateES() that will fail if TrackFillConfig fails thus not putting the track b_ok.
And TrackUpdateFormat() that will now fail if TrackFillConfig fails too but that will require François' validation.
VideoLAN code repository instance