Playback issue with MP4 file contains CBR opus track

VLC fails to open a MP4 file that contains only a opus (CBR encoded) track, with the following error messages:

opus error: Error: corrupted stream?
main error: buffer deadlock prevented

When a MP4 file also contains video track, VLC opens it and starts playback, but audio stutters badly, with these warnings:

main warning: timing screwed (drift: -100020 us): stopping resampling
main warning: playback too early (-119375): down-sampling
main warning: playback way too early (-139979): playing silence
main debug: inserting 6718 zeroes
main warning: playback too early (-59521): down-sampling
main warning: timing screwed (drift: -119687 us): stopping resampling
main warning: playback too early (-119731): down-sampling
main warning: playback way too early (-140126): playing silence
main debug: inserting 6726 zeroes

Opus track was encoded by opusenc with --hard-cbr option specified. Remux from opus to mp4 is done by ffmpeg (-c:a copy specified).