avcodec: prioritize RGB with matching mask
The VLC_RGB
macro maps the RGB with mask to/from VLC formats. But it also maps one of the FFmpeg variants with no mask at all. That means it might be picked even though there's another format with exactly the proper mask.