Add common code to handle mask to/from chroma

These are various code changes needed to handle the new RGB chromas without masks. These changes are used in almost all Merge Requests dealing with RGB chroma+mask.

It's better to resolve issues like !4201 (comment 410290) here rather than in each MR.

MR will be a draft on top of this one.

All the masks are put in vlc_es.h but we could hide this from most of the code and export vlc_RGBChromaToMask/vlc_RGBMaskToChroma from the core. Then we could use a static table to do the mapping faster.

---

My original comment:

For a 24-bit mask it's tricky to explain the "native endianness", since there is no uint24_t, but for uint32_t or uint16_t it's clearer.

A mask is basically used with the & operator. But in this case the value is a RGB color that is described in memory order. So when you have VLC_COCEC_BGRX:

uint32_t red_mask = 0x0000ff00;
uint32_t *memory_order_rgb = XXX;
uint32_t red = *memory_order_rgb & red_mask;

On Big-Endian you get only the (shifted) red value. But on Little-Endian you get the (shifted) green value. As you mention below, we could use hton32() so the mask can always be used as-is. But historically that's not what we have used. I might rename this to "host" endianness and/or mention to use the mask with hton32().

But in reality, we never have to use hton32() in any places we need a mask. We certainly didn't use it so far. We sometimes read the mask a Little-Endian (from AVI) but that gives a uint32_t value in native/host endianness.

---

Also the hton32() wouldn't work for 24-bit masks. 0x0000ff (blue in RGB) would end up being 0xff000000 which wouldn't make much sense when only 3 bytes are used.