DXGI1.6 colorspace selection has unexpected result

I first filed this as a forum topic and on investigation I believe I have narrowed it down to a particular bug. I don't know how to do actual debugging on the code unfortunately, so I'm basing this report on some inferences about it, specifically differences in behavior seen between VLC 3.0.16 and yesterday's VLC4 nightly.

Here's the relevant output from the VLC 3.0.16 win64 release build:

direct3d11 debug: Created the D3D11 device type 1 level b100.
direct3d11 debug: NVIDIA WDDM driver 30.0.14.7168
direct3d11 debug: supports colorspace RGB Rec.709 gamma:22 range:FULL
direct3d11 debug: supports colorspace RGB Rec.709 gamma:22 range:STUDIO
direct3d11 debug: supports colorspace RGB Rec.2020 gamma:22 range:STUDIO
direct3d11 debug: supports colorspace RGB Rec.2020 gamma:22 range:FULL
direct3d11 debug: supports colorspace RGB Rec.2020 gamma:2084 range:FULL
direct3d11 debug: supports colorspace RGB Rec.2020 gamma:2084 range:STUDIO
direct3d11 debug: Output max luminance: 400.0, colorspace RGB Rec.709 gamma:22 range:FULL, bits per pixel 8
direct3d11 debug: using colorspace RGB Rec.709 gamma:22 range:FULL
direct3d11 debug: Using pixel format VA_P010 for chroma DX10

When viewing the video rendered in RGB Rec.709 gamma:22 range:FULL it is clear to me that that colorspace selection is wrong. The colors are very desaturated, which is obvious even in a screencap:

The right hand half of that frame is from VLC 4 (nightly-win64/20210915-0421), which gives this log output:

direct3d11 debug: Using pixel format VA_P010 for chroma DX10
direct3d11 debug: UpdateRects source offset: 0,0 visible: 3840x1600 decoded: 3840x1664
direct3d11 debug: UpdateRects image_dst coords: 0,0 3840x1600
direct3d11 debug: supports colorspace RGB Rec.709 gamma:22 range:FULL
direct3d11 debug: supports colorspace RGB Rec.709 gamma:22 range:STUDIO
direct3d11 debug: supports colorspace RGB Rec.2020 gamma:22 range:STUDIO
direct3d11 debug: supports colorspace RGB Rec.2020 gamma:22 range:FULL
direct3d11 debug: supports colorspace RGB Rec.2020 gamma:2084 range:FULL
direct3d11 debug: supports colorspace RGB Rec.2020 gamma:2084 range:STUDIO
direct3d11 debug: using colorspace RGB Rec.2020 gamma:2084 range:FULL

The interesting thing here is that the "Output max luminance" line is missing, which indicates that VLC 4 is not running the DXGI6 colorspace selection algorithm, and is instead falling back to its own basic colorspace selection algorithm, which selects what I believe to be the correct colorspace: RGB Rec.2020 gamma:2084 range:FULL.

I see two potential causes of this discrepancy:

• HAVE_DXGI1_6_H is unset on the nightly build server, but was set for the release build.
• One of the SUCEEDED() checks failed.

I find the first more likely as the calls subject to SUCEEDED checks are more or less the same between versions, and my system is the same each time.

What I don't know is why the DXGI colorspace selection code is the way it is. It appears to me that it has an obvious flaw. It goes through the colorspaces in the same order that they are present in the debugging output and selects the first colorspace that it is possible to convert to. It is possible to covert to RGB Rec.709 gamma:22 range:FULL, so we do. To my eye that indicates that some incorrect implicit assumptions were made about how the items in this list would be ordered. In particular that assumption appears to have been introduced in this commit.