Expose libplacebo upscalers in OpenGL vout

This is the last patchset to enable libplacebo upscalers (and downscalers) from the OpenGL vout (it relies on the new OpenGL filters API from !801 (merged)).

There are 2 parts in this patchset:

1. add support for output resize in the OpenGL filters API
2. implement a pl_scale filter, and enable it on demand in the OpenGL vout

Context

First, here is some context to understand the next section.

As you know, there is a specific API for OpenGL filters:

• an OpenGL filter implements vlc_gl_filter;
• it can be run in an OpenGL filter chain (vlc_gl_filters).

In practice, this "OpenGL filter chain" is used in 2 different places:

• in the OpenGL vout (creation, draw), to run the renderer filter;
• in the opengl video filter (filter_t), to allow executing OpenGL filters as VLC video filters (this is for example what the OpenGL blend deinterlace filter does).

Resize handling

An upscaler must render according to the output size: when the window is resized, we want to re-apply the filter with the new target size.

This is not supported by the video filter (filter_t) architecture:

• a video filter produces an output according to the input size, not the output size;
• the filter chain output format must match the filter chain input format.

If the output format of the video filter chain is different from the input format, a converter is automatically inserted by the core. In practice, a swscale filter is added to scale back to the initial size, defeating the purpose of an upscaler.

It is potentially a lot of work to change how video filters (filter_t) work, so I decided to apply the upscaler/downscaler filter in the OpenGL vout directly. In other words, the goal of this MR is to implement an OpenGL video filter (vlc_gl_filter) intended to be run from the OpenGL vout (before the renderer, which manages features like aspect-ratio, zoom…).

However, the OpenGL filter API does not handle output resizing either. So the first step consists in adding an API to handle it.

Although renderer is just one particular OpenGL filter (it "inherits" vlc_gl_filter), it is also used via its concrete type, to configure (among others) the window aspect ratio.

The idea here is to make the mechanism more generic, and expose a new vlc_gl_filter operation request_output_size() (the renderer can still compute the aspect ratio from the output size).

In an OpenGL filter chain, let's call a "responsive" filter a filter intended to render according to its output size (for example, renderer is a responsive filter). Such filters implement the request_output_size() callback.

If there are several responsive filters in the filter chain (for example, pl_scale and renderer), it remains to decide which filter should handle the resizing. Let's add the constraint that all "responsive" filters are at the end of the chain (we don't need more complex cases for now).

Concretely, what we want is the following:

• if there is only the renderer, then it should handle the resizing;
• if there are pl_scale and renderer, then pl_scale should handle the resizing.

Here is the "generic" resize mechanism this MR implements to achieve this goal without hardcoding the desired behavior (which would require to use the concrete types of renderer and pl_scale directly).

On vout resize, the last filter (if its request_output_size is not NULL) receives the requested output size (vlc_gl_tex_size). In this callback, it is allowed to require an "optimal input size". If it does, then request_output_size() (if not NULL) is called on the previous filter with that optimal size. And so on. A filter can reject the requested output size (on error for example, or it may even provide a new size, but we don't need it). The actual change is notified to the next filter by on_input_size_change() (but that does not trigger another output size change).

During this resize handling, the OpenGL filter engine also recreates, for each filter, the internal renderbuffer and texture storage at the new size.

Concretly, here is what happens in the first scenario, where there is only the renderer:

• the renderer receives the vout output size via request_output_size()
• it adapts its state to properly handle the window aspect ratio
• it requests its optimal input size, i.e. the content output size (which is not the same as the vout output size, since it takes aspect ratio into account, there might be black borders)
• there are no other filters, so its input size has not changed and it will handle the (linear) resize itself

And here is what happens in the second scenario, where there is also pl_scale before renderer:

• the renderer receives the vout output size via request_output_size()
• it adapts its state to properly handle the window aspect ratio
• it requests its optimal input size
• pl_scale receives the optimal size via its request_output_size()
• it saves the requested size internally to use it for the next draw calls
• it does not request an optimal input size, so resizing is not propagated to previous filters (there are none anyway)
• pl_scale will now output pictures at the optimal size requested by the renderer (so in practice it will take care of the resizing)
• the renderer could be notified of the actual pl_scale output via on_input_size_change() (but it does not care)

Libplacebo scale filter

Since !801 (merged), an OpenGL filter receives the input textures (using a sampler to access them via the GLSL function vlc_texture() is now optional).

This is exactly what we need for an OpenGL filter to delegate the work to libplacebo: we don't create a sampler at all. Instead, the filter wraps the input textures and output framebuffer into libplacebo types, converts format data (flip, crop…) to libplacebo formats, then requests libplacebo to render to the framebuffer using the specified upscaler and downscaler.

Unfortunately, this filter requires a version of libplacebo very very recent (master branch from 20 days ago). With previous versions, libplacebo (as used by this filter) would make VLC assert/crash by closing random file descriptors (fix). If the version is < 167, the filter returns VLC_EGENERIC on Open().

Results

The filter may be enabled by --pl-upscaler=NUMBER or --pl-downscaler=NUMBER (similar to what the libplacebo vout display does):

./vlc --pl-upscaler=6 video.mp4

Note that this is expected to already work without this MR with the libplacebo vout:

./vlc -Vlibplacebo --pl-upscaler-preset=6 video.mp4

(I'm not sure whether I should use --pl-upscaler for both or --pl-upscaler-preset for both.)

Here is the matching number/filter.

It can also be configured in the settings:

Here is a 426×240 video to test:

sample_to_upscale

Its original size:

Rendered to 2560×1440 without --pl-upscaler (please zoom):

Upscaled to 2560×1440 with --pl-upscaler=6 (please zoom):

Remaining task

• support 90°-rotated video (the support has been added recently in libplacebo) done

Refs #26066 (closed)