Custom Shaders (mpv .hook syntax)

libplacebo supports the same custom shader syntax used by mpv, with some important changes. This document will serve as a complete reference for this syntax.

Overview

In general, user shaders are divided into distinct blocks. Each block can define a shader, a texture, a buffer, or a tunable parameter. Each block starts with a collection of header directives, which are lines starting with the syntax //!.

As an example, here is a simple shader that simply inverts the video signal:

//!HOOK LUMA
//!HOOK RGB
//!BIND HOOKED

vec4 hook()
{
    vec4 color = HOOKED_texOff(0);
    color.rgb = vec3(1.0) - color.rgb;
    return color;
}

This shader defines one block - a shader block which hooks into the two texture stages LUMA and RGB, binds the hooked texture, inverts the value of the rgb channels, and then returns the modified color.

Expressions

In a few contexts, shader directives accept arithmetic expressions, denoted by <expr> in the listing below. For historical reasons, all expressions are given in reverse polish notation (RPN), and the only value type is a floating point number. The following value types and arithmetic operations are available:

• 1.234: Literal float constant, evaluates to itself.
• NAME.w, NAME.width: Evaluates to the width of a texture with name NAME.
• NAME.h, NAME.height: Evaluates to the height of a texture with name NAME.
• PAR: Evaluates to the value of a tunable shader parameter with name PAR.
• +: Evaluates to X+Y.
• -: Evaluates to X-Y.
• *: Evaluates to X*Y.
• /: Evaluates to X/Y.
• %: Evaluates to fmod(X, Y).
• >: Evaluates to (X > Y) ? 1.0 : 0.0.
• <: Evaluates to (X < Y) ? 1.0 : 0.0.
• =: Evaluates to fuzzy_eq(X, Y) ? 1.0 : 0.0, with some tolerance to allow for floating point inaccuracy. (Around 1 ppm)
• !: Evaluates to X ? 0.0 : 1.0.

Note that + and * can be used as suitable replacements for the otherwise absent boolean logic expressions (|| and &&).

Shaders

Shaders are the default block type, and have no special syntax to indicate their presence. Shader stages contain raw GLSL code that will be (conditionally) executed. This GLSL snippet must define a single function vec4 hook(), or void hook() for compute shaders.

During the execution of any shader, the following global variables are made available:

• int frame: A raw counter tracking the number of executions of this shader stage.
• float random: A pseudo-random float uniformly distributed in the range [0,1).
• vec2 input_size: The nominal size (in pixels) of the original input image.
• vec2 target_size: The nominal size (in pixels) of the output rectangle.
• vec2 tex_offset: The nominal offset (in pixels), of the original input crop.
• vec4 linearize(vec4 color): Linearize the input color according to the image's tagged gamma function.
• vec4 delinearize(vec4 color): Opposite counterpart to linearize.

Shader stages accept the following directives:

HOOK <texture>

A HOOK directive determines when a shader stage is run. During internal processing, libplacebo goes over a number of pre-defined hook points at set points in the processing pipeline. It is only possible to intercept the image, and run custom shaders, at these fixed hook points.

Here is a current list of hook points:

• RGB: Input plane containing RGB values
• LUMA: Input plane containing a Y value
• CHROMA: Input plane containing chroma values (one or both)
• ALPHA: Input plane containing a single alpha value
• XYZ: Input plane containing XYZ values
• CHROMA_SCALED: Chroma plane, after merging and upscaling to luma size
• ALPHA_SCALED: Alpha plane, after upscaling to luma size
• NATIVE: Merged input planes, before any sort of color conversion (as-is)
• MAIN: After conversion to RGB, before linearization/scaling
• LINEAR: After conversion to linear light (for scaling purposes)
• SIGMOID: After conversion to sigmoidized light (for scaling purposes)
• PREKERNEL: Immediately before the execution of the main scaler kernel
• POSTKERNEL: Immediately after the execution of the main scaler kernel
• SCALED: After scaling, in either linear or non-linear light RGB
• PREOUTPUT: After color conversion to target colorspace, before alpha blending
• OUTPUT: After alpha blending, before dithering and final output pass

MAINPRESUB

In mpv, MAIN and MAINPRESUB are separate shader stages, because the mpv option --blend-subtitles=video allows rendering overlays directly onto the pre-scaled video stage. libplacebo does not support this feature, and as such, the MAINPRESUB shader stage does not exist. It is still valid to refer to this name in shaders, but it is handled identically to MAIN.

It's possible for a hook point to never fire. For example, SIGMOID will not fire when downscaling, as sigmoidization only happens when upscaling. Similarly, LUMA/CHROMA will not fire on an RGB video and vice versa.

A single shader stage may hook multiple hook points simultaneously, for example, to cover both LUMA and RGB cases with the same logic. (See the example shader in the introduction)

BIND <texture>

The BIND directive makes a texture available for use in the shader. This can be any of the previously named hook points, a custom texture define by a TEXTURE block, a custom texture saved by a SAVE directive, or the special value HOOKED which allows binding whatever texture hook dispatched this shader stage.

A bound texture will define the following GLSL functions (as macros):

• sampler2D NAME_raw: A reference to the raw texture sampler itself.
• vec2 NAME_pos: The texel coordinates of the current pixel.
• vec2 NAME_map(ivec2 id): A function that maps from gl_GlobalInvocationID to texel coordinates. (Compute shaders)
• vec2 NAME_size: The size (in pixels) of the texture.
• vec2 NAME_pt: Convenience macro for 1.0 / NAME_size. The size of a single pixel (in texel coordinates).
• vec2 NAME_off: The sample offset of the texture. Basically, the pixel coordinates of the top-left corner of the sampled area.
• float NAME_mul: The coefficient that must be multiplied into sampled values in order to rescale them to [0,1].
• vec4 NAME_tex(vec2 pos): A wrapper around NAME_mul * textureLod(NAME_raw, pos, 0.0).
• vec4 NAME_texOff(vec2 offset): A wrapper around NAME_tex(NAME_pos + NAME_pt * offset). This can be used to easily access adjacent pixels, e.g. NAME_texOff(-1,2) samples a pixel one to the left and two to the bottom of the current location.
• vec4 NAME_gather(vec2 pos, int c): A wrapper around NAME_mul * textureGather(pos, c), with appropriate scaling. (Only when supported¹)

Rotation matrix

For compatibility with mpv, we also define a mat2 NAME_rot which is simply equal to a 2x2 identity matrix. libplacebo never rotates input planes - all rotation happens during the final output to the display.

This same directive can also be used to bind buffer blocks (i.e. uniform/storage buffers), as defined by the BUFFER directive.

SAVE <texture>

By default, after execution of a shader stage, the resulting output is captured back into the same hooked texture that triggered the shader. This behavior can be overridden using the explicit SAVE directive. For example, a shader might need access to a low-res version of the luma input texture in order to process chroma:

//!HOOK CHROMA
//!BIND CHROMA
//!BIND LUMA
//!SAVE LUMA_LOWRES
//!WIDTH CHROMA.w
//!HEIGHT CHROMA.h

vec4 hook()
{
    return LUMA_texOff(0);
}

This shader binds both luma and chroma and resizes the luma plane down to the size of the chroma plane, saving the result as a new texture LUMA_LOWRES. In general, you can pick any name you want, here.

DESC <description>

This purely informative directive simply gives the shader stage a name. This is the name that will be reported to the shader stage and execution time metrics.

OFFSET <xo yo | ALIGN>

This directive indicates a pixel shift (offset) introduced by this pass. These pixel offsets will be accumulated and corrected automatically as part of plane alignment / main scaling.

A special value of ALIGN will attempt to counteract any existing offset of the hooked texture by aligning it with reference plane (i.e. luma). This can be used to e.g. introduce custom chroma scaling in a way that doesn't break chroma subtexel offsets.

An example:

//!HOOK LUMA
//!BIND HOOKED
//!OFFSET 100.5 100.5

vec4 hook()
{
    // Constant offset by N pixels towards the bottom right
    return HOOKED_texOff(-vec2(100.5));
}

This (slightly silly) shader simply shifts the entire sampled region to the bottom right by 100.5 pixels, and propagates this shift to the main scaler using the OFFSET directive. As such, the end result of this is that there is no visible shift of the overall image, but some detail (~100 pixels) near the bottom-right border is lost due to falling outside the bounds of the texture.

WIDTH <expr>, HEIGHT <expr>

These directives can be used to override the dimensions of the resulting texture. Note that not all textures can be resized this way. Currently, only RGB, LUMA, CHROMA, XYZ, NATIVE and MAIN are resizable. Trying to save a texture with an incompatible size to any other shader stage will result in an error.

WHEN <expr>

This directive takes an expression that can be used to make shader stages conditionally executed. If this evaluates to 0, the shader stage will be skipped.

Example:

//!PARAM strength
//!TYPE float
//!MINIMUM 0
1.0

//!HOOK MAIN
//!BIND HOOKED
//!WHEN intensity 0 >
//!DESC do something based on 'intensity'
...

This example defines a shader stage that only conditionally executes itself if the value of the intensity shader parameter is non-zero.

COMPONENTS <num>

This directive overrides the number of components present in a texture. For example, if you want to extract a one-dimensional feature map from the otherwise 3 or 4 dimensional MAIN texture, you can use this directive to save on memory bandwidth and consumption by having libplacebo only allocate a one-component texture to store the feature map in:

//!HOOK MAIN
//!BIND HOOKED
//!SAVE featuremap
//!COMPONENTS 1

COMPUTE <bw> <bh> [<tw> <th>]

This directive specifies that the shader should be treated as a compute shader, with the block size bw and bh. The compute shader will be dispatched with however many blocks are necessary to completely tile over the output. Within each block, there will be tw*th threads, forming a single work group. In other words: tw and th specify the work group size, which can be different from the block size. So for example, a compute shader with bw = bh = 32 and tw = th = 8 running on a 500x500 texture would dispatch 16x16 blocks (rounded up), each with 8x8 threads.

Instead of defining a vec4 hook(), compute shaders must define a void hook() which results directly to the output texture, a writeonly image2D out_image made available to the shader stage.

For example, here is a shader executing a single-pass 41x41 convolution (average blur) on the luma plane, using a compute shader to share sampling work between adjacent threads in a work group:

//!HOOK LUMA
//!BIND HOOKED
//!COMPUTE 32 32
//!DESC avg convolution

// Kernel size, 41x41 as an example
const ivec2 ksize = ivec2(41, 41);
const ivec2 offset = ksize / 2;

// We need to load extra source texels to account for padding due to kernel
// overhang
const ivec2 isize = ivec2(gl_WorkGroupSize) + ksize - 1;

shared float inp[isize.y][isize.x];

void hook()
{
    // load texels into shmem
    ivec2 base = ivec2(gl_WorkGroupID) * ivec2(gl_WorkGroupSize);
    for (uint y = gl_LocalInvocationID.y; y < isize.y; y += gl_WorkGroupSize.y) {
        for (uint x = gl_LocalInvocationID.x; x < isize.x; x += gl_WorkGroupSize.x)
            inp[y][x] = texelFetch(HOOKED_raw, base + ivec2(x,y) - offset, 0).x;
    }

    // synchronize threads
    barrier();

    // do convolution
    float sum;
    for (uint y = 0; y < ksize.y; y++) {
        for (uint x = 0; x < ksize.x; x++)
            sum += inp[gl_LocalInvocationID.y+y][gl_LocalInvocationID.x+x];
    }

    vec4 color = vec4(HOOKED_mul * sum / (ksize.x * ksize.y), 0, 0, 1);
    imageStore(out_image, ivec2(gl_GlobalInvocationID), color);
}

Textures

Custom textures can be defined and made available to shader stages using TEXTURE blocks. These can be used to provide e.g. LUTs or pre-trained weights.

The data for a texture is provided as a raw hexadecimal string encoding the in-memory representation of a texture, according to its given texture format, for example:

//!TEXTURE COLORS
//!SIZE 3 3
//!FORMAT rgba32f
//!FILTER NEAREST
//!BORDER REPEAT
0000803f000000000000000000000000000000000000803f00000000000000000000000
0000000000000803f00000000000000000000803f0000803f000000000000803f000000
000000803f000000000000803f0000803f00000000000000009a99993e9a99993e9a999
93e000000009a99193F9A99193f9a99193f000000000000803f0000803f0000803f0000
0000

Texture blocks accept the following directives:

TEXTURE <name>

This must be the first directive in a texture block, and marks it as such. The name given is the name that the texture will be referred to (via BIND directives).

SIZE <width> [<height> [<depth>]]

This directive gives the size of the texture, as integers. For example, //!SIZE 512 512 marks a 512x512 texture block. Textures can be 1D, 2D or 3D depending on the number of coordinates specified.

FORMAT <fmt>

This directive specifies the texture format. A complete list of known textures is exposed as part of the pl_gpu struct metadata, but they follow the format convention rgba8, rg16hf, rgba32f, r64i and so on.

FILTER <LINEAR | NEAREST>

This directive specifies the texture magnification/minification filter.

BORDER <CLAMP | REPEAT | MIRROR>

This directive specifies the border clamping method of the texture.

STORAGE

If present, this directive marks the texture as a storage image. It will still be initialized with the initial values, but rather than being bound as a read-only and immutable sampler2D, it is bound as a readwrite coherent image2D. Such texture scan be used to, for example, store persistent state across invocations of the shader.

Buffers

Custom uniform / storage shader buffer blocks can be defined using BUFFER directives.

The (initial) data for a buffer is provided as a raw hexadecimal string encoding the in-memory representation of a buffer in the corresponding GLSL packing layout (std140 or std430 for uniform and storage blocks, respectively):

//!BUFFER buf_uniform
//!VAR float foo
//!VAR float bar
0000000000000000

//!BUFFER buf_storage
//!VAR vec2 bat
//!VAR int big[32];
//!STORAGE

Buffer blocks accept the following directives:

BUFFER <name>

This must be the first directive in a buffer block, and marks it as such. The name given is mostly cosmetic, as individual variables can be accessed directly using the names given in the corresponding VAR directives.

STORAGE

If present, this directive marks the buffer as a (readwrite coherent) shader storage block, instead of a readonly uniform buffer block. Such storage blocks can be used to track and evolve state across invocations of this shader.

Storage blocks may also be initialized with default data, but this is optional. They can also be initialized as part of the first shader execution (e.g. by testing for frame == 0).

VAR <type> <name>

This directive appends a new variable to the shader block, with GLSL type <type> and shader name <name>. For example, VAR float foo introduces a float foo; member into the buffer block, and VAR mat4 transform introduces a mat4 transform; member.

It is also possible to introduce array variables, using [N] as part of the variable name.

Tunable parameters

Finally, the PARAM directive allows introducing tunable shader parameters, which are exposed programmatically as part of the C API (pl_hook).²

The default value of a parameter is given as the block body, for example:

//!PARAM contrast
//!DESC Gain to apply to image brightness
//!TYPE float
//!MINIMUM 0.0
//!MAXIMUM 100.0
1.0

Parameters accept the following directives:

PARAM <name>

This must be the first directive in a parameter block, and marks it as such. The name given is the name that will be used to refer to this parameter in GLSL code.

DESC <description>

This directive can be used to provide a friendlier description of the shader parameter, exposed as part of the C API to end users.

MINIMUM <value>, MAXIMUM <value>

Provides the minimum/maximum value bound of this parameter. If absent, no minimum/maximum is enforced.

TYPE [ENUM] <DEFINE | [DYNAMIC | CONSTANT] <type>>

This gives the type of the parameter, which determines what type of values it can hold and how it will be made available to the shader. <type> must be a scalar GLSL numeric type, such as int, float or uint.

If a type is ENUM, it is treated as an enumeration type. To use this, type must either be int or DEFINE. Instead of providing a single default value, the param body should be a list of all possible enumeration values (as separate lines). These names will be made available inside the shader body (as a #define), as well as inside RPN expressions (e.g. WHEN). The qualifiers MINIMUM and MAXIMUM are ignored for ENUM parameters, with the value range instead being set implicitly from the list of options.

The optional qualifiers DYNAMIC or CONSTANT mark the parameter as dynamically changing and compile-time constant, respectively. A DYNAMIC variable is assumed to change frequently, and will be grouped with other frequently-changing input parameters. A CONSTANT parameter will be introduced as a compile-time constant into the shader header, which means thy can be used in e.g. constant expressions such as array sizes.³

Finally, the special type TYPE DEFINE marks a variable as a preprocessor define, which can be used inside #if preprocessor expressions. For example:

//!PARAM taps
//!DESC Smoothing taps
//!TYPE DEFINE
//!MINIMUM 0
//!MAXIMUM 5
2

//!HOOK LUMA
//!BIND HOOKED
const uint row_size = 2 * taps + 1;
const float weights[row_size] = {
#if taps == 0
    1.0,
#endif

#if taps == 1
    0.10650697891920,
    0.78698604216159,
    0.10650697891920,
#endif

#if taps == 2
    0.05448868454964,
    0.24420134200323,
    0.40261994689424,
    0.24420134200323,
    0.05448868454964,
#endif

    // ...
};

An example of an enum parameter:

//!PARAM csp
//!DESC Colorspace
//!TYPE ENUM int
BT709
BT2020
DCIP3

//!HOOK MAIN
//!BIND HOOKED
const mat3 matrices[3] = {
    mat3(...), // BT709
    mat3(...), // BT2020
    mat3(...), // DCIP3
};

#define MAT matrices[csp]
// ...

Full example

A collection of full examples can be found in the mpv user shaders wiki, but here is an example of a parametrized Gaussian smoothed film grain compute shader:

//!PARAM intensity
//!DESC Film grain intensity
//!TYPE float
//!MINIMUM 0
0.1

//!PARAM taps
//!DESC Film grain smoothing taps
//!TYPE DEFINE
//!MINIMUM 0
//!MAXIMUM 5
2

//!HOOK LUMA
//!BIND HOOKED
//!DESC Apply gaussian smoothed film grain
//!WHEN intensity 0 >
//!COMPUTE 32 32

const uint row_size = 2 * taps + 1;
const float weights[row_size] = {
#if taps == 0
    1.0,
#endif

#if taps == 1
    0.10650697891920,
    0.78698604216159,
    0.10650697891920,
#endif

#if taps == 2
    0.05448868454964,
    0.24420134200323,
    0.40261994689424,
    0.24420134200323,
    0.05448868454964,
#endif

#if taps == 3
    0.03663284536919,
    0.11128075847888,
    0.21674532140370,
    0.27068214949642,
    0.21674532140370,
    0.11128075847888,
    0.03663284536919,
#endif

#if taps == 4
    0.02763055063889,
    0.06628224528636,
    0.12383153680577,
    0.18017382291138,
    0.20416368871516,
    0.18017382291138,
    0.12383153680577,
    0.06628224528636,
    0.02763055063889,
#endif

#if taps == 5
    0.02219054849244,
    0.04558899978527,
    0.07981140824009,
    0.11906462996609,
    0.15136080967773,
    0.16396720767670,
    0.15136080967773,
    0.11906462996609,
    0.07981140824009,
    0.04558899978527,
    0.02219054849244,
#endif
};

const uvec2 isize = uvec2(gl_WorkGroupSize) + uvec2(2 * taps);
shared float grain[isize.y][isize.x];

// PRNG
float permute(float x)
{
    x = (34.0 * x + 1.0) * x;
    return fract(x * 1.0/289.0) * 289.0;
}

float seed(uvec2 pos)
{
    const float phi = 1.61803398874989;
    vec3 m = vec3(fract(phi * vec2(pos)), random) + vec3(1.0);
    return permute(permute(m.x) + m.y) + m.z;
}

float rand(inout float state)
{
    state = permute(state);
    return fract(state * 1.0/41.0);
}

// Turns uniform white noise into gaussian white noise by passing it
// through an approximation of the gaussian quantile function
float rand_gaussian(inout float state) {
    const float a0 = 0.151015505647689;
    const float a1 = -0.5303572634357367;
    const float a2 = 1.365020122861334;
    const float b0 = 0.132089632343748;
    const float b1 = -0.7607324991323768;

    float p = 0.95 * rand(state) + 0.025;
    float q = p - 0.5;
    float r = q * q;

    float g = q * (a2 + (a1 * r + a0) / (r*r + b1*r + b0));
    g *= 0.255121822830526; // normalize to [-1,1)
    return g;
}

void hook()
{
    // generate grain in `grain`
    uint num_threads = gl_WorkGroupSize.x * gl_WorkGroupSize.y;
    for (uint i = gl_LocalInvocationIndex; i < isize.y * isize.x; i += num_threads) {
        uvec2 pos = uvec2(i % isize.y, i / isize.y);
        float state = seed(gl_WorkGroupID.xy * gl_WorkGroupSize.xy + pos);
        grain[pos.y][pos.x] = rand_gaussian(state);
    }

    // make writes visible
    barrier();

    // convolve horizontally
    for (uint y = gl_LocalInvocationID.y; y < isize.y; y += gl_WorkGroupSize.y) {
        float hsum = 0;
        for (uint x = 0; x < row_size; x++) {
            float g = grain[y][gl_LocalInvocationID.x + x];
            hsum += weights[x] * g;
        }

        // update grain LUT
        grain[y][gl_LocalInvocationID.x + taps] = hsum;
    }

    barrier();

    // convolve vertically
    float vsum = 0.0;
    for (uint y = 0; y < row_size; y++) {
        float g = grain[gl_LocalInvocationID.y + y][gl_LocalInvocationID.x + taps];
        vsum += weights[y] * g;
    }

    vec4 color = HOOKED_tex(HOOKED_pos);
    color.rgb += vec3(intensity * vsum);
    imageStore(out_image, ivec2(gl_GlobalInvocationID), color);
}

---

1. Because these are macros, their presence can be tested for using #ifdef inside the GLSL preprocessor. ↩

2. In mpv using --vo=gpu-next, these can be set using the --glsl-shader-opts option. ↩

3. On supported platforms, these are implemented using specialization constants, which can be updated at run-time without requiring a full shader recompilation. ↩