This allows using more specific implementations for chroma/luma, e.g. we can make assumptions on filterSize being constant, thus avoiding that test at runtime.