2D Graphics on Modern GPU (2019)

> someone's 2020 fork

pcwalton was the developer behind pathfinder at Mozilla (but was part of past summer's layoff).

> instead just blits glyphs from an atlas texture rendered with Freetype on the CPU

Correct.

> has high enough path rendering quality (and performance) to render glyphs purely on the GPU

Do you have screenshots showing quality, and performance measures showing speed? Ideally from Raspberry Pi 4?

> This makes it clear you didn't even perform a cursory investigation of the project

I did, and mentioned in the docs, here’s a quote: “I didn’t want to experiment with GPU-based splines. AFAIK the research is not there just yet.” Verifiable because version control: https://github.com/Const-me/Vrmac/blob/bbe83b9722dcb080f1aed...

For text, I think bitmaps are better than splines. I can see how splines are cool from a naïve programmer’s perspective, but practically speaking they are not good enough for the job.

Vector fonts are not resolution independent because hinting. Fonts include a bytecode of compiled programs who do that. GPUs are massively parallel vector chips, not a good fit to interpret byte code of a traditional programming language. This means whatever splines you gonna upload to GPU will only contain a single size of the font, trying to reuse for different resolution will cause artifacts.

Glyphs are small and contain lots of curves. Lots of data to store, and lots of math to render, for comparatively small count of output pixels. Copying bitmaps is very fast, modern GPUs, even low-power mobile and embedded ones, are designed to output ridiculous volume of textured triangles per second. Font face and size are more or less consistent within a given document/page/screen. Apart from synthetic tests, glyphs are reused a lot, and there’re not too many of them.

When I started the project, the very first support of compute shaders on Pi 4 was just introduced in the Mesa upstream repo. Was not yet in the official OS images. Bugs are very likely in versions 1.0 of anything at all.

Finally, even if Pi 4 had awesome support for compute shaders back them, the raw compute power of the GPU is not that impressive. Here in my Windows PC, my GPU is 30 times faster than CPU in terms of raw FP32 performance. With that kind of performance gap, you can probably make GPU splines work fast enough, after spending enough time on development. Meanwhile, on Pi 4 there’s no difference, the quad-core CPU has raw performance pretty close to the raw performance of the GPU. To lesser extent same applies to low-end PCs: I only have a fast GPU because I’m a graphics programmer, many people are happy with their Intel UHD graphics, these are not necessarily faster than CPUs.

> Does your approach allow for quality optimizations like sub pixel rendering for arbitrary curves?

The library only does grayscale AA for vector graphics.

Subpixel rendering implemented for text but comes with limitations: only works when text is not transformed (or transformed in specific ways, like rotated 180°, or horizontally flipped), and you need to pass background color behind the text to the API. Will only look good if the text is on solid color background, or on slow-changing gradient.

Sub pixel AA is hard for arbitrary backgrounds. I’m not sure many GPUs support the required blend states, and workarounds are slow.

> does that include drawing text as part of your transparent pass

Yes, that includes text and bitmaps. Here’s two HLSL shaders who do that, the GPU abstraction library I’ve picked recompiles the HLSL into GLSL or others on the fly: https://github.com/Const-me/Vrmac/blob/master/Vrmac/Draw/Sha... https://github.com/Const-me/Vrmac/blob/master/Vrmac/Draw/Sha... These shaders are compiled multiple times with different set of preprocessor macros, but I did test with all of them enabled at once.

piet-gpu contributor here. You're right that supersampling is the easiest way to achieve AA. However, its scalability issues are immense: for 2D rendering it's typically recommended to use 32x for a decent quality AA, but as the cost of supersampling scales linearly (actually superlinearly due to memory/register pressure), it becomes more than a magnitude slower than the baseline. So if you want to do anything that is real-time (e.g. smooth page zoom without resorting to prerendered textures which becomes blurry), supersampling is mostly an unacceptable choice.

What is more practical is some form of adaptive supersampling: a lot of pixels are filled by only one path and don't require supersampling. There's also some more heuristics that can be used: one that I want to try out in piet-gpu is to exploit the fact that in 2D graphics, most pixels are only covered by at most two paths. So as a base line we can track only two values per pixel plus a coverage mask, then in rare occasions of three or more shapes overlapping, fall back to full supersampling. This should keep the cost amplification more under control.

The short answer — it depends.

What you described is called super-sampling. Supersampling is indeed not terribly hard to implement, the problem is performance overhead. Many parts of graphics pipeline scale linearly with count of pixels. If you render at 16x16 upscaled resolution, that gonna result in 256x more pixel shader invocations, and 256x the fill rate.

There’s a good middle ground called MSAA https://en.wikipedia.org/wiki/Multisample_anti-aliasing In practice, 16x MSAA often delivers very good results for both 3D and 2D. In case of 2D, even low-end PC GPUs are fast enough with 8x or 16x MSAA level.

Initial versions of my library used that method.

The problem was, Raspberry Pi 4 GPU is way slower than PC GPUs. The performance with 4x or 2x MSAA was too low for 1920x1080 resolution, even just for 2D. Maybe the problem is actual hardware, maybe it’s a performance bug in Linux kernel or GPU drivers, I have no idea. I didn’t want to mess with the kernel, I wanted a library that works fast on officially supported 32-bit Debian Linux. That’s why I bothered to implement my own method for antialiasing.

High quality supersampling is about equally hard in 2D and 3D, since the result is 2D. It is also the most common solution in both 2D and 3D graphics, so your instinct is reasonably good.

But, font and path rendering, for example in web browsers and/or with PDF or SVG - these things can benefit in both efficiency and in quality from using analytic antialiasing methods. 2D vector rendering is a place where doing something harder has real payoffs.

Just a fun aside - not all supersampling algorithms are equally good. If you use the wrong filter, it's can be very surprising to discover that there are ways you can take a million samples per pixel or more and never succeed in getting rid of aliasing artifacts. (An example is if you just average the samples, aka use a Box Filter.) I have a 2D digital art project to render mathematical functions that can have arbitrarily high frequencies. I spent money making large format prints of them, so I care a lot about getting rid of aliasing problems. I've ended up with a Gaussian filter, which is a tad blurrier than experts tend to like, because everything else ends up giving me visible aliasing somewhere.

Good question.

If you use the fillRectangle method https://github.com/Const-me/Vrmac/blob/1.2/Vrmac/Draw/iDrawC... to draw them I think you should get solid fill. That particular API doesn’t use AA for that shape. Modern GPU hardware with their rasterization rules https://docs.microsoft.com/en-us/windows/win32/direct3d11/d3... is good at that use case, otherwise 3D meshes would contain holes between triangles.

If you render them as 2 distinct paths, filled, not stroked, and anti-aliased, you indeed gonna see a thin line between them. Currently, my AA method shrinks filled paths by about 0.5 pixels. For stroked paths it’s the opposite BTW, the output is inflated by half of the stroke width (the mid.point of the strokes correspond to the source geometry).

You can merge boxes into a single path with 2 figures https://github.com/Const-me/Vrmac/blob/1.2/Vrmac/Draw/Path/P..., in this case C++ code of the library should collapse the redundant inner edge and the output will be identical to a single box, i.e. solid fill. Will also render slightly faster because less triangles in the mesh.

It's referenced in slug's algorithm description paper [1], the main disadvantage with Loop-Blinn is the triangulation step that is required, and at small text sizes you lose a bit of performance. Slug only needs to render a quad for each glyph. That is not to say that any one method is better than the other though! They both have advantages and disadvantages. I think the two most advanced techniques for rendering vector graphics on the GPU are "Massively Parallel Vector Graphics" [2] and "Efficient GPU Path Rendering Using Scanline Rasterization" [3]. Though I don't know of any well known usage of them. Maybe it's because it's very hard to implement them, the sources attached to them are not trivial to understand, even if you've read the papers. They also use OpenCL/Cuda if I remember correctly.

[1] "GPU-Centered Font Rendering Directly from Glyph Outlines" http://jcgt.org/published/0006/02/02/

[2] http://w3.impa.br/~diego/projects/GanEtAl14/

[3] http://kunzhou.net/zjugaps/pathrendering/

EDIT: I've only now seen that [2] and [3] are already mentioned in the article

EDIT2: To compensate for my ignorance, I will add that one of the authors of MPVG has a course on rendering vector graphics: http://w3.impa.br/~diego/teaching/vg/

A signed distance field approach can be good depending on what you're after. https://github.com/libgdx/libgdx/wiki/Distance-field-fonts

You can always render the text to a texture offline as a signed distance field and just draw out quads as needed at render time. This will always be faster than drawing from the curves, and rendering from an SDF (especially multi-channel variants) scales surprisingly well if you choose the texture/glyph size well.

A little more info:

https://blog.mapbox.com/drawing-text-with-signed-distance-fi...

MIT-licensed open-source multi-channel glyph generation:

https://github.com/Chlumsky/msdfgen

The only remaining issue would be the kerning/layout, which is admittedly far from simple.

FOSS does not magically circumvent patents.

I've once implemented the basic idea behind the algorithm used in slug(described in the paper [1], though without the 'band' optimization, I just wanted to see how it works), and I agree with you, the real innovation is in that quadratic root-finder. It can tell you whether you are inside or outside just by manipulating the three control points of a curve, it's very fast, what remains to be done is to use an acceleration data structure so that you don't have to check for every curve. That works very well for quadratic Bézier curves, in the paper it says that it can be easily extended to cubics, though no example is provided(and I doubt it's trivial). What I think would be hard with Slug's method is extending it to draw gradients, shadows, basically general vector graphics like you say. Eric Lengyel on his twitter showed a demo [2] using Slug to render general vector graphics, but I'm not sure of how many features it supports, but it definitely supports cubic Bézier curves. I'd also like to add that the algorithm didn't impress me with how the text looks at small sizes, which I think is very important in general, though maybe not so much for games(maybe I just didn't implement it correctly).

[1] "GPU-Centered Font Rendering Directly from Glyph Outlines" http://jcgt.org/published/0006/02/02/

[2] https://twitter.com/EricLengyel/status/1190045334791057408

It doesn't seem to be a finished work. I guess this is more of a journal entry on the author's initial takeaways from a week-long coding retreat.

It's still there on all NVIDIA GPUs as an extension, just nobody uses it.

IMO it didn't catch on because all three of these points:

1. It only works on NVIDIA GPUs, and is riddled with joint patents from NVIDIA and Microsoft forbidding anyone like AMD or Intel from supporting it.

2. It's hard to use: you need to get your data into a format it can consume, usage is non-trivial, and often video game artists are already working with rasterized textures anyway so it's easy to omit.

3. Vector graphics suck for artists. The number of graphic designers I have met (who are the most likely subset of artists to work with vector graphics) that simply hate or do not understand the cubic bezier curve control points in Adobe, Inkscape, and other tools is incredible.

Stencil-and-cover approaches like NV_path_rendering have a lot of drawbacks like documented below, but probably biggest of all, they're still mostly just doing the tessellation on CPU. A lot of the important things, like winding mode calculations, are handled on the CPU. Modern research is looking for ways out of that.

Tanks for the amazing article! I wonder if you met any performance annoyances / bottlenecks when doing actual GUI / game dev with this?

Enjoyable article, thanks!

Thanks for the write up. Yeah i can see the huge performance difference, one thing that GPU bothers me is now every vendor provides a different set of API, you kinda have to use a hardware abstraction layer to use the GPU if you really want cross platform, and that's often a huge effort or dependency and hard to do right, even in OpenGL days it's easy because you only have to deal with one API instead of three vulkan/metal/d3d. With CPU if ignore the performance lack it's just a plain pixel array that can be easily displayed on any environment and you have control over any bits of it, I just can't get over the lightness and elegance difference between the two..