Personally I wouldn't teach OpenGL ES 3.0 or any OpenGL because its state machine (https://webgl2fundamentals.org/webgl/lessons/resources/webgl...) and horrible error system means tracking down bugs is hard. Instead I'd teach WebGPU as it's portable and has great error messaging.
I'm sure people will objected but I'd argue WebGPU is easier to understand, has better errors, has no state machine, works C++, Rust, JS/TS and other languages, and closer to modern GPU programming if you want to move to Metal or D3D12 or Vulkan later.
Further, OpenGL (and ES) are a nightmare of bugs and incompatibilities students might run into that WebGPU fixes.
That said, the article mentions Apple platforms where you can use ANGLE if you happened to have students stuck there.
I mostly agree regarding GL's programming model, but tbf some problems have been fixed by WebGL2 (much better validation and error reporting - e.g. very helpful error messages right on the browser console - and it's probably the least buggy and most portable 3D API implementation of them all, including the current state of WebGPU.
For teaching I would generally use the web browser via WebGL2 or WebGPU, just because those APIs require much less boilerplate in the browser environment to get to your first triangle and the edit-test-loop is pretty much instant.
In real life there’s lots of bugs around and difficult to read error messages.
Re. the boilerplate just have students clone a pre-fabbed repo with the boilerplate handled for them.
The real question should be whether you want students to know how to program in a low programming language or not. Whether they should see what the reality of programming is or not.
If they can’t program in C/C++ and debug the error messages how are they going to survive in the wild? I guess they’ll use an LLM.
> If they can’t program in C/C++ and debug the error messages how are they going to survive in the wild?
The problem is that you don't get any error messages from native GL in the first place (unless you wire up a debug-message callback via the optional KHR_debug extension which isn't available everywhere), and the builtin error system via glGetError() is pretty much useless.
Other APIs don't suffer from this problem, D3D, Vulkan and Metal all have excellent validation layers which give very helpful messages, and WebGL or WebGPU running in browsers basically implements such a validation layer out of the box.
WebGPU will however always suffer from being an API designed for the Web, and targeting 2015 thereabouts hardware.
Better than OpenGL, but don't expect that much improvements coming into their way, other than what the Chrome team might feel like pushing, as I doubt Firefox or Safari teams have much left to say about where it goes beyond version 1.0.
I have found OpenGL 4.5/4.6 to be a sweet spot for students currently learning graphics in a semester course. SDL_GPU is otherwise a good intermediate I would consider.
I remember for myself: learning a new API (opengl), new paradigm (gpu programming), and new language(glsl) all on top of learning graphics concepts is a lot. OpenGL 4.5 and 4.6 are modern enough to get started and there are good resources to learn the 3.3+ stuff that I have built on.
See my website(mshah.io) for my paper on some of the 'why'.
The tutorials and hands on coding examples made me understand how to program with GPUs. I loved them because at the time, khronos was still in its infancy and just started out with GLES extensions and the whole WebGL ecosystem.
Meanwhile I kind of believe that learning where it came from (fixed function pipeline, buffer/texture and uniform problems etc) makes a lot of sense to learn first so you start thinking in shader computation pipelines much easier, as you understand the whys and whens much better.
I've learned more graphics writing software rasterizers and ray tracers than using any APIs. If the goal is to teach GPU rendering specifically, then OpenGL is indeed more amenable to teaching than the others. But if the goal is to teach graphics and rendering concepts, then OpenGL et al are just miserable and counter-productive.
Disagree.
Trying to express all of graphics and rendering through the lens of OpenGL will of course fail and doing 3D rendering itself via raytracing is simply not possible in rasterization focused OpenGL without dirty workarounds. That is obvious.
A fundamental shift in thinking that got me into high performance graphics programming and parallelized image processing, the "Aha" Moment, is being forced to express how to calculate the final pixel independently, without having access to neighboring pixels of the same output buffer, confined to the straight jacket of the fragment shader. OpenGL with it's combination of hand-holding and relative simplicity as compared to Vulkan and limits of a predefined shader pipeline as compared to a pure software implementation is exactly the thing that I needed to get into graphics programming. It's an excellent jump in point I think.
I agree with Adam Sawicki that it's not that relevant any more but also I do believe it's valuable lesson to tech students where the modernity comes from.
I just don't see personally sense to focus on it longer than just 1mo of intro, tutorial, get through simple example, show limitations, carry on to Vulkan/WGPU with background story that there is also DX12 and Metal that follow similar principles.
> carry on to Vulkan/WGPU with background story that there is also DX12 and Metal that follow similar principles
Except that's a minefield all of its own. DirectX 12 is its own special beast that is unlike the others. Vulkan is already mired in 3 revisions of architecture changes. WGPU is based on the (outdated) first cut of Vulkan, not the current model. Of all of these, only Metal has a reasonable on-ramp for newcomers to 3D graphics.
Tbf, it all depends on how deep you look into the fractal ;) The main differences between the modern APIs are in how resource binding is done, but many other concept overlap quite well between Vulkan, D3D12 and Metal (especially in more recent Vulkan revisions).
> Of all of these, only Metal has a reasonable on-ramp for newcomers to 3D graphics.
...unless you start with Metal 4, which is bringing in a lot of Vulkan/D3D12 concepts (for better or worse, it's definitely not great for 'convenience' but may help with porting Vulkan or D3D12 code over).
Nice breakdown. I've been wondering why it's so complicated to program on the GPU compared to the CPU. Is it a language paradigm issue or maybe a hardware design limitation? I always think of this when looking at the graphic APIs, espeically Vulkan.
It's mostly hardware design limitations. GPU hardware is way more diverse than CPU hardware, so you need to distribute shaders as either source code or a portable intermediate representation, in both cases compiled at runtime for the user's actual GPU. There are also many fixed function parts of the graphics pipeline with wildly different behavior on different hardware which you need to abstract over. And it all runs in parallel with the CPU, often with a separate pool of RAM, so you need implicit or explicit synchronization and data transfer which adds a lot of complexity especially in error cases.
If you really want to see why all the cruft is there, and how much you could remove if you dropped support for older hardware, look no further than Sebastian Aaltonen's "No Graphics API" blog post (which he actually implemented too): https://www.sebastianaaltonen.com/blog/no-graphics-api/
I think most of it comes down to 'historical accidents' tbh.
E.g. if CPUs would have closed ISAs and heavily guarded IPs, you'd also need a 'driver' and an 'OpenCPU API' to make any use of them (of course the public ISA of a CPU is essentially the driver interface baked into the hardware).
The other big difference is focus on decent single-thread performance for CPUs versus performing thousands of small tasks in parallel on GPUs, but each one slowly. This difference causes a different programming model (e.g. running an entire application in a single 'main' function from start to finish, versus spawning small specialized tasks which are running across hundreds of threads).
And the specific problem with Vulkan is that it tried to solve two conflicting requirements: on one hand being an explicit low-level API without 'magic abstractions', on the other hand supporting very different GPU architectures (especially the mobile GPU zoo). The resulting mess was quite predictable to be honest...
First, GPUs aren't simply "highly parallel CPUs", they also contain a whole lot of fixed function hardware that is specialized for real-time computer graphics. This hardware isn't exposed as "fancy instructions", it is set up in other ways, which is achieved through a device driver.
Second: GPUs are coprocessor, separate from the CPU where another OS is running. This also necessitates a device driver to manage scheduling, synchronization, etc. with the main OS. Because, in case you were not aware, the GPU itself runs its own little operating system, although it is rarely called that.
> they also contain a whole lot of fixed function hardware that is specialized for real-time computer graphics. This hardware isn't exposed as "fancy instructions", it is set up in other ways, which is achieved through a device driver.
Those hardwired fixed-function parts are getting fewer and less important though (ok, and then raytracing units entered the scene...). Also AFAIK something like a texture sampling operation is indeed a 'fancy instruction'(?), even though the actual texture sampling is done by a 'fixed function' hardware unit.
> This also necessitates a device driver to manage scheduling, synchronization, etc. with the main OS.
I bet this type of GPU/CPU handshaking work could also be baked into the hardware (similar to CPU mirocode) and exposed via a standardized memory mapped IO interface (or whatever the current mechanism is for letting the CPU control the GPU). E.g. a CPU also implements similar protocols to talk to the rest of the computer system (e.g. memory and other CPUs), but this is mainly implemented as "hardware protocols" - drivers "just" offer more flexibility for hardware vendors because they can delegate the standardization into software.
...also a lot of the "CPU / GPU software ecosystem" was probably shaped by CPU and GPU vendors historically being different companies, and once the big consolidation started, the conventions were already set in stone.
part of the reason is historical cruft & history path dependency. GPUs didn't start as general computing device for parallelism but a device for the specific purpose of accelerating graphics with very specific fixed functions for that job and only that. with time that fixed pipeline became more complex and more configurable, not fixed/more flexible and hacks were in parallel used to make use of it for tasks outside graphics those hacks were later made into something more formal (CUDA/OpenCL).
CUDA being the winner (closed source, not really usable on non-nvidia GPUs) between those 2 and nvidia not wanting to open their competitive advantage to the competition for the niche of non gamings uses / scientific computing (and later AI, which became a huge win for nvidia).
vulkan came around after as this open thing (based on a previous attempt by AMD called mantle) that's trying to break down the leaky abstractions and the divide between language for graphics & video games and separate language/frameworks for gpu non-graphic computing with the promise of giving more control and more room for squeezing performance from the hardware at the cost of more complexity with the idea that you'll probably build your own abstractions on top of it for your subset of applications.
If GPU were from the get go designed as a parallelism tool first and video game tools second the hardware architecture and the language would probably be different than today and simpler (and if the competition between the duopoly didn't sabotage things)
don't forget that the drivers for GPUs would until recently get updates to patch specific video games for performance issues (not sure if they still do that ?)
> If GPU were from the get go designed as a parallelism tool first and video game tools second the hardware architecture and the language would probably be different than today and simpler (and if the competition between the duopoly didn't sabotage things)
Sony tried that with the PS3's Cell processor, just a wise parallel vector processor instead of dedicated hardware.
If you knew how to use it, you could get great results, which is how Naughty Dog's games became so spectacular. But if you just tried to port an existing game, things went downhill quickly, which is why GabeN talked on stage how much he hated the PS3.
Oh and this complexity led to lots of games just switching to one of the existing engines that supported it natively, and started the modern dominance of Unreal Engine. But it's also why engines introduced more parallel architectures, which a few years later became very useful when quad-core processors in PCs became mainstream.
Good summary, thanks. Sounds like it's all still pretty early, even for graphics but much more for parallel compute, which seems the direction the hardware is going in. I'm sure there is a CUDA equivalent for CPU but I would guess it's all the way back in the 70s or earlier. Hopefully open standards gets some traction.
> If GPU were from the get go designed as a parallelism tool first and video game tools second the hardware architecture and the language would probably be different than today and simpler (...)
The market demand was for video game tools, though.
People didn't wanted faster compute, they wanted their computer games to work smoothly on the hardware they already own. Therefore, they preferred to spend 100-200€ on a point upgrade, much like RAM, than scrapping their 2k€ PC.
To give people a reason to buy these upgrades, games had to use them. Therefore video cards were designed specially for the usecases and development flow that game developers already followed.
If it wasn't for the video game market, NVidia wouldn't exist as we know it today.
One thing that helped me understand it was to reframe it as a network application, where you have a client running on the CPU and a server running on the GPU (the shaders).
A bunch of the complexity comes from having to write both client and server, and the communication protocol between them, then some more comes from the client being responsible for spinning up the server. As with every network service, client/server traffic is the slowest part of the architecture, so a bunch of complexity then comes from having to minimise the amount of network calls (or draw calls, in this case).
Then you hit the part where all of this is performance critical (soft-realtime requirements), so the whole thing is geared towards “writing your own userspace IP stack in C++” rather than “writing a flask application in Python”.
The thing that makes it "hard" is that it's all indirect. You compile a program for the GPU (a shader). Then you have to setup data (textures and buffers) for it. Then, you need to spend a bunch of time telling the GPU how to use that data, offsets where each piece of data is, what formats the data is in,, how to advance to the next piece of data, etc.. etc..
There are a few libraries that try to abstract over all this indirection but all the graphics APIs themselves are lower-level so you're left to do all of that yourself.
It's not more complicated, you have things like CUDA which is basically C/C++ that compiles to GPU.
But GPUs have thousands of cores, which means if you want any benefit, the smallest data type you must use is something like array[n * 1024], or your GPU is wasted doing nothing. So you must shape your programs into that memory model or you're just wasting your time (and GPU).
Right, that makes sense. There's a ton of overhead in the setup though, I'm assuming that's because everything runs on the CPU by default. Is this then an OS design choice? Thanks for your input.
There can be overhead, though GPU memory + drivers are some of the most optimized software in the world.
Some OS/hardware have separate GPU memory (Nvidia), some have unified memory (Macos, modern consoles), but moving memory is usually not the problem.
Things don't "run on CPU by default", it's up to the programmer where to run it. But the CPU and GPU paradigm of "running a program" are very different, since on GPU you're running SIMD (i.e. running your code on arrays) or else there is no point to use a GPU.
For me, if-branching is the hardest to wrap my head around. "If"s are so natural to CPU programming that it's second nature to any developer but on GPU, it's expensive.
instead of iterating over an array and if-branching to one of two paths for each item, conceive of your conditionals as logic operations which test for set membership (i.e. "is this array item a member of the set i care about?") and with this array mask of "do the thing or don't do the thing" flags, you can mutate your data 0 or 1 times.
the key for me is thinking in terms of sets—much of your conditional logic is probably about sorting things into sets anyway.
It's partially historical; I'm sure if we redesigned GPGPU today from the ground up, we'd have an entirely different pipeline. And partially due to hardware vendor lock-in; you can't really design an alternative to Vulkan/d3d12 even if you had the skill and drive to do so.
But also partly in the fact that the GPU is specialized for certain operations over a CPU. It wants and need to obtain data in a different way from a CPU to maximize throughput. So language constructs are designed around making sure developers stay on this fast path. The complexity can be seen as some intentional gatekeeping to signal that this kind of programming is only for specific kinds of problems.
actually yes you could design an alternative API and even implement it....
there's HW references + the linux drivers for AMD (both UMD and KMD), there's refs for NV and their open linux KMD (no UMD though), intel also has HW refs and open drivers for linux
I think it would be possible to put together as long as you don't mind only working for certain hardware, like a prototype you know
I might give it a shot to make a proof-of-concept "high level low level" graphics API ngl if no one else feels like so...
I think there's space for a "two-level" design like have the driver consist of the HW-specific parts, expose things in terms of GPU VAs, explicit sync, explicit memory management, command buffers etc. then have a high-level wrapper over it to provide vaguely GL-ish semantics, and you could intermix the two APIs, obviously you wouldn't be able to get the VA for a high-level object because that'd break the autotracking, if you use low-level you're responsible for syncing, cycling etc.
GLES isn't a bad way to teach graphics but I wish there was a middle ground between it and VK.
The worst part of GL is the mysterious global state you will never understand without studying the spec. The world could use an API that's basically GL but without global hidden singleton (per thread!) BS behind the scenes.
WebGPU backend (has a couple more demos which use features not available on GLES3.0/WebGL2, like compute shaders): https://floooh.github.io/sokol-webgpu/
dx11 isn't bad but it has a looot of cruft around it, writing a good constant buffer manager is witchcraft (remember, there's no "loose uniforms" injected into the command stream, it's all constant buffer updates, drivers do all sorts of writecombining magic to mitigate it, but the naive map / unmap cycle without any driver optimisations would absolutely tank....
btw you can run dx11 on other platforms because dxvk exists, but while d3d9vk is a solid idea (high-level API, lowering to vulkan often makes it much faster than the crummy MS d3d9 runtime + DDI combo), d3d11vk is just often a wash or even a regression. I've had to debug dxvk-only perf problems a bit too many times...
anyway, point is, it's not really an easy-to-use or intuitive API
Nothing is an easy to use intuitive api with all the features and all the performance, it's (Dx11) the sweet spot of where I felt like it was easy to learn and easy to use without worrying about performance too much.
Sure if you are building the newer rendering pipelines DirectX 11 leaves a lot on the table but it brought a lot of nice things with it, Debug Runtime alone was worth moving to it, 99/100 times your app written in DirectX 11 even if not optimised properly will perform better than DirectX 9 in my experience.
This holds with OpenGL though the modern OpenGL 4.6 and stuff wasn't bad, but it's a lot of cruft still.
Either way there are no perfect APIs, a truly general purpose API that just gives you basics like Texture handling and io and data mapping would be enough today imho. Fully general purpose compute APIs are the solution with some additional tweaks to make graphics stuff convenient. IMHO
Though it's definitely much harder than plain and simple paint colors and textures on vertices & geometry uv approach of old.
> writing a good constant buffer manager is witchcraft
Tbh, I think the main lesson is simply to not try to work around the specific 3D API conventions too much, e.g. don't try to shoehorn mechanisms into the D3D11 layer that would be straightforward in Vulkan or D3D12 but would disable the special fast-paths in D3D11.
E.g. for a traditional "one uniform update per draw call" loop with one small constant buffer per shader updated via plain old UpdateSubresource(), D3D11 is incredibly fast (you can easily do hundreds-of-thousands update+drawcall pairs like this per 60Hz frame on a somewhat recent - e.g. post-2016 or so - mid-level gaming PC.
Of course trying to emulate D3D11 on Vulkan or D3D12 might have all sorts of issues that don't exist on a good "native" D3D11 driver.
Also somewhat related: in a GL shim it's typically still faster to update uniform data via the old glUniform*v calls instead of using uniform buffers, apparently most GL drivers lack the special fast-path optimizations of small D3D11 constant buffers, and the more abstract glUniform calls seems to give the driver more 'wiggle room' to implement an optimal upload path.
I mean sure you can do a generic layer and common denominator stuff, basically the kinda stuff webgpu or those rendering abstractions do and that's fair enough, it works well for simpler stuff!
The problem really is that if you do that you'll end up with random weird stalls or perf problems you can't really debug sadly :\
Re GL ye glUniform is usually much faster because you just do a lot less, you're writing a small amount of data directly into constant memory, it doesn't need resource tracking. UBOs / constant buffers come back to the theoretically slow but can-be-made-faster-with-driver-heuristics approach...i.e. if you see a small UBO which is only used as a UBO, you get the same direct constant bank upload
On NV it's quite fast, on AMD it's not really optimised, on intel just use buffersubdata because everything else sucks....
TL;DR because WebGL 2 is basically equivalent to OpenGL ES 3.0, making it the most portable GPU API in the world by far, running on everything from laptops and phones to smart fridges and car head units.
Web browsers have also done a lot of behind-the-scenes work on compatibility. The behavior of WebGL 2 is much, much more consistent across platforms than native OpenGL [ES] ever was.
Personally I wouldn't teach OpenGL ES 3.0 or any OpenGL because its state machine (https://webgl2fundamentals.org/webgl/lessons/resources/webgl...) and horrible error system means tracking down bugs is hard. Instead I'd teach WebGPU as it's portable and has great error messaging.
I'm sure people will objected but I'd argue WebGPU is easier to understand, has better errors, has no state machine, works C++, Rust, JS/TS and other languages, and closer to modern GPU programming if you want to move to Metal or D3D12 or Vulkan later.
Further, OpenGL (and ES) are a nightmare of bugs and incompatibilities students might run into that WebGPU fixes.
That said, the article mentions Apple platforms where you can use ANGLE if you happened to have students stuck there.
I mostly agree regarding GL's programming model, but tbf some problems have been fixed by WebGL2 (much better validation and error reporting - e.g. very helpful error messages right on the browser console - and it's probably the least buggy and most portable 3D API implementation of them all, including the current state of WebGPU.
For teaching I would generally use the web browser via WebGL2 or WebGPU, just because those APIs require much less boilerplate in the browser environment to get to your first triangle and the edit-test-loop is pretty much instant.
In real life there’s lots of bugs around and difficult to read error messages.
Re. the boilerplate just have students clone a pre-fabbed repo with the boilerplate handled for them.
The real question should be whether you want students to know how to program in a low programming language or not. Whether they should see what the reality of programming is or not.
If they can’t program in C/C++ and debug the error messages how are they going to survive in the wild? I guess they’ll use an LLM.
> If they can’t program in C/C++ and debug the error messages how are they going to survive in the wild?
The problem is that you don't get any error messages from native GL in the first place (unless you wire up a debug-message callback via the optional KHR_debug extension which isn't available everywhere), and the builtin error system via glGetError() is pretty much useless.
Other APIs don't suffer from this problem, D3D, Vulkan and Metal all have excellent validation layers which give very helpful messages, and WebGL or WebGPU running in browsers basically implements such a validation layer out of the box.
WebGPU will however always suffer from being an API designed for the Web, and targeting 2015 thereabouts hardware.
Better than OpenGL, but don't expect that much improvements coming into their way, other than what the Chrome team might feel like pushing, as I doubt Firefox or Safari teams have much left to say about where it goes beyond version 1.0.
I have found OpenGL 4.5/4.6 to be a sweet spot for students currently learning graphics in a semester course. SDL_GPU is otherwise a good intermediate I would consider.
I remember for myself: learning a new API (opengl), new paradigm (gpu programming), and new language(glsl) all on top of learning graphics concepts is a lot. OpenGL 4.5 and 4.6 are modern enough to get started and there are good resources to learn the 3.3+ stuff that I have built on.
See my website(mshah.io) for my paper on some of the 'why'.
On a related note, I'd like to give a shout-out to NeHe (Neon Helium) for his brilliant OpenGL tutorials back in the 90s.
I, and I suspect many others, benefited immensely from him lessons.
Oh my I totally forgot about NeHe! I probably still have some of the lessons printed out at my parents house somewhere.
Good times.
Yep, can confirm.
The tutorials and hands on coding examples made me understand how to program with GPUs. I loved them because at the time, khronos was still in its infancy and just started out with GLES extensions and the whole WebGL ecosystem.
Meanwhile I kind of believe that learning where it came from (fixed function pipeline, buffer/texture and uniform problems etc) makes a lot of sense to learn first so you start thinking in shader computation pipelines much easier, as you understand the whys and whens much better.
I've learned more graphics writing software rasterizers and ray tracers than using any APIs. If the goal is to teach GPU rendering specifically, then OpenGL is indeed more amenable to teaching than the others. But if the goal is to teach graphics and rendering concepts, then OpenGL et al are just miserable and counter-productive.
Disagree. Trying to express all of graphics and rendering through the lens of OpenGL will of course fail and doing 3D rendering itself via raytracing is simply not possible in rasterization focused OpenGL without dirty workarounds. That is obvious.
A fundamental shift in thinking that got me into high performance graphics programming and parallelized image processing, the "Aha" Moment, is being forced to express how to calculate the final pixel independently, without having access to neighboring pixels of the same output buffer, confined to the straight jacket of the fragment shader. OpenGL with it's combination of hand-holding and relative simplicity as compared to Vulkan and limits of a predefined shader pipeline as compared to a pure software implementation is exactly the thing that I needed to get into graphics programming. It's an excellent jump in point I think.
The way I’m interpreting their post is, you could far more easily teach things like introductory projective geometry without a GPU.
I agree with Adam Sawicki that it's not that relevant any more but also I do believe it's valuable lesson to tech students where the modernity comes from.
I just don't see personally sense to focus on it longer than just 1mo of intro, tutorial, get through simple example, show limitations, carry on to Vulkan/WGPU with background story that there is also DX12 and Metal that follow similar principles.
> carry on to Vulkan/WGPU with background story that there is also DX12 and Metal that follow similar principles
Except that's a minefield all of its own. DirectX 12 is its own special beast that is unlike the others. Vulkan is already mired in 3 revisions of architecture changes. WGPU is based on the (outdated) first cut of Vulkan, not the current model. Of all of these, only Metal has a reasonable on-ramp for newcomers to 3D graphics.
And then there are also the NVN and LibGNM(X) in the corner that folks forget about, due to NDAs and such.
Ah, and DirectX on XBox is not the same as on Windows.
Tbf, it all depends on how deep you look into the fractal ;) The main differences between the modern APIs are in how resource binding is done, but many other concept overlap quite well between Vulkan, D3D12 and Metal (especially in more recent Vulkan revisions).
> Of all of these, only Metal has a reasonable on-ramp for newcomers to 3D graphics.
...unless you start with Metal 4, which is bringing in a lot of Vulkan/D3D12 concepts (for better or worse, it's definitely not great for 'convenience' but may help with porting Vulkan or D3D12 code over).
Nice breakdown. I've been wondering why it's so complicated to program on the GPU compared to the CPU. Is it a language paradigm issue or maybe a hardware design limitation? I always think of this when looking at the graphic APIs, espeically Vulkan.
It's mostly hardware design limitations. GPU hardware is way more diverse than CPU hardware, so you need to distribute shaders as either source code or a portable intermediate representation, in both cases compiled at runtime for the user's actual GPU. There are also many fixed function parts of the graphics pipeline with wildly different behavior on different hardware which you need to abstract over. And it all runs in parallel with the CPU, often with a separate pool of RAM, so you need implicit or explicit synchronization and data transfer which adds a lot of complexity especially in error cases.
If you really want to see why all the cruft is there, and how much you could remove if you dropped support for older hardware, look no further than Sebastian Aaltonen's "No Graphics API" blog post (which he actually implemented too): https://www.sebastianaaltonen.com/blog/no-graphics-api/
I think most of it comes down to 'historical accidents' tbh.
E.g. if CPUs would have closed ISAs and heavily guarded IPs, you'd also need a 'driver' and an 'OpenCPU API' to make any use of them (of course the public ISA of a CPU is essentially the driver interface baked into the hardware).
The other big difference is focus on decent single-thread performance for CPUs versus performing thousands of small tasks in parallel on GPUs, but each one slowly. This difference causes a different programming model (e.g. running an entire application in a single 'main' function from start to finish, versus spawning small specialized tasks which are running across hundreds of threads).
And the specific problem with Vulkan is that it tried to solve two conflicting requirements: on one hand being an explicit low-level API without 'magic abstractions', on the other hand supporting very different GPU architectures (especially the mobile GPU zoo). The resulting mess was quite predictable to be honest...
You underestimate the effect of two factors.
First, GPUs aren't simply "highly parallel CPUs", they also contain a whole lot of fixed function hardware that is specialized for real-time computer graphics. This hardware isn't exposed as "fancy instructions", it is set up in other ways, which is achieved through a device driver.
Second: GPUs are coprocessor, separate from the CPU where another OS is running. This also necessitates a device driver to manage scheduling, synchronization, etc. with the main OS. Because, in case you were not aware, the GPU itself runs its own little operating system, although it is rarely called that.
> they also contain a whole lot of fixed function hardware that is specialized for real-time computer graphics. This hardware isn't exposed as "fancy instructions", it is set up in other ways, which is achieved through a device driver.
Those hardwired fixed-function parts are getting fewer and less important though (ok, and then raytracing units entered the scene...). Also AFAIK something like a texture sampling operation is indeed a 'fancy instruction'(?), even though the actual texture sampling is done by a 'fixed function' hardware unit.
> This also necessitates a device driver to manage scheduling, synchronization, etc. with the main OS.
I bet this type of GPU/CPU handshaking work could also be baked into the hardware (similar to CPU mirocode) and exposed via a standardized memory mapped IO interface (or whatever the current mechanism is for letting the CPU control the GPU). E.g. a CPU also implements similar protocols to talk to the rest of the computer system (e.g. memory and other CPUs), but this is mainly implemented as "hardware protocols" - drivers "just" offer more flexibility for hardware vendors because they can delegate the standardization into software.
...also a lot of the "CPU / GPU software ecosystem" was probably shaped by CPU and GPU vendors historically being different companies, and once the big consolidation started, the conventions were already set in stone.
If you care about every last bit of performance, CPUs are difficult to program for as well, possibly more difficult than a GPU.
Generally GPUs punish you more by bad performance if you don't optimise your code.
part of the reason is historical cruft & history path dependency. GPUs didn't start as general computing device for parallelism but a device for the specific purpose of accelerating graphics with very specific fixed functions for that job and only that. with time that fixed pipeline became more complex and more configurable, not fixed/more flexible and hacks were in parallel used to make use of it for tasks outside graphics those hacks were later made into something more formal (CUDA/OpenCL).
CUDA being the winner (closed source, not really usable on non-nvidia GPUs) between those 2 and nvidia not wanting to open their competitive advantage to the competition for the niche of non gamings uses / scientific computing (and later AI, which became a huge win for nvidia).
vulkan came around after as this open thing (based on a previous attempt by AMD called mantle) that's trying to break down the leaky abstractions and the divide between language for graphics & video games and separate language/frameworks for gpu non-graphic computing with the promise of giving more control and more room for squeezing performance from the hardware at the cost of more complexity with the idea that you'll probably build your own abstractions on top of it for your subset of applications.
If GPU were from the get go designed as a parallelism tool first and video game tools second the hardware architecture and the language would probably be different than today and simpler (and if the competition between the duopoly didn't sabotage things)
don't forget that the drivers for GPUs would until recently get updates to patch specific video games for performance issues (not sure if they still do that ?)
> If GPU were from the get go designed as a parallelism tool first and video game tools second the hardware architecture and the language would probably be different than today and simpler (and if the competition between the duopoly didn't sabotage things)
Sony tried that with the PS3's Cell processor, just a wise parallel vector processor instead of dedicated hardware.
If you knew how to use it, you could get great results, which is how Naughty Dog's games became so spectacular. But if you just tried to port an existing game, things went downhill quickly, which is why GabeN talked on stage how much he hated the PS3.
Oh and this complexity led to lots of games just switching to one of the existing engines that supported it natively, and started the modern dominance of Unreal Engine. But it's also why engines introduced more parallel architectures, which a few years later became very useful when quad-core processors in PCs became mainstream.
Good summary, thanks. Sounds like it's all still pretty early, even for graphics but much more for parallel compute, which seems the direction the hardware is going in. I'm sure there is a CUDA equivalent for CPU but I would guess it's all the way back in the 70s or earlier. Hopefully open standards gets some traction.
> If GPU were from the get go designed as a parallelism tool first and video game tools second the hardware architecture and the language would probably be different than today and simpler (...)
The market demand was for video game tools, though.
People didn't wanted faster compute, they wanted their computer games to work smoothly on the hardware they already own. Therefore, they preferred to spend 100-200€ on a point upgrade, much like RAM, than scrapping their 2k€ PC.
To give people a reason to buy these upgrades, games had to use them. Therefore video cards were designed specially for the usecases and development flow that game developers already followed.
If it wasn't for the video game market, NVidia wouldn't exist as we know it today.
One thing that helped me understand it was to reframe it as a network application, where you have a client running on the CPU and a server running on the GPU (the shaders).
A bunch of the complexity comes from having to write both client and server, and the communication protocol between them, then some more comes from the client being responsible for spinning up the server. As with every network service, client/server traffic is the slowest part of the architecture, so a bunch of complexity then comes from having to minimise the amount of network calls (or draw calls, in this case).
Then you hit the part where all of this is performance critical (soft-realtime requirements), so the whole thing is geared towards “writing your own userspace IP stack in C++” rather than “writing a flask application in Python”.
The thing that makes it "hard" is that it's all indirect. You compile a program for the GPU (a shader). Then you have to setup data (textures and buffers) for it. Then, you need to spend a bunch of time telling the GPU how to use that data, offsets where each piece of data is, what formats the data is in,, how to advance to the next piece of data, etc.. etc..
There are a few libraries that try to abstract over all this indirection but all the graphics APIs themselves are lower-level so you're left to do all of that yourself.
https://docs.swmansion.com/TypeGPU/
https://github.com/google/toucan
It's not more complicated, you have things like CUDA which is basically C/C++ that compiles to GPU.
But GPUs have thousands of cores, which means if you want any benefit, the smallest data type you must use is something like array[n * 1024], or your GPU is wasted doing nothing. So you must shape your programs into that memory model or you're just wasting your time (and GPU).
That's >50% of the problem.
Right, that makes sense. There's a ton of overhead in the setup though, I'm assuming that's because everything runs on the CPU by default. Is this then an OS design choice? Thanks for your input.
There can be overhead, though GPU memory + drivers are some of the most optimized software in the world.
Some OS/hardware have separate GPU memory (Nvidia), some have unified memory (Macos, modern consoles), but moving memory is usually not the problem.
Things don't "run on CPU by default", it's up to the programmer where to run it. But the CPU and GPU paradigm of "running a program" are very different, since on GPU you're running SIMD (i.e. running your code on arrays) or else there is no point to use a GPU.
For me, if-branching is the hardest to wrap my head around. "If"s are so natural to CPU programming that it's second nature to any developer but on GPU, it's expensive.
instead of iterating over an array and if-branching to one of two paths for each item, conceive of your conditionals as logic operations which test for set membership (i.e. "is this array item a member of the set i care about?") and with this array mask of "do the thing or don't do the thing" flags, you can mutate your data 0 or 1 times.
the key for me is thinking in terms of sets—much of your conditional logic is probably about sorting things into sets anyway.
It's partially historical; I'm sure if we redesigned GPGPU today from the ground up, we'd have an entirely different pipeline. And partially due to hardware vendor lock-in; you can't really design an alternative to Vulkan/d3d12 even if you had the skill and drive to do so.
But also partly in the fact that the GPU is specialized for certain operations over a CPU. It wants and need to obtain data in a different way from a CPU to maximize throughput. So language constructs are designed around making sure developers stay on this fast path. The complexity can be seen as some intentional gatekeeping to signal that this kind of programming is only for specific kinds of problems.
actually yes you could design an alternative API and even implement it....
there's HW references + the linux drivers for AMD (both UMD and KMD), there's refs for NV and their open linux KMD (no UMD though), intel also has HW refs and open drivers for linux
I think it would be possible to put together as long as you don't mind only working for certain hardware, like a prototype you know
I might give it a shot to make a proof-of-concept "high level low level" graphics API ngl if no one else feels like so...
I think there's space for a "two-level" design like have the driver consist of the HW-specific parts, expose things in terms of GPU VAs, explicit sync, explicit memory management, command buffers etc. then have a high-level wrapper over it to provide vaguely GL-ish semantics, and you could intermix the two APIs, obviously you wouldn't be able to get the VA for a high-level object because that'd break the autotracking, if you use low-level you're responsible for syncing, cycling etc.
GLES isn't a bad way to teach graphics but I wish there was a middle ground between it and VK.
The worst part of GL is the mysterious global state you will never understand without studying the spec. The world could use an API that's basically GL but without global hidden singleton (per thread!) BS behind the scenes.
> but I wish there was a middle ground between it and VK
There's WebGPU (probably the best option for teaching), D3D11 and Metal.
And (shameless plug): sokol-gfx tries to find a balance between simplicity and modern 3D APIs:
https://github.com/floooh/sokol
Demos with links to C source and shader code:
WebGL2 backend: https://floooh.github.io/sokol-html5/
WebGPU backend (has a couple more demos which use features not available on GLES3.0/WebGL2, like compute shaders): https://floooh.github.io/sokol-webgpu/
That middle ground is often called DirectX 11 but sadly it's stuck to a specific platform.
dx11 isn't bad but it has a looot of cruft around it, writing a good constant buffer manager is witchcraft (remember, there's no "loose uniforms" injected into the command stream, it's all constant buffer updates, drivers do all sorts of writecombining magic to mitigate it, but the naive map / unmap cycle without any driver optimisations would absolutely tank....
Look at a GDC talk (https://d29g4g2dyqv443.cloudfront.net/sites/default/files/ak...), efficient dx11 is also "reinvent a diy memory allocator" level except that you don't have much control.
btw you can run dx11 on other platforms because dxvk exists, but while d3d9vk is a solid idea (high-level API, lowering to vulkan often makes it much faster than the crummy MS d3d9 runtime + DDI combo), d3d11vk is just often a wash or even a regression. I've had to debug dxvk-only perf problems a bit too many times...
anyway, point is, it's not really an easy-to-use or intuitive API
Nothing is an easy to use intuitive api with all the features and all the performance, it's (Dx11) the sweet spot of where I felt like it was easy to learn and easy to use without worrying about performance too much.
Sure if you are building the newer rendering pipelines DirectX 11 leaves a lot on the table but it brought a lot of nice things with it, Debug Runtime alone was worth moving to it, 99/100 times your app written in DirectX 11 even if not optimised properly will perform better than DirectX 9 in my experience.
This holds with OpenGL though the modern OpenGL 4.6 and stuff wasn't bad, but it's a lot of cruft still.
Either way there are no perfect APIs, a truly general purpose API that just gives you basics like Texture handling and io and data mapping would be enough today imho. Fully general purpose compute APIs are the solution with some additional tweaks to make graphics stuff convenient. IMHO
Though it's definitely much harder than plain and simple paint colors and textures on vertices & geometry uv approach of old.
> writing a good constant buffer manager is witchcraft
Tbh, I think the main lesson is simply to not try to work around the specific 3D API conventions too much, e.g. don't try to shoehorn mechanisms into the D3D11 layer that would be straightforward in Vulkan or D3D12 but would disable the special fast-paths in D3D11.
E.g. for a traditional "one uniform update per draw call" loop with one small constant buffer per shader updated via plain old UpdateSubresource(), D3D11 is incredibly fast (you can easily do hundreds-of-thousands update+drawcall pairs like this per 60Hz frame on a somewhat recent - e.g. post-2016 or so - mid-level gaming PC.
Of course trying to emulate D3D11 on Vulkan or D3D12 might have all sorts of issues that don't exist on a good "native" D3D11 driver.
Also somewhat related: in a GL shim it's typically still faster to update uniform data via the old glUniform*v calls instead of using uniform buffers, apparently most GL drivers lack the special fast-path optimizations of small D3D11 constant buffers, and the more abstract glUniform calls seems to give the driver more 'wiggle room' to implement an optimal upload path.
I mean sure you can do a generic layer and common denominator stuff, basically the kinda stuff webgpu or those rendering abstractions do and that's fair enough, it works well for simpler stuff!
The problem really is that if you do that you'll end up with random weird stalls or perf problems you can't really debug sadly :\
Re GL ye glUniform is usually much faster because you just do a lot less, you're writing a small amount of data directly into constant memory, it doesn't need resource tracking. UBOs / constant buffers come back to the theoretically slow but can-be-made-faster-with-driver-heuristics approach...i.e. if you see a small UBO which is only used as a UBO, you get the same direct constant bank upload
On NV it's quite fast, on AMD it's not really optimised, on intel just use buffersubdata because everything else sucks....
Very interesting and I think it makes sense to teach it, though I'd prefer something that works on MacOS
(what I don't get is why this page is breaking pgUp/pgDown)
These low level details have become unimportant since the ai can generate and debug them well imo.
The linked webpage refuses keyboard navigation e.g. cursors, pageup/down and home/end seem to do nothing.
Awful.
I think it's because of the webgl demo at top stealing focus
Specifically, it registers a keydown handler on the <html> that swallows everything.
Sounds like a great way to make sure students understand fundamental concepts.
[flagged]
[flagged]
TL;DR because WebGL 2 is basically equivalent to OpenGL ES 3.0, making it the most portable GPU API in the world by far, running on everything from laptops and phones to smart fridges and car head units.
Web browsers have also done a lot of behind-the-scenes work on compatibility. The behavior of WebGL 2 is much, much more consistent across platforms than native OpenGL [ES] ever was.
[dead]