Summary
Consumer hardware has come a long way, to the point where some data-center workloads are now being replicated at desktop scale. To play with local AI, desktop GPUs are ideal companions for complex parallel processing, especially RTX cards with dedicated AI hardware. However, a bulky card is overkill for smaller AI tasks that don’t need a separate GPU. That’s where newer specialized components come in to handle these lighter workloads, like a dedicated AI accelerator: a PCIe device that looks like a GPU and has the same form factor, but it doesn’t output video and is built to handle lighter AI workloads much more efficiently compared to a GPU. These AI accelerator cards had a huge wave around 2023/24, but a market for them never really formed, and the hype quickly faded because they were trying to fill a gap already covered. What a PCIe AI accelerator is A GPU-shaped card that doesn’t draw a frame AI accelerators are specialized expansion cards that look like GPUs and are explicitly designed to run AI inference. They don’t have input or output ports; they sit in a PCIe slot or come as M.2 devices. The processors inside an AI accelerator, in particular, handle parallel processing and matrix multiplication, operations used in AI workloads. An AI accelerator typically has onboard memory, but it’s not a replacement for a GPU; it’s meant to work alongside it by offloading AI inference to the accelerator while the GPU handles graphics and other AI tasks. Modern GPUs like the RTX series and newer RX cards already have dedicated AI hardware like NVIDIA Tensor Cores and AMD AI accelerators, so they themselves are technically AI accelerators. Overall, the term “AI accelerators” applies to GPUs, NPUs, and TPUs, but here it refers to dedicated PCIe AI cards. Who they’re actually for Low-power vision tasks that don’t need a full GPU Most AI accelerators are comparatively less powerful than a GPU. Even though they pack a punch relative to their power draw, they’re mostly advertised for light workloads like object detection, creating live models, transcription, or running small language models. On the other hand, they can also help with content creation by improving audio and video quality with background noise reduction using AI, image enhancements, and lighter video editing tasks, too. A GPU can already handle these workloads easily, but the pitch for AI accelerators is that they can handle the same mundane tasks that don’t require a full GPU’s attention, lowering the total power draw. A prime example of an AI accelerator that I’m talking about is the Hailo-8 PCIe, which packs up to 208 TOPS, depending on the number of processors inside. Individually, each processor inside outputs 26 TOPS while only consuming around ~2.5 W, making a PCIe card with up to 8 modules draw 20W in total. The pitch for an AI accelerator doesn’t make sense A PCIe AI accelerator can’t compete with a GPU at the same price The main selling point for these cards over a GPU is the power savings. However, if you compare the price of an AI accelerator against a GPU, the pitch doesn’t quite land. The Hailo-8 PCIe card, for example, goes around $400, which is an awful lot for just 206 TOPs. In contrast, the RTX 5060 retails for the same price even after the price hike and inflated MSRP. The GPU features a total of 614 TOPs — triple the amount compared to the Hailo-8, and only consuming 145–150W. Therefore, a separate AI accelerator on your desktop wouldn’t make sense if a GPU could already handle these tasks, unless you want to save a handful of watts that would barely amount to anything for your electricity bill. Where I see a practical application of AI accelerators is in robotics and other edge applications. Since there isn’t an option to install a GPU on a single-board computer, an M.2 AI accelerator would fit fine. Even then, the use case is minimal because, individually, these AI accelerators aren’t that fast, unless you opt for an add-in that fits into a GPU slot. Dedicated AI accelerators sound great, but don’t get tricked In their current state, AI accelerators rarely solve any problem, regardless of their scale. For the desktop, a GPU alone should be a no-brainer, or an NPU can do the same job as an AI accelerator. Even then, NPUs are considered useless. Software support is another problem, and there are no proper benchmarks to showcase the performance of AI accelerators. In contrast, Nvidia’s CUDA ecosystem is already too broad, and no developer will be convinced to buy a separate chip because there are no equivalent tools. Unfortunately, I don’t see proper support even coming in the near future because AI accelerators never really took off despite the initial hype.