Summary
In the business world, a moat is a quality of a business that makes it difficult for competitors to take that company’s profits. With how hard it is to train models for large language models (LLMs) and generative AI, it might seem like Anthropic, Open AI, and other LLM companies would have huge moats given the amount of compute it takes to build models. But open source models are quickly draining that moat, and now the only thing standing in the way of a customer using one of these models on their own hardware instead one from the larger companies is physical computing resources. [TerminalBytes] demonstrates a few of these models on personally owned computers to show the current state of the art. [TerminalBytes] started off running the 27B version of the Qwen3.8 on a Mac Studio with 256 GB of unified RAM, which is plenty for this task. But it’s also enough to benchmark a few different models. Qwen3.6 is compared to 3.8, and then the different quants of each model are also compared. Quants are compressed versions of models that need fewer bits to store weights, meaning that the same models can run in less memory with smaller losses in fidelity. Many of these quants run on machines with 32 GB of RAM or less, encompassing many average gaming PCs. There’s even a 1-bit quant that [TerminalBytes] tested which can easily run on a machine with 16 GB, although with mixed results. Keep in mind that this is just the current state of affairs with open LLMs. Future versions of these models are likely to optimize the number of tokens produced per unit time, or otherwise increase quality of responses while requiring less computer resources. We don’t really think that the ease of running local models will be the sole reason that the AI bubble pops, though. The fact that not every computer user is running Linux is proof enough of that. Why is the article mixing inference and training? Qwen still required lots of hardware to train. The moats are being destroyed by models you can run locally, even if you can’t train them yourself. That’s the point. The barrier to entry might be high, but it can be democratized across a population. (Imagine a co-op that spends the money to access a datacenter for a specific data processing task / and output.) You pay your 5 in, and you and everyone else in the coop get a full fledged model out. Sure, hours of GPU time to train are relevant to the final model, but that ends up a sunk cost. I have several LLMs installed on my several year old MacBook Pro. Admittedly some aren’t overly performant, but they do work. I still default to using chatGPT / codex for now. I’d be more than happy to run them locally however and forgo the token cost. (Which for me isn’t cost really, it’s a ceiling that I try to avoid hitting within a time period). If you’re concerned about the environmentalism aspect, you should give up on MMORPG, Massive FPS, etc. and get out of your mother’s basement. You should also not take up Golf as an alternative. You shouldn’t work for a multinational corporation either. As they say, there’s plenty of swords in the world, but there’s only one Sword of Fury in Rookgaard. If you know then you know what I mean. The whole AI thing is a mess but it won’t fall just because RAM is expensive. (I played Tibia for a very long time btw, almost 24 years.) True, and also quite the understatement. Perhaps worth clarifying that the main version [TerminalBytes] focuses on, the Q4_k_m quant, will run on a computer with either 32GB of shared memory, or a 24GB dedicated GPU, with performance remarkably close to the un-quantized version. That admittedly still limits it to pretty high-end machines, but you don’t need to run out and spend 7000 on a mac studio (or even 5 is hardly relevant. It’s readily obtainable. i would’ve said, a quick googling shows 24GB of VRAM means you’re looking at spening 2000 for 2x gpu over oculink, with 32gb vram, and a computer for tinkering with AI. Qwen 3.8 at q5 and 128k context pulls about mid 40’s for tokens/sec in generation. Don’t get me wrong, I don’t just let AI build my stuff, but it sometimes hits upon viable options, or what it proposes can be used with some tweaks. Exponential growth almost always leads to a crash, disaster or something. I have been running Qwen3.6 quantized the same way with llama.cpp server for a while. On an Intel Arrow Lake’s ARC GPU with 32GB of unified memory and limiting the context to 64k tokens, it’s nice and accurate for background tasks but my experience so far is it’s about 8x slower than the Apple M3 in this blog. You can easily send into la la land by including a Chinese sensitive string in the prompt however, often degrading sufficiently hard to require restarting the server. Don’t bother running under 4-bit. 1-bit quant sucks.