Microsoft and Nvidia are trying to take the AI Underground mainstream
Open weight models are getting more efficient and for many everyday tasks, their capabilities are more than enough.
Last year I wrote about the AI underground, the creators going deep into AI tools to go beyond just prompts to get the results and outputs they want. A lot of what is happening in that space was predicated not on what Google or OpenAI let creators do through their API or subscriptions, but on the open weight models those creators could run on their own machines leveraging platforms like ComfyUI.
Since I wrote that article, the AI underground, while not as advanced, is moving at similar pace of development as the mainstream frontier models from OpenAI and Anthropic. Models like Qwen 3.6 have been proving that, with a good enough setup, you don’t need a subscription to Claude or ChatGPT to get good reasoning results or to do basic agentic coding. Now, the most important software and AI hardware companies in the world are validating that point.
In the AI Underground ComfyUI has continued growing rapidly, being the preferred platform by most creators to generate and edit images using open weight models like Z-Image-Turbo and Flux.2-Klein. Open weight audio generation models like Stable Audio 3 allows generating serviceable SFX and music. New nodes are released from the community everyday, driven by creators vibe-coding solutions to their platform needs.
At the frontier, the best models from OpenAI, Anthropic, and Google are solving decades-old math problems, discovering vulnerabilities in the software everyone builds on, and building operating systems from scratch. The reasoning capabilities of the latest frontier models are remarkable at helping to solve complex, previously unsolvable problems. It’s like having a team of geniuses in every field at your prompting fingertips.
This is where a divergence starts to appear. For common, everyday tasks like cross-referencing a PDF, or adding a trivial feature to an app, the team of expensive token hoarding geniuses are not needed. Savvy users understand what tasks need the best available models and what tasks can be completed with less capable open weight models that are roughly a year or so behind in capability.
Why does this matter? Why not just use the frontier models via a subscription or API? The answer comes down to money and flexibility.
I’ve been working on my realtime storytelling project for a few months now and at the start I relied on the gpt-realtime model from OpenAI as the realtime voice service provider for the project. It wasn’t long until I realized that wouldn’t be sustainable. Gpt-realtime is one of the more expensive models to run because it has to work across different modalities (voice, text) seamlessly with enough reasoning capability to remain coherent.
It also wasn’t sustainable to build on. Building applications on top of external APIs is very common in software development but when OpenAI updated gpt-realtime, it completely changed how the characters in my project performed. I can’t build a storytelling project where the characters change personalities overnight due to changes in a backend service.
I had always planned to use open models for my project, but the experience with gpt-realtime brought more urgency so I made an early detour to the AI underground. Using ComfyUI and LocalAI as the AI inference engines, my project is now able to run entirely offline without the need for cloud AI services. I’ll share more about that later. Today, I want to focus on the hardware required to make it happen and how Nvidia and Microsoft are trying to lead the way so we don’t need the cloud for AI.
At Build, Microsoft’s developer conference last week Satya Nadella said something that made my ears perk up, “unmetered intelligence is right there at the edge”. Just before that, Jensen Huang, CEO of Nvidia said something notable as well at Computex, “In the future, AI wants to run on a device, too, because we want to have assistants running with us all the time.” Nadella was excited to talk about Windows running on Nvidia’s newly announced RTX Spark, a chip designed largely to do AI inference work. The emphasis Microsoft put on running AI locally, or at the edge as Microsoft likes to put it, was surprising to me. Not because local inference wasn’t already a big conversation point about AI, but because so much media attention has been going to how the data center buildout has been propping up the economy.
This makes sense from a competitive standpoint. Microsoft doesn’t have in-house models, having spent the last years investing heavily in OpenAI. And at this point, it would be hard for them to catch up to OpenAI, Anthropic, and Google - though they are trying. It makes sense for them to pivot to local on Windows where they can unlock agentic experiences on the Windows platform without inducing even more subscription fatigue. It also speaks to how capable systems are becoming and how efficient models are getting.
For me, being able to efficiently run AI locally meant a deep dive into the current state of GPUs and PC components. I currently have a 64GB M1 Max Macbook Pro which has for years now been able to take anything I throw at it…except for AI inference.
I’m not ready to upgrade to an M5, which has very decent AI inference performance and I didn’t want to build a whole new PC, especially with RAM prices as high as they are currently. I knew I needed to get a GPU and likely an Nvidia one because so much AI software is built with CUDA in mind. This all left me without many options. MacOS doesn’t support external GPUs but there are a few “AI Underground” projects.
One project by scottjg caught my eye. Basically, he was able to pass an external GPU connected to the mac to a Linux virtual machine. Linux has great support for Nvidia GPUs, after all that’s what most of the servers in the cloud are running. I snagged an RTX PRO 4000 Blackwell workstation card, threw it into an eGPU enclosure and with a tiny bit of setup, I was off to the races being able to get high token per second AI generation.
The AI Underground is going mainstream. With big tech promising more powerful PCs and capable open weight models being developed, soon I won’t have to find a hobby project and jury-rig a VM to be able to get good inference performance without breaking the bank. We’re going to see a lot more interesting experiences as AI intelligence moves from the cloud to the devices we use everyday. This will unlock new pathways of creative expression and change how we use our devices.