Telling stories at the speed of now with ComfyUI Realtime
I've been building in ComfyUI an open-source way to run realtime conversational AI locally with open-weight models.
In January I wrote that I was leaving slow fiction behind to tell real-time stories. My vision was “stories being distributed and consumed as quickly as they’re made.” It harkens back to how stories through history were passed down around the campfire, and the stories my siblings and I would tell each other under the moonlight when we would occasionally lose power at home.
I started prototyping a storytelling app that orchestrates AI characters to act out dramatic scenes under the real-time performance and plot direction of the human storyteller. Along the way, I set on the project an important constraint. I didn’t want it to be limited by cost-per-token limits or inevitable changes imposed by cloud AI and compute providers.
So a few months ago I built ComfyUI Realtime, a custom node package for ComfyUI. The idea was simple: just like OpenAI offers real-time conversational speech-to-speech interaction via its Realtime API, I could have the same thing locally through ComfyUI using open-weight AI models. ComfyUI wasn’t designed for real-time in this way, but that’s what makes ComfyUI so popular within the AI creative community. You can mold it to be whatever you need it to be.
ComfyUI Realtime takes advantage of open-weight LLM, TTS, and STT models. I wanted it to be as malleable as possible. So if I wanted to swap out the intelligence driving the conversational AI (like switching between Gemma and Qwen), I could. And I wanted to have flexibility with TTS models as well, so I could switch between VibeVoice, Pocket TTS, or any of the many TTS models out there.
I wanted to be able to run it without needing beefy hardware, so I made sure ComfyUI Realtime workflows could run on my old M1 MacBook Pro without breaking a sweat. Serendipitously, since the initial build of ComfyUI Realtime, a great open-source project called audio.cpp sprouted and I’m in the middle of incorporating it into ComfyUI Realtime for even more performance, flexibility, and capability.
Even though intelligent real-time conversational systems are just one of the innumerable applications for AI technology, they illustrate what AI is foundationally enabling at a faster rate across all domains. AI is driving the broadest latency collapse in the history of humanity.
Humans despise latency when we are afforded the means to reduce it. Amazon is one of the most successful companies in the world because it exploits our impulse to get what we want NOW! In line at a theme park? Pay a premium to skip the line and ride NOW! Buy the fastest broadband internet speed to instantly watch movies and listen to music NOW! There is no concept of “fast enough” for us.
In my organization, I’ve seen AI become a kind of super-serum that my colleagues use to produce faster. Iteration loops get tighter. Efficiency improves and more projects get completed more quickly. Again, when there is a lower latency option, people tend to gravitate there.
Creativity is not immune to the low-latency craving. We graduated from paints to photography as the preferred medium for freezing moments in time because photography is faster and more efficient. Even within that medium some professionals today prefer to use their phones rather than their DSLRs to capture images because the phone makes it easier and faster to share.
Storytelling is natively low-latency. We dress it up with mediums that make it slow and asynchronous (and there is nothing inherently wrong with that), but at the core story only requires the voice of the person telling the story and the ear of the audience to receive it. Improv, theater, stand-up, even live-streaming… these are native storytelling mediums. And now with generative AI, mediums like film are evolving toward being more low-latency.
I built ComfyUI Realtime for my storytelling app with that idea as a driving motivation. Telling stories at the speed of now is not enabled by technology. It is innate to our species and how we communicate. But over time AI technology will (across any medium) enable experiencing all of our stories natively again… if we choose to experience our stories in that more human way.