Last time (Getting Voice Onto a CPU) I got Fulloch’s CPU tier speaking and listening with zero PyTorch: Qwen3-ASR on ONNX for the ears, Kokoro-82M on ONNX for the voice. Since then I have created a comparison video showing this setup running the Qwen3-0.6–ASR model on my 2022 Macbook Air M2 with 8GB and on my development PC, an AMD Ryzen 9 7900 with 32GB RAM.
Kokoro is fast but it’s a fixed catalogue of voices, no cloning, and it skips words it can’t pronounce. The state-of-the-art open text-to-speech (TTS) with voice cloning is Qwen3-TTS, so the question was whether it could run acceptably on CPU too.
The ONNX quantisation path#
Qwen3-TTS is an autoregressive codec LM: a talker (28-layer Qwen3 LM emitting one codebook per frame) feeding a code predictor (15 sequential calls per frame filling the remaining codebooks) feeding a vocoder. Trying to run this on CPU is tricky, especially due to the code predictor running 15 sequential calls. Running this at full fp32 size gave as a real-time factor (RTF) of around 4.7 on the development PC. So 1 second of audio would take 4.7 seconds to generate. You can forget about using this for a real-time voice assistant. I won’t even bother talking about the initial results from the M2.
I then proceeded trying to see if quantisation of the different parts could get the ONNX model speed down to a usable RTF…
talker_decode(the 28-layer body): int4MatMulNBitswas the first thing to show promise, doing this shrinks the weights by 6.4× for this section, ~1.5× faster per step, no quality loss from our statistical tests.code_predictor(the fine codebook detail): This was the one that was really bottlenecking on the CPU but int4 audibly degrades the output. This makes sense as the job of this section is to crispen up and sharpen what the talker_decode produces. fp16 restored quality but cost speed and seemed to be upcasted to fp32 on the CPU anyway. A later QDQ dynamic-quant int8 build matched fp16 quality on paper and took the RTF from 2.4–3.1× down toward 1.4×, that statistical tests looked good but it still needed me to hear it properly in real-world examples.vocoder: stays fp32. This section creates the final audio signal waveform, any quantisation here just broke the output completely.
So 1.4x on the development system might be workable if we could chunk and overlap some of the processing. So break a sentence up into parts where natural pauses might happen and then process them concurrently to simulate a real-time TTS.
On Apple Silicon (M2), the picture got worse, RTF ranging 5–54× on an 8GB machine that was visibly RAM-starved during the runs. The int4 win on the talker_decode step also didn’t transfer to the M2 and it wasn’t clear anymore what was saving storage space vs increasing speed across different CPU’s. “Same models on both machines” was never going to work with this setup.
CrispASR: a full-pipeline alternative#
CrispASR is a mature ggml/GGUF runtime with ready-made Qwen3-TTS and Qwen3-ASR conversions. We are already using the GGUF models from Unsloth for our small language model (SLM) AI tests, so why not use it for the voice as well.
Running our statistical tests over some samples gave good enough results: CrispASR’s Q8_0 TTS (1.5–2.1× RTF) beat ONNX’s default (2.4–3.1×) but lost to our quantised ONNX build (1.4×). I tested the ASR here as well and CrispASR matched the quality of our existing ONNX setup but ran slower.
So, the statistical tests still pointed to our quantised ONNX build as the best model and it might just be usable on our development PC, but probably not the M2 in its current form.
And then I heard it…#
Statistical tests and RTF numbers are good for sorting through builds, but in the end whether a human can stand to listen to the output is the only real metric that matters. Every one of the ONNX quantisations was audibly broken. Robotic, stuttering, weird auto-tune sounding voices that were painful to listen to or actually made you worried they were reciting some pagan text to summon demons from the pits of hell.
Every ONNX quantisation build that looked good on a stopwatch was unusable to a human ear.
The GGUF models from CrispASR were the only models that produced usable audio. But they were still too slow on CPU. Buuut, if they were same quality as the full models and they took up a lot less VRAM, these were swap in candidates for a system with a smaller, older GPU. Still an interesting proposition…
One limitation. The CrispASR backend currently can’t share the process with the SLM on GPU in our current setup. This is something that needs further testing and exploration. So for now these models are experimental options on the CPU only.
And don’t forget the speed tests. CrispASR Qwen3-O.6B RTF on CPU is 1.5–2.1×. So it takes one-and-a-half to two seconds of compute per second of speech. That’s too slow for real assistant usage on CPU. Even doing some smart chunking and concurrent processing doesn’t prevent those few seconds of “dead air” while a user waits for a response and every full stop and comma created a too long pause that was not natural. This just can’t replace the Kokoro’s real-time voice output.
So we are back where we started#
The CPU tier’s TTS story ends where it started with Kokoro-82M ONNX. It is a great model and deserves its spot here. The only things missing are voice cloning and some longer, less common words being skipped. So, good enough for now.
If you have a GPU with only 8GB VRAM, you could run the CPU tier of Fulloch for ASR and TTS. Then load the brilliant Qwen3.5-9B-MTP-GGUF from Unsloth onto the GPU. The UD-Q4_K_XL will fit with around 12K of context, which should be more than enough for Fulloch to run comfortably. The full stack running in realtime on a PC with a “cheap” consumer-grade GPU. Not bad, not bad at all!
The next tests will be on the ASR and TTS GGUF models from CrispASR on the GPU. I’m thinking we could get the full stack GPU resident in under 12GB VRAM. Full SoTA Qwen3 1.7B ASR and TTS with a Qwen3-9B LLM orchestrating in real-time on an RTX 3060. Now that would be amazing!
All code is at GitHub, fulloch. The discussion regarding int4 speed is at ONNX Runtime #23004.