Getting Voice Onto a CPU: CPU-Only ASR and TTS for Fulloch

Fulloch, my fully local home voice assistant, was built around the RTX 5060 Ti graphics card. The assistant runs Qwen3 1.7B ASR and TTS plus the Qwen3.5 9B language model, squeezing everything out of the 16GB VRAM available. It works really well, but a graphics card humming away all day to turn off the lights is a hard sell for even the most hardcore Home Assistant enthusiast.

I had started my voice assistant journey trying to repurpose an old PC with an RTX 1050 Ti, a GPU that I didn’t even use for any of those first experiments, like it would have been able to do anything anyway. With everything I had learnt since those early days and advancements made in the last 12 months, could a genuine CPU version of Fulloch be feasible?

I knew by now that the LLM just was never going to work on a CPU setup (see Watching a Voice Assistant Get Dumber), but I could get regex to cover a lot of the most common use cases. Anything more advanced could be funneled off to an OpenAI endpoint running on another GPU server in your house or somewhere else (Fulloch might need to be callled “Parloch” if it’s only Partially local now). If the OpenAI endpoint isn’t available you can still turn off and on lights, set timers, and play/stop your music, if it is running you can use all the more advanced features, best of both worlds!

This post is about trying to get speech in and out of a computer using only the CPU (developed and tested on an AMD Ryzen 9 7900, 32GB RAM).

Legacy Rerun: Let’s check how those old CPU models work in our latest setup

#

I’d already tested trying to run the voice assistant on an old CPU computer in the early experiment days and still had the models and code for that setup.

After much testing back then I had landed on Moonshine for ASR and Kokoro-82M for TTS. Both models were very tiny and incredibly good for their size. You could run them on an old mini pc no problem. I felt the ASR was better than Whisper models that were still commonly used in Home Assistant setups and the speech from Kokoro was leagues ahead of Piper.

• The ASR wrapper in our GPU setup exposed the same context to the ASR, so we could easily plug in our old Moonshine script, the only thing we lost was the context bias that helped pickup the “Hey Atticus” wakeword (and any other custom entity names) in our Qwen3 models.
• The TTS module provided the same surface for the Kokoro model, but we lost the voice cloning option and had to use the available Kokoro models for voice.

The GPU setup had been built on these initial CPU experiments so I knew the small models were usable.

But this first cut still depended on torch. transformers pulls it in for Moonshine, and the Kokoro build was torch-native. I wanted a slim final image for our CPU setup, so wanted to see how we could minimise the footprint of torch and still run the best ASR and TTS models possible.

GPU to CPU: Passing on the torch

#

Kokoro gave good voice but missed complicated words like “meteorological”, just skipping them completely. But the quality was generally good and for most tasks good enough.

Moonshine base model was also “OK” but the missing context bias meant the wakeword in transcript method we were using for our GPU setup just made it much harder for the wakeword to be picked up. Yelling at your voice assistant five times in a row to brighten the living room lights and then have it misunderstand “brighten” for “brighton” when you are standing just a few metres away from the light switch doesn’t make sense anymore. Casually asking the voice assistant to turn off all the upstairs lights as you go down the stairs and it just does it makes more sense. The ASR is actually much more important than the TTS in a home voice assistant.

We needed better ASR and the best open-source ASR is Qwen3, that was clear from all the tests we had done on the GPU. So, how can we run it on a CPU?

The answer was ONNX Runtime: export the model graph once, then run it with a lean runtime that has fast CPU kernels and zero deep-learning-framework baggage. Searching Hugging Face showed that ONNX versions of the Qwen3 ASR models existed and that Kokoro also had an ONNX version.

Migrating both backends off torch is what could really make the CPU image slim, its entire ML footprint became onnxruntime plus a couple of small helpers (librosa for mel features, tokenizers, misaki for grapheme-to-phoneme), instead of torch + transformers + CUDA.

Backends in Fulloch are selected through a registry (core/backends.py), so swapping the torch backends for ONNX ones and the GPU Qwen models for CPU ONNX ones is a config change, not a rewrite. The two images, GPU :latest and CPU :cpu, share the same orchestrator and differ only in which backend modules they can load.

So, can we get the context bias and improved speech recognition of Qwen3 ASR on our old CPU mini PC?

Speech in: from Moonshine to Qwen3-ASR-0.6B on ONNX

#

The Moonshine starting point

#

The first CPU ASR I ported after first experiments with Whisper was Moonshine, a purpose-built, tiny English ASR that runs happily on edge hardware. It works, and it’s still in the repo as the option for very constrained edge devices, in both base (~62M) and tiny (~27M) sizes. But it doesn’t have any context biasing, so wakeword pickup is harder and it mistranscribes often enough to be annoying for anything but the simplest commands.

Fulloch’s wakeword is “Hey Atticus”, and this isn’t a common dictionary word. Moonshine will cheerfully transcribe it as “attic us”, “Oticus”, “Adicus”, etc. and the wakeword regex misses. On the GPU tier I solve this by feeding the ASR a context hint (“Technical terms: Atticus, …”) that biases the decoder toward the right spelling. Moonshine doesn’t have this ability, so you need to make sure you pronounce Atticus as clearly as possible for it to work.

Switching to Qwen3-ASR-0.6B

#

I found an int8 ONNX export of the 0.6B variant (Daumee/Qwen3-ASR-0.6B-ONNX-CPU) on Hugging Face. This could be the one. It’s the same Qwen3 ASR chat-template architecture as the big GPU model, so the context-biasing wakeword detection would work on CPU. Plus it’s multilingual (30 languages), something I am keen to test in some future version of Fulloch.

core/asr_onnx.py adapts the model’s bundled ONNX inference into the same QwenASRPipelineWrapper the GPU backend uses, Whisper-compatible mel features computed with librosa/numpy, the encoder/decoder ONNX sessions, and the special-token handling for Qwen3 ASR’s audio frames. From the orchestrator’s point of view it’s just another streaming generator over the mic queue. No torch anywhere.

Performance

#

On the dev CPU box it transcribed at RTF ~0.16–0.21 (real-time factor, so about 5x faster than real-time).

The wakeword trap

#

Yes, we got context biasing and our 0.6B Qwen3 ASR would reliably pickup the wakeword… The problem was it always picked up the wakeword! When the model is acoustically uncertain (e.g., it hears a cough), it has to choose between trusting the audio and trusting the context bias

The 0.6B model leaned so hard on the prior that it started hallucinating the wakeword out of every little cough, sneeze or tap on the table. If we turned the bias off, the model scattered the spelling into variations like “Adikis” or “Eddie Kiz,” and the assistant ignored us. We were trapped: we could either have a system that woke up to coughs, or a system that ignored our commands. There was no nuance the knob was either fully on or fully off.

Band-aids and the dedicated model detour

#

My first instinct was to try to put guards and filters all around the wakeword detection. I tried several options:

• Bare-wakeword loudness gate: Automatically rejects a command-less wakeword if it is at or below the room’s background noise baseline.
• Unbiased re-transcribe: Re-runs the ASR on the same buffer with the bias blanked out; if the wakeword vanishes, it was an echo.
• Prompt-echo marker: Drops any transcript that literally hallucinates the “technical terms” scaffolding.

These worked, and were good guards to have in place anyway but they were band-aids on the capabilities of our 0.6B model. It forced me to reconsider a path I had previously rejected and still wasn’t happy about: adding a dedicated wakeword model.

This is what most voice assistants do and it should kill the hallucination problem, while also dropping the always-on CPU compute for ASR. I evaluated several candidates that could run without PyTorch on our slim CPU image:

openWakeWord was still the recommended tool. I could technically train a new wakeword model for “Hey Atticus” using the TTS synthetic data, but I would also need to add in some real recordings and either simulate or run more recordings with background noise and mumblings, coughs etc. This could be work for a later date, and would be a big project in itself. But before I did any implementation work I went back to my early experiment notes. Those early experiments had used the openWakeWord model way back then and the results were disappointing… “Alexa” was the only wakeword that worked reliably and the accuracy in picking it up was about as good as what I already got with Moonshine ASR using my custom “Hey Atticus” wakeword. This wouldn’t be the upgrade I thought it would be, I would just be back where I started.

Until a new open-source wakeword model comes out or a big upgrade to openWakeWord is released, I couldn’t justify retrying it.

Brute forcing with a bigger ASR

#

Before giving up and just falling back to Moonshine, I wanted to test one last theory: what if we just brute-forced the problem by running a stronger ASR model on the CPU?

I found a community export of the 1.7B Qwen3 model (andrewleech/qwen3-asr-1.7b-onnx). It wasn’t a drop-in replacement—it unified the encoder, used int4 quantization, and required some wrapper adaptation, but it ran entirely on ONNX Runtime and theortically a CPU.

I spent an afternoon writing an adapter and benched the 1.7B int4 model against our current 0.6B int8 model using synthetic generated voice clips from 31 different TTS voice models:

The results were staggering. The 1.7B model was flawless across every voice tested. While it cost about 1.5× the latency of the smaller model, it still ran ~4.6× faster than real-time on the dev box CPU.

More importantly, I tested it live with a microphone. When I said “Hey Atticus,” it transcribed it perfectly without scattering. When I made gibberish sounds or coughed, it transcribed nonsense and suppressed it, I had zero hallucinated wakewords with wakeword bias in the context. I was getting the state of the art open-source ASR on a CPU and wakeword biasing was now working as well as it did on the GPU. It needed a solid 5-6GB of RAM just for the ASR, but as this was the key to a good voice assistant I saw it as a necessary resource requirement.

Speech out: the Kokoro latency saga

#

After all the back and forth with setting up the ASR, I thought the TTS would be an easy win. Kokoro was astoundingly good for such a small model, shouldn’t need much work to just plug that back into our CPU tier. But I was wrong…

The model itself never changed: Kokoro-82M, a small, high-quality model with built-in named voices (no cloning but good voice selection). It had started as the torch build in initial tests, but I wanted to move it to the ONNX build to drop that torch dependency from our CPU image.

It sounded great. The problem was latency: once on ONNX, spoken replies had ~5 seconds of dead air before any audio. That’s worse than the GPU pipeline, and subjectively worse than the torch Kokoro had felt even on the CPU. Was I going to need some special “streaming” Kokoro ONNX build?

No. It turned out to be two independent, self-inflicted causes, and a third bug I created while fixing them.

Wrong quant

#

Kokoro ships several ONNX variants. I’d defaulted to the int8 model_quantized.onnx on the assumption that “smaller = faster on CPU”. That’s wrong. The dequantisation (QDQ) overhead dominates, and int8 came out ~5x slower than fp16/fp32. On CPU, fp16 kernels just upcast to fp32 for the actual compute, so fp16 ≈ fp32 speed at half the disk size.

Benchmark on a single 8.2s utterance, best of 3 after warmup, CPUExecutionProvider:

Just changing the default away from int8 took RTF from 0.76 → 0.14, a 5x speedup.

Full sentence synthesis

#

Kokoro is non-autoregressive: it does one forward pass per input span, so time-to-first-audio equals the synth time of whatever you hand it first. I was handing it the whole sentence. A long opening sentence meant several seconds of silence before the first sample played.

I briefly investigated the msgflux/Kokoro-82M-streaming-onnx build, thinking I needed real streaming. It turned out its “streaming” is just input-text chunking with fixed token buckets, exactly the thing I could do myself, plus a quant choice I’d already made. Not worth a new dependency and fixed-bucket padding.

The fix was to split the input on clause/sentence punctuation and synthesise one clause at a time on the existing producer/consumer worker thread, so the first clause plays while the rest renders.

An honest aside: I first over-engineered this with a “ramp”: a tiny opening fragment growing to a steady-state cap. I’d even built out specific config levers for it before realizing it was the wrong path:

knob	default	meaning
first_fragment_words	5	opening fragment size → first-audio latency
fragment_ramp_words	2	growth per fragment (buffer outruns synth)
fragment_max_words	14	steady-state cap (prosody/throughput)

That was a latency mechanism for the slow int8 model. Once fp16 landed and synth was well under real-time, the fine-grained fragments started causing audible mid-sentence gaps and choppy speech. I removed the ramp out and reverted to a plain clause split. Simpler code, gap-free playback. The lesson: don’t keep optimisations whose premise you’ve already removed.

The combined effect on a multi-sentence reply (~15s of audio):

From 5.6 seconds of dead air to under half a second.

That is not a number

#

Happy with fp16, I went to pre-render a demo clip for each of Kokoro’s 28 voices for the setup wizard’s voice picker. Several came back as harsh noise instead of speech, including af_heart, which was the default voice and recommended voice for Kokoro. I thought it might just be some random error so reran it and exactly the same errors came out in the same places, this was a built-in failure.

Further analysis revealed that the fp16 ONNX model emits NaN samples for 11 of the 28 voices. Those voices’ style vectors push the fp16 graph out of numeric range; the output is 36–65% NaN, and on write soundfile’s PCM-16 saturates NaN to full-scale, which reads as clipping/noise. Some real-world testing showed the same voices producing the same garbage in live playback.

The bad voices could be identified by clip% ≫ 0 with RMS ≈ −2 to −4 dBFS indicating a mostly-saturated waveform, versus the clean voices sitting around −18 to −27 dBFS. You can see this clearly in the sample from a raw signal analysis below:

So, checking all the voice models that come with Kokoro we see the below when running the fp16 ONNX version:

To confirm that NaN-free didn’t automatically mean correct speech, I ran an intelligibility cross-check on the voices that looked like they worked. Each clip was transcribed with the already on-disk Qwen3-ASR ONNX (resampled 24k→16k) and scored against the source text using difflib similarity (ratio) and the percentage of expected words present (recall):

The 17 clean voices were intelligible, while the broken control scored terribly because the ASR only captured the clean intro before the NaN noise completely overtook the audio. All voices skipped over the difficult to pronouce meteorological word, that was a limitation of the Kokoro model not an fp16 ONNX issue.

Get our voices back

#

My first fix was to trim the voice list to the 17 clean ones and just allow those with our CPU image. But that traded 40% of the voice catalogue and a correctness bug for ~160 MB of saved download. Also, on CPU, fp16 buys no speed, it upcasts to fp32 to compute anyway. The exclusion list was also never really sound: the fp16 NaN was input-dependent and I hadn’t tested on lots of different transcripts, so a “clean” voice could still NaN on an unlucky phrase and real-world testing showed it to be very flaky. “Clean” voices were not necessarily clean and could NaN at anytime.

So I re-ran the analysis on the fp32 model.onnx and got the below results:

Clean across the board! The identical ~7% WER across every voice (measured by transcribing each clip back with the on-disk Qwen3-ASR ONNX) is the key signal: the recogniser understands every voice equally, so none is acoustically degraded. The residual 7% was the ASR consistently dropping the one deliberately-hard word, “meteorological” included in the transcript, the Kokoro limitation we already knew about.

The cost of fp32 over fp16: RTF 0.15 vs 0.14 (within noise, both far under real-time) and ~326 MB vs ~163 MB download. That ~160 MB is the entire price for zero NaNs and all 28 voices.

Where we landed

#

Two models, both in ONNX, both comfortably real-time on a CPU, and zero PyTorch in the image:

flowchart LR
    Mic[(Microphone)] --> ASR[Qwen3-ASR-1.7B
ONNX, CPU]
    ASR -- "'hey Atticus'
regex match" --> Route{Regex
fast-path}
    Route -- "common commands" --> Tools[Tools / Home Assistant]
    Route -- "everything else" --> LLM[Regex-only
or remote LLM]
    LLM --> TTS[Kokoro-82M
ONNX, CPU]
    Tools --> TTS
    TTS --> Speaker[(Speaker)]

At RTF ~0.15–0.21 on my dev CPU, even a 3–4x slower mini pc should stay well under real-time. I need to do lots more testing and would love to hear how this runs on other peoples setups.

All code is at GitHub, fulloch. The CPU image ships as :cpu; pick the default tier in the first-run setup wizard and it downloads the recommended models for you.