[{"content":"","date":"11 July 2026","externalUrl":null,"permalink":"/tags/asr/","section":"Tags","summary":"","title":"Asr","type":"tags"},{"content":"","date":"11 July 2026","externalUrl":null,"permalink":"/categories/","section":"Categories","summary":"","title":"Categories","type":"categories"},{"content":"","date":"11 July 2026","externalUrl":null,"permalink":"/tags/cpu/","section":"Tags","summary":"","title":"Cpu","type":"tags"},{"content":"","date":"11 July 2026","externalUrl":null,"permalink":"/tags/edge/","section":"Tags","summary":"","title":"Edge","type":"tags"},{"content":"Fulloch (Fully Local Home) is a private voice assistant that runs entirely on your home computer: wake word, speech-to-text, language model, and voice all on-device. No cloud, no subscriptions, nothing leaving your network.\n","date":"11 July 2026","externalUrl":null,"permalink":"/","section":"Fulloch","summary":"","title":"Fulloch","type":"page"},{"content":"Last time (Getting Voice Onto a CPU) I got Fulloch\u0026rsquo;s CPU tier speaking and listening with zero PyTorch: Qwen3-ASR on ONNX for the ears, Kokoro-82M on ONNX for the voice. Since then I have created a comparison video showing this setup running the Qwen3-0.6–ASR model on my 2022 Macbook Air M2 with 8GB and on my development PC, an AMD Ryzen 9 7900 with 32GB RAM.\nYour browser does not support the video tag. Comparing the latency of ASR-TTS between a Mac M2 and a Dev PC Kokoro is fast but it\u0026rsquo;s a fixed catalogue of voices, no cloning, and it skips words it can\u0026rsquo;t pronounce. The state-of-the-art open text-to-speech (TTS) with voice cloning is Qwen3-TTS, so the question was whether it could run acceptably on CPU too.\nThe ONNX quantisation path # Qwen3-TTS is an autoregressive codec LM: a talker (28-layer Qwen3 LM emitting one codebook per frame) feeding a code predictor (15 sequential calls per frame filling the remaining codebooks) feeding a vocoder. Trying to run this on CPU is tricky, especially due to the code predictor running 15 sequential calls. Running this at full fp32 size gave as a real-time factor (RTF) of around 4.7 on the development PC. So 1 second of audio would take 4.7 seconds to generate. You can forget about using this for a real-time voice assistant. I won’t even bother talking about the initial results from the M2.\nI then proceeded trying to see if quantisation of the different parts could get the ONNX model speed down to a usable RTF…\ntalker_decode (the 28-layer body): int4 MatMulNBits was the first thing to show promise, doing this shrinks the weights by 6.4× for this section, ~1.5× faster per step, no quality loss from our statistical tests. code_predictor (the fine codebook detail): This was the one that was really bottlenecking on the CPU but int4 audibly degrades the output. This makes sense as the job of this section is to crispen up and sharpen what the talker_decode produces. fp16 restored quality but cost speed and seemed to be upcasted to fp32 on the CPU anyway. A later QDQ dynamic-quant int8 build matched fp16 quality on paper and took the RTF from 2.4–3.1× down toward 1.4×, that statistical tests looked good but it still needed me to hear it properly in real-world examples. vocoder: stays fp32. This section creates the final audio signal waveform, any quantisation here just broke the output completely. So 1.4x on the development system might be workable if we could chunk and overlap some of the processing. So break a sentence up into parts where natural pauses might happen and then process them concurrently to simulate a real-time TTS.\nOn Apple Silicon (M2), the picture got worse, RTF ranging 5–54× on an 8GB machine that was visibly RAM-starved during the runs. The int4 win on the talker_decode step also didn’t transfer to the M2 and it wasn’t clear anymore what was saving storage space vs increasing speed across different CPU’s. \u0026ldquo;Same models on both machines\u0026rdquo; was never going to work with this setup.\nCrispASR: a full-pipeline alternative # CrispASR is a mature ggml/GGUF runtime with ready-made Qwen3-TTS and Qwen3-ASR conversions. We are already using the GGUF models from Unsloth for our small language model (SLM) AI tests, so why not use it for the voice as well.\nRunning our statistical tests over some samples gave good enough results: CrispASR\u0026rsquo;s Q8_0 TTS (1.5–2.1× RTF) beat ONNX\u0026rsquo;s default (2.4–3.1×) but lost to our quantised ONNX build (1.4×). I tested the ASR here as well and CrispASR matched the quality of our existing ONNX setup but ran slower.\nSo, the statistical tests still pointed to our quantised ONNX build as the best model and it might just be usable on our development PC, but probably not the M2 in its current form.\nAnd then I heard it… # Statistical tests and RTF numbers are good for sorting through builds, but in the end whether a human can stand to listen to the output is the only real metric that matters. Every one of the ONNX quantisations was audibly broken. Robotic, stuttering, weird auto-tune sounding voices that were painful to listen to or actually made you worried they were reciting some pagan text to summon demons from the pits of hell.\nEvery ONNX quantisation build that looked good on a stopwatch was unusable to a human ear.\nThe GGUF models from CrispASR were the only models that produced usable audio. But they were still too slow on CPU. Buuut, if they were same quality as the full models and they took up a lot less VRAM, these were swap in candidates for a system with a smaller, older GPU. Still an interesting proposition…\nOne limitation. The CrispASR backend currently can\u0026rsquo;t share the process with the SLM on GPU in our current setup. This is something that needs further testing and exploration. So for now these models are experimental options on the CPU only.\nAnd don’t forget the speed tests. CrispASR Qwen3-O.6B RTF on CPU is 1.5–2.1×. So it takes one-and-a-half to two seconds of compute per second of speech. That\u0026rsquo;s too slow for real assistant usage on CPU. Even doing some smart chunking and concurrent processing doesn’t prevent those few seconds of “dead air” while a user waits for a response and every full stop and comma created a too long pause that was not natural. This just can’t replace the Kokoro’s real-time voice output.\nSo we are back where we started # The CPU tier\u0026rsquo;s TTS story ends where it started with Kokoro-82M ONNX. It is a great model and deserves its spot here. The only things missing are voice cloning and some longer, less common words being skipped. So, good enough for now.\nIf you have a GPU with only 8GB VRAM, you could run the CPU tier of Fulloch for ASR and TTS. Then load the brilliant Qwen3.5-9B-MTP-GGUF from Unsloth onto the GPU. The UD-Q4_K_XL will fit with around 12K of context, which should be more than enough for Fulloch to run comfortably. The full stack running in realtime on a PC with a “cheap” consumer-grade GPU. Not bad, not bad at all!\nThe next tests will be on the ASR and TTS GGUF models from CrispASR on the GPU. I’m thinking we could get the full stack GPU resident in under 12GB VRAM. Full SoTA Qwen3 1.7B ASR and TTS with a Qwen3-9B LLM orchestrating in real-time on an RTX 3060. Now that would be amazing!\nAll code is at GitHub, fulloch. The discussion regarding int4 speed is at ONNX Runtime #23004.\n","date":"11 July 2026","externalUrl":null,"permalink":"/posts/getting-voice-onto-a-cpu-part-2/","section":"Posts","summary":"ONNX quantisation tuning produced great RTF numbers and unusable audio. A GGUF runtime with worse numbers won on the only test that mattered.","title":"Getting Voice Onto a CPU (Part 2): More Attempts at Qwen3 TTS","type":"posts"},{"content":"","date":"11 July 2026","externalUrl":null,"permalink":"/tags/gguf/","section":"Tags","summary":"","title":"Gguf","type":"tags"},{"content":"","date":"11 July 2026","externalUrl":null,"permalink":"/tags/local-ai/","section":"Tags","summary":"","title":"Local-Ai","type":"tags"},{"content":"","date":"11 July 2026","externalUrl":null,"permalink":"/tags/onnx/","section":"Tags","summary":"","title":"Onnx","type":"tags"},{"content":"","date":"11 July 2026","externalUrl":null,"permalink":"/posts/","section":"Posts","summary":"","title":"Posts","type":"posts"},{"content":"","date":"11 July 2026","externalUrl":null,"permalink":"/categories/projects/","section":"Categories","summary":"","title":"Projects","type":"categories"},{"content":"","date":"11 July 2026","externalUrl":null,"permalink":"/tags/","section":"Tags","summary":"","title":"Tags","type":"tags"},{"content":"","date":"11 July 2026","externalUrl":null,"permalink":"/tags/tts/","section":"Tags","summary":"","title":"Tts","type":"tags"},{"content":"","date":"11 July 2026","externalUrl":null,"permalink":"/tags/voice-assistant/","section":"Tags","summary":"","title":"Voice-Assistant","type":"tags"},{"content":"","date":"11 July 2026","externalUrl":null,"permalink":"/tags/voice-cloning/","section":"Tags","summary":"","title":"Voice-Cloning","type":"tags"},{"content":"Fulloch, my fully local home voice assistant, was built around the RTX 5060 Ti graphics card. The assistant runs Qwen3 1.7B ASR and TTS plus the Qwen3.5 9B language model, squeezing everything out of the 16GB VRAM available. It works really well, but a graphics card humming away all day to turn off the lights is a hard sell for even the most hardcore Home Assistant enthusiast.\nI had started my voice assistant journey trying to repurpose an old PC with an RTX 1050 Ti, a GPU that I didn\u0026rsquo;t even use for any of those first experiments, like it would have been able to do anything anyway. With everything I had learnt since those early days and advancements made in the last 12 months, could a genuine CPU version of Fulloch be feasible?\nI knew by now that the LLM just was never going to work on a CPU setup (see Watching a Voice Assistant Get Dumber), but I could get regex to cover a lot of the most common use cases. Anything more advanced could be funneled off to an OpenAI endpoint running on another GPU server in your house or somewhere else (Fulloch might need to be callled \u0026ldquo;Parloch\u0026rdquo; if it\u0026rsquo;s only Partially local now). If the OpenAI endpoint isn\u0026rsquo;t available you can still turn off and on lights, set timers, and play/stop your music, if it is running you can use all the more advanced features, best of both worlds!\nThis post is about trying to get speech in and out of a computer using only the CPU (developed and tested on an AMD Ryzen 9 7900, 32GB RAM).\nCould an old Nucbox paired with a cheap conference speaker like this one run Fulloch 24/7? Legacy Rerun: Let\u0026rsquo;s check how those old CPU models work in our latest setup # I\u0026rsquo;d already tested trying to run the voice assistant on an old CPU computer in the early experiment days and still had the models and code for that setup.\nAfter much testing back then I had landed on Moonshine for ASR and Kokoro-82M for TTS. Both models were very tiny and incredibly good for their size. You could run them on an old mini pc no problem. I felt the ASR was better than Whisper models that were still commonly used in Home Assistant setups and the speech from Kokoro was leagues ahead of Piper.\nThe ASR wrapper in our GPU setup exposed the same context to the ASR, so we could easily plug in our old Moonshine script, the only thing we lost was the context bias that helped pickup the \u0026ldquo;Hey Atticus\u0026rdquo; wakeword (and any other custom entity names) in our Qwen3 models. The TTS module provided the same surface for the Kokoro model, but we lost the voice cloning option and had to use the available Kokoro models for voice. The GPU setup had been built on these initial CPU experiments so I knew the small models were usable.\nBut this first cut still depended on torch. transformers pulls it in for Moonshine, and the Kokoro build was torch-native. I wanted a slim final image for our CPU setup, so wanted to see how we could minimise the footprint of torch and still run the best ASR and TTS models possible.\nGPU to CPU: Passing on the torch # Kokoro gave good voice but missed complicated words like \u0026ldquo;meteorological\u0026rdquo;, just skipping them completely. But the quality was generally good and for most tasks good enough.\nMoonshine base model was also \u0026ldquo;OK\u0026rdquo; but the missing context bias meant the wakeword in transcript method we were using for our GPU setup just made it much harder for the wakeword to be picked up. Yelling at your voice assistant five times in a row to brighten the living room lights and then have it misunderstand \u0026ldquo;brighten\u0026rdquo; for \u0026ldquo;brighton\u0026rdquo; when you are standing just a few metres away from the light switch doesn\u0026rsquo;t make sense anymore. Casually asking the voice assistant to turn off all the upstairs lights as you go down the stairs and it just does it makes more sense. The ASR is actually much more important than the TTS in a home voice assistant.\nWe needed better ASR and the best open-source ASR is Qwen3, that was clear from all the tests we had done on the GPU. So, how can we run it on a CPU?\nThe answer was ONNX Runtime: export the model graph once, then run it with a lean runtime that has fast CPU kernels and zero deep-learning-framework baggage. Searching Hugging Face showed that ONNX versions of the Qwen3 ASR models existed and that Kokoro also had an ONNX version.\nMigrating both backends off torch is what could really make the CPU image slim, its entire ML footprint became onnxruntime plus a couple of small helpers (librosa for mel features, tokenizers, misaki for grapheme-to-phoneme), instead of torch + transformers + CUDA.\nBackends in Fulloch are selected through a registry (core/backends.py), so swapping the torch backends for ONNX ones and the GPU Qwen models for CPU ONNX ones is a config change, not a rewrite. The two images, GPU :latest and CPU :cpu, share the same orchestrator and differ only in which backend modules they can load.\nSo, can we get the context bias and improved speech recognition of Qwen3 ASR on our old CPU mini PC?\nSpeech in: from Moonshine to Qwen3-ASR-0.6B on ONNX # The Moonshine starting point # The first CPU ASR I ported after first experiments with Whisper was Moonshine, a purpose-built, tiny English ASR that runs happily on edge hardware. It works, and it\u0026rsquo;s still in the repo as the option for very constrained edge devices, in both base (~62M) and tiny (~27M) sizes. But it doesn\u0026rsquo;t have any context biasing, so wakeword pickup is harder and it mistranscribes often enough to be annoying for anything but the simplest commands.\nFulloch\u0026rsquo;s wakeword is \u0026ldquo;Hey Atticus\u0026rdquo;, and this isn\u0026rsquo;t a common dictionary word. Moonshine will cheerfully transcribe it as \u0026ldquo;attic us\u0026rdquo;, \u0026ldquo;Oticus\u0026rdquo;, \u0026ldquo;Adicus\u0026rdquo;, etc. and the wakeword regex misses. On the GPU tier I solve this by feeding the ASR a context hint (\u0026ldquo;Technical terms: Atticus, …\u0026rdquo;) that biases the decoder toward the right spelling. Moonshine doesn\u0026rsquo;t have this ability, so you need to make sure you pronounce Atticus as clearly as possible for it to work.\nSwitching to Qwen3-ASR-0.6B # I found an int8 ONNX export of the 0.6B variant (Daumee/Qwen3-ASR-0.6B-ONNX-CPU) on Hugging Face. This could be the one. It\u0026rsquo;s the same Qwen3 ASR chat-template architecture as the big GPU model, so the context-biasing wakeword detection would work on CPU. Plus it\u0026rsquo;s multilingual (30 languages), something I am keen to test in some future version of Fulloch.\ncore/asr_onnx.py adapts the model\u0026rsquo;s bundled ONNX inference into the same QwenASRPipelineWrapper the GPU backend uses, Whisper-compatible mel features computed with librosa/numpy, the encoder/decoder ONNX sessions, and the special-token handling for Qwen3 ASR\u0026rsquo;s audio frames. From the orchestrator\u0026rsquo;s point of view it\u0026rsquo;s just another streaming generator over the mic queue. No torch anywhere.\nPerformance # On the dev CPU box it transcribed at RTF ~0.16–0.21 (real-time factor, so about 5x faster than real-time).\nASR backend params torch? biasing languages notes Qwen3-ASR-0.6B ONNX 0.6B no yes 30 Tiny-tier default; RTF ~0.16–0.21 on dev CPU Moonshine Base ~62M no no English smaller/faster fallback for edge devices Moonshine Tiny ~27M no no English smallest (~27M), most constrained devices The wakeword trap # Yes, we got context biasing and our 0.6B Qwen3 ASR would reliably pickup the wakeword\u0026hellip; The problem was it always picked up the wakeword! When the model is acoustically uncertain (e.g., it hears a cough), it has to choose between trusting the audio and trusting the context bias\nThe 0.6B model leaned so hard on the prior that it started hallucinating the wakeword out of every little cough, sneeze or tap on the table. If we turned the bias off, the model scattered the spelling into variations like \u0026ldquo;Adikis\u0026rdquo; or \u0026ldquo;Eddie Kiz,\u0026rdquo; and the assistant ignored us. We were trapped: we could either have a system that woke up to coughs, or a system that ignored our commands. There was no nuance the knob was either fully on or fully off.\nBand-aids and the dedicated model detour # My first instinct was to try to put guards and filters all around the wakeword detection. I tried several options:\nBare-wakeword loudness gate: Automatically rejects a command-less wakeword if it is at or below the room\u0026rsquo;s background noise baseline. Unbiased re-transcribe: Re-runs the ASR on the same buffer with the bias blanked out; if the wakeword vanishes, it was an echo. Prompt-echo marker: Drops any transcript that literally hallucinates the \u0026ldquo;technical terms\u0026rdquo; scaffolding. These worked, and were good guards to have in place anyway but they were band-aids on the capabilities of our 0.6B model. It forced me to reconsider a path I had previously rejected and still wasn\u0026rsquo;t happy about: adding a dedicated wakeword model.\nThis is what most voice assistants do and it should kill the hallucination problem, while also dropping the always-on CPU compute for ASR. I evaluated several candidates that could run without PyTorch on our slim CPU image:\nModel Approach CPU cost Custom phrase No-torch? openWakeWord CNN on speech embeddings very low yes (synthetic Piper TTS) ✅ microWakeWord tiny streaming CNN tiniest (MCU-class) yes (synthetic Piper TTS) ✅ sherpa-onnx KWS transducer KWS low yes (write keyword tokens, no training) ✅ EfficientWord-Net few-shot embedding match low yes (record 3-4 samples) ✅ openWakeWord was still the recommended tool. I could technically train a new wakeword model for \u0026ldquo;Hey Atticus\u0026rdquo; using the TTS synthetic data, but I would also need to add in some real recordings and either simulate or run more recordings with background noise and mumblings, coughs etc. This could be work for a later date, and would be a big project in itself. But before I did any implementation work I went back to my early experiment notes. Those early experiments had used the openWakeWord model way back then and the results were disappointing\u0026hellip; \u0026ldquo;Alexa\u0026rdquo; was the only wakeword that worked reliably and the accuracy in picking it up was about as good as what I already got with Moonshine ASR using my custom \u0026ldquo;Hey Atticus\u0026rdquo; wakeword. This wouldn\u0026rsquo;t be the upgrade I thought it would be, I would just be back where I started.\nUntil a new open-source wakeword model comes out or a big upgrade to openWakeWord is released, I couldn\u0026rsquo;t justify retrying it.\nBrute forcing with a bigger ASR # Before giving up and just falling back to Moonshine, I wanted to test one last theory: what if we just brute-forced the problem by running a stronger ASR model on the CPU?\nI found a community export of the 1.7B Qwen3 model (andrewleech/qwen3-asr-1.7b-onnx). It wasn\u0026rsquo;t a drop-in replacement—it unified the encoder, used int4 quantization, and required some wrapper adaptation, but it ran entirely on ONNX Runtime and theortically a CPU.\nI spent an afternoon writing an adapter and benched the 1.7B int4 model against our current 0.6B int8 model using synthetic generated voice clips from 31 different TTS voice models:\nModel WER Mean latency RTF Qwen3-ASR-1.7B int4 0.00 % (0/871) 2.40 s/clip 0.215× Qwen3-ASR-0.6B int8 (current) 3.33 % (29/871) 1.59 s/clip 0.142× The results were staggering. The 1.7B model was flawless across every voice tested. While it cost about 1.5× the latency of the smaller model, it still ran ~4.6× faster than real-time on the dev box CPU.\nMore importantly, I tested it live with a microphone. When I said \u0026ldquo;Hey Atticus,\u0026rdquo; it transcribed it perfectly without scattering. When I made gibberish sounds or coughed, it transcribed nonsense and suppressed it, I had zero hallucinated wakewords with wakeword bias in the context. I was getting the state of the art open-source ASR on a CPU and wakeword biasing was now working as well as it did on the GPU. It needed a solid 5-6GB of RAM just for the ASR, but as this was the key to a good voice assistant I saw it as a necessary resource requirement.\nSpeech out: the Kokoro latency saga # After all the back and forth with setting up the ASR, I thought the TTS would be an easy win. Kokoro was astoundingly good for such a small model, shouldn\u0026rsquo;t need much work to just plug that back into our CPU tier. But I was wrong\u0026hellip;\nThe model itself never changed: Kokoro-82M, a small, high-quality model with built-in named voices (no cloning but good voice selection). It had started as the torch build in initial tests, but I wanted to move it to the ONNX build to drop that torch dependency from our CPU image.\nIt sounded great. The problem was latency: once on ONNX, spoken replies had ~5 seconds of dead air before any audio. That\u0026rsquo;s worse than the GPU pipeline, and subjectively worse than the torch Kokoro had felt even on the CPU. Was I going to need some special \u0026ldquo;streaming\u0026rdquo; Kokoro ONNX build?\nNo. It turned out to be two independent, self-inflicted causes, and a third bug I created while fixing them.\nWrong quant # Kokoro ships several ONNX variants. I\u0026rsquo;d defaulted to the int8 model_quantized.onnx on the assumption that \u0026ldquo;smaller = faster on CPU\u0026rdquo;. That\u0026rsquo;s wrong. The dequantisation (QDQ) overhead dominates, and int8 came out ~5x slower than fp16/fp32. On CPU, fp16 kernels just upcast to fp32 for the actual compute, so fp16 ≈ fp32 speed at half the disk size.\nBenchmark on a single 8.2s utterance, best of 3 after warmup, CPUExecutionProvider:\nONNX variant size synth time RTF notes model_quantized.onnx (int8, old default) 92 MB 6.21s 0.76 dequant overhead dominates a small model model_q8f16.onnx (int8+fp16) 86 MB — — segfaults on the CPU EP — avoid model_fp16.onnx 163 MB 1.18s 0.14 half the size of fp32, same CPU speed model.onnx (fp32) 326 MB 1.20s 0.15 safe, correct fallback Just changing the default away from int8 took RTF from 0.76 → 0.14, a 5x speedup.\nFull sentence synthesis # Kokoro is non-autoregressive: it does one forward pass per input span, so time-to-first-audio equals the synth time of whatever you hand it first. I was handing it the whole sentence. A long opening sentence meant several seconds of silence before the first sample played.\nI briefly investigated the msgflux/Kokoro-82M-streaming-onnx build, thinking I needed real streaming. It turned out its \u0026ldquo;streaming\u0026rdquo; is just input-text chunking with fixed token buckets, exactly the thing I could do myself, plus a quant choice I\u0026rsquo;d already made. Not worth a new dependency and fixed-bucket padding.\nThe fix was to split the input on clause/sentence punctuation and synthesise one clause at a time on the existing producer/consumer worker thread, so the first clause plays while the rest renders.\nAn honest aside: I first over-engineered this with a \u0026ldquo;ramp\u0026rdquo;: a tiny opening fragment growing to a steady-state cap. I\u0026rsquo;d even built out specific config levers for it before realizing it was the wrong path:\nknob default meaning first_fragment_words 5 opening fragment size → first-audio latency fragment_ramp_words 2 growth per fragment (buffer outruns synth) fragment_max_words 14 steady-state cap (prosody/throughput) That was a latency mechanism for the slow int8 model. Once fp16 landed and synth was well under real-time, the fine-grained fragments started causing audible mid-sentence gaps and choppy speech. I removed the ramp out and reverted to a plain clause split. Simpler code, gap-free playback. The lesson: don\u0026rsquo;t keep optimisations whose premise you\u0026rsquo;ve already removed.\nThe combined effect on a multi-sentence reply (~15s of audio):\nconfig time-to-first-audio RTF gaps int8, whole-sentence (before) 5.58s 0.76 — int8 + fragmentation 2.5s 0.88 sub-second early gaps fp16 + clause split (after) 0.42s 0.17 none From 5.6 seconds of dead air to under half a second.\nThat is not a number # Happy with fp16, I went to pre-render a demo clip for each of Kokoro\u0026rsquo;s 28 voices for the setup wizard\u0026rsquo;s voice picker. Several came back as harsh noise instead of speech, including af_heart, which was the default voice and recommended voice for Kokoro. I thought it might just be some random error so reran it and exactly the same errors came out in the same places, this was a built-in failure.\nFurther analysis revealed that the fp16 ONNX model emits NaN samples for 11 of the 28 voices. Those voices\u0026rsquo; style vectors push the fp16 graph out of numeric range; the output is 36–65% NaN, and on write soundfile\u0026rsquo;s PCM-16 saturates NaN to full-scale, which reads as clipping/noise. Some real-world testing showed the same voices producing the same garbage in live playback.\nThe bad voices could be identified by clip% ≫ 0 with RMS ≈ −2 to −4 dBFS indicating a mostly-saturated waveform, versus the clean voices sitting around −18 to −27 dBFS. You can see this clearly in the sample from a raw signal analysis below:\nvoice peak clip% rms dBFS sil% gap flag af_alloy 0.825 0.00 -22.5 29 1.0 ok af_bella 1.000 63.17 -2.0 22 0.8 NaN af_jessica 1.000 38.11 -4.2 24 0.6 NaN af_aoede 1.000 0.00 -18.4 25 1.0 ok (peak-touch only) So, checking all the voice models that come with Kokoro we see the below when running the fp16 ONNX version:\nstatus voices NaN fraction Malformed af_bella, af_kore, am_michael, bm_lewis, bf_emma 62–65% Malformed af_river, am_liam, af_sky, bm_george, af_jessica, bf_alice 36–39% Clean (17) af_alloy, af_aoede, af_heart, af_nicole, af_nova, af_sarah, am_adam, am_echo, am_eric, am_fenrir, am_onyx, am_puck, am_santa, bf_isabella, bf_lily, bm_daniel, bm_fable 0% To confirm that NaN-free didn\u0026rsquo;t automatically mean correct speech, I ran an intelligibility cross-check on the voices that looked like they worked. Each clip was transcribed with the already on-disk Qwen3-ASR ONNX (resampled 24k→16k) and scored against the source text using difflib similarity (ratio) and the percentage of expected words present (recall):\nvoices ratio recall transcript all 17 clean voices 0.94 91% \u0026ldquo;A rainbow is a meteorological phenomenon that is caused by refraction,\u0026rdquo; af_bella (broken control) 0.47 32% \u0026ldquo;A rainbow is a meteorological phenomenon that is caused by refraction.\u0026rdquo; The 17 clean voices were intelligible, while the broken control scored terribly because the ASR only captured the clean intro before the NaN noise completely overtook the audio. All voices skipped over the difficult to pronouce meteorological word, that was a limitation of the Kokoro model not an fp16 ONNX issue.\nGet our voices back # My first fix was to trim the voice list to the 17 clean ones and just allow those with our CPU image. But that traded 40% of the voice catalogue and a correctness bug for ~160 MB of saved download. Also, on CPU, fp16 buys no speed, it upcasts to fp32 to compute anyway. The exclusion list was also never really sound: the fp16 NaN was input-dependent and I hadn\u0026rsquo;t tested on lots of different transcripts, so a \u0026ldquo;clean\u0026rdquo; voice could still NaN on an unlucky phrase and real-world testing showed it to be very flaky. \u0026ldquo;Clean\u0026rdquo; voices were not necessarily clean and could NaN at anytime.\nSo I re-ran the analysis on the fp32 model.onnx and got the below results:\nmetric result on all 28 voices (fp32) non-finite (NaN) samples 0 across all 28 clip ceiling (|x| ≥ 0.999) 0% on 26; af_aoede \u0026amp; am_puck touch peak 1.000 at a single sample (0.0%) — benign peak-touch ASR word error rate 6.9% on every voice (mean = median = max) ASR character error rate 10.1–10.7% (mean 10.5%) flagged (NaN / WER\u0026gt;15% / clip\u0026gt;1%) none Clean across the board! The identical ~7% WER across every voice (measured by transcribing each clip back with the on-disk Qwen3-ASR ONNX) is the key signal: the recogniser understands every voice equally, so none is acoustically degraded. The residual 7% was the ASR consistently dropping the one deliberately-hard word, \u0026ldquo;meteorological\u0026rdquo; included in the transcript, the Kokoro limitation we already knew about.\nThe cost of fp32 over fp16: RTF 0.15 vs 0.14 (within noise, both far under real-time) and ~326 MB vs ~163 MB download. That ~160 MB is the entire price for zero NaNs and all 28 voices.\nWhere we landed # Two models, both in ONNX, both comfortably real-time on a CPU, and zero PyTorch in the image:\nComponent CPU tier technology RTF on dev CPU ASR Qwen3-ASR-1.7B ONNX (int4 decoder, wakeword biasing) ~0.16–0.21 TTS Kokoro-82M ONNX (fp32, 28 built-in voices) ~0.15 Wakeword tolerant regex on the ASR transcription — LLM regex-only, or a remote OpenAI-compatible endpoint — flowchart LR Mic[(Microphone)] --\u003e ASR[Qwen3-ASR-1.7BONNX, CPU] ASR -- \"'hey Atticus'regex match\" --\u003e Route{Regexfast-path} Route -- \"common commands\" --\u003e Tools[Tools / Home Assistant] Route -- \"everything else\" --\u003e LLM[Regex-onlyor remote LLM] LLM --\u003e TTS[Kokoro-82MONNX, CPU] Tools --\u003e TTS TTS --\u003e Speaker[(Speaker)] At RTF ~0.15–0.21 on my dev CPU, even a 3–4x slower mini pc should stay well under real-time. I need to do lots more testing and would love to hear how this runs on other peoples setups.\nAll code is at GitHub, fulloch. The CPU image ships as :cpu; pick the default tier in the first-run setup wizard and it downloads the recommended models for you.\n","date":"26 June 2026","externalUrl":null,"permalink":"/posts/getting-voice-onto-a-cpu/","section":"Posts","summary":"How Fulloch’s CPU tier runs speech recognition and text-to-speech on a CPU-only PC","title":"Getting Voice Onto a CPU: CPU-Only ASR and TTS for Fulloch","type":"posts"},{"content":"","date":"18 June 2026","externalUrl":null,"permalink":"/tags/benchmarks/","section":"Tags","summary":"","title":"Benchmarks","type":"tags"},{"content":"","date":"18 June 2026","externalUrl":null,"permalink":"/tags/llm/","section":"Tags","summary":"","title":"Llm","type":"tags"},{"content":"","date":"18 June 2026","externalUrl":null,"permalink":"/tags/quantization/","section":"Tags","summary":"","title":"Quantization","type":"tags"},{"content":"","date":"18 June 2026","externalUrl":null,"permalink":"/tags/self-hosted/","section":"Tags","summary":"","title":"Self-Hosted","type":"tags"},{"content":"In the first year diary I made a fairly blunt claim: that Fulloch needed at least a 4B model and really the 9B model if you wanted anything beyond basic tool calls and responses. Anything smaller was a dead end. That was the lived experience of a year of tinkering, but it was never a clean experiment. I\u0026rsquo;d swap models, change five other things at the same time, and make my conclusion. So with the World Cup on and after a genuine conversation I had with the 9B version, I finally sat down and ran some experiments on how well this real world use case transferred down the parameter ladder.\nThe test is simple. I took one conversation, one I actually had with the thing, and then repeated it a further three times against the different smaller brains: Qwen3.5 at 9B, 4B, 2B and 0.8B were all given the same scenario. Every one is the Q5_K_M quant. Everything else, the ASR, the TTS, the prompt, the tool registry, the temperature, stays exactly the same. The only variable is the size of the model doing the thinking.\nThe conversation was always the same six-ish turns:\nWhat time is it? - A quick warm-up question to make sure the thing was running properly, regex fast path should capture it What\u0026rsquo;s the weather forecast? - Another one to actually trigger the LLM and see it can make a straight tool call When is Australia\u0026rsquo;s next game in the World Cup? - Now we need web search, summarisation and the follow on queries to link it together while holding context Can you add those events into the calendar? Who are the favourites? What about the next two Australian matches, who are the favourites in those? It\u0026rsquo;s a deliberately ordinary and short sort of conversation to have with the AI but it touches on some unique challenges and quickly tests tool calling, context and reasoning in just a few questions. The conversation is exactly the kind of thing you\u0026rsquo;d ask while you\u0026rsquo;re doing something else or just want some quick info without reaching for your phone.\nWhat follows is, frankly, a slow-motion lobotomy.\nQuick note: In the process of running this experiment, I realised there was a lot of context bloat being fed through the agent loop. Good to show how the models handled this unoptimised scenario, but triggered quite a few optimisations that can be seen in the v2.1.9 release\nThe 9B: Slow, but it Actually Knows Things # Qwen3.5-9B: grounded, accurate, and in no particular hurry. The 9B is the brain Fulloch ships with, and it sets the bar. Ask it when Australia plays next and it fires off a web search, reads the result, and comes back grounded: Australia vs USA at Lumen Field in Seattle on June 20, followed by Paraguay at Levi\u0026rsquo;s Stadium on June 26. Then it tells the user it has logged the answer to today\u0026rsquo;s note without being asked.\nTell it to add that like both those to the calendar and even with a slight stumble in the question it does the thing that actually matters, it remembers what \u0026ldquo;those\u0026rdquo; refers to, and creates two correct events: Australia vs USA on the 20th and Australia vs Paraguay on the 26th. Three tool calls in a single agent loop, no fuss.\nAsk who the favourites are and it searches again, returning France as the outright favourite at +400, Spain at +500, then England, Portugal and Argentina trailing with real odds attached. And when I ask about the favourites for Australia\u0026rsquo;s next two matches specifically, it reasons over what it already has: USA are favoured against Australia, Paraguay are favoured in the other. No tool call needed, it just used context.\nThe catch is the clock. Those web-search turns took over 20 seconds each. A large chunk of that is the agent loop making three round-trips, with ~3.8s time-to-first-token while llama.cpp prefills an 8K+ token prompt for each trip. It generates at a steady 14-16 tokens a second. It is, to be honest, on the slow side of conversational. But every single answer was correct and grounded, and that turns out to be the whole ballgame.\nVRAM sat at 15.2 GB of the 5060 Ti\u0026rsquo;s 15.5 GB usable. Right on the edge, which is exactly where the diary left it.\nThe 4B: Faster, and Starting to Fib # Qwen3.5-4B: quicker off the mark, but the grounding is already slipping. Drop to 4B and the first thing you notice is speed. Weather comes back in 3.6s, favourites in 4.3s, and VRAM falls to 12.7 GB. For a beat it feels like a free upgrade.\nThen you read the answers. The next-game search still mostly works, USA on June 19 in Seattle, though it\u0026rsquo;s now disagreeing with the 9B on the date (a timezone it didn\u0026rsquo;t quite resolve) and volunteering that Australia already played Türkiye on the 14th. Useful, but noisier.\nThe cracks show on the calendar turn. Asked to add the next two games, it added Australia vs USA on the 19th and Australia vs Turkey on the 14th, a match that has already been played. It heard \u0026ldquo;Turkey\u0026rdquo; in its own previous answer and bolted it onto the calendar without registering that it\u0026rsquo;s in the past.\nThen it stops bothering to check its work. Asked who the favourites are, it skips the web search entirely and answers straight from some trained memory: \u0026ldquo;France, Spain, and Brazil\u0026hellip; Australia is considered a mid-table contender.\u0026rdquo; Brazil has quietly appeared from nowhere but the whole answer sounds legit, only a check of the logs shows that this information never existed anywhere in the tool result history and was completely fabricated. On the per-match follow-up it does search, but comes back with a shrug, couldn\u0026rsquo;t find specifics, here are the tournament favourites instead. Still coherent, still mostly sensible, but you can feel it starting to guess. This model could be dangerous if you trust it with too much.\nThe 2B: Confidently Wrong # Qwen3.5-2B: still fast, but now answering a different question entirely. At 2B (10.7 GB VRAM, answers in 2-4 seconds) the speed is genuinely lovely and the accuracy is gone.\nAsked for the next game, the web search actually returns the correct schedule, Group D, Turkey on the 14th, USA on the 19th, Paraguay on the 25th, but the model reads all of that and replies that Australia\u0026rsquo;s next game is June 14 against Turkey. It picked the first date it saw and handed me a fixture that happened days ago. The calendar turn then made it worse, adding two vague \u0026ldquo;World Cup - Australia\u0026rdquo; events on June 11 and June 14, the tournament opener and a past game, with all the specifics dissolved.\nAnd then the favourites question goes completely off the rails:\nThe main favourites are Liverpool, Man City, Arsenal, and Chelsea.\nThat\u0026rsquo;s not a World Cup answer. That\u0026rsquo;s the Premier League. The model has lost the thread of the conversation so badly it\u0026rsquo;s answering about English club football. The per-match follow-up doubles down with an invented \u0026ldquo;New Zealand vs Australia on June 17\u0026rdquo; fixture that simply doesn\u0026rsquo;t exist.\nThe 2B hits an interesting intersection. Fast, fluent, and confidently feeding you fixtures and favourites that are flat wrong. You can\u0026rsquo;t trust this thing at all. The only things it was good at were the basic tool calls that a decent regex capture pattern or automation could achieve. So you could tune it on simple tasks to make it function on a narrow focus area, but why? Just automate that and use the bigger models to actually reason and connect the dots on those things you don\u0026rsquo;t want or aren\u0026rsquo;t able to automate.\nThe 0.8B: A Different Sport Entirely # Qwen3.5-0.8B: fast, cheap, and convinced you\u0026rsquo;re asking about Formula 1. The 0.8B is where it stops being a comparison and starts being comedy. 9.9 GB VRAM, sub-2-second replies, and almost nothing it says is true.\nAsked when Australia plays next, its first move was to call get_weather_forecast(), the wrong tool entirely for a sports question. I repeated myself, and it then hallucinated a game \u0026ldquo;Thursday, June 18, 2026 at 12:00 PM\u0026rdquo; with a cheerful \u0026ldquo;you can see the schedule in your calendar or on the main site.\u0026rdquo; The calendar turn produced a malformed tool call with a weekly recurrence and garbled arguments, then declared \u0026ldquo;Done.\u0026rdquo;\nThen the favourites:\nThe top three finishers were Antonelli, Hamilton, and Hadjar. Hadjar avoided a penalty.\nWe have changed sports again. That\u0026rsquo;s Formula 1. The model has decided the World Cup is a Grand Prix, and on the next turn it cheerfully lists \u0026ldquo;Antonelli (1st), Hamilton (2nd), Hadjar (3rd)\u0026rdquo; as the favourites for Australia\u0026rsquo;s upcoming football matches against, apparently, Argentina and Italy. This list of F1 drivers was taken almost verbatim from an intent example given in the system prompt, so the model just copy and pasted that in rather than understand it was just an example.\nThe finale was the most telling. I explicitly asked it to search the web for the next game. It ran the search, got a garbled result, and then got stuck calling search_notes over and over with the same query, the same four times in a row, the agent loop spinning with no exit. That\u0026rsquo;s not a knowledge gap, that\u0026rsquo;s the model being too small to reliably drive the tool-calling machinery at all.\nSide by Side # The same conversation, four brains:\n9B 4B 2B 0.8B VRAM 15.2 GB 12.7 GB 10.7 GB 9.9 GB Web-search turn ~20s ~14s ~11s ~3s Simple tool turn ~3-5s ~3.6s ~2s ~1.6s Gen speed 14-16 t/s 15-22 t/s 11-22 t/s up to 35 t/s Next game ✅ Correct \u0026amp; grounded ⚠️ Mostly right, noisy ❌ Gave a past game ❌ Hallucinated + wrong tool Calendar ✅ Both events correct ⚠️ Added a past match ❌ Wrong dates, vague titles ❌ Malformed call Favourites ✅ Searched, real odds ⚠️ Guessed, no search ❌ Premier League clubs ❌ Formula 1 drivers Per-match reasoning ✅ Used context ⚠️ Hedged ❌ Invented a fixture ❌ F1 drivers again Agent loop Reliable Reliable Shaky Broke (infinite loop) The pattern is brutally clean. Latency and VRAM scale down smoothly and pleasantly. Capability does not, it falls off a cliff. The 9B is the only one that\u0026rsquo;s actually trustworthy. The 4B is a usable assistant that needs supervision and a tight focus. The 2B and below aren\u0026rsquo;t a smaller version of the same assistant, they\u0026rsquo;re a different, unreliable product wearing the same voice.\nWhat This Actually Settles # A year ago I wrote \u0026ldquo;4B or nothing\u0026rdquo; and meant it as a gut feeling. This test turns it into something I can point at. The interesting wrinkle is how the models fail, because it isn\u0026rsquo;t graceful.\nThe first thing to go isn\u0026rsquo;t fluency, it\u0026rsquo;s grounding. The 4B already prefers to answer favourites from memory rather than spend a tool call checking, and that\u0026rsquo;s the moment an assistant becomes a confident guesser. By 2B the model still happily reads a correct search result and then ignores it. The web search machinery works fine the whole way down; what breaks is the model\u0026rsquo;s discipline to actually trust the result over its own training-data instincts.\nThe second thing to go is task identity. The slide from \u0026ldquo;World Cup\u0026rdquo; to \u0026ldquo;Premier League\u0026rdquo; to \u0026ldquo;Formula 1\u0026rdquo; is the model leaning harder and harder on statistical priors and system prompt examples as it loses the ability to hold the actual conversation in its head. Football favourites, English clubs, F1 drivers, they\u0026rsquo;re all \u0026ldquo;sport-shaped competitive rankings\u0026rdquo; in some fuzzy latent sense, and a small enough model just reaches for the nearest shape.\nThe third, at the very bottom, is tool-calling competence itself. The 0.8B reaching for the weather tool to answer a football question, emitting malformed arguments, and then deadlocking in a search_notes loop, that\u0026rsquo;s the agent loop being too much machinery for the model to operate. No prompt tweak fixes that.\nSo the conclusion holds, and now I understand it better. Fulloch is an agent, not a chatbot. It lives or dies on picking the right tool, trusting what the tool returns, and remembering the thread across turns, and those are exactly the three capabilities that evaporate as the model shrinks. The 9B\u0026rsquo;s over 20 second web searches are an annoyance I\u0026rsquo;ll keep chipping away at. A 2B telling me, smoothly and instantly, that Liverpool are favourites to win the World Cup is not a trade I\u0026rsquo;d make.\nWhere to From Here? # At some point the agent orchestration optimisations will bring only incremental gains and it will get more interesting to start swapping in different models again. There is a new open-source model being released every other week that seems to break preheld notions of what is possible with limited resources, might we get a lightning fast 2B model that is trustworthy and can hold context in the near future or does it already exist and I just need to plug it in?\nAlso, if shrinking the brain breaks the agent, the obvious question is what happens when you go the other way. I am constrained to the 16GB GPU at the moment but a lot of power users aren\u0026rsquo;t. If you\u0026rsquo;ve already got a serious LLM running on a home server, plumbing that into Fulloch over an OpenAI-compatible endpoint would let the assistant punch well above the current 9B model weight. The even bigger advantage is you don\u0026rsquo;t have that model tied up with just Fulloch anymore, it can still be used for other tasks but Fulloch can tap into it when needed.\nI would love to see what the Qwen3.6-27B at Q8_0 would unlock capability-wise\u0026hellip;\nAll code is at GitHub, fulloch. Same machine, same RTX 5060 Ti, no cloud — just a worse and worse model each run.\n","date":"18 June 2026","externalUrl":null,"permalink":"/posts/watching-a-voice-assistant-get-dumber/","section":"Posts","summary":"Same conversation, same prompt, same tools, four model sizes. Watching Fulloch degrade from a capable agent into something that confuses the World Cup with Formula 1.","title":"Watching a Voice Assistant Get Dumber: Qwen3.5 from 9B Down to 0.8B","type":"posts"},{"content":"The idea wasn\u0026rsquo;t unique. We had a house full of Alexa devices and enjoyed being able to quickly ask for lights to be turned on or off, set timers and play music on command. However, in March of 2025 Amazon announced all audio would be sent to their cloud servers for processing. It wasn\u0026rsquo;t clear how much of our day to day chatter was being recorded and saved into Bezos\u0026rsquo; computers, and we became a little uncomfortable having little microphones everywhere recording our every utterance. Why couldn\u0026rsquo;t we run our own voice assistant on a computer in the house without needing to send anything to the cloud?\nI had a very old computer (i5-3570 + 8GB RAM + 1050 Ti GPU) that I thought I could repurpose for a simple private voice assistant. The GPU wouldn\u0026rsquo;t be able to do much, but I\u0026rsquo;d already been doing lots of testing of small open-source LLMs and thought there might be some tiny model that could handle the basic tasks we used most frequently.\nSo the story starts with that idea and that old computer, and a year of mostly going in wrong directions before any of it really worked.\n1. The Wyoming Pipeline: Following the Well-Trodden Path # It started, like most people\u0026rsquo;s did, by reading the forums and discovering that everyone who\u0026rsquo;d ditched Alexa was walking the same path. The standard community build was using Wyoming, the Rhasspy team\u0026rsquo;s simple TCP protocol for stringing voice services into a pipeline, and they\u0026rsquo;d already built Wyoming-compatible containers for all the popular models. I put together a Docker Compose file and a controller script to tie it together, OpenWakeWord listening, Whisper for speech-to-text, Ollama for the LLM, Piper for the voice. On the surface it looked clean and modular.\nflowchart LR Mic[(Microphone)] --\u003e Controller[[\"Controller(app.py)\"]] Controller --\u003e Speaker[(Speaker)] Controller \u003c--\u003e|Wyoming| OWW[OpenWakeWordwakeword detection] Controller \u003c--\u003e|Wyoming| Whisper[Whisperspeech-to-text] Controller \u003c--\u003e|HTTP REST| Ollama[Ollamagemma3:1b] Controller \u003c--\u003e|Wyoming| Piper[Pipertext-to-speech] One issue that came up straight away was the OpenWakeWord system. When OpenWakeWord fired its detection event the controller stopped the stream and re-armed the microphone from scratch, so any words spoken right after the wakeword had already gone past, unrecorded. You had to say \u0026ldquo;hey Alexa\u0026rdquo;, pause, wait for Whisper to buffer and process, then wait again for the reply. It demanded robotic commands and killed the conversational vibe, when it worked! But it felt like it was 50:50 if the model would detect the wakeword on my old crummy microphone. The only wakeword model that kind of worked was the \u0026ldquo;Alexa\u0026rdquo; one.\nNext issue was the precise phrasing these small models needed to get even close to giving the right tool call response. \u0026ldquo;Turn off downstairs lights\u0026rdquo; worked but \u0026ldquo;Lights off downstairs\u0026rdquo; didn\u0026rsquo;t. Malformed json tool calls were pretty common and regex could only fill some of the gap.\nMy research had led me straight down a well-trodden path and reading the forums, I saw everyone was running into the exact same problems.\n2. The Buffering Wars: Attacking Latency on Both Sides # I didn\u0026rsquo;t want to give up yet and decided to focus on that delay between wakeword and transcription first. This was the biggest pain point for me initially, the current delay made interacting with the assistant a painful experience.\nThe first end-to-end run quantified the pain: 4 seconds from wakeword to transcription. For a smart speaker that\u0026rsquo;s an eternity; you say \u0026ldquo;hey, turn off the lights\u0026rdquo; and stand there staring at it, wondering if it even heard you. The kids would yell questions and get ignored because the command was gone before the mic re-armed. So OpenWakeWord was scrapped entirely in favour of always-on Whisper, transcribing continuously and just watching the text for the wakeword. It worked surprisingly well: no ring buffers to capture the gaps, no separate model to train, you could say the wakeword and keep talking without pausing, and the wakeword became a string you typed in config rather than a model you trained, you could set just about any wakeword you wanted instantly! Who cared if the computer transcribed all day, the compute load was minimal.\nAnother issue in this early prototype was the fixed input buffer that ran until timeout before transcribing, this meant we had to wait for that buffer to fill before it would even process the audio and it could cut you off on longer commands. This was replaced with a voice activity detection system that watched the audio stream RMS energy and gated when below a given threshold, so transcription could start the moment the user stopped talking instead of always waiting for the worst case. That reduced latency a bit when the environment wasn\u0026rsquo;t too noisy, which wasn\u0026rsquo;t often in our house! It wasn\u0026rsquo;t until much later that SileroVAD was used to monitor actual voice activity rather than just noise, improving use in the actual household environment and not just in my relatively quiet office.\nHowever, another buffering problem lived entirely on the output. Streaming text-to-speech models hand you audio in chunks as the model runs, and the trick is to start playing the first chunks immediately while the rest generate. Otherwise you are waiting for everything to be generated before playing it and this created a noticable wait for response. But threading that correctly, managing the buffer, and handling playback catching up to generation was a multi-week affair. A custom queue was tried and then reverted to a deque; underruns, where the buffer emptied before new audio arrived, were logged and chased across commit after commit.\nFixing the plumbing exposed the real bottleneck, though: the small language models themselves simply weren\u0026rsquo;t up to the job.\n3. 4B or Nothing: Hitting the Small-Model Wall # With the latency in the pipeline finally tolerable, attention turned to the brains of the operation, and the tiny models buckled. Gemma3:1b and Qwen3:1.7b were both bad at tool calling and worse at conversation, even after I split them across two Ollama containers: one with an \u0026ldquo;intent\u0026rdquo; prompt focused only on routing, another \u0026ldquo;chat\u0026rdquo; prompt for short exchanges. My commit message when I gave up on Gemma says it best: \u0026ldquo;gemma chat is rough.\u0026rdquo; One attempt to rescue the routing was a vector database with a createdb.py to build an index and an intents.py to match speech against pre-written examples. It was more reliable, but it was a band-aid: more complexity, another model to load, and a fresh class of failure modes bolted onto small LLMs that simply weren\u0026rsquo;t fit for the task.\nThe new PC ready to be built, with an RTX 5060 Ti to run the bigger models. By August I\u0026rsquo;d accepted I needed a newer computer. Not wanting to spend crazy money but wanting to run the next tier of models properly, I went for an RTX 5060 Ti, 16GB of VRAM was the sweet spot for the bigger models, and replaying my old PC games on ultra settings was a nice bonus! Mid-August the intent model moved up to Qwen3:4b, and by September I\u0026rsquo;d caved and set both intent and chat to Qwen3:4b-instruct. Anything smaller just wasn\u0026rsquo;t usable. The verdict was clarifying but also a little deflating: this project was never going to run on a repurposed old computer or an edge device. Bigger models opened the door to more ambitious features, but the first few attempts at them turned out to also be dead ends.\n4. Experiments That Didn\u0026rsquo;t Survive: KGLLM and Voice ID # With real model capacity available, two ambitious personalisation ideas got their turn, and both taught their lessons by failing. The first was the \u0026ldquo;knowledge-graph LLM\u0026rdquo;, KGLLM: rather than cramming personal context into the system prompt as raw text, I built a structured family_facts.json of names, preferences and routines and augmented a separate 4B-instruct model with it, all wired through a nearly 300-line ai_knowledge.py. The second was voice identification, using SpeechBrain\u0026rsquo;s speaker-recognition model, with the goal of having the assistant respond differently to different household members.\nVoice ID \u0026ldquo;kinda\u0026rdquo; worked. You had to record and train a model per person, it failed entirely on the kids\u0026rsquo; voices, and it was genuinely jarring when it called you the wrong name; in a noisy family house it landed maybe one time in five. A cool tech demo, but not useful day to day.\nThe knowledge graph met a similar fate, its complexity never earned its keep, and the surviving idea was far simpler: write a fact in plain prose, load it into the prompt at startup, done. Between September 2025 and January 2026 the project basically went into sleep mode. I tinkered occasionally, but the setup was still janky and wasn\u0026rsquo;t anywhere near a replacement for the Echo\u0026rsquo;s I still had sitting in storage under the house.\nA fresh mind after the Christmas break reframed the whole thing. Not more features, but a radical simplification.\n5. The Monolith and the Qwen3 Moment: Collapsing the Stack # Coming back energised, the instinct was to tear down the modular sprawl rather than add to it. I didn\u0026rsquo;t like the forest of separate containers and the Wyoming plumbing holding them together, so I stripped it all away and rebuilt everything as a single monolithic Python script. The LLM moved in-process via llama-cpp-python running a Qwen3-4B GGUF, Moonshine replaced the ageing Whisper, and Kokoro replaced Piper; smaller, faster, and a noticeable upgrade in voice quality. The whole thing got dramatically smaller, felt faster and, to my surprise, just worked.\nThen, in late January 2026, Alibaba\u0026rsquo;s Qwen team open-sourced standalone ASR and TTS models. I wanted to try them immediately. Switching to Qwen3-ASR-1.7B was a real bet, Moonshine was purpose-built and working fine, but the Qwen models were the new state of the art and I really wanted to see what they were capable of. Also, it meant running all Qwen models and collapsing three ecosystems into one and making the dependency footprint vastly simpler. The TTS earned its place fast. Instead of preset voices it clones a speaker from a few seconds of reference audio and a transcript, drop a name.wav and name.txt into a folder and the assistant speaks in that voice. I scared my wife by having it talk in hers, decided that was a bit too creepy, and instead cloned Morgan Freeman from a documentary; the video I made drew a lot of attention on Reddit (though I kept that voice out of the public repo to avoid any legal grief). All of this coincided with the project finally getting a name and a shape, Fulloch, Fully Local Home, with proper core/, tools/ and utils/ modules, a YAML config, and a first basic Home Assistant integration.\nA clean single-family pipeline was a huge leap forward but there were still some wrinkles in how audio was piped through the Docker containers that needed to be ironed out. A small job I thought\u0026hellip;\n6. Docker Audio Hell and Using the GPU: Earning Stability # The new pipeline looked great in a demo, but containers have no natural access to the host\u0026rsquo;s audio hardware and echo cancellation features, so the setup broke whenever I changed a speaker or mic on my computer. Most of February went to nothing but trying to stabilise that. ALSA, PulseAudio, and the container\u0026rsquo;s own audio stack all had to be coaxed into cooperating. The solution that stuck made PulseAudio the authoritative routing layer, with PULSE_SOURCE and PULSE_SINK environment variables in compose.yml telling the container which sources and sinks to use.\nWith the pipeline stable, it was time to see how much I could push the hardware I\u0026rsquo;d bought. In May the SLM jumped from Qwen3-4B to Qwen3.5-9B-Q5_K_M with the commit message saying it plainly, \u0026ldquo;make use of the 5060 Ti\u0026rsquo;s 16GB VRAM.\u0026rdquo; At ~6.6GB the 9B fills a serious chunk of the card, and fitting it alongside the 1.7B versions of ASR and TTS took some genuine GPU trickery. Context was kept to 8K on the SLM and compiling the TTS with torch.compile\u0026rsquo;s reduce-overhead mode reuses its activation buffers across decode steps via CUDA graphs, saving ~4-5GB of VRAM, the difference between fitting and an out-of-memory crash. This only worked once I\u0026rsquo;d pinned all TTS generation to a single long-lived worker thread, because the graph manager lives in thread-local storage and a fresh thread per turn tripped an assert deep in the compiler. On the LLM side, deleting a stray defensive reset() call let llama.cpp reuse the prefilled system prompt across turns, dropping per-turn latency from ~1100ms to ~250-400ms, and a startup cache-prime plus a cleanup pass that shaved the tool registry from 228 lines to 139 trimmed the rest.\nA bigger, stable model and audio pipeline finally made one of the harder interaction problems worth solving properly: interrupting the assistant mid-sentence.\n7. Barge-In, Stalls, and Memory: Making It Feel Responsive # With a capable model running on stable audio, the focus shifted to how the thing actually felt to talk to, starting with the feature that took the most iterations of anything in the project. Barge-in, interrupting the assistant mid-speech, landed in phases. First came a simple threading primitive, a TtsSession that could signal the worker to stop generating and abort playback. Then the harder half: the mic stays live during playback, so the assistant hears its own voice and might transcribe it as a wakeword. That self-echo problem was solved with a combination of PulseAudio echo cancellation reducing the leakage and a timing heuristic treating any transcription arriving within a narrow window of TTS ending as suspect. Getting a clean interrupt meant cancelling three distinct things in the right order, the LLM\u0026rsquo;s token stream, any stall phrase still playing, and the TTS audio output. All of which took careful coordination between threads.\nTwo smaller additions did a lot for the feel. First, stall phrases like \u0026ldquo;one moment\u0026rdquo;, \u0026ldquo;let me check that\u0026rdquo; are pre-rendered into audio at startup and cached, so when there\u0026rsquo;s a gap while the LLM thinks or a tool is awaiting a response, the user gets feedback that the assistant is doing something instead of leaving dead silence. Second, a deliberately simple notes system arrived as a successor to the old knowledge graph experiments. This system involved plain markdown files with tools provided to create new notes or read exising ones. A full-text search, and a BGE-small semantic search helped make retrieving information from the notes fast and accurate. A single facts markdown file provided more important long term information that could be loaded into the system prompt at startup for the LLM to utilise directly.\nA responsive, memory-equipped assistant set the stage for the final leap in intelligence and integration.\n8. The Agent Loop, HA Consolidation, and HACS: Closing the Loop # Now that we were using a 9B model we could stop just matching intents and start genuinely reasoning over tools. Instead of the old \u0026ldquo;hear request → match intent → call tool → speak\u0026rdquo;, the LLM could become an orchestrator that could dispatch tools and form responses based on feedback. The trick is catching the easy stuff and making it quick to respond while still having the flexibility of longer agentic loops available when the user wants it. Tthis is ongoing and probably never-ending work, a small change to intent examples or prompting can improve one type of agentic response while completely wrecking five others! A regex fast-path still catches the common stuff (\u0026ldquo;play something\u0026rdquo;, \u0026ldquo;stop\u0026rdquo;, \u0026ldquo;set a timer for ten minutes\u0026rdquo;) before the LLM is ever called, that was an early trick I implemented that has survived through the whole project.\nThe same period brought a big cleanup. The project had accumulated direct integrations with seven smart-home systems: Spotify, Hue, LG ThinQ, WebOS TV, Pioneer AVR, Airtouch HVAC, Google Calendar. Each integration came with its own auth flow and API quirks. All of them were retired in favour of a single Home Assistant integration. I had not used Home Assistant up until now but saw the attraction immediately. Home Assistant had a much better setup for integrating all the possible smart home devices a person could have and the community supporting it was amazing. A HACS component makes the relationship bidirectional, Fulloch talks to Home Assistant to control devices, and HA can talk back, speaking through Fulloch\u0026rsquo;s voice from an automation (\u0026ldquo;it\u0026rsquo;s bin night\u0026rdquo; at 8pm), surfacing its state as sensors, and reacting to wakeword events. Another Reddit post in the Home Assistant community showed real interest in the project, keeping me motivated.\nI did some tests using the VoiceDesign TTS model from Qwen, but speaker drift within conversations made it to weird to interact with, like everytime a different person was answering you. I switched back to fixed-reference Base cloning, which locks the voice by construction. VoiceDesign can still be used to generate new base models. It is a fun afternoon describing the voice you want for your assistant and seeing what it generates, some of the voices that were generated made me really wonder what data these models were trained on!\nFinally, I wanted to give my assistant it\u0026rsquo;s own name/wakeword. I\u0026rsquo;d tried the project\u0026rsquo;s name as the wakeword, but \u0026ldquo;Fulloch\u0026rdquo; has no entry in the ASR\u0026rsquo;s vocabulary and came out as \u0026ldquo;fulik\u0026rdquo;, \u0026ldquo;full lock\u0026rdquo; and a dozen other things. A lot of regex pattern matching tests made me realise this was a dead end. After testing about 20 to 30 different possibilities, I finally landed on \u0026ldquo;Atticus\u0026rdquo;. A consonant-anchored real name with a natural \u0026ldquo;Hey\u0026rdquo; prefix that the ASR transcribes reliably, I just had to make sure it got properly filtered from transcriptions, otherwise every other response was asking me if I wanted to know more about the book \u0026ldquo;To Kill a Mockingbird\u0026rdquo;!\nWhat had started as a brittle chain of containers was now a single coherent system, and after twelve months it was worth taking stock of where it actually landed.\nWhere Things Stand # Twelve months from the first plan, the pipeline looks like this:\nComponent Technology ASR Qwen3-ASR-1.7B (always-on, streaming generator) Wakeword Tolerant regex on ASR transcription (\u0026ldquo;hey Atticus\u0026rdquo;) SLM Qwen3.5-9B-Q5_K_M via llama.cpp + GBNF grammar TTS Qwen3-TTS-12Hz-1.7B-Base, voice-cloned Smart home Home Assistant REST API Notes Local markdown store with BGE-small semantic search Web search Self-hosted SearXNG Echo cancellation PulseAudio module-echo-cancel Deployment Docker Compose with host audio passthrough flowchart TB Mic[(Microphone)] --\u003e ASR[Qwen3-ASR-1.7Balways-on streaming] ASR -- \"'hey Atticus'regex match\" --\u003e FastPath{Regexfast-path} FastPath -- \"everything else\" --\u003e Agent[[\"Agent loopQwen3.5-9B + GBNF grammar\"]] subgraph Tools[\"Tool registry\"] direction TB HA[Home Assistant+ HACS] Notes[NotesBGE-small search] Web[Web searchSearXNG] Utils[Calculator, timers+ date/unit conversion] end FastPath -- \"common commands(skip the LLM)\" --\u003e Tools Agent \u003c--\u003e|\"pick a tool, get result,repeat up to N turns\"| Tools Agent --\u003e TTS[Qwen3-TTS-12Hz-1.7Bvoice-cloned] Tools --\u003e TTS TTS --\u003e Speaker[(Speaker)] Everything runs on a single machine with an RTX 5060 Ti. No cloud. No subscriptions. No requests leaving the network (except for any web searches through SearXNG).\nIt still needs a lot of work. Echo cancellation, barge-in and agentic capabilities will require never-ending tweaking. I want to see how well a cheap conference speaker can work as the \u0026lsquo;satellite\u0026rsquo; for the main Fulloch server, could I have more than one in multiple rooms around the house all linking back to the same server? What about those power-users who already have an LLM running on a home server, can they just connect that in through an OpenAI protocol rather than having a dedicated 9B model running just for Fulloch? So many questions and ideas still to be explored.\nSo now that I have built Fulloch, what do I actually use it for? # For the \u0026ldquo;turn off the lights from the couch\u0026rdquo; use case, I am not really using it. It is a bit much to have a GPU running all day just for that.\nWhere it gets more interesting is as a work from home colleague. I can turn it on when my work day starts and ask it questions about ideas or thoughts as they come to me. It is all kept completely local, so don\u0026rsquo;t need to worry about any private or work-related information being shared to the cloud. While I\u0026rsquo;m reviewing something for work I\u0026rsquo;ll ask it to summarise today\u0026rsquo;s news, check what time those outdoor motion sensors went off last night or read out the weekend weather forecast without breaking focus or opening a browser. That\u0026rsquo;s the version of this I keep coming back to, and the one I\u0026rsquo;m most excited to push further: a private assistant I can brainstorm with, that helps juggle the work-and-family calendar, that genuinely feels like a colleague rather than a gimmick.\nIt\u0026rsquo;s early, and it\u0026rsquo;s far from finished but it works, and nothing you say to it ever has to leave your home. If that idea appeals to you, the whole thing is open source and I\u0026rsquo;d love for you to try it. Here\u0026rsquo;s how to get started.\nAll code is at GitHub, fulloch. The HACS integration can be installed directly from the Home Assistant Community Store.\n","date":"14 June 2026","externalUrl":null,"permalink":"/posts/building-a-fully-local-home-voice-assistant/","section":"Posts","summary":"A first year development diary of Fulloch, the Fully Local Home voice assistant.","title":"A Year Building a Fully Local Home Voice Assistant","type":"posts"},{"content":"","date":"14 June 2026","externalUrl":null,"permalink":"/tags/home-assistant/","section":"Tags","summary":"","title":"Home-Assistant","type":"tags"},{"content":" Hi, I\u0026rsquo;m Liam Pettigrew, the person tinkering away on Fulloch.\nI started Fulloch after getting uncomfortable with always-on cloud microphones at home, and wanted to see how far a fully local, self-hosted voice assistant could go on modest hardware.\nYou can find the project and all of its code on GitHub.\n","date":"13 June 2026","externalUrl":null,"permalink":"/about/","section":"Fulloch","summary":"","title":"About the creator","type":"page"}]