Add native Coqui XTTS v2 voice cloning

[DrewThomasson]

Summary

• add a native Coqui XTTS v2 family for offline multilingual TTS and zero-shot voice cloning
• implement reference mel frontends, conditioning encoder + Perceiver, speaker encoder, 30-layer autoregressive GPT, and speaker-conditioned HiFiGAN decoder with ggml
• add BPE tokenization, request validation, session/registry integration, conversion tooling, documentation, and F16/Q8_0 GGUF packages
• preserve the conditioning Perceiver in F32 to avoid half-precision NaNs and keep convolution/embedding/logit-sensitive tensors at safe precision in Q8_0
• match Hugging Face repetition-penalty semantics, including XTTS's synthetic prefix ids, so generation emits natural EOS instead of becoming rough or overlong

Model artifacts

Hugging Face PR: https://huggingface.co/audio-cpp/audio.cpp-gguf/discussions/9

• xtts-v2-q8_0.gguf: 616,802,944 bytes, SHA-256 e07fd16d7365575fc9fe3946a70e9510f73f904983459d8eef060e91e62cfd4a
• xtts-v2-f16.gguf: 976,290,816 bytes, SHA-256 1db573bd06510f9fb8513913cc4a20b3b4340d23f918dbaf7dde695dda1ce055

Both packages embed the tokenizer, config, CPML license, and this model spec, and both pass the conditioning → GPT → HiFiGAN native probe.

Validation

• complete CPU build with model and unit-test targets enabled
• native speaker embedding vs Coqui: correlation 0.999999999993, RMSE 1.66e-7
• native HiFiGAN vs Coqui for identical inputs: correlation 0.99999999983, RMSE 6.31e-7
• Q8_0 GPT prefill vs Coqui: logits correlation 0.9999105, identical argmax token 808
• final Q8_0 CLI synthesis: 24 kHz mono, 2.965 seconds, natural EOS
• independent Whisper ASR of final demo: Hello? This is a native voice clone. for input Hello, this is a native voice clone.
• CTest: 54 passed, 4 skipped for unavailable model fixtures, one unrelated existing macOS /var vs /private/var alias failure in fun_asr_nano_assets_test

The XTTS model and generated output are governed by the Coqui Public Model License 1.0.0 and are non-commercial.

[DrewThomasson]

Native Q8_0 synthesis demo

Generated end-to-end by this PR's audiocpp_cli using the published Q8_0 GGUF and Coqui's bundled English reference clip. The input text is: “Hello, this is a native voice clone.” The MP4 contains H.264 video plus 24 kHz AAC audio so it plays inline on GitHub.

xtts-v2-audiocpp-demo.mp4

Independent Whisper-base.en transcription: “Hello? This is a native voice clone.”

[DrewThomasson]

Same as usual if this passes your requirements this also needs to be merged

https://huggingface.co/audio-cpp/audio.cpp-gguf/discussions/9

👍

[DrewThomasson]

Synthetic voice-cloning demo — alternate reference

This is an explicitly labeled synthetic test using the user-supplied DavidAttenborough.wav reference. It was generated end-to-end by this PR's native Q8_0 audiocpp_cli; it is not a recording of David Attenborough.

Spoken text: “This is a synthetic voice clone generated locally by audio dot C P P. It is not a real recording of David Attenborough.”

xtts-v2-attenborough-demo.mp4

[DrewThomasson]

Hm that output is good but... perhaps quantization the model has lowered the voice cloning ability

[mirek190]

That model is VERY old ...so you can not expect would work as good as nowadays models.

[DrewThomasson]

No....

We use XttsV2 all the time in Ebook2audiobook and the voice cloning as well as pronunciation quality is MUCH better when using full precision

some quality reduction has defiantly occurred when turning this into a gguf, perhaps I need to shoot for a higher than f16

[DrewThomasson]

I mean this was a output of out xttsV2 raw which sounds a lot more like David

shortStory.mp4

Hmmmm

[mirek190]

Did you ask your AI agent to compare audio quality during making a port?
Then could catch when that is happening.

I think the easiest way i star from beginning and comparing quality on each step.