How to Use Text to Speech

Neuphonic's primary offering is its text-to-speech technology, which serves as the foundation for various features, including Agents. Visit our Playground to experiment with different models and voices, and then continue reading below to learn how to implement this in code.

note

Don't forget to visit the Quickstart guide to obtain your API Key and get your environment set up.

Speech Synthesis

You can generate speech using the API in two ways: Server Side Events (SSE) and WebSockets.

Server Side Events (SSE)

SSE is a streaming protocol where you send a single request to our API to convert text into speech. Our API will then stream the generated audio back to you in real-time, providing the lowest possible latency. Below are some examples of how to use this endpoint.

The SDK examples demonstrate how to send a message to the API, receive the audio stream from the server, and play it through your device's speaker.

cURL

# Replace <API_KEY> with your actual API key.
# To switch languages, replace the lang_code in the path parameter (e.g., /en) with the desired language code.
curl -N --request POST \
  --url https://api.neuphonic.com/sse/speak/en \
  --header 'Content-Type: application/json' \
  --header 'X-API-KEY: <API_KEY>' \
  --header 'Accept: text/event-stream' \
  --data '{
  "text": "Hello, world!"
}'

Python SDK

import os
from pyneuphonic import Neuphonic, TTSConfig
from pyneuphonic.player import AudioPlayer

# Ensure the API key is set in your environment
client = Neuphonic(api_key=os.environ.get('NEUPHONIC_API_KEY'))

sse = client.tts.SSEClient()

# TTSConfig is a pydantic model so check out the source code for all valid options
tts_config = TTSConfig(
    lang_code='en', # replace the lang_code with the desired language code.
    sampling_rate=24000,
)

# Create an audio player with `pyaudio`
# Make sure you use the same sampling rate as in the TTSConfig
with AudioPlayer(sampling_rate=24000) as player:
    response = sse.send('Hello, world!', tts_config=tts_config)
    player.play(response)

JavaScript SDK

import fs from 'fs';
import { createClient, toWav } from '@neuphonic/neuphonic-js';
const client = createClient({ apiKey: '<API_KEY>'});

const msg = `Hello how are you?<STOP>`;

const sse = await client.tts.sse({
  speed: 1.15,
  lang_code: 'en'
});

const res = await sse.send(msg);

// Saving data to file
const wav = toWav(res.audio);
fs.writeFileSync(__dirname + '/sse.wav', wav);

WebSockets

WebSockets offer a powerful way to create a two-way communication channel between your application and the server. This means you can send messages to the server and receive responses back in real-time. WebSockets are ideal for situations where you need the lowest possible latency, as they allow you to send text to the server in small, easy-to-manage pieces.

This protocol is especially useful for applications that involve continuous text streams, such as when connecting Neuphonic TTS to a language model like GPT-4o, which generates responses word by word. This setup ensures minimal latency and makes it easy to integrate with other services, as audio is streamed back to you as soon as it is generated.

Because WebSockets work by streaming data, you need to use special <STOP> tokens to tell the server when a piece of text is complete and ready to be processed. An appropriate example of when the <STOP> symbol could be used is at the end of a stream of messages output from a language model.

This is all demonstrated in the SDK examples below. They demonstrate how to send messages to the API and attach event handlers to play the received audio through your device's speaker.

wscat

# First do `npm install -g wscat`
wscat -c "wss://api.neuphonic.com/speak/en?api_key=<API_KEY>&speed=1.0&lang_code=en&sampling_rate=24000"
# And then type "Hello, World! <STOP>" and hit "Enter"

Python SDK

import os
import asyncio
from pyneuphonic import Neuphonic, TTSConfig, WebsocketEvents
from pyneuphonic.models import APIResponse, TTSResponse
from pyneuphonic.player import AsyncAudioPlayer

async def main():
    client = Neuphonic(api_key=os.environ.get('NEUPHONIC_API_KEY'))

    # Create a websocket object from the client
    ws = client.tts.AsyncWebsocketClient()

    # Set the desired configuration, see the TTSConfig model for more configuration options
    tts_config = TTSConfig(lang_code='es',voice_id='<VOICE_ID>', sampling_rate=24000, speed=1.0, encoding="pcm_linear")

    # Create and start the audio player, which will output audio to your device's speaker
    player = AsyncAudioPlayer()
    await player.open()

    # Attach event handlers. Check WebsocketEvents enum for all valid events.
    async def on_message(message: APIResponse[TTSResponse]):
        """Play audio through the speaker as it is received."""
        await player.play(message.data.audio)

    async def on_close():
        """Close the audio player when the websocket closes."""
        await player.close()

    ws.on(WebsocketEvents.MESSAGE, on_message)
    ws.on(WebsocketEvents.CLOSE, on_close)

    await ws.open(tts_config=tts_config)  # create the connection with the server

    # A special symbol ' <STOP>' must be sent to the server, otherwise the server will wait for
    # more text to be sent before generating the last few snippets of audio
    await ws.send('¡Hola, cómo estás!', autocomplete=True)
    await ws.send('¡Hola, cómo estás! <STOP>')  # Both the above line, and this line, are equivalent

    await asyncio.sleep(15)  # let the audio play
    player.save_audio('output.wav')  # save the audio to a .wav file
    await ws.close()  # close the websocket and terminate the audio resources

asyncio.run(main())

JavaScript SDK

import fs from 'fs';
import { createClient, toWav } from '@neuphonic/neuphonic-js';
const client = createClient({ apiKey: '<API_KEY>'});

const msg = `Hello how are you?<STOP>`;

const ws = await client.tts.websocket({
  speed: 1.15,
  lang_code: 'en'
});

let byteLen = 0;
const chunks = [];

// Websocket allow us to get chunk of the data as soon as they ready
// which can make API more responsve
for await (const chunk of ws.send(msg)) {
  // here you can send the data to the client
  // or collect it in array and save as a file
  chunks.push(chunk.audio);
  byteLen += chunk.audio.byteLength;
}

// Merging all chunks into single Uint8Array array
let offset = 0;
const allAudio = new Uint8Array(byteLen);
chunks.forEach(chunk => {
  allAudio.set(chunk, offset);
  offset += chunk.byteLength;
})

// Saving data to file
const wav = toWav(allAudio);
fs.writeFileSync(__dirname + '/ws.wav', wav);

await ws.close(); // closing the socket if we don't want to send anything

Return Configuration

If you call the websocket directly the returned response is a JSON object with a structure that contains audio data, sampling rate, the input text and other details. Below is a breakdown of the format:

{
  "data": {
    "audio": "<audio_data_string>",
    "text": "<input_text>",
    "sampling_rate": 24000 // Default value
  }
}

With timestamps=true each message also carries an alignment object. See Word Timings.

The websocket connection has a 60 second timeout, so if you don't send any messages within that time, the connection will close automatically.

warning

The chosen voice needs to be available for the chosen language.

Text-to-Speech Configuration

The settings for Text-to-Speech generation can include the following parameters.

Name	Type	Description
lang_code	string	Language code for the desired language. See the full Languages page for all supported codes. Common values: 'en' (English), 'es' (Spanish), 'zh' (Chinese)
voice_id	string	The voice ID for the desired voice. Based on what voice_id you chose different models will be leveraged. Examples: '8e9c4bc8-3979-48ab-8626-df53befc2090'
speed	float	Playback speed of the audio. Supported values snap to 0.7 (slow), 1.0 (normal), or 1.5 (fast). Any value <1.0 is treated as slow, >1.0 as fast.
sampling_rate	int	Sampling rate of the audio returned from the server. Options: 8000, 16000, 22050, 24000. Default: 24000
encoding	string	Encoding of the audio returned from the server. Options: 'pcm_linear', 'pcm_mulaw'
timestamps	boolean	Return word timings alongside the audio, for highlighting text as it is spoken. See Word Timings. Default: false
ipa_key	string	Use a regional variant's sounds for <phoneme:...> tags. See Pronunciation. Options: 'es-castilian', 'pt-br'

Word Timings

Set timestamps=true to get the start time and length of every word alongside the audio, so you can highlight text as it is spoken.

wss://api.neuphonic.com/speak/en?api_key=<API_KEY>&timestamps=true

Messages then carry an alignment next to the audio:

{
  "data": {
    "audio": "<audio_data_string>",
    "sampling_rate": 24000,
    "alignment": {
      "units":            ["The", "lens", "costs", "$42", "today."],
      "unitStartTimesMs": [100, 240, 560, 900, 1780],
      "unitDurationsMs":  [140, 300, 340, 880,  380],
      "charStartOffsets": [0, 4, 9, 15, 19],
      "charEndOffsets":   [3, 8, 14, 18, 25]
    }
  }
}

The five arrays line up: index i of each describes the same word.

Field	Meaning
units	The word, as you sent it
unitStartTimesMs	When it starts, in milliseconds from the beginning of the audio
unitDurationsMs	How long it lasts, in milliseconds
charStartOffsets	Where the word starts in your text
charEndOffsets	Where it ends (exclusive)

Reading them

Collect entries as they arrive, keyed by index across the five arrays, and highlight whichever one covers the current playback position (`ms >= unitStartTimesMs[i] && ms < unitStartTimesMs[i]

• unitDurationsMs[i]) by selecting myText.slice(charStartOffsets[i], charEndOffsets[i])`. Times are measured from the start of the stream, so they stay correct whatever your text's length, and some messages carry no entries while others carry several — just keep appending.

Things to expect

• Offsets point at your original text, not at expanded speech — $42 comes back as one entry spanning its whole read-out, and 東京 stays 東京, not its pronunciation.
• Not every character is covered: spaces and most punctuation have no sound of their own.
• Some voices have no timings. alignment is then null, with alignment_unavailable explaining why, so you can tell "none coming" from "not yet".
• Every language is supported; for scripts written without spaces (Japanese, Chinese, Thai) a "word" is the smaller unit natural to that script.

Pronunciation

If a word is said wrongly, look it up in the pronunciation tool on our dashboard to get its correct IPA transcription, then wrap it in a <phoneme:...> tag inline in your text:

"Please welcome <phoneme:ˈdeɪtə>, our guest."

Only the tagged word changes; the rest of the sentence is spoken as normal. Bare, /slashed/, or [bracketed] IPA all work, and ˈ marks the stressed syllable.

Available in ten languages (English, German, Dutch, French, Italian, Spanish, Portuguese, Russian, Hindi and Korean) — use the phonemes of the language you're synthesising, or the request is rejected. For Castilian Spanish or Brazilian Portuguese, add ipa_key=es-castilian or ipa_key=pt-br to use that variant's sounds.

Inline Pauses

Insert silence anywhere in your text for pacing, using <pause:500ms>, <pause:2s>, or SSML's <break time="500ms"/>:

"Welcome to Neuphonic. <pause:1s> Your audio is being generated."
"And the winner is ... <break time="2s"/> you!"

Billed at 1 character per second of silence.

Providing a context ID

You can provide a context ID to uniquely identify the audio chunks generated for a given text input. This ID is returned with the corresponding audio, allowing you to track, interrupt, or associate the audio with its original request.

Python SDK

client = Neuphonic()

ws = client.tts.AsyncWebsocketClient()
audio_bytes = bytearray()

async def on_message(message: APIResponse[TTSResponse]):
    nonlocal audio_bytes
    audio_bytes += message.data.audio
    print(message.data.context_id) # here the context_id will be "1"

ws.on(WebsocketEvents.MESSAGE, on_message)

await ws.open()

await ws.send(
    {"text": "This is message one example.<STOP>", "context_id": "1" }
)

JavaScript SDK

import { createClient } from '@neuphonic/neuphonic-js';

const client = createClient();

const ws = await client.tts.websocketCb({
  lang_code: 'en'
});

const msg = `Hello how are you?<STOP>`;

let byteLen = 0;
const chunks: Uint8Array[] = [];

ws.onMessage((chunk) => {
  byteLen += chunk.audio.byteLength;
  chunks.push(chunk.audio);
  console.log({ ...chunk, audio: chunk.audio.byteLength }); // here the context_id will be "1"
});

ws.send(msg, {  context_id: '1' });

Resetting the Stream

Send <RESET> as a message to discard everything currently buffered — any text not yet spoken and the voice's carried-over prosody — without closing the connection. The next text you send starts a fresh utterance, and its word timings (if timestamps=true) start again from zero:

await ws.send('<RESET>')

Use it to interrupt what's currently being generated, e.g. on barge-in, without paying the cost of reopening the websocket.

More Examples

To see more examples, check out our Python SDK examples and JavaScript SDK examples on GitHub.