How to Use Agents

Agents allow you to create, manage, and interact with intelligent multilingual AI assistants. Follow the example below to create a simple agent and start speaking with it. Navigate the tabs to see how you can easily change from english speaking agents to any available language.

import os
import asyncio

# See AgentConfig model for full list of parameters to configure the agent
from pyneuphonic import Neuphonic, Agent, AgentConfig

async def main():
    client = Neuphonic(api_key=os.environ.get('NEUPHONIC_API_KEY'))

    agent_id = client.agents.create(
        name='Agent 1',
        prompt='You are a helpful agent. Answer in 10 words or less.',
        greeting='Hi, how can I help you today?'
    ).data['agent_id']

    # All additional keyword arguments (such as `agent_id` and `tts_model`) are passed as
    # parameters to the model. See AgentConfig model for full list of parameters.
    agent = Agent(client, agent_id=agent_id, tts_model='neu_hq')

    try:
        await agent.start()

        while True:
            await asyncio.sleep(1)
    except KeyboardInterrupt:
        await agent.stop()

asyncio.run(main())

import os
import asyncio

# See AgentConfig model for full list of parameters to configure the agent
from pyneuphonic import Neuphonic, Agent, AgentConfig

async def main():
    client = Neuphonic(api_key=os.environ.get('NEUPHONIC_API_KEY'))

    agent_id = client.agents.create(
        name='Agent 1',
        prompt='You are a helpful agent. Answer in 10 words or less.',
        greeting='Hi, how can I help you today?'
    ).data['agent_id']

    # All additional keyword arguments (such as `agent_id` and `tts_model`) are passed as
    # parameters to the model. See AgentConfig model for full list of parameters.
    agent = Agent(client, agent_id=agent_id, tts_model='neu_hq')

    try:
        await agent.start()

        while True:
            await asyncio.sleep(1)
    except KeyboardInterrupt:
        await agent.stop()

asyncio.run(main())

import os
import asyncio

# See AgentConfig model for full list of parameters to configure the agent
from pyneuphonic import Neuphonic, Agent, AgentConfig

async def main():
    client = Neuphonic(api_key=os.environ.get('NEUPHONIC_API_KEY'))

    agent_id = client.agents.create(
        name='Asistente',  # Assistant
        prompt='Eres un agente español útil.',  # You are a helpful spanish agent.
        greeting='¿Cómo puedo ayudarte hoy?',  # How can I help you today?
    ).data['agent_id']

    # All additional keyword arguments (such as `agent_id`) are passed as
    # parameters to the model. See AgentConfig model for full list of parameters.
    agent = Agent(
        client,
        agent_id=agent_id,
        lang_code='es',
        voice_id='<VOICE_ID>',
    )

    try:
        await agent.start()

        while True:
            await asyncio.sleep(1)
    except KeyboardInterrupt:
        await agent.stop()


asyncio.run(main())s

When you run the example, the agent will greet you with “Hi, how can I help you today?”. Your spoken input is then converted into text by a speech recognition system. The prompt you set above is used as the system prompt for a GPT-4o model, which generates the responses. These responses are then converted to audio using Neuphonic’s TTS and played back to you. You can keep the conversation going for as long as you want!

After creating an agent with client.agents.create, you can access and manage it in the Voice Agents dashboard. The returned agent_id allows you to reuse this agent configuration across multiple sessions or applications. You can also modify the agent’s prompt, greeting, and other settings directly through the dashboard interface for quick iterations without changing your code.

​

Editing the Callback

The Agent interface described above utilizes a WebSocket connection to facilitate real-time audio streaming between your microphone and the server. The server processes this audio and sends back responses, which can be categorized into the following types:

{
    "type": "user_transcript",
    "text": "<TRANSCRIPTION>"
}

{
    "type": "user_transcript",
    "text": "<TRANSCRIPTION>"
}

{
    "type": "stop_audio_response",
}

{
    "type": "llm_response",
    "text": "<LLM_RESPONSE>"
}

{
    "type": "audio_response",
    "audio_url": "<AUDIO_BYTES>"
}

The Agent class, by default, plays audio_response messages and prints llm_response and user_transcript messages to the console. To add custom behavior, you can attach a custom event handler which perform extra actions on each event, as shown below.

import os
import asyncio

from pyneuphonic import Neuphonic, Agent, AgentConfig
from pyneuphonic.models import APIResponse, AgentResponse

def on_message(message: APIResponse[AgentResponse]):
    if message.data.type == 'user_transcript':
        print(f'Received user_transcript')
    elif message.data.type == 'llm_response':
        print(f'Received llm_response')
    elif message.data.type == 'audio_response':
        print(f'Received audio_response. Playing audio.')
    elif message.data.type == 'stop_audio_response':
        print(f'Received stop_audio_response.')

async def main():
    client = Neuphonic(api_key=os.environ.get('NEUPHONIC_API_KEY'))

    agent_id = client.agents.create(
        name='Agent 1',
        prompt='You are a helpful agent. Answer in 10 words or less.',
        greeting='Hi, how can I help you today?'
    ).data['agent_id']

    agent = Agent(
        client,
        agent_id=agent_id,
        tts_model='neu_hq',
        on_message=on_message,  # attach the custom callback
    )

    try:
        await agent.start()

        while True:
            await asyncio.sleep(1)
    except KeyboardInterrupt:
        await agent.stop()

asyncio.run(main())

If you need more control than the Agent class provides, you can implement a custom solution using the underlying websocket API. This approach gives you direct control over audio input/output through custom websocket event handlers, allowing for more advanced or specialized implementations. For an example on how to do this, see this example on the pyneuphonic GitHub examples section.

​

View Agents

To retrieve a list of all your existing agents

curl -X GET "https://eu-west-1.api.neuphonic.com/agents" \
     -H "X-API-KEY: <API_KEY>"

The previous response does not include prompt and greeting because they can be quite lengthy, especially for complex agents with extensive system prompts. The following request will return all details for the specified agent_id.

curl -X GET "http://eu-west-1.api.neuphonic.com/agents/<AGENT_ID>" \
     -H "X-API-KEY: <API_KEY>"

​

More Examples

To see more examples with our Python SDK, head over to the GitHub repo agents examples section.