Speechmatics Speech-to-Text

Overview

SpeechmaticsSTTService enables real-time speech transcription using Speechmatics’ Agent STT WebSocket API with partial and final results, speaker diarization, and server-side or client-side turn detection.

Speechmatics provides its own user turn start and end detection. When using TurnDetectionMode.VAD, the service runs its own VAD and closes turns itself, automatically requesting ExternalUserTurnStrategies at start. The service proposes each turn boundary and those strategies resolve it, so should_interrupt is the control for barge-in. See User Turn Strategies for more details. A VAD in the transport (such as SileroVADAnalyzer) is optional when Speechmatics drives turn detection; include it if you want useful STT metrics.

Speechmatics Agent STT Reference

The Agent STT WebSocket API this service speaks

Example Implementation

Complete example with interruption handling

Speechmatics Documentation

Speechmatics’ own guide to this integration

Speaker Diarization Guide

Learn about separating different speakers in audio

Installation

To use Speechmatics services, install the required dependencies:

uv add "pipecat-ai[speechmatics]~=1.10.0"

Prerequisites

Speechmatics Account Setup

Before using Speechmatics STT services, you need:

Speechmatics Account: Sign up at Speechmatics
API Key: Generate an API key from your account dashboard
Feature Selection: Configure transcription features like speaker diarization

Select Endpoint

Speechmatics Agent STT supports the following endpoints (defaults to EU2):

Region	Environment	STT Endpoint
EU	EU2 (Default)	`wss://eu2.rt.speechmatics.com/v2/agent`
All regions	Global	`wss://global.rt.speechmatics.com/v2/agent`
EU	EU1	`wss://eu.rt.speechmatics.com/v2/agent`
US	US1	`wss://us.rt.speechmatics.com/v2/agent`
AU	AU1	`wss://au.rt.speechmatics.com/v2/agent`

global.rt.speechmatics.com routes each connection to the nearest region for lowest latency. It may route to any region, so use a regional endpoint if you have data residency requirements. For the full list, see Supported endpoints.

Required Environment Variables

SPEECHMATICS_API_KEY: Your Speechmatics API key for authentication
SPEECHMATICS_RT_URL: Speechmatics endpoint URL (optional, defaults to EU2)

Configuration

SpeechmaticsSTTService

str

default:"None"

Speechmatics API key. Falls back to the SPEECHMATICS_API_KEY environment variable.

str

default:"None"

Base URL for the Speechmatics API. Falls back to SPEECHMATICS_RT_URL environment variable, then defaults to wss://eu2.rt.speechmatics.com/v2/agent.

int

default:"None"

Audio sample rate in Hz. When None, uses the pipeline’s configured sample rate.

AudioEncoding

default:"AudioEncoding.PCM_S16LE"

Audio encoding format. Init-only — not part of runtime-updatable settings.

SpeechmaticsSTTService.InputParams

default:"None"

deprecated

Additional configuration parameters. Deprecated in v0.0.105. Use settings=SpeechmaticsSTTService.Settings(...) instead.

SpeechmaticsSTTService.Settings

default:"None"

Runtime-configurable settings for the STT service. See Settings below.

bool

default:"True"

Whether to interrupt bot output when Speechmatics detects user speech. Only applies when turn_detection_mode is set to VAD. See User Turn Strategies if you pass your own user_turn_strategies.

float

default:"0.74"

P99 latency from speech end to final transcript in seconds. Override for your deployment. See stt-benchmark.

Settings

Runtime-configurable settings passed via the settings constructor argument using SpeechmaticsSTTService.Settings(...). These can be updated mid-conversation with STTUpdateSettingsFrame. See Service Settings for details.

Parameter	Type	Default	Description
`model`	`str`	`"linden-1"`	Transcription model to use. `linden-1` is the only model Agent STT accepts. (Inherited from base STT settings.) Supersedes `operating_point`.
`language`	`Language \| str`	`Language.EN`	Language code for transcription. (Inherited from base STT settings.)
`domain`	`str`	`None`	Domain for Speechmatics API (e.g. for bilingual transcription).
`turn_detection_mode`	`TurnDetectionMode`	`EXTERNAL`	How turns are closed. `EXTERNAL` has the caller drive turns via `finalize()` (e.g. Pipecat’s own VAD); `VAD` lets the STT service run its own VAD and close turns itself.
`speaker_active_format`	`str`	`"{text}"`	Formatter for speaker output. Available attributes: `{speaker_id}`, `{text}`, `{ts}`, `{lang}`. Example: `"@{speaker_id}: {text}"`.
`known_speakers`	`list[SpeakerIdentifier]`	`[]`	Known speaker labels and identifiers for speaker attribution.
`additional_vocab`	`list[AdditionalVocabEntry]`	`[]`	Additional vocabulary to boost recognition of specific words.
`operating_point`	`Model \| str`	`None`	Deprecated since 1.10.0. Resolves to `model`, warning when used. If both are given they must name the same value, otherwise a `ValueError` is raised. Removed in 2.0.0.
`enable_partials`	`bool`	`None`	Emit interim segments as `AddPartialSegment`, updating until the final segment arrives. Agent STT enables partials by default. Set to false to disable partials.
`punctuation_overrides`	`dict`	`None`	Custom punctuation overrides for the STT engine.
`enable_diarization`	`bool`	`None`	Enable speaker diarization to attribute words to unique speakers.
`speaker_sensitivity`	`float`	`None`	Diarization sensitivity. Higher values help distinguish similar voices.
`max_speakers`	`int`	`None`	Maximum number of speakers to detect. Only use when the speaker count is known.
`prefer_current_speaker`	`bool`	`None`	Give extra weight to grouping nearby words as the same speaker.

Turn Detection

The Speechmatics Agent STT service supports two turn detection modes:

External mode (Default)

The default TurnDetectionMode.EXTERNAL hands endpointing to the caller. The service does not endpoint on its own; instead, the caller drives turns by calling finalize(), typically from Pipecat’s own VAD. Use this mode when you want Pipecat’s VAD to control turn boundaries.

transport_params = TransportParams(
    audio_in_enabled=True,
    audio_out_enabled=True,
    vad_analyzer=SileroVADAnalyzer(),  # Use Pipecat's VAD
)

...

stt = SpeechmaticsSTTService(
    api_key=os.getenv("SPEECHMATICS_API_KEY"),
    settings=SpeechmaticsSTTService.Settings(
        language=Language.EN,
        speaker_active_format="<{speaker_id}>{text}</{speaker_id}>",
    ),
)

VAD mode

TurnDetectionMode.VAD lets the Speechmatics service run its own VAD and close turns itself. The service emits StartOfTurn and EndOfTurn events, proposes turn boundaries to Pipecat’s turn strategies, and handles endpointing server-side. In this mode, you do not need a VAD in the transport — the service manages turn detection.

stt = SpeechmaticsSTTService(
    api_key=os.getenv("SPEECHMATICS_API_KEY"),
    settings=SpeechmaticsSTTService.Settings(
        language=Language.EN,
        turn_detection_mode=SpeechmaticsSTTService.TurnDetectionMode.VAD,
        speaker_active_format="<{speaker_id}>{text}</{speaker_id}>",
    ),
)

When using VAD mode, remove any vad_analyzer or turn_analyzer from your transport configuration to avoid conflicts — Speechmatics handles turn detection itself.

Speaker Diarization

Speechmatics STT supports speaker diarization, which separates out different speakers in the audio. The identity of each speaker is returned in the TranscriptionFrame objects in the user_id attribute. If speaker_active_format is provided, then the text output for the TranscriptionFrame will be formatted to this specification. Your system context can then be updated to include information about this format to understand which speaker spoke which words. Examples:

<{speaker_id}>{text}</{speaker_id}> → <S1>Good morning.</S1>.
@{speaker_id}: {text} → @S1: Good morning..

Available attributes

Attribute	Description	Example
`speaker_id`	The label of the speaker	`S1`
`text`	The transcribed text	`Good morning.`
`ts`	Start time of the segment, in seconds	`12.34`
`lang`	The language of the transcription	`en`

Language Support

Refer to the Speechmatics docs for more information on supported languages.

Speechmatics STT supports the following languages and regional variants. Setting a language can be done using the language parameter when creating the STT object. The exception to this is English / Mandarin which has the code cmn_en.

Language Code	Description	Locales
`Language.AR`	Arabic	-
`Language.BA`	Bashkir	-
`Language.EU`	Basque	-
`Language.BE`	Belarusian	-
`Language.BG`	Bulgarian	-
`Language.BN`	Bengali	-
`Language.YUE`	Cantonese	-
`Language.CA`	Catalan	-
`Language.HR`	Croatian	-
`Language.CS`	Czech	-
`Language.DA`	Danish	-
`Language.NL`	Dutch	-
`Language.EN`	English	`en-US`, `en-GB`, `en-AU`
`Language.EO`	Esperanto	-
`Language.ET`	Estonian	-
`Language.FA`	Persian	-
`Language.FI`	Finnish	-
`Language.FR`	French	-
`Language.GL`	Galician	-
`Language.DE`	German	-
`Language.EL`	Greek	-
`Language.HE`	Hebrew	-
`Language.HI`	Hindi	-
`Language.HU`	Hungarian	-
`Language.IA`	Interlingua	-
`Language.IT`	Italian	-
`Language.ID`	Indonesian	-
`Language.GA`	Irish	-
`Language.JA`	Japanese	-
`Language.KO`	Korean	-
`Language.LV`	Latvian	-
`Language.LT`	Lithuanian	-
`Language.MS`	Malay	-
`Language.MT`	Maltese	-
`Language.CMN`	Mandarin	`cmn-Hans`, `cmn-Hant`
`Language.MR`	Marathi	-
`Language.MN`	Mongolian	-
`Language.NO`	Norwegian	-
`Language.PL`	Polish	-
`Language.PT`	Portuguese	-
`Language.RO`	Romanian	-
`Language.RU`	Russian	-
`Language.SK`	Slovakian	-
`Language.SL`	Slovenian	-
`Language.ES`	Spanish	-
`Language.SV`	Swedish	-
`Language.SW`	Swahili	-
`Language.TA`	Tamil	-
`Language.TH`	Thai	-
`Language.TR`	Turkish	-
`Language.UG`	Uyghur	-
`Language.UK`	Ukrainian	-
`Language.UR`	Urdu	-
`Language.VI`	Vietnamese	-
`Language.CY`	Welsh	-

For bilingual transcription, use the language and domain parameters as follows:

Language Code	Description	Domain Options
`cmn_en`	English / Mandarin	-
`en_ms`	English / Malay	-
`Language.ES`	English / Spanish	`bilingual-en`
`en_ta`	English / Tamil	-

Usage Examples

Examples are included in the Pipecat project:

Using Speechmatics STT service -> 07a-interruptible-speechmatics.py
Using Speechmatics STT service with VAD -> 07a-interruptible-speechmatics-vad.py
Transcribing with Speechmatics STT -> 13h-speechmatics-transcription.py

Sample projects:

Guess Who -> Guess Who
Guess Who Board Game -> Guess Who

Basic Configuration

Initialize the SpeechmaticsSTTService and use it in a pipeline:

from pipecat.services.speechmatics.stt import SpeechmaticsSTTService
from pipecat.transcriptions.language import Language

# Configure service
stt = SpeechmaticsSTTService(
    api_key="your-api-key",
    settings=SpeechmaticsSTTService.Settings(
        language=Language.FR,
    )
)

# Use in pipeline
pipeline = Pipeline([
    transport.input(),
    stt,
    context_aggregator.user(),
    llm,
    tts,
    transport.output(),
    context_aggregator.assistant()
])

With Diarization

Enable diarization to attribute transcribed words to unique speakers. Initialize the SpeechmaticsSTTService and use it in a pipeline:

from pipecat.services.speechmatics.stt import SpeechmaticsSTTService
from pipecat.transcriptions.language import Language

# Configure service
stt = SpeechmaticsSTTService(
    api_key="your-api-key",
    settings=SpeechmaticsSTTService.Settings(
        language=Language.EN,
        turn_detection_mode=SpeechmaticsSTTService.TurnDetectionMode.VAD,
        enable_diarization=True,
        speaker_active_format="<{speaker_id}>{text}</{speaker_id}>",
    )
)

# Use in pipeline
pipeline = Pipeline([
    transport.input(),
    stt,
    context_aggregator.user(),
    llm,
    tts,
    transport.output(),
    context_aggregator.assistant()
])

Additional Notes

Connection Management: Automatically handles WebSocket connections and reconnections with exponential backoff. Transient failures trigger automatic reconnection while audio is buffered; permanent errors (auth rejection, bad configuration) mark the service unusable.
Sample Rate: The default sample rate is 16000 Hz in pcm_s16le format
VAD Integration: Server-side VAD and turn detection are available in VAD mode

The InputParams / params= pattern is deprecated as of v0.0.105. Use Settings / settings= instead. See the Service Settings guide for migration details.

Event Handlers

In addition to the standard service connection events (on_connected, on_disconnected, on_connection_error), Speechmatics provides:

Event	Description
`on_speakers_result`	Speaker identification result received

@stt.event_handler("on_speakers_result")
async def on_speakers_result(service, message):
    print(f"Speaker result: {message}")

Get Started

Pipecat Server

Client SDKs

Pipecat Flows

Pipecat Cloud

CLI

Pipecat Context Hub

Speechmatics Speech-to-Text

Overview

Speechmatics Agent STT Reference

Example Implementation

Speechmatics Documentation

Speaker Diarization Guide

Installation

Prerequisites

Speechmatics Account Setup

Select Endpoint

Required Environment Variables

Configuration

SpeechmaticsSTTService

Settings

Turn Detection

External mode (Default)

VAD mode

Speaker Diarization

Available attributes

Language Support

Usage Examples

Basic Configuration

With Diarization

Additional Notes

Event Handlers

​Overview

Speechmatics Agent STT Reference

Example Implementation

Speechmatics Documentation

Speaker Diarization Guide

​Installation

​Prerequisites

​Speechmatics Account Setup

​Select Endpoint

​Required Environment Variables

​Configuration

​SpeechmaticsSTTService

​Settings

​Turn Detection

​External mode (Default)

​VAD mode

​Speaker Diarization

​Available attributes

​Language Support

​Usage Examples

​Basic Configuration

​With Diarization

​Additional Notes

​Event Handlers

Overview

Installation

Prerequisites

Speechmatics Account Setup

Select Endpoint

Required Environment Variables

Configuration

SpeechmaticsSTTService

Settings

Turn Detection

External mode (Default)

VAD mode

Speaker Diarization

Available attributes

Language Support

Usage Examples

Basic Configuration

With Diarization

Additional Notes

Event Handlers