OpenTelemetry Tracing

Overview

Pipecat includes built-in support for OpenTelemetry tracing, allowing you to gain deep visibility into your voice applications. Tracing helps you:

Track latency and performance across your conversation pipeline
Monitor service health and identify bottlenecks
Visualize conversation turns and service dependencies
Collect usage metrics and operational analytics

Installation

To use OpenTelemetry tracing with Pipecat, install the tracing dependencies:

pip install "pipecat-ai[tracing]"

For local development and testing, we recommend using Jaeger as a trace collector. You can run it with Docker:

docker run -d --name jaeger \
  -p 16686:16686 \
  -p 4317:4317 \
  -p 4318:4318 \
  jaegertracing/all-in-one:latest

Then access the UI at http://localhost:16686

Basic Setup

Enabling tracing in your Pipecat application requires two steps:

Initialize the OpenTelemetry SDK with your preferred exporter
Enable tracing in your PipelineTask

import os
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from pipecat.utils.tracing.setup import setup_tracing
from pipecat.pipeline.task import PipelineTask, PipelineParams

# Step 1: Initialize OpenTelemetry with your chosen exporter
exporter = OTLPSpanExporter(
    endpoint="http://localhost:4317",  # Jaeger or other collector endpoint
    insecure=True,
)

setup_tracing(
    service_name="my-voice-app",
    exporter=exporter,
    console_export=False,  # Set to True for debug output
)

# Step 2: Enable tracing in your PipelineTask
task = PipelineTask(
    pipeline,
    params=PipelineParams(
        enable_metrics=True,                              # Required for some service metrics
    ),
    enable_tracing=True,                                  # Enable tracing for this task
    enable_turn_tracking=True,                            # Enable turn tracking for this task
    conversation_id="customer-123",                       # Optional - will auto-generate if not provided
    additional_span_attributes={"session.id": "abc-123"} # Optional - additional attributes to attach to the otel span
)

For complete working examples, see our sample implementations:

Jaeger Tracing Example - Uses gRPC exporter with Jaeger
Langfuse Tracing Example - Uses HTTP exporter with Langfuse for LLM-focused observability

Trace Structure

Pipecat organizes traces hierarchically, following the natural structure of conversations:

Conversation (conversation)
├── turn
│   ├── stt
│   ├── llm
│   └── tts
└── turn
    ├── stt
    ├── llm
    └── tts
    turn...

For real-time multimodal services like Gemini Live and OpenAI Realtime, the structure adapts to their specific patterns:

Conversation (conversation)
├── turn
│   ├── llm_setup (session configuration)
│   ├── stt (user input)
│   ├── llm_response (complete response with usage)
│   └── llm_tool_call/llm_tool_result (for function calls)
└── turn
    ├── stt (user input)
    └── llm_response (complete response)
    turn...

This hierarchical structure makes it easy to:

Track the full lifecycle of a conversation
Measure latency for individual turns
Identify which services are contributing to delays
Compare performance across different conversations

Exporter Options

Pipecat supports any OpenTelemetry-compatible exporter. Common options include:

OTLP Exporter (for Jaeger, Grafana, etc.)

from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

exporter = OTLPSpanExporter(
    endpoint="http://localhost:4317",  # Your collector endpoint
    insecure=True,  # Use False for TLS connections
)

HTTP OTLP Exporter (for Langfuse, etc.)

from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

exporter = OTLPSpanExporter(
    # Configure with environment variables:
    # OTEL_EXPORTER_OTLP_ENDPOINT
    # OTEL_EXPORTER_OTLP_HEADERS
)

See our Langfuse example for details on configuring this exporter.

Console Exporter (for debugging)

The console exporter can be enabled alongside any other exporter by setting console_export=True:

setup_tracing(
    service_name="my-voice-app",
    exporter=otlp_exporter,
    console_export=True,  # Prints traces to stdout
)

Cloud Provider Exporters

Many cloud providers offer OpenTelemetry-compatible observability services:

AWS X-Ray
Google Cloud Trace
Azure Monitor
Datadog APM

Check the OpenTelemetry documentation for specific exporter configurations: OpenTelemetry Vendors

OpenInference

Arize-ai provides OpenInference instrumentation, compatible with OpenTelemetry.

openinference-instrumentation-pipecat

See Observability community integrations for more details.

Span Attributes

Pipecat enriches spans with detailed attributes about service operations:

TTS Service Spans

gen_ai.system: Service provider (e.g., “cartesia”)
gen_ai.request.model: Model ID/name
voice_id: Voice identifier
text: The text being synthesized
metrics.character_count: Number of characters in the text
metrics.ttfb: Time to first byte in seconds
settings.*: Service-specific configuration parameters

STT Service Spans

gen_ai.system: Service provider (e.g., “deepgram”)
gen_ai.request.model: Model ID/name
transcript: The transcribed text
is_final: Whether the transcription is final
language: Detected or configured language
vad_enabled: Whether voice activity detection is enabled
metrics.ttfb: Time to first byte in seconds
settings.*: Service-specific configuration parameters

LLM Service Spans

gen_ai.system: Service provider (e.g., “openai”, “gcp.gemini”)
gen_ai.request.model: Model ID/name
gen_ai.operation.name: Operation type (e.g., “chat”)
stream: Whether streaming is enabled
input: JSON-serialized input messages
output: Complete response text
tools: JSON-serialized tools configuration
tools.count: Number of tools available
tools.names: Comma-separated tool names
system: System message content
gen_ai.usage.input_tokens: Number of prompt tokens
gen_ai.usage.output_tokens: Number of completion tokens
metrics.ttfb: Time to first byte in seconds
gen_ai.request.*: Standard parameters (temperature, max_tokens, etc.)

Multimodal Service Spans (Gemini Live & OpenAI Realtime)

Setup Spans

gen_ai.system: “gcp.gemini” or “openai”
gen_ai.request.model: Model identifier
tools.count: Number of available tools
tools.definitions: JSON-serialized tool schemas
system_instruction: System prompt (truncated)
session.*: Session configuration parameters

Request Spans (OpenAI Realtime)

input: JSON-serialized context messages being sent
gen_ai.operation.name: “llm_request”

Response Spans

output: Complete assistant response text
output_modality: “TEXT” or “AUDIO” (Gemini Live)
gen_ai.usage.input_tokens: Prompt tokens used
gen_ai.usage.output_tokens: Completion tokens generated
function_calls.count: Number of function calls made
function_calls.names: Comma-separated function names
metrics.ttfb: Time to first response in seconds

Tool Call/Result Spans (Gemini Live)

tool.function_name: Name of the function being called
tool.call_id: Unique identifier for the call
tool.arguments: Function arguments (truncated)
tool.result: Function execution result (truncated)
tool.result_status: “completed”, “error”, or “parse_error”

Turn Spans

turn.number: Sequential turn number
turn.type: Type of turn (e.g., “conversation”)
turn.duration_seconds: Duration of the turn
turn.was_interrupted: Whether the turn was interrupted
conversation.id: ID of the parent conversation

Conversation Spans

conversation.id: Unique identifier for the conversation
conversation.type: Type of conversation (e.g., “voice”)

Usage Metrics

Pipecat’s tracing implementation automatically captures usage metrics for LLM and TTS services:

LLM Token Usage

Token usage is captured in LLM spans as:

gen_ai.usage.input_tokens
gen_ai.usage.output_tokens

TTS Character Count

Character count is captured in TTS spans as:

metrics.character_count

Performance Metrics

Pipecat traces capture key performance metrics for each service:

Time To First Byte (TTFB)

The time it takes for a service to produce its first response:

metrics.ttfb (in seconds)

Processing Duration

The total time spent processing in each service is captured in the span duration.

Configuration Options

PipelineTask Parameters

bool

default:"True"

Enable or disable tracing for the pipeline

bool

default:"False"

Whether to enable turn tracking.

Optional[str]

default:"None"

Custom ID for the conversation. If not provided, a UUID will be generated

Optional[dict]

default:"None"

Any additional attributes to add to top-level OpenTelemetry conversation span.

setup_tracing() Parameters

str

default:"pipecat"

Name of the service for traces

Optional[SpanExporter]

default:"None"

A pre-configured OpenTelemetry span exporter instance

bool

default:"False"

Whether to also export traces to console (useful for debugging)

Example

Here’s a complete example showing OpenTelemetry tracing setup with Jaeger:

import os
from dotenv import load_dotenv
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.services.deepgram.stt import DeepgramSTTService
from pipecat.services.openai.llm import OpenAILLMService
from pipecat.services.cartesia.tts import CartesiaTTSService
from pipecat.utils.tracing.setup import setup_tracing

load_dotenv()

# Initialize tracing if enabled
if os.getenv("ENABLE_TRACING"):
    # Create the exporter
    otlp_exporter = OTLPSpanExporter(
        endpoint=os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://localhost:4317"),
        insecure=True,
    )

    # Set up tracing with the exporter
    setup_tracing(
        service_name="pipecat-demo",
        exporter=otlp_exporter,
        console_export=bool(os.getenv("OTEL_CONSOLE_EXPORT")),
    )

# Create your services
stt = DeepgramSTTService(api_key=os.getenv("DEEPGRAM_API_KEY"))
llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"))
tts = CartesiaTTSService(
    api_key=os.getenv("CARTESIA_API_KEY"),
    voice_id="your-voice-id"
)

# Build pipeline
pipeline = Pipeline([
    transport.input(),
    stt,
    context_aggregator.user(),
    llm,
    tts,
    transport.output(),
    context_aggregator.assistant(),
])

# Create pipeline task with tracing enabled
task = PipelineTask(
    pipeline,
    params=PipelineParams(
        enable_metrics=True,
        enable_usage_metrics=True,
    ),
    enable_tracing=True,
    enable_turn_tracking=True,
    conversation_id="customer-123",                       # Optional - will auto-generate if not provided
    additional_span_attributes={"session.id": "abc-123"} # Optional - additional attributes to attach to the otel span
)

# Run the pipeline
runner = PipelineRunner()
await runner.run(task)

Troubleshooting

If you’re having issues with tracing:

No Traces Visible: Ensure the OpenTelemetry packages are installed and that your collector endpoint is correct
Missing Service Data: Verify that enable_metrics=True is set in PipelineParams
Debugging Tracing: Enable console export with console_export=True to view traces in your logs
Connection Errors: Check network connectivity to your trace collector
Collector Configuration: Verify your collector is properly set up to receive traces

Pipecat Server

Pipecat Flows

Pipecat Cloud

CLI

OpenTelemetry Tracing

Overview

Installation

Basic Setup

Trace Structure

Exporter Options

OTLP Exporter (for Jaeger, Grafana, etc.)

HTTP OTLP Exporter (for Langfuse, etc.)

Console Exporter (for debugging)

Cloud Provider Exporters

OpenInference

Span Attributes

TTS Service Spans

STT Service Spans

LLM Service Spans

Multimodal Service Spans (Gemini Live & OpenAI Realtime)

Setup Spans

Request Spans (OpenAI Realtime)

Response Spans

Tool Call/Result Spans (Gemini Live)

Turn Spans

Conversation Spans

Usage Metrics

LLM Token Usage

TTS Character Count

Performance Metrics

Time To First Byte (TTFB)

Processing Duration

Configuration Options

PipelineTask Parameters

setup_tracing() Parameters

Example

Troubleshooting

References

​Overview

​Installation

​Basic Setup

​Trace Structure

​Exporter Options

​OTLP Exporter (for Jaeger, Grafana, etc.)

​HTTP OTLP Exporter (for Langfuse, etc.)

​Console Exporter (for debugging)

​Cloud Provider Exporters

​OpenInference

​Span Attributes

​TTS Service Spans

​STT Service Spans

​LLM Service Spans

​Multimodal Service Spans (Gemini Live & OpenAI Realtime)

​Setup Spans

​Request Spans (OpenAI Realtime)

​Response Spans

​Tool Call/Result Spans (Gemini Live)

​Turn Spans

​Conversation Spans

​Usage Metrics

​LLM Token Usage

​TTS Character Count

​Performance Metrics

​Time To First Byte (TTFB)

​Processing Duration

​Configuration Options

​PipelineTask Parameters

​setup_tracing() Parameters

​Example

​Troubleshooting

​References

Overview

Installation

Basic Setup

Trace Structure

Exporter Options

OTLP Exporter (for Jaeger, Grafana, etc.)

HTTP OTLP Exporter (for Langfuse, etc.)

Console Exporter (for debugging)

Cloud Provider Exporters

OpenInference

Span Attributes

TTS Service Spans

STT Service Spans

LLM Service Spans

Multimodal Service Spans (Gemini Live & OpenAI Realtime)

Setup Spans

Request Spans (OpenAI Realtime)

Response Spans

Tool Call/Result Spans (Gemini Live)

Turn Spans

Conversation Spans

Usage Metrics

LLM Token Usage

TTS Character Count

Performance Metrics

Time To First Byte (TTFB)

Processing Duration

Configuration Options

PipelineTask Parameters

setup_tracing() Parameters

Example

Troubleshooting

References