How do I give memory to an AI voice assistant that doesn't rely on text history?

Voice assistants face a starker version of the memory problem than text chatbots. In a voice interaction, there is no persistent message array. The audio is processed, transcribed, and turned into a response — then it is gone. Each voice session is a clean slate unless you explicitly build state around it.

Why storing full transcripts does not work

Voice transcripts are verbose. A 3-minute conversation might produce 500 words of transcript — expensive to embed and retrieve. More critically, voice conversations include filler words, false starts, and conversational noise that degrades retrieval quality if stored raw. The signal-to-noise ratio in voice transcripts is significantly lower than in typed text.

The right pattern: extract, not store

After each voice session, pass the transcript to an extraction pipeline rather than storing it wholesale. The pipeline identifies facts worth preserving and writes only those to the memory store. The transcript is discarded or archived separately for compliance. On the next session, the extracted facts are retrieved and injected into the system prompt before the first response is generated.

from mem0 import Memory

memory = Memory()

# After voice session ends — transcript from Whisper or Deepgram
transcript = [
    {"role": "user", "content": "Yeah so I want reminders at 8am, I'm an early riser"},
    {"role": "assistant", "content": "Got it, I'll set your default reminder time to 8am"},
    {"role": "user", "content": "And I'm training for a half marathon so fitness stuff is important"},
]

# Extract and store — Mem0 identifies what is worth keeping
memory.add(transcript, user_id="voice_user_88")

# Extracted and stored:
# "User prefers 8am reminders"
# "User is training for a half marathon"
# "Fitness is a priority topic for this user"

# Next session — inject before first response
def start_voice_session(user_id: str, first_utterance: str) -> str:
    memories = memory.search(first_utterance, user_id=user_id, limit=5)
    return "\n".join([m["memory"] for m in memories["results"]])

Integrating with voice platforms

Voice AI platforms including Vapi, ElevenLabs Conversational AI, and LiveKit Agents all support custom system prompt injection at session start and webhook callbacks at session end. The integration pattern is: on session start — retrieve memories, inject into system prompt, begin voice session. On session end — receive transcript via webhook, extract facts, write to memory store.

This gives users a voice assistant that greets them by name, remembers their preferences from past sessions, and skips re-asking questions it already knows the answers to.