AvatarLayer — pluggable SDK for realtime conversational avatars

AvatarOrchestrator is a multi-character orchestration layer that coordinates multiple avatars from a single sendMessage() call. It handles LLM communication, stream parsing, and turn routing so you can build panel discussions, Socratic dialogues, and multi-character scenes without wiring the plumbing yourself.

Quick start

import { AvatarOrchestrator } from "avatarlayer";
import { VRMLocalRenderer } from "avatarlayer/renderers";

const orchestrator = new AvatarOrchestrator({
  llm,
  characters: [
    { id: "alpha", name: "Alpha", systemPrompt: "You are Alpha, a confident strategist.", tts, renderer: alphaRenderer },
    { id: "beta",  name: "Beta",  systemPrompt: "You are Beta, a sharp-eyed analyst.", tts, renderer: betaRenderer },
  ],
});

orchestrator.on("turn", (turn) => {
  console.log(`[${turn.characterName}]: ${turn.text}`);
});

await orchestrator.sendMessage("What should our next move be?");

// Play all dialogue in sequence
await orchestrator.speakAll();

Modes

The orchestrator supports three modes that control how LLM calls are made and how characters interact.

`"single-llm"` (default)

One LLM call produces tagged dialogue for all characters. The orchestrator builds a combined system prompt describing all characters and instructs the model to tag output with [Name]: prefixes. A streaming parser routes each tagged line to the matching character.

const orchestrator = new AvatarOrchestrator({
  llm,
  characters: [...],
  mode: "single-llm",
});

Characters are fully aware of each other and can reference, agree with, or push back on what other characters say — because a single model is writing all of the dialogue. This produces the most natural multi-character conversations.

`"round-robin"`

Each character gets a separate LLM call, sequentially. The first character responds to the user's message. The second character sees the user's message plus the first character's response, and so on. Each character uses its own system prompt and speaks only as itself.

const orchestrator = new AvatarOrchestrator({
  llm,
  characters: [...],
  mode: "round-robin",
});

Characters react to prior speakers, creating a structured dialogue. Each character can optionally use a different LLM via the llm field on OrchestratorCharacter.

`"parallel"`

All LLM calls fire concurrently — one per character. Each character responds independently to the user's message without awareness of the other characters' responses.

const orchestrator = new AvatarOrchestrator({
  llm,
  characters: [...],
  mode: "parallel",
});

Fastest mode (latency = max of individual calls). Useful when you want multiple independent perspectives on the same question.

Constructor options

interface AvatarOrchestratorConfig {
  llm: LLMProvider;
  characters: OrchestratorCharacter[];
  mode?: OrchestratorMode;
  emotions?: boolean;
  reasoningEffort?: ReasoningEffort;
  promptStrategy?: (characters: OrchestratorCharacter[], history: ChatMessage[]) => string;
}

Option	Type	Default	Description
`llm`	`LLMProvider`	—	Required. The shared LLM adapter.
`characters`	`OrchestratorCharacter[]`	—	Required. At least one character.
`mode`	`OrchestratorMode`	`"single-llm"`	How LLM calls are structured.
`emotions`	`boolean`	`false`	Parse emotion markers from LLM output.
`reasoningEffort`	`ReasoningEffort`	—	Extended thinking budget passed to the LLM.
`promptStrategy`	`function`	—	Custom system prompt builder for single-llm mode. Overrides the default multi-character prompt.

OrchestratorCharacter

interface OrchestratorCharacter {
  id: string;
  name: string;
  systemPrompt?: string;
  characterCard?: CharacterCard;
  tts?: TTSProvider;
  renderer: AvatarRenderer;
  llm?: LLMProvider;
}

Field	Type	Description
`id`	`string`	Unique identifier for this character.
`name`	`string`	Display name. Used as the tag in single-llm mode (`[Name]: text`).
`systemPrompt`	`string`	Character personality/instructions. Used in round-robin/parallel modes and in the default single-llm prompt.
`characterCard`	`CharacterCard`	Alternative to `systemPrompt`. The card's description is used as the character's prompt.
`tts`	`TTSProvider`	TTS adapter for this character's speech.
`renderer`	`AvatarRenderer`	Required. The avatar renderer for this character.
`llm`	`LLMProvider`	Per-character LLM override. Only used in round-robin and parallel modes.

Methods

Method	Description
`sendMessage(text)`	Send a user message and generate responses from all characters.
`speak(characterId, turnIndex)`	Synthesize and play a single turn on demand.
`speakAll()`	Play all turns from the last response in sequence.
`interrupt()`	Cancel any in-flight LLM generation or speech playback.
`destroy()`	Tear down the orchestrator and unmount all character renderers.

Properties

Property	Type	Description
`turns`	`readonly Turn[]`	Accumulated turns from the current/last response. Cleared on each `sendMessage()`.
`history`	`readonly ChatMessage[]`	Shared conversation history across all characters.
`characters`	`readonly OrchestratorCharacter[]`	The character roster.
`characterIds`	`readonly string[]`	Character IDs in roster order.

Events

orchestrator.on("turn", (turn) => { /* ... */ });
orchestrator.on("complete", () => { /* ... */ });
orchestrator.on("error", (characterId, err) => { /* ... */ });

Event	Payload	Description
`turn`	`Turn`	Fired for each sentence-sized text turn produced by a character.
`complete`	—	Fired when all characters have finished responding.
`error`	`(characterId: string, err: Error)`	Fired when a character's pipeline encounters an error.

Turn type

interface Turn {
  characterId: string;
  characterName: string;
  index: number;
  text: string;
  emotion?: EmotionPayload;
}

Turns are cleared at the start of each sendMessage() call. The turns getter exposes the accumulated turns from the current or most recent response. The emotion field is populated when emotions: true is set.

Custom prompt strategy

In single-llm mode, the orchestrator builds a default system prompt that describes all characters and instructs the LLM to use [Name]: tagging. You can override this entirely:

const orchestrator = new AvatarOrchestrator({
  llm,
  characters,
  mode: "single-llm",
  promptStrategy: (characters, history) => {
    return `You are moderating a panel discussion between ${characters.map(c => c.name).join(" and ")}...`;
  },
});

Your custom prompt should instruct the LLM to tag each line with [CharacterName]: — the orchestrator's stream parser depends on this format.

Integration with AvatarStage

The orchestrator manages the conversation; AvatarStage manages the rendering layout. They compose naturally:

import { AvatarOrchestrator, AvatarStage } from "avatarlayer";
import { VRMLocalRenderer } from "avatarlayer/renderers";

const alphaRenderer = new VRMLocalRenderer({ modelUrl: "/alpha.vrm" });
const betaRenderer  = new VRMLocalRenderer({ modelUrl: "/beta.vrm" });

// Stage handles layout
const stage = new AvatarStage({ layout: "row", gap: 8 });
stage.mount(document.getElementById("root")!);
await stage.add(alphaRenderer, { id: "alpha" });
await stage.add(betaRenderer, { id: "beta" });

// Orchestrator handles conversation
const orchestrator = new AvatarOrchestrator({
  llm,
  characters: [
    { id: "alpha", name: "Alpha", systemPrompt: "...", tts, renderer: alphaRenderer },
    { id: "beta",  name: "Beta",  systemPrompt: "...", tts, renderer: betaRenderer },
  ],
});

orchestrator.on("turn", (turn) => {
  // Display dialogue in your UI
});

await orchestrator.sendMessage("Discuss the future of AI");
await orchestrator.speakAll();

Playback patterns

// Play a specific character's turn
await orchestrator.speak("alpha", 0);

// Play all turns in order
await orchestrator.speakAll();

// Interrupt playback
orchestrator.interrupt();

Since the orchestrator doesn't auto-play audio, you have full control over when and how dialogue is spoken. This is useful for building UIs where users can click individual lines to hear them, or for orchestrating sequential playback with visual highlighting.

AvatarOrchestrator

On this page