AvatarLayer — pluggable SDK for realtime conversational avatars

AvatarLayer can parse inline emotion markers from LLM output, strip them from TTS text, and apply expression changes to the avatar renderer automatically.

Setup

Enable emotions in the session config:

import {
  AvatarSession,
  OpenAIAdapter,
  ElevenLabsAdapter,
  VRMLocalRenderer,
  getEmotionSystemPromptSuffix,
} from "avatarlayer";

const session = new AvatarSession({
  llm: new OpenAIAdapter({ apiKey: "sk-..." }),
  tts: new ElevenLabsAdapter({ apiKey: "..." }),
  renderer: new VRMLocalRenderer({ modelUrl: "/models/avatar.vrm" }),
  systemPrompt: "You are a friendly avatar assistant.\n\n" + getEmotionSystemPromptSuffix(),
  emotions: true,
});

The getEmotionSystemPromptSuffix() function returns instruction text that tells the LLM how to emit emotion markers.

How it works

The LLM outputs text with inline markers like <|ACT {"emotion":"happy","intensity":0.8}|>
The session's marker parser detects and extracts these markers from the stream
Markers are stripped from the text before it reaches TTS
The parsed emotion is applied to the renderer via update() with the appropriate expression preset
The emotion event fires on the session

Marker format

Emotion markers

<|ACT {"emotion":"happy","intensity":0.8}|>

The JSON payload contains:

Field	Type	Description
`emotion`	`string`	Emotion name (see values below)
`intensity`	`number`	Intensity from 0 to 1

Delay markers

<|DELAY:500|>

Inserts a pause (in milliseconds) in the speech output.

Emotion values

The Emotion enum defines the supported emotion labels:

Value	VRM preset	Live2D expression
`happy`	`happy`	`happy`
`sad`	`sad`	`sad`
`angry`	`angry`	`angry`
`surprised`	`surprised`	`surprised`
`relaxed`	`relaxed`	`relaxed`
`neutral`	`neutral`	`neutral`

The EMOTION_VRM_MAP and EMOTION_LIVE2D_MAP exports provide the full mapping from emotion labels to renderer-specific expression names.

Events

Event	Payload	Description
`emotion`	`{ name: string, intensity: number }`	Fired each time an emotion marker is parsed from the LLM stream

session.on("emotion", ({ name, intensity }) => {
  console.log(`Emotion: ${name} at ${intensity}`);
});

Utilities

Export	Description
`getEmotionSystemPromptSuffix()`	Returns the instruction text for the LLM
`Emotion`	Enum of supported emotion labels
`EMOTION_VALUES`	Array of all emotion label strings
`EMOTION_VRM_MAP`	Map from emotion label to VRM expression preset
`EMOTION_LIVE2D_MAP`	Map from emotion label to Live2D expression name
`parseEmotionMarker(text)`	Parse an emotion marker string into an `EmotionPayload`
`parseDelayMarker(text)`	Parse a delay marker string into milliseconds
`createMarkerParser()`	Create a stateful parser for streaming marker extraction

Emotions