AvatarLayer — pluggable SDK for realtime conversational avatars

LemonSliceRenderer connects to a LemonSlice avatar service through LiveKit. Audio is streamed to the service as raw PCM, and lip-synced video is streamed back.

Installation

npm install livekit-client

Usage

LemonSlice requires server-side session creation (to generate LiveKit tokens). The renderer accepts a createSession function that you implement:

import { LemonSliceRenderer } from "avatarlayer/renderers";

const renderer = new LemonSliceRenderer({
  createSession: async () => {
    const resp = await fetch("/api/lemonslice", {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({
        lemonsliceApiKey: "...",
        livekitUrl: "wss://...",
        livekitApiKey: "...",
        livekitApiSecret: "...",
        agentImageUrl: "https://...",
      }),
    });
    return resp.json();
  },
  onVideoStream: (stream) => console.log("Video:", stream),
  onAudioStream: (stream) => console.log("Audio:", stream),
  onStateChange: (state) => console.log("State:", state),
});

Constructor options

Option	Type	Description
`createSession`	`() => Promise<LemonSliceSession>`	Required. Returns LiveKit connection info.
`onVideoStream`	`(stream: MediaStream \| null) => void`	Called when video track is received/lost
`onAudioStream`	`(stream: MediaStream \| null) => void`	Called when audio track is received/lost
`onStateChange`	`(state: string) => void`	Called on connection state changes

LemonSliceSession shape

The object returned by createSession:

interface LemonSliceSession {
  sessionId: string | null;
  roomName: string;
  livekitUrl: string;
  livekitToken: string;
  avatarIdentity: string;
  userIdentity: string;
}

How it works

mount() calls your createSession function to get LiveKit credentials
Connects to the LiveKit room and subscribes to video/audio tracks
When speak(audio) is called, the audio blob is decoded to raw PCM
A persistent byte stream to the avatar participant is opened via LiveKit's streamBytes API on the first call, and reused for subsequent utterances — no per-sentence stream teardown/recreation
PCM chunks are written to the stream at real-time pace
LemonSlice renders lip-synced video from the audio in realtime
interrupt() closes the byte stream and clears the avatar's audio buffer via RPC; the next speak() opens a fresh stream

Server-side setup

The demo app includes a ready-made /api/lemonslice route that handles LiveKit room creation and token generation. You need:

A LemonSlice API key
LiveKit server URL, API key, and API secret

The server-side route creates a LiveKit room, generates participant tokens, and returns the connection info to the client.