Docs / AI Infrastructure / Streaming chat

Streaming chat

The AI chat is a real streaming endpoint, not a canned demo. The route in app/api/teams/[teamId]/ai/chat/route.ts streams tokens from the model as they're generated, and the client renders them as they arrive.

The server stream

The route calls streamText from the ai package and returns a plain text stream built from result.fullStream:

const body = result.fullStream
  .pipeThrough(
    new TransformStream<TextStreamPart<ToolSet>, string>({
      transform(part, controller) {
        if (part.type === "text-delta") controller.enqueue(part.text);
        else if (part.type === "error") controller.error(part.error);
      },
    }),
  )
  .pipeThrough(new TextEncoderStream());

It doesn't use toTextStreamResponse() because that helper drops error parts: a provider failure (bad key, rate limit) would reach the client as a 200 with an empty body. Erroring the stream aborts the response, so the client's reader rejects and shows a retryable error.

maxOutputTokens caps each reply so a single generation can't run up an unbounded provider bill. Credits are reserved before the model runs, see Usage & credits for that side.

The client stream

components/app/ai-chat.tsx posts the message history to the route, then reads the response body directly with a ReadableStream reader:

const reader = res.body.getReader();
const decoder = new TextDecoder();
for (;;) {
  const { done, value } = await reader.read();
  if (done) break;
  content += decoder.decode(value, { stream: true });
  // update the assistant bubble with the growing content
}

The assistant bubble fills in live as chunks decode. An AbortController backs the stop button: stopping before any text arrives drops the empty bubble, stopping mid-reply keeps the partial text (it was charged). A reply that ends with no text is shown as an error. After every attempt the client re-reads the balance from /api/teams/[teamId]/credits instead of subtracting locally, since only the server knows whether that request was charged or refunded.

Message and context limits

The route validates the request body with Zod before it reaches the model:

• Each message is capped at 8000 characters (MAX_MESSAGE_CHARS).
• The whole conversation is capped at 24,000 characters (MAX_CONVERSATION_CHARS), roughly 6k input tokens.
• The history is capped at 50 messages per request, minimum 1.

These bound how much context a single request can ship, which matters because flat credit pricing charges the same regardless of token count.

Rate limiting

The route wires in an Upstash sliding-window limiter (getRatelimiter("ai-chat", "1 m", 20)), 20 requests per minute per user, returning a 429 when exceeded.