Docs / AI Infrastructure / Streaming chat

Streaming chat

The AI chat is a real streaming endpoint, not a canned demo. The route in app/api/teams/[teamId]/ai/chat/route.ts streams tokens from the model as they're generated, and the client renders them as they arrive.

The server stream

The route calls streamText from the ai package and returns it as a plain text stream:

const result = streamText({
  model,
  maxOutputTokens: 2048,
  system: "You are the Keel Assistant, ...",
  messages: parsed.data.messages,
  onFinish: async ({ totalUsage }) => { /* charge credits */ },
});

return result.toTextStreamResponse();

maxOutputTokens caps each reply so a single generation can't run up an unbounded provider bill. Credits are charged in onFinish, after the stream completes, so an aborted or errored generation never bills the team. See Usage & credits for that side.

The client stream

components/app/ai-chat.tsx posts the message history to the route, then reads the response body directly with a ReadableStream reader:

const reader = res.body.getReader();
const decoder = new TextDecoder();
for (;;) {
  const { done, value } = await reader.read();
  if (done) break;
  content += decoder.decode(value, { stream: true });
  // update the assistant bubble with the growing content
}

The assistant bubble fills in live as chunks decode. An AbortController backs the stop button, and an aborted stream drops the partial bubble rather than leaving an error state.

Message and context limits

The route validates the request body with Zod before it reaches the model:

• Each message is capped at 8000 characters (MAX_MESSAGE_CHARS).
• The history is capped at 50 messages per request, minimum 1.

These bound how much context a single request can ship, which matters because flat credit pricing charges the same regardless of token count.

Rate limiting

The route wires in an Upstash sliding-window limiter (getRatelimiter("ai-chat", "1 m", 20)), 20 requests per minute per user, returning a 429 when exceeded.