LLM API Integration: Streaming, Retries, and Cost Control

Calling an LLM API for the first time takes about four lines. Running one in production takes considerably more, and the gap between those two states is where most of the surprises live: requests that hang for two minutes, bills that are ten times the estimate, and a JSON parse that works for a week and then does not.

This walks through the parts that are not in the quickstart. Examples are TypeScript against the Anthropic SDK, because that is what I use, but the shape of the problems is the same whichever provider you pick.

The baseline request

npm install @anthropic-ai/sdk

import Anthropic from '@anthropic-ai/sdk';

// Reads ANTHROPIC_API_KEY from the environment. Do not hardcode a key.
const client = new Anthropic();

const response = await client.messages.create({
  model: 'claude-opus-5',
  max_tokens: 16000,
  messages: [{ role: 'user', content: 'Summarise this in one sentence: ...' }],
});

// content is a discriminated union of blocks, not a string.
for (const block of response.content) {
  if (block.type === 'text') console.log(block.text);
}

That last loop is the first thing people get wrong. Reaching for response.content[0].text compiles only if you assert the type, and it breaks the moment a response leads with a different block kind. Narrow on block.type and the compiler keeps you honest.

Never put the key in the browser

Worth stating plainly because it still happens. An API key in client-side code is a public API key. Bundlers inline environment variables at build time, so a key in a NEXT_PUBLIC_ variable ends up in JavaScript anyone can read, and the first thing that finds it will be a scraper, not a user.

The call belongs on the server. In the Next.js App Router that means a Route Handler or a Server Action, and the model call never crosses the boundary into client code. The mechanics of that boundary are in React Server Components and Server Actions.

// app/api/summarise/route.ts
import Anthropic from '@anthropic-ai/sdk';

const client = new Anthropic();

export async function POST(request: Request) {
  const { text } = await request.json();

  // Validate before spending money. An unbounded body is an unbounded bill.
  if (typeof text !== 'string' || text.length > 50_000) {
    return Response.json({ error: 'Invalid input' }, { status: 400 });
  }

  const response = await client.messages.create({
    model: 'claude-opus-5',
    max_tokens: 1024,
    messages: [{ role: 'user', content: `Summarise:\n\n${text}` }],
  });

  const summary = response.content.find((b) => b.type === 'text')?.text ?? '';
  return Response.json({ summary });
}

Streaming, and why you need it

A non-streaming request holds the connection until the entire response is generated. For anything long that runs into HTTP timeouts, and for anything user-facing it means a blank screen for several seconds.

const stream = client.messages.stream({
  model: 'claude-opus-5',
  max_tokens: 64000,
  messages: [{ role: 'user', content: prompt }],
});

for await (const event of stream) {
  if (event.type === 'content_block_delta' && event.delta.type === 'text_delta') {
    process.stdout.write(event.delta.text);
  }
}

// The complete message, after the stream finishes.
const final = await stream.finalMessage();
console.log(final.usage);

Note finalMessage(). If you only need the whole response and are streaming to dodge timeouts rather than to render tokens, that helper is the entire implementation. Do not hand-roll a Promise around event handlers to reassemble the text, which is a pattern I still see copied around.

Getting the stream to the browser

Server-Sent Events are the right transport here. The response is one-directional, it is text, and it reconnects on its own. WebSockets are more machinery than this problem needs, an argument I make at length in WebSockets vs SSE vs polling.

export async function POST(request: Request) {
  const { prompt } = await request.json();

  const stream = client.messages.stream({
    model: 'claude-opus-5',
    max_tokens: 64000,
    messages: [{ role: 'user', content: prompt }],
  });

  const encoder = new TextEncoder();

  return new Response(
    new ReadableStream({
      async start(controller) {
        try {
          for await (const event of stream) {
            if (
              event.type === 'content_block_delta' &&
              event.delta.type === 'text_delta'
            ) {
              controller.enqueue(
                encoder.encode(`data: ${JSON.stringify(event.delta.text)}\n\n`)
              );
            }
          }
          controller.enqueue(encoder.encode('data: [DONE]\n\n'));
        } catch (error) {
          // The client sees a clean end rather than a hung connection.
          controller.enqueue(encoder.encode('event: error\ndata: {}\n\n'));
        } finally {
          controller.close();
        }
      },
    }),
    {
      headers: {
        'Content-Type': 'text/event-stream',
        'Cache-Control': 'no-cache, no-transform',
        Connection: 'keep-alive',
      },
    }
  );
}

The no-transform in that cache header is not decoration. Some proxies buffer responses to compress them, which holds every token until the stream ends and gives you a non-streaming stream that took longer to build. If your tokens arrive in one burst at the end, that header is the first thing to check.

Error handling that distinguishes retryable from fatal

One broad catch block around the call is the most common shape, and it throws away the information you need. A 429 should be retried. A 400 should not, because it will fail identically forever.

import Anthropic from '@anthropic-ai/sdk';

try {
  const response = await client.messages.create({ /* ... */ });
} catch (error) {
  if (error instanceof Anthropic.BadRequestError) {
    // Your request is malformed. Retrying changes nothing.
    throw error;
  } else if (error instanceof Anthropic.AuthenticationError) {
    // Bad or missing key. Page someone.
    throw error;
  } else if (error instanceof Anthropic.RateLimitError) {
    // Back off and try again.
    return scheduleRetry();
  } else if (error instanceof Anthropic.APIError) {
    console.error(`API error ${error.status}`, error.message);
    throw error;
  }
  throw error;
}

Check from most specific to least. Matching on error message strings is the alternative, and it breaks silently the first time a provider rewords a message. A 429 here means you hit the provider's own LLM rate limit, which is separate from any limit you impose on your own callers.

On retries: the SDK already retries connection errors, 408, 409, 429 and 5xx twice by default. Before you write your own retry loop, check whether you are building a second one on top of the first. Two layers of retry with a ten minute default timeout can keep a single request alive for half an hour.

Structured output from an LLM

Asking for JSON in the prompt and calling JSON.parse on the result works most of the time, and "most of the time" is the problem. The failure is a leading sentence of explanation, or a code fence, and it arrives in production rather than in testing.

Constrain the output instead of parsing hopefully. Structured outputs let you hand over a schema and get a response that conforms to it:

const response = await client.messages.create({
  model: 'claude-opus-5',
  max_tokens: 4096,
  output_config: {
    format: {
      type: 'json_schema',
      schema: {
        type: 'object',
        properties: {
          sentiment: { type: 'string', enum: ['positive', 'neutral', 'negative'] },
          topics: { type: 'array', items: { type: 'string' } },
        },
        required: ['sentiment', 'topics'],
        additionalProperties: false,
      },
    },
  },
  messages: [{ role: 'user', content: review }],
});

Even with a schema, validate on the way in before the value reaches your database. A schema constrains shape, and your code still owns whether the contents make sense; the parsing patterns are in validating API input with Zod.

The API is stateless, and that has a cost

There is no conversation on the server. Every request carries the entire history, which means turn twenty sends everything from turns one through nineteen again and pays input tokens for all of it. A chat that feels cheap at the start gets quadratically more expensive as it runs.

import Anthropic from '@anthropic-ai/sdk';

// Use the SDK's types rather than inventing a ChatMessage interface.
const messages: Anthropic.MessageParam[] = [];

async function send(userText: string) {
  messages.push({ role: 'user', content: userText });

  const response = await client.messages.create({
    model: 'claude-opus-5',
    max_tokens: 16000,
    messages,
  });

  // Push the whole content array, not just the extracted string. Dropping
  // non-text blocks here is how tool calls and reasoning get lost between turns.
  messages.push({ role: 'assistant', content: response.content });

  return response;
}

Pushing only the text back is a bug that hides for a while. It works fine until the response contains anything other than a plain text block, at which point the model loses the thread of its own previous turn with no error to point at.

Count tokens before you send

You do not have to guess whether a request will fit or what it will cost. There is an endpoint for it, and it is free.

const count = await client.messages.countTokens({
  model: 'claude-opus-5',
  system,
  messages,
});

if (count.input_tokens > 150_000) {
  // Trim history or summarise before spending anything.
  await compactHistory();
}

Do not reach for tiktoken or a characters-divided-by-four heuristic to do this. Tokenisers differ between providers and between model generations, so a local estimate is approximately right in a domain where being approximately right about your context limit means a 400 at the worst moment.

Cost, and the two levers that matter

Pricing is per million tokens, split between input and output, and output costs several times more than input. That asymmetry drives most of the optimisation.

Prompt caching is the first lever, and it is nearly free. If you send the same large system prompt or document on every request, cache it. Cached reads are around a tenth of the normal input price.

const response = await client.messages.create({
  model: 'claude-opus-5',
  max_tokens: 16000,
  system: [
    {
      type: 'text',
      text: largeStableInstructions,
      cache_control: { type: 'ephemeral' },
    },
  ],
  messages: [{ role: 'user', content: userQuestion }],
});

// Zero across repeated calls means something is invalidating the prefix.
console.log(response.usage.cache_read_input_tokens);

Caching is a prefix match, so any byte that changes early invalidates everything after it. A timestamp in the system prompt, a request ID, or a tool list built by iterating an object with non-deterministic key order will each quietly reduce your hit rate to zero while looking correct. The way to catch it is to log cache_read_input_tokens and alert when it stays flat.

The second lever is not sending as much. Retrieval instead of stuffing the whole knowledge base into context, tighter output instructions, and a lower max_tokens on routes that only need a classification. If your prompt is growing because you keep appending documents to it, the answer is a RAG pipeline rather than a bigger context window, built on chunking and embeddings that decide what gets sent at all.

Timeouts and the request you forgot to bound

The SDK default timeout is ten minutes. In a serverless function with a 30 second execution limit that is meaningless, and in a long-running Node process it means a stuck request holds a slot for ten minutes and then retries twice, which is half an hour of one connection.

// TypeScript SDK timeouts are in milliseconds, unlike the Python one.
const client = new Anthropic({ timeout: 60_000, maxRetries: 2 });

// Or per request, when one route legitimately needs longer.
const response = await client.messages.create(params, { timeout: 120_000 });

Set it deliberately. Wall clock worst case is roughly the timeout multiplied by retries plus one, so a 60 second timeout with two retries can still take three minutes before it gives up.

What I would build first

A server route with auth and a rate limit in front of it. Streaming, even if the UI does not render tokens yet, because it removes a class of timeout problem. Typed error handling that separates retryable from fatal. Logging of usage on every response, because token counts are the only way you will understand the bill later.

Everything after that depends on the product. If you are answering questions about your own documents, retrieval is the next thing to build. If the model needs to take actions rather than produce text, that is tool calling and agent loops, which is a different set of problems and a different set of ways to overspend.