How to Add Rate Limiting to an AI-Built App

How to add rate limiting to an AI-built app comes down to building two separate rate limiters, not one. The first protects your own API endpoints from abusive traffic, the job rate limiting does for any web service. The second manages outbound calls to the model provider, which enforces its own request and token limits and returns a 429 once exceeded. Most guides cover only the first, but skipping the second gets your app throttled even with perfectly protected endpoints.

How to Add Rate Limiting to an AI-Built App: Two Problems to Solve

The first problem is standard API protection. Someone finds your /api/generate route and hammers it with a script, running up your model bill and degrading latency for everyone else. This is the rate limiting most tutorials describe: per-user or per-IP caps at your server or gateway.

The second problem is specific to AI apps and rarely covered. Every call to a model provider counts against that provider's own limits, usually requests and tokens per minute (RPM/TPM), applied per key or tier. Ten users generating at once can trip the TPM ceiling before any limit you set yourself, and the resulting 429 looks like a raw failure without a plan for it.

Problem 1: How to Add Rate Limiting to Your Own AI Endpoints

Why AI Endpoints Need Tighter Abuse Prevention

A typical CRUD endpoint costs a database query; an AI endpoint costs a model call, billed per token and often taking seconds. A script hitting /api/generate 500 times a minute is a line on your invoice and holds open connections that block real users. Set limits below what your infrastructure could handle, matched to what you are willing to pay for.

Token Bucket Rate Limiting

A token bucket gives each user, key, or IP a bucket holding a fixed number of tokens. Each request consumes one token, and the bucket refills at a steady rate, so short bursts are allowed but sustained hammering is not.

interface TokenBucketOptions {
  capacity: number;   // max tokens the bucket can hold
  refillRate: number; // tokens added per second
}

class TokenBucket {
  private tokens: number;
  private lastRefill: number;

  constructor(private opts: TokenBucketOptions) {
    this.tokens = opts.capacity;
    this.lastRefill = Date.now();
  }

  tryConsume(cost = 1): boolean {
    this.refill();
    if (this.tokens >= cost) {
      this.tokens -= cost;
      return true;
    }
    return false;
  }

  private refill() {
    const now = Date.now();
    const elapsedSeconds = (now - this.lastRefill) / 1000;
    this.tokens = Math.min(
      this.opts.capacity,
      this.tokens + elapsedSeconds * this.opts.refillRate
    );
    this.lastRefill = now;
  }
}

// One bucket per user, keyed by user ID or API key
const buckets = new Map<string, TokenBucket>();

function getBucket(userId: string): TokenBucket {
  if (!buckets.has(userId)) {
    // 5 requests of burst, refilling at 1 every 6 seconds (10/min sustained)
    buckets.set(userId, new TokenBucket({ capacity: 5, refillRate: 1 / 6 }));
  }
  return buckets.get(userId)!;
}

// Express middleware
app.use('/api/generate', (req, res, next) => {
  const userId = req.headers['x-user-id'] as string;
  if (!userId) return res.status(401).json({ error: 'Missing user identity' });

  const bucket = getBucket(userId);
  if (!bucket.tryConsume(1)) {
    return res.status(429).json({ error: 'Too many requests, slow down.' });
  }
  next();
});

This in-memory version works for a single process. Run more than one instance and each has its own map, so a user gets the full limit from each. For multi-instance deployments, move the counter to Redis with INCR and a TTL, or a Lua script for atomic check-and-decrement.

Sliding Window Rate Limiting

Token buckets are imprecise at the edges. A sliding window counts actual requests inside a rolling time frame, avoiding the fixed-window problem where a user sends the full limit at 0:59 and again at 1:00, doubling their rate for two seconds. A sliding window log, backed by a Redis sorted set, counts only timestamps still inside the window.

import Redis from 'ioredis';

const redis = new Redis();

async function isAllowed(
  userId: string,
  limit: number,
  windowMs: number
): Promise<boolean> {
  const key = `ratelimit:${userId}`;
  const now = Date.now();
  const windowStart = now - windowMs;

  const pipeline = redis.pipeline();
  pipeline.zremrangebyscore(key, 0, windowStart); // drop entries outside the window
  pipeline.zadd(key, now, `${now}-${Math.random()}`);
  pipeline.zcard(key);
  pipeline.pexpire(key, windowMs);

  const results = await pipeline.exec();
  const count = results?.[2]?.[1] as number;

  return count <= limit;
}

// Usage: allow 10 requests per rolling 60 seconds
const allowed = await isAllowed(userId, 10, 60_000);
if (!allowed) {
  // reject with 429
}

Sliding window logs cost more memory and CPU than a token bucket, but for most AI apps the precision is worth it: per-user volume is naturally low, and boundary bursts do not slip through.

Problem 2: Respecting the Upstream Model API's Rate Limits

Every major model provider enforces its own limits on top of whatever you build. OpenAI's rate limit documentation measures requests and tokens per minute, tied to your usage tier. Anthropic's rate limits guide tracks requests and input/output tokens per minute separately, with a retry-after header on every 429. These apply regardless of your endpoint setup, since the provider limits you as one client across all users combined.

Handling 429s With Exponential Backoff

The correct response to a 429 is not to retry immediately. Wait, honor whatever retry-after value the provider sent, and add a small random delay (jitter) so you are not retrying in lockstep with other throttled requests.

async function callModelWithBackoff(
  makeRequest: () => Promise<Response>,
  maxRetries = 5
): Promise<Response> {
  let attempt = 0;

  while (true) {
    const res = await makeRequest();

    // 429 = rate limited, 529 = provider overloaded (Anthropic-specific)
    if (res.status !== 429 && res.status !== 529) {
      return res;
    }

    attempt++;
    if (attempt > maxRetries) {
      throw new Error(`Model API still rate limited after ${maxRetries} retries`);
    }

    const retryAfterHeader = res.headers.get('retry-after');
    const waitMs = retryAfterHeader
      ? parseInt(retryAfterHeader, 10) * 1000
      : Math.min(2 ** attempt * 1000, 30_000); // exponential fallback, capped at 30s

    const jitterMs = Math.random() * 250;
    await new Promise((resolve) => setTimeout(resolve, waitMs + jitterMs));
  }
}

The Retry-After header is a standard HTTP header, not AI-specific, so this pattern works against any rate-limited service. Treat its value as a floor and let the exponential fallback handle cases where a provider omits it.

Throttling AI API Requests With a Queue Instead of Blind Retries

Reactive backoff handles occasional 429s fine, but if your app regularly saturates the provider's RPM or TPM limit, every user during a spike experiences a delay. A better approach is proactive: queue outbound calls behind a concurrency limiter under the provider's published limits, so you rarely trigger a 429. A simple version caps requests in flight; a fuller one also tracks a rolling token budget. p-queue, bottleneck, or a plain semaphore cover this without a custom scheduler.

Combining Both Layers in Practice

The two layers need to agree, or the inner one is pointless. Say your provider caps you at 60 requests per minute app-wide, but your endpoint limiter allows 100 per user. With more than one active user, you hit the provider's ceiling before any single user hits yours. The upstream 429s become the real, worse rate limiter, returning errors instead of the clean responses your own limiter can produce. Size your limits and queue concurrency against your actual upstream budget, not your hardware's theoretical capacity.

As one example, apps generated on Swarmz that call a model API include both layers by default: a per-user limiter on backend routes, and a queued client for outbound calls that honors retry-after and backs off on 429s. Neither layer substitutes for the other.

Choosing Sensible Limits

• Start from the provider's published RPM and TPM for your tier, not a round number that feels reasonable.

• Divide that budget across expected concurrent users, then leave headroom since usage rarely distributes evenly.

• Set your endpoint limit below that per-user share so one user cannot consume the app's whole upstream budget.

• Log every upstream 429. Regular occurrences mean your queue's concurrency is too high, not just your endpoint limit.

• Re-check limits after any model or plan change, since a quiet provider upgrade can silently change what your queue should allow.

Testing Before Users Find the Edges

Load test both layers separately before shipping. Hit your own endpoint with a burst script and confirm 429s come back with a clear error, not a stack trace. Then, with a real (low-traffic) provider key in staging, simulate concurrent generations to see how close you get to the published RPM and TPM before your queue starts holding requests. Better to find that ceiling in staging than during a launch.

FAQ

How do you prevent API abuse in an AI app?

Rate limit by an identity you can trust, a logged-in user ID or API key rather than IP alone, since IPs are shared and spoofable. Combine that with a token bucket or sliding window on your own endpoints, and cap the cost-heavy actions, generation calls, more tightly than cheap ones like status checks.

What is the difference between rate limiting your API and rate limiting the AI model API?

Rate limiting your API controls how often your users can call your endpoints. Rate limiting the AI model API is about respecting limits the model provider imposes on your app as a whole, measured in requests and tokens per minute. You control the first with your own code, the second you can only work around with backoff, queueing, and by staying under the provider's published limits.

How do you throttle AI API requests without hurting user experience?

Queue requests instead of rejecting them outright when you are close to a limit, show a short generating state instead of an error, and reserve hard rejections for genuine abuse rather than normal load. A queue with a few seconds of added latency reads as normal to a user, a raw 429 reads as broken.

What HTTP status code means an AI API is rate limiting you?

429 Too Many Requests is the standard code both OpenAI and Anthropic return when you exceed a rate limit. Anthropic also returns 529 for server overload, which is a capacity issue on their side rather than a limit you exceeded, but it should be retried the same way, with backoff.

Should you rate limit by user, IP address, or API key in an AI app?

Prefer user ID or API key whenever you have authenticated users, since that ties the limit to an identity that cannot be trivially rotated. Fall back to IP-based limiting only for unauthenticated routes, and expect it to be a blunter, more easily evaded tool.

Related reading

• the how to build an app with AI pillar

• adding payments to an AI-built app

• choosing a database for an AI-built app

• adding multi-language support to an AI-built app

• why an AI-built app might be slow