Dashboard

What Is a Load Balancer? When an AI-Built App Needs One

A load balancer shares traffic between copies of your app. Most small apps do not need one yet, and the three things that break first are not what you expect.

Cecilia Iona
Cecilia Iona
Senior Editor, AI & Product
29 October 20261 min read

A load balancer is a service that spreads incoming traffic across several copies of your app, so no single copy is overwhelmed and one failing copy does not take the site down. Visitors see one address. Behind it, the balancer picks which server answers each request.

Before the how, the more useful question: do you need one? For most apps built with AI, not yet.

Do you need a load balancer? A quick test

You probably do not, if all of these are true:

  • One server handles your traffic with room to spare.

  • You can accept a few minutes of downtime when you deploy or when something crashes.

  • You are on a platform that scales for you.

That last point matters most. Hosting platforms and serverless setups run a load balancer on your behalf, which is why you may never configure one. If you deploy to such a platform, the balancing already exists and is not your job. The question of how much server you need at all is covered in do I need a backend for my app.

You start to need one when a single machine is the limit (it runs out of processor or memory under real load) or when downtime is expensive enough that you want a second copy ready.

What it actually does

Three jobs:

  1. Distributes requests. The default method in nginx is round-robin, which hands requests to each server in turn. Other methods send each request to the server with the fewest active connections, or pick a server based on the client's address. These are described in the nginx load balancing documentation.

  2. Notices failures. If a server returns errors, the balancer stops sending it traffic for a while and probes it again later. nginx calls this passive health checking, controlled by max_fails and fail_timeout.

  3. Lets you change servers without downtime. Take one out, update it, put it back.

Here is the smallest nginx version:

nginx
upstream myapp {
    server 10.0.0.11:3000;
    server 10.0.0.12:3000;
}

server {
    listen 80;
    location / {
        proxy_pass http://myapp;
    }
}

If that looks like a reverse proxy config, it is one with a list of servers instead of one. A load balancer is a reverse proxy with more than one destination.

What breaks when you add a second copy

This is the part tutorials skip. An app that works on one machine can fail in odd, intermittent ways on two, because it quietly relied on being alone. AI-generated apps lean on three shortcuts in particular.

1. Sessions kept in memory

The login state lives in the app's memory. Request one hits server A and logs you in. Request two hits server B, which has never heard of you, and logs you out. It looks random because it is: it depends on which server answered.

The fixes are to store sessions in a shared place such as a database, or to use signed tokens that carry the information themselves. A workaround called sticky sessions (nginx's ip_hash is one way) pins each visitor to one server. It works, but it undermines the point of balancing and does not survive a server dying.

2. Files saved to the local disk

An upload lands on server A's disk. The next request, handled by server B, cannot find it. Images appear and vanish. The fix is shared storage, such as an object storage bucket, instead of the app's own folder.

3. Scheduled jobs that run twice

A task that sends a daily email runs inside the app. With two copies, it runs twice, and every customer gets two emails. Move scheduled work to one place, or use a lock so only one copy runs it. If the work is heavy, a queue is the cleaner design.

Ask your coding assistant to search the project for each of the three before you scale: where session state is stored, where uploads are written, and what runs on a timer. That one review prevents most of the surprises.

Load balancer versus reverse proxy versus CDN

They overlap, which is why the terms blur:

  • A reverse proxy stands in front of your app. One destination is enough.

  • A load balancer chooses between several destinations.

  • A CDN caches and serves copies of your content from locations near visitors. It reduces load but does not replace a balancer for dynamic requests.

Many tools do all three roles, and cloud providers sell them as a single product.

Streaming and long requests

AI features add a wrinkle. A streaming answer from a model can keep a connection open for many seconds. Balancing by fewest active connections handles this better than plain round-robin, since a server busy with long responses looks busier. Make sure the balancer's timeouts are longer than your slowest legitimate response, or it will cut off answers mid-sentence.

Where to go from here

If you reach the point of running your own servers, the checklist in how to deploy an app built with AI comes first, and the broader path is laid out in the guide to building an app with AI.

Frequently asked questions

What is the difference between a load balancer and a reverse proxy?

A reverse proxy forwards requests to a backend. A load balancer is a reverse proxy that chooses among several backends. Many products do both.

Do I need a load balancer for a small website?

Usually not. One server, or a hosting platform that scales for you, is enough until traffic or uptime needs say otherwise.

What is round-robin load balancing?

Requests are handed to each server in turn: first to server one, then two, then three, then back to one. It is the default method in nginx.

What are sticky sessions?

A setting that sends the same visitor to the same server every time, so server-held login state keeps working. It is a workaround for apps that keep state in memory.

How did this land?

About the author

Cecilia Iona
Cecilia Iona

Senior Editor, AI & Product

Cecilia leads the Swarmz editorial desk. She has spent a decade turning complex AI and product topics into writing people actually finish, and she owns the blog's quality bar.

Share

Get the next post in your inbox

One email a month. Product updates, engineering posts, and the best of Built with Swarmz.

I agree to receive emails about AI building tips and Swarmz product news. Unsubscribe any time.