What Is Batch Size in AI Training? The Tradeoffs

Batch size is the number of training examples a model processes before it updates its weights once. If you train on 1,000 examples with a batch size of 100, the model makes 10 updates to cover the whole dataset. A small batch gives frequent, noisy updates and needs little memory. A large batch gives fewer, smoother updates and needs more memory. Choosing one is mostly a question of what your hardware can hold.

The three sizes

Google's machine learning course lays out the spectrum. Stochastic gradient descent uses a single example per update, a batch size of one. Full-batch gradient descent uses the entire dataset for each update. Mini-batch sits in between, where a small batch behaves like the stochastic version and a larger one behaves like full-batch. The course also says the right size depends on the dataset and the available compute, which is the honest answer to "what should I pick". You can read the definitions in Google's hyperparameters lesson.

Nearly all real training uses mini-batches, because full-batch needs too much memory and one-example updates waste the parallel hardware.

Counting updates

The arithmetic is simple and worth doing once, because it explains most of the behaviour.

An epoch is one full pass through the data. The batch size decides how many updates happen inside that pass. Ten epochs at batch size 100 is 100 updates. Ten epochs at batch size 10 is 1,000 updates, each based on less evidence.

What changes when you move it

• Memory. Every example in the batch is held in memory while the model works out its error. Doubling the batch size roughly doubles that load, which is why batch size is often capped by the GPU you have, not by preference.

• Noise. A small batch gives a rougher estimate of the right direction, so each step wobbles. Some wobble can help the model avoid settling into a poor spot. Too much makes the loss curve jumpy.

• Speed. Larger batches use parallel hardware better, so each epoch finishes sooner, though you also get fewer updates per epoch.

• Interaction with learning rate. Batch size and learning rate are tuned together. A common habit is to raise the learning rate somewhat when you raise the batch size, then check the loss curve, because a smoother gradient can tolerate bigger steps.

A practical way to choose

1. Find the largest batch that fits in memory without errors.

2. Back off a notch, since memory use can creep up with longer inputs.

3. Run a short test and watch the loss. If it is very jumpy, try a larger batch or a lower learning rate.

4. Compare results on data the model did not train on, to catch overfitting.

If you are fine-tuning through a hosted service, the provider often picks the batch size for you or exposes it as one optional setting. Leave it at the default until you have a reason to change it.

Not the same as batch inference

The word also appears in a different context. Batch inference means sending many requests to a trained model in bulk to save money. That is about running a model, while batch size in training is about teaching one. The two share a name and little else. For the wider picture, see how AI models work.

Frequently asked questions

What is a good batch size?

There is no universal value. Pick the largest size your memory allows, then adjust based on how stable the loss curve looks. The right size depends on the dataset and the compute available.

Does a bigger batch size train a better model?

Not automatically. A bigger batch trains faster per epoch and gives smoother updates, but the model gets fewer updates, and results can be no better or slightly worse without retuning the learning rate.

What is the difference between batch size and epoch?

Batch size is how many examples go into one update. An epoch is one full pass through the whole training set. Together they set how many updates a run makes.

Is batch size the same as batch inference?

No. Batch size is a training setting. Batch inference is a way of sending many requests to an already trained model at once.