LLM inference is the process of using a trained Large Language Model (LLM) to generate an answer or prediction from an input.
Think of it as:
Training = teaching the model Inference = asking the trained model to do something
Simple example
You type:
“Explain photosynthesis in simple words.”
During inference, the LLM:
- Receives your text as input.
- Converts it into tokens.
- Processes those tokens through its neural network.
- Predicts the next token.
- Repeats this process until it has generated the response.
Training vs. inference
| Training | Inference | |
|---|---|---|
| Purpose | Learn patterns/parameters | Generate an output |
| Input | Huge datasets | Your prompt |
| Computation | Usually very expensive | Usually cheaper, but can still be expensive |
| Happens | During model development | Every time you use the model |
| Example | Teaching GPT language patterns | GPT answering your question |
Why “inference” matters
When people talk about inference cost, inference speed, inference hardware, or inference optimization, they're talking about making the process of running a trained LLM faster, cheaper, and more efficient.
For example, techniques such as quantization, batching, KV caching, and model parallelism are commonly used to optimize LLM inference.
