9 LLM Engineering interview questions and answers

Models read text as subword tokens, usually produced by byte-pair encoding. Cost, context limits and latency are all counted in tokens.

Some languages, including Hindi, need more tokens per word than English, so the same message costs more. Tokenisation also explains quirks with spelling, counting letters and arithmetic.

Temperature scales the logits before softmax: low values make output more deterministic, high values more varied.

Top-k keeps only the k most likely tokens. Top-p keeps the smallest set of tokens whose probabilities add up to p. Use low temperature for extraction and classification, higher for creative writing.

Each token is projected into a query, key and value. Attention scores are Q·Kᵀ divided by √dₖ, passed through softmax, and used to take a weighted sum of the values.

In decoder models a causal mask stops a token attending to later tokens. Multiple heads learn different relationships. Cost grows with the square of sequence length.

During generation, the keys and values for earlier tokens are stored so each new token only computes its own, instead of recomputing the whole sequence.

Its memory grows with layers × KV heads × head size × sequence length × batch size × 2, and it often limits batch size and context length. Grouped-query attention shrinks it by sharing keys and values across heads.

Start with prompting. Use RAG when the model needs knowledge that changes, is private, or must be cited.

Fine-tune to change behaviour, format or style, or to get a smaller model to do a narrow task cheaply. Fine-tuning is a poor way to add fresh facts.

The base model's weights are frozen. For chosen layers, usually the attention projections, LoRA learns a low-rank update ΔW = B·A with rank r much smaller than the layer size.

This cuts trainable parameters and optimiser memory dramatically, and the update can be merged back into the weights for serving. QLoRA also stores the base model in 4-bit to fit larger models on one GPU.

Prefill processes the whole prompt in parallel and is limited by compute. Decode produces one token at a time, and each step must read all the weights and the KV cache from memory, so it is limited by memory bandwidth.

Batching many requests shares each weight read across users. Track time to first token for prefill and time per output token for decode.

Build an eval set from real user queries with expected answers or grading criteria. Use automatic checks where you can: exact match, schema validity, retrieval recall@k.

For open answers, use an LLM judge calibrated against human labels, and watch for its biases toward position, length and its own outputs. Run evals on every prompt or model change, and keep monitoring in production.

For a fixed compute budget, loss is lowest when parameters and training tokens grow together, roughly 20 tokens per parameter for compute-optimal training.

Many teams now deliberately train smaller models on far more tokens than that, because a smaller model is cheaper to serve.