AI Concepts: Quick Revision

An embedding is a dense vector (say 384 to 3072 floats) that a model produces for a piece of text, an image or a user, placed so that similar things land close together. Distance in that space stands in for meaning.

• How they're made: an encoder (BERT-style sentence encoders, OpenAI or Cohere embedding models, CLIP for images) is trained with contrastive objectives so that related pairs are pulled together and unrelated pairs are pushed apart.
• What they enable: semantic search, RAG retrieval, deduplication, clustering, recommendations (two-tower models) and cheap features for downstream classifiers.
• Similarity: usually cosine similarity, or a dot product on normalised vectors.

Interview gotchas: the query and the documents must be embedded with the same model, and changing the model means re-embedding the whole corpus. Generic embeddings can miss domain jargon, which is when you fine-tune them or add keyword search alongside.

LLMs don't read characters or words. They read tokens: subword pieces from a fixed vocabulary, built with algorithms like BPE (GPT models), WordPiece (BERT) or SentencePiece/Unigram (Llama, T5). Common words become one token and rare words split into several ("tokenization" → token + ization).

Why it matters in a design:

• Cost and limits: pricing, rate limits and the context window are all counted in tokens. English is roughly 4 characters per token; many Indian languages and code use far more tokens per word, so the same text costs more.
• Latency: output is generated one token at a time, so the number of output tokens drives response time.
• Odd behaviour: character-level tasks (counting letters, reversing strings) and arithmetic on long numbers are hard because the model never sees individual characters.
• Chunking: RAG chunk sizes should be set in tokens, not characters.

Always count with the model's own tokenizer (for example tiktoken). Different tokenizers give different counts.

In self-attention every token builds a query, a key and a value vector. A token's output is a weighted mix of all the value vectors, where the weights are softmax(Q·Kᵀ / √d), which measures how relevant each other token is to it. Multi-head attention runs several of these in parallel so that different heads can track different relationships (syntax, coreference, position). A decoder-only LLM adds a causal mask so each token only attends to earlier ones.

Why long context is expensive:

• Attention compares every token with every other token, so compute and memory grow as O(n²) with sequence length during prefill.
• During generation the KV cache grows linearly with context and often becomes the memory bottleneck.

Ways around it: FlashAttention (exact attention, computed without materialising the n×n matrix), sliding-window or sparse attention, grouped-query attention (GQA), which shrinks the KV cache, and retrieval (RAG), so that you don't send the whole corpus as context.

At each step the model outputs a probability distribution over the vocabulary, and decoding decides which token to emit.

• Temperature rescales the logits before the softmax. Low values (0 to 0.3) sharpen the distribution toward the top token and give focused, repeatable output. High values (0.8 to 1.2) flatten it and give more varied, creative output. A temperature of 0 is effectively greedy decoding.
• Top-k samples only from the k most likely tokens.
• Top-p (nucleus) samples from the smallest set of tokens whose probabilities add up to p (for example 0.9). Unlike top-k, it adapts to how confident the model is.

How to choose: use low temperature for extraction, classification, code and anything you parse as JSON. Use moderate temperature with top-p for chat and brainstorming.

Temperature 0 is not a guarantee of determinism: batching and floating-point non-determinism on GPUs can still change outputs. If you need reproducibility, cache responses.

RAG (retrieval-augmented generation) retrieves relevant documents at query time and puts them in the prompt. Fine-tuning changes the model's weights using examples.

Rule of thumb: start with prompting, add RAG for knowledge, and fine-tune (often LoRA) only when the model reliably fails at a skill or you need a smaller, cheaper model for a narrow task. They combine well: a fine-tuned model can be better at using retrieved context.

Ingestion (offline): parse documents (PDF, HTML, tables) → clean them → chunk → embed → store the vectors plus metadata (source, section, permissions, timestamp) in a vector DB, usually with a keyword (BM25) index alongside.

Query (online): optionally rewrite the query (fix typos, expand it, or turn a follow-up into a standalone question) → hybrid retrieval (vector + BM25) with permission filters → rerank the top 50 down to the top 5 with a cross-encoder → build the prompt with citations → generate → check the answer is grounded.

Chunking choices:

• Size: often 200 to 800 tokens with 10 to 20% overlap. Chunks that are too small lose context; chunks that are too large dilute the embedding and waste context window.
• Structure-aware: split on headings, paragraphs and code blocks rather than at fixed character counts. Keep a table with its header.
• Parent-child: retrieve small precise chunks, then send their larger parent section to the LLM.
• Contextual headers: prepend the document title and section path to each chunk before embedding.

Tune with an evaluation set (questions with known source passages), measuring recall@k for retrieval separately from answer quality.

Vector (dense) search matches meaning: "car won't start" finds "vehicle ignition failure". It is weak on exact tokens such as product codes, error IDs, names and rare jargon.

Keyword (sparse) search such as BM25 matches exact terms and is great for identifiers, but it misses synonyms and paraphrases.

Hybrid search runs both and merges the results, commonly with Reciprocal Rank Fusion (score = Σ 1/(k + rank)), which needs no score calibration between the two systems. It almost always beats either one alone.

Reranking: first-stage retrieval uses bi-encoders (query and document embedded separately), which is fast but coarse. A cross-encoder reranker (for example bge-reranker or Cohere Rerank) reads the query and each candidate together and scores relevance much more accurately. It is too slow to run over millions of documents, so the usual pattern is: retrieve 50 to 100 candidates cheaply, rerank them, keep the top 3 to 10.

The result is higher precision in the context window, which means fewer tokens, lower cost and fewer hallucinations.

Exact nearest-neighbour search compares the query with every vector, which is O(N·d) and too slow at millions of vectors. Vector DBs use approximate nearest neighbour (ANN) indexes, which trade a little recall for big speedups.

HNSW (Hierarchical Navigable Small World) builds a multi-layer graph. The upper layers are sparse and act like express lanes; the bottom layer contains every vector. A search starts at the top, greedily walks toward the query, and drops a layer each time it stops improving, which gives roughly O(log N) search.

• M: links per node. Higher means better recall and more memory.
• efConstruction: build-time effort. Higher means a better graph and slower builds.
• efSearch: query-time candidate list. This is the main recall vs latency knob.
• Downsides: it is memory-hungry (the graph plus full vectors are kept in RAM), deletes are awkward, and builds are slow.

Alternatives: IVF clusters vectors and searches only the nearest nprobe clusters, using less memory. PQ (product quantization) compresses vectors 10 to 30 times at some cost in accuracy. IVF-PQ and DiskANN handle billion-scale indexes that don't fit in RAM.

Always report recall@k against exact search when you tune these indexes, and remember that metadata filters can hurt recall unless the DB supports filtered HNSW.

A decoder generates one token at a time, and each new token attends to all previous tokens. Without caching, every step would recompute the keys and values for the whole sequence. The KV cache stores each layer's key and value tensors for tokens already processed, so each step only computes them for the new token.

Two phases:

• Prefill: process the whole prompt in parallel and fill the cache. This is compute-bound and sets time to first token (TTFT).
• Decode: generate tokens one by one, reading the cache each time. This is memory-bandwidth-bound and sets time per output token (TPOT).

Why serving engineers care: cache size ≈ 2 × layers × kv_heads × head_dim × seq_len × batch × bytes. For long contexts it can exceed the model weights and caps how many requests fit on a GPU.

Optimisations: PagedAttention (vLLM) allocates the cache in pages like OS virtual memory, which removes fragmentation. Prefix caching reuses the cache for shared system prompts. GQA/MQA reduce the number of KV heads. A quantized KV cache (FP8/INT8) and offloading to CPU memory help too.

Quantization stores weights (and sometimes activations or the KV cache) at lower precision: FP16/BF16 → INT8, FP8 or 4-bit. A 70B model needs about 140 GB in FP16 and about 35 GB in 4-bit.

Benefits: less GPU memory (fewer or cheaper GPUs, room for bigger batches), and faster decoding, because decoding is memory-bandwidth-bound and smaller weights move faster.

Approaches:

• Post-training quantization (PTQ): no retraining. Examples are GPTQ and AWQ (both use calibration data to protect important weights), plus bitsandbytes and llama.cpp GGUF formats.
• Quantization-aware training (QAT): simulates low precision during training. Better accuracy, more work.
• QLoRA: fine-tunes LoRA adapters on top of a frozen 4-bit base model.

Trade-offs: some quality loss. 8-bit is usually near-lossless; 4-bit is often fine for chat but can hurt maths, code and long reasoning. Results depend on hardware and kernel support. Always re-run your evaluation suite on the quantized model instead of trusting benchmark claims.

GPUs are efficient only when they work on many sequences at once, but requests arrive at different times and have different lengths.

• Static batching waits for a batch to fill, then runs every sequence until the longest one finishes. Short requests wait and GPU slots sit idle.
• Continuous (in-flight) batching (vLLM, TGI, TensorRT-LLM) schedules per decoding step: finished sequences leave and new ones join immediately. This gives large throughput gains.

Other levers:

• PagedAttention for KV-cache memory, so more sequences fit.
• Prefix caching for shared system prompts.
• Chunked prefill, so a long prompt doesn't stall other users' decoding.
• Speculative decoding: a small draft model proposes several tokens and the big model verifies them in one pass. Same output distribution, lower latency.
• Quantization and tensor parallelism across GPUs.
• Routing: send easy queries to a smaller model.

Metrics to state: TTFT, TPOT or tokens per second per user, total throughput, p95/p99 latency, GPU utilisation and cost per 1M tokens. Throughput and per-user latency trade off, so set an SLO and maximise batch size under it.

Full fine-tuning updates every weight, which needs optimizer state for billions of parameters and a full copy of the model for every variant.

LoRA (Low-Rank Adaptation) freezes the base model and learns a small update for chosen weight matrices: W' = W + B·A, where A and B have a small rank r (4 to 64). Usually only 0.1 to 1% of the parameters are trained.

Why it's popular:

• Much less GPU memory, so it trains on a single GPU, especially with QLoRA (a 4-bit base model plus LoRA adapters).
• Adapters are megabytes, not gigabytes. You can keep one per customer or task and hot-swap them on a shared base model; multi-LoRA serving in vLLM or LoRAX batches requests for different adapters together.
• The adapter can be merged into the weights for zero extra inference latency.
• Less catastrophic forgetting, because the base weights don't change.

Knobs: rank r, scaling alpha, which layers to adapt (attention projections, sometimes the MLP), learning rate. Other PEFT methods include prefix/prompt tuning and adapters, but LoRA is the default.

1. Pre-training: next-token prediction on huge text corpora. The model knows a lot but doesn't follow instructions well.

2. SFT (supervised fine-tuning): train on curated (instruction, ideal response) pairs. This teaches format and instruction following.

3. Preference optimisation, so outputs match what people prefer (helpful, honest, harmless):

• RLHF: collect human rankings of response pairs, train a reward model on them, then optimise the policy with RL (usually PPO), with a KL penalty that keeps it close to the SFT model. It is powerful but complex and unstable, and it needs several models in memory.
• DPO (Direct Preference Optimization): skips the reward model and the RL loop. It is a simple classification-style loss directly on (chosen, rejected) pairs. Easier and more stable, which makes it the common choice now.
• Variants: RLAIF / Constitutional AI (AI feedback instead of human feedback), KTO, ORPO.

Risks to mention: reward hacking (the model exploits flaws in the reward model, for example by getting verbose or sycophantic), drift away from base capabilities (hence the KL term), and the cost and consistency of human labels.

LLMs are trained to produce plausible next tokens, not true ones. When knowledge is missing, out of date or ambiguous, the most plausible continuation can be a confident invention: fake citations, APIs that don't exist, wrong numbers.

Mitigations, layered:

• Ground answers: use RAG with good retrieval, tell the model to answer only from the provided context, and require citations to chunk IDs.
• Allow abstaining: say explicitly "if the answer isn't in the context, say you don't know", and reward that behaviour in evaluations.
• Use tools for facts: calculators, database queries and search instead of model memory. Return structured output validated by a schema.
• Lower temperature for factual tasks.
• Verify after generating: NLI or LLM-judge groundedness checks (is every claim supported by a source?), self-consistency (sample several times and compare), or validate citations against the retrieved text.
• Product UX: show sources, flag low confidence, and keep a human in the loop for high-stakes outputs.

Measure it: keep a factuality and groundedness evaluation set and track the hallucination rate per release. You can't eliminate hallucination, only reduce it and make it visible.

Prompt injection happens when untrusted text changes the model's instructions. It can be direct (a user types "ignore previous instructions and...") or indirect (instructions hidden in a web page, email, PDF or tool result that the model reads). It is the LLM version of SQL injection, except there is no reliable escaping, because instructions and data share one channel.

Impact grows with what the model can do: leaking the system prompt or other users' data, exfiltrating data through links or tool calls, or taking unwanted actions (sending email, deleting files) in agents.

Defence in depth:

• Least privilege: scope tools and tokens tightly, prefer read-only, and filter retrieved documents per user permissions before they reach the model.
• Human confirmation for consequential actions (payments, sends, deletes).
• Treat model output as untrusted: validate tool arguments, don't render arbitrary links or images (they are an exfiltration channel), sandbox code execution.
• Separate and label untrusted content in the prompt, and use input/output classifiers to detect injection attempts. These help but can be bypassed.
• Design patterns: dual-LLM / quarantine (a privileged model never reads raw untrusted text), plan-then-execute.
• Monitoring and red-teaming, with logs of tool calls.

Give the honest answer: no single filter solves it. Limit the blast radius.

Guardrails are checks around the model that enforce policy and reliability. They are separate from the model itself, so they can be tested, versioned and changed without retraining.

Input side (before the LLM):

• Moderation and toxicity classifiers, and topic restrictions (a banking bot declines medical advice).
• PII detection and redaction before sending text to a third-party API.
• Prompt-injection and jailbreak detection, plus length and rate limits.

Output side (after the LLM):

• Moderation of the response, and PII or secret leakage checks.
• Schema validation for structured output (JSON Schema or Pydantic/zod), with retry or repair on failure.
• Groundedness or citation checks for RAG answers, and business rules (no promising refunds over ₹X).

Action side (agents): tool allowlists, argument validation, spend limits and human approval steps.

Design considerations: each check adds latency and cost, so run cheap classifiers first, run checks in parallel with generation, and moderate streaming output in chunks. Track false positives (over-refusals annoy users) as closely as misses. Tools include Llama Guard, NeMo Guardrails, Guardrails AI and the moderation APIs from model providers.

Treat evaluations like unit tests for a product whose behaviour is non-deterministic.

Offline, before release:

• Build a golden dataset of real or realistic inputs with expected outputs or grading criteria. Cover edge cases, adversarial inputs and every key intent; a few hundred well-chosen examples beat thousands of random ones.
• Metrics by task: exact match or F1 for extraction, pass@k or unit tests for code, retrieval recall@k and MRR for RAG, faithfulness/groundedness and answer relevance for generation, plus format validity, latency and cost per request.
• Graders: deterministic checks where possible, then LLM-as-judge with a clear rubric, then human review for calibration.
• Run the suite in CI on every prompt, model or retrieval change, and block regressions.

Online, after release:

• Implicit signals (regenerations, copies, abandonment) and explicit ones (thumbs up or down).
• A/B tests of prompt or model variants on business metrics.
• Sample production traces into review queues and add failures to the golden set.

BLEU and ROUGE correlate poorly with quality for open-ended generation. Prefer task-specific checks and rubric-based judging.

LLM-as-a-judge uses a strong model to grade outputs, either scoring one answer against a rubric or picking the better of two (pairwise comparison). It scales evaluation far beyond what human labelling can.

Known biases and pitfalls:

• Position bias: in pairwise mode it favours the first (or second) answer. Fix: evaluate both orders and average.
• Verbosity bias: it prefers longer answers. Fix: tell it to ignore length, and control for length.
• Self-preference: it rates outputs from its own model family higher. Fix: use a judge from a different family.
• Vague rubrics give noisy scores. Fix: specific criteria, low temperature, ask for reasoning before the score, use binary or small scales (pass/fail, 1 to 5) instead of 1 to 100, and give few-shot graded examples.
• Weak at verification: judges miss subtle factual or maths errors. Use reference answers, tools or execution where possible.

Make it trustworthy: calibrate against human labels (measure agreement, for example Cohen's kappa), re-check whenever you change the judge prompt or model, and keep a human-reviewed slice in every evaluation run. Cost is real too: a judge call per test case on every CI run adds up, so cache the results.

From the confusion matrix (TP, FP, FN, TN):

• Precision = TP / (TP + FP): of everything flagged, how much was right? Use it when false positives are costly, for example a spam filter hiding real mail.
• Recall = TP / (TP + FN): of all the real positives, how many were caught? Use it when misses are costly, for example fraud, cancer screening or content safety.
• F1 is the harmonic mean of the two. Use it when you need one number and care about both.
• ROC-AUC is the probability that a random positive is ranked above a random negative, across all thresholds. It is threshold-independent, but it looks optimistic on heavily imbalanced data, because the large number of true negatives keeps the false-positive rate low.
• PR-AUC (average precision) is better for rare positives such as fraud at 0.1%.
• Accuracy is misleading under imbalance: predicting "not fraud" every time is 99.9% accurate.

In a system design answer: pick the metric from the business cost of each error type, choose the operating threshold from that (for example "recall ≥ 95% at precision ≥ 50%"), and report metrics per segment (country, device, new users), not just overall.

Models degrade silently after deployment because the world changes.

• Data (covariate) drift: the input distribution shifts (new user segments, a new app version logs a feature differently, seasonality). P(X) changes.
• Concept drift: the relationship between inputs and the label changes (fraudsters adapt, user tastes shift). P(Y|X) changes, so the model is wrong even on familiar-looking inputs.
• Label shift: the class balance changes. P(Y) changes.

Monitoring:

• Input monitoring: per-feature statistics (null rates, ranges, cardinality) and distribution distances against the training data, such as PSI, KL/JS divergence or the KS test. For embeddings, track centroid distance or cluster share.
• Output monitoring: shifts in the prediction distribution and in confidence. These are an early warning when labels arrive late.
• Performance monitoring: the real metric once delayed labels arrive (chargebacks can take weeks). Use proxies until then.
• Data quality and pipeline checks: schema validation, freshness, and training-serving skew (the same feature computed differently online and offline).

Response: alerts with sensible thresholds, scheduled or triggered retraining, a shadow or canary release for the new model, and a quick rollback path.

Offline metrics (AUC, NDCG) don't guarantee business impact, so you validate online.

Rollout ladder:

1. Shadow mode: the new model scores live traffic but its predictions aren't used. Compare its outputs, latency and errors at no user risk.
2. Canary: a small percentage of real traffic, with automated guardrail checks.
3. A/B test: randomly split users (by user ID, not by request, so each person gets a consistent experience) into control and treatment.

Design the test:

• Pre-register one primary metric (CTR, conversion, watch time) plus guardrail metrics (latency, revenue, complaints, unsubscribes).
• Compute sample size and duration up front from the minimum detectable effect and statistical power. Run at least one full weekly cycle.
• Don't peek and stop early when the result looks good; use sequential testing if you must monitor.
• Watch for novelty effects, network effects (marketplaces, social features) and sample ratio mismatch.

Alternatives: interleaving for rankers (mixing both models' results in one list is far more sensitive), and multi-armed bandits when you want to limit regret during exploration.

Ship only if the primary metric wins and no guardrail regresses.

Training-serving skew means the model sees different feature values in production than in training, so offline metrics look great and online performance quietly drops.

Causes:

• Two code paths: features computed in SQL or Spark for training and re-implemented in Java or Python for serving, with subtle differences (time zones, null handling, rounding).
• Data leakage / point-in-time errors: training used values that weren't yet known at prediction time, for example a user's 7-day click count computed after the event.
• Staleness: a batch feature refreshed daily at serving time but treated as real-time in training.

A feature store (Feast, Tecton, Vertex or SageMaker Feature Store) provides:

• One feature definition that computes both the offline store (history for training) and the online store (low-latency key-value lookups for serving).
• Point-in-time correct joins, so training rows only see feature values as of the event timestamp.
• Reuse and discovery of features across teams, plus monitoring of freshness and drift.

Also: log the features actually used at serving time and train on those logs, which is the most reliable way to remove skew.

• Bias: error from an over-simple model that misses real patterns (underfitting). Training and validation error are both high.
• Variance: error from a model so flexible it fits noise in the training data (overfitting). Training error is low and validation error is much higher.

More capacity lowers bias and raises variance. The goal is the best generalisation, not the lowest training loss. Large deep models complicate the classic U-curve ("double descent"), but the practical advice stands: judge on held-out data.

Detect it: use a proper train / validation / test split (time-based for temporal data, grouped by user to avoid leakage), learning curves, and cross-validation for small datasets.

Fix overfitting: more or better-labelled data, data augmentation, regularisation (L1/L2, dropout, weight decay), simpler models or fewer features, early stopping, ensembling (bagging and random forests reduce variance).

Fix underfitting: a richer model, better features, longer training, less regularisation.

Tip: a gap between offline validation and online results is often leakage or skew, not classic overfitting.

Data level:

• Undersample the majority class: fast, but throws away data. Often fine at very large scale.
• Oversample the minority class, or generate synthetic examples (SMOTE). Do this only on the training split, never before splitting.

Algorithm level:

• Class weights or cost-sensitive loss (weight positives about 1000×), or focal loss, which focuses on hard examples.
• Tree ensembles (XGBoost/LightGBM with scale_pos_weight) handle tabular imbalance well.
• Anomaly detection (isolation forest, autoencoders) when positive labels are very scarce.

Evaluation: don't use accuracy. Use PR-AUC, recall at a fixed precision, or cost-weighted metrics, and use stratified splits.

Decision level:

• Tune the threshold on validation data to match business costs. The default 0.5 is almost never right.
• Recalibrate probabilities (Platt scaling, isotonic regression) if you resampled, so that scores mean real probabilities again.

System angle: combine the model with rules, send uncertain cases to human review, and watch for label delay and adversaries who adapt (concept drift).

A multi-stage funnel, because you can't score millions of items per request with a heavy model:

1. Candidate generation (retrieval): reduce millions of items to about 1000 using several sources: a two-tower model (user and item embeddings, ANN search), collaborative filtering, "similar to recently watched", trending and follows.
2. Ranking: a heavier model (GBDT or a deep network such as DLRM or DCN) scores the candidates on rich features to predict engagement (click, watch time, like), often multi-task.
3. Re-ranking / policy: diversity, freshness, de-duplication, business rules and fairness constraints.

Two-tower: the user tower and item tower are trained so that the dot product predicts interaction. Item embeddings are precomputed and indexed in an ANN index, and the user embedding is computed at request time.

Cold start:

• New users: popular or trending items by region, onboarding preferences, context features (device, location, time), and quick adaptation from in-session clicks.
• New items: content features (text and image embeddings, category, creator) so the item tower can embed them without interaction history, plus an exploration budget (bandits) to collect signal.

Pitfalls: feedback loops and popularity bias, position bias in the training data, and the gap between optimising clicks and long-term satisfaction.