The Complete
Study Guide

1.1 What Is an Encoder?

An encoder takes raw input — an image, a sentence, a signal — and turns it into a smaller, compressed form that still carries the important information. Think of it like summarizing a 300-page book into 3 paragraphs. You lose some detail, but the core meaning stays.

The compressed form is called a latent representation (or embedding). It lives in a space called latent space. Latent space is lower-dimensional, meaning it has fewer numbers but packs in more meaning per number.

1.2 How Encoders Work

An encoder is typically a neural network — layers of matrix multiplications followed by activation functions (like ReLU). Each layer transforms the data a bit more, gradually compressing and abstracting it.

1.3 Types of Encoders

2.1 What Is a Decoder?

A decoder is the reverse of an encoder. It takes a compressed latent representation and reconstructs an output — either the original input, or something generated in the same format.

The output does not have to be exactly the original input. In generative models, the decoder generates new things from new latent vectors never seen during training. This is how we generate new images or text.

2.2 How Decoders Work

Decoders mirror encoders in structure but do the reverse operation.

2.3 Encoder-Only vs Decoder-Only vs Encoder-Decoder

This distinction is critical for understanding modern architectures.

3.1 The Core Idea

An autoencoder (AE) is a neural network trained to take an input, compress it into a small latent vector, and then reconstruct the original input from that vector — as accurately as possible.

The bottleneck forces the network to learn what is actually important in the data. If you compress a 784-dimensional image into a 32-dimensional vector, the network has no choice but to learn the essential structure.

3.2 Architecture

Input x (784-dim)
     ↓
[Encoder Layer 1: 784 → 256]  ReLU
     ↓
[Encoder Layer 2: 256 → 128]  ReLU
     ↓
[Bottleneck: 128 → 32]        ← Latent code z
     ↓
[Decoder Layer 1: 32 → 128]   ReLU
     ↓
[Decoder Layer 2: 128 → 256]  ReLU
     ↓
[Output Layer: 256 → 784]     Sigmoid

3.3 Applications

3.4 The Problem with Vanilla AEs — Motivating the Variants

This is exactly why we need the variants: Denoising AE, Stacked AE, Contractive AE, and Variational AE (VAE).

4.1 The Main Idea

A Denoising Autoencoder (DAE) deliberately corrupts the input before feeding it to the encoder — then trains the network to recover the clean original. The training signal says: "here is noisy data, now reconstruct the clean version."

4.2 Types of Noise

4.3 Why This Works Better

By forcing the encoder to see corrupted inputs and still reconstruct clean outputs, you push it to learn the underlying data manifold — the smooth surface in high-dimensional space where real data lives. Noise pushes data off the manifold; the network learns to project it back.

5.1 The Idea

A stacked autoencoder is multiple autoencoders layered on top of each other. The compressed output of one AE becomes the input to the next. The motivation is greedy layer-wise pretraining — a technique from 2006 that made it possible to train deep networks before modern tricks like batch normalization existed.

5.2 Training Procedure

Phase 1: Layer-wise Pretraining

Phase 2: Fine-tuning (Unrolling)

Stack all encoders and all decoders (in reverse). Train the whole network end-to-end with a small learning rate to refine everything together.

5.3 Why Layer-wise Pretraining Helps

The concept lives on in modern deep pretraining — BERT pretraining, LLM pretraining, MAE — all train hierarchically in some sense.

6.1 Motivation

The contractive AE (CAE) addresses a specific problem: a vanilla AE's encoder is sensitive to small changes in input. If you perturb x slightly, the latent code z might change wildly. We want the opposite — a stable encoder that ignores small, irrelevant variations.

6.2 The Contractive Penalty

Where J_f(x) is the Jacobian matrix of the encoder — it measures how much each output of the encoder changes when each input dimension changes. ||·||²_F is the Frobenius norm (sum of squared entries).

Minimizing the Frobenius norm pushes the encoder to be insensitive to small input changes. This is the "contractive" property.

6.3 CAE vs DAE Comparison

6.4 Missing Variant: Variational Autoencoder (VAE)

7.1 The Problem Attention Solves

Before attention, sequence models (RNNs) processed sequences step by step and had to compress the entire history into a fixed-size vector. Long sequences were a disaster — by the time you got to the end, the beginning was mostly forgotten.

Attention solves this by letting every position in the output look directly at any position in the input. No compression. No forgetting.

7.2 Query–Key–Value Framework

Think of attention like a database lookup:

The division by √d_k prevents dot products from growing too large (which would push softmax into flat saturation regions, killing gradients).

7.3 Step-by-Step Walkthrough

Sentence: "The cat sat on the mat." We want the representation of "sat" that incorporates context.

7.4 Self-Attention vs Cross-Attention

7.5 Multi-Head Attention

Instead of one attention function, compute h parallel attention functions with different learned projections:

MultiHead(Q, K, V) = Concat(head₁, ..., headₕ) × W_O
head_i = Attention(Q·W_Qi, K·W_Ki, V·W_Vi)

Each head can focus on different relationship types: one head might track syntactic dependencies, another semantic similarity, another positional patterns. Richer than single-head attention.

7.6 Positional Encoding

Attention has no notion of order — it treats input as a set, not a sequence. Positional encodings are added to input embeddings to inject order information.

8.1 The Big Picture

The Transformer (Vaswani et al., "Attention Is All You Need," 2017) replaced recurrent networks for sequence-to-sequence tasks. It is built entirely on attention — no RNNs, no convolutions in the core architecture. The key insight: attention can replace sequential processing with parallel processing.

8.2 Full Architecture

INPUT TOKENS
     ↓
Token Embedding + Positional Encoding
     ↓
┌─────────────────────────────┐
│  ENCODER BLOCK (×N layers)  │
│  ┌───────────────────────┐  │
│  │ Multi-Head Self-Attn  │  │
│  └──────────┬────────────┘  │
│      + residual connection   │
│       Layer Normalization    │
│  ┌───────────────────────┐  │
│  │ Feed-Forward Network  │  │
│  └──────────┬────────────┘  │
│      + residual connection   │
│       Layer Normalization    │
└─────────────────────────────┘
     ↓ Encoder Output (context)
┌─────────────────────────────┐
│  DECODER BLOCK (×N layers)  │
│  ┌───────────────────────┐  │
│  │ Masked Self-Attention │  │ ← Causal: can't see future
│  └──────────┬────────────┘  │
│      + residual + LayerNorm  │
│  ┌───────────────────────┐  │
│  │  Cross-Attention      │  │ ← Attends to encoder output
│  └──────────┬────────────┘  │
│      + residual + LayerNorm  │
│  ┌───────────────────────┐  │
│  │ Feed-Forward Network  │  │
│  └──────────┬────────────┘  │
│      + residual + LayerNorm  │
└─────────────────────────────┘
     ↓
Linear + Softmax → token probabilities

8.3 Key Components

8.4 Computational Complexity

Self-attention is O(n²d) in time and O(n²) in memory, where n is sequence length. This quadratic scaling is the main bottleneck for long sequences.

9.1 The Core Idea

ViT (Dosovitskiy et al., 2020) asked a simple question: what if we just split an image into patches and treat them like a sequence of tokens, then apply a standard transformer? Turns out — it works extremely well, especially at scale.

9.2 Step-by-Step Architecture

9.3 ViT vs CNN

9.4 Important ViT Variants

10.1 The Core Idea — A Two-Player Game

A GAN (Goodfellow et al., 2014) consists of two networks trained in opposition:

10.2 Training Loop

For each training step:

1. Sample batch of real data {x₁, ..., xₘ}
2. Sample batch of noise {z₁, ..., zₘ}
3. Generate fakes: {G(z₁), ..., G(zₘ)}

Update Discriminator:
  Real samples → D should output 1
  Fake samples → D should output 0
  Minimize: −[log D(x) + log(1 − D(G(z)))]

Update Generator:
  Want D(G(z)) ≈ 1 (fool D)
  Minimize: −log D(G(z))   ← non-saturating loss

10.3 Why the Non-Saturating Loss?

The original G loss was log(1 − D(G(z))). Early in training, D can easily spot fakes (since G is terrible), so D(G(z)) ≈ 0 and the gradient ≈ 0 — G cannot learn.

The fix: use −log D(G(z)). When D(G(z)) ≈ 0, this loss is large, giving G a strong gradient to improve.

10.4 Common GAN Problems

10.5 Evaluation Metrics

11.1 DCGAN (2015)

The first practically successful image GAN. Key contributions: transposed convolutions in G, strided convolutions in D (no pooling), batch normalization, LeakyReLU in D. This architecture guideline became the universal GAN starting point.

11.2 Conditional GAN (cGAN, 2014)

Feed a class label (or other conditioning) to both G and D. G(z, y) → image conditioned on label y. Lets you control what the generator produces — ask for a "7" and get a 7.

11.3 Pix2Pix (2017)

Conditional GAN for paired image-to-image translation. Given (input image, target image) pairs, learn to translate between domains: sketch→photo, day→night, satellite→map. Loss = adversarial loss + L1 pixel-level reconstruction loss.

11.4 CycleGAN (2017)

Unpaired image-to-image translation. No need for matched pairs — just two sets of images (horses and zebras). Uses two generators and two discriminators, plus a cycle consistency loss: translating A→B→A should recover A.

11.5 StyleGAN (2019) / StyleGAN2 (2020)

Radical generator architecture redesign. Key ideas:

StyleGAN2 fixes blob artifacts from AdaIN. Powers "This Person Does Not Exist."

11.6 WGAN (Wasserstein GAN, 2017)

Addresses GAN instability theoretically. Replaces Jensen-Shannon divergence (implicit in standard GAN) with Wasserstein distance — a more stable training signal.

11.7 Progressive Growing (ProGAN, 2018)

Start generating 4×4 images, gradually increase to 8×8, 16×16, ... 1024×1024. Add new layers with smooth blending transitions. Makes training stable because the task starts easy (4×4) and the model builds on what it has already learned.

12.1 What Is Transfer Learning?

Transfer learning uses knowledge gained from solving one problem to help solve a different (but related) problem. The core empirical observation: neural networks trained on one task learn feature representations that are useful for many other tasks.

12.2 Why It Matters

Training large models from scratch requires millions of labeled examples, weeks of GPU time, and enormous cost. Transfer learning lets practitioners start from a powerful pretrained model and adapt it with a small dataset in hours.

12.3 When to Use Which Strategy

12.4 Negative Transfer

13.1 What Is Fine-Tuning?

Fine-tuning means taking a pretrained model and continuing training it on a new (typically smaller) dataset, adjusting weights to fit the new task. Unlike feature extraction (frozen backbone), fine-tuning lets the internal representations shift.

13.2 Standard Procedure

13.3 Parameter-Efficient Fine-Tuning (PEFT)

Modern LLMs have billions of parameters — full fine-tuning is expensive and risks overfitting. PEFT methods update only a small fraction of parameters:

13.4 Catastrophic Forgetting

When you fine-tune on a new task, the network can "forget" what it learned before. New gradient updates overwrite old knowledge. Mitigation: small LR, replay (mix old + new data), EWC (Elastic Weight Consolidation), or PEFT methods (frozen backbone prevents forgetting).

14.1 The Core Idea

Knowledge distillation (Hinton et al., 2015) transfers knowledge from a large, accurate model (the teacher) into a smaller, faster model (the student). The student learns from the teacher's soft output probabilities, not just hard labels.

14.2 Why Soft Labels Carry More Information

A hard label says: "this image is a cat." That is all. The teacher's output probabilities say: "95% cat, 4% dog, 1% tiger." These soft labels reveal which classes are similar — a dog is more like a cat than a car is. This extra signal helps the student learn better representations.

14.3 Distillation Loss

Where T is temperature — higher T makes soft labels softer (less peaked), making the small probabilities more readable and useful for the student.

14.4 Variants of Distillation

14.5 Real-World Applications

15.1 Broader Than Distillation

Knowledge distillation is one specific method of knowledge transfer. Knowledge transfer is the broader idea — any method by which knowledge learned in one setting is made useful in another. This includes: transfer learning, fine-tuning, distillation, multi-task learning, zero-shot transfer, and few-shot transfer.

15.2 Zero-Shot Transfer

The model is tested on a task it was never explicitly trained on. Works when the model has learned sufficiently general representations. CLIP is the canonical example — trained to align image and text, it can classify images into any category by comparing image embeddings to text embeddings of class names. No task-specific training needed.

15.3 Few-Shot Transfer

The model sees only K examples (K = 1, 5, or 10) of the new task and must generalize. This is the goal of meta-learning — training models to be good at learning quickly.

16.1 The Problem

You train a model on data from one distribution (source domain) and deploy it on data from a different distribution (target domain). Performance degrades — sometimes catastrophically. Example: train on California sunshine photos, deploy in winter fog in Germany.

16.2 Types of Distribution Shift

16.3 Key Methods

17.1 The Setting

You have a large pool of unlabeled data. Labeling is expensive (requires human annotators). You have a limited labeling budget. The question: which examples should you label to maximize model performance?

17.2 The Active Learning Loop

1. Start with small labeled set L, large unlabeled pool U
2. Train model on L
3. Score each unlabeled example using an acquisition function
4. Select top-K highest scoring examples from U
5. Have humans label those K examples
6. Add to L; go to step 2
7. Repeat until labeling budget is exhausted

17.3 Acquisition Functions

17.4 Batch Active Learning

Most real scenarios require selecting a batch of K examples at once. Naive approach: pick top-K by individual score. Problem: they might all be similar (redundant). Better: balance informativeness with diversity.

18.1 Combining Two Paradigms

Semi-supervised learning (SSL) uses both labeled and unlabeled data to improve the model. Active learning selects which examples to label. Together: use unlabeled data to improve the model, then use the improved model to better choose what to label next. A virtuous cycle.

18.2 Semi-Supervised Learning Methods (Background)

18.3 VAAL (Variational Adversarial Active Learning)

Trains a VAE on all data (labeled + unlabeled). Trains a discriminator to distinguish labeled from unlabeled data in latent space. Query examples the discriminator cannot tell apart from labeled data — regions of latent space not yet covered by labeled examples.

18.4 Why Combine SSL and AL?

19.1 What Is a Foundation Model?

A foundation model is a large model trained on vast amounts of broad data that can be adapted to many downstream tasks. Term coined by Stanford HAI in 2021. Properties: massive scale (billions of parameters), generality (not one task), emergent capabilities (appear at scale), adaptability (fine-tune / prompt for specific applications).

19.2 Pretraining Paradigms

19.3 Emergent Abilities

Certain capabilities appear only above a scale threshold. Below: near-random performance. Above: sudden, dramatic improvement. Examples: few-shot learning (GPT-3+), chain-of-thought reasoning, multi-step arithmetic, instruction following without explicit training. Not fully understood — active research area.

19.4 RLHF — Making Models Behave

Reinforcement Learning from Human Feedback shapes model behavior toward being helpful, honest, and harmless.

20.1 Architecture

Modern LLMs are transformer decoder-only models: causal self-attention only, trained to predict the next token autoregressively. No encoder, no cross-attention.

20.2 Scaling Laws

Chinchilla (2022) showed the optimal relationship: train on ~20 tokens per parameter. Many earlier models were over-parameterized and under-trained (too big, too little data). Loss follows power laws with both model size and training data size.

20.3 In-Context Learning

GPT-3 showed that LLMs can perform few-shot learning just from prompt formatting — no gradient updates. Give the model a few examples in the prompt; it infers the task from context and answers. The model is learning from context at inference time, without updating weights.

20.4 Chain-of-Thought Prompting

Adding "Let's think step by step" dramatically improves performance on reasoning tasks. The model generates intermediate reasoning steps before the final answer. The final answer is conditioned on the reasoning chain — externalizing computation that would otherwise need to happen implicitly.

21.1 What Are LVMs?

Large Vision Models are foundation models for visual data — pretrained on massive image datasets, adaptable to many vision tasks. Same philosophy as LLMs: pretrain at scale, adapt as needed.

21.2 Key Models

21.3 Self-Supervised Methods for Vision (Important Background)

22.1 What Makes a Model Multi-Modal?

A multi-modal model processes data from multiple modalities simultaneously — text, images, audio, video. The challenge is learning a joint representation space where different modalities can communicate with each other meaningfully.

22.2 CLIP (OpenAI, 2021)

Trained on 400 million (image, text) pairs from the internet. Learns to align image and text embeddings in a shared space.

22.3 BLIP (2022)

Addresses CLIP's weakness: noisy web training data. BLIP bootstraps cleaner training data.

22.4 BLIP-2 (2023)

Freeze a pretrained image encoder AND a pretrained LLM. Bridge them with a lightweight Q-Former — 32 learnable query tokens that attend to image features. Only the Q-Former is trained. Extremely parameter-efficient.

Frozen Image Encoder → Q-Former → Frozen LLM
                         ↑
               32 learnable query tokens

22.5 Flamingo (DeepMind, 2022)

Connects a frozen vision encoder to a frozen LLM using cross-attention layers inserted between LLM layers. Trained on interleaved image-text sequences. Enables few-shot visual question answering — give a few (image, question, answer) examples, then ask about a new image.

22.6 ImageBind (Meta, 2023)

Extends CLIP to 6 modalities: image, text, audio, depth, thermal, IMU. Uses images as the binding modality — aligning all modalities to images implicitly aligns them to each other. Enables cross-modal retrieval between modalities that were never directly paired in training (e.g., retrieve audio from text).

23.1 What Is LLaVA?

LLaVA (Large Language and Vision Assistant, Liu et al., 2023) connects a CLIP vision encoder to a LLaMA/Vicuna LLM using a simple linear projection layer. Despite its architectural simplicity, it achieves strong performance on visual dialogue and question answering. It democratized multi-modal LLM research.

23.2 Architecture

Image Input
    ↓
CLIP Image Encoder (frozen ViT-L/14)
    ↓  visual features [N × D_clip]
Linear Projection Layer (trainable)
    ↓  visual tokens [N × D_llm]
    ↕
LLM (LLaMA / Vicuna) — processes [visual tokens + text tokens] together
    ↓
Text output (answer to the question)

The visual features are projected into the LLM's embedding space and prepended to the text token sequence. The LLM sees them as "extra tokens" at the start of the sequence.

23.3 Training Data — The Key Innovation

LLaVA's training data was generated using GPT-4. The authors gave GPT-4 image captions and bounding boxes (text descriptions of visual content — not the actual images) and asked it to generate:

Total: 158K samples generated this way. High quality instruction-following data without expensive human annotation.

23.4 Two-Stage Training

23.5 LLaVA-1.5 and Beyond

23.6 Why LLaVA Matters