How Fine-Tuning Changes LLM Weights & Next-Word Prediction (LoRA / DoRA)

🚀 Welcome! This Interactive Guide Walks You Through the Complete LLM Fine-Tuning Lifecycle

You are about to trace how a general-purpose AI (like GPT / Llama) is transformed into a domain-specific financial analyst — step by step. The 6 stages below mirror exactly how production ML teams at companies like Bloomberg, Morgan Stanley, and Google fine-tune LLMs:

1. CPT — Feed raw domain text (SEC filings)

2. SFT — Teach Q&A format with loss masking

3. Adapters — Attach LoRA/DoRA sidecar modules

4. Softmax — Watch word probabilities shift

5. Training — Simulate loss descent in real-time

6. Alignment — Prefer concise, accurate answers

💡 Tip: Click any word, hover any matrix cell, drag sliders, and toggle between LoRA/DoRA/Base to explore interactively. Every visual updates in real-time!

Stage 1: Continual Pre-Training (CPT)

Ingesting Unstructured Domain Knowledge (SEC 10-Ks)

CPT: Raw Domain Text

📚 Think of CPT like reading a financial library cover-to-cover:
• Unstructured Text: The model reads raw SEC 10-K filings line-by-line.
• No Loss Masking (L = 1): The AI practices predicting every single next word (Loss weight L = 1.0).
• Goal: Ingests specialized finance vocabulary ("EBITDA", "amortization", "margins") into the base model before instruction tuning.
• ⚠️ Pro Tip — Replay Buffer: Production CPT pipelines (like BloombergGPT, CodeLlama) mix in 5–10% general English data alongside domain text to prevent catastrophic forgetting — the model losing standard grammar while learning finance.
➡️ What comes next: After absorbing domain vocabulary, the model needs to learn how to structure answers — that's what Stage 2 (SFT) teaches via question-answer flashcards.

📐 Formal Loss Formulation (Autoregressive Cross-Entropy)

$$\mathcal{L}_{\text{CPT}}(\theta) = - \sum_{t=1}^{T} \log P_\theta(x_t \mid x_{\lt t})$$ Every token x_t in the domain corpus is active (L = 1.0). The model learns to predict the next word at every position, absorbing domain vocabulary and writing patterns into its weight matrices.

💡 Click any word below or use the stepper to trace continuous next-word predictions:

🎯 Selected Token: Click any word above to inspect its token ID and next-word training target.

Stage 2: Supervised Fine-Tuning (SFT)

Instruction Tuning & Prompt Loss Masking (L = -100)

SFT: Q&A Flashcards

💬 Think of SFT like practicing Q&A flashcards with a mentor:
• Prompt Masking (L = -100): The user question is completely masked out with loss weight -100 so the AI is never graded or penalized for reading what the user asked.
• Assistant Response (L = 1): Gradients and loss are only computed on the assistant's answer, teaching the model how to follow instructions and format answers as a professional financial analyst!
• KV Cache Contribution: Even though prompt tokens are masked from loss, they still populate the attention KV cache — the model reads and fully understands the question, it's just never penalized for the question text itself.
➡️ What comes next: SFT teaches answer formatting, but standard full fine-tuning updates ALL model weights (expensive!). Stage 3 introduces tiny LoRA/DoRA adapter modules that achieve the same result while training only 0.4% of parameters.

📐 Completion-Only Loss (PyTorch ignore_index = -100)

$$\mathcal{L}_{\text{SFT}}(\theta) = - \sum_{t=|x|+1}^{|x|+|y|} \log P_\theta(y_t \mid x, y_{\lt t})$$ Loss is computed only over assistant response tokens y. Prompt tokens x have label = -100 (PyTorch's ignore_index), which zeroes their gradient contribution while still letting them inform the attention context.

💡 Click words to compare Masked Prompt (-100) vs Graded Response (L=1.0):

🎯 Selected Token: Notice how prompt tokens have L = -100 (Ignored), while assistant tokens have L = 1.0 (Graded!).

Stage 3: Inside The AI Brain

How Adapters (LoRA & DoRA) Nudge Thoughts

🧠 The "Giant Brain + Tiny Sidecar" Intuition:
• The Giant Brain (W₀): 16+ Million numbers. We keep it 100% frozen so it never forgets standard English.
• The Tiny Sidecar (LoRA A × B): Instead of retraining the giant brain, we attach two tiny filters. Matrix A compresses the word into an 8-number summary (r = 8). Matrix B turns that summary into a gentle nudge (Δh) that steers the AI toward domain concepts (saves 99.6% memory!).
• DoRA: Adds a Volume Knob (m) that controls how loudly the new concept speaks!
➡️ What comes next: The adapter nudges the hidden state vector, but how does that change which word the AI outputs? Stage 4 shows the Softmax probability race where domain words overtake generic ones.

🎮 How to explore this stage: Toggle between Base / LoRA / DoRA tabs above to compare architectures. Switch between ⚡ Electrical Pathway (circuit flow) and 🧮 2D Matrix (numeric weights) views below. Drag the Rank (r) and Alpha (α) sliders to see how hyperparameters change adapter behavior in real-time.

💡 Toggle views to see circuit flow vs 2D weight matrix math

⚡ Electrical Thought Pathway:

⚡ Hover or click any block: See how electrical current passes through each module!

What the AI is Thinking (Concept Neurons)

Word: " EBITDA"

Watch how the adapter dials down generic chatter and dials up specialized domain concepts:

• Step 0: The adapter hasn't learned yet. All concept nudges are neutral until you click Train!

Step 1 🔒 Freeze Base Matrix (W₀)

• What it is: 16.7M weights per layer (Billions total) that understand standard English grammar and general knowledge.
• Why we lock it: Training base weights requires massive GPU clusters and risks catastrophic forgetting (the AI forgetting how to speak English). Freezing keeps it 100% safe.

Step 2 — LoRA 🤏 Compress via Strips (B × A)

• The Low-Rank Secret: Weight adjustments (ΔW) have a low "intrinsic rank".
• Matrix A (2×6): Compresses incoming tokens into a compact summary.
• Matrix B (6×2): Re-expands the summary into domain-specific nudges.
• Multiplying B × A synthesizes the full 6×6 ΔW grid while saving 99.6% memory!

Step 3 — LoRA ⚡ 0-Latency Weight Merging

• Training Time (Modular): ΔW is saved as a tiny 20MB file. You can swap Finance, Legal, or Medical adapters on a single server without restarting the model!
• Deployment (Merged): Add ΔW directly into W₀ once: W_new = W₀ + (α / r) · (B × A). Runs at 100% full original speed with 0ms extra compute!

Summary Rank (r) — Adapter Detail Level r = 8 (99.6% savings)

🧠 What it does: Controls the adapter's capacity. r = 8 compresses 4,096 dimensions → 8 summary numbers. Common values: r = 4 (lightweight, chatbots), r = 8–16 (standard fine-tuning), r = 64 (complex reasoning). Higher r = more expressive, but trains more parameters.

Nudge Strength (α) — Scaling Multiplier α = 16 (Scale = 2.00×)

📢 What it does: Controls the adapter's influence strength: Scale = α / r = 2.0×. Practitioner rule of thumb: set α = 2 × r to start. Too high → unstable training (loss spikes). Too low → adapter cannot express sufficient domain shift.

Stage 4: The Next-Word Race

Which Word Does the AI Guess Next?

LoRA Adapted

🏁 The Word Race: The AI converts its nudged thought into percentage chances for every possible word. Given the current word , notice how untrained models bet on generic words (), but after adapter training, the target domain word () surges into 1st place!
➡️ What comes next: We've seen probabilities shift for one word — Stage 5 lets you simulate an entire training run and watch the loss curve drop from high confusion to near-certainty.

📐 Softmax Probability (LM Head Projection)

$$P(w_i \mid x) = \frac{\exp(z_i / T)}{\sum_{j=1}^{|V|} \exp(z_j / T)}, \quad z_i = W_{\text{head}, i}^\top h$$ The adapted hidden state h ∈ ℝ⁴⁰⁹⁶ is projected through the vocabulary head (50,000+ words) to produce logits z, then normalized via Softmax. T = 1.0 for standard greedy decoding. As the adapter trains, the logit z for domain words like "margins" increases, pushing its probability bar upward.

Word Context:

Adapter Training:

Target Correct Word: " margins"

Before fine-tuning, the Base Model guessed generic words (" growth", " revenue"). Adapter training successfully shifts the model's guess to the correct domain term.

Stage 5: Interactive Training Simulation

Watch the AI Learn & Error Drop (Interactive Simulation)

Step 0 / 10

📉 AI Confusion Score (Click any step node on the chart to jump!) Current Error: 2.842

📉 Confusion Score

2.842

High Confusion (Untrained)

🎯 Target Word Confidence

Surges to >90% with training

📊 Perplexity (PPL = e^Loss)

17.15

Guessing among ~17 words (Untrained)

💾 Memory & Weights Saved

99.6%

Only trains 0.4% adapter params

Stage 6: Post-Training Alignment

Teaching AI Quality Preferences (RLHF vs. DPO vs. SimPO)

🤔 Why do we need alignment after training? After SFT and LoRA training (Stages 1–5), the model is accurate but not necessarily helpful. It may generate verbose, rambling, or repetitive answers because standard cross-entropy loss doesn't penalize wordiness — a 120-word fluff answer and a crisp 15-word answer both get graded the same if they contain the target words. Alignment teaches the model to prefer quality and conciseness.

💾 GPU VRAM Footprint & Model Architecture: 50% Memory Savings vs DPO

⚡ SimPO Architecture: Only 1 model loaded in GPU VRAM (Reference-Free). Fast, lightweight, and memory-efficient!

Preference Battle: Chosen (yw) vs. Rejected (yl) Prompt: "Summarize Q3 EBITDA"

🟢 Candidate A: Winning Answer (yw) 15 tokens (Crisp)

"Q3 EBITDA grew 18% YoY to $42M, driven by 120bps margin expansion in Enterprise SaaS."

Avg Log-Prob: -0.42 / tok SimPO Reward: +1.60

🔴 Candidate B: Rejected Answer (yl) 120 tokens (Verbose)

"EBITDA is an acronym that stands for earnings before interest, taxes, depreciation, and amortization, which is a financial metric that many executives..."

Avg Log-Prob: -1.85 / tok SimPO Reward: -0.74

🎯 Reward Separation Gap (Δr = r_w - r_l): +2.34

Target Margin Threshold (γ = 0.8) ✅ Target Margin Satisfied!

Target Margin (γ) — Quality Gate γ = 0.8 (Strict Gate)

🧠 What it does: Forces the chosen answer's reward to exceed the rejected answer by at least γ. SimPO innovation only. Higher γ = stricter quality gate. Set γ = 0.8 for standard tasks, γ = 1.2–1.5 for high-precision domains.

Beta Temperature (β) — Sensitivity β = 2.0 (Balanced)

⚖️ What it does: Controls how aggressively log-probabilities scale into preference reward gradients.

📊 Side-by-Side Comparison: RLHF vs. DPO vs. SimPO

Method	Models in GPU VRAM	Reference Model Needed?	Length-Bias / Verbosity Immune?	Target Margin (γ)?	Relative Training Speed
🤖 RLHF (PPO)	4 Models (~96 GB)	Yes (Frozen π_ref)	❌ Vulnerable (Exploits length)	❌ No margin	1.0× (Slow 4-step loop)
⚖️ DPO	2 Models (~48 GB)	Yes (Frozen π_ref)	❌ Vulnerable (Raw log-prob sum)	❌ No margin	2.0× (Simpler loss)
⚡ SimPO (SOTA)	1 Model (~24 GB)	✅ NO (Reference-Free!)	✅ 100% Immune (1/\|y\| Normalized)	✅ Enforced (γ Margin)	3.5× Faster (Max throughput)