Bugyal
📖 Interactive Blueprint 🧭 Guided Journey
1
Stage 1: Continual Pre-Training (CPT)

Ingesting Unstructured Domain Knowledge (SEC 10-Ks)

CPT: Raw Domain Text
📚 Think of CPT like reading a financial library cover-to-cover:
• Unstructured Text: The model reads raw SEC 10-K filings line-by-line.
• No Loss Masking (L = 1): The AI practices predicting every single next word (Loss weight L = 1.0).
• Goal: Ingests specialized finance vocabulary ("EBITDA", "amortization", "margins") into the base model before instruction tuning.
• ⚠️ Pro Tip — Replay Buffer: Production CPT pipelines (like BloombergGPT, CodeLlama) mix in 5–10% general English data alongside domain text to prevent catastrophic forgetting — the model losing standard grammar while learning finance.
➡️ What comes next: After absorbing domain vocabulary, the model needs to learn how to structure answers — that's what Stage 2 (SFT) teaches via question-answer flashcards.
📐 Formal Loss Formulation (Autoregressive Cross-Entropy)
$$\mathcal{L}_{\text{CPT}}(\theta) = - \sum_{t=1}^{T} \log P_\theta(x_t \mid x_{\lt t})$$ Every token x_t in the domain corpus is active (L = 1.0). The model learns to predict the next word at every position, absorbing domain vocabulary and writing patterns into its weight matrices.
💡 Click any word below or use the stepper to trace continuous next-word predictions:
🎯 Selected Token: Click any word above to inspect its token ID and next-word training target.
Stage 1 of 6: CPT Domain Ingestion
2
Stage 2: Supervised Fine-Tuning (SFT)

Instruction Tuning & Prompt Loss Masking (L = -100)

SFT: Q&A Flashcards
💬 Think of SFT like practicing Q&A flashcards with a mentor:
• Prompt Masking (L = -100): The user question is completely masked out with loss weight -100 so the AI is never graded or penalized for reading what the user asked.
• Assistant Response (L = 1): Gradients and loss are only computed on the assistant's answer, teaching the model how to follow instructions and format answers as a professional financial analyst!
• KV Cache Contribution: Even though prompt tokens are masked from loss, they still populate the attention KV cache — the model reads and fully understands the question, it's just never penalized for the question text itself.
➡️ What comes next: SFT teaches answer formatting, but standard full fine-tuning updates ALL model weights (expensive!). Stage 3 introduces tiny LoRA/DoRA adapter modules that achieve the same result while training only 0.4% of parameters.
📐 Completion-Only Loss (PyTorch ignore_index = -100)
$$\mathcal{L}_{\text{SFT}}(\theta) = - \sum_{t=|x|+1}^{|x|+|y|} \log P_\theta(y_t \mid x, y_{\lt t})$$ Loss is computed only over assistant response tokens y. Prompt tokens x have label = -100 (PyTorch's ignore_index), which zeroes their gradient contribution while still letting them inform the attention context.
💡 Click words to compare Masked Prompt (-100) vs Graded Response (L=1.0):
🎯 Selected Token: Notice how prompt tokens have L = -100 (Ignored), while assistant tokens have L = 1.0 (Graded!).
Stage 2 of 6: SFT Instruction Tuning
3
Stage 3: Inside The AI Brain

How Adapters (LoRA & DoRA) Nudge Thoughts

🧠 The "Giant Brain + Tiny Sidecar" Intuition:
• The Giant Brain (W₀): 16+ Million numbers. We keep it 100% frozen so it never forgets standard English.
• The Tiny Sidecar (LoRA A × B): Instead of retraining the giant brain, we attach two tiny filters. Matrix A compresses the word into an 8-number summary (r = 8). Matrix B turns that summary into a gentle nudge (Δh) that steers the AI toward domain concepts (saves 99.6% memory!).
• DoRA: Adds a Volume Knob (m) that controls how loudly the new concept speaks!
➡️ What comes next: The adapter nudges the hidden state vector, but how does that change which word the AI outputs? Stage 4 shows the Softmax probability race where domain words overtake generic ones.
🎮 How to explore this stage: Toggle between Base / LoRA / DoRA tabs above to compare architectures. Switch between ⚡ Electrical Pathway (circuit flow) and 🧮 2D Matrix (numeric weights) views below. Drag the Rank (r) and Alpha (α) sliders to see how hyperparameters change adapter behavior in real-time.
💡 Toggle views to see circuit flow vs 2D weight matrix math
⚡ Electrical Thought Pathway:
⚡ Hover or click any block: See how electrical current passes through each module!
What the AI is Thinking (Concept Neurons)
Word: " EBITDA"
Watch how the adapter dials down generic chatter and dials up specialized domain concepts:
• Step 0: The adapter hasn't learned yet. All concept nudges are neutral until you click Train!
Summary Rank (r) — Adapter Detail Level r = 8 (99.6% savings)
🧠 What it does: Controls the adapter's capacity. r = 8 compresses 4,096 dimensions → 8 summary numbers. Common values: r = 4 (lightweight, chatbots), r = 8–16 (standard fine-tuning), r = 64 (complex reasoning). Higher r = more expressive, but trains more parameters.
Nudge Strength (α) — Scaling Multiplier α = 16 (Scale = 2.00×)
📢 What it does: Controls the adapter's influence strength: Scale = α / r = 2.0×. Practitioner rule of thumb: set α = 2 × r to start. Too high → unstable training (loss spikes). Too low → adapter cannot express sufficient domain shift.
Stage 3 of 6: LoRA / DoRA Adapters
4
Stage 4: The Next-Word Race

Which Word Does the AI Guess Next?

LoRA Adapted
🏁 The Word Race: The AI converts its nudged thought into percentage chances for every possible word. Given the current word " EBITDA", notice how untrained models bet on generic words (" growth"), but after adapter training, the target domain word (" margins") surges into 1st place!
➡️ What comes next: We've seen probabilities shift for one word — Stage 5 lets you simulate an entire training run and watch the loss curve drop from high confusion to near-certainty.
📐 Softmax Probability (LM Head Projection)
$$P(w_i \mid x) = \frac{\exp(z_i / T)}{\sum_{j=1}^{|V|} \exp(z_j / T)}, \quad z_i = W_{\text{head}, i}^\top h$$ The adapted hidden state h ∈ ℝ4096 is projected through the vocabulary head (50,000+ words) to produce logits z, then normalized via Softmax. T = 1.0 for standard greedy decoding. As the adapter trains, the logit z for domain words like "margins" increases, pushing its probability bar upward.
Word Context:
Adapter Training:
Target Correct Word: " margins"

Before fine-tuning, the Base Model guessed generic words (" growth", " revenue"). Adapter training successfully shifts the model's guess to the correct domain term.

Stage 4 of 6: Softmax Word Race
5
Stage 5: Interactive Training Simulation

Watch the AI Learn & Error Drop (Interactive Simulation)

Step 0 / 10
📉 Interactive Simulation: Play the simulation below or drag the Step Scrubber to travel through time and see how the AI's thoughts evolve at each learning step!
➡️ What comes next: The model is now accurate, but it might still write verbose or rambling answers. Stage 6 (Alignment) teaches the AI to prefer concise, high-quality responses using preference optimization (SimPO).
Scrub Step: 0 / 10
📉 AI Confusion Score (Click any step node on the chart to jump!) Current Error: 2.842
📉 Confusion Score
2.842
High Confusion (Untrained)
🎯 Target Word Confidence
4%
Surges to >90% with training
📊 Perplexity (PPL = eLoss)
17.15
Guessing among ~17 words (Untrained)
💾 Memory & Weights Saved
99.6%
Only trains 0.4% adapter params
Stage 5 of 6: Loss & Optimization
6
Stage 6: Post-Training Alignment

Teaching AI Quality Preferences (RLHF vs. DPO vs. SimPO)

🤔 Why do we need alignment after training? After SFT and LoRA training (Stages 1–5), the model is accurate but not necessarily helpful. It may generate verbose, rambling, or repetitive answers because standard cross-entropy loss doesn't penalize wordiness — a 120-word fluff answer and a crisp 15-word answer both get graded the same if they contain the target words. Alignment teaches the model to prefer quality and conciseness.
⚡ SimPO Intuition (Simple Preference Optimization):
• Reference-Free (1 Model in GPU): Unlike DPO and RLHF, SimPO needs no reference model loaded in GPU memory (saves 50% to 75% VRAM!).
• Length-Normalized Reward (1/|y|): Divides log-probabilities by response length, completely eliminating the verbosity/wordiness exploit.
• Target Margin (γ): Enforces that the winning answer beats the losing answer by a strict quality margin.
💾 GPU VRAM Footprint & Model Architecture: 50% Memory Savings vs DPO
⚡ SimPO Architecture: Only 1 model loaded in GPU VRAM (Reference-Free). Fast, lightweight, and memory-efficient!
Preference Battle: Chosen (yw) vs. Rejected (yl) Prompt: "Summarize Q3 EBITDA"
🟢 Candidate A: Winning Answer (yw) 15 tokens (Crisp)
"Q3 EBITDA grew 18% YoY to $42M, driven by 120bps margin expansion in Enterprise SaaS."
Avg Log-Prob: -0.42 / tok SimPO Reward: +1.60
🔴 Candidate B: Rejected Answer (yl) 120 tokens (Verbose)
"EBITDA is an acronym that stands for earnings before interest, taxes, depreciation, and amortization, which is a financial metric that many executives..."
Avg Log-Prob: -1.85 / tok SimPO Reward: -0.74
🎯 Reward Separation Gap (Δr = r_w - r_l): +2.34
Target Margin Threshold (γ = 0.8) ✅ Target Margin Satisfied!
Target Margin (γ) — Quality Gate γ = 0.8 (Strict Gate)
🧠 What it does: Forces the chosen answer's reward to exceed the rejected answer by at least γ. SimPO innovation only. Higher γ = stricter quality gate. Set γ = 0.8 for standard tasks, γ = 1.2–1.5 for high-precision domains.
Beta Temperature (β) — Sensitivity β = 2.0 (Balanced)
⚖️ What it does: Controls how aggressively log-probabilities scale into preference reward gradients.
📊 Side-by-Side Comparison: RLHF vs. DPO vs. SimPO
Method Models in GPU VRAM Reference Model Needed? Length-Bias / Verbosity Immune? Target Margin (γ)? Relative Training Speed
🤖 RLHF (PPO) 4 Models (~96 GB) Yes (Frozen π_ref) ❌ Vulnerable (Exploits length) ❌ No margin 1.0× (Slow 4-step loop)
⚖️ DPO 2 Models (~48 GB) Yes (Frozen π_ref) ❌ Vulnerable (Raw log-prob sum) ❌ No margin 2.0× (Simpler loss)
⚡ SimPO (SOTA) 1 Model (~24 GB) ✅ NO (Reference-Free!) ✅ 100% Immune (1/|y| Normalized) ✅ Enforced (γ Margin) 3.5× Faster (Max throughput)
Stage 6 of 6: SimPO Preference Alignment