BOT-TRICKS // Abliterate Your First Small Model for n00bs

00 READ ME FIRST

A step-by-step walkthrough that takes you from "I run local models sometimes" to "I have abliterated my first small open-source model, evaluated it on both axes, and I understand what every step did." Smart but new; no ML background needed beyond the companion guide's five facts.

A tool for making production attack models. An abliterated model will answer harmful requests — that's the technique, and the risk. Only open-weight models you run yourself; use it on models you own, on hardware you control, for research and testing, in accordance with the law. If you're doing this to attack other people's systems, close the tab and go touch grass.

An abliteration is judged on TWO numbers together — refusal rate on held-out harmful prompts AND KL divergence from the original on harmless prompts. Refusal rate alone hides capability damage. Any tutorial (or model card) that only reports how much it "uncensored" something is selling you half the truth. You will measure both before you call anything done.

01 PICK YOUR FIRST MODEL

Start small. A 0.5-2B instruct model teaches you everything a 70B would, at a tenth of the cost and a hundredth of the heartache. Good first targets, in order of gentleness:

Qwen2.5-0.5B-Instruct — tiny, well-behaved, refuses reliably on its safety categories. The training-wheels pick: everything runs on a laptop.
Qwen2.5-1.5B-Instruct — same family, a bit more capability, still laptop-friendly.
Qwen3-0.6B / Qwen3-1.7B — the current-generation pick, with one big trap: thinking mode (warn box below).

What works in general: dense, Llama-style instruct models (Llama, Qwen, Gemma, Mistral families) — that covers almost everything you'd run locally. MoE and multimodal are supported by the newer tools; exotic architectures are not.

Qwen3 models are "hybrid thinking" — they can emit a <think>...</think> reasoning block before answering. If you evaluate with short generation budgets, the reply gets truncated inside the thinking block: every output reads as reasoning text, refusal classifiers score 0% (false "answered!"), and your numbers are garbage. mlabonne hit this on his Qwen3 abliterations — "the reasoning mode interfered with non-reasoning refusals" — and the fix is to disable thinking everywhere with tokenizer.apply_chat_template(..., enable_thinking=False), and record that you did. Want a quieter first run? Pick a Qwen2.5.

02 WHAT YOU NEED

Hardware (rough, fp32 math — the technique wants 32-bit floats):

Model size	Measurement + editing	Notes
0.5B	~4 GB RAM	any laptop, CPU is fine, just slow
1.5-2B	~8-16 GB RAM	desktop RAM or a 16GB+ GPU
4B	~16-24 GB VRAM/RAM	the classic "3090 class" target
8B+	a real GPU	or 4/8-bit loading for measurement

You need the model in memory roughly twice at some point — a pristine copy for comparison plus the working copy. "CPU-only" is not a disqualifier for 1-4B: it works, it's deterministic with greedy decoding, it's just slow. Patience is a component.

Software: Python 3.10+, pip install -U transformers torch, and a few GB of disk for the model download. That's the whole stack for the manual path; the easy path adds one package.

Keep a pristine copy of the original model somewhere safe — never edit in place. You will compare against it constantly; every serious workflow caches "original model logits on my evaluation prompts" once and reuses them for every candidate.

03 CHOOSE YOUR PATH

PATH A — Heretic, the easy way. Two commands, automated search, ~30 min on a modest GPU for a small model, co-minimizes refusals and KL for you. Best quality-per-effort. Do this if you want a working model today.
PATH B — the manual way. You run each stage yourself: measure, difference, test, edit, verify, evaluate. Slower, fiddlier, and the only way you actually learn where the traps live. Do this once in your life even if you use Path A afterwards.

They're not exclusive, and the smart order for a n00b is: run Path A for the win, then walk Path B on the same model to understand what Heretic just did for you.

04 PATH A: HERETIC

$ Install

pip install -U heretic-llm

$ Run it on your model

heretic Qwen/Qwen2.5-1.5B-Instruct

That's the whole interface. It loads the model, runs the search, prints a report, and when it finishes offers to save, upload to HF, chat with the result, or benchmark it. For a small model on a GPU think minutes-to-tens-of-minutes; on CPU, go make coffee. Small-VRAM? quantization = bnb_4bit loads bigger models for measurement.

$ Read the report like a skeptic

It reports refusals AND KL — check both. A run with 2% refusals and KL 0.5 is a worse model than 5% refusals and KL 0.05 for most purposes. Then do the thing no automated tool does for you: spot-read full generations (§7).

$ Save and use it

Save from Heretic's prompt (or upload), then treat it as a normal model — quantize to GGUF, drop it in Ollama/LM Studio, whatever your pipeline does (§8).

05 PATH B: MANUAL

The full pipeline, stage by stage — mlabonne's walkthrough as systematized by Fahlgren with grimjim's refinements. The NousResearch/llm-abliteration repo gives you script-shaped versions of steps 2-6 (measure.py, analyze.py, sharded_ablate.py with --normpreserve for MPOA, chat.py) if you want plumbing help. Doing it once with your own hands teaches you more than running any tool ten times.

Get your datasets

Two prompt sets: harmful (things the model refuses) and harmless (things it answers). Labonne's are on HF: mlabonne/harmful_behaviors and mlabonne/harmless_alpaca. ~128 prompts per class is plenty — more burns memory for marginal gain.

Filter to behavior, not label

Run every prompt through the model once and keep only the ones that behave as labeled: harmful prompts the model refuses, harmless prompts it answers. Fahlgren kept 517/520 harmful and 591/600 harmless. A "harmful" prompt the model happily answers adds noise to your mean. Also hold out the last N of each set for evaluation — never measure and evaluate on the same prompts.

Capture activations

Run both filtered sets through the model with output_hidden_states=True, formatted with the chat template (apply_chat_template(..., add_generation_prompt=True)), and save the residual-stream vector at the last token of the formatted prompt — the moment right before the model writes its first reply token, where the refuse-or-comply decision lives. Do the math in fp32 even if the model ships in bf16 — outlier activations derail the mean difference in 16-bit. On Qwen3, template with enable_thinking=False. Extreme activation magnitudes (Gemma-family)? Optionally zero the top ~0.5% of magnitudes per layer before averaging.

Compute the mean-difference direction at every layer

refusal_dir[l] = mean(harmful_l) − mean(harmless_l) → normalize

No training, no gradients. Rank layers by a quality metric — grimjim's composite is SNR × (1 − cosine similarity of the two means); good sources are mid-to-late-mid layers.

Test candidates with inference-time steering

For each top candidate, hook the model so that at every generated token you subtract the activation's projection onto the direction. Then Fahlgren's three tests, all three required:

Remove — harmful prompts stop being refused? (lower = better)
Add — direction added to harmless prompts makes the model refuse them? (proves it's really the refusal direction)
KL — harmless answers barely change? (stay near 0.1 or below)

Pick the layer — the traps live here

Do not trust the deepest layers just because they score best — on paper they always look great; in full replies the model finds new ways to say no in different words. Fahlgren's 32-layer data: only layer 11 — early-middle — passed all three tests (KL 0.06); the late band that looked best on numbers failed the spot-read. Read the full replies of every candidate, not just classifier scores.

Apply the edit

Take your chosen direction (better: per-layer directions over a band of layers — single-layer edits get ~70% self-repaired by the rest of the network, the hydra effect) and orthogonalize the writer matrices: attention o_proj, MLP down_proj per target layer, plus the embedding matrix once.

W' = W − r̂ (r̂ᵖ W)

Two details that bite: orientation — o_proj and down_proj are [d_model, d_in] with the residual stream in their ROW space, while the embedding is [vocab, d_model] with the stream in its COLUMN space; same projection, transposed, and the wrong one is a silent no-op at worst. And off-by-one layer indexing — hidden-states position l sits between decoder layers l−1 and l, so a mis-indexed edit lands on the wrong layer and everything downstream still runs green.

Once this works, redo it with grimjim's upgrades — project the refusal direction against the harmless mean first (biprojection), then try norm-preserving MPOA — and compare the KL. The variant math is in the companion guide, §5.

Verify the edit landed

Diff the edited weights against the pristine copy: max|dW| > 0 on exactly the intended layers, ~0 everywhere else. Cheapest insurance in the whole pipeline against a silent no-op. For orthogonalization, r̂ᵖW' should be ~0 on edited matrices; for MPOA it's small-but-not-zero by design.

06 SAVE YOUR MODEL

model.save_pretrained("my-abliteration")
tokenizer.save_pretrained("my-abliteration")

From here it's a normal HF-format model. Optionally upload it — and label it honestly: a model card stating the base checkpoint, the method, and both eval numbers.

07 EVALUATE BOTH AXES

Refusal rate on the held-out harmful tail. A cheap classifier works: refusals almost always open with "I" (99.4% of Fahlgren's harmful replies) — "starts with I / contains 'I cannot' / 'Sorry'..." catches nearly everything at zero cost. Build one; don't eyeball 128 outputs and call it 3%.
KL divergence on harmless prompts vs. the pristine copy — how much the model drifted on things it used to do fine. Cache the pristine logits once and reuse them per candidate.
Spot-read full replies. The classifier sees first words; covert noncompliance — the model answers in ways that quietly don't do what was asked — survives first-word metrics, especially in Llama bases. Also watch for refusal style-shift: one refusal circuit removed and the model reroutes into different refusal vocabulary. Read a sample of full generations every run.

Calibration points from the literature: Fahlgren's manual run took refusals 100% → ~1% at KL 0.058; Heretic's automated gemma-3-12b hit 3/100 refusals at KL 0.16 (vs. 1.04 for an earlier manual abliteration of the same model).

Evaluate with thinking disabled and short budgets double-checked, or the truncation trap from §1 poisons every number.

08 ACTUALLY USE IT

From HF format, the usual routes: quantize to GGUF (llama.cpp's convert_hf_to_gguf.py, then your favorite quant), ollama create with a Modelfile pointing at the GGUF, or serve it with vLLM like any other model.

Keep the pristine original around — every interesting question about your edit ("did this hurt its coding?") is answered by diffing against it, and the abliterated copy is a research artifact, not a replacement daily driver.

09 TRAPS RECAP

Deep-layer trap — best-on-paper layers refuse in new words; mid layers win.
Hydra effect — one-layer edits get ~70% self-repaired; edit a band.
bf16 math — fp32 intermediates for measuring and editing, always.
Silent no-ops — wrong embedding orientation, off-by-one layers; diff against pristine and check where dW landed.
Thinking-mode evals (Qwen3) — enable_thinking=False or your metrics lie to you.
Covert noncompliance — spot-read full generations.
Refusal style-shift — count the styles, not just the count.
One number is zero numbers — refusal without KL is marketing.

10 SOURCES

The Complete n00b's Guide to Abliteration — the companion overview this walkthrough assumes.