LoRA Fine-Tuning, Crash-Tested

I wanted to know what local LoRA fine-tuning actually feels like end-to-end on a Mac, not just in theory: pick a model, pick a dataset, watch it train, and see if I could tell the difference afterwards. It turned into a small debugging saga against Apple's Metal allocator, and a more useful result than I expected once I stopped trusting the loss curve and started comparing actual outputs.

The setup

Machine: M1 Max, 64GB unified memory, no dedicated GPU — everything runs through mlx-lm, Apple's own training/inference library for MLX. Model: mlx-community/Qwen3.5-4B-MLX-4bit, a 4-bit quantized 4B-parameter reasoning model, already sitting in the local Hugging Face cache. Dataset: CodeAlpaca-20k, converted into mlx-lm's chat-format jsonl — 18,816 train / 600 validation / 600 test examples of short instruction → code pairs.

Method: LoRA, rank 8, on 16 layers, batch size 2, learning rate 1e-5, seed fixed at 0 throughout so every run was directly comparable. A short calibration run first (20 iterations) to measure real throughput before committing to anything longer: ~60 tokens/sec, ~29.7GB peak memory. Comfortably inside 64GB, or so I thought.

Three crashes

Crash 1: 2% battery. The very first training run died with a cryptic [METAL] Command buffer execution failed: Impacting Interactivity error. A quick check of pmset -g batt explained it: the Mac was on battery power at 2%, "Final" low-battery warning, mid-command-buffer. Plugged in the charger, re-ran the identical command, and it completed cleanly. Lesson zero: fine-tuning on a laptop that isn't plugged in is not a great idea.

Crash 2: an actual reboot. Feeling good about batch-size 2, I kicked off a batch-size 4 comparison run — and the whole machine crashed and rebooted mid-run. No adapter was saved, but nothing important was lost either: the batch-2 calibration and the CodeAlpaca data prep both survived on disk, so the setup work wasn't wasted.

Crash 3: the real one. With the Mac back up, I launched the actual 1,500-iteration run at batch-size 2 — the config that had already worked. It crashed again, with the same Insufficient Memory Metal error, at iteration 30, every single time I reran it, byte-for-byte identical peak memory (29.664GB) each time. Because the seed is fixed, the batches are deterministic, so this wasn't random bad luck — something was reliably running out of memory at the same point in the data.

First guess was competing memory pressure: Docker Desktop was running ~15 containers (a Overleaf/sharelatex instance, a Twenty CRM stack, mongo, several redis/postgres instances) with its Linux VM configured for a 16GB ceiling. I quit Docker entirely and reran — it crashed at the exact same iteration 30 again, ruling that out completely.

The real cause was MLX's own allocator: mlx_lm.lora exposes a --clear-cache-threshold flag that defaults to 0, meaning the Metal buffer cache is never proactively released during training. The "peak mem" the trainer reports is real allocations only, not the cache sitting behind it — so memory can be silently climbing well past what the logs show, until Metal itself refuses to allocate. The fix was two flags: --grad-checkpoint (recompute activations instead of keeping them all resident, dropping peak memory from 29.7GB to 7.9GB immediately) and --clear-cache-threshold 15G (force MLX to release its cache once it exceeds 15GB). That got the run from crashing at iteration 30 to iteration ~1,310 out of 1,500 — still not perfectly stable over very long runs, but with --save-every 300 already checkpointing along the way, the second crash cost nothing: four checkpoints (300, 600, 900, 1200) were already safe on disk, which turned out to matter more than finishing the last 200 iterations.

Does more training actually help?

Rather than trust a single final loss number, I set the run to checkpoint every 300 iterations specifically so I could compare behavior across the run, not just its endpoint. The validation loss itself was not encouraging on its own:

That's ambiguous on its own — it could mean the model plateaued, or it could just mean the 4-batch validation set is too small to be a stable signal. So I ran the same fixed set of five held-out test prompts (temperature 0, for reproducibility) through the un-adapted base model and through each of the four checkpoints. The difference between the base model and checkpoint 300 is immediate and obvious:

The user wants me to generate code to sort an
array in ascending order. The array is
[3, 5, 7, 2, 1].

I should provide a solution that works in a
common programming language. Since no specific
language was mentioned, I'll provide solutions
in a few popular languages like JavaScript,
Python, and maybe a few others.
</think>

Here are code snippets to sort the array
`[3, 5, 7, 2, 1]` in ascending order in several
popular programming languages:

### JavaScript
```javascript
let arr = [3, 5, 7, 2, 1];
arr.sort((a, b) => a - b);
console.log(arr); // Output: [1, 2, 3, 5, 7]
```

### Python
```python
arr = [3, ...  (truncated, still going)

The user wants to sort an array in ascending
order. I'll generate a code snippet to do this.

let arr = [3, 5, 7, 2, 1]

arr.sort((a, b) => a - b);
console.log(arr);
</think>

let arr = [3, 5, 7, 2, 1];
arr.sort((a, b) => a - b);
console.log(arr);

Prompt: "Generate a code snippet to sort the given array in ascending order. let arr = [3, 5, 7, 2, 1]". The base model's answer was still going when I cut it off at 200 tokens — it planned to cover Python, and probably more languages after that.

The pattern holds across the other prompts: the base model reasons at length, then answers with multiple languages, markdown headers and an "Explanation" section — a totally reasonable thing for a general instruct model to do. The fine-tuned checkpoint answers the way CodeAlpaca examples are written: one language, no preamble, no headers. That's the adapter doing exactly what LoRA fine-tuning is supposed to do — nudging output style toward the training distribution — and it's visible after just 300 iterations, roughly 5–6 minutes of training on this machine.

What's more interesting is what didn't change. Comparing checkpoints 300 → 600 → 900 → 1200 on the same five prompts, three of the five answers came back byte-for-byte identical at every checkpoint (the SQL query, the Python countdown loop, and the "how many times does this loop run" reasoning question). The other two — the sort snippet and a pandas dataframe example — only shifted in cosmetic ways (a stray comment line, "Jan" vs "January"). Nine hundred more iterations of training, and the model's answers to fixed test prompts barely moved. That matches the flat, noisy validation loss above: for a small rank-8 LoRA adapter on a fairly homogeneous instruction dataset, almost all of the useful adaptation happened early, and the rest was mostly the model reshuffling around the same behavior.

Takeaways

• MLX's Metal allocator cache can grow well past what the trainer's own "peak mem" log reports. If a long-ish run dies with a Metal Insufficient Memory error even though the numbers look fine, try --grad-checkpoint and --clear-cache-threshold before assuming it's a data or hardware problem.
• --save-every is cheap insurance. The second crash, at iteration 1,310 of 1,500, cost nothing because four checkpoints were already on disk — and it turned out the run had already said everything it was going to say by iteration 300 anyway.
• A loss curve alone was ambiguous here. Generating from fixed prompts at fixed checkpoints and diffing the actual text made the "diminishing returns after ~300 steps" finding concrete instead of a guess.
• Charge the laptop first.

What to try next

Unsloth is the obvious next stop if this moves off a laptop and onto a GPU box. It's built around custom kernels (Triton by default) that its docs claim get roughly 2x faster training and ~70% less VRAM than a standard Hugging Face fine-tune, with even larger gains claimed for MoE fine-tuning, plus much broader method coverage than mlx-lm — QLoRA, full fine-tuning, GRPO/RL, vision and TTS models. Apple Silicon support is real but new: as of the "Unsloth Studio" launch this year, native MLX training was still listed as "coming very soon" rather than shipped, since the kernels are Triton-based and Triton doesn't run on Macs. A community project, mlx-tune, already offers an Unsloth-compatible API natively on MLX in the meantime — worth a look before Unsloth's own Mac story lands.

nanochat (Karpathy's "best ChatGPT $100 can buy") is the opposite kind of project: an ~8,300-line, MIT-licensed pipeline that trains a chat model completely from scratch — tokenizer, pretraining, midtraining, SFT, RL, and a CLI to talk to the result. It's genuinely worth reading as a from-first-principles tour of how a ChatGPT-style model gets built. It is not, despite the framing, a lightweight local option: the flagship "speedrun" needs 8×H100s for ~3–4 hours (~$100), and a homelab attempt on two DGX Spark desktops (128GB unified memory each, Blackwell-class, network-linked) still took ~73 hours and hit memory-exhaustion issues that real datacenter GPUs don't. There is a documented CPU/MPS path (runs/runcpu.sh), but the README says so itself: it shrinks the model down to fit "a few ten minutes" and "you will not get strong results in this way." Which is a good way to frame the contrast with this weekend's project: LoRA only nudges a small adapter on top of a model that's already pretrained, which is why it fits in a laptop's unified memory in a couple of hours, while nanochat's job — building the pretrained model itself — is a fundamentally bigger amount of compute, toy CPU demo aside.

Adapters, eval prompts and full training logs from this run are on disk if I want to revisit them; next experiment is probably a larger, more diverse dataset to see if the same "plateaus fast" pattern holds, or if CodeAlpaca specifically is just narrow enough to saturate a rank-8 adapter quickly.