Nuclei Segmentation on a Shoestring

This started as a simple question: how far can I push a serious computer-vision benchmark when the machine doing the thinking has no GPU at all? My home server is an old dual-socket box with 48 CPU threads and 256 GB of RAM. It cannot train a vision model in any reasonable time. What it can do is run a lot of small language models at once. So I split the work. The CPU box runs a little swarm of local models that brainstorm methods, read papers and sanity-check results, and the Snellius supercomputer runs the actual training jobs. The CPU thinks, the GPU trains.

The task is a PanNuke-style problem from pathology: take an H&E-stained microscopy image, find every individual cell nucleus, and label each one with its type. The genuinely hard part is separating nuclei that touch. It is scored with Panoptic Quality (PQ), which rewards both finding a nucleus and outlining it well. The class-averaged version, mPQ, is what the field competes on, and it sits around 0.50 at the top.

Climbing a ladder, cheapest step first

I went from cheap to expensive on purpose, measuring every rung. A plain U-Net with a watershed step to split instances already reached a binary PQ of 0.52, which was well above the naive 0.38 I had guessed before starting. A no-retrain post-processing tweak (thresholding the distance transform before the watershed) nudged that to 0.5362 by cutting over-segmentation.

Then the swarm earned its keep. In brainstorm mode it proposed adding a second network head that predicts a learned splitting field: high at the centre of each nucleus, near zero at the borders, so the watershed runs on the model's own landscape instead of a naive distance transform. This is the Deep Watershed idea, and here it worked: binary PQ jumped to 0.5709, with both false positives and false negatives falling, because the learned field gives two fused nuclei two separate basins.

To reach the real state of the art you need a real state-of-the-art model, so I reproduced LKCell-L, a large-kernel transformer segmenter with open code and MIT-licensed weights. On my own leakage-free two-fold protocol it scored mPQ 0.4989, which is 0.009 under the paper's reported 0.508. Test-time augmentation and per-class threshold calibration were the standard extra squeeze on top.

The rare class is the whole game

mPQ is a plain average over five classes, so the weakest class drags the number down. Here that is the "Dead" class, which is rare and sits at 0.182 while the strong classes are near 0.58. Lifting Dead from 0.18 to 0.30 on its own would add about 0.024 to mPQ. That is exactly where the remaining headroom is, and also where things went wrong.

The most useful thing I did was throw a number away

An ensemble experiment reported a much better mPQ of 0.5810, with the Dead class suddenly jumping from 0.157 to 0.428. That kind of leap on the hardest class is a red flag, so I audited it. Reading each model checkpoint's config file, rather than trusting its folder name, showed that the second model had been trained on the exact fold I was evaluating on. The number was a training-set score, not a result. Re-run on its real held-out data the model scores 0.5021 and the Dead class collapses back to 0.184. Catching that leak, and deleting my own "record", was the single most valuable moment of the project.

An honest failure

The obvious way to lift the Dead class is to oversample it and paste extra Dead nuclei into training images. Every attempt produced Loss = NaN. The cause took a while to pin down: degenerate tiny masks make the loss non-finite, the mixed-precision scaler skips those weight updates, but BatchNorm keeps updating its running statistics in the forward pass, unprotected, and those quietly go bad. The model showed a healthy training score while its validation score was zero the entire time, a complete mirage. Even the mildest version went NaN, which told me the real issue was that fine-tuning from this particular checkpoint is unstable in my setup. I stopped and kept the honest baseline. A clean, well-understood failure is worth more than a lucky number.

So what is this, honestly?

It is not a new record, and the segmentation core is a reproduction of published work. What I think is worth showing is the rest: a working pipeline that lets a GPU-less machine drive real HPC experiments, a genuine improvement that a local-model swarm proposed, a data leak caught by reading configs instead of trusting file names, and a failure I could explain down to the BatchNorm layer. The whole GPU campaign cost about 1,600 of my 60,000 supercomputer credits. The thinking was free.

Bonus. As a side probe I ran a 744-billion-parameter mixture-of-experts model on the same GPU-less box, streaming its experts off disk. It reasoned correctly, if slowly, at about 0.06 tokens per second, which was a fun reminder that frontier-scale inference does not strictly need a GPU, only patience and disk bandwidth.

---