DetCls and SegCls: Two-Stage Pipelines for Class Mistakes

Detection models can localize an object perfectly and still write the wrong class on it. On most datasets this isn’t an issue; on a few it stays a problem no matter how much data goes into the training set. When that happens, classification is the part that’s failing, and a bigger detector won’t help.

The fix is a second model that only does classification, fed by the boxes the first one found. We implemented this on our platform as a chained detection + classification pipeline (and the segmentation equivalent), so the detector finds the object and the classifier names it.

What the Pipelines Do

DetCls chains a detection model and a classification model. The detector finds objects and produces boxes; each box is cropped and sent to the classifier; the classifier’s labels replace the detector’s labels in the output.

SegCls is the same idea with a segmentation backbone — pixel-accurate masks, then a per-instance classifier on each cropped mask region.

Both pipelines emit the same result format as their single-stage equivalents, so any postprocessing already in place keeps working unchanged.

A Real Case: Stamped Characters on Meat Cuts

A meat processor stamps traceability characters on every cut — codes like 3, 8, V, K. The cut moves under a top-down camera; the system has to read the character and pass it on, and the model regularly confused some of these characters between each other.

Single-stage detection on this dataset failed in two consistent ways.

Right box, wrong label. The detector localized each stamp perfectly but emitted the wrong character often enough to break traceability.

Two overlapping boxes, different classes. The same stamp fired twice, once with one class and once with another. Standard NMS didn’t merge them because they belonged to different classes, so both boxes survived into the output.

How DetCls Fixes It

Train the detector to find a single class — call it stamp. Now stage 1 finds the stamp, and stage 2 reads what’s on it. Localization stops competing with classification for backbone capacity, and the classifier sees a tight, axis-aligned crop at its native input resolution rather than a small patch of feature map.

After the classifier runs, the pipeline performs a class-scoped IoU pass. Two boxes that both end up labeled 8 collapse into one. Two boxes labeled 8 and 0 stay separate, because they are most likely two real, neighboring stamps.

Why Two Stages Help

Task interference

On the meat dataset the bottleneck wasn’t capacity, but that the same backbone had to learn features that help draw a tight box and features that tell a 3 from an 8. Those two jobs don’t always agree, and can pull the network in conflicting directions. As we labeled more data the boxes kept tightening, but the character confusions stuck.

Input resolution

A detection head reads each candidate through a small portion of feature map. A classifier sees the cropped object filling its full input. On targets as small as a stamped character, that’s the difference between “round shape” and reading the digit.

How Training and Inference Work

The platform auto-splits the dataset into two sub-datasets when you train a detcls or segcls model:

• A detector dataset with the original geometry, labeled either with a single unified class or with the original per-class labels (depending on the chosen strategy).
• A classifier dataset of cropped ground-truth instances, plus optional negative crops drawn from areas with no annotations to teach the classifier an unknown class.

Each model trains independently. There is no end-to-end backprop through the pipeline — that’s deliberate. Stage 1 and stage 2 can be retrained, swapped, or upgraded in isolation, and a debug failure tells you immediately which model is to blame.

At inference, the pipeline runs five steps per image:

1. Detect.
2. Crop each target box.
3. Classify the full batch of crops in a single call.
4. Merge: the classifier label replaces the detector label, and the two confidences combine into a single number — the minimum, the product, or whichever side you trust most.
5. Class-scoped NMS removes duplicates that share a label after step 4.

A Note on Class Strategy

Don’t unify visually distinct classes. If the detector is asked to find bottle labels, caps, and bodies all under the same umbrella, the detection head has to learn an extremely broad notion of “object,” and recall drops everywhere. Unify only what shares enough visual structure that a single detector class is a fair description, such as in the meat example.

When to Reach for It

What’s failing	Use
Boxes are right, labels swap between classes the model confuses	DetCls or SegCls
Multiple boxes / masks for a single object	DetCls or SegCls
Boxes themselves are wrong or the detector misses objects	Not this — fix detection first

A pipeline is a tool for a specific failure mode. It cannot rescue a model that fails to localize.

Pitfalls

• Unifying visually different classes. Stage 1 starts hallucinating, recall collapses, and no amount of stage-2 cleverness recovers it.
• Routing every box through the classifier when only one class is confused. You pay double the latency and introduce a second source of error for boxes that were already correct.
• Expecting stage 2 to recover missed detections. If the detector never saw the object, the classifier has nothing to relabel. DetCls fixes labels on found boxes; it does not improve recall.
• Training the classifier without negative crops. Without an unknown class, the classifier has to assign every false-positive crop to one of the real classes, and stage 1’s mistakes leak through unfiltered.
• Skipping the class-scoped NMS step. Without it, the duplicate boxes the classifier just collapsed to the same label stay duplicated in the final output.

Summary

Reach for DetCls or SegCls when detector boxes are right and the labels slip. The capacity and resolution arguments make the two-stage version strictly more capable on hard classification tasks, at the cost of one extra model to train and a few extra milliseconds per frame. Otherwise, stick with single-stage — adding a second model is paying for a problem you don’t have.