The case for full automation isn't what it used to be

Five years ago there was a credible case for fully-automated geospatial annotation. Detect-everything models were improving fast, labeled training data was easier to come by, and the cost differential between human-in-the-loop and fully-automated workflows was wide enough to matter at scale.

The case for full automation has narrowed since. Three reasons.

First, the easy part of the work has already been automated. Detecting a clearly-visible stop sign in good lighting on a clean image is solved. Most production geospatial extraction pipelines do this part automatically. The remaining work is the hard part — and the hard part is what humans do well.

Second, the cost of being wrong has gone up. Geospatial data feeds infrastructure inventories, regulatory submissions, safety-critical routing, and litigation evidence. A 95% accurate model sounds great until you realize 5% of a million features is fifty thousand wrong records. For some downstream consumers that's a recall, a lawsuit, or a missed inspection that ends in someone getting hurt.

Third, the labor cost differential has narrowed. The annotators who do this work well are skilled GIS professionals, not crowd-task workers. They're not cheap, but they're not the bottleneck either. The bottleneck is the cost of having an automated model produce subtly wrong output that consumers can't audit.

What "human-in-the-loop" actually means in our workflow

Three places humans live in the pipeline.

Up front: schema definition

Before any model runs, a human-with-domain-expertise locks the schema. What classes exist, how they're defined, what the edge cases are, how the output is structured. Models can't do this — it requires judgment about what the downstream consumer needs.

This is invisible work that determines whether the rest of the pipeline produces anything useful. We document the schema in writing, get sign-off from the client, and use it as the reference for every later QA decision.

In the middle: review of pre-labeled output

For most production projects we run an automated pre-labeling pass first. A trained model — sometimes ours, sometimes the client's — produces initial labels on every frame. Humans then review.

Review is not "look at every label one by one." Review is: the model proposed 100,000 labels; let's confirm the ones with confidence above 0.85, spend more time on the 0.55-0.85 band, and re-label the ones below 0.55 from scratch. This is where humans add the most value per minute: they're working on the cases where the model is uncertain, which are usually the cases that actually matter.

At the end: spatial QA against authoritative ground truth

A senior reviewer with GIS expertise runs a final pass that cross-references labels against authoritative spatial layers. Sign labels are checked against road right-of-way polygons. Utility pole labels are checked against utility company GIS inventories where available. Pavement marking labels are checked against direction-of-travel from road network data.

Spatial QA catches a class of errors that pixel-level QA misses entirely — labels that look correct in the image but are geographically impossible. It's the difference between annotation that looks right and annotation that is right.

What humans do better than models in this domain

Four things, consistently.

Ambiguous classes

When a feature could be one of two related classes (R1-1 stop sign vs R1-5 stop-here-on-red, single yellow line vs double yellow), models tend to pick the majority class regardless of context. Humans use surrounding context — what intersection type is this, what's the speed limit, what's the road geometry — to disambiguate.

Novel asset types

When a captured frame contains an asset type the model hasn't seen, the model classifies it as the nearest known class. A human notices it's new, flags it, and we add it to the schema. Models don't surface this automatically — by the time you discover the schema gap, the gap has been silently filled with wrong labels for weeks.

Quality judgments

"Is this pavement marking in good condition or poor condition" is a judgment that requires context the model doesn't have. Lighting changes between frames bias condition scores in fully-automated workflows. Humans normalize for lighting.

Defensibility

When the work is challenged, "the model said so" is not a defense. A human reviewer who signed off on a label is. For regulatory submissions, litigation exhibits, and safety-critical work, we keep humans visibly in the loop precisely because the work needs to defend itself.

What models do better than humans

Three things, also consistently.

Speed at the easy cases. Confident detections on clean imagery in known classes. Consistency at scale (no annotator fatigue). These are real strengths and we use them — pre-labeling pipelines are part of every production project at meaningful volume.

But we use them as acceleration, not as replacement. The model speeds up the boring work so humans can focus on the work that actually requires judgment.

The ratio that works in practice

For most production geospatial annotation projects we run, the pipeline ends up roughly:

  • 60-70% of features confirmed by a human reviewer after model pre-labeling
  • 20-30% re-labeled from scratch by a human (low-confidence model output, edge cases, novel features)
  • 5-10% rejected and removed by spatial QA
  • <1% added by humans that the model missed entirely

That ratio shifts by domain. Densely-captured urban roadway imagery skews toward more model confirmations. Rural utility inventories with rare asset types skew toward more human work. Custom schemas always skew toward human work in the first few weeks until the model is trained on enough domain-specific examples.

What this means for your project

Two takeaways.

If a vendor pitches "fully automated geospatial annotation at human-level accuracy," ask for the confusion matrix on a held-out test set in your domain. The answers we see are almost always 85-92% F1 on aggregate, which means the 8-15% wrong is concentrated in exactly the cases you most need to be right about — the edge cases and rare classes.

If you're building infrastructure-grade geospatial data, plan a human-in-the-loop workflow from the start. The cost premium over fully-automated is small. The cost of having to redo the work because the automated output isn't defensible is large.

Want to see how we run it? Start a 500-sample pilot. You'll see the model pre-labeling output, the human review pass, and the spatial QA pass — all on your data. Two to four business days. Fixed pilot fee.