Back to Academy
Relevance 7/10Operations and WorkflowIntermediate6 min read
Uncertainty Sampling
Uncertainty sampling selects instances where model confidence is low for human annotation.
Why it matters for annotators
It maximizes information gain from human effort.
Visual mental model
Low-confidence examples -> annotation -> retrain loop.
Examples (bad vs good)
Scenario: Real annotation scenario involving Uncertainty Sampling
Bad: Labeling quickly without applying project rubric.
Good: Applying rubric criteria, documenting rationale, and escalating uncertainty.
Common mistakes
- Skipping guideline details for edge cases.
- Applying inconsistent criteria across similar samples.
- Avoiding escalation even when uncertain.
Submission checklist
- Read the latest guideline update before each batch.
- Apply rubric dimensions explicitly in each decision.
- Escalate ambiguous items with concise rationale.