AI Training Glossary

The language used by AI labs and labeling teams, defined in one place. 115 terms covering annotation quality, prompting, evaluation, safety and the day to day workflow.

Human data startups and teams, partner with us

115 terms

Audio and Speech

Audio Event Labeling: Audio event labeling tags sounds such as alarms, music, speech, or environmental noise.
Speaker Diarization Labeling: Speaker diarization labeling identifies who spoke when in audio streams.
Transcription Quality: Transcription quality measures accuracy and formatting consistency in speech-to-text labels.

Computer Vision

3D Point Cloud Annotation: 3D point cloud annotation labels LiDAR points and objects in spatial scenes.
Bounding Box Annotation: Bounding box annotation draws rectangular boxes around target objects in images.
Frame-Level Classification: Frame-level classification assigns labels to individual video frames.
OCR Annotation: OCR annotation labels text regions and transcriptions in images and documents.
Video Event Annotation: Video event annotation labels actions and events over time in video streams.

Data and Metrics

Benchmark Contamination: Benchmark contamination means evaluation data was seen during training or tuning.
Class Imbalance: Class imbalance means some labels appear far less often than others.
Confidence Scoring: Confidence scoring indicates how certain an annotator is about a decision.
Error Analysis: Error analysis clusters failure patterns and identifies root causes.
Error Bucketing: Error bucketing groups failures into standardized categories for analysis.
Hard Negative Mining: Hard negative mining collects confusing non-target examples that models frequently misclassify.
Label Leakage: Label leakage occurs when target information unintentionally appears in features or prompt context.
Precision and Recall for Labelers: Precision measures correctness of predicted labels; recall measures coverage of true labels.
Retrieval Ground Truth Curation: Retrieval ground truth curation builds high-quality relevance judgments for search and RAG evaluation.
Root Cause Analysis: Root cause analysis identifies the underlying source of repeated quality failures.
Schema Coverage Analysis: Schema coverage analysis checks whether all classes are sufficiently represented in labeled data.
Temporal Consistency Labeling: Temporal consistency labeling checks whether labels remain consistent across time-linked events or frames.
Train-Test Contamination: Train-test contamination happens when overlapping information appears in both training and evaluation sets.

Operations and Workflow

Active Learning: Active learning selects uncertain samples for annotation to improve model learning efficiency.
Adjudication Latency: Adjudication latency is the turnaround time to resolve disputed labels.
Annotation Cost per Accepted Label: This metric estimates effective cost after accounting for rejected or reworked labels.
Annotation Guidelines: Annotation guidelines define exactly how to classify data, handle ambiguity, and escalate edge cases.
Annotation Throughput: Annotation throughput measures volume completed over time at target quality.
Appeal Workflow: Appeal workflow defines how annotators can contest review outcomes and receive clarifications.
Audit Trail: An audit trail records who changed labels, when, and why.
Data Validation: Data validation checks labels and metadata against schema and quality constraints before export.
Dataset Versioning: Dataset versioning tracks schema, labels, and policy changes across releases.
Deduplication: Deduplication removes exact duplicate samples from datasets.
Escalation Policy: Escalation policy defines when and how uncertain or high-risk items should be routed for review.
Escalation Rationale Writing: Escalation rationale writing documents why a sample was escalated and what evidence supports uncertainty.
Human-in-the-Loop (HITL): Human-in-the-loop workflows combine model automation with human review and correction.
Model-Assisted Prelabeling: Model-assisted prelabeling generates initial labels for human correction.
Near-Duplicate Detection: Near-duplicate detection finds highly similar samples that are not exact matches.
Ontology Alignment: Ontology alignment maps concepts across different schemas or taxonomies.
Post-Editing Workflow: Post-editing workflow improves machine-generated outputs through human edits.
Quality-Weighted Sampling: Quality-weighted sampling prioritizes samples based on expected quality impact.
Schema Migration: Schema migration transitions labeling data from one taxonomy version to another.
Task Routing Optimization: Task routing optimization assigns work to annotators based on skill, language, and quality profiles.
Taxonomy and Label Schema: A taxonomy defines classes and rules for assigning labels consistently.
Uncertainty Sampling: Uncertainty sampling selects instances where model confidence is low for human annotation.

Prompting and Evaluation

Chain of Verification: Chain of verification validates outputs through structured checks instead of single-pass acceptance.
Citation Quality: Citation quality evaluates whether references are relevant, valid, and correctly used.
Code Correctness Evaluation: Code correctness evaluation checks whether generated code satisfies requirements and expected behavior.
Code Review Annotation: Code review annotation labels code quality issues such as bugs, style violations, and security concerns.
Context Window Adherence: Context window adherence checks whether responses use available context without ignoring key evidence.
Conversation Coherence Scoring: Coherence scoring evaluates whether responses remain logically consistent with prior turns.
Fact-Checking for LLM Evaluation: Fact-checking verifies whether model claims are supported by trusted context or references.
Groundedness: Groundedness measures whether outputs are supported by provided context.
Hallucination: A hallucination is a plausible-looking model claim that is unsupported or false.
Helpfulness Score: Helpfulness scoring measures whether output is useful, clear, and actionably relevant.
Honesty Score: Honesty scoring checks whether the model states uncertainty and avoids fabricated certainty.
Instruction Following Evaluation: Instruction-following evaluation checks whether outputs satisfy explicit constraints from prompts.
Instruction Hierarchy Awareness: Instruction hierarchy awareness applies system and policy instructions before user preferences.
Long-Context Evaluation: Long-context evaluation tests whether models use and retain relevant information across large context windows.
Math Reasoning Evaluation: Math reasoning evaluation checks intermediate logic and final numeric correctness.
Model Response Ranking Consistency: Ranking consistency measures whether similar response pairs receive similar judgments over time.
Pairwise Ranking: Pairwise ranking compares two candidate outputs and chooses the better one.
Policy-Compliant Refusal Writing: Policy-compliant refusal writing produces safe refusals that are clear, non-judgmental, and policy-aligned.
Preference Ranking: Preference ranking compares model outputs and selects the better answer using a rubric.
Prompt Engineering: Prompt engineering designs instructions to elicit reliable model behavior.
Refusal Quality: Refusal quality evaluates whether unsafe requests are declined clearly and safely.
Rubric-Based Evaluation: Rubric-based evaluation scores outputs across clear dimensions such as correctness, safety, and completeness.
Semantic Search Relevance Labeling: Semantic search relevance labeling scores whether retrieved items satisfy intent and context.
Summarization Evaluation: Summarization evaluation scores summary faithfulness, coverage, and clarity.
Tool Use Evaluation: Tool use evaluation scores how accurately models decide when and how to invoke external tools.

Quality and QA

Acceptance Rate: Acceptance rate is the percentage of submitted work approved by review.
Active Quality Monitoring: Active quality monitoring tracks quality metrics continuously during production.
Adjudication: Adjudication resolves conflicting labels into a final canonical decision.
Ambiguity Resolution: Ambiguity resolution handles uncertain cases through structured escalation instead of guessing.
Calibration: Calibration aligns annotators on the same guideline interpretation before and during production.
Disagreement Mining: Disagreement mining identifies and analyzes patterns where annotators frequently diverge.
Edge Case: An edge case is a rare but valid sample that stresses normal labeling rules.
Gold Set: A gold set is a verified benchmark set used to audit annotator quality.
Guideline Drift Detection: Guideline drift detection identifies when annotator behavior diverges from current written policy.
Inter-Annotator Agreement (IAA): Inter-Annotator Agreement measures how consistently multiple annotators label the same sample using the same guideline.
Quality Assurance (QA) in Annotation Ops: QA in annotation operations combines audits, review policies, and feedback loops to maintain label quality.
Reviewer Consistency: Reviewer consistency measures whether QA reviewers apply standards uniformly.
Reviewer Feedback Quality: Reviewer feedback quality measures clarity, actionability, and consistency of reviewer comments.
Rubric Drift: Rubric drift occurs when evaluators gradually apply scoring criteria inconsistently over time.

Safety and Policy

Adversarial Example Awareness: Adversarial example awareness identifies inputs crafted to trigger model errors.
Content Moderation Labeling: Content moderation labeling classifies content by policy categories and severity.
Differential Privacy Awareness: Differential privacy awareness means understanding privacy-preserving techniques that limit individual data exposure.
Harmlessness Score: Harmlessness scoring measures risk reduction in model responses.
Hate Speech Taxonomy: A hate speech taxonomy defines classes and scope for protected-target abuse labeling.
Jailbreak Detection: Jailbreak detection identifies prompts intended to bypass model safety constraints.
Misinformation Labeling: Misinformation labeling flags unsupported, deceptive, or manipulated claims.
PII Redaction: PII redaction finds and masks sensitive personal information.
Policy Violation Severity: Severity scoring measures how serious a policy violation is.
Privacy-Preserving Annotation: Privacy-preserving annotation minimizes exposure to sensitive data during labeling.
Prompt Injection Detection: Prompt injection detection identifies attempts to override system behavior or safety constraints.
Response Safety Grading: Response safety grading scores model outputs across defined safety risk dimensions.
Safety Policy Enforcement: Safety policy enforcement labels and evaluates content against harm and misuse policy rules.
Self-Harm Labeling: Self-harm labeling identifies risk-related content and intent levels.
Toxicity Annotation: Toxicity annotation labels harmful or abusive language patterns.

Text and NLP

Coreference Annotation: Coreference annotation connects mentions that refer to the same entity across text.
Document Classification: Document classification assigns documents to categories based on content.
Entity Linking: Entity linking maps entity mentions to canonical knowledge base entries.
Intent Classification: Intent classification labels the underlying user goal in text or voice requests.
Linguistic Quality Assurance: Linguistic QA audits grammar, style, and semantic integrity in language data.
Locale Sensitivity Labeling: Locale sensitivity labeling evaluates cultural and regional appropriateness of outputs.
Multi-Turn Dialogue Annotation: Multi-turn dialogue annotation labels conversational quality across turns, including coherence and policy compliance.
Multilingual Annotation: Multilingual annotation applies label standards consistently across multiple languages.
Named Entity Recognition (NER): NER labels spans of text as people, organizations, locations, and other entity types.
Relation Extraction Labeling: Relation extraction labeling marks semantic relationships between entities.
Slot Filling Annotation: Slot filling labels parameter values tied to an intent, such as date, location, or product.
Translation Quality Estimation: Translation quality estimation scores adequacy and fluency of translated outputs.

Training Paradigms

Data Augmentation: Data augmentation creates modified examples to improve model robustness.
Reinforcement Learning from Human Feedback (RLHF): RLHF uses human rankings and critiques to teach models preferred behavior.
Rejection Sampling: Rejection sampling keeps model outputs that pass quality criteria and discards low-quality outputs.
Reward Model: A reward model predicts human preference signals from ranked examples.
Supervised Fine-Tuning (SFT): SFT trains models on high-quality human-curated instruction and response pairs.
Weak Supervision: Weak supervision uses imperfect labeling signals such as heuristics or programmatic rules.