AlignList Academy
Learn the language used by AI labs and labeling teams. Terms are ranked by relevance so you can focus on what improves real annotation performance first.
Showing 115 terms, sorted by relevance score (10 → 1).
Adjudication
Adjudication resolves conflicting labels into a final canonical decision.
Annotation Guidelines
Annotation guidelines define exactly how to classify data, handle ambiguity, and escalate edge cases.
Gold Set
A gold set is a verified benchmark set used to audit annotator quality.
Inter-Annotator Agreement (IAA)
Inter-Annotator Agreement measures how consistently multiple annotators label the same sample using the same guideline.
Quality Assurance (QA) in Annotation Ops
QA in annotation operations combines audits, review policies, and feedback loops to maintain label quality.
Reinforcement Learning from Human Feedback (RLHF)
RLHF uses human rankings and critiques to teach models preferred behavior.
Supervised Fine-Tuning (SFT)
SFT trains models on high-quality human-curated instruction and response pairs.
Ambiguity Resolution
Ambiguity resolution handles uncertain cases through structured escalation instead of guessing.
Calibration
Calibration aligns annotators on the same guideline interpretation before and during production.
Edge Case
An edge case is a rare but valid sample that stresses normal labeling rules.
Fact-Checking for LLM Evaluation
Fact-checking verifies whether model claims are supported by trusted context or references.
Hallucination
A hallucination is a plausible-looking model claim that is unsupported or false.
Instruction Following Evaluation
Instruction-following evaluation checks whether outputs satisfy explicit constraints from prompts.
Preference Ranking
Preference ranking compares model outputs and selects the better answer using a rubric.
Rubric-Based Evaluation
Rubric-based evaluation scores outputs across clear dimensions such as correctness, safety, and completeness.
Safety Policy Enforcement
Safety policy enforcement labels and evaluates content against harm and misuse policy rules.
Taxonomy and Label Schema
A taxonomy defines classes and rules for assigning labels consistently.
Acceptance Rate
Acceptance rate is the percentage of submitted work approved by review.
Active Learning
Active learning selects uncertain samples for annotation to improve model learning efficiency.
Active Quality Monitoring
Active quality monitoring tracks quality metrics continuously during production.
Bounding Box Annotation
Bounding box annotation draws rectangular boxes around target objects in images.
Class Imbalance
Class imbalance means some labels appear far less often than others.
Code Correctness Evaluation
Code correctness evaluation checks whether generated code satisfies requirements and expected behavior.
Confidence Scoring
Confidence scoring indicates how certain an annotator is about a decision.
Content Moderation Labeling
Content moderation labeling classifies content by policy categories and severity.
Data Validation
Data validation checks labels and metadata against schema and quality constraints before export.
Dataset Versioning
Dataset versioning tracks schema, labels, and policy changes across releases.
Error Analysis
Error analysis clusters failure patterns and identifies root causes.
Groundedness
Groundedness measures whether outputs are supported by provided context.
Hate Speech Taxonomy
A hate speech taxonomy defines classes and scope for protected-target abuse labeling.
Human-in-the-Loop (HITL)
Human-in-the-loop workflows combine model automation with human review and correction.
Instruction Hierarchy Awareness
Instruction hierarchy awareness applies system and policy instructions before user preferences.
Intent Classification
Intent classification labels the underlying user goal in text or voice requests.
Jailbreak Detection
Jailbreak detection identifies prompts intended to bypass model safety constraints.
Math Reasoning Evaluation
Math reasoning evaluation checks intermediate logic and final numeric correctness.
Misinformation Labeling
Misinformation labeling flags unsupported, deceptive, or manipulated claims.
Model Response Ranking Consistency
Ranking consistency measures whether similar response pairs receive similar judgments over time.
Multi-Turn Dialogue Annotation
Multi-turn dialogue annotation labels conversational quality across turns, including coherence and policy compliance.
Multilingual Annotation
Multilingual annotation applies label standards consistently across multiple languages.
Named Entity Recognition (NER)
NER labels spans of text as people, organizations, locations, and other entity types.
Pairwise Ranking
Pairwise ranking compares two candidate outputs and chooses the better one.
PII Redaction
PII redaction finds and masks sensitive personal information.
Policy Violation Severity
Severity scoring measures how serious a policy violation is.
Policy-Compliant Refusal Writing
Policy-compliant refusal writing produces safe refusals that are clear, non-judgmental, and policy-aligned.
Precision and Recall for Labelers
Precision measures correctness of predicted labels; recall measures coverage of true labels.
Privacy-Preserving Annotation
Privacy-preserving annotation minimizes exposure to sensitive data during labeling.
Prompt Engineering
Prompt engineering designs instructions to elicit reliable model behavior.
Prompt Injection Detection
Prompt injection detection identifies attempts to override system behavior or safety constraints.
Refusal Quality
Refusal quality evaluates whether unsafe requests are declined clearly and safely.
Response Safety Grading
Response safety grading scores model outputs across defined safety risk dimensions.
Retrieval Ground Truth Curation
Retrieval ground truth curation builds high-quality relevance judgments for search and RAG evaluation.
Reviewer Consistency
Reviewer consistency measures whether QA reviewers apply standards uniformly.
Reward Model
A reward model predicts human preference signals from ranked examples.
Root Cause Analysis
Root cause analysis identifies the underlying source of repeated quality failures.
Self-Harm Labeling
Self-harm labeling identifies risk-related content and intent levels.
Semantic Search Relevance Labeling
Semantic search relevance labeling scores whether retrieved items satisfy intent and context.
Summarization Evaluation
Summarization evaluation scores summary faithfulness, coverage, and clarity.
Toxicity Annotation
Toxicity annotation labels harmful or abusive language patterns.
Transcription Quality
Transcription quality measures accuracy and formatting consistency in speech-to-text labels.
3D Point Cloud Annotation
3D point cloud annotation labels LiDAR points and objects in spatial scenes.
Adversarial Example Awareness
Adversarial example awareness identifies inputs crafted to trigger model errors.
Audio Event Labeling
Audio event labeling tags sounds such as alarms, music, speech, or environmental noise.
Audit Trail
An audit trail records who changed labels, when, and why.
Benchmark Contamination
Benchmark contamination means evaluation data was seen during training or tuning.
Chain of Verification
Chain of verification validates outputs through structured checks instead of single-pass acceptance.
Citation Quality
Citation quality evaluates whether references are relevant, valid, and correctly used.
Code Review Annotation
Code review annotation labels code quality issues such as bugs, style violations, and security concerns.
Context Window Adherence
Context window adherence checks whether responses use available context without ignoring key evidence.
Conversation Coherence Scoring
Coherence scoring evaluates whether responses remain logically consistent with prior turns.
Coreference Annotation
Coreference annotation connects mentions that refer to the same entity across text.
Data Augmentation
Data augmentation creates modified examples to improve model robustness.
Disagreement Mining
Disagreement mining identifies and analyzes patterns where annotators frequently diverge.
Document Classification
Document classification assigns documents to categories based on content.
Entity Linking
Entity linking maps entity mentions to canonical knowledge base entries.
Error Bucketing
Error bucketing groups failures into standardized categories for analysis.
Escalation Policy
Escalation policy defines when and how uncertain or high-risk items should be routed for review.
Escalation Rationale Writing
Escalation rationale writing documents why a sample was escalated and what evidence supports uncertainty.
Guideline Drift Detection
Guideline drift detection identifies when annotator behavior diverges from current written policy.
Hard Negative Mining
Hard negative mining collects confusing non-target examples that models frequently misclassify.
Harmlessness Score
Harmlessness scoring measures risk reduction in model responses.
Helpfulness Score
Helpfulness scoring measures whether output is useful, clear, and actionably relevant.
Honesty Score
Honesty scoring checks whether the model states uncertainty and avoids fabricated certainty.
Label Leakage
Label leakage occurs when target information unintentionally appears in features or prompt context.
Linguistic Quality Assurance
Linguistic QA audits grammar, style, and semantic integrity in language data.
Locale Sensitivity Labeling
Locale sensitivity labeling evaluates cultural and regional appropriateness of outputs.
Long-Context Evaluation
Long-context evaluation tests whether models use and retain relevant information across large context windows.
Model-Assisted Prelabeling
Model-assisted prelabeling generates initial labels for human correction.
OCR Annotation
OCR annotation labels text regions and transcriptions in images and documents.
Ontology Alignment
Ontology alignment maps concepts across different schemas or taxonomies.
Post-Editing Workflow
Post-editing workflow improves machine-generated outputs through human edits.
Quality-Weighted Sampling
Quality-weighted sampling prioritizes samples based on expected quality impact.
Rejection Sampling
Rejection sampling keeps model outputs that pass quality criteria and discards low-quality outputs.
Relation Extraction Labeling
Relation extraction labeling marks semantic relationships between entities.
Reviewer Feedback Quality
Reviewer feedback quality measures clarity, actionability, and consistency of reviewer comments.
Rubric Drift
Rubric drift occurs when evaluators gradually apply scoring criteria inconsistently over time.
Schema Coverage Analysis
Schema coverage analysis checks whether all classes are sufficiently represented in labeled data.
Schema Migration
Schema migration transitions labeling data from one taxonomy version to another.
Slot Filling Annotation
Slot filling labels parameter values tied to an intent, such as date, location, or product.
Speaker Diarization Labeling
Speaker diarization labeling identifies who spoke when in audio streams.
Task Routing Optimization
Task routing optimization assigns work to annotators based on skill, language, and quality profiles.
Tool Use Evaluation
Tool use evaluation scores how accurately models decide when and how to invoke external tools.
Train-Test Contamination
Train-test contamination happens when overlapping information appears in both training and evaluation sets.
Translation Quality Estimation
Translation quality estimation scores adequacy and fluency of translated outputs.
Uncertainty Sampling
Uncertainty sampling selects instances where model confidence is low for human annotation.
Video Event Annotation
Video event annotation labels actions and events over time in video streams.
Weak Supervision
Weak supervision uses imperfect labeling signals such as heuristics or programmatic rules.
Adjudication Latency
Adjudication latency is the turnaround time to resolve disputed labels.
Annotation Cost per Accepted Label
This metric estimates effective cost after accounting for rejected or reworked labels.
Annotation Throughput
Annotation throughput measures volume completed over time at target quality.
Appeal Workflow
Appeal workflow defines how annotators can contest review outcomes and receive clarifications.
Deduplication
Deduplication removes exact duplicate samples from datasets.
Differential Privacy Awareness
Differential privacy awareness means understanding privacy-preserving techniques that limit individual data exposure.
Frame-Level Classification
Frame-level classification assigns labels to individual video frames.
Near-Duplicate Detection
Near-duplicate detection finds highly similar samples that are not exact matches.
Temporal Consistency Labeling
Temporal consistency labeling checks whether labels remain consistent across time-linked events or frames.