Grading the Unannotated: Insights from the WorkRB Skill Extraction Challenge at RecSys-HR 2026
[Sztuczna inteligencja] Autorzy: Warre Veys, Luis Gasco, Jens-Joris Decorte
Skill extraction benchmarks score models against a few human-annotated positives and treat every other skill in the vocabulary as equally wrong. At the scale of ESCO (13,939 skills) this binary view can reward models that rank the annotated positive high for the wrong reason, and hides semantically nonsensical predictions. The WorkRB Challenge at RecSys-HR 2026 replaces it with graded relevance: every candidate skill in a model's ranking window carries a 0 to 4 grade, and the leaderboard scores nDCG@100 against these grades. Since ternary judgements proved too coarse for objective annotation, relevance is decomposed into a cascade of four binary questions (same domain, activity mentioned, right granularity, replacement of a gold positive) that a smaller LLM judge can answer reliably; the labels are validated with blind human and frontier-model re-annotation. This paper documents the ann…
- Kategoria arxiv
- cs.AI
- Kategoria
- Sztuczna inteligencja