Gold Standards, Uneven Ground

Rethinking Evaluation for Low-Resource Language Technology

Abstract

Evaluation sets are frequently called gold standards even when annotation conditions, speaker coverage, and domain assumptions differ sharply across languages. This paper proposes a transparent reporting framework for low-resource evaluation and demonstrates how uncertainty-aware interpretation changes comparisons between systems.