The argument in this short paper (6 page PDF) is that "measurement tasks involved in evaluating GenAI systems are highly reminiscent of measurement tasks found throughout the social sciences" and thus "the ML community would benefit from learning from and drawing on the social sciences when developing approaches and instruments for measuring concepts related to the capabilities, impacts, opportunities, and risks of GenAI systems." That doesn't mean "naïvely transferring measurement instruments designed for humans," but rather, adopting a framework based on four levels, "the background concept, the systematized concept, the measurement instrument(s), and the instance-level measurements themselves," as described in the paper.
Today: Total: [Direct link] [Share]
