[论文解读] Response to Moffat's Comment on "Towards Meaningful Statements in IR Evaluation: Mapping Evaluation Measures to Interval Scales"

Marco Ferrante, Nicola Ferro+1 more|arXiv (Cornell University)|Dec 22, 2022

Advanced Text Analysis Techniques被引用 24

一句话总结

本文回应 Moffat 的批评，阐明测量的表征理论、意义性以及对 IR 评估度量的区间化方法。

ABSTRACT

Moffat recently commented on our previous work. Our work focused on how laying the foundations of our evaluation methodology into the theory of measurement can improve our knowledge and understanding of the evaluation measures we use in IR and how it can shed light on the different types of scales adopted by our evaluation measures; we also provided evidence, through extensive experimentation, on the impact of the different types of scales on the statistical analyses, as well as on the impact of departing from their assumptions. Moreover, we investigated, for the first time in IR, the concept of meaningfulness, i.e. the invariance of the experimental statements and inferences you draw, and proposed it as a way to ensure more valid and generalizabile results. Moffat's comments build on: (i) misconceptions about the representational theory of measurement, such as what an interval scale actually is and what axioms it has to comply with; (ii) they totally miss the central concept of meaningfulness. Therefore, we reply to Moffat's comments by properly framing them in the representational theory of measurement and in the concept of meaningfulness. All in all, we can only reiterate what we said several times: the goal of this research line is to theoretically ground our evaluation methodology - and IR is a field where it is extremely challenging to perform any theoretical advances - in order to aim for more robust and generalizable inferences - something we currently lack in the field. Possibly there are other and better ways to achieve this objective and these proposals could emerge from an open discussion in the field and from the work of others. On the other hand, reducing everything to a contrast on what is (or pretend to be) an interval scale or whether all or none evaluation measures are interval scales may be more a barrier from than a help in progressing towards this goal.

研究动机与目标

• 澄清关于在 IR 评估中的测量表征理论的误解。
• 论证意义性作为在允许的尺度变换下的不变性的重要作用。
• 捍卫所提出的区间化方法，以在 IR 指标中保留用户视角。
• 讨论测量公理（差异结构）与 IR 中的区间尺度之间的联系。

提出的方法

• 回顾测量和尺度类型的基础概念（名义、序数、区间、比值）。
• 将可解性（等距刻度）解释为区间尺度的关键公理。
• 将意义性定义为在允许的变换下陈述的不变性。
• 呈现区间化作为将排序转换为区间尺度的过程，同时保留底层排序。
• 提供关于区间化如何影响跨数据集的统计分析的实证评估。

实验结果

研究问题

• RQ1 在测量的表征理论下，IR 评估度量的有效区间尺度应具备哪些条件？
• RQ2 IR 评估度量是否能在区间尺度下获得有意义的解释？对统计分析有何后果？
• RQ3 区间化在实现有意义推断的同时，是否能保留用户视点？
• RQ4 偏离区间尺度假设对 IR 评估与推断有何影响？

主要发现

• 区间尺度需要等距的刻度与仿射的允许变换；并非所有 IR 指标都满足这一点。
• 意义性关乎在允许的变换下陈述的不变性，而非主观可解释性。
• 区间化在保持度量给出的排序的同时，使区间尺度分析和检验成为可能。
• 作者提供广泛的实证研究，展示尺度假设对标准 IR 任务中的统计分析的影响。
• 回应重申，其目标是为 IR 的健壮且可推广的推断提供理论依据，而不是把所有度量强行放入区间尺度。