Measuring progress in robotics: Benchmarking and the ‘measure-target confusion’

In Fabio Bonsignorio, John Hallam, Elena Messina & Angel P. Del Pobil (eds.), Metrics of sensory motor coordination and integration in robots and animals. Springer. pp. 169-179 (2019)
  Copy   BIBTEX

Abstract

While it is often said that robotics should aspire to reproducible and measurable results that allow benchmarking, I argue that a focus on benchmarking can be a hindrance for progress in robotics. The reason is what I call the ‘measure-target confusion’, the confusion between a measure of progress and the target of progress. Progress on a benchmark (the measure) is not identical to scientific or technological progress (the target). In the past, several academic disciplines have been led into pursuing only reproducible and measurable ‘scientific’ results – robotics should be careful to follow that line because results that can be benchmarked must be specific and context-dependent, but robotics targets whole complex systems for a broad variety of contexts. While it is extremely valuable to improve benchmarks to reduce the distance be- tween measure and target, the general problem to measure progress towards more intelligent machines (the target) will not be solved by benchmarks alone; we need a balanced approach with sophisticated benchmarks, plus real-life testing, plus qualitative judgment.

Author's Profile

Vincent C. Müller
Universität Erlangen-Nürnberg

Analytics

Added to PP
2018-06-09

Downloads
645 (#33,373)

6 months
119 (#41,744)

Historical graph of downloads since first upload
This graph includes both downloads from PhilArchive and clicks on external links on PhilPapers.
How can I increase my downloads?