Measurement Theory #1 - “What Gets Measured, Gets Managed”
I first heard the phrase in a conversation with a Data Scientist, and the concept resonated with me (the cliche that it has become, I had no idea until I started digging). I am very interested in quantifying various domains of athleticism, but my studies often confirm that we are very good at identifying people that are good and bad at tests, but do these tests actually tell us anything insightful about performance?
These thoughts are not meant to resolve this issue. Instead, I am writing out loud about the distinct possibility that quantitative performance measurements may have undesirable consequences for overall performance. Specifically, sport scientists need to ensure that a measure (or composite of measures) does not define the ultimate goal.
One example that comes to mind, was a study published in The Lancet correlating grip strength and health. Essentially, the stronger your grip, the healthier you are (and it’s this unfortunate reductionism central to concept). The specific details are unimportant, but the motivation and behaviours as a consequence of this measure are interesting. I only came across this study as I searched online stores for a grip dynamometer to provide a measure of fatigue / physiological readiness in my elite athletes. What I found was dozens of devices being marketed as a way to increase your grip strength so that you could be healthier. Nowhere does the Lancet study equate the two, but now you can buy the solution.
A critical thinker will realize the grip strength is an agent - it acts on behalf of another concept. Changing the second concept (health) will (likely) change grip strength, but targeting a change in grip strength will not necessarily change health. And since I’m throwing catch-phrases around (this one is from Ridgway, 1956):
“Not everything that matters can be measured.
Not everything that can be measured matters. “
When we are measuring athletes, one must continually ask themselves:
Does the outcome change an intervention? Is it meaningful?
a. What is the most important trailing measure(s) / outcome(s)?
b. (I’ll bookmark this spot for a future link on another data cliche: “All models are wrong. But some are useful.”)
Does the measurement measure what I think it measures? Is it specific?
a. How do I standardize a measurement for a movement / physiology that by definition is experienced in unstandardized conditions?
b. Are my interventions / treatments targeting sport performance, or to excel in testing?
I’ll refine this list as my studies mature.
- Updated June 2020