Hey Everyone,
This is now an awkward letter, almost a week late.
Sorry for that! I chose a topic and two essays to build it around but then couldn’t get myself to really dig into the material and think real hard.
Maybe I’m losing steam and should consider creating newsletter seasons.
Anyway, I think the lack of substantial commentary and analysis on my part is made up by the things themselves. After all, as always, they stand on their own.
The topic today is measurement and metrics. It’s an interesting and important topic because so much of our lives are shaped by it! Think: grades, GRE scores, Company KPIs, and so on.
It’s useful to have or try to have numeric measures and metrics. But they have their own pitfalls. The Map is not the Territory. The measure is not what it is measuring.
It can be easy to forget that. Furthermore, complex adaptive systems complicate things even more…
Goodhart’s Law and Why Measurement is Hard
David Manheim | 18 mins
Measurement is important. It allows us to map the abstract and vague non-standardized ideas into the domain of numbers.
David lists three uses for measurement:
Measurement replaces intuition, which is often fallible. It replaces trust, which is often misplaced. It finesses complexity, which is frequently irreducible. So faulty intuition, untrusted partners, and complex systems can be understood via intuitive, trustworthy, simple metrics.
A measure’s best use seems to be as metrics—i.e when they are used to make some decisions.
However, here we start seeing Goodhart’s Law come into play. It’s commonly paraphrased as:
When a measure becomes a metric, it ceases to be a good measure.
David then goes on to talk about how the three uses have their own pitfalls:
Measuring and Intuition
There are some decisions in certain domains aren’t based on metrics but still are effective. These are domains where “raw intuition” beats reflection.
So focusing on metrics here can be counterproductive.
Measuring and Trust
Using measurements requires trusting the data used for the measurements, and the methods used for transforming the data.
We have to trust that the measurements are good. Else, we’ll have to verify the results ourselves. But that’s only feasible when the verification is easy. This can lead to choosing easy-to-verify metrics over good metrics.
Measuring and Complexity
Measures and metrics help simplify complexity by summarizing certain aspects of it. But that comes at a cost of reduced fidelity.
Measures can summarize, but they don’t reduce the complexity. This means that measures hide problems, or create them, instead of solving them.
Optimizing for Mistakes
All of the above means that we are very much likely to create metrics misaligned with our true goals.
Systems using measures are incentivized to perform certain ways – they self optimize. Building systems using bad metrics doesn’t stop their self-optimization, they just optimize towards something you didn’t want.
Overpowered Metrics Eat Underspecified Goals
David Manheim | 28 mins
Most organizations and people have “underspecified goals”—goals that “haven’t been articulated clearly enough”.
It might be benign—or even work well in some cases. But it does leave the organization vulnerable. One danger is that of overpowered metrics—metrics that shape the direction and functioning of the organization more substantially than the actual goals.
David touches upon a lot of ideas in this essay, a lot of which I actually need to chew on further.
Some pointers that caught my eye:
Startups are like short stories. When they start growing bigger, it’s like a story being turned into a novel. There is more of a chance of having fillers in.
“The study of most complex systems shows that disparate goals can lead to harmony despite conflict; alignment in the absence of clearly imposed goals, organizational or systemic, is clearly possible. For example, capitalism, at its best, uses competition to harnesses the desires of individuals to achieve a synthesis that is better for everyone”.
Central BHAG goals are achieved by SMART goals of the individual and teams.
Management's job is to align them.“Like people, organizations drift when they don’t have clear goals, and like people, they implicitly look for metrics to justify their decisions”.
The Lesson to Unlearn
Paul Graham | 18 mins
The most damaging thing you learned in school wasn't something you learned in any specific class. It was learning to get good grades.
In this essay, Paul contends that the usual schooling—with its high emphasis on grades—doesn’t just miss an opportunity of imparting knowledge effectively, it’s actually actively harmful.
First, tests are given way too much importance.
In theory you shouldn't have to prepare for a test in a class any more than you have to prepare for a blood test.
Second (the main problem): tests are no good.
The real problem is that most tests don't come close to measuring what they're supposed to.
… nearly all tests given to students are terribly hackable.
So given the value placed on grades as the measure and metrics of learning, students are incentivized to optimize for that; they (well, we) go on tangents:
The result is that students compete to maximize the difference between learning and getting good grades.
And the worse thing isn’t this wasted, misdirected, effort; it’s more subtle:
But wasting your time is not the worst thing the educational system does to you. The worst thing it does is to train you that the way to win is by hacking bad tests.
We start looking for hacks—easy answers, ways to crack “tests”—everywhere.
So what’s the takeaways?
Realize that there is this subtle lesson to unlearn.
We must identify bad tests to avoid them. To avoid being incentivized to play the “hack-the-test” game.
How can we identify bad tests? Paul posits that most tests that are imposed by authorities are bad tests.
Tests can be divided into two kinds: those that are imposed by authorities, and those that aren't. Tests that aren't imposed by authorities are inherently unhackable, in the sense that no one is claiming they're tests of anything more than they actually test.
François Chollet: Measures of Intelligence | Lex Fridman Podcast
Lex Fridman and François Chollet | 2 hours 30 mins
Very interesting conversation which, among other things, centers around how to measure intelligence. (See: On the Measure of Intelligence).
Chollet defines Intelligence as:
The intelligence of a system is a measure of its skill-acquisition efficiency over a scope of tasks, with respect to priors, experience, and generalization difficulty.
So it's the efficiency with which a system learns rather than the output of such a learning.
The ability to play chess isn’t intelligence. Rather, it’s the ability to learn to play chess.
However, we usually do or can only measure the outputs (the skills) rather than the meta skill. In that sense, they are misdirected and limited.
A better test/measure would directly look at the meta-skill. Furthermore, it needs to be actionable and should also account for constraints (the priors, environments, etc).
Chollet’s attempt at an answer was the Abstraction and Reasoning Challenge which tries to test for the meta-skill of skill acquisition.
💻 Making Badass Developers
Kathy Sierra | 23 mins
An interesting talk on a way to get better as a developer.
Takeaways:
There are roughly three skills in relation to you: (A) those that you can’t do, (B) those that you can do but with difficulty, and (C) those that have become automatic.
Having a lot of skills at (B) is bad because they take up a lot of our limited cognitive capacity. Thus, the goal when learning should be to make the transition from A->B->C be as smooth and quick as possible.
One great way to do that is to be exposed to a high quantity and quality of the skill you want to get better at.
💻 Learn X in Y Minutes
Mostly programming languages focused but at any rate, a pretty cool resource.
I found it (and found it useful) when looking for a resource to quickly get the gist of the `toml` file format.
🎵 Music
Insanely good! Everything about this!
// Wholesome
I can see myself going for this research area, haha.
Oh, and one more thing:
I have shared this series already but wanted to share this specific episode I just got around to watching this week. It’s just so good.
Anyway, that’s it from me.
Happy Weekend!
With Immeasurable Love,
Bijay