Blog of Sara Jakša

Goals and Measurements

There is an interesting phenomena, when it comes to the goal setting and measurement. In economics, it is called the Goodhard's Law, but it also exists in the artificial intelligence and I have heard anecdotally, that it also exists in the hiring. Basically, what it means is, that when some statistical regularity is used as a target, then this target can break this regularity. It means, that just because something worked in the past, if we take it as a goal, it stops being a good predictor.

Lets take an very cliche and cartoony example from the personal health. But I think it is really understandable for explaining the principle. I don't know a lot about health studies, but I think there is a connection between health and not being overweight. Otherwise, they would not talk about obesity epidemic and be all panicky about children being more obese, because it is bad for their health? I mean, it is an assumption, but if it is wrong, then a lot of discourse in media is just... misleading :). So, lets now say, that because we want to be more healthy, the weight will be the crude measurement of that (I am from social sciences, so I am very ok with crude measurements). So we start dieting, and at one point, the people send us to the hospital, because we can't function normally anymore - which can be the effect of severe anorexia.

I guess I could say, that the very act of measurement can change the phenomena itself. I could bring quantum physics here, but I think that then I would only prove, that I don't know much about it. I can see this with myself. The very act of me putting every scientific article that I read in my bib file makes me more likely to try and finish the article, even if it is not that interesting. And that is without me checking how many of them do I read very frequently. I think the first time that I checked was for the New Year, and that was after more than a year of starting collecting. And I have not checked since. The books are the same, but there the effect is a bit different. Now, I am slightly more favoring the short books, than the long ones.

Well, reading a lot of books, short or long, and finishing more scientific articles, whenever they are relevant or not, is not a bad. But I am not checking constantly. Imagine how things change for people, that keep track of more aspects of their life. Quantified self movement comes to mind. How are these very subtle effect coming together to shape their lives?

Not to mention, that this is sort of done to us on a regular basis. Lanier in the book Ten arguments for Deleting Your Social Media Accounts Right Now calls them BUMMER (Behaviours of Users Modified, and Made into an Empire for Rent), which are the social media and search engines and personalized advertising and so on (the book itself is quite good, I can recommend it). How their algorithms collect the information about us, and then they use this as a target to predict our behaviour. Can you see the problem, in relation to the problem defined above?

What does this means for us? It means, that just because something can be measured, that does not necessary means, that it is a good goal. That we need to be more careful about how do we measure the goals for ourselves. But it also gives us hope. One thing that economics figured out, that when the governments were using inflation as a target, the Goodhart's Law kicked in and it became useless for prediction. But once it stopped being the measurement, it again become the good predictor. So, on the short term, I don't think it is that bad to use measurements like that. Lets say that one wants to be healthy. First one could try losing weight (if overweight), then start exercise more frequently and so on. Each measurement would only be under this effect for a short amount of time.

I guess even better would be to use multiple ones, since it is hard to go too much in one direction with multiple of them. It there really needs to be a measurement, then they could be combined, but summing them (better with regards to avoid overlearning) or as some sort of principle-component index (because not everything is a good indicator).

The best, but probably not so easy way, would be to use all the evidence and have a more holistic way of looking at the situation. Not everything can be measured and maybe our long-term improvement in live needs a bit more quantitative approach, not just qualitative.