As a society, we are charging headlong into the era of big data, and being a data scientist is going to be the ‘sexiest job in the 21st century’. Apparently.
Numbers and data are everywhere. It’s being produced, collated, crunched and used by everyone – there’s a tidal wave of the stuff.
Nowhere is this more apparent than in professional sports, where data companies are generating masses of data for use by professional teams, the media and gamblers.
But is all this knowledge making us smarter?
In many cases, no. At least, not yet.
This is because collectively we aren’t all that good at interpreting data at the moment. We need more practice. Big data, the science of data analysis, and even the raw computer processing power capable of storing and crunching enormous amounts of data is all relatively new, so as a society we’re still finding our feet.
But also when it comes to data analysis we are still afflicted by biases that have evolved in our brains, making us not nearly as good at spotting true patterns in data as we like to think we are. And time and again we see that having lots of knowledge doesn’t actually help in making us act any smarter.
A football club can know every last detail about what it’s players have done in previous matches and in training, all their stats, even in some cases going back to when they were kids. But how does that help the team win more? It doesn’t. Simply having knowledge is worthless.
Knowledge alone can of course have an aesthetic quality, like a pretty picture hanging on a wall – we can enjoy owning it, and looking at it. But for it be be productive, to have predictive utility and for it to actually be used to make us better at doing something, we have to find insight. And then translate that insight into ways of actually doing things better; in other words – being more efficient. Insight and efficiency. Good data analysis includes regularly asking ‘what’s the point?’. If what you’re doing doesn’t lead in some way to greater efficiency in the way you behave, then you’re doing analysis to hang on a wall, not to get better.
As we plunge boldly into this age of mass data, there are plenty of different roles to be played. There are people who collect data, store it and help us retrieve it. And there are people whose job it is to present it, often in attractive graphs and charts so that we can see it and understand it.
But perhaps the most important role is that of the ‘analyst’ – the person who looks at the data and tries to makes sense of it. An analyst is looking for meaning in the numbers, to detect underlying patterns, to learn lessons from it that can improve our understanding of the subject. An analyst is looking for insight.
The insight can be used to make smarter decisions, such as by building models to forecast what will happen in the future, built on the lessons the data teaches us about what happened in the past.
This is ‘analytics’. Analytics can handily be defined as “getting insight from numbers, and using that insight to be more efficient”.
Analytics is undoubtedly fashionable, and increasingly prevalent in the operations of professional sports teams and betting/investing operations. After an initial period of unrealistic expectations among sports teams in particular, that accompanied the publication of seminal sports analytics book Moneyball, a more level-headed and realistic understanding of the practical uses of analytics is emerging.
Analytics is not a magic wand that can be waved by a sports team to make them win. It is not a way for gamblers to predict the future that ensures their bets always come in. Analytics is not a strategy, a plan or an ethos. Analytics is just a tool that can be used as part of a process focussed on making smarter, evidence based decisions. That’s all.
Like with any tool, if it’s used poorly then it can easily do more harm than good. And like with any tool, there are some people who are just better at using it than others.
So what does it take to be good at analysing data?
The 3 C’s are ‘Context, Correlation and Causation’. Good analysts need to ‘get’ the 3 C’s.
Human beings are hard-coded with a basic ability to discern patterns in things. Everyone reading this will be able to spot the pattern in a series of numbers such as 2 – 4 – 6 – 8 – 10 – ….. but in the animal kingdom, the majority of animals would not pick out the pattern in an equivalent pattern presented to them in their ‘language’.
Humans can spot basic patterns. But a big part of the skill in making proper sense of what you see in a set of data, is being able to put it in context. The ability to see data in its proper context is definitely not common to everyone.
Isaiah Berlin wrote an essay called ‘the hedgehog and the fox’ that split human thinkers into two groups. The majority of human beings think like ‘hedgehogs’. Hedgehogs acquire beliefs based on gut instincts, cognitive biases and relying on things they have been told are true, being true. They have convictions. They seek evidence only to confirm their beliefs, not to challenge them. Evidence against their belief is ignored or excused. Hedgehogs make terrible data analysts.
Fox-like thinking involves being open-minded. Rather than convictions, foxes have theories which they test through experimentation and seeking evidence, the results of which they look at objectively. They don’t mind their theories being proved wrong by the evidence. They seek to understand the universe better out of curiosity, not to confirm the universe is how they have always believed it to be.
Hedgehogs see the world in black and white, foxes in shades of grey. Only foxes make good data analysts.
Perhaps the most common ‘hedgehog’ failing is to develop a conviction based on an insufficient sample size. In their eagerness to spot patterns, and to confirm the beliefs that they hold so dear, hedgehehogs jump to conclusions too quickly.
They see a footballer score a wonder goal and conclude that he must be a fantastic player, when there’s every chance the skill he showed will rarely if ever be repeated. When a team loses its first four matches under a new coach, they conclude that the coach must be rubbish. A betting tipster gives six winning bets in a row, meaning he must be worth following. A business makes three consecutive quarters of profit growth, and the conclusion is that its fundamentals must be sound and should make a good investment. Most people do this all the time, they can’t help ourselves. Hedgehog thinkers makes up the majority of our society.
Another way of thinking of context is ‘an ability to see the bigger picture’. When we jump to conclusions based on a too-small sample size, we run the serious danger of missing the bigger picture, through focussing too hard on just the small picture we can currently see.
If you stood at a roulette wheel for a while watching the results of each spin, and the bets being placed and paid out, you might well start to see patterns emerging. Patterns such as; ‘red 32 sure comes up a lot’, ‘when there’s two blacks in a row there’s always a third’, ‘that guy over there betting big knows what he’s doing, he must make money betting on roulette’.
None of these statements is true. There are no such patterns. 32 red will come up exactly 2.7% of the time if you stand there long enough. A third black will follow any previous consecutive blacks exactly 48.6% of the time. And the guy betting big on roulette will lose in the long run (unless he’s colluding with the dealer, or cheating in some other way). Nobody wins legally at roulette. The laws of probability which govern the game are immutable and unbeatable. Those patterns that you thought you saw were illusions, little fictional stories created in an over-active, pattern loving part of your hedghehogy brain.
Things that seem obvious and definite in a narrow context can often seem far less so once a wider view is considered. Amateur bettors can be attracted to statements like ‘Barcelona have been leading at half time in all five of their previous away matches, so bet on them to be ahead again at HT tonight’. This is a failure to understand context – that a small sample of their last five games is nothing like enough to be sure how likely it is that Barca will be leading at HT tonight. But more than that, expecting something to happen today, just because it happened yesterday is also a fundamental misunderstanding of how our universe works. Everything (virtually) is subject to randomness.
In analysing footballers, context is important and can be more art than science to weigh up. A question a football analyst might be confronted with is; if striker X is currently playing in the Dutch Eredivisie and is scoring loads of goals, will they continue to score loads of goals if we sign him to play for us in the English Premier League?
Anyone who says they know for sure either way is an idiot or a liar. They can’t know, not for sure. Sometimes successful strikers from the Dutch league are smash hits (Ruud van Nistelrooy, Luis Suarez). Sometimes they do ok (Dirk Kuyt). But sometimes they bomb (Jozy Altidore). Anybody who says they know X definitely will/will not score lots of goals in the EPL is not someone who would make it working for a professional betting syndicate.
The smart way to think about such a problem is to develop a probabilistic projection of how likely striker X is to succeed. So, rather than saying he WILL be a success, a smart projection would contain a range of probabilities like; 10% chance of being a smash hit, 60% do fine, 30% bomb.
In working towards these probabilities there are various contextual considerations you would need to allow for. The Dutch league is naturally high-scoring, so goals scored in the Eredivisie are arguably worth less than goals scored in, say, the French Ligue Un where goals are overall much rarer.
Was the striker playing for one of the top teams in Holland, or getting loads of goals for a relegation contender? How old are they? How many games have they played, how many minutes? What language(s) do they speak? Have they played in a formation similar to the one your club uses? How many assists have they had? How often do they make key passes? What % of their team’s total goals have they scored? What is their conversion rate of shots:goals? What is their character like? Do they get booked a lot? Do they like to party? Are they dedicated, highly motivated? Will they be ok sitting on the bench for a season or so?
There are some more in-depth technical contexts that a really thorough analyst might want to look at, to contextualise the raw data. Have they been substituted off or subbed on a lot in games? This matters to a player’s stats because football is a game with a progressive scoring rate, and the level of cumulative fatigue of opponents is a factor, so it is ‘easier’ for a striker to score goals late in games, especially if he is fresh off the bench playing against guys who started the game.
So while it is undoubtedly better to measure player scoring rates in ‘per 90 minutes’ rather than ‘per appearance’, it would technically be more accurate to also control for which minutes a player played. A striker who often comes on late in games may have an artificially embellished per 90 scoring rate, and be unlikely to be able to replicate it if he is starting games. Teams also score goals at a higher rate when they are behind, so a striker coming on and getting a late equaliser should probably be given less credit for it, not more – as theorised in a piece of analysis in The Numbers Game book, which assumed that scoring important goals was somehow a repeatable skill, and didn’t control for any of these real contextual factors.
All of these considerations can lead you to a smarter projection of the range of probabilities. What it can’t do it let you predict for certain how they will do. You can guess, and it’s possible you will be right. But that’s doesn’t mean that you knew. You didn’t. You couldn’t, because our universe doesn’t allow for accurate predictions of things in the future which are subject to randomness.
Thinking probabilistically is smart, but admittedly it can also be a bit boring and appear maddeningly vague. Guys chatting/arguing about football in a pub don’t want to think in terms of a range of probabilities. They want a prediction; something to brag about, or laugh at their mates about later. But thinking probabilistically is how professional bettors have to think because they are seeking efficiency in handling probability, rather than looking for ‘winners’, or bragging rights with their pals. Thinking probabilistically about the future is also how good data analysts need to think.
But it’s reasonable for a ‘normal’ person to ask ‘what’s the point of all this analysis, and all of the taking into account of contextual factors, if at the end of the day I still won’t know for sure if the guy is going to be a success?”. Welcome to the world of professional gambling and investing, and the reality of doing data analysis. What we do is all about estimating probabilities, not making predictions. Just like science can be defined as ‘seeking to improve our current best understanding of the universe’, so analytics is ultimately about seeking to be smarter and more efficient. It’s not the search for truth or perfection, or the power to be right every time. If that doesn’t seem right to you, you’re not cut out to be a scientist, an analyst or a professional gambler.
Imagine that an alien arrives on our planet, and starts observing human behaviour. She starts to observe a pattern that involves the putting up of umbrellas. She notes that when human beings put umbrellas up, puddles then appear on the ground. She reports back her findings to the mothership; something she has learned is that on Earth, the putting up of umbrellas causes puddles to form.
This is a basic case of confusing correlation for causation. Correlation is when two things happen together. Causation is when one thing causes a second thing to happen. Correlation and causation are two different things, though often they can be related.
Though it may be tempting to scoff at the stupidity of our hapless alien, earth-dwelling humans confuse correlation for causation all the time.
Our brains are primed to reach false conclusions, mostly because we like to see patterns, even when they are not there. We get suckered by ‘illusions of causality’ like the umbrellas and the puddles, tiger prints and tigers round the corner, far more often than you might think. This weakness in our cognitive reasoning is sometimes referred to as the ‘post hoc ergo propter hoc fallacy’ (Latin for “after it, therefore because of it”). Most of us are attracted to the this idea of simple order. One thing happened after another, therefore it was caused by it.
But it’s hardly ever true. Umbrellas don’t cause puddles. And ice cream doesn’t cause people to drown. But several decades ago a report by eminent American statisticians concluded that eating ice cream was dangerous and caused drowning. This was based on their findings in data that days with increased ice cream sales were closely correlated with days that had increased numbers of drowning deaths. The authors weren’t earth-ignorant martians. They were highly educated, intelligent data scientists who simply missed the fact that warm weather causes people to both go swimming more, and to buy more ice cream.
Ice cream and drowning, like umbrellas and puddles, is an example of a correlation which has a third, common causal factor. One thing which causes the other two to happen. Another example is that people who go to bed with their shoes on are also more likely to wake up in the morning with a sore head. This is the first kind of correlation, where a common factor causes two things to appear related, despite there being no actual causal connection between them.
The second kind of correlation is where there is no causal link at all, they are just coincidences. Plotted on a graph, it can be made to look for instance that the growth of Facebook users caused the Greek debt crisis, Maine’s divorce rate is linked to the US per capita consumption of margarine, and that the level of Norway’s national crude oil exports will affect how many people will be killed each year in collisions with trains.
Viewed graphically these unrelated sets of data will appear to mirror each other, looking for all the world like they must be related. But they aren’t, of course. If you look for long enough at enough data you will always find correlations. That’s just a product of the law of big numbers. But some of them will be pure coincidence.
The third type of correlation is where there is a direct causal link between two things. They are different things, but they will always mirror each other (virtually) exactly. For example, the rate at which people chop off their fingers doing DIY, will be virtually perfectly correlated to the rate of hospital admissions for severed digits.
An inverse correlation is the fourth kind. Here the patterns of data will go in opposite directions. For instance, as the slope of a hill increases, so the speed at which a cyclist can pedal decreases. An increase in alcohol intake is inversely correlated with good judgement. A football club starting at the bottom tier of a pyramid can show an inverse correlation between the level at which it plays and how many games it wins, as the quality of opposition gets harder as it climbs the leagues.
The fifth and last kind of correlation is a loose correlation. This where things get tricky. Here there is a bit of causation, but the relationship is not simply one-to-one. Examples include the height of parents and the height of their children, and the price of a good/service and the level of demand for it.
In football there is a loose correlation between the number of team shots taken and team success. Very broadly, a football team’s success will be correlated and partially caused by the number of shots it takes, and the number it restricts its opponents to taking. But the relationship is not perfectly causal, because shot quality matters.
In an extreme example, if a team took a shot every time it got the ball into its opponent’s half it would compile superficially impressive looking shot statistics. But such an extreme tactic would actually result in an inverse correlation with success.
No real-world team plays such an exaggerated version of that tactic, but plenty use a milder version, having the habit of taking shots from unpromising situations, including from great distance to the goal. So football analysts have refined their ways of measuring playing statistics in recent years, meaning they use better data than a simple tally/ratio of total shots; such as shots on target, or shots from inside the penalty area, or expected goals based on shot location.
None of these is perfect, and in such a low scoring sport as football, even if analysts found a perfect causal relationship between a shot stat and actual goals scored, the perfect correlation would only become evident over a large sample of games. The inherent randomness of a football match will always defeat any effort to find perfect correlation between a stat and success in a single game.
“Correlation does not imply causation” is the phrase often used to describe the chance that statistically significant looking relationships are not, in fact, inter-related. But it’s probably not a great phrase; it’s fine for a correlation to IMPLY a causation, it’s just not smart to assume it PROVES it.
It’s because correlations can have strong, loose or non existent causal relationships that we need data analysts, not just a bunch of robots to analyse data for us. Understanding the nature of relationships isn’t something that can be left to an algorithm – at least not yet. No matter how much data you gather, and no matter what computer processing power you employ, data needs to be interpreted to make sense of it.
So if someone suggests that statistics ‘tell’ you something or other, you can tell them that they are wrong. Statistics are inanimate. They don’t think or talk, so they can’t ‘tell’ us anything. The lessons and insight they can reveal needs to be extracted and interpreted by us.
Football statistics analysts have been especially prone to confusing correlation and causation. When team possession stats were first produced and published it was quickly apparent that there was a correlation between teams having a lot of possession, and teams who won lots of matches. So some analysts made the logical conclusion that a high possession% causes teams to win.
But this is just wrong. It is umbrellas and puddles. Being a really good football team (compared to the teams you play against) is the third common factor. Good teams will win lots of games, and will generally have more possession than weaker teams. But not always. Leicester City and Atletico Madrid are good examples of teams whose recent success is clearly not caused by high possession. If anything, because of their style, having high possession is detrimental to their chances in any given game. Possession % in football is correlated to and caused by playing style. It does not have a direct causal effect on success.
High profile people at football clubs often misunderstand correlation and causation too. The former Tottenham and Liverpool Director of Football Damian Comolli (ironically hired for his supposed proficiency with analytics) found a correlation between teams winning games, and the number of headers won, crosses made, and ‘final third regains’ that teams made. This is worse than relying on possession stats. He spent/wasted millions of pounds in transfer fees on players who fitted in with this theory. He was buying umbrellas thinking they would give him puddles.
Jose Mourinho espouses an analytical theory; ‘The game is won by the team who commit fewer mistakes. Whoever has the ball is more likely to make a mistake. Whoever does not have the ball is therefore stronger’. This is just as wrong. It’s like concluding that warm weather is the only thing that causes people to go swimming. It ignores team playing style. So while it might be a decent proxy for describing how Atletico Madrid, Leicester and some of Jose’s teams have achieved success during certain seasons, it does not hold for all teams or even specific teams indefinitely.
Opponents adapt to styles and tactics. Believing this ‘error avoiding’ tactic is some sort of key to playing long-term successful football is little better than believing that making lots of final third regains is going to make you into world-beaters, or that warm weather causes people to drown.
A high profile case of people hopelessly mistaking correlation for causation has received a lot of attention in recent years; this is the so-called 10,000 hour rule.
This theory espouses the idea that excellence in anything can be achieved through a lot of good practice, and that natural talent is at least unimportant, and perhaps irrelevant. The idea stems from studying ‘outlying’ high performers (musicians, sportsmen, businessmen, chess players etc) and finding in the data that a lot of them had practiced in their chosen field for around 10k hours.
The 10k hour theory is complete and utter nonsense, and a perfect example of how to do data analysis very badly.
All living animals on our planet are born with a set of aptitudes for doing physical and mental tasks. These aptitudes are largely genetically hard-coded, derived from generations of evolution and passed on to us by our parents. In every individual there is also a sprinkling of randomness and the chance of genetic mutations which contribute to our aptitudes. We can improve upon our natural aptitudes by training and practice, but the ceilings of our ability in any field are set and you can’t break through them.
Giraffes are born with an aptitude for reaching leaves high up in trees. Moles are born with an aptitude for burrowing underground. Saying that a human being can become outstanding at ‘anything’ with enough practice is as patently wrong as saying that the only thing holding back a giraffe from being a great burrower is practice, or that a mole could learn to pick off high-up leaves if only he would apply himself properly to the task for 10k hours.
Millions of dollars are spent each year on young racehorses. If the 10k hour were true then all this money would be being squandered. All that rich racehorse owners would need to do was buy any old horse, train it to run fast for 10k hours, and then unleash it on the track.
The buying of yearling horses for racing is an excellent distillation of how the universe of high performance really works. Owners will pay millions of dollars for year-old horses who have never yet run in a race, based on three factors; breeding, conformation and temperament.
The genes that relate to a horse being able to run fast are passed down by its parents to each foal. Just like in humans and all animals, genetics is the largest factor which governs the aptitude the young horse will be born with for the capacity to run fast. This is why the progeny of the fastest racehorses now retired to stud are most likely to be able to run fastest, and why they fetch the highest prices.
Because of randomness and the potential for genetic mutation, occasionally a foal with an unpromising genetic profile will emerge as a champion runner. And regally bred yearlings can turn out to be slow. But in a big sample of horses, selecting young horses to buy using their breeding beats any other method out of sight.
Among the impeccably bred foals who pass through the big show-rings, there are small differences in physiological and mental make-up that can impact upon the horse’s ability to run. Experts are looking out for signs of these small differences, positive and negative, when they make their recommendations to owners about which young horses to buy.
The small differences which manifest themselves in the way horses look, go towards making up what is referred to as the horse’s ‘conformation’. The way the horse behaves is ‘temperament’, and the yearling experts are looking for clues as to a young horse’s likely attitude to training and hard competition. It’s not an exact science of course, and like with any data sample there are extremes examples of this method of selecting young racehorses getting it wrong.
But on the whole, the horse-trading market using this ‘likely aptitude’ method of identifying future excellence is formidably efficient. The training the horses receive makes a contribution to their ability to run fast of course, but only up to the ‘ceiling’ to its maximum potential speed, faster than which it cannot physically go. A good trainer can help it get up to that ceiling. But the height of the ceiling (aka ‘the range of it’s potential’) is hard-coded by its natural aptitude for running fast.
Humans are exactly the same. We all have a range of aptitudes with ceilings of varying heights, for a whole range of tasks such as running fast, playing sports, doing music, producing art etc. The idea that you can do ANYTHING if you put your mind to it and practice hard enough, is as illogical, wrong and pernicious as saying there is no limit to how fast any horse can run.
The people who promote the 10k hour rule are guilty of five basic failures of data analysis;
- Context. They are guilty of looking too narrowly at a small and select sample size. They only looked at high performers and how many hours they practiced. What about all the millions of people in the world who have practiced something for 10k hours and remain useless/average at it? What about all the people who have attained excellence in a field with far less than 10k hours of practice? This is classic hedgehog thinking – they found that 10k hours was a ‘magic’ number because they wanted to find it.
- Correlation does not imply causation. Lots of practice, and excellence in a given field is like ice cream sales and drowning deaths. One is not caused by the other. It is a classic case of correlation caused by a third factor, which is natural aptitude. If a person is naturally talented at something they will be more likely to enjoy it, and also more likely to be afforded the opportunity to practice it more. So people who are really good at something are likely to have practiced the thing far more than someone who is only mediocre. There is a loose causal relationship, because (good) practice makes people better at a thing. But the relationship is nothing remotely like one-to-one.
- Absence of evidence is not evidence of absence. Just because you can’t see evidence of something in your set of data does not mean that that thing does not exist. You may not be able to see the evidence that on-field leadership makes a difference to a football team’s performance, but that doesn’t mean that it doesn’t. It’s possible that your data isn’t good enough, or that you aren’t good enough at interrogating it. For 10k hour proponents to say that they can find no evidence of natural aptitude being a key factor in high performance is almost certainly a case of hedgehog thinking; they didn’t find any evidence because they really didn’t want to find it.
- The smell test. Using fancy analytical techniques is all very well, but a big part of a good analyst’s armoury is being able to look at something and just telling when it looks, and smells wrong. For example, what would a 10k hour believer make of AB de Villiers? The South African cricket captain is currently the world’s top rated one-day batsman, and rated no. 6 in the Test ratings. By his late teens he had been selected by his national hockey and football squads and was the captain of his country’s junior rugby team. He still holds six national school swimming records, and the record for the 100 metres dash on the track. He played junior Davis Cup tennis for his country and was South African U-19 badminton champion. And he plays golf off scratch. If the 10k rule was really true, then to have achieved excellence in all these different sports, AB would have had to start practicing all day every day from several years before he was conceived. In other words it stinks. The smell test of the 10k hour tells you it must be nonsense simply by looking at a person like de Villiers. Some people are just born with outrageous, outlying physical gifts, just like some people are born with an extreme mental capacity, or great aptitude for producing art or music.
- Commercialism over good science. The 10k hour is so obviously wrong that it’s fair to question just how much it’s authors really believe in it. The human capacity for sticking to a hedgehog conviction in the face of irrefutable evidence is considerable, but you have to wonder in this case if the commercial benefits (such as book sales, research grants, speaking engagements etc) have helped to blind from reality the people who should have been critically testing their theory through scientific research and experimentation. The 10k hour rule is a ‘sexy’ idea that has sold well, but it’s poor science and terrible data analysis.
The 10k hour rule is a myth. Being naturally talented at the ‘something’ is the common third factor that makes practice and high performance correlate, like rain does for puddles and umbrellas, warm weather for ice cream and drowning. Practicing will not make you world class, unless you have the disposition of aptitude that allows you to reach that level.
Of course practicing something will invariably make a person better at something. And if it’s particularly good practice; such as receiving high class coaching with great facilities, or getting to train with people who are better than you, then improvements can be rapid and significant. But the idea that anyone can carry on improving to become world class simply through a lot of practice is simply a misunderstanding of how the universe of high performance really works.
The 10k hour myth is a great example of how intelligent, but not smart people can misunderstand the basics of context, correlation and causation. It is also the sort of really poor analysis that can give data analysts a bad name. It is possible for analytics to make significant contributions to our understanding of complex events, and help us get much better at doing things. But high profile hoaxes like 10k hours sets the whole science back.
Being good at data analysis is not the same as being good at doing maths. In fact we can say the two things, like practice and high performance are only loosely causally related. Analysing data is about the clever interpretation of evidence in search of insight, not just about finding correlations in data – as we have seen above. It is largely about being a smart, cunning fox who sees the big picture all around him, unlike a hedgehog scuttling along with his head down, pursuing the one thing that he can see in his limited range of vision.
Interpretation requires imagination. People whose brains who naturally work in neat straight lines tend to be good at storing knowledge, and performing the logical procedures of mathematics. Such thinkers are unlikely though to also have a great capacity for the tangental thought that is required for making imaginative connections, required in creative disciplines and in data analysis. In other words, maths people are unlikely to also be good at distinguishing the causal links among all the correlations that they find.
Albert Einstein was clearly an exception, whose unique brain had an enormous capacity for both types of thought, But he was pretty clear on what mattered more; “imagination is more important than knowledge” he said.
So if you are hiring any kind of analyst and your job description says something like ‘must have a degree in maths based subject’ then scrub it out. You want maths guys to be collecting, organising and storing your data for you, for sure. But you want guys with imagination, curiosity, subject knowledge and street-smarts to be doing the analysis for you. It takes that sort of brain-type to be really good at figuring out the underlying signals behind, and in amongst the stats. Anybody who can do maths and show data in a graph can spot correlations. Understanding the degree of causation is a different job entirely.
The future is inherently random and unpredictable; whether it’s football results, rates of ice cream sales or telling how much rain will fall from the sky. Knowing what happened in the past is only useful in helping you to say how likely things are to happen in the future. It doesn’t allow you to know for sure what is going to happen, no matter how much data you collect and how many correlations you think you see in the patterns that it creates.
So if you fancy doing the ‘sexiest job of the 21st century’ but you weren’t much use at maths at school, don’t despair. Albert Einstein wasn’t all that good at it either. Imagination is more important than knowledge. Practicing analytics for 10k hours isn’t going to help you if you don’t ‘get’ the 3 C’s. Analysing data – doing analytics – is ultimately about being smart, not brainy.