Six Years after Moneyball

On September 19th, 2011 the one movie that affected my career more than any other was released: Moneyball. It blew me away! To put it most simply, it showed how math and data could transform business success. Sixteen days later, Steve Jobs died. I really looked up to Mr. Jobs and this was a period of great reflection for me. 

A month later on December 16th, the historically catastrophic Zynga IPO happened. Probably aided by the hype around Moneyball, the leaders of Zynga touted their company as "revolutionized by data science". Investors were falling all over themselves to grab a piece of this and trusted media sources were completely on board. When tragedy struck, I remember shocked investors asking "why didn't anyone see this coming?" on the Forbes website. So I published a link to Zynga Analysis in the thread on Forbes and it became clear that one person had indeed predicted the Zynga collapse in detail months before. Was this luck? Psychic ability? The author described in detail what was going on inside the company without being inside the company. Events that hadn't even happened yet were detailed. 

This is important because it is an example of predictive analysis and in this case was potentially worth billions of dollars if utilized at the time it was written. This is the same sort of predictive analysis described in Moneyball. 

Sometime in January of 2012, while reflecting on all of this, I wrote Moneyballification. The paper is fairly short, but I go into more depth as to what was really going on in Moneyball than what was communicated in the movie. Then I explain how that success can be translated to other industries.

So if I'm this visionary evangelist for data science, why did I predict the Zynga IPO failure? Shouldn't I have been Zynga's biggest fan? If the insertion of a mathematician into a baseball team in Moneyball caused money to fly out at hair-raising speeds, shouldn't the same thing have happened at Zynga? Or at (insert almost any big company today)?

The problem is that to have the success described in Moneyball, you need two people (unless they are the same person). One is a data scientist. The other is a Domain Expert (also called a Subject Matter Expert or SME). In the movie the data scientist transforms the company in the course of a year. Predictive modeling really is that powerful. But he's only able to do this by using the models developed by the Domain Expert. What isn't mentioned in the movie is that this took said Domain Expert (Bill James) 30 years. Going back in time to talk about this other person would have made the movie too long so he was just deleted. 

Leaving out the Domain Expert did a big disservice to all the business people that saw the movie and decided this was too good to pass up. They were given the false impression that you just hire a data scientist and unleash them on your business and run off to spend money before the bank explodes. Demand for data scientists skyrocketed and now they are treated like superstars with mind boggling paychecks. I'm not suggesting they aren't worth it, or that you don't need them, but the success you were hoping for isn't going to happen because you missed something. The part that was left out of the movie. 

The Domain Expert.

What is a Domain Expert? In regards to software I like this definition from the wiki: "In software development, as in the development of "complex computer systems" (e.g., artificial intelligence, expert systems, control, simulation, or business software) an SME is a person who is knowledgeable about the domain being represented (but often not knowledgeable about the programming technology used to represent it in the system). The SME tells the software developers what needs to be done by the computer system, and how the SME intends to use it."

Someone that understands an industry and its consumers so well that they can predict and model consumer behavior and product outcome. Typically these are Super Consumers that become so involved in their passion for some particular thing that they take ownership of it and do their best to understand it completely.

When a data scientist attempts quantitative analysis on a system, their results depend on the use of all of the most important variables, and knowing how those variables interact. Some of these variables and interactions are invisible to all but the Domain Expert, and this is why Data Scientists working without the aid of the DE have limited and not always positive results. In the area of data science, if your math isn't good enough to be reliably predictive, it's not worth much. If your BI team can tell you what just happened, that's a start. But telling you what is going to happen, before it happens, is much more useful.

So here we are in 2017 and in the last six years business intelligence departments in game companies have gone from non existent to pervasive. I credit Moneyball and Zynga, even though in the latter case this was a red herring used to promote their IPO. So you have all these data scientists, and the people that hired them have very high expectations that they will transform profits as was done in the movie. This is a lot of pressure and in most cases is unrealistic if there is no domain expert to help guide the BI team. In some cases the data scientist will just do their best to pass as the domain expert because this is, after all, what is expected of them.

While every data scientist I have talked to in the last six years has agreed that they perform much more effectively when paired with a domain expert, where are the domain experts in game dev? What are they called? Where do you find them? There are no schools for "Domain Expertise". How do you assess expertise? If someone knows more than you in some area, you can tell they know more than you in that area. But you can't tell how much MORE they know than you in that area. Is it 10% more? 100% more? 1000% more? You have no way to tell. 

Two Types of Domain Expert

Even if you do engage a domain expert, you can get very mixed results if you don't know which type you are engaging and how to make best use of them. How do you even tell which type you are engaging? The first step in even finding a DE is to know the two types and what they look like.

By far the most common type of DE is what I would describe as the Existing Paradigm Expert. Using the system described in Moneyballification, I will describe this person as the "C-DE". C because they operate in the current paradigm, C. Their skill sets, resume, and income trajectory are linear and reliable. They command respect in their field. Professionally at least, they are not particularly controversial. Employees operating in the existing paradigm will experience elevated morale when led, at least in part, by a C-DE. 

The other type of DE is initially something of a unicorn. They are the Future Paradigm Expert, or D-DE as they are operating in generation D. Their ideas challenge the existing paradigm and are considered silly or impractical at first. They will have a very difficult time even getting the opportunity to test their ideas. Once they do, you might expect that they will be welcomed with open arms by industry. Industry spends a lot on capital to gear up to operate according to the prevailing paradigm. This capital includes human capital. All C-paradim capital becomes much less valuable in the D-paradigm. As people do not like losing value, they will move aggressively to suppress any move to the D-paradigm if they are in the C-paradigm. This is rational, at least in the short term, from the perspective of self interest.

Thus if a D-DE is dropped into a company operating under the rules of the existing paradigm, they are going to be seen as disruptive, controversial, and threatening. If the D-DE lacks authority, their actions will be neutralized to maintain the existing paradigm and actions will be taken to purge them from the work place. You can see how this looks in the movie Moneyball, and you can see this also with Steve Jobs how his success led to C level employees successfully petitioning the board to replace him with someone less controversial. If utilized successfully, the D-DE delivers results that render the C paradigm (and all the capital associated with it if it can't be retooled) obsolete and "D" becomes the new "C". 

Which Type is Best for Your Company?

If you are asking this question, then presumably you already have a company and that probably means you are operating under the existing paradigm. To be frank, companies operating under one paradigm tend to continue on until that paradim is no longer profitable and those same companies often then go out of business. Kodak is a classic example of a company that invented the future tech that eventually replaced it, but did not want to expend the effort transitioning to the new tech paradigm. 

As shown in Moneyball, the Red Sox were willing to take a risk on the future paradigm because they really had nothing to lose at that point. Failure was assured for them on their present course. This doesn't mean everyone was happy about trying something new. 

In the auto industry, it seems pretty clear at this point that the days of the internal combustion engine are numbered. Investors are racing to embrace what they see as next, even though complications and setbacks are all but certain. Elon Musk seems to have what it takes to usher in the future paradigm for auto production. But if you dropped Mr. Musk into General Motors, the results would likely be catastrophic. Morale would drop, production would drop, and the company just might not survive the retooling process without an emergency influx of capital. 

So ask yourself, is your company more like Tesla or more like GM? This might require some painful reflection. If you aren't sure, then you are almost certainly like GM. If you are using a model embraced as the standard by your industry, then you are like GM. The existing paradim works great and is very profitable until it isn't. At some point profits drop to zero, what is called "perfect competition". Beyond that point, profits turn negative and obsolescence has arrived. 

If you are in the last situation, it's probably time to either throw in the towel or attempt to convert to the future paradigm if you know what that is and it is available. Success is not assured. If you are just in the process of creating a new company, and you have a choice between C and D paradigms, you can save yourself a lot of grief right away by designing the company around the D paradigm. If your company is currently profitable under the existing paradigm but you would like a significant boost from your BI team (and perhaps other departments), then you should seek out a C-DE. 

Filtering Out Posers

In the case of the C-DE, they have a long and easy to evaluate career that is pretty well understood even by those that don't do what they do. Methods to evaluate their efficacy are already well established. With someone claiming to be a D-DE, this is much more difficult. You likely don't understand what they claim to understand, you have no established tools to measure their abilities, and any successes they may have had with their new methods may have been luck. 

In this case a standard interview will be more of a popularity contest than an effective predictor of value. My favorite method of determining potential value in this case is the one used by both Microsoft and Wargaming during my evaluation. A question or scenario is given that there is no internal solution for in the company. If the prospective DE can solve the problem in a way that seems rational and likely to succeed, despite being something the existing employees have not previously thought of, then you have a winner. If they are a poser, they will say all kinds of things that sound pretty awesome, but you know they won't work because you've already tried those things or the pitch leaves core problems unresolved.

What's Next?

Okay so you saw the movie, you hired your data scientits, and you got some positive results. The results didn't catapult you from last place to first place, but here you are and since everyone else is here too, this is now the standard paradigm. A BI team with no DE. Get a DE. The better the DE, the bigger the boost. Give your DE sufficient authority to do their job. In Eastern (Europe or Asia) studios, this is rarely a problem. Workers follow instructions. 

Presumably any DE (C or D) is going to propose changes to how things are done in your company. That's why you hired them. You should be prepared to back them up. If you don't have confidence in them, then get rid of them. Lack of resolve at this stage is a breakdown in leadership and management. If you saw the movie Moneyball, and saw how easy it was for them to go from a losing team to a winning team, then you must have been watching a different version from the one I saw. 

This applies to your whole BI team as well. You hired them to tell you how to transform and optimize your business. If they are operating without a DE, then they are basically just measuring what you are doing and telling you how you just did. That has some value and is pretty simple. If they are functioning properly with a DE, they will come to you with "We are underperforming because of 'ABC' and can improve performance by doing 'XYZ'". They will guide you through the process and then report on the results. They don't wait to be told there is a problem. The DE should be actively assessing all aspects they are expert in and autonomously identifying systems and creating measurable solutions. 

I look forward to further discussing any of the topics raised here, in the comments section.