We have discussed the difficulties in implementing open innovation.  Netflix did an amazing job of leveraging open innovation with Netflix Prize. For a while they were receiving amazing results from the exercise. That is why, I was surprise when I read the article Netflix never used its $1 million algorithm due to engineering costs:

“Netflix awarded a $1 million prize to a developer team in 2009 for an algorithm that increased the accuracy of the company’s recommendation engine by 10 percent. But today it doesn’t use the million-dollar code, and has no plans to implement it in the future,”

Let us dig in to see what we can learn…
First of kudos to Netflix for engaging a very wide community in the innovation.  But more importantly, Netflix was great at setting up tools to keep the community engaged. The second point is overlooked by many organizations engaging in open innovation portals.  Let us go through each of the four challenges we identified about Open Innovation and see how Netflix was able to address them:

1. Valley of Death is when organizations are unable to incorporate outside innovation into delivered products even after acquiring it. Netflix focused open innovation around a problem critical to their business – predicting what movies customers will like.  If the outside innovators were able to demonstrate those results, it would be hard for internal experts to resist implementation (because of not-invented-here mentality).  More importantly, there would be management attention on the subject because of its importance to overall business – which would surely help overcome the valley of death.
2. Trade Secret Protection: Netflix was in a unique position because they did not have to disclose their current implementation in anyway.  All they had to do was to publish the output of their algorithm.  They were able to provide data to the outside innovators to test performance relative to Netflix’s performance.  In most Open Innovation problems, it will be hard for organizations to set up a problem so that they do not have to disclose any internal know-how.  But something everyone should consider…
3. Evaluation / Management Costs: Although Netflix had to set up an extensive infrastructure to administer Open Innovation, the costs were some what mitigated.  Netflix was able to device an approach where the community was able to test their algorithms internally before sending it to Netflix.  Furthermore, Netflix provided clear guidelines and test data for the outside innovators.  This self evaluation by inventors reduced the overhead required to manage/test innovation ideas submitted for consideration. 
4. IP Liability: Netflix bypassed the entire IP problem by only requiring the “implemented” algorithms be provided for evaluation against the test set.  The details of the algorithm need only be discolosed if the algorithm actually produced required results.  Furthermore, by requiring that the results be published (thereby leveraging advantages open source provides to open innovation). I am not sure how many companies will be able to the IP liability issues this way…

Even so, Netflix was not able to get full benefit from the $1M prize because of two factors. First, the cost of implementing the algorithm was so high that they could not close the business case:

We evaluated some of the new methods offline but the additional accuracy gains that we measured did not seem to justify the engineering effort needed to bring them into a production environment.

Second, the market had changed from DVD rentals at the time of innovation challenge to to on-line streaming so that the benefit of the innovation was minimized:

Also, our focus on improving Netflix personalization had shifted to the next level by then.

This is an important lesson for all R&D managers: Even if we can overcome most of the challenges in implementing Open Innovation, several other factors may still prevent us from gaining full benefit of the investment.  However, all was not lost. Netflix was able to use some of the algorithms developed at early stages of the challenge to gain significant benefits.

Netflix notes that it does still use two algorithms from the team that won the first Progress Prize for an 8.43 percent improvement to the recommendation engine’s root mean squared error (the full $1 million was awarded for a 10 percent improvement).

That too is an interesting idea: Set up intermediate goals for open innovation and incorporate them into the overall R&D planning process.