How to measure R&D Effectiveness?

Metrics that measure effectiveness of ongoing R&D have been difficult to obtain. Four types of R&D metrics have traditionally been used:
  • R&D Investments / Expenses
    • Total R&D Headcount
    • Total R&D Expense
    • R&D Expense as % of Revenue
    • R&D Expense increase/decrease from prior year
    • R&D Expense compared with peers/industry average
  • Project execution status
    • Performance relative to plans (costs and schedule)
    • Concept to Market Time
    • Number of Projects in the Pipeline
  • Historic results-based Metrics
    • Fraction of Revenues from New Products
    • Number of Patents generated
    • Number of Papers Published
    • Customer satisfaction with new products
  • ROI-based metrics
    • Return on Innovation Investment
    • Target NPV for each new product
The problem with most of these metrics is that they are not actionable. As we discussed earlier, R&D effectiveness needs to drive real business results. It is hard for managers to take concrete actions using these metrics to improve R&D effectiveness. All metrics need to be able to aggregate information across the R&D portfolio to help managers see trends and make decisions. Good metrics should allow segregation of management decisions into individual project level actions.
The first two types of metrics are not directly related to ongoing R&D and its projected results. For example, what would happen if we increased our headcount or reduced it? There is no easy way to understand the impact on R&D pipeline. Will it improve revenues? When? Are there key technologies that will take time to develop no matter how much money you invest? It is not easy to tie headcounts to R&D results in the future.
Historic results-based metrics such as number of patents generated are all approximate indicators of R&D performance several years in the past. Managers can gain little direct insight into ongoing R&D effectiveness from these metrics. Nothing a manager can do now will impact number of patents issued or number of papers published. More importantly, it might be better for
ROI-based metrics tend to work better with product development projects near delivery, but they are very hard to use for early stage development. Furthermore, it is hard (if not impossible) to develop ROI on technology development effort that might impact a feature in several different products (think a new type of metal that can be used in different types of cars). ROI computation becomes even more difficult for disruptive innovation.
Over the next few weeks, we will discuss new kinds of metrics that can help managers improve R&D effectiveness…

How to Manage R&D Risks?

R&D is about building something new and delivering it in the future. The process is inherently uncertain, and hence involves risk. The more disruptive the technology, the more risk is likely to be introduced. The risk comes from several sources:

  1. Technology Uncertainties
  2. Integration Uncertainties
  3. Manufacturing Uncertainties
  4. Market Uncertainties

It is not possible to eliminate these risks, but it is definitely possible to manage them. An interesting case study can be found in The mysterious story of the battery startup that promised GM a 200-mile electric car

Image Source: Quartz.com

Image Source: Quartz.com

Energy storage systems are probably one of the most important current R&D challenges. From mobile phones to automobiles to airplanes, the implications of better performance are huge — both for the companies that develop new batteries and companies that start providing products using them. Because of this promise, a small company called Envia got an opportunity to work directly with GM:

It was to contain a deal rare to an industry newcomer—a contract worth tens and possibly hundreds of millions of dollars to provide the electric central nervous system for two showcase GM models including the next-generation Chevy Volt. Untested small suppliers almost never get in the door of the world’s major automakers, which regard them as too risky to rely on. But GM was won over by what seemed to be the world’s best lithium-ion battery—a cell that, if all went well, would catapult the company to a commanding position in the industry with a middle-class electric car that traveled 200 miles on a single charge and rid motorists of the “range anxiety” that disquieted them about such vehicles. GM would have the jump on the high-end Tesla S, the only other major model with that range but one that would cost much more. For Envia, the contract could lead to an IPO that would make both men rich.

The article points out many steps that both GM and Envia took to manage the four R&D risks we laid out above. Let us get started with Market Uncertainty. Clearly, GM would have needed to take some risk in projecting demand for a new extended range car. However, for start ups and small firms like Envia, market uncertainties potentially overwhelm other types of risks. The discussion around a potential agreement with GM took a long time (as many such agreement with large companies take):

But the talking had gone on so long and with such uncertainty that neither man had even told Envia’s staff scientists of the impending deal. Even if they felt more confident, they could not have said anything, since such news could affect GM’s share price. Word had leaked around the Envia lab anyway. An edginess hung over the lunch. … Bay and Redpoint invested another $7 million in the company a year or so after the first tranche. But the startup was burning through the cash while potential customers were slow to commit.

Unfortunately, market risks are not easy to mitigate. Pretty much the only approach might be diversification: to pursue many alternate paths so that the impact can be managed if there are any changes. Many startups such as Envia may not get the opportunity to do so. Larger companies can try alternate sources of technologies or alternate product roadmaps to mitigate market risk. For managers, this means gaining visibility into, guiding and synchronizing many alternate development paths. For R&D organizations, this means managing multiple projects. Product and Technology roadmaps are a key solution that organizations can use to address the challenge of increased R&D complexity. That brings us to the biggest source of risk in this case: Technology Uncertainty. Most new technologies take years to develop. Most disruptive technologies might even take decades to mature. Both Envia and GM took several steps to manage technology uncertainty.

  1. Acquire Partially Developed Technologies: Less mature the technology, more risky it is likely to be. An easy approach to mitigating that risk is to find out if someone else who has solved the problem. Open Innovation is a common term used for this access. There are many challenges to accessing innovation and technology (differentiation, control, innovation valley of death), but that is a topic for another post.

    Kumar and the company’s early team perused patents and journal papers and consulted experts before settling on a promising cathode invented by Argonne National Laboratory outside Chicago. The cathode combined nickel, manganese and cobalt into an exceptional composite that astonishingly had not attracted a single licensee

  2. Modify Technologies Developed Elsewhere: If a small manufacturing or processing step can let a company build on research performed elsewhere, it can mitigate technology risk while maintaining differentiation.

    The cell reported to Arpa-E and sent to GM contained anode material purchased in a confidential deal from Shin-Etsu, a Japanese supplier. Kumar said Shin-Etsu’s role was unimportant—the anode’s true value emerged in the processing steps he had developed that allowed the anode to cycle hundreds of times without shattering.

  3. Leverage new sources for funding disruptive R&D: Many sources of funding such as government research grants may be available for funding development of disruptive / innovative technologies. However, other sources of funding can bring many distractions to a company and can potentially dilute the focus and energy.

    So Kumar applied for the Arpa-E competition in collaboration with Argonne. … It won a $4 million Arpa-E grant for the work to be carried out jointly with Argonne.

  4. Setup multiple intermediate gates: Divide development in phases and only commit resources for the next phase. Develop alternate plans to mitigate technology risks. Key challenge to deploying effective intermediate gates is: quality plans; clearly defined pass /fail criteria; and misaligned development pace/cycle across different components.

    But given the stringent deadline, every day counted. GM decided to delay publicly announcing the deal. Instead, the carmaker rushed a planning team to Newark three days later—on Dec. 3—to create a quarter-by-quarter schedule of milestones that ended with delivery of the battery technology for its two signature models. As Envia hit each quarterly target, it would receive $2 million.

Beyond setting up multiple approaches for mitigating technology risks, GM and Envia did many other things right. They set up clear deliverables:

The draft contract went on to be quite specific: For the 200-mile car, Envia was to provide a working battery delivering around 350 watt-hours per kilogram that could endure 1,000 charge-discharge cycles.

They defined a clear timeline:

Kumar’s deadline for the 200-mile battery was October 2013. After that, adjustments could be made to optimize the performance until Aug. 15, 2014. But that was a full-stop deadline—Kumar could make no changes to the battery after the latter date. This point was critical to GM because once the battery was ready, all the other deadlines could follow, ending with the pure electric car’s actual launch in 2016.

And the defined intermediate milestones:

Instead, the carmaker rushed a planning team to Newark three days later—on Dec. 3—to create a quarter-by-quarter schedule of milestones that ended with delivery of the battery technology for its two signature models. As Envia hit each quarterly target, it would receive $2 million.

Envia tested and demonstrated disruptive capabilities:

It was Kumar and his Envia team. Envia, he said, had just reported (pdf) the achievement of “the world record in energy density of a rechargeable lithium-ion battery.” It had produced a prototype car battery cell that demonstrated energy density between 378 and 418 watt-hours per kilogram. Envia said the achievement had been validated by Crane, the Indiana-based testing facility of the US Naval Surface Warfare Center, which cycled the cell 22 times.

That brings us to Number 2 on our R&D risks: Integration Uncertainties. Technology capabilities that are demonstrated in a lab environment may not translate to real world results. More importantly, as the technology is integrated with other components necessary to get it to market, new problems may arise. R&D plans and roadmaps need to clearly account for these uncertainties and allow plenty of time to address them. That did not quite happen in Envia’s case. They had less than a year to move from a hand tweaked battery to mass producible system.

Increasingly alarmed queries piled up from GM in phone calls and meetings. The Arpa-E results could not be reproduced—not by a long shot. Meeting a team from the carmaker on March 4, Kumar “struggled to allay GM’s concerns,” according to Kapadia’s lawsuit. A document provides a sense of why GM was concerned.

The team never even got to a stage where they would need to address manufacturing uncertainties (Number 3 in our list of risks). Many times we can produce small batches of widgets, but processes required for mass production may not be supported. Again, the only way to mitigate this risk is careful planning, intermediate milestones (that verify manufacturability) and adequate resources…

A year later, the deal is in tatters, GM has accused Envia of misrepresenting its technology, and a document suggests why the carmaker may be right. The startup’s unraveling is a blow for GM as it transitions to a new regime next month under CEO-designate Mary Barra, setting back its ambitions in the potentially gigantic future electric-car industry. It also risks making Envia, the recipient of several small federal grants, another punching bag for critics of US government funding of advanced battery companies.

Here is the summary: R&D is risky. Some R&D efforts will fail — and that is a good thing as it means that the organization is encouraging responsible risk taking. However, the risks should be managed carefully and failures should be caught early, before significant resources have been expended. A good R&D planning process and robust tools may help.