I asked a colleague what he wanted out of a collaboration project, and he said “A good relationship.”
“What does good mean?” I asked. “How will we know when we’ve gotten there?”
He skewered me with an expression that reminded me that he’s quite a lot senior to me, and he said: “We’ll know because I’ll tell you.”
He’s probably right in many ways. He’s very experienced, and he’s got a feel for “good.” But what about the rest of us? How do we know when a research collaboration is delivering on its objectives? Especially when the objectives don’t involve money but more ephemeral things like prestige and reputation or credibility?
Since I’m on the industry side, the research collaborations I work on fall in three different categories
- Relationships and image building: Projects where we want to contribute/support in to the sector and be seen to be doing the right thing. These are things like open data infrastructure projects, research integrity projects and big EU/NIH projects.
- Credibility building: Projects where we want to show how great the company’s data or analytical capabilities are. So we might do pro-bono analysis for government reports or give away data for academic research.
- Capability building: Projects where we want to learn how to do something or give advice on improving a product.
And for all of these, we’ve distilled the potential measurable outcomes of research collaborations. There are really only six.
- Thought leadership and executive engagement: These are things like board meetings, executive visits and round tables.
- Media: This could be press releases, or articles in our online magazine, articles in our partner’s publications or general media coverage. Yes, these should probably be weighted, based on their reach/prestige, but I haven’t got there yet.
- Research Results: These are academic articles and conference proceedings – things like that.
- Sales: The company I work for is not generally looking for licensing income, they’re looking to sell/license data, information, books, journals, solutions and services. A sale is not a usual outcome of a research collaboration, but if we’re beta-developing a product or service together, it could be.
- IP/product implementation: This is of course the Holy Grail. A research project with an outcome that’s actually useful in a product. Sometimes the collaboration outcome is straightforward advice and feedback on a product or service. Sometimes it’s a Result that we’d license, like that magical machine learning algorithm that changes the world.
- Talent recruitment: We do a lot of projects with graduate students and postdocs. Having the opportunity to snap them up when they’re looking for employment is a big win.
I’ve found those six buckets pretty much do it all. My office keeps track of the outcomes of projects (just a tally at the moment) and that helps us track the outcomes per project. It’s not perfect, because of course these aren’t weighted. Who’s to say that one research article is more impactful than one board meeting? But at least we can easily identify under-performing projects and pay them more attention. And we can assess if the kinds of outcomes we’re getting are aligned with the objectives of the collaboration. We’d expect more media and thought leadership/executive engagement from research projects where we’re hoping to build up relationships and image. We expect more IP outcomes from projects where we’re co-developing technology and building capabilities.
Which brings me to planning
The point of the ROI exercise is not to scrutinize the project constantly. After all, research projects can take years to bloom. And some can yield outcomes long after they’re complete. Besides, the investment on the company side is really in time and data more than money. But by keeping tabs on those projects that are not producing anything, we can avoid Black Hole Collaborations, where we give something (data, analysis, funding) and…silence. The outcome tracking system keeps a bit of accountability on both sides and makes sure that there aren’t unaddressed issues.
The point of the ROI exercise is not to judge the project or pull the plug early. The point is to ensure that the project is being nurtured, and that everyone’s getting out of it what they want. We don’t look at the cost-per-outcome (we can, but it’s usually not very helpful), but we look at the return compared to the objectives. As long as the outcomes are in the areas the participants are hoping for, we’re happy.
We do a planning cycle yearly for the main projects, checking on the project’s objectives and planning the kinds of outcomes we’d like to have. It’s far more useful than just checking in if the relationship is “good” and the project is “successful.”
The outcome tracking system will continue to evolve in our office and heck, it may not work for you at all. But it gives us an internal framework to discuss. And, if pressed, I can always add a note in the project database that says “Sponsor says ROI = ‘good’.”