Measuring Collaboration and Why it Matters
The C-Score Suite of tools and processes, conceived by the Rapid Science team in 2013 and evolving with valued partners ever since, is designed to enable, incentivize, and reward all contributors engaging in open collaborative research.
The current reward system for researchers employs metrics based on publication in top journals, resulting in closely held incremental work and often considerable delays in reporting discoveries. Collegial feedback and formal peer review occurs when the research project is completed, absent complete data and code and documentation that cannot be improved upon, replicated or updated as new findings develop. This is exacerbated by the motivation to submit selective outcomes that offer the most striking headlines. In addition, meaningful credit to the work is attributed to a handful of investigators so that many contributors are passed over by hiring and tenure committees and funding agencies. All of these factors are widely acknowledged to slow and impede progress and fresh insights in the scientific establishment.
The C-Score Suite of Solutions
Solutions presented here address research initiatives involving multiple institutions and disciplines, where sharing results and interacting outside of one’s team and understanding is critical to achieve reliable and valuable outcomes. These mega research initiatives are increasingly being funded as disciplines become more and more specialized. However, there is a dearth of tools and processes that can support the open sharing and collaboration that’s intended.
To succeed as a collaborative venture, new strategies must be in place that do not impede researchers’ valuable time and that minimize the frustration of adopting an unfamiliar workflow. It requires well crafted policies, funding, infrastructure oversight and ongoing facilitation to ensure meaningful connections and timely exchanges of findings between team members and the wider community. There must also be incentives to collaborate and the means to assess and reward all team members for the desired behaviors.
A Case Study
Components of the C-Score Suite
Lay the Foundation
Collaboration Workspace
This is the infrastructure that enables interaction among team members, including their profiles, research goals, lab resources that can be shared, access to incremental findings and null results, and tools for discussion, review, and revision. A key component is the Research Output Management System (ROMS), a centralized methodology for tracking and recording the teams’ research findings. Standardized tagging of protocols and outputs can trigger team discussions, which can also be tracked and measured to incentivize and reward behaviors such as early output sharing, data reuse and revisions, and interactivity with peers.
Facilitate and Communicate
Editorial Facilitator and Open Narrative
Meaningful collaboration in multi-institutional transdisciplinary initiatives requires professional facilitation to encourage and ease the burden of communication among research teams who have disparate goals. In such initiatives, building a network of trust and convergence of expertise is key to innovative problem solving. To extend this trust outside of the funded project, communicating and eliciting peer review on early outcomes is a challenge that can only be met by a dedicated, trained facilitator with subject matter and editorial expertise.
The Editorial Facilitator is a funded team member – with skills and knowledge akin to editors of scholarly journals – who promotes early and frequent sharing, discussing, and revising. In addition, the EF is ideally positioned to provide a birds-eye view of how project outcomes provide new context to the current understanding of the broader field of inquiry, and vice versa. A continually updated Open Narrative, embedding snapshots of the latest findings, is maintained as the scaffold for researchers’ interactive review, periodically shared outside of the closed workspace for community feedback.
See the progression of a project’s findings from lab to workspace to preprint.
Measure and Reward Success
The C-Score: Incentivizing Collaboration
The Collaboration Score is a metric that incentivizes and rewards researchers for sharing research findings early, often, and openly; and for an array of interactions deemed to be collaborative. Both quantitative and qualitative measures are established (ideally jointly by the funder and grantees) to confer credits of varying magnitude for desired behaviors. Gaming of scores is minimized by team members’ input through surveys and interviews, and by archiving and making searchable participants’ contributions for future evaluation.