Forget the format, focus on the functions: research evaluation, re-evaluated

July 24 2015 / By Charlie Rapple

This week I’ve been inspired and challenged by discussions at the inaugural Sydney Conference organised by JoAnne Sparks at Macquarie University. It was my first unconference; a group of approximately 100 people loosely identified as researchers, librarians or (very much in the minority) publishers and service suppliers divided itself into 5 threads: peer review, “containers”, barriers, roles and data management. Tough call - the concept rested on us sticking in one thread for all three days, however much we wanted to participate in all of them. On the basis that it was a “no passengers” affair and I was going to be expected to contribute some kind of knowledge, I opted for “containers” (“it’s not all about the packaging” - what “units” of scholarly content should we value or measure?)

Our discussions were broadly shaped by the Merlin approach: visualise the future, then work backwards to figure out how to progress towards it. Before this method was revealed, one of our group unwittingly proposed the year 2472 as our starting point. It was almost impossible to imagine such a far-off time, and we found ourselves diverted into discussions about climate change, resource scarcity, changing alliances and enmities, greed, human nature … I say “diverted” because at first I was champing at the bit to pull the discussions back to the scholarly information microcosm. But I came to recognise the value in the “diversion”: it helped us learn more about each other’s viewpoints, and it loosened up our thinking by taking us so dramatically beyond today’s status quo. It also brought into play some of the concepts of “Seventh Generation Sustainability” - thinking far ahead to people we won’t meet, and considering what they would make of our discussions and decisions.

We boiled down some interesting ideas about why people do research (in the future, it will be edutainment - what if we treat it that way now?), and the forms in which knowledge is created, shared, evaluated and preserved (collections of both facts and multi-media / multi-sensual expressions of experience - broken down to the lowest component part, nanopublications tagged for flexible regrouping into different formats). While focussing our discussions “beyond the paywall” in line with the conference’s overarching theme, we considered “ownership" of these components, particularly (for example) when the “raw data” might be a person and their story. We talked about the need to avoid walled gardens (disciplines, publishers, business models, political frameworks, geography, cultural silos and so on). We dreamed of metadata baked into the very fabric of every type of research output as part of the process of its creation (à la microdots?), to enable structuring and navigation of knowledge. We thought about how to experience research from different perspectives, like testing software in Browserstack, or re-playing games as different characters. We oscillated back and forth on authority and crowdsourcing, and settled for something in the middle.

Our theme became “anti-containers”. We envisaged research outputs as a dashboard of tiles - for data, for multimedia sources and expressions, for interpretation, for translation, for validation. We anticipated ongoing iterations, with timestamped versions to capture milestones or other points in time, and the ability for open participation - here referencing Github. We recognised that whatever forms and processes prevail, people will always need mentoring to learn how best to express and share knowledge. We acknowledged the need for research to convey itself to a range of audiences, and the respective roles of plain language and field-specific language as well as translation beyond the researchers’ own native tongue(s), and expressions beyond the textual / digital / verbal.  We touched on the importance of broad definitions of, and therefore diverse measurements of, impact - when asked what headline we’d like to see in ten years’ time, we finally chose “Research breaks the internet”.

We concluded that the processes and functions of scholarly communications are the only manner of container that is required, i.e. it doesn’t matter what format work is presented in, if it can meet some or all of the criteria below. These are based on the “forces and functions in scientific communication” proposed by Roosendaal and Geurts and referenced by Van de Sompel et al. We emphasised that we don’t see them as a linear process; any function can happen at any time, and at multiple times, and iteratively (with acknowledgement of versioning).

Registration – Certification – Awareness – Re-use – Archiving – Recognition

These elements could (should?) be decoupled, and need not all happen within one vehicle. Finally, we talked about badges as a means of demonstrating fulfilment of each criterion. We acknowledged that much of our conversation reflected existing projects, declarations, products / services and ideas and determined that our next step would be to liaise with those existing “stakeholders” to further progress our proposed framework into a functional model / matrix for evaluating container-less research.


Big thanks to all the participants, and especially the thread 2 moderators JoAnne Sparks and Sherman Young, for such a fascinating and thoughtful few days.

Enter your email address to receive notifications of new posts by email.