Forget the format, focus on the functions: research evaluation, re-evaluated

This week I’ve been inspired and challenged by discussions at the inaugural Sydney Conference organised by JoAnne Sparks at Macquarie University. It was my first unconference; a group of approximately 100 people loosely identified as researchers, librarians or (very much in the minority) publishers and service suppliers divided itself into 5 threads: peer review, “containers”, barriers, roles and data management. Tough call – the concept rested on us sticking in one thread for all three days, however much we wanted to participate in all of them. On the basis that it was a “no passengers” affair and I was going to be expected to contribute some kind of knowledge, I opted for “containers” (“it’s not all about the packaging” – what “units” of scholarly content should we value or measure?)

Our discussions were broadly shaped by the Merlin approach: visualise the future, then work backwards to figure out how to progress towards it. Before this method was revealed, one of our group unwittingly proposed the year 2472 as our starting point. It was almost impossible to imagine such a far-off time, and we found ourselves diverted into discussions about climate change, resource scarcity, changing alliances and enmities, greed, human nature … I say “diverted” because at first I was champing at the bit to pull the discussions back to the scholarly information microcosm. But I came to recognise the value in the “diversion”: it helped us learn more about each other’s viewpoints, and it loosened up our thinking by taking us so dramatically beyond today’s status quo. It also brought into play some of the concepts of “Seventh Generation Sustainability” – thinking far ahead to people we won’t meet, and considering what they would make of our discussions and decisions.

We boiled down some interesting ideas about why people do research (in the future, it will be edutainment – what if we treat it that way now?), and the forms in which knowledge is created, shared, evaluated and preserved (collections of both facts and multi-media / multi-sensual expressions of experience – broken down to the lowest component part, nanopublications tagged for flexible regrouping into different formats). While focussing our discussions “beyond the paywall” in line with the conference’s overarching theme, we considered “ownership” of these components, particularly (for example) when the “raw data” might be a person and their story. We talked about the need to avoid walled gardens (disciplines, publishers, business models, political frameworks, geography, cultural silos and so on). We dreamed of metadata baked into the very fabric of every type of research output as part of the process of its creation (à la microdots?), to enable structuring and navigation of knowledge. We thought about how to experience research from different perspectives, like testing software in Browserstack, or re-playing games as different characters. We oscillated back and forth on authority and crowdsourcing, and settled for something in the middle.

Our theme became “anti-containers”. We envisaged research outputs as a dashboard of tiles – for data, for multimedia sources and expressions, for interpretation, for translation, for validation. We anticipated ongoing iterations, with timestamped versions to capture milestones or other points in time, and the ability for open participation – here referencing Github. We recognised that whatever forms and processes prevail, people will always need mentoring to learn how best to express and share knowledge. We acknowledged the need for research to convey itself to a range of audiences, and the respective roles of plain language and field-specific language as well as translation beyond the researchers’ own native tongue(s), and expressions beyond the textual / digital / verbal.  We touched on the importance of broad definitions of, and therefore diverse measurements of, impact – when asked what headline we’d like to see in ten years’ time, we finally chose “Research breaks the internet”.

We concluded that the processes and functions of scholarly communications are the only manner of container that is required, i.e. it doesn’t matter what format work is presented in, if it can meet some or all of the criteria below. These are based on the “forces and functions in scientific communication” proposed by Roosendaal and Geurts and referenced by Van de Sompel et al. We emphasised that we don’t see them as a linear process; any function can happen at any time, and at multiple times, and iteratively (with acknowledgement of versioning).

Registration – Certification – Awareness – Re-use – Archiving – Recognition

These elements could (should?) be decoupled, and need not all happen within one vehicle. Finally, we talked about badges as a means of demonstrating fulfilment of each criterion. We acknowledged that much of our conversation reflected existing projects, declarations, products / services and ideas and determined that our next step would be to liaise with those existing “stakeholders” to further progress our proposed framework into a functional model / matrix for evaluating container-less research.


Big thanks to all the participants, and especially the thread 2 moderators JoAnne Sparks and Sherman Young, for such a fascinating and thoughtful few days.

3 thoughts on “Forget the format, focus on the functions: research evaluation, re-evaluated

  1. Thanks Charlie – I enjoyed your comments both in the sessions and now this response.
    I found myself thinking about all this post-non-conference and coming back to the value of the container. We like Basho’s haikus and Shakespeare’s sonnets because they are haiku’s and sonnets. We don’t expect to find the craft, skill and insight of the poet somewhere else when we are reading the poem. And they got such joy from producing a good one. As a researcher, I quite like writing to the disciplined from of the journal article or the book chapter (not solely – I do enjoy singing at students, and the nourishing and much less structured conversations of PhD supervision sessions for example), and I wondered if we might have been a bit harsh on particular forms in our enthusiasm for getting our heads around process and relationship.
    I am quite chuffed when I produce a good journal paper. And I’m sometimes an ‘abstract thinker’ – I try to capture a whole thought in the form of a title and abstract for an imagined future paper and file it away. These ideas that I capture as an abstract are bigger and more refined than a tweet, but less refined and coherent than a manuscript ready for submission. I put them away for later attention and development (and transformation). Many of them languish … but some come to fruition.
    I like words, but I recognise that they are (some of) the raw materials for thought and proposition. I don’t mistake them for the finished product of scholarship. Today I’ve serendipitously stumbled upon Guattari’s use of ‘dissensus’ in The Three Ecologies and been led on to thinking about the value of dissent in complex and pluralist and creative scholarship. In the conference someone warned of throwing out babies when we clean out the bathwater … and I’ve reflected on the risks of keeping all the bathwater in order to be able to reconstruct the baby’s momentary embodied experience of bathing. While we would be awed by the value of Monet’s painting palette as an artefact, perhaps we should not mistake it for the finished paintings (plural and diverse).
    What ever the containers, the gift of scholarship – the deliberate and considered scholarly communication – should be valued and treasured and encouraged.
    Thanks for your provocations.

    • Hi Richie, thanks for your thoughts (how great that we can continue our discourse despite the thousands of miles that now separate us!) What a good point you make, that sometimes the structure of a particular form can be helpful to developing our thinking, expression. I have experienced this too and now that you bring the thought to mind, I fully agree that it’s a strong reason not to be too hasty in abandoning the current forms.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.