This is about measuring reuse of results. In a healthy field, results are reused, replicated, built upon and so on. In some basic views, it appears that software engineering research is not at all like this, however the authors of this paper suggest this is because you are measuring the wrong way.
Good quote: “we assert that researchers do more than write papers. Rather, we are all engaged in long-term stewardship of ideas; as part of that stewardship, we generate more than just papers. Yet, of all our prouducts, it is only our papers that are used, mostly in some annual bibliometric analysis of our worth. We view this as an inadequate way to measure what researchers do.” ok other than the excess of commas and the phrase “products” this is a paper-banger. “long-term stewardship” as the primary of activity of researchers. yes.
usually “bibliometric” studies just mean whatever is stored on professional databases. it’s a case of the object being reduced to the part that is easy to observe/query with an api. this paper proposes a methodology for also measuring using guidelines from one paper, or locally using code from a third.
a full-scale citation analysis is a lot of work, as are other ways of measuring this reuse. instead, they have made a collaborative ‘reuse-graph’ where researchers can submit what work they re-use and then it is associated into a large graph. they are checked, validated by the community (via github…).
uses include: - academics can check their contributions are being properly recorded - when applying for promotions or new positions, this provides evidence other than citations for your work - graduate students can direct their attention to areas that are new (recent nodes) and productive (nodes with an unusually large number of edges) - organisers of conference can select speakers from who is contributing a lot - growth patterns - okay they’re getting capitalist/techy about it now. anywayyyyy – it is a good idea! but academia/tech is not.
The graphs were produced in a pretty interesting way: journals/proceedings were identified and these were grouped into packets of 10, had github issues associated, and these issues were self-assigned by various groups who read the papers for examples of reuse. Two people examined each paper independently. Then disagreement was checked/controlled for before being included in the graph.
The sorts of reuse considered were: - new ideas are benchmarked against the previous state-of-the-art, reusing it - guidance on how to correctly apply statistics to your field is reused - metrics and methodology that are specific to a research area - datasets - sense checks, or best practices - tools and applications used, codes. there are certainly more sorts of reuse that could be considered too.
the researchers found that this method worked, was reasonably quick to identify these kinds of reuse and disagreement was low (where reviewers were reasonably experienced).
oh hey! they mentioned ‘severe’ tests. that’s a blast from my past (c/o deborah mayo)
other fields have lots of reuse, in physics and astromony, sharing datasets is extremely common (because our experiments are expensive and centrally managed independent of people who use the data). preprint servers enable a kind of reuse. in machine learning, reuse comes mostly from use of the previous-best as a stepping stone. in some fields, there is increasing trend towards sharing aertifacts (ie, codes) along with your papers. it is often found that these codes are not widely used, at least in their entirety. this is not particularly surprising tbh.
awww, it seems this project died a death and has not really looked at many papers since. they suggest ways to encourage reuse in the field, an index rewarding researchers who reuse or produce reused research.
i like this framing of academia. stewards of ideas.
this is a really cool idea. good application of open science. i hope it goes well/goes somewhere? what would this look like in mri? idk lol. i’ve reused gras’ and orzada’s work. that’s reuse. i think the ideas in this paper would even help a lot of the issues in science, and that the ways they suggest to work on/address it are realistic and could work. they should do that!