So last night I discovered an extremely interesting paper, \’The First Myriapod Genome Sequence Reveals Conservative Arthropod Gene Content and Genome Organisation in the Centipede Strigamia maritima\’ by Richards and colleagues. After reading the abstract, for genomics papers I always read the methods sections next. I like to know how the authors did what they did and if I should believe it. Anyway, when I want to the methods section of this paper, I found a single paragraph which said, and I paraphrase, we assembled using the data available here, and annotated using Maker and custom perl scripts and did a bunch of other stuff. For all these details, see the supplementary.
I was in a bit of a tizzy seeing this custom perl script BS and a grossly inadequate methods section in the main text, and I tweeted this. I will say that the 53 page supplement is reasonably OK, but the custom perl scripts are not detailed.. Didn\’t we decide as a community that this was NOT cool???
— Matt MacManes (@PeroMHC) December 8, 2014
I was a little surprised by the response – most of which was like \’Dude, is the 53 page supplemental not good enough\’
— PLOS Biology (@PLOSBiology) December 8, 2014
So the question is, Is this sufficient?? I\’m happy to hear peoples opinions on this, but my gut tells me that it is not. I mean, if you\’re OK with this, then why not ax the entire section and put 100% of the methods in supplementary? I am not advocating that the full 53 pages of supplementary be included in the main text, but these surely is some middle ground here, right? Something between 1 paragraph and 53 pages. For me, this would be a thorough description of the key analyses the form the basis for all the other things you did. Without putting too much thought into it, maybe this would include the assembly and annotation steps. Put those in the main text, then all the details about, for instance, how you made a tree of gene X in the supplementary. I understand that for phylogenomics people that these might be the analyses they think are most interesting, but they are not as fundamental as the assembly and annotation. This is an online only journal after all, so it\’s not really like a few more paragraphs add substantial cost to the publication.
Why should these things go into the main text? There are at least a few reasons – First is that methods matter, and sad is it may be, downloading that supplementary file and opening it is a threshold many (including reviewers) won\’t cross. So, put it in the supplementary and basically no one will ever read it (see https://www.youtube.com/watch?v=5NiFibnbE8o). Isn\’t one of the primary goals of scientific publication to report the methods used to generate findings? If these methods are published in way that makes them functionally inaccessible to the majority of people, are we fulfilling this basic goal/responsibility? I kinda think we are not.. Also, there is the issue of the message it sends. One one side of big data science we have a strong push for reproducibility and detailed reporting of methods, and on the other we have methods (including the infamous custom perl scripts) that are tucked away out of sight. What message does this send? Maybe that the methods are so very straightforward that they don\’t even need to be reported? That they are boring, uninteresting, or worse not important?
So what do you guys think?