On the quantity of methods sections

So last night I discovered an extremely interesting paper, \’The First Myriapod Genome Sequence Reveals Conservative Arthropod Gene Content and Genome Organisation in the Centipede Strigamia maritima\’ by Richards and colleagues. After reading the abstract, for genomics papers I always read the methods sections next. I like to know how the authors did what they did and if I should believe it. Anyway, when I want to the methods section of this paper, I found a single paragraph which said, and I paraphrase, we assembled using the data available here, and annotated using Maker and custom perl scripts and did a bunch of other stuff. For all these details, see the supplementary.

I was in a bit of a tizzy seeing this custom perl script BS and a grossly inadequate methods section in the main text, and I tweeted this. I will say that the 53 page supplement is reasonably OK, but the custom perl scripts are not detailed.. Didn\’t we decide as a community that this was NOT cool???

A genome paper in @PLOSBiology with a 1 PARAGRAPH METHODS SECTION.. Shameful #methodsmatter http://t.co/TrLb9L1IXA

— Matt MacManes (@PeroMHC) December 8, 2014

I was a little surprised by the response – most of which was like \’Dude, is the 53 page supplemental not good enough\’

…with a clear link to a >50-page methods supplement! Yay! #methodsreallydomatter @PeroMHC http://t.co/MIz7012LXI pic.twitter.com/gbAmp5Q96t

— PLOS Biology (@PLOSBiology) December 8, 2014

So the question is, Is this sufficient?? I\’m happy to hear peoples opinions on this, but my gut tells me that it is not. I mean, if you\’re OK with this, then why not ax the entire section and put 100% of the methods in supplementary? I am not advocating that the full 53 pages of supplementary be included in the main text, but these surely is some middle ground here, right? Something between 1 paragraph and 53 pages. For me, this would be a thorough description of the key analyses the form the basis for all the other things you did. Without putting too much thought into it, maybe this would include the assembly and annotation steps. Put those in the main text, then all the details about, for instance, how you made a tree of gene X in the supplementary. I understand that for phylogenomics people that these might be the analyses they think are most interesting, but they are not as fundamental as the assembly and annotation. This is an online only journal after all, so it\’s not really like a few more paragraphs add substantial cost to the publication.

Why should these things go into the main text? There are at least a few reasons – First is that methods matter, and sad is it may be, downloading that supplementary file and opening it is a threshold many (including reviewers) won\’t cross. So, put it in the supplementary and basically no one will ever read it (see https://www.youtube.com/watch?v=5NiFibnbE8o). Isn\’t one of the primary goals of scientific publication to report the methods used to generate findings? If these methods are published in way that makes them functionally inaccessible to the majority of people, are we fulfilling this basic goal/responsibility? I kinda think we are not.. Also, there is the issue of the message it sends. One one side of big data science we have a strong push for reproducibility and detailed reporting of methods, and on the other we have methods (including the infamous custom perl scripts) that are tucked away out of sight. What message does this send? Maybe that the methods are so very straightforward that they don\’t even need to be reported? That they are boring, uninteresting, or worse not important?

So what do you guys think?

Titus Brown

1) Page limits are dumb legacies of paper journals.

2) Technically I suppose it’s OK to put details in Supplemental Material.

3) If their assembly isn’t reproducible to a fairly high degree (w/data posted), I would have hoped the reviewers would have pushed on it ’til it was.
- Matt MacManes
  
  regarding point 2, by details to you mean *all* details, or should some basic reporting occur within the main text of the manuscript?
  - Titus Brown
    
    I have to admit I don’t really see a difference… reproducibility and data availability is far more important than *where* the methods are. If all the details are available for review and reproduction, then I don’t have a strong negative reaction to them being put in the supplement. Pick your battles, I say 😉
- Harold Pimentel
  
  (1): Are limits really “dumb legacies”? While it doesn’t directly result in well-written papers, I find it often encourages me to be concise and direct. I often find myself refactoring sections over and over to make them read better. I certainly don’t want to read someones 10K word section to find the only relevant piece of information is the aligner they used. What do you propose as an alternative?
  - Titus Brown
    
    Hard page limits? I think so. Put guidelines for concision, and/or ask reviewers to address it. But limiting pages to a hard number, and limiting references to a hard number, doesn’t serve a purpose in and of itself.
Björn Brembs

If people still wanted a printed version, then they could have a “printed supplement” with a page or two summarizing the results. Online, there is no reason to shorten anything, just a lack of layout options for easy reading. For data or discussion that lie outside the main story, one could use a supplement (formerly known as ‘footnotes’), but the methods are clearly not a footnote of the research.
Harold Pimentel

Thanks for opening this up for discussion, Matt. I find this issue incredibly annoying. While a 50+ page supplement has some sense of value, the longer it is, they often result in diminishing returns.

If the summary of the result can be written in a few paragraphs, why can’t the summary of the methods section be a comparable length? Just like there are usually caveats with the results, there are certainly caveats in the analysis. Supplementary should contain the caveats, not the summary.

The other part that absolutely drives me insane is that many papers usually aren’t reviewed as well as they should be — how does this bode for an absurdly long supplement? I’m sure most reviewers never even look at the supplement, let alone read it carefully. Many of the supplementary sections I’ve read look like dumping grounds to justify that they didn’t write any methods. “What’s the main metric you’re scoring by?” “Oh, just look in this 10,000 page supplement. It’s in section 3.22.9.1.A”.