Defiant: (DMRs: easy, fast, identification and ANnoTation) identifies differentially Methylated regions from iron-deficient rat hippocampus

1 Defiant: (DMRs: easy, fast, identification and ANnoTation) identifies differentially Methylated regions from iron-deficient rat hippocampus

David E. Condon, Phu V. Tran, Yu-Chin Lien, Jonathan Schug, Michael K. Georgieff, Rebecca A. Simmons and Kyoung-Jae Won, Defiant: (DMRs: easy, fast, identification and ANnoTation) identifies differentially Methylated regions from iron-deficient rat hippocampus, 2018, BMC Bioinformatics, 19:31.

https://doi.org/10.1186/s12859-018-2037-1

1.1 Summary

The paper considers identification of differentially methylated regions (DMRs) from bisulfite sequencing data (BSSEQ). A new package (defiant) is introduced. The paper claims that shows superior performance to other approaches as shown in analyses of a series of benchmarking tests on artificial and real data.

1.2 Study outcomes

1.2.1 Outcome O1

The performance of Defiant, MethylKit, MethylSig, Metilene, RADmeth and RnBeads was compared in terms of precision, recall and false negatives. The following result is obtained:

• Metilene and Defiant clearly outperform MethylKit, MethylSig, RADmeth and RnBeads
• Sometimes precision and recall equal to zero is obtained which which is a strange outcome because the inverse predictor (i.e. predict DMRs by non-DMR calls of the algorithm) would have better performance.

Outcome O1 is presented as Figure 3 in the original publication. Since "Defiant" is occuring twice in the figure legend and RnBeads is missing, we guess that the magenta markers belong to RnBead (i.e. like the colors chosen in Fig. 4).

1.2.2 Outcome O2

The performance of Defiant, MethylKit, MethylSig, Metilene, RADmeth and RnBeads was compared in terms of the F1-score which is the harmonic mean of precision and recall. The following outcome is observed:

• Metilene is superior
• Metilene and Defiant clearly outperform MethylKit, MethylSig, RADmeth and RnBeads

Outcome O2 is presented as Figure 4 in the original publication.

1.2.3 Further outcomes

• Defiant was fastest, followed by Metilene. Therefore, both methods which showed best performance are also fastest.
• Defiant, Metilene and RADMeth required less memory (RAM) than MethylKit, MethylSig and RnBeads

1.3 Study design and evidence level

1.3.1 General aspects

• The paper presents a new approach (defiant) and at the same times provides several analyses for comparing the performance of the new approach with existing algorithms. Such a study setting is very frequently found in the literature but has a high risk for biased outcomes. One reason for such a bias might be that typically application examples are selected to nicely demonstrate performance benefits. Moreover, new approaches are often established if existing methods have minor performance in a new application setup. For such a setup, a new approach then has good chances to outperform and it remains rather unclear how performance comparisons translates to new application settings.
• The same simulation data was used as in [27]. Therefore, the outcomes of both studies are comparable, at least with respect to the analyzed data.

1.3.2 Design for Outcome O1 and O2

• 16 "benchmark" data sets were analyzed taken from [27]
• The outcome of O1 might indicate switched class labels. The outcome O2 would also be in agreement with switched class labels because also F1-score equals to zero occurs. If this guess holds true, then the performance of MethylKit, MethylSig, RADMeth and RnBeads is underestimated.
• It is not specified, how wonfiguration parameters of the individual methods were chosen.

1.4 Further comments and aspects

None.

1.5 References

[27] Jühling F, Kretzmer H, Bernhart SH, Otto C, Stadler PF, Hoffmann S. Metilene: fast and sensitive calling of differentially methylated regions from bisulfite sequencing data. Genome Res. 2015; https://doi.org/10.1101/gr.196394.11