Difference between revisions of "Defiant: (DMRs: easy, fast, identification and ANnoTation) identifies differentially Methylated regions from iron-deficient rat hippocampus"
(→General aspects) |
(→General aspects) |
||
| Line 32: | Line 32: | ||
* The paper presents a new approach (defiant) and at the same times provides several analyses for comparing the performance of the new approach with existing algorithms. Such a study setting is very frequently found in the literature but has a high risk for biased outcomes. One reason for such a bias might be that typically application examples are selected to nicely demonstrate performance benefits. Moreover, new approaches are often established if existing methods have minor performance in a new application setup. For such a setup, a new approach then has good chances to outperform and it remains rather unclear how performance comparisons translates to new application settings. | * The paper presents a new approach (defiant) and at the same times provides several analyses for comparing the performance of the new approach with existing algorithms. Such a study setting is very frequently found in the literature but has a high risk for biased outcomes. One reason for such a bias might be that typically application examples are selected to nicely demonstrate performance benefits. Moreover, new approaches are often established if existing methods have minor performance in a new application setup. For such a setup, a new approach then has good chances to outperform and it remains rather unclear how performance comparisons translates to new application settings. | ||
* The same simulation data was used as in [27]. Therefore, the outcomes of both studies are comparable, at least with respect to the analyzed data. | * The same simulation data was used as in [27]. Therefore, the outcomes of both studies are comparable, at least with respect to the analyzed data. | ||
| − | * On top of the presented benchmark results, experimental data is investigated in the paper. | + | * On top of the presented benchmark results, experimental data is investigated in the paper. However, only qualitative comparisons were feasible for this real data sets. |
==== Design for Outcome O1 and O2 ==== | ==== Design for Outcome O1 and O2 ==== | ||
Revision as of 08:30, 4 June 2019
Contents
1 Defiant: (DMRs: easy, fast, identification and ANnoTation) identifies differentially Methylated regions from iron-deficient rat hippocampus
David E. Condon, Phu V. Tran, Yu-Chin Lien, Jonathan Schug, Michael K. Georgieff, Rebecca A. Simmons and Kyoung-Jae Won, Defiant: (DMRs: easy, fast, identification and ANnoTation) identifies differentially Methylated regions from iron-deficient rat hippocampus, 2018, BMC Bioinformatics, 19:31.
https://doi.org/10.1186/s12859-018-2037-1
1.1 Summary
The paper considers identification of differentially methylated regions (DMRs) from bisulfite sequencing data (BSSEQ). A new package (defiant) is introduced. The paper claims that shows superior performance to other approaches as shown in analyses of a series of benchmarking tests on artificial and real data.
1.2 Study outcomes
1.2.1 Outcome O1
The performance of Defiant, MethylKit, MethylSig, Metilene, RADmeth and RnBeads was compared in terms of precision, recall and false negatives. The following result is obtained:
- Metilene and Defiant clearly outperform MethylKit, MethylSig, RADmeth and RnBeads
- Sometimes precision and recall equal to zero is obtained which which is a strange outcome because the inverse predictor (i.e. predict DMRs by non-DMR calls of the algorithm) would have better performance.
Outcome O1 is presented as Figure 3 in the original publication. Since "Defiant" is occuring twice in the figure legend and RnBeads is missing, we guess that the magenta markers belong to RnBead (i.e. like the colors chosen in Fig. 4).
1.2.2 Outcome O2
The performance of Defiant, MethylKit, MethylSig, Metilene, RADmeth and RnBeads was compared in terms of the F1-score which is the harmonic mean of precision and recall. The following outcome is observed:
- Metilene is superior
- Metilene and Defiant clearly outperform MethylKit, MethylSig, RADmeth and RnBeads
Outcome O2 is presented as Figure 4 in the original publication.
1.2.3 Further outcomes
- Defiant was fastest, followed by Metilene. Therefore, both methods which showed best performance are also fastest.
- Defiant, Metilene and RADMeth required less memory (RAM) than MethylKit, MethylSig and RnBeads
1.3 Study design and evidence level
1.3.1 General aspects
- The paper presents a new approach (defiant) and at the same times provides several analyses for comparing the performance of the new approach with existing algorithms. Such a study setting is very frequently found in the literature but has a high risk for biased outcomes. One reason for such a bias might be that typically application examples are selected to nicely demonstrate performance benefits. Moreover, new approaches are often established if existing methods have minor performance in a new application setup. For such a setup, a new approach then has good chances to outperform and it remains rather unclear how performance comparisons translates to new application settings.
- The same simulation data was used as in [27]. Therefore, the outcomes of both studies are comparable, at least with respect to the analyzed data.
- On top of the presented benchmark results, experimental data is investigated in the paper. However, only qualitative comparisons were feasible for this real data sets.
1.3.2 Design for Outcome O1 and O2
- 16 "benchmark" data sets were analyzed taken from [27]
- The outcomes O1 and O2 shows analyses, where precision and recall are zero. In such a case, the inverse predictor has better performance. It might also indicate switched class labels. If this guess holds true, then the performance of MethylKit, MethylSig, RADMeth and RnBeads could be underestimated.
- It is not specified in the paper, how configuration parameters of the individual methods were chosen.
- No dependency on configuration parameters is investigated.
1.4 Further comments and aspects
None.
1.5 References
[27] Jühling F, Kretzmer H, Bernhart SH, Otto C, Stadler PF, Hoffmann S. Metilene: fast and sensitive calling of differentially methylated regions from bisulfite sequencing data. Genome Res. 2015; https://doi.org/10.1101/gr.196394.11
