Gene set analysis methods: a systematic comparison

1 Citation

Mathur, R., Rotroff, D., Ma, J., Shojaie, A., & Motsinger-Reif, A. , Gene set analysis methods: a systematic comparison, 2018, BioData mining, 11(1), 8.

Permanent link to the paper

2 Summary

Approaches for gene set analyses were assessed by using simulated data that were generated based on a real experimental data set.

There are competitive tests (COMP) that uses the distribution of a reference gene set (e.g. all gene that are not in the gene set) as reference and self-contained (SELF) approaches that do not rely on a reference.

• The authors compared four different methods:
  • Gene Set Enrichment Analysis (GSEA-SELF and GSEA-COMP)
  • Significance Analysis of Function and Expression (SAFE) based on the t-test as gene-wise test and offers Wilcoxon rank sum, Fisher’s Exact Test, Pearson’s Chi-squared type statistic and a t-statistic as global (gene set wide) tests
  • sigPathway
  • Correlation Adjusted Mean Rank (CAMERA)

3 Study outcomes

3.1 Outcome O1: False positives under null distribution

The frequency of false-positives was assessed by using a significance level alpha=0.05. Consequently all approaches (except FET-1k) showed around 5% false-positive or less. FET-1k ("SAFE with Fisher's Exact test as global statistic") had around than 20%.

Outcome O1 is presented as Figure 2 in the original publication for the prostate data template and in the "Additional File 1" for the other templates.

Baseline of this outcome is that all approaches excep FET-1k perform similarly well in terms of false-positives.

3.2 Outcome O2: GSEA vs. SAFE vs. sigPathway vs. CAMERA

• sigPathway showed superior performance
• The order in terms of performance seems to be sigPathway > SELF-GSEA-FDR > SELF-GSEA-Q > COMP-GSEA-FDR > SELF-GSEA-Q > CAMERA > SAFE-Wilcoxon
• SAFE-Wilcoxon could NOT detect any differentially regulated pathway(s)
• In general, the performance increases with increasing fraction of regulated genes (parameter \pi in the paper). Similarly, it also increases with increasing effect size \tau. However, "COMP GSEA Q" shows counterintuitive dependency of performance on effect size.

Outcome O2 is presented as Figure 3 in the original publication and in supplemental figures. The numbers are also provided in the supplement.

3.3 Outcome O3: Power of SAFE for different configurations

• SAFE again performs weak for most configurations
• Only "aveDiff-boot" seems to have a good power that also improves with increasing magnitudes \tau of the regulation
• FET-1k, FET-10k (SAFE with Fisher's Exact Test) work partly: both approaches could identify the regulated pathway but the show counterintuitive performance (i.e. decreasing performances for increasing magnitudes of regulation)

Outcome O3 is presented as Figure 4 in the original publication.

3.4 Outcome O4: Power of GSEA

• COMP-GSEA-FDR and Self-GSEA-FDR showed superior performance
• Comp-GSEA-Q and SELF-GSEA-Q showed counterintuitive performance, i.e. the performance deceases with increasing effect size \tau
• Overall, the performance seems to be: SELF-GSEA-FDR > COMP-GSEA-FDR > SELF-GSEA-Q > COMP-GSEA-Q

3.5 Outcome O5: Overlapping gene sets

Some genes may occur in several gene sets. For gene sets that overlap with the target gene sets, the power only behaves most reasonable for COMP-GSEA-FDR.

Here, reasonable means that and increasing effect size and an increasing proportion of regulated genes should lead to and increasing power.

4 Study design and evidence level

4.1 General aspects

• The authors consider different sizes of the gene sets
• The authors consider different proportions of regulated genes in the gene sets
• The authors consider different magnitudes of the underlying effect size (i.e. log-fold-changes)
• The authors consider three null simulations (without regulation) as reference for outcome O1
• In this publication, the authors published a novel simulation approach termed (FANGS)
• The simulation approach is available in this R package (FANGS) offers the opportunity to reproduce the simulations and repeat the analysis for other gene set methods.
• The authors provide a comprehensive list of the used configuration parameters
• The authors evaluated the following alternative configurations
  • For GSEA one alternative
  • For SAFE five alternative setups
  • For sigPathway and CAMERA no other configurations were considered
• Three experimental data sets were used as foundations for simulating data
  • prostate cancer (264 cases, 160 controls)
  • ischemic stroke (20 cases, 20 controls)
  • normal brain tissue (21 cases, 20 controls)

4.2 Design for Outcome O1

• The authors consider three null simulations (without regulation) as reference:
  • permutation of class labels
  • independently sampled expression of all features (=genes)
  • centering the simulated data, i.e. set effect size to zero
• Default configuration parameters and the alternative parameters described above were evaluated
• Only the prostat cancer data set was considered as template for simulations

4.3 Design for Outcome O2 and O4

For GSEA, only GSEA-Q is shown in the main figure 2 (prostate cancer). In contrast, the supplemental figures also show the results for GSEA-FDR (stroke and brain data sets). For outcome O4 also both, GSEA-Q and GSEA-FDR were considered (and GSEA-FDR outperformed GSEA-Q). Figure 4 contains the missing comparison of GSEA-Q and GSEA-FDR

• In total, six analyses were performed (3 data sets x 2 regulated pathways):
  • One simulated data set is based on simulating differential expression of one pathway
  • The analysis was repeated for all three data sets as template
  • For each of the three data sets the analysis was repeated by selecting two different pathways as differentially regulated.
• Default configuration parameters were chosen

4.4 Design for Outcome O3

• The weak performance of SAFE for the default configuration in O2 seems to be the motivation for investigation of other configurations for SAFE
• The outcome O3 was only generated for one data set (prostate cancer) and two regulated pathways

4.5 Design for Outcome O5

At least to my understanding, the discussion and interpretation of this outcome is not very clear in the paper.

5 Further comments and aspects

• Gene sets from MSigDB were used
• The authors are aware of the fact that different null hypotheses are tested by the different approaches
• sigPathway and CAMERA offers other options that are discussed in the article but not evaluated. For SAGE, multiple setups were investigated.