Download TSA user manual - Copenhagen Trial Unit

Transcript
User Manual for TSA
Copenhagen Trial Unit 2011
Pr(|Z|≥c | H0 is true) ≤ α
(2)
Pr(|Z|=c | H0 is true) = α
(3)
For the remaining theoretical sections on repeated significance testing
(sections 2.2.2 to 2.2.5), we will assume that all statistical tests are two-sided.
We will also assume that all test statistic values, Z, are absolute values. We
assume the latter because the involved algebra becomes much simpler by
doing so. For example, in defining two-sided thresholds for a non-absolute
test statistic, one would need to consider the probability that Pr(Z≤-c or Z≥c |
... ) rather than Pr(|Z|≥c | ... ).
Problems with repeated significance testing
Conventional single significance tests can be considered reliable if ‘enough’
data has accumulated. In meta-analysis, a single significance test can be
considered reliable once the required information size is surpassed.14;6;11;12;20;59
If we perform a single test for statistical significance at or after a
meta-analysis has surpassed its required information size, statistical
significance testing simply entails determining an appropriate threshold, c,
that will make equations (2) and (3) true. For example, for α=5% we would
consider c=1.96 appropriate if the meta-analysis data had not previously been
subjected to significance testing.
When a cumulative meta-analysis is subjected to significance testing more
than once (before surpassing its required information size), the situation
becomes more complex. Consider the example where a meta-analysis is
updated once and where the conventional 5% maximum type I error is used.
In this situation, the first meta-analysis yields a Z-value, Z1, and the metaanalysis update yields another, Z2. If the first meta-analysis yields a Z-value
larger than 1.96, the two investigated interventions are declared significantly
different. However, if the first meta-analysis is not significant (i.e., Z1<1.96),
the two interventions can still be declared statistically significant if the metaanalysis update yields a Z-value larger than 1.96 (i.e., if Z2≥1.96). By the laws
- 36 -