Statistics: Sample size - validity

From MathWiki

There are at least two criteria that need to be considered in determining a minimal sample size: power and validity. In contrast with power calculations, there is no simple way to determine the minimal sample size to achieve a reasonable assurance of validity. It depends on the nature of data and statistical relationships in the area under study. Frank Harrell (2001) Regression Modeling Strategies p. 61:

Studies in which models are validated on independent datasets <<add references>> have shown that in many situations a fitted regression model is likely to be reliable when the number of predictors (or candidate predictors if using variable selection) p is less than n / 10 or m / 20, where m is the "limiting sample size" given in [the table below]:
Type of response variable Limiting Sample Size mNotes
Continuous n (total sample size)
Binary min(n1,n2) If one considers the power of a two-sample binomial test compared with a Wilcoxon test if the response could be made continuous and the proportional odds assumption holds, the effective sample size for a binary response is 3 n_1 n_2/n \approx 3 min(n_1, n_2) if n1 / n is near 0 or 1. Here n1 and n2 are the marginal frequencies of the two response levels.
Ordinal (k categories) n - \frac{1}{n^2}\sum_{i=1}^k n_i^3 Based on the power of a proportional odds model two-sample test when the marginal cell sizes for the response are n1,...,nk, compared with all cell sizes equal to unity (response is continuous). If all cell sizes are equal, the relative efficiency of having k response categories compared to a continuous response is 1 - {1 \over {k^2}}, for example, a five-level response is almost as efficient as a continuous one if proportional odds holds across category cutoffs.
Failure (survival) time number of failures This is approximate, as the effective sample size may sometimes be boosted somewhat by censored observations, especially for nonproportional hazards methods such as Wilcoxon-type tests.

Harrell's table is based on experience with biomedical data. The standards could be different in other areas. In some fractional-factorial designs used, for example, to monitor chemical processes in industry, the number of parameters is equal to the number of observations! However, these models are highly validated by being used continuously on the same process.

PLEASE ADD TO THIS PAGE: If you are aware of a similar prescription for other areas, such as MANOVA, please add it to this page.