Heather's attempt at link

From MathWiki


Review of the multivariate normal: marginal and conditional expectation and variance, the concentration ellipse for the bivariate normal, Spectral Decomposition Theorem. Linear regression: confidence regions and intervals for β, relationship to Var(X) Introduction to the theory of mixed models

A First Look at Multilevel and Longitudinal Models (http://www.math.yorku.ca/~georges/Slides/CourseNotes.pdf) pp 1-51

Re Simpsons' and Robinson's Paradox: See the first few pages of Some practical issues applying mixed longitudinal models for observational data (http://www.math.yorku.ca/~georges/Slides/TalkOnContextualEffectsv2.pdf)

Introduction to the Analysis of Hierarchical and Longitudinal Data - Part 1 (http://www.math.yorku.ca/~georges/Slides/IntroHierLong-1.pdf) pp 1-42

On Friday, we had a look at Pine Trees, Comas and Migraines: Asymptotic functions of time (http://www.math.yorku.ca/%7Egeorges/Slides/TalkOnComasAndMigraines.pdf) to illustrate multilevel models (longitudinal models in this case) that have either a continuous normal response or a dichotomous response.

We also considered the difference between fixed effects and random effects classification models discussed in Chapter 2 of the text and in pp 52-60 of A First Look at Multilevel and Longitudinal Models (http://www.math.yorku.ca/~georges/Slides/CourseNotes.pdf).

  • Relationships between models and contextual effects:
Mixed Models -- Behind the Scenes (http://www.math.yorku.ca/~georges/Slides/HandoutM.pdf)
Contextual Effects (http://www.math.yorku.ca/~georges/Slides/TalkOnContextualEffectsv2.pdf)
  • R scripts
MATH 6643 hsintro.R
MATH 6643 hsgraph.R

  • Diagnostics for Mixed Models:
Mixed Model Influence Diagnostics by Oliver Schabenberger, SAS Institute Inc., Cary, NC (http://www2.sas.com/proceedings/sugi29/189-29.pdf)
Judith Singer on SAS PROC MIXED (http://www.jstor.org/view/10769986/ap040018/04a00030/0)
  • Plan for Friday, June 14:
Finish proof re Mixed Model with contextual effects.
Fitting models MATH 6643 hsdetail.R GM comment: Fix this
Interpreting T (http://www.math.yorku.ca/~georges/Slides/NMakingVariance.pdf)
Longitudinal models A First Look at Multilevel and Longitudinal Models: A second look ... pp.67ff (http://www.math.yorku.ca/~georges/Slides/CourseNotes.pdf)
Interpreting R output:
Introduction to Hierachical and Longitudinal Models Part 1 (http://www.math.yorku.ca/~georges/Slides/IntroHierLong-1.pdf)
Introduction to Hierachical and Longitudinal Models Part 2 (http://www.math.yorku.ca/~georges/Slides/IntroHierLong-2.pdf)
Longitudinal models A First Look at Multilevel and Longitudinal Models: A second look ... pp.86ff (http://www.math.yorku.ca/~georges/Slides/CourseNotes.pdf)
Introduction to non-linear models:
Comas and Migraines: Non-linear models for time (http://www.math.yorku.ca/~georges/Slides/TalkOnComasAndMigraines.pdf)
PROC NLMIXED Summary (http://www.math.yorku.ca/~georges/Slides/PROC%20NLMIXED%20SUMMARY.pdf)
NLME from José C. Pinheiro
User's guide (http://stat.bell-labs.com/NLME/UGuide.pdf)
Help files of NLME function (http://cm.bell-labs.com/cm/ms/departments/sia/project/nlme/HelpFunc.pdf)

FROM the Workshop Wiki


This course uses classical repeated measure, univariate and multivariate, as a point of departure for studying methods for the analysis of longitudinal and hierarchical data using mixed models. Mixed models allow the analysis of repeated measures data when the data are ‘unbalanced’ and classical models do not work, e.g. subjects are observed at different times or time-varying covariates are included in the model. The ability to analyze a wider range of data comes at a price. Not only do you need to learn new techniques, you also need to become aware of concepts that are not as salient in the analysis of ‘balanced’ data.

The course will emphasize the visualization of the basic concepts to help you develop a good understanding of the strengths and limitations of these methods. The proposed list of topics includes: Classical univariate and multivariate repeated measures models, extensions to mixed models. The structure of the linear mixed model: fixed effects, random effects, variance and covariance components. How mixed models are used to fit longitudinal data. Statistical control with observational data. Borrowing strength, shrinkage and bias in random effects models. Contextual versus compositional effects. Model building and diagnostics. Consequences of measurement error and approaches to adjustment. Modelling correlation. Missing data patterns. Modelling panel attrition. Logistic regression for binary responses. Non-linear models for binary and categorical responses. Day 1: October 15

   * Why go beyond repeated measures? What can we do with longitudinal data analysis?
   * Some concepts in multiple regression.
   * Preparation for day 2.

Newer methods are needed for:

1. Balanced data with missing occasions 2. Observations at irregular times: recovery of IQ after brain injury: Pine trees, comas and migraines 3. Time-varying covariates: Schizophrenic symptoms over time (p. 18) 4. Non-normal outcome Daily subject log recording presence or absence of migraine: Pine trees, comas and migraines (p. 93ff).

   * Classical methods vs. mixed models:
         o Mixed models give greater flexibility: missing data, time-varying covariates, different times for different subjects.
         o At cost of having to understand possible consequences
         o See UCLA:Longitudinal Data Analysis Using Multilevel Models

Day 1: Unorthodox review of regression This material is intended to help visualize some statistical concepts in regression that are useful for longitudinal data analysis.

   * Normal contours (Population and sample variance ellipse: regression to mean  Visualizing Regression: Correlation and Regression (first section only)
   * Simple vs. multiple regression: 
         o Visualizing Regression: Multiple Regression and R scripts:
               + VisualizingMultipleRegressionPart1.R
               + VisualizingMultipleRegressionPart2.R
               + VisualizingMultipleRegressionPart3.R
         o More more on coffee, heart damage example: Visualizing Regression: Correlation and Regression (p. 159)
         o More on height and weight example: Visualizing Regression: Correlation and Regression (p. 190)
         o Topics:
               + Variance ellipse, data ellipse, confidence ellipse
               + Where confidence intervals come from?
               + Model selection: statistical fit or based on theory?
               + Strategies for model selection: 2 types of data and 2 purposes of statistical analysis:
                     # Types of data: observational or experimental
                     # Purposes of analysis: prediction or causal
                     # Statistical vs theoretical considerations
               + Consequences of measurement error
               + Leverage and influence, multivariate outliers
   * Interpreting main effects with interactions
         o Parameter estimates and SAS Type III sums of squares with numerical and class predictors and handout
   * Review:
         o UCLA online seminar: Regression with SAS
   * Links:
         o SAS input file for artificial coffee data: coffee.sas
         o SAS input file for artifical height and weight data: hw.sas

Preparation for Day 2

 Materials on multilevel models:
   * A First Look at Hierarchical and Longitudinal Data Analysis
   * UCLA web movie on Repeated Measures vs Mixed Models
   * UCLA: Online seminar on Repeated Measures And Longitudinal Analysis in SAS (with movies)
   * UCLA: Online seminar on Repeated Measures and Longitudinal Analysis in SPSS

Day 2: October 22

   * Classical repeated measures with SAS
   * Analyzing these models with PROC MIXED
   * Introduction to mixed models: "A First Look at Multilevel and Longitudinal Models"
   * Multilevel models to 'composite models'
   * What mixed models really estimate
   * Analyzing longitudinal data using mixed models
   * Analysis using hierarchical (e.g. HLM) vs MIXED (See Analyzing Longitudinal Models using Multilevel Modeling: Part 3) [mixed models called 'composite models']
   * Numerical exercise in SAS:

For an introduction to repeated measures, we take advantage of the excellent web materials on using SAS for repeated measures prepared at UCLA. A parallel but less developed document discusses using SPSS for repeated measures.

UCLA: Repeated Measures Analysis in SAS (with movies) [Excellent if only for the medium] UCLA: Repeated Measures Analysis in SPSS Notes on Mixed and Longitudinal Models : A First Look at Multilevel Models


Wolfinger and Chang: "Comparing the SAS GLM and MIXED Procedures for Repeated Measures"

Day 3: October 29

   * We continue studying mixed models in "A First Look at Multilevel and Longitudinal Models"
   * SAS workshop slides
   * Additional notes for section 8.1 can be found in "What a mixed model really estimates"
   * Additional notes for section 8.2 can be found in "Variance -- Making T simpler"
   * Interpreting T, the variance matrix for random effects
   * Contextual and compositional effects. Role of contextual variables.
   * Age, cohort effects.: Miyazaki, Raudenbush (2000) "Tests for Linkage of Multiple Cohorts in an Accelerated Longitudinal Design," Psychological Methods, 5, 44-63
   * For data and additional notes, go to notes on the Wiki page for the course.

Day 4: November 5

   * Self-extracting data sets -- download each file into a convenient directory and run it to create SAS data sets.
         o Panel Study of Income Dynamics: psid.exe
         o Coffee-Stress example: coffee.exe
         o Migraine example (partial data): migraine.exe
         o IQ recovery example (partial data): iq.exe
         o Pothoff and Roy jaw growth data: pr.exe
         o High School Math Achievement and SES: hs.exe
   * Longitudinal models in "A First Look at Multilevel and Longitudinal Models"
   * SAS PROC MIXED: New ODS GRAPHICS output in Version 9: Version.doc

Day 5: November 12

   * Non-linear models with PROC NLMIXED:
         o Introduction to non-linear models for longitudinal data: Pine Trees, Comas and Migraines
         o Summary: Modeling Change in Time
         o Summary: Periodic functions of time
         o Logistic longitudinal regression: Modeling Binary Outcomes
         o Using PROC NLMIXED: Summary notes on PROC NLMIXED for non-linear and logistic regression
   * Power calculations and design of longitudinal studies:
         o Latest version of PinT: http://stat.gamma.rug.nl/multilevel.htm#progPINT
         o Power for longitudinal studies
         o PinT
   * Example with contextual/compositional effects
         o BeyondRepeated-2007-02-22-FittingModels.R
         o Practical Issues Applying Mixed Models to Observational Data

R scripts

What IS a mixed model?

  1. Fixed effects models are models with only fixed factors and optional covariates as predictors. Most models in analysis of variance, regression, and GLM are fixed effects models, which are by far the most common type in social science. An example would be a study of job satisfaction by gender, controlling for salary level. Job satisfaction would be the dependent variable, gender the fixed factor, and salary the covariate, treated as fixed.
  1. Random effects models are models with only one or more random factors and optional covariates as predictors. If there are covariates, they are treated as fixed effect variables, so the random effects model becomes a mixed model though some authors may still call it a "random effects model." An example would be a study of job satisfaction by city, controlling for salary level. Job satisfaction would be the dependent variable, city the random factor (assuming only a random sample of cities was studied), and salary the covariate.
  1. Mixed models have both fixed and random factors as well as optional covariates as predictors. An example would be a study of job satisfaction by gender by city, controlling for salary level. Job satisfaction would be the dependent variable, gender the fixed factor, city the random factor, and salary the covariate.
   * Hierarchical linear models (HLM) are a type of mixed model with hierarchical data - that is, where data exist at more than one level (ex., student-level data and school-level data). HLM models focus on differences between groups (ex., schools) in explaining a dependent variable. Put another way, the focus is on the group effect on a dependent in relation to predictor covariates.
         o Unconditional models are often used as a comparison baseline. They focus on the variability in the dependent when there are no fixed effects which condition (control, moderate) the variability.
         o Conditional models include one or more fixed factors. Covariance parameter estimates are conditional on the model's fixed effects. The likelihood ratio test (model chi-square difference test) can be used to assess the difference in fit between a conditional model and the corresponding unconditional model.

And why do we need them? The general linear model (GLM) includes such procedures as t-tests, analysis of variance, correlation, regression, and factor analysis, to name a few. The linear mixed model (LMM) is a further generalization of GLM to better support analysis of a continuous dependent for:

  1. random effects: where the set of values of a categorical predictor variable are seen not as the complete set but rather as a random sample of all values (ex., the variable "product" has values representing only 5 of a possible 42 brands).
  2. hierarchical effects: where predictor variables are measured at more than one level (ex., reading achievement scores at the student level and teacher-student ratios at the school level).
  3. repeated measures: where observations are correlated rather than independent (ex., before-after studies, time series data, matched-pairs designs)



   * Fixed factors are categorical variables where all possible category values (levels) are measured, even if one of the values is "other." (Ex., religion = Protestant, Catholic, Jewish, Other). Fixed factors may be the primary variables of interest in a research study. Fixed factors ave different, varying intercepts for each group, but the regression slope is the same for each group
   * Random factors are categorical variables where only a random sample of possible category values are measured. (ex., city = Philadelphia, Miami, Denver, Detroit, Los Angeles). Random factors often reflect the sampling and data collection design of a research study. That is, random factors may be grouping factors in multilevel analysis (ex., individual-level voting data grouped by city, so that voter attributes are level 1 fixed factors and city is a level 2 random factor). It is important to note, however, that a given effect may be modeled as a random or as a fixed factor. For instance, ses (socioeconomic status) in a study of employee performance scores grouped by agency might be modeled as a random covariate if it is thought its regression coefficient varied randomly by agency, but if the regression coefficient is assumed to be constant across agencies, ses might be modeled as a fixed factor. Designating a level 1 (ex., employee) variable as a random factor means the researcher assumes its coefficient varies randomly across level 2 (ex., agency) groups.
   * Covariates are continuous predictor variables. (ex., income). SPSS syntax is dependentvariable BY factors WITH covariates. Factors and covariates are entered in the main "Linear Mixed Models" dialog box in SPSS, then either the Fixed or Random buttons or both are clicked to specify how factors and covariates are to be modeled.
   * Moderator variables in a multi-level model are the level 2 or higher level independent variables. Levels are modeled as random factors. For instance, school budget at level 2 may moderate the effect of socio-economic status (SES) on test performance at the base (student) level. The theory, presumably, would be that school budget compensates for some of the educational resources implicit in SES at the individual level. The regression coefficients connecting the moderator variables at level 2 to the regression slopes and intercepts at the individual level are assumed not to vary across groups (schools) and hence these are fixed coefficients, in contrast to the random coefficients at the base level.
   * Cross-level interactions. Multi-level modeling will identify significant cross-level interactions (the joint effect of a variable at a base level in conjunction with a variable at an upper level). When such an interaction is present, the estimated coefficient of the direct variable estimates the effect of that variable when the other variable in the interaction is controlled (is zero). Sometimes, of course, a zero value is impossible for that variable because it is outside the permissible range. Interpretation may be improved in such cases by centering the variable: subtract the mean value from all cases for that variable. Interactions can also be visualized by plotting separate scatterplots of one of the interacting variables with the dependent, getting a separate plot for each value of the other interacting variable. 


Why use LMM instead of regression?

     Data with multiple levels involve group effects on individuals which may be assessed invalidly by traditional statistical techniques. When grouping is present (ex., students in schools), observations within a group are often more similar than would be predicted on a pooled-data basis.That is, simple regression ignores grouping effects and violates the assumption of independence of observations. Mixed (multi-level) modeling handles this by using variables at upper levels (ex., school-level budgets at level 2) to adjust the regression of base level dependent variables on base level independent variables (ex., predicting student-level performance from student-level socioeconomic status scores).
     Multi-level modeling in LMM is particularly helpful in the analysis of covariance when data are sparse. For instance, in a study of a Social Security agency office, there may be too few minority employees to enable valid statistical inferences on performance evaluations, using traditional regression models. However, if multi-level data are available on employees and multiple SSA offices, then multi-level models can use not only the individual data in the SSA office but also information in the pooled data for all offices. The resulting prediction equation applied to the given SSA office will use coefficients reflecting both their own and also pooled data. For agencies with a large number of minorities, the multi-level and ordinary regression models will be similar. For agencies with sparse data -- few minorities -- it is true their estimate will rely considerably on the pooled data, but the advantage is that the pooling involved in multi-level models affords a "borrowing of strength" that supports statistical inference in a situation where no inference would be possible using traditional methods.
     Traditional regression models vs. LMM analysis. There were three traditional approaches to regression modeling of multilevel data:
        1. Simple regression, also called naive regression, simply ignored higher level effects (ex., ignored class or school effects in a study of students). This is appropriate, of course, only when the researcher can be sure there are no higher level effects. More often, there are such effects and simple regression leads to too low estimates of standard error, a higher rate of Type I errors and too-narrow confidence limits compared to multilevel modeling of the same data.
        2. Fixed effects regression. A popular traditional approach was to disaggregate data to the base level (ex., each student is assigned various school-level variables such as funding level per student, and all students in a given school have the same value on these contextual variables, and students are used as the unit of analysis). In fixed effects regression, sampling error is taken into account only for level 1 (the base) level, and sampling error at level 2 (or higher) is ignored. That is, information from fewer units at the upper level is wrongly treated as if it were independent data for the many units at the base level, and this error in treating sample size led to over-optimistic estimates of significance. Also, there was the danger of the ecological fallacy: there is no necessary correspondence between individual-level and group-level variable relationships (ex., race and literacy correlate little at the individual level but correlate well at the state level, since Southern states have many African-Americans and many illiterates of all races). Finally, under fixed effects regression, the number of dummy variables increases as the number of clusters increases, making estimation inefficient. While adding crosslevel interaction terms is possible and does represent an improvement, it is inferior to multilevel modeling in LMM, which will model separate intercepts and slopes for individuals in each level 2 (or higher level) group, where the grouping variable is treated as a random effect.
        3. Summary measures regression. Another traditional approach to multi-level problems was to aggregate data to a higher level (ex., student performance scores are averaged to the school level and schools are used at the unit of analysis). Aggregated data was often centered (the mean was subtracted so the average value was zero). Ordinary OLS regression or another traditional technique was then performed on the unit of analysis chosen. A problem with summary measures regression is that under aggregation, fewer units of analysis at the upper level replace many units at the base level, resulting in loss of statistical power. As with simple regression, summary measures regression regression leads to too low estimates of standard error, a higher rate of Type I errors and too-narrow confidence limits compared to multilevel modeling of the same data. Summary measures regression also suffers from the possibility of ecological fallacy. Finally, summary measures regression prevents the valid analysis of covariate interactions due to loss of individual-level information. 

Hierarchical Models Hierarchical data involve measurement at multiple levels such as individual and group as, for example, a study of certain variables studied in terms of individual students' opinions, their classes, and their schools. In fact, much early work on multi-level modeling focused on educational settings. In general, hierarchical data are obtained by measurement of units grouped at different levels, such as a study of children nested within families; employees nested within agencies; soldiers nested within platoons, divisions, and armies; or subjects nested within studies.

   * Groups. Individuals are clustered within groups. In two-level mixed models, the base layer (level 1) is individuals (ex., students) who are clustered within the groups formed by the upper level layer (ex., level 2 = schools).
   * Multistage sampling. Hierarchical data are normally obtained by multistage sampling. For instance, one might sample schools within school districts, then sample students within sampled schools. 

Mixed Models

simple complex mulitlevel

Longitudinal Data

Using R Using SAS Using R to run SAS


   * Booth, J.G. & Hobert, J.P. (1998). Standard errors of prediction in generalized linear mixed models. Journal of the American Statistical Association 93, 262 -272.
   * Breslow, N. E., & Clayton, D. G. (1993). Approximate inference in generalized linear mixed models. Journal of the American Statistical Association 88, 9-25.
   * Bryk, A.S., and Raudenbush,S.W. (1992). Hierarchical linear models. Newbury Park, Sage Publications.
   * Bryk, A. S., S. W. Raudenbush, and R. Congdon (1996). Hierarchical Linear and nonlinear modeling with the HLM/2L and HLM/3L Programs. Chicago: Scientific Software International.
   * Burton, B. (1993). Some observations on the effect of centering on the results obtained from hierarchical linear modeling. Washington, DC: National Center for Education Statistics, U. S. Department of Education.
   * Cheung, M. W.-L. & Au, K. (2005). Applications of multilevel structural equation modeling to cross-cultural research. Structural Equation Modeling 12(4), 598-619.
   * Curran, P. J. (2003). Have multilevel models been structural equation models all along? Multivariate Behavioral Research 38(4): 529–569.
   * de Leeuw, J. and I. G. G. Kreft (1986). Random coefficient models for multi-level analysis. Journal of Educational Statistics 11: 57-86.
   * Diggle, P. J. (1988). An approach to the analysis of repeated measurements. Biometrics 44, 959-971.
   * Duncan, T. E., Duncan, S. C., Strycker, L. A., Li. F, & Alpert, A. (1999). An introduction to latent variable growth curve modeling: Concepts, issues, and applications. Mahwah, NJ: Erlbaum
   * du Toit, S. H. C. & du Toit, M. (2005). Multilevel structural equation modeling. In de Jeeuw, J. & Kreft, I. G. G., eds., .Handbook of quantitative multilevel analysis. Boston: Kluwer Acad.
   * Ehlers, Margaret (2004). Assessment of covariance selection strategies with repeated measures data. 34th Annual SCASA Meeting, American Statistical Association. Retrieved 4/223/07 from http://www.stat.sc.edu/scasa/Apr04t.html.
   * Goldstein, H. (1991). Nonlinear multi-level models with an application to discrete response data. Biometrika 78: 45-51.
   * Goldstein, H. (1995). Multilevel statistical models. London, Edward Arnold; New York, Halstead Press.
   * Hoffman, D. A. & Gavin, M. B. (1998). Centering decisions in hierarchical linear models: Implications for research organizations. Journal of Management 24(5), 623-641.
   * Hox, J. J. (1995). Applied multi-level analysis, 2nd Ed.. Amsterdam: TT-Publikaties. A basic, non-technical introductory text. Available at http://www.fss.uu.nl/ms/jh/publist/amaboek.pdf.
   * Hox J. J. (2002). Multilevel analysis: Techniques and applications. Mahwah, NJ: Erlbaum.
   * Hox, J. J. & Maas, C. J. M. (2001). The accuracy of mulitlevel structural equation modeling with pseudobalanced groups and small samples. Structural Equation Modeling 8, 157-174.
   * Jiang, Jiming (2007). Linear and generalized linear mixed models and their applications Springer Series in Statistics. New York: Springer.
   * Julian, M. W. (2001). The consequences of ignoring multilevel data structures in nonhierarchical covariance modeling. Structural Equation Modeling 8, 325-352.
   * Kreft, Ita G. G. (1996). Are multi-level techniques necessary? An overview, including simulation studies. Obtained online on Nov. 22, 1999, at http://www.calstatela.edu/faculty/ikreft/quarterly/quarterly.html.
   * Kreft, Ita G. G., Jan de Leeuw, J., and Aiken, Leona S. (1995). The effect of different rorms of centering in hierarchical linear models. Multivariate Behavioral Research 30(1), 1-21.
   * Kreft, Ita G. G., Jan de Leeuw, J., and R. van der Leeden (1994). Review of five multi-level analysis programs: BMDP-5V, GENMOD, HLM, ML3, VARCL. American Statistician 48(4): 324-335.
   * Kreft, Ita G. G. and Leeuw, Jan de (1991). Model-based ranking of schools. International Journal of Education 15: 45-59.
   * Kreft, Ita G. G. and Leeuw, Jan de (1998). Introducing multi-level modeling. Thousand Oaks, CA: Sage Publications. Introducing Statistical Methods Series. A user-oriented introductory text with a minimum of formal mathematics. Includes worked examples using real datasets using MLn, the predecessor of MLWin.
   * Littell, Ramon C., George A. Milliken, Walter W. Stroup, and Russell D. Wolfinger (1999). SAS System for mixed models. Cary, NC: Sas Institute.
   * Longford, N.T. and Muthen,B.O. (1992). Factor analysis for clustered populations. Psychometrika, 57, 581-97.
   * Longford, N.T. (1993). Random coefficient models. Oxford, Clarendon Press.
   * Luke, Douglas (2004). Multilevel modeling. Thousand Oaks, CA: Sage Publications. Vol. 142 in Quantitative Applications in the Social Sciences Series.
   * McCulloch, Charles E. (1997). Maximum likelihood algorithms for generalized linear mixed models. Journal of the American Statistical Association 92, 162-170.
   * McCulloch, Charles E. & and Searle, Shayle R.(2001). Generalized, linear, and mixed models. NY: Wiley-Interscience.
   * McCulloch, Charles E. (2003). Generalized linear mixed models. Regional Conference Series. Bethesda, MD: Institute of Mathematical Statistics
   * McDonald, R.P. (1994). The bilevel reticular action model for path analysis with latent variables. Sociological Methods and Research, 22, 399-413.
   * McDonald, R.P. and Goldstein, H. (1988). Balanced versus unbalanced designs for linear structural relations in two level data. British Journal of Mathematical and Statistical Psychology, 42, 215-32.
   * Moerbeek, Mirjam; van Breukelen, Gerard J. P.; & Berger, Martijn P. F. (2003). A comparison between traditional methods and multilevel regression for the analysis of multicenter intervention studies. Journal of Clinical Epidemiology 56, 341-350.
   * Morris, C. (1995). Hierarchical models for educational data - an overview. Journal of Educational and Behavioral Statistics 20(2).
   * Muthén, B.O. (1989). Latent variable modelling in heterogeneous populations. Psychometrika, 54, 557-85.
   * Muthén, B.O. (1994). Multilevel covariance structure analysis. Sociological Methods and Research 22: 376-399.
   * Raudenbush, Stephen W. and Anthony S. Bryk (2002). Hierarchical linear models: Applications and data analysis methods, Second edition. Thousand Oaks, CA: Sage Publications. Advanced Quantative Techniques in the Social Sciences Series No. 1.
   * Singer, J. (1998). Using SAS Proc Mixed to fit multilevel models, hierarchical models, and individual growth curves. Journal of Educational and Behavioral Statistics 24(4), 323-355.
   * Singer, J. & Willett, J. (2003). Applied logitudinal data analysis. Oxford, UK: Oxford University Press.
   * Snijders, T. & Bosker, R. (2000). Multilevel analysis: An introduction to basic and advanced multilevel modeling. London: Sage.
   * Thompson, R. (1985). A note on restricted maximum likelihood estimation with an alternative outlier model, Journal of the Royal Statistical Society. Series B (Methodological). 47(1), 53-55.
   * Tomarken, A. J. & Waller, N. G. (2005). Structural equation modeling: Strengths, limitations, and misconceptions. Annual Review of Clinical Psychology 1, 31-65.
   * Verbeke, Geert & Molenberghs, Geert (1997). Linear mixed models in practice: A SAS-oriented approach. Lecture Notes in Statistics. New York: Springer.
   * Verbeke, G., & Molenberghs, G. (2000). Linear mixed models for longitudinal data. New York: Springer.
   * West, Brady; Welch, Kathleen B.; & Galecki, Andrzej T. (2006). Linear mixed models: A practical guide using statistical software. Boca Raton, FL: Chapman & Hall/CRC.
   * Wong, G. and W. Mason (1985). The hierarchical logistic regression model for mulitlevel analysis. Journal of the American Statistical Association 80: 513-524.