A Meta-Analysis Package for R https://mail.metafor-project.org/ 2026-04-17T15:45:44+00:00 https://mail.metafor-project.org/ https://mail.metafor-project.org/lib/exe/fetch.php/wiki:logo.png text/html 2022-11-27T19:01:52+00:00 Anonymous (anonymous@undisclosed.example.com) https://mail.metafor-project.org/doku.php/tips:assembling_data_or?rev=1669575712 Assembling Data for a Meta-Analysis of (Log) Odds Ratios Suppose the goal of a meta-analysis is to aggregate the results from studies contrasting two groups (e.g., treatment versus control) and each study measured a dichotomous outcome of interest (e.g., treatment success versus failure). A commonly used effect size measure used to quantify the size of the group difference (i.e., the size of the treatment effect) is then the odds ratio.$\hat{\mu} = -0.75$$-1.11$$-0.38$$0.47$$0.33$$0.68$$1 - 0.4… text/html 2023-05-17T13:52:46+00:00 Anonymous (anonymous@undisclosed.example.com) https://mail.metafor-project.org/doku.php/tips:assembling_data_smd?rev=1684331566 Assembling Data for a Meta-Analysis of Standardized Mean Differences Suppose the goal of a meta-analysis is to aggregate the results from studies contrasting two groups (e.g., treatment versus control) and each study measured an outcome of interest using some quantitative scale. A commonly used effect size measure used to quantify the size of the group difference is then the standardized mean difference (also commonly known as Cohen's d).$\hat{\mu} = -0.54$$-1.14$$0.07$$\tau^2$$I^2$$Q$$$d = t \… text/html 2022-08-03T11:30:41+00:00 Anonymous (anonymous@undisclosed.example.com) https://mail.metafor-project.org/doku.php/tips:bootstrapping_with_ma?rev=1659526241 Bootstrapping with Meta-Analytic Models The use of bootstrapping in the meta-analytic context has been discussed by a number of authors (e.g., Adams, Gurevitch, & Rosenberg, 1997; van den Noortgate & Onghena, 2005; Switzer, Paese, & Drasgow, 1992; Turner et al., 2000). The example below shows how to conduct parametric and non-parametric bootstrapping using the metafor and boot packages in combination. The example is based on a meta-analysis by Collins et al. (1985) examining the effectiveness o… text/html 2023-08-10T11:20:03+00:00 Anonymous (anonymous@undisclosed.example.com) https://mail.metafor-project.org/doku.php/tips:ci_for_r2?rev=1691666403 Confidence Intervals for $R^2$ in Meta-Regression Models The $R^2$ statistic that is shown in the output for meta-regression models (fitted with the rma() function) estimates how much of the total amount of heterogeneity is accounted for by the moderator(s) included in the model (Raudenbush, 2009). An explanation of how this statistic is computed is provided $R^2$$k$$R^2$$R^2$$R^2$$Q_M(\textrm{df} = 4) = 10.01, p = .04$$p = .0041$$R^2 = 18.3\%$$R^2$$R^2$$R^2$$0\%$$57.6\%$$k = 46$$R^2$$0\%$$100\… text/html 2022-08-03T11:31:16+00:00 Anonymous (anonymous@undisclosed.example.com) https://mail.metafor-project.org/doku.php/tips:clogit_paired_binary_data?rev=1659526276 Conditional Logistic Regression for Paired Binary Data This is just a short illustration of how to fit the conditional logistic regression model for paired binary data using various functions, including the rma.glmm() function from the metafor package. text/html 2021-11-08T15:48:04+00:00 Anonymous (anonymous@undisclosed.example.com) https://mail.metafor-project.org/doku.php/tips:comp_mh_different_software?rev=1636386484 Comparison of the Mantel-Haenszel Method in Different Software The Mantel-Haenszel method is an approach for fitting meta-analytic equal-effects models when dealing with studies providing data in the form of 2x2 tables or in the form of event counts (i.e., person-time data) for two groups (Mantel & Haenszel, 1959). The method is particularly advantageous when aggregating a large number of studies with small sample sizes (the so-called sparse data or increasing strata case).$(1 - .299) \times 10… text/html 2024-06-18T19:28:37+00:00 Anonymous (anonymous@undisclosed.example.com) https://mail.metafor-project.org/doku.php/tips:comp_two_independent_estimates?rev=1718738917 Comparing Estimates of Independent Meta-Analyses or Subgroups Suppose we have summary estimates (e.g., estimated average effects) obtained from two independent meta-analyses or two subgroups of studies within the same meta-analysis and we want to test whether the estimates are different from each other. A Wald-type test can be used for this purpose. Alternatively, one could run a single meta-regression model including all studies and using a dichotomous moderator to distinguish the two sets. Bo… text/html 2024-09-18T20:01:45+00:00 Anonymous (anonymous@undisclosed.example.com) https://mail.metafor-project.org/doku.php/tips:computing_adjusted_effects?rev=1726689705 Computing Adjusted Effects Based on Meta-Regression Models After fitting a random-effects model and finding heterogeneity in the effects, meta-analysts often want to examine whether one or multiple moderator variables (i.e., predictors) are able to account for the heterogeneity (or at least part of it). Meta-regression models can be used for this purpose. A question that frequently arises in this context is how to compute an 'adjusted effect' based on such a model. This tutorial describes how t… text/html 2022-10-09T10:47:37+00:00 Anonymous (anonymous@undisclosed.example.com) https://mail.metafor-project.org/doku.php/tips:convergence_problems_rma_mv?rev=1665312457 Convergence Problems with the rma.mv() Function Model fitting with the rma.mv() function makes use of iterative methods that attempt to maximize the log likelihood function over the model parameters. The optimization techniques that are used for this purpose sometimes do not converge, in which case the function will inform the user that the optimizer did not achieve convergence. Below, I illustrate this problem with an example and discuss some remedies.$\sigma^2_1$ text/html 2022-03-26T14:50:31+00:00 Anonymous (anonymous@undisclosed.example.com) https://mail.metafor-project.org/doku.php/tips:convergence_problems_rma?rev=1648306231 Convergence Problems with the rma() Function Some routines within the rma() function are not based on closed-form solutions, but require numerical (iterative) methods. In particular, when using the ML (maximum likelihood), REML (restricted maximum likelihood), and EB (empirical Bayes) estimators for $\tau^2$$\tau^2$$10^{-5}$$\tau^2$$\tau^2$$\tau^2$ text/html 2024-06-12T17:43:10+00:00 Anonymous (anonymous@undisclosed.example.com) https://mail.metafor-project.org/doku.php/tips:diff_omnibus_vs_individual_tests?rev=1718214190 Difference Between the Omnibus Test and Tests of Individual Predictors When a meta-regression model includes multiple predictors, one can examine the significance of each individual predictor (i.e., coefficient), but also the significance of the model as whole. For the latter, we can conduct an $z = 2.94, p = .003$$z = -2.19, p = .029$$z = -2.24, p = .025$$z = -2.20, p = .028$$Q_M$$Q_M = 5.97, \mbox{df} = 3, p = 0.11$$k=13$$Q_M = 9.53, \mbox{df} = 4, p = 0.049$ text/html 2024-06-18T19:26:54+00:00 Anonymous (anonymous@undisclosed.example.com) https://mail.metafor-project.org/doku.php/tips:different_tau2_across_subgroups?rev=1718738814 Allowing $\tau^2$ to Differ Across Subgroups In a meta-analysis, we often want to examine if the size of a particular effect differs across different groups of studies. While the focus in such a 'subgroup analysis' tends to be on the size of the effect across the different groups, we might also be interested in examining whether the amount of heterogeneity differs across groups (i.e., whether the effect sizes are more/less consistent in some of the groups). Below, I illustrate different methods… text/html 2024-09-18T19:51:01+00:00 Anonymous (anonymous@undisclosed.example.com) https://mail.metafor-project.org/doku.php/tips:forest_plot_with_aggregated_values?rev=1726689061 Forest Plot with Aggregated Values In the simplest case of a meta-analysis, each study provides a single (effect size) estimate to the analysis. A standard forest plot then shows the estimates of the studies (with corresponding confidence intervals) and the summary estimate based on the meta-analysis at the bottom of the figure. See $k = 100$$I^2$$Q$$k=100$ text/html 2024-09-18T19:44:23+00:00 Anonymous (anonymous@undisclosed.example.com) https://mail.metafor-project.org/doku.php/tips:forest_plot_with_exact_cis?rev=1726688663 Forest Plot with Exact Confidence Intervals A forest plot is a commonly used visualization technique in meta-analyses, showing the results of the individual studies (i.e., the estimated effects or observed outcomes) together with their (usually 95%) confidence intervals (CIs). A four-sided polygon, sometimes called a summary 'diamond', is added to the bottom of the plot, showing the summary estimate based on the model (with the center of the polygon corresponding to the estimate and the left/ri… text/html 2024-09-18T19:39:11+00:00 Anonymous (anonymous@undisclosed.example.com) https://mail.metafor-project.org/doku.php/tips:handling_missing_data?rev=1726688351 Handling Missing Data in Output/Figures In many cases, the dataset to be used for a meta-analysis will contain studies for which insufficient information is available to compute the observed outcomes (e.g., the risk/odds ratios or the raw/standardized mean differences) or for which the values of potentially relevant moderators/covariates are unknown. We can use the dataset for the BCG vaccine meta-analysis (Colditz et al., 1994) as an illustration. text/html 2022-08-03T11:32:45+00:00 Anonymous (anonymous@undisclosed.example.com) https://mail.metafor-project.org/doku.php/tips:hunter_schmidt_method?rev=1659526365 Hunter and Schmidt Method The meta-analytic methods developed by Hunter and Schmidt (1990, 2004, 2014), sometimes called "psychometric meta-analysis", are commonly used to conduct meta-analyses in industrial/organizational psychology and related areas. A question that comes up on a regular basis is how one can conduct such meta-analyses using the metafor package. Adding such functionality is on my to-do list (see $$\text{Var}[r_i] = \frac{(1-\rho_i^2)^2}{n_i - 1}$$$n_i$$n_i - 1$$\rho_i$$r_i$$\r… text/html 2024-06-09T16:23:52+00:00 Anonymous (anonymous@undisclosed.example.com) https://mail.metafor-project.org/doku.php/tips:i2_multilevel_multivariate?rev=1717950232 $\boldsymbol{I^2}$ for Multilevel and Multivariate Models The $I^2$ statistic was introduced by Higgins and Thompson in their seminal 2002 paper and has become a rather popular statistic to report in meta-analyses, as it facilitates the interpretation of the amount of heterogeneity present in a given dataset.$I^2$$$I^2 = 100\% \times \frac{\hat{\tau}^2}{\hat{\tau}^2 + \tilde{v}},$$$\hat{\tau}^2$$\tau^2$$$\tilde{v} = \frac{(k-1) \sum w_i}{(\sum w_i)^2 - \sum w_i^2},$$$w_i = 1/v_i$$i^{th}$$\tilde… text/html 2022-05-24T12:13:15+00:00 Anonymous (anonymous@undisclosed.example.com) https://mail.metafor-project.org/doku.php/tips:increasing_tau2_when_adding_moderators?rev=1653394395 Increasing Value of $\tau^2$ When Adding Moderators In the meta-analytic random-effects model, the parameter $\tau^2$ denotes the amount of heterogeneity (also called 'between-study variance'), that is, the variability in the underlying true effects or outcomes. When the data suggest that the underlying true effects or outcomes vary (i.e., the estimate of $\tau^2$$\tau^2$$\tau^2$$\tau^2$$\tau^2$$\hat{\tau}^2 = 0.0465$$\tau^2$$R^2$$Q_M$$\tau^2$$R^2$$\tau^2$ text/html 2024-01-15T16:54:45+00:00 Anonymous (anonymous@undisclosed.example.com) https://mail.metafor-project.org/doku.php/tips:input_to_rma_function?rev=1705337685 Specifying Inputs to the rma() Function Unfortunately, I have seen a number of cases where users of the metafor package have misspecified the inputs to the rma() function, which will then lead to incorrect results. To explain the problem in more detail, let's examine the arguments of the function (note: only the first three arguments are shown below): text/html 2023-01-09T11:10:14+00:00 Anonymous (anonymous@undisclosed.example.com) https://mail.metafor-project.org/doku.php/tips:meta_regression_with_log_rr?rev=1673262614 Interpreting Coefficients in Meta-Regression Models with (Log) Risk Ratios In this tutorial, we will examine ways of interpreting the coefficients in meta-regression models when the log risk ratio is used as the effect size measure. Data Preparation $Q_M$$2.63 / 1.68 \approx 1.56$$\tau^2$$\tau^2$$1.86 / 0.78 \approx 2.39$$2.49 / 1.04 \approx 2.39$ text/html 2022-10-13T06:07:09+00:00 Anonymous (anonymous@undisclosed.example.com) https://mail.metafor-project.org/doku.php/tips:model_selection_with_glmulti_and_mumin?rev=1665641229 Model Selection using the glmulti and MuMIn Packages Information-theoretic approaches provide methods for model selection and (multi)model inference that differ quite a bit from more traditional methods based on null hypothesis testing (e.g., Anderson, 2008; Burnham & Anderson, 2002). These methods can also be used in the meta-analytic context when model fitting is based on likelihood methods. Below, I illustrate how to use the metafor package in combination with the $2^7 = 128$$.1439$$.1439 + … text/html 2019-05-05T08:46:31+00:00 Anonymous (anonymous@undisclosed.example.com) https://mail.metafor-project.org/doku.php/tips:model_selection_with_glmulti?rev=1557045991 Model Selection using the glmulti Package Please go here for the updated page: Model Selection using the glmulti and MuMIn Packages. This also covers how to use the MuMIn package for the same types of analyses. text/html 2024-06-18T19:29:11+00:00 Anonymous (anonymous@undisclosed.example.com) https://mail.metafor-project.org/doku.php/tips:models_with_or_without_intercept?rev=1718738951 Meta-Regression Models With or Without an Intercept A question that comes up frequently is the proper interpretation of meta-regression models with or without an intercept term. Below I illustrate the difference using the dataset for the BCG vaccine meta-analysis (Colditz et al., 1994).$b_0$$b_1$$b_2$$\mbox{H}_0{:}\; \beta_1 = \beta_2 = 0$\begin{align} &H_0: (0 \times \beta_0) + (1 \times \beta_1) + (0 \times \beta_2) = 0 \; \Leftrightarrow \; \beta_1 = 0 \\ &H_0: (0 \times \beta_0) + (0 \times… text/html 2024-09-04T07:21:01+00:00 Anonymous (anonymous@undisclosed.example.com) https://mail.metafor-project.org/doku.php/tips:multiple_factors_interactions?rev=1725434461 Models with Multiple Factors and Their Interaction The example below shows how to test and examine multiple factors and their interaction in (mixed-effects) meta-regression models. Data Preparation For the example, we will use the data from the meta-analysis by Raudenbush (1984) (see also Raudenbush & Bryk, 1985) of studies examining teacher expectancy effects on pupil IQ ($b_0 = 0.4020$$p = .0020$$0.1472$$0.6567$$b_1 = -0.2893$$b_2 = -0.4422$$p = .0334$$p = .0025$$p = .0103$$p = .1323$$b_3 =… text/html 2022-08-03T11:35:32+00:00 Anonymous (anonymous@undisclosed.example.com) https://mail.metafor-project.org/doku.php/tips:multiple_imputation_with_mice_and_metafor?rev=1659526532 Multiple Imputation with the mice and metafor Packages A meta-analytic dataset often looks like Swiss cheese -- there are lots of holes in it! For example, due to missing information, it may not be possible to compute the effect size estimates (or the corresponding sampling variances) for some of the studies. Similarly, it may not be possible to code certain moderator variables for some studies. Below, I illustrate how to use multiple imputation as a possible way to deal with the latter issue, … text/html 2023-08-31T08:18:21+00:00 Anonymous (anonymous@undisclosed.example.com) https://mail.metafor-project.org/doku.php/tips:non_linear_meta_regression?rev=1693469901 Modeling Non-Linear Associations in Meta-Regression In some situations, we may want to model the association between the observed effect sizes / outcomes and some continuous moderator/predictor of interest in a more flexible manner than simply assuming a linear relationship. Below, I illustrate the use of polynomial and spline models for this purpose.$p < .0001$ text/html 2022-10-02T10:46:52+00:00 Anonymous (anonymous@undisclosed.example.com) https://mail.metafor-project.org/doku.php/tips:regression_with_rma?rev=1664707612 Linear Regression and the Mixed-Effects Meta-Regression Model The standard linear regression model is given by $$y_i = \beta_0 + \beta_1 x_{i1} + \beta_2 x_{i2} + \ldots + \beta_p x_{ip} + e_i,$$ where $e_i \sim N(0, \sigma^2)$. Models of this sort can be fitted with the R function lm(). The mixed-effects meta-regression model is given by $$y_i = \beta_0 + \beta_1 x_{i1} + \beta_2 x_{i2} + \ldots + \beta_p x_{ip} + u_i + e_i,$$ where $u_i \sim N(0, \tau^2)$ and $e_i \sim N(0, v_i)$, where $v_i$… text/html 2019-05-05T08:47:54+00:00 Anonymous (anonymous@undisclosed.example.com) https://mail.metafor-project.org/doku.php/tips:rma_vs_lm_and_lme?rev=1557046074 A Comparison of the rma() and the lm() and lme() Functions Please go here for the updated page: A Comparison of the rma() and the lm(), lme(), and lmer() Functions. This now also covers the lmer() function. text/html 2023-11-14T08:00:04+00:00 Anonymous (anonymous@undisclosed.example.com) https://mail.metafor-project.org/doku.php/tips:rma_vs_lm_lme_lmer?rev=1699948804 A Comparison of the rma() and the lm(), lme(), and lmer() Functions Commonly used meta-analytic models are just special cases of the general linear (mixed-effects) model with the only peculiar aspect that the variances of the error terms (i.e., the sampling variances) are known. Not surprisingly, the question therefore comes up occasionally why the $\sigma^2_e$$\hat{\sigma}_e$$$y_i = \mu + u_i + e_i,$$$u_i \sim N(0, \tau^2)$$e_i \sim N(0, v_i)$$v_i$$$y_i = \mu + u_i + e_i,$$$u_i \sim N(0, \tau^… text/html 2022-08-03T11:36:55+00:00 Anonymous (anonymous@undisclosed.example.com) https://mail.metafor-project.org/doku.php/tips:rma.uni_vs_rma.mv?rev=1659526615 A Comparison of the rma.uni() and rma.mv() Functions The main workhorse of the metafor package is the rma.uni() function (same as rma()), which fits meta-analytic equal-, fixed-, random-, and mixed-effects models using what is often referred to as the "inverse-variance method" (for random/mixed-effects models, this is also called the "normal-normal model", as it assumes normally distributed sampling distributions with known sampling variances and normally distributed random effects for the tota… text/html 2017-03-19T21:24:00+00:00 Anonymous (anonymous@undisclosed.example.com) https://mail.metafor-project.org/doku.php/tips:speeding_up_model_fitting?rev=1489958640 Speeding Up Model Fitting Fitting models to large datasets and/or models involving a large number of random effects (for the rma.mv() function) can be time consuming. Admittedly, some routines in the metafor package are not optimized for speed and efficient memory usage by default. However, there are various ways for speeding up the model fitting, which are discussed below.$k = 4000$$\mu = 0.50$$\tau^2 = 0.25$$k$$k$$k \times k$$k$ text/html 2024-09-04T07:21:55+00:00 Anonymous (anonymous@undisclosed.example.com) https://mail.metafor-project.org/doku.php/tips:testing_factors_lincoms?rev=1725434515 Testing Factors and Linear Combinations of Parameters The example below shows how to test factors and linear combinations of parameters in (mixed-effects) meta-regression models. Testing Factors Meta-regression models can easily handle categorical predictors (factors) via appropriate coding of factors in terms of dummy variables. We will use the dataset for the BCG vaccine meta-analysis (Colditz et al., 1994) as an illustration.$k=13$$b_0$$b_1$$b_2$$b_3$$b_4$$\mbox{H}_0{:}\; \beta_1 = \beta_2… text/html 2022-08-03T11:38:30+00:00 Anonymous (anonymous@undisclosed.example.com) https://mail.metafor-project.org/doku.php/tips:two_stage_analysis?rev=1659526710 Two-Stage Analysis versus Linear Mixed-Effects Models for Longitudinal Data Longitudinal or growth curve data (where individuals are repeatedly measured over time) are often analyzed using (linear) mixed-effects models. It can be instructional to examine how the results from such models can be approximated by a (simpler) two-stage analysis (e.g., section 4.2.3 in Diggle et al., 2002; section 3.2 in Verbeke & Molenberghs, 2000; Stukel & Demidenko, 1997).$b_0 = 22.04$$SE = .420$$b_1 = 0.66$$SE = … text/html 2024-09-18T19:56:39+00:00 Anonymous (anonymous@undisclosed.example.com) https://mail.metafor-project.org/doku.php/tips:weights_in_rma.mv_models?rev=1726689399 Weights in Models Fitted with the rma.mv() Function One of the fundamental concepts underlying a meta-analysis is the idea of weighting: More precise estimates are given more weight in the analysis then less precise estimates. In 'standard' equal- and random-effects models (such as those that can be fitted with the $w_i = 1 / v_i$$v_i$$i$$w_i = 1 / (\hat{\tau}^2 + v_i)$$\hat{\tau}^2$$v_i$$\hat{\tau}^2$$\hat{\tau}^2 + v_i$$w_i = 1 / (\hat{\tau}^2 + v_i)$$$\hat{\mu} = \frac{\sum_{i=1}^k w_i y_i}{…