Research Synthesis Methods最新文献

This study quantifies how prompting strategies interact with large language models (LLMs) to automate the screening stage of systematic literature reviews (SLRs). We evaluate six LLMs (GPT-4o, GPT-4o-mini, DeepSeek-Chat-V3, Gemini-2.5-Flash, Claude-3.5-Haiku, and Llama-4-Maverick) under five prompt types (zero-shot, few-shot, chain-of-thought [CoT], CoT-few-shot, and self-reflection) across relevance classification and six Level-2 tasks, using accuracy, precision, recall, and F1. Results show pronounced model-prompt interaction effects within this SLR-automation corpus: zero-shot maximizes recall for high-sensitivity relevance screening; CoT-few-shot yields stronger precision and F1; and self-reflection underperforms due to over-inclusivity and instability across models. GPT-4o and DeepSeek provide robust overall performance, while GPT-4o-mini performs competitively at a substantially lower dollar cost. A cost-performance analysis for relevance classification (per 1,000 abstracts) reveals large absolute differences among model-prompt pairings; GPT-4o-Mini remains low-cost across prompts, and structured prompts (CoT/CoT-few-shot) on GPT-4o-Mini offer attractive F1 at a small incremental cost. The cost-performance analysis also suggests that future screening workflows could incorporate staged, cost-aware procedures: (1) using lower-cost, high-recall configurations for initial screening and (2) reserving higher-cost or more precision-oriented configurations for later triage or uncertain records. These findings highlight LLMs' uneven but promising potential to support screening within corpora similar to the SLR-automation corpus examined in this study. Further validation is needed before generalising these results to broader systematic review screening tasks. By systematically analyzing prompt-model interactions, we provide a comparative benchmark and practical guidance for task-adaptive LLM deployment.

Meta-analysis is often viewed as a gold standard for synthesizing available evidence, but it faces challenges as evidence becomes available gradually and studies may be disseminated selectively and at different times. When publication timing is systematically related to study results, meta-analytic conclusions at any given time can be biased; for example, studies with statistically significant or larger effects may be published earlier than those with null or smaller effects. This phenomenon, commonly referred to as time-lag bias, is a key reason why early meta-analytic conclusions may later weaken, reverse, or become more heterogeneous as additional evidence accrues. To address the limitations of existing definitions and detection approaches, which are largely tied to monotone, effect-size-driven dissemination, we broaden the concept of time-lag bias beyond the traditional yet simple notion that "positive" studies are published earlier. Specifically, we view time-lag bias more generally as a manifestation of time-dependent evidence availability, whereby the timing of study dissemination can distort meta-analytic conclusions at different stages of evidence accrual. We illustrate this broader perspective through several real-world case studies, including classical time-lag bias, inverse time-lag bias, non-monotone temporal patterns, and anomalous evidence localized within specific time windows. These examples show that time-lag bias can arise through multiple dissemination mechanisms and may exaggerate, attenuate, or even reverse pooled estimates over time. They highlight the importance of explicitly accounting for the temporal dimension of evidence accumulation and motivate the routine use of diagnostic and sensitivity strategies to assess robustness to selective dissemination over time.

We show that the covariance matrix of the treatment effect estimates in a network meta-analysis can be obtained without matrix inversion using a geometric series of diffusion matrices. This property extends to the hat matrix and provides a connection between parameter estimation in regression analysis and random walks on the network graph. We also provide a number of visualization tools implemented in R.

Systematic reviews are the gold standard for evidence synthesis, but the manual title and abstract screening of thousands of references creates a severe bottleneck. Existing automated tools have historically struggled to achieve the near-perfect recall (sensitivity) required for reliable reviews. We developed MechaScreener as a 'zero-shot' automated screening tool that utilises a large language model to rank article relevance. The tool requires no initial training data or manual pre-screening, as MechaScreener directly applies user-provided question elements (PICO) or inclusion/exclusion criteria to assign an ordinal relevance score (1-5) to each reference. We evaluated the tool in two phases: a development phase using five reference libraries to optimise prompts, and an independent evaluation phase using 10 diverse Cochrane review libraries (comprising both randomised controlled trials [RCTs] and non-RCTs) containing over 58,000 references. In the evaluation dataset, MechaScreener achieved a perfect recall of 1.00 (100%) in all 10 reviews, ensuring that no relevant articles were missed (pooled record-level 95% CI: 0.98-1.00). Concurrently, it achieved an overall macro-average specificity of 0.61 (61%, cluster bootstrap 95% CI: 0.46-0.75; pooled record-level specificity 0.59, 95% CI: 0.59-0.60). Specificity varied from 0.21 in broad public health topics to 0.91 in precise pharmacological interventions, reflecting the tool's built-in conservatism when evaluating ambiguous abstracts. By safely eliminating over 60% of irrelevant literature during the initial screening phase without compromising recall, MechaScreener functions as a highly reliable but low-effort 'first-pass' filter, allowing researchers to substantially reduce manual workloads and reallocate resources towards full-text review and data extraction.

Meta-analytic structural equation modeling (MASEM) is a flexible modeling framework that combines the strengths of meta-analysis and structural equation modeling. Despite the increasing popularity of the MASEM technique, its usage and scope are limited to relatively simple meta-analytic data structures and moderator hypotheses. Within-study variability in moderator variables and dependence in primary studies' correlation estimates tend to be handled using ad hoc methods. In this study, we demonstrate use of multilevel multivariate modeling with robust variance estimation (RVE) for handling within-study dependence when pooling studies' correlation estimates and we propose how to incorporate categorical moderator values directly into the structural equation model for moderators with values varying within studies. We outline the proposed approach using three stages: (a) specification of the structural equation model using moderator variable categories, (b) pooling correlation matrix elements using the three-level multivariate model with RVE, and (c) testing the moderator hypotheses through either model comparison or direct estimation of structural equation model's path differences. We illustrate the approach using three examples to demonstrate its flexibility across different MASEM models. This framework broadens the scope of moderator hypotheses that can be tested using MASEM and enhances the methodological rigor for handling dependencies in studies' correlation estimates for MASEM.

Evidence synthesis (ES) is costly and time-intensive. Artificial intelligence (AI) tools are increasingly used to automate or semi-automate ES tasks. We conducted a living rapid review of AI tools that automate or semi-automate ES tasks. We searched for published articles and preprints through MEDLINE, Embase, and the Association for Computing Machinery Digital Library from January 1, 2021 through October 15, 2025, supplemented by hand-searching. We included existing reviews to capture evaluations before 2021. Screening was conducted in duplicate at the title/abstract level, with single full-text screening and data extraction verified by a second reviewer. We included 134 studies. Performance varied by task and evaluation approach. Search automation tools showed poor recall (median 17%), although human-in-the-loop approaches improved performance. Tools for identifying randomized controlled trials demonstrated high median recall (98%) and precision (92%). Semi-automated abstract screening tools using active learning reduced screening burden by approximately 50% while maintaining 95% recall. Large language models demonstrated variable accuracy for abstract screening (median 88% recall), full-text screening (median 96% recall), and data extraction (median 63% accuracy). Risk of bias (RoB) assessment tools showed moderate agreement with human reviewers for the Cochrane RoB tool version 1 (median 71%) but performed less well with other RoB tools. AI tools can meaningfully reduce burden in study design identification and abstract screening but perform inconsistently across other evidence synthesis tasks. Tools for search automation, data extraction, and RoB assessment require further development. No existing tool can currently replace human expertise.

In current meta-analytic methods, researchers must choose which model to use-the common effect (CE), fixed effects (FES), or random effects (RE) model. The CE and FES models allow for estimation of the person-average effect size while conditioning on the studies included in the analysis, while the RE model allows for generalizations to a broader population of studies, yet may estimate the study-average effect rather than the person-average effect. These meta-analysis methods (CE, FES, and RE) do not always allow for both estimation of the person-average effect and generalizations beyond the studies included in the analysis. In this article, we consider situations in which the person-average and study-average effect size can differ. Then, we propose a new framework that models directly how studies are sampled into a meta-analysis. We consider three different sampling designs-simple random sampling (SRS), stratified SRS, and cluster sampling-and investigate how a meta-analysis could be conceived under these design frameworks. Through this new framework, we introduce assumptions, estimators, and weights that can be used to estimate person-average effects with inference that is no longer conditional on the observed studies. We then illustrate how these new models and estimators differ using examples based upon data in the What Works Clearinghouse and a pre-existing systematic review. We conclude with a discussion of the implications of these differences in the context of the proposed sampling estimators, as well as suggestions for using these models and future research.

Meta-analyses in social science fields face multiple methodological challenges arising from how primary research studies are designed and reported. One challenge is that many primary studies report multiple relevant effect size estimates, leading to a data structure with dependent observations. Another is selective reporting bias, which arises when the availability of study findings is influenced by the statistical significance of results. Although many selective reporting diagnostics and bias-correction methods have been proposed, few are suitable for meta-analyses involving dependent effect sizes. Among available methods, step-function selection models are conceptually appealing and have shown promise in previous simulations, but tend to understate uncertainty in parameter estimates if dependent effects are ignored. We study methods for estimating step-function models from data involving dependent effect sizes, focusing specifically on estimating parameters of the marginal distribution of effect sizes and accounting for dependence using cluster-robust variance estimation or bootstrap resampling. We describe two estimation strategies, demonstrate them by re-analyzing data from a previous synthesis on ego depletion effects, and evaluate their performance through an extensive simulation study under both univariate and multivariate selection processes. Simulation findings indicate that selection models provide low-bias estimates of average effect size and clustered bootstrap confidence intervals provide acceptable coverage levels for meta-analyses with at least 30 studies. Although unadjusted estimates are more efficient when selective reporting is absent or minimal, the proposed methods provide a robust and useful approach for evaluating the potential biases arising when selective reporting is present in meta-analyses with dependent effects.

Automation-assisted title-abstract screening is routinely evaluated using metrics such as recall, precision, and workload reduction, and results are increasingly summarized across studies. Synthesis presupposes that evaluation results refer to comparable target quantities. This article examines whether screening evaluation results produced under diverse automation-assisted workflows can meaningfully support cumulative inference, and under what conditions cross-study comparison is warranted. This conceptual article treats screening evaluation targets as workflow-induced rather than task-inherent. Evaluation design is decomposed into four workflow dimensions (representation, inference, governance, and evaluation) plus the reference-decision set produced or specified. Recurring choice patterns across these elements define evaluation configurations. Two study-level configurations and one synthesis-level pattern are illustrated: endogenous adjudication, inherited adjudication, and aggregation of heterogeneous targets. Across configurations, evaluation targets are induced by workflow-specific reference-decision sets and, in some designs, model-based target domains rather than by a common reference set. Nominally similar metrics can therefore estimate different quantities across studies, and apparent cumulativeness can arise from pooling estimates of different quantities. Standard heterogeneity statistics cannot distinguish variation in method effectiveness from variation in what is being estimated. Comparison is warranted only when targets align before pooling. Differences across screening evaluations may reflect differences in what is being estimated rather than statistical heterogeneity around a common estimand. When evaluation targets are not aligned across the framework, aggregate summaries can describe reported results but cannot support cumulative inference. Improved aggregation requires both better study-level specification of evaluation targets and explicit attention to configuration variation at synthesis .

Component network meta-analysis (CNMA) is an evidence synthesis method that can estimate the effectiveness of components and component combinations of complex interventions. Individual participant data (IPD) meta-analysis is considered the gold standard for evaluating medical interventions. In this article, we propose CNMA models that simultaneously synthesise IPD and aggregate data to evaluate the effectiveness of individual components and combinations of components. We extend the models to adjust for individual-level covariates, by separating the within-study and between-study interaction effects. The methods are applied to a Cochrane systematic review dataset of 11 studies, of which eight have IPD available, to evaluate the effectiveness of complex home safety practices for promoting the safe storage of cleaning products to prevent childhood poisonings. Our findings demonstrate the benefits of combining IPD with aggregate data beyond a standard network meta-analysis or CNMA based solely on aggregate data, such as improved precision, changed conclusions, and addressing aggregation bias. For models without covariate adjustment, inclusion of IPD reduced the between-study heterogeneity and reduced the width of credible intervals (CrIs). However, across analyses, many CrIs are wide, particularly in models that separate the within- and between-study interactions. The models proposed in this article maximise the use of available evidence and provide valuable insights into which combinations of components are potentially the most effective and in which subgroups of the population. Further research is required to assess the generalisability of CNMA that combine IPD and aggregate data and their potential to enhance healthcare decision making.