Introduction

In Lesson 8, the Cedar Valley evidence team extracted data, appraised each study's risk of bias and rated the certainty of evidence for loneliness. Those ratings depend on a step that this lesson teaches: synthesis, the process of bringing the results of several studies together to answer the review question. A synthesis says what the studies, taken together, show about a comparison and an outcome, and how consistent they are.

A quantitative synthesis combines numerical results, such as differences in loneliness scores between program and comparison groups; Lesson 10 introduces the synthesis of qualitative findings. Section 1 of this lesson tabulates and groups the studies. Section 2 introduces synthesis without meta-analysis and the SWiM reporting guideline. Section 3 explains why counting studies by statistical significance misleads and presents better displays. Section 4 asks when pooling results in a meta-analysis is justified, a method whose computation HSCI 230 Lesson 2 teaches.

This lesson works with the 36 studies that report quantitative results: the 33 quantitative studies and the quantitative strands of the 3 mixed-methods studies. The 6 qualitative studies and the qualitative strands of the mixed-methods studies go to the qualitative evidence synthesis in Lesson 10. The figure shows the route from the extraction forms to the synthesis.

1.1 Why Synthesis Starts with Tables

A common first draft of a review's results describes the included studies one at a time: "Smith and colleagues found that ... Jones and colleagues found that ...". Readers of such a draft must hold every study in mind and do the synthesis themselves, and the writer has no structure for noticing patterns. A structured table solves both problems. It lays the studies side by side on the same columns, so that similarities, differences and gaps become visible before anyone writes a sentence of interpretation.

The guidance on narrative synthesis by Popay and colleagues (2006) describes four elements: developing a theory of how the intervention works, why and for whom; developing a preliminary synthesis; exploring relationships within and between studies; and assessing the robustness of the synthesis. Tabulation is the main tool of the preliminary synthesis, and the Cochrane Handbook chapter on synthesis without meta-analysis (McKenzie and Brennan, 2019) likewise recommends structured tables. A review usually needs three kinds of table.

1.2 Grouping Studies for Synthesis

A review question is usually broader than any single synthesis. A statement that "community-based interventions reduce loneliness" would combine programs that work in different ways and would help no one plan a service. The review therefore divides its studies into groups for synthesis, each defined by a population, intervention, comparator and outcome. The Cochrane Handbook calls this the "PICO for each synthesis" and treats it as part of planning the review (McKenzie et al., 2019). Design can also define groups, since randomized and non-randomized studies are usually synthesized separately.

Grouping involves a trade-off called lumping and splitting. Lumping places broadly similar programs in one group, giving more studies per synthesis but possibly combining programs that differ in ways that matter. Splitting places studies in narrow groups, giving specific answers from as few as one or two studies each. The right level depends on the decision the review supports. The Cedar Valley planning team must choose a kind of program to fund, so the team grouped by the main component of each program.

Groups should be planned in the protocol, because a group defined after seeing the results can be drawn to make a program look better or worse. When plans must change, for example because a category contains no studies, the change is reported with its reason, as the SWiM guideline in Section 2 asks.

1.3 Cross-Tabulating the Cedar Valley Studies

The team's first table counted the 42 included studies by intervention category and design. Five of the six categories contain studies; no study fell into "other".

The table shows a pattern that matters for the planning team. Community connector programs, the model Cedar Valley intends to launch, have 12 studies, second only to group activities (14), yet none is a randomized trial; all 14 trials evaluated group or one-to-one programs. This explains in advance why Lesson 8 rated the connector evidence as very low certainty. The table also shows that the intergenerational and technology-based categories hold too few controlled studies for a confident synthesis, which the review should state plainly.

1.4 Choosing One Result per Study

Many studies report several results that could fill one cell of a results table: loneliness on two scales, at three time points, by two kinds of analysis. Reviewers who choose among these after seeing them may pick, without meaning to, the result that fits their expectations. The safeguard is a set of decision rules, written before extraction, that select one result per study for each synthesis, as the Cochrane Handbook recommends (McKenzie et al., 2019). The accordion lists the Cedar Valley rules.

1.5 Building a Results Table

With groups defined and one result chosen per study, the team built a results table for each synthesis. To compare results from different loneliness scales, it expressed each as a standardized mean difference (SMD): the difference in mean scores between program and comparison groups divided by a standard deviation, which puts every scale in standard deviation units. HSCI 230 Lesson 2 shows the calculation. The table gives the results for the 8 trials of group-based programs.

SMD: standardized mean difference, program minus comparison; values below zero mean less loneliness in the program group. CI: confidence interval, the range of effects reasonably compatible with the trial's data. Trials C and G report estimates adjusted for clustering. The 8 trials analyzed 1,236 participants in total. All results are illustrative.

Three features make this table useful. Every result points in a stated direction, so a reader sees at once that seven of the eight estimates fall below zero. The rows are ordered by risk of bias and then by size, so the two trials at high risk sit together and a reader can check whether they differ from the rest; here one shows almost no difference and the other a small benefit. And the table includes columns, such as program length, that might explain variation (Section 2). A study that reported only "no significant difference" still gets a row, marked "not estimable", so that readers can see how much evidence lacks usable data.

The table does not yet state a conclusion. A reader might be tempted to count the confidence intervals that exclude zero (two of eight) and decide that group programs mostly fail. Section 3 explains why that reading is wrong and what to do instead, and Section 2 first sets out the methods and reporting standard for synthesis without meta-analysis.

Introduction

Many reviews in public health do not combine their results in a meta-analysis, because the studies are too few, too varied or too incompletely reported, or because the review maps evidence. Such reviews still synthesize, and many have described what they did as a narrative synthesis. The term has a defined meaning in the guidance of Popay and colleagues (2006), but it has often been used for any synthesis that is not a meta-analysis, including study-by-study descriptions with no stated method.

Campbell and colleagues (2019) examined a sample of Cochrane reviews that synthesized results without meta-analysis and found that most did not describe their synthesis methods or link their conclusions clearly to the data. The same group then led the development of the Synthesis Without Meta-analysis (SWiM) reporting guideline (Campbell et al., 2020), which sets out nine items to report when a review synthesizes quantitative effects without pooling them.

2.1 When a Review Synthesizes Without Meta-Analysis

The Cochrane Handbook chapter on synthesis without meta-analysis (McKenzie and Brennan, 2019) lists the circumstances that lead reviews to other methods. Effect estimates may be incompletely reported, for example as "no significant difference" with no numbers. Studies may use effect measures that cannot be converted to a common metric. The evidence may be at such high risk of bias that a pooled estimate would give misleading precision. And the studies may differ so much that an average effect would have no clear meaning. Section 4 returns to these conditions.

The Cedar Valley review meets several of them. Its protocol (Lesson 2) planned no meta-analysis, because a twelve-week rapid scoping review aims to map evidence. Its community connector studies are non-randomized and mostly at serious risk of confounding, and several categories contain only a handful of controlled studies. Even so, the planning team asked what the evidence says about loneliness, and the review needs a method that answers that question transparently.

2.2 Three Methods That Synthesize Without Pooling

McKenzie and Brennan (2019) describe three methods that are acceptable when effects are not pooled. Each answers a different question and needs different data. The table summarizes them.

Summarizing effect estimates

When the studies in a group report effects in a common metric, such as the standardized mean difference introduced in Section 1, the review can describe their distribution with three statistics. The median is the middle value when the estimates are put in order. The interquartile range (IQR) is the span of the middle half of the estimates, from the 25th to the 75th percentile. The range runs from the smallest to the largest estimate. These statistics describe the spread of observed results and carry no weighting by study size.

For the 6 one-to-one befriending or telephone trials, the 5 with estimable results had a median of −0.20 (IQR −0.24 to −0.16; range −0.28 to −0.12). Trial L reported only that there was no significant difference, so the summary must state that it rests on 5 of the 6 trials.

This method needs a standard metric. A difference of 2 points means something different on the 20-item UCLA scale, scored from 20 to 80, than on the three-item scale, scored from 3 to 9, and standard deviation units put the two on a common footing. For binary outcomes, such as the proportion who often feel lonely, the metric would be a ratio such as the odds ratio or risk ratio, which HSCI 230 Lesson 2 teaches.

Combining P values

Methods such as Fisher's method combine the exact P values from several studies into one test of the hypothesis that the program has no effect in any study. A small combined P value suggests a real effect in at least one study but says nothing about its size. The Cedar Valley team did not use these methods, since the planning team asked about the size and consistency of effects.

Vote counting based on the direction of effect

The third method counts how many studies found an effect in the direction of benefit and how many in the direction of harm, whatever the size or significance of each result. It needs the least data, and Section 3 explains how it differs from the flawed practice of counting studies by statistical significance.

2.3 The SWiM Reporting Guideline

SWiM is a reporting guideline: it lists what a review should tell readers about a synthesis that does not pool effects, so that they can judge whether the conclusions follow. It was developed through a consensus process and is designed to be used alongside PRISMA 2020 (Campbell et al., 2020). Its nine items cover the methods (items 1 to 7), the results (item 8) and the discussion (item 9).

The accordion describes each item and shows how the Cedar Valley review reports it.

SWiM allows any defensible method, provided the review reports it against the nine items. A reader of a review that follows SWiM can find which studies contributed to each statement, what was done with their results, and why.

2.4 Investigating Variation Without Meta-Analysis

Results almost always vary across studies, partly by chance and partly, perhaps, because of real differences between programs, populations or methods. Without the statistical tools of meta-analysis, a review can tabulate or plot results against characteristics that might explain the variation, as SWiM item 5 asks. The characteristics should be few, chosen in the protocol and grounded in a reason to expect a difference, because a search through many characteristics after seeing the results will find patterns by chance.

The Cedar Valley team examined program length for the 8 group-based trials, reasoning that longer programs might allow friendships to form. The table divides the trials at 12 weeks.

The medians are similar, and the shorter programs show a wider spread, including both the largest benefit and the two results near zero. With five and three trials, this pattern could easily arise by chance, so the team reported that program length did not explain the variation; tabulations by setting and by risk of bias gave the same conclusion. The unexplained variation is why GRADE rated down for inconsistency in Lesson 8.

2.5 Writing the Synthesis

SWiM item 8 asks that each written statement address a comparison and an outcome, report the findings and their certainty, and name the contributing studies. The tabs contrast a study-by-study draft with a structured synthesis of the same eight trials.

Introduction

Section 1 ended with a temptation: to see that only two of eight confidence intervals for group-based programs exclude zero and conclude that the programs mostly failed. Lesson 1 showed the same trap with eight simulated trials of one program, of which only three reached significance. This section explains why counting by significance gives wrong answers, presents vote counting based on the direction of effect, and shows two displays designed for synthesis without meta-analysis.

3.1 Vote Counting by Statistical Significance

A result is called statistically significant when its P value falls below a chosen threshold, usually 0.05, or equivalently when its 95 percent confidence interval excludes the value that means no effect (zero for a difference). Vote counting by statistical significance sorts studies into three piles, significant benefit, significant harm and no significant difference, and then reaches a conclusion from the tallies. If most studies are significant and favourable, the program "works"; if most are non-significant, it "does not work"; and if the piles are similar, the evidence is "mixed". The method still appears in published reviews, including one that the Cedar Valley team verified in Lesson 6, and the Cochrane Handbook advises against it (McKenzie and Brennan, 2019).

3.2 Why Counting by Significance Misleads

A non-significant result is not a finding of no effect

Whether a study reaches significance depends on the size of the true effect and on the study's precision, which rises with the number of participants. A small study of an effective program will often have a confidence interval wide enough to include zero. Altman and Bland (1995) summarized the error of reading such a result as proof of no effect in the title of a short paper: "Absence of evidence is not evidence of absence". A non-significant result means that the study could not distinguish the effect from zero. Its confidence interval often includes both no effect and a worthwhile benefit.

The figure plots the Cedar Valley results for group-based programs. It looks like a forest plot, which HSCI 230 Lesson 2 teaches, but it has no pooled estimate at the bottom; SWiM item 7 lists this kind of ordered display as a way to present results without meta-analysis.

Six of the eight intervals include zero, and all six of those also include reductions in loneliness of 0.3 standard deviations or more. The non-significant trials are compatible with a small benefit, and seven of the eight point estimates favour the program. A significance count reads this pattern as "two successes, six failures"; a direction count shows seven of eight trials pointing toward benefit, a pattern compatible with a small benefit measured imprecisely.

Small studies have low power

Statistical power is the probability that a study will produce a statistically significant result when the program truly has an effect of a given size. The table shows the power of a two-group trial to detect a standardized mean difference of 0.2, a small effect of the size the Cedar Valley trials suggest, using a two-sided test at the 0.05 level.

Calculated with the normal approximation for a comparison of two means. Most Cedar Valley trials analyzed between 64 and 230 participants.

A trial with 100 participants has about a one-in-six chance of reaching significance even when the program truly reduces loneliness by 0.2 standard deviations, so a significance count across such trials will usually declare an effective program ineffective.

More studies can make the answer worse

One might expect that adding studies would correct the error. Hedges and Olkin (1980) showed the opposite for the majority rule. When each study has power below one half, the proportion of significant results tends toward that power as studies accumulate, so the chance that most studies are significant falls toward zero. The table applies this to trials of 100 participants each, with a true standardized mean difference of 0.2 and power of 17 percent.

With 40 such trials, a vote count is almost certain to conclude that a program with a real effect does not work. As the number of trials grows, a majority-rule count becomes more likely to reach the wrong conclusion.

Size, weight and false conflict

Vote counting by significance also discards information. A trial of 64 people counts the same as a trial of 230, and a large benefit counts the same as a trivial one that reached significance in a very large study. The method also manufactures conflict: when some trials are significant and others are not, reviewers call the evidence "mixed" even when the estimates lie close together, as in Figure 3.1.

3.3 Vote Counting Based on the Direction of Effect

The acceptable alternative counts each study by the direction of its effect estimate alone: favouring the program or favouring the comparison, whatever its size or significance (McKenzie and Brennan, 2019). The review reports the number and proportion of studies favouring the program, ideally with a confidence interval for that proportion, and can apply a sign test. The sign test asks how likely a split at least as uneven as the observed one would be if the program had no effect, so that each study were as likely to point one way as the other, like a fair coin.

These P values need care. With only five studies, the sign test cannot produce a two-sided P value below 0.06, because even five out of five has a probability of 2 in 32 under the hypothesis of no effect. Reading such a P value as proof of no effect would repeat the error of Section 3.2, so the team reports the counts, the proportion with its interval and the P value together, and lets the effect summaries from Section 2 describe size.

Direction counting has its own limits, which SWiM item 9 asks reviews to state. It ignores the size of effects, so an estimate of 0.03 counts as a full vote against the program although it is essentially zero. It gives a small study the same vote as a large one. It requires every result to be aligned in direction before counting (Section 1, Rule 6). And it counts studies, so a study that reports five loneliness measures still casts one vote in each synthesis, chosen by the decision rules.

3.4 Effect Direction Plots

Direction counts become harder to follow when studies report several outcomes measured in different ways. The effect direction plot, developed by Thomson and Thomas (2013) for reviews of complex public health interventions, displays the direction of effect for every study and every outcome domain in one grid. Each row is a study and each column an outcome domain. An upward arrow shows an effect favouring the intervention, a downward arrow an effect favouring the comparison, and a sideways arrow a domain in which the study's several outcomes conflict. The size of the arrow shows the study's sample size, and rows are ordered or shaded by design and risk of bias. Hilton Boon and Thomson (2021) revised the plot to follow the 2019 Cochrane Handbook guidance, so that arrows record the direction of effect regardless of statistical significance and each column can be summarized with a sign test.

When a study reports several outcomes within one domain, such as depressive symptoms and anxiety within mental health, the team needs a rule for the domain's arrow. The Cedar Valley team gave a domain a direction when at least 70 percent of its outcomes in the study pointed the same way and marked it as conflicting otherwise, a threshold of the kind Thomson and Thomas (2013) described. It also defined the direction of each domain in advance: for health service use, fewer unplanned visits to primary care or emergency departments counted as benefit.

For loneliness, all five controlled studies favour the program, as do four of the five uncontrolled studies. On a two-sided sign test, five of five gives P = 0.06, the smallest value possible with five studies. Social isolation and well-being point the same way in every controlled study that measured them. Mental health and health service use are mixed: Studies Q and T recorded more unplanned visits among program participants, which the team interpreted with caution, since connectors may link people to care they had been going without. Because the controlled studies are non-randomized and three of five are at serious risk of confounding, the plot supports a statement about consistency of direction, and the GRADE rating from Lesson 8 (very low certainty) governs how strongly the review can state it.

Following SWiM item 4, the uncontrolled studies appear below a labelled divider, visible to readers, and the review's statements about effect rest on the controlled studies above it.

3.5 Harvest Plots

The harvest plot was developed by Ogilvie and colleagues (2008) to synthesize evidence about the differential effects of interventions, such as whether they narrow or widen health inequalities, and is now used for many kinds of synthesis without meta-analysis. It is a matrix. Rows usually represent outcomes, hypotheses or categories of intervention, and columns represent categories of finding. Each study is drawn as a bar in the cell that matches its finding, and the height and shading of the bar encode study characteristics chosen by the review team, such as design and risk of bias, with a label above the bar identifying the study. The plot combines the information of a vote count with a picture of the strength of the evidence behind each vote.

The Cedar Valley team drew one harvest plot for loneliness, covering the 24 controlled studies across the five intervention categories. It placed studies by direction of effect, consistent with Section 3.3, so that the plot shows a direction-based vote count with design and risk of bias added.

Twenty of the 24 controlled studies favour the program, one has an unclear direction and three favour the comparison. The tall bars, the randomized trials, are concentrated in the group and one-to-one rows. The community connector row holds only short bars, and three of its five are red. The intergenerational and technology rows hold so few studies, mostly at serious risk of bias, that no pattern can be read from them. A planning team can see in one figure that the most consistent and best-designed evidence concerns programs other than the one it plans to launch, which is the same message the cross-tabulation in Section 1 and the GRADE ratings in Lesson 8 gave.

3.6 Choosing a Display

The three displays suit different data, and a review often uses more than one. The table compares them. Other displays exist, such as the albatross plot, which plots each study's P value against its sample size (Harrison et al., 2017), and the choice among them should follow the question and the data, as SWiM item 7 asks reviews to explain.

Each of these methods describes a body of evidence without combining its results into a single estimate. Section 4 asks when a review should go further and pool the results statistically, and what HSCI 230 Lesson 2 adds when it does.

Introduction

The methods in Sections 2 and 3 describe a body of evidence without combining its results into a single number. A meta-analysis goes further: it combines the effect estimates from several studies into one weighted average, with a confidence interval, using statistical methods. Gene Glass (1976) coined the term to describe "the analysis of analyses", and the method is now standard in reviews of the effects of health interventions. This section does not teach how to compute a meta-analysis. HSCI 230 Lesson 2 does that, covering fixed-effect and random-effects models, forest plots, statistical heterogeneity, publication bias and influential studies. This section teaches the decision that comes first: whether pooling is justified for a given group of studies, and how to report the decision either way.

4.1 What Pooling Adds and What It Cannot Fix

Pooling adds three things. First, it combines the information in several imprecise studies into one more precise estimate, which addresses the problem of small studies that Section 3 described. The eight Cedar Valley trials of group-based programs, none with more than 230 participants, would together give an estimate based on 1,236. Second, it estimates the size of the effect with a confidence interval, which a direction count cannot do. Third, it provides statistical tools for describing how much results vary between studies and for exploring why, as well as for examining signs of publication bias.

Pooling cannot fix three other problems. It cannot remove bias from the included studies: if every trial overstates the benefit because participants who know their group report less loneliness, the pooled estimate will overstate it too, with a narrower confidence interval that makes the error look more certain. It cannot give meaning to an average of things that should not be averaged: a pooled effect of "community programs" that mixes group activities, befriending calls and connector referrals would describe no program anyone could implement. And it cannot recover studies that were never published or found, which is why the searching methods in Lessons 4 and 5 matter. These limits explain why the decision to pool depends on judgement about the studies as well as on whether the numbers are available.

4.2 Five Questions Before Pooling

The Cochrane Handbook treats the decision as a matter of whether a meta-analysis would be meaningful and appropriate for a given synthesis, considering the diversity of the studies, the data available and the risk of bias (Deeks, Higgins and Altman, 2019; McKenzie and Brennan, 2019). Those considerations can be organized as five questions, which the Cedar Valley team asked of each comparison.

4.3 Common Misconceptions About Pooling

Students and some review authors hold beliefs about pooling that lead to poor decisions in both directions. The cards describe five of them.

4.4 Worked Example: Should the Cedar Valley Team Pool?

The team applied the five questions to each comparison for loneliness. The table records its reasoning.

The first row needed the most thought. The group-based trials meet the conditions that the Cochrane Handbook sets out, and a full systematic review of these programs would normally pool them. The team considered amending its protocol to add a meta-analysis and decided against it for two reasons. The decision facing the planning team concerns community connector programs, for which pooling is not justified, and a precise estimate for group programs would not change that decision. And an amendment made after the results table had been seen would be open to the criticism that the method was chosen by the result. The review reports that the data for group-based programs would support a meta-analysis and recommends one in any full review. A team with different aims might reasonably decide the other way, provided it reports the change and its reason, as SWiM item 1b and PRISMA 2020 require.

4.5 From Synthesis to Statements for Decision-Makers

The synthesis ends in statements that combine the findings from Sections 2 and 3 with the GRADE ratings from Lesson 8, written in the standard wording of Santesso and colleagues (2020). The table gives the Cedar Valley statements for loneliness that will go into the evidence brief in Lesson 12.

These statements show how far a careful synthesis without meta-analysis can go. The planning team learns which kinds of program have the most consistent evidence, roughly how large their effects appear to be, how confident it can be, and that the evidence for the model it intends to launch is very uncertain. Lesson 8 drew the practical implication: the Cedar Valley program should be launched with an evaluation built in, so that the program adds to the evidence about community connector programs.

4.6 Connecting to Meta-Analysis

Students who take HSCI 230 Lesson 2, before or after this course, will find that the work of this lesson is the preparation a meta-analysis needs. The groups for synthesis define which studies enter each meta-analysis. The decision rules choose one result per study. The results table, with standardized mean differences, confidence intervals and risk-of-bias judgements, is the data set the meta-analysis uses, and the ordered estimate plot in Figure 3.1 becomes a forest plot once a pooled estimate is added. The exploration of variation in Section 2.4 becomes subgroup analysis, and the restriction to studies at lower risk becomes a sensitivity analysis. HSCI 230 Lesson 2 adds the statistical model, the weights and the heterogeneity statistics, and its section on when pooling is inappropriate returns to the questions asked here.

Bringing It All Together

This lesson followed the Cedar Valley evidence team from extracted data to statements about effect. The team first made the evidence visible: a characteristics table described every study, a cross-tabulation showed that all 14 randomized trials concerned group or one-to-one programs while the 12 community connector studies were all non-randomized or uncontrolled, and results tables set out one result per study, chosen by rules written in advance and aligned in direction. Groups for synthesis, defined by population, intervention, comparator, outcome and design, kept unlike comparisons apart.

Because the review did not pool results, it used the methods that the Cochrane Handbook accepts for synthesis without meta-analysis and reported them against the nine SWiM items. Summaries of standardized mean differences described the size and spread of effects, and counts by the direction of effect, with sign tests, replaced the misleading practice of counting studies by statistical significance, which in the Cedar Valley trials would have reported two successes in eight and none in six. Effect direction plots and harvest plots displayed these counts alongside design, sample size and risk of bias.

Finally, five questions about the question, the data, the designs, the risk of bias and the plan determined whether pooling was justified. The group-based trials could support a meta-analysis of the kind HSCI 230 Lesson 2 teaches, while the connector studies could not. Combined with the GRADE ratings from Lesson 8, the synthesis produced clear statements for decision-makers: group and one-to-one programs may reduce loneliness, and the evidence for community connector programs is very uncertain.

The final reflection asks you to explain the lesson's main ideas to a decision-maker who has read a misleading headline about the evidence.

Final Knowledge Assessment

These terms, tools and people appear in this lesson on synthesizing quantitative findings.