1.1 What Grey Literature Is

In Lesson 4 you built a search for bibliographic databases such as MEDLINE, Embase and CINAHL, which index articles published in journals. A large share of the evidence on health programs, however, never appears in a journal. It sits in reports posted on government websites, in theses held by university libraries, in abstracts presented at conferences and in records of trials that were registered but never published. This material is called grey literature (spelled "gray" in American usage), and a review that ignores it may give decision-makers a partial and sometimes misleading picture.

The most widely cited definition was agreed at the International Conference on Grey Literature in Luxembourg in 1997 and extended in New York in 2004. In paraphrase, grey literature is material produced at all levels of government, academia, business and industry, in print and electronic formats, whose publication is not controlled by commercial publishers, meaning that publishing is not the main activity of the organization that produces it. The definition concerns who controls publication. A grey document can be a careful evaluation with a large sample and clear methods, or it can be a promotional summary with no methods at all, so its quality has to be judged separately (Section 1.5).

Some writers contrast grey literature with commercially published, indexed journal articles on one side and with unpublished material, such as an internal evaluation that was never posted, on the other. Unpublished material can be reached only by asking the people who hold it. The figure shows how each zone is reached.

1.2 Why Reviews Search Beyond Journals

Four reasons to search for grey literature apply with particular force to questions about community programs.

Publication bias

Publication bias occurs when the results of a study influence whether, where and how quickly it is published. Studies that find a statistically significant or favourable effect are more likely to be submitted and accepted than studies that find no effect. A review that includes only published studies will then overestimate how well an intervention works. Hopewell and colleagues (2007) reviewed methodological studies for the Cochrane Collaboration and found that trials published in journals tended to show larger intervention effects than trials found only in the grey literature. Searching grey literature is one of the main ways a review team reduces this bias at the search stage. Statistical methods that look for signs of publication bias after studies have been collected, such as funnel plots, are taught in HSCI 230 Lesson 2.

Time-lag bias

Time-lag bias is a related problem: studies with favourable results tend to be published sooner, while studies with null results can take years to appear or never appear. Conference abstracts, preprints and trial registry records show recent or ongoing work that may change a review's conclusions.

Practice-based evidence

Many community programs are evaluated by the organizations that deliver them or by the agencies that fund them. A non-profit that runs a friendly-visiting program may commission an evaluation for its funder and post the report on its website. A health authority may evaluate a pilot and present the results to its board. These evaluations are rarely submitted to journals, yet they often describe exactly the kinds of programs a planning team is considering. For the Cedar Valley question, the team expects much of the evidence on Canadian community programs to be found in this form.

Context for decisions and for the scan

Grey literature also describes who runs programs, how they are funded and how they are delivered. These details matter to decision-makers even when a document contains no outcome data, and they feed the environmental scan, which Lesson 11 teaches in full.

1.3 Kinds of Grey Literature

The cards describe the kinds of grey literature that matter most to health evidence reviews, with a Cedar Valley example of each. Select a card to read more.

Trial registries in more detail

Trial registries deserve particular attention because they can reveal studies that would otherwise be invisible. Since 2005, the International Committee of Medical Journal Editors has required that trials be registered at or before the enrolment of the first participant as a condition of consideration for publication in the journals that follow its recommendations. A registry record therefore exists for many trials whose results were never published, and comparing a registered protocol with the later publication can show whether outcomes were changed after the data were seen (Lesson 8). Registration may be less consistent for trials of social and community interventions than for drug trials, so a registry search for Cedar Valley will be incomplete, but it remains worthwhile.

1.4 Canadian and British Columbian Sources

A grey-literature search is organized around the organizations that produce relevant documents. The table lists sources that a team working on a British Columbia question about older adults would consider, as a starting point to which each team adds organizations discovered while searching.

Theses Canada does not hold every recent Canadian thesis, because deposit practices vary by university, so searchers also check institutional repositories directly. When the search reaches documents produced by or about First Nations, Métis or Inuit communities, the team should respect the community's ownership of its information. The First Nations principles of OCAP (ownership, control, access and possession) describe how First Nations data and information should be governed, and the First Nations Health Authority is the appropriate starting point for First Nations health information in British Columbia.

1.5 Judging Grey Literature with AACODS

Grey documents have not passed through a journal's peer review, so the team needs a quick, consistent way to judge whether a document is credible enough to include and how much weight to give it. Tyndall (2010), at Flinders University in Australia, developed the AACODS checklist for this purpose. Its name lists six criteria: authority, accuracy, coverage, objectivity, date and significance. The accordion shows the questions each criterion asks, applied to a fictional evaluation report that the Cedar Valley team found on a non-profit's website.

AACODS is a credibility screen, and a document that passes it may still have a weak design; Lesson 8 introduces the risk-of-bias tools. Formal critical appraisal is often optional in a scoping review (Lesson 10), but recording an AACODS judgement helps the team describe the grey evidence in its report.

2.1 Four Complementary Strategies

There is no single index of grey literature, so a grey-literature search combines several strategies. Godin and colleagues (2015) set out a widely cited model in a case study that looked for Canadian guidelines on school breakfast programs. They used four strategies: searching grey-literature databases, searching customized Google search engines, searching the websites of targeted organizations, and asking content experts by email. For each Google search, they decided in advance to screen the first ten pages of results, about 100 hits.

Their results show why a combination is needed. The strategies together produced 302 potentially relevant items, of which 15 publications were included. Targeted website searching found 14 of the 15, content experts identified 9, and the grey-literature databases found only 1. Eleven of the 15 publications were found by more than one strategy, but four were found by only one, so dropping any single strategy would have produced a less complete result. The yields will differ for other topics. For a question about community programs, such as the Cedar Valley question, the case study suggests that organizational websites and contacts with the people who run programs are likely to be productive.

2.2 Targeted Website Searching

Targeted website searching means identifying, in advance, the organizations most likely to publish relevant documents and searching each of their websites in a planned way. Stansfield, Dickson and Bangpan (2016) describe the problems this raises for systematic reviews: site search functions vary in quality, the volume of material can be large, and the steps taken are easy to forget. Their advice, which this lesson follows, is to plan which sites to search, how to search each one and how far to go, and to keep a record of each decision.

Choosing the websites

The list of websites follows from the review question. For Cedar Valley, the question concerns community-based interventions for loneliness and social isolation among adults aged 65 and older, so the team asks which organizations fund, deliver, evaluate or set policy for such programs. The librarian proposes candidates from the Canadian sources listed in Section 1.4, the intern adds organizations named in the documents found so far, and the evidence officer adds organizations suggested by members of the planning team. The final list is recorded in the search plan with a short reason for each site.

Searching a website

Each website is searched with up to three methods, and the log records which were used.

• The searcher browses the pages where documents are listed, such as "Publications", "Reports", "Resources" or "Research and evaluation", and scans every item listed under relevant headings.
• The searcher uses the site's own search box with a few simple terms, since most site search boxes do not support Boolean operators, phrases or truncation reliably.
• The searcher runs a Google query restricted to the site with the site: operator, which often finds PDF reports that the site's own search misses.

Health authority and government websites are reorganized often, so documents may move or disappear. Section 4 explains how to capture what is found.

The Grey Matters checklist

Grey Matters: A Practical Tool for Searching Health-Related Grey Literature was developed by CADTH (the Canadian Agency for Drugs and Technologies in Health) and is now published by Canada's Drug Agency. It lists sources of health-related grey literature by type and by jurisdiction, including health technology assessment agencies, clinical practice guideline sources, drug and device information, trial registries and health statistics, and it is laid out as a checklist on which a searcher can record which sources were checked. Its emphasis on drugs and health technologies means that it covers only part of what a community-program review needs, but it is a useful way to make sure that no major agency is overlooked, and the completed checklist becomes part of the search record.

2.3 Google Advanced Operators

A search operator is a word or symbol that tells a search engine how to treat part of a query. Google supports a small set of operators that make web searching more precise. The table lists those most useful for grey literature. Operators are typed with no space between the operator, its colon and the term, so site:gov.bc.ca works and site: gov.bc.ca does not.

Several features that searchers learn in databases do not carry over to Google. Google has no truncation, so isolat* does not retrieve "isolated" and "isolation"; instead Google matches some word variants automatically. Google combines terms with an implied AND, ignores most punctuation and symbols, and limits a query to 32 words. It has also retired operators over time, including the plus sign in 2011 and the tilde for synonyms in 2013, and in 2024 it removed its links to cached copies of pages. Because of these differences, a database search string from Lesson 4 cannot be pasted into Google. It has to be rewritten as several short queries, each combining one or two key concepts with operators.

Building queries for Cedar Valley

The intern drafts a set of short queries, each aimed at one group of sources. The tabs show four of them, what each is meant to find and how the team refined it after looking at the first page of results.

2.4 Google Scholar

Google Scholar indexes scholarly material found on the web, including journal articles, theses, preprints, books and reports posted on academic and organizational sites. It supports quotation marks, OR, the minus sign, intitle: and allintitle:, author: and source:, and an advanced search form that restricts terms to the title, an author, a publication or a range of years. It does not support truncation or nested Boolean logic reliably, it ranks results by an undisclosed method, it displays at most 1,000 results for any query, and it cannot export a full result set in one step.

Two studies by Haddaway and colleagues help define its role. Haddaway, Collins, Coughlin and Kirk (2015) tested Google Scholar against case studies of environmental science systematic reviews. They found that its results contained moderate amounts of grey literature, that much of it appeared well beyond the first pages of results, and that Google Scholar missed important studies in five of six case studies. They recommended that searches of article titles for grey literature focus on the first 200 to 300 results and concluded that Google Scholar should not be used alone. Gusenbauer and Haddaway (2020) compared 28 academic search systems and concluded that Google Scholar is inadequate as the principal search system for a systematic review, chiefly because its queries cannot be controlled or reproduced precisely. The Cedar Valley team therefore uses Google Scholar as a supplementary source with a fixed stopping rule of 200 results, and it uses Google Scholar's "Cited by" links for forward citation chasing, which Section 3 describes.

2.5 Making Web Searching More Systematic

Two further tools reduce the haphazard quality of web searching. Google's Programmable Search Engine (formerly called Custom Search Engine) lets a team build a search box that searches only a chosen list of websites. Godin and colleagues searched two customized engines of this kind, one for Canadian public health information and one for Canadian government documents. The Cedar Valley team could build one engine covering the provincial government, the five regional health authorities, the First Nations Health Authority, the Office of the Seniors Advocate and United Way British Columbia, and then run the same queries across all of them at once. The team must still record the list of sites included in the engine.

The second tool is a stopping rule, a decision made before searching about how many ranked results will be examined for each query. A fixed number, such as the first 100 Google results or the first 200 Google Scholar results, is the most common rule, and it can be combined with an early stop when, for example, two consecutive pages contain nothing relevant. Setting the rule in advance keeps effort consistent across queries and lets the team report exactly how much of each ranking was examined.

If the protocol includes French-language documents, each query is also written in French, with terms such as "isolement social", solitude, aînés and "prescription sociale", and French-language sources such as the Institut national de santé publique du Québec are added to the website list.

3.1 What Citation Chasing Is

Every research article cites earlier work and is later cited by newer work. Citation chasing uses these links to find studies that keyword searches miss. If a study included in the Cedar Valley review cites an evaluation of a befriending program, that evaluation is likely to be relevant too, whatever words its title and abstract happen to use. Citation chasing therefore finds studies by their relationships to known relevant studies, which makes it a useful complement to searches that depend on matching words.

The method has gone by many names, including snowballing, pearl growing, reference list checking, citation tracking and citation searching. To bring order to these terms, Hirt and colleagues (2024) ran a Delphi consensus study with international methods experts and published the TARCiS statement (Terminology, Application and Reporting of Citation Searching), which contains ten recommendations. This lesson uses "citation chasing" as the everyday name and the TARCiS terms for the specific methods. The cards define them.

3.2 When Citation Chasing Helps

Citation chasing adds most when a topic is hard to search with words, as loneliness interventions are. The same kind of program may be called social prescribing, community referral, a link worker service, a community navigator program, befriending, friendly visiting or a telephone check-in service, and many evaluations describe their outcomes as well-being or social participation, with loneliness mentioned only in the full text. A search string cannot anticipate every label, but studies on the same topic tend to cite one another.

Greenhalgh and Peacock (2005) audited the sources of a systematic review of complex evidence on the diffusion of innovations in health service organizations. They found that fewer than a third of the primary sources had come from the database and hand searches set out in the protocol, while about half had been found by "snowballing", that is, by following references of references. That review covered a scattered literature from many disciplines, so its proportions are unusually high, but it showed how much a word-based search can miss. A Cochrane methodology review by Horsley, Dingwall and Sampson (2011) found limited but supportive evidence that checking reference lists identifies additional studies for systematic reviews. A scoping review by Hirt and colleagues (2023) mapped how citation tracking is used and reported, and it informed the TARCiS statement.

TARCiS turns this evidence into practical advice. For topics that are difficult to search, the team should seriously consider backward and forward citation searching as supplementary methods. For topics with a clear vocabulary and a highly sensitive search, they are not explicitly recommended, although checking the reference lists of included records can still test whether the database search was sensitive enough. Citation searching should not replace extensive database searching in a review that aims to find all relevant studies, because it can only find documents that are linked to the seeds.

3.3 Backward Citation Searching

Backward searching starts from the seed references and retrieves everything they cite. TARCiS recommends, where possible, retrieving the cited references from a citation index so that titles and abstracts can be screened, since a reference list read by hand usually gives only titles. The steps are as follows.

1. Assemble the seed set, normally all studies included after full-text screening, and record the list.
2. Retrieve the cited references from one or more citation indexes, such as Web of Science or Lens.org, and check by hand the reference lists of any seeds that the indexes do not cover.
3. Export the retrieved records to the reference manager, remove duplicates within the set, and remove records that were already screened from the database and grey-literature searches.
4. Screen the remaining records against the eligibility criteria, using the same screening process as for the main search (Lesson 7).

Backward searching is also one of the best ways to find grey literature. Authors of included studies often cite program reports, government documents and theses, and these citations point to documents that no index covers. The Cedar Valley team therefore reads the reference lists of the included grey-literature documents by hand as well, and counts any new documents found this way with its grey-literature results.

3.4 Forward Citation Searching

Forward searching retrieves the documents that cite each seed. It is the only systematic way to move forward in time from a known study, and it often finds recent evaluations and follow-up studies. Web of Science shows a "Times Cited" count and list, Scopus and Google Scholar show "Cited by" links, and Lens.org records citing works. These indexes cover different sets of documents, so the same seed can have quite different citing lists in each. TARCiS suggests considering two citation indexes to widen coverage, especially when some seeds are missing from one index. Google Scholar's citing lists include more theses and reports than the subscription indexes, but they cannot be exported in bulk, so they are most practical for a small number of seeds.

Citation counts grow over time, so the date on which forward searching was done must be recorded. Some indexes also let a user set an alert that sends an email whenever a seed is newly cited, which is useful for keeping a review current, as Lesson 12 discusses.

3.5 Citation-Network Tools

Two older ideas from information science underlie most citation-network tools. Co-citation, described by Small (1973), links two documents that are cited together by a later document: the more often they are cited together, the more closely related they are assumed to be. Bibliographic coupling, described by Kessler (1963), links two documents that cite the same earlier work: the more references they share, the more closely related they are assumed to be. Both build on the citation index, which Eugene Garfield proposed in 1955 and which became the Science Citation Index, the ancestor of Web of Science. In TARCiS terms, retrieving co-cited documents and co-citing documents are the indirect methods of citation searching.

Citation-network tools retrieve these relationships automatically and display them as lists or maps. The tabs describe examples available in 2026. Products change quickly, so the point is to understand what kind of relationship each tool uses and what data it draws on.

Whichever tool is used, the team records its name, the date of use, the data source it drew on, the seeds entered and any settings, and it exports the full list of retrieved records so that they can be de-duplicated and screened like any other search result. Tools that show only a selection of related papers are best used during scoping and search development. Lesson 6 extends this discussion to artificial intelligence tools that summarize the context of citations and suggest related papers, and it explains how to verify what they report.

3.6 Worked Example: Citation Chasing for Cedar Valley

A check of the database search

Before screening began, the librarian used citation relationships to check the sensitivity of the database search. The team had found three earlier reviews of interventions for loneliness and social isolation in older people during scoping: Dickens and colleagues (2011), Gardiner, Geldenhuys and Gott (2018), and Fakoya, McCorry and Donnelly (2020). The intern listed the studies cited in these reviews that appeared to meet the Cedar Valley criteria and checked whether each one was among the 2,480 database records. Suppose that two such studies were missing and that both described their programs as "friendly visiting". The librarian would add that phrase and related terms to the search strategy and rerun it, as Lesson 4 described, and the team would report the check and the change.

Direct citation searching after full-text screening

Once screening is complete (Lesson 7), the team will use the 38 studies included from the database searches as seed references and run backward and forward searching on all of them, retrieving records through citationchaser (Lens.org) and Web of Science and checking by hand the reference lists of two seeds that neither index covered fully. The table shows how the records are reduced to a set for screening. These numbers come from the completed review, which later lessons describe.

The 4 studies found by citation chasing bring the review from 38 to 42 included studies. TARCiS suggests considering a further iteration with the newly included studies as seeds. The Cedar Valley team decided against a second iteration because of its twelve-week deadline and because the first iteration had added few studies relative to the number screened, and it reported this decision and its reason. A team with more time, or one whose first iteration had found many new studies, might reasonably have continued.

4.1 The Documentation Problem

A database search is reported by pasting the full strategy for each database, a practice Lesson 4 taught with PRISMA-S. Grey-literature and web searches are much harder to report, and studies of published reviews show that they are often reported poorly. Briscoe (2015) examined systematic reviews published in the United Kingdom's Health Technology Assessment programme between 2004 and 2013. Of 300 systematic reviews, 108 reported searching the web. Most of these gave only the names of the websites (54 reviews) or the search engines (33 reviews) used, and only 6 reached the highest standard of reporting, which included the search terms, dates and results. Briscoe concluded that this level of reporting did not allow readers to judge or repeat the searches, and that full reproducibility is difficult in any case because websites and search engines change.

Mahood, Van Eerd and Irvin (2014) found that grey-literature searching was hard to document because each source had its own interface and few allowed results to be exported. Godin and colleagues (2015) noted that web documents and their addresses are transient, that Google results could not be exported into their record-management spreadsheet, and that personalization of search results may have affected what they saw. For these reasons, documentation is often the weakest part of a grey-literature search. The remedy is a plan written before searching and a log kept during it.

4.2 Transparent and Reproducible Searches

Two related standards apply. A search is transparent when a reader can see exactly what was done: which sources were searched, with which queries and settings, on which dates, how far down each list of results the searcher went, and what was kept. A search is reproducible when another person could repeat those steps and obtain the same results. Lesson 3 explained why web searches cannot be fully reproducible: search engines personalize and re-rank results, their indexes change daily, and web pages move or disappear. A database search run on the same date with the same strategy is close to reproducible, while a Google search rarely is.

The practical goal is therefore full transparency and as much reproducibility as the sources allow. The team records enough detail for someone else to rerun each search, and it keeps copies of everything it saved so that the evidence can be checked after the original pages change.

4.3 The Grey-Literature Search Plan

A grey-literature search plan sets out, before searching begins, which sources will be searched, how, how far, by whom and when. It belongs in the protocol (Lesson 2), and any later change is recorded with its reason. Writing the plan first forces the team to justify each source from the review question, fixes stopping rules before anyone sees the results and divides the work among team members. The plan has the following elements.

• The plan lists the source groups and the specific sources in each, with a reason for including each one.
• The plan states the method for each source, such as exporting all records, browsing publication pages, using site search or running site-restricted queries.
• The plan gives the queries or search terms, any date, language or file-type limits, and a stopping rule for every ranked source.
• The plan names who will search and check each source and when, and how found items will be captured, logged and passed to screening.

4.4 The Web Search Log

A web search log is a table with one row for every search of a website, search engine or other online source, completed at the time the search is run. It is the record from which the methods section and the search appendix are written. The template shows the fields.

Results counts reported by Google are estimates and can be noted, but they are not a measure of how many results were examined. Google now often shows an AI-generated overview above the results. The log can note that one appeared, but the overview is not a source, and Lesson 6 explains how to treat AI-generated answers.

4.5 Capturing and Preserving What You Find

Health authority and government websites are reorganized regularly, and documents are removed when programs end. Addresses that stop working, a problem known as link rot, can make a review's evidence base impossible to check. The team therefore saves a copy of every item at the time it is found and records where it came from.

• Each saved item receives an ID that combines the search ID with a sequence number, such as W04-03, and the file is saved in the team's shared folder under that ID.
• Each item is added to the reference manager with its address and access date; Zotero (Lesson 7) records the access date automatically.
• For web pages, as opposed to PDF files, the team also saves an archived copy, for example with the Internet Archive's Wayback Machine "Save Page Now" function, and records the archived address.
• For each search-engine query, the team saves a screenshot or PDF of the first page of results, which documents the ranking it saw.
• Any persistent identifier, such as a DOI or a repository handle, is recorded when one exists.

4.6 Worked Example: The Cedar Valley Plan and Log

This worked example shows the grey-literature search plan and the web search log for the fictional Cedar Valley review. The numbers are illustrative.

The search plan

The web search log

All Google and Google Scholar searches were run on google.ca in a private browsing window, signed out of all Google accounts, with English as the language, Verbatim off and no date filter; the location Google reported was Burnaby, British Columbia. Google Scholar searches excluded citations and patents. A screenshot of the first results page was saved for each query.

The log records W12, which found nothing relevant, because readers need to know the source was searched. Twelve searches saved 63 items: 32 from the five Google queries, 14 from Google Scholar and 17 from the six websites. The table combines these with the other sources in the plan.

The PRISMA 2020 flow diagram, which Lesson 7 builds, places study registers beside databases in its left-hand column and counts websites, organizations and citation searching in a separate column for other methods. The Cedar Valley team ran its registry, thesis, preprint and conference searches in week 5 as part of the grey-literature search, after the database records had been de-duplicated and screening had begun, and it screened the 557 records and items in the table above as one set. It therefore reported all of these sources in the column for other methods and explained the adaptation in its methods. A team that searches registers together with its databases and screens their records as one set counts them in the left-hand column, as the template intends. Of the 557 records and items, 61 were sought as full reports, and none of the registry records led to an additional study. In the completed review, grey-literature and website searching contributed 26 documents, which the team charted separately from the research studies, and citation searching added 4 studies to the 38 found through the databases, for a total of 42 included studies.

Reporting the web searches

From the log, the team writes a short methods paragraph and places the full log in an appendix. "Between 17 and 20 February 2026, one reviewer searched Google (google.ca) with five queries and Google Scholar with one query, in a private browsing window while signed out; Appendix 3 gives each query exactly as run. We screened the first 100 Google results and the first 200 Google Scholar results for each query, or all results when fewer were shown, and a second reviewer checked the first page of results for each query. We searched six organizational websites by browsing their publication pages or using their site search. All saved documents were archived with their addresses and access dates." Together with the appendix and the plan, it addresses the PRISMA-S items in Section 4.2.

Bringing It All Together

This lesson extended the Cedar Valley search beyond bibliographic databases. Grey literature is defined by who controls its publication, and it matters to a review because publication bias and time-lag bias distort a journal-only evidence base, because community programs are often evaluated in reports that never reach a journal, and because reports describe who runs and funds programs. Its quality varies, so the AACODS checklist offers a consistent first judgement of each document.

Finding grey literature combines grey-literature databases, trial registries, preprint servers, targeted websites, search engines, contacts and citation searching, since each finds documents the others miss. Google and Google Scholar operators make web searching more precise, but ranked results must be sampled with a stopping rule. Backward and forward citation searching, described in the terminology of the TARCiS statement, find studies through their links to included studies, and citation-network tools can support exploration without replacing complete citation searches.

Documentation holds these methods together. A plan written before searching, a web search log completed during it and archived copies of every item make the search transparent, and PRISMA-S and the PRISMA 2020 flow diagram set out how to report it. The final reflection asks you to plan these searches for a new question, and the knowledge check integrates all four sections.

The final reflection asks you to plan grey-literature, web and citation searches for a new review question about support for family caregivers of people living with dementia.

Final Knowledge Assessment

Terms, tools and people introduced in this lesson on grey literature, web searching and citation chasing.