Most acupuncture trials are run in China, and many are published in Chinese-language journals that the databases most English-language reviews search do not index. Where a reviewer looks therefore decides which trials a review counts, and the trials found in different places differ in design, in risk of bias and in the size of the effects they report. In 1998 Andrew Vickers and colleagues showed that acupuncture trials from some countries were uniformly positive; Nothing in this literature ever fails reports their figures and the share of registered trials that never publish. This page covers what later work has measured: how many trials come from where, how they are indexed, how they were randomised and registered, and how their results differ by language.
The 1998 study and what it asked of reviewers
Vickers and colleagues, at the Research Council for Complementary Medicine in London, read abstracts of controlled trials in Medline from 1966 to 1995. Reviewers who did not know a trial’s country judged whether it was positive, and its country was classified separately. To check whether the pattern was peculiar to acupuncture, they ran a second study of trials of other treatments from the same countries against trials published in England, and found the same pattern. Publication bias, the failure of negative trials to reach print, was the explanation they offered as possible. They advised authors of systematic reviews to consider carefully how to handle trials from those countries. The 2014 US Veterans Affairs evidence map cited the paper and lowered its confidence in any effect that rested mostly on trials not indexed in PubMed; Evidence maps describes how.
How many trials, and from where
| Source | What was counted | From China |
|---|---|---|
| Tang, Zhan and Ernst, 1999 | Randomised trials in 28 of 100 Chinese traditional medicine journals, hand-searched to the end of 1996 | 2,938 trials found; about 7,500 estimated across all 100 journals |
| Matsuura and colleagues, 2020 | 1,758 acupuncture trials registered with the WHO registry platform, 1990 to 2018, from 43 countries | China, the US and South Korea together 61% |
| Gang and colleagues, 2022 (FAMOUS) | 584 acupuncture trials with more than 100 participants, published 2015 to 2019 | 92.5% conducted in China |
| Lu and colleagues, 2022 | 2,471 systematic reviews of acupuncture in Web of Science, 2000 to 2020, by first author’s country | 40.3%, then the US 14.5%, the UK 12.8%, South Korea 10.5% |
| Li and colleagues, 2023 | Trials in Cochrane acupuncture reviews that searched Chinese databases | 54% of trials were in Chinese, against 15% in reviews that did not search them |
Tang and colleagues found that the number of trials in China had doubled every two to three years since the early 1980s, that only 15% used blinding, and that few enrolled 300 patients or more. Their funnel plot of 49 acupuncture trials for stroke suggested that positive trials had been published selectively.
The Chinese databases
Chinese medical journals are indexed in Chinese bibliographic databases. A 2008 survey by Xia, Wright and Adams described five large ones: the China National Knowledge Infrastructure (CNKI), Wanfang, VIP, CBM and CMCC. Together they indexed about 2,500 journals, fewer than 6% of which were indexed for Medline, and that their access, cost and search features varied widely; all of them were searched best in simplified Chinese. A review that searches only English-language databases therefore misses most of the journals in them. Li and colleagues found that 31 of 84 Cochrane acupuncture reviews, 37%, had searched Chinese databases.
AcuiQ reads its protocols from the same literature, and the sources it holds that resolve to no index entry are mostly Chinese-language clinical studies; Nothing in this literature ever fails explains why they were kept and labelled.
Whether a trial called randomised was randomised
Taixiang Wu and colleagues at West China Hospital in Chengdu tested that directly in 2009. From 37,313 articles in CNKI on 20 common diseases, published from 1994 to 2005, they identified 3,137 reports that described themselves as randomised trials, of conventional and traditional treatments alike, and interviewed the authors of 2,235 of them about how patients had been allocated. Only 207, or 6.8%, had used a method that counted as randomisation. The rate did not differ between trials of traditional and conventional treatments. Trials run at hospitals affiliated to medical universities were more often genuine, 56.3% of them, and every pre-market drug trial in the sample was genuine. The authors attributed the rest to authors not understanding trial design and called for better peer review. Reading an acupuncture trial reports how seldom Chinese-journal acupuncture trials described their randomisation and blinding up to 2012.
Results by language and population
Li and colleagues, at Lanzhou University with co-authors in Canada, Hong Kong, Sweden and Australia, re-ran the meta-analyses in Cochrane acupuncture reviews separately for Chinese-language and other trials. The Chinese-language trials reported larger effects (pooled ratio of odds ratios 0.51, 95% confidence interval 0.29 to 0.91). They were also more often at high or unclear risk of bias for blinding of patients and practitioners (97% against 51%) and of outcome assessors (93% against 47%). Higher risk of bias went with larger effects in both groups, but the authors found no evidence that it inflated the Chinese-language results more than the rest. They also found acupuncture appeared more effective in Chinese than in non-Chinese populations, and concluded that the difference in effects was probably associated with differences between the populations studied.
Researchers in China describe the split too. Gang, Gong and Jing, at the China Academy of Chinese Medical Sciences and the American Academy of Acupuncture and Oriental Medicine, wrote in 2022 that Chinese trials overwhelmingly report positive outcomes while Western trials generally find no difference from placebo, and compared the two in purpose, design, investigators, treatment regimens and methodological quality. They proposed that acupuncture research proceed in stages, each with agreed protocols and reporting norms. In the FAMOUS analysis, country of origin was associated with effect size when considered alone but not once other trial features were accounted for, and the authors cautioned that the sample was too unbalanced between countries to generalise.
Registration
A trial registered before it starts records in public what it will measure, which lets a reader check whether the published outcome is the one planned. Journals that follow the International Committee of Medical Journal Editors (ICMJE) consider a trial that began on or after 1 July 2005 only if it was registered before its first patient was enrolled. Since June 2007 the committee has accepted any primary registry of the WHO International Clinical Trials Registry Platform, as well as ClinicalTrials.gov and the other registries it names. China’s primary registry, the Chinese Clinical Trial Registry (ChiCTR), is run by the Chinese Cochrane Centre at West China Hospital in Chengdu.
Two studies compared registered plans with what acupuncture trials published.
| Study | Sample | Registered before the trial started | Primary outcome changed |
|---|---|---|---|
| Su and colleagues, 2015 | 88 completed trials found in 15 registries, 74% from Western countries and 7 from mainland China | 17 of 88 | 32 of 71 publications that could be compared; 15 of the 21 changes with a clear direction moved toward a statistically significant outcome |
| Won and colleagues, 2019 | 322 English-language trials published 2013 to 2017, 56% from East Asia | 74 registered by a month after the start, and 61 more later; 187 not registered | 25 of 64 prospectively registered trials with a stated primary outcome; 15 of the 25 changes favoured a statistically significant finding |
Won and colleagues also found that more than half of the trials whose primary outcome was not statistically significant, 44 of 78, reported it in a way that suggested benefit, most often by claiming efficacy without addressing the primary result. Of the registered trials in their sample, 45% were on ClinicalTrials.gov and 21% on ChiCTR. Their sample was limited to trials published in English.
What the sources cannot settle
The studies here establish that trials from China dominate the literature by number, that the Chinese databases hold much of it outside Medline, that many reports called randomised in one large sample were not, and that Chinese-language trials report larger effects and carry more risk of bias. They do not settle why the effects differ: Vickers and colleagues offered publication bias as a possible cause, Li and colleagues pointed to differences between patient populations, and Gang and colleagues to differences in how trials are designed and run. None of the later studies cited here repeated the 1998 count of positive results country by country, and the registration studies sampled mostly English-language publications. Each AcuiQ protocol carries its citation, so a reader can see where and in what language the study behind it was published.