Does the Chinese Literature Indicate Larger Effect Sizes? This Might Indeed be the Case

Article information

Perspect Integr Med. 2023;2(3):202-203

Publication date (electronic) : 2023 October 23

doi : https://doi.org/10.56986/pim.2023.10.009

Tae-Hun Kim¹, Jung Won Kang²

¹Korean Medicine Clinical Trial Center, Korean Medicine Hospital, Kyung Hee University, Seoul, Republic of Korea

²Department of Acupuncture & Moxibustion, College of Korean Medicine, Kyung Hee University, Seoul, Repulic of Korea

^*Corresponding author: Tae-Hun Kim, Korean Medicine Clinical Trial Center, Korean Medicine Hospital, Kyung Hee University, Seoul, Republic of Korea, Email: rockandmineral@gmail.com

Received 2023 October 11; Revised 2023 October 16; Accepted 2023 October 16.

Dear editor,

We have recently read with interest, a meta-epidemiological study on effect sizes in Chinese literature, published in BMC Medical Research Methodology [1]. The issue has long been the subject of debate, addressed by the authors, and has been recognized by scholars as having empirical evidence [2]. Having been involved in complementary and alternative medicine research for several years and having encountered a significant amount of Chinese literature, we have always been mindful of these concerns. We are grateful that this meta-epidemiological study revisited this issue and provided conclusions that support our long-standing observations on this matter. According to the authors’ findings, acupuncture trials reported in Chinese showed significantly larger effect sizes than those in non-Chinese languages (ratio odds ratio 0.51, 95% confidence intervals 0.29–0.91) and they concluded that high risk of bias and study population are significant factors affecting the large effect size [1]. We would like to address and discuss some of these points.

First, the research methodology of this study needs to be revisited. Rather than directly searching for original trials to be included in the analysis, the authors located individual studies from Cochrane reviews, which may have inadvertently introduced selection bias. Depending on author composition and research objectives, Chinese literature may not always be included in Cochrane reviews [3]. According to the authors, only 37% of the 84 systematic reviews included in this study were found in Chinese databases. Thus, there is a considerable possibility of limiting the number of Chinese publications.

Second, we would like to mention the method used to analyze differences between comparison groups. The authors calculated ratio odds ratios to examine differences between articles published in Chinese and non-Chinese languages, or between Chinese and other populations. However, it might have been more appropriate to analyze the impact of publication language and study population on effect size using meta-regression.

Third, the reasoning process leading to the conclusion seems to have some issues. The authors attempted to demonstrate that the language of the trial (Chinese or non-Chinese) is not an influential factor in explaining large effect sizes by comparing studies with a high or unclear risk of bias. However, estimating the actual impact through a comparative analysis of all studies and presenting comparisons between high-risk and low-risk subgroups may provide greater insight. Similarly, in the context of comparing Chinese and non-Chinese populations, a fair interpretation can solely be achieved through the concurrent presentation of subgroup results for studies involving both high and low risk of bias. Using the methodology employed in this study to analyze data could result in a potentially biased perspective. For instance, this methodology does not completely address Chinese-language publications with significant effect sizes and with a low risk of bias. Furthermore, the correlation between Chinese-language publications and effect size in studies of the Chinese population is not covered in the illustrated method. Evidently, issues related to the research question are inadequately covered by the current method.

In conclusion, it is critical to exercise caution when conducting a comprehensive review of the literature, including publications from China. Whether in a positive or negative way, the volume of clinical evidence on acupuncture is significantly influenced by studies conducted and reported from China. Continual endeavors to enhance the caliber of research are imperative, irrespective of its location.