The only major flaw I see in the methodology is the single-blinded nature of treatment delivery, which may be enough for me to discount the conclusion but how would one refute such a study publicly?
I see a major flaw, but firstly lets look at other possibilities for bias, none of them particularly crucial, but bias nonetheless:
Patients were recruited by advertisements in local media
This means you do not have a random sample of patients with PAR entering your study, you have patients who specifically put themselves forward to be subjects in a trial of acupuncture for their PAR. This introduces selection bias.
The acupuncturist was not blinded, but was instructed not to communicate with the participants about treatment procedures and responses
Difficult to control for this one, but patients may be able to detect from the acupuncturists responses/reactions whether he is giving true or sham needling. Also as Zep has indicated, participants may have had acupuncture before - it would have been better if an exclusion criterion was "any acupuncture in the past".
For sham acupuncture, the insertion sites were 1–1.5 cm from the acupoints used for real treatment, and shorter needles (diameter, 0.25 mm; length, 13 mm; Suzhou Medical Appliance Factory) were applied in a shallow needling technique (3–5 mm).
13
Ideally a better study would have had subgroups in the design -
(1) "true" acupuncture"
(2) using sham acupoints with normal acupuncture technique and needles, and
(3) Shallow needle insertion at true acupoints but with shallow neeles and no de-qi being elicited.
This would have enabled the researchers to differentiate whether acupuncture works because sticking needles into someone elicits a response, or because it uses ancient chinese meridians and gobbledegook mysticism
four nasal symptoms (nasal obstruction, sneezing, rhinorrhoea and nasal itch) were self-assessed daily
Potential for bias. Objective outcomes are always best, but perhaps hard for something like PAR.
The sample size was determined on the basis of a 70% reduction in TNSS with real acupuncture and a 30% reduction with sham acupuncture.
13 Thus, a sample size of 36 for each group would provide 80% power with a type I error rate of 5% (two-tailed).
Am I being overly suspicious in noting that of the original 182 patients invited to submit to the study, only 80 finally entered (conveniently making just sufficient in each group for their power stats analysis)?
Now the major flaw:
Regression to the mean:
Look at box 3. This shows that at baseline, the 2 groups had quite a marked difference in symptoms (and this is
before any acupuncture was given!) The TNSS is 47 for the group that were selected to go on to have acupuncture, and 39 for the group to recieve acupuncture. The first group had worse symptoms (p = 0.058, quite a big difference really)
However, the researchers, quite cleverly, only consider changes from baseline. This is wrong! The reduction in symptoms in the true acupuncture group might well be largely explained by regression of symptoms to the mean.
The researchers should never do an analysis comparing groups from baseline unless the baseline was the same in both groups, which it clearly wasn't.
What they should have done was only randomise people to sham/true acupuncture
after they submitted baseline TNSS scores, to ensure that baseline scores were similar for the groups.
IMO this is the most serious design flaw with this study, and if this had been addressed then a statistically valid difference would not have been obtained for the overall analysis. The authors make no mention of this in the study discussion.
