Lessons in disappointment and some New Year’s resolutions

Broken your New Year’s resolutions yet? Research from the US suggests just 8% of people achieve theirs and we know that if you’re hanging onto them into February you’re doing really well. When I thought about making a list of resolutions which clinical trialists and systematic reviewers might like to follow (the cheek of it!), it looked like a welcome distraction from deciding I’m going to take up running (again), eat only healthy snacks and get 10 hours sleep a night. But I find I am coming to this with a rising sense of sadness, disappointment and, well, outrage about some of the evidence gaps. Gaps which leave people struggling with health problems, and those trying to help them, without clear guidance on which interventions might help; and, indeed, on which may make things worse.

I saw the titles of several newly published or updated Cochrane reviews at the tail end of the year and thought, great, these look important. But most of them were ’empty reviews’, with no evidence at all, or fell into that all too familiar category of reviews where the reviewers were unable to come to clear conclusions because of problems with the evidence.

Pass the parcel – what no present?!

Pass the parcel – a lesson in disappointment!

Many of us will remember this game from our childhood. A parcel is handed round and when the music stops the person holding it gets to unwrap a layer of paper. The last one finds a present, but only after lots of people have had their hopes raised only to find nothing under the wrapper. It’s a bit like this with ’empty reviews’. The review team sets out to answer an important clinical question by devising complicated search strategies and using them to hunt through vast databases of clinical trials, looking for some relevant studies to include. Then find – none.

Here are the pressing needs addressed in those recent reviews:

• alternatives to hospital follow-up after treatment for gynaecological cancer
• effective ways to manage faecal incontinence, constipation and urinary catheters in people with neurological diseases
• guidance on the use of prophylactic antibiotics for people with penetrating abdominal injuries
• preventing the spread of MRSA among older people in nursing homes
• whether end-of-life care pathways are an effective means of providing high quality care

This is really important stuff, right? Take the first one, follow-up after gynaecological cancer treatment to detect recurrences, a new review published in November. This usually involves multiple hospital visits that cause anxiety among the women, its cost effectiveness is questionable, and there’s no evidence that this method identifies recurrences earlier or improves overall survival. Patient groups have called for a consideration of alternatives, such as using patient-reported outcome measures (PROMs) questionnaires through which people can give detailed information about their health and associated quality of life. These are increasingly being used as part of routine follow-up. Bring on a systematic review. The review team didn’t expect to get randomized controlled trials (RCTs) so looked for studies of other designs too. They found none.

You might think that it’s just a matter of research catching up with changes in practice. But we have been doing things for a long time for which there’s still no research evidence. Since World War II antibiotics have been given to people with penetrating abdominal injuries in the hope of preventing wound infection. But is this best practice? Problems of antibiotic resistance, side effects and costs are reasons not to use antibiotics unnecessarily and then there’s debate about timing. A Cochrane review on this has recently been updated for the second time and still there were no trials to include. Guidelines, of necessity, remain expert-based not evidence-based.

Then there’s bladder and bowel business. Can there be many things which would make more difference to the quality of life of people who have severe problems with these than things to help manage them effectively and alleviate problems? For the person with nerve damage affecting their bladder, leaking urine or problems with passing it, and a risk of infection and kidney damage, have to be managed, but how best to do it? How about a permanent catheter, or perhaps use one intermittently, or forget the catheter and pass urine at timed intervals? It’s important to know how effective these are and also how easy to use, how comfortable and, for the NHS purse, how they compare in terms of costs. A review addressing this was first published a decade ago and has recently been updated again, but still found no trials and can’t offer any evidence to guide practice.

A review on managing faecal incontinence and constipation in people with neurological diseases has just been updated for the third time and despite the addition of 15 trials, bringing the total to 20, the reviewers are still unable to draw any reliable conclusions due to the limitations of the evidence. These displayed some all too familiar problems, not least poor reporting. ‘Usual care’ is a common comparator in trials but utterly meaningless unless you explain what the ‘usual care’ was. In most of the trials in this review, the authors left us to guess.

Small sample sizes in the trials included in this review meant that confidence intervals were wide, so we can’t be very sure that the findings are reliable. Then there was such variety in the choice of outcome measures that putting together the results was rendered inappropriate. Even identifying where future research activity should be focused was made difficult for the reviewers because of the lack of condition-specific, validated outcome measures, little evidence to support any intervention and, crucially, “an absence of reliable information about what is important to patients regarding the outcomes of bowel management”. Which brings me to the New Year’s resolutions list. The high quality research we need demands attention to all aspects of the way it’s designed and carried out, but here are the things at the top of my list. I hope we might have a blog on each of these points over the coming months, but right now I’m going to point you to some places you should go if you’re interested.

• Identify priorities for research (both primary research and systematic reviews). The James Lind Alliance brings patients and clinicians together in Priority Setting Partnerships to prioritise topics for research and you can find them here
• Get the outcomes right. This should include the development of core outcome sets for specific populations. Validated measures that focus on things that matter to patients and are common to research investigating the same area will really help the evaluation of interventions and the pooling of results in systematic reviews. The COMET (Core Outcome Measures in Effectiveness Trials) Initiative brings together people interested in the development and application of agreed standardised sets of outcomes. Trialists wouldn’t be restricted to that set; rather, it would be expected that these outcomes would be collected and reported, along with others that the researchers wish to explore. Read more here
• Better reporting (poor reporting is the killer for much research that may be good but is rendered unusable by all that’s left out at the reporting stage). You need to dash over to The EQUATOR network, which “works to improve the reliability and value of medical research literature by promoting transparent and accurate reporting of research studies” and has links to so many useful resources on this topic  http://www.equator-network.org/

‘Empty’ things sometimes contain something valuable!

When ’empty’ isn’t entirely empty

Empty reviews lack the stuff we hoped for – relevant trials to help answer the question – but they do contain other valuable things. As Joanne Yaffe and colleagues pointed out in their study of empty Cochrane reviews, the absence of studies might reflect a topic area that is too new to be ready for review, or perhaps priorities of health care decision makers and researchers don’t much overlap, but either way the absence of evidence might help stimulate appropriate research. The hope is that the review will then be updated with eligible studies. Good reporting is vital here too; you can see a poster on this and more at the Cochrane Empty Reviews Project website.

Happy New Year!

Links:

Nama V, Nordin A, Bryant A. Patient-reported outcome measures for follow-up after gynaecological cancer treatment. Cochrane Database of Systematic Reviews 2013, Issue 11. Art. No.: CD010299. DOI: 10.1002/14651858.CD010299.pub2.

Brand M, Grieve A. Prophylactic antibiotics for penetrating abdominal trauma. Cochrane Database of Systematic Reviews 2013, Issue 11. Art. No.: CD007370. DOI: 10.1002/14651858.CD007370.pub3

Jamison J, Maguire S, McCann J. Catheter policies for management of long term voiding problems in adults with neurogenic bladder disorders. Cochrane Database of Systematic Reviews 2013, Issue 11. Art. No.: CD004375. DOI: 10.1002/14651858.CD004375.pub4.

Coggrave M, Norton C. Management of faecal incontinence and constipation in adults with central neurological diseases. Cochrane Database of Systematic Reviews 2013, Issue 12. Art. No.: CD002115. DOI: 10.1002/14651858.CD002115.pub4.

Hughes C, Tunney M, Bradley MC. Infection control strategies for preventing the transmission of meticillin-resistant Staphylococcus aureus (MRSA) in nursing homes for older people. Cochrane Database of Systematic Reviews 2013, Issue 11. Art. No.: CD006354. DOI: 10.1002/14651858.CD006354.pub4.

Chan RJ, Webster J. End-of-life care pathways for improving outcomes in caring for the dying. Cochrane Database of Systematic Reviews 2013, Issue 11. Art. No.: CD008006. DOI: 10.1002/14651858.CD008006.pub3.

Yaffe J, Montgomery P, Hopewell S, Shepard LD (2012) Empty Reviews: A Description and Consideration of Cochrane Systematic Reviews with No Included Studies. PLoS ONE 7(5): e36626, doi:10.1371/journal.pone.0036626