There are, according to Mark Twain, lies, damn lies, and statistics. And you can, of course, prove just about anything with statistics, 66% of which, according to one wag, are wrong. Even the quote attributed to Mark Twain was apparently said first by British prime minister and novelist Benjamin Disraeli. But Walter Bagehot (the first editor of The Economist), the First Earl of Balfour, Sir Charles Dilke, and others have all been credited with the quotation. Nine out of ten people, though, still say it was Twain.
I’ve never really understood where a lot of statistics come from. When I read that Barack Obama has a less than 50% approval rating, or that 45% of people believe we should pull out of Afghanistan, who’s being asked? I never have been. Ever. And anyway, who cares about vox populi? I’ve always felt that the so-called Man in the Street has no business there, but should be in a library researching his facts before treating us – or the pollsters – to his opinions. And remember, these street sages are some of the same people who believe in flying saucers, or in ‘death panels’, or that the moon landing was actually staged in a Hollywood studio.
According to This Week magazine, 23% of all psychiatrists in the US are in New York City, causing one to wonder whether the mental health of New Yorkers is better than, say, that of the denizens of San Francisco. Then there was the USA Today report that counties with high levels of rainfall had concurrent high levels of autism, despite the fact that autism diagnoses have increased in all climates.
An outfit called STATS (published from George Mason University, USA) honors some of the worst abuses of statistics and science in the previous year in their ‘Dubious Data’ awards. They call out the ‘scorecard on US Health System Performance’ published by the Commonwealth Fund in 2008, which aimed to “measure and monitor health care outcomes, quality, access, efficiency, and equity” in the US. The results were, not surprisingly, dismal and growing steadily worse. Yet, STATS notes that many of the findings came not from patient outcomes but from surveys of patient satisfaction. “In other words, patients with no medical training decided whether the treatment they received was mistaken or unnecessarily repetitious.” Moreover, says STATS, “the scorecard mashed together studies without adjusting for different methodologies, sample sizes and collection techniques.”
So where was the 19th century French physician Claude Bernard when we needed him? He said that, “the first requirement in using statistics is that the facts treated shall be reduced to comparable units.”
Maybe we’re best off with Andrew Lang, a Scottish scholar (1844-1912) who is quoted in Familiar Medical Quotations thus: “He uses statistics as a drunken man uses lampposts – for support rather than illumination.” Interestingly, Lang was not a medical man but rather a polymath, best known for producing an annual volume of fairy tales.
What all this probably means for would-be statisticians and economists is more rigour in gathering and presenting statistics, and a healthy skepticism in interpreting and using them. Oh, and don’t forget to ask my opinion. Like a stopped clock, I’m right twice a day.
Dr Woods is the former editor of Health Outcomes Communicator.
We’ve all seen our share of silly headlines. Indulge me – my favourite one from just this week: ‘California state senator says he’s gay after DUI arrest’ (msnbc.com). Who knew that arrest for drunk driving changed one’s sexual orientation?
A surprising one from the BMJ: ‘Restricting sales of barbecue charcoal helps in suicide prevention, study says’. I had to read the article to figure out why barbecue charcoal would in any way increase the risk of suicide. As it turns out, suicide by carbon monoxide poisoning induced by charcoal burning in a sealed room was a ‘popular’ form of suicide in the late 1990s in Asia and is now beginning to occur outside Asia. Maybe the editor wanted an ambiguous headline, to prompt me to read the article?
We can laugh about these editorial blunders (except perhaps the senator), but they become a more serious matter when health or money is at stake.
I am forever trying to impress upon the millennial generation that words mean things, and in a world of abbreviated, impersonal, asynchronous communication, clarity is sine qua non. And although mathematics and scientific proofs in general often are described as elegant or beautiful, because of their inherent logic and order, they can be also be used to mislead. Careless miscommunication is one thing; misleading the audience is another story.
Dr David Woods provides an amusing and intelligent consideration of how we are unwittingly misled by statistics every day.
By Sian Claire Owen
The UK National Institute for Health and Clinical Excellence (NICE) has released its updated guide to their technology appraisal appeals process. Interested parties have until 22 March 2010 to comment on this document before the new version is set in stone. At 30 pages, it is dry material indeed, but important nonetheless.
Established in 1999, NICE was created to provide guidance on the use of health technologies within the UK National Health Service (NHS). For a new drug or technology to be available on the NHS, it must be approved by NICE. Technologies are assessed on clinical efficacy and cost-effectiveness.
Producing clinical guidance is a laborious process, taking up to 24 months (11 to 13 months if guidance is urgently needed), and the chance to appeal the Final Appraisal Decision (FAD) must be squeezed into this time frame. And ‘squeeze’ is the operative word – appellants have only 15 working days to appeal an FAD from the day it is published in any medium, including on the NICE website.
There is no doubt as to the importance of the appeals process. Barbara McLaughlan is the Campaigns Manager for the Royal National Institute for the Blind (RNIB) and Chair of Patients Involved in NICE (PIN), a group of patient organisations involved in NICE decision-taking. As she explains: “Having a NICE FAD appeals process is essential because the health economics that support NICE decision-making are not an exact science.”
“Decisions can give grounds to appeal and it is not in the interest of any stakeholders to forgo that option.” However, she adds that: “Any controversies about the assumptions used in the decision-making process should be ironed out during the ACD and FAD consultation process. Having to go to appeal is not in the interest of anybody, least the patient who is hoping to gain access to new treatments.”
The past year has seen some fundamental changes in the technology appraisal process to oil the wheels and make the system more efficient. Updated guides to multiple and single technology appraisal processes, and the Patient Access and Flexible Pricing Schemes were introduced, all of which have impacted the appraisal process – hence, the revised appeals manual.
Alice Law, a spokesperson for NICE says: “The aim of these changes is to ensure that the grounds of appeal are consistent with those outlined in the Secretary of State’s Directions. But the revised manual also provides greater clarification on the appeal process.”
This clarification gives us more information on the modus operandi of NICE appeals, as well as details on exactly what is acceptable and what is not. There also has been some ‘textual pruning’ to eliminate repetition between sections. This is undoubtedly useful information, saving first-time appellants precious time, and preventing them from becoming tangled in a bureaucratic quagmire.
Centred at the core of the FAD appeal process are three grounds, and any appeal is strictly limited to the scope of these grounds:
Ground One: The institute has failed to act fairly and in accordance with the appraisal procedures set out in the Institute’s Guide to the Technology Appraisal Process
Ground Two: The Institute has prepared guidance, which is perverse in the light of evidence submitted.
Ground three: The Institute has exceeded its powers.
These remain largely unchanged, with one minor revision in Ground Two, which now states:
Ground Two: The Institute has formulated guidance, which cannot reasonably be justified in the light of evidence submitted.
“This amendment was not intended to change the scope of appeals,” says Law. “We felt that ‘cannot reasonably be justified’ means the same as ‘perverse’ but was easier to understand,” which is in line with their aim of keeping it clear.
McLaughlin was involved in the NICE FAD appeals process brought forward by Derbyshire Primary Care Trust (PCT) on the decision not to approve Macugen (used in the treatment of macular degeneration) for use on the NHS. She agrees that greater clarification is needed. “Personally I would have expected one or two of the appeal grounds put forward by Derbyshire PCT and the manufacturer of Macugen to succeed. It was surprising to me that none of them did, and clarification of the process may have helped.”
The proposed appeals process also irons out any discrepancies in the time it takes to consider appeals – the final appeal decision should be considered within 10 working days of receipt, and the final guidance published on the NICE website 15 days after the Guidance Executive meeting. New updates recommend that the Department of Health apply funding direction as soon as the NICE guidance is published, in any medium, so consistency is key.
Given her experience with this process, McLaughlin adds that if she personally could introduce further changes to the guidelines, she would allow the introduction of new evidence and arguments in cases where the appeals process becomes lengthy. “Where appraisals stretch to a year and more there may well be occasions where new evidence becomes available that has a bearing on the decision about the safety, efficacy and cost-effectiveness of a new treatment,” she says. “In these cases it should be possible for the appeals panel to refer a decision back to the Appraisal Committee.”
So will McLaughlin and colleagues be commenting on these updated guidelines? “We are in the process of putting together a response,” she says. “Our draft will then be circulated among members of PIN, and then we will submit a joint response to strengthen the patient voice in these discussions.”
All in all, these alterations are designed purely to make the appeals process quicker and slicker. When asked who would benefit most from these changes, Law says: “The appeals manual is targeted towards consultees involved in technology appraisals who would like to lodge an appeal.” In other words everyone’s a contender, time willing.
The updated guide clarifies and standardizes the acceptable grounds for appeal and provides a clearer picture of the appeal process.
Mary Gabb, Editor
When I first read of the updated guidance from the UK’s National Institute for Health and Clinical Excellence (NICE) on their technology appeals process, I must admit – I yawned. It is important information…but, as an editor, it doesn’t exactly make for a sexy headline.
The temptation is to ignore such news, relegating it to the reading piles of health care policy wonks/boffins. But, guidance rules and appeals procedures affect and are important to several stakeholders – developers and manufacturers, health care providers, and patients. When you’ve got skin in the game (ie, when you have invested thousands of hours and enormous sums of money in trying to bring a new product to market), you want to know that everyone is playing by the same rules, and that you are getting fair consideration. Patients depend on appropriate NHS guidance to know they are getting the best treatment available and that their tax dollars are being correctly and fairly used to critically evaluate each new therapy or intervention, to get the best value for the pound (or dollar or euro as other countries study NICE procedures for possible use in their own countries). Health care providers rely on the technology appraisal process to ensure that they have the necessary tools to practice medicine effectively, without worrying about backroom deals or poor oversight.
Sian Claire Owens gives us a brief review of the updated NICE technology appeals guidance. She also spoke with NICE representatives to find out what the update really means for manufacturers.
With the growing push toward used of electronic health records (EHRs), will the computer get in the way of the physician-patient interaction – literally or figuratively? One study, possibly the first of patient satisfaction with EHRs in psychiatric office practice, provides a unique opportunity to observe the effect of computers in the office on the physician-patient interaction and suggests that the disruption – at least for the psychiatrist-patient relationship – was not as was feared. Psychiatry is an area of special concern because communication is the interaction between physician and patient and confidentiality is a sensitive issue.
Dr. Randall F. Stewart and his colleagues at the University of New Mexico Health Sciences Center, Albuquerque (USA), surveyed psychiatric outpatients about their satisfaction with care before and after the introduction of EHRs. They found that these chronically mentally ill patients were equally satisfied with their psychiatrists’ care regardless of whether the doctor was using a paper-based record-keeping system or a computer. Furthermore, the patients who were expected to have the greatest difficulty with computers-those with a psychotic disorder-did not have any decrease in satisfaction when EHRs were introduced.
Because EHRs have not been widely adopted in the US, outcomes researchers now have a window, as they are introduced, to explore their effects using observational studies, Dr. Stewart said. Only about 20% of US hospitals use EHRs; the proportion is lower for outpatient medical practices, and in psychiatry the subject of office computers has received virtually no attention.
The study was conducted in a clinic that provided care for 20 to 40 patients a day. Most patients were seen for medication management, though psychotherapy was provided to a minority. Clinically stable, adult psychiatric outpatients were administered a modified form of the validated Patient Satisfaction Questionnaire 18 (PSQ-18), with some subscales dropped and some additional, locally-developed subscales added, having to do with anxiety, confidentiality, and concerns about computer use. The final composite questionnaire consisted of 23 questions: 1 measuring general satisfaction and 7 subscales addressing satisfaction with a variety of aspects of the doctor-patient interaction, most not directly concerned with computer use (Table 1).
|Table 1: Subscales of the composite patient satisfaction questionnaire|
Overall. General satisfaction with psychiatric care.
Technical. Satisfaction with knowledge and abilities of the psychiatrist and how the office is equipped.
Interpersonal. Manner of psychiatrist, whether businesslike and impersonal or friendly and courteous.
Communication and Education. How well the psychiatrist explains the reasons for tests, answers questions, and understands and pays attention to what the patient says.
Time. Spending sufficient time with the patient.
Confidentiality of the patient record.
Anxiety about the future or about psychiatric care.
Computer use. Is the patient comfortable with it, or does it get in the way?
The questionnaires were administered to patients upon discharge from outpatient treatment during a period of paper record keeping and another period, beginning 4 months after electronic charting had been implemented so that psychiatrists had a chance to become comfortable with it.
The researchers found no difference in overall satisfaction or in any subscale of satisfaction between the 161 patients surveyed pre-EHR and the 141 surveyed post-implementation. This result was somewhat surprising, according to Philip J. Kroth, MD, a co-author. “We found the effect of the computer on patient satisfaction in psychiatry is really no different than previous studies have shown for traditional medical patients. There are apparently no special problems in the behavioural health care arena.” When the investigators stratified the results by primary diagnosis, they found another surprising result: patients with schizophrenia, despite difficulty processing reality and a tendency to view the recording of information with suspicion, were no more likely to dislike electronic than paper records, Dr Kroth said. The study was statistically powered to detect a 7% difference in satisfaction between encounters using paper records and the EHR; thus it is unlikely that the study failed to detect a large difference in satisfaction.
Anecdotally, Dr Stewart noted: “some physicians use computers very well. They greet the patient and give him their undivided attention, excusing themselves when they want to record or look something up on the computer.” Some even use the computer as a teaching tool-for example, showing the patient a correlation between blood drug levels and depression scores. “Patients tend to like it when the physician shares with them the information that’s on the computer screen,” he said. In contrast, in some offices the computer is placed against the wall and “the physician has his back to the patient and is asking questions over his shoulder while typing away.” Dr Stewart said that the study deliberately did not measure specific aspects of how psychiatrists implemented the EHR. Though this is a limitation of the study, the investigators thought it better to avoid intruding into the privacy of the office and inducing stress.
Another limitation, he said, is that the study did not measure outcomes directly, only patient satisfaction, a surrogate. “We were really interested in the dynamics of the psychiatrist-patient relationship, and patient satisfaction seemed the most readily available way to measure the quality of the interaction,” he said. He added that patient satisfaction is widely measured as a surrogate for quality of care, and the PSQ-18 provides a benchmark to compare this with other studies; also it is validated, freely available and easy to complete. However, it should be noted that their modified PSQ-18 was not validated.
Finally, the authors note that the participating patients may have been a limitation, in that the sampling strategy may have been biased toward patients who were more likely to participate, precisely because they were satisfied with the encounter. One can imagine that many patients are likely to say to their physician only, “I’m doing fine, thanks,” especially if they are there just to get a prescription refill.
The transition to EHRs is presenting outcomes researchers with ‘golden opportunities’ to study their effect using before-and-after designs, opportunities that will not always be available, he said.
In the UK, adoption of EHRs has been in fits and starts. While their impact in psychiatric practice has not been researched, it would not be surprising if they had little effect on the psychiatrist-patient relationship there too, according to Stefan Priebe, FRCP, Professor of Social and Community Psychiatry at Barts and the London School of Medicine and Dentistry, Queen Mary University of London, and an investigator of physician-patient communication.
Unlike the US, electronic records in the UK are being implemented from the top down, by the National Health Service via the country’s regional Trusts, he said. Implementation has been slow and uneven, delayed by controversy and unexpected costs, and has been a political issue since the effort began in 2002.
Most psychiatrists and other clinicians in the UK view EHRs as an additional burden, of uncertain benefit, Dr Priebe said. Where EHRs are used in the mental health setting, most psychiatrists prefer to keep a paper record and enter only the minimal required data on the computer when the patient is no longer present. The detailed notes, including sensitive information such as family history and suicide risk, typically don’t go into the electronic record, he said. Because patients’ EHRs are centrally accessible, there are real confidentiality issues; unlike the US, many NHS employees may have access to patients’ personal information. However, Dr. Priebe said that in his clinical experience, confidentiality of medical information is not a very important concern for UK patients with severe mental illness.
Stewart RF, Kroth PJ, Schuyler M, Bailey R. Do electronic health records affect the patient-psychiatrist relationship? A before & after study of psychiatric outpatients. BMC Psychiatry. 2010;10:3. doi:10.1186/1471-244X-10-3.
McCabe R, Priebe S, Communication and psychosis: it’s good to talk, but how? Br J Psychiatry. 2008;192:404-405. doi: 10.1192/bjp.bp.107.048678
I have a love-hate relationship with technology. I admire the way old-order groups such as the Amish or Mennonites restrict the use of new technologies until their societal effects have been carefully considered and analysed. Yet, rapid uptake of new technology is essential for me to work from a home office, access medical articles when I cannot get to a medical library, and travel for business meetings while still being (at least somewhat) productive.
There is a big push now to use electronic medical records, but how will this work in a medical setting, especially psychiatry? The idea of pouring out my heart and soul to a psychiatrist is intimidating on its own, let alone imagining him or her tapping away on a keyboard as I explain my deepest insecurities.
And yet, when it comes right down to it, patients don’t seem to mind the use of a computer in the doctor’s office/surgery. Am I being oversensitive?
It may be that most patients, when speaking with their physicians, are probably too focused on their own concerns and being able to vocalize them clearly to be worried about the etiquette or awkwardness of computer use.
Kate Casano describes the results of a study among psychiatric patients whose physicians began to incorporate electronic health records into the physician-patient encounter. The results are surprisingly… dispassionate. And as the study authors point out, attention to communication style, interpersonal manner, and computer proficiency are probably bigger players in how well technology is incorporated into such a sensitive setting. Click here to view her article.