Training session for SPs

Why standardized patients? Relative to other methods of assessing quality such as direct clinical observations, inspection of medical records (where they exist), and patient exit interviews, the use of standardized patients (SPs) presents several advantages, and is therefore widely regarded as the “gold standard” in assessing the quality of medical care.¹

First, data from SPs yield an assessment of provider practice that is free from potential observation bias, where the doctor changes his/her behavior because she knows she is being observed,² is less vulnerable to recall bias than patient exit interviews,³ and is more complete than what doctors might record themselves in medical records.⁴

Second, SPs permit estimates of case detection rates since illnesses are pre-specified in the study design. We will show below that providers’ diagnoses are often inaccurate, so that methods based on medical records or clinical observations may not yield accurate data on the true illness of the observed patients. ⁵

Finally, because all case presentations are standardized, the SP methodology allows for valid quality comparisons across different types of doctors and clinics. Poorer patients or patients with more complicated symptoms might choose particular providers. Thus, data based on real patients could confound true differences in provider quality with differences in the characteristics of providers’ patients.

SPs and Team in Indore preparing for village deployment

For more exploration of SP methodology please visit the University of Toronto, Standardized Patient Program (SPP)

 http://www.spp.utoronto.ca

and 

The Association of Standardized Patient Educators  

http://www.aspeducators.org

large group exit questionnaire training session

Cases: The 3 cases selected for presentation by the SPs in this study (MI, asthma, or dysentery of an absent child) allow assessment of provider quality across a range of conditions that are relevant in the Indian context. Rates of cardiovascular and respiratory disease have been increasing, and diarrheal disease kills more than 200,000 children per year. ^6-8 ­.

Checklists: The Indian government’s National Rural Health Mission (NRHM) has developed triage, management, and treatment protocols for MI, asthma, and dysentery in public clinics, ^9-11suggesting clear steps to be taken for patients presenting with any of these conditions.

Detection: Follow-up visits to the private providers in Delhi yielded detection rates below 1 percent of interactions, suggesting that the SP-based presentations did not arouse suspicion and cases were treated as they would be under normal circumstances. In MP, no provider voiced any suspicions and the detection rate can thus be considered zero.

Limitations: There are several limitations to the SP-based approach. First, to avoid potential harm to SPs, cases were restricted to those that do not require invasive examinations, including the use of a thermometer. Thus, the estimates of quality may not generalize to communicable diseases.

Second, SPs were unknown to the providers they visited, which potentially biased interactions towards emergency care. In the case of proxy-dysentery, providers may have behaved differently had the patient actually been present. However, data from direct observations of clinical encounters between real patients and providers in the same clinics suggest similar process quality measures.

Third, the study did not audio-record interactions to verify SP recall, although other studies have shown high rates of recall and the one hour limit on the time elapsed between the interaction and the SP exit interviews should have minimized the risks of recall bias.

Finally, there is some inherent subjectivity in determining which history questions and exams are essential and recommended. The low completion rates of most items suggest that deleting some of the items from the checklists would not appreciably change the percentage of completed items. Adding items would only make the quality estimates even lower than they currently appear.

We refer you to our field guide, manual and sample instruments, Standardized Patients and Measurement of Healthcare Quality: http://www.spp.utoronto.ca/images/stories/Resources/spmanualfieldguide_012012.pdf

Average  quality outcomes in SP-provider interactions: full sample and by case

Exhibit 3

Exhibit A-1

Exhibit A-2

Variations by sector and qualifications (Asthma Only)

Completion of specific checklist item: For unstable angina, providers asked about the location of pain in 62% of interactions. All other checklist items, including a question about pain radiation, were completed in less than 46% of interactions. Although most providers possessed the required equipment, completion of vital signs or basic examinations such as pulse rate (36%), blood pressure (32%) and chest auscultation (43%) was notably low.

Completion rates were even lower for asthma, where adherence to protocol occurred most often for chest auscultation (52%) and questions about the presence of cough (61%) and symptom onset (51%). All other checklist items were completed in 37% or fewer interactions.

For dysentery, the child’s age was asked in 93% of interactions, but other important questions much less frequently: stool quality (20%), stool frequency (29%), and presence of a fever (18%).

Diagnosis and treatment: In cases where a diagnosis was articulated, the proportions of interactions that resulted in a correct diagnosis were 8.8% (95% CI:  3.8 to 13.8) for unstable angina, 22.5% (95% CI: 14.9 to 30.1) for asthma, and 9.5% (95% CI: 0.27 to 18.8) for dysentery. The proportions of interactions that resulted in partially correct diagnoses were 43.2% (95% CI: 34.4 to 52.0) for unstable angina, 36.7% (95% CI: 27.9 to 45.4) for asthma, and 54.8% (95% CI: 39.1 to 70.5) for dysentery.

Partially correct diagnoses refer to the following: heart-related problems for the unstable angina case; allergies or breathing problems for the asthma case; and diarrhea for the dysentery case. It is important to note that the SPs themselves mentioned a breathing problem or diarrhea for the asthma and dysentery cases. Wrong diagnoses refer to the following: blood pressure problem, gas, muscle problem, the weather, or other problems for the unstable angina case; blood pressure problem, gas, heart problem, the weather, or other problems for the asthma case; and the weather or other problems for the dysentery case.

These estimates of correct diagnosis, however, have not been calculated from all interactions since SPs received a diagnosis only 31% of the time. The true proportion of correct diagnosis in the entire sample depends on the unobserved likelihood of a correct diagnosis in interactions where no diagnosis was articulated. If the providers that did not articulate a diagnosis were more likely to have made an incorrect diagnosis had they been forced to articulate one, then our estimates of correct diagnosis represent upper-bounds of the true case detection rate in the sample. Other empirical patterns in the data suggest that this may indeed be the case since articulation of a diagnosis is positively associated with other quality measures, as is articulation of a correct diagnosis. Providers who articulated any diagnosis completed more essential items (42%, 95% CI: 39 to 45) than those who said nothing (29%, 95% CI: 27 to 31), and among those articulating any diagnosis, those who diagnosed correctly completed more essential items (53%, 95% CI: 46 to 60) than those who diagnosed incorrectly (32%, 95% CI: 27 to 36). Thus, it would be reasonable to assume that those who said nothing would have been less likely to make a correct diagnosis.   

For unstable angina, adherence to guidelines was low: 32% referred the SP to another provider, 24% referred the SP for an ECG, 5% prescribed nitroglycerin, and 4% prescribed aspirin. For asthma, SPs were prescribed an inhaler in 2% of interactions and steroids in 20%.

When medicines came with a label, there was widespread use of antibiotics: 15% for unstable angina, 33% for asthma, and 56% for dysentery (when the provider did not refuse to see the SP without the child). These figures are suggestive of the indiscriminate use of antibiotics considered to be contributing to high rates of antibiotic resistance in India. While the results for dysentery suggest that many providers could be treating the case appropriately, among providers who did articulate a diagnosis, providers who made a wrong diagnosis for dysentery were just as likely to prescribe antibiotics (73%, 95% CI: 48 to 99) as providers who gave a correct or partially correct diagnosis  (70%, 95% CI: 52 to 89).

^6.Jindal SK, Gupta D, Aggarwal AN, Agarwal R. Guidelines for management of asthma at primary and secondary levels of health care in India (2005). Indian J Chest Dis Allied Sci. 2005; 47(4):308-343

^7. Ischaemic heart Disease: Acute Mycardial Infarction [Internet]. Available from: http://mohfw.nic.in/nrhm/stg/Contents.htm

^8. Integrated Management of Neonatal and childhood illness: Physician Chart Booklet [Internet]. Available from: http://www.mohfw.nic.in/NRHM/IMNCI/IMNCI_Physician_Chart_Booklet.pdf

^9. Das J, Hammer J. Money for nothing: the dire straits of medical practice in Delhi, India. J Deve Econ. 2007; 83:1-36.

^10. Chaudhury N, Hammer J, Kremer M, Muralidharan K, Rogers FH. Missing in action: teacher and health worker absence in developing countries. J Econ Perspect. 2006; 20(1):91-116.

^11.  Sood R. Medical Education in India. Med Teach. 2008; 30(6):585-591.