This material is abstracted with permission from the RAND Corporation.
The RAND/UCLA Appropriateness Method (RAM) was developed in the mid-1980s, as part of the RAND Corporation/University of California Los Angeles (UCLA) Health Services Utilisation Study, primarily as an instrument to enable the measurement of the overuse and underuse of medical and surgical procedures. An appropriate procedure is one in which "the expected health benefit (e.g., increased life expectancy, relief of pain, reduction in anxiety, improved functional capacity) exceeds the expected negative consequences (e.g., mortality, morbidity, anxiety, pain, time lost from work) by a sufficiently wide margin that the procedure is worth doing, exclusive of cost" (Brook et al., 1986; Park et al., 1986).
The rationale behind the method is that randomised clinical trials—the "gold standard" for evidence-based medicine—often are either not available or cannot provide evidence at a level of detail sufficient to apply to the wide range of patients seen in everyday clinical practice. Although robust scientific evidence about the benefits of many procedures is lacking, physicians must nonetheless make decisions every day about when to apply them. Consequently, it was believed a method was needed that would combine the best available scientific evidence with the collective judgment of experts to yield a statement regarding the appropriateness of performing a procedure at the level of patient-specific symptoms, medical history and test results.
To measure underuse, the RAND/UCLA method was expanded to measure the necessity of clinical procedures (Kahan et al., 1994). Necessity is a more stringent criterion than appropriateness and refers to procedures that must be offered to patients fitting a particular clinical description. Necessity is more difficult to measure than appropriateness, however, because it involves identifying a group of patients who might have benefited from the procedure, but did not receive it.
Overview of Method
First, a detailed literature review is performed to synthesize the latest available scientific evidence on the procedure to be rated. At the same time, a list of specific clinical scenarios or "indications" is produced in the form of a matrix which categorizes patients who might present for the procedure in question in terms of their symptoms, past medical history and the results of relevant diagnostic tests. These indications are grouped into “chapters” based on the primary presenting symptom leading to a patient’s being referred for treatment or considered for a particular procedure.
A panel of experts is identified, often based on recommendations from the relevant medical societies. The literature review and the list of indications, together with a list of definitions for all terms used in the indications list, are sent to the members of this panel. For each indication, the panel members rate the benefit-to-harm ratio of the procedure on a scale of 1 to 9, where 1 means that the expected harms greatly outweigh the expected benefits, and 9 means that the expected benefits greatly outweigh the expected harms. A middle rating of 5 can mean either that the harms and benefits are about equal or that the rater cannot make the judgment for the patient described in the indication.
The panelists rate each of the indications twice, in a two-round "modified Delphi" process. In the first round, the ratings are made individually at home, with no interaction among panelists. In the second round, the panel members meet for 1-2 days under the leadership of a moderator experienced in using the method. Each panelist receives an individualized document showing the distribution of all the experts’ first round ratings, together with his/her own specific ratings. During the meeting, panelists discuss the ratings, focusing on areas of disagreement, and are given the opportunity to modify the original list of indications and/or definitions, if desired. After discussing each chapter of the list of indications, they re-rate each indication individually. No attempt is made to force the panel to consensus. Instead, the two-round process is designed to sort out whether discrepant ratings are due to real clinical disagreement over the use of the procedure ("real" disagreement) or to fatigue or misunderstanding ("artifactual" disagreement).
Finally, each indication is classified as "appropriate," "uncertain" or "inappropriate" for the procedure under review in accordance with the panelists' median score and the level of disagreement among the panelists. Indications with median scores in the 1-3 range are classified as inappropriate, those in the 4-6 range as uncertain, and those in the 7-9 range as appropriate. However, all indications rated "with disagreement," whatever the median, are classified as uncertain. "Disagreement" here basically means a lack of consensus, either because there is polarization of the group or because judgments are spread over the entire 1 to 9 rating scale.
If necessity criteria are also to be developed, a third round of ratings takes place in which panelists are asked to rate the necessity of those indications that have been classified as appropriate by the panel. The definition of necessity (Kahan et al., 1994a) is that:
- The procedure is appropriate, i.e., the health benefits exceed the risks by a sufficient margin to make it worth doing.
- It would be improper care not to offer the procedure to a patient.
- There is a reasonable chance that the procedure will benefit the patient.
- The magnitude of the expected benefit is not small.
All four of the preceding criteria must be met for a procedure to be considered as necessary for a particular indication. To determine necessity, indications rated appropriate by the panel are presented for a further rating of necessity. This rating is also done on a scale of 1 to 9, where 1 means the procedure is clearly not necessary and 9 means it clearly is necessary. If panelists disagree in their necessity ratings or if the median is less than 7, then the indication is judged as "appropriate but not necessary." Only appropriate indications with a necessity rating of 7 or more without disagreement are judged "necessary."
Figure 1: The RAND/UCLA Appropriateness Method
For more information please see the RAND manual at http://rand.org/pubs/monograph_reports/MR1269/index.html
Brook RH, Chassin MR, Fink A, et al. A method for the detailed assessment of the appropriateness of medical technologies. International Journal of Technology Assessment in Health Care. 1986; 2(1): 53-63.
Kahan JP, Bernstein SJ, Leape LL, et al. Measuring the necessity of medical procedures. Medical Care 1994; 32: 357-365.
Park RE, Fink A, Brook RH, et al. Physician ratings of appropriate indications for six medical and surgical procedures. American Journal of Public Health. 1986; 76 (7): 766-772
Expert Panel Meeting
Expert Panelists, Moderators and Principal Investigator
From left to right: Back Row - Sam Wiebe, Richard Huntsman, Jeptha Davenport, David Steven, , Jorge Burneo, Walter Hader, Matt Wheatley, Carter Snead, Bill Ghali,
Front Row - Lionel Carmant, Mark Sadler, Nathalie Jetté, Fred Andermann, Jean-Pierre Farmer, Peter Camfield, Don Gross
- Seizure free: free of all seizure types at the time of the clinical assessment based on the physician’s opinion.
The “seizure free” category should be considered less severe than the “less than one seizure per year” category.
Seizure severity: Disabling seizures
- Disabling seizures are serious enough to cause injuries or other significant medical problems (e.g. cardiac arrhythmia) or which result in psychosocial consequences (e.g. inability to work, causing serious psychological distress, etc). In children, especially those under age 3, disabling seizures are seizures associated with lack of expected developmental progress.
- Whether a seizure is disabling or not is based on the patient/caregiver’s opinion along with the physician’s assessment.
- Non disabling seizures were defined as “minimally impactful”. Examples:
- Potentially non disabling simple partial seizures: Seizures which involved few seconds of tingling in ones non dominant hand without functional or social impairment.
- Potentially non disabling complex partial seizures or generalized tonic clonic seizures could be seizures which only occur at night infrequently in any patient or those which are brief (<30 seconds) and do not affect activities of daily living in a developmentally delayed patient.
Antiepileptic drugs (AEDs)
- # of AEDs refers to number of adequate AED trials (in monotherapy or polytherapy). An adequate AED trial refers to:
- Adequate dose for patient age/weight
- Therapeutic serum levels (if applicable)
- No side effects that resulted in discontinuation or insufficient dose
- Appropriate for epileptic syndrome (i.e. localization related or partial epilepsy)
- Any response to a drug which is noxious and unintended, and which occurs at doses normally used in humans for prophylaxis, diagnosis, or therapy of disease, or for the modification of physiological function (WHO, 1972).
- Abnormal EEG includes any of:
- Focal slowing
- Any epileptiform discharges
- Any focal abnormalities that are not consistent with a normal variant (e.g. focal bursts of paroxysmal fast activity would be considered as a relevant focal abnormality; however wicket spikes or RMTD also known as rhythmic mid temporal theta of drowsiness would not be relevant as these are normal variants).
- Abnormal MRI
- Any lesions which could potentially be epileptogenic (e.g. changes consistent with mesial temporal sclerosis, tumor, encephalomalacia, vascular lesion, etc).
- Excludes incidental findings (e.g. pineal gland cyst, pituitary abnormalities) and lesions involving the brainstem or cerebellum.
- Caution: normal MRI = normal based on typed radiologist’s report.
- It is best to have an epilepsy-specific MRI protocol before calling an MRI normal. However, this may not be available. The best available MRI protocol should be used.
Engel, J. Seizures and epilepsy. Philadelphia: F.A. Davis Company, 141-157, 1989.
Glauser, TA and Morita DA. (2006). Infantile Spasm (West Syndrome). Available: http://emedicine.medscape.com/article/1176431-overview. Last accessed 4 January 2009.
World Health Organization, International drug monitoring: The role of national centres. Techn Rep Series WHO, Geneva, n. 498, 1972.