Question
Why do residencies place so much weight on USMLE scores in the interview decision and application process?
Answer
Great question! Program Directors know for sure there is a lot more to being a great anesthesiologist than test scores. Attributes such as professionalism, communication, work ethic, interest in lifelong learning, working to improve our practice, and advocacy are all essential (and often considered for AOA status). Our very best residents also distinguish themselves by excelling in those areas, but since no objective measures exist for these attributes most residencies fall back on USMLE scores as one way to screen applicants for interviews. Understanding that not everyone tests well, we look at the potential of the person as a whole, and what unique things they offer.
Although I have not formally studied the theory of written test assessments in education, I do know that the USMLE as currently designed is a criterion based test, not a norm based test. Most tests created by high school teachers, for example, are criterion-referenced tests where the objective is to see whether or not the student has learned the material. On the other hand, a norm-referenced test yields an estimate of the position of the tested individual in a particular population. The SAT is a norm based test. This normative assessment refers to the process of comparing one test-taker to his or her peers.
To illustrate the potential difficulty in using a criterion based test such as the USMLE for ranking applicants, let us say that to be a competent doctor, knowledge wise, it has been determined that you have to get more than 65% of the questions correct on the USMLE. Theoretically, you could have a situation where everyone that takes step 1 (or step 2) gets 98% of the questions correct. From a criterion point of view all the students know the material, and would qualify for medical licensure by the state. But the person answering only 90% of questions correctly (if the USMLE were also used for normative purposes) would be deemed to be in a low percentile of his cohort!
Another disadvantage of a norm-referenced test like the SAT is that it cannot measure increases in knowledge of the population as a whole, for instance as might occur after completing a math or english class. The norm referenced test indicates only where individuals fall within the whole, so that if the entire class knows more and scores higher the middle student will still be 50th percentile.
For the USMLE, as you know, the 3-digit score is calculated to ensure that scores from different yrs are on a common scale. The 2-digit score is derived from the 3-digit score and is not a percentile!
I worry that people that should know better fall in the trap of incorrectly thinking the 2-digit score is a percentile. The reason we have this 2-digit score is to meet requirements by state medical licensing authorities that the passing score be reported as 75. As a result, a 185 to pass in the exam corresponds to a 2-digit score of 75. The 2-digit score is derived such that a 75 always corresponds to the minimum passing score. Nationally, about 8% of students get below a 185 (75). The recommended minimum passing level for each USMLE Step examination is reviewed periodically and may be adjusted at any time.
I use the following guidelines to roughly translate the 3-digit score into a national percentile. A 3-digit score of 200 equals a 2-digit score of 82 which is 20th percentile nationally. Many of the competitive residencies use this as a minimum cutoff score for interview invites.
50th percentile score = 217 for 3-digit score and 90 for 2-digit
75th percentile score = 232 (97)
81st percentile score = 237 (99)
90th percentile score = 246 (99)
95th percentile score = 255 (99)
98th percentile score = 264 (99)
As I mentioned in an earlier blog, the USMLE score is just one piece of the overall picture. The selection committee looks at grades (especially in the 3rd yr clinical core rotations), class rank if provided, medical school attended, commitment to the specialty of anesthesia (a very subjective assessment, but for example we would rather not hear that a person is switching to anesthesia from another specialty because anesthesia is easier or less demanding), research experience and potential for a future academic career, communication skills, compassion & humanism, and an other category that might include: exceptional achievements, their personal statement, or brilliant letters of recommendation.
Studies have found that applicants that do well on the USMLE are more likely to score highly on the in-training written exams. Adequate knowledge is necessary for overall satisfactory performance during anesthesia residency. One of our expectations is that all Stanford graduates will pass both the ABA written and oral exam to achieve board certification. This is particulary important now as most hospitals now require board certification to obtain clinical privileges. We are committed to helping residents achieve their highest professional potential.