Home page
Home This Week's Issue Jobs Education Video Contact Us
  Search:  
Login
  spacer
  Archive
  Earn CPD Points
Services
Resources
 
Med school tests under fire 17-Mar-2008

By Paul Smith
 
 MJA Standardised selection tests for graduate medical school entry are virtually worthless in predicting students’ performance in examinations, one of the biggest studies of student selection in Australia has found.

The study, published in the Medical Journal of Australia, found GAMSAT scores were a “poor predictor” of success and had no significant link with overall exam performance.

The research, which is likely to fuel debate on the methods used to select future doctors, looked at the overall entry scores of 706 students selected by the University of Queensland medical school based on three criteria: their grade point average (GPA) in a previous degree; scores in face-to-face interviews; and their results in the Graduate Australian Medical School Admissions Test (GAMSAT).

 The analysis then linked the entry scores with students’ exam results at years one and four of their medical degree.

 The GAMSAT test — which applicants usually pay to sit — is used by almost all medical schools for admission to their graduate programs. It has also led to thousands of students spending millions on so-called medical entry courses that claim to coach students to pass the test.

 The study found GPA scores were significantly linked with overall exam performance and were a stronger predictor than interview scores – although interview scores proved the best predictor of performance in the final clinical exam.

However, despite being statistically significant, the selection process used by the University of Queensland only explained 21.9% of variation in overall exam scores achieved by students, and this fell to just 17.7% in a student’s fourth-year exams.

 In light of the study, the university has controversially scrapped interviews from its 2008 selection system.

 Professor David Powis, professor of health,professional education, at the University of Newcastle, said the study was flawed because it only measured the value of the selection process on its ability to predict academic performance at medical school.

 “Given the current view that graduating doctors should be more than just academically competent, it is of some concern that academic performance was the sole outcome measure of the study.”

 Insisting interviews still had value in identifying personal characteristics, he added: “Is the interview [at the University of Queensland] being abandoned because it is too expensive in terms of time and resources? Is GAMSAT being retained because it costs the medical school nothing, since applicants pay for it, and its numerical scores are a convenient way to differentiate between applicants?”

 Professor David Wilkinson, head of the University of Queensland medical school, denied the motivation was financial.

“The interviews never cost us anything, other than staff time,” Professor Wilkinson said. “We used volunteers and staff to do the interviews. We have not cut any staff as a result of this change.”

 He said the school had retained GAMSAT because it offered a reliable test that all students took under the same conditions.

 “This makes ranking defensible,” he said. “It is not about saying you are a better student or will make a better doctor, it is just that it offers a test that leads to a ranking.”

 The Australian Council for Education Research, which is responsible for GAMSAT, defended the value of the test.

“Medical selection tests ... are designed to identify people who are likely to have the ability to succeed in demanding medical degrees and in later practice,” the council said. “Their function is to identify, from a pool of academically high achieving applicants, those who demonstrate the highest levels of ability in reasoning, problem solving, critical thinking and written communication in the case of GAMSAT.

 “Neither GAMSAT nor UMAT claims to predict accurately course outcomes or to identify who will make the best doctors at the end of medical training.”

 MJA 2008: 188;323-24, 349-54.

Latest Comments

  • Part of what motivated me to start medicine in the first place was observing the debate around the definition of a "good" doctor and how to identify good raw material and design a manufacturing process.

    A large part of the problem arises because "good" is contextual, depending on what and where the doctor is practising.

    In any production process it makes sense to take care with the raw materials but quality control (assessment) at the end of the process is probably more important. At either end of the process, testing to see if you have met your definition of good will be difficult but it needs to be done.

    So far I have not been contacted by UQ medical school for any sort of post-marketing surveilance. It seems to me that this should be an important part of assessing the performance of the medical school.

    Posted by Charles Evill 24/03/2008 12:06:29 PM

  • It came as a shock to me to realise that medical schools only measured success as graduating from medical school, as Prof David Wilkinson pointed out with his comment that he uses the GAMSAT even though it "is not about saying you ... will make a better doctor".

    Of course, thinking people are interested in postgraduate performance -- that is, making a better doctor. Which is why we use everything, including interviews, to try to pick candidates who we feel will make good doctors -- not just graduate.

    And let's face it: undergraduate doctors find postgraduate doctors a bit challenging. They are a bit too old, articulate and self-assured compared with the young, naive, accepting undergraduates.

    It will take about 20-30 years to validate any education system like medicine. And having a look at evidence like the effectiveness and efficiency of doctors selected by interview compared with mere academic performance will help.

    And it would be helpful if the undergraduate doctors could please try to remember the plural of "anecdote" is not "evidence". We all know doctors who don't know enough anatomy, or whatever. But that is not evidence of systemic failure in selection or teaching. In fact, it's not evidence of anything. Portraying it as such is, however, quite illuminating in another way.

    Posted by Dr Julian Fidge 19/03/2008 10:28:39 AM

  • Professor David Powis is correct. If you were advertising for a PA, would you hire him/her sight unseen? You would not only require a skilled person selected by a test or exam but one who was of pleasant demeanour and had an ability to understand your needs. The same applies to aspiring clinicians.

    On the other hand, not all doctors will wish to be clinicians and may opt for academia and research. Such persons may not require any form of interview other than to ascertain that they have at least a prospect of interacting with colleagues.

    Posted by Anthony Floyd 18/03/2008 12:10:35 PM

  • This may be a crazy idea, but how about using a test that does identify who will make the best doctors at the end of medical training?

    Posted by Leigh McKenzie 17/03/2008 5:54:02 PM

  • I was involved in the early interviews of potential medical students and as far as I am concerned the only value was you could tell the few who should not be allowed to do medicine and a very few who had the attributes to make good doctors.

    But for the vast majority it was useless.

    Posted by Michael Cohn 17/03/2008 5:04:24 PM

Submit your feedback here:

Full name:
Email address:
Emailaddress is used for verification only, we will not publish it.
Your comments:
Security Code:
   

Please tick here if you do not want your comment to be considered for publication in the print edition

Remember my details

(So you don't have to retype your details each time you send feedback.)
Actions
Email to a friend/colleague
Print this article

Related Articles - Medical Education / training
Perfect match (9/3/2010)
Students’ future in doubt (24/2/2010)
Weighing the evidence (23/2/2010)
Med students unprepared for real world (1/2/2010)
Graduate ‘tsunami’ about to hit (27/1/2010)
Subscribe to our daily newsletter




 
 

Australian Doctor