Shifting Assessment to Help Residents Become Their Best Selves

Written by Andrew Schwartz

Because of the ways that medical education has evolved along with the expectations of students and trainees, educators in the UCSF Department of Anesthesia and Perioperative Care have spent the last few years adapting the way they deliver training for residents. One of the primary concerns has been revamping the way faculty members assess and deliver feedback to trainees.

“We believe it’s important for us to move from assessment of knowledge toward assessment for learning and growth,” says Associate Chair for Education Kristina Sullivan, MD. While the concept is evolving, “assessment for learning” generally means that faculty members provide deeper and more constructive feedback than the number-based grades that have dominated assessment of medical residents for decades.

Noting the contrast between formative assessments, which happen repeatedly throughout a day or rotation, and summative assessments, which come in the form of tests or overall rotation evaluations, Sullivan says that assessment for learning relies on trainees receiving multiple formative assessments. In theory, this enables the residents to better understand what they are expected to learn and to what standard.                

The Drivers

“Medical education has demanded that we do better in terms of defining what our ‘apprenticeship’ outcomes should be,” says Sullivan. “What are the skills and attitudes we expect our trainees to gain? How can we help get to this defined place? Since everyone works and learns differently, each individual trainee may move along this path differently. Setting up expectations and continuously giving them feedback on where they've been, where they are now, where they are going and, perhaps most importantly, how to get there is an important part of training in the modern era.”                                   

“Being an outstanding educator has become much more complex – and often requires a very different skill set than being an outstanding clinician,” says Associate Residency Program Director for Guidance and Development Gabriel Sarah, MD. “Teaching goes way beyond providing anesthesia and our feedback system has to reflect that.”

Communication skills with increasingly diverse resident cohorts – perhaps especially during assessment – are particularly important. Numerous studies, including those conducted by UCSF educators, have shown gaps and biases in communication and assessment that can negatively affect some groups of learners. 

Sullivan and Sarah want both faculty and residents to understand that feedback should help the residents grow into competent perioperative physicians and to identify where within the specialty they would like to concentrate. Feedback to faculty should focus on helping them improve their skills as educators.

“We want to impress upon people the importance of a growth mindset,” says Sullivan. “Trainees should strive to receive constructive feedback and coaching tips to help them get to the next level. All of us should always be striving to advance and do better.”

An Enhanced Role for EPAs

Sarah, who recently earned his master’s degree in education from UC Berkeley, used his master’s program to understand how to create just such a system. In one paper, he surveyed residents in the department about the first iteration of the new approach to assessment. “We got great feedback and it created a place for us to learn and roll out our program again in a different way,” he says.

Part of the survey gauged residents’ interest in engaging with the Entrustable Professional Activities (EPAs) app. EPAs are tasks or responsibilities that residents perform in the clinical environment and are part of the catalog of skills that are deemed “required to master” to be marked competent for graduation by the American Board of Anesthesiology and the American Council for Graduate Medical Education.

Sarah hopes that EPAs could be an important part of any assessment for learning, because they offer feedback through an easily downloadable app that enables faculty to simply scan a resident’s QR code and rate the resident on a given EPA. The faculty member can also note a number of potential reasons for a given rating. Perhaps the resident had trouble identifying structures on an ultrasound or putting in an IV. This would provide important feedback, without imposing an excessive burden on faculty. Nearly three-quarters of the residents surveyed think the program could be helpful for them becoming a competent anesthesiologist.

Nevertheless, Sarah’s research found there are hurdles to overcome. For one, only slightly more than half of residents have the app on their phone and few were using it, in part because they were skeptical that faculty members would use it. In addition, some expressed concerns that traditional biases would remain and wind up being used against people of color, women, or LGBTQ+ individuals. Sarah says those are legitimate concerns, but believes a carefully constructed rollout can address them and that it’s important to continue to test ways to improve how residents receive feedback.

That’s why this past summer the education team rolled out an EPA Qualtrics survey for the CA1 residents during their onboarding month, which included a short list of tasks for the residents to become more proficient in during their first month of training.

“We need to move away from the numbers game,” says Sarah. “We get so fixed on whether someone met expectations and we’d like this to be more about: even if you meet expectations, how can you continue to improve?”

“If we can create an assessment system that builds toward that, we will have achieved a great deal,” says Sullivan.