Using Assessments to Energize Your Innovation

Presenter First Name: 
Lawrence
Presenter Last Name: 
Rizzolo, PhD
Year of Presentation: 
2010

 

Lawrence J Rizzolo, Ph.D.
Dept of Surgery
Yale University

William C Rando, Ph.D.
Director, McDougal Graduate Teaching Center
Yale University

Overview:

Most educational innovations fail, especially on the first try. In this module, we will use a case study to demonstrate techniques that will allow you to design and implement a successful education innovation. Specifically, we will discuss using:

  • NEEDS ASSESSMENTS to identify and involve stakeholders
  • FORMATIVE ASSESSMENTS to guide design and implementation throughout the process
  • both FORMATIVE AND SUMMATIVE ASSESSMENTS to convince stakeholders that the innovation is working and worthy of continued support.

Our case study describes our attempt to introduction of a new way to teach anatomy called Clinically-Engaged Anatomy. Details have been published that describe the assessments used during the design phase5 and the implementation and evaluation phases7 of the project.

Case Study

Clinically-Engaged Anatomy: A shorter, more efficient and more effective way to design an anatomy course.
The Goal: To develop an anatomy curriculum for medical students that can be taught effectively in 100 scheduled course hours.

Strategy: Use clinical cases to situate anatomical learning within the context of actual clinical practice, and supplement labs, learning groups and other in-class instruction with interactive web-based resources. We proposed that the clinical cases commonly encountered by students during clerkship training should determine the course content. The course activities would be organized around a patient’s symptoms, physical exam, imaging studies, and medical or surgical resolution. Web resources would be developed to guide in-class activities and out-of-class-time preparation and study.

Assessment in the initial phase: Identify stakeholders and determine their needs.
Since stakeholders will make or break your innovation, it is essential to find out how they think and what they need. Data from Stakeholder Assessments will not only inform the basic design of the innovation but will also form the basis of your ongoing negotiations with your stakeholders.

  • What do they have that I need?
  • What do I have that they need?
  • My vision guides my negotiation strategy
  • But I have to be prepared to re-evaluate and refine my vision

Through interviews, focus groups, and surveys, we identified and assessed the needs of the following stakeholders for our project:

Stakeholder
Need
Measurable Outcome
Dean
Decreased costs and course hours for anatomy
Preserve student performance on USMLE Boards
Clinical Faculty
Determine course objectives
Increased face time with first-year students
Increased residency applications for their subspecialty
Anatomy Faculty
Why change?
Is this worth it?
Students
Comfort Zone
Is this what I need?
Is this what my peers are doing?
Pass Boards
Do well in clerkships
Don't forget yourself!
Develop vision for the future
Other programs adopt the method


Note that the measurable outcomes desired by the stakeholders could not be determined until several years had passed. Short-term assessments were needed to guide the implementation of the innovation and assure the stakeholders of continued progress towards their long-term goals

Assessment in the Design Phase I: creating the innovation.
Using what we learned from our Stakeholder Assessments, we refined our understanding of what we were trying to accomplish. We also learned how to frame our innovation in ways that would make sense to our stakeholders.

We framed the essential goals of our innovation in terms of 4 key learning outcomes for students:

  • Basic Knowledge
  • Structure/Function Relationships
  • Spatial Reasoning
  • Clinical Reasoning

We then identified six pedagogical strategies.

  • Learning Societies -- To promote professionalism and group process.
  • Clinical Cases -- To serve as the framework for presenting anatomical details and concepts.
  • Web Assisted Instruction -- To guide students in and out of the classroom/laboratory.
  • Clinical Faculty -- To focus study on clinically relevant issues.
  • Faculty Development -- To develop the skills for a student-centered approach.
  • Exam Questions -- To test and reinforce the essential goals.

Assessment in The Design Phase II: What, How, and Why are Students Learning.
A tempting, but premature, assessment would ask:

  • Learning societies: Do learning societies promote professionalism and group process?
  • Clinical cases: Do cases place dissections in a clinical context & encourage the acquisition of fundamental anatomical principals?
  • Computer Assisted Instruction: Are knowledge and reasoning refined with usage?
  • Clinical faculty: Do clinical faculty relate dissection to clinical issues?
  • Faculty Development: Are faculty members able to encourage reasoning over memorization; discovery over passive learning?
  • Exam questions: Do exam questions match objectives?

These are good questions and ultimately need to be answered, but asking them now is premature. These "summative" questions have "yes" or "no" answers. They tell what worked or failed, but not why. They yield few insights into how the innovation might be developed. Given that most innovations fail to one extent or another in their first iteration, "formative" assessments are needed to show the way forward.

We needed to find out:

  • What students were experiencing and how were they behaving?
  • What faculty were experiencing and how were they behaving?

For example:

  • Focus groups asked students how they prepared, which resources they used and  how they dealt with contradictory information.
  • Surveys asked which resources were used for different activities and why.
  • Faculty development explored techniques for managing group discussions

By formulating these questions during the design phase, we were prepared to gather the data that helped us refine our tools and approaches. Here are some examples of the questions we asked:

Focus groups
Sample Q 1: How do you prepare for lab?
Sample Q 2: Which resources do you prefer for preparation?
Sample Q 3: How do you handle contradictory information?
Generates data on students’ behaviors, which provides feedback to faculty
Surveys
Sample Q: You have participated in the following: Lecture, Dissection. lab, Radiology lab, Hallway radiology/holograms, Web Instruction.
Which 2 or 3 helped you learn Clinical Reasoning. Describe how.
Asks students to reflect on pedagogical strategies in the context of essential goals
Exam
A neuromuscular exam of the lower limb reveals….. Where is the most likely site of nerve injury?  Justify you conclusion with the data that support it and that rule out alternatives.
Reflect Essential Design Goals precisely

Implementation phase
The cycle of formative assessments, refinement, re-implementation, formative assessment was repeated continuously over six years. Performance on the summative questions (the "premature" questions of Design Phase II) gradually improved. In addition, we followed students into their clinical years to demonstrate an improvement in long-term recall. A detailed discussion of our assessment program and the analysis of the data has been published for medical students7 and physician assistant students8.

Because the long-term goals of the stakeholders listed above have largely been realized, they are very happy and supportive of the anatomy program. Even the initial reluctance of the anatomy faculty has been overcome.

Laws of Course Design and Assessment
Stakeholders will make or break your innovation

  • Learn how they think and what they want.

Even the best planned innovations fail at first.

  • Assessments that only ask if it worked lead to premature death.
  • Assessments that ask “What’s happening?” lead to improvement and success.
  • Formative and summative assessments are most effective when formulated in conjunction with the innovation's design.

Course assessment is an iterative process.

References:
Anatomy Clinic website: http://info.med.yale.edu/surgery/anatomy/

Texts and Reviews:
1. Brown SC , Stevens RA, Troiano PF, Schneider MK. 2002. Exploring complex phenomena: Grounded theory in student affairs research. J Coll Student Dev 43:173-183.
2. Creswell JW. 2008. Educational Research: Planning, Conducting, and Evaluating Quantitative and Qualitative Research. 3rd Ed. Upper Saddle River , N.J. : Pearson/Merrill Prentice Hall. 640 p.
3. Heifetz RA, Linsky M. 2002. Leadership on the line: staying alive through the dangers of leading. Boston , Mass. : Harvard Business School Press. 252p.
4. Kern DE , Thomas PA, Howard DM, Bass, EB. 1998. Curriculum development for medical education: A six-step approach. Baltimore , MD , The Johns Hopkins University Press. 178 p.

Examples of Applications:
5. Rizzolo LJ, Stewart WB, O'Brien M, Haims A, Rando W, Abrahams J, Dunne S, Wang S, Aden M. 2006. Design principles for developing an efficient clinical anatomy course. Med Teach 28:142-151.
6. Rizzolo LJ, Drake RL. 2008. Anatomists debate the value of a teaching credential. Anat. Sci. Educ. 1:60-67.
7. Rizzolo LJ, Rando W, O'Brien M, Haims A, Abrahams J, Stewart WB. 2010. Design, implementation and evaluation of an innovative anatomy course . Anat Sci Ed. 3:109-120.
8. Rizzolo LJ, Rando W, O'Brien M, Garino A, Stewart WB. 2011. Effectiveness of a shortened, clinically engaged anatomy course for physician assistant students. Anat Sci Ed. 4:64-70.
Copyright © 2011 Lawrence J. Rizzolo and William C. Rando

American Association of Anatomists

9650 Rockville Pike Bethesda, Maryland 20814-3998
Tel: 301-634-7910 | Fax: 301-634-7965

 

Copyright © 2014. All rights reserved.           
home ~ contact us