Reliability Evidence and Pharmacist Performance on the Vaccine Assertiveness Assessment (1090-004006) (Research Abstract Professor Rounds: Group 2)
Start time: Thursday, January 28, 2021, 10:00 AM End time: Thursday, January 28, 2021, 11:00 AM Session Type: Research Abstracts (Completed Studies)
Despite available, efficacious vaccinations, the most recent rates of adults being immunized against pneumococcal disease remain below goals set by Health People 2020. As the passive offering of vaccinations to patients has stifled efforts to improve rates, we contend that improved counseling that is personalized and includes a strong, evidence-based recommendation improve immunization efforts. To address this gap, research is needed to equip pharmacists with tools and training that can lead to more effective vaccine promotion and increased immunization rates. Therefore, the research questions for this study are: (1) What is the reliability of the Vaccine Assertiveness Rubric? (2) Do different raters agree with each other about their ratings? Is there any difference in the learner assessment score between stations? (3) What is the overall reliability of learner assessment score across raters, stations, and rubric items?
Twenty-three pharmacists participated in this study and completed two assessment stations on the topic of vaccine assertiveness. Both stations focused on pneumococcal vaccination. Three faculty members rated each pharmacist’s performance using the Vaccine Assertiveness Rubric, which was developed by multiple pharmacy faculty. It has 6 items (focusing on presumptive recommendations made, the pharmacists’ patient care process, and communication) per station and is on a 5-point behavior-anchored scale, where 1 – Not done, 3 – Partially done, and 5 – Well done (2 is between 1 and 3 while 4 is between 3 and 5). We conducted a generalizability study using a fully crossed rater design, which means each learner was evaluated by each rater in each station on all rubric items.
We examined the internal consistency for the Vaccine Assertiveness Rubric using Cronbach’s a (n=23). It was acceptable (.74) for station 1 and was excellent in station 2 (.90). The interrater reliability was examined among three raters by calculating intraclass correlation (ICC) coefficient. The lowest and highest ICC in station 1 were on items 4 (.51) and 6 (.81) and station 2 were on items 1 (.65) and 6 (.86). We averaged ratings to create a final score for each learner. T-tests were conducted to detect any difference between learner performance for station 1 and station 2. The only significant difference between stations was their performance on item 1 (p<.01). Participants performance in station 2 performed higher than in station 1. Finally, we conducted a Generalizability study to examine the overall reliability. The Generalizability coefficient was .42. The two largest variance components were from person x station (26.1%) and person x station x item (20.6%).
We found the Vaccine Assertiveness Rubric was a reliable tool for assessing pharmacists’ competency for making a presumptive pneumonia vaccine recommendation. Future research is warranted to determine if this tool is reliable for assessing pharmacists’ competency to make presumptive recommendations for other vaccinations. Good interrater reliability results were obtained from each station. A future study may focus on how to improve the overall reliability through better rater evaluation training.