WEBVTT 00:00:07.500 --> 00:00:12.338 >> Christopher: It's a pleasure to be with you today 00:00:12.338 --> 00:00:16.676 for the APRN Roundtable, and I'm looking forward to 00:00:16.676 --> 00:00:19.579 having a discussion about transforming assessments 00:00:19.579 --> 00:00:23.416 of clinician knowledge and sharing with you the 00:00:23.416 --> 00:00:26.319 NBCRNA's take and perspective on comparing 00:00:26.319 --> 00:00:28.721 traditional standardized methodologies compared to 00:00:28.721 --> 00:00:30.189 longitudinal assessment modalities. 00:00:30.189 --> 00:00:33.559 So, with that, I will briefly introduce myself. 00:00:33.559 --> 00:00:34.127 My name is Christopher Gill. 00:00:34.127 --> 00:00:37.830 I have the privilege of being in the role of chief 00:00:37.830 --> 00:00:40.199 credentialing officer at the National Board of 00:00:40.199 --> 00:00:41.868 certification and recertification of nurse 00:00:41.868 --> 00:00:42.368 anesthetist's. 00:00:42.368 --> 00:00:47.407 I have been in my position for the past three plus 00:00:47.407 --> 00:00:51.210 years, and it has been an awesome, tremendous 00:00:51.210 --> 00:00:54.047 experience to delve into certification and 00:00:54.047 --> 00:00:56.916 regulatory affairs at a deeper level. 00:00:56.916 --> 00:01:00.286 Today I am going to focus on a couple of objectives 00:01:00.286 --> 00:01:01.487 to discuss with you. 00:01:01.487 --> 00:01:04.991 I want to really compare and contrast knowledge 00:01:04.991 --> 00:01:07.593 versus competence, with the differences are, 00:01:07.593 --> 00:01:10.663 similarities, historical methods that the NBCRNA has 00:01:10.663 --> 00:01:13.266 used to assess CRNA knowledge specifically. 00:01:13.266 --> 00:01:17.537 I want to talk to you and have a discussion around 00:01:17.537 --> 00:01:19.872 our randomized controlled trial assessing and 00:01:19.872 --> 00:01:22.575 evaluating two methods of knowledge assessment for 00:01:22.575 --> 00:01:26.479 CRNAs, and to share with you and hope you understand 00:01:26.479 --> 00:01:30.350 how the study findings link to the development of a 00:01:30.350 --> 00:01:31.918 new continued certification program. 00:01:31.918 --> 00:01:36.622 Just some baseline concepts to start with, knowledge 00:01:36.622 --> 00:01:37.824 versus competence. 00:01:37.824 --> 00:01:40.393 Clinical knowledge is largely theoretical 00:01:40.393 --> 00:01:40.893 information. 00:01:40.893 --> 00:01:45.131 We talk about diseases, treatments, procedures, and 00:01:45.131 --> 00:01:49.369 guidelines, to kind of draw a distinction. 00:01:49.369 --> 00:01:53.773 Competence is more broad but inclusive of practical 00:01:53.773 --> 00:01:57.610 skills, judgment, and behaviors of our profession, 00:01:57.610 --> 00:02:02.048 so much more related to the practical day-to-day. 00:02:02.048 --> 00:02:05.351 And this is not unfamiliar material for most of you, 00:02:05.351 --> 00:02:08.621 but really the knowledge is what we acquire in our 00:02:08.621 --> 00:02:11.591 initial training and then have to maintain as we 00:02:11.591 --> 00:02:12.091 practice. 00:02:12.091 --> 00:02:14.394 But it is this clinical competence that really helps 00:02:14.394 --> 00:02:17.163 guide us once we are out in the working world. 00:02:17.163 --> 00:02:21.701 I think the key takeaways here I want to drive home, 00:02:21.701 --> 00:02:24.203 both dimensions are crucial for knowledge, 00:02:24.203 --> 00:02:27.106 evaluation and come brands of measurement of 00:02:27.106 --> 00:02:29.976 knowledge assessed is what you know, competence 00:02:29.976 --> 00:02:34.113 assesses how you apply what you know. And it is 00:02:34.113 --> 00:02:37.016 really balancing measurement of these that enhances 00:02:37.016 --> 00:02:39.919 health care quality and patient care outcomes. 00:02:39.919 --> 00:02:43.489 When we talk about entry-to-practice, it is this 00:02:43.489 --> 00:02:47.060 knowledge mastery required to enter and begin and 00:02:47.060 --> 00:02:51.097 become, as I like to call it, baseline confident. 00:02:51.097 --> 00:02:54.033 We are lifelong learners by nature. 00:02:54.033 --> 00:02:56.803 For registered nurses, advanced practice registered 00:02:56.803 --> 00:03:00.006 nurses, physicians, PAs, all individuals in the 00:03:00.006 --> 00:03:03.676 health care milieu, we are really responsible for 00:03:03.710 --> 00:03:06.913 understanding knowledge to begin practice, and this 00:03:06.913 --> 00:03:10.116 is measured by standardized examinations. These 00:03:10.116 --> 00:03:13.820 standardized examinations are what we call kind of 00:03:13.820 --> 00:03:18.391 the public litmus test, and it really tells us what 00:03:18.391 --> 00:03:22.528 an individual must have in order to begin safely 00:03:22.528 --> 00:03:26.699 practicing. I like to use the quote that my 00:03:26.699 --> 00:03:30.803 colleague and my superior at the NBCRNA says, John 00:03:30.803 --> 00:03:31.304 Preston. 00:03:31.304 --> 00:03:34.574 "No one likes going to the DMV, but everybody 00:03:34.607 --> 00:03:37.176 appreciates the freedom and ability that driving 00:03:37.176 --> 00:03:37.677 provides." 00:03:37.677 --> 00:03:41.581 So it is really this entry-to-practice and 00:03:41.581 --> 00:03:44.450 regulatory place to function in. 00:03:44.450 --> 00:03:48.354 Where you want to center our discussion today, 00:03:48.354 --> 00:03:49.322 continued practice. 00:03:49.322 --> 00:03:52.225 Is there a best practice for continued practice? I 00:03:52.225 --> 00:03:55.161 don't know that we have it completely nailed down. 00:03:55.161 --> 00:03:59.232 Do the same mechanisms capture the initial 00:03:59.265 --> 00:04:02.168 mechanisms, capture what continued practice 00:04:02.168 --> 00:04:05.672 requires? Some do, some don't. 00:04:05.672 --> 00:04:08.841 And what happens when current programs fall short of 00:04:08.841 --> 00:04:09.542 stated goals? 00:04:09.542 --> 00:04:12.311 How do we reevaluate and come in many ways, 00:04:12.311 --> 00:04:15.048 reimagine a program when we have fallen short of 00:04:15.048 --> 00:04:15.682 stated goals? 00:04:15.682 --> 00:04:18.418 More questions than answers here, but I'm going to 00:04:18.418 --> 00:04:20.253 give you some answers very shortly. 00:04:20.286 --> 00:04:23.489 There is a range of current mechanisms that are used 00:04:23.489 --> 00:04:25.091 today to understand continued practice. 00:04:25.091 --> 00:04:30.763 We have continuing education units, CEUs we are 00:04:30.763 --> 00:04:32.865 asked to do. 00:04:32.899 --> 00:04:35.201 Semisynchronous and some are a synchronous in 00:04:35.201 --> 00:04:35.702 nature. 00:04:35.702 --> 00:04:40.006 We have standardized exams we can apply to 00:04:40.039 --> 00:04:43.943 individuals, both in the initial certification and 00:04:43.943 --> 00:04:45.078 continued certification. 00:04:45.078 --> 00:04:47.680 The health care simulation has shown promise for 00:04:47.680 --> 00:04:50.283 certain avenues but might not be broadly applicable 00:04:50.283 --> 00:04:51.918 for a number of reasons. 00:04:51.918 --> 00:04:55.121 Cost to travel there, cost to run with the 00:04:55.121 --> 00:04:58.324 personnel, as well as the time away from clinical 00:04:58.324 --> 00:05:01.160 practice that an individual must submit to you. 00:05:01.160 --> 00:05:05.098 In the last thing I just threw in there is a 00:05:05.098 --> 00:05:07.233 portfolio submission. Some regulatory and licensing 00:05:07.233 --> 00:05:11.170 boards have chosen to go the realm of looking at a 00:05:11.170 --> 00:05:14.007 portfolio of what our current practitioner is doing, 00:05:14.007 --> 00:05:16.509 measuring that in terms of the benchmark. 00:05:16.509 --> 00:05:20.580 So there's a lot of tools out there, but is there a 00:05:20.580 --> 00:05:21.247 best one? 00:05:21.247 --> 00:05:21.748 I don't know. 00:05:21.748 --> 00:05:23.950 Ideal characteristics are certainly helpful to guide 00:05:23.950 --> 00:05:28.755 us to get into a best one, so I think when we are 00:05:28.755 --> 00:05:30.990 talking about clinical professionals and individuals 00:05:30.990 --> 00:05:34.694 that are out in practice, we have to have things 00:05:34.694 --> 00:05:37.296 that are practical for practicing clinicians. We 00:05:37.296 --> 00:05:41.000 have to look at what strengths they have and what 00:05:41.000 --> 00:05:42.101 liabilities they have. 00:05:42.101 --> 00:05:47.407 The other thing that I think is highly advantageous 00:05:47.407 --> 00:05:50.343 for practicing professionals is suggesting 00:05:50.343 --> 00:05:52.111 personalized educational opportunities. 00:05:52.111 --> 00:05:54.180 When we have identified liabilities, we generally 00:05:54.180 --> 00:05:57.450 want to fill them as clinicians and we want to know 00:05:57.450 --> 00:06:01.020 what we should do and what will get us to a place 00:06:01.020 --> 00:06:03.723 that we need to be and we should be. 00:06:03.723 --> 00:06:08.761 So I want to talk to you briefly before I go into 00:06:08.761 --> 00:06:11.297 our randomized controlled trial about our 00:06:11.297 --> 00:06:13.833 certification journey. No certification journey is 00:06:13.833 --> 00:06:17.637 complete without talking about the past, and I won't 00:06:17.637 --> 00:06:22.275 spend too much time on this, but we had a legacy 00:06:22.275 --> 00:06:25.211 recertification program that function for over 40 00:06:25.211 --> 00:06:25.712 years. 00:06:25.712 --> 00:06:30.783 It was a good program by a lot of measures, but 00:06:30.783 --> 00:06:35.021 times have changed, the public and the patients we 00:06:35.021 --> 00:06:39.225 serve have really asked us for more than verifying 00:06:39.225 --> 00:06:42.061 contact information, verifying practice hours or 00:06:42.061 --> 00:06:45.331 certain practice arrangements, and just saying "is 00:06:45.331 --> 00:06:48.634 your licensure good and unencumbered?" they expect 00:06:48.634 --> 00:06:51.904 more of us than doing passive CEUs. 00:06:51.904 --> 00:06:56.376 So our Legacy program focused on making sure we had 00:06:56.376 --> 00:06:59.946 the adequate contact information to reach out to 00:06:59.946 --> 00:07:03.516 people, that they were practicing, and they had 00:07:03.516 --> 00:07:05.752 licensure requirements that were met. 00:07:05.752 --> 00:07:10.156 We moved from that to present day and a lot of times 00:07:10.156 --> 00:07:13.459 we had questions from our certificate body in terms 00:07:13.459 --> 00:07:17.497 of why we would move from a legacy program to a 00:07:17.497 --> 00:07:18.965 contemporary continued professional certification 00:07:18.965 --> 00:07:19.465 program. 00:07:19.465 --> 00:07:22.635 This was largely predicated on changing 00:07:22.635 --> 00:07:23.736 accreditation requirements. 00:07:23.736 --> 00:07:30.643 The NBCRNA, like many of you in the certification 00:07:30.643 --> 00:07:33.713 space, has two accreditors. 00:07:33.713 --> 00:07:36.749 accreditors are asking more of certifiers in terms 00:07:36.749 --> 00:07:40.186 of what they are doing to measure knowledge after 00:07:40.219 --> 00:07:42.488 that initial training point in time. 00:07:42.488 --> 00:07:46.192 Changing needs, certainly we know that knowledge and 00:07:46.192 --> 00:07:50.363 skills are developed over a time and based on 00:07:50.363 --> 00:07:51.264 practice arrangements. 00:07:51.264 --> 00:07:53.900 They may vary based on individuals. 00:07:53.900 --> 00:07:57.670 We also wanted to make sure we recognize the 00:07:57.670 --> 00:08:01.007 activities that CRNAs were doing, so these were 00:08:01.007 --> 00:08:03.910 really strong reasons to change our program. 00:08:03.910 --> 00:08:05.311 The last is technology. 00:08:05.311 --> 00:08:07.280 Technology has changed. 00:08:07.280 --> 00:08:08.614 I talked about simulation. 00:08:08.614 --> 00:08:13.119 Virtual reality, skills check-offs now exist. 00:08:13.119 --> 00:08:16.989 I'm going to talk to you in large part about 00:08:16.989 --> 00:08:19.325 longitudinal assessment and answering questions over 00:08:19.325 --> 00:08:20.493 a time period. 00:08:20.493 --> 00:08:22.628 But there's a lot of promising opportunities on the 00:08:22.628 --> 00:08:24.797 horizon, so he felt those were compelling enough to 00:08:24.797 --> 00:08:25.531 change our program. 00:08:25.531 --> 00:08:32.238 What we did is we split off our class of CEUs to two 00:08:32.238 --> 00:08:33.272 different versions. 00:08:33.272 --> 00:08:37.510 Class a was kind of our similar to the previous 40 00:08:37.510 --> 00:08:42.115 CEUs, but we upped it from 40 to 60 in a 4-year 00:08:42.115 --> 00:08:42.615 cycle. 00:08:42.615 --> 00:08:46.819 For CRNAs this has to be prior approved by the 00:08:46.819 --> 00:08:50.289 educational accreditation, and it had to have an 00:08:50.289 --> 00:08:52.458 assessment, much like previous CEUs. 00:08:52.458 --> 00:08:57.430 Individuals can always have more, but they can't 00:08:57.430 --> 00:08:59.932 have less than 60. 00:08:59.932 --> 00:09:02.035 Class B was a category we added. 00:09:02.035 --> 00:09:05.171 This was looking at professional development 00:09:05.171 --> 00:09:07.774 activities, leadership, scholarly work, exercising 00:09:07.774 --> 00:09:10.910 new techniques in a clinical realm. 00:09:10.910 --> 00:09:14.313 So we don't have an assessment for this. 00:09:14.313 --> 00:09:16.082 It is not prior approved. 00:09:16.082 --> 00:09:18.918 And they can have up to 40 CEUs in this category. 00:09:18.918 --> 00:09:22.588 But we broadly said it had to support patient 00:09:22.588 --> 00:09:25.458 safety, enhance the knowledge of nurse anesthesia 00:09:25.458 --> 00:09:28.728 practitioners, and relate to a broader health care 00:09:28.728 --> 00:09:29.228 environment. 00:09:29.228 --> 00:09:32.398 Probably the most contentious part of our current 00:09:32.398 --> 00:09:35.268 program is that we added an assessment. 00:09:35.268 --> 00:09:39.439 So CRNAs that have been in the certificate pool 00:09:39.439 --> 00:09:44.110 prior to 2016 really did not have to take a 00:09:44.110 --> 00:09:47.380 formalized assessment outside the initial entry into 00:09:47.380 --> 00:09:47.880 practice. 00:09:47.880 --> 00:09:51.150 So we introduced what is called the continued 00:09:51.150 --> 00:09:53.653 professional certification assessment, CPCA, kind of 00:09:53.653 --> 00:09:54.487 a mouthful. 00:09:54.487 --> 00:09:59.158 It's not like our National certified exam, or entry 00:09:59.158 --> 00:10:00.727 practice gatekeeping assessment. 00:10:00.727 --> 00:10:04.430 The CPCA focus on clinical knowledge required of 00:10:04.430 --> 00:10:08.101 experienced CRNAs, knowledge that was common for all 00:10:08.101 --> 00:10:10.403 CRNAs regardless of practice focus. 00:10:10.403 --> 00:10:14.707 I used to say, CRNAs don't necessarily always 00:10:14.707 --> 00:10:17.377 provide cardiovascular anesthesia for open-heart 00:10:17.377 --> 00:10:22.215 cases or valve replacements, but every patient has a 00:10:22.215 --> 00:10:26.519 heart, so an interest in cardiac physiology was 00:10:26.519 --> 00:10:27.053 essential. 00:10:27.053 --> 00:10:31.524 This outline is available on our website and is 00:10:31.524 --> 00:10:34.027 developed using professional practice analysis 00:10:34.027 --> 00:10:34.527 methodology. 00:10:34.527 --> 00:10:38.531 The CPCA was not a mechanism whereby individuals 00:10:38.531 --> 00:10:40.500 would lose their certification. 00:10:40.533 --> 00:10:44.404 If they fell below and establish performance 00:10:44.404 --> 00:10:48.307 benchmarking, we requested them to take additional 00:10:48.307 --> 00:10:51.644 focused CE, one for each category. 00:10:51.678 --> 00:10:56.215 If they didn't need to, they were good to go through 00:10:56.215 --> 00:10:57.850 the next certification cycle. 00:10:57.850 --> 00:11:02.288 So a majority of our certificate pool will complete 00:11:02.288 --> 00:11:07.260 this and is completing this now, and will have it 00:11:07.260 --> 00:11:12.198 done by 7/31/24 or 7/31/25. This leads us to the 00:11:12.198 --> 00:11:16.669 question, where are we making a critical error from 00:11:16.669 --> 00:11:17.637 the start? 00:11:17.637 --> 00:11:22.308 Could it be that we had the assumption that we 00:11:22.308 --> 00:11:25.545 launched CRNAs into practice of capabilities that 00:11:25.545 --> 00:11:29.282 they would remain at or above performance levels 00:11:29.282 --> 00:11:30.683 until they retire? 00:11:30.683 --> 00:11:32.352 This is really a strong question. 00:11:32.352 --> 00:11:35.755 We know from other realms, just personal, we 00:11:35.755 --> 00:11:39.992 certainly can have mastery at one point in time, but 00:11:39.992 --> 00:11:43.429 without active work, may not maintain mastery in 00:11:43.429 --> 00:11:47.667 certain topical areas if we are not going to them 00:11:47.667 --> 00:11:48.968 over and over. 00:11:48.968 --> 00:11:54.173 As a certifier who led both staff and Board of 00:11:54.173 --> 00:11:58.845 Directors, we have taken the view of competency for 00:11:58.845 --> 00:11:59.912 lifelong learning. 00:11:59.912 --> 00:12:02.882 So there's multiple tools and methodologies whereby 00:12:02.882 --> 00:12:06.719 you can assess this and measure this and evaluate 00:12:06.719 --> 00:12:10.556 it, but it really centers on knowing one's practice, 00:12:10.556 --> 00:12:13.960 being able to scan the environment for changes 00:12:13.960 --> 00:12:16.963 occurring or that have occurred, managing learning 00:12:16.963 --> 00:12:20.366 in practice, so this is kind of self-directed 00:12:20.366 --> 00:12:24.637 activities. You have to be able to raise and answer 00:12:24.637 --> 00:12:28.474 questions, and you have to assess and enhance your 00:12:28.474 --> 00:12:31.044 practice over time after you graduated. 00:12:31.044 --> 00:12:34.914 So we weren't led by problems in the space, we 00:12:34.914 --> 00:12:38.751 really want to lead into the future with a positive 00:12:38.751 --> 00:12:39.919 perspective and methodology. 00:12:39.919 --> 00:12:43.389 If you look on the left, you can see certification, 00:12:43.389 --> 00:12:44.090 acceptable performance. 00:12:44.090 --> 00:12:48.294 We have met the benchmark, past our initial 00:12:48.294 --> 00:12:49.328 certifying exam. 00:12:49.328 --> 00:12:53.166 But is really once and then good for life? Many 00:12:53.166 --> 00:12:56.669 people will stay, many will go above, but there are 00:12:56.669 --> 00:13:00.139 some individuals who will fall below and we want to 00:13:00.139 --> 00:13:02.575 identify them and gently bring them up. 00:13:02.575 --> 00:13:06.279 So if you move to the right hand graph, we can see 00:13:06.279 --> 00:13:09.382 the competence that we enter with or maybe even more 00:13:09.382 --> 00:13:09.982 specifically novice. 00:13:09.982 --> 00:13:13.986 You pass, you have set baseline knowledge, you enter 00:13:13.986 --> 00:13:17.990 as a novice, you move to advanced, you achieve 00:13:17.990 --> 00:13:21.527 competency hopefully in all categories but in core 00:13:21.561 --> 00:13:26.432 areas for sure, and what we really want to do is 00:13:26.432 --> 00:13:29.969 move them from competent to proficient and expert 00:13:29.969 --> 00:13:33.106 and maintain them at that high level. 00:13:33.106 --> 00:13:37.243 What that requires outside of the initial training 00:13:37.243 --> 00:13:40.346 is deliberate professional practice and evaluation. 00:13:40.346 --> 00:13:46.552 Setting the stage for the future, we wanted to make 00:13:46.552 --> 00:13:47.787 some assumptions. 00:13:47.787 --> 00:13:49.355 CRNA education has not specifically focused on 00:13:49.355 --> 00:13:51.391 skills required to be a lifelong learner but has 00:13:51.391 --> 00:13:52.291 focused on expert clinicians. 00:13:52.291 --> 00:13:56.562 People buy in large gravitate toward learning they 00:13:56.562 --> 00:13:57.630 don't need. 00:13:57.630 --> 00:14:01.300 Why? When we are interested in a topic we tend to 00:14:01.334 --> 00:14:04.003 choose consuming more of that information, and we 00:14:04.003 --> 00:14:07.006 tend to shy away from the things that don't 00:14:07.006 --> 00:14:08.007 necessarily excite us. 00:14:08.007 --> 00:14:11.911 CRNAs must enter practice with a set of learning 00:14:11.911 --> 00:14:15.848 competencies like all APRNs that will enable them to 00:14:15.848 --> 00:14:19.318 critically assess and revise their practice to make 00:14:19.318 --> 00:14:22.355 sense of complex situations and learn from 00:14:22.355 --> 00:14:25.858 experience. So it is really this interconnected web 00:14:25.858 --> 00:14:28.461 of knowledge, complex situations, and clinical 00:14:28.461 --> 00:14:32.365 judgment, and growing over time, that we want to 00:14:32.365 --> 00:14:34.100 build our programs around. 00:14:34.100 --> 00:14:38.271 With that in mind, we set out for the future, so one 00:14:38.271 --> 00:14:41.741 of the things we think is highly promising and that 00:14:41.741 --> 00:14:45.578 I'm going to present to you in brief, kind of our 00:14:45.578 --> 00:14:48.014 study findings on a randomized controlled trial 00:14:48.014 --> 00:14:50.783 looking at longitudinal assessment, if you have to 00:14:50.783 --> 00:14:54.587 define it and think about it in your mind, it is 00:14:54.587 --> 00:14:57.023 simply a cross-sectional methodology that seeks to 00:14:57.023 --> 00:14:59.826 apply principles of adult learning in measurement of 00:14:59.826 --> 00:15:00.326 knowledge. 00:15:00.326 --> 00:15:03.696 So I'm very distinct in saying, not competence, but 00:15:03.696 --> 00:15:04.197 knowledge. 00:15:04.197 --> 00:15:06.199 Because we believe we can measure knowledge. 00:15:06.199 --> 00:15:09.769 We cannot necessarily 100% measure competence. 00:15:09.769 --> 00:15:12.538 But having the requisite knowledge will certainly 00:15:12.538 --> 00:15:15.274 set individuals up for competent practice, and 00:15:15.274 --> 00:15:18.411 there's other stakeholders in this process that are 00:15:18.411 --> 00:15:21.914 on this call, probably, that have a role in 00:15:21.914 --> 00:15:22.715 assessing competence. 00:15:22.715 --> 00:15:27.053 Even more basically it can be thought of as a tool 00:15:27.053 --> 00:15:29.022 that evaluates individual knowledge dynamically. 00:15:29.022 --> 00:15:31.824 We often see longitudinal assessment applied to 00:15:31.824 --> 00:15:35.028 practicing clinicians as a set number of questions 00:15:35.028 --> 00:15:38.631 per quarter that they are introduced, and it's these 00:15:38.631 --> 00:15:41.034 shorter periodic assessments with immediate feedback 00:15:41.034 --> 00:15:43.836 that shows significant promise in helping practicing 00:15:43.836 --> 00:15:45.838 professionals increase their knowledge and 00:15:45.838 --> 00:15:49.042 understanding on topics and also reinforce what they 00:15:49.042 --> 00:15:53.046 have today. So when I say adult principles of 00:15:53.046 --> 00:15:55.848 learning and longitudinal assessment, these are the 00:15:55.848 --> 00:16:00.653 ones I'm kind of driving home or want to give you a 00:16:00.653 --> 00:16:03.856 sense of. Learning linked to testing, the more 00:16:03.856 --> 00:16:06.659 frequently we see repetitive assessments, the more 00:16:06.659 --> 00:16:10.263 likely that information is to become durable in our 00:16:10.263 --> 00:16:13.066 mind, and the connections will become stronger. 00:16:13.066 --> 00:16:16.235 That way, when we are asked to recall in the 00:16:16.235 --> 00:16:18.738 clinical realm or professional realm, we are more 00:16:18.771 --> 00:16:21.607 likely to do it expeditiously then we were before. 00:16:21.607 --> 00:16:24.677 Space learning, exposure to materials interspersed 00:16:24.677 --> 00:16:26.212 with other activities. 00:16:26.212 --> 00:16:28.214 Again, strengthening neural connections. 00:16:28.214 --> 00:16:32.051 When you take a deck of cards, you are in grade 00:16:32.051 --> 00:16:34.821 school and trying to learn different subjects, if 00:16:34.821 --> 00:16:37.957 you space to learning and move between math and 00:16:37.957 --> 00:16:41.427 science and English, then you are going to have a 00:16:41.427 --> 00:16:44.564 better exposure and a stronger connection to all of 00:16:44.564 --> 00:16:47.367 those topics and if you just studied them 00:16:47.367 --> 00:16:47.867 separately. 00:16:47.867 --> 00:16:49.836 Subject matter, simultaneous presentation of 00:16:49.836 --> 00:16:53.673 different topics, if you took those cards and you'd 00:16:53.673 --> 00:16:57.944 took some and mix them all up, that would even 00:16:57.944 --> 00:17:00.947 further enhance your ability to recall information. 00:17:00.947 --> 00:17:03.916 Providing instant and immediate feedback is highly 00:17:03.916 --> 00:17:06.052 advantageous, learning through repetition, which 00:17:06.052 --> 00:17:09.022 I've talked about, offering convenience and learning 00:17:09.022 --> 00:17:12.859 platforms is really key with adults, and there has 00:17:12.859 --> 00:17:16.295 to be some self-direction and determinism to a 00:17:16.295 --> 00:17:16.796 point. 00:17:16.796 --> 00:17:19.332 We set out to do a research study. 00:17:19.332 --> 00:17:21.200 We had some study aims. 00:17:21.200 --> 00:17:25.204 He wanted to compare pass rates and mean scaled 00:17:25.204 --> 00:17:28.775 scores on the assessment among CRNAs in a 00:17:28.775 --> 00:17:31.911 traditional assessment, CPCA group, as a parent, 00:17:31.911 --> 00:17:34.580 compared to a longitudinal assessment group. 00:17:34.580 --> 00:17:38.317 We saw differences in the perceptions and attitudes 00:17:38.351 --> 00:17:42.088 between the two methodologies using scales, and we 00:17:42.088 --> 00:17:45.391 wanted to understand satisfaction and whether the 00:17:45.391 --> 00:17:47.260 tools promoted lifelong learning. 00:17:47.260 --> 00:17:51.030 The third and kind of final studying was that we 00:17:51.030 --> 00:17:54.033 wanted to be able to describe the longitudinal 00:17:54.033 --> 00:17:55.902 assessment participant's experience in engagement 00:17:55.902 --> 00:17:58.905 using data triangulation focus groups, which we did. 00:17:58.905 --> 00:18:02.408 So requirements between the two groups, the 00:18:02.408 --> 00:18:05.378 traditional assessment group versus the longitudinal 00:18:05.378 --> 00:18:06.379 assessment group. 00:18:06.379 --> 00:18:09.215 The traditional assessment group, we really just ask 00:18:09.215 --> 00:18:12.418 them to do what individuals are already doing today. 00:18:12.418 --> 00:18:17.056 We had to solicit a group that hadn't completed the 00:18:17.056 --> 00:18:21.227 activity to date, but it was a single 3-hour, 00:18:21.227 --> 00:18:24.030 150-question assessment that was either in-person 00:18:24.030 --> 00:18:27.266 through our Pearson Vue Center, our contracted 00:18:27.266 --> 00:18:31.004 vendor, or online, again through Pearson Vue, and 00:18:31.004 --> 00:18:35.174 that had to be completed before the study ended. 00:18:35.174 --> 00:18:37.777 The longitudinal assessment cohort really had to 00:18:37.777 --> 00:18:41.114 answer when hundred 35 questions, a total of 135 00:18:41.114 --> 00:18:42.982 questions come over four quarters. 00:18:42.982 --> 00:18:45.651 He could see them listed there. 00:18:45.651 --> 00:18:48.554 Some similarities between the groups, all questions 00:18:48.554 --> 00:18:51.824 were developed using the same item bank. The 00:18:51.858 --> 00:18:55.128 question distribution was similar and used the same 00:18:55.128 --> 00:18:58.431 content outline, and even though we use different 00:18:58.431 --> 00:19:02.568 sets of questions with the two groups, we did pulled 00:19:02.568 --> 00:19:07.106 them from one item bank, and it was a created using 00:19:07.106 --> 00:19:08.341 a common scale. 00:19:08.341 --> 00:19:12.145 Outcome measures, in brief, we sought to understand 00:19:12.145 --> 00:19:15.448 what the performance differences were between two 00:19:15.448 --> 00:19:19.252 groups. The perceptions and attitudes dig into the 00:19:19.252 --> 00:19:22.088 satisfaction and promotion of lifelong learning 00:19:22.088 --> 00:19:24.957 through longitudinal assessment focus groups, and 00:19:24.957 --> 00:19:28.261 then we wanted to understand what engage 00:19:28.261 --> 00:19:32.532 participants the most and what they felt was most 00:19:32.532 --> 00:19:34.901 usable as a practicing clinician. 00:19:34.901 --> 00:19:38.004 When we talk about participant recruitment, we were 00:19:38.004 --> 00:19:41.874 fortunate enough that he will see the diagram on the 00:19:41.874 --> 00:19:46.112 screen in front of you -- we put out a call for 00:19:46.112 --> 00:19:49.582 volunteers in 2022, we conducted the survey in a 00:19:49.582 --> 00:19:53.419 large part of 2023. We sought to elicit a thousand 00:19:53.419 --> 00:19:56.522 volunteers, and during the call for volunteers we 00:19:56.522 --> 00:19:58.458 had over 10,000 interested individuals. 00:19:58.691 --> 00:20:02.595 So that was highly advantageous for us and we were 00:20:02.595 --> 00:20:04.530 very proud of that accomplishment. 00:20:04.530 --> 00:20:08.501 We randomized the individuals between two groups, 00:20:08.501 --> 00:20:14.173 matched them one to one look at gender, age, and 00:20:14.173 --> 00:20:18.144 years of practice as representative of the 00:20:18.144 --> 00:20:21.547 practicing continuing certification cohorts we had. 00:20:21.547 --> 00:20:27.754 So we randomized 500 CRNAs into the CPCLA cohort and 00:20:27.754 --> 00:20:32.125 the traditional assessment, the CPCA group, where 00:20:32.125 --> 00:20:34.594 they took the assessment. 00:20:34.594 --> 00:20:38.931 Our power analysis said that we needed at least 320 00:20:38.931 --> 00:20:43.269 per group, and we exceeded that and we are very 00:20:43.269 --> 00:20:43.770 fortunate. 00:20:43.770 --> 00:20:47.206 These demographics are highly representative of 00:20:47.206 --> 00:20:53.079 practicing CRNAs, but we see in large part that we 00:20:53.079 --> 00:20:57.784 had similar distributions between CPCLA and CPCA in 00:20:57.784 --> 00:21:00.119 terms of gender distribution. 00:21:00.119 --> 00:21:05.825 Again, this is consistent with what we know about 00:21:05.825 --> 00:21:07.727 the CRNA population. 00:21:07.760 --> 00:21:11.431 The age range was similar between the groups, a 00:21:11.431 --> 00:21:14.734 slight difference, but largely similar in terms of 00:21:14.734 --> 00:21:17.203 average age and years of practice. 00:21:17.203 --> 00:21:20.573 And then, in terms of geographic distribution, these 00:21:20.573 --> 00:21:24.777 are known in states where we have the highest kind 00:21:24.777 --> 00:21:26.045 of CRNA population. 00:21:26.045 --> 00:21:31.451 But the CPCLA group had Pennsylvania, Texas, 00:21:31.451 --> 00:21:34.520 Florida, Ohio, and Tennessee. 00:21:34.520 --> 00:21:37.990 And within the traditional assessment, Pennsylvania, 00:21:37.990 --> 00:21:42.595 to Texas, Ohio, Florida, and Illinois, all states 00:21:42.595 --> 00:21:44.897 with the highest populations. 00:21:44.931 --> 00:21:46.566 The longitudinal assessment specifications where 00:21:46.566 --> 00:21:49.836 that individuals in that cohort had a duration of 12 00:21:49.836 --> 00:21:50.336 months. 00:21:50.336 --> 00:21:53.806 You can see the start and end dates they are. They 00:21:53.806 --> 00:21:57.110 had to answer a set number of questions, the content 00:21:57.110 --> 00:21:59.445 was balanced according to our professional practice 00:21:59.445 --> 00:22:02.115 analysis, the blueprint was the same with the 00:22:02.115 --> 00:22:02.749 traditional assessment. 00:22:02.749 --> 00:22:07.153 They had to answer between 30-35 items per quarter. 00:22:07.153 --> 00:22:10.790 They were given immediate feedback as well as 00:22:10.790 --> 00:22:14.427 rationale for why the answer was correct or 00:22:14.427 --> 00:22:17.630 incorrect, a reference, and then confidence ratings 00:22:17.630 --> 00:22:19.899 were solicited from the individuals. 00:22:19.899 --> 00:22:24.270 They had one minute per item, they were not allowed 00:22:24.270 --> 00:22:28.641 to skip any questions, but they could see -- that is 00:22:28.641 --> 00:22:32.578 where the 35 questions comes in -- in quarters 2-4, 00:22:32.578 --> 00:22:33.880 five repeat questions. 00:22:34.280 --> 00:22:37.650 They could also look at the question history as well 00:22:37.650 --> 00:22:40.987 as some scoring and normative data. So what were our 00:22:40.987 --> 00:22:42.321 results of the study? 00:22:42.321 --> 00:22:45.692 While getting into the first research question is 00:22:45.692 --> 00:22:48.661 the performance, whether it's comparable, the answer 00:22:48.661 --> 00:22:49.495 is yes. 00:22:49.529 --> 00:22:54.033 Between the groups we saw that, within the two 00:22:54.033 --> 00:22:59.539 groups come in the L.A. group we had 85%, almost 86% 00:22:59.539 --> 00:23:03.042 of individuals meeting the performance standard, and 00:23:03.042 --> 00:23:06.546 in the CPCA, 94% of individuals. Interestingly 00:23:06.546 --> 00:23:10.550 enough, when we added in if questions were 00:23:10.550 --> 00:23:14.554 readministered if they got the question correct on 00:23:14.554 --> 00:23:19.058 the readministration, we counted it correct as if it 00:23:19.058 --> 00:23:24.564 was correct from the start. We moved up from a very 00:23:24.564 --> 00:23:27.600 similar percentage of performance meeting the 00:23:27.600 --> 00:23:30.603 standard, so 91 compared to 94%. 00:23:30.603 --> 00:23:35.308 So that was really reassuring in terms of what our 00:23:35.341 --> 00:23:36.743 research question was. 00:23:36.743 --> 00:23:41.047 When you look at the initial response scored versus 00:23:41.047 --> 00:23:44.851 most recent, when scoring incorrectly on the initial 00:23:44.851 --> 00:23:49.155 attempt, the mean scaled score for the L.A. group, 00:23:49.155 --> 00:23:53.926 the mean score was 649, and that was significant a 00:23:53.926 --> 00:23:58.231 higher than the mean score of those that we 00:23:58.231 --> 00:24:01.100 classified based on first response, 562. 00:24:01.100 --> 00:24:04.637 When we talk about perception results, the question 00:24:04.637 --> 00:24:08.608 is if there is a difference in perceptions and 00:24:08.608 --> 00:24:11.678 attitudes and methodology, we had some data 00:24:11.678 --> 00:24:13.012 collection time points. 00:24:13.012 --> 00:24:16.683 For the CPCLA we had the fortune of feeding the 00:24:16.683 --> 00:24:19.952 individuals because you have them in a year time 00:24:19.952 --> 00:24:23.256 period, and four Strong touch points at the ending 00:24:23.256 --> 00:24:24.323 of the quarter. 00:24:24.357 --> 00:24:27.694 We delivered post quarterly assessment surveys in 00:24:27.694 --> 00:24:32.031 addition to a final usability survey, and offer the 00:24:32.031 --> 00:24:35.368 ability for focus groups which were optional. 00:24:35.368 --> 00:24:39.005 The CPCA was a post assessment exit survey only, and 00:24:39.038 --> 00:24:41.941 centered on that kind of one time point. 00:24:41.941 --> 00:24:47.647 But what we learned when comparing the two groups is 00:24:47.647 --> 00:24:51.651 that the satisfaction for both testing experiences 00:24:51.651 --> 00:24:53.953 was roughly the same. 00:24:53.953 --> 00:24:56.723 CRNAs were satisfied with their test taking 00:24:56.723 --> 00:25:00.293 experience regardless of the format they took it in. 00:25:00.293 --> 00:25:03.696 The CRNAs that took CPCA were slightly more 00:25:03.696 --> 00:25:06.666 satisfied with their testing experience than those 00:25:06.666 --> 00:25:10.870 in the L.A. format, however, what we notice is that 00:25:10.903 --> 00:25:14.707 the participants in the CPCLA group rated most other 00:25:14.707 --> 00:25:18.077 items higher than participants in the CPCA group. 00:25:18.077 --> 00:25:21.748 Overall, participants were most satisfied with CPCLA 00:25:21.748 --> 00:25:27.020 in terms of the ability to help with knowledge gaps, 00:25:27.020 --> 00:25:32.258 and they felt it was an accurate reflection of core 00:25:32.258 --> 00:25:35.962 knowledge required of all parents and professionals. 00:25:35.962 --> 00:25:38.131 Interestingly, the lowest indoor statement by the 00:25:38.131 --> 00:25:40.967 L.A. group was that it provides better care to 00:25:40.967 --> 00:25:42.535 patients by helping maintain knowledge. 00:25:42.535 --> 00:25:46.039 We have seen that before. 00:25:46.039 --> 00:25:51.277 It is hard to extol the virtues and the benefits and 00:25:51.277 --> 00:25:54.113 the value of certification to participants. 00:25:54.113 --> 00:25:56.883 They know they needed for practice, but where does 00:25:56.883 --> 00:25:59.952 the rubber meets the road in terms of what they 00:25:59.952 --> 00:26:03.022 actually get out of it, outside of the fact that 00:26:03.022 --> 00:26:05.792 they have the ability to earn a livelihood and 00:26:05.792 --> 00:26:08.561 provide care to patients and practice in a certain 00:26:08.561 --> 00:26:11.330 realm? We looked at individuals in the number of 00:26:11.330 --> 00:26:12.565 hours they spent studying. 00:26:12.565 --> 00:26:16.969 On average, how many hours did you study per week? 00:26:16.969 --> 00:26:22.642 The L.A. group, we saw 64% of people were really not 00:26:22.642 --> 00:26:25.745 studying, versus the CPCA, the traditional 00:26:25.745 --> 00:26:29.349 assessment, 37% went into that without studying. 00:26:29.349 --> 00:26:32.085 So more people taking the traditional assessment. 00:26:32.085 --> 00:26:33.086 That stands to reason. 00:26:33.086 --> 00:26:35.588 When you take a traditional assessment, you want to 00:26:35.588 --> 00:26:37.557 go back through the books and study. 00:26:37.557 --> 00:26:41.127 When we are given this longitudinal assessment with 00:26:41.127 --> 00:26:44.263 interspersed repetitive questions, we tend to look 00:26:44.263 --> 00:26:47.834 at it as less stressful, perhaps, and more 00:26:47.834 --> 00:26:52.271 relatable, so we had -- we are less likely to study 00:26:52.271 --> 00:26:53.639 in that methodology. 00:26:53.706 --> 00:26:57.176 In terms of looking at what individuals preferred, 00:26:57.176 --> 00:27:01.481 we saw that, when we asked the L.A. group which 00:27:01.481 --> 00:27:04.951 methodology they would prefer in the future, they 00:27:04.951 --> 00:27:07.553 largely said longitudinal assessment, because they 00:27:07.553 --> 00:27:11.457 were experiencing it and seeing the benefits of that 00:27:11.457 --> 00:27:12.759 tool in action. 00:27:12.759 --> 00:27:16.295 The CPCA group was kind of split, probably because 00:27:16.295 --> 00:27:19.465 most of those individuals are not experiencing the 00:27:19.465 --> 00:27:21.834 benefits of the longitudinal assessment platform, 00:27:21.834 --> 00:27:25.405 and/or may not have been as familiar with the 00:27:25.405 --> 00:27:25.905 methodology. 00:27:25.905 --> 00:27:28.808 In the third research question, we sought to elicit, 00:27:28.808 --> 00:27:31.511 is use of the L.A. platform feasible, acceptable, 00:27:31.511 --> 00:27:32.178 and usable? 00:27:32.178 --> 00:27:36.783 What we saw was an average overall rating of 4.3 out 00:27:36.783 --> 00:27:41.354 of 5 stars, and we felt that was very strong and 00:27:41.354 --> 00:27:44.257 highly advantageous in terms of potential for 00:27:44.290 --> 00:27:45.925 adoption in the future. 00:27:45.925 --> 00:27:50.730 When we kind of sought to dig into topics at a 00:27:50.730 --> 00:27:54.634 deeper level, you will notice the average rating for 00:27:54.667 --> 00:27:58.571 all the questions was really 3.08, which we felt 00:27:58.571 --> 00:28:01.174 indicated that participants were generally satisfied 00:28:01.174 --> 00:28:02.475 with the platform. 00:28:02.475 --> 00:28:06.012 The highest rated question was completing 30-35 00:28:06.012 --> 00:28:10.016 questions per quarter was feasible with my schedule, 00:28:10.016 --> 00:28:15.021 and that scored 3.62. I would take this format 00:28:15.021 --> 00:28:15.988 again, 3.5. 00:28:16.022 --> 00:28:20.626 Some of the ones that fell into the lower stratum, 00:28:20.626 --> 00:28:23.830 individual said participating in L.A. increase my 00:28:23.830 --> 00:28:25.698 knowledge base and anesthesia. 00:28:25.698 --> 00:28:32.872 And then, L.A. helped change how I practiced nurse 00:28:32.872 --> 00:28:33.673 anesthesia. 00:28:33.673 --> 00:28:37.410 So it does beg the question, although it is beyond 00:28:37.410 --> 00:28:39.679 parts of this particular randomized controlled 00:28:39.679 --> 00:28:43.416 trial, it does beg the question if over time we 00:28:43.416 --> 00:28:47.520 would see these items kind of move up in value as 00:28:47.520 --> 00:28:50.890 individuals saw some of those areas of liability or 00:28:50.890 --> 00:28:53.159 weakness become more obvious to them. 00:28:53.159 --> 00:28:56.896 As they started to fill those voids with 00:28:56.896 --> 00:28:59.198 personalized educational opportunities, would they 00:28:59.198 --> 00:29:01.067 then see the value. 00:29:01.067 --> 00:29:05.438 But as I stated before, I think extolling the value 00:29:05.438 --> 00:29:08.508 of certification is really a difficult charge. 00:29:08.508 --> 00:29:12.945 We try to establish value through a number of focal 00:29:12.945 --> 00:29:17.417 points, but sometimes it can be nebulous to get to 00:29:17.417 --> 00:29:18.751 the end user. 00:29:18.751 --> 00:29:22.088 When we look at the platform ratings, we saw that 00:29:22.088 --> 00:29:25.091 login process was easy, the platform was easy to 00:29:25.091 --> 00:29:28.094 navigate without too much effort, and they felt the 00:29:28.094 --> 00:29:31.097 knowledge dashboard was helpful in that it was easy 00:29:31.097 --> 00:29:32.098 to track performance. 00:29:32.098 --> 00:29:36.536 Those are all characteristics that we sought and had 00:29:36.536 --> 00:29:37.537 ideals for. 00:29:37.537 --> 00:29:41.574 Things that kind of showed up at the lower level was 00:29:41.574 --> 00:29:44.911 that information on the review page was helpful, or 00:29:44.911 --> 00:29:47.480 probably readily accessible to them, references with 00:29:47.513 --> 00:29:50.450 answers to questions were useful, and questions were 00:29:50.450 --> 00:29:53.419 clearly written at the appropriate ability level, a 00:29:53.419 --> 00:29:54.520 little bit lower. 00:29:54.520 --> 00:29:57.523 We see those all his opportunities for development 00:29:57.523 --> 00:30:01.627 in our platform over time, so it is highly useful to 00:30:01.627 --> 00:30:02.729 get that information. 00:30:02.729 --> 00:30:06.632 When we talk about the overall usability score, the 00:30:06.632 --> 00:30:10.069 usability scoring matrix allowed us a score between 00:30:10.069 --> 00:30:13.506 0-100, and anything above 68 was considered above 00:30:13.506 --> 00:30:14.007 average. 00:30:14.007 --> 00:30:19.746 You will see the reference thereto the tool used. 00:30:19.746 --> 00:30:23.316 But the CPCA/LA longitudinal assessment platform 00:30:23.316 --> 00:30:26.853 elicited a usability percentage of 80%. 00:30:26.853 --> 00:30:29.288 So we were really happy with that outcome. 00:30:29.288 --> 00:30:32.925 When the asked individuals if they would recommend a 00:30:32.925 --> 00:30:36.195 longitudinal assessment to a CRNA collie, 95% of 00:30:36.195 --> 00:30:40.233 them said yes, so we were pleased with that finding. 00:30:40.233 --> 00:30:43.903 When we looked at the perceptions and attitudes, in 00:30:43.903 --> 00:30:46.739 summary, we found overall participants related to 00:30:46.739 --> 00:30:49.575 migrated usability of the platform highly, that's 00:30:49.575 --> 00:30:53.246 why talk about the 4.3 out of 5 stars. 00:30:53.246 --> 00:30:55.648 They also found participants preferred a more 00:30:55.648 --> 00:30:58.418 continuous L.A. format in the future, according to 00:30:58.418 --> 00:30:59.452 the survey results. 00:30:59.452 --> 00:31:02.422 And although these findings were not statistically 00:31:02.422 --> 00:31:04.957 significant, there were some differences in 00:31:04.957 --> 00:31:07.927 perceptions and attitudes when we compared the 00:31:07.927 --> 00:31:10.463 ratings. The longitudinal assessment group ratings 00:31:10.463 --> 00:31:13.866 were higher than the CPCA traditional assessment, as 00:31:13.866 --> 00:31:16.002 rated in promoting lifelong learning. 00:31:16.002 --> 00:31:19.172 So that is highly advantageous. 00:31:19.172 --> 00:31:21.874 The L.A. platform showed above average usability, 00:31:21.874 --> 00:31:24.977 95% of the individuals would recommend the platform 00:31:24.977 --> 00:31:29.215 to a colleague, so he really felt that all of those 00:31:29.215 --> 00:31:32.318 things were driving us in the right direction. 00:31:32.318 --> 00:31:34.954 So these are the primary research aims, and these 00:31:34.954 --> 00:31:35.822 are our findings. 00:31:35.822 --> 00:31:39.425 There was a difference in the performance that was 00:31:39.425 --> 00:31:41.828 statistically significant between the L.A. and 00:31:41.861 --> 00:31:45.064 traditional assessment group, in scoring the item on 00:31:45.064 --> 00:31:46.666 first versus second attempt. 00:31:46.666 --> 00:31:51.104 So we saw a bit of a difference, but if we counted 00:31:51.104 --> 00:31:54.440 that second answer is correct, and we give them 00:31:54.440 --> 00:31:58.144 credit for getting it correct, in total, moved up to 00:31:58.144 --> 00:32:01.481 a level commensurate with what we are seeing and 00:32:01.481 --> 00:32:05.218 what we have seen with CRNAs who have taken the 00:32:05.218 --> 00:32:06.686 traditional assessment to date. 00:32:06.686 --> 00:32:10.656 The average ratings are perceived higher on 00:32:10.656 --> 00:32:14.627 self-reported agreement scales on the L.A. versus 00:32:14.627 --> 00:32:16.329 the traditional assessment. 00:32:16.329 --> 00:32:20.700 And the feedback on L.A. was overall positive, 00:32:20.700 --> 00:32:22.902 eliciting above average usability. 00:32:22.902 --> 00:32:26.839 Leaf suggested these findings suggest it is a 00:32:26.839 --> 00:32:30.276 feasible and usable format in the future. 00:32:30.276 --> 00:32:33.880 Was the future for our CPCA program? 00:32:33.880 --> 00:32:39.185 We have had a lot of feedback over time, most of it 00:32:39.185 --> 00:32:43.156 good and useful, and probably, like many of you, 00:32:43.156 --> 00:32:46.693 feedback is always solicited or elicited and some 00:32:46.693 --> 00:32:51.564 comes on its own, but we have heard that our current 00:32:51.564 --> 00:32:55.535 program has cycle lengths asking them to do things 00:32:55.535 --> 00:32:56.869 at different times. 00:32:56.869 --> 00:33:02.241 They find this incredibly confusing and, honestly, 00:33:02.241 --> 00:33:09.148 as a fellow certificant, I understand and agree with 00:33:09.148 --> 00:33:09.916 that. 00:33:09.916 --> 00:33:12.985 We are moving to a repeated four year cycle with the 00:33:12.985 --> 00:33:15.188 same requirements in each cycle that an individual 00:33:15.188 --> 00:33:15.755 goes through. 00:33:15.755 --> 00:33:20.993 So what we are going to have is 60 credits, much 00:33:20.993 --> 00:33:25.264 like the current program, the 40 class B credits, 00:33:25.264 --> 00:33:28.601 again, like the current program, and individuals 00:33:28.601 --> 00:33:32.405 will have to meaningfully -- we have a definition, 00:33:32.405 --> 00:33:36.175 meaningful participation over a 4-year period. So we 00:33:36.209 --> 00:33:39.045 are transitioning from the current traditional 00:33:39.045 --> 00:33:41.414 assessment of continued professional certification 00:33:41.414 --> 00:33:44.751 assessment to using longitudinal assessment in terms 00:33:44.751 --> 00:33:49.022 of measuring knowledge of CRNAs who desire to have 00:33:49.022 --> 00:33:49.956 continued certification. 00:33:50.023 --> 00:33:53.760 And one thing they are removing from our program is 00:33:53.760 --> 00:33:57.096 we had a component called the core modules, which 00:33:57.096 --> 00:34:00.466 was really four domain areas of focus learning that 00:34:00.466 --> 00:34:02.669 all CRNAs were asked to do. 00:34:02.669 --> 00:34:05.805 The goal is to infuse contemporary knowledge into 00:34:05.805 --> 00:34:09.308 the profession and keep us all within a certain 00:34:09.308 --> 00:34:10.476 realm and level. 00:34:10.476 --> 00:34:14.480 Those tools, despite having wonderful ideals, we 00:34:14.480 --> 00:34:19.585 felt did not materialize to their full benefit, so 00:34:19.585 --> 00:34:24.157 we are removing that as a requirement. However, 00:34:24.157 --> 00:34:28.695 individuals, vendors and the space who spent the 00:34:28.695 --> 00:34:33.232 time and effort will likely see those mechanisms 00:34:33.232 --> 00:34:37.804 repurposed to other CEUs and probably class A 00:34:37.804 --> 00:34:38.371 credit. 00:34:38.371 --> 00:34:41.774 With that, I want to stop and say thank you very 00:34:41.774 --> 00:34:43.910 much for the opportunity to present today. 00:34:43.943 --> 00:34:47.780 I look forward to answering any questions you have 00:34:47.780 --> 00:34:52.952 during the live Q&A, and I just want to make a side 00:34:52.952 --> 00:34:56.356 point of saying that the research study, the 00:34:56.389 --> 00:34:59.792 randomized controlled trial I've talked to you about 00:34:59.792 --> 00:35:03.663 today, has been accepted for publication and will be 00:35:03.663 --> 00:35:06.666 published in the practical assessment research and 00:35:06.666 --> 00:35:09.669 evaluation Journal at some point in 2024. 00:35:09.669 --> 00:35:12.171 However, we haven't heard exactly what issue it is 00:35:12.171 --> 00:35:13.539 going to be published in. 00:35:13.573 --> 00:35:17.410 But it will be published in 2024, so you will be 00:35:17.410 --> 00:35:20.213 able to delve into these findings of randomized 00:35:20.213 --> 00:35:23.383 controlled trial at a deeper level once that is 00:35:23.383 --> 00:35:23.883 published. 00:35:23.883 --> 00:35:36.095 Again, I want to thank you for the opportunity to 00:35:36.129 --> 00:35:49.742 speak with you today and I would like to take any 00:35:49.742 --> 00:35:55.948 questions that you have now. 00:35:55.948 --> 00:35:59.485 >> Michelle: Thank you so much, Dr. Gill, for your 00:35:59.485 --> 00:36:01.954 presentation and sharing your study results, and 00:36:01.954 --> 00:36:04.757 talking with us about your innovative approach to 00:36:04.757 --> 00:36:05.258 recertification. 00:36:05.258 --> 00:36:10.163 We do have some questions, so we are going to jump 00:36:10.163 --> 00:36:14.701 right in. What kind of feedback did you receive from 00:36:14.701 --> 00:36:17.003 participants regarding the immediate feedback 00:36:17.003 --> 00:36:21.107 feature of the CPCA/LA platform, and how did this 00:36:21.107 --> 00:36:23.376 feedback influence their learning process? 00:36:23.376 --> 00:36:26.879 >> Christopher: For sure. 00:36:26.879 --> 00:36:29.949 The immediate feedback as well as the convenience 00:36:29.949 --> 00:36:33.019 features that are inherent kind of within the 00:36:33.019 --> 00:36:35.688 platform itself were well regarded by the 00:36:35.688 --> 00:36:36.189 participants. 00:36:36.189 --> 00:36:39.092 They said that was something that could really make 00:36:39.092 --> 00:36:41.761 a difference in terms of day-to-day and maintaining 00:36:41.761 --> 00:36:43.096 a fund of knowledge. 00:36:43.096 --> 00:36:45.998 There is certainly a number of aspects of adult 00:36:45.998 --> 00:36:48.901 learning theory that is different then when you are 00:36:48.901 --> 00:36:50.870 going through grade school, through collegiate. 00:36:50.870 --> 00:36:56.576 So I've touched on a number of them, but really we 00:36:56.576 --> 00:37:01.781 want to make people aware of areas that they may 00:37:01.781 --> 00:37:04.917 have opportunities for improvement, really, as 00:37:04.917 --> 00:37:08.554 adults we have an illusion of knowing. 00:37:08.554 --> 00:37:11.424 We think that we have a greater fund of knowledge 00:37:11.424 --> 00:37:14.027 and certain topical areas and maybe perhaps we do. 00:37:14.027 --> 00:37:19.632 So by putting that information in real time in front 00:37:19.632 --> 00:37:24.103 of practicing clinicians, we think we'll move them 00:37:24.103 --> 00:37:26.339 to the next level. 00:37:26.339 --> 00:37:28.074 >> Michelle: Terrific, thanks. 00:37:28.074 --> 00:37:30.076 Next question. 00:37:30.076 --> 00:37:34.313 How have the recertification changes affected CRNA 00:37:34.313 --> 00:37:36.749 costs related to recertification? 00:37:36.749 --> 00:37:40.853 Secondly, I clinicians reporting increased CE costs 00:37:40.853 --> 00:37:44.924 or time burdens associated with maintaining their 00:37:44.924 --> 00:37:45.525 certifications? 00:37:45.525 --> 00:37:49.429 >> Christopher: Absolutely. 00:37:49.429 --> 00:37:52.899 From the kind of legacy program to the current 00:37:52.899 --> 00:37:54.834 continued professional certification program, there 00:37:54.834 --> 00:37:57.537 was a change in the cost component. 00:37:57.537 --> 00:38:01.808 So I'm not going to mince words on that one. 00:38:01.808 --> 00:38:05.111 In hearing feedback from our certificants, Liv 00:38:05.111 --> 00:38:09.782 understood that the pain point for them, and we have 00:38:09.782 --> 00:38:13.052 sought to reconfigure and reimagine our programs 00:38:13.052 --> 00:38:15.888 with cost and value in mind. 00:38:15.888 --> 00:38:20.293 So, as we moved to the new program, which we are 00:38:20.293 --> 00:38:24.697 going to end up having a new name for, we have 00:38:24.697 --> 00:38:28.701 sought to really control costs at a point where they 00:38:28.701 --> 00:38:31.904 are kind of just on par with cost-of-living 00:38:31.904 --> 00:38:32.405 increases. 00:38:32.405 --> 00:38:33.706 We have increased the cost of the program. 00:38:33.706 --> 00:38:37.243 So we think we will be able to enhance the value the 00:38:37.243 --> 00:38:40.179 end user is going to see and received without really 00:38:40.179 --> 00:38:41.948 seeing a significant increase in cost. 00:38:41.948 --> 00:38:47.020 In relation to CE costs, it is quite varied, 00:38:47.020 --> 00:38:49.822 actually. I will say that. 00:38:49.822 --> 00:38:54.027 Where you working where you practice, you may have 00:38:54.027 --> 00:38:57.296 continuing medical education funds available to you. 00:38:57.296 --> 00:39:01.634 Your employer may provide a class A, the CE that we 00:39:01.634 --> 00:39:06.005 see as being costly in terms of per credit cost. So 00:39:06.005 --> 00:39:09.575 there's kind of a great variety in that. There 00:39:09.575 --> 00:39:13.546 certainly was a bit of an increased cost going from 00:39:13.546 --> 00:39:18.284 40 to 60 CEUs, but as we increase the length of the 00:39:18.317 --> 00:39:21.487 certification cycle, we felt that was the right 00:39:21.487 --> 00:39:22.655 thing to do. 00:39:22.655 --> 00:39:25.925 >> Michelle: Thank you. Next question. You had 00:39:25.925 --> 00:39:29.962 mentioned that one of the issues that was an impetus 00:39:29.962 --> 00:39:34.033 for you doing this work and looking at changing the 00:39:34.033 --> 00:39:37.704 assessment process had to do with the evolution and 00:39:37.704 --> 00:39:39.739 the development of new technologies. 00:39:39.739 --> 00:39:43.443 Can you discuss that further? 00:39:43.443 --> 00:39:46.279 Were there particular technologies? We talked about 00:39:46.279 --> 00:39:47.480 AI earlier today. 00:39:47.480 --> 00:39:51.284 Can you talk about some of those new technologies 00:39:51.284 --> 00:39:55.054 that you observed evolving that prompted you guys to 00:39:55.054 --> 00:39:56.322 take this step? 00:39:56.322 --> 00:39:59.659 >> Christopher: Sure. 00:39:59.659 --> 00:40:03.863 I think, fundamentally, the Board of Directors and 00:40:03.863 --> 00:40:08.601 the staff at the organization has explored, I would 00:40:08.601 --> 00:40:13.373 say, in a very academic-minded manner, what is out 00:40:13.373 --> 00:40:13.873 there. 00:40:13.906 --> 00:40:15.808 First you have to understand the question I talked 00:40:15.808 --> 00:40:17.276 about at the outset of the presentation. 00:40:17.276 --> 00:40:22.749 What are the ideals we are seeking to achieve and 00:40:22.749 --> 00:40:23.282 meet? 00:40:23.282 --> 00:40:28.921 For the practicing clinician, that is markedly 00:40:28.921 --> 00:40:34.560 different than someone who is entering the 00:40:34.560 --> 00:40:38.598 profession for the first time. 00:40:38.598 --> 00:40:43.169 We explored a number of different aspects, whether 00:40:43.169 --> 00:40:46.606 it be simulation lab, virtual-reality, augmented 00:40:46.606 --> 00:40:48.875 reality to some degree. 00:40:48.875 --> 00:40:52.912 So it is getting the cost and value proposition, I 00:40:52.912 --> 00:40:56.149 think, perfectly correct -- or at least as correct 00:40:56.149 --> 00:41:00.186 as you can -- in that moment that is the hardest 00:41:00.186 --> 00:41:00.687 part. 00:41:01.020 --> 00:41:03.723 As longitudinal assessment platforms have matured, 00:41:03.823 --> 00:41:07.093 where individuals can do questions, there are 00:41:07.093 --> 00:41:10.830 security safeguards in place, like a permanent time 00:41:10.830 --> 00:41:14.567 clock or other kind of security features that 00:41:14.567 --> 00:41:17.837 prevent loss of intellectual property, as those 00:41:17.837 --> 00:41:21.107 technologies have matured, we have really followed 00:41:21.107 --> 00:41:22.942 closely with those vendors. 00:41:22.942 --> 00:41:28.448 We saw this as the right time to remove that 00:41:28.448 --> 00:41:33.386 technology, because we think it is getting closer to 00:41:33.386 --> 00:41:37.790 the stated goal of enhancing provider quality and 00:41:37.790 --> 00:41:39.425 maintaining patient safety. 00:41:39.425 --> 00:41:43.830 >> Michelle: Thank you. Next question. 00:41:43.830 --> 00:41:48.034 How did technology play a role in facilitating or 00:41:48.034 --> 00:41:51.337 hindering the effectiveness of the CPCA/LA platform? 00:41:51.337 --> 00:41:55.541 And what improvements could be made to enhance the 00:41:55.541 --> 00:41:56.476 user's experience? 00:41:56.476 --> 00:41:59.345 >> Christopher: Share. 00:41:59.345 --> 00:42:01.848 The positive things that people have reported on 00:42:01.848 --> 00:42:03.082 where the convenience functionality. 00:42:03.082 --> 00:42:07.186 That they can do questions anywhere, at any time, as 00:42:07.186 --> 00:42:11.324 long as they can access a device that is supported. 00:42:11.324 --> 00:42:14.160 The immediate feedback is highly sought after. We 00:42:14.160 --> 00:42:18.064 want to know what we didn't get right, and we want 00:42:18.064 --> 00:42:21.601 to have the time to go and research that when -- 00:42:21.601 --> 00:42:22.669 usually right after. 00:42:22.769 --> 00:42:26.039 We don't like not knowing something, especially when 00:42:26.039 --> 00:42:28.474 we are taking care of patients. 00:42:28.474 --> 00:42:31.911 So that immediate feedback coupled with providing 00:42:31.911 --> 00:42:35.314 rationales was advantageous, and people saw this 00:42:35.314 --> 00:42:38.751 being highly favorable. The nonpunitive nature of 00:42:38.751 --> 00:42:42.655 the system was also something they highly regarded. 00:42:42.655 --> 00:42:46.192 Things they kind of reported that they felt created 00:42:46.192 --> 00:42:49.762 some stress where the one minute of time per 00:42:49.762 --> 00:42:53.299 question that maybe made them feel a little bit 00:42:53.299 --> 00:42:53.800 rushed. 00:42:53.800 --> 00:42:56.903 We think over time there's going be a more positive 00:42:56.936 --> 00:42:59.505 assimilation of that, where you are not really 00:42:59.505 --> 00:43:00.473 having to think. 00:43:00.473 --> 00:43:03.076 It is walking around knowledge we are trying to 00:43:03.076 --> 00:43:05.645 capture, that we feel confident that one is correct. 00:43:05.645 --> 00:43:08.448 The study findings confirmed that 97% of 00:43:08.448 --> 00:43:11.651 participants were able to answer the question in 00:43:11.651 --> 00:43:12.852 around 30 seconds. 00:43:12.852 --> 00:43:16.789 So we are getting around that knowledge. 00:43:16.789 --> 00:43:19.225 The software platform and technology we have used 00:43:19.225 --> 00:43:21.961 has an advanced functionality where the end user has 00:43:21.994 --> 00:43:25.031 to click, yes, I want to answer the next question. 00:43:25.031 --> 00:43:27.967 That was kind of frustrating for some individuals, 00:43:27.967 --> 00:43:32.005 but we see that as being something we want to leave 00:43:32.005 --> 00:43:34.974 in place, because you might be answering questions 00:43:34.974 --> 00:43:37.910 and get interrupted, and we don't want individuals 00:43:37.910 --> 00:43:40.847 to get unfairly docked in terms of points. 00:43:40.847 --> 00:43:44.584 They are going to the next question, they have to 00:43:44.584 --> 00:43:45.318 select that. 00:43:45.318 --> 00:43:50.490 So that was another pain point. 00:43:50.490 --> 00:43:54.660 We are going to grow the functionality where the end 00:43:54.660 --> 00:43:58.398 user can see statistics and metrics that talk about 00:43:58.398 --> 00:44:00.466 areas of strength and weakness. 00:44:00.466 --> 00:44:03.903 From a topical and domain level, that is something 00:44:03.903 --> 00:44:05.038 that we have. 00:44:05.038 --> 00:44:08.474 They do have that functionality today, but we see 00:44:08.474 --> 00:44:11.878 that as being the potential for enhancement in the 00:44:11.878 --> 00:44:16.082 future where we may take it to the point to create 00:44:16.082 --> 00:44:19.852 kind of a personalized grid where the end user can 00:44:19.852 --> 00:44:24.057 go to CE vendors and say, look, my certifier and my 00:44:24.057 --> 00:44:26.325 maintenance of certification program, my L.A. 00:44:26.325 --> 00:44:27.460 results, show this. 00:44:27.460 --> 00:44:32.432 What do you think you can do in terms of creating a 00:44:32.432 --> 00:44:34.500 personalized CE plan for me? 00:44:34.500 --> 00:44:37.437 I do think, overall, highly positive enhancements in 00:44:37.437 --> 00:44:41.107 the future, and this is something we are all going 00:44:41.107 --> 00:44:43.676 to be tracking closely in the space. 00:44:43.676 --> 00:44:45.111 Sorry, a little long-winded. 00:44:45.111 --> 00:44:45.611 [L[Laughs] 00:44:45.611 --> 00:44:47.613 >> Michelle: No, that was terrific. 00:44:47.613 --> 00:44:48.114 Thanks, Chris. 00:44:48.114 --> 00:44:51.818 I think you are giving us a lot to think about and 00:44:51.818 --> 00:44:54.287 an entirely different way to look at these 00:44:54.320 --> 00:44:54.821 processes. 00:44:54.821 --> 00:45:01.394 We have kind of a comment, it's a quote. 00:45:01.427 --> 00:45:03.696 "People gravitate toward learning what they don't 00:45:03.696 --> 00:45:04.197 need." 00:45:04.197 --> 00:45:05.231 You said that. 00:45:05.231 --> 00:45:08.768 And she says -- the person says, "this certainly 00:45:08.768 --> 00:45:09.268 resonates. 00:45:09.335 --> 00:45:12.338 It's all about what I can get for free rather than 00:45:12.405 --> 00:45:13.806 reflecting on gaps of knowledge. 00:45:13.806 --> 00:45:17.844 And this is from the perspective of a state that 00:45:17.844 --> 00:45:19.479 requires CE for nurses." 00:45:19.479 --> 00:45:21.647 This is what the comment is. 00:45:21.681 --> 00:45:27.020 So someone agreeing with your previous comment. 00:45:27.020 --> 00:45:30.523 I think that is about the end of our time. 00:45:30.523 --> 00:45:34.260 Thank you again, Dr. Gill. 00:45:34.260 --> 00:45:38.564 This was incredibly interesting, and we look forward 00:45:38.564 --> 00:45:43.436 to seeing your work and how your organization moves 00:45:43.436 --> 00:45:44.504 us forward. 00:45:44.537 --> 00:45:45.872 Thank you so much. 00:45:45.872 --> 00:45:47.674 >> Christopher: Wonderful. 00:45:47.674 --> 00:45:48.274 Thank you.