WEBVTT 00:00:04.878 --> 00:00:08.110 position:50% align:middle - [Crystal] Hello. I'm Crystal Tillman, and I'm from North Carolina. 00:00:08.110 --> 00:00:12.170 position:50% align:middle I also serve as the CEO for the North Carolina Board. 00:00:12.170 --> 00:00:18.400 position:50% align:middle And I've had the pleasure of serving a chair for the NCLEX Examination Committee. 00:00:18.400 --> 00:00:28.050 position:50% align:middle And today I'm partnering with Phil Dickison, COO of NCSBN, soon to be the next CEO in October. 00:00:28.050 --> 00:00:33.330 position:50% align:middle And we're going to be doing the NCLEX Examination Committee update. 00:00:33.330 --> 00:00:37.920 position:50% align:middle We're going to do a little bit differently this year, which I think is exciting. 00:00:37.920 --> 00:00:46.258 position:50% align:middle So being on the NEC is much like the great doctor, Dr. Seuss, his book, 00:00:46.258 --> 00:00:49.780 position:50% align:middle "Oh, the Places You'll Go," on NEC. 00:00:49.780 --> 00:00:53.560 position:50% align:middle So here we're doing [inaudible] just got it lifted off. 00:00:53.560 --> 00:00:57.120 position:50% align:middle And then Phil says, "Okay, let's go and jump in into AI." 00:00:57.120 --> 00:00:58.500 position:50% align:middle So here we are. 00:00:58.500 --> 00:01:04.140 position:50% align:middle And if you were fortunate enough to be at Midyear, you got to hear his great presentation. 00:01:04.140 --> 00:01:11.240 position:50% align:middle And we're going to recap some of that today and show you some other things as well. 00:01:11.240 --> 00:01:18.190 position:50% align:middle So I'd like a special thank you to our committee members and the NCSBN staff that works with NEC. 00:01:18.190 --> 00:01:20.410 position:50% align:middle You're all exceptional. 00:01:20.410 --> 00:01:28.570 position:50% align:middle The NEC's primary responsibility is advising on the matters related to the NCLEX examination process. 00:01:28.570 --> 00:01:34.000 position:50% align:middle For the past several years, the NEC has focused on the impact of the NCLEX, 00:01:34.000 --> 00:01:40.710 position:50% align:middle and the changes in landscape of health care and the effects the pandemic has had on the 00:01:40.710 --> 00:01:43.140 position:50% align:middle health care worldwide. 00:01:43.140 --> 00:01:49.000 position:50% align:middle It's important to note that the NCLEX experienced a reduction in the pandemic, 00:01:49.000 --> 00:01:58.420 position:50% align:middle but certainly did not experience a disruption of exam administration, which is huge that we were able to keep 00:01:58.420 --> 00:02:01.000 position:50% align:middle going during the pandemic. 00:02:01.000 --> 00:02:09.010 position:50% align:middle The exam continues to support validity and reliability, and the NCSBN persists in its focus on increasing 00:02:09.010 --> 00:02:14.800 position:50% align:middle efficiency and effectiveness in the licensure process. 00:02:14.800 --> 00:02:18.020 position:50% align:middle Today, I'd like to present the NEC work. 00:02:18.020 --> 00:02:21.080 position:50% align:middle And I brought the expert with me. 00:02:21.080 --> 00:02:26.440 position:50% align:middle During the 2023 Midyear Meeting, you received an update on the work to support the 00:02:26.440 --> 00:02:33.190 position:50% align:middle development of a secure, reliable, remote proctoring and advanced AI system. 00:02:33.190 --> 00:02:38.150 position:50% align:middle With the implementation of this project, some of or all testing to remote 00:02:38.150 --> 00:02:43.820 position:50% align:middle proctoring administration, and offer increased security and workflow efficiencies 00:02:43.820 --> 00:02:49.710 position:50% align:middle for the nurse and regulatory bodies with better software tools, and AI support systems. 00:02:49.710 --> 00:02:57.170 position:50% align:middle So folks, put your seatbelt on, tighten your strap, because you're about to blast into space with Phil. 00:02:57.170 --> 00:03:04.740 position:50% align:middle And David, what you did today was a great prelude to what we will be talking about today. 00:03:04.740 --> 00:03:11.420 position:50% align:middle So, Phil, can you give us a recap of the information presented during the Midyear Meeting? 00:03:11.420 --> 00:03:14.430 position:50% align:middle - [Phil] Yes, I can. 00:03:14.430 --> 00:03:15.990 position:50% align:middle She asked if I could. 00:03:15.990 --> 00:03:18.440 position:50% align:middle She didn't ask if I'd do it, but that's okay. 00:03:18.440 --> 00:03:21.020 position:50% align:middle I can. Oh, I will do it. 00:03:21.020 --> 00:03:23.390 position:50% align:middle I have to...you know I can't stand still. 00:03:23.390 --> 00:03:25.850 position:50% align:middle So this is going to have to be not in front of that podium. 00:03:25.850 --> 00:03:26.860 position:50% align:middle I'm sorry. 00:03:26.860 --> 00:03:30.200 position:50% align:middle I'll be back here because I can't do it without talking with my hands. 00:03:30.200 --> 00:03:34.430 position:50% align:middle But at Midyear, we talked about where were we going? 00:03:34.430 --> 00:03:36.150 position:50% align:middle What was the map for remote proctoring? 00:03:36.150 --> 00:03:37.660 position:50% align:middle What did we have to solve? 00:03:37.660 --> 00:03:39.010 position:50% align:middle I'm not going to go all over those. 00:03:39.010 --> 00:03:41.030 position:50% align:middle You should have seen them. 00:03:41.030 --> 00:03:42.970 position:50% align:middle We can make them available to you. 00:03:42.970 --> 00:03:48.740 position:50% align:middle But ultimately, the idea was if we were going to use remote proctoring, if we were going to use AI, 00:03:48.740 --> 00:03:50.790 position:50% align:middle what were the things that we had to solve? 00:03:50.790 --> 00:03:56.830 position:50% align:middle And through these points that we've made here, we said that we got to solve 27 points of validity, 00:03:56.830 --> 00:03:58.460 position:50% align:middle or threats to validity. 00:03:58.460 --> 00:04:03.140 position:50% align:middle And they all came back to, as I told you, a validity triangle, right? 00:04:03.140 --> 00:04:11.600 position:50% align:middle Validity triangle, the strongest building block you can ever get is a triangle, and it has three legs, content, 00:04:11.600 --> 00:04:13.690 position:50% align:middle psychometrics, and security. 00:04:13.690 --> 00:04:17.420 position:50% align:middle We have the greatest content in the world, and we have the greatest psychometrics in the world. 00:04:17.420 --> 00:04:20.850 position:50% align:middle But when you go to AI, you're changing all the security. 00:04:20.850 --> 00:04:27.370 position:50% align:middle So a lot of the focus was going to be on security, and how we then could get this administered 00:04:27.370 --> 00:04:28.870 position:50% align:middle in a remote environment. 00:04:28.870 --> 00:04:35.400 position:50% align:middle And so we continue to focus the research, ensuring that we have data interoperability, 00:04:35.400 --> 00:04:39.790 position:50% align:middle those sort of things that we're working there, the technology solutions. 00:04:39.790 --> 00:04:41.030 position:50% align:middle All of this. 00:04:41.030 --> 00:04:46.040 position:50% align:middle And I showed you what the candidate experience before the exam might look like. 00:04:46.040 --> 00:04:48.460 position:50% align:middle We started to talk about the tech check. 00:04:48.460 --> 00:04:50.300 position:50% align:middle And we said, "We've got this AI built. 00:04:50.300 --> 00:04:55.830 position:50% align:middle We could actually build a tech check that a person would log in to a site three days before the exam. 00:04:55.830 --> 00:04:58.160 position:50% align:middle We could check their..." I showed it to you, right? 00:04:58.160 --> 00:05:00.520 position:50% align:middle We can check their open browsers. 00:05:00.520 --> 00:05:02.870 position:50% align:middle We can check what was on that machine. 00:05:02.870 --> 00:05:04.090 position:50% align:middle We could lock all that down. 00:05:04.090 --> 00:05:08.600 position:50% align:middle And then, oh, by the way, you remember, we could get their face graphed, 00:05:08.600 --> 00:05:15.260 position:50% align:middle take their voice so that when they came back three days later, we would have a voiceprint and a face graph. 00:05:15.260 --> 00:05:17.700 position:50% align:middle So we would know it was exactly the same person. 00:05:17.700 --> 00:05:18.950 position:50% align:middle We went through all that. 00:05:18.950 --> 00:05:22.010 position:50% align:middle We talked about graphing, and how we could do that. 00:05:22.010 --> 00:05:25.890 position:50% align:middle And what I want to show you...wanted to show you, but you'll just have to believe me now, 00:05:25.890 --> 00:05:32.630 position:50% align:middle and I haven't lied to you before, so please believe me, is what we've done since then. 00:05:32.630 --> 00:05:34.480 position:50% align:middle Remember, I was showing you single things. 00:05:34.480 --> 00:05:39.970 position:50% align:middle We can map the face, we can map a voice, we can take an item. 00:05:39.970 --> 00:05:46.140 position:50% align:middle And actually, if you're trying to do a snapshot or a screengrab of it, make it go away with QR codes. 00:05:46.140 --> 00:05:51.270 position:50% align:middle I talked about having little DNA, and what I called cyber DNA in there so that if you 00:05:51.270 --> 00:05:57.600 position:50% align:middle ever showed it, I'd know exactly with that barcode when you took it, and at what machine you're on when 00:05:57.600 --> 00:05:59.420 position:50% align:middle you took it. We showed you all that. 00:05:59.420 --> 00:06:01.980 position:50% align:middle But [inaudible] we had to start, what? 00:06:01.980 --> 00:06:04.080 position:50% align:middle Stitching it together. 00:06:04.080 --> 00:06:05.860 position:50% align:middle So let me tell you how far we've gotten. 00:06:05.860 --> 00:06:07.190 position:50% align:middle And it is rocket speed. 00:06:07.190 --> 00:06:09.620 position:50% align:middle So David's talking 2050. 00:06:09.620 --> 00:06:11.681 position:50% align:middle I'm talking 2025. 00:06:11.681 --> 00:06:15.200 position:50% align:middle So we have a different speed. 00:06:15.200 --> 00:06:19.104 position:50% align:middle His is larger, but I think we can do all this by 2025. 00:06:19.104 --> 00:06:21.580 position:50% align:middle And what am I saying? So we started stitching together. 00:06:21.580 --> 00:06:27.830 position:50% align:middle What if we wanted our AI to get a second person detection, right? 00:06:27.830 --> 00:06:31.100 position:50% align:middle And so some of the slides I was going to show you is really cool. 00:06:31.100 --> 00:06:34.370 position:50% align:middle So we started with, "Okay, if a second person was in a room, 00:06:34.370 --> 00:06:35.830 position:50% align:middle could our AI pick it up?" 00:06:35.830 --> 00:06:39.170 position:50% align:middle Guess what. Yeah, almost any AI can pick that up right now. 00:06:39.170 --> 00:06:40.150 position:50% align:middle That's not that good. 00:06:40.150 --> 00:06:43.220 position:50% align:middle So I said, "Well, that's...we need better than that. 00:06:43.220 --> 00:06:48.010 position:50% align:middle What if all they could see was a hand and a foot? 00:06:48.010 --> 00:06:51.590 position:50% align:middle Could our AI tell it was a human?" 00:06:51.590 --> 00:06:53.210 position:50% align:middle We've done it. 00:06:53.210 --> 00:06:55.460 position:50% align:middle I said, "Well, okay. That's kind of cool. 00:06:55.460 --> 00:06:58.840 position:50% align:middle But what if they weren't actually in the room? 00:06:58.840 --> 00:07:01.890 position:50% align:middle You just caught their head peeking around the window. 00:07:01.890 --> 00:07:06.300 position:50% align:middle Could we get them?" We did. 00:07:06.300 --> 00:07:07.780 position:50% align:middle That's cool, right? 00:07:07.780 --> 00:07:10.250 position:50% align:middle So I said, "Well, that's probably not good enough either. 00:07:10.250 --> 00:07:12.680 position:50% align:middle What about just a shadow? 00:07:12.680 --> 00:07:14.470 position:50% align:middle What if we could find it with just a shadow? 00:07:14.470 --> 00:07:18.460 position:50% align:middle We don't even actually see the person, but there's a shadow of one. 00:07:18.460 --> 00:07:19.500 position:50% align:middle Can we find that?" 00:07:19.500 --> 00:07:20.860 position:50% align:middle Guess what? 00:07:20.860 --> 00:07:22.870 position:50% align:middle We did. 00:07:22.870 --> 00:07:27.010 position:50% align:middle That's the type of thing that...that's how fast we've gotten since March. 00:07:27.010 --> 00:07:28.060 position:50% align:middle We couldn't do that in March. 00:07:28.060 --> 00:07:30.440 position:50% align:middle We can do that now. Very cool. 00:07:30.440 --> 00:07:35.500 position:50% align:middle So in stitching it together, what about that second voice? Right? 00:07:35.500 --> 00:07:36.640 position:50% align:middle Can we pick up a second voice? 00:07:36.640 --> 00:07:38.420 position:50% align:middle Well, let me back up before I do that. 00:07:38.420 --> 00:07:41.000 position:50% align:middle So I showed you the graphing of the face. 00:07:41.000 --> 00:07:42.390 position:50% align:middle Some of you remember that, right? 00:07:42.390 --> 00:07:45.800 position:50% align:middle And the guy with his mouth, and there was graphing around, right? 00:07:45.800 --> 00:07:47.410 position:50% align:middle You could watch it move, right? 00:07:47.410 --> 00:07:48.980 position:50% align:middle You could see their eyes moving. 00:07:48.980 --> 00:07:51.110 position:50% align:middle And we graphed all that. 00:07:51.110 --> 00:07:56.730 position:50% align:middle What if they were moving their mouth, and we could actually get our AI to lip read? 00:07:56.730 --> 00:08:00.000 position:50% align:middle That's kind of cool. 00:08:00.000 --> 00:08:01.670 position:50% align:middle Yeah. Yeah, we can. 00:08:01.670 --> 00:08:06.750 position:50% align:middle So what if there's a second voice in the room? 00:08:06.750 --> 00:08:10.070 position:50% align:middle Can we tell it's a second voice? 00:08:10.070 --> 00:08:11.460 position:50% align:middle Yeah. We did it. 00:08:11.460 --> 00:08:13.760 position:50% align:middle We stitched it together. We can do it. 00:08:13.760 --> 00:08:18.720 position:50% align:middle What if that second voice is not a human voice, it's an electronic voice? 00:08:18.720 --> 00:08:23.220 position:50% align:middle I.e., let's assume... How it happened, I don't know. 00:08:23.220 --> 00:08:30.250 position:50% align:middle But let's assume for a minute that our AI missed an earpiece, or missed a phone. 00:08:30.250 --> 00:08:33.130 position:50% align:middle Maybe they got attached somewhere else. 00:08:33.130 --> 00:08:35.180 position:50% align:middle So what we did was put a phone... 00:08:35.180 --> 00:08:36.620 position:50% align:middle Love this one. 00:08:36.620 --> 00:08:40.170 position:50% align:middle Put a phone in the room over here, and had our person taking a test. 00:08:40.170 --> 00:08:44.140 position:50% align:middle We called that phone, opened the line, and we spoke through that line. 00:08:44.140 --> 00:08:53.940 position:50% align:middle Not only were we able to tell it was an electronic voice, we were able actually to have the AI print the 00:08:53.940 --> 00:08:59.830 position:50% align:middle text out of what was being said on that phone. 00:08:59.830 --> 00:09:04.780 position:50% align:middle We actually put a earphone in, and could actually know something was coming 00:09:04.780 --> 00:09:07.360 position:50% align:middle through there, but it wasn't loud enough for us to translate it. 00:09:07.360 --> 00:09:11.800 position:50% align:middle But we did even pick up an earphone that was speaking to the person. 00:09:11.800 --> 00:09:16.160 position:50% align:middle This is outside a door where you got lines over it. 00:09:16.160 --> 00:09:17.950 position:50% align:middle Didn't miss the person at all. 00:09:17.950 --> 00:09:22.630 position:50% align:middle Look at this one. This one is distorted and we still got it. 00:09:22.630 --> 00:09:25.050 position:50% align:middle There's even a cooler one. 00:09:25.050 --> 00:09:26.660 position:50% align:middle This one, show this. Yeah. 00:09:26.660 --> 00:09:28.370 position:50% align:middle Look back there behind the door. 00:09:28.370 --> 00:09:31.080 position:50% align:middle Do you see that person back there? 00:09:31.080 --> 00:09:33.010 position:50% align:middle Our AI did. 00:09:33.010 --> 00:09:34.600 position:50% align:middle They're not even in the room. 00:09:34.600 --> 00:09:37.225 position:50% align:middle They're just walking past the door. 00:09:37.660 --> 00:09:39.320 position:50% align:middle Kind of cool. 00:09:39.320 --> 00:09:41.910 position:50% align:middle This is all the things our AI detected. 00:09:41.910 --> 00:09:43.190 position:50% align:middle Remember what David said? 00:09:43.190 --> 00:09:44.630 position:50% align:middle This is a really cool thing to remember. 00:09:44.630 --> 00:09:46.500 position:50% align:middle He is absolutely correct. 00:09:46.500 --> 00:09:49.410 position:50% align:middle AIs right now don't make decisions. 00:09:49.410 --> 00:09:51.080 position:50% align:middle They provide data. 00:09:51.080 --> 00:09:54.940 position:50% align:middle Every one of these flags our AI got, it timed when they got it, 00:09:54.940 --> 00:09:59.810 position:50% align:middle and what it required was for a human proctor, right, 00:09:59.810 --> 00:10:05.160 position:50% align:middle who's watching too, to check each of those out, and either pass them or not. 00:10:05.160 --> 00:10:05.620 position:50% align:middle Right? 00:10:05.620 --> 00:10:07.730 position:50% align:middle And so it found this person. 00:10:07.730 --> 00:10:10.160 position:50% align:middle It found... Let me see. I think there's a better picture. 00:10:10.160 --> 00:10:11.990 position:50% align:middle That's the voice. Look at this. 00:10:11.990 --> 00:10:16.550 position:50% align:middle The voice here, this was the actual organic voice here. 00:10:16.550 --> 00:10:19.320 position:50% align:middle The magnetic voice said, "Do you need any help? 00:10:19.320 --> 00:10:20.620 position:50% align:middle I can help you with this." 00:10:20.620 --> 00:10:22.770 position:50% align:middle That is what they actually said on that phone. 00:10:22.770 --> 00:10:27.230 position:50% align:middle And it, in fact, caught that text and wrote it down for us. 00:10:27.230 --> 00:10:28.700 position:50% align:middle Very cool. 00:10:28.700 --> 00:10:31.880 position:50% align:middle So now you're at your next question. We're back on time, right? 00:10:31.880 --> 00:10:33.240 position:50% align:middle - Yeah, we are back. 00:10:33.240 --> 00:10:37.170 position:50% align:middle Isn't this great information now that you have slides that you can see? 00:10:37.170 --> 00:10:42.730 position:50% align:middle So, Phil, can you tell us more about the work done to protect the exam items? 00:10:42.730 --> 00:10:43.690 position:50% align:middle - Yeah. 00:10:43.690 --> 00:10:49.490 position:50% align:middle So what we're getting ready to do, and this is where I'm saying that we're ready 00:10:49.490 --> 00:10:50.960 position:50% align:middle to get operational. 00:10:50.960 --> 00:10:52.140 position:50% align:middle We know our AI can do it. 00:10:52.140 --> 00:10:55.120 position:50% align:middle We know we have a CAT engine that can deliver this. 00:10:55.120 --> 00:10:57.110 position:50% align:middle We know the AI is getting smart. 00:10:57.110 --> 00:11:00.730 position:50% align:middle It only gets smarter every day in a very controlled way. 00:11:00.730 --> 00:11:02.670 position:50% align:middle We control it. Right? 00:11:02.670 --> 00:11:08.250 position:50% align:middle That's supervised, unsupervised reinforcement happens all the time. 00:11:08.250 --> 00:11:16.120 position:50% align:middle But the next step is part of our very bylaws is that we should help reduce the workload of the 00:11:16.120 --> 00:11:17.250 position:50% align:middle boards of nursing. 00:11:17.250 --> 00:11:19.830 position:50% align:middle That's written into our bylaws. 00:11:19.830 --> 00:11:22.010 position:50% align:middle And one of the things we want to do is actually do that. 00:11:22.010 --> 00:11:27.790 position:50% align:middle So we are going to start using AI behind these consoles, what we call consoles. 00:11:27.790 --> 00:11:31.570 position:50% align:middle What is important is this item banking console. I'll show you it in a minute. 00:11:31.570 --> 00:11:34.640 position:50% align:middle We've got a CAT engine. We're working on an admin console. 00:11:34.640 --> 00:11:38.020 position:50% align:middle And NRB console, that's yours. 00:11:38.020 --> 00:11:45.130 position:50% align:middle That's yours where you can actually get data real time, and have the AI look at it for you. 00:11:45.130 --> 00:11:50.680 position:50% align:middle We've got a testing site console, a financial console, a candidate console, all in one system. 00:11:50.680 --> 00:11:54.380 position:50% align:middle [inaudible] that mean the AI can put all that data together for you, 00:11:54.380 --> 00:11:59.760 position:50% align:middle and give you data of which to analyze to make decision. 00:11:59.760 --> 00:12:01.920 position:50% align:middle So these are the four we're working on right now. 00:12:01.920 --> 00:12:06.870 position:50% align:middle Candidate, NRB, item banking, and CAT engine. 00:12:06.870 --> 00:12:09.890 position:50% align:middle I just took some screenshots, or we created some screenshots. 00:12:09.890 --> 00:12:10.610 position:50% align:middle This is really cool. 00:12:10.610 --> 00:12:17.470 position:50% align:middle I know there is probably no testing vendor out there in the world right now today that actually has an AI that 00:12:17.470 --> 00:12:19.320 position:50% align:middle can do what we call bank health. 00:12:19.320 --> 00:12:28.560 position:50% align:middle So our AI can tell us where in our...where we're building items, pool production, pool prediction, 00:12:28.560 --> 00:12:31.810 position:50% align:middle where we need more items, the content we need. 00:12:31.810 --> 00:12:33.150 position:50% align:middle And that means what? 00:12:33.150 --> 00:12:39.450 position:50% align:middle We can bring people in in a very defined way to write items for our bank, instead of saying, "Well, 00:12:39.450 --> 00:12:44.490 position:50% align:middle bring in 10 nurses, and hopefully we'll overproduce items," which, by the way, 00:12:44.490 --> 00:12:48.390 position:50% align:middle I appreciate that we did that because we have the biggest bank in the entire world, 00:12:48.390 --> 00:12:51.090 position:50% align:middle but that's not necessarily sustainable. 00:12:51.090 --> 00:12:55.017 position:50% align:middle So this creates a way for AI to do this for us. 00:12:55.150 --> 00:12:59.970 position:50% align:middle The other thing is your regulatory body, this is just a snapshot of what it can do, 00:12:59.970 --> 00:13:01.350 position:50% align:middle not what it will do. 00:13:01.350 --> 00:13:11.930 position:50% align:middle But my point is we can make this and it's...it will be customizable at the state level for what you need 00:13:11.930 --> 00:13:13.140 position:50% align:middle it to do. 00:13:13.140 --> 00:13:14.850 position:50% align:middle So we will be able to provide you what you want. 00:13:14.850 --> 00:13:18.680 position:50% align:middle But if you need to create a different workflow because all of you have different licensing workflows, 00:13:18.680 --> 00:13:22.700 position:50% align:middle different task workflows, we're going to be able to do that. 00:13:22.700 --> 00:13:27.980 position:50% align:middle While we provide you, like I said, the nice things at the top that are sort 00:13:27.980 --> 00:13:31.550 position:50% align:middle of dashboard-y, you can get behind those, see deeper. 00:13:31.550 --> 00:13:32.850 position:50% align:middle That's where we're going with that. 00:13:32.850 --> 00:13:35.010 position:50% align:middle And the AI is helping us with that. 00:13:35.010 --> 00:13:36.340 position:50% align:middle Candidate console. 00:13:36.340 --> 00:13:39.810 position:50% align:middle That's also kind of tied into your system, right? 00:13:39.810 --> 00:13:43.720 position:50% align:middle They're different consoles, but these are all laced together in a single system 00:13:43.720 --> 00:13:50.244 position:50% align:middle with AI security, with the content, psychometric security added to it. 00:13:52.170 --> 00:13:59.970 position:50% align:middle I love this one because I know nobody today, without using virtual machines, that can actually launch a CAT 00:13:59.970 --> 00:14:03.450 position:50% align:middle exam without what I call latency problems. 00:14:03.450 --> 00:14:08.150 position:50% align:middle Remember... It's Loretta [SP], right? Yeah. 00:14:08.150 --> 00:14:12.620 position:50% align:middle So, Loretta, if you were sitting in an exam and had to wait five seconds for the next item to come up, 00:14:12.620 --> 00:14:15.110 position:50% align:middle how happy would that make you? 00:14:15.110 --> 00:14:16.087 position:50% align:middle - [Loretta] I'd want it faster. 00:14:16.087 --> 00:14:17.550 position:50% align:middle - You want it faster, right? 00:14:17.550 --> 00:14:19.170 position:50% align:middle And so we call latency that. 00:14:19.170 --> 00:14:23.310 position:50% align:middle So we need a latency to be less than a blink of an eye. 00:14:23.310 --> 00:14:25.600 position:50% align:middle And we've got to be able to do that over the internet. 00:14:25.600 --> 00:14:28.920 position:50% align:middle So we're very close. We can do that. 00:14:28.920 --> 00:14:34.500 position:50% align:middle I don't...no other group that can finally do that and not lose security or any of the things that we've tried 00:14:34.500 --> 00:14:35.250 position:50% align:middle to build into this. 00:14:35.250 --> 00:14:38.700 position:50% align:middle So it's really exciting what we've been able to do. 00:14:38.700 --> 00:14:44.990 position:50% align:middle I will tell you, we are partnering with a group called ExamRoom.AI. 00:14:44.990 --> 00:14:53.560 position:50% align:middle ExamRoom.AI, with our guidance, has hired some of the brightest, 00:14:53.560 --> 00:14:56.840 position:50% align:middle smartest people I've ever met in my life. 00:14:56.840 --> 00:15:00.000 position:50% align:middle That means... By the way, how many have seen "Big Bang?" 00:15:01.260 --> 00:15:08.870 position:50% align:middle Yeah. That's a room full of Sheldons. 00:15:08.870 --> 00:15:13.920 position:50% align:middle All right? I just need you to know it. 00:15:13.920 --> 00:15:16.420 position:50% align:middle But they are so bright. 00:15:16.420 --> 00:15:19.210 position:50% align:middle They've came...we've got them from Google, we've got them from Amazon, 00:15:19.210 --> 00:15:22.260 position:50% align:middle we've got them from the Canadian Stock Exchange. 00:15:22.260 --> 00:15:28.020 position:50% align:middle We've got them from individuals who've solved the UPS logistics problems. 00:15:28.020 --> 00:15:30.090 position:50% align:middle All over the world, we've been able to do it. 00:15:30.090 --> 00:15:34.380 position:50% align:middle That's why we're being able to move fast, because this is their only role. 00:15:34.380 --> 00:15:35.980 position:50% align:middle This is what they're doing. 00:15:35.980 --> 00:15:42.110 position:50% align:middle And they are totally focused on making sure that we're able to do this successfully as a council. 00:15:42.110 --> 00:15:47.680 position:50% align:middle And by the way, those that went to the focus group, remember, when I say council, it's not the staff. 00:15:47.680 --> 00:15:52.090 position:50% align:middle The council is we. It's not us and them, it's we. 00:15:52.090 --> 00:15:55.290 position:50% align:middle This group is working for you to ensure this happens. 00:15:55.290 --> 00:15:58.510 position:50% align:middle The other great thing that I told them is it has to be cloud agnostic. 00:15:58.510 --> 00:16:02.650 position:50% align:middle So who knows what's going to happen to the cloud out there, right? 00:16:02.650 --> 00:16:05.520 position:50% align:middle So we need to build it so it didn't matter. 00:16:05.520 --> 00:16:08.810 position:50% align:middle I don't care who's the frontrunner, don't care who's doing anything. 00:16:08.810 --> 00:16:10.540 position:50% align:middle We're building this cloud agnostic. 00:16:10.540 --> 00:16:19.810 position:50% align:middle And that is a huge solution set for us, too, as well as the idea at the end of this is also to build 00:16:19.810 --> 00:16:25.260 position:50% align:middle it with our own cloud if we need to, so that if you're ever de-platformed anywhere, 00:16:25.260 --> 00:16:29.740 position:50% align:middle the council can go on, you won't be able to buy your stuff on Amazon, 00:16:29.740 --> 00:16:33.340 position:50% align:middle but we'll be able to give you a test. 00:16:33.340 --> 00:16:35.680 position:50% align:middle And if you ever expand, we can become Amazon. 00:16:35.680 --> 00:16:37.810 position:50% align:middle All right. Just saying. 00:16:37.810 --> 00:16:39.560 position:50% align:middle I don't really want us to do that. It's a joke. 00:16:39.560 --> 00:16:43.740 position:50% align:middle But the point is, if they ever de-platformed you, wouldn't matter because we can run it ourselves. 00:16:43.740 --> 00:16:45.890 position:50% align:middle So that's the long-term goal. 00:16:45.890 --> 00:16:54.590 position:50% align:middle We're looking to make sure this is real to you starting somewhere post January 2025, 00:16:54.590 --> 00:16:57.380 position:50% align:middle somewhere in that year is what we're...that is our hope. 00:16:57.380 --> 00:16:58.360 position:50% align:middle That is our goal. 00:16:58.360 --> 00:17:01.500 position:50% align:middle There is no reason that I believe we will not meet that. 00:17:01.500 --> 00:17:03.695 position:50% align:middle With that, I'm going to turn it back over to you. 00:17:03.695 --> 00:17:06.250 position:50% align:middle - All right. Great. Thank you, Phil, for the update. 00:17:06.250 --> 00:17:13.290 position:50% align:middle And since we have a few minutes left, does anyone have questions for Phil regarding the 00:17:13.290 --> 00:17:15.890 position:50% align:middle remote proctoring, AI? 00:17:20.700 --> 00:17:22.000 position:50% align:middle - [Lory] Hey, Phil. - Hey. 00:17:22.000 --> 00:17:24.070 position:50% align:middle - So, Lori from California. 00:17:24.070 --> 00:17:31.430 position:50% align:middle Not necessarily remote proctoring for AI but the step right before that where we get the applicant 00:17:31.430 --> 00:17:32.700 position:50% align:middle at our boards. 00:17:32.700 --> 00:17:37.610 position:50% align:middle Can AI be used for all the transcripts and all the documents received so that we no longer have fraud, 00:17:37.610 --> 00:17:39.980 position:50% align:middle so that when we send them to there, they're qualified? 00:17:40.702 --> 00:17:43.230 position:50% align:middle - So the answer is yes. 00:17:43.230 --> 00:17:44.680 position:50% align:middle Interesting, we're working on that. 00:17:44.680 --> 00:17:50.440 position:50% align:middle The idea is that we can actually have a bot that can evaluate curriculum. 00:17:50.440 --> 00:17:55.110 position:50% align:middle We give it...it can evaluate certificates, all of that. 00:17:55.110 --> 00:18:05.200 position:50% align:middle So yes, the answer is that our intent is to have that in that nursing regulatory board console. 00:18:05.200 --> 00:18:11.000 position:50% align:middle Where you're getting at is what we're calling...it's one of the initiatives of the council, 00:18:11.000 --> 00:18:13.760 position:50% align:middle which is called licensure process reform. 00:18:13.760 --> 00:18:18.860 position:50% align:middle So I want to be clear, do not call it licensure reform process. 00:18:18.860 --> 00:18:24.429 position:50% align:middle It's licensure process reform, and it's building those efficiencies for you. 00:18:26.100 --> 00:18:27.520 position:50% align:middle Yes. 00:18:27.520 --> 00:18:31.190 position:50% align:middle - [Theresa] Theresa Delahoyde, President of the Nebraska State Board of Nursing. 00:18:31.190 --> 00:18:35.270 position:50% align:middle And my full-time job is dean of an undergraduate nursing program, BS program. 00:18:35.270 --> 00:18:36.790 position:50% align:middle So this is very near and dear to my heart. 00:18:36.790 --> 00:18:43.350 position:50% align:middle Can you explain just a little bit more about when you were talking about no longer having 10 nurses coming 00:18:43.350 --> 00:18:46.570 position:50% align:middle in to write questions, and how you're going to do this a little bit more 00:18:46.570 --> 00:18:48.506 position:50% align:middle efficiently and with AI? 00:18:48.506 --> 00:18:49.210 position:50% align:middle - I can. 00:18:49.210 --> 00:18:53.940 position:50% align:middle I'm going to step back up here so everybody can see me, but I will look at you. 00:18:53.940 --> 00:18:56.350 position:50% align:middle All right? 00:18:56.350 --> 00:18:57.610 position:50% align:middle There's a couple of things we're going to do. 00:18:57.610 --> 00:19:01.560 position:50% align:middle One, it doesn't mean that we'll get rid of nurses, but we're working on two things. 00:19:01.560 --> 00:19:03.550 position:50% align:middle So I always think this needs to be a hybrid. 00:19:03.550 --> 00:19:07.100 position:50% align:middle Maybe someday, 10 years in the future, it changes. All right? 00:19:07.100 --> 00:19:11.020 position:50% align:middle But right now, to me, everything has to be thought in a hybrid way. 00:19:11.020 --> 00:19:17.160 position:50% align:middle And so where I'm getting at is what we'll be able to do is say we need to write on OB, 00:19:17.160 --> 00:19:22.500 position:50% align:middle and it needs to be focused on these three tasks in OB. 00:19:22.500 --> 00:19:26.670 position:50% align:middle We can actually go out and recruit those individuals to come to Chicago, and write for us the way 00:19:26.670 --> 00:19:28.080 position:50% align:middle they do today. 00:19:28.080 --> 00:19:33.150 position:50% align:middle But in the background, we're working on automated item generation 00:19:33.150 --> 00:19:39.720 position:50% align:middle so that our AI can actually develop items which then, instead of bringing you in to write, 00:19:39.720 --> 00:19:43.390 position:50% align:middle I bring you in to review and edit. 00:19:43.390 --> 00:19:48.330 position:50% align:middle And we could generate thousands upon thousands upon thousands, as long as you'll come and help 00:19:48.330 --> 00:19:50.130 position:50% align:middle us review them. Right? 00:19:50.130 --> 00:19:53.440 position:50% align:middle And then they still have to be put in an experimental pool, and do all the things. 00:19:53.440 --> 00:19:57.690 position:50% align:middle But this will be huge for item development. 00:19:57.690 --> 00:20:01.930 position:50% align:middle It will change the whole world in terms of test development if we're successful. 00:20:01.930 --> 00:20:02.510 position:50% align:middle - Thank you. 00:20:02.510 --> 00:20:05.260 position:50% align:middle - Yeah. Yes? 00:20:05.260 --> 00:20:13.130 position:50% align:middle - So for item validation, are you going to partner that with the NCSBN individual 00:20:13.130 --> 00:20:20.300 position:50% align:middle number when you're tracking workforce and experience, and target the people with the best experience to be 00:20:20.300 --> 00:20:21.330 position:50% align:middle able to validate that? 00:20:21.330 --> 00:20:24.470 position:50% align:middle - So the reality is that's already done. 00:20:24.470 --> 00:20:27.140 position:50% align:middle If I pulled your record, right? 00:20:27.140 --> 00:20:30.000 position:50% align:middle I'd be able to go back and go, because of your ID number, 00:20:30.000 --> 00:20:34.360 position:50% align:middle I know exactly what items you took, and how you respond to those items. 00:20:34.360 --> 00:20:36.980 position:50% align:middle I know that. 00:20:36.980 --> 00:20:39.330 position:50% align:middle We may become more efficient at that. 00:20:39.330 --> 00:20:41.630 position:50% align:middle It may be... I mean, I get where you're going. 00:20:41.630 --> 00:20:48.480 position:50% align:middle I think it's a great question, but we can do that now, and we may be able to do it more efficiently. 00:20:48.480 --> 00:20:51.170 position:50% align:middle - I might have asked the question wrong. 00:20:51.170 --> 00:21:00.000 position:50% align:middle The people that validate the questions that your AI is right, how do you choose those people? 00:21:00.000 --> 00:21:01.360 position:50% align:middle And will you... - You help us. 00:21:01.360 --> 00:21:03.490 position:50% align:middle - Well, I know we help you right now. 00:21:03.490 --> 00:21:09.130 position:50% align:middle But if you're using the NCSBN identifier number, and you're tracking their workforce, 00:21:09.130 --> 00:21:15.250 position:50% align:middle and you're tracking their competency and their ability, would you then partner and utilize that information 00:21:15.250 --> 00:21:18.000 position:50% align:middle as well to target people that are most competent? 00:21:18.000 --> 00:21:22.230 position:50% align:middle - Well, I think we could do that, but I don't think we'd ever step away from working 00:21:22.230 --> 00:21:28.670 position:50% align:middle with the board to continue to validate because there's more than just that that I wouldn't be able to check. 00:21:28.670 --> 00:21:32.060 position:50% align:middle So I think it adds credibility to it. 00:21:32.060 --> 00:21:33.120 position:50% align:middle - [Male] [inaudible] 00:21:33.120 --> 00:21:36.120 position:50% align:middle - Yes. Right, right. 00:21:36.120 --> 00:21:38.630 position:50% align:middle So, yes, the idea... I get where you're going. 00:21:38.630 --> 00:21:44.250 position:50% align:middle We can say our numbers say this is what we know about this person. 00:21:44.250 --> 00:21:49.830 position:50% align:middle I would still think, at this point, maybe we become comfortable at a while, we'd say, 00:21:49.830 --> 00:21:55.490 position:50% align:middle "Loretta, this is what we...we want this person, this is the credentials, this is their experiences, 00:21:55.490 --> 00:22:00.320 position:50% align:middle everything we have. Will you approve them to be part of that?" 00:22:00.320 --> 00:22:05.880 position:50% align:middle That's what I still think needs to happen at this point, because it's not just us. 00:22:05.880 --> 00:22:08.240 position:50% align:middle I don't think, at this point, a regulator should step that far away 00:22:08.240 --> 00:22:09.360 position:50% align:middle from what we're doing. 00:22:09.360 --> 00:22:12.384 position:50% align:middle You should be involved in it. Right? 00:22:18.070 --> 00:22:21.920 position:50% align:middle No other questions? Really? 00:22:21.920 --> 00:22:23.071 position:50% align:middle Come on. 00:22:25.570 --> 00:22:28.240 position:50% align:middle All right. Crystal, I turn it back over to you. 00:22:28.240 --> 00:22:29.970 position:50% align:middle - All right. Thank you, Phil. 00:22:29.970 --> 00:22:33.660 position:50% align:middle And thank you for the presentation, and the questions. 00:22:38.500 --> 00:22:42.559 position:50% align:middle And I'll turn it over to David, and hope you all have a wonderful night.