WEBVTT 00:00:00.590 --> 00:00:04.345 position:50% align:middle - [Woman] Brendan Martin is the Director of Research for NCSBN. 00:00:04.345 --> 00:00:09.090 position:50% align:middle He has more than 13 years in quantitative modeling and consulting. 00:00:09.090 --> 00:00:14.950 position:50% align:middle Brendan has extensive graduate-level statistical training in the fields of mathematics and 00:00:14.950 --> 00:00:17.190 position:50% align:middle public health sciences. 00:00:17.190 --> 00:00:25.663 position:50% align:middle His research interests include post-secondary access, biostatistics, healthcare reform, and regulation. 00:00:31.090 --> 00:00:32.170 position:50% align:middle - [Brendan] Hello. 00:00:32.170 --> 00:00:36.330 position:50% align:middle My name is Brendan Martin, and I am the director of NCSBN's research department. 00:00:36.330 --> 00:00:41.550 position:50% align:middle I'm here today to discuss the results of a recently completed study examining the operational efficiency 00:00:41.550 --> 00:00:46.420 position:50% align:middle and effectiveness of nursing regulatory bodies as it relates to nurse discipline. 00:00:46.420 --> 00:00:50.140 position:50% align:middle For today's presentation, we are going to cover a few major points. 00:00:50.140 --> 00:00:55.410 position:50% align:middle To start, I'll provide a bit of background on the study to give you all the necessary context for why we wanted 00:00:55.410 --> 00:00:59.060 position:50% align:middle to pursue this study in the first place and what we hope to achieve. 00:00:59.060 --> 00:01:03.890 position:50% align:middle I'll then share a brief overview of the study methodology, so that we are clear on the study sample, 00:01:03.890 --> 00:01:08.240 position:50% align:middle how we went about collecting the data, and how we analyzed the responses. 00:01:08.240 --> 00:01:12.390 position:50% align:middle Then we'll get into the meat of the presentation, in which I will cover the results in detail 00:01:12.390 --> 00:01:15.110 position:50% align:middle before wrapping things up with a few key takeaways. 00:01:15.110 --> 00:01:21.150 position:50% align:middle As always, I'll attempt to leave ample time at the end for any follow-up questions or necessary clarification. 00:01:21.150 --> 00:01:26.640 position:50% align:middle So, please feel free to use the chat box to submit your comments as I go through the material. 00:01:26.640 --> 00:01:31.600 position:50% align:middle Operational efficiency and effectiveness in promoting public safety are key performance indicators 00:01:31.600 --> 00:01:34.260 position:50% align:middle for nursing regulatory bodies worldwide. 00:01:34.260 --> 00:01:39.620 position:50% align:middle Nonetheless, these same regulatory bodies charged with ensuring public protection are often understaffed 00:01:39.620 --> 00:01:41.510 position:50% align:middle and poorly resourced. 00:01:41.510 --> 00:01:47.160 position:50% align:middle Aging patient populations, workforce shortages, and an increasingly mobile professional class require 00:01:47.160 --> 00:01:52.540 position:50% align:middle agile regulatory systems that facilitate the efficient disciplinary process and, when appropriate, 00:01:52.540 --> 00:01:54.980 position:50% align:middle safe return to practice. 00:01:54.980 --> 00:02:00.680 position:50% align:middle In the U.S., differences in staffing, operations, terminology, and other critical measures have 00:02:00.680 --> 00:02:04.940 position:50% align:middle historically made it difficult for nursing regulatory bodies to create a standardized method 00:02:04.940 --> 00:02:07.620 position:50% align:middle for objectively evaluating performance. 00:02:07.620 --> 00:02:12.740 position:50% align:middle This is particularly evident in how discipline cases brought against nurses are managed. 00:02:12.740 --> 00:02:18.450 position:50% align:middle This pilot study aimed to examine the steps involved in the disciplinary process to identify the most efficient 00:02:18.450 --> 00:02:25.740 position:50% align:middle and effective models for case management, and thereby, develop an evidence-based, uniform discipline process. 00:02:25.740 --> 00:02:29.900 position:50% align:middle Regarding the methodology, the study utilized a longitudinal survey design. 00:02:29.900 --> 00:02:33.410 position:50% align:middle NCSBN partnered with investigative staff at 10 U.S. 00:02:33.410 --> 00:02:38.730 position:50% align:middle nursing regulatory bodies to enter detailed, step-by-step information across five disciplinary case 00:02:38.730 --> 00:02:43.530 position:50% align:middle categories into an online data repository between June 2018 and June 2020. 00:02:43.530 --> 00:02:50.580 position:50% align:middle The five case categories tracked for the analysis were professional conduct, impairment or diversion, 00:02:50.580 --> 00:02:54.200 position:50% align:middle practice error, criminal, and a random category. 00:02:54.200 --> 00:02:59.830 position:50% align:middle The random category was a free choice option for which participants could enter information for any new case 00:02:59.830 --> 00:03:03.510 position:50% align:middle that fell into one of the four fixed case categories. 00:03:03.510 --> 00:03:08.290 position:50% align:middle To minimize selection bias, all participating boards of nursing were asked to track 00:03:08.290 --> 00:03:12.570 position:50% align:middle case details only for complaints that were logged in the two-year window. 00:03:12.570 --> 00:03:17.570 position:50% align:middle Further, participants were instructed to initially select cases that aligned with one of the four case 00:03:17.570 --> 00:03:23.080 position:50% align:middle categories as they came in, for example, the first instance of a professional conduct complaint 00:03:23.080 --> 00:03:25.218 position:50% align:middle after June 2018. 00:03:25.218 --> 00:03:29.870 position:50% align:middle The fifth or random case was then afforded more flexibility as it could align with any form 00:03:29.870 --> 00:03:32.530 position:50% align:middle of the aforementioned cases. 00:03:32.530 --> 00:03:38.030 position:50% align:middle Detailed instructions on what data should be tracked and definitions for key terms were provided to all 00:03:38.030 --> 00:03:43.840 position:50% align:middle participating boards through a series of regularly scheduled training webinars prior to study launch. 00:03:43.840 --> 00:03:48.470 position:50% align:middle Regular contact was maintained throughout the two-year period to ensure that questions that came up were 00:03:48.470 --> 00:03:52.820 position:50% align:middle resolved in an efficient manner as possible. 00:03:52.820 --> 00:03:58.960 position:50% align:middle The 10 states that participated in the study were Oregon, Georgia, Minnesota, Florida, Idaho, 00:03:58.960 --> 00:04:03.450 position:50% align:middle North Dakota, Ohio, Texas, Wyoming, and New Mexico. 00:04:03.450 --> 00:04:10.010 position:50% align:middle This sample allowed us to capture not only geographical but also operational diversity within our sample. 00:04:10.010 --> 00:04:14.900 position:50% align:middle We used Microsoft's Forms functionality within SharePoint to collect a baseline board and case 00:04:14.900 --> 00:04:18.190 position:50% align:middle information as well as detailed case records. 00:04:18.190 --> 00:04:22.330 position:50% align:middle Detailed case records included the number of steps involved in resolving a complaint, 00:04:22.330 --> 00:04:27.110 position:50% align:middle the associated dates of those steps, broader descriptions as to the nature of the steps 00:04:27.110 --> 00:04:30.180 position:50% align:middle taken in the case, and detailed narratives regarding all activities. 00:04:30.180 --> 00:04:36.990 position:50% align:middle For the analysis, we used generalized estimating equation models to assess case resolution and time 00:04:36.990 --> 00:04:38.500 position:50% align:middle to case resolution. 00:04:38.500 --> 00:04:42.990 position:50% align:middle This approach appropriately accounted for our clustered data collection process. 00:04:42.990 --> 00:04:47.540 position:50% align:middle In other words, because we were investigating operational efficiency within select nursing 00:04:47.540 --> 00:04:52.050 position:50% align:middle regulatory bodies, we wanted to ensure that any internal consistency, 00:04:52.050 --> 00:04:56.360 position:50% align:middle meaning the likelihood that an efficient board was consistently efficient across cases, 00:04:56.360 --> 00:04:58.850 position:50% align:middle or vice-versa, was captured. 00:04:58.850 --> 00:05:04.500 position:50% align:middle This approach also allowed for additional flexibility to assess a binary outcome and adjust for other 00:05:04.500 --> 00:05:07.290 position:50% align:middle important covariates as necessary. 00:05:07.290 --> 00:05:12.310 position:50% align:middle The primary dependent variable for this study was operational efficiency measured as number of days 00:05:12.310 --> 00:05:14.750 position:50% align:middle required to resolve a complaint. 00:05:14.750 --> 00:05:19.710 position:50% align:middle We also explored case resolution in general as a supplemental outcome. 00:05:19.710 --> 00:05:24.860 position:50% align:middle We then extended the analysis to incorporate participants' open-ended narrative responses using 00:05:24.860 --> 00:05:26.960 position:50% align:middle natural language processing. 00:05:26.960 --> 00:05:32.790 position:50% align:middle You'll see shortly that we kept the presentation of these results fairly high-level for today's discussion. 00:05:32.790 --> 00:05:38.440 position:50% align:middle But the overall goal was to objectively align response trends, meaning word frequency and choice, 00:05:38.440 --> 00:05:40.260 position:50% align:middle with our primary outcome. 00:05:40.260 --> 00:05:43.030 position:50% align:middle More on this in a few slides. 00:05:43.030 --> 00:05:46.790 position:50% align:middle As I mentioned earlier, 10 states participated in the study. 00:05:46.790 --> 00:05:52.570 position:50% align:middle While participants were only asked to provide data on up to five cases, some voluntarily exceeded this total, 00:05:52.570 --> 00:05:57.030 position:50% align:middle such as Minnesota, Georgia, and Oregon, while one fell just short. 00:05:57.030 --> 00:06:01.180 position:50% align:middle Across the four case categories, professional conduct was the most common, 00:06:01.180 --> 00:06:05.410 position:50% align:middle followed by impairment or diversion, practice error, and criminal. 00:06:05.410 --> 00:06:10.730 position:50% align:middle Nearly three-quarters of all cases included in the two-year review period were resolved. 00:06:10.730 --> 00:06:16.380 position:50% align:middle The median open case load across boards was approximately 500, but the distribution varied greatly, 00:06:16.380 --> 00:06:19.140 position:50% align:middle as is evident from the interquartile range. 00:06:19.140 --> 00:06:25.150 position:50% align:middle The 25th percentile was 125, and the 75th percentile was 787. 00:06:25.150 --> 00:06:29.380 position:50% align:middle This underscored the range of complaint volume across participating boards. 00:06:29.380 --> 00:06:33.510 position:50% align:middle Similarly, the median investigator count was three. 00:06:33.510 --> 00:06:37.360 position:50% align:middle But some boards reported only one dedicated staff was assigned to complaints, 00:06:37.360 --> 00:06:43.550 position:50% align:middle or even just a proportion of one staff's time was allotted, while other larger boards reported over 30 00:06:43.550 --> 00:06:45.570 position:50% align:middle investigators on staff. 00:06:45.570 --> 00:06:48.930 position:50% align:middle To account for this variability, we also calculated a case load 00:06:48.930 --> 00:06:51.070 position:50% align:middle to investigator ratio variable. 00:06:51.070 --> 00:06:57.230 position:50% align:middle The median number of cases per investigator was 60, with an interquartile range of 29 to 131, 00:06:57.230 --> 00:06:59.830 position:50% align:middle affording us a bit more precision. 00:06:59.830 --> 00:07:06.220 position:50% align:middle The median number of steps involved in each case was 10, and it took approximately 177 days for each 00:07:06.220 --> 00:07:08.790 position:50% align:middle complaint to be resolved. 00:07:08.790 --> 00:07:12.650 position:50% align:middle Regarding the demographic characteristics of the nurses involved in the complaints, 00:07:12.650 --> 00:07:18.480 position:50% align:middle the average age was approximately 43 years old, and a majority were female. 00:07:18.480 --> 00:07:22.390 position:50% align:middle As previously noted, generalized estimating equation logistic regression 00:07:22.390 --> 00:07:25.040 position:50% align:middle models were employed for the analysis. 00:07:25.040 --> 00:07:29.910 position:50% align:middle Initially, we explored the independent associations between board, case, and nurse characteristics, 00:07:29.910 --> 00:07:33.550 position:50% align:middle and the odds of the case being resolved. 00:07:33.550 --> 00:07:38.950 position:50% align:middle As you can see from the detailed results on this slide, only two variables emerged as marginal drivers 00:07:38.950 --> 00:07:40.700 position:50% align:middle of case resolution. 00:07:40.700 --> 00:07:45.130 position:50% align:middle Those were open case load and number of active investigators. 00:07:45.130 --> 00:07:50.490 position:50% align:middle Overall, for every 100 additional cases, boards were about 10% more likely to resolve 00:07:50.490 --> 00:07:56.380 position:50% align:middle a complaint, perhaps suggesting efficiencies gained through familiarity with certain case types. 00:07:56.380 --> 00:08:02.090 position:50% align:middle Less surprising, we also observed that boards were roughly 13% more likely to resolve a complaint for each 00:08:02.090 --> 00:08:04.690 position:50% align:middle additional active investigator. 00:08:04.690 --> 00:08:09.510 position:50% align:middle Nonetheless, while interesting trends, we did not document any significant correlations 00:08:09.510 --> 00:08:15.970 position:50% align:middle between board, case, and nurse characteristics, and the odds of the case being resolved. 00:08:15.970 --> 00:08:21.180 position:50% align:middle We then proceeded to explore our primary dependent variable of operational efficiency. 00:08:21.180 --> 00:08:26.210 position:50% align:middle This measure was defined as the number of days required to resolve a complaint. 00:08:26.210 --> 00:08:31.320 position:50% align:middle To simplify the modeling process and to objectively delineate between efficient and inefficient processes, 00:08:31.320 --> 00:08:36.990 position:50% align:middle we settled on a binary cut point aligned with the median number of days to resolve a complaint, 00:08:36.990 --> 00:08:40.410 position:50% align:middle which was 177 days. 00:08:40.410 --> 00:08:43.980 position:50% align:middle As we were primarily interested in barriers to efficient case resolution, 00:08:43.980 --> 00:08:49.180 position:50% align:middle complaints that took more than 177 days to resolve were of primary interest. 00:08:49.180 --> 00:08:54.310 position:50% align:middle Thus, these models highlighted the possible determinants of inefficient case resolution or the 00:08:54.310 --> 00:08:59.980 position:50% align:middle likelihood that a case would take more than 177 days to resolve given a particular board, case, 00:08:59.980 --> 00:09:03.030 position:50% align:middle or nurse characteristic. 00:09:03.030 --> 00:09:07.330 position:50% align:middle As before, the independent associations between board, case, and nurse characteristics, 00:09:07.330 --> 00:09:12.780 position:50% align:middle and the odds of inefficient case resolution were initially the focus of our analysis. 00:09:12.780 --> 00:09:18.650 position:50% align:middle Unlike our case resolution outcome, however, several notable trends emerged in our review. 00:09:18.650 --> 00:09:21.840 position:50% align:middle Overall, case volume contributed to backlog. 00:09:21.840 --> 00:09:26.150 position:50% align:middle For every 100 additional cases, a complaint was about 10% more likely 00:09:26.150 --> 00:09:28.540 position:50% align:middle to be inefficiently resolved. 00:09:28.540 --> 00:09:34.750 position:50% align:middle To be more specific, in terms of operations, for every 10 additional cases per investigator, 00:09:34.750 --> 00:09:40.000 position:50% align:middle boards documented an 8% increase in inefficient case resolution. 00:09:40.000 --> 00:09:45.080 position:50% align:middle In addition, the number of steps involved in a case were reviewed as a possible indicator of the complexity 00:09:45.080 --> 00:09:46.720 position:50% align:middle of that case. 00:09:46.720 --> 00:09:52.250 position:50% align:middle Like case volume and the case-to-investigator ratio metric, for every additional step required in a 00:09:52.250 --> 00:09:59.840 position:50% align:middle case investigation, a board documented a 9% increase in the likelihood of inefficient case resolution. 00:09:59.840 --> 00:10:03.800 position:50% align:middle For those of you who prefer figures over a table, we also created a figure 00:10:03.800 --> 00:10:06.330 position:50% align:middle to illustrate these associations. 00:10:06.330 --> 00:10:10.910 position:50% align:middle What you see before you is a forest plot of the odds ratios that highlights the three significant 00:10:10.910 --> 00:10:15.560 position:50% align:middle associations we just discussed, as well as the marginal alignment between inefficient 00:10:15.560 --> 00:10:20.400 position:50% align:middle case resolution and the number of active investigators. 00:10:20.400 --> 00:10:25.430 position:50% align:middle Having documented the independent associations between board, case, and nurse characteristics, 00:10:25.430 --> 00:10:31.180 position:50% align:middle and the odds of inefficient case resolution, we shifted our focus to multivariable analysis, 00:10:31.180 --> 00:10:37.200 position:50% align:middle meaning we further explored the significant univariable trends, adjusting for other important covariates. 00:10:37.200 --> 00:10:40.820 position:50% align:middle In this first table, we further controlled for case category to better 00:10:40.820 --> 00:10:46.920 position:50% align:middle understand if some of these barriers to efficient case resolution related to the underlying case type. 00:10:46.920 --> 00:10:53.030 position:50% align:middle For instance, we wanted to learn if criminal complaints or cases of professional misconduct, as examples, 00:10:53.030 --> 00:10:56.430 position:50% align:middle might exacerbate or mitigate some of these patterns. 00:10:56.430 --> 00:10:58.460 position:50% align:middle Clearly, they did not. 00:10:58.460 --> 00:11:04.240 position:50% align:middle So, in the second table, we ran a multivariable model, including both the case-to-investigator ratio and the 00:11:04.240 --> 00:11:06.740 position:50% align:middle number of steps involved in a case. 00:11:06.740 --> 00:11:13.360 position:50% align:middle Importantly, both criteria retained at least a marginal alignment with the efficient processing of complaints. 00:11:13.360 --> 00:11:18.650 position:50% align:middle This highlights the critical importance of each investigator's case load at any given time and the 00:11:18.650 --> 00:11:22.600 position:50% align:middle complexity of those cases to operational efficiency. 00:11:22.600 --> 00:11:28.120 position:50% align:middle To further explore these two predictors, we generated receiver operating characteristic or ROC 00:11:28.120 --> 00:11:32.700 position:50% align:middle curves to identify specific cut points at which boards could expect to see a drop-off 00:11:32.700 --> 00:11:35.160 position:50% align:middle in operational efficiency. 00:11:35.160 --> 00:11:40.090 position:50% align:middle Beginning with our case-to-investigator metric, we identified a meaningful cut point at the 35th 00:11:40.090 --> 00:11:42.330 position:50% align:middle observation in the sample. 00:11:42.330 --> 00:11:45.740 position:50% align:middle Specifically, this aligned with the case load per investigator of 38. 00:11:45.740 --> 00:11:51.950 position:50% align:middle As you can see, the area under the curve, or AUC, which is a measure of model fit, 00:11:51.950 --> 00:11:56.090 position:50% align:middle is good and just a bit below the 0.8 excellent threshold. 00:11:56.090 --> 00:12:02.070 position:50% align:middle This suggests an accurate and predictive reference point for our case-to-investigator ratio variable. 00:12:02.070 --> 00:12:09.370 position:50% align:middle Further, we noted a strong positive predictive value or PPV indicating investor case loads over 38 correctly 00:12:09.370 --> 00:12:15.140 position:50% align:middle identified 81% of the cases that would ultimately run over the median closure time. 00:12:15.140 --> 00:12:21.230 position:50% align:middle It also does a fairly good job of discriminating as it accurately identifies 69% of the cases that 00:12:21.230 --> 00:12:23.590 position:50% align:middle close more efficiently. 00:12:23.590 --> 00:12:28.950 position:50% align:middle We followed a similar strategy to further investigate the number of steps involved in the case management. 00:12:28.950 --> 00:12:34.230 position:50% align:middle We identified a meaningful cut point at the 43rd observation in the sample. 00:12:34.230 --> 00:12:38.250 position:50% align:middle Specifically, this aligned with a case step count of 11. 00:12:38.250 --> 00:12:45.680 position:50% align:middle As you see, the AUC is again strong, also just a bit below the 0.8 excellent threshold. 00:12:45.680 --> 00:12:51.310 position:50% align:middle This suggests an accurate and predictive reference point for our case step count variable as well. 00:12:51.310 --> 00:12:57.160 position:50% align:middle In this instance, we observed a slightly weaker PPV as the cut point of 11 case steps identifies about 68% 00:12:57.160 --> 00:13:01.070 position:50% align:middle of the cases that run over the median closure time. 00:13:01.070 --> 00:13:06.580 position:50% align:middle However, it does a somewhat better job discriminating than a ratio variable as it also accurately identifies 00:13:06.580 --> 00:13:11.210 position:50% align:middle 75% of the cases that close more efficiently. 00:13:11.210 --> 00:13:16.130 position:50% align:middle Using participants' own narrative accounts, we then utilized natural language processing to align 00:13:16.130 --> 00:13:23.180 position:50% align:middle response trends, meaning word frequency and choice, with our primary operational frequency outcome. 00:13:23.180 --> 00:13:27.250 position:50% align:middle While fairly high level, these initial results presented an interesting snapshot 00:13:27.250 --> 00:13:31.600 position:50% align:middle into investigators' experiences in managing these cases. 00:13:31.600 --> 00:13:35.610 position:50% align:middle For inefficient cases, administrative themes associated with subpoenas, 00:13:35.610 --> 00:13:41.320 position:50% align:middle notice letters, requests, calls, and emails drove participants' descriptions. 00:13:41.320 --> 00:13:46.010 position:50% align:middle While administrative steps dominated both groups, interestingly, the pronounced emphasis on the 00:13:46.010 --> 00:13:52.070 position:50% align:middle words send, call, and subpoena with inefficient cases perhaps aligns with our overall finding of case 00:13:52.070 --> 00:13:57.060 position:50% align:middle complexity measured as the number of case steps, suggesting built-in delays with more 00:13:57.060 --> 00:13:59.140 position:50% align:middle administratively burdensome complaints. 00:13:59.140 --> 00:14:05.040 position:50% align:middle By contrast, more cut-and-dried criminal complaints, in which compounding factors such as felony arrest or 00:14:05.040 --> 00:14:09.340 position:50% align:middle concealment are present, may typify more efficient cases. 00:14:09.340 --> 00:14:12.200 position:50% align:middle So, what are the key takeaways? 00:14:12.200 --> 00:14:16.910 position:50% align:middle Despite differences in staffing, operations, terminology, and other critical measures that have 00:14:16.910 --> 00:14:21.910 position:50% align:middle historically made it difficult for nursing regulatory bodies to create a standardized method for objectively 00:14:21.910 --> 00:14:26.070 position:50% align:middle evaluating performance, there are indicators of operational efficiency that 00:14:26.070 --> 00:14:28.920 position:50% align:middle transcend individual jurisdictions. 00:14:28.920 --> 00:14:34.620 position:50% align:middle To this end, monitoring investigator workload and case complexity is critically important to ensure the 00:14:34.620 --> 00:14:39.890 position:50% align:middle efficient management of complaints against nurses and, when possible, the quick and safe return 00:14:39.890 --> 00:14:42.050 position:50% align:middle of nurses to practice. 00:14:42.050 --> 00:14:46.600 position:50% align:middle Finally, as with all things, our shared goal of agile regulatory systems is best 00:14:46.600 --> 00:14:51.070 position:50% align:middle achieved and supported by standardized and systematic data collection. 00:14:51.070 --> 00:14:56.530 position:50% align:middle The important findings in this study were only made possible by the 10 boards of nursing that generously 00:14:56.530 --> 00:14:59.760 position:50% align:middle volunteered their time and rigorously tracked their activities. 00:14:59.760 --> 00:15:06.040 position:50% align:middle Moving forward, routinizing basic data collection standards can facilitate ongoing case monitoring and, 00:15:06.040 --> 00:15:10.150 position:50% align:middle thereby, active management of operational efficiency. 00:15:10.150 --> 00:15:14.090 position:50% align:middle With that, I will open the floor to discussion and any questions you might have. 00:15:35.970 --> 00:15:37.170 position:50% align:middle Hello, everyone. 00:15:37.170 --> 00:15:42.320 position:50% align:middle The floor is now open for any questions that you might have. 00:15:42.320 --> 00:15:50.230 position:50% align:middle As we wait for some of the questions to roll in, I did want to share a few updates since this 00:15:50.230 --> 00:15:52.890 position:50% align:middle presentation was prepared. 00:15:52.890 --> 00:15:58.980 position:50% align:middle The first is just to note that there will be an extension of the natural language processing analysis. 00:15:58.980 --> 00:16:05.480 position:50% align:middle So we are looking to delve a bit further into some of the free text or unstructured responses. 00:16:05.480 --> 00:16:11.670 position:50% align:middle And then, in addition to that, we have now done access to some other kind of metadata 00:16:11.670 --> 00:16:18.110 position:50% align:middle that we are going to overlay on the data that we collected ourselves directly from board participants. 00:16:18.110 --> 00:16:25.410 position:50% align:middle So our hope here is to augment the analysis somewhat, in particular, with board characteristics. 00:16:25.410 --> 00:16:32.530 position:50% align:middle So we'll be able to get a little bit more nuanced insight into the governance structure of the boards and 00:16:32.530 --> 00:16:37.250 position:50% align:middle build that information into the analysis as yet another data point that might inform on our 00:16:37.250 --> 00:16:38.790 position:50% align:middle outcome of interest. 00:16:38.790 --> 00:16:45.920 position:50% align:middle So, with that, I will wait to see all the questions that come in. 00:16:45.920 --> 00:16:49.280 position:50% align:middle I think we have our first question rolling in. 00:16:49.280 --> 00:16:52.420 position:50% align:middle Give me one second. 00:16:52.420 --> 00:17:00.134 position:50% align:middle So, in the definition... Here, let's make sure I can see the whole thing. 00:17:00.134 --> 00:17:06.760 position:50% align:middle "In your definition of complaint resolution, was it based on completion of the investigation? 00:17:06.760 --> 00:17:12.910 position:50% align:middle I ask because efficiency of the actual investigation process may not have any influence over the proceedings 00:17:12.910 --> 00:17:15.180 position:50% align:middle occurring after the investigation is complete." 00:17:15.180 --> 00:17:17.090 position:50% align:middle Yes, that's an excellent question. 00:17:17.090 --> 00:17:22.100 position:50% align:middle So, we did have two primary outcomes of interest. 00:17:22.100 --> 00:17:26.610 position:50% align:middle Obviously, our primary outcome of interest was that dependent variable associated 00:17:26.610 --> 00:17:28.360 position:50% align:middle with efficient case resolution. 00:17:28.360 --> 00:17:32.930 position:50% align:middle But in terms of what we designated as the kind of formal close of the case, 00:17:32.930 --> 00:17:39.880 position:50% align:middle that was what investigators reported to us as the final step associated with formally resolving the case. 00:17:39.880 --> 00:17:45.740 position:50% align:middle So that was one of the things that kind of stood out to us in this analysis and dealing with the type of data 00:17:45.740 --> 00:17:47.200 position:50% align:middle that we had come in. 00:17:47.200 --> 00:17:52.820 position:50% align:middle Many of the steps were largely outside of the control of the direct investigator, 00:17:52.820 --> 00:17:58.960 position:50% align:middle which ultimately oftentimes prolonged the timeline associated with the resolution of these cases. 00:17:58.960 --> 00:18:03.710 position:50% align:middle So that is why we looked at essentially both efficient case resolution, looking at number of days 00:18:03.710 --> 00:18:08.610 position:50% align:middle to case closure, but also did initially look at just case closure overall. 00:18:08.610 --> 00:18:13.480 position:50% align:middle So that was defined at the board level by the investigator. 00:18:13.480 --> 00:18:15.760 position:50% align:middle So, a second question came in. 00:18:15.760 --> 00:18:19.000 position:50% align:middle "Did you ask New York State to participate?" 00:18:19.000 --> 00:18:23.020 position:50% align:middle Yes. So, this was initially positioned as a pilot study. 00:18:23.020 --> 00:18:27.340 position:50% align:middle Despite the fact that it was a pilot study, we did ask all states if they would be 00:18:27.340 --> 00:18:28.590 position:50% align:middle willing to participate. 00:18:28.590 --> 00:18:36.820 position:50% align:middle This goes back to basically spring of 2018, because the bookend for this data collection period, 00:18:36.820 --> 00:18:40.430 position:50% align:middle if you remember back in the slide, was June 2018 to June 2020, 00:18:40.430 --> 00:18:43.910 position:50% align:middle so we just recently closed the data collection. 00:18:43.910 --> 00:18:47.570 position:50% align:middle But initially, we did ask all boards for their participation. 00:18:47.570 --> 00:18:53.220 position:50% align:middle But we did also recognize, I'm very quick to point out, that this was a pretty significant lift. 00:18:53.220 --> 00:18:59.360 position:50% align:middle So, as you saw from some of the slides, there was pretty significant variability at the board 00:18:59.360 --> 00:19:03.600 position:50% align:middle level for the number of active investigators that they had on staff, the number of cases that they were 00:19:03.600 --> 00:19:06.590 position:50% align:middle looking to kind of try to resolve. 00:19:06.590 --> 00:19:12.200 position:50% align:middle And so, for many of these boards, asking them to essentially take more of their time 00:19:12.200 --> 00:19:15.160 position:50% align:middle to support this study was a pretty heavy ask. 00:19:15.160 --> 00:19:18.650 position:50% align:middle So we were thrilled with the level of participation we got. 00:19:18.650 --> 00:19:23.680 position:50% align:middle We had about, as you saw, 10 boards participate from start to finish. 00:19:23.680 --> 00:19:28.510 position:50% align:middle We would have loved all 50 to participate, but we felt as though we were able to aggregate 00:19:28.510 --> 00:19:32.410 position:50% align:middle thousands of data points, both in terms of unstructured kind of free text 00:19:32.410 --> 00:19:38.150 position:50% align:middle responses as well as more kind of fixed or, like, finite criteria associated with board profile 00:19:38.150 --> 00:19:39.590 position:50% align:middle or respondent profile. 00:19:39.590 --> 00:19:45.380 position:50% align:middle So we were thrilled with the sample we had, but obviously, more would have been even better. 00:19:45.380 --> 00:19:50.410 position:50% align:middle And so, let's see if I have a… Yes. 00:19:50.410 --> 00:19:55.210 position:50% align:middle And I'll just echo what Cathy Russell [SP] is mentioning in the chat. 00:19:55.210 --> 00:19:59.090 position:50% align:middle The full slide deck is available in the details. 00:19:59.090 --> 00:20:02.400 position:50% align:middle But you can also feel free to reach out to me directly, and I'm happy to share. 00:20:02.400 --> 00:20:08.370 position:50% align:middle In addition, if you have any questions that you think of later, please feel free to reach out to me directly. 00:20:08.370 --> 00:20:11.590 position:50% align:middle My email is bmartin@ncsbn.org. 00:20:11.590 --> 00:20:13.600 position:50% align:middle And I'll be happy to go back and forth with you. 00:20:13.600 --> 00:20:16.690 position:50% align:middle So, I do see another question come here. 00:20:16.690 --> 00:20:21.710 position:50% align:middle "Did you compare efficiency with types of outcomes?" 00:20:21.710 --> 00:20:28.340 position:50% align:middle So, in terms of what was the ultimate action taken, we did not. 00:20:28.340 --> 00:20:34.020 position:50% align:middle So we looked at, effectively, the type of case category that it fell into. 00:20:34.020 --> 00:20:38.060 position:50% align:middle That's where we got into some of the nuance about what they were looking at or what they were 00:20:38.060 --> 00:20:39.490 position:50% align:middle attempting to investigate. 00:20:39.490 --> 00:20:43.680 position:50% align:middle But we did not look at efficiency to different outcome measures, if what you're talking about is 00:20:43.680 --> 00:20:48.560 position:50% align:middle like license revocation, temporary suspension, or whatnot. 00:20:48.560 --> 00:20:56.790 position:50% align:middle We looked at specifically essentially the timeline writ large 50,000-foot view to case resolution in any manner 00:20:56.790 --> 00:20:58.580 position:50% align:middle that they deemed appropriate at the board level. 00:20:58.580 --> 00:21:01.900 position:50% align:middle And then I see another question. 00:21:01.900 --> 00:21:05.190 position:50% align:middle "Did you take into account the types of outcomes of the case?" 00:21:05.190 --> 00:21:07.810 position:50% align:middle Oh, yeah. So this is a very related question. 00:21:07.810 --> 00:21:16.740 position:50% align:middle So, we do have some of that information available to us, but we have not yet investigated that. 00:21:16.740 --> 00:21:24.120 position:50% align:middle What we felt was most appropriate in kind of the initial pass of the data was to essentially just look 00:21:24.120 --> 00:21:29.830 position:50% align:middle at that timeline to, again, case resolution, be it whatever it is, 00:21:29.830 --> 00:21:35.650 position:50% align:middle whatever the board determined to be sufficient and appropriate for that particular case. 00:21:35.650 --> 00:21:43.610 position:50% align:middle I will say we did run analyses trying to get at some of the complexities associated with these cases, 00:21:43.610 --> 00:21:46.810 position:50% align:middle which I think some of these questions kind of allude to. 00:21:46.810 --> 00:21:51.430 position:50% align:middle And also, I mentioned this at the beginning of the live Q&A session. 00:21:51.430 --> 00:21:56.760 position:50% align:middle We do plan on pursuing kind of an extension of the natural language processing or kind 00:21:56.760 --> 00:21:59.100 position:50% align:middle of machine learning analysis. 00:21:59.100 --> 00:22:06.390 position:50% align:middle And the real core rationale behind that is to get at some of what you were kind of asking in the 00:22:06.390 --> 00:22:08.510 position:50% align:middle Q&A chat here. 00:22:08.510 --> 00:22:15.430 position:50% align:middle So we do want to understand some of the nuance associated with if it was kind of a larger, 00:22:15.430 --> 00:22:20.360 position:50% align:middle more complex case, but in certain instances, it was potentially more black and white. 00:22:20.360 --> 00:22:22.170 position:50% align:middle There were issues of concealment. 00:22:22.170 --> 00:22:27.450 position:50% align:middle There were things that essentially lended themselves to more kind of efficient processing. 00:22:27.450 --> 00:22:33.480 position:50% align:middle We're hoping some of those trends will become more apparent kind of regardless of how the case was 00:22:33.480 --> 00:22:36.750 position:50% align:middle resolved within some of the free text responses. 00:22:36.750 --> 00:22:39.320 position:50% align:middle So the analysis is not over. 00:22:39.320 --> 00:22:46.037 position:50% align:middle So we will certainly look at different outcomes as well. 00:22:52.120 --> 00:22:55.910 position:50% align:middle And one of the things that I will note, just as I wait to see if any additional questions 00:22:55.910 --> 00:23:02.950 position:50% align:middle come in, is we are looking to, obviously, publish the results of this analysis. 00:23:02.950 --> 00:23:10.230 position:50% align:middle It will likely be closer to the end of this year as we are looking to extend some of these analyses and make 00:23:10.230 --> 00:23:18.261 position:50% align:middle sure that essentially we have a comprehensive picture both of the unstructured and the structured data. 00:23:22.850 --> 00:23:25.970 position:50% align:middle All right. So, I think we're at time. 00:23:25.970 --> 00:23:31.930 position:50% align:middle I just wanted to thank everyone for attending today's session and for your great questions. 00:23:31.930 --> 00:23:37.270 position:50% align:middle If you do have questions that come to top of mind after this session closes, please, again, 00:23:37.270 --> 00:23:40.320 position:50% align:middle feel free to contact me at bmartin@ncsbn.org. 00:23:40.320 --> 00:23:44.210 position:50% align:middle Otherwise, have a great day, and I hope you enjoy the rest of the program. 00:23:44.210 --> 00:23:48.174 position:50% align:middle Bye.