(a) F1 score for CatBoost model trained on data from 1 previous year and evaluated on the next year.
(b) F1 score for CatBoost model trained on the data from 3 previous years and evaluated on the next year.
This conflict between efficiency and efficacy arises even when no eligibility scores are used explicitly. Classical statistical hypothesis testing teaches that efforts to reduce the odds of doing the wrong thing (a Type I error) generally increase the odds of not doing the right thing (a Type II error). It is therefore reasonable to assume that efforts to prevent improper overpayments correspondingly make it more likely that improper underpayments will occur.
This conflict can play out along multiple causal pathways. For example, fraud might be reduced by requiring claimants to prove having qualified dependents in triplicate instead of a single source. This requirement may cause some recipients to become ineligible even though they are, in fact, eligible. Underpayment errors would result on the margin for the appropriate dependent allowance, or in full if the recipient’s claim is placed on administrative hold pending determination or if a claimant elects to not to complete their application or appeal in light of the additional documentation required.
It should be noted that different errors are associated with vastly different practical results. Failing to detect an underpayment is different from failing to detect an overpayment, both from ethical and economic perspectives. From the claimants perspective, failing to detect an underpayment is usually worse than failing to detect an overpayment. The practical consequences of not receiving claims because an agency mistakenly determined them to be ineligible are severe (De La Garza, 2020; Eubanks, 2018).
This conflict between efficiency and efficacy is relevant for two reasons. First, the conflict can inform an analysis of the legal and policy history. We argue in the next section that social insurance programs in the US tend to focus on efficiency. Second, the conflict motivates our investigation into the use of AI to identify administrative errors, which we undertake in the subsequent sections. Avoiding under- and over-payment is immediately relevant to automatic classification (because these two values are related to avoiding false negatives and positives respectively). AI could be a way of partially overcoming the conflict between efficiency and efficacy.
### 2.4.2 Conflicts of Integrity and Equity
Two further conflicts within these public values should be mentioned — very briefly since in the literature on the ethics of AI, both are considered rather general and well-known tradeoffs (Andrus et al., 2021; Kleinberg et al., 2016, 2018). First, equity and privacy (a dimension of integrity) might conflict. To check whether a system upholds equity, one might need to reduce privacy. Specifically, to check that a system does not discriminate based on protected characteristics—such as gender, age, sexual orientation, or race—one needs data on these protected characteristics, which, depending on the nature of the data, can be a significant privacy risk.
Second, there is a public values conflict internal to *equity*. The two dimensions of equity, *error parity* and *predictive parity*, are themselves in tension with one another under realistic conditions (Kleinberg et al., 2016). This is a so-far under-investigated and un-addressed challenge for equity in welfare systems. Some contend that this conflict can be seen as a conflict between procedural fairness (such as equal treatment, here: predictive parity) and reasons of justice (here: error parity), similarly to how affirmative action furthers justice at the expense of procedural fairness (Vrendenburgh, Forthcoming).
## 2.5 Empirical Context
Unemployment Insurance in the United States was created by the 1935 Social Security Act. Employees who meet eligibility criteria can apply to receive these benefits if their current employment tenure ends. It is a devolved program; the federal government covers overhead costs and provides broad-based oversight, while State governments finance and implement the program individually. This devolution also applies to eligibility and benefit amount criteria, which can vary substantially across States and within any given State over time. Such changes are frequent, because UI is both a primary tool for countercyclical economic stimulus in recessionary periods when unemployment increases, and a highly visible and politicized program at both the state and national level. This combination of institutional variation and high political salience makes UI a highly complex administrative program. Its complexity, in turn, increases the risk of administrative errors — mistakes in determining the eligibility status of a claim and/or the amount of benefit the claimant is entitled to receive. When such errors occur frequently enough in programs that are as large as welfare, Medicare, or UI, the result is a significant misallocation of public money.
### 2.5.1 Institutional History: Focus on Overpayments
The US federal government began to take a hardline approach on combating administrative errors in UI during the 1970s in a “war on fraud and error” (Brodkin & Lipsky, 1983; Matt & Cook, 1993). As the name of this ‘war’ would suggest, the political goal of reducing or eliminating UI payments made to individuals who were not only receiving benefits they were statutorily ineligible for, but were actively defrauding the government in the process (Mendeloff, 1977).
The current process for auditing unemployment insurance claims is known as the Benefits Accuracy Measurement (BAM) program. BAM auditors take a statistically representative, randomized sample of all claims submitted to each State unemployment agency and audit this sample to find errors in both claimant data and administrative decisions on eligibility and benefit level determination. Over the past 20 years federal oversight has increasingly focused on the identification and recovery of improper payments — particularly overpayments — made through social insurance policies, including unemployment insurance. As part of this focus Congress has passed several laws producing performance management requirements for State unemployment administrators to demonstrate that they are proactively seeking, identifying, and correcting overpayments in their unemployment insurance programs. However, no such statutory focus exists with respect to the *under*payment of unemployment insurance benefits. This discrepancy recently came to the fore during the COVID-19 induced unemployment surge in the United States during 2020. Individual and collective harms from systematic wrongful denial and underpayment of unemployment insurance benefits proved to be substantial. At the same time, little has been done at the policy level to identify solutions for this problem going forward.
This research tests whether AI systems can help administrators identify all types of administrative errors in unemployment claim decisions — both overpayments and underpayments, as well as any other systematized errors. This is particularly relevant for practitioners because unemployment insurance is such a highly complex and complicated benefit program. The high level of variation between States and within States over time suggests that the underlying error rate of UI administration is high. To the extent that AI-based automation can both increase the scope and scale while decreasing the time and labor costs required to perform such audits, there are strong normative claims for using this technology on efficiency as well as efficacy grounds. But we argue that a more complete normative consideration also requires an empirical assessment of whether such AI systems may also improve our ability to identify and correct underpayments to eligible beneficiaries.
# Methods
## 3.1 Data
The dataset used in this analysis was sampled from the Benefits Accuracy Measurement system of the Department of Labor. This dataset contains information about randomly sampled investigations into UI claims: specifically, federally-collected information of improper UI payments. There are 785,159 observations in the final, analysis-ready dataset. Each observation contains information about one unemployment benefits claim made during the years 2002–2018 in the form of 228 features (variables). These features contain personal information of the claimant (date of birth, gender, race, etc), information about the last employment of the claimant (occupation code, salary, etc), information about the interaction between the claimant and agency (how the claim was filed, whether it was submitted on time or not, etc.), as well as other information. Additionally, we include data on state-level policy differences in UI eligibility requirements and benefit determination criteria, merged by State and corresponding year.
## 3.2 Analytic Strategy
In this work we treat the problem of detecting administrative errors as a classification problem. Classification (sometimes called categorization) is one of the most fundamental problems in machine learning. It assumes that all observations or datapoints (sometimes called data samples) belong to a limited set of categories or classes, and the goal is to predict the category of the datapoint from its features. A simple example of a classification problem is spam detection, in which new incoming emails should be categorized into two categories: “spam” and “non-spam.” Each email has a set of features, such as its content, its length and the words used, as well as from where the email was sent. To solve the classification problem, an algorithm uses these features of past emails together with each email’s known category — spam or non-spam — to train a new algorithm, a model, that describes those features that distinguish spam emails from non-spam emails. This new algorithm is then applied to incoming emails to determine whether they are spam or not.
In this work, we similarly use a supervised learning approach. Such an approach assumes that the categories, denoted by labels (such as “spam” and “non-spam”), are given for the sample, so that patterns between features and labels can be learned. In this work, we use machine learning algorithms to predict administrative errors. Thus, the features in our data describe various pieces of information about UI claims and class labels describe the types of possible errors (e.g., “No error,” “Underpayment,” etc). Different methods are available to train a model with the labeled samples in order to predict labels on the unlabeled instances. Next, we describe the various machine learning approaches that we used and their obtained results.
### 3.2.1 Overview of Machine Learning Classification
As discussed above, classification in machine learning can be defined as a problem of predicting labels of the data points (data samples). Each data point has a corresponding vector of the features, which describe that sample. For example, for the spam detection problem, possible features include the number of words in the text, specific words (or their combinations) used in the email, url addresses in the email (which can be used to detect suspicious websites), time of the email, etc. Features can appear in different formats: categorical features, which have values drawn from a limited set of possible values (for example, dog vs. cat); numerical features, which take on a numerical value; text features; image-based features; and others. Many machine learning algorithms expect input features to be in numerical form, and various approaches exist to convert non-numerical features into numerical features.
In addition to these features, a machine learning algorithm also needs a subset of data for which samples are labeled with their category or class. Ultimately, the machine learning model will be trained on this subset, and patterns connecting the features to the class label will be extracted. In supervised machine learning, these categories are assumed to be known — for example, they could be annotated manually by humans, or the process of data collection may be such that categories are obtained alongside data, and so on.
These labeled samples are used to train the model. During the training process, the machine learning algorithm tries to infer patterns in the features to distinguish different categories — for example, “spam” emails may have some specific words that are rarely used in the usual emails, or they may be sent from some specific set of domains. In contrast, “non-spam” emails may be more likely to come from the same domain as the recipient of the email, or sent from addresses that have already been replied to by the recipient. The trained model is used to predict categories of non-labeled samples in an automated way. In addition to making predictions about unlabeled data points, ML algorithms can be used to better understand the data. For instance, algorithms can identify those features that are most important for classification, giving the analyst insight into properties distinguishing between classes of data points.
For certain classes of algorithms (e.g., those based on decision trees), feature importance can be extracted directly from the model. For other algorithms, it is possible to infer the most important features through local permutation of the samples (i.e., make slight modifications to feature values and observe how the predicted label changes) (Baehrens et al., 2010). In the case of longitudinal data, such algorithms may help to identify changes over time. Additionally, as different ML models operate under specific assumptions about structure of the patterns that may exist in the data, even the performance of various algorithms may reveal additional information, leading to better understanding of the dataset.
The choice of a classification algorithm gives rise to public values conflicts. Insofar as different machine learning algorithms are better or worse at enabling an increased understanding of the data or of the classification, as described here, the algorithms fulfill the public values of integrity, especially its dimensions of audit and explanation, to different degrees. Some algorithms might be more accurate at the expense of fewer insights into feature importance. Or some algorithms might be more accurate at the expense of needing more, and perhaps more sensitive data — a potential privacy risk. The choice of a classification algorithm thus may give rise to conflicts between efficacy and efficiency on the one hand and integrity on the other.
### 3.2.2 CatBoost: Decision Tree-Based Classification
For the sake of performing a thorough analysis, we use several classification algorithms, which are listed in the Experimental Setup section. Here, for purposes of illustration, we describe one of them: the CatBoost algorithm. CatBoost belongs to the family of gradient boosting classifiers. This type of model combines multiple decision trees into a single model. A decision tree is a flowchart-like object in which feature values are used to determine which branch in the flowchart to take, until a prediction is arrived at (Jain & Srivastava, 2013). During construction of the decision tree, the algorithm iteratively selects the feature that is the most useful for predicting the label, and builds a corresponding branch of the tree, thus splitting the dataset according to the selected feature.
CatBoost (and gradient boosting models in general) are based on an important observation: combining predictions of multiple simple models is better than using a single model. An entire class of models builds a set of decision trees such that every new tree is trained to correct errors of previously trained trees. Several models operate under this general framework, including XGBoost (eXtreme Gradient Boosting (Chen et al., 2015)), LightGBM (Ke et al., 2017), AdaBoost (Friedman et al., 2000), and CatBoost.
Although these algorithms often perform similarly, the main advantage of CatBoost over other algorithms from the family of gradient boosting models is that it was created to address a problem known as *prediction shift*. Prediction shift can be defined as a variation of target leakage, which happens when classifiers during training implicitly get access to the information about the label of the sample in the way that is impossible for unlabeled data during prediction. This happens because when trees are trained iteratively, the gradients of the error reveal information about the target variables.
CatBoost addresses prediction shift with ordered boosting (a weighted sampling method). In this work, we use the “MultiClass” optimization from the original CatBoost implementation, which uses a Multiclass Cross-Entropy Loss (log-loss) function given by:
$`MCE\ = \ \frac{\sum_{i = 1}^{N}w_{i}log(\frac{e^{a_{it_{i}}}}{\sum_{j = 0}^{M - 1}e^{a_{ij}}})}{\sum_{i = 1}^{N}w_{i}}`$ ,
where $`a_{ij}`$ represents the predicted probability that element $`i`$ belongs to class $`j`$ and the $`w_{i}`$ values represent weights associated with each element.
Additionally, CatBoost generates new features during the training process. These features are defined as a combination of existing features. As the number of possible combinations of features on the big dataset is huge, CatBoost constructs new features according to a greedy heuristic to approximate the true global optima.
## 3.3 Experimental Setup
### 3.3.1. Preprocessing of the dataset
To prepare the dataset for analysis, we first removed all features that could act as a direct proxy to the target variables. Next, for all models except CatBoost, all non-numeric features were converted to a categorical binary representation (one-hot encoding). As this representation is very sparse, some of the features were removed to reduce dimensionality of the data. CatBoost is able to work with some non-numeric (e.g., text features) features by design, assuming their categorical type.
### 3.3.2. Algorithms tested
We evaluated several different ML classifier algorithms to test for variation in performance against the audit data. These include:
1. **Logistic regression** (LR). This algorithm is simple, but is commonly used for classification problems. While originally designed for the binary classification, it can be used for multi-class classification with one-vs-all scheme (where a different LR model is trained for each class).
2. **Random Forests** (RF). This algorithm is an ensemble-based algorithm which uses multiple decision trees trained on the different subsets of the data.
3. **CatBoost**. This algorithm, described in more detail above, is based on the idea of Gradient Boosting.
4. Several deep learning-based models:
1. **TabNet** (Arik & Pfister, 2019). This DL model was created by Google for neural network learning on tabular data. It uses sequential attention and, similarly to the Gradient Boosting models, allows insight into structural patterns in the data.
2. **DeepFM** (Guo et al., 2018). This model was created for use in recommender systems and uses factorization machines together with neural networks. While it was not created for tabular data specifically, it may be used with this type of data, and we include it for completeness.
3. **WideDeep** (Cheng et al., 2016). This neural network was created by Google for recommender systems. It combines “wide” linear models and “deep” neural networks. Similarly to DeepFM, it can be used for the tabular data.
4. **Deep & Cross Network** (DCN) (Wang et al., 2017). This model is based on the idea of feature crossing (generation of new features based on existing ones). This model also works well for the tabular data.
To evaluate the performance of a model, we need to compare predictions on a subset of data with known labels. This subset of data is usually referred to as a “test” set or “holdout” set. This data should be excluded from the training process, as it is important that the model is not familiar with the testing data.
Two scenarios were considered. In the first scenario the whole dataset was split into two parts — the “training” set and the “test” set, consisting of 80% and 20% of the data, respectively. The “training” set was used for the training of the model and the “test” set was used for the evaluation. In the second scenario, dataset models were evaluated only on one specific year of claims and trained on the data from one (a) or three (b) previous years. This setting is more realistic, as in this case the model is not able to utilize patterns observable in the current year and has to rely only on previous observations.
## 3.4 Evaluation Metrics
Each sample of the dataset is a UI claim that was randomly selected to be investigated for incorrect payments. There are several different categories that datapoint can belong to: “No error,” “Overpayment,” “Underpayment,” or “Wrong issue.” A “No error” label means that the claim was processed without any errors, “Overpayment” denotes a claim with excessively paid benefits, and “Underpayment” labels claims with incorrectly low benefits. “Wrong issue” means that an error unrelated to payment in the claim was made.
**Table 1. Audit determination outcomes in the data.**
| | | |
|:--:|:--:|:--:|
| **Outcome** | **Count** | **Proportion** |
| No Error | 629,445 | 0.807 |
| Overpayment | 74,983 | 0.096 |
| Underpayment | 59,080 | 0.076 |
| Wrong Issue | 16,090 | 0.021 |
| *Total* | 779,598 | 1 |
The dataset is highly imbalanced, as shown in Table 1. Most of the claims belong to the “No Error” class. This fact needs to be accounted for during model evaluation. To understand why, suppose that we evaluate the model just by percentage of claims with correctly inferred class. In this case a simple classifier that makes the “No Error” prediction every time would have a relatively high accuracy of 80%, while being absolutely useless at the same time. To address this, one can use the *F1-score* to evaluate accuracy. To define F1-score we need to define some other metrics that are commonly used in evaluation of ML models:
1. Precision: the fraction of the correctly predicted instances from a class over all predicted instances of the class. In other words, precision with respect to a particular class is defined as the number of true positives divided by sum of true positives and false positives:
``` math
P\ = \ TP/(TP + TN).
```
2. Recall: the fraction of the relevant instances which were correctly predicted. This can also be defined as the number of true positives divided by the sum of true positives and false negatives:
``` math
R\ = \ TP/(TP + FN).
```
The F1-score is the harmonic mean of these two measures:
``` math
F_{1} = \frac{2P*R}{P + R}\
```
For the multi-class setting there are two ways to compute the F1-score:
1. Precision, recall, and F1-score may be computed separately for each class. The *macro-F1 score* is then computed as the unweighted mean of these class F1-scores.\
2. Precision and recall may be computed globally for all classes at the same time. The *micro-F1 score* is computed from this single precision and recall value.
## 3.5 Addressing Imbalances in the Dataset
As the dataset is very imbalanced, each model tends to predict most of the claims as belonging to the “No error” class. This leads to low recall performance of less frequent classes (i.e., many members of these classes are not identified).
Such an imbalanced dataset is typical for any empirical context that involves the identification of administrative errors, insofar as errors are the exception. How such imbalances can be addressed is thus a relevant question for any context where AI is used to detect administrative errors.
The issue of low recall performance can be addressed in different ways; for example, by using different weighting schemes or employing techniques aimed to rebalance dataset (undersampling, oversampling, generation of balanced synthetic data). Most of these approaches exhibit a tradeoff between precision and recall. Precision may be sacrificed in order to obtain higher recall.
This tradeoff affects public values. It constitutes another instance of the conflict between efficacy and efficiency. On the one hand, reducing precision for detecting over- and underpayment reduces efficiency. More claims are flagged as administrative errors by mistake, and investigated without finding an error (*cost*). On the other hand, increasing recall for underpayment errors is a matter or efficacy. It means that more of the underpayment errors, which might otherwise go unnoticed, are detected by the algorithm (*avoid underpayment*).\
At the same time, increasing recall for overpayment, however, increases efficiency again, since more of the true overpayments errors are detected as such (*avoid overpayment*). Given the normative and empirical relevance of imbalanced datasets, we explore different methods, we perform two experiments, described next.
### 3.5.1 Weighting schemes
The simplest way to increase recall of the minority classes is to change class weights in the model (for those algorithms that support class weights). Intuitively, class weights affect the penalty for misclassification of different classes. The higher class weight is comparatively to the weights of other classes, the higher is the expected penalty for misclassification for this category.
There are multiple different ways to set class weights. In our experiments, we set weights according to the “balanced” heuristics suggested by (King & Zeng, 2001):
> $`w_{c} = N/(\ k\ *|c|)\`$,
where $`w_{c}`$ denotes the weight of class $`c`$, $`N`$ stands for the size of the dataset, $`k`$ denotes the number of classes in the dataset, and $`|c|\`$denotes the number of samples that belong to class $`c`$.
### 3.5.2 Synthetic data
Another common technique for addressing the problem of an imbalanced dataset is to generate synthetic data which follows a similar distribution as samples from the minority classes. This synthetic data is merged into the training set to make proportions of the classes balanced. Ultimately, the model is evaluated on the test set (which does not have any synthetic samples). For this experiment, we used the SMOTENC (Chawla et al., 2011) algorithm for generating synthetic data, which operates under the following general principle:
1. Select a random sample from the least frequent class;
2. Find the $`k\`$most similar samples in the dataset (in our experiments $`k`$ was set to 5; similarity between samples is defined as Value Difference Function (Cost & Salzberg, 1993); and
3. Generate a new sample in the next fashion: sample numeric features between values of random neighbor and the features original example. Set categorical features to be equal to the most common category from the neighbors of the same class.
# 4. Results
## 4.1 Descriptive Statistics
Administrative politics and economic situations change over time, which affects patterns of administrative errors as well. Figure 2 shows the trend of increasing overpayment error rates with an approximately 25% increase from 2002 to 2018. The rate of underpayment was slowly decreasing over the period of observation.
**Figure 2. Improper payment error trends over the sample period.**
This descriptive statistic can be evaluated with the public values framework outlined above. Insofar as underpayment errors are trending downward and overpayment errors upward over time, the UI system would have improved over time in efficacy (*avoid underpayment*) but worsened in efficiency (*avoid overpayment*).
The error types also vary by State, as seen in Figure 3, which shows aggregated results for all years. These differences may be explainable by differences in State policies. For example, Ohio has a high rate of underpayment errors (one of the highest in the country), but a comparatively low rate of fraud. Non payment-related errors appear to be particularly clustered in the New England region of the Northeast, particularly in New Hampshire. Across all states and outlying territories, the total cost of overpayments across all years of observation is \$69 million. Total underpayments are approximately \$5 million. This variance might also be indicative of different decisions in resolving public values conflicts.
**Figure 3. Relative rate of improper payment errors by type of error across states.**
## 4.2 Classification Analysis
In this section we report our estimates of whether machine learning can be used to predict administrative errors made in processing claims using features of the claim and prior information on administrative error, and utilize trained models to obtain better understanding of the data. Our goal is to predict the class (category) of the claim. Recall from Section 3 that we considered two experimental settings. In the first setting, we combined all years of data and performed a randomized split of the data into training and test sets. Results of this setup can be found in Tables 2 and 3. The best-performing classifier’s score for both precision and recall by each class of outcome are bolded. As can be seen, the performance of the CatBoost is the highest, with Random Forest in second place. Logistic Regression failed to generalize and performed poorly.
**Table 2. F-Scores (micro/macro) by classifier type by class using data from all time periods.**
| Classifier | F1-score (macro) |
F1-score (micro) |
| LR | 0.230 | 0.721 |
| RF | 0.452 | 0.805 |
| CatBoost | 0.491 | 0.823 |
| TabNet | 0.366 | 0.781 |
| DeepFM | 0.274 | 0.730 |
| Wide & Deep | 0.271 | 0.733 |
| DCN | 0.384 | 0.777 |
| No errors | Overpayment | Underpayment | Wrong Issue | |||||
| Classifier | Pr. | Rec. | Pr. | Rec. | Pr. | Rec. | Pr. | Rec. |
| Logistic Reg. | 0.807 | 0.995 | 0.250 | 0.015 | 0.061 | 0.001 | 0.000 | 0.000 |
| Random Forest | 0.851 | 0.986 | 0.756 | 0.374 | 0.651 | 0.113 | 0.702 | 0.119 |
| CatBoost | 0.865 | 0.980 | 0.739 | 0.486 | 0.686 | 0.171 | 0.745 | 0.106 |
| TabNet | 0.842 | 0.975 | 0.627 | 0.404 | 0.605 | 0.036 | 0.500 | 0.001 |
| DeepFM | 0.818 | 0.952 | 0.277 | 0.168 | 0.071 | 0.003 | 0.000 | 0.000 |
| Wide & Deep | 0.815 | 0.971 | 0.394 | 0.122 | 0.044 | 0.006 | 0.000 | 0.000 |
| DCN | 0.835 | 0.983 | 0.710 | 0.290 | 0.438 | 0.039 | 0.475 | 0.090 |
(a) F1 score for CatBoost model trained on data from 1 previous year and evaluated on the next year. |
(b) F1 score for CatBoost model trained on the data from 3 previous years and evaluated on the next year. |
**Table 4. Description of important features.**
| | |
|:--:|:--:|
| **Feature Name** | **Feature Description** |
| High quarter wages | Highest wages reported in the fiscal quarter before investigation. |
| Remaining Balance | Remaining Balance (RB) of claim as of key week ending date. |
| Base period wage | Base period wage before the investigation. |
| Monetary redetermination | Whether the State redetermined claimant’s monetary eligibility. |
| Last tax rate | Last tax rate for the claimant. |
| Occupation code | Occupation code of the claimant’s last employment. |
| Claim was paid | Money was paid for the claim (claim was not denied). |
| Reason for monetary denial | Reason for monetary denial before investigation. |
| \# Job Contacts | Number of Job Contacts Listed from any source. |
| Determination Appeal results | Results of Appeal of Initial Determination that denied eligibility. |
| Nonseparation Determ. Reason | Reason for Nonseparation Determination Before Investigation. |
| Filling method for KW claim | Filing method for claim |
| Benefit Year Beginning | Effective date of most recent new or transitional (not reopened or additional) claim. |
| Date of first comp. wek | Date of first compensable week. |
| Most recent ERP date | Date of claimant's most recent eligibility review up to Key Week. |
| Week of claim filed | Week in which claim was filed (beginning date). |
| Numeric State identifier | State |
| Work Search Requirements | Subject to Work Search Requirements |
| \# of Labor Exchange Referrals | Number of times Employment Services referred claimant for employment during the current benefit year. |
| Separation Determ. Reason | Reason for Separation Determination Before Investigation. |
## 4.2 Weighting schemes and synthetic data
Results for CatBoost, trained with the “balanced” heuristic weight scheme can be found in Table 5. There is significant improvement of the recall of minority classes at the expense of precision. At the same time recall of the “No errors” class dropped.
**Table 5. Results for CatBoost using ‘balanced’ weighting scheme**
| Audit Outcome Class | ||||
| Metric | No Errors | Overpayment | Underpayment | Wrong Issue |
| Precision | 0.944 | 0.342 | 0.236 | 0.110 |
| Recall | 0.532 | 0.815 | 0.502 | 0.773 |
| F-1 score (macro) | 0.419 | |||
| F-1 score (micro) | 0.624 | |||
| Audit Outcome Class | ||||
| Metric | No Errors | Overpayment | Underpayment | Wrong Issue |
| Precision | 0.862 | 0.678 | 0.554 | 0.491 |
| Recall | 0.967 | 0.446 | 0.180 | 0.150 |
| F-1 score (macro) | 0.488 | |||
| F-1 score (micro) | 0.813 | |||
| Alternative Training Method | Metric | Outcome Class | |||
| No Errors | Overpayment | Underpayment | Wrong Issue | ||
| Weighting Schema | Precision | 9% | -54% | -66% | -85% |
| Recall | -46% | 68% | 194% | 629% | |
| F-1 score (macro) |
|
||||
| F-1 score (micro) |
|
||||
| Synthetic Data | Precision | 0% | -8% | -19% | -34% |
| Recall | -1% | -8% | 5% | 42% | |
| F-1 score (macro) |
|
||||
| F-1 score (micro) |
|
||||