"All that Glitters": Approaches to Evaluations with Unreliable Model and Human Annotations

Full reproductions⁸⁸8Small differences in the reported values here compared to the original study arise from random human rater selection required in the procedure, which were done at the segment level. All families and model evaluations used the same random sample of human raters for comparison. of all reliability metrics and calculation processes exactly as described in the NCTE Main Study Appendix Section 2 were conducted. (Kane et al., 2015). Following their same procedures, replicated calculations were extended to the model families, replacing a human rater score with a specified or random model for evaluations of individual models and model families, respectively. Intra-class correlations (ICCs) are with the calculation methods in Appendix F. Reproduced human results and model results, including additional metrics in this section, are fully reported in Appendix F.1 and all item results can be found in the online supplement.

The results also include three additional correlation and reliability metrics: Quadratic Weighted Kappa (QWK) typically used in ordinal classification tasks to penalize distance quadratically (squared error) while accounting for categorical agreement by chance (e.g., Shermis (2014); Hardy (2021); Wang and Demszky (2023)), Pearson correlation $r$, (e.g., Whitehill and LoCasale-Crouch (2024)) Spearman correlation $\rho$ (e.g., Wang and Demszky (2023); Xu et al. (2024)), and Kendall correlation $\tau$ (e.g., Liu et al. (2023b)). Figure 2 shows Spearman correlations ($\rho$) and confidence intervals for all model families and for models from Xu et al. (2024). The table in Figure 2 contains the $\rho$ estimates.

Using nearly any standardized combination of metrics across all items from Section 5.1, Encoder models perform better than the single highest performing expert human rater. The human labels assigned for the four focus MQI have very low reliabilities, despite the significant training and calibration for human raters described in 3.2. Overall, the human labels are highly unreliable, but if a researcher were trying to compare the model to human performance, they could be displayed as they are in Table 1. For metrics of agreement and reliability, each encoder model outperformed humans on average, whilst each GPT model underperformed humans on every metric and every item. Table 1 has a summary of the full panel of lesson segment-level inter-rater reliability metrics for each MQI item. Specific metrics for the four focus MQI items in this study are in Panel (b) in Figure 4, and the full individual model-item comparisons for all MQI items and metrics in this section are in Table LABEL:tab:tab:full. Additionally, the detailed full results for all models and metrics, MQI, and CLASS rubrics can be found in the supplementary materials online.

Using only these metrics and without further testing, one might assume that the encoder models are therefore ready to help with the task of automated annotations of teaching quality or that GPT models show improvement to ICC measures and could be helpful. Implications: Basic statistics in the presence of unreliable labels can mislead interpretations of model performance. Researchers should be wary of studies reporting few metrics in the presence of low reliabilities.

Generalizability Study (g-study) (Brennan, 2001a, 2013, b; Hill et al., 2012b) designs utilize random effect estimates across possible configurations of different sources of variance to quantify how generalizable labels. This is done by estimating the extent to which given labels would persist if sources of variation changed (e.g., same teacher, different day; same lesson, different rater; human rater vs model rater; etc.). $\mathbf{E}\rho^{2}$ is a measure of the relative generalizability of a rating (i.e., is rating order preserved), and $\mathit{\Phi}$, accounting for absolute error, is a measure of label dependability: how likely specific ratings would be numerically the same with different sources of variation. These two reliability-like estimates can help quantify how "golden" labels are.

The multifaceted g-study design used to estimate the how much variation ($\nu$) in individual teachers’ instructional quality, $i$, contributed to a rating label, $X$, annotated for a section of a lesson, $s$, during an observation, $o$, on rubric item $j$ by rater $r$ is known as a Item-by-Rater-by-Segment-within-Observation-within-Individual Teacher design: $J\times R\times(S:O:I)$. Overall estimates across all MQI items for a given rater family, $\mathbb{F}$, are in Table 2. For item-level reliabilities, we simplify the expression by holding the item fixed, resulting in a $R\times(S:O:I)$ design. Using nested random effects notation, the estimation model is:

$\displaystyle X_{s:o:ir}^{(j)}=\mu+\nu_{i}+\nu_{o:i}+\nu_{s:o:i}+\nu_{ir}+\nu_{r}+\nu_{s:o:ir},\forall j\in\textbf{J}$

(1)

where $j$ indicates the item index.⁹⁹9For the estimates in Fig. 4 (c), for dependability metrics of Section 5.3, and for comparability with human baselines(Hill et al., 2012b; Kane et al., 2015; Ho and Kane, 2013; Kane and Staiger, 2012), a simplified model, an by-item $R\times(O:I)$ design, was conducted for the human expert rater family with results in Appendix H.1. The simplified model is $X_{o:ir}^{(j)}=\mu+\nu_{i}+\nu_{o:i}+\nu_{ir}+\nu_{r}+\nu_{o:ir}$ The full model structures of Eq. 1, 2 and 3 are used for Section 5.6. Code for the model specification is in Appendix H.3. Then, $\mathbf{E}\rho^{2}$ (Equation 2) and $\mathit{\Phi}$ (Equation 3) are easily estimated from the random effects for raters in rater family $\mathbb{F}$:

$\displaystyle{\mathbf{E}\mathit{\rho}^{2}_{\mathbb{F}}}^{(j)}=\frac{\nu_{ij}}{\nu_{ij}+\nu_{o:ij}+\nu_{s:o:ij}+\nu_{irj}+\nu_{s:o:irj}}\text{, }$

(2)

$\displaystyle\mathit{\Phi}_{\mathbb{F}}^{(j)}=\frac{\nu_{ij}}{\nu_{ij}+\nu_{o:ij}+\nu_{s:o:ij}+\nu_{irj}+\nu_{rj}+\nu_{s:o:irj}}\text{, }$

(3)

$\forall r\in\mathbb{F}$, where the individual item-rating-segment variation, $\nu_{s:o:irj}$, is confounded with error variation. These results are found in Table 2. A figure comparing the $\mathbf{E}\hat{\rho}^{2}_{j}$ item values to item-level reliability estimates related to Guttman’s $\lambda_{6}$ , $\rho^{\lambda_{6}}_{jj\prime}$, from Classical Test Theory (Zijlmans et al., 2018a, b), can be found in Appendix H.2. Additionally an illustration of sources of variance including descriptions can be found in Appendix H, color-coded to support interpretation of sources of variance with the table of results.

Humans, on average, produce labels that are both more reliable and generalizable. The full results for human rater labels, decomposed into variance components, can be found in H.3¹⁰¹⁰10Appendix 2.c of Kane et al. (2015) provided a g-study, but, surprisingly, not using the data from the study. and estimates for $\mathbf{E}\rho^{2}$ and $\mathit{\Phi}$ can also be found in panel (c) of Figure 4. The Encoder models outperform humans on nearly every item in terms of inter-rater reliability metrics (Table 1) , but not in generalizable reliability metrics as seen in panel (c) tables of Figure 4. Importantly, the large difference between $\mathbf{E}\hat{\rho}^{2}$ and $\mathit{\hat{\Phi}}$ for Humans and Encoders is due to properties of individual items, which accounted for over 75% of the variation in those families. GPT models, on the other hand, did not change ratings very much on different items, consistent with literature on these models not understanding such prompts Liu et al. (2023a); Webson and Pavlick (2022); Heo et al. (2024).

Table 2 shows that Encoder model still performs better than humans on the majority of items, but it is no longer as clear. Interestingly, as mentioned in Section 4, the encoder models did not receive any annotations outside of the transcript, including speaker. This means that the model would struggle to identify teacher explanations (EXPL) from student explanations (STEXPL). This shift in interpreting encoder family performance from superhuman to zero reliability adds validity to the argument that these metrics provide valuable insight, showing that the relationships found in some of the variables could be explained by variance unrelated to the label construct. Implications: Measures of generalizability and dependability derived from structured variance decomposition can meaningfully quantify label quality.

Dependability and generalizability do not guarantee accuracy, but even at these very low levels, they can be used in indirect tests of convergent validity to see whether correlations between humans and models are low because of measurement error, such as poor rubric item construction, or because the two sets are really uncorrelated. If an individual teacher’s latent instructional ability $\theta_{i}$ is about the same from lesson to lesson with the same students, we can correlate $\hat{\theta}_{i}$ for human ($\mathbb{h}$) and model ($\mathbb{m}$) family ratings for different lessons coming from the same teacher and correct for measurement error by disattenuating the correlations by each rater family’s $\mathbb{F}$ label generalizability, $\mathbf{E}\hat{\mathit{\rho}}_{\mathbb{F}}^{(j)}$, for a given item $j$. The disattenuated correlation,$\mathbf{\varrho}_{\mathbb{hm}}^{(j)}$, between humans and a family of models for item, $j$, can be estimated:

$\displaystyle\mathbf{\varrho}_{\mathbb{hm}}^{(j)}=\frac{\operatorname{Corr}[\operatorname{\tilde{\mathcal{X}}_{\mathbb{h}}}(i,\mathfrak{L},j,r_{\mathbb{h}}),\operatorname{\tilde{\mathcal{X}}_{\mathbb{m}}}(i,\neg\mathfrak{L},j,r_{\mathbb{m}})]}{\sqrt{{\mathbf{E}\hat{\mathit{\rho}}^{2}_{\mathbb{h}}}^{(j)}{\mathbf{E}\hat{\mathit{\rho}}^{2}_{\mathbb{m}}}^{(j)}}}$

(4)

where $\tilde{\mathcal{X}}_{\mathbb{F}}$ is score retrieval function for individual teacher $i$ on item $j$ by a random member $r$ of rater family $\mathbb{F}$ in relation to some observed lesson $\mathfrak{L}$ with family label generalizability, ${\mathbf{E}\hat{\mathit{\rho}}^{2}_{\mathbb{F}}}^{(j)}$ defined in Equation 2. In other words, the numerator (represented in red in Figure 3) is the correlation in scores whenever two different lessons from the same teacher were scored by raters from different families (human and model). The denominator then adjusts for based on the reliabilities of raters from each family to account for the known tendency of low reliability to diminish observed correlations.

Figure 4 panel (b) has the disattenuated correlations and their respective 95% confidence intervals, calculated at $\alpha=0.05$ using empirical confidence scaling methods defined by Charles (2005), which produces more conservative confidence intervals on this data than traditional Fisher normalization (Kromrey et al., 2008), which is preferable given the low levels of reliability in Section 5.2 which can lead to overcorrection. Reported disattenuated correlations of 1.0 do not mean perfect correlation: it generally means that measurement error is not randomly distributed.

Disattenuated correlations are not directly comparable¹¹¹¹11For example, reported disattenuated correlations of 1.0 do not mean perfect correlation: it generally means that measurement error is not randomly distributed. to the measures of correlation in Section 5.1 (Muchinsky, 1996). However, failure of disattenuation to identify viable human-model correlations for items that previously such showed correlated relationships in Section 5.1 suggests the prior correlations may be spurious. Disattenuation does not change the low reliability across items nor the quality of the measurement, but it offers indirect evidence for discerning model predictive validity by quantifying the how changes in the underlying construct result in changes in the same direction for both human and model.

Results for disattenuated correlations described in Section 5.3 and their confidence intervals are in Figure 3. Most items show correlated relationships after disattenuation, and most with confidence intervals above 0.5, suggesting that the encoder models and the humans are likely identifying similar sources of underlying teacher variation for those items.

Disattenuation analyses and Section 5.2 suggest that the Encoder model family’s SOTA-level correlations on the EXPL and STEXPL item may have been spurious (likely identifying speech patterns associated with higher teacher performance, and not necessarily specific to explanations), a direct result of low generalizabilities found in Section 5.2. Additionally, we see see very large confidence intervals for the encoders for items where item score distributions are most imbalanced (MGEN, MAJERR), suggesting that correlations found are not justified in the presence of low reliabilities. Items where the disattenuated correlations are lower (e.g., LCP, MMETH) suggests that models and humans interpreted observational features differently. Implications: when measurement error is high, disattenuating model and human correlations can help identify whether items with high or similar correlations have spuriousness or are responding to similar features.

This method only minimally provides evidence for investigating accuracy and validity, but, for the Encoder models, evidence can be built upon by comparing how the more continuous ratings of the models and humans change and correlate over the course of a given observation. While not explicitly part of this study, an example of how Encoders’ and humans’ ratings change from the start to the end of a class for a randomly chosen lesson observation is illustrated in Figure 15. Investigating the validity of a construct would require more robust qualitative review of the content.

Rater biases in complex tasks are usually not directly measurable, but we can estimate latent constructs that quantify the effects of individual raters’ behaviors using methods commonly used to estimate latent attributes of rubric items (e.g., item difficulty) and latent attributes individuals (e.g., ability) throughout Item Response Theory (IRT). If the data had no variation due to raters, various polytomous IRT methods could help estimate "true scores"/"gold" labels ($\xi_{ij}$) during classroom observations, teacher instructional abilities ($\theta_{i}$), and the various individual item effects. For tasks with human-mediated labels, human raters introduce additional sources of measurement error for each classification and the data may include multiple measures from multiple raters for a single observation (leading to an accumulation of information at overlap observation points). To address this, hierarchical rater modeling (HRM) (Patz et al., 2002; Decarlo, 2003; DeCarlo et al., 2011) combines an IRT model with a first stage estimation defined by a signal detection theory (SDT) relationship. The latter asks the question, "given the presence of the ’true’ score, can a rater detect it?" as the former asks, "given the inputs, can we estimate the ’true’ score accounting for differences in the tasks used to measure it?". The hierarchical structure addresses the problem of accumulation of information in the estimates. HRMs consist of three components:

$$\displaystyle\text{HRM}\begin{cases}\boldsymbol{\theta}_{i}\sim\text{MVN}(\textbf{0}_{M\times 1},\textbf{I}_{M\times M})\text{,}\\ \xi_{oij}\sim\text{{IRT model}: Equation \ref{eq:MHRM_IRT}}\\ X_{soijr}\sim\text{{SDT model}: Equation \ref{eq:MHRM_SDM}}\end{cases}$$

(5)

where an IRT model estimates the "gold" label score $\xi_{soij}$ for a given item for some time segment $s$ in teacher $i$’s $o$-th observed lesson for item $j$, which arises from $i$’s $M$-dimensionally distributed latent instructional ability/needs ($\boldsymbol{\theta}_{i}$), and a Signal Detection Theory (SDT) model component disentangles individual rater biases from each recorded score, $X_{soijr}$, by quantifying the latent attributes that mediate whether rater $r$ correctly detects the true score, i.e., $p_{\xi kr}=\ P\left[X_{soijr}=k\ |\xi_{oij}=\xi\ \right]$.

The IRT component of Equation 5 estimating the the true scores based on rubric item- and teacher-specific parameters is a $K_{j}$-category multidimensional generalized partial credit model (MGPCM) (Muraki, 1992; Adams et al., 1997; Cui et al., 2024; Casabianca, 2021). Distributional challenges of negatively worded items can be addressed through a multidimensional parameterization of the underlying latent teacher instructional abilities, with between-item dimensionality confirmatorily defined by the factors in Blazar et al. (2017). The MGPCM item discrimination parameters, $\boldsymbol{\alpha}_{j}=\alpha_{jm}$, a vector of dimension-specific traits $\boldsymbol{\theta}_{i}=\theta_{im}$ are separated for $m\in M$ latent dimensions, and parameters for item difficulties $\gamma_{jk}$ exist for each possible score category $k$ in item $j$:

$\displaystyle P\left[\xi_{oij}=\xi\ |\boldsymbol{\theta^{\prime}}_{i},\ \boldsymbol{\alpha}_{j\ },\ \gamma_{j\xi},o\right]=\frac{\exp\left\{(k-1)\boldsymbol{\alpha}_{j}\boldsymbol{\theta^{\prime}}_{i}-\sum_{k=1}^{k}\gamma_{jk}\right\}}{\sum_{h=1}^{K_{j}}\exp\left\{(k-1)\boldsymbol{\alpha}_{j}\boldsymbol{\theta^{\prime}}_{i}-\sum_{k=1}^{h}\gamma_{jk}\right\}},\forall s\in o$

(6)

where $oi=1,...,N$ lessons observed for teacher $i$, $j=1,...,J$ items, $r=1,...,R$ raters, and $k=1,...,K$ possible scores.

As parameterized by Patz et al. (2002), the base-level SDT model of the HRM represents the measurement error induced by rater $r$ whose ability to "detect" the true score changes according to an individual rater’s item-specific biases, $\phi_{jr}$ and variabilities, $\psi_{jr}$ , on the x and y axes of Figure 4:

$\displaystyle p_{\xi kr}\propto\exp\left\{-\ \frac{1}{2\psi_{jr}^{2\ }}\left[k-\left(\xi\ +\ \phi_{jr}\right)\right]^{2}\right\}\$

(7)

where $\boldsymbol{\phi}_{jr}=\textbf{Y}_{jr}\eta$ is a linear model for rating bias for items and with design matrix $\textbf{Y}_{jr}$ of dimensions $(RJ)\times(R+J)$ and $\eta=(\phi_{1},...,\phi_{R},\eta_{1},...\eta_{J})^{T}$ for $R$ raters and $J$ items, as parameterized in Mariano and Junker (2007). Correspondingly, we update $\ln{\psi_{jr}^{2}}=\textbf{Y}_{jr}(\ln{\tau^{2}})$ where $\ln{\mathbf{\tau}^{2}}=(\ln{\psi_{1}^{2}},...,\ln{\psi_{R}^{2}},\ln{\tau_{1}^{2}},...,\ln{\tau_{J}^{2}})^{T}$ . The complete rater estimates from these models are displayed in Figure 10. The Bayesian estimates were calculated via Markov-chain Monte Carlo (MCMC) simulation using Gibbs sampling across four chains using JAGS (Plummer, 2003) in R using very weakly-informative priors, converging with $\hat{R}<1.1$ for each parameter. A structural plate diagram and JAGS code for the full extended model can be found in Appendix G.

Individual annotator tendencies and behaviors can be measured and indiciate significant differences. The vertical dashed lines on the graphs in panels (d) and (e) in Figure 4 represent 0.5 standard deviations of difference for individual raters from the mean. GPT models show significantly different rater behavior. Implications: even tasks where there is minimal overlap of observations to individual raters, behaviors can still be modeled and removed. This allows for improved curation of datasets and model selection.

Disentangling individual rater biases further, across sensitive attributes, can provide a measure of fairness for labels and identify raters (human or model) that display discriminatory biases. Variables representing a sensitive attribute, $\varsigma$ (e.g., race/ethnicity, gender, age, etc.) should be independent of observed score $X_{soijr}$ given the true score $\xi_{soij}$ if ratings are fair: $X\perp\varsigma\Rightarrow P_{\varsigma=a}(X_{jr}|\xi_{j})=P_{\varsigma=b}(X_{jr}|\xi_{j}),\forall a,b$ . In the notation used for disentangling rater effects, there should be no difference in variation in scoring from rater $r$ on item $j$ is fair with respect to attribute $\varsigma$ given $\varsigma\perp\xi$:

$\displaystyle P[X_{soijr}|\xi_{soij},r,j,\varsigma_{i}]=P[X_{soijr}|\xi_{soij},r,j]$

(8)

To measure a rater’s item-level fairness with respect to some sensitive teacher attribute, $\varsigma$, the rater parameter vectors are easily updated where $\phi_{jr\varsigma}=\textbf{Y}_{jr\varsigma}\eta$ is now a linear model for rating bias for items and with $\textbf{Y}_{jr\varsigma}$ is a design matrix of dimensions $(RJ\Sigma)\times(R+J+\Sigma)$ and $\Sigma=\{B,W\}$ for Black and White self-identified teachers respectively. In this case, where $\varsigma_{i}\in\{B,W\}$, we can update the vector explicitly to illustrate those values $\eta=(\phi_{1_{B}},\dots,\phi_{R_{B}},\phi_{1_{W}},\dots,\phi_{R_{W}},\eta_{1_{B}},...\eta_{J_{B}},\eta_{1_{W}},...\eta_{J_{W}})^{T}$ for $R$ raters, $J$ items, , and , $\ln{\psi_{jr\varsigma}^{2}}=\textbf{Y}_{jr\varsigma}(\ln{\tau^{2}})$ is similarly updated such that $\ln{\mathbf{\tau}^{2}}=(\ln{\psi_{1}^{2}},\dots,\ln{\psi_{R}^{2}},\ln{\tau_{1}^{2}},...,\ln{\tau_{J}^{2},\tau_{B}^{2},\tau_{W}^{2}})^{T}$.

By approaching the estimation this way, where $\phi_{jr\varsigma}$ is estimated as a parameter, we disentangle contributions to rater scores based on teacher race. This simplifies the task of evaluating for fairness using the metric of group independence, $X\perp\varsigma$, where we can directly calculate $P[X_{soijr}|\xi_{oij},\phi_{jr\varsigma},\varsigma_{i}]=P[X_{soijr}|\xi_{oij},\phi_{jr\varsigma}]$. Thus, $X\perp\varsigma\measeq\phi_{B}-\phi_{W}\approxeq 0$.

When estimated, less than 1% of parameter estimates had $\hat{R}\geq 1.1$, whose differences in posterior distributions have no material effect on results or discussion; all rater-item-specific 95% credible intervals for biases are represented as horizontal lines in Figure 4, in panel (e). Appendix G has full JAGS code used for the formula specification for all items and dimensions, including initial value parameters. Additionally, a plate diagram for MCMC modeling can be found in Figure 9.

Racial bias at the individual rater level is significiantly measurable. The GPT model families show a negative bias trend against Black teachers relative to White teachers on most items, as seen in the comparison of those models across panels (d) and (e) in Figure 4. Potentially more precisely, GPT models’ rating centrality seemed to diminish when rating Black teachers, especially with the "reasoning" model, adding evidence that these foundation models may be sensitive to linguistic differences found in African-American English (AAE) (Hofmann et al., 2024b; Fleisig et al., 2024), possibly due to historical data or models’ relative unfamiliarity with AAE Rickford and King (2016). These results alone should give pause to edtech developers relying on prompt-engineering of foundation LLMs, as subtleties in biases exist in very complex tasks. Additionally, it is not just GPT models showing biases. For some types of items, such as negatively worded items, individual human rater effects could be detected where abnormal rater biases, either positive or negative, towards teachers with some sensitive attribute.

Overall, encoders displayed much less bias than humans. However, while not as severe as the GPT or human biases, the encoder models did not avoid issues of racial bias. On the worst performing item for both human and encoder models, MGEN, all of the encoder models found spurious relationships in some language feature while overfitting with a negative bias against Black teachers. The reasons are likely to do with label sparcity and underrepresentativeness across label categories: with so few examples of ratings in the higher categories in the training dataset, overfit on a biased sample was not adequately controlled for, showing a microcosm of alignment to poor data that GPT exhibits in macrocosm. Fortunately for the encoders, many earlier data had already suggested that neither the models nor humans (see Appendix F.1 and Hill et al. (2012b)) could sufficiently distinguish between the item’s categories.

Implications: even tasks where there is minimal overlap of observations to individual raters, bias can still be modeled and removed. This allows for improved curation of datasets and model selection. The techniques can be used for evaluation of biases from given populations.

A Decision Study (D-study) estimates how reliabilities of ratings could improve by adjusting measured facets of variation, much like Ho and Kane did to motivate the case study. To estimate the reliability in a human-in-the-loop scenario, multiple g-studies and d-studies would need to be constructed to combine the variance contributions across a set rater families, $\mathbb{F}$. For this work, only two different types of families are consider in each d-study, and one of them will always be human, as automated rating models, even high-performing Encoders, are not yet ready to produce ratings independent from human confirmation. For a human-in-the-loop decision study, $\mathbb{F}$ would consist of families $\mathbb{f}$ that have humans only and models only, and a combined human-model family. For a $(S:O:i)\times R$ study estimated dependability of ratings provided to teachers $i$ on item $j$, $\tilde{\Phi}_{j}$ is, in the joined "universe" $\mathbb{F}^{\prime}$ where estimations are represented by $\mathbf{K}$, the collection of unique parameterizations and estimates, $\varkappa$, for the facets of variance in each D-study:

$\displaystyle\widetilde{\Phi}_{j,{\mathbb{F^{\prime}_{\varkappa}}}\sim\mathbf{K}}=\frac{\sum_{\mathbb{f}}^{\mathbb{F}}{\sigma^{2}(i_{\varkappa})}_{j\mathbb{f}}}{\sum_{\mathbb{f}}^{\mathbb{F}}{\sigma^{2}(i_{\varkappa})}_{j\mathbb{f}}+{\sigma^{2}(\Delta_{\varkappa})}_{j\mathbb{f}}}$

(9)

where the summations in Equation 9 combines the variation across the familial "universes", indexed by $\varkappa$, of different rater families in $\mathbb{F}$ and ${\sigma^{2}(i_{\varkappa})}_{j}$ and ${\sigma^{2}(\Delta_{\varkappa})}_{j}$ represents the "universe" variability for teacher $i$ and the absolute error for dependability, respectively, at the teacher-year-level ($i$) across the combined parameterization set $\mathbf{K}$. Structurally, Equation 9 shares similarities with the two-stage ICC calculation of Eq. 12. These values are represented in the ratio for calculating dependability, $\Phi_{j}$, as found in Equation 3 ${\sigma^{2}(\Delta)}_{j}\equiv\nu_{o:ij}+\nu_{s:o:ij}+\nu_{irj}+\nu_{rj}+\nu_{s:o:irj}$. The absolute error for a rater family ($\mathbb{f}$) indexed by $\varkappa$ across any permutation of decision values in this study:

$\displaystyle\sigma^{2}(\Delta_{\varkappa})$

$\displaystyle=\frac{\sigma^{2}(r_{\varkappa})}{n_{r_{\varkappa}}^{\prime}}+\frac{\sigma^{2}(o_{\varkappa}:i)}{n_{o_{\varkappa}}^{\prime}}+\frac{\sigma^{2}(r_{\varkappa}i)}{n_{r_{\varkappa}}^{\prime}}+\frac{\sigma^{2}(s_{\varkappa}:o_{\varkappa}:i)}{n_{s_{\varkappa}}^{\prime}n_{o_{\varkappa}}^{\prime}}+\frac{\sigma^{2}(s_{\varkappa}:o_{\varkappa}:ir_{\varkappa})}{n_{s_{\varkappa}}^{\prime}n_{o_{\varkappa}}^{\prime}n_{r_{\varkappa}}^{\prime}}$

(10)

where the decision values vary across design facets and whose contribution is weighted by the combined count $n_{k}^{\prime}$ of a given facet $k$ for ratings generated only by the family indexed by $\varkappa$, $n_{k_{\varkappa}}$ and those facets, if any, shared between families, $n_{k_{\mathbb{F}^{\prime}}}$: $n_{k_{\varkappa}}^{\prime}=n_{k_{\varkappa}}+n_{k_{\mathbb{F}^{\prime}}}\forall k\in\{s,o,r\},n_{r_{\mathbb{F}^{\prime}}}=0$. These distinct sets of parameter values for each design study are represented in Equation 9. For human-in-the-loop only use cases, $\varkappa_{\text{HIL}}$, the value $n_{k_{\mathbb{F}^{\prime}}}$ represents those sources of variation that are shared between rater families, and for a model family $\mathbb{f}=\mathbb{m}$, where there would be no observations made by a model without a human, the model would not have any independent observations $n_{o_{\mathbb{m}}}=0$. To represent these $n$ values where a human $\mathbb{h}$ observes a classroom for 15 minutes¹²¹²12For the MQI instrument, observation segments are 7.5 minutes long. with a model and where a single model $\mathbb{m}$ continues to observe for the remainder of the class (an additional 45 minutes), $\mathbf{K}_{n\in\varkappa_{\text{HIL}}}=\{n_{o_{\mathbb{m}}}=0,n_{o_{\mathbb{h}}}=0,n_{o_{\mathbb{F}^{\prime}}}=1,n_{s_{\mathbb{m}}}=6,n_{s_{\mathbb{h}}}=0,n_{s_{\mathbb{F}^{\prime}}}=2,n_{r_{\mathbb{m}}}=1,n_{r_{\mathbb{h}}}=1,n_{o_{\mathbb{F}^{\prime}}}=0\}$ and where the variance components are solved similarly to the coefficients of Eq. 1.

Estimates of impacts of model use can be reconstructed from measurable variances. The estimates for $\widetilde{\Phi}_{j,{\mathbb{F^{\prime}}}}$ are in Figure 4 panel (f) with complete results for all items in Figure 14. As conducting actual human annotated classroom observation ratings is immensely expensive, the decision study analyses of Section 5.6 offer methods for estimating the improvement gained by using a model or model family. Parameterizing the decision conditions to reflect "human-in-the-loop" scenarios can even offer insight into whether the variation offered from automated ratings adds or detracts from human rating quality, offering a means of estimating research questions before more expensive trials.

Constructs that are relatively infrequent, such as LANGIMP, could greatly benefit automated ratings, since sufficient human observations for identifying that construct would be expensive. Having encoder models listen in for three entire classes yields reliabilities for that construct that are twice that of the combined efforts of multiple human raters stopping by a teacher’s classroom 10 times, fifteen minutes each time—a net savings of two hours for the principal and a potential savings of over 10 hours if such a level of reliability were desire and were these trends to continue. Implications: Not all variance contributes equally, and its careful deconstruction and reconstruction can anticipate future effects before setting up more expensive studies.