Statistical Framework not only for Precision Livestock Management and Ethology, but for performing Concordance Analysis and MSA for Scoring Data

This repository contains a comprehensive, open-source statistical framework developed at the Department of Agricultural and Biosystems Engineering (University of Kassel). It is specifically designed to evaluate the reliability, consistency, and diagnostic accuracy of human observers and sensor systems in ethology, precision livestock farming, and beyond. Because human observation inevitably contains cognitive noise, this toolset provides robust mathematical methods to validate these "imperfect gold standards".

The framework is divided into two highly specialized modules

Module 1: Concordance Analysis Framework:
This framework is designed for the broad evaluation of inter-rater and intra-rater reliability for scoring data. It handles categorical, ordinal, and metric data, offering robust chance-corrected metrics (including Cohen’s Kappa, Gwet’s AC, Krippendorff’s Alpha, and PABAK), as well as metric equivalence tests (Deming Regression, Bland-Altman Analysis, and TOST). To ensure robust statistical inference, it calculates asymmetric confidence intervals via percentile bootstrapping. Beyond global omnibus metrics, the framework conducts detailed pairwise analyses to trace specific rater disagreements and classwise analyses to pinpoint exactly which score categories carry the highest uncertainty. For in-depth graphical exploration, it implements Bangdiwala’s B Agreement Plot, providing an intuitive visual approach to identifying systematic diagnostic deviations and marginal imbalances.
Module 2: Attributive Gage R&R and "Noisy Labels" Framework:
This framework performs a deep Measurement System Analysis (MSA) specifically tailored for ordinal scoring data. It mathematically partitions human measurement error into Repeatability (intra-rater cognitive noise) and Reproducibility (inter-rater variation). Furthermore, it identifies systematic bias using dynamic control limits (Variability Charts) and automated algorithm-based diagnostics (e.g., contrasting Kendall’s W against Cohen’s Kappa). Finally, the framework aggregates the filtered rater data to export a "Global Consensus" dataset. These mathematically purified "Noisy Labels" are optimized as ground-truth data for training Machine Learning algorithms or validating technical sensor systems.

Technical Implementation & Open Science

Championing the principles of Open Science, the entire framework is built on R, a free and open-source statistical programming language. It strictly separates backend calculation from user configuration, requiring no advanced programming skills from the end-user. Utilizing the power of Quarto and LaTeX, the toolchain automatically translates the statistical outputs into fully formatted, publication-ready PDF reports. These dynamic documents contain all relevant cross-tabulations, advanced visualizations, and an automatically generated methodological decision-making guide, ensuring maximum transparency and full reproducibility for the scientific community.

Invitation for Collaboration and Error Correction

To conclusively validate the reliability and mathematical limits of these heuristics, further research is imperative. In particular, the Gage R&R tool must be systematically stress-tested through comprehensive Monte Carlo simulations across broad parameter ranges. This entails stochastically securing the behavior of the control limits and bias algorithms under varying scale widths, extreme shifts in prevalence, and different rater team sizes. The data simulator already implemented within the scope of this project provides an initial foundation for this in the spirit of Open Science.

Pending full simulative validation, the diagnoses derived from the framework should therefore always be interpreted as exploratory decision-making aids. They possess the potential to significantly enrich the critical discourse between ethologists and data scientists during the generation of ground-truth data, but they do not replace a professional plausibility check of the resulting machine learning dataset.

The present framework is not intended as a static final product, but rather as a dynamic, iteratively growing tool in the spirit of Open Science. Despite careful algorithmic implementation and extensive test-driven validation via stochastic simulations, unforeseen edge cases can always arise when analyzing complex field data. In particular, experimental heuristics - such as the transfer of control limits to ordinal scales or automated bias diagnostics - benefit enormously from practical application and stress-testing across diverse diagnostic contexts.

Therefore, users, researchers, and data scientists are explicitly invited to critically review the provided scripts and Quarto templates and to adapt them to their own specific research questions. Feedback regarding methodological inaccuracies, programming errors (bugs), or suggestions for functional extensions - such as the integration of alternative concordance measures or specific weighting matrices for novel animal welfare indicators - is highly welcome. Through this open, interdisciplinary exchange, the framework will be continuously refined in order to sustainably and collaboratively bridge the methodological gap between practical ethology and robust data science.