Search papers, labs, and topics across Lattice.
This paper introduces the Rule Violation Score (RVS), a novel metric designed to evaluate the logical compliance of predictive models beyond traditional accuracy measures. By distinguishing between hard and soft rules, RVS provides a framework for assessing how well models adhere to predefined logical constraints, which is crucial in high-stakes domains like healthcare and finance. The evaluation across multiple benchmarks shows that models with similar predictive accuracy can significantly differ in their logical compliance, highlighting the limitations of conventional metrics in capturing model behavior.
Two models can achieve the same accuracy but differ dramatically in logical compliance, revealing a critical oversight in standard evaluation practices.
Machine learning models are predominantly evaluated through predictive performance metrics such as ranking quality, prediction error, or classification accuracy. While these metrics effectively quantify how closely predictions match the ground truth, they do not assess whether model outputs respect predefined logical or domain-specific constraints. In high-stakes applications, including healthcare, finance, and autonomous systems, logical consistency can be as critical as predictive accuracy, yet no standard metric captures this dimension. We introduce the Rule Violation Score (RVS), a complementary evaluation metric that quantifies the extent to which a predictive model respects a given set of logical rules, independently of predictive accuracy. RVS treats hard rules (strict constraints) and soft rules (statistical regularities) differently, can be evaluated on any dataset and on any predictive model expressed over a relational vocabulary, and can be computed using SQL queries that are automatically generated for Horn rules. Beyond evaluating models, RVS can also evaluate the logical consistency of training datasets and help identify poorly defined rules. We evaluate RVS on three benchmarks covering knowledge graph link prediction and relational regression, including rule-based, embedding-based, and neuro-symbolic predictive models. Our results demonstrate that two models achieving comparable predictive accuracy can exhibit substantially different levels of logical compliance, revealing differences in model behavior that standard metrics fail to capture.
