Scaling Policy Compliance Assessment in Language Models with Policy
  Reasoning Traces
Policy compliance assessment is a fundamental task of evaluating whether an
input case strictly complies with a set of human-defined rules, more generally
known as policies. In practice, human experts follow a systematic, step-by-step
process to identify violations with respect to specific stipulations outlined
in the policy. However, such documentation of gold-standard, expert-level
reasoning processes is costly to acquire. In this paper, we introduce Policy
Reasoning Traces (PRT), a form of specialized generated reasoning chains that
serve as a reasoning bridge to improve an LLM's policy compliance assessment
capabilities. Our empirical evaluations demonstrate that the use of PRTs for
both inference-time and training-time scenarios significantly enhances the
performance of open-weight and commercial models, setting a new
state-of-the-art for HIPAA and GDPR policies. Beyond accuracy gains, we also
highlight how PRTs can improve an LLM's ability to accurately cite policy
clauses, as well as influence compliance decisions through their high
utilization from the raw chains of thought.
68e39ec573e20ab57784218c
Joseph Marvin Imperial
68e39ec573e20ab57784218d
Harish Tayyar Madabushi
Scaling Policy Compliance Assessment in Language Models with Policy
  Reasoning Traces
Policy compliance assessment is a fundamental task of evaluating whether an
input case strictly complies with a set of human-defined rules, more generally
known as policies. In practice, human experts follow a systematic, step-by-step
process to identify violations with respect to specific stipulations outlined
in the policy. However, such documentation of gold-standard, expert-level
reasoning processes is costly to acquire. In this paper, we introduce Policy
Reasoning Traces (PRT), a form of specialized generated reasoning chains that
serve as a reasoning bridge to improve an LLM's policy compliance assessment
capabilities. Our empirical evaluations demonstrate that the use of PRTs for
both inference-time and training-time scenarios significantly enhances the
performance of open-weight and commercial models, setting a new
state-of-the-art for HIPAA and GDPR policies. Beyond accuracy gains, we also
highlight how PRTs can improve an LLM's ability to accurately cite policy
clauses, as well as influence compliance decisions through their high
utilization from the raw chains of thought.
Policy Reasoning Traces (PRT) enhance LLMs' policy compliance assessment by providing detailed reasoning chains, improving accuracy and policy clause citation.
Policy Reasoning Traces PRT LLM policy compliance assessment reasoning chains HIPAA GDPR