Search papers, labs, and topics across Lattice.
This paper analyzes conflicts and dilemmas faced by LLMs, modeling their decision-making process as a priority graph where nodes represent instructions/values and edges represent context-specific priorities. The analysis reveals that achieving a unified and stable LLM alignment is challenging due to the dynamic and inconsistent nature of the priority graph across different contexts. The authors propose a runtime verification mechanism to enhance robustness against priority hacking, while acknowledging the inherent limitations in resolving philosophically irreducible ethical and value dilemmas.
LLM alignment is fundamentally challenged by dynamic, context-dependent value hierarchies that can be manipulated via "priority hacking," suggesting current safety measures are more brittle than believed.
As Large Language Models (LLMs) become more powerful and autonomous, they increasingly face conflicts and dilemmas in many scenarios. We first summarize and taxonomize these diverse conflicts. Then, we model the LLM's preferences to make different choices as a priority graph, where instructions and values are nodes, and the edges represent context-specific priorities determined by the model's output distribution. This graph reveals that a unified stable LLM alignment is very challenging, because the graph is neither static nor necessarily consistent in different contexts. Besides, it also reveals a potential vulnerability: priority hacking, where adversaries can craft deceptive contexts to manipulate the graph and bypass safety alignments. To counter this, we propose a runtime verification mechanism, enabling LLMs to query external sources to ground their context and resist manipulation. While this approach enhances robustness, we also acknowledge that many ethical and value dilemmas are philosophically irreducible, posing a long-term, open challenge for the future of AI alignment.
