Search papers, labs, and topics across Lattice.
To diagnose syntax degradation that breaks downstream agents and code extractors, the authors introduce LatentMD, a 4,179-prompt benchmark that explicitly decouples semantic correctness from CommonMark fence-boundary compliance. Across 9 LLMs and ~37,600 generations, 38.0% of content-correct outputs were found to be boundary-broken, driven primarily by symmetric delimiter collisions (e.g., nested backticks or docstrings) rather than structural nesting depth alone. These syntactic breakages persist even under explicit prompt hinting, exposing an unmeasured failure mode in how current models handle structured text generation.
Content correctness masks widespread structural failure: 38% of otherwise accurate LLM outputs silently break CommonMark delimiter boundaries, crippling downstream parsers and execution pipelines.
Large language models (LLMs) increasingly generate Markdown that is consumed by renderers, agents, code extractors, and structured downstream pipelines. Yet existing evaluations often conflate content quality with format adherence, leaving Markdown boundary failures under-measured. We introduce LatentMD, a benchmark and evaluation protocol for diagnosing CommonMark-level fence-boundary failures in LLM-generated Markdown. LatentMD separates content correctness from boundary correctness, enabling detection of outputs that are content-correct but boundary-broken. The benchmark contains 4,179 prompts and a CLI for scoring arbitrary model outputs. Across 9 LLMs and roughly 37,600 generations, we find that Markdown boundary failures are widespread: 38.0% of valid main-grid outputs are content-correct but boundary-broken, with substantial boundary breakage under unspecified prompts and in a small human-authored validation set. Ablations show that failures are driven primarily by same-family symmetric-delimiter collisions rather than nesting alone, are only partially mitigated by prompt hints, and generalize to Python triple-quote docstrings while JSON remains robust as an asymmetric-delimiter control. LatentMD provides a reproducible diagnostic target for parser-sensitive LLM evaluation.
