Search papers, labs, and topics across Lattice.
This paper introduces Alipay-PIBench, a benchmark designed to evaluate coding agents on realistic payment integration tasks, which involve selecting products, implementing client-server flows, and ensuring transaction consistency. The benchmark features nine product-specific projects with 18 task instances categorized into Basic and Advanced scenarios, and it employs a combination of deterministic checks and LLM-assisted assessments to evaluate performance. Results show that coding agents achieve a mean rubric pass rate (RPR) of 68.58% to 91.37%, with a notable 10.31 percentage point improvement when agents have access to the alipay-payment-integration skill, highlighting the benchmark's utility in diagnosing model capabilities in payment integration contexts.
Coding agents can significantly improve payment integration performance with targeted skills, achieving up to 91.37% success in complex scenarios.
Payment integration is a demanding repository-level software task: agents must select a suitable product, implement coordinated client-server flows, verify payment outcomes, and preserve consistency between transaction and business states. We introduce Alipay-PIBench, a benchmark for evaluating coding agents on realistic Alipay payment integration. It contains nine product-specific projects and 18 task instances, each organized into Basic functional-completion and Advanced risk-aware hardening scenarios. Scenario-specific rubrics support deterministic static, unit, integration, and end-to-end checks, supplemented by LLM-assisted assessment for semantic requirements. We evaluate six coding-agent models and report rubric pass rate (RPR). Under the with-skill condition, mean RPR ranges from 68.58% to 91.37%. Access to the alipay-payment-integration skill improves mean RPR by 10.31 percentage points on average relative to the without-skill condition, with gains varying across models, products, and scenarios. Method-level results distinguish source-level completion, executable payment behavior, and payment-domain requirements. Alipay-PIBench provides a controlled setting for diagnosing model capability and evaluating structured guidance in payment integration.
