Search papers, labs, and topics across Lattice.
The paper introduces DEEP, a Docker-based platform for automated execution, evaluation, and statistical analysis of machine translation and optical character recognition models. DEEP addresses the challenge of reliably comparing system performance by providing a standardized environment for running dockerized models and scoring their outputs against references. The platform incorporates a clustering algorithm based on statistical significance testing to identify performance clusters and provides a visualization web-app for interpreting the results.
Stop manually running and scoring MT/OCR models: DEEP automates the entire evaluation pipeline, from execution to statistical analysis and visualization.
Comparative evaluation of several systems is a recurrent task in researching. It is a key step before deciding which system to use for our work, or, once our research has been conducted, to demonstrate the potential of the resulting model. Furthermore, it is the main task of competitive, public challenges evaluation. Our proposed software (DEEP) automates both the execution and scoring of machine translation and optical character recognition models. Furthermore, it is easily extensible to other tasks. DEEP is prepared to receive dockerized systems, run them (extracting information at that same time), and assess hypothesis against some references. With this approach, evaluators can achieve a better understanding of the performance of each model. Moreover, the software uses a clustering algorithm based on a statistical analysis of the significance of the results yielded by each model, according to the evaluation metrics. As a result, evaluators are able to identify clusters of performance among the swarm of proposals and have a better understanding of the significance of their differences. Additionally, we offer a visualization web-app to ensure that the results can be adequately understood and interpreted. Finally, we present an exemplary case of use of DEEP.
