Search papers, labs, and topics across Lattice.
VersaDB is a high-performance database specifically designed to unify and optimize access to multimodal AI datasets, addressing the inefficiencies of existing frameworks. By implementing a page-based storage system and B+ tree indexing, it significantly enhances data processing speeds, achieving up to 5.35x acceleration in experiments. This unified approach not only simplifies data management but also ensures consistent performance across various parallelism levels, making it a critical tool for AI researchers and practitioners.
Achieving up to 5.35x faster data processing for multimodal AI datasets could revolutionize how researchers handle and access diverse data formats.
The AI field has been rapidly developing, leading to the emergence of a large number of AI training datasets of various types. These datasets contain different modalities, including text, images, audio, etc., and may come in various data storage formats. With the advancement of AI hardware, AI computation units like GPUs, TPUs, and NPUs can greatly accelerate the training speed of AI models, which in turn increases the demand for faster data processing. When using existing AI processing frameworks to handle datasets with different modalities and storage formats, processing speeds may be suboptimal due to issues such as data layout and the way users handle the data. Therefore, using a unified database to store multiple data formats can better manage and optimize data access. In this paper, we introduce VersaDB, a database designed specifically for AI datasets with various modalities. We implemented a page-based storage system, separating structured and unstructured data. Additionally, we generated B+ tree-based index files to accelerate data access. VersaDB supports automatic sharding and maintains a hierarchical metadata management system, with corresponding metadata maintained at the page, shard, and global levels, forming the foundation for the efficient operation of the database. We also focused on ease of use by providing APIs for directly converting datasets into VersaDB, as well as APIs for converting popular AI data storage formats (e.g., CSV, TFRecord, .bin) into VersaDB.Our experiments show that using VersaDB can achieve up to 5.35x acceleration and maintain consistent performance across different parallelism levels.
