Yiming Lin (UC Berkeley): AI-Powered Data Systems for Multimodal Analytics

We live in a world overflowing with data, and the emergence of AI, such as Large Language Models (LLMs), is revolutionizing data analytics. However, directly using AI to process massive and complex data is neither effective nor scalable.

In this talk, I introduce my work on building database systems powered by AI to analyze and process multimodal data at scale, focusing on tables and documents. On one hand, when analyzing tables, AI is often used to prepare data, such as cleaning, enriching, or synthesizing data prior to query processing. This becomes prohibitively expensive when the data scale is large. To support scalable analysis over expensive data ingestion, my work leverages the fact that not all data are needed to answer a query and explores a set of techniques to reduce AI operations unnecessary to analytics by optimizing the query engine in the database. On the other hand, when analyzing documents, current systems treat them as plain text and ignore underlying structures, leading to limited accuracy and performance. In this regard, we exhaustively identified three document structures that encompass most real-world documents we have encountered, and we designed tools and systems to extract their structures and leverage them for accurate and efficient document analytics. Finally, I’ll share my vision for building data systems for multimodal analytics, including aspects of trustworthy systems, interaction with hardware, and co-optimization among different data modalities.