BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Information Systems Group - ECPv6.4.0.1//NONSGML v1.0//EN
CALSCALE:GREGORIAN
METHOD:PUBLISH
X-ORIGINAL-URL:https://isg.ics.uci.edu
X-WR-CALDESC:Events for Information Systems Group
REFRESH-INTERVAL;VALUE=DURATION:PT1H
X-Robots-Tag:noindex
X-PUBLISHED-TTL:PT1H
BEGIN:VTIMEZONE
TZID:America/Los_Angeles
BEGIN:DAYLIGHT
TZOFFSETFROM:-0800
TZOFFSETTO:-0700
TZNAME:PDT
DTSTART:20250309T100000
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0700
TZOFFSETTO:-0800
TZNAME:PST
DTSTART:20251102T090000
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DTSTART;TZID=America/Los_Angeles:20250716T110000
DTEND;TZID=America/Los_Angeles:20250716T120000
DTSTAMP:20260717T113726
CREATED:20250717T035358Z
LAST-MODIFIED:20250717T040900Z
UID:2245-1752663600-1752667200@isg.ics.uci.edu
SUMMARY:Michal Shmueli-Scheuer (IBM Research): GenAI Benchmarking and Evaluation
DESCRIPTION:Title: GenAI Benchmarking and Evaluation\nWednesday (July 16)\, 11 am – noon\, DBH 6011\, UC Irvine \nAbstract:\nIn this seminar\, I will present a practical perspective on benchmarking in the era of generative AI. We will begin with an overview of goals and motivations\, followed by key definitions such as benchmarks and leaderboards. I’ll review prominent existing benchmarks and highlight several core challenges that limit their effectiveness in evaluating today’s models. In response\, I’ll introduce a new methodology for benchmarking that focuses on flexibility\, reuse\, and deeper insights. This includes a closer look at ToRR\, a new table reasoning benchmark\, and Unitxt\, an open-source tool designed to simplify the creation and management of benchmarks—accompanied by a brief demo. Finally\, I will outline emerging directions in agentic evaluation\, pointing to what’s next in assessing increasingly complex AI systems. \nBio:\nMichal Shmueli-Scheuer is a Distinguished Engineer for AI Benchmarking and Evaluation at IBM Research. Her expertise spans Natural Language Generation (NLG) and Natural Language Processing (NLP)\, with a focus on evaluation\, data-to-text generation\, conversational bots\, scientific document summarization\, and affective computing. \nShe has authored over 40 academic papers in top NLP and AI conferences and journals. Michal is a co-organizer of the GEM-squared workshop on evaluation (ACL) and recently delivered the “Navigating the Modern Evaluation Landscape” tutorial at LREC-COLING. She will also deliver a tutorial on “Evaluating LLM-based Agents: Foundations\, Best Practices\, and Open Challenges” at IJCAI. \nActively involved in the NLP community\, she has organized multiple workshops and shared tasks. Michal earned her Ph.D. in Computer Science from the University of California\, Irvine\, in 2009.
URL:https://isg.ics.uci.edu/event/genai-benchmarking-and-evaluation/
LOCATION:DBH 6011
END:VEVENT
END:VCALENDAR