This 21th Eindhoven Data Community meetup will feature two sessions focused on leveraging AI technologies for performance evaluation and operational efficiency. The first session will discuss the challenges of evaluating Large Language Models (LLMs) at scale, highlighting the use of LLM-as-a-Judge systems and the implementation of a scalable evaluation framework using Vertex AI and Gemini on Google Cloud. The second session will introduce Menu AI at Just Eat Takeaway, a solution designed to automate the transcription of restaurant menus, significantly reducing the time and effort required for this task. You will learn about the cloud architecture and multimodal models used for parsing menu images and extracting structured data. Overall, the event will showcase innovative strategies for enhancing AI application evaluations and improving operational processes.
Location: Skybar, Microlab Eindhoven, Kastanjelaan 400, Eindhoven.
Evaluating LLM applications at scale
As LLM deployments grow, evaluating their performance effectively and efficiently becomes of critical performance. Standard metrics struggle with the heterogeneity of LLM outputs, and manual expert review doesn't scale. LLM-as-a-Judge systems promise automation but require careful implementation to handle specific jargon and ensure alignment with human standards.This session dives into practical solutions for these evaluation challenges, grounded in a large-scale project evaluating a customer-facing chatbot handling over one million conversations per year. We will explore strategies for overcoming both conceptual hurdles (like judge alignment and context awareness) and technical bottlenecks (including cost optimization, data throughput, and robust API interaction). Learn how we leveraged the power of Vertex AI and Gemini on Google Cloud to implement a scalable, reliable, and insightful LLM evaluation framework.
Decoding Culinary Complexity: Transforming Menu Transcription Menu AI at Just Eat Takeaway
Step into the shoes of a team faced with the herculean task of transcribing restaurant menus by hand. Each menu, a labyrinth of culinary offerings, can take an excruciating 2-4 hours to decode, demanding unwavering attention to detail amidst a jungle of artistic fonts and intricate designs. Now multiply that by 1700, the number of menus that land on the desks of JET every month only in UK. This is the very pain point that Menu AI promises to alleviate, setting the scene for a transformative solution we're eager to share with you. Join us for an insightful session on how JET integrates restaurant menus into its platform. Our talk will delve into the intricacies of Menu AI, from cloud architecture to the parsing of pictures of restaurant menus and how it and augments the productivity of humans in the loop. We will also get under the hood on how to leverage the power of multimodal models and their vision capabilities on parsing photos, describing menus in structured data, and the importance of mapping relationships among menu items. Moreover, we'll share insights into the significant cost savings realized by the project in Customer Services Operation costs.
Program
- 17:00 – 18:00 Food
- 18:00 – 18:10 Welcome
- 18:10 – 19:00 Sander van Donkelaar: Evaluating LLM applications at scale
- 19:00 – 19:15 Break
- 19:15 – 20:00 Caio Benatti Moretti: Decoding Culinary Complexity: Transforming Menu Transcription with
- 20:00 - 21:00 Drinks
Sander van Donkelaar | AI/ML Engineer at Xebia Data
Sander is an AI/ML Engineer skilled in building AI products and platforms. Experienced across diverse industries, Sander has a proven track record of delivering innovative solutions At the core of Sander's expertise is the ability to translate complex business problems into tangible AI-powered solutions, driving efficiency, innovation, and data-driven decision-making for organizations.
Caio Benatti Moretti | AI consultant at Xebia Data
Caio holds a PhD in Computer Science and has been acting as a DS/MLE both in academia and industry since 2014. Currently working as an AI Consultant at Xebia, he created SlackGPT and is particularly keen on neural networks in its many forms and applications. His enthusiasm even led him to make a neural network fit inside a business card. Apart from practical experience, Caio has been giving seminars and trainings on how to empower businesses with LLMs from use cases to technical tooling. He is focused on how LLMs can augment human productivity and hence helping businesses to leverage novel technologies to achieve their goals.
