Responsibilities

About the Team Flow is ByteDance's AI-native innovation business team, focusing on developing products in emerging fields such as LLM. Our product portfolio includes Doubao (China) and Dola (Global). The Model Evaluation Team plays a key role within Flow, providing accurate and timely assessment signals for product iterations. We are responsible for the entire evaluation process, from designing methodology to executing evaluations, identifying issues, and proposing feasible improvement suggestions. Join us to explore innovative opportunities in AI, grow in a dynamic team environment, and create real value for users! Responsibilities - Evaluation Framework Establishment: Design evaluation frameworks for the Dola App including standards, test cases, and workflows. Create metrics that measure performance across key dimensions and enable data-driven decisions in existing and upcoming use scenarios - Human-Machine Evaluation Integration: Build workflows combining human expert and automated evaluation. Ensure automated processes align with human judgment, identify performance gaps, and increase testing coverage to supplement human evaluation - Evaluation Results Analysis: Interpret and analyze evaluation findings with depth, transforming nuanced case analysis into actionable insights. Develop data production and product design strategies aimed at enhancing model capabilities and/or user experience - Cross-functional Collaboration: Align product, engineering and evaluation teams across cultural and professional boundaries. Facilitate communication, resolve differences, and build consensus around overall evaluation methodologies - Evaluation Methodology Research: Stay at the forefront of LLM evaluation by researching industry-leading assessment methodologies and reports. Integrate insights with practical applications to continuously improve our evaluation framework

Qualifications

Minimum Qualifications - Bachelor's degree or above in Computer Science, Data Science, Business Analytics, or related fields - 3+ years of experience as a Product Manager, Business Analyst, or Operations Specialist in technology companies or other highly dynamic environment - Strong data analysis skills with experience in designing metrics for product performance - Project management capabilities with the ability to balance priorities and handle ambiguous requirements - Quick adaptation to new tools, technologies and concepts, with demonstrated learning agility - Genuine passion for AI/LLM technologies - Professional proficiency in both English and Mandarin is required for collaboration with key stakeholders and businesses based in China and across APAC. This collaboration is necessary to support the development of new technology and drive improvements to the company's platforms Preferred Qualifications - Experience in AI/ML product management, particularly with conversational AI - Hands-on experience with LLM evaluation methodologies - Background in developing metrics frameworks for measuring product performance - Excellent stakeholder management and communication skills

AI Product Manager (Evaluation) - Flow