AI Operations Specialist

Accountabilities: Identify, assess, and prioritize AI automation opportunities based on business impact, operational efficiency, feasibility, and expected value. Collaborate with process owners to map existing workflows, identify bottlenecks, and determine which tasks can be automated with AI and which require human intervention. Design, build, and implement AI-powered workflows, including prompt development, pipeline configuration, quality controls, and performance monitoring. Establish and maintain quality standards for AI-driven processes, ensuring outputs remain accurate, consistent, and aligned with business requirements. Develop evaluation datasets and test cases to measure AI performance using metrics such as accuracy, precision, recall, and sampling-based quality assessments. Evaluate and calibrate large language model (LLM) judges against human reviewer decisions to improve the reliability and consistency of automated evaluations. Partner with product teams to define launch-readiness criteria for AI features, generate performance evidence, and support go/no-go decisions. Translate complex or ambiguous policies into clear, actionable evaluation criteria that reviewers can apply consistently across different scenarios. Work with subject matter experts to develop labeling guidelines and produce high-quality, human-verified labels using real-world data. Coordinate large-scale data labeling and review operations with external vendor teams, validating their work through agreement metrics, sampling, and quality checks. Monitor deployed AI workflows for performance degradation, quality drift, and emerging issues, implementing corrective actions when necessary. Take ownership of assigned workstreams from initial planning and setup through implementation, quality validation, and ongoing monitoring. Document workflows, evaluation methodologies, lessons learned, and best practices to make solutions reusable and accelerate future deployments. Stay informed about evolving AI models, tools, evaluation techniques, and vendor capabilities, continuously improving existing solutions and retiring approaches that are no longer effective. Communicate findings, performance metrics, and recommendations clearly to business stakeholders, product managers, and engineering teams. Requirements: Between 5 and 7 years of professional experience in operations, program management, analytics, consulting, applied AI, or a related field, with demonstrated accountability for delivering measurable results. Hands-on experience using AI tools regularly and building an AI-powered workflow, automation, or analytical solution that has delivered tangible business impact. Strong understanding of performance measurement, including metrics such as accuracy, precision, recall, and sampling methods. Demonstrated ability to transform ambiguous, complex, or inefficient processes into structured workflows with clear criteria, measurable outcomes, and repeatable quality standards. Experience in quality assurance, auditing, review operations, guideline development, or comparable quality management activities. Strong written and verbal communication skills, including the ability to document processes clearly and collaborate effectively with business leaders, product managers, engineers, and other stakeholders. Ability to work independently, manage competing priorities, take ownership of projects, and make sound decisions with limited supervision. Strong analytical thinking and problem-solving skills, with a practical, results-oriented approach to evaluating AI performance and improving operational processes. Willingness to learn new AI evaluation methods, tools, and technologies as the scope of responsibilities evolves. A technical degree or formal technical background is not required. Previous experience designing AI evaluations or calibrating LLM judges is also not mandatory, as training will be provided. Preferred: Practical experience with SQL, Python, APIs, or other