AI Product Analyst

An AI product can look impressive in a demo. But how do you prove it''s actually intelligent, accurate and delivering value? That''s where you come in.We''re working with an established business making significant investments in AI and generative technology, and we''re looking for a commercially minded, technically strong AI Product Analyst to help answer the questions that really matter.Is the model producing accurate answers? Are users getting what they need? Is retrieval working properly? Are hallucinations decreasing? Are product changes actually improving the experience?This isn''t simply about building dashboards or reporting on usage. You''ll be evaluating the performance and quality of AI models and the products they power, helping shape what gets improved, developed and launched next. What will you actually be doing?Evaluating LLM and RAG product outputs, focusing on accuracy, groundedness, hallucinations, relevance and answer quality.Developing evaluation datasets, scoring frameworks and KPIs to measure model and product performance.Investigating why AI responses fail, why certain queries underperform and where improvements are needed.Using SQL and Python to analyse product behaviour, identify patterns and uncover actionable insights.Measuring the impact of model changes, retrieval improvements and new product features.Designing experiments, analysing user behaviour and identifying opportunities to improve the experience.Working alongside AI Engineers, Product ..... full job details .....