Is your AI product actually working? How to develop the right metric system
1 min read
Summary
In order to ensure an AI product is working well and serving its purpose, it is imperative to track its progress and success rate, argues Sharanya Rao in a piece for VentureBeat.
She recounts her experiences as a product manager for a machine learning (ML) product that contained both external and internal customers, and the difficulty of defining metrics that showed the product’s impact.
She advised that not tracking these metrics is akin to a pilot landing a plane without air traffic control, and also risks the creation of multiple subjective versions of the same ‘accuracy’ or ‘quality’ metric.
The first step to rectifying this, she says, is to figure out what you want to measure and track to inform your business decisions, then to define a set of sub-questions related to this, and finally to figure out methods for gathering this data.
She concludes by saying that this method is transferable to ML-based products, using specific examples such as a search function or generating listing descriptions.