Since the start At ChatGPT, a stampede of tech company leaders was chasing the hype: Everywhere you turned, there was another company touting its groundbreaking AI feature. But real business value comes from delivering product capabilities that matter to users, not just from using new technology.
We’ve achieved a 10x better return on engineering efforts with AI by starting with the core principles of what users need from your product, building the AI capability that supports that vision, and then measuring adoption to make sure it’s hitting the target.
Our first AI product feature was not aligned with this idea, and it took a month to reach a disappointing 0.5% adoption among returning users. After refocusing on our core principles of what users need from our product, we developed an “AI as Agent” approach and shipped new AI capability with 5% adoption in the first week. This AI success formula can be applied to almost any software product.
Waste of hype in a hurry
Many startups, like ours, are often seduced by the temptation to incorporate the latest technologies without a clear strategy. So, following the groundbreaking release of various incarnations of OpenAI’s Generative Pre-Trained Transformer (GPT) models, we started looking for a way to use Large Language Models (LLM) AI technology in our product. We quickly got our place on board the hype train with a new AI-driven element of production.
This first AI capability was a small summarization feature that used GPT to write a short paragraph describing each file that our user uploaded to our product. It gave us something to talk about and we created some marketing content, but it had no tangible impact on our user experience.
Many startups are often attracted by the allure of incorporating the latest technologies without a clear strategy.
We knew this because none of our key metrics showed a meaningful change. Only 0.5% of returning users interacted with the description in the first month. Furthermore, there was no improvement in user activation and no change in the frequency of user subscriptions.
When we thought about it from a broader perspective, it was clear that this feature would never change those metrics. Our core product value proposition revolves around analyzing big data and using data to understand the world.
Generating a few words about the uploaded file will not lead to any significant analytical insight, which means it will not do much to help our users. In our rush to deliver something related to AI, we have missed delivering actual value.
Success with AI as an agent: 10 times better return
The AI approach that has given us success is the “AI as Agent” principle that enables our users to interact with the data in our product via natural language. This recipe can be applied to any software product built on API calls.
After the initial AI feature, we ticked the box, but we weren’t satisfied because we knew we could do better for our users. So we did what software engineers have been doing since the invention of programming languages, which is get together for a hackathon. From this hackathon, we implemented an AI agent that acts on behalf of the user.
The proxy uses our own product by making API calls to the same API endpoints that our web front-end calls. It creates API calls based on a natural language conversation with the user, trying to fulfill what the user is asking it to do. Proxy actions appear in our web UI as a result of API calls, just as if the user had taken the actions themselves.