Ashish Shubham
AI traded accuracy for ease of use
In this interview, Ashish Shubham, VP of Engineering and Engineering Fellow at ThoughtSpot, explains how AI is changing business intelligence. He describes why a simple question like "what was my revenue last year?" is still hard when a database has eight date columns, how ThoughtSpot scopes memory to users, departments, and organizations without ever sharing it across customers, and how it keeps customer data safe with user-level permissions and zero-retention LLM contracts. He argues that moving from deterministic queries to a non-deterministic AI layer made accuracy the new bottleneck, walks through ThoughtSpot's offline and online evals, and discusses how AI is reshaping hiring, sprint expectations, and the anxiety engineers feel about keeping up.
Ashish Shubham is Vice President of Engineering and an Engineering Fellow at ThoughtSpot, the AI-powered analytics company used by some of the world's largest enterprises.
Ashish still calls himself a software engineer first. He is one of ThoughtSpot's earliest engineers and has spent more than a decade designing many of the platform's foundational components. He led the creation of ThoughtSpot Embedded and now helps set the architectural vision for Spotter, ThoughtSpot's AI agent, and the company's frontend organization.
“If you see that your colleague is also facing the same thing, then it feels like, it's just not me. It's the model's gaps.”
Ashish's core argument is that AI made analytics far easier to use but harder to trust. Putting a non-deterministic layer between a user's question and the answer turned accuracy, not language parsing, into the bottleneck, which is why his team now spends most of its time on evals.
"Historically, BI has been deterministic. You ask a question, we give you an answer," Ashish explains. "We had to take two steps back on accuracy for this ease of use."
The conversation
In this conversation: Josh Rubin (Host, CTO Studio) and Ashish Shubham (Vice President of Engineering and Engineering Fellow, ThoughtSpot).
I like to say I work as a software engineer. That's what I've been doing most of my life, but I work as a Vice President of Engineering and Engineering Fellow at ThoughtSpot.
And what is ThoughtSpot?
ThoughtSpot is data analytics software. We build analytics software used by some of the biggest enterprises in the world. We use AI and agentic analytics these days, as everyone does, to basically help users get insights from their data.
So it's a BI tool, ultimately.
You can say that, but now the lines are blurring. Traditionally, BI has been working on structured data, like data which is in your warehouse. But now a lot of data context or information is in unstructured or semi-structured data which is lying in your apps: your Google Docs, your Slack, your Jira, your Salesforce. So now we are also connecting to these sources to get the insight to you.
Is that data and information something that was only unlocked with these new agentic AI tools, generative AI, LLMs?
You can say kind of. They helped merge the two data sets together. Earlier, you would have separate systems of record which would query these data sets, right? You would have a BI tool like ThoughtSpot to query data in your data warehouse, and you would just use Salesforce to get data out of Salesforce. Salesforce always had APIs to read data out of Salesforce, but there was no way to actually marry these together in a cohesive way for someone to understand in real time. And with AI, we are able to do that: to present insights and ground your insights in data from various sources.
So I guess that's the deeper question here. You've been at ThoughtSpot for what, a decade at this point? So how big of a difference is there in your analytics platform, in these data layers that you're putting together, between five years ago, when you're obviously using machine learning and finding insights, versus today?
ThoughtSpot was always about providing data insights to the end user. Traditionally, analytics software was built for analysts, where there would be an analyst who would create reports, which would be consumed by your product managers, your CXOs, executives, or business users, as we call them. But ThoughtSpot was always about making it more self-service, giving the tool to the business users directly so they could query their data. So we've been doing search-based analytics for the last 10 years, actually 12 years since the company started. You could type a question, right? "Hey, what's my revenue in Orange County been over the last five years, year over year, or quarter over quarter?" And you would get a chart of how your revenue has been trending. AI has just made that mission easier for us. We are now able to understand the user's questions more effectively and also present them insights more effectively, while getting data out of multiple sources, like the data warehouse or their Salesforce or their Slack or even their Google documents.
If I'm understanding you correctly, you've been dealing with natural language querying for a while.
Yes.
But now you've been able to offload some of that understanding of the user's natural language. I imagine you were spending an inordinate amount of time as developers just building the tools to parse that language before the advent of these LLMs.
Yes, exactly. We had these BERT models in the past which would be used to understand the user's intent, and then you would have to tune them to actually know what the user is saying. I'll give you an example. If you say, "What was my revenue last year?" and you just put any AI model on top of it and say, "Okay, what does the user mean?", it can tell you, yes, the user means revenue from last year. But there are eight date columns in your database. So which one of those eight columns does the user mean when they say last year? Is it the booked date, the commit date, the ship date, the order date? Which date, last year? So that disambiguation is the key part. Even though we use AI models now, and the LLMs are really good at understanding intent, this disambiguation is still the piece we need to figure out. And having the context of the conversation, right? What does this user usually mean when they say last year? Maybe I had a question before this one which hints me towards finding the right date column. Those sorts of things become easier with LLMs now.
I imagine you're also building context and memory around both the users at large, but also individual users.
Yes.
How this person queries the date.
Yes, yes. And memory is the most important piece here, because that's what makes the system better tomorrow than it is today, right? You just gather this context over time. So we scope our memory. There's user memory, which is the most common sort of memory. If you use GPT or Claude, you'll just have user memory. But we also need to have organizational memory, which is memory shared across the org, or department-specific memory, which is shared within that department. When people are creating memories, we try to scope each memory item into one of these scopes. Is it user specific or org specific? Because if I'm teaching the system something, I want my colleague to benefit from what I taught the system. So that organizational memory piece helps there a lot.
Is there crossover not just from user to user or org to org, but writ large across all of your clients, like learnings that you're tuning in the moment?
Yeah, that is a compliance concern. We cannot just do that, right? Just take memories from someone and put them in another org.
So compliance and IP, that was going to be my next question. When you're using a BI tool, obviously people are sharing or giving access to proprietary data for y'all to parse. You're doing that in a compliant fashion, but what does that mean? How do you protect that data? How are you keeping this stuff safe, is the easiest question.
Yeah. So there are various levels there. I'll start from when the data is extracted from the data warehouse. When we extract the data from the database, we only extract the data this user is permitted to see. Within an org itself, the CEO may have broader permissions to see the data than some engineer on the team. So whenever our agent is accessing the data, it always carries the identity of the user. It's always accessed as if the user themselves is accessing the data. So when we are getting the data out of the system, it is always the data the user has access to. It's never something beyond that. Now, once this data is in our domain, in our system, we will have to send some part of this data to an LLM for the LLM to give insights. So we have contracts with the LLM providers where we make sure they don't train their systems on this data, and it's basically a zero retention policy.
Where is the data hosted? Increasingly I'm hearing that we've gone from a cloud-based architecture to people wanting to host their data on prem. Are you providing some of those services?
Yeah, so actually that goes full circle. When ThoughtSpot started 10 years ago, we were shipping boxes of our software with hardware to our customers. It sounds unbelievable today, right? But that was because customers were paranoid about their data leaving their premises. ThoughtSpot had big banks and big retail customers, so we would just ship compute boxes, with storage, memory, and compute, to these customers' data centers. They would plug them in, and that's how our software would run on top of their data.
Are you back in the hardware game now?
No. What we are exploring, and we are not there yet, is going into the customer's own cloud installation and being plugged in there, rather than being a SaaS service.
So they're on an AWS server that's cut off from everything else, and you're inserting yourself within that architecture.
Yeah. Although when it comes to cloud providers, there are not a lot of choices, right? You have AWS, GCP, or Azure. These are the major ones. So we are already working within these cloud providers. If a customer is already in AWS, our system is also in AWS, and we are able to dedicate our stack to run for that customer. But that still means it's installed and deployed by us. So a step ahead in that journey is that the system is deployed within the customer's perimeter.
There are a lot of conversations happening right now about cost and going with frontier models. Obviously you can plug into Claude and the frontier models here, but they might not be necessary for some of this data. Are any of your customers seeking out more open source options?
Yeah. And we actually make it really open anyway. We have this thing called bring your own model, where a customer may have their own open source model which they have hosted themselves. We can connect to that, and we will use it for our LLM inferencing. We give them the choice. If they want to use OpenAI, Claude, Gemini, or any other open source model, they can.
Are you seeing any signal of more people moving into open source versus frontier models, or is it still too early?
People want value. They want to see the system working with high accuracy, and they want to be able to trust the insights generated from the system and make business decisions on top of them. Once that is there, then they will start optimizing on cost. So right now we are making sure that people are getting that value, and for that we use the best model possible, and people don't mind that.
How do you understand, and then communicate to your customer, when you've achieved that level of fidelity? "We think we've got it tuned correctly, your data has been spot checked, your data is true. Now it's time to get your costs in line."
Yeah, so it's an evaluation scheme, right? I'm sure...
Is it continuous? Is there a specific time?
So there are two sorts of evaluations. I'm sure many people sitting here will have talked about evaluations before me, because it's such a popular thing these days, and it's so important. The first and most common is offline evaluations. You have fixed evaluation sets which you run on top of your system on a periodic basis, just to make sure the system hasn't regressed, and also to see how your system is improving over time. The other part of evals is online evals, where you run your evals on live customer queries: how is the system behaving on live customer queries? That's really important, and that is what gives you the best confidence in the value. That is the data you can present back to your customer: this is how the system has been doing.
Where does the human layer exist in your eval process?
Humans are building the evaluation sets. We have a curated set of hundreds of questions we run every other day to make sure our system doesn't regress, and that evaluation set is built by humans.
It's built by humans, but I imagine at this point it's also informed by AI. "Give me a good eval set," and then it's vetted.
Yeah, we can review it with AI, and maybe it is also seeded by AI to begin with. And there is also a lot of literature. It's not just AI, right? When it comes to SQL queries, or getting insights out of data, there is a lot of literature and a lot of research. One of the most popular sets for evaluating any SQL system, or any querying system, these days is called Spider. That evaluation set is one of the most popular ones. So you basically pick some of these standard evaluation sets, tune them to meet your use cases, and then leverage them as much as possible.
Where have you found the bottleneck in your process moving to? Like we talked about, five years ago the huge bottleneck was around parsing human language. Where is it now?
It's actually achieving accuracy.
That's a great place to spend your time, frankly.
Yeah, that's where we spend most of our time right now, making sure our system is accurate. Historically, BI has been deterministic. You ask a question, we give you an answer. There is no sense of doubt that the answer we gave matches the question you asked, because it was very deterministic. But with AI, that determinism cannot be assumed now. So that is the shift in the industry: going from deterministic querying and answering to this non-deterministic layer of AI in the middle, which sits between understanding your intent and firing the right queries to the system to get back the result. So we are basically two steps behind there. We had to take two steps back on accuracy for this ease of use. We traded off accuracy for ease of use, at least in the BI world.
It's because we've inserted misunderstanding into the mix where it wasn't. If you could only ask a certain set number of queries before, arguably two plus two equals... But now there's a level of interpretation the AI is bringing to the mix: it gave me an answer, but that's actually not what I meant. Which makes sense of why a lot of people are saying engineers need to understand the whole of the business and the user's use case, because it's intent that matters, and how you're engaging that matters. How has this sea change affected how you hire and build the kinds of engineers and teams that can solve for these problems?
The hiring has been changing, from the interview process to the kind of people we want on our team. Most people have been talking about one person wearing multiple hats, right? The engineer has to also think like a designer. The engineer has to also think like a product manager. And similarly, the product manager has to think like an engineer and a designer as well. So these lines are blurring a bit. As an engineer, I have great power now, using these coding tools to write my code so quickly. With that power, I get the responsibility of using these tools to build the right thing for my user, and making sure I am researching and doing my iterations on coming up with the right plan for my output.
Has it changed how you think when you're going into whatever we're calling sprints now?
Not fundamentally. At the end of the day, you are trying to build a product which serves the needs of your user, and sprint by sprint you are going towards that goal. So it's not fundamentally changing the goal of product building. But it tactically shifts some things. You can achieve more in a sprint now than you could have achieved three years ago. I would say three weeks ago, because it's changing so fast. But it also comes with expectations. I feel the expectations of software engineering are also changing. The leadership, or your system, now expects you to actually be more productive. So there is also a level of anxiety it brings to software engineers: am I being productive enough, am I using these tools enough? So that's the other trade-off.
Is that anxiety useful or destructive, in your experience?
With any anxiety, it has to be balanced. Some sort of anxiety is good, because it pushes you to become better. But sometimes this anxiety can morph into FUD, and by FUD I mean fear, uncertainty, and doubt. A lot of times we hear insane stories. Some time ago there was a Spotify news article where, during a Wall Street earnings call, they said, "Hey, our engineers now commit code while they are commuting on the metro, using their phone." I don't think that's true. But these kinds of episodes put unnecessary anxiety onto you: are we lagging behind? So there has to be a healthy balance. You have to sometimes see through some of these over-glorified events and see where the reality actually is.
How are you maintaining a healthy balance of anxiety with your peers and colleagues at work?
I think it's the community. The more you talk amongst engineers, the more you talk about how you yourself are using these tools, helping others use the tools effectively, and learning from each other's experience. "This is good for doing certain things. I use this model for certain things, and this worked well for me, or this did not work well for me." Having these sorts of conversations just instills some confidence in you. If you saw something while you were building, and you see that your colleague is also facing the same thing, then it feels like, it's just not me. It's the model's gaps, or the system's gaps. And it works vice versa as well. There might be something my colleague is able to do which I can learn from them. So I think having this community of engineers talking more to each other helps a lot. If you are a single engineer working by yourself, not talking to anyone about this shift happening in the industry, I feel you might get really anxious, and you might not be as productive as you could be.
So AI is forcing us to need other people.
I think it's about any shift. Whenever some industry is going through a shift, as individuals, we need a community to go through that shift. As an individual you might just feel overwhelmed sometimes with the rate of change. And that community can be online. You could be on Reddit threads talking to people. It's not necessarily someone who needs to work at your company.
No one has ever recommended Reddit for your mental health. I suppose there could be some use cases for that.
Yeah, I would not recommend LinkedIn for mental health. But Reddit is far better than LinkedIn when it comes to that.
I agree with you. You can get some more real conversations.
Yeah, it's anonymous, and people are generally not always trying to sell.
Be part of the
conversation.
Whether you're a CTO who wants to be featured, a company looking to sponsor, or an engineering leader wanting a seat in the room — there's a place for you here.