Douglas Lawrence

I am a CTO at CLARA Analytics. They are a Silicon Valley insurtech company.

Right, so we build an intelligence platform. We sell our product to casualty claims insurance companies. That's primarily workers' comp and auto liability. If you're familiar with claims, they're managed by adjusters, the people that gather all the data about a particular claim and see it all the way through, and interact with customers. We can provide an intelligence layer that's powered by AI to help them predict outcomes, benchmark against industry standards, and make it much more efficient for them to work with volumes of claims.

Correct.

Yeah, we always say CLARA Analytics is the OG. We've been using machine learning and AI even prior to the current wave of LLMs. We use a hybrid mix of everything from traditional machine learning all the way up to advanced capabilities of LLMs and fine-tuned models.

I think, like most people, you're skeptical of using a new tool. And it's really, really hard to frame how you felt two years ago versus what's happened in the last six months. The pace of change and the rate of new capabilities being launched by Anthropic or OpenAI has just been incredible, and we have an incredibly hard time keeping up. So when you look at what you were doing with ChatGPT two years ago versus what you can do with Claude Code today, in terms of creating a planned workflow, deploying an agent that interacts with employees or with customers, it's a world different than it was even two years ago. So it's hard to judge, because even people that are getting started today are really just quickly adopting these tools, and they see a completely different world from what it looked like two years ago.

Yeah, for CLARA Analytics, we obviously work in the insurance space, so we're heavily regulated and we have a lot of compliance. All of our AI has to be governed and predictable as much as possible. So we don't wing new tools, we evaluate. We are an AI-native company. We have data scientists, machine learning folks. So that's just part of the core of how we build things: understanding the characteristics, the non-deterministic nature of how outputs are generated, and putting evaluations and experiments in place to understand where's the variance. And then we also put in guardrails. Our core platform has guardrails integrated. We evaluate the outputs, we have gold data sets, and from those gold data sets we can evaluate whether it actually improves the intelligence that we build into our products as well.

So, we're the OG, we started with doing traditional machine learning, and for most companies that's a huge value for us. The reason is the way that we approach it: when you looked at traditional machine learning, you typically didn't start with training some models. You typically would start with characterizing a large data set, and having a statistical model. Then we can put in evals that are guarded and guardrailed by grounded outputs that we have years' worth of, for the more non-deterministic flows in LLMs.

I think 100%, but I would call it a playbook. When you look at how we've built our solutions to ensure that you get deterministic outputs, there are guardrails, we evaluate over time, and there's benchmarked and grounded data to make sure you have good-quality intelligence being presented to the adjuster. That's a playbook that you use, for example, if you're at OpenAI when they're building OpenAI health. That same system that we built, with grounded data, characterizing the data, building statistical models, then opening it up to LLMs, and then evaluating the outputs that you get from LLMs, and what guardrails you put in place to guide the path that the LLM takes through navigating the interaction with the consumer. It's the same playbook, in some sense. It's just applied to a different industry.

Correct. I think the differentiation is, if you're interacting with an LLM, as people will say, it will lie with a very high degree of confidence. In the systems that we build, we provide intelligence, predictions. With those predictions, there's a human that evaluates those results and takes action because of them. That's a choice that we've made in terms of the products that we built. That's a choice also that OpenAI, Anthropic, Gemini or Google, or Microsoft make in terms of the way that they build user interactions with their LLMs. Right now, the primary interface for that is a chat. Our primary interface is interacting with data and workflows. That's very different in terms of the products that we build. There's a human involved. We synthesize data, we summarize content, we have predictions that we show. But a human looks at the grading and the scoring, and it's benchmarked, and they understand those benchmarks enough to say, okay, I will now approve this decision that you're making as a system.

Yes, I would emphatically say 100%. I think what we all struggle with as engineering leaders is to really fully understand where does that productivity and velocity come from. It's certainly easy to generate code now. It's hard to evaluate whether it's still great code, whether it's building meaningful outcomes for the company based on the products and capabilities you build. So we've leveraged a couple of different frameworks that are out there to try to understand where do we get that productivity from. We do a lot of surveys with our engineers: are they getting more productive in terms of code generation, test-case generation, to understand where that comes from? And at the end of the day, software development is a team sport. So we also have to look at, when we can generate code and build product faster, where does it put back-pressure on the rest of the system that's being used to build? So it goes back to specification, having clarity around the roadmap, how do we prioritize, how do we evaluate quality in the products that we build, making sure that we have test-automation coverage. So it's not just that one slice. It's the full spectrum of, how do you create a roadmap, define the products, and make sure that it ladders up to building outcomes for your customers.

I'm still a big fan of, I don't think AI changes the nature of having the thoughtful conversations around what you need to build that drives the company forward for outcomes. So what we do is we still hunker down and we do roadmaps. We understand what's a priority. We understand what we need to build for our customers, that's the voice of the customer. AI doesn't replace that from an engineering perspective for us. So we focus on making sure we have a clear roadmap in terms of what moves the needle forward. It certainly allows us to move forward a lot faster and potentially build better quality for those things that we want to get out into the market.

We have about 35 people in the engineering org.

Well, I think in general, the great engineers are curious by nature and they love to learn. So this is an amazing time for them to be in, because using LLMs unlocks a lot of the toil that they would have had to deal with in traditional software development. So I feel like in general, great engineering orgs are already incentivized to use these tools and move faster and figure it out as a team. And we've done pretty well at that, I feel.

The harder part is knowing what fast looks like and what good quality looks like when you can now get everybody aligned with using Claude Code. You can generate a lot of code very quickly. Now, making sure you've got eyes on it, making sure you've got your automation built, and everybody feels good about the pace that you're building at. I think we're working through that, of making sure that the pace at which we can build, what we have on the roadmap we want to build, and what we put in front of customers is good quality and meets the original intent of what we wanted to build, because we are moving pretty quickly.

Well, I think, like most companies, at the end of the day, it's a very customer-centric focus. Are customers happy with the product? Many companies choose to run NPS surveys. What's our bug-escape ratio? Do we have new bugs going into the product? Can we remediate them faster? I think there are very grounded measures that you can use. Are customers churning? Are they seeing you continue to deliver more value in your product, potentially faster, so that they don't see another competitor that can outpace you? I think that's the general pressure that's happening in Silicon Valley now: everybody has access to these tools, it's a level playing field. Can you continue to build and deliver the customer value that somebody else can't build, and displace you in a very competitive market?

Yeah, we work in the insurance industry, the adjuster workflow. It is a human-driven workflow. I can only speak to what we build, but we obviously have to include our customers as part of getting new capabilities out there. We have to make sure that they're trained. So again, this is where AI creates back-pressure in other parts of the delivery chain that we would ultimately want to move faster, but the customer can only learn new capabilities, can learn new features, so fast. I think every industry will have different variants of this based on customers using tools. You would see this as well with consumer tools that you use. If it changed every day, you would be frustrated as well. So now the back-pressure is really understanding where's the customer sentiment based on the pace of change that you can introduce to them, and can they keep up?

The way that I approached this with my engineering org... It's hard to even evaluate, because the last six months have been the highest rate of change, based on the models that came out in the December, January timeframe. And for us, it's around the time when Claude Code matured to the point where it is the de facto tool that every engineer uses to build, to varying degrees, across the way that we build. The thing that I acknowledged, when I noticed what was happening with the quality of the tooling, is that I think we all feel there's a bit of an identity crisis. I've been in the industry for a very long time. We're all grounded on, my experience adds this kind of value to this organization. What if now somebody can prompt you and displace the value that you add to a team? And if you look at the layoffs that are happening, that starts to feed that narrative in a lot of people's own personal way that they work. So they don't have trust. They feel like they might be displaced. The things that they added value for, for many, many years before, as senior engineers, may now be something you can simply prompt. So what we've tried to tell people is, create a safe space of, we are not trying to displace skills. We're trying to learn a new way of developing. It's like, what does it look like as we learn to get better at development, build better product, build outcomes for our customers faster, make customers happier in terms of the way that they use our product? Let's focus on those outcomes, and that will ultimately allow us to drive the business, because we have plenty of ideas that we can act on to make the product better and build new capabilities for customers.

For us, it's pretty positive. I would say we just ran a survey, and more than 80% of the team is super happy with the tooling that we use. They're happy with the way that we're adopting the tools in a very thoughtful way. We put in guardrails and the way that we're approaching the build. That's for my team. I think other teams feel maybe certain kinds of pressure from management or the board to adopt tools faster, or maybe unrealistic outcomes. We expect about a 20 to 30% lift in productivity or velocity every quarter based on maturity of our adoption of the tools. I know other leaders are expecting 2x, 3x, 4x, because if you spend some time, there's obviously a lot of dubious claims that are happening in the industry about, if you adopt these tools you can get 2x, 3x, 4x. When would you actually achieve those? We're trying to chart that path. I run a lot of surveys that get feedback. What can we do to be more thoughtful and gain insights about how do we use the tools? It's just the engagement style. I think it's just the nature of the way that you have to work through adopting a new tool.

Yeah, there are two things. We focus on people that are super comfortable with ambiguity. At this rate of change, you could have something that shows up, either on the roadmap, or a new product capability that we want to build, or even in the tools that we use, that creates a lot of ambiguity. And that can make people uncomfortable. How do I get my job done while I'm learning a tool, and there are expectations that we move 2x on something that gets defined very, very quickly? If you compress time, you get into spots where you're like, I have to figure these things out for myself, I don't have all the traditional specifications that I expected. And so we look for engineers who are very comfortable operating in that kind of environment, where they can navigate the unknowns, learn things, and also work with business partners to make sure they have the details enough that they can create the way of the work very dynamically. On the flip side, we look for learning velocity, the ability for engineers to learn while they do. One of the hardest things you can do as an engineer is accept a deadline for building something very, very quickly while you're using new technology, and while it's not typically specced out to the level that you would want. Now you're navigating a lot of ambiguity and you're learning something new, and you're expected to still deliver at the same time, which is a lot of pressure. So we look for engineers, and one of the things we ask is, can you cite an example where you learned a new technology while you were delivering a very ambitious project? How did you navigate that? How do you learn? For the high-caliber engineering orgs, engineers are putting in additional time outside of their traditional workload to figure out these tools and keep up with the industry. It is just reality. The people that survive into the next wave of what engineering looks like, they're going to be the ones that put in that time and energy to stay on top of the tools, learn them as they build, and adapt as they go.

No doubt there are dopamine hits for a certain category of engineers who feel comfortable with the pace at which you can move using something like Claude Code. Because I see a certain subset of our engineers, and they're like, oh my God, I was able to build this thing in a weekend. And they actually feel that the capability opened up, that they can now, they're not blocked by other specialties or capabilities that they traditionally would have been blocked by. They can now build a front end if they're a backend engineer. They can now provision platform capabilities if they're a front-end engineer. It is unlocking all these capabilities that maybe they were traditionally blocked by. And so now it's starting to also collapse skill sets, where one person can cover a lot more territory and stuff that they want to build.

At some point it will settle, I feel like. At some point, I would think you settle into, this is what the new normal is, this rate of change can continue in a society that functions. I think it's hard. We know that there will be a new version of what the rubric looks like, what skills, what titles we use. They're starting to change as well. And I think if somebody says they know what it's going to look like a couple of years from now, they're literally just making it up. I don't think we can tell. Part of it is we're just in the middle of all this and trying to work through it. I don't think you can really make a call to say exactly what it's going to look like a couple of years from now. Maybe that's my point with my engineering team: we know we can build better, we know we can build faster, we know we can improve the outcomes for our customers and for the company. Let's just focus on that, working together as a team, navigating all this together, and we're making adjustments and calls based on the dynamics of everything that's changing around us. That's really where I lean into, just the navigating of all the dynamics of the way that things are changing.