Ben Levine

I'm Ben Levine. I'm the Chief Product Officer at Candidly.

Not a co-founder. I joined at seed stage. I was the first data scientist at the company. I joined Candidly immediately after dropping out of a PhD program at Columbia. Prior to that, I had been a data scientist in industry. I knew that I really enjoyed being a data scientist in industry, and I was excited about jumping back in at an earlier stage company and driving impact for them.

What I really enjoyed about academia, the research and the teaching, I felt I could still find within industry. I'm still publishing research, and having grown out a team brings fulfillment in a similar way to what teaching did. So I was able to carry that into my time at Candidly. In addition, it's hard not to be excited about the immediate feedback cycles of producing products that users can touch and feel and respond to immediately, and being able to drive impact for them. Putting a $2.5 billion dent in the universe feels really amazing, and that's the impact we've been able to drive for our users, reducing the burden of their student debt.

Both of my parents are academics, and both sets of grandparents are also academics, so it runs in the blood. I was a data scientist at Twitter, now X, before starting the PhD program, and I looked at the data scientists whose careers I admired, and they all had PhDs. So I thought, whether I go into academia or return to industry, this will unlock doors for me and give me a tighter lens and a greater tool set for whatever problems I encounter.

When I made the decision to leave the PhD program, I knew I wanted to be somewhere earlier stage. At that point, Twitter was a public company. I joined Twitter after it became public, and if you've read Hatching Twitter, that was the antecedent to when I joined. It had a colorful cast of characters as executives, and I learned a lot from the leadership team there. But I didn't know the first thing about choosing startups. There's a little bit of kismet in how Candidly and I found each other. I thought, I have no idea how to choose a startup, I'm a PhD student. But it's some people's job to find good startups. Maybe I can put my resume out to those folks and they can route me, and it's in their best interest too. So I sent my resume to a bunch of VCs and heard back from almost none of them. I was getting along in the job search with a few other companies, and then I get a call at 6 p.m., I was living in New York at that point, from a number I don't recognize. He says, hey, this is so-and-so at a VC, I'm on my way to a founder dinner, tell me about yourself in 20 seconds. I fumbled through my elevator pitch, which I had not practiced, and he just goes, okay, thanks, and hangs up. Turns out he was seated next to Laurel Taylor, our founder and CEO, at that dinner. It came up that she was looking for a data scientist, and he said, I think I have a guy for you. A week later, she was walking me through slides of her vision for the company. She got about 30 seconds in, and I said, I'm in. What do I need to prove to show you that I belong?

We exist to crush debt and build wealth. We got our start helping folks through decisions around college planning, sending their kids to college, and paying down their own student loans, because that's often people's first really large interaction with the US financial system. The average loan to go to undergrad is $38,000, and total student debt in the US has reached 1.7 or 1.8 trillion dollars. As far as driving impact, that felt like a great place to start. Since then, folks' financial journey doesn't end when their student debt ends, so we've expanded to help across both sides of the balance sheet and the fullness of their financial journey: helping them budget, save for emergencies, save for retirement, and reach other goals they have in their lives.

Congratulations, and I'm sorry. What we do is provide AI financial guidance. We can meet your daughter wherever she is in the journey, however she feels about taking on that debt, and however you feel about her taking on that debt. We help her think through her options for how to best extinguish it. For example, there are still federal programs that put folks on the path to some amount of loan forgiveness. There's also the opportunity to accelerate the pay-down. If she was in school a few years ago, those loans were issued at around an 8 percent interest rate, so devoting additional dollars to paying that down once she's in the workforce is a really high ROI. There are also opportunities for refinancing and other repayment plans we can match her to, to personalize that guidance and get her on the best journey for her.

What's great is that we're not conflicted, and we're always free to the user. We're distributed by financial institutions, banks, retirement record keepers, and others, alongside employers. So we're distributed either as a benefit you get from being a member at one of those institutions, or a benefit you receive from your employer.

It's not just student debt. You're an employer, and you have employees making a range of financial decisions. You're already offering a retirement plan and a set of benefits that may or may not be getting utilized. And what you don't want is your employees stressed out about their finances on the job. The data show that the effects on productivity are super adverse when folks are preoccupied by their finances. So you and Candidly get introduced, and we launch with an email to your employees inviting them into the platform. From there, our AI assistant Cait can help them connect the information they need, or match them to benefits you're already offering, so they get the most out of what you're offering them as an employer.

Cait is spelled C-A-I-T, and it stands for conversational AI tool. Cait serves a dual purpose. One is as an assistant throughout the platform, interwoven throughout the digital experience, that helps you actually take action: open an account, start saving, extinguish your student debt. In addition, she's an expert on all of those topics. As far as how we built Cait, we've taken a modular approach where we endow Cait with the ability to call deterministic tools that return calculations to the user. That means the large language model is never creating those calculations anew, never inventing that math. We leverage pre-built, audited calculations to return things like, here's how we think you should allocate the funds available to you between making progress on your debt, saving for retirement, and saving for an emergency. We need to do that to be enterprise-ready. And we need to do that to feel good about the product we're delivering to users. I would never want to deliver a product to a user that I wouldn't feel comfortable using myself, that I wouldn't feel comfortable giving to friends and family.

In a highly regulated domain like finance and personal financial guidance, the math needs to be right every single time, and that's a degree of certainty that only a deterministic system, a deterministic function or calculation, can provide. So we've offloaded a lot of the mathematical reasoning to deterministic systems that are overlaid by generative AI, by LLMs. That allows us to leverage the precision and consistency of those deterministic calculations and then present them to the user in a way that's easy to understand and easy to take action on.

I'm extremely hesitant to use words like always and never in a field that's moving as quickly as LLMs and AI. If you look at some of the predictions made six months or 18 months ago versus where the systems are today, I would be putting myself in a position to look silly. What you're getting at is exactly the balance and the taste we're striking for our partners: where do we feel comfortable leveraging the magic that is generative AI, versus what needs to be right, literally every single time, in a way I can prove to you?

We're language model agnostic. Lots of our partners have made particular choices about model providers, and many of them want to retain that option value. So we needed to be model agnostic, and that's been a technical challenge we've had to build around for the last few years. The architecture is twofold. Cait has essentially two different registers. The first is what passes back and forth with the user. The second is the analytical side. On the analytical side, we've taken a skills and scripts approach, where Cait has access to a sandbox with references she can pull from: here's how you should approach thinking about somebody's retirement savings, for example. And here are the tools, the scripts you can invoke and parameterize. Take the user's savings from this database, or from what they've told us, and place them in this function, which returns the projected growth over time. All of that to say, we've created a hybrid AI system where the parts that need to be deterministic, precise, and consistent always occur in that deterministic code. And the joy that AI brings, being able to stitch together and synthesize information from different calculations, explain things in analogies that make sense to you, and create visual components that make it easier to take action, those retain the magic the foundation models deliver.

That's partner dependent. One of the things it depends on is which environment we're launching in. Are we launching within their VPC, or is it a Candidly-hosted service? If we're within their VPC, whether it's a closed source or open source model, they may have existing technical infrastructure they want to leverage rather than offloading it to us.

For us, the happy path is the happy path of the partner. For some partners, we need to be more deeply integrated, and that's something we relish, because with that deeper integration comes the opportunity to more deeply interweave what we bring into their experience. Many folks just aren't there yet, so they want to start simple. We're agnostic, and we don't view them as mutually exclusive, but maybe as a path from one to the other.

Our AI engineering effort began with our data science team, primarily because we wanted to make evaluations, evals, a first-class concern. Being able to measure uncertainty is extremely important and helpful in creating those systems, and it maps really nicely to what we need in order to be enterprise ready. You can't just say, trust me, bro. You have to bring data to bear. As far as forward deployed engineering, it's the same group of folks pushing the frontier of Cait's capabilities who are forward deployed within our partners' environments, because any delta between the two really shortchanges partners, in my belief. We want to bring the partners as close to the metal as possible. That makes it very difficult to hire, because it's a unique set of skills and attributes, but I love the team, and they've been phenomenal partners to me.

Distributed by design. We have an office here in San Francisco. We have some folks in Austin and throughout the East Coast, and then our first engineering team is in Portugal.

It predates me, but I can provide the context. They were our first engineering team. Our founder didn't want an outsourced shop. She wanted them to be full-time engineers, and the Portuguese education system has historically invested a lot in English as a second language, so they're all perfectly fluent and phenomenal engineers.

I did not know that. I've been to Montevideo and La Pedrera, a small beach town. Wonderful people and great steak. Underrated. Argentina gets all the credit.

The full company is around 75 people, and the team focused on building this AI product is 15. Models are always changing, and that's been a first-class consideration for a long time. We've made a few architectural decisions that have compounded and allowed us to make use of those advancing capabilities, things like giving Cait a code sandbox, so that when models improve in their ability to reason mathematically or through code, we leverage those gains rather than saying, shoot, we need to rethink this from the ground up. That's how I think about technical development: what are the bets we can make that take advantage of an exponential rather than fighting against it?

Interestingly, and this is an occupational hazard as a data scientist, I pulled the last two years of data from GitHub, Jira, Notion, and the Excel files we had historically used for quarterly planning. I constructed a longitudinal panel of what somebody was working on, the team they were working with, the actual code diff, an analysis of that diff, and any downstream impacts. So for the first time ever, I have an empirical view of management. Here are the folks who are the central nodes in the review graph that elevate everybody else. Here are quantitative measures of performance and which teams work best together. We found that teams with a directly responsible member of the executive team moved 25 percent faster than teams that didn't, in part because only 20 percent of the software development lifecycle was actually code gen and code review. The rest is planning and QA. So for the first time, we're able to use AI not just for code gen but also to evaluate the teams. To answer your question directly, 85 percent of our code is generated by AI.

Thank you. Occupational hazard. A tiger can't change his stripes. I view that quantitative analysis as informing questions rather than immediately leading to conclusions, just the beginning of a conversation. But it dramatically accelerates and focuses those conversations. And having lived it, it all feels a lot more connected than maybe it does on the page.

Every year we sit down as an executive team and map out the three to four pillars for the year that aren't going to change as goals. That comes out of conversations with the board, with our customers, and at the management level. Then we go through a quarterly OKR cycle, like many companies. Very recently we shifted EPD, engineering, product, and design, planning to 45 days, down from quarterly. We're moving so fast that it's more overhead to plan for the full 90 days than for 45-day cycles. What's interesting is the EPD team would have preferred 30 days. Our stakeholders said, we have to coordinate externally and internally, so doing it twice a quarter is cleaner than on a truly ongoing basis. It's the balance between planning overhead and giving folks a heads up on what we're doing.

When we are jointly able to reach nirvana, you should let me know how you achieved it. It's like Sisyphus pushing the rock up the hill, except the rock keeps changing. We've experienced that as well. To put a few numbers on it, ticket lifecycle two years ago was roughly 27 calendar days. Those tickets now get through in 14. So we've dramatically reduced it in aggregate, but also compositionally, like you were saying, the bottleneck has shifted from engineering to getting stakeholders aligned, doing the planning, and making sure what we build rests on a cohesive, coherent understanding of what we're trying to accomplish as a product and as a company.

Shifting to 45-day plans, I feel great about. The other change I feel great about is moving from pods of seven, each led by a product manager, which is very traditional, to what we've called atomic units: two to three people, sometimes led by a product manager, sometimes by an engineering lead. It flattens the organization and brings each unit closer to its stakeholders. And because we're planning every 45 days, stakeholders come into the planning process on a more frequent cycle. It's early days, but we're really excited about some of the early data. To be clear, this wasn't some grand plan. It was iterative learning.

As a data scientist, you're working so closely with product anyway. And as we knew the product was going to take the next step toward being more AI native, it made more and more sense to have somebody leading the team who deeply understood the technology we were building on and also understood how to measure it. Shortening and consolidating that cycle under a single function has paid dividends for us. I still love the discipline, and I also really enjoy people. One of the elements of moving to a startup that I've really enjoyed is nobody ever said, stay in your lane. Everybody is appreciative of an additional helping hand, and it just so happened that outstretched hand kept going deeper and deeper into the product. But I still love picking apart the data with the team.

How we think about trust is really important for people to know. I think about it along three spokes. The first is security and privacy. For going on nine years, we've been entrusted by some of the largest financial institutions in the world to help folks with some of the most intimate financial decisions. So the math has to be right every single time, and the way we treat that data has to be right every single time. The second is accuracy and correctness. There it's less, hey, just trust me, and more, let us bring data to bear that shows you, just as self-driving vehicles are multiple times safer than a lot of human drivers, that providing AI financial guidance through Candidly is actually safer than allowing generative beings, just call them humans, to do this on their own. And the third is memory. In the same way that in the movie 50 First Dates Adam Sandler had to earn the trust of his date 50 times in a row, we view the opportunity to compound learnings throughout that relationship as a way to both gain trust and make the user experience better, more personalized, and more actionable.

Many people view the financial system as it exists today as relatively frictionless. What we observe in the data from our users is that it's just not. There are two sources of friction. The first is between where you are and the knowledge and comfort it would take to actually take action. The second is, now that I know what action I need to take, let me go into my financial institution and do the 17 clicks it takes to transfer funds from here to here, open or close an account, or make this trade. Over the next ten years, as AI helps close both of those gaps, we'll finally appreciate how big those frictions have been.

In the reading I've done around quantum and finance, it's primarily focused on security and cryptography. What does it mean to look toward a world where technology can be both the unstoppable force breaking through security and the immovable object making sure our financial systems are secure and safe for us to use? In the same way that opening digital portals allowed new attack vectors and also new security mechanisms, it's the next realm of that, as I read it.

I have friends who work in the space who are really bright, specifically on the side of helping secure those accounts, and I'm glad there are a lot of smart people working on the problem. Beyond that, I don't want to crash out. I'm not working on the problem right now, so I'm entrusting our safety to them.

Thanks for having me. Looking forward to talking again soon.