AI, Actually – Episode 26: Multiplayer AI, Harnesses, and Why Pacing the Frontier Won’t Change Your Roadmap

AI to multiplayer AI. And I’m just gonna give my just sort of my view of things and would love for you guys to just sort of chime in and and and correct it where it makes sense. And the way I think about single player AI is just this idea of, hey, we we we’ve got access to you know to codex and we’ve got access to clawed code and cowork and all these things that pr pretty well are are centered around me as an individual and making myself more productive, giving myself a a personal assistant and a brain and so on. But then when it comes to

enabling larger workflows or really automating larger processes across the company, more than one person is very, very often involved. And I I personally find it really challenging to take things that I designed for myself and then share them with the team. So my my view of the world is just that multiplayer AI is just that where we have a group of people working together. They’re trying to achieve a certain objective, they’re trying to automate a process, and that

that in itself becomes is is a little bit challenging today. So be curious to maybe alone get your perspective on that, where that sits today and and what companies and sort of where that leads

Alon (01:57)

Sure.

Pete (01:57)

into things like harnesses and and and so on.

Alon (02:00)

Yeah. No, I I think it’s a it’s super interesting. obviously any work that you do by yourself has limitations and in any organization where there are teens, that’s a big part of the process is how how do we collaborate, how do we compound the work because we can specialize in some ways or we can bounce back and forth.

So I kind of viewed a few different ways. There there is multiplayer in the sense of it’s still you and maybe a team of agents that you’re working with.

Pete (02:27)

Mm, that’s a good point.

Alon (02:28)

which is you know, one level up from from sort of you working directly with a with a harness and a kind of a single purpose. and and that, you know, like maybe Grokbot is a good example of for those of you who’ve played with that of of that environment, you start to get a feel for like, well, I’ve got

You know, I I’ve said there’s a my chief of staff agent and I’ve got, you know, some specific agents to do research or doing analysis or producing documents. And and that even that collaboration gives you a sense of like, it’s it’s it’s a little more challenging to have conversations there because sometimes you want direct traffic at a specific agent who’s supposed to do the work. Sometimes there’s an orchestration layer that tries to have that

smartly decide who’s supposed you know whose turn it is to talk and when. As or already gives you a sense before you introduce even more people into the equation, which is okay, like if the four of us are trying to work on something. and so I think the tooling is is evolving to help, but there’s still a long way to go. I think we’re really at the early stages of how

multiplayer should work and it’s not like here here is the the ideal surface yet. you know, I think of like when we collaborate normally, if it’s a message-based kind of thing, it’s Slack or Teams, you have some expectation around like turnaround time and when somebody contributes to it, like, you know, is this part of their expertise or is this just an opinion that they have? and so you you kind of learn how how to interact over a message-based surface.

And then you have the whole other area, which is like a whiteboard, right? I’m trying to make something that’s more real-time, not not a not as asynchronous. And that work feels a little different. And so I guess what what what what I’m seeing is the there’s already the the first stages of the thing we call workspace, let’s say in in in many of our software and other it’s sort of a common thing, common I guess vocabulary around this.

A workspace, let’s say, is a place where you invite a bunch of people and there’s an agreed to there’s a set of resources beyond who who’s there. and there’s, you know, rights and and and access to that workspace, which sets up the ground rules for how to collaborate and and what you’re here to talk about. I think that’s sort of like, you the the base infrastructure that you need. And then the mechanism by which multiplayer happens is, you know, ha like if there’s an objective, right?

let’s say it’s it’s a workspace where we’re supposed to be collaborating on let’s say an an annual plan. You know that exercise is gonna bounce back and forth and and there’s deadlines of when things have to get delivered and there’s assignments and all that kind of stuff. it’s just like even early implementations feel super powerful in the sense that

the there’s somebody to keep you on track, right? If if somebody like in today’s world, most time you have somebody who’s in charge of the project and they’re the ones keeping track and nudging people like I need this from you by here by this date and so on. it’s just so much more

constructive to have an agent that has that and you know sort of like no no no judgment, right? Just sending you nudges like when it needs things and you you know react to it emotionally and and so I think just the facilitation of collaboration is is one thing that can get unlocked by having kind of an always on present agent.

And then of course the the the higher value is if agents can start doing work. So if an agent that’s part of this workspace as a team member has a a designated role, like your job is to balance you know, whatever, let’s say our our last year’s plan to whatever is coming up.

It’s constantly on and it’s constantly evaluating. And you can ask it questions about, well, why were we off? Or you know, explain more to me about this. Today, agents work best when they have a very well-defined objective and and kind of how they go about their work. And so that that’s the reason for wanting multiplayer in the first place is you want to break out these agents to have, you know, very specific objectives. and so that defines how you interact with them and it helps them define the work they do. So

I I think we’re starting to see this level of of technology emerge. And it’s really a lot about maturing the application layers and the UX as opposed to the models needing to be smarter in order for us to get more value. smarter models will obviously help a lot, but I think it’s a UX problem first and foremost.

Pete (06:30)

Yeah, I would say U UX, but one of the here’s one of the things I struggle with and you guys have heard me talk about this, is I created a bunch of agents around helping our helping us out with our financial close process. The process of providing those to other parts of the team seamlessly has been a real challenge.

And have been blocked by things like, you you don’t want to initially put in your own authorization to act to access things up in sort of a cloud, like in a cloud cloud manage agent scenario. Or your alternative is to provide someone else with a version of the skill, which of course isn’t going to continue to change, and then you’ve got to connect them to an API, which is just a whole scenario. So what are you guys seeing there? Like how are people starting to overcome some of those types of challenges? I don’t know, Stu or Mike, do you

have you have a perspective on that?

Mike (07:19)

Yeah, I can jump in. So while it’s it’s certainly true that that there’s the the friction points that you’re mentioning, Pete, it’s it’s also true that those friction points go even deeper in the sense that the the all the way down at the platform level, even how the models are trained, the the these super intelligent agents that we’re working with are still expecting, you know, just one user and one assistant to be trading messages back and forth. Right. So it it’s

Pete (07:40)

Hmm. Interesting.

Mike (07:43)

kind of like the way email is become such a global thing, but it’s just but it’s based on a very primitive protocol.

We’re still working at the bottom of the stack, we’re still working with a really primitive kind of protocol that the AI uses to talk, no matter which agent it is. So my point is there’s friction all the way up that stack. and and you know, alone’s pointing out, yep, they we haven’t figured out the UX yet, but we also haven’t figured out how to share these things. We when we and really we’re still riding on old rails all the way down at the bottom. So it it just proves the the the lack of maturity in this space. you know, that said, I I do think the the these kind of problems,

Are the sort of problems where the the solution is only emergent based on real experiences. So you to to give one one example of that in this case, Pete, you you’re you had to pioneer a very specific task with a very specific goal. Just you and your agents together are working on that, right?

Pete (08:34)

Mm-hmm. Yeah. Yeah.

Mike (08:37)

And and now we have the challenge of scaling that out to everybody else. If we had taken a committee of all those people and given all those people the goal of making that

clothes happen, we’d we just have people doing it still. We wouldn’t we wouldn’t have the opportunity to automate something, right?

Pete (08:48)

Mm-hmm. Right. Right. Yeah.

Mike (08:52)

and so so I think it’s it’s sort of the the the the problem exists because we’ve figured out that AI is a way to do it. and now now the goal is to scale it up, which I I would argue is is mostly engineering. And in fact, this whole transition to multiplayer that we’re talking about is super interesting because not only do we have the challenge Alon was addressing of saying, well I’ve got

one human and I want to add more humans, or Pete, you were bringing in the same one. But the fact that we have one agent and now I want to bring in more agents is is super important to this. And it’s and that that that division is is interesting because like you might say, well, one agent with one tool or five agents with five tools, it’s all the same, right? You’re just calling an AI with a context. So why not just make it one loop doing that work over and over. And the fact is there’s a big engineering challenge there.

Which is the same one that humans face, which is people specialize. Specialize because we can only fit

Pete (09:45)

Yeah.

Mike (09:46)

so much in our training, we can only fit so much in our brain.

Pete (09:48)

Right.

Mike (09:48)

So you need five different people, you know, accountants and lawyers and engineers to each do their job. And on the AI side, it’s exactly the same. There there’s not enough room in the context space. even as we make these models smarter and faster and better and bigger, there’s not enough room to hold everything that needs to be there. So breaking up the problem is part of solving it better. And and so I think

between these two fronts that we we have the we we have the existence proof. We know that this problem can be solved with AI when one person works with the you know their agents to do it. Now the the fact that we need to add engineering to make multiple people able to interact with it. We need to add additional agents to make sure that context works for all of them. again all all this to me points to diving in dissecting these problems at a detailed level and working these these smaller issues

that are much more tractable than the original problem that we were trying to solve.

Alon (10:41)

I was gonna say I I think just to put this in context though, this is a very solvable problem. We’re already solving it on

Pete (10:45)

Yeah.

Alon (10:46)

a you know on a customer by customer basis. You’re obviously experiencing it because y you you are y you yourself are going through that process. so there’s nothing here that that is unsolvable today with some energy.

I think twelve months from now we’ll look back and go, this is just the norm, right? This is sort of what’s expected and it’s out of the box for many many experiences.

There’ll still always be the need for specialization. Like, you know, in in our company we do things this way and and you know, therefore there’s some sensitivity to some areas that maybe aren’t addressed in out-of-the-box solutions. But I I think you’re you’re pointing the finger, you’re like six months ahead of the curve here, I think, in terms of the friction that you’re facing that will will melt away as more repetitions

Pete (11:28)

Well

Alon (11:28)

and improving these processes happen in you know the application layers.

Pete (11:32)

I I I’ve sat here and said, well, is the answer. I just need to build an application that just sort of wraps all this together that I can hand then I can just give Diana. So Diana is our controller, and I’m sitting here with her yesterday trying to connect her to the QuickBooks API and it was a complete pain in the rare end. And I thought, I just need to build an app and go host it somewhere that you can just easily get to and and run it.

Stew (11:54)

Yeah.

Alon (11:55)

yeah.

Stew (11:55)

I

I yeah. I think there’s like a lot of the challenges map to just multiplayer companies, you know, like multi human companies, right? Like there’s

Pete (12:04)

Yeah, yeah.

Stew (12:05)

always if you’ve managed people in the past, there’s certain challenges around how do I get everyone to to at the same level of knowledge, right? So have have all of the same context. How do I get people aligned around common goals? How do I

how do I gate certain things, right? Like in in terms of approval and access, finance is a great example, right? You might want to have multiple players engaging in this, but maybe not everyone can see everyone’s payroll information, right? Maybe not everyone can can approve the final action. So you’ve got to have ideas of these kind of like authority levels among the different players that are involved,

Pete (12:47)

Hmm. Right.

Stew (12:49)

both.

humans and agents. context wise, you’ve it’s very similar as well, right? Like, like who has access to what? And I think there’s an idea of kind of like gated context of like, you know, there’s shared context. All of the agents and humans here have access to this data. And then there’s kind of like gated context. I you know, I can choose to share this additional context that I got through, right? A more private channel.

Right. That t to the overall bit, but there’s some amount of decision making, right, on whether it’s appropriate and necessary to share that type of information. So all of those type of challenges come up with this multiplayer idea. I think it’s this is I mean we’ve been talking about this for a while. We’ve been working on on this for a while. I agree with Alon that it’s you know, it’s it’s doable today. it takes work.

Right, it it’s very situational and takes a lot of work for each situation today. W I think the tooling and UI and all of that people are learning better constructs for how to organize that. But that’s that’s where we’re headed and it’s you know, and I th I think we’re slowly getting there.

Mike (13:58)

But all all of the knowledge and capabilities that we’re gathering today apply to the future version where it’s a lot easier to do all the tech bits, right? In other words, if if you wait for that future to arrive, you’re still gonna have to organize and collate all the knowledge that you have to put in the system. So you might as well get it done now, even if some parts of it are harder, right? Start getting the business value and and essentially th that that’s where the competitive edge is gonna come from in the agentic age, right? In the AI age. It’s gonna be first

to get an AI that’s capable of doing a thing is going to be the beginning of a really strong competitive advantage.

Stew (14:34)

And it’s why companies exist. Companies

Pete (14:34)

Important.

Stew (14:36)

exist and thrive because they got good at understanding a domain and organizing a workforce and other capital around how to, you know, like do

Pete (14:46)

Yeah.

Stew (14:46)

something interesting in that domain. This is really the same thing as just going into steroids with AI. And I still think you’re gonna have specialization around topics. You’re gonna have, right? Like you’re gonna have

companies that build out these systems in in better ways than others, just like companies have organized themselves, you know, differently or better or have better cultures than others. but it’s gonna be a real big advantage, competitive advantage to be good at this over the long haul.

Pete (15:17)

In in and what I keep hearing more about, we’re sort of onto the ne the sort of segue into this idea of a harness. So I keep hearing, you know, be people are building these deeper vertical harnesses and so on to attack various problems. So maybe Alon, maybe you could just start with, all right, well, what the heck is a harness and how does that relate to what we’re talking about or not?

Alon (15:39)

Yeah.

Pete (15:39)

maybe just give us an overview.

Alon (15:41)

Yeah. probably the easiest way to think of a harness is it’s sort of the the outer loop, the thing that is is coordinating the work and and kind of you know, if you think of there’s work to be done, something is monitoring and and and dishing out or delegating the work. And so to name harnesses, right, things like codex and claude.

code are are well-known harnesses that people use every day and and you know on the technical side. I think as you start looking at work that is not necessarily software development, there’s room for specialization in those harnesses. and so if if a generic harness basically is something that has you know whatever tools you give it, you know what whatever context you give it, it’s it’s looping around saying, okay, you asked the question,

I’m gonna start answering it and I’m gonna be thinking about the, you know, each each time I go get a piece of information, I’m gonna think about okay, does that help me? Does that get me all the way to the end? No, I need to keep thinking, I need to keep iterating. the idea of of specialized har harnesses, you know, that are more focused on an industry or or a function, are a a lot about like what can you do better if you were smarter about some task. So if you think about let’s say do an accounting.

For example, you know, or finance, let’s say broadly, in inside of a company as opposed to you know, I guess a financial services company. you might say, okay, the the questions that we face a lot of is we gotta close the books every month. we’ve gotta you know write up management reports about the outcome and we gotta do cash flow analysis and we gotta do all these things. The the the harness that coordinates all that activity.

essentially you know can be tuned in some combination of the harness and the models that are are being asked for that domain. And so something that’s trained on the idea of what does it mean to close the books and what does it mean to close the books in your company, even more specifically, can go from a little bit of context where instructions are fed, entire documents are fed, or even models are are you know baked into the model ways is training, you know, past

experiences of how to to do this work. Like what tools should you call? What is the successful, you know, what are the ten steps that you have to take and what’s the sequence between them? So you know, I I think harnesses are are are a thing that stays with us, right? That if the model is the brain, the harness is the th you know, is the entire body that holds that brain and it has the arms and legs and the tools to to do the work. Maybe I don’t know if that’s what happens. Yeah.

Pete (18:10)

Now could it could it be that like

why why why do people need something just beyond a Claude or a or a codex?

Mike (18:17)

Yeah. So so can I jump in? So

so think of it this

Alon (18:19)

yeah, jump in, Mike.

Mike (18:20)

way. You you imagine you what you want is you want the AI to make a report and the and verify that the report is correct, right? So so so why don’t you just tell the model to do that? Just say, hey Claude, make a report and then verify it. Well, because you can’t really trust the model to actually follow the instructions at a detailed level. And even if it does that step of verifying, you’re still gonna have to verify the verifier, right? So

The harness just steps out one level and says, okay, I’m gonna make sure that two steps happen. First, I’m gonna tell Claude to make a report. Second, I’m gonna tell Claude to verify it. And it turns out if you do it that way, in those two separate steps, you’re much more assured that the model is gonna do what you’re asking it to. And and and so could you do it just by asking Claude? Yeah, you can give it a shot. but but do you have the certainty that it executed your business processes correctly?

that it produced the output in the format that you wanted it, that it’s reliably gonna do this the same way next time, not unless you did it with a harness, right? So the harness is what makes it predictable. The model is what makes it smart.

Alon (19:24)

Yeah. And I think to your question, Pete, why do we need other harnesses? remember the your custom harness may start with, you know, under the hood with codex or or cloud code or you know, some derivative of that. So it’s not you’re not necessarily inventing stuff from scratch each and every time. What you’re trying to find is is the efficiencies. Like what what what can I throw away that I don’t need to know or care about? Like remember the the the general purpose harness has to sort of think very big and very broad.

If you have more specific tasks in mind, you may say, Well, I don’t need, you know, I don’t need to know anything about how to play chess. I don’t need to know anything about, you know, the the history of the world. I need to know about how to triage a customer support case. And therefore, I could be much tighter fit to that problem. Now that there’s a there’s a a balancing act there, which is you don’t want to throw away general reasoning because often the thing that happens that that’s that we feel like is smart in AI is that

it’s able to reason outside of the of you know a very tight narrow definition. So you’re trying to find that that balancing I was like, I know it’ll go faster and cheaper if I could throw out a lot of extra stuff that I I don’t need, but you don’t want to throw away so much stuff that you suffer from becoming too dumb, you know, not not able to generalize certain triage cases and understand when things are should even be triaged versus not triage.

Stew (20:41)

I think there’s an element here, if you think of it at more like business strategy level, I I think there’s you know, there’s always been an idea of of seeking out your competitive advantage and being smarter around what you outsource or don’t outsource, you know, where you’re building your secret sauce, right, versus where you just wanna get payroll done and get it done accurately. The fact that you, you know.

that you’re the best in the world at payroll probably doesn’t is you know it doesn’t matter that much than another company as long as you’re able to give accurate payroll checks on time, right? The so I think when you start, you know, for a lot of the generic things, right? I’m just pull right, the things that aren’t that are necessary for your business but aren’t core to your competitive advantage. I think the basic harness is like

Codex and Claude or whatever are gonna be fantastic. I think when you start to figure out like what’s the real secret sauce of this business, what’s unique about my processes and how I need to run things, then then you’ve you know, then you start to need things that you have more control over, right? And that you can you can bring, you know, a very bespoke knowledge and tools and other things to bear.

and I think it’s it kind of goes back to that classic trade off that’s always been in business. and it’s just kind of coming now to to this tech.

Pete (22:07)

It feels to me like everything you just said, and this idea that within a harness I can be much more precise about who gets to see what, who has what authorization to do XYZ, what human in the is in the loop for what process, and so on. So the combination of that and and all the things you talked about, Stu, to me that that feels like what that’s this is why you would create a harness.

Stew (22:32)

And then also like it’s also a classic business problem of of de-risking across vendors as well, right? So obviously if if if

Pete (22:42)

Good point.

Stew (22:44)

the secret sauce of your business only works on claud or only works on open AI technology or harness, that’s that’s a familiar problem in business, right? There’s all sorts of businesses that

Pete (22:57)

Mm-hmm.

Stew (22:58)

that you know are

Only work on SAP, you know what I mean? And

Pete (23:01)

Yeah.

Stew (23:02)

replacing SAP would be a decade-long project for them.

Pete (23:05)

Right.

Stew (23:06)

You know, there’s times when that makes sense and it’s the right choice for your business, but but you also have to realize there’s an inherent

Pete (23:11)

Yeah.

Stew (23:12)

risk there that if you use something that that that works well for you, but is a little more able to to work across solutions.

Alon (23:22)

Yeah. And I I don’t know if this is like the way I would think about it, there’s always a harness. It’s not like a harness is an option.

Pete (23:27)

Right.

Alon (23:28)

You don’t ever really interact directly with the model as as a typical user. When if you’re going directly to the API and sending it one question again, one response, okay, you’re you’re kinda at the raw level. But

Even when you’re talking to chat GPT, there’s a harness. You’re not talking to the model and you know, there

Pete (23:42)

Yeah. Yeah.

Alon (23:44)

there’s a thing there that’s doing a bunch of stuff, like deciding whether it should even answer your question or, you know, what what to do next. so so the harness is is infrastructure, if you will, you know, to to hold the brain, right? To to let it do its thing.

And then you can go from sort of the you know, harness is basically just to interact back and forth and sits in and something as dumb as just a loop that says, okay, what should I do next? And and keep asking that same question to something much more sophisticated, which understands, well, I’m running in an operation. Maybe I’ve got real-time sensor data coming in. And when when that comes in, I should funnel it to a specific model that does something quickly, or maybe it doesn’t even go to a model, maybe it goes to a specific tool that does all the work. and so it as diverse as

as as your your use case is, it creates some specialization needs to you know to optimize. If all you’re doing is let’s say generic and I don’t know exactly what this would follow, but you know knowledge work broadly, you’ll have choices of models and and choices of harnesses that are good for doing knowledge work.

If you start looking at, you know, what does your company do? What do you do every day that’s that’s got some you know some IP in it, something that’s different than everybody else, then you gotta start asking the question, okay, how do we make that faster and repeatable and and you know more accurate for this specific job?

Pete (25:03)

How should companies be thinking about this right now? Should they be thinking, well, we’ll just wait that eventually a foundation, you know, provider’s harness will be good enough for a lot of what we’re trying to do, or no, you should build your own or some other variation? What what are you recommending to people these days?

Mike (25:21)

A combination of both. So you’re always gonna wanna use the the best thing that’s coming out of the foundation model providers because they’re investing billions and yeah.

Pete (25:29)

And for things like coding and and other use cases that are world class, yeah.

Mike (25:33)

Broadly speaking, yes. but at the same

Alon (25:35)

Yeah, but even in coding, I mean

you’ve got you’ve got other companies that are, you know, right? I mean there’s

Pete (25:38)

Good point.

Alon (25:39)

th there is, you know, factory and and you know, I mean cursor got got bought, but there’s it’s not like because you’re the model providing investing billions, you’re gonna be the

Pete (25:46)

Yeah, yeah.

Alon (25:46)

best at it. It’s like okay, even in coding, there’s all kinds of specializations. Like I I’m you know, I do large migrations and here’s my needs or my team works in this way. yeah, anyway, not to cut you off, Mike, but I I think it

Mike (25:57)

Yeah, yeah, no, so so we’re

definitely gonna want to be on the latest, greatest third party, whether it’s foundation model or not, to Alon’s point. but at the same time, y that’s what your competitors have too, right? So so you so you

Pete (26:07)

Yeah, point.

Mike (26:09)

have to be building above that as well, right? Getting closer to the to the secret sauce that drives your business, serving your customers better, going faster. That’s why a lot

Pete (26:15)

Yeah. Yeah, to two’s point. Yeah, yeah.

Mike (26:18)

of times if I get asked about, you know, how should we measure AI? Should we use lines of code that it wrote or pull requests that it wrote?

My answer is neither. It should change your business. You should be measuring customer satisfaction. You should be measuring churn. You should be measuring pro you know, productivity. You should be measuring, you know, all sorts of of factors because AI is strong enough right now to change the high level metrics of your business. You don’t have to measure it, you know, kind of deep down in some esoteric value that you don’t even understand.

Pete (26:46)

And given how easy this is, it sounds like we should just pause all progress on AI. What do you would so th

Mike (26:51)

Yeah.

Pete (26:52)

this was the this was the we must pace the frontier of the the new famous or infamous, depending on where you see things, note blog post, I guess, by Dario Amodei. you know, and look, there’s all sorts of different perspectives on this. It’d be it’d be interesting, I think, to just get you guys perspective on.

Yeah, how much should we maybe leaning in in one way or the other? But I think more importantly trying to figure out, okay, I’m a business. I’m just trying to figure out how to take advantage of this technology. How should I be thinking about this? What should I be doing? And maybe go around the horns. Well Stu, if you don’t mind, I think I’ll start with you. What what’s your per

Stew (27:27)

I mean, yeah, it’s it’s it’s a complex topic and and you know there’s potential motivations from different players around you know who gets advantaged from pauses,

Pete (27:37)

Mm.

Stew (27:38)

right? And and it wasn’t just Dario, who’s the CEO of Anthropic, right? We had

Pete (27:41)

Yeah, yeah.

Stew (27:43)

Sam Altman and and Elon Musk and ever you know, lots of people echoed the

Pete (27:48)

Yep.

Stew (27:48)

same statement. I think the

In in some ways that’s what they’re really talking about though that we have to remember is is the stuff we don’t even have access to. Right?

Pete (27:59)

Stuff we can’t see. Fair.

Stew (28:01)

The stuff we can’t see yet. And and you know, these guys are always into the future. And I think a lot of of kind of wise commentary came back to obviously no one should no company should release a a a product that that that

that hasn’t gone through the right safety reviews and other sorts of things, right? But you know, that that can be handled product by product, company by company, to some level with some controls. But I think it’s for businesses right now, right? I mean, I think it’s though you know, I I think all of the opportunities there, I think if you did pause AI, there’s years of productivity improvements we could go just from, you know,

getting better at implementing

Pete (28:45)

they are.

Stew (28:46)

what we have today. so I don’t think it takes any of the opportunity off the table. If it does anything, it’s probably a little bit of risk management. Again, right? Kind of, you know, it does bring open it does bring open the point

Pete (28:57)

To your earlier point. Yeah.

Stew (28:59)

that this this is very powerful technology. It’s can control the top frontier of this is controlled by a handful of companies. There’s a lot of interest in governments

across the globe, not just the US around this technology. So trying to figure out, hey, how am I protecting my business from, you know, from a situation where this, you know, a particular vendor could be banned in my country or could, you know, have some sort of restriction or something come up quickly, I think is is certainly worth considering in your strategies.

Pete (29:33)

Yeah, so what what if there is a okay, yep, there there’s some sort of pause and some form of, I don’t know, government oversight or something like that. Wha how does that impact the some of the companies that we deal with Alon?

Alon (29:45)

You know, I think if if if AI miles get regulated more heavily, I

Pete (29:49)

Mm-hmm.

Alon (29:50)

think the downstream effect is the on businesses that have adopted it. And I think of of of our own, right? You’re you’re gonna try to use the latest best available to you, whatever that is. Th there isn’t you know, th there’s an opportunity where if if the proprietary models get regulated, the open source becomes a bigger factor and moves, you know, is able to move faster than

Them in the proprietary models. it’s probably a a bit of time before, you know, that that catch up game happens. And so I think in terms of of your choices, probably as as Stu said, the the best thing you could do is make sure that you’re set up in a world where you can change model and model providers as as smoothly as possible because the future is uncertain in that respect. That may be true with or without regulations. The y there may be bottle models that are better for you anyway. So

To some degree, you you know, you want to have that flexibility wherever possible. and really the the I’m not sure that the net effect of it in terms of the actual usage of AI, doesn’t feel like it’s gonna decline. Like if miles were frozen, we know there’s years of work ahead of us to take advantage and deploy them just as we talked about earlier. The the whole multiplayer thing doesn’t need a better mile. It needs a bunch of improved

applications and and you know bespoke code in some cases. So I I don’t think it’s a a kind of event that you you would think of as like, okay, that we have to you know alter our overall business strategy versus let’s just make sure we’re well positioned to take advantage of whatever the tools that are available are.

Mike (31:21)

That’s right. I mean,

this is not like a financial crisis, right, where nobody could get any credit or or even Y two K when there was just no talent that was available, right? Or or the you know, I don’t know, the energy embargo when there, you know, you just couldn’t get raw materials. It’s nothing like that, right? It’s it’s we’re literally sitting on the equivalent of, you know, a decade of inventory, if you want to think of it that way, with AI exactly where it is right now.

Pete (31:45)

Yeah.

Mike (31:45)

And so the question is, you know, how to best take advantage of it. And the issue of you know, does does AI take over the world and that’s why we should slow down and all of those things, that’s like geopolitical risk to to your business. You you just take it into account and keep operating, right? You can’t stop and say, let me see what’s gonna happen, because your competitors not gonna stop. They’re gonna they’re gonna keep going. The I I was fortunate to be with customers this week a couple of days and in on two different occasions, this topic kind of came up because it was hot off the presses.

And and it was kind of a l a big shrug because there’s always some existential thing. There’s always is it gonna take away my job or you know, is it gonna use up all the water or you know, there’s always those kinds of things in the background. This is just one more. The real question

Pete (32:28)

No.

Mike (32:28)

that actually came up was, are my tokens gonna be too expensive? Right. Because the amount of usage, right.

Pete (32:33)

Well, that’s the one that’s the one I

keep thinking about because one implication potentially of the regulation is that open open source, open weights get essentially shut down. What’s what’s your perspective

Alon (32:45)

I I

Pete (32:46)

on them?

Alon (32:46)

I I I think there’s low risk of that. I it’s it’d be very hard to regulate the open source models in the way that you can’t use That’s constructive, I guess I would say. I mean sure the we can have a panic over things, but look open source providers are are global, right? They’re from around the world. So you can’t put the genie back in a bottle. We’re we’re gonna have increasingly better models no matter what we do. And you know, I mean

the US has always been sort of the the place to do business. I I’d be shocked if, you know, we were the first one to slam the brakes

Pete (33:17)

Yeah.

Alon (33:17)

on you can’t have better models while your competitors globally do have better models. So you know, hopefully that’s that’s not what it comes down to.

Pete (33:25)

Yeah.

Yeah.

so we’re getting we’re into the extra credit topic for the day. And I think Stu, you were the first one to mention it. maybe talk us talk to us a little bit about Jev.

Stew (33:35)

Okay, yeah. So so Jev is some we’re recording this on a Friday, September 18th. So Jev was for us just released literally yesterday. So we’re trying to absorb this. and it was released from from a startup that was founded by some exopenai guys that were involved in kind of the original Chat GPT release. it what’s interest so if you you know

Ever read the book which was super popular for a long time called Thinking Fast and Slow? I think it’s by like Daniel Kahneman, right? You know, where it

Pete (34:07)

I haven’t read it. Daniel Connelly. Mm-hmm.

Stew (34:10)

basically says, like, hey, there’s there’s two types of thinking. There’s kind of like your fast blink reaction, highly intuitive

Pete (34:15)

Yep. Right.

Stew (34:17)

type of thinking. And then there’s more you know, where you have to sit back, think about it, reason on it, those sort of things. Jev is kind of focusing more on that first case.

Right. Like how do I get really intuit cheap, fast, intuitive driven answers. And it turns out I think there’s a big backlog of interesting use cases around this, right, that people need to do. Like I’ve got an agent I’m cheating and going and looking at every once in a while while we have this conversation right now, doing effectively a classif classification exercise, right?

Trying to say, like, okay, if I give you all of this data, can you classify that data and into the correct bucket out of these five choices? Right. Very common thing, something, you know, maybe you you could have built a more deterministic model around that was super fast but had some limitations, or you might have built your own machine learning model around that you would have had to have trained.

Or maybe you use an LLM on, which would be slow and expensive for what’s a pretty routine task. This tries to come into that like more generalized intelligence to do those fast type of tasks. And I think it’s gonna have, you know, the other thing it seems to be good at. I haven’t experimented with this with it yet, is is like visual recognition as well. I’ve seen some pretty good demos on the internet of it doing like browser use type stuff.

pretty quickly ’cause a lot of that is kind of again, just quick recognition of certain things and and working intuitively on that. So I’m it’s pretty interesting, Tech.

Pete (35:57)

Mike, if you play with it?

Alon (35:57)

Yeah, I’ll I’ll I’ll

Pete (35:58)

go ahead, Alan, sorry.

Alon (35:59)

I’ll just pile in. I think the thing Stu pointed out, I think there is a backlog of use cases that fall into the sweet spot that is is is a gap. So I think there is there’s pent up demand for something better at this kind of intersection of I’ve got really smart models. and you know, and and the there’s a whole range of L ons, right? I mean you could go like Luna on on, you know, it can be faster and cheaper.

But not so fast and cheap that you want to send it a thousand examples a second to classify, right?

Pete (36:28)

Mm.

Alon (36:28)

Like there’s there’s still for some types of of event monitoring, you want it faster, cheaper than you can with a language model, which has to think in terms of tokens, like what’s predict the next token. here the sweet spot is I need you to tell me you know 10 things, and these 10 things all have to have, you know, some choices that that you make between them. And I want it fast and cheap.

And so if you think about triaging support tickets, right? like an anything where you you you’re kind of trying to say, need to know where it goes, it goes to you know, Q1, two, three, or four, depending on the thing that it is, and depending who it comes from, then I I want further classification and so on. there is a missing bit of efficient technology that hasn’t received a ton of attention because the world of of traditional classifiers has been around you know for a long time, and you know, language models as classifiers is

is new but has downsides. So I’m I’m very curious to see like we’re playing with it. So I I think if we’re playing with it, lots of other people will play with it. It’s got enough notoriety on on launch.

and it just goes to share with the pen top demand more than anything else. And I think, you know, it’s an incremental improvement. It’s it doesn’t replace LLMs I think, for the vast majority of things that people want LLMs to do, but it gives you another tool in the toolbox to to kind of become much more efficient at at doing certain kind of work, which sometimes then opens up additional work that you wouldn’t have done before because now it’s cheap and fast enough to do it.

Mike (37:54)

Yeah, the w the way that I think about this is the the LLMs have become this Swiss Army knife because you can tell to do anything and they’ll they’ll do a pretty good job at almost anything. You want a spoon, here’s the spoon. You want a knife, here’s the knife, right? and and and it’s not just like the normal ten tools in a Swiss Army knife. We’ve they’ve become like this thousand tool Swiss Army mess now, right? That that are is getting used for absolutely everything in our lives. What kind of music do you want to listen to? And

What color should this room be? And you know, just all kinds of things. And that’s great. Language models are great for that. The problem is sometimes you don’t need anything but a steak knife, right? So can I just get a steak knife, please, that does exactly

Pete (38:31)

Mm-hmm.

Mike (38:32)

what a steak knife needs to do and it’s easy to use and I can and that that’s where this fits in. And there there was an interesting implementation that we had back in, you know, about about four years ago when when LLMs were new and they weren’t that capable, had really, really tiny windows that you could work with.

and we there was one case where we had 14 LLMs that we used all in a row to get the result we needed. And so it was like it tell me one LLM would tell me what date the customer is talking about, another one would tell me what brand was important, and a third one would say whether this was a long question or a short question, right? So 14 of these classifications, and and that’s what this kind of

Pete (39:10)

Mm.

Mike (39:11)

of classifier is doing now. So imagine being able to to handle, you know, handle one big question with

14 little ones. That’s what this is kind of giving us that capability to do. and so like a really concrete example that that we’ve implemented using JEV is you know, our customers have inbound messages that land in their general sales inbox. Very first thing you need to do is look at that and say, is this spam or not? Right? Is this a legitimate sales opportunity or is this somebody selling me something or wanting to be a partner or apply for a job, et cetera, et cetera? Today or before Jev, that was an LLM call.

Would you would say to an LLM, please look at this information and decide if it’s a

Pete (39:50)

Yeah.

Mike (39:51)

legitimate sales, our business is whatever our business is, and send that to an LLM and then it’s going to stream back to you, yes, this is a good opportunity, or no, that right. under this JEV scenario, you you flash it the the problem and it flashes you back yes or no. Right. And so it’s it’s much faster, much cheaper. Now it doesn’t do anything else, it does just

Pete (40:10)

Right.

Mike (40:11)

that one job, but it does it really well and really fast, right? And so

So again, it’s that state knife versus Swiss Army knife. you use the right one at the right time. and and both have advantages and and it’s like anything. Specialization brings better performance.

Pete (40:25)

And then to so then to bring it bring it all home, maybe I could ask each of you guys to summarize just w where you see things back in terms of the multiplayer harness and then the the sort of the AI pause and maybe what companies should be thinking about it. Stu, what’s the what’s the what’s the headline do you think for folks that are maybe listening? Yeah. But then you could you could you could break it up, but

Stew (40:45)

all three of those. I mean I think the headline is man it’s we’re

racing ahead pretty quickly into more and more advanced use cases. And I think the you know the fact that that some people were calling for a pause, especially people with access to models we haven’t seen yet, or maybe access to information around what those models have have been able to do that that we certainly don’t have access to are you know

starting to, you know, be blown away by how fast this is moving. And then the fact that you’re getting new technology like like the JEV, which I think is just, you know, solving right little niche problems and then will probably vastly improve, be another thing that will be able to vastly improve on its own track over the next few years as a type of technology. And then bringing in this multiplayer element, I mean it’s moving fast. And it’s you know,

It’s not really slowing down. You know, they might you know, even if it slows down, I think you know, it’s it’s slowing down at some it’s it’s slowing

Pete (41:48)

Yeah.

Stew (41:49)

down a rapid exponential curve to maybe a slightly slower exponential curve. So

Pete (41:56)

Right.

Stew (41:56)

it’s you know, it’s it’s it’s not getting less challenging, you know, to to keep track of the space and and to figure out how to deploy it best in your business. But but it there is a

vastly larger set of things you can do today than you could do a year ago. And I think one of the things we all have to be reminded is is just to try to be more ambitious. You know, that a lot more is possible today than than

Pete (42:20)

I think that’s really good point. Yeah. Yeah.

Stew (42:23)

a year ago and we need to recalibrate our ambition.

Pete (42:26)

Yep.

Agree. That’s a good point. Mike?

Mike (42:28)

Yeah, expect to

be surprised. Expect to be surprised. Hit the gas. Adapt to this technology. You know, everything is dynamic. n nothing is going to stay the same. even if things as that we slow down with the development of, you know, the IPOs of a couple of giant companies, it’s not gonna affect most people’s business and there’s a lot of work to do to take advantage of the tech that’s already there. Hit the gas.

Pete (42:49)

Yeah, yeah. Yeah. Alon,

you get the last word.

Alon (42:55)

I’m I’m most interested in seeing how this gets rolled out over the next year or so in terms of like do I talk to one agent that’s like my agent that does everything on my behalf or do I talk to a bunch of specialized agents and and I know when to talk to one. You know, maybe I have something on my behalf that sort of orchestrates some things, but

I think it is gonna be fascinating tension between like the generalist that I I just always go to and that’s my yeah, it it knows me and it memorize what I like and all that. versus you know, like at our company we have, you know, five experts. And when I have a conversation with expert one, I don’t wanna, you know, I don’t want to debate and and confuse other issues that are not necessarily like what we’re talking about.

and I think w however that manifests, I I think is gonna be fascinating in terms of just the amount of interaction that you’ll you’ll take for granted. You know, like today, maybe if you if you do knowledge work, you sit on a keyboard a good bit of time, you know, constructing things, all that, how much of that will will go away and it’ll just be, you know, this conversation that you’re having with whoever your your favorite expert is, I guess on a topic that you wanna get progress on.

Pete (44:02)

Interesting times. Guys, gr great

Alon (44:03)

Yeah. Yeah.

Pete (44:05)

great conversation. Appreciate you guys joining us. Happy Friday. See you next week.

Alon (44:12)

Have a good one.