Practical multi agent automations with CrewAI

[00:00:00] Joao: there, everyone. Nice to meet you all. My name is hard to pronounce for any new Portuguese speaker.

[00:00:04] Joao: So I go by Joe, that makes my sales calls a little faster, a little easier as well. I gotta say you gotta got a little crazy a few months ago where we had a, an agent to actually do lead enrichment scoring and then schedule meeting for us, but we didn't block our calendars for a lunch or anything.

[00:00:23] Joao: So what do we end up with was a bunch of like back to back to back to back. And there were like 15 minute calls, just get to know the other person. I would jump and go straight to my name is Joe. Let's like, let's get things going. And and I think that it stuck a little bit with me.

[00:00:38] Joao: I got to say, first of all, it's very good to be here. Great audience. We're going to be talking about some interesting stuff today about agents. I'm going to tell you a little bit on how we are using agents internally. And how you're seeing other companies use externally as well. I'm very excited to share some of the stuff that I'm going to share with you.

[00:00:56] Joao: All been a quick question is this video going to be ever public for everyone to watch just so that I know how much I'm able to share.

[00:01:04] Ben: No, there's only four pro members. Yeah.

[00:01:07] Joao: Oh, nice. All right. So I can get a pretty savvy. I like this. I love this, NDAs. And now that you never know, like how much you can tell about certain things, but hey given that this is just us here, I think this is going to be very interesting.

[00:01:19] Joao: Let me get my slides going real quick. I'm going to share my screen so that you all can see it. I got a few different slides, I'm going to be jumping against them just so that we can we can get the best of both worlds real quick. This would be the first ones. So yes, we got, there was a record last week where we executed 322, 000 crews.

[00:01:46] Joao: Each crew usually has between. Two, three, maybe five agents if you push it. We have seen some that have up to 21 agents. So it was one day with over a million agents executed. It's [00:02:00] crazy. You go back 60 days, we are running like already like over 10 million agents in a month. So I assume everyone knows what an agent is.

[00:02:08] Joao: But if you want me to level set, I can spend two minutes on that. Yeah, I think that'd be helpful. Alright, so let's talk about it. You all know like Jet, GBT, or entropic, like this LLMs and your ability to chat with them. They're good at creating content. They're so good at creating content that they always have like reasoning.

[00:02:26] Joao: So if you're giving them two options, they're going to choose it. You can say Hey, go left or go right. And they're going to say, Oh, you should go right. Because X, Y, and Z. So they can reason about certain topics. If you exploit that ability and you get this models to either chat with themselves or to chat with a copy of themselves, then you leave the room.

[00:02:45] Joao: And this agents now have autonomy, agents born. They can choose what actions to take in what order and then achieve like an outcome or a task that you're trying to achieve. And the reason why this matters is because if you all have been in tech for a while or company building in a while, that automation is a big piece of that.

[00:03:05] Joao: And it starts like with simple A to B, but very quickly expand to all those different use cases. And that is like how headaches are born. But with the beauty of AI agents now is you don't need to link all those different points. You can give the options and the agents choose on how to explore them.

[00:03:22] Joao: So if they go to A or B or C or D, depending on what they're seeing out there. So this real time reaction kind of helps a lot. I have a quick demo that I like to show, and this is, Assuming it's a simple use case, but I think it highlights some of the power of things that you can do. So let's say that I'm going to jump into a call with the CEO of Zendesk Tom.

[00:03:44] Joao: And I know Tom's email is his name at Zendesk. I can just copy that, and I'm going to send that to a couple agents. These agents are going to start researching Tom in Zendesk. We're going to learn that Tom has a background in sales, marketing, and leadership. He sits on the board of Zendesk [00:04:00] and then we also learn Zendesk was founded in 07 and its mission is simplify complexity.

[00:04:06] Joao: And then another agent writes a landing page, custom made for Tom, basically selling CrewAI to Tom around his expertise around Zendesk's mission and Zendesk. Use cases and mentioning out of that. And this is something that we eventually upgraded this and something that we actually are using internally now so that we can generate support sales materials to follow up on calls and things like that.

[00:04:30] Joao: So again, like a simple example, there was a lot more cutting edge stuff that you can go for, and we can talk about those in a second. But this highlights and like how powerful some of the systems can be. And this was not a sped up in any way, by the way, I was using Grok to power this agent.

[00:04:44] Joao: It was a very fast inference model. So before I keep going, quick breaks, any questions about AI agents, what they are? We're going to talk about a few use cases in a second, but I don't know if anyone has any questions. All

[00:05:01] Joao: Let's keep rocking that. So these agents usually start very simple, right? They have a lab in the middle, you have the tools, they give them a task and they're going to go out and use these tools in order to give you like a result. But then once you got into production, a few things started to pop up, you started to worry about, oh, wow, I need caching because I don't want these tools to be used over and over.

[00:05:21] Joao: I don't want to run out of like credits on these other systems or overload these other systems. And then, oh, I see you have Shack going. Have this open. I'm just gonna pop it up here. All right. Then you have memory. Hey, if my agents have done one thing once, I want to make sure that they remember this, they learn from this.

[00:05:41] Joao: So if they do it again, like they can leverage that. I want to train them, feel like onboarding our agents. Like once I told them what to do, now I want to really like instruction on how to do it. I want to like specific things. Foreman, or when a specific number of paragraphs. So you want to, there's a bunch of massaging that you want to do there.

[00:05:58] Joao: And then the classical guard [00:06:00] rails, preventing hallucinations and all that. And when you go to Muti agents, that kind of still are like one of the main problems. You still want to cache in memory, the training, the guard rails, but now you want them to be shared, right? You want your agents to share the memory, your agents to share.

[00:06:13] Joao: If they use it, something they learned about it, they both learn about it, or the three or the four. For the five of them learn about it. So they don't need to go over and over it. And you can go one step further something that I'm going to show to you firsthand that we haven't showed anyone out in the world yet today, that's called flows.

[00:06:30] Joao: Ability to connect multiple crews together and also execute Python between them or before them and all that. I think it's going to be a good one. We're going to probably launch it as week. And then, yeah, different ways to orchestrate these agents. You can get them to work similar to teams, right?

[00:06:46] Joao: Sequentially hierarchically, there's a bunch of different ways that you can make them work in a way, but overall it's a very good change of pace in terms of the way that we have traditionally viewed software, like a traditionally software has been very strongly type, what is going in, what is going out.

[00:07:00] Joao: going inside, what's going out. And with this model is getting better and more agency, getting more fuzzy. Like you don't necessarily know what's going in. The model can be somewhat black box and then you don't know necessarily what is going out. And you can take chat GPT as an example of that.

[00:07:15] Joao: The person can paste a recipe in there or a PhD thesis, then things do works the same. And people love it for that. I see a few questions popping up in there. Nice. All right. We're going to get that to the end. I love it. So again, I'm going to skip the part that I introduced myself, even though we're talking about it.

[00:07:31] Joao: Company is spread all over the place today. We have people in Brazil, US, Canada, UK, Australia. We have been growing quite a lot and we started by using agents. We started by doing a marketing crew. That was the main thing that kind of got us off where we had four agents. A content creator specialist, a media analyst, a content writer, and a chief content officer.

[00:07:53] Joao: And then what we would do is we would basically tell those two. agents to write like some piece [00:08:00] of content for us based on a very high level rough idea. And this is a node crew, but I'm going to see if we can get it to run today so I can show you, but it would look through X and LinkedIn, what other people have been talking about this topic, would search the internet, look at my previous experiences, and then Write an incredible draft and we had the final agent there actually be a fine tuned model that was fine tuned on Content that we already wrote so it would write like us What made this like a like just super easy for us to edit before posting?

[00:08:33] Joao: And we end up being able to put way more content out there. I'm seeing we're getting way more views After that, we need to do lead qualification. Another one I'm going to show you today. Again, three agents analysts and research in a planner. And then they get together. And once they get those lead responses, they basically analyze them.

[00:08:52] Joao: They compare with a CRM data. They research the industry and they give us like a lead score and use case analysis, complexity score points for the meeting. Very interesting. Oh, and then. The calendar madness that I told you about. There you go. You can see that it was a little insane, but we talk with 200 customers in two weeks and it has been a lot of fun.

[00:09:13] Joao: Since then we expanded a lot. So now even our docs, if you go over our docs, all that is written by agents. We still write out like brand new ones here and there, but they're the ones that are reviewing the docs and then comparing the code with the new one and make sure that everything checks out.

[00:09:29] Joao: Before I keep going on this one, I'm going to pop up to another one that I think might have more interesting information on the market before we get more technical. How does that sounds like a plan? That's good. Nice. All right. So boom. There we go. Let me bring this bad boy up. So I, I think like one thing that is pretty clear at least from our point of view is like the market's this genie's not going back into the bottom people like not going to [00:10:00] stop using AI and feels like AI agents like are a thing happening no matter what.

[00:10:04] Joao: It's a matter of like how we're going to get there and what that's going to look like. But we already seem like major companies out there and we're talking with some of them, like adopting, like some of the technologies in order to try to optimize processes. And the way that we are looking at the market as it grows a little bit insane.

[00:10:20] Joao: We started in the early days, hearing a lot about. Pilots, a lot of lead adopters and experimentation. And a lot of that is still happening, but we are now already starting to see a lot of light RPA replacements. So a lot of the conversations that we're having with some of these companies Oh, I want to replace or expand what RPA is not able to do for my company, people saying Oh, I want to cancel like this SAS.

[00:10:41] Joao: And I want to replace with something internally. And can you help me with that? And eventually we're going to just keep growing on what is this new internet that is permeated with like agents and what that looks like. So that's like how we are seeing the market and we're seeing a bunch of companies that we are already working with.

[00:10:57] Joao: Carlyle, an interesting financial use case, PwC interesting co development use case. We're doing something with SAP projects with them. Very interesting. Now they're expanding for PepsiCo. Aurecon, very interesting company in Australia infrastructure, a lot of operations automations with them.

[00:11:15] Joao: Very interesting use cases have bought the marketing company for pharma. And that's the department that we were talking in there. Also interesting use cases. ABM BAM is a classical reply, not replacing, but expanding on RPA. So they have some RPA set up, but they want to like drastically expand on that.

[00:11:33] Joao: So a lot of interesting use cases across this companies that we have been chatting and working with has been very interesting, but also And a bunch of people actually on the waiting list, I think we have over 40, 000 people on the waiting list that we're trying to figure out how to get to.

[00:11:48] Joao: And the reason, I think like one thing that we realize is that the way that we make this a reality companies and enterprises to adopt these agents is there's a few boxes that need to check. You need to [00:12:00] be enterprise ready. And what that means is like from the silly features like, hey, I need like a proper support.

[00:12:06] Joao: I need like ability to connect with SAML. I need to be universal. I don't want to be locked into a single vendor. I don't want to be on OpenAI forever in case Entropic launches a big model. So when I be able to like help with that you have to support complexity and complex use cases. And you need to be fast.

[00:12:23] Joao: It's almost like the economics need to make sense, right? If people want to experiment with all these different automations to see what sticks. If that is a big lift and you have to deploy a huge team on it. That kind of screw up the economics a little bit and then these companies are just like not willing to go there But if you make it like so damn simple for them to throw thing at the wall Then the economics flip and they're like, alright, so let's try all the automations that we can and see what works So there's a interesting like argument in there like making it simple We have some codes in here to kill off Gemini for PwC Folks from Carlisle has been very interesting.

[00:13:01] Joao: All right Enough of that. Let's let's check in a little bit on what this looks like. Let's just start more technical and then get a less technical. How technical is the audience, by the way? And just to make sure that I get there.

[00:13:13] Ben: I think a mix would be good. I'm definitely not technical, so I'm bringing that that level down anyway, if no one else is.

[00:13:20] Ben: So yeah, so obviously non technical would be great for me. But there's definitely some more technical people in the crowd.

[00:13:26] Joao: I love that. And by the way, I got a little bit like corporate here and enterprise y. I could expand on that, but I assume that you folks want to see some action.

[00:13:33] Joao: But if you ever want me to get back, there's slides that I skipped talking about specific use cases, ROI distribution. Keep me honest here if you ever want me to go back and revisit some of these topics. Otherwise, let's get some action. All right. So let me move some of the screens around, make sure that we grab something.

[00:13:55] Joao: So the way that a CrewAI started was with [00:14:00] open source, right? So it starts in the open source framework, very much focused on engineers and just make it super easy for them to build this agent. There's a GitHub and people would come in here and download. There's over like almost over 300, 000 daily downloads now of the library where people would come in and build this.

[00:14:19] Joao: And we made it exceptionally easy to build things with code to the point that we have a bunch of no coders and non technical people that are jumping into this and like building themselves, even if it's their first time doing Python. And we basically call this the engine, right?

[00:14:36] Joao: And we see all these people using the engine and like putting miles on it. And then what I realized is that people really wanted the full car. And that's what we went for. And then we built an enterprise version on top of that. So let me show you some of the examples of the. Open source. So you can see how this works in the wild.

[00:14:55] Joao: And then we can show you what is the difference between that and the enterprise and how that makes things a little easier, right? Moving this here. Boom. Alright, so a city example, but that I use a lot is preparing for a meeting. I don't know if you folks have that, but I, sometimes my calendar still got a little crazy and I got jumped into this random meetings that I was not expecting.

[00:15:23] Joao: Either because I didn't see them in a calendar and I don't know what is the person that I'm talking to and sometimes the customer, sometimes something else. And I just I have no materials to go into that because it's five minutes away. Okay. And I just realized that I got that. So I use, I put together a crew of AI agents just to help me prep for that.

[00:15:42] Joao: So let me show you what that looks like. So we have, I need to move this into here. There we go. So again, don't get too scared about the code. You're going to see that it's not that, they're complex. I'm going to close a bunch of this just to make it more straightforward. [00:16:00] So when you're talking about CrewAI on the technical side for engineers, what you have is the ability to create this project as code.

[00:16:09] Joao: And it's very simple. You go into your terminal and you're going to type CrewAI, create crew. And you give it a name and what that will do is set up this directory for you with this specific files. So everything that you need to get things started, your agents and tasks are YAML files.

[00:16:30] Joao: So anyone can read them like they need to be an engineer to understand what is going on in there. So you can see that we have an overall researcher and we have a final report agent. And then you have a role, a go and a backstory. So this is something that a query I batted early on that all these agents, they do better jobs if their role playing.

[00:16:50] Joao: So all your agents will always have a role, backstory. And then for the tasks it's a similar, you have a description and an expected output. So you can see that the first task is basically conducting a research around a specific domain and a specific individual. Make sure there's interesting relevant information.

[00:17:10] Joao: And the second one is analyzing this information in the context of a meeting. And then there is the context of the meeting in here. And basically expanding each topic into a full section for the report. So you can see that it's a research and then writing a report. This is actually a common pattern that we are seeing, and we can talk a little bit more about that in a second.

[00:17:28] Joao: But this is like the crew. It's very similar. Then you do have some code specific around what is the format of the information that I want to get out of this. So I want a person profile with name, email, professional history, social media presence. I want a name, domain, background. Round recent news, a bunch of information about this company, and I want to submitting insights and meeting preparation notes.

[00:17:52] Joao: And things can get a little more complex if you want to, but this would be very straightforward. Your agents in here, they have the ability to search Google [00:18:00] and to scrape websites. So this would be like what an engineer would be. And you're like this is great, but I can't do this, or I can't do this in my company.

[00:18:08] Joao: That's where the enterprise makes things a lot easier, because then you can build some of that yourself with no code, or you can even deploy that. The way that I use this crew, actually, is not by running this myself. Is I, in my enterprise version, I have deploy this prep for meeting template right here.

[00:18:28] Joao: And when I want to execute that, I actually get a directly integration with Slack. And that makes it so much easier for me. So let me load that up and bring you down. All right. So if I look into Slack, for example, I can now do a slash Kickoff crew and then it's going to basically tell me like what crew you want to do It's like I want the prep for me and it's gonna ask for inputs.

[00:18:57] Joao: I'm gonna say this is an initial sales call with Director The name the email I'm gonna say I'm gonna use the email that I use back in clear bed And I'm going to say this is a corporate company and I can do kickoff. So what happens from here is that kickoffs my crew for me, and it's going to send me the input back right in my Slack.

[00:19:24] Joao: As that happens, I can show you a little bit about what is happening behind the scenes. We've got a bunch of features in here that allow us to keep track of things. So we can see active execution. So you can see our execution in here already happening. Let me zoom in a little bit just to make it easier.

[00:19:42] Joao: I see some of you squinting your eyes there. Yeah. Dan, you also have liability again, some of the things that I mentioned, right? You can train this agent to behave in a certain way. You can see the execution, you can monitoring what exactly is happening in one of those executions. What are these agents [00:20:00] actually doing?

[00:20:00] Joao: And that is very interesting as well. And the other thing that I do is you get a bunch of extra testing is also very good. So you can compare multiple models. This is very helpful. I can show you an example of that. And then you have metrics about overall execution, how much tokens, how much time it took, and then you got an extra log.

[00:20:22] Joao: If you want to get a very technical, everything that is going on in their extra settings for you to monitor quality of her crew over time in a bunch more stuff. All right, Enough of this. Let's look at what is the response looks like. So if I had to jump with this, I can see you all. It was senior engineer manager at Clearbit.

[00:20:41] Joao: And there was CTO at Palpateros. I'm very heavily on LinkedIn. Some information about Clearbit found in 2015 B2B marketing. That feels good. Relationship with CRM's recent news. Oh, Clearbit was acquired by 115 million by HubSpot. All right. So this might be good for me to mention on my call. With myself financial health raised 17 million.

[00:21:03] Joao: This is good. Industry position, leading position on the B2B data sector. All right. Do you using a lot of live language models and they have a user base over 400, 000. All right. That is good. All right. There are some key points. So I can discuss about the acquisition by Clearbit. I can talk about the marketing strategies.

[00:21:20] Joao: I can talk about Juval's leadership visions for the future. All right. I'm ready to jump in this call. And it would be something that I would do. So I actually use this crew quite a lot. It does helps me to get information. And then I jump into a call with a random person that I was not ready. And I can talk.

[00:21:34] Joao: Oh, I saw that you folks just opened a new factory in Australia. That's amazing. How's that going for you? That feels like I have done a better homework that I have just because of how much you have me have me working. All I'm going to quick break in here before we talk about other use cases.

[00:21:49] Joao: I don't know if there's any burning questions that we have time at the end. Anything. No. Nice. All right, let's keep [00:22:00] rocking. So in terms of other use cases, we feel there's a few other ones that I like as well. We have the lead enrichment score. We also have this user data gather. This is very interesting.

[00:22:11] Joao: We use this, I can show you how. And this is kinda like a test that we use every now and then. This test is very interesting because I think that I ran a. Test on it. And this is interesting. Yeah, there you go. I run a test on this to compare all the different like a major opening models. So this is you can see models in how they are impacting the quality of your agents.

[00:22:32] Joao: So let's dissect this real quick. Let's look at the averages. The final column there in the right. So if you look at the averages, you can see there's two tasks that these agents are doing. The first one got a very low score in terms of quality. The second one, a very high score, the crew average. So the final crew output get a 6.

[00:22:52] Joao: 3 and takes 11 seconds to run. And it costs 3 million tokens. So DPT 4. 0, 11 seconds, 6. 3, 3 million tokens. That feels good. But if you go for a 4. 0 mini. We see a bump, an 80. 4 score, but a little longer, even though it's a smaller model. Why is that? So when you're using a smaller models, what happens is that they fail sometimes they try to use something and they need to reassess.

[00:23:19] Joao: So even though they're faster, sometimes they might take more tries or more attempts to get to something. But you can see that due to that and the self healing properties of running agents like this, you end up getting a better outcome. So you got an 8. 4. Okay. And you don't use that many more for tokens and it's a way cheaper model.

[00:23:39] Joao: Then you compare with some of the big models, the new old one preview, like the bigger model that that open AI has now, it has a 9. 4 score. You can definitely feel that test. Number one, 9. 03, test number two, 9. 5, very solid, but 74 seconds in over 10 million tokens on a super expensive [00:24:00] model honestly, I might be better off with dpt40mini, so that's what I would use.

[00:24:05] Joao: So this is an example of how we sometimes compare some of those models, and you'll also see that I used some grok in here, so I could run this comparison like, alright, now I want to compare dpt40mini, and I want to compare I don't know with Yammer 3. 1 and I can basically run this comparison and get some of the results.

[00:24:24] Joao: So it's very interesting for us to basically make sure that we're choosing the right models for some of our agents and all that. Any tech support use cases? I'm seeing the chat. How's the quality? Oh, this is a good question. How is the quality score calculated? Thank you for reminding me that I should have talked about it So there's a thing called LLM as a judge where you use an LLM to judge Another one or another, an output.

[00:24:48] Joao: And this is what we're doing here. If you look back at the crew in the early days, we made a decision. We're very conscious about the conventions that we wanted to have on how you create agents and tasks in some of the engineering community. I didn't necessarily saw eye to eye on that.

[00:25:05] Joao: And that was funny because early Dan didn't make much sense. For example, every task got to have a description. And an expected output. So we talked about that, but why would you have to give an expected output if you're already describing what do you want the task to be? So people would ask us that all the time.

[00:25:22] Joao: And the reason was so that we could do things like this. If we know what you expect it to be, we can compare it against what you actually got and you can see like, all right, how actually close or how actually accurate the output is from what you expected it to be. And then we can draw conclusions around.

[00:25:40] Joao: How the quality is, how good this is. So this helps not only on the testing, but also on training. When you're trying to optimize your models to get better and better, we can basically have this run over and see like, all right, am I getting closer to what the expected result was? And then they just keep [00:26:00] tweaking the prompt up to the point that the agents get a super optimized and that helps a lot.

[00:26:07] Joao: In terms of metrics, it's a little bit of early days, we're adding a little more in here. You can see some of the qualitative metrics, so average prompts, average execution time, prompt tokens, but what we're plotting now is we're bringing that quality component into here. So because we don't want to store any of our prompts, you can see that we do a quality sampling.

[00:26:28] Joao: That means that 10 percent of all the executions that you run, is You're going to start to, we're going to be using to assess quality over time so that you can see actually like how your quality is going. And if you ship a new version of crew and your quality is a million drops, you know that whatever you change in there kind of impact the quality of your agent.

[00:26:49] Joao: So a lot of things that helps in here. Let me talk about another example that I think is interesting. Lead scoring is a good one. Yes, let's look at this one. So this is a lead scoring use case. Let me bring it into, let me move the meeting prep out of the way, bingoing score in the way. All right.

[00:27:18] Joao: So the lead scoring use case is basically research someone a little deeper than what we did for the prep one and then actually trying to understand if they fit on our ICP, if they're a good match or not. We executed a version of this for quite a long time. We're now updating it a little bit. So if you look at our agents, we have three agents.

[00:27:39] Joao: We have a lead analysis agent. Zoom in a little bit. We have a researcher agent and we have a scoring and planning agent. So three agents in here to help us get to things. And we have a few tasks and lead analysis task. It takes a product name, a company, an ICP description, [00:28:00] the lead responses. What is your product name and product description?

[00:28:04] Joao: Because there's a lot of information that goes into this Then you have a research task and then you have a scoring and planning task. So all this goes into kind of like creating the score and research dispersion. Again, the screw doesn't use very complex tools. Again, searching the internet, scraping websites, very to the point.

[00:28:23] Joao: You can have like custom tools if you want to. And we support that on the enterprise where you have a private repository in case you want to connect with internal data sources and things like that. And these are all the information that we want to extract from this, right? So I want to lead score. I want to use case summary.

[00:28:40] Joao: I want to talking points. I want to industry, company size, marketing position, recent news. So there is a bunch in there. And then the cool thing about a crew like this is that once that you deploy it and I have it in here It's this lead enrichment score crew that you see We are actually trying out some integrations and one of those integrations It's an alpha feature, but one of the integrations is HubSpot So we are actually a HubSpot customer so we can go into HubSpot and we can say all right Every time that someone do something you can kick off a crew.

[00:29:12] Joao: This is the crew that I want you to kick off. And then we have an automatically translating layer that makes sure that we track all the information from HubSpot for that crew to be executed. And then you can get that code back and you can do like a bunch of things with it. You can Push it back. You can save this enriched data into your HubSpot and you now have it for you for always.

[00:29:34] Joao: You can send an email in case the quality is over a certain threshold. There's a lot that you can do in there. Before I show Flows, because Flows is super excited, what happens if you're not a technical person? We're working on something, and this is me very much showing you behind the scenes, as you can tell by the development environment in here.

[00:29:52] Joao: This is me just sharing a little bit of early days. We have a new feature called Crew Studio. So crew [00:30:00] studio is basically to allow anyone in a company to view this AI agents and automations. In this helps people a lot. Let me just move this out of the way real quick so that we can there we go.

[00:30:12] Joao: So what a crew studio does is it's going to allow you to basically view any use cases that you want. Without having to know how to code and being able to leverage your LLM connections and also any specific tool that you might have. So let's say that your team, for example, built an internal to you for, I don't know you have a big like Mongo database, or you have a very specific CRM or ERP connector that only you use.

[00:30:44] Joao: You can have your team build that for you. And then, if you're Sorry. And then you can automatically use that in case like this is something that That like some people in your company needs to get access. Let me just do one thing real quick I want to make sure that I don't get blocked by this screen It's like an onboarding session that we are working.

[00:31:07] Joao: So let me remove that Second this is UI studio There we go. I want to drop this line Boom, let's refresh this. All right. There we go. This is now refreshed And now what I want is one second Yes[00:32:00]

[00:32:00] Joao: And in here, boom, the quirks of live demoing. Oh, he's interesting. Oh,

[00:32:17] Joao: what a shame. All right. I'm going to show you directly in production. It's not the same version. The local one is a little better. But hey, guess what? Oh, all right. Let's go through this. And Lauren. So interesting enough, what happens in here is because when I make crew studio super custom and personalized, we actually go over and have a crew research your profile.

[00:32:45] Joao: So because you don't need your name to sign up to the. Project. You basically use your email. We ask your name to this point so that it can make sure that we find information on the right person, because the UI studio comes previewed without this information about who are you, what is your company, what is our position, what do you do and everything.

[00:33:03] Joao: So that helps us to make sure that we get like a super custom experience as they're trying to build this agents, because it understands given your position, what are like automations that you might be trying to do, or what are the what are the things that you necessarily care about, but this is brand new.

[00:33:20] Joao: So this is all like behind the feature flag now. So people don't see it because it's under heavy. Testing and breaking and all that. I'd love to show you so cross fingers. I hopefully it works. Meanwhile, that a process or a profile that it shouldn't take that long. I'm going to show you a management UI.

[00:33:38] Joao: So we're talking about all this crews and how you run them and you connect them to all those different things. What if you have someone that is non technical and you want them to be able to execute our crews, for example, or prepare for meeting crew. You can run rankly on the UI and this kind of like a UI that people are already familiar with.

[00:33:56] Joao: This is like Trello or Jira for managing your agents. So if I [00:34:00] want a new agent to like prep for a meeting for me, I could say the same thing here. I'll say, Oh, this is the initial initial sales call. This is the email of the person that I'm meeting. We're a bit. com. And this is the domain and I can say, Hey, start my crew.

[00:34:20] Joao: And you can see your agents automatically picking up that card and starting to work on that. And you can see how that goes over time. And once that's done, you can like X spend on it. And all right, this is how many tokens is the prompt tokens. This was the actual output. So the final report on the recent news on the acquisition and everything.

[00:34:39] Joao: And if I want to see individual tasks, I can also see other ninja tasks. What were the outputs of that tasks? So this is like a friendly UI for people to manage this agents like on the go once like they're actually running. So if you have something that is automating, business proposals or writing down like demos or things like that.

[00:35:00] Joao: This is where someone that is non technical could spend their time making sure these agents are doing work and then just reusing some of their output. So there's a bunch of work here about unblocking, like this use cases for people that are non technical. Let's see if this is done. Nah, it's not.

[00:35:16] Joao: All right. I know they were closing in the 15 minutes. I'm going to give it a try, but meanwhile, why don't we get to some questions?

[00:35:23] Ben: Yeah, sounds good. I think john you've had a few questions. If you are happy to jump on if I can find your name to unmute you

[00:35:38] Ben: go

[00:35:38] Joao: for it.

[00:35:39] Guest: Hi there. My experience in using any kind of agents is it takes Forever two, three minutes for simple agents to talk among themselves and come up with conclusions. This is using a, like open AI is a pro version as the agent called or an API version of that. [00:36:00] What are you using? What have you been demonstrating to get these particular response times?

[00:36:05] Guest: I'm seeing, 14 seconds. The largest I've seen is 90 seconds. You're doing an example right now that was still completing. How are you speeding up? What are the response times you're seeing on basically the pro and free versions of OpenAI?

[00:36:23] Joao: I gotta say we use a lot of OpenAI and we tend to use GPT 40 Mini.

[00:36:28] Joao: That is a very famous So that works a lot. So I think all the examples that I show you today are using are using 4. 0 Mini. And that's where you get a lot of that speed. That initial one that shows the Zendesk customization, that is using Grok. Grok is a way faster inference model, so you don't have inference provider.

[00:36:46] Joao: You don't have the same models, but if you're okay using like LLAMA 3. 2 or LLAMA 3. 1, they can be super fast. I think some of the beauty of Create. AI is we're like a platform and we are very much universal. So you can choose what models to use. If you want to optimize for quality for a specific agent, you can use a faster and smaller model.

[00:37:08] Joao: If you want to optimize for for speed, if you want to optimize for a more deep. Quality work. You can use like bigger models that are going to be a little slower. So you have different options there, but I'll do one.

[00:37:20] Guest: So with grok though, you getting what, five times, 10 times faster than you were getting on the open AI mini.

[00:37:28] Joao: Yes, I don't know what specifically is, but definitely more than five times faster. Yeah.

[00:37:32] Guest: Okay, so is the implication here that for those of us who are looking for practical information should take away, and that is we should deploy CrewAI really on grok if speed is at all a factor that's important to us.

[00:37:45] Joao: I think if the speed is a factor to the point where you need things to be running like seconds,

[00:37:50] Guest: Yeah, so like, if you're using regular perplexity or Google search, you expect an answer within 910 seconds of the longest. [00:38:00]

[00:38:00] Joao: Yeah, I would say if you want to get those kind of like timeframes, I would say Grok, Cerberus would be like two providers that provide very fast like inference for models.

[00:38:12] Joao: What

[00:38:12] Guest: was that second one? Severus?

[00:38:13] Joao: Cerberus. Yes, I can get you their their name. I think it's Cerberus. Oh, Cerberus.

[00:38:18] Guest: Okay.

[00:38:20] Joao: Yeah. Yeah. Servers. There you go.

[00:38:22] Guest: Okay. Thank you.

[00:38:23] Joao: There's another provider that is super fast, so 20x faster than GPUs.

[00:38:29] Guest: Excellent. Okay. This sounds like a super practical problem that I keep experiencing.

[00:38:35] Joao: Yeah. I gotta say, speed for some use cases, especially if it involves user interaction, are like top of mind. I think what we are seeing more out there is like being used for internal automations, where Even if it takes minutes, it's like taking away what it used to take hours for someone to do.

[00:38:50] Joao: So people are okay with that. But if you're talking about a user facing, yes. You want to be mindful of a time.

[00:38:56] Guest: Your first example of preparing for a sales call, I would use in the same way. I'm sitting down just before the call. I want it. It's not going to happen the night before and I'm, I can take three minutes to do it.

[00:39:08] Guest: So

[00:39:09] Joao: yeah, it was a hundred percent.

[00:39:13] Ben: Awesome. Also a few people when you mentioned earlier about ROI of agents and a few people wanted to see or hear some expanded views on that. So if you got anything you can share on the ROI of agents and

[00:39:27] Joao: yes, let's talk a little bit about that. Distribution of use cases that are seeing a lot of operation automations and there's a lot of sales in there as well.

[00:39:36] Joao: And then. Marketing, co development, research, education, but more than specific verticals, what we are finding out is like a pattern where people, most of the use cases, try to do these four things. Research, analysis, summarization, and reporting. And. The research can happen in different places, the analysis can happen in different places, the summarization can happen in different ways, and the reporting as [00:40:00] well.

[00:40:00] Joao: In terms of ROI, we actually have an ROI calculator. So this was a use case that we did for and by the way, sorry, this one is in Portuguese. But this was an a use case for a pharmaceutical factory, and they're trying to automate monitoring that. So they had four different agents the average execution time of that crew was close to three minutes.

[00:40:21] Joao: That would cost them 3, 000, 30, 000 tokens for prompts and 6, 000 tokens for completion. We are talking that this crew Which saved them, let's say, half an hour. And actually, it was way more than that. But let's downplay it a little bit. Let's say that saved them half an hour. And this is how much they would cost.

[00:40:41] Joao: Given that you would use Upt4 Turbo. And what were their prices back then? If you run this crew 5, 000 times a month, seven times, seven times per hour, what do you see is you're saving basically 2, 500 hours in a month. So your cost to run this on GPT, we're talking about 2, 500 given GPT. T4 turbo.

[00:41:03] Joao: Some of the new models are going to make it even less. If you're paying like an average employer of a rate of 25, you're saving close to 60, 000 a month by running an automation like this. Again, this doesn't necessarily like. This is something that once it collects, right? You find the right automation, it works, it's saving the time.

[00:41:22] Joao: Then if you bring this data like to a month, you're like, all right, I'm saving like 6, 000 a month assuming that I'm doing 5, 000 runs of this basically running seven times every hour. What was reasonable for that case that was monitoring live data coming out from factory machines. So this is a little bit on how we were Thinking about ROI when we go to this, and we are adding some features on enterprise for that.

[00:41:43] Joao: So when you have a crew, you can say how much time you used to spend on that. So that can calculate the ROI for you and just give you a sense. Is this worth it? Is this going in the right direction? Or do I need to drop and do something else? So I could dive deeper, but I'm going to stop here real second.

[00:41:59] Joao: Just in case [00:42:00] folks went to, want to know more.

[00:42:02] Ben: Yeah, that's good. There's a question here from Robin, which I think is You've touched on slightly, and I think everyone, a lot of other people would want to know this, which is, can you use things like replica agent and or cursor to help create an execute on cruise?

[00:42:18] Ben: I know you're building like a UI. So non technical people can use it there as well. Like how. What are the best ways people can create and execute them?

[00:42:26] Joao: Yeah, it's a shame that I was not able to show the studio because that works like a charm and basically like you can chat your way into an automation So you can basically say what do you want to do?

[00:42:37] Joao: And then you go to a tabular view and eventually you go to a node view so it's a shame that I was not able to show it because I would say that's the simplest way for you to get Things going It's a lot of what we are now using it internally as well. It's like just going through that emotion.

[00:42:52] Joao: And that does helps a lot. So I would say that would probably be the main way. You can use repl. it agents for sure. We're actually very close with the repl. it folks. So if you want to use repl. it agents, that should work. And we are also bringing some templates. I think they call recipes in repl.

[00:43:09] Joao: it so that you can use them in there. And In that, that probably will make it easier. We're going to probably make a codejourn announcements of like custom recipes in there, probably sometime late October.

[00:43:20] Ben: Cool. I had a follow up then, which my brain is just gone. How, what about Julie just asked, how are customers using crew alongside other tools like Copilot?

[00:43:31] Ben: Like how do they connect the two? If they do at all.

[00:43:35] Joao: Yeah, I gotta say we're not seeing a lot of co pilot integrations What we're seeing is like internal systems integrations, right? It's very commonly for enterprises for people to integrate things with sharepoint For example, if they clearly be like microsoft shop.

[00:43:49] Joao: It's we already have all these sharepoint stuff tables that are being generated by our power apps. But now you want to get that information, process that, do some legal processing, because sometimes that case could be [00:44:00] illegal and and then push it back into another SharePoint table so that we can take it from there.

[00:44:04] Joao: So it usually like. You can imagine between those four things, you have an existing system in one end and existing systems in the other end. So what do we see is like integrations with this existing systems going on. Not so much with Copilot per se. And some customers that need like more, let's say they're not tech, like they're not technically savvy enough, or they have a very complex use case.

[00:44:27] Joao: What we're seeing them is going with consistency. So we have some partnerships. I think IBM now is offering like some services to implement CrewAI. PwC is also doing it. We also see some partnerships with things like Mongo, helping like some of people that have Mongo installations to use it.

[00:44:44] Joao: And And some companies are now hiring for a crew. So we're seeing Amazon hiring for a crew, Cloudflare hiring for a CrewAI, getting that expertise within house as well.

[00:44:54] Ben: Awesome. Actually on, on that, there's some other frameworks like I just saw that on autogen. How does CrewAI compare with other similar agent frameworks?

[00:45:05] Joao: Yeah, I would say that we are positioning a very unique way. So the way that I'm looking into the. Market right now. It's like a spectrum. So in one end of the spectrum, you have the big companies, you have Google, Apple, OpenAI, all those were like building in here. And because of like the business model and how they're set up, they have built something for masses, right?

[00:45:24] Joao: So it's got to be consumer based most of the times for them. And I believe a lot of people are going to experience AI agents in their day to day lives through like an iPhone. For example, but that for me is great because once that you experience the agents in your personal life, your next question is how I deploy this.

[00:45:40] Joao: In my professional life. So if you go into all the other end of this spectrum, what is vertical agents for specific companies. So you have a vertical agent for sales, an agent for marketing, and people are selling that. There's a lot of money to be made here, but it's getting very crowded very quickly.

[00:45:57] Joao: And I also believe there is a play here that [00:46:00] once that market starts to stabilize a little bit, the main thing that I have seen throughout the years is consolidation. People don't want to pay five different vendors for five different agents. And that's like where Create. AI sits as a platform.

[00:46:15] Joao: So in the spectrum, we're Towards the middle, a little bit skewed close to the bigger guys where you can build your custom use cases, or you can also use templates and we have templates for sales, marketing support, a bunch of those different ones. So helping people attack from different angles.

[00:46:32] Joao: The way that I would compare is with AutoGem, for example, is AutoGem came out of a research project. So it's a research coming from Stanford with Microsoft. It's not a product that Microsoft is necessarily selling, right? They do, as part of their consulting, they talk about AI agents, but there's not like a proper enterprise offering like what we do.

[00:46:51] Joao: And what we hear from the community when compared with AutoGem specifically is that they're too complex. Thanks. Too complex. It's hard to understand. They have a good edge on conversational use cases. People do like that. If they're trying to do like a kind of more like support bot like things that kind of works.

[00:47:09] Joao: But I think where we're going is like more full blown out to nations, very much focused on enterprise with a bunch of like security layers, control layers Things that I didn't necessarily showed up in here, but ability to control specific like roles, permissions, central way for you to control environment variables, LLM connections, ability for you to deploy this self hosted on your own cloud, private VPCs, a very fine grained control of specific verticals.

[00:47:38] Joao: So I would say even like on the open search side, we might compare a little bit with AutoGem. I think what we're doing business wide is way bigger. Yeah, that makes

[00:47:47] Ben: sense. We've got two, two final questions. One is from Ritesh and one is from me. The one from Ritesh, it says, We're building a voice agent for handling scheduling.

[00:47:58] Ben: Can CrewAI be used to have [00:48:00] multiple smaller agents? How can we achieve that without making it feel like a user is getting transferred from one person, in quotes, to another person?

[00:48:08] Joao: Yeah, I gotta say I have seen people do CrewAI for like conversational before what they do is they basically have a crew to decide on how to go about it, do the research and everything.

[00:48:20] Joao: But in the end of the day, they call like an 11 labs API, there's a generate the voice and then that plays the same voice all the time. So they what the agents are doing is just like collectively like guiding the conversation behind the scenes and understanding like Oh, if this is the person, if they're talking about, let's push this direction and they degenerate this one single voice and push it back for use cases like that as John pointed out then speed is at essence.

[00:48:46] Joao: So I would say you want to keep the number of agents to a minimal, the shatter to a minimum, and make sure that you focus on very fast inference providers. And then there's a bunch of things that we didn't touch. That is. Technically, you can have a lot of control in this agents on like how many interactions they do how much shadow they do so that you can make them run faster if you want to.

[00:49:09] Ben: Got it. Yeah. So the final question for me is how can people on this call or pro members get access? I think it was a few people waiting on the waiting list. I know there's some of this newer stuff coming out as well. So how can we all benefit from that?

[00:49:24] Joao: Yes to let you all know we are planning and this is not a public yet So if you don't mind them sharing it just yet, we're planning to do a big launch in October around like late October So we're gonna be we're gonna be coming to market with a bunch of news announcing some big partnerships with bigger players Also doing a press release.

[00:49:45] Joao: We're not very public about the business side of things. So people don't know You But we actually have raised threes, three rounds of funding. We have grown our team quite considerably and we just haven't been public about out of that. So we're going to hit all of that at once on [00:50:00] this late October and start onboarding people.

[00:50:02] Joao: If any of you want to skip the line, I would say, reach out to me directly and let me know that you're a part of this. Like a session and I can hook you up with our onboarding team and we can get you access right away even before that, but that's basically how we how we are working towards.

[00:50:18] Joao: It's this big push at the end of this month. There's going to be a quite fun. And as part of that, There is a AI agents week that we're going to be marketing. If you want to sign up for that, you can go into week. create. com and sign up. It's going to be a series of three webinars October 21, 22, and 23rd.

[00:50:37] Joao: We're going to be talking about again, a longer version of this Gary, like very practical building some of these use cases and you can see what's going to be happening in there. It is going to be around that lounge as well. So this is part of that.

[00:50:50] Ben: Awesome. If I can just squeeze one more in because Lucas asked a good question where how can people find Crews that the community have built to use those instead of just the templates that you've got an offer.

[00:51:03] Joao: Yes, we are building a new page on the website. There's going to be an ecosystem page where we're going to highlight like specific things that people viewed. So it's going to be a new manual in here and then probably drop in on. Thursday, I would say. The other thing is if you go in YouTube and search CrewAI, there's a lot of content in there.

[00:51:24] Joao: There is a bunch of things that you can find on how to build crew. So I would recommend that as well.

[00:51:30] Ben: Amazing. Joe, Joao, we we really appreciate your time. Thanks so much for showing us all of this and yeah I'll work on getting edited, getting, editing. until a crew can do that for me.

[00:51:40] Ben: And yeah, we really appreciate your time. And yeah, thanks so much.

[00:51:44] Joao: Thank you. It was great folks. Have a good one. Bye. Bye.

‍