Building custom GPTs from training calls to streamline team knowledge

Mike (00:00)
Yeah, cheers Ben. Yeah, so I'm Mike, Mike Taylor. I come from a marketing background. I actually built a marketing agency called Lada. I grew it to 50 people. We were one of the first growth hacking agencies, if anyone remembers the growth hacking trend. And now I'm on the AI train. I left the agency in...

2020 and got GPT-3 access same year. So I've been diving into that and now call myself a prompt engineer. like my LinkedIn says prompt engineer, which is a bit of a gimmick, I admit, but you know, people are interested at least. And I work a couple of days a week on client projects, like helping them optimize their prompts and optimize the AI workflows, kind of get more out of the AI tools that they're using or building in some cases. And then I build some

my own things as well. a couple of my own projects I've worked on. have a Udemy course which has been taken by a hundred thousand people on prompt engineering actually which is pretty nuts and way more successful than I expected. And then we just released a book through O'Reilly as well.

engineering. yeah, doing the info product game, it's been fun, but also trying to build some SaaS tools and stuff like that as well. But yeah, the thing I wanted to talk about today was really something from my agency days, which was a huge pain point when I was running an agency. If you can imagine it, you have like 50 people on the team, any of which could reach out to you on any given day to ask you questions about stuff. And I was, you know, spending most of my time doing training.

either recruiting new team members or teaching them how to get up to speed. as an agency, agency margins aren't that thick. Maybe we weren't running the agency properly, I'm not sure. But we couldn't really afford to hire very experienced people. And therefore a lot of it was just me kind of teaching recent graduates how to do Facebook ads, how to do landing page tests, how to do various different growth activities.

What I realized when AI came out and we started to get these, you know, call recording AIs like Grain.com if you guys have used that, which is the one I use, or there's like Auto AI, Zoom has their own kind of AI recording functionality. I kind of realized, hey, a lot of this really interesting information that you have on calls is completely lost as soon as you hang up. And what I wanted to do was really figure out how to capture that.

in a repository of some sort. This is the sort of thing that I would have done if I was still running the agency when these tools came out. unfortunately, I didn't solve this problem at the agency, but I played around with Zapier just to kind of see how it might work. A few of my friends work in agencies still and kind of built something for them to show them how to do this workflow. So that's what I'm gonna walk through. It's 100 % live demo, so almost definitely something will go wrong, but we'll roll the punches and...

I'll just kind of give you a directional steer on how to fix it if something does go wrong. So I'll share my screen now. I'll bring up my Zapier and then we can kind of walk through it. And by the way, feel free to ask any questions in the chat. I'm happy to make it interactive. We can also maybe test some stuff out and add something to this if anyone has any good ideas. Cool, so let me just share my screen.

Alright, hopefully everyone can see my Zapio here. So this is the Zapio I published and I'm just going to talk you high level through what it does. But first, just kind of see what the output is. We have here in my Google Drive a list of core transcripts. Sorry, core recordings. Yeah, I already broke something. Probably in a different drive folder.

Okay, yeah, it's in here.

Okay, here we go. So yeah, we have a list of transcripts. And these are transcripts after the end of each call that I do with different people. So with Grain, I specifically invite people in, invite the AI recording, you know, recording tool into the chat, if it's going to be an educational session. So this was a chat I had with one of my friends, Ellis, and he was teaching me how to use PocketBase, which is like a technical tool that you can use like as a really simple backend database.

if you're building something. So you don't have to worry too much. Like if you're not a programmer, don't worry about that. The topic really doesn't matter. But whenever you have like a training session where someone is teaching you something or if you're teaching someone something, if you're the trainer, then this is a valuable thing to have. So the call is recorded by Grain and then it dumps the transcript into a Google Doc for me. So we have this full transcript, but then we're going one step further and making it useful.

we have a set of instructions here. And this is the one that's I think really useful, but specifically it's written a SOP or a standard operating procedure for me on how to do the thing that Ellis was teaching me.

So in this case, it actually does like pull out and create some code, which is pretty nice. It created the folder structure based on the transcript that we were talking. And this is done by AI. It takes the transcript, passes it to ChatGBT through Zapier, and then creates basically like a playbook of how to do the thing that...

Alice was teaching me. So, you know, it doesn't matter what it is that you're being taught or teaching someone. It might be how to set up a Facebook ad campaign. It might be how to write a blog post. It might be how to promote, you know, get internal links, you know, whatever the internal process you have, you can create one of these SOPs automatically based on the individual call that you had. And you don't have to spend all the time to create these.

And that's really powerful because AI is just really good at this. It's just great at being able to take a unstructured document and add some structure to it. without this, basically this almost never gets done. I I was very diligent with writing standard operating procedures at my agency because I really hated meetings. So if you can avoid a meeting, it's always worth the time. instead of having a meeting with someone and walking them through how to do something, I would

write a standard operating procedure, which is kind of like an internal blog post if you think about it that way. And then I would give them this SOP and say, okay, if you're trying to do this, like here's a link to SOP, you can just follow these instructions and just give me a shout if you get stuck. And that means that you can work a lot more asynchronously. Like we had a team in Poland, we had a team in New York, we had a team in London and a few other people remote. So it was just basically impossible to schedule calls across that whole team. And this is the sort of thing that really saved us.

The other thing which is really powerful by the way just tactically or strategically maybe is that I started turning all this stuff into blog content So like if you go on the ladder blog, you know, I'm not at the agency anymore, but they still have all my old blog posts up You know, there's you know hundreds and hundreds of blog posts and videos and these are all kind of like SOPs basically that was the first instance that they were born and when someone asked me the same thing multiple times I would then turn that into an SOP

and then eventually I would pass that to the SEO guy in our team who would do some keyword research, take that SOP and then turn it into a blog post. And we got to the point where 60 % of our leads actually came from the blog. So most agencies rely almost entirely on referrals or they have to pay for ads or pay to sponsor events or something like that. For us, we almost always drove our leads through content, which is pretty nice.

nice little extra benefit you can get from from setting something up like this. Cool. So let me dig into the actual workflow and this should, you know, basically give you a bit of an idea of what to do and what can be done. You might not follow this exactly. mean, Ben would probably know better than anyone, but like potentially I might be able to share this with.

like as a template with someone. I haven't done that before, but we could figure that out. And if you guys did want to copy it, but yeah, you'll see essentially it's a pretty straightforward workflow. All right, so first thing I did, by the way, was I brought up this thing, the copilot.

And it actually works pretty well. I did run into some errors with it, but I built this really just by asking it. I I want to take green.com recordings and transcripts and then put them in Google Drive. And then from there, I want to...

run them into ChanGPT and then ChanGPT is going to write up a standard operating procedure for me.

And I'll save that.

into drive. So I did that and didn't get the whole thing in one shot, which I didn't expect it to, to be honest, but because it knows all about Zapier and I, you I use Zapier like I'll, I'll use Zapier really aggressively for a few days and then I won't even look at it for several months. It's my kind of usage of Zapier when I need to automate something I'll go in here and

and do this and then I forget like how it works. So this copilot I think has been quite useful in kind of diving back into the tool and you can see it gives you all the different steps and then you can click in to like drop all those steps in or you could drag you know grab the individual steps and add them in and so that that was pretty helpful. I'm not going to replace this one here because I have it working but I just kind of give you a sense of basically how dumb my prompt was you know I didn't have to do a lot of prompt engineering here and I got a general sense of how

this works. But the other thing that I found useful, by the way, is I would just Google Zapier Grain. And then I would look at the page. And then I would see, like, what can I get from Grain? And, and you're like, what what is like in the API? So these different triggers. So these are things that can happen, you know, when, when something happens, we want to trigger something, you know,

workflow and then we have like specific actions in this case you can just get a recording and when you get a recording that's how you get the transcript by the way that's something I figured out afterwards so let's dive into the details so the first setup that we did was we wanted to trigger on the recording added so you know I had to connect my grain account here you know if you go in and just you know search grain it's pretty straightforward and

that you just authenticate with your account. And then you have this recording added trigger and you can just search for that here. So that's pretty straightforward. And then if you go to configure, you want to grab it from my recordings.

That's where all of my recordings go in in Grain. And then you can run a test. if you look at this, I always find this is really useful. I mean, I do know how to program. I do know how to code, but I don't code if I don't have to because I'm not great at coding and I definitely don't want to maintain stuff if I don't have to. So what I try and do is actually look at the details like I would if I was building, if I was coding this, I I would take a look at the API and see what.

what are the actual fields that they're giving me here. And in this case, you can see that we've got the summary overview, got the recording URL, know, got who is involved, you know, different things like that. Summary points, but importantly, it just has the notes and it doesn't have a transcript. So that was something I had to figure out. So anyway, recording added, and then I go get recording.

And when you get the recording, so here I'm passing this recording ID, which is the actual ID from this field. And then I'm getting from that recording, then I can get the actual transcript. So that's when.

you get the transcript in VTT format. So yeah, a little bit of a convoluted thing. It took me a little bit of time to figure out how to do that. But basically, when the recording is added, then you have to get the recording. So that's like the only gotcha really that I ran into. But once you got the transcript, then you can dump that in Google Drive. And that's super easy. I just paste in here.

So we have the file name, the file that's recording URL, and then for the file name, I just put the title and then a dash and then the recording ID. So if you look in here, that's what we have. So you can see the title here.

So and then we have the recording ID. The reason why I added the recording ID, which is another programming thing, you want the name to be unique because otherwise it's going to go and mess up your existing files. If I have another call with Ellis, you can see it's like 30 minute meeting between Ellis Crosby and Mike Taylor. So if I have another call with that same title, it's not going to be unique and it might mess things up and like delete the previous one. And so that's why I dumped the ID in there. Cool. So once that's in there, then you can do more stuff, right?

So then I created, so that's uploaded a file. You could also create a file from the text. So I just grabbed the actual transcript. So this is taking the file that I created with the recording ID and the title, and then it's filling in the content of that file. And you can put whatever you want in here. So like I added the recording title and the participant names, and then I added like,

hashtag, hashtag summary. So this is just some text, right? So you can see, if we look in here, hashtag, hashtag summary. So, you know, by the way, these hashtags is just a convention. I write it for markdown, which is another way to store plain text, but you can use whatever you want. But yeah, essentially I wanted the summary in there, and then I wanted the transcript afterwards as well.

But you can add more stuff to this if you wanted. So, you know, if you just click on the plus and you can say, okay, what do I have from the recording? I have the emails if I wanted them. I have some tags, like I don't use tags so much in Grain, but some people will have tagging logic. So, you know, if the call is with a client, then it might be a client call, in which case you might want to actually add a filter and not save the client calls. Or maybe, you know, if you get someone on your team to go in and tag all the training calls,

then you could make sure that there are training tags in here. You can add some logic. You can also add this recording end to date time, and then you can click continue and then you'd have that date at the end of the transcript, which is maybe useful. Cool. That's how that works. Then you find the file again. Basically, this is just a step to make sure that the file was created. That's why I added that.

And then you continue on to Chat GPT. So this is where the actual AI magic happens. All the way up until now, we haven't really used AI other than Grain uses AI to transcribe the conversation. So kind of use the Grain AI there, but we didn't really have to do any prompt engineering or anything like that. Now, when you...

talk to ChatGPT through Zapier, you're using the OpenAI API. So I think it's a little bit of a misnomer that they call it ChatGPT because technically ChatGPT is like the front end thing. But I can see why they do that because it's probably got the most SEO search traffic. yeah, specifically you're talking to the OpenAI API and you authenticate it just like normal. But then you can pass in your prompt.

So this is the user message. This is what actually gets sent to chat GBT. And what I'm passing in here is I'm saying here's a meeting, here's a summary, here's the transcript. So that's what I'm passing that's different every time. Every time there's a recording, there's going to be that. But I'm also adding the assistant instructions, which is the system message. If you're using chat GBT, you could think of this as the custom instructions.

This is basically the instructions for what I wanted to do. And these shouldn't really change every time. This is like general system instructions. here I've told it, you turn meeting call transcripts into standard operating procedures, SOP, for the task being taught. First think about what the main task would be here and come up with a descriptive name for the task being done. So this little tactic here where I've asked it to think first is called chain of thought.

and it's a common prompting technique. In fact, the new model from OpenAI, the O1 model, O1 preview, if you've played with that, one that's really good at reasoning, that has chain of thought basically built into the training of the model. So you might not have to do that for much longer, but here I'm using GPT-40, the normal model. So I'm asking it to think first, and I'm going to get higher quality SOPs just by asking it to think first. Just like if you asked a person to

to plan out what you're gonna write about first before you write, they're gonna do a better job as well. Then I'm asking it to write a comprehensive SOP for that task that anyone in the team could do even if they hadn't attended the call. And I found that like after some testing, this was really important instruction because sometimes it would just refer to things that happened on the call. And I wanted it to know that like the call would not be available. The transcript would not be available to the person doing this SOP. And then the last thing

I trying to do with these prompts is to tell it what type of format I want back, especially if I'm using something as app here or from coding something myself, I need to get it in a reliable format, like something that I can then add to the next step in the chain. So in this case, I've just gone with something simple with it. I've just said I want the SOP title and the SOP body. You know, I could always kind of search for that in the document then, either programmatically or manually in order to get those

structured fields. But you can also do JSON, which is a JavaScript object notation. like a type of format that developers use. And that's a little bit easier to parse if you're building an automated system. But we can keep it simple for this, which is nice.

The other things that other settings here people always ask about what the settings do. But just in general, the only things you really need to worry about are max tokens. That's basically how many words you can get back. And in this case, I've dumped this quite high. Token is about three fourths of a word. You know, so this is about 2500 something words, which should be fine for us. You know, that should be plenty. We don't want it to cut off, obviously. And then temperature. So you can think about temperature in terms of creativity.

If the temperature is too high, then you're going to get back something that's like really crazy and off the wall and maybe very hallucinated. It's going to take liberties, right? So the average temperature would be one. So you probably won't run into any issues if you set temperature as a one. Zero is like very straight-laced and repetitive and boring and functional. It's not going to take any risks. It's going to give you the most highly probabilistic word, you know, that every time it

predicts the next token. So there's no randomness. Actually, if you run it again multiple times, you should get the same response if the temperature is set to zero. So for this type of task where it's important that it follows the structure of the content fairly well, I just dialed it down a little bit to 0.7.

And then you don't have to worry about top P. Very, very few people use it unless you're like an expert. But if we keep going, then I get the response from chat GBT and I need to set up a formatter, right? Because I need to grab what I said before about the structure. I want it to come back with SOP title. I want to grab that title separately because I want to put that as the name of the document. So here I'm using this is something

maybe a little bit more complicated if you haven't done any programming but this is something you can also ask Chatchie BT to write for you is I'm using some regex here so I'm looking for the word title

So here it says SOP title. And then I'm telling it to grab anything after the word title. So this the dot star means just get everything. And then I want it to end on SOP because the next thing after SOP title is SOP description. So what that's going to do is grab everything after the end of SOP title and get everything all the way up until it starts saying SOP description.

So yeah, if that didn't make any sense, could also just ask Churchie Petit, I just want to grab everything here. Here's an example of the output. How do I write the regex to capture this? And then you could just test it. So that's how that works. when we just run the test here, just come to see.

Yeah, so here we go. You can see the output. We've got just the title here. Setting up fast API with htmax, lpine.js and pocketbase. So that's the title that ChetjubyT came back with.

Now we have everything we need. done all the hard work, right? We've got an SOP, we just need to save it. So it's the same thing again. We just choose in Google Drive where we want to save it. We put the file content in there, which is just the reply from ChatGBT. then I think I've also set the... Yeah, the file name is just output zero here, which is the regex that we used. So this output from this regex, which is the title.

So there we go. We have an SOP with a friendly name that people will understand and they could search for, and we have the content in Google Drive. Now this is where it gets a little bit more advanced, which is you want to basically create like a custom GPT.

to make these things accessible. And if you haven't used the custom GPTs before, I'm just going to kind of show you what I mean by that. But me just get into my other account. the general, yeah, here we go. The general problem that you have with these SOPs is that once you get enough of them, so like here I've got a bunch of them now.

then I found that like, you I had like a hundred SOPs or something in my folder at one point. And my team would keep coming back to me and say like, hey, how do we do search query reports? And I'm like, did you not check the SOP folder? Because I know I have one in there and you know, I'd have to like go and find it for them and send it to them. you run into like a discoverability problem. You've done all this hard work of dumping all this content into Google Drive, but then

you're like, wait, what do I do now? Like, have, like, the team are still not being able to find those instructions. And maybe instead of search query report, they searched for, you know, search term report and they didn't find it. Right. So that becomes a problem. And this is where we can use AI again, which is quite nice. So you can create a custom GPT. And this is just basically your own personal version of chat GPT. And you can share this with the rest of team.

and push updates to it. And what you can do is you can put in some instructions and I'll talk about those in a second. But the most important thing is you can upload your files here, right? So you can, if we just grab this one, just gonna download it.

I'm just going to download in plain text because it's usually a little bit more digestible.

there. Now what this has done is it's given chatgbt special knowledge of one of your processes and you know when you've when you've done that then you can ask it specific situations, specific things so you can say you know how do I what database should I use with fast API

Right. And hopefully what they'll do is it's going to match to this document. So here you can see it searching my knowledge and now it's found that SOP. So it says the SOP for setting up fast API suggests using PocketBase. Right. And this is really great because now we're very happy because the customer, you know, the like the actual people in your agency can use this custom GPT and it's got your specific company knowledge in it.

So it's going to answer the way that you would answer. And it's matching pretty well because I said fast API and therefore it's found that basically that keyword or similar keywords in that document. If I went out of this and I went into ChatGPT, I said, what database should I use for fast API?

it would give me a completely different answer, right? So it's recommending SQL databases, SQL Alchemy, PostgreSQL, MySQL. It actually doesn't even mention PocketBase. And that's because PocketBase isn't that popular yet, right? Like it's...

a small project that I found really useful because I'm dumb and I don't know how to maintain databases properly. And it does a lot of that hard work for me. So I quite like using it. I've discovered it's really useful and it saves me from having to learn a bunch of stuff that I don't care about. And therefore I want to recommend PocketBase when I have someone on my team that's doing a new tool build or something. But if they just ask ChatGBT, they're not going to get the

advantage of that knowledge. Whereas with my new SOP guide, then they can ask it and they're going to get the benefit of my experience that other companies don't have. And in general, think one of the real tricks to AI is getting the knowledge that you already have, the experiences that you have, and putting them into a structured format so that you can benefit from them with AI. And if you have some information that the average chat GPT user

doesn't have and that can really help your team, then you want to be able to make that as accessible as possible. That's going to give you an edge versus everyone else that's just using the box standard chat GPT. So that's that is like

pretty much the end-to-end approach that I've been taking here. I think once you figure that out, you can really think about a lot of different situations where you could take unstructured data and dump it into a useful place. But yeah, I maybe want to open it up now. I've just been talking for about half an hour, so I thought it would be good to come up for air and see what everyone thinks and if you have any questions.

Ben (29:20)
Yeah.

Yeah, I'm going to kick off some and then there's a few in the chat. So if you have any questions in the audience, please pop them in the chat and we'll get to them. With that GBT, you were uploading an SOP individually, but there is a way to connect Google Drive to a custom GBT. Have you tried playing with that at all?

Mike (29:52)
Yeah, do you know what? When I initially set this up, there wasn't that connector yet. And I did try and set it up automatically within Zapier. from what I understand, actually, I looked at the API the other day. It doesn't seem to be way, unless I'm missing it, a way to like...

programmatically upload files to a custom GPT. So I haven't checked out the Google Drive Connector. That might be a good hack around it. But you can upload files to OpenAI. But those files can't be used in a custom GPT or assistant, as far as I know. So I'm sure that that's something that'll change, or maybe something you already know how to do. But yeah, that was not clear to me.

Ben (30:38)
Yeah, there's, we did a tutorial on this, I think last week, which was setting up a custom GBT that connects with your Google Drive automatically. It is a bit, we've got it marked down as like an intermediate tutorial, because it's not as simple as like click a few buttons, you're connected, you've got to like configure your Google Cloud with the API stuff, and you've got to do it that way. But it is.

Mike (30:49)
Nice.

Mm.

Ben (31:07)
It is possible. So you've to use the actions in the custom GBT. There's like a schema, which is some code you have to basically put in.

Mike (31:13)
so you can go up to Google Drive and then it can conduct a search through the Google Drive API.

Ben (31:19)
Yeah, exactly. Yeah. So that's how we did it with our, with our tutorial, but yeah, that's definitely a way to then programmatically keep adding to it. So your files, then your Zapier workflow just continues to add to your GBT instead of manually uploading everything. So, yeah, I mean, it's definitely not, it's not very simple to straight to set up. Obviously we've got a tutorial on it, but yeah.

Mike (31:27)
Mm.

That would be better. Yeah. Yeah.

Yeah, it's also not ideal and actually this is worth maybe laboring the point because most people don't seem to understand how...

those file uploads really work, like how they're used within ChatGVT. So it would be good to make that distinction actually. So if you just paste something into a prompt, then it has that information every time you talk to it. If you upload a file, it does what's called a vector search. So it searches by similarity. So even though I said fast API, and that's a keyword that was contained in my document, a vector search will still match even if I said like, if I want to set up a Flask API.

Because Fast API and Flask API are quite similar. They're both in Python. Most developers who use Fast API have also set up a Flask API. So there's a lot of semantic similarity. Like they're found quite often together on the internet in the same tutorials, right? So chat GPT, the vector search will basically be context aware. It will know that, hey, Fast API is actually quite similar to Flask. So even if they ask about Flask, we'll still get that SOP coming up, which is quite nice.

The problem with hooking up the Google Drive API is that it's just doing a keyword search, I think, of Google Drive. So you might lose some of these potentially relevant SOPs that don't have the exact keywords they searched for. Like if they search search term report instead of search query report, it might not be a strong enough of a match based on the keywords. So yeah, that is something that's not very well explained, I think.

OpenAI and most people think that if you upload the file then the AI just has read it and knows about it and that's not the case.

Ben (33:31)
Yeah, makes sense. There's a question from Robin and Jasper, which is similar. So Robin asks, does Chabach's GBT strictly only use your knowledge documents and no hallucinations? And Jasper similarly asked, how do you deal with the knowledge management of the SAP guide, ensuring accuracy, updating information and stuff? So there's sort of two parts there. One is the accuracy of stuff you've uploaded. And then, yeah, the next one is keeping things up to date.

Mike (34:01)
Yeah, so I would say it doesn't by default always, you know...

Be it it's not it's not 100 % accurate all the time like like anything with AI In fact, you know, even even if it does do the right search and get the right context Sometimes it you know that it has so much training data about a topic that it might still kind of override what you've what you've said in your SOP so I would say the key is to really just test it and and have like a series of

user queries that you think people will use or ones that people have complained about, like they said, I searched for this and I didn't get anything. Then you can maybe keep a Google Doc or

a G sheet of test cases, I'll call them, of different things that people might search, and then what you'd expect as an answer. And then when you make changes to the prompt, you can see how often then it hallucinates versus giving the right answer. And then eventually you'll find a version of the prompt that decreases the hallucination. So that's a whole topic, like prompt A, testing you can go deep into. But the thing I found that worked here is

Specifically, I said, told it to always search. You must always search your knowledge for the relevant SOP and only use information from those SOPs in your answer. Because I found a lot of the times it would fail to do the search. So that was like one of the common failure modes. It would just answer without doing a search in the knowledge base. And I found that when I told it you must always search regardless, then it did a much better job.

Ben (35:44)
That's interesting. And if we can spend a few minutes on prompt A B testing, because obviously you did, you've got a book on prompt engineering. You've got a course on prompt engineering and something I noticed recently. can't remember if it was another workshop we were in where people were sort of testing. Okay. I'm using this prompt and I want to try four zero. That was it. We were doing it with, crew AI. So they're an agent, framework.

Mike (35:50)
Yeah.

Yeah.

Ben (36:14)
And there's loads of different variations where someone can have one prompt and use different models, different temperatures or whatever it is, and then end up with completely different results. And the prompt changes just ever so slightly. Like, how do you go about testing prompts like this? Is it just, you've got a Google sheet somewhere with just like a bunch of text in and you're testing it? Like, how do you do it?

Mike (36:22)
Mmm, yeah.

Yeah, good question. So first of all, I'd say I just start with the naive prompt because most of the time for most tasks, it just kind of works like if you first time like if you're using the best model like GPT-4.0 or Claude, then you don't really need prompt engineering every time. I would say you should only really use it when you find it's failing on something specific or if you're having to use a cheaper, smaller model.

So with your Kuro AI example, it depends on what you're using, a lot of the reason... When people are building AI tools, they don't want to use the big expensive model because they pay every time for every call and that can add up quite quickly, especially if your users aren't paying that much for the AI or if you have free users. So for example, Notion's AI is a much, much smaller model than GPT-4 and therefore it would make a lot of mistakes.

if they hadn't done a lot of prompt engineering and testing and fine tuning on their own data. the way that I usually approach it is there's two different ways. The first way is always the vibe check, which is basically does the result look good? Like, am I happy with it? And the trick there is basically just run it a few times, see, is it making mistakes? Are there some patterns in the types of mistakes it's making?

and then just trying to address that by wording the prompt differently. you know, typically one thing that I try and do is I tell it to do something positive rather than avoid something negative. It's usually much better at that. So, instead of saying, you know, you must not do something, like you must not make up your own answer. Instead, you should say, you must search the SOPs, you know, saying something positive tends to work a lot better.

And then, you know, there are all sorts of tricks and techniques. Like I read a bunch of these AI papers or ask Claude to summarize them for me. And they come up with good techniques like chain of thought or emotion prompting or these kind of other things you might have heard about. But what I found is universally, it doesn't, none of them work every time. Some of them actually make things worse in certain cases. So you want to build a list of test cases of, when you're not just vibe checking, like I've run it a few times, it seems to work.

But if you're going to be relying on this prompt again and again, you want to figure out, okay, what are the things that it struggles with? So with the SOPs, sometimes, maybe with certain questions, it doesn't do the search properly. Or with other questions, maybe you found that it doesn't recall the right parts of the document.

And then you kind of find these edge cases as you go. Maybe some topics it struggles with more than others. And if you make a list of those, maybe it could be like 10 different searches that you've tried or queries that you've tried. And you can then write what are the expected results. And then you have something to compare it against. did the result that it gave me match up with what I expected? And if it doesn't, then you can keep changing the prompt until it does get better.

As a programmer, what I typically do is I'll run that at larger scale. It's very manual to make a change to the prompt, run it a bunch of times, and then for each of the test cases, that gets really onerous. One thing I do is I'll set up a Jupyter Notebook, which is something you can run in Google Colab in the cloud if you don't want to set up a programming environment, or you can run it locally on your computer. I use cursor, like a code editor, and then I have

Jupyter notebook and then I just, I'll write like a function to call the OpenAI API and then I'll just ask cursor like run this prompts a hundred times and then run this evaluation metric and then tell me what the average score was, you and it will write the code for me quite often. And then I'll run another prompt, I'll run another prompt and just see which ones improve the score.

Ben (40:59)
Yeah, that makes sense. Yeah, I think it seems like there's definitely ways for developers to do this easily, but not the everyday person like me. I think it is always.

Mike (41:05)
Yeah.

Yeah, the tools are still being built. Yeah, like I've talked to a bunch of people who have built like prompt optimization tools or like, you know, they're building like a

like A-B testing tools, there's a tool, like I talked to the founder of Lang Watch, I think it's called yesterday and that seemed pretty cool. But yeah, it's still really early days because we're still kind of getting used to using these things in the business setting at large scale. And I'm sure in the next year, it'll be fairly common that like, as a non-technical user, you'll have some basic tools that you're using to A-B test your prompts.

Ben (41:49)
Yeah, because it feels like people just get the... I don't know, it's a strange topic, prompt engineering, because I think there's two arguments to it from two very different use cases. One is the developer use case where I need to know that this output is going to be similar or the same kind of feeling thing. You need to do a specific thing every time. And then there's the other side of prompt engineering, which is I'm having a conversation with ChatGBT to do something, write a blog post or whatever it is.

Mike (41:58)
Mmm.

Ben (42:19)
And they're very different. Like I argue, and you can tell me if you think I'm wrong, because I suppose you have both, both those hats, right? And I argue that if you're having a conversation and you're going back and forth, a lot of that context gets stuffed in. Instead of you stuffing that in at the very start without one prompt and you're hoping the best output is going to come out, it's actually going to come out through the conversation rather than just through one really perfect prompt.

Mike (42:28)
Yeah.

Ben (42:46)
Whereas obviously on the developer side, it's very different. it's sort of, you see that when you're building like that as app your workflow where you have a specific system instruction because this is the thing you found produces the document or the output that you want. Do you think that is relatively right or different opinion?

Mike (43:06)
Yeah, I agree. think, you know, when I first started using GPT-3, you had to do a load of tricks just to get it to output anything sensible. And therefore prompt engineering really took off because people realized, like, actually, if I do all these tricks, then I can actually get something useful out of it. But I would say that it's becoming less and less important to know anything about prompt engineering. I would say that's specifically for that first use case of like, I'm just using ChantGPT as a

to help me get my work done. I would say like a lot of the platforms are building these techniques in to their UX and stuff behind the scenes. So I don't think you need to worry about it too much unless you're like an advanced or like aggressive AI adopter because it will start to get better. I would say it actually converges more on management training. Like if you're managing an employee and you don't give them a good brief and you don't really explain what type of work you want out of it,

it, then you're also going to be terrible at managing an AI employee, you know, and all the things that like I used to have to learn the hard way as a manager, like now I'm basically applying it when I'm chatting to chat, I'm kind of teaching it like an employee who, know, maybe like intern level, who, doesn't quite understand what it is I want, you know, so I can just kind of dump that context in. yeah, I think as a programmer as well, I don't know if it will be called prompt engineering in the future. I think it's kind of like, it's a bit like calling you

an Excel engineer, right? you know, like it's going to be something that, you know, if you're building AI tools, you'll need to understand how these techniques are and how to get the best out of the tools so they work.

you know, flawlessly at scale. but, you know, it might not be called prompt engineering just in the same way that like, you know, growth hacking was very trendy at one point and we started a growth hacking agency. But then as we got bigger and start to work with bigger clients, we called it growth marketing and, now it's just marketing, you know, like all marketing has gotten a lot more technical now, you know, but most marketers know how to run an AB test on the landing page. and you don't need to know how to code to do that now either. you have all these drag and drop landing page builder.

And I think you're going to see the same thing with prompt engineering. Most people will know how to work with AI. They'll be able to A-B test without having to run code, that sort of thing.

Ben (45:29)
Yeah, and I mean, like you said, 0 or 01, many in preview. I think you're not supposed to say, think step by step because it's that's that's in its system instructions or in in the model itself. It is interesting to see that there's coming out that have like that and eliminate part of what your prompt would have been before.

Mike (45:43)
Yeah.

Yeah.

Yeah, I feel like there's two approaches here. Either one is like lean in very aggressively to the forefront and try and automate all of your work with AI and like learn how to code. Just because then you have like a three to six monthly time on everyone else. Like I was using stable diffusion when it first came out and you you just have to know how to write code to use it, right? But then, but now like, you know, it's built into Twitter or like actually flux, the better like newer model is like built into Twitter now. So you don't have to know how to

to use it. I would say, you can get that three, six, 12 monthly time if you know how to code. It get more technical. And you don't have to be very technical. These tools are not like rocket science to use. They're rocket science to build, but not to use as a developer. It's just an API call. And you can use Zapier as well. You don't have to necessarily know how to code. You just kind of have to think about the system. But the other approach could just be, let's sit back. I'm going to wait until it becomes available for non-technical users, and then I'm just going to adopt it.

and I'm still going to be ahead of like 50-60 % of people so who cares, we'll see what happens. I think that both of those approaches are valid.

Ben (47:05)
Yeah, exactly. I think you're right. What else do you use? GBTs for any other sort of AI workflows that you're using regularly?

Mike (47:19)
Yeah, so call recording was a big one. Every single call really I do is recorded and transcribed now for various reasons. I use it a lot for summarization and kind of reading for me essentially. So I...

almost never read anything raw now. I just like dump it into Claude and then say like summarize this for me and then I'll decide whether I want to read it or not based on that summary. So I'm actually handing over a lot of determinism to AI though which is kind of funny because it might not be a faithful summary but it's like good enough that I save a bunch of time like reading things that are low know like not reading things that are low value to me. I also use it a

for brainstorming like strategy so when I'm trying to come up with a new blog post or I'm trying to come up with a new you know a new like product that I'm building or whatever I will try and use it to kind of build a good sense of what a comprehensive a comprehensive map of the topic so like what are all the different things about this topic that I you know other people have written about that that that I need to understand right like what are the normal features

and data architecture for this type of product, right? And if I know that, then I can pick out, okay, where do I want to innovate and be different from everyone else? Because it's really useful to know, like, what is the average standard? And then you can say, okay, I want to do this differently because that's unique to me. So I'd say it's not a replacement for creativity or innovation, but it can really help you draw a map of where you're operating, if that makes sense.

Ben (49:07)
Yeah, that makes sense. And yeah, I use it a lot for summarizing in the same way where previously I would just save, this looks like a good thing I should read, save it for later in one of those apps that I would never then revisit or never look at again. And instead I will do the same thing. I will save it to notes, but I will get a summary of it first.

Mike (49:18)
Yeah.

Yeah.

Ben (49:28)
And then I know for certain if I am going to go and then look like it sort of piqued my interest or hasn't. And if it hasn't and I've got the full, like even if it's a really long YouTube video, it's like, okay, I get the gist. Like they would have talked about these three things that would have, I would have found interesting in that hour long conversation. But actually this has done a good enough job. But then yeah, if it's supposed, like I think the founder of Anthropic.

Mike (49:49)
Yeah.

Ben (49:53)
did a really, really long post on like the, the bull case for AI and where AI could take us in sort of five to 10 years. And it's a very, very long post. And I was like, I want to read this because I should read this. And I did a summary first. And then I was sort of like, okay, it's going to stick on my sidebar until I actually go and read it in full, because I think this, this is worth my time. But it's funny how you sort of change the perception of like something other than just.

Mike (50:00)
Mmm.

You

Ben (50:23)
a link to the post or like the summary at the top or the intro or whatever, none of that almost matters anymore because everyone's going to say, I've got my own process of figuring out whether I'm going to go and spend time reading about this.

Mike (50:34)
Yeah, exactly. You kind of use it as this like proxy of like, you you go figure this out, like your time is worth less than mine. So you just go figure this out and then let me know what I should pay attention to. Notebook LM from Google, I think it should get a mention here because that was like one of those really shockingly good implementations where you can just dump in whatever document you want and it will turn it into a podcast. And I've done this with a ton of things. Like actually the really cool thing, that was the other one I was thinking of maybe doing on this, on this webinar.

Ben (50:46)
Yeah.

Mike (51:04)
was I did a bunch of user research for one of my businesses, an education business that I run. we did like, I had a researcher like go and do like 30 customer interviews. And I didn't have time to watch all the recordings and stuff. So I just dumped all the transcripts into Notebook LM and listened to the podcast. And I was like, my God, I actually like, it really did pull out the best, like interesting insights. And then I could dive deeper into those individual transcripts. And I actually sent it to him and

said, what do you think? how accurate is this based on like, you were on all of these calls? And he was like, it absolutely nailed it.

Ben (51:42)
It's crazy. Yeah. I've liked it so much that it's one of my pin tabs now that I don't actually use the podcast feature though. And maybe I did. Yeah. I've used it to just like test it out, but I haven't used it for like a real use case. I don't think, but maybe I should. Yeah.

Mike (51:47)
Mmm.

Not really.

Yeah, well it's good for those long documents and stuff. Like that's quite helpful. Also if you ever want like a boost to your ego, you just upload your CV or your resume or your LinkedIn profile and they're so enthusiastic about your accomplishments. It just makes you feel great, you know.

Ben (52:16)
you

Yeah, nice, I'll do that. We'll finish on one question from Jasper and something that I'm always thinking about as well. So I've tried Super Normal, I tried Fireflies, I've tried the Zoom transcript one. I used Grain a few years ago when they first came out actually and I can't remember what they were originally for. I think it might have been transcriptions but it was before sort of AI was...

fully in its now form. Do you have perspective on the accuracy, which is the best in terms of a transcript, do you think?

Mike (53:01)
Yeah, do you know what? I know you're a big tools guy, given your background with the last business you ran in.

and everything, like, I'm really not like, I very rarely compare and contrast these tools. And unless I have like a really catastrophic problem with them or like the, know, they really massively increase the pricing. So, like I just asked a friend, like which one should I use? And he said, grain. said, great, I just did it. I don't, really thought about it again. I would say like the only reason I would bother comparing all the different tools

is like if I heard from some of my smart friends that like one was significantly better than the other. But I think in general, most of them tend to converge on the same sort of quality. And audio transcription is actually like.

relatively open compared to the other types of AI. Like you have image generation, which is all open source, right, with flux and stable diffusion. you have, but text generation is mostly closed source, like, you know, GPT-4 and Claude. But, you know, transcription is actually pretty widely open source because OpenAI has actually been very open in that space. they keep open sourcing like the whisper model, which is, which is really good. So I suspect that either all these tools use whisper or they use

something that has to be at least as good as Whisper because otherwise, how could you justify that to CTO if you're getting worse results than the free open source model that's out there? So yeah, it's not something I spend a lot of time thinking about. I tend to think more about processes than tools.

Ben (54:44)
Yeah, that makes sense because the grain integrated into Zapier then helps all of that. yeah, I've used, so one I use now is called Granola, which I prefer over everything else because it doesn't require like joining a course. There's not some bot sitting there. And Robin, you just mentioned one called Jamie, which is a bot free one. So it's like, it's not another thing in a call where the AI note takers outnumber the people on the call. It sits in the background. And one thing that I've found

Mike (55:08)
Mmm.

I've had that, yeah.

Ben (55:15)
One of the best things I think of it, and I'm not an investor or anything, but I was close to being, one of the best things I've found is when you're doing a Slack huddle. So I always do a huddle with my business partner, Shanice, and it now comes up and says huddle detected, and then it will take the transcript and summarize that. Whereas there's no other way that actually I can get that transcript, because we huddle all the time and we just forget what we've said or what there is.

Mike (55:21)
Mm-hmm.

nice.

Ben (55:44)
like action items or anything like that so granola for me has been really good because I like the

Mike (55:49)
Yeah, and that one runs locally on your computer, does it?

Ben (55:54)
Yeah, I mean, it's just, yeah, an app and they've got a really generous free tier and the paid tier is like $10 a month or something really small. and yeah, I had, I also tried Crisp because that was one that was before AI. then I think the reason I ever switched from these is not cause I'm a tools guy, which I am, but it's because they wouldn't, wouldn't do the job properly. like super normal and Fireflies often wouldn't join a call. I was like, well, that's the first thing that you need.

Mike (56:18)
Yeah.

Ben (56:23)
in order to work. if I can't do that, then I'm already like out the door. And then every time I was signing up in the flyer flies, I had to like re-authenticate with Google every time it was like I signing in as a new user and just stuff like that. was like, I don't need any of this. And so then yeah, I went to Granola and I haven't looked back since. So yeah, I'm liking that one at the moment. And actually Riverside luckily does a transcription. I mean, all of the, like so many of these tools just have it.

Mike (56:25)
Mm.

Yeah, agreed.

Nice.

Ben (56:51)
baked in, which like you said, they're all probably using whisper. So it's all going to be fairly accurate. so yeah, I'm finding it because I think with Riverside, I need anything that's doing it sort of on the fly. So granola is doing a transcript on the fly as well. But previously when I did zoom for workshops and I put dump that video into the script to do the editing, it's generating a transcript based on the video rather than real time, which I think is where

Mike (57:14)
Mm-hmm.

Ben (57:21)
It actually is better when it's real time than after the fact. So I've seen less issues with spelling of like chatGBT for example from that experience anyway. that's my do then.

Mike (57:23)
interesting.

Yeah. Yeah, my D script transcripts always say chat-jip-ty and... Yeah.

Ben (57:39)
Yeah, exactly. Yeah, you think that would be one of the things that like they can recognize and like already do. But yeah, that comes.

Mike (57:45)
Yeah, I'm not sure why the Descript one is so bad because they've been so far ahead on AI. But yeah, I use Descript a lot for recording stuff. I find the AI features are quite nice for that.

Ben (57:51)
I

Yeah, although now I've left that behind and on Riverside. yeah, anyway, Mike, thanks so much for your time today and chatting us through your process and then diving into prompt engineering. That was really good. Yeah, very appreciated. And thanks everyone for joining. We will see everyone on the next one.

Mike (58:03)
Yeah.

Yeah, it's pleasure.

Cool, thanks everyone.

Ben (58:19)
Good luck. Bye.

‍