What We Built And Why

Speaker 0:00

Hello, and welcome back to SEO is Not That Hard. It's me here, Ed Dawson, as always, and today I'm gonna talk about what I've been basically been up to for the last 11 months Which is building Salients at salients.com. So what is Salients? What is this product that we've been building? Let's take a step back, okay? There's- A whole load of data that every website collects that most people tend not to look at for a few reasons. First reason being most people find it hard to get hold of the data, and I'll explain more about that later. And secondly, because even if you do get hold of that data, it's not particularly user-friendly to read. Okay?

Server Logs Explained Simply

Speaker 0:44

So what is a server log? So we'll start at the very basics.

Speaker 2 0:47

A server log is where every single request that's made to your website is stored. And by a request, that is every single time a web browser or a crawler or an AI agent makes a request for any content on your site, every item that is requested will be recorded in the server log. So the time of the request, the IP address of who asked for it, the user agent, the path that they requested from. Sometimes you get referrer information, geogra-geography information, like where they're coming from, a whole bunch of stuff that can be up to th-30-plus items, depending on how your web server's been configured. So this is like the ground zero of every single request that comes into your site, and there is no getting out of this. Every request is recorded in this server log. It's not like Google Analytics, where it requires JavaScript running to fire that the request has been made. It doesn't require anyone to accept a cookie or anything like that. Basically, every single request will come in here, and there's no way that anyone can g- can get out of it. Now, this is great because you get to see all the stuff that you can't see in a traditional analytics. So you get to see when Google Bot came, when it-- what content it has looked at. You get to see when AI agents have

What Logs Reveal Beyond Analytics

Speaker 2 2:08

come to your site. You get to see when normal people have come to your site. You get to see All the activity that is normally hidden And this is a huge amount of first-party data that you own that you probably are completely unaware of. Now, the reason is these server logs aren't user-friendly to read and they are huge. You will-- some sites can get millions of requests a day. Even a small site can get thousands of requests a day. And also, they're not that easy to get hold of because a lot of the time they're sat in file systems on a server that you may not have access to The whole point of Salient is to make gaining access to those logs simple, one-time thing And secondly, to then help you make sense of those log files, to give you an an- an analytics dashboard that allows you to understand what's going on, who's requesting data from your site, when they're coming, what they're doing And this is essentially to help turn all that data into plain answers for you. What's been requested, how the site's responded, what needs attention, whether you're blocking crawlers that you do or don't want to crawl, what missing pages people are requesting, e-errors only a bot ever sees, sus- suspicious traffics, people trying to hack your site. That's really good for seeing hacking and things like that. And the fantastic thing is you haven't got to actually add anything to your site. There's no tracking script, there's no cookies. All we need is to connect your logs to Salient and then and it's a one-time t-thing to do that, and then we can keep an eye on it for you and help you make sense of it And it's particularly important in this new AI age because you can get to see what ChatGPT user is requesting on your site, what Claude user is requesting, what Perplexity user is requesting. When a real person asks a real question about you, y- you don't see that in analytics because that person doesn't come to your site. They're going through an AI agent. But you can see the AI agent making a request to your site, which you won't see in analytics because they don't trigger the JavaScript that Google Analytics require-- relies on You can also find out if you're blocking things that you shouldn't be because you get to see not just the requests that succeeded, but also the ones that failed. Sometimes fail because you've blocked it. You can get to check whether your robot's text is being respected By third-party bots. There's a huge amount of stuff you can do, so I'm really just scratching the surface here. So that's what we do in a nutshell, is we help you access and then understand your server logs, and then take action on everything that surfaces from them.

From Sick Sofa MVP To Product

Speaker 2 4:49

So how it was built, and this is the bit that would've been very different a few years ago. So I mentioned in, I think it was maybe last episode or the episode before, that Christmas 2025, I wasn't feeling very well. I had the flu or COVID, one of the two, and I was ill on the sofa with my laptop, and I was playing with, I think Claude Code. It was Claude Code. I was just getting into Claude Code at the time and playing about with it, and I just wondered if I could get at my server logs and build something to help me understand them. Now, as I mentioned before, this is not a solo project of mine. I'm actually-- we've built a company me, Carl Hendy, and Nick Reading

Speaker 3 5:36

Now we'd initially been playing about with the idea of trying to do prompt tracking, which w- a lot of people and a lot of companies are now doing. And we'd actually built some prompt tracking software of our own, but we were finding the issues with prompt tracking, which is something I'll go into another day. But Karl had always been saying to me, "I wish we could get access to server logs more easily. I... There's so much in server logs we could use. It's s- it's so important for technical SEO. There's things in there that using server logs, you can improve sites. The insights it gives you for making improvements is just, it's just incredible." So there was me sat on the sofa feeling ill, sorry for myself, and I just thought, "I wonder if I can do something about logs," because it kept coming up in my mind that this is what Karl wanted to do. And yeah, I spent that Christmas period getting the very first sort of minimum viable product together for streaming log files from some of my websites into a database, essentially, into a data warehouse to be able to then perform analytics on them and perform analysis and try and learn from them. And yeah, I managed to get a very basic, raw, not very optimized, but basic system working, and I demoed it to Carl and Nick, and they loved it. They just thought, "Yeah this is a, this is-- this could be a really good product." So we then have basically, in a nutshell, spent the last 11 months building it out and taking it from that very basic proof of concept And turning it into a product that is robust, scalable, secure and enterprise-ready And we've Been using it with real clients since February, with clients of Carl and Nick's consulting agency. And these are some-- I can't name them obviously because it's all it's all under non-disclosure. But they're, they're very big companies. Some sites with hundreds of millions of requests a month. We've put huge amount of volume of traffic through it. We've found all the holes, we've patched all the problems, and we brought it to the public about two weeks ago. Yeah in mid-October time. We didn't make too much noise but just enough to start getting a few people on. And we've now got quite a lot of sites on there and yeah, the, it, it-- we're getting some really fantastic feedback.

Building With A Software Factory

Speaker 3 8:18

Now, the thing that we've done different in how we've built it, couple of years ago, we'd have needed a team of dozens of people to build this. But with the advent of AI agents, it's the kind of thing that we've now been able to build, not in a vibe coded way. Now this is, wanna be very clear, this is not vibe coding. This has been built to industry standard protocols. We've got a full proper pipeline for launching, testing, and rolling out new functionality

Speaker 4 8:53

What we've built is essentially a software factory. And the idea of a software factory is that you stop treating building software as a craft done one piece at a time, and you start to treat it like a production line with stages, quality gates, and measurement built in. So where people own the decisions and machines own the repetition, essentially. So that's what we've ended up building around Salient, and it works like this. So I will decide the scope and write a brief, the problem, and an example of what it is that we're trying to get done, what we're trying to build, what we're trying to add to the product, and what done looks like, and what tests must pass. This is where the software engineering comes out, okay? I'm saying to it, this is what we've gotta do. This is h- this is the debriefer on how we're gonna do it, and this is how we're gonna prove that it's been done." And I then give it to a manager agent, which will essentially turn that into a bounded specification, and it'll give it to an implementer agent, which will then work in its own isolated copy of the code with no power to put anything live or launch anything. And when it's cons- when it's decided it's finished, a reviewer agent will then look at what it's done, run the test against it, run browser checks, and will send corrections back to that agent if it's not got it right or if there are things that it's missed. And the- they will loop around until they think they have reached the point where they've met the specification. And at that point, it will go back to the manager agent, who will do, again, a whole bunch of ch- tests. Automated tests will go on it. And when it has decided that it is good enough, a preview app... Sorry, a preview version of the app will be built so that then a human, like myself, or if it's Nick or Carl, can then run through that preview version of the app, check that it works, be happy that it works, and then, and only then, will they decide to merge it into the actual application All this happens through a system of call, it's called continuous integration. So even at that point when it gets merged in, then on the actual live app, a whole load more tests get done. If that fails and those tests fail, it will roll back and send it back to the agents to be looked at. Every release gets recorded with what's changed, why, who did it, which agents did it And this is all recorded all in GitHub, so no one has to remember these things. So it means that at any point, if we ever find a problem further down the line, we can go back and we can revert versions all the way back. So this is what I mean about it not being vibe coded. This is proper application development like large scale teams would do. It's just that we can use agents now for doing a lot of the work so we can get so much work done in a period of time that just would have been absolutely impossible even a year ago. Things have just, have moved on even since just a year ago But it still very much requires a human in the loop. It's only a human that decides something's gonna get merged. A human has the final say. And we will still look at code if we need to. If we're not happy, we'll we will interrogate the agents in terms of what, what decisions they've made. We will p-pick up issues if issues have come in, and we w- we can go back to them with a, with feedback on it. And this, this loop, this continuous integration, this continuous testing, this model is this software factory model that we've built it with. And I think you're probably gonna hear more about software factories from people in the industry going forward. This is the new paradigm for producing robust, secure, scalable software at speed, is this factory model So what does Salience actually do over and above just like getting your server logs in and then giving you some dashboards?

Getting Logs In From CDN

Speaker 4 12:59

First of all, it reads the data, your server or your CDN. If you don't know what CDN is, that's a content delivery network. If that's things like Cloudflare, CloudFront Which so many sites use now, and that's the method that we use to get the data in. So how it works is you come to... You go to Sadiance.com, you put in your domain. We work out how your website is exposed to the internet. If it's through a CDN, say it's through Cloudflare, we'll say, "Here, this is-- You're on Cloudflare. Click this button, auth- authenticate." We'll get it set up, and then Cloudflare will st- send that log data to us. If you've got a setup where you're directly serving from Nginx or Apache, we've got ways to get that streaming in. You can, It runs you through, the onboarding's really good. It runs you through how to set the stream up, and then your logs start streaming to us And from this point, we can then identify who made every single request to you, whether it's person, Googlebot, Bingbot, GPTbot, Claude Bot, Perplexity bot. With over 200 named agents that we know so far, and we're picking up more all the time. We'll verify them. A lot of traffic claims, claims to be Googlebot when it isn't. This is people trying to get around, you stop, get around scraping controls. They try and pretend to be Googlebot, so you'll give the content to them. But what we do is we do reverse DNS, forward DNS, and look at published IP ranges, so we can work out whether you're actually getting real Googlebot traffic or whether it's spoofed. We can then turn that into plain answers so we can show you what's been requested, how the site responded, what needed attention. For example, one example might be, are you serving Googlebot really slow content? That's an issue you need to fix if you are, and we can tell

Bot Verification And Practical Fixes

Speaker 4 14:39

you which bits of content are serving slowly if they are. We can confirm which c-crawlers are blocked. We can find the missing pages. We can find the errors only bots are seeing, and we can see suspicious traffic. We pick up so many hacking attempts, it's unbelievable. And yeah, as I said, there is no tracking script. This is w- once you've done that one-time setup, then we see everything. And I'm only just scratching the surface here. The-- I will do more on Salient in future podcasts that come through this feed So who should care about this and who shouldn't? I think everyone should care about this. Okay, if you're a small brochure site with only a few hundred visits a month, then this, it's not really gonna do a huge amount for you. But at that volume level, it's free. That's the great thing. We, we only charge people once you get past half a million requests a month. If you're getting just a few thousand visits a month or a few hundred visits a month, you're not gonna go anywhere near that, and you c- you can access this for free. So while it's not necessarily gonna make huge amounts of difference probably to your life if you're that kind of site, it's all still good information to know. But if org- if organic traffic is your business, then you need to see the bots because the share of your audience that's now machines reading on behalf of a person, those agents, is growing every

Who It’s For Pricing And Trials

Speaker 4 15:57

month, and currently you're blind to it if you're not looking at your server logs. If you're an agency or a consultant, this is the audit layer that you have been doing by hand probably before. There is stuff in here that you can go to pretty much immediately for, to your clients with actionable stuff. We will f- this finds those problems. Even if it's just like you've been blocking GPT bot all this time. No wonder you're not getting, getting recommended by Claude, by ChatGPT. You're blocking the, you're blocking all the ChatGPT bots. We'll find things like that immediately And you've gotta remember that these AI referrals are few. You see a few of them at the moment in Google Analytics when someone's been referred by an AI, but you're not seeing all the times that agents or crawlers from the AI LLMs have visited your site in all the conversations that preceded that person coming on and making a sale And yeah, going back to the prompt tracking, which I mentioned before, is so hard because there's no short tail. We can pick up when prompts have triggered a fetch. We can't tell you what the prompt is yet, but we can tell you the times when actual prompts have triggered a fetch. This is the kind of thing that you just cannot get in normal analytics Now, if you wanna try it, just go to sellience.com. There's a full demo on there. You can just click the demo. It's around one of my-- goes around one of my actual sites, and it shows you the entire interface so you can see exactly how it works. If you wanna try it out for free, I say if you're under half a million requests a month, it's free anyway. Anything over that, we've got free trials, fourteen-day trials, no credit card required, so you can try it, get connected. If you don't learn anything from it, I'd be hugely surprised. But yeah, I don't wanna pitch, even though I've just spent probably the past thirty seconds pitching. I just wanted to explain what it is we've been building and why.

What’s Next Subscribe And Review

Speaker 5 17:58

So what comes next on this feed? I'm not going back to three episodes a week on SEO Is Not That Hard Because I just don't have the time at the moment. It's so busy with Salient and with other projects keywords people use and my other sites, i've got a million and one things on at the moment. What's fantastic is I-- the productivity levels you can get with AI nowadays are unbelievable. But it still does leave me short. There are only 24 hours in a day. I'm asleep for a good number of them, eating for some of the others, and I do wanna have a bit of a life as well. So t- three episodes a week on a series is not that hard. It's, it's-- would be a stretch. But I will definitely be back with episodes a bit more frequently than once every 11 months, I hope. There are gonna be other podcasts, videos, all sorts of things coming with Salient, with Nick and Carl, and some other things around AI specific content that I wanna put out that will not be SEO necessarily. And I'm probably gonna do another podcast for them. But I will put a copy of all these things into this feed as well you won't miss it. If you stay subscribed to this feed, you w- you won't miss it. If you're just listening to this first time and you haven't subscribed, then do subscribe. And subscribe as they say. Comment even better. Leave a review, all those kind of things, all would be brilliant

Speaker 6 19:26

So thanks for listening. Really appreciate it. As I say, I won't be 11 months. I'll be back before then. And yeah, thanks for listening, and take care of yourself, and I'll see you soon