Downloading is free and you can get up to 500 URL crawls under the free version with lots of functionality. Note we are not affiliated in any way but this is a well known and accepted tool for SEOs in the industry. Another check is also on the backend of your website if your CMS allows for you to perform a search. In this case we can search with the following ‘lifetime warranty site:which will result based on googles indexed pages for pages that are relevant to this search term. What we also did in this exercise is to check with a google search with an advanced operator. Screaming frog is a super powerful tool and for those who work on campaigns with small websites can take advantage of the fact that it will crawl 500 URLs for free. Select the dropdown and ‘Ignore robots.txt’Īs with any tool, it’s not always going to be 100% so it always pays to double check. Go to robots.txt settings by clicking on the menu and going to Configuration > robots.txt > Settings Step 2 – Ignore robots.txt What this does is makes the crawler effectively crawl everything it can and not stop even if robots.txt tells it not to. To go that one step further, we make our robot a little naughty and tell it to ignore robots.txt. However, this does not mean there isn’t a page linked somewhere on the site which might still contain the text you are looking for.
Pro Tip: Robots.txt – This is an important oneĪs a good crawler should, Screaming Frog is set to obey robots.txt as a good robot should.
The reason being is that there is a piece of code which noted something like ‘id_warranty’ in the HTML so this crawler has picked up on it. The reality of this particular website is that ‘warranty’ does not appear on the webpage when you land. You will see in the above screenshot that there are 27 instances of ‘warranty’.
From there, you can check through the resulting URLs to then go through and update the relevant pages on your live website. The overview will show your search results and you can also select which filter you want to look at on the left hand side. You will find your search results under the ‘custom’ tab. I’ll explain later why this is important to know. What this means is it searches the coding of the page and not necessarily what you see as a user. Note: As you can see in the screenshot, this searches the source code of the internal HTML page. It would be much easier filtering through the pages where the code doesn’t appear rather than potentially hundreds of pages where the code does appear. Often this might be something like your chat tool or a tool like Hotjar, Google Tag Manager or Google Analytics. When would you use this? if you have a piece of code that needs to appear on many pages or all pages. Pro Tip: ‘Does Not Contain’ is a search option. In the screenshot below, you can see we searched ‘lifetime’, ‘lifetime warranty’ and ‘warranty’. Great if you need to change an email address and a phone number. You can enter multiple search terms or ‘filters’ so you only need to perform one crawl to get data for multiple searches. This is where you input the search terms you are looking for. You’ll then get to the ‘Custom Search’ screen. On the menu, go to Configuration > Custom > Search
Solution: This is how to use Screaming Frog to find that Specific Text Load your Screaming Frog applicationĮnter in URL details as you normally would. All references to ‘lifetime warranty’ had previously been removed but our clients were concerned that there may have ben an instance of ‘lifetime’ lingering around on the website. We now offer a very lengthy warranty period but no longer advertise ‘lifetime’. A website we work with once advertised and offered a ‘lifetime warranty’ on its products. The example today where we use Screaming Frog to make what could be an arduous task easy is this. Case Study Scenario why we used Screaming Frog to find specific content For example, a company changes phone number or email address and we need to find any page that this information exists. This can be so useful in so many scenarios.
In context of todays topic, we can also use it to find specific text on a website.
After scraping your website, it can help you to visualise the structure of the website, finds broken links and is full of useful information for anything from H1s, Metas, Canonicals. Screaming frog is one of the many SEO tools available with plenty of great features for SEOs. Download Screaming Frog SEO Spider Tool.Pro Tip: Robots.txt – This is an important one.Solution: This is how to use Screaming Frog to find that Specific Text.Case Study Scenario why we used Screaming Frog to find specific content.