How to Make a Sitemap w/ Screaming Frog SEO Spider (+Video)

Introduction (Video Below)

The following is a transcription of the video that appears below

Hi. This is Michael David from Tasty Placement. I’m the author of WordPress Search Engine Optimization, now out on 2nd Edition on Packt Publishing. My agency is a Google Marketing Platform (analytics) certified agency and Google Certified Adwords Partners.

What You’ll Need

For today’s tutorial, we are going to use Screaming Frog SEO Spider to create a full sitemap of a website. First, what you’ll need. We’re using Screaming Frog version 5.1 which is current in October 2015 and a website. Let’s go. The video quality blows because I’m using CamStudio, but you’ll be able to get through it.

Let’s do the Screaming Frog tutorial on how to create a sitemap

Why do we want a sitemap? Well, we want to create an XML sitemap and I’m not talking about an HTML sitemap page that you put on your website with a link to all of your pages. I don’t think users really use those. Your site should be big enough, if you want to make money, that a page like that is going to be too huge anyways. We’re talking about an XML sitemap, a sitemap.xml file, that we can submit to Google Webmasters or Bing Webmasters to help search engines crawl our site and find what’s important. By the way, don’t use the free sitemap tools online. We have a huge client, 600 employees, and one of the IT guys there used one of the free sitemap programs online and posted the sitemap. Upon later investigation, it didn’t include all the pages on the site. Screaming Frog is a very pro way to do this. You’re going to get a deep crawl, a full crawl, and a very good sitemap out of this that is completely compliant with search engines. So this is the way to do it.

Create the XML Sitemap: Step 1

First step. We’ve opened Screaming Frog and I’m using version 5.1, which is current in October of 2015. Always set the mode, we want to make sure we’re in Spider mode. We’re going to spider a single site. We are going to use tastyplacement.com which is our website. I’m going to start it and then it takes about 3 1/2 minutes to crawl so I’ll start it and then pause the video and then just resume when it finishes. So we’ve started it and you can see it’s already chewing and moving pretty quickly. I’ll be back in a sec.

After Screaming Frog Completes the Website Crawl

Okay, Screaming Frog just completed its crawl of tastyplacement.com and you can tell when it’s finished because this bar will progress eventually to 100%. This is a good time to sorta look through what you’ve got here. How do we do that? We can go over here and you can see we’ve crawled 798 elements and that includes JavaScript HTML, CSS files, images, PDFs. If you click on HTML here, you’re going to see only HTML files in this window and you’ll see a count here, 304 pages. That’s the number of pages we’ve got and that’s including some 301s and no index pages. You can also see you images here if you’re into that. But we’re really interested in creating a XML site of our HTML pages. This is a good point, once you do just a basic spider basic crawl in Screaming Frog, from this window you’re able to see if maybe one of your pages is lacking a title tag. You’ll see blanks here if there’s no title tag. If you scroll to the right you can see whether there’s a better description of what the lengths of those are. You can see we’ve got 155s, 124s, and 138s. So we’re really up to that 155 character standard there. This gets pretty deep though so we will do another video about that at another time.

This is a 2 phase process. First we crawl the site and the second step is we generate the sitemap. We’re 70% done. This crawled effectively, we’ve got the proper page count, we know Screaming Frog crawled it properly, so we can move on from here. Now if you’re at this point and you know you’ve got more than 300 pages and Screaming Frog didn’t find them all, it means you have a crawl problem. Bingo! Another benefit of Screaming Frog because if Screaming Frog can’t find the pages on your site because there aren’t links to all those pages, then a search engine spider may not find them either. That indicates a problem.

Generating the Sitemap

So from here, we’ve done our crawl and Screaming Frog makes this very easy. We’re going to go up to the title bar here to sitemaps and we’re going to create an XML sitemap. You could also create an image of sitemap, we’ll do another video. We’re going to click this and we’re going to get a little menu here. You progress through these tabs before hitting next. We do not want to include no index pages. We’ve no indexed them for a reason. If you know what you’re doing, don’t include no index pages here. These we all leave blank except PDFs. We’ve got about two PDFs we want Google to find. Last modified we leave alone. This is very sound, so if priority. These are good priority numbers, change frequency, it’s totally fine. And images, we do not want to include images in our main XML sitemap. We click next, it’s prenamed sitemap.xml. We’ll overwrite the file I created in the test, save, and that’s it. Let’s take a look at that file to review it. Go over to our temp directory and we’ll open this with notepad ++, which is a great text editor for all kinds of files. There it is and it looks good. We’ve got nice priorities here, higher priorities on the key pages up front. We are set.

Next step would be just to use our XML sitemap wherever we want it. We upload it to Google Webmasters, which is now called Search Console, or Bing Webmaster.

Just another one of the many things that Screaming Frog can do and at Tasty Placement we use this for a couple dozen individual tasks. This is definitely one of the easier ones because they’ve included the sitemap creation right the menu here. It’s 99 a year for Screaming Frog, but when you learn about all the things it can do it’s definitely high level stuff that pros use. Anyway, if you have any questions you can email me at michael@tastyplacement.com. As always, visit our site or follow us on Facebook. We’re always doing super, super high level tips on everything from WordPress to high level SEO and a lot of really advanced Google Analytics and Adwords stuff. Thanks for watching. I hope this helps you.

Create an XML Sitemap With Screaming Frog SEO Spider from michael david on Vimeo.