Inbound Marketing vs. Outbound Marketing
February 1, 2022
How Can I Learn SEO on My Own?
February 3, 2022Sitemap: How to Set-up & Structure Your Sitemap for SEO
Sitemaps are a critical and often overlooked element of your site structure, and they play a crucial role for search engines looking to index information about your site.
Major sitemap issues include:
- Lack of a sitemap or sitemap that’s not submitted to Google Search Console (GSC)
- Outdated sitemap
- Overly large or complicated sitemap (a.k.a. sitemap not optimized for a limited crawl budget )
- General sitemap code errors (e.g. 301 redirects or 404 errors)
If your sitemap falls out of date and you don’t take measures to correct it, you could pay the price with reduced search visibility and therefore, less traffic to your site.
Why Sitemaps Are Beneficial
Onsite sitemaps are a link on your main site that contain a structured data layout of all your existing pages.
They’re crawled by Google and inform the search engine about the remainder of your onsite pages.
XML Sitemaps are files that you can build and submit to Google directly. Essentially, these files are a condensed map that lays out the structure and hierarchy of your site.
By submitting a sitemap to Google, you’re telling the search engine to review and index the pages of your site—and this is a critical first step when you’re launching something new.
Sitemaps are basically instructions that allow search engines to find your pages faster and more accurately. Keeping them updated ensures that Google has the best understanding of your overall website, and the greatest number of your pages are showing up for the appropriate searches.
How to Implement a Sitemap
If you’re using WordPress as your CMS, then I would suggest the Yoast SEO plugin to create and manage your sitemaps.
When it comes to structure, we suggest the simplest sitemap structure possible.
For instance, in our sitemap, we have removed many of the extensive sub-sitemaps available in the Yoast plugin, including:
- Categories
- Tags
- Authors (we keep author archives “on” but remove the Author sitemap)
- Portfolios
- Templates
- Downloads
Unless we want specific indexation, the only sitemaps we include in our sitemap at SEO.co are for “Pages” and “Posts”:
The name of the game is K.I.S.S. (keep it simple stupid). Stick to “meat and potatoes” content.
Google only has to crawl the following three pages to access ALL the content on our site:
- https://seo.co/sitemap_index.xml (with only two simple links to the other two sitemaps)
- https://seo.co/post-sitemap.xml
- https://seo.co/page-sitemap.xml
If you have a shop in Woocommerce with numerous SKUs, then indexing your store products with a separate sitemap is advised.
But the ideal sitemap will have the least number of steps for the webcrawlers to get access to the content.
Crawlers can be fickle and rarely will crawl (let alone index or re-index) your entire site in a single pass.
You can turn off additional sub-maps in the Yoast features:
If you’re not on WordPress, you can use a custom XML sitemap generator. I would suggest https://www.xml-sitemaps.com/ for a simple, free solution.
The Dangers of an Outdated Sitemap
If your sitemap isn’t up-to-date, you could be providing inaccurate data to search engines.
Depending on the severity of your inaccuracies, this could have major consequences or minimal impact.
For example, if one of your hundred product pages drops off, you won’t see much of an impact.
However, if you’ve restructured your entire navigation, search engines could be confused when they attempt to crawl your site, and you may lose indexed pages as a result. In addition to having a smaller number of indexed, searchable pages, your domain authority could even take a hit.
The bottom line here is that an outdated sitemap will send outdated information to Google—and while Google, in some cases, is smart enough to make sense of these discrepancies on its own, the safer play is to ensure your sitemaps are always up-to-date.
How Your Sitemap Can Become Outdated
Sitemaps don’t become obsolete on their own. Only through a deliberate change in your site, usually an increase or decrease in the number of pages, can make your previously submitted sitemap outdated. Keep a close eye on the changes you make to your site, and if you do make a significant change, take efforts to keep your sitemap updated accordingly.
Adding and Removing Pages
By far the most common reason for a sitemap becoming outdated is the addition or removal of a core page. Even traditional, static websites experience the need for change from time to time—whether that’s the addition of a new service page or the removal of a special offers page that’s no longer relevant. While some regularly updated sections of your website (such as a blog or press page) will be routinely scanned by Google, any major page changes will need to be reflected in an updated sitemap.
Redesigning the Site or Navigation
Restructuring your site will also require an update to your sitemap. In addition to simply listing out the pages of your site, the sitemap is responsible for showcasing the hierarchy of your web presence, outlining the most important pages in a very specific order. If you make major changes to your navigation or restructure your page-based priorities, you’ll need to update your sitemap.
Adding or Removing Products or Listings
E-commerce sites and sites with classified-style postings (like job opportunities) tend to be the most vulnerable to sitemaps falling into obsolescence. Since most of these sites have large volumes of products and listings, sometimes numbering in the thousands, it’s common for new blog posts to be written and old postings to be taken down. Fortunately, a dynamic sitemap can spare you the pain of manually updating a sitemap every time you make a minor change, but you will have to routinely check to ensure your sitemap is accurate and up-to-date.
Sitemap Diagnostics
The easiest way to test whether your sitemap is current or outdated is to check it using Google Webmaster Tools. If you haven’t yet uploaded a sitemap here, you can start from scratch. If you need help creating your sitemap from scratch, be sure to read up on Google’s guidelines for building a sitemap.
Once submitted, you might encounter errors during the upload process:
- If you see a Compression Error, Empty Sitemap, HTTP Error (specific code), Incorrect Namespace, or Incorrect Sitemap Index Format, there is likely a problem with the format of the sitemap you submitted. These problems are generally easily fixed, and do not necessarily indicate a problem with the links and structure included in your map.
- If you see Invalid or Missing errors, a Parsing Error, or a Path mismatch, it generally means there is a formatting error in the body of your sitemap that needs to be corrected.
And once the sitemap is accepted, you may find errors with your sitemap. Perform a test by clicking on your intended sitemap, and clicking Test Sitemap in the top right corner. From there, you’ll be able to Open Test Results and view the results of the test.
The test will tell you what type of content was submitted, in a quantifiable data table, including the number of web pages and videos that were submitted in the test. Any errors that Google encountered, which prevented it from indexing a page that was submitted, will be displayed. Some errors arise when there isn’t a page present where one should be according to the sitemap. Others are based on outside factors, such as server-related problems, or the presence of a robots.txt file blocking Google crawlers from discovering it.
How to Test Your Sitemap and Site Structure
Unfortunately, your sitemap may not reveal hidden SEO issues (e.g. 4XXs, 3XXs, title tag, UTM, meta and internal link) that could be harming your rankings and hampering the webcrawlers from accessing specific content.
You can either hire an SEO to complete a comprehensive SEO audit or, alternatively you can use a combination of the following tools to find errors in your site structure, including your sitemap:
- Google Search Console. If Google sees issues with your site, they will generally let you know by providing email and alerts through the GSC console. But, even as the holy grail, GSC may not be the first to pick up issues with your site. And, it’s better for you to find an issue with your site (with the help of ancillary tools) than for Google to find it first.
- ScreamingFrog. As a crawler, ScreamingFrog (SF) looks at your site as Google would. You won’t see the issues that Googlebot does, but a SF scan will eek those out. It’s $150/year for unlimited scans. We use it on every managed SEO account before we do ANYTHING else.
- Ahrefs. From time to time, issues may popup unawares. That’s why a regular automated audit can help find issues even when you’re not looking. Ahrefs has a feature that will automatically crawl your site, looking for issues at regular intervals. For instance, we have Ahrefs scan all of our and our clients sites once a week. We receive the report via email and fix emails as they arise.
Example result of an Ahrefs site scan.
Each of these tools has its place and sometimes one will eek out a different finding than the others. Consequently, it’s helpful to use them all in tandem to ensure your site maintains a clean bill of health and doesn’t include issues your sitemap may not be telling.
If you notice any of errors preventing your sitemap from being indexable, accurate or up-to-date, take a closer look at the breakdown in GSC, make a list of any corrections you need to make, and start making them.
Submitting a New Sitemap
Once ready with your new sitemap, head to the Google Search Console homepage and enter through the site you wish to submit the sitemap for.
Under the Crawl header, click on Sitemaps, select the sitemap you wish to resubmit, and click the Resubmit Sitemap button.
Once resubmitted successfully, you’ll be able to re-run the test you used to find the errors in the first place. Hopefully, all of these errors have been corrected in your revision. If not, you’ll have another opportunity to make corrections and resubmit a new sitemap.
If you’re using static XML sitemaps and you run an e-commerce site or another type of site where pages come and go regularly, you’re in for a lot of work. With a static XML sitemap, you’ll have to manually change and resubmit your work with every change. Instead, you can build a dynamic sitemap and setup automated “pings” to notify the search engine whenever there is a major change.
No matter how you look at it, sitemaps are an important piece of the puzzle when it comes to making sure Google has the right information about your website. Take measures to ensure your sitemap is up-to-date at all times, and you’ll be rewarded with more indexed pages, and more search traffic as a result.
He is a recurring speaker at the Search Marketing Expo conference series and a TEDx Talker. Today he works directly with high-end clients across all verticals to maximize on and off-site SEO ROI through content marketing and link building. Connect with Sam on Linkedin.
- Ultimate Guide and Review of Ahrefs for 2023 - November 10, 2022
- White Label Link Building: Pros & Cons of of Private Label Backlink Services - November 3, 2022
- What Are the Common Reasons Clients Leave SEO Agency? - November 1, 2022
