How Frequently Google Crawls & Indexes Pages

In a webmaster interact, a person asked how fast Google remove pages from the index if they are mentioned as noindex nofollow. He said that he added noindex but the page still remains in Google’s index. Google’s John Mueller responded with an answer that described how often some pages are indexed.

John Mueller said that URLs are crawled at different intervals. That’s somewhat well understood. What was of more interest was that he said some URLs can be crawled as little as once every six months.

The publisher stated:

“We’re seeing stuff that’s from a long time ago, where we’ve changed the noindex nofollow but we’re still seeing it in the index. And this is several months after we’ve changed this.”

Also Read: What Are the Major Changes in SEO 2020

John Mueller answered:

“We don’t crawl URLs with the same frequency all the time. So some URLs are crawled daily, some maybe weekly and remaining URLs every couple of months, maybe even every once half year or so.

So this is the way to make the right balance so that we don’t overload your server.

And if you make changes on your website then there are chances that a lot of changes are noticed up fairly quickly but there will be some leftover ones.

So if you do things like site queries then probably you’ll watch those URLs that get crawled like once every half year. They’ll still be there after months.

It means that if you want that these URLs should not index more, then you should upload the sitemap file with the last modification date so that Google read this and try to double-check these a little bit faster than otherwise.”

Also Read: How to do SEO for Small Businesses

Use Sitemap for Updated Crawling

John Mueller suggested to update the site map and let Googlebot discover the last modification date and using that as a hint for it to go out and crawl the old web pages.