If you’re a website owner or operator, you’ve likely heard about Googlebot, the digital spider that crawls the internet to index websites. Understanding how frequently Googlebot visits your website is a crucial piece of knowledge for anyone looking to enhance their online presence. In this article, I’ll share my insight into mysteries of Googlebot’s crawling behavior with a look at my own websites and provide insights into optimizing your site’s crawl frequency.
Key Takeaways On Googling Crawling Of Websites;
- Surprisingly The Googlebot will crawl even small or new sites at least once a day (data).
- It is important to add your site to Google Search Console.
- Crawling and indexing is not the same. Indexing takes longer (tips below).
- Backlinks, social signals and brand recognition are major factors for being indexed fast.
- Having topical authority and good Core Web Vital performance helps index faster.
This may be a surprise to most website owners but even if you have a new site with little content the Googlebot will most likely craw; it a few times per day! However it may not update the Google index for several days or weeks. The frequency on which Googlebot will index new content and update their index for old content depends on the value of the site. I’ll address some of those factors later on in the post.
First, let me show you an example of site I have that is brand new. Actually this is hardly a new site. Just about 2 weeks ago a ‘coming soon’ page was put in the new down. There’s still no other content on it. And boom look at this, getting crawled a few times per day.
From the image above, you can see Google is regularly crawling this site… waiting for the day I build it out lol.
Now let’s look at another small site.
This one is well established with quality content on a very specific topic, and I would consider it a ‘topical authority’. This site is about 7 years old, has about 150 pages of content and gets about 200 visitors per day. It’s a small site relative to what’s out there in the online world.
As you can see. It gets about 200 Crawl requests per day. When I publish on this site, the content is indexed and usually shows up in Google within 3 hours
These screenshots showing the crawl stats of my sites were taken from Google Search Console which you can use to see if Google is crawling your site. The other option
Google Will Crawl Automatically But The Are A Few Things You Can Do To Speed It Up.
The Google bot is always crawling the web looking for new sites and content, but it’s best if you do somethings to point the crawlers in the right direction.
Add it to Google Search Console.
Google Search Console is a free and important tool that show webmasters details on how their site is performing, interacting, and behaving with the search and engine and it’s bots. You can sign up here, and once you sign up, you’ll have to verify ownership of the site (usually by adding a tag in your domain name registrar).
Once your domain is verified, Google will crawl your website daily as you can see from the screen shot of my website example.
How to find the crawl stats report.
The crawl stats report in Google Search Console provides details on when your site was last crawled and the frequency. It is a bit tricky to find though. This screen shot from my account should help you navigate.
Create & Submit a site map.
Ah the wonderful world of sitemaps.
Sitemaps are arguably irrelevant in 2023 as the Googlebot is a very effective at finding and indexing new content. If your site is aged and established you really don’t need a sitemap. However with a newer site or site that is having problems getting traffic, a sitemap provides Google direction on exactly where it should look for updates.
Personally speaking I don’t have sitemaps on my older and established sites, but creating a sitemap is a simple ‘best practice’ that Google recommends.
Creating a sitemap on WordPress is easy — just use a plugin like free Yoast SEO (this is what I use) or one one of the other free sitemap programs. There is really no ‘best’ sitemap, so just use whatever one feels the most intuitive to you.
Once the sitemap is created and you have a link to the sitemap, it’s time to submit it in Search Console. Navigate to ‘Sitemaps’ on the left navigation and then input the URL to your sitemap. It’s that easy!
Once your sitemap is submitted, Google will crawl it very quickly, within a day in my experience. At that point you’re done. You don’t need to resubmit the sitemap or submit it again next month. My own sitemaps have been left untouched for years.
Now let’s talk about indexing.
Just because your site is getting crawled doesn’t mean it is getting indexed, or indexed fast. Indexing is the process of taking the data the crawler found on your website and getting into the index that is served in search results. This could be showing a new page, or showing new content on a page that was updated.
In other words, just because your site is being crawled, do not mean that it is being indexed and showing up in Google!
My preferred method of checking if my site or a specific page has been indexed is simply going to Google Search and doing a search for site:mywebsite.com. That will show nearly all pages that have been indexed the results near the top are the pages on the site Google prioritizes in ranking. To check indexing of a specific page, just use the full URL with site:mywebsite.com/the-awesome-new page.
Here’s what you can do to get your website indexing to happen faster, so your content show in Google Search.
Optimize website performance and experience.
You may have heard of ‘Core Web Vitals’. These metrics measure the performance of your website and experience your site delivers to users, and Google uses them as a search ranking factor as of 2021.
In my experience sites, that get a passing score both on mobile and desktop get indexed much quicker than those that do not. For that reason, I have made it a priority to ensure all of my sites pass Core Web Vitals and maintain that passing score.
Passing Core Web vitals can be tricky. I use a fast hosting service, and WP rocket to get the passing score on my sites.
Once sites receive a fair sample size of traffic, the Core web vitals will be reported in search console. Until then you can check that each score of the 6 Core Web Vital metrics are within passing range here at the google web dev speed tool.
Build topical authority (semantic relevance) for faster indexing.
When it comes to getting your website’s content indexed faster by Google, it’s not just about technical optimizations. Building topical authority, also known as semantic relevance, is a crucial and often overlooked aspect that can significantly expedite the indexing process.
One common trend I have noticed when analyzing sites that are having indexing problems is that their content topics are ‘all over the place’. When this happens Google sees no topical authority within the site and gives it a lower priority. This hodge-podge of content also leads to poor site structure.
On the flip site, sites I work with that have depth, semantic relevance, and strong interlinking on the topics they cover index very quickly.
Let me give you an example of topical authority problem to solution. A site has 1,000 blog posts on ‘pickup trucks’. All of the topics are in the category of ‘pickup trucks’ but they cover dozens of different models and year ranges. The site is NOT indexing fast and performing poorly in search.
To harness the power of topical authority a site should narrow down and focus on a few specific sub categories covering every possible aspect of it. In this truck niche example, an subcategories might be Toyota Tacoma, Dodge Ram, and Nissan Titan. To keep everything consistent, only publishing on the late model vehicles might be ideal. The site should ‘go deep’ on topics for those models until they’ve covered nearly everything on it and they are a topical authority. No-indexing or removing unrelated content is likely advisable until the site shows growth in the areas for which they are working to achieve topical authority.
Backlinks, social signals and branded traffic.
In the realm of search engine optimization (SEO), backlinks, social signals, and branded traffic are powerful factors that show Google that other users online find your content valuable. And when Google considers your content valuable it gets indexed faster.
These aspects are broad and ever changing so I am not going to cover them in depth in this post.
At a simple level, when you publish new content, post it on social media, and send it out to your email list. While it may seem like fruitless task these small things will add up to faster indexing and better rankings when done consistently for the long haul.
Problems With Crawling & Indexing.
This is a rolling FAQ to be expanded upon as needed.
How do I fix ‘crawled but not indexed?
This message comes up frequently in search console. The simple reason for it is that Google thinks the content is not valuable enough to index and show in your search results. Sites that get this message for many pages usually lack topical authority and semantic relevance. To fix it you should focus on establishing topical authority by going in depth on the topics and pruning (deleting or no-indexing) content that does not add to that depth. Improving page experience and core web vitals, along with increasing social signals and inbound links will also help fix it. Look to my section on ‘talk about indexing’ for more information on these aspects.