What exactly does crawling and indexing mean and why’s it so important to SEO?

Introduction

Picture this, you run a company here in the great city of Bristol, you’ve had a company website for a long time, yet management has now decided to invest in digital marketing.

You want the whole of Bristol to know about your company, you want everyone living near Clifton Suspension Bridge, through to apartments in the centre of Bristol, through to the docklands to know what your company offers, the great service you offer, you want the phone ringing off the hook, yet when you call the first seo agency for a quote, they say they will run an audit, then they ring you back, and they say this “your websites not even indexed?- but what does that even mean?

What does not in Google’s index mean?

Well, let us explain, grab a coffee, let’s begin, imagine your in a library, every book has its space, its section, its own number, and this makes sense, as the librarian, well he or she, when a book’s returned is not going to just throw the book into a massive pile, as otherwise their work becomes much harder, someone might ask, I would like this book, and the librarian would have to point to the heap of books, and say its in their somewhere, not a good system, it’s a mess.

Well, what’s needed is organisation, a system where books have their place, they are logged in and out of system, the librarian knows if its out on loan, in the library or needs replacing as bottle of wine was accidentally spilt over it.

Now apply this logic for SEO, Google is a massive library, it knows the status of millions, if not billions of websites, are they live, are they being built, have they been improved.

So, how is this done, through a group of loyal librarians, nope, they would be exhausted after a few seconds, its done my Google’s mighty algorithm, an algorithm that’s so good, so impressive, that good SEO agencies, worth their weight in gold, well they will spend most of their waking hours either figuring out how to get a business to the top of the ranks, or working out what changes the search engine, i.e Google or Bing have introduced, so that they keep their white hat SEO knowledge up-to-date.

So, what does Google’s index mean?

Well, at this point, all you need to think of is all the businesses here in Bristol, and right around the globe, that have a website, that can be indexed are in a sort of library, there rated by Google’s algorithm.

So, what does “crawling and indexing” mean when in terms of organic SEO?

Well, let’s say you run a business, your passionate about coffee, so in Clifton, in Bristol, you run a coffee shop, and your mighty good at it, yet Bristolians, they cant get enough of your coffee, they keep banging the door “we-need-a-coffee-fix” they cry, so you’ve decided to branch-out, to hire a digital marketing agency, that’s here in the fine city of Bristol to build an e-commerce website, which sells coffee, dispatches it and gets out to customers in less than 48 hours, so the great public of Bristol, can make coffee while your coffee shop is not open.

But what does crawling and indexing mean?

Well, it simply means this, your coffee shop website, that now sells coffee online, is regularly visited by “Googlebot” its “ranking signals” are rated, no not by Google’s PageRank, nope, things are more sophisticated now, Google’s PageRank is widely thought by many SEO agencies to be replaced by over 200 ranking factors, such as backlinks, the site title, the anchor text, the internal links, the quality of the ever green content marketing.

Yet, websites are updated, there’s hundreds of SEO companies in Bristol working hard for their clients, they want their client at the top, they need to retain their clients, otherwise the company will pick another SEO company which will deliver, so things don’t stay the same for long.

This is why Googlebot may visit some Bristol businesses website a few times a day, crawling and indexing the site-map, following each backlink, to see have any SEO improvements been made?

Google’s mighty algorithm is then working out, where should the website be ranked? Is this coffee shop the business with the strongest SEO? If not, how strong is it? Is it worthy of being on page 1 for say local coffee shops, should it be on page 2.

Let’s say that you’ve hired a new seo consultant, yet, let’s say that they then go onto build the backlinks, in the wrong way, for example let’s say that they haven’t followed Google’s WebMaster Guidelines- they have built all of the backlinks incorrectly, they haven’t followed white hat methods at all!

So this means that the business, could incur what’s known as a penalty, an algorithmic or a manual penalty, that could mean that the business gets “de-indexed”- so you see, its possible to get a companies website, indexed, and this is important, so that the ranking factors can be read by Googlebot, but if the seo agencies, or let’s say the seo consultants get it wrong, or your business gets the organic seo wrong, then the business could incur a penalty, such as a Google Penguin penalty, this means then the website will be “de-indexed” sometimes, which means that the website may not longer appear in Google’s results, because the organic seo hasn’t been implemented correctly.

So, this is why when shopping around here in Bristol England, to hire a new say SEO business, make sure that you hire the right company, the right marketing agency, which follow’s Google’s WebMaster Guidelines.

Why is having an indexable website so important?

Well, quite simply put, GoogleBot, must be able to “crawl and index” your companies website, it must be “indexable”, as otherwise the companies website will not appear in Google’s SERP’s, which simply stands for “Google’s search engine results pages SERP’s, so basically speaking, if you have a website, that you want to improve its organic seo, well, the web designer, and the SEO agencies must make sure that the website can be indexed.

So, what is Googlebot?

So, at this point, you may well be sipping your cup of coffee, and thinking well, what is GoogleBot?

I’ve heard of Google, obviously, your thinking, but never of this Googlebot, so what is Googlebot, and what is its function?

Well, Googlebot is a very busy bot, its ultimately just code, the codes job is to regularly go over companies’ websites, to report back if SEO changes have been made, so for example, if the main site title, on your WordPress website changes from Coffee shops Clifton, to Coffee retailer in Bristol, well, you guessed it in one, Googlebot will notice the change, it will notice that the web designer or lets say the SEO agencies have changed the WordPress site title, it will then decide if the companies website should change in terms of where it ranks in the SERP’s, so for example, Google’s algorithm will think, well, this company might no longer be relevant for “Clifton Bristol Coffee Shops” its more relevant now to say retailing coffee more generally, perhaps through a WordPress e-commerce website, so, the business may well be trying to deliver coffee, rather than get customers to visit the coffee shop, so that’s right, the company will change where it is in the SERP’s.

So, that this process is a bit clearer, here’s an example of what happened to that coffee shop, once it changed it’s site title:

• Googlebot spotted the site title had changed
• It told Google’s algorithm that an important ranking factor, the site title had changed.
• Google’s algorithm then worked out that the business might not be that relevant for selling coffee in Clifton Bristol anymore, instead, its selling coffee now online, to a much wider customer base, that it wants to sell coffee all over the U.K.
• Google’s algorithm, and Google’s AI technology RankBrain, that’s Google RankBrain AI, will then workout where that business should be ranked, for example, should it now rank say 22nd for “coffee delivery companies” or “buy-coffee-online” and no longer rank for “Clifton local coffee shops” anymore? AI and Google’s mighty algorithm will work this out.

What are Google crawl errors?

So, let’s say that your web developer places code, such as a “no-index” code on a page, that can mean that it tell’s Googlebot now to crawl and index that page, so its kind of like locking a door, it cant be accessed, so, if you want that website, or that page to be indexed, to show in Google’s SERP’s, then the page needs to be indexed, it needs Googlebot to crawl and index it, without this happening, the website might be de-indexed, so it no longer appears in Google’s SERP’s anymore.

Your SEO agency, whether you hire a local SEO agency here in Bristol, in England, or anywhere else, the agency should make sure the companies website can be indexed, if you want it to appear on Google, if it cant be indexed, then the company website wont show in the SERP’s

So, I run a company here in Bristol, how often will my company website be crawled and indexed?

This does depend, for example, some company websites are crawled and indexed much more than others in Bristol, this is why when you meet your SEO consultant, and over a cup of tea, they start talking about “Google’s Crawl Budget”- and how often “GoogleBot” crawls your site, you may be thinking, well-what on earth are they talking about?

Well, here’s the thing, let’s say you own a rather fancy restaurant right here in Bristol, you’re the “bee’s knees” at what you do, yet, that website might be crawled and indexed say once a week, because lets say that the restaurant business, has let’s say 600 organic visitors every week, and you know this because your SEO agency has stated that in an SEO report, where the organic SEO information was taken from Google Analytics.

So, you get 600 visitors a week, to your restaurant business, and lets say the companies website gets crawled by GoogleBot once a week also, and you notice your website’s SERP’s flux say once a week also.

Now let’s compare this to another company in Bristol, lets say the other business is a major chain of dentists, and instead that company gets 10,000 organic visitors every week, with people travelling from all over the South West of England to use the dental practice, they come from the centre of Bath, they come from all over Bristol, it’s a very popular business, and as such the business uses a local seo company, and they invest a lot of time every month improving the organic seo, they are regularly adding top-quality content marketing, they are also working hard on building quality white hat “do-follow” backlinks, and GoogleBot crawls that company website every single day.

So, you see the difference in “Google’s Crawl Budget” for one business, the restaurant, because the SEO work carried out each month is less, it only get’s crawled once a week, with ranking flux happening, so the SERP’s change often once a week, then compare this to the major chain of dentists, where people from Bath, and all over Bristol, come to that dentists, and because the amount of SEO work is much higher, also the website has many more organic visitors, that websites crawl budget is much higher, in matter of fact, GoogleBot, crawls and indexes that site every day, so you got it, the crawl budget for Google, for that company is much higher.

Now why does this even matter? Well, it matters simply because any good SEO companies, will tell you, it matters because where businesses get “crawled and indexed” more, is sometimes because the business sector is much more competitive, there’s more seo work occurring to companies websites, so Googlebot wants to know about these changes, so that Google’s algorithm and Google’s AI, that’s artificial intelligence can work out, well, should that business rank higher in Bristol?

Where that restaurant was say be number 3 in the SERP’s for “local Italian restaurants”- because of the SEO work, it might now move up, yet the point we are trying to make is this, that business will fluctuate much more, if GoogleBot regularly indexes the site, as it may spot more changes occurring to the seo.

What does my SEO agency mean when they say we should create quality content marketing so that Google’s algorithm will reward us for such work?

Whether you run a coffee shop, or you are a major online retailer, that retails thousands of products, all of the content marketing, should be white hat, and top-quality.

So basically, Googlebot reports back to Google’s algorithm?

Yes, so let’s say that your web developer completely changes the wording on your homepage, they alter it, so that the text is different, Googlebot will notice this, it will tell Google’s Algorithm, and the business may alter where it ranks.

But what are Google ranking factors?

There are well over

What are entities?

What is Google’s Knowledge Vault?

Why it crucial to keep your website’s information up to date?

Is your companies offsite details correct?

Is your businesses NAP information correct?

Will Googlebot notice if I have conflicting NAP information?

So, how regularly should my company, here in Bristol, be adding new blog posts?

So, is this part of the reason why organic SEO is such a slow process? Because we have to wait for Googlebot to crawl and index the work?

Is there coding that Googlebot struggles to crawl?

What does my SEO company mean, when they say that the page, or the entire website is now indexed?

What does it mean that Googlebot and Google’s algorithm has issued a penalty, why is this bad news, and what does “de-indexed” mean?

Why does good web design matter?

So, will some businesses here in Bristol, England get indexed more than others?

What does Google crawl budget mean?

Why do all the various SEO agencies that I call, say that I should invest in organic seo monthly, surely it can all be done in one go?

How can your agency help?

 

 

 

 

 

when you visit your digital marketing agency you may hear them talk about your company website getting crawled and indexed, but what does this even mean?

Well crawling and indexing is a very important part of SCO, as this is what is used to update the search engine with what work had been carried out to your company website.

So let’s say for example that you sell coffee beans, and you decide to start selling flavoured coffee beans, that’s the say you want to start selling vanilla flavoured coffee beans.

Now this is a new product, that’s the say you have never sold this product on your company website before, so obviously you want this product to be available in the search engine so somebody can click on it and purchase the product, for that to happen that page needs to be crawled and indexed.

What does crawling and indexing mean?

Well quite simply put, most search engines use an automated bot.

This automated bot will crawl and index your website.

It will therefore discover when you have added new work, such as adding a new product such as vanilla flavoured coffee beans.

Therefore this is obviously a very important part of the SEO process, as you want the SEO work that you have applied to your company website to be noticed by the search engine, as otherwise if it is not crawled and indexed, the search engine would never know that you have applied the SEO work.

What does indexed mean?

When you publish a new piece of work on your company website, so that new piece of work could be a blog post obviously you wanted to appear in the search engine as soon as possible, for this to happen it needs to be indexed, for it to be indexed needs to be crawled and indexed first.

Index therefore means that the page can then possibly appear in the search engine

so for example let’s say you have just published your brand-new page which sells flavoured coffee beans, you want that page to be indexed as soon as possible so that it appears in the search engine.

What are bots

you may hear your SEO agency talk about box, there are many different types of bots the carry out different tasks, for example some bots are used by the search engine companies to spot changes made to your company website

other bots might carry out different tasks, for example bots are sometimes used to scrape email addresses offer website so that they can start to for a data base of email addresses for marketing purposes for example.

How do the search engines crawl and index a website?

It is widely thought by many SEO companies, that backlinks are used as the mechanism to crawl and index a website.

So for example let’s say that you sell mountain bikes, that’s the say you sell mountain bikes and you sell electric bikes and you sell bike accessories and you also sell bike helmets.

Now on the homepage on your bike website you are going to have links to all these various different pages, that the say a link to the bike accessories page, a link to the mountain bike page, a link to the electric bike page a link to the bike helmets page

Therefore the bots will use these links to crawl and index your website, for example if there is not a site map, then the bot might use your homepage to go to all the various pages on your company website.

So what pages are indexed?

All pages should be indexed, that’s the say the pages which your company wants to have appear in the search engine.

So for example if you sell mountain bikes, electric bikes, helmets, bike accessories and you want all these different pages to appear in the search engine then you should make sure that all of these pages can be indexed.

Therefore it is the SCO agency’s job to make sure that these pages can be indexed by the bots.

What type of content marketing should my company be adding to our website?

So now you know it’s important that the bots can crawl and index your company website, it’s important to start adding high quality, white hat content marketing.

The work should be high quality and white hat and it is really good quality then this can help to improve your businesses organic SEO.

So for example the bike company, might want to review various electric bikes, if this work is really good quality then this can help to improve the businesses organic SCO.

Why do the algorithms rely on bots to crawl and index a company website?

Without the bots crawling and indexing a company website the algorithm will never be able to make a calculation as to how strong your businesses SEO is.

So, for example, let’s say that your business invests heavily in “organic SEO”, this work would never get noticed if it is never crawled and indexed, therefore if it is never crawled and indexed the algorithm will have no information on how strong your businesses SEO is.

This is why it’s really important that your company website gets regularly crawled and indexed so if you have made any SEO improvements this can get noticed by the algorithm.

Then the algorithm deems the work to be high quality and white hat then this can help to improve your businesses organic SEO.

However we must stress that you should only invest in white hat SEO.

What does crawl budget mean exactly?

It is widely thought by many online marketing companies that the search engines apply a crawl budget to company websites.