If we want to appear in the search results, the first thing we must do is to get Google<\/strong> or any other search engine to interpret the website correctly. If they can’t read the content, or don’t know what they can read…. How will they know which pages to show and which not to show?<\/p>\n<\/div> How Google works: indexing and crawling<\/span><\/i><\/a><\/p>\n<\/blockquote>\n<\/div> Related to web indexation, there are some elements that we must take into consideration: the robots.txt file<\/strong>, through which we will indicate to the robots the pages and files that they can access from our website<\/strong>. A key aspect to design any SEO strategy correctly.<\/p>\n<\/div> The robots.txt files are a text file through which we provide information to the robots (bots, crawlers…) about the pages or files that can request information or not on a website<\/strong>. Through this file we can “communicate” directly with the crawlers.<\/p>\n Thus, mainly the robots.txt file<\/strong> is used to avoid overloading the server with requests and to manage the traffic of robots on the websit<\/strong>e, because in this file we indicate the content that should crawl and not on a website. Here we have to point out that blocking or not blocking pages, has a different use than the “no-index” tag<\/strong> (explained below).<\/p>\n The robots.txt file is located at the root of the website (www.myweb.com\/robots.txt).<\/p>\n<\/div> The robots.txt file is a text file that is implemented in the root of the main domain,<\/strong> for example: www.myweb.com\/robots.txt. This is where we will include different elements to tell the crawlers which pages should be crawled<\/strong>, and which should not.\u00a0<\/strong><\/p>\n Robots.txt files can be created in any web editor, just be aware that it can create standard UTF-8 text files.<\/p>\n<\/div> In general, by default the robots.txt file will be implemented in WordPress with the use of SEO plugins (Yoast SEO or Rank Math), like this one:<\/p>\n<\/div> We also have the option to edit this file in several ways. On the one hand, directly from the FTP of our hosting or through different plugins that you can implement in WordPress as Yoast SEO or Rank Math, from where you can edit the file. It should be borne in mind, however, that editing this file incorrectly, can significantly affect the positioning results of the web<\/strong>, so it will be important to inform us well of what each parameter means and how it can affect our website.<\/p>\n In the case of using Rank Math in WordPress, for example, to edit the file we should go to Rank Math > General Settings > Edit Robots.txt files.<\/p>\n<\/div> For a correct implementation, it is important to consider the following aspects, as highlighted by <\/span>Google<\/span><\/a><\/p>\n Next, we are going to define some of the main elements that are important to know in order to interpret and implement robots.txt files:<\/p>\n There are several things to consider when setting the allow and disallow directives in the file. An incorrect implementation can affect the results of the page in the SERPs. So we must take into account the following parameters.<\/p>\n If we leave the robots.txt file as follows, it will not block any directory:<\/p>\n<\/div> But if, for some reason (not recommended unless it is for a consistent reason), we put only the slash (\/), we would block the crawling of the entire web page <\/strong>(and we would not appear in search engines):<\/p>\n<\/div> On the other hand, if between \/…\/ we define a directory<\/strong>, we will block only this one from the crawle<\/strong>r. How it would be \/wp-admin\/. It is important to emphasize, that if we do not include the final \/, the robots will not crawl any page that will begin with \/wp-admin.<\/p>\n<\/div> Likewise, if within this exclusion, we want to indicate any subdirectory that could be traced<\/strong>, we will include it as allow:<\/p>\n<\/div> We have seen that by means of the user-agent, as well as directories or URLs with the parameter allow or disallow, we can indicate to the robots different parameters that they can crawl or not of the web.<\/p>\n Next, we are going to detail some that we can find and that can be useful to us depending on the objectives of our project. But it should be noted that each website is different, so you should analyze well if you are interested in any of them and why.<\/p>\n<\/div> Using user-agent: *<\/strong> is one of the easiest ways to indicate to all robots<\/strong> (Google, Bing, Baidu…) that they can crawl the entire website.<\/strong> Since it is a wildcard we use * to indicate that “any” robot can crawl the site.<\/p>\n So, if you use it in user-agent, you indicate to the robots that they can crawl the entire website, and you will only have to detail with the disallow element, those directories that you do not want them to access<\/p>\n<\/div> On the other hand, you can also use * in URLs, either at the beginning or in the middle, and it has the same meaning: all\/any. In this case, for example, we block any URL such as www.myweb.com\/retail\/red\/jumper or www.myweb.com\/retail\/small\/jumper.<\/p>\n<\/div> To indicate to robots the end of a URL, you can use the $ symbol<\/strong>. For example, it can be used with disallow: *.php$, which would block the URL ending like this, but it would accept another one like: .php\/whateverend. That would be for if we want it to crawl some particular URLs of these files.<\/p>\n<\/div> Depending on our strategy (check this before), we may not want specific robots to crawl the site<\/strong>. This aspect would be indicated as follows:<\/p>\n<\/div> If we want to comment on any aspect, but without addressing the robots, we will do it with the #. Since the robots will not read everything after #.<\/p>\n<\/div> Other elements that we can use in the robots.txt file are those that will help us to block the entry of robots to URLs,<\/strong> such as URLs with parameters, which are created when users use the search engine of a website or filter products according to specific parameters, or when we do not want a page to be indexed in search engines because it is not relevant (privacy policy …).<\/p>\n<\/div> Before including this directive as a disallow in a web page, it is very important to correctly assess whether including it can be very useful or not. A decision that, as we have already mentioned, will vary depending on our website and objectives.<\/p>\n<\/div> With the “robots” meta tag, we can specify at the page level how content should be indexed in searches, in case we do not want a page to be displayed in the results.<\/p>\n But in order to indicate this directive to a robot, it is important that they can read it. So blocking a URL that includes the tag “no index” in robots.txt, will be a mistake. Since the robots will not be able to read that page and, although it may be difficult for them to find it, it may be indexed after a while, since the robots end up reaching it through links from other websites.<\/p>\n<\/div> Leaving aside that it will vary depending on the objectives of each website, what we have to ask ourselves is:<\/p>\n Depending on the answer<\/strong>, we will design our strategy<\/strong>.<\/p>\n<\/div> Finally, another option that we can point out is that, although we decide that we are interested in having the searches on the website performed by users blocked, we may want to make an exception for some specific ones,<\/strong> as they may be terms of interest<\/strong>, which can help us to increase visibility<\/strong>. In this case, for example, we would put:<\/p>\n<\/div> The canonical tag is used to avoid duplicate content<\/a> on a web page<\/strong>. These tags are sometimes used in parameterized URLs – which have very similar content to the main product or category – to prevent creating duplicate content.<\/p>\n<\/div>\n
What are robots.txt files?<\/h2>\n
How to implement robots.txt files?<\/h2>\n
How to implement or modify the robots.txt file in WordPress?<\/h2>\n
![\"\"](\"https:\/\/www.gemmafontane.com\/wp-content\/uploads\/2020\/08\/robots-txt-predeterminado-google.png\" "\\"robots")
<\/span><\/div><\/div>![\"\"](\"https:\/\/www.gemmafontane.com\/wp-content\/uploads\/2020\/08\/editar-archivos-robots.txt-rank-math.png\" "\\"editar"){kind=icon}
<\/span><\/div><\/div>Aspects to take into account for a correct implementation<\/h3>\n
\n
\n
Basic guide to robots.txt files. What parameters should we know?<\/h2>\n
\n
\n
Allow and disallow, when we give or deny access to robots in a personalized way<\/h3>\n
![\"\"](\"https:\/\/www.gemmafontane.com\/wp-content\/uploads\/2020\/08\/disallow-robotstxt-1.png\" "\\"disallow")
<\/span><\/div>![\"\"](\"https:\/\/www.gemmafontane.com\/wp-content\/uploads\/2020\/08\/disallow-robotstxt-1.2-toda-la-web.png\" "\\"disallow")
<\/span><\/div>![\"\"](\"https:\/\/www.gemmafontane.com\/wp-content\/uploads\/2020\/08\/disallow-robotstxt-2.png\" "\\"disallow")
<\/span><\/div>![\"\"](\"https:\/\/www.gemmafontane.com\/wp-content\/uploads\/2020\/08\/disallow-robotstxt-3.png\" "\\"disallow")
<\/span><\/div>Other elements to take into account for the robots.txt file<\/h2>\n
The *, to indicate “any” in the robots crawl.<\/h3>\n
![\"\"](\"https:\/\/www.gemmafontane.com\/wp-content\/uploads\/2020\/08\/user-agent-acceso-robots.png\" "\\"user")
<\/span><\/div>![\"\"](\"https:\/\/www.gemmafontane.com\/wp-content\/uploads\/2020\/08\/bloquear-red-jumper-robots.png\" "\\"bloquear")
<\/span><\/div>Indicates the end of a URL with the $ symbol<\/h3>\n
![\"\"](\"https:\/\/www.gemmafontane.com\/wp-content\/uploads\/2020\/08\/bloquear-archivos-disallow-concretos-final.png\" "\\"bloquear")
<\/span><\/div>Preventing crawlers from visiting the website<\/h3>\n
![\"\"](\"https:\/\/www.gemmafontane.com\/wp-content\/uploads\/2020\/08\/impedir-un-robot-concreto-visitar-sitio-web.png\" "\\"impedir")
<\/span><\/div>The #, to explain comments<\/h3>\n
![\"\"](\"https:\/\/www.gemmafontane.com\/wp-content\/uploads\/2020\/08\/almoadilla-comentarios-robots.jpg\" "\\"almoadilla")
<\/span><\/div>Disallow or no-index tag?<\/h2>\n
Indexing content with the “robots” meta tag<\/h3>\n
What will be our best option when we have URLs with parameters, “no-index” or disallow?<\/h3>\n
\n
![\"\"](\"https:\/\/www.gemmafontane.com\/wp-content\/uploads\/2020\/08\/robots-url-parametros-1.png\" "\\"robots")
<\/span><\/div>![\"\"](\"https:\/\/www.gemmafontane.com\/wp-content\/uploads\/2020\/08\/robots-allow-palabra-clave.png\" "\\"robots")
<\/span><\/div>Blocking URLs with canonicals tags<\/h3>\n