.Google's John Mueller answered a concern concerning why Google.com marks pages that are actually prohibited from crawling by robots.txt and why the it's secure to ignore the similar Look Console documents regarding those creeps.Robot Visitor Traffic To Question Criterion URLs.The person talking to the inquiry recorded that robots were actually creating web links to non-existent inquiry criterion URLs (? q= xyz) to webpages along with noindex meta tags that are also blocked in robots.txt. What cued the concern is actually that Google is actually crawling the links to those web pages, acquiring shut out by robots.txt (without watching a noindex robots meta tag) at that point obtaining reported in Google.com Search Console as "Indexed, though obstructed through robots.txt.".The person talked to the complying with question:." Yet below is actually the huge inquiry: why would Google.com index pages when they can't also observe the web content? What is actually the benefit during that?".Google's John Mueller confirmed that if they can not creep the page they can't find the noindex meta tag. He likewise helps make an appealing reference of the internet site: hunt operator, suggesting to overlook the end results due to the fact that the "average" users won't observe those outcomes.He wrote:." Yes, you are actually proper: if our company can not creep the web page, we can not view the noindex. That stated, if our company can not creep the web pages, after that there's not a whole lot for us to mark. So while you could view several of those pages with a targeted website:- concern, the ordinary user won't view them, so I wouldn't bother it. Noindex is also alright (without robots.txt disallow), it just means the URLs are going to find yourself being crawled (as well as find yourself in the Browse Console document for crawled/not catalogued-- neither of these conditions trigger problems to the remainder of the web site). The integral part is that you don't make them crawlable + indexable.".Takeaways:.1. Mueller's solution affirms the limits in using the Web site: search progressed search driver for analysis causes. One of those factors is actually given that it's certainly not connected to the regular hunt index, it is actually a distinct factor altogether.Google.com's John Mueller talked about the website hunt driver in 2021:." The quick solution is that an internet site: query is certainly not implied to become full, neither made use of for diagnostics functions.An internet site inquiry is actually a certain type of search that restricts the end results to a particular web site. It's essentially merely words web site, a bowel, and afterwards the website's domain name.This query restricts the end results to a certain web site. It's not implied to be a comprehensive selection of all the pages from that site.".2. Noindex tag without using a robots.txt is actually alright for these sort of scenarios where a bot is actually linking to non-existent web pages that are actually receiving uncovered through Googlebot.3. URLs with the noindex tag will definitely produce a "crawled/not listed" entry in Search Console which those will not have an adverse effect on the rest of the internet site.Read through the inquiry as well as address on LinkedIn:.Why would Google.com mark pages when they can not also view the content?Included Picture through Shutterstock/Krakenimages. com.