.Google's John Mueller responded to a question concerning why Google.com indexes pages that are actually prohibited from crawling by robots.txt and also why the it's secure to ignore the related Search Console reports concerning those crawls.Crawler Web Traffic To Inquiry Parameter URLs.The individual inquiring the question chronicled that robots were generating hyperlinks to non-existent query guideline URLs (? q= xyz) to pages along with noindex meta tags that are likewise shut out in robots.txt. What cued the question is actually that Google is creeping the web links to those web pages, receiving shut out by robots.txt (without seeing a noindex robots meta tag) at that point getting reported in Google Browse Console as "Indexed, though shut out through robots.txt.".The individual talked to the observing concern:." But right here's the huge question: why would certainly Google.com mark web pages when they can't also view the content? What is actually the perk because?".Google.com's John Mueller verified that if they can't creep the page they can not view the noindex meta tag. He also creates an interesting reference of the website: search driver, suggesting to dismiss the results given that the "common" customers won't find those results.He wrote:." Yes, you are actually appropriate: if our team can not crawl the web page, our experts can't view the noindex. That pointed out, if our experts can not creep the webpages, then there's not a whole lot for our company to mark. Thus while you might find several of those pages along with a targeted site:- query, the normal consumer will not view them, so I wouldn't bother it. Noindex is actually also great (without robots.txt disallow), it simply indicates the URLs will definitely find yourself being crept (as well as wind up in the Explore Console record for crawled/not indexed-- neither of these standings cause concerns to the remainder of the web site). The important part is that you don't make them crawlable + indexable.".Takeaways:.1. Mueller's answer confirms the constraints in using the Internet site: search progressed hunt driver for diagnostic reasons. Some of those reasons is actually since it is actually not linked to the frequent search mark, it is actually a different thing entirely.Google's John Mueller commented on the web site hunt operator in 2021:." The short response is actually that a web site: query is actually certainly not meant to be complete, neither made use of for diagnostics reasons.An internet site concern is actually a certain sort of search that confines the end results to a specific website. It is actually basically just words site, a digestive tract, and afterwards the web site's domain.This question limits the end results to a particular web site. It's not implied to become a thorough selection of all the web pages from that internet site.".2. Noindex tag without making use of a robots.txt is fine for these type of circumstances where a bot is connecting to non-existent web pages that are actually receiving found out by Googlebot.3. Links along with the noindex tag will certainly produce a "crawled/not indexed" entry in Browse Console and also those won't possess an unfavorable result on the rest of the website.Go through the concern and also answer on LinkedIn:.Why would certainly Google mark pages when they can not also observe the content?Featured Photo by Shutterstock/Krakenimages. com.