User-agent: * # --- RSS feeds: blocked to stop wasted Google crawl budget --- # Per the Coverage Drilldown report, ~960 of 1000 URLs flagged as # "Crawled - currently not indexed" were exactly these /feed/ URLs. Disallow: /feed/ Disallow: /*/feed/ Disallow: /*/feed/rss/ Disallow: /*/feed/atom/ Disallow: /*/feed/rss2/ Disallow: /comments/feed/ Disallow: /*/comments/feed/ Disallow: /product/*/feed/ Disallow: /product-tag/*/feed/ Disallow: /product-category/*/feed/ Disallow: /tag/*/feed/ Disallow: /category/*/feed/ Disallow: /author/*/feed/ Disallow: /*?feed= # --- login/register URL parameters: already properly canonicalized by --- # --- Google, blocked here only to save extra crawl budget --- Disallow: /*?login=true Disallow: /*?type=register Disallow: /*&type=register Disallow: /*&login=true # --- WordPress attachment (media) pages: thin/low-value auto-generated --- # --- pages, blocked to save crawl budget --- Disallow: /*?attachment_id= Sitemap: https://ajanskonkur.ir/sitemap_index.xml