search - How to disallow specific pages in robots.txt, but allow everything else? -

search - How to disallow specific pages in robots.txt, but allow everything else? -

is way it?

user-agent: * allow: / disallow: /a/*

i have pages like:

mydomaink.com/a/123/group/4 mydomaink.com/a/xyz/network/google/group/1

i don't want allow them appear on google.

your robots.txt looks correct. can test in in google's webmaster tools account if want 100% sure.

fyi, blocking pages in robots.txt doe snot guarantee not show in search results. prevents search engines crawling pages. can still list them if want to. prevent page beingness indexed , listed need utilize x-robots-tag http header.

if utilize apache can place file in /a/ directory next line block pages:

<ifmodule mod_headers.c> header set x-robots-tag: "noindex" </ifmodule>

search seo robots.txt