seo - Are local robots.txt files read by Facebook and Google? -

seo - Are local robots.txt files read by Facebook and Google? -

i have folder half public: url not linked, people know url few friends (which not link it) , cryptic plenty create sure nobody lands there accident.

however, link send via googlemail , facebook messages. there way tell facebook , google in local robots.txt file not index page?

when add together "global" robots.txt file takes there see in /secret-folder-12argoe22v4 might interesting. not that. facebook / google @ /secret-folder-12argoe22v4/robots.txt?

the content be

user-agent: * disallow: .

or

user-agent: * disallow: /secret-folder-12argoe22v4/

as cbroe mentioned, robots.txt file must @ top level of site. if set in subdirecory, ignored. 1 way can block directory without publicly revealing total name block part of it, this:

user-agent: * disallow: /secret

this block url starts "/secret", including "/secret-folder-12argoe22v4/".

i should point out above not 100% reliable way maintain files out of search engines. maintain search engines straight crawling directory, can still show in search results if other site links it. may consider using robots meta tags instead, won't prevent straight next off-site link. reliable way maintain directory private set behind password.

facebook seo robots.txt