From time to time people ask me about hiding information from search engines. For one reason or another, these individuals have published information on the web, but don't seem to want anyone to find that information. There are certain rare situations where this makes sense, but in most cases, these individuals have missed the key component to publishing information on the web, that being the word publishing. Publishing is by definition "to make known to another or to the public generally". So before you stop robots from indexing portions of your website by using your robots.txt file, there are two questions you should be asking yourself.
The first question you should ask yourself is "Why am I putting this information on the web?" Many people who want to hide documents from search engines are trying to create a semi-private workspace where they can make documents available to their friends, family or colleagues, but no one else. If this is what you want to accomplish, you have to password protect your website using htaccess or find some other solution that forces users to authenticate themselves. Keeping documents hidden from search engines is not sufficient protection for even semi-private data.
So if the answer to "Why am I putting this information on the web?" is anything except to allow the public to have access to it, the simplest solution is not put this information on the web in the first place.