Several ways exist to deny bots access to files and folders:
The best, yet quite tedious, way is to use ><meta name="robots" content="noindex,nofollow">
in the header of every page you don't want spidered. I use this method and it doesn't take long to implement since I use php templates to do most of the dirty work for me.
You can, of course, use the robots.txt file to control robot access to your sites contents. However, if you are at all security conscious I would recommend against this method. Any divulgence of your site's directory structure, file names, naming conventions, or types of extensions used will make it that much easier for knowledgeable folks to gain access to it and use it imporperly.
The only 100% (well, almost) guaranteed method is to password protect a directory or file. This will almost surely hinder usability but works well for folders containing only support files (e.g. cgi-bin, javascript, includes, stylesheets, images, etc...).
You can use >deny from <ip>
in .htaccess to deny certain bots, but you would have to know the ip/domain of every bot to be denied. This isn't feasable for large-scale denial, but is good for denying those "rogue" bots that ignore robots.txt and robots meta tags. You can do a google search for bad or rogue robots to find the names and ips of many of the worst offenders.
After the fact if you find any of your pages indexed on a search engine you can usually submit a request for removal of any and all pages from your domain.