Hi Robert.
A couple of thoughts; certainly not a complete answer:
Do you mean folders outside www? There are no non-public folders below www.
I don't know how spiders handle such things, but I have had some success by putting a PHP check at the top of any page I don't want seen by unauthorized viewers:
Then of course the login page does the appropriate authentication and redirects back to the requested page. Once the user is authenticated, the pages just appear normally until the browser is closed.
As I say, not a complete answer by any means, but it may shed some light, or at least occasion further discussion.