I would bet they can, if you can decipher the cryptic directory structure c5 uses to store files. You can lock down that directory using "file permissions", but that can get complicated.

As far as web spiders go, see here:
http://www.robotstxt.org/robotstxt.html...

You can use the robot.txt file in your webroot or using META tags in your header.

Hope it helps.

Here is some proof. If fancy indexing is enabled in apache, here is what you get, assuming you know where to look

So how is everyone keeping their files from being sucked off the site. If you have a members area that is protected via c5 premissions and contains private documents in the asset area, anyone can could find the files?

I am running just a "stock out of the box c5". I don't know if an asset is protected using user/group permissions at the file level. They may very well be (one of the experts can answer to that), but just playing with the stock setup, you can get to them. It aint easy though :)

You can protect files themselves with a password, but search engine spiders cannot find the files anyways unless they are linked somewhere on your site, and if they are only linked on protected pages, then those pages aren't indexed and neither are the files, so you are in good shape.

Wow, that was long sentence.

You can go even a step futher by using mod_rewite rules to prevent off-site linking. mod_rewrite is very powerful, you just have to RTFM *icky*

Go into Dashboard > Files > Access and change the location of your files. If you specify a location that is outside your web root, the files will be available when requested through concrete5 but you'll never be able to browse to them directly.

Note - this will increase overhead, as all files have to pass through concrete5 to be viewed/access/downloaded but it will make it so that there's no possible way files that aren't available to users permission-wise will be able to accessed through other means.

@andrew: Good tip. If your files are outside of your webspace, apache can't see them :)

There is a mod_rewrite method to prevent "hotlinking". This will prevent any hotlinking to gif & jpg files :) Something like this in your .htaccess:
RewriteCond %{HTTP_REFERER} .
RewriteCond %{HTTP_REFERER} !^http://(www\\.)?myc5site\\.com [NC]
RewriteRule \\.(gif|jpe?g)$ - [NC,F]

Just to make it perfectly clear: there is NO safe way to do this except for the method Andrew describes.

Relying on search engines not being able to guess file names is dangerous, I've experienced engines indexing the weirdest names out of the blue. Also, if you have a link that is accessible without authentication, anyone with access to the file could just spread the link around - or they could be eavesdropped on, by checking out the browser history for example.