[lazarus] Component page revisited.

[Michael A. Hess]

Ronan van Riet wrote:
> 
> There are ways of disallowing crawlers on your site by simple adding a
> robots.txt.
> Has anyone had any experience with this?

If it is a reputable crawler this works. It is based on the honor
system. They are suppose to first read robots.txt and not process files
or directories listed within. It doesn't stop them in anyway from just
going ahead and doing it.

Most are but if it is a crawler after email addresses for SPAM I'm not
sure if they would honor this file since the activity they are
performing isn't very honorable in the first place.

-- 
==== Programming my first best destiny! ====

Michael A. Hess      Miracle Concepts, Inc.
mhess at miraclec.com   http://www.miraclec.com