Managing Search Engines and AI Bots with robots.txt and llms.txt

Search engine crawlers and AI bots hit every site, so we created clear guidelines on what they can access.

robots.txt Highlights

• Allowed bots to crawl all public pages and static assets like /images/ and /portfolio/.
• Blocked sensitive areas such as /api/, /_next/, and /admin/.
• Added a sitemap reference and a modest Crawl-delay to reduce server strain.
• Included specific rules for bots like GPTBot and ChatGPT-User to limit API and admin access.

llms.txt Guidelines

• Outlined which site content AI models may quote, analyze, and index.
• Prohibited training on client data, custom implementations, and pricing strategies.
• Listed acceptable uses, such as answering questions about public services.
• Provided contact emails for AI/LLM inquiries and permissions.

By publishing these files, we encourage good-faith indexing while protecting proprietary information and client privacy.