Include expected and beneficial files
Websites should include a range of expected and standard beneficial files to improve search engine optimization, user experience, transparency, and overall site health. Search engines and browsers regularly request these files by default. If they don't exist, this leads to unnecessary requests, potential errors, and increased emissions. Including these files avoids these issues while also providing SEO, user experience, and other benefits. They each have a low carbon footprint, so while they do create emissions, it's worth including them for the benefits they provide.
Criteria: Expected file formats
Include the favicon.ico, robots.txt, opensearch.xml, site.webmanifest, and sitemap.xml documents. Additionally, ensure that any such files defined in future web standards or specifications are included.
Resources
- About /robots.txt
- AssetConfig
- Build and submit a sitemap
- Define a favicon to show in search results
- Favicon Generator
- GPF - General Policy Framework (PDF) - 1.9 - Strategy (Interoperable Technologies)
- How Google interprets the robots.txt specification
- OpenSearch
- OpenSearch Protocol
- Sitemaps Protocol
- The Carbon Impact of Web Standards (PDF)
- Web Application Manifest
Criteria: Beneficial file formats
Include beneficial files such as ads.txt, carbon.txt, humans.txt, security.txt. Additionally, ensure that any such files defined in future web standards or specifications are included.
Resources
- Ads.txt guide
- All about humans. Humans.txt, actually
- AssetConfig
- Google Says LLMs.Txt Comparable To Keywords Meta Tag
- GPF - General Policy Framework (PDF) - 1.9 - Strategy (Interoperable Technologies)
- Meet LLMs.txt, a proposed standard for AI website content crawling
- security.txt
- slash pages
- The ads.txt Standard
- The Carbon Impact of Web Standards (PDF)
- The carbon.txt Proposed Standard
- The humans.txt Standard
- The llms.txt Proposed Standard
- The security.txt Proposed Standard
- What Is llms.txt, and Should You Care About It?
Impact: Low, Effort: Low
GRI | Impact |
---|---|
materials | Low |
energy | Low |
water | Low |
emissions | Low |
Benefits of this guideline
- Environment: Search engines or browsers request certain files by default, ensuring they are in place will reduce loading errors, and may provide efficiency enhancements in how visitors find or interact with a site. Plaintext requires no rendering. If visitors (or search engines) know about these useful files (like carbon.txt) they can load quicker and with less CPU / GPU impact than any HTML website.
- Accessibility: OpenSearch.xml enables the browser's default search box rather than a custom solution to be integrated with your website search, which may aid accessibility as it encourages the use of a browser native component (and / or keyboard shortcuts) rather than a website or application which may suit certain accessibility requirements better.
- Performance: Files that are expected will produce HTTP requests, ensuring they are met will satisfy the products making them and potentially reduce the requests once they are discovered. Plaintext files contain no links, no markup, and have no imprint. Putting credits (for example) in such a file will reduce data transfer and have a lower rendering footprint.
- Economic: Robots.txt and Sitemap files can be utilized by search engines to help make your website more findable, this could lead to more visitors and potentially more customers as a result. The ads.txt file is part of a scheme to reduce advertising fraud, it could be useful.
- Conversion: Robots.txt files can be used to target specific search engines, helping to ensure content is correctly indexed to get a good placement so that visitors can find you easily.
- Transparency: The humans.txt file provides credit to people involved in a site's creation, and security offers critical points of contact if an issue is discovered. Both are valuable additions to a project.
Example
-
code
User-agent: * Disallow: /cgi-bin/ -
content
About /robots.txt.
Tags:
- Assets
- Compatibility
- Marketing
- Patterns
- Security
- UI