| Internet-Draft | Robots Exclusion Protocol Extension to m | October 2024 |
| Canel & Madhavan | Expires 24 April 2025 | [Page] |
This document extends RFC9309 by specifying additional rules for controlling usage of the content in the field of Artificial Intelligence (AI).¶
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶
This Internet-Draft will expire on 24 April 2025.¶
Copyright (c) 2024 IETF Trust and the persons identified as the document authors. All rights reserved.¶
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶
While the Robots Exclusion Protocol enables service owners to control how, if at all, automated clients known as crawlers may access the URIs on their services as defined by [RFC8288], the protocol doesn't provide controls on how the data returned by their service may be used in training generative AI foundation models.¶
Application developers are requested to honor these tags. The tags are not a form of access authorization however.¶
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.¶
The possible values of the rules complementing existing allow, disallow rules are:¶
DisallowAITraining - instructs the parser to not use the data for AI training language model.¶
AllowAITraining - instructs the parser that the data can be used for AI training language model.¶
The values are case insensitive and honor the same matching logic as Allow and disallow rules. When Allow and Disallow rules define if the content can be downloaded, AllowAITraining and DisallowAITraining rules only apply rules on usage of the content for AI training.¶
The same rules can also be set in the Application Layer Response Header:¶
DisallowAITraining - instructs the parser to not use the data for AI training language model.¶
AllowAITraining - instructs the parser that the data can be used for AI training language model.¶
The values are case insensitive and honor the same matching logic as Allow and disallow rules.¶
Same rules can also be set via an HTML meta tag:¶
TODO: https://www.rfc-editor.org/rfc/rfc9110.html#name-field-name-registry¶