-
Notifications
You must be signed in to change notification settings - Fork 3
Open
Labels
questionFurther information is requestedFurther information is requested
Description
https://developers.google.com/search/reference/robots_txt#order-of-precedence-for-user-agents
Only one group is valid for a particular crawler. The crawler must determine the correct group of lines by finding the group with the most specific user-agent that still matches. All other groups are ignored by the crawler. The user-agent is case-sensitive. All non-matching text is ignored (for example, both googlebot/1.2 and googlebot* are equivalent to googlebot). The order of the groups within the robots.txt file is irrelevant.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
questionFurther information is requestedFurther information is requested