Skip to content
This repository has been archived by the owner on Jan 31, 2023. It is now read-only.

block all crawlers #11

Merged
merged 2 commits into from
Nov 1, 2022
Merged

block all crawlers #11

merged 2 commits into from
Nov 1, 2022

Conversation

tonypls
Copy link

@tonypls tonypls commented Nov 1, 2022

Adbot needs to be specifically named, all other crawlers are covered by *

@madsnedergaard
Copy link
Member

Maybe add a noindex meta tag too to be sure?
https://developers.google.com/search/docs/crawling-indexing/block-indexing#meta-tag

<meta name="theme-color" content="#42b883" />
<title>Vitamin</title>
<meta name="robots" content="noindex" />
<title>Avocado Maps</title>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

😂

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to really throw those crawlers off 😆

@tonypls tonypls merged commit 7943081 into master Nov 1, 2022
@madsnedergaard madsnedergaard deleted the block-crawlers branch November 1, 2022 22:07
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
2 participants