Skip to content
This repository has been archived by the owner on Feb 9, 2024. It is now read-only.

Create robots.txt #14

Open
fisher60 opened this issue Jun 8, 2023 · 0 comments
Open

Create robots.txt #14

fisher60 opened this issue Jun 8, 2023 · 0 comments
Labels
enhancement New feature or request

Comments

@fisher60
Copy link
Member

fisher60 commented Jun 8, 2023

Summary

It is general good practice have a robots.txt file. We must ensure that we allow search engine crawlers to index our sites. We can simply allow all scraping from all clients on abandontech site, but it is good to explicitly state this.
@fisher60 fisher60 added the enhancement New feature or request label Jun 8, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@fisher60