Parse robots.txt and sitemaps using dotnet. Supports the proposed RFC9309 standard, as well as the following common, non-standard directives:
- Sitemap
- Host
- Crawl-delay
This library is based upon HttpClient, making it very familiar, easy to use and adaptable to your needs. Since you have full control over the HttpClient, you are able to configure custom message handlers to intercept outgoing requests and responses. For example, you may want to add custom headers on a request, configure additional logging or set up a retry policy.
Some websites can have very large sitemaps. For this reason, async streaming is supported as the preferred way of parsing sitemaps.
There is also the possibility to extend this library to support protocols other than HTTP, such as FTP.
| Name | Supported | Priority |
|---|---|---|
| HTTP/HTTPS | ✔️ | |
| FTPS/FTPS | ❌ | 0.1 |
Wildcard (*) User-agent |
✔️ | |
| Allow & disallow rules | ✔️ | |
End-of-match ($) and wildcard (*) paths |
✔️ | |
| Sitemap entries | ✔️ | |
| Host directive | ✔️ | |
| Crawl-delay directive | ✔️ | |
| RSS 2.0 feeds | ❌ | 0.8 |
| Atom 0.3/1.0 feeds | ❌ | 0.8 |
| Sitemaps XML format | ✔️ | |
| Simple text sitemaps | ✔️ | |
| Async streaming of sitemaps | ✔️ | |
| Cancellation token support | ✔️ | |
| Memory management | ✔️ |
Install the package via NuGet.
dotnet add package Robots.Txt.ParserFirst, create an implementation of IWebsiteMetadata for the host address that you wish to use.
public class GitHubWebsite : IWebsiteMetadata
{
public static Uri BaseAddress => new("https://www.github.com");
}Next, create an instance of RobotWebClient<TWebsite>.
public void ConfigureServices(IServiceCollection services)
{
services.AddHttpClient<IRobotWebClient<GitHubWebsite>, RobotWebClient<GitHubWebsite>>();
}using var httpClient = new HttpClient();
var robotWebClient = new RobotWebClient<GitHubWebsite>(httpClient);Optionally, specify message handlers to modify the HTTP pipeline. For example, you may be attempting to crawl the website and therefore will want to reduce the rate of your requests, to do so responsibly. You can achieve this by adding a custom HttpMessageHandler to the pipeline.
public class ResponsibleCrawlerHttpClientHandler : DelegatingHandler
{
protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
{
var response = await base.SendAsync(request, cancellationToken);
await Task.Delay(TimeSpan.FromSeconds(1), cancellationToken);
return response;
}
}public void ConfigureServices(IServiceCollection services)
{
services.TryAddTransient<ResponsibleCrawlerHttpClientHandler>();
services.AddHttpClient<IRobotWebClient<GitHubWebsite>, RobotWebClient<GitHubWebsite>>()
.AddPrimaryHttpMessageHandler<ResponsibleCrawlerHttpClientHandler>();
}var httpClientHandler = new HttpClientHandler
{
InnerHandler = new ResponsibleCrawlerHttpClientHandler()
};
using var httpClient = new HttpClient(httpClientHandler);
var robotWebClient = new RobotWebClient<GitHubWebsite>(httpClient);var robotsTxt = await robotWebClient.LoadRobotsTxtAsync();
// providing a datetime only retrieves sitemap items modified since this datetime
var modifiedSince = new DateTime(2023, 01, 01);
// sitemaps are iterated asynchronously
// even if robots.txt does not contain sitemap directive, looks for a sitemap at {TWebsite.BaseAddress}/sitemap.xml
await foreach(var item in robotsTxt.LoadSitemapAsync(modifiedSince))
{
}var robotsTxt = await robotWebClient.LoadRobotsTxtAsync();
// if rules for the specific robot are not present, it falls back to the wildcard *
var anyRulesDefined = robotsTxt.TryGetRules(ProductToken.Parse("SomeBot"), out var rules);
// even if no wildcard rules exist, an empty rule-checker is returned
var isAllowed = rules.IsAllowed("/some/path");var robotsTxt = await robotWebClient.LoadRobotsTxtAsync();
// host value will fall back to TWebsite.BaseAddress host, if no directive exists
var hasHostDirective = robotsTxt.TryGetHost(out var host);var robotsTxt = await robotWebClient.LoadRobotsTxtAsync();
// if rules for the specific robot are not present, it falls back to the wildcard *
// if no Crawl-delay directive exists, crawl delay will be 0
var hasCrawlDelayDirective = robotsTxt.TryGetCrawlDelay(ProductToken.Parse("SomeBot"), out var crawlDelay);Issues and pull requests are encouraged. For large or breaking changes, it is suggested to open an issue first, to discuss before proceeding.
If you find this project useful, please give it a star.