r-robotstxt 0.7.15
Propagated dependencies: r-stringr@1.5.1 r-spiderbar@0.2.5 r-magrittr@2.0.3 r-httr@1.4.7 r-future-apply@1.11.3
Channel: guix-cran
Home page: https://docs.ropensci.org/robotstxt/
Licenses: Expat
Synopsis: 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker
Description:
This package provides functions to download and parse robots.txt files. Ultimately the package makes it easy to check if bots (spiders, crawler, scrapers, ...) are allowed to access specific resources on a domain.
Total results: 1