3.11. Crawl Rate Limiter#
Note
This page is under construction.
- Plugin Key
?
- Plugin Value Type
?
- Description
Fine-grained control of the maximum rate at which URLs may be fetched, based on media type, URL pattern, day of week or time of day.
Table of Contents
LOCKSS System
LOCKSS Guides
- LOCKSS 1.x to 2.x Migration Guide
- LOCKSS Network Administrator Guide
- LOCKSS Plugin Developer Guide
- 1. Introduction
- 2. Identification
- 3. Crawl Control
- 3.1. Start URLs
- 3.2. Crawl Seed
- 3.3. Permission URLs
- 3.4. Per-Host Permission Path
- 3.5. Permitted Host Pattern
- 3.6. Crawl Rules
- 3.7. Crawl Window
- 3.8. Recrawl Interval
- 3.9. Refetch Depth
- 3.10. Fetch Pause Time
- 3.11. Crawl Rate Limiter
- 3.12. Crawl Pool
- 3.13. Response Handler
- 3.14. URL Normalizer
- 3.15. Link Extractor
- 3.16. Crawl Filter
- 3.17. URL Fetcher
- 3.18. URL Consumer
- 4. Crawl Validation
- 5. Poll Control
- 6. Hash Filtering
- 7. Metadata Extraction
- 8. Web Replay
- 9. Inheritance
- 10. Appendix
- LOCKSS Software Developer Guide
Navigation
- LOCKSS Program Web Site
- ยป LOCKSS Documentation Portal
- LOCKSS Community Wiki
- LOCKSS Community Discussions