3.12. Crawl Pool
- Plugin Key
plugin_fetch_rate_limiter_source
- Plugin Value Type
- Description
A specification for how AUs are split into crawl pools. Only one new content crawl from a given crawl pool can happen at once. In the default crawl pool scheme, there is one crawl pool per plugin, but this plugin key can be used to define a new behavior.
The possible values are:
au
: Each AU has its own crawl pool.plugin
: The pool name is the Plugin Identifier.key:str
: The pool name is the stringstr
.host:urlparamkey
: The pool name is the stringhost:
followed by the value of the AU's parameter namedurlparamkey
. If there is no such parameter, or if the parameter is not of type URL, the default crawl pool scheme applies.title_attribute:auattrkey
andtitle_attribute:auattrkey:dflt
: The pool name is the stringattr:
followed by the value of the AU's attribute namedauattrkey
. When the attribute is unset, if the longer form is used then usedflt
as the value instead; otherwise the default crawl pool scheme applies.