3.12. Crawl Pool

Plugin Key

plugin_fetch_rate_limiter_source

Plugin Value Type

String

Description

A specification for how AUs are split into crawl pools. Only one new content crawl from a given crawl pool can happen at once. In the default crawl pool scheme, there is one crawl pool per plugin, but this plugin key can be used to define a new behavior.

The possible values are:

  • au: Each AU has its own crawl pool.

  • plugin: The pool name is the Plugin Identifier.

  • key:str: The pool name is the string str.

  • host:urlparamkey: The pool name is the string host: followed by the value of the AU's parameter named urlparamkey. If there is no such parameter, or if the parameter is not of type URL, the default crawl pool scheme applies.

  • title_attribute:auattrkey and title_attribute:auattrkey:dflt: The pool name is the string attr: followed by the value of the AU's attribute named auattrkey. When the attribute is unset, if the longer form is used then use dflt as the value instead; otherwise the default crawl pool scheme applies.