Crawlers, floods and fleets: a year of keeping our digital collections open stacks.wellcomecollection.org

Wellcome Collection’s Robert Kenny and Raphaëlle Cantin describe how automated traffic from crawlers, distributed search floods, browser fleets and bulk harvesters drove up costs and disrupted access to its digital collections from June to September 2026. The team reduced data transfer costs by more than 90% through compression, added crawler rules and traffic limits, and used browser challenges to protect search pages; a fleet still evades detection on item pages, while the IIIF API remains open with usage budgets.