scrapyd

Last released Oct 2, 2024

A service for running Scrapy spiders, with an HTTP API

itemloaders

Last released Sep 30, 2024

Base library for scrapy's ItemLoader

scrapy-zyte-smartproxy

Last released Aug 5, 2024

Scrapy middleware for Zyte Smart Proxy Manager

scrapy-poet

Last released Jul 18, 2024

Page Object pattern for Scrapy

form2request

Last released Jul 15, 2024

Build HTTP requests out of HTML forms

w3lib

Last released Jun 12, 2024

Library of web-related functions

Scrapy

Last released May 14, 2024

A high-level Web Crawling and Web Scraping framework

itemadapter

Last released May 7, 2024

Common interface for data container classes

queuelib

Last released May 4, 2024

Collection of persistent (disk-based) and non-persistent (memory-based) queues

parsel

Last released Apr 8, 2024

Parsel is a library to extract data from HTML and XML using XPath and CSS selectors

Protego

Last released Apr 5, 2024

Pure-Python robots.txt parser with support for modern conventions

web-poet

Last released Mar 4, 2024

Zyte's Page Object pattern for web scraping

xtractmime

Last released Jan 16, 2024

Implementation of the MIME Sniffing standard (https://mimesniff.spec.whatwg.org/)

andi

Last released Dec 26, 2023

Library for annotation-based dependency injection

scrapy-splash

Last released Feb 3, 2023

JavaScript support for Scrapy using Splash

scrapyd-client

Last released Jan 30, 2023

A client for Scrapyd

cssselect

Last released Oct 27, 2022

cssselect parses CSS3 Selectors and translates them to XPath 1.0

scrapy-deltafetch

Last released Sep 20, 2021

Scrapy middleware to ignore previously crawled pages

splash

Last released Jun 16, 2020

A javascript rendered with a HTTP API

scrapely

Last released Nov 28, 2019

A pure-python HTML screen-scraping library

scrapy-po

Last released Aug 28, 2019

Page Object pattern for Scrapy

webstruct

Last released Dec 29, 2017

A library for creating statistical NER systems that work on HTML data

PyPyDispatcher

Last released Jul 3, 2017

Multi-producer-multi-consumer signal dispatching mechanism

adblockparser

Last released Oct 17, 2016

Parser for Adblock Plus rules

loginform

Last released Aug 16, 2016

Fill HTML login forms automatically

scrapy-splitvariants

Last released Jul 18, 2016

Scrapy spider middleware to split an item into multiple items on a multi-valued key

scrapy-hcf

Last released Jul 18, 2016

Scrapy spider middleware to use Scrapinghub's Hub Crawl Frontier as a backend for URLs

scrapy-querycleaner

Last released Jun 30, 2016

Scrapy spider middleware to clean up query parameters in request URLs

scrapy-magicfields

Last released Jun 30, 2016

Scrapy middleware to add extra "magic" fields to items

scrapy-djangoitem

Last released May 4, 2016

Scrapy extension to write scraped items using Django models

scrapyjs

Last released Mar 25, 2016

JavaScript support for Scrapy using Splash

scrapy-jsonrpc

Last released Apr 13, 2015

Scrapy extenstion to control spiders using JSON-RPC