Crawlingo is a Rust-powered web scraping framework with self-healing selectors, stealth TLS emulation, and Python, Node.js, and Rust SDKs. It achieves 3,500+ req/s with zero-copy architecture.

How does self-healing work in Crawlingo?

Crawlingo caches DOM element fingerprints (tag, class, id, text, structure). When a selector fails after a website redesign, the Rust engine scores all candidate nodes using Jaro-Winkler and Jaccard similarity, automatically binding the best match above 94% confidence.

How do I install Crawlingo?

Install via pip: `pip install crawlingo` (Python 3.8+), npm: `npm install crawlingo` (Node.js 18+), or cargo: `cargo add crawlingo` (Rust 1.70+).

Is Crawlingo free and open source?

Yes. Crawlingo is fully open source under the MIT License. You can use it freely in personal and commercial projects.

Python SDK

Complete method reference for the Crawlingo Python SDK (PyO3 bindings). Requires Python 3.8+.

12 min read Updated July 2026

ℹ️

Installation

pip install crawlingo — Python 3.8+. Pre-built wheels for Linux (x86_64, aarch64), macOS (Intel, Apple Silicon), Windows AMD64.

Session

Central config container. All operations bind to a Session and share its FetchManager, rate limiter, connection pool, middleware stack, and fingerprint store. Use as a context manager for automatic cleanup.

Method	Returns	Description
`Session() / Session.from_config(path)`	Session	Create from defaults or TOML/env config
`.headers(dict)`	Self	Default HTTP headers for all requests
`.cookies(dict)`	Self	Default cookies for all requests
`.proxy(str)`	Self	Single proxy URL
`.proxy_pool(list)`	Self	Round-robin proxy rotation list
`.proxy_provider(str)`	Self	Remote proxy list endpoint URL
`.rate_limit(float)`	Self	Per-host req/s (0.0 = unlimited)
`.auto_match(bool)`	Self	Enable self-healing selector repair
`.auto_match_weights(*floats)`	Self	Fingerprint similarity weights (tag, class, id, attr, text, parent, pos, sibling, depth)
`.timeout(int)`	Self	Request timeout in seconds
`.fetcher_tier(str)`	Self	"standard" or "stealthy"
`.browser_profile(str)`	Self	"chrome", "firefox", or "safari"
`.cache_enabled(bool)`	Self	Enable response caching (Cache-Control, ETag aware)
`.retry_base_delay(int)`	Self	Initial retry delay (ms)
`.retry_max_delay(int)`	Self	Maximum retry delay (ms)
`.retry_multiplier(float)`	Self	Exponential backoff factor
`.basic_auth(user, pass)`	Self	HTTP Basic Auth
`.bearer_auth(token)`	Self	Bearer token auth
`.header_auth(name, value)`	Self	Custom auth header
`.api_key_auth(param, value)`	Self	Query parameter API key
`.auth_oauth2(client_id, secret, token_url)`	Self	OAuth2 with auto-refresh on 401
`.page(url)`	Page	Fetch a page using this session
`.metrics()`	dict	Lock-free metrics snapshot
`.clone()`	Session	Clone session and its config
`.destroy()`	None	Destroy session and release resources

Page

Method	Returns	Description
`Page(url, session?)`	Page	Fetch and parse URL (sync)
`.title()`	str	<title> text content
`.html()`	str	Raw HTML string
`.markdown()`	str	GitHub-flavored markdown
`.status`	int	HTTP response status code
`.css(selector)`	ElementList	CSS query — returns ElementList
`.xpath(expr)`	ElementList	XPath query — returns ElementList
`.regex(pattern)`	MatchList	Regex matches against all visible text
`.find_text(text)`	ElementList	SIMD text anchor lookup
`.after_text(text)`	ElementList	Sibling element after text anchor
`.before_text(text)`	ElementList	Sibling element before text anchor

ElementList & ElementRef

element_ref.py

python

1els = page.css("div")
2els.first()     # First match or None
3els.last()      # Last match or None
4els.at(i)       # i-th match
5len(els)        # Count of matches
6for e in els:   # Iterable
7    pass
8
9e = els.first()
10e.text()            # Trimmed text content
11e.html()            # Inner HTML
12e.outer_html()      # Including element tag
13e.attr("href")      # Attribute value
14e.tag()             # Tag name (lowercase)
15e.classes()         # ["class-a", "class-b"]
16e.parent()          # Parent ElementRef
17e.children()        # Direct children
18e.next_sibling()    # Next sibling
19e.prev_sibling()    # Previous sibling

Dataset

Method	Returns	Description
`Dataset(url, session?)`	Dataset	Create for a URL
`.field(name, selector, selector_type?, transform?, default?)`	Self	Add extraction field
`.with_schema(schema)`	Self	Apply DatasetSchema validation
`.build()`	DatasetResult	Execute extraction synchronously
`.build_async()`	Awaitable[DatasetResult]	Execute extraction asynchronously
`.build_structured()`	list[dict]	Extract multi-row table records from a page
`→ .to_dict()`	dict	Fields as Python dict
`→ .to_json(path)`	None	Write JSON file
`→ .to_csv(path)`	None	Write CSV file
`→ .to_parquet(path)`	None	Write Apache Parquet file
`→ .df()`	DataFrame	Return as Pandas DataFrame

Extraction Types

Type	Input → Output	Use Case
`text`	" Hello " → "Hello"	General text trimming (default)
`price`	"$1,234.56" → "1234.56"	Currency normalization
`datetime`	"Jan 15, 2024" → "2024-01-15"	Date standardization to ISO 8601
`url`	"/path" → "https://base.com/path"	Relative URL resolution
`datalink_url`	<a href="..."> → href value	Link extraction
`datalink_email`	"mailto:a@b.com" → "a@b.com"	Email extraction
`datalink_phone`	"tel:+1234" → "+1234"	Phone number extraction