v0.1.0·Open Source · MIT License

Web Scraping
That Never Breaks.

Crawlingo is a Rust-powered web scraping framework with self-healing selectors, stealth TLS, and first-class Python, Node.js, and Rust SDKs. Achieve 3,500+ req/s with a 2.4 MB memory footprint.

$pip install crawlingo·$npm install crawlingo
3,500+
requests / second
Standard mode (50 concurrent)
5
selector types
CSS · XPath · Regex · Text · Anchor
3
language SDKs
Python · Node.js · Rust
2.4 MB
memory (idle session)
94× lighter than Playwright
Developer First

One unified API.
Three languages.

The same Rust core powers all three SDKs with zero-copy memory sharing and identical semantics. Switch languages without rewriting your scraping logic.

  • 🔧Self-healing selectors auto-fix when websites change
  • 🛡️Stealth TLS rotates browser fingerprints per request
  • 📦Export to JSON, CSV, or Parquet in one method call
  • 👁️Watch monitors fire webhooks on any DOM change
1from crawlingo import Page, Session, Dataset, Crawl, Watch
2
3# Fetch a page instantly
4page = Page("https://example.com")
5print(page.title()) # "Example Domain"
6print(page.status) # 200
7print(page.markdown()[:80]) # Clean markdown output
8
9# CSS, XPath, Regex, Text Anchors — all in one API
10h1 = page.css("h1").text()
11paras = page.xpath("//p")
12prices = page.regex(r"\$[\d.]+")
13el = page.find_text("Price:") # SIMD-accelerated
14
15# Self-healing session
16with Session() as s:
17 s.auto_match(True).fetcher_tier("stealthy").rate_limit(5)
18
19 result = (Dataset("https://shop.example.com", session=s)
20 .field("title", "h1")
21 .field("price", ".price", extract_type="price")
22 .field("stock", ".stock-badge")
23 .build())
24
25 print(result.to_dict()) # {"title": "...", "price": "99.99"}
26 result.to_json("data.json")
27 result.to_parquet("data.parquet")
Features

Built for production.
Designed for developers.

Everything you need to build reliable data pipelines that survive website changes.

Self-Healing Selectors

DOM fingerprints + Jaro-Winkler similarity scoring auto-repair selectors when websites change their structure. Never write a CSS fix again.

Stealth TLS Browsing

Bypass Cloudflare Turnstile and bot detection via raw HTTP/2 with JA3/TLS fingerprint rotation. Chrome, Firefox, Safari profiles built-in.

SIMD-Accelerated

Text anchor search runs at 2.1M ops/s using parallel CPU vectors (memchr + Rayon). Extract tables even when CSS classes are absent.

Proxy Rotation

Round-robin static pool, remote proxy list provider, or per-host rotation with exponential backoff and Retry-After header support.

Change Monitoring

Poll-based DOM watchers publish typed callbacks (on_change, on_price_change, on_stock_change) and webhooks when content drifts.

Zero-Copy Rust Core

Compiled Rust engine with memory-mapped shared state across Python, Node.js, and Rust SDKs. 3,500+ req/s on commodity hardware.

Architecture

One engine. Zero compromises.

A Rust core compiled once, wrapped by thin FFI layers for each language SDK.

Python SDK
Node.js SDK
Rust SDK
FFI
Rust Core
Engine
Parser
Selectors
Dataset
Crawl
Watch
Metrics
HTTP/2
Stealth TLS
Proxy Pool
Rate Limiter
Fingerprint DB
Performance

Benchmarks don't lie.

Crawlingo outperforms pure-Python scrapers by 7×, and headless browsers by 70×.

MetricCrawlingoScrapyPlaywright
Throughput (50 concurrent)3,500 req/s~500 req/s~50 req/s
Memory (session idle)2.4 MB~50 MB~200 MB
p50 latency12 ms120 ms800 ms
Self-healing selectorsBuilt-in
Stealth TLSBuilt-in
Change monitoringBuilt-in
Multi-language SDKsPython, Node, RustPythonJS, Python, C#

Start building in 2 minutes.

Install via your language's package manager. Pre-built wheels for Linux, macOS, and Windows.

PythonPython 3.8+
pip install crawlingo
Node.jsNode.js 18+
npm install crawlingo
RustRust 1.70+
cargo add crawlingo