Crawlingo is a Rust-powered web scraping framework with self-healing selectors, stealth TLS emulation, and Python, Node.js, and Rust SDKs. It achieves 3,500+ req/s with zero-copy architecture.

How does self-healing work in Crawlingo?

Crawlingo caches DOM element fingerprints (tag, class, id, text, structure). When a selector fails after a website redesign, the Rust engine scores all candidate nodes using Jaro-Winkler and Jaccard similarity, automatically binding the best match above 94% confidence.

How do I install Crawlingo?

Install via pip: `pip install crawlingo` (Python 3.8+), npm: `npm install crawlingo` (Node.js 18+), or cargo: `cargo add crawlingo` (Rust 1.70+).

Is Crawlingo free and open source?

Yes. Crawlingo is fully open source under the MIT License. You can use it freely in personal and commercial projects.

Architecture

How Crawlingo's Rust core works under the hood — FFI layers, the fetch pipeline, parallelism model, and the reasoning behind every design decision.

10 min read Updated July 2026

Overview

Crawlingo is structured as a single compiled Rust library that exposes thin FFI boundaries for Python (via PyO3) and Node.js (via napi-rs). All heavy computation — HTTP/2 fetching, HTML parsing, selector evaluation, DOM fingerprinting, and dataset streaming — happens inside the Rust process with zero-copy shared memory.

System Architecture

User Code (Python / Node.js / Rust)
         │
         ▼
  ┌──────────────────────────────┐
  │     FFI Boundary (PyO3 / napi-rs)     │
  └──────────────┬───────────────┘
                 │
         ▼
  ┌──────────────────────────────────────────────────────────────┐
  │                        RUST CORE                             │
  │                                                              │
  │  Session ──► FetchManager ──► [Standard | Stealthy] Fetcher  │
  │      │           │                     │                      │
  │      │           ▼                     ▼                      │
  │      │    Rate Limiter          HTTP/2 + TLS                  │
  │      │    (governor)            Fingerprint                   │
  │      │           │                     │                      │
  │      │           ▼                     ▼                      │
  │      │      CachingLayer        Retry (backoff)               │
  │      │           │                     │                      │
  │      │           ▼                     ▼                      │
  │      │       Middleware          html5ever                    │
  │      │    (Metrics, Auth)         Parser                      │
  │      │                               │                        │
  │      │                               ▼                        │
  │      │                    DOM (scraper crate)                 │
  │      │                               │                        │
  │      ├──── Selectors ────────────────┤                        │
  │      │   CSS │ XPath │ Regex │ Text Anchor │ After/Before     │
  │      │                               │                        │
  │      ├──── Dataset Builder ──────────┤                        │
  │      │   Fields │ Extraction Types │ Streaming                │
  │      │                               │                        │
  │      ├──── Crawl (BFS) ─────────────┤                        │
  │      │   Frontier │ Concurrency │ robots.txt                  │
  │      │                               │                        │
  │      ├──── Watch ────────────────────┤                        │
  │      │   Polling │ Diff │ Callbacks                           │
  │      │                               │                        │
  │      └──── Fingerprint Store ────────┘                        │
  │            (sled embedded DB)                                 │
  │                                                              │
  │   Metrics: DashMap + atomics (lock-free)                     │
  └──────────────────────────────────────────────────────────────┘

Fetch Pipeline

Every request flows through a layered middleware stack before hitting the wire:

Page("url")

→

Cache?

→

Middleware

→

Rate Limit

→

Transport

→

HTTP/2

→

Retry

→

html5ever Parser

→

DOM

Parallelism Model

Rayon (CPU-bound)

•DOM fingerprint scoring during auto-match
•Parallel field extraction per document
•Dataset streaming chunks
•Work-stealing scheduler across all CPU cores

Tokio (I/O-bound)

•HTTP/2 fetching (concurrent requests)
•Crawl task management (Semaphore)
•Watch polling intervals
•Async task pool with work-stealing

DashMap + Atomics

•Metrics counters (lock-free reads)
•Connection pool cache (moka LRU)
•Session config (copy-on-write)
•Fingerprint hot-path cache

ℹ️

Backpressure in streaming

The streaming dataset uses bounded Tokio channels. Producers block when the consumer channel is full, preventing unbounded memory growth when processing millions of URLs. The crawl engine uses a Tokio Semaphore to cap concurrent in-flight requests.

Core Dependencies

Crate	Purpose
`tokio`	Async runtime (I/O-bound work)
`rayon`	Parallel iteration (CPU-bound work — DOM scoring, field extraction)
`wreq / wreq-util`	HTTP/2 client + TLS fingerprint emulation
`html5ever / scraper`	Spec-conformant HTML5 parser + DOM traversal
`regex`	Regex selector engine
`memchr`	SIMD text anchor search (2.1M ops/s)
`sled`	Embedded fingerprint database (ACID, no external deps)
`governor`	Per-host token bucket rate limiter
`moka`	LRU response cache
`dashmap`	Lock-free concurrent hash maps (metrics, fingerprint store)
`serde / serde_json`	Serialization / deserialization
`pyo3`	Python FFI (PyO3 bindings)
`napi / napi-derive`	Node.js N-API FFI (auto .d.ts generation)
`tracing`	Structured async logging
`toml / envy`	Config file parsing + env var override

Design Decisions

Decision	Choice	Reasoning
Core language	`Rust`	Memory safety, zero-cost abstractions, excellent FFI compatibility
HTML parser	`html5ever + scraper`	Spec-conformant, handles malformed real-world HTML
Async runtime	`Tokio`	Industry standard, work-stealing scheduler, native async/await
CPU parallelism	`Rayon`	Work-stealing thread pool, simple data-parallel API
FFI (Python)	`PyO3`	Mature, async-safe, maturin build tooling
FFI (Node.js)	`napi-rs`	Type-safe, auto .d.ts generation, N-API stability guarantees
Config	`toml + envy`	Human-readable + env var override for container deployments
Fingerprint store	`sled`	Embedded, ACID transactions, zero external process deps
Rate limiting	`governor`	Token bucket, per-key, async-aware

Error Types

FetchErrorHTTP errors, DNS failures, TLS handshake, timeouts

ParseErrorInvalid HTML, unknown encoding

SelectorErrorInvalid CSS/XPath/regex syntax

ExtractionErrorValue normalization failure, unknown extraction type

ConfigErrorMissing file, parse error, invalid value

IoErrorFile read/write failure (export methods)

SdkErrorSerialization, FFI callback exceptions