Crawlingo is a Rust-powered web scraping framework with self-healing selectors, stealth TLS emulation, and Python, Node.js, and Rust SDKs. It achieves 3,500+ req/s with zero-copy architecture.

How does self-healing work in Crawlingo?

Crawlingo caches DOM element fingerprints (tag, class, id, text, structure). When a selector fails after a website redesign, the Rust engine scores all candidate nodes using Jaro-Winkler and Jaccard similarity, automatically binding the best match above 94% confidence.

How do I install Crawlingo?

Install via pip: `pip install crawlingo` (Python 3.8+), npm: `npm install crawlingo` (Node.js 18+), or cargo: `cargo add crawlingo` (Rust 1.70+).

Is Crawlingo free and open source?

Yes. Crawlingo is fully open source under the MIT License. You can use it freely in personal and commercial projects.

Configuration & CLI

Configure Crawlingo sessions via JSON/TOML files, CRAWLINGO_* environment variables, or launch pre-configured CLI sessions.

6 min read Updated July 2026

Configuration Precedence

Crawlingo session configuration follows a strict overlay hierarchy. Values loaded from higher layers override those below:

Default Engine Config

Built-in hardcoded Rust defaults.

File Config (TOML / JSON)

Parsed at initialization via crawlingo.toml or crawlingo.json.

Environment Variables

Process-wide CRAWLINGO_* overrides.

Runtime Setters

Explicit method calls (e.g. s.rate_limit(5)) take final precedence.

Configuration Specs

Field Name	Type	Default	Description
`headers`	Map<String, String>	`{}`	Default HTTP headers for all outgoing requests.
`proxy`	String	`None`	Single proxy server URL (e.g., http://127.0.0.1:8080).
`proxy_pool`	List<String>	`[]`	Round-robin proxy rotation pool list.
`proxy_provider_url`	String	`None`	HTTP endpoint returning newline-separated proxy lists to dynamically reload.
`rate_limit_rps`	Float	`0.0`	Requests-per-second limit per host (0.0 = unlimited).
`auto_match`	Boolean	`false`	Enable self-healing selector fingerprint repair (auto-matching).
`timeout_seconds`	Integer	`30`	Request connection, handshake, and read timeout in seconds.
`fingerprint_path`	String	`None`	Path to storage directory for sled ACID self-healing fingerprints database.
`fetcher_tier`	String	`"standard"`	"standard" for raw high-speed HTTP/2; "stealthy" to activate TLS fingerprint rotation.
`browser_profile`	String	`"chrome"`	Stealth client signature profile: "chrome", "firefox", or "safari".

Connection Pool Settings

Config Field	Environment Variable	Default	Description
`pool.max_idle_per_host`	`CRAWLINGO_POOL_MAX_IDLE_PER_HOST`	`32`	Maximum idle connections retained per host.
`pool.max_clients`	`CRAWLINGO_POOL_MAX_CLIENTS`	`1024`	Hard limit on concurrent in-flight clients.
`pool.idle_timeout_secs`	`CRAWLINGO_POOL_IDLE_TIMEOUT_SECS`	`90`	Duration in seconds after which idle connections are closed.

Retry Policy Settings

Config Field	Environment Variable	Default	Description
`retry.base_delay_ms`	`CRAWLINGO_RETRY_BASE_DELAY_MS`	`100`	Initial delay in milliseconds for backoff.
`retry.max_delay_ms`	`CRAWLINGO_RETRY_MAX_DELAY_MS`	`10000`	Maximum cap on retry backoff delays.
`retry.multiplier`	`CRAWLINGO_RETRY_MULTIPLIER`	`2.0`	Exponential backoff rate multiplier.

Example Config Files

crawlingo.toml

toml

proxy = "http://myproxy.com:8000"
rate_limit_rps = 3.5
auto_match = true
timeout_seconds = 45
fetcher_tier = "stealthy"
browser_profile = "firefox"

[pool]
max_idle_per_host = 16
idle_timeout_secs = 60

[retry]
base_delay_ms = 250
multiplier = 1.5

crawlingo.json

json

{
  "proxy": "http://myproxy.com:8000",
  "rate_limit_rps": 3.5,
  "auto_match": true,
  "timeout_seconds": 45,
  "fetcher_tier": "stealthy",
  "pool": {
    "max_idle_per_host": 16
  },
  "retry": {
    "base_delay_ms": 250
  }
}

CLI Interface

The Python SDK comes with a preconfigured command-line interface. Start standard scrapers, test query rules, or launch local agent server controllers instantly.

1. Quick Selector Extraction

bash

# Extract all H1 text content
crawlingo extract https://news.ycombinator.com --css "td.title a"

# Use XPath and output as a clean JSON map
crawlingo extract https://example.com --xpath "//h1" --json

# Run with self-healing auto-matching enabled
crawlingo extract https://shop.com --css "button.add-to-cart" --auto-match

2. Preloaded REPL Shell

bash

# Spin up Python REPL with Crawlingo Page/Session/Dataset classes pre-imported
crawlingo shell

# Fetch a page at load and place it inside the context as "page"
crawlingo shell https://example.com

3. Model Context Protocol Server

bash

# Start the local HTTP/SSE server for AI Agent integration
crawlingo mcp --host 127.0.0.1 --port 8000

💡

Pre-built Matched Binaries

If running inside server environments or containers, environment variables automatically take precedence over configurations loaded from TOML files, making it easy to configure proxies via K8s secrets.