Crawlingo is a Rust-powered web scraping framework with self-healing selectors, stealth TLS emulation, and Python, Node.js, and Rust SDKs. It achieves 3,500+ req/s with zero-copy architecture.

How does self-healing work in Crawlingo?

Crawlingo caches DOM element fingerprints (tag, class, id, text, structure). When a selector fails after a website redesign, the Rust engine scores all candidate nodes using Jaro-Winkler and Jaccard similarity, automatically binding the best match above 94% confidence.

How do I install Crawlingo?

Install via pip: `pip install crawlingo` (Python 3.8+), npm: `npm install crawlingo` (Node.js 18+), or cargo: `cargo add crawlingo` (Rust 1.70+).

Is Crawlingo free and open source?

Yes. Crawlingo is fully open source under the MIT License. You can use it freely in personal and commercial projects.

Features

A complete overview of Crawlingo's capabilities — from self-healing DOM selectors to AI-ready dataset export.

8 min read Updated July 2026

Self-Healing

Self-Healing DOM Fingerprinting

When websites redesign their HTML — renaming CSS classes, restructuring divs, renumbering IDs — traditional scrapers break silently. Crawlingo solves this by caching element layout fingerprints and using similarity matching to self-heal drifted selectors on the fly, without any code changes from you.

Auto-Match Lifecycle

First match succeeds

Crawlingo evaluates your selector (e.g. button#submit.btn-primary), caches a fingerprint: tag name, class set, id, attributes, text content, parent tag, sibling tags, depth, child index.

Subsequent visits update cache

On each successful match, the fingerprint is updated to account for minor layout drift — keeping the snapshot fresh.

Selector fails after redesign

The Rust engine intercepts the mismatch. It loads the current DOM and isolates candidate nodes within the same parent coordinates as the cached element.

Similarity scoring & rebind

All candidates are scored in parallel (Rayon) using Jaro-Winkler + Jaccard. The best match above 50% confidence is automatically bound and the cache is updated. Your code receives the correct element.

auto_match.py

python

1from crawlingo import Session, Page
2
3with Session() as s:
4    s.auto_match(True)
5
6    # First scrape: button#submit found and fingerprinted
7    page = s.page("https://example.com/checkout")
8    btn = page.css("button#submit").text()   # "Place Order"
9
10    # [Website redesigns: button becomes div.cta-button]
11    # Second scrape: auto_match silently finds the new element
12    page2 = s.page("https://example.com/checkout")
13    btn2 = page2.css("button#submit").text()  # Still works!
14
15    # Fine-tune fingerprint weights
16    s.auto_match_weights(
17        tag=1.0, class_name=0.8, id=0.6,
18        attributes=0.4, parent_tag=0.5, depth=0.1
19    )

Stealth Browsing

Stealth Browser Impersonation

Crawlingo compiles a raw HTTP/2 client inside the Rust core that rotates JA3/TLS handshake fingerprints, user-agent headers, and request timing gaps — bypassing Cloudflare Turnstile and similar bot detection systems without a headless browser.

TLS Fingerprint Rotation

JA3 signature emulates real Chrome, Firefox, and Safari TLS handshakes.

Multi-Profile Support

Switch between Chrome, Firefox, and Safari browser identity profiles per request.

Timing Randomization

Request gaps and header ordering randomized to avoid detection patterns.

stealth.py

python

1from crawlingo import Session, Page
2
3with Session() as s:
4    # Enable stealth mode
5    s.fetcher_tier("stealthy")
6
7    # Choose browser identity
8    s.browser_profile("chrome")   # "chrome" | "firefox" | "safari"
9
10    # Optional: custom headers on top
11    s.headers({
12        "Accept-Language": "en-US,en;q=0.9",
13        "Accept-Encoding": "gzip, deflate, br",
14    })
15
16    # Now fetches look like a real browser
17    page = s.page("https://cloudflare-protected-site.com")
18    print(page.status)  # 200 instead of 403

⚠️

Ethical use

Always respect a website's robots.txt, Terms of Service, and applicable laws. Crawlingo's stealth features are intended for legitimate data collection, research, and testing.

SIMD Text Anchors

SIMD-Accelerated Text Anchors

Many websites lack meaningful CSS classes or IDs. Text anchor selectors let you locate elements by their visible content — using memchr SIMD instructions for 2.1M ops/s throughput, making it faster than CSS selectors on large DOMs.

Selector Type Comparison

Type	API	Speed	Best For
CSS	`page.css("h1")`	850K/s	Standard element targeting
XPath	`page.xpath("//p")`	310K/s	Complex DOM traversal
Regex	`page.regex(r"\$[\d.]+")`	1.2M/s	Pattern-based extraction
Text Anchor	`page.find_text("Price:")`	2.1M/s	Visible content lookup
After/Before	`page.after_text("Price:")`	1.8M/s	Table/sibling extraction

text_anchors.py

python

1from crawlingo import Page
2
3page = Page("https://example.com/product")
4
5# Find element by exact visible text content
6title_el = page.find_text("Product Title")
7
8# Extract sibling AFTER a label text — perfect for price tables
9price = page.after_text("Price:").first().text()
10# → "€49.99"
11
12# Extract sibling BEFORE a unit label
13amount = page.before_text("USD").first().text()
14# → "1,234.56"
15
16# Works great for extracting data tables without CSS classes
17rows = page.after_text("Processor:").all()
18for row in rows:
19    print(row.text())

Change Monitoring

Reactive Watch Monitors

Watch polls target DOM nodes on configurable intervals, instantly publishing typed callbacks or webhooks when content, prices, stock status, or element presence changes.

watch.py

python

1from crawlingo import Watch
2import threading
3import requests
4
5def alert(event):
6    msg = f"[{event.field}] {event.old_value} → {event.new_value}"
7    requests.post("https://hooks.slack.com/services/...", json={"text": msg})
8
9w = (
10    Watch("https://shop.example.com/product/42")
11    .field("price", ".price")
12    .field("stock", ".stock-badge")
13    .interval(300)                   # every 5 minutes
14    .on_price_change(alert)
15    .on_stock_change(lambda e: print(f"Stock: {e.old_value} → {e.new_value}"))
16)
17
18# Run in background thread
19t = threading.Thread(target=w.run)
20t.start()
21# w.stop()

Dataset Export

Structured Dataset Export

The Dataset builder extracts multiple fields in one pass and exports them directly to JSON, CSV, or Parquet.

JSON

.to_json("file.json")

Web APIs, general analysis

CSV

.to_csv("file.csv")

Spreadsheets, data import

Parquet

.to_parquet("file.parquet")

Data warehouses, Spark, BigQuery

dataset_export.py

python

1from crawlingo import Dataset
2
3# Single-page extraction
4ds = (
5    Dataset("https://example.com/products")
6    .field("title",    "h1")
7    .field("price",    ".price")
8    .field("date",     "time.posted")
9    .field("url",      "")
10    .field("email",    r"[\w.]+@[\w]+\.[\w]+", selector_type="regex")
11    .build()
12)
13
14print(ds.to_dict())
15ds.to_json("data.json")
16ds.to_csv("data.csv")
17ds.to_parquet("data.parquet")
18
19# ─── Streaming large URL lists ────────────────────────────────
20urls = ["https://example.com/item/" + str(i) for i in range()]
21
22for record in Dataset(urls).field("title", "h1").field("price", ".price").stream():
23    # Processes at constant memory via bounded async channels
24    process(record.data)