Crawlingo is a Rust-powered web scraping framework with self-healing selectors, stealth TLS emulation, and Python, Node.js, and Rust SDKs. It achieves 3,500+ req/s with zero-copy architecture.

How does self-healing work in Crawlingo?

Crawlingo caches DOM element fingerprints (tag, class, id, text, structure). When a selector fails after a website redesign, the Rust engine scores all candidate nodes using Jaro-Winkler and Jaccard similarity, automatically binding the best match above 94% confidence.

How do I install Crawlingo?

Install via pip: `pip install crawlingo` (Python 3.8+), npm: `npm install crawlingo` (Node.js 18+), or cargo: `cargo add crawlingo` (Rust 1.70+).

Is Crawlingo free and open source?

Yes. Crawlingo is fully open source under the MIT License. You can use it freely in personal and commercial projects.

Model Context Protocol (MCP) Tools

Equip LLMs with high-speed stealth web search, structured data extraction, and BFS crawling capabilities via standard MCP schemas.

7 min read Updated July 2026

Overview

Crawlingo includes native support for the Model Context Protocol (MCP). By starting the server, you can interface with autonomous agents (like Claude Desktop, Gemini, or custom LLM frameworks) to let them fetch, parse, and crawl web content using Crawlingo's self-healing and stealth engines.

Starting the Server

The MCP server runs over an HTTP server exposing an SSE (Server-Sent Events) endpoint. Launch it via the CLI:

bash

# Start server on default port (http://127.0.0.1:8000/sse)
crawlingo mcp

# Bind to custom host and port
crawlingo mcp --host 0.0.0.0 --port 9000

Transport Protocol

Crawlingo implements the SSE transport specification for MCP client-server communication:

SSE Handshake (GET)

GET /sse

Establishes client streaming connection. Exposes initialization event containing the message path header.

Message Endpoint (POST)

POST /message

Clients POST JSON-RPC payloads containing commands. Responses are pushed to the SSE stream.

Exposed Tools Reference

TOOL 1

fetch_page

Fetches a single web page and extracts its title, HTTP response status, and trimmed body text.

Parameter	Type	Required	Description
`url`	`string`	Yes	The target web page URL.
`auto_match`	`boolean`	No (default: false)	Enable self-healing DOM fingerprint repair.
`timeout`	`integer`	No (default: 30)	Fetch timeout in seconds.

JSON-RPC Example Call

json

// POST request JSON-RPC payload
{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/call",
  "params": {
    "name": "fetch_page",
    "arguments": {
      "url": "https://news.ycombinator.com",
      "auto_match": true
    }
  }
}

TOOL 2

extract_data

Extracts customized fields from a single URL using CSS or XPath selectors with optional self-healing.

Parameter	Type	Required	Description
`url`	`string`	Yes	The target web page URL.
`fields`	`array[object]`	Yes	List of extraction fields. Each object requires: { name, selector, selector_type? }.
`auto_match`	`boolean`	No (default: false)	Enable self-healing selector recovery.

JSON-RPC Example Call

json

// POST request JSON-RPC payload
{
  "jsonrpc": "2.0",
  "id": 2,
  "method": "tools/call",
  "params": {
    "name": "extract_data",
    "arguments": {
      "url": "https://example.com",
      "auto_match": true,
      "fields": [
        { "name": "headline", "selector": "h1" },
        { "name": "cta", "selector": "//a[@class='button']", "selector_type": "xpath" }
      ]
    }
  }
}

TOOL 3

crawl_site

Starts a BFS crawler from a seed URL, following links, enqueuing pages, and extracting defined fields.

Parameter	Type	Required	Description
`start_url`	`string`	Yes	The seed page to start crawling.
`follow_selector`	`string`	Yes	CSS selector of link anchors to traverse.
`fields`	`array[object]`	Yes	Custom fields to extract from each crawled page.
`limit`	`integer`	No (default: 10)	Maximum number of pages to crawl.
`depth`	`integer`	No (default: 2)	Maximum traversal hops from the seed.

JSON-RPC Example Call

json

// POST request JSON-RPC payload
{
  "jsonrpc": "2.0",
  "id": 3,
  "method": "tools/call",
  "params": {
    "name": "crawl_site",
    "arguments": {
      "start_url": "https://shop.com/products",
      "follow_selector": "a.product-link",
      "limit": 25,
      "depth": 3,
      "fields": [
        { "name": "title", "selector": "h1.product-title" },
        { "name": "price", "selector": "span.price" }
      ]
    }
  }
}