Benchmarks
Performance measurements for Crawlingo's standard and stealthy fetch modes, auto-match scoring, streaming datasets, and selector types.
5 min read Updated July 2026
Methodology: All benchmarks run on a Linux x86_64 VM (8 vCPU, 16 GB RAM). HTTP target is an internal echo server with ≤1ms response time to measure Crawlingo overhead. Auto-match benchmarks use synthetic DOM trees of varying size. Streaming benchmarks use a URL list pointing to the echo server.
Request Throughput
Standard (HTTP/2, 50 concurrent)3,500 req/s
3,500 r/s
Stealthy (TLS fingerprint, 50 concurrent)1,800 req/s
1,800 r/s
Scrapy (Python async, 50 concurrent)~500 req/s
500 r/s
Playwright (browser-based)~50 req/s
50 r/s
| Mode | Throughput | p50 | p95 | p99 | Memory |
|---|---|---|---|---|---|
| Standard (50 concurrent) | 3,500 | 12 ms | 45 ms | 120 ms | 2.4 MB |
| Stealthy (50 concurrent) | 1,800 | 28 ms | 95 ms | 250 ms | 3.1 MB |
Selector Performance
| Selector Type | Throughput |
|---|---|
| CSS | 850K ops/s |
| XPath | 310K ops/s |
| Regex | 1.2M ops/s |
| Text Anchor | 2.1M ops/s |
| After/Before | 1.8M ops/s |
Auto-Match Scoring Time
Time to score all candidate nodes when a selector fails (Jaro-Winkler + Jaccard via Rayon parallel scoring).
| DOM Size | Scoring Time |
|---|---|
| 100 nodes | 45 μs |
| 1,000 nodes | 380 μs |
| 10,000 nodes | 3.2 ms |
Streaming Dataset Memory Usage
| URL Count | Peak Memory | Elapsed Time |
|---|---|---|
| 100 URLs | 8 MB | 0.8 s |
| 1,000 URLs | 42 MB | 7.5 s |
| 10,000 URLs | 85 MB | 72 s |
Full Comparison Table
| Metric | Crawlingo | Scrapy | Playwright |
|---|---|---|---|
| Throughput (50 concurrent) | 3,500 req/s | ~500 req/s | ~50 req/s (browser) |
| p50 latency | 12 ms | ~120 ms | ~800 ms |
| p99 latency | 120 ms | ~2 s | ~8 s |
| Memory (idle session) | 2.4 MB | ~50 MB | ~200 MB |
| Self-healing selectors | Built-in | Manual | Manual |
| Stealth TLS | Built-in | (full browser) | |
| Change detection | Built-in | ||
| Selector types | 5 (CSS/XPath/Regex/Text/Anchor) | 2 (CSS/XPath) | 3 (CSS/XPath/text) |
| Multi-SDK | Python, Node.js, Rust | Python only | JS, Python, C#, Java |
| Open source | MIT | BSD | Apache 2.0 |