FAQ & Troubleshooting
Answers to common questions about installation, usage, stealth browsing, auto-match, and performance.
General
Q: What is Crawlingo?
Crawlingo is a Rust-powered, high-performance web scraping and monitoring framework with native bindings for Python, Node.js (TypeScript), and Rust. It features self-healing DOM selectors, stealth TLS fingerprinting, and reactive change monitors.
Q: Is Crawlingo free and open source?
Yes! Crawlingo is 100% open-source software licensed under the permissive MIT License.
Installation & Build
Q: pip install fails on Linux
Ensure glibc 2.28+ is installed on your Linux distribution (ldd --version). Alternatively, install from source using: pip install --no-binary crawlingo crawlingo (requires Rust 1.70+).
Q: npm install fails
Ensure you are using Node.js 18+. Check that platform-specific prebuilt binaries exist for your OS/architecture. If compiling from source, ensure Visual Studio Build Tools (Windows) or build-essential (Linux) are installed.
Usage & Troubleshooting
Q: Getting HTTP 403 Forbidden on protected sites
The target website blocks generic HTTP clients. Enable stealth mode in your session: session.fetcher_tier("stealthy").browser_profile("chrome") to emulate genuine browser TLS/HTTP2 handshakes.
Q: Selector returns empty or fails after layout changes
Enable Auto-Match on your Page or Session: session.auto_match(True). Crawlingo will use stored DOM fingerprints to automatically heal broken selectors in production.
Performance & Benchmarks
Q: What is Crawlingo's throughput?
Crawlingo can achieve up to 3,500+ requests per second for lightweight page fetches on commodity hardware, using Tokio async I/O and Rayon parallel CPU processing.
Q: What is the memory footprint?
Idle memory footprint is around ~2.4 MB. Multi-page crawls require ~85 MB constant memory when using the streaming dataset API.