Changelog
Release notes for every version of Crawlingo. All notable changes, additions, bug fixes, and breaking changes.
3 min read Updated July 2026
All notable changes to Crawlingo are documented here. Crawlingo follows Semantic Versioning.
High-performance optimizations, FFI binding completion, and sitemap decompression.
Added
- +FFI Pagination Config: Python and JS client classes supporting NextLink, PageNumber, and UrlPattern selectors
- +FFI Dataset Schema validation: dynamic field constraints check (required status and type assertions)
- +Sitemap Gzip support: auto-decompresses gzipped sitemaps (.xml.gz) via flate2
- +Monotonic B-Tree PersistentFrontier layout optimization ($O(1)$ writes, $O(log n)$ LIFO reads)
- +Non-blocking FingerprintStore writes (asynchronous flushes on session teardown)
- +Batched Parquet serialization stream (default chunk size of 1000 records)
- +Pre-compiled regex pattern queries in pagination to avoid hot-spot compilations
Breaking
- !Removed individual rate_limit_rps field from FetchRequest (rate limiting moved to Session/FetchManager level)
- !Converted ExtractionRule to a type alias for DatasetField to remove layer duplication
Initial public release of Crawlingo.
Added
- +Core Rust engine with HTTP/2 standard and stealthy fetcher tiers
- +Python SDK (PyO3) — Page, Session, Dataset, Crawl, Watch
- +Node.js SDK (napi-rs) — full TypeScript typings, auto-generated .d.ts
- +Rust native crate API
- +CSS, XPath, Regex, Text Anchor, After/Before Text selectors
- +Self-healing auto-match with DOM fingerprinting (Jaro-Winkler + Jaccard)
- +Stealth TLS with Chrome, Firefox, Safari browser profiles
- +SIMD-accelerated text anchor search via memchr
- +Proxy rotation: static pool, round-robin, remote provider
- +Per-host rate limiting via governor (token bucket)
- +Reactive Watch monitors with typed callbacks and webhook support
- +Dataset streaming with bounded channels (constant memory)
- +Export to JSON, CSV, Parquet
- +Middleware stack: MetricsLayer, CachingLayer, AuthLayer (Basic, Bearer, API Key, OAuth2)
- +TOML config + CRAWLINGO_* env var support
- +Lock-free metrics counters (DashMap + atomics)
- +BFS Crawl with depth, concurrency, robots.txt, domain allowlist
- +Pre-built wheels: Linux x86_64/aarch64, macOS x86_64/arm64, Windows AMD64
Internal alpha — Rust core proof of concept.
Added
- +Rust core with basic HTTP/2 fetcher
- +HTML parser (html5ever)
- +CSS selector support
- +PyO3 Python bindings prototype