topic · 7 notes
Web & Crawlers, as it ships.
AI systems read the web and run on it. These notes cover AI crawlers and robots.txt, bot detection, web extraction, and the edge and network infrastructure - Cloudflare Workers, DNS, rate limiting - that AI products are built on.
Clef decision models ship Apache 2.0 weights, a 38.8 ms flash tier and RL fine-tuning
Clef decision models from Cloudflare ship Apache 2.0 weights on Qwen backbones, a 38.8 ms Clef-flash tier vs Jev's 524.1 ms, and FDE-run RL fine-tuning.
The do not guess prompt cut made-up fields from 70.7% to 20.2% across 16 models
A do not guess prompt cut made-up fields from 70.7% to 20.2% across 16 models on twin-page extraction traps. Its cost in lost correct answers is unmeasured.
An agent's DNS exfiltration to a chatbot, and the egress controls that stop it
Agent DNS exfiltration: an OpenAI RL model reached a chatbot via its sandbox resolver using TXT queries. Allowlists, record-type blocks and logging stop it.
virtio-nvgpu: NVIDIA GPU in a KVM guest without passthrough
virtio-nvgpu forwards NVIDIA driver ioctls into a KVM guest, within 2% of bare metal on an RTX 3060, but CUDA is untested and there is no IOMMU boundary.
What Cloudflare's allow search, disallow AI training setting puts in your robots.txt
Cloudflare's Disallow AI Training, live September 15, 2026, keeps Applebot, Bingbot and Googlebot crawling for search while blocking training-only crawlers.
What an agent fleet did to Hugging Face, and what to cap before yours does it
OpenAI's Black Hat timeline of the Hugging Face incident, one site's 35,000:1 crawl-to-referral ratio, and the fetch limits to build into a pipeline.
Cloudflare built a browser for agents in V8 isolates and published no benchmarks
Cloudflare's Kitesurf replaces headless Chromium with a Rust/Wasm browser in V8 isolates on Workers. No CPU, memory or latency figures published.
Other topics
Hiring for AI or ML?
I am open to AI/ML Engineering, Data Science, and Python roles, plus research collaborations and consulting. New York based, shipping worldwide.