40:"/_next/static/chunks/0m2qavasnhva7.js" 41:I[16093,["$7","$8","$9","$a","$b","$c","$40"],"default"] 42:T1c00,# Case Study: Building a High-Performance Sitemap Generator with Node.js & Crawlee ## Introduction In the world of Technical SEO, discoverability is everything. You can have the best content on the web, but if search engines can't find it, it doesn't exist. The humble **XML Sitemap** is the roadmap we give to Google, Bing, and other crawlers to ensure every piece of valuable content is indexed. However, for large, dynamic, or legacy websites, maintaining an up-to-date sitemap is a logistical nightmare. plugin-based solutions often fail on decoupled CMS architectures, and enterprise SaaS tools can be prohibitively expensive. I recognized a gap in the market for a **developer-first, highly configurable automation tool** that could crawl any website—regardless of its tech stack—and generate compliant XML, HTML, and TXT sitemaps. This case study breaks down how I built the **Fast Sitemap Generator**, a high-performance actor on the Apify platform, designed to solve the problem of SEO visibility at scale. ## The Challenge: Why Not Just Use a Plugin? Building a sitemap generator sounds simple on paper: *visit a page, grab links, repeat.* But when you move from a 10-page blog to a 10,000-page e-commerce site, complexity explodes. I faced three core technical hurdles: 1. **Infinite Loops & Spider Traps**: Many modern sites have calendar widgets, faceted search parameters (e.g., `?price=low&color=red`), or relative link structures that can trap a naive crawler in an infinite loop. 2. **Politeness & Compliance**: A responsible crawler must respect `robots.txt`. Ignoring standard exclusion protocols can get IPs banned or negatively impact the target site's server load. 3. **Performance vs. Depth**: How do you crawl deep into a site structure (Level 5+) without the process taking hours or consuming excessive memory? My goal was to build a tool that was **robust enough for enterprise sites** but **simple enough for a junior dev to configure**. ## The Architecture: Built on Giant Shoulders To tackle these challenges, I chose a stack optimized for speed and reliability: * **Runtime**: Node.js (for non-blocking I/O). * **Crawler Engine**: **Crawlee** (specifically `CheerioCrawler`). * **Platform**: Apify (for serverless execution and dataset storage). * **Language**: TypeScript (for type safety in complex configuration objects). ### Why Crawlee? I opted for [Crawlee](https://crawlee.dev/) because of its intelligent queue management. Unlike a simple recursive function, Crawlee handles the **RequestQueue** persistently. If the crawler crashes (or the server restarts) after 5,000 pages, the state is saved, and it resumes exactly where it left off. I specifically chose `CheerioCrawler` over `PlaywrightCrawler` or `PuppeteerCrawler`. * **CheerioCrawler**: Downloads raw HTML and parses it. It's lightning-fast and consumes minimal CPU. * **Browser-Based Crawlers**: Render JavaScript. While powerful, they are slower and costlier. For sitemaps, we rarely need to execute JS just to find `` tags, making Cheerio the efficient choice. ```typescript // Simplified view of the Crawler setup const crawler = new CheerioCrawler({ requestQueue, maxRequestsPerCrawl: input.maxPagesPerCrawl, requestHandler: async ({ request, $, log }) => { const title = $('title').text(); log.info(`Processing ${request.url}...`); // Extract links and add to queue if they match patterns await enqueueLinks({ globs: input.includePatterns, exclude: input.excludePatterns, }); // Store data for sitemap generation await Dataset.pushData({ url: request.url, title }); }, }); ``` ## The "Aha!" Moment: Solving the Regex Puzzle The biggest UX challenge was giving users control over *what* to crawl. A simple "match domain" rule isn't enough. Users needed to exclude `/admin` panels, `/login` pages, or dynamic tags like `/tag/*/feed`. I implemented a dual-layer filtering system using **Regex (Regular Expressions)** within the `enqueueLinks` strategy. The breakthrough came when I realized I could map the user's simple configuration directly to Craylee's `globs` and `exclude` options, but I had to sanitize them to prevent ReDoS (Regex Denial of Service) attacks. By strictly typing the input schema and validating regex patterns *before* the crawl starts, I ensured the actor fails fast with a helpful error message rather than hanging indefinitely on a bad pattern. ### Respecting Robots.txt One of the most complex features was respecting `robots.txt`. I integrated a parsing library that fetches the robots file at the *start* of the run, creates a rule set, and checks every potential URL against it before adding it to the queue. This ensures the generator is a "good citizen" of the web. ## Performance & Results Since its release, the Fast Sitemap Generator has achieved: * **100% Success Rate**: On all runs on the Apify platform. * **Speed**: Capable of crawling **1,000 pages in under 120 seconds** (network dependent). * **Efficiency**: Runs on the lowest tier of Apify compute units (0.1 CPU), making it extremely cost-effective. Users have reported using it to generate sitemaps for single-page applications (SPAs) and legacy PHP sites alike, bridging the gap between development and marketing teams. ## Technical Deep Dive: Generating the XML Writing the XML file itself posed a memory challenge. For a 50,000-page site, storing a massive array of URL objects in memory is risky. I implemented a **stream-based approach**. As the crawler progresses, it writes results to a Key-Value Store. Once the crawl finishes, a separate "Merge & Export" step iterates through the dataset and streams the lines into the final `sitemap.xml` file. ```xml https://example.com/ 2023-10-10 daily 1.0 ... ``` This decoupling of *crawling* and *generating* ensures that the process is scalable and fault-tolerant. ## Future Roadmap The tool is functional and stable, but I have ambitious plans for v2: 1. **Broken Link Reporter**: Since we are already visiting every page, why not report 404s? This turns the tool into a site health auditor. 2. **Visual Site Tree**: generating a JSON graph of the site structure to visualize internal linking silos. 3. **Proxy Configuration**: Adding support for residential proxies to crawl sites with strict firewalls (WAFs). ## Conclusion Building the Fast Sitemap Generator taught me that "simple" tools often require complex architectural decisions to be reliable at scale. By leveraging Node.js streams and Crawlee's robust queueing, I built a tool that saves developers hours of manual work. If you are looking to automate your technical SEO or need a reliable crawler for your next project, give it a spin. --- ### **Ready to Optimize Your SEO?** * **Try it Live**: [Run the Actor on Apify](https://apify.com/eunit/sitemap-generator) 14:["$","div",null,{"className":"animate-in fade-in duration-500","children":["$","$L41",null,{"projects":[{"id":"ed11ba8d-b9be-4241-91b0-372bbd4ea294","title":"Building a High-Performance Sitemap Generator with Node.js & Crawlee","slug":"high-performance-sitemap-generator-crawlee","description":"A scalable, customizable sitemap generator capable of crawling thousands of pages while respecting robots.txt and regex filters. Built to automate technical SEO tasks.","techStack":["Node.js","TypeScript","Crawlee","Apify SDK","XML","Cheerio"],"link":"","github":"","githubOwner":null,"githubRepo":null,"imageUrl":"https://evhjogolxocysfqkhzlz.supabase.co/storage/v1/object/public/media/gdcx0m4twoc.png","content":"$42","status":"published","isCaseStudy":true,"problemStatement":"Maintaining up-to-date sitemaps for large, dynamic websites is a manual nightmare, often missing deep pages or failing to automatically exclude non-indexable content.","solutionDescription":"An automated, configuration-driven crawler that generates compliant XML, HTML, and TXT sitemaps with precision control over depth, frequency, and regex-based exclusion patterns.","myRole":"Lead Full Stack Developer & Architect","lessonsLearned":"Mastered the complexities of web crawling politeness policies (robots.txt), optimizing BFS traversal for depth control, and handling large datasets in memory efficiently using streams.","githubStats":null,"category":"Tool"}],"title":"Projects built with Crawlee","description":"Found 1 projects involving Crawlee."}]}]