# How to Build an Automated SEO Pipeline: Connecting Sitemap Generator and Google Indexer to Notion Canonical: https://eunit.me/blog/build-automated-seo-pipeline-sitemap-generator-and-google-indexer-to-notion Authors: Emmanuel Uchenna Published: 2026-08-01T00:00:00+00:00 Category: Technical Build an automated SEO pipeline for instant content indexing. Stop waiting for Googlebot; use Apify's Fast Sitemap Generator & Google Indexer to push URLs to Google, track with Notion. Generating an XML sitemap is only the first step in modern technical Search Engine Optimization (SEO). If you have ever published a new blog post, launched a product line on an e-commerce platform, or updated critical landing pages, you know the frustration of waiting days, weeks, or even months for Google to crawl and index your content. Relying on search engine crawlers to discover changes organically is no longer viable for high-growth websites. If you run a dynamic content platform, an online store with rapidly changing inventory, or a news publication, delayed indexing directly equates to lost traffic, lower visibility, and missed revenue. In this guide, we will walk through the process of building an automated, enterprise-grade SEO pipeline. We will start by covering automated URL discovery using the [Fast Sitemap Generator](https://apify.com/eunit/sitemap-generator) [Apify Actor](https://www.apify.com?fpr=eunit). Then, we will look at programmatically pushing those URLs to Google using the [Google Indexer & Instant SEO Submitter](https://apify.com/eunit/google-indexing) Actor. Finally, we will solve the critical tracking challenge by using [Model Context Protocol (MCP)](https://blog.apify.com/what-is-model-context-protocol/) Connectors to stream real-time indexing logs directly into a [Notion](https://www.notion.so/) database. ![Build an Automated SEO Pipeline: Connecting Sitemap Generator and Google Indexer to Notion](https://i.postimg.cc/02kthKDQ/image.png) ## The evolution of programmatic indexing Historically, website owners managed SEO by submitting a static Extensible Markup Language (XML) sitemap through [Google Search Console](https://search.google.com/search-console). While essential, submitting a sitemap is fundamentally a passive notification. You are effectively dropping a note in Google's mailbox, asking Googlebot to visit your site whenever it gets around to it. For small, static websites, this passive approach works. However, for modern dynamic sites, it falls short due to several structural challenges: 1. **Crawl budget constraints**: Search engines assign a finite "crawl budget" to every domain. On large sites with thousands of pages, Googlebot may exhaust its budget on legacy pages, completely ignoring newly published or updated content. 2. **Orphaned or deep pages**: Content buried deep within subdirectories or pagination chains often fails to receive internal link equity, making it nearly invisible to organic crawling. 3. **Time-sensitive content**: E-commerce price updates, breaking news, real estate listings, and job postings require instant indexation. Waiting two weeks for organic discovery renders the content stale before it ever reaches search results. 4. **Desktop crawler overhead**: Traditional site auditors like Screaming Frog run locally, consuming heavy system resources, tying up developer machines, and requiring manual CSV exports to process updates. ### Moving from passive crawling to active push pipelines To solve these limitations, search engines introduced programmatic indexing APIs. Instead of waiting for Googlebot to discover changes, you can actively send a push notification to Google the instant a page is created or updated. By combining cloud-based crawling with active API submissions and real-time database logging, you can replace manual SEO workflows with a fully automated "set and forget" pipeline. ```mermaid graph TD A["Target Website / Domain"] -->|"1. Cloud Crawl"| B["Fast Sitemap Generator (Apify Actor)"] B -->|"2. Extracted Dataset ID"| C["Google Indexer & Instant SEO Submitter"] C -->|"3. Indexing API Push"| D["Google Search Index"] C -->|"4. Trigger on Completion"| E["Notion Uploader (Apify Actor)"] E -->|"5. Sync via MCP Connector"| F["Notion SEO Database"] ``` --- ## Stage 1: Automated URL discovery with Fast Sitemap Generator Before you can submit URLs to Google, you need an accurate, deduplicated list of active links from your website. Hand-coding XML files or relying on CMS plugins that crash on large databases creates maintenance bottlenecks. The [Fast Sitemap Generator](https://apify.com/eunit/sitemap-generator) Actor runs on the [Apify platform](https://apify.com) as a serverless cloud program. It uses a Direct Connection to crawl websites at high speeds directly from data centers, without requiring proxy infrastructure. ### Configuring the URL crawler Setting up automated URL discovery requires configuring key parameters inside the [Apify Console](https://console.apify.com/sign-up?fpr=eunit) to balance thoroughness with execution speed. Here is an example input configuration for a production crawl: ```json { "startUrls": [ { "url": "https://example.com" } ], "maxCrawlDepth": 3, "maxPagesPerCrawl": 5000, "includePatterns": [ ".*/blog/.*", ".*/products/.*" ], "excludePatterns": [ ".*/admin/.*", ".*/login.*", ".*/cart.*", ".*/checkout/.*" ], "sitemapFormats": [ "xml", "html", "txt" ], "respectRobotsTxt": true, "changefreq": "daily", "defaultPriority": 0.8, "includeImages": false } ``` ### Key configuration settings explained - **Start URLs (`startUrls`)**: The entry point for the crawler. Usually, this is your primary domain homepage. - **Max Crawl Depth (`maxCrawlDepth`)**: Controls how deep the crawler traverses link structures. A depth of `3` to `5` is typically sufficient to discover all essential content while preventing infinite loops. - **Regex Filtering (`includePatterns` and `excludePatterns`)**: Prevents cluttering your index with utility pages. You should explicitly exclude administrative paths like `.*/admin/.*`, checkout pages `.*/cart.*`, and query parameters that create duplicate content. - **Robots.txt Respect (`respectRobotsTxt`)**: Ensures your pipeline adheres to site governance rules and crawler restrictions. ### Output storage and Dataset generation When execution completes, the Actor stores output artifacts in two locations: 1. **Key-Value Store**: Contains compiled `sitemap.xml`, `sitemap.html`, and `sitemap.txt` files ready for static hosting. 2. **Apify Dataset**: Stores structured JSON records for every discovered page. The Dataset ID produced during this run serves as the input payload for the next stage in our pipeline. --- ## Stage 2: Programmatic submission via Google Cloud infrastructure With your site URLs discovered and structured into an Apify Dataset, the next phase is notifying Google. This step relies on the official Google Indexing API, which allows site owners to submit batch update requests directly to Google's indexing pipeline. ### Step 1: Setting up Google Cloud credentials Before making programmatic requests, you must configure authentication through the [Google Cloud Console](https://console.cloud.google.com/). ![Google Cloud Console](https://i.postimg.cc/g2k1vKkd/screenshot-2025-12-29-at-12-22-16.png) 1. **Create a Google Cloud Project**: Log in to Google Cloud Console, click the project dropdown, and select **New Project**. Name it `seo-automation-pipeline`. 2. **Enable the Indexing API**: Navigate to **APIs & Services > Library**, search for "Web Search Indexing API", and click **Enable**. 3. **Create a Service Account**: Go to **IAM & Admin > Service Accounts**, click **Create Service Account**, name it `indexing-bot`, and assign it the **Project > Editor** role. 4. **Generate a JSON Key**: Select your new service account, go to the **Keys** tab, click **Add Key > Create new key**, select **JSON**, and download the file. Keep this file secure, as it grants API permissions. ```txt Service Account Email Example: indexing-bot@seo-automation-pipeline.iam.gserviceaccount.com ``` ![Enable Indexing API](https://i.postimg.cc/tJsYZ6Z8/screenshot-2025-12-29-at-12-25-57.png) ### Step 2: Granting Google Search Console access Enabling the API in Google Cloud is not enough on its own. Google requires verified domain ownership to prevent unauthorized indexing requests. 1. Open [Google Search Console](https://search.google.com/search-console). 2. Select your verified domain property. 3. Navigate to **Settings > Users and permissions**. 4. Click **Add user**, paste your service account email address, set the role to **Owner**, and save. > IMPORTANT > Google Search Console requires **Owner** permissions for Service Accounts calling the Indexing API. Setting lower permission levels such as "Full" or "Restricted" will cause authentication errors during API calls. --- ### Bypassing DIY limitations: Python vs. Apify Actors Developers often attempt to write custom scripts to interact with the Google Indexing API. Below is a standard Python implementation using official Google authentication libraries: ```python import json import requests from google.oauth2 import service_account from google.auth.transport.requests import AuthorizedSession SERVICE_ACCOUNT_FILE = "google-key.json" INDEXING_API_ENDPOINT = "https://indexing.googleapis.com/v3/urlNotifications:publish" def get_authenticated_session(key_path): scopes = ["https://www.googleapis.com/auth/indexing"] credentials = service_account.Credentials.from_service_account_file( key_path, scopes=scopes )     return AuthorizedSession(credentials) def submit_url(session, url, action_type="URL_UPDATED"): payload = {         "url": url,         "type": action_type } response = session.post(INDEXING_API_ENDPOINT, json=payload)     return response.status_code, response.json() if __name__ == "__main__": session = get_authenticated_session(SERVICE_ACCOUNT_FILE) target_url = "https://example.com/blog/automated-seo-pipeline" status, result = submit_url(session, target_url)     print(f"Status Code: {status}")     print(f"Response: {json.dumps(result, indent=2)}") ``` ### Why custom scripts break at scale While the DIY Python script works for single URLs, operating it in production exposes major infrastructure hurdles: - **Rate limit handling**: Google enforces daily quotas (typically 200 URL notifications per day for standard projects). Custom scripts will fail with `HTTP 429 Too Many Requests` unless you write exponential backoff algorithms. - **State management**: A basic script does not remember which URLs were submitted yesterday. Running it daily resubmits duplicate URLs, wasting your daily quota. - **Execution server maintenance**: You must deploy the script to a cloud server, manage cron schedules, and store credentials securely. ### The solution: Google Indexer & Instant SEO Submitter Instead of writing and hosting custom boilerplate code, we use the [Google Indexer & Instant SEO Submitter](https://apify.com/eunit/google-indexing) Actor. ![Google Indexer & Instant SEO Submitter](https://i.postimg.cc/rwzd266m/https-apify-com-eunit-google-indexing.png) | Feature | Custom Python Script | Google Indexer Actor | | --- | --- | --- | | **Quotas & Rate Limiting** | Manual handling required | Built-in automatic rate limiting & backoff | | **Input Source** | Hardcoded or manual files | Direct integration with Apify Datasets & Sitemaps | | **Pricing Model** | Server hosting costs | Pay-Per-Event | | **Test Mode** | Requires code flags | Built-in dry-run toggle | | **Monitoring** | Custom logging setup | Real-time console logs & dataset outputs | To configure the Actor, pass the JSON credentials and your Dataset ID from Stage 1 directly into the input schema: ```json { "datasetId": "YOUR_STAGE_1_DATASET_ID", "serviceAccountJson": { "type": "service_account", "project_id": "seo-automation-pipeline", "private_key_id": "abcdef123456789", "private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQC...\n-----END PRIVATE KEY-----\n", "client_email": "indexing-bot@seo-automation-pipeline.iam.gserviceaccount.com" }, "action": "URL_UPDATED", "testMode": false } ``` --- ### Stage 3: The missing link: Logging to Notion via MCP Connectors Automating crawls and API submissions solves the execution side of SEO. However, engineering teams and SEO leads still face a visibility gap: **How do you monitor pipeline results without manually checking console logs after each run?** Exporting CSVs after every run is tedious, and setting up custom webhooks requires maintaining dedicated receiver endpoints. This is where [MCP connectors](https://docs.apify.com/integrations/mcp-connectors) fit into the pipeline. ### Understanding MCP connectors [The Model Context Protocol (MCP)](https://blog.apify.com/what-is-model-context-protocol/) is an open standard that lets Actors call third-party services like Notion, Slack, and GitHub on your behalf, using your own credentials, without the Actor ever holding your token directly. MCP connectors on Apify work differently from the Apify MCP server. The [Apify MCP server](https://mcp.apify.com) exposes Apify Actors as tools to external AI clients (such as Claude or Cursor). MCP connectors do the opposite: they let Actors running on the Apify platform call external MCP-compatible services. The two are independent and can be used together. In this pipeline, we use the [Notion Uploader](https://apify.com/filip_cicvarek/notion-uploader) Actor together with an Apify MCP connector to push the Google Indexer's results into a Notion database automatically after each run. ```mermaid sequenceDiagram     participant Indexer as Google Indexer Actor     participant Google as Google Indexing API     participant Uploader as Notion Uploader Actor     participant MCP as Apify MCP Proxy     participant Notion as Notion Database Indexer->>Google: POST /urlNotifications:publish (Target URL) Google-->>Indexer: 200 OK (Submission Received)     Note over Indexer: Run completes, dataset saved Uploader->>MCP: notion_create_page(row data) MCP->>Notion: Create row (URL, Status, HTTP Code, Timestamp) Notion-->>MCP: Page Created MCP-->>Uploader: Confirmation Success ``` ### Setting up the Notion database Create a dedicated database in your Notion workspace titled **SEO Indexing Tracker**. Configure the database columns with the following properties: | Column Name | Property Type | Description | | --- | --- | --- | | **URL** | Title | The target webpage URL submitted to Google | | **Submission Status** | Select | `Indexed`, `Failed`, or `Quota Exceeded` | | **HTTP Code** | Number | The status response code returned by Google (e.g., `200`, `429`, `403`) | | **Last Updated** | Date | Timestamp of when the request occurred | | **Source Actor** | Text | Name of the Apify Actor that executed the request | ### Authorizing the Notion MCP connector MCP connectors are authorized once at the account level, not on individual Actor or task pages. To set one up: 1. Open [Apify Console](https://console.apify.com/sign-up?fpr=eunit) and go to **Settings → API & Integrations → MCP connectors**. 2. Click **Add connector** and select **Notion**. 3. Complete the OAuth flow — you will be redirected to Notion to authorize access to your workspace. The Apify platform never sees your token directly; authentication is handled server-side via the MCP proxy. 4. Once authorized, the connector appears as a selectable option when running any compatible Actor. ![Notion MCP Connector](https://i.postimg.cc/mkGfWZRw/Notion-MCP-Connector.png) ### Connecting the Notion Uploader Actor The [Notion Uploader](https://apify.com/filip_cicvarek/notion-uploader) Actor reads any Apify dataset and uploads its rows into Notion. It accepts your MCP connector ID so you do not need to paste a Notion API key anywhere. When the Notion Uploader runs as an Actor-to-Actor integration (triggered by the completion of the Google Indexer run), it automatically picks up the triggering run's output dataset — no explicit `datasetId` is needed in the payload template. Here is an example input for the Notion Uploader when configured as a standalone run: ```json { "notionConnector": "", "notionDatabaseId": "https://www.notion.so/workspace/SEO-Indexing-Tracker-", "datasetId": "", "dedupeMode": "upsert", "keyProperty": "URL", "dryRun": false } ``` > **Tip**: Enable `"dryRun": true` on your first run. It resolves the field mapping, validates it against your Notion schema, and shows exactly what would be written — without creating any rows. --- ## Orchestrating the full "set and forget" pipeline Now that all three Actors are configured, we can chain them into an automated workflow: [Fast Sitemap Generator](https://apify.com/eunit/sitemap-generator) discovers URLs, [Google Indexer & Instant SEO Submitter](https://apify.com/eunit/google-indexing) submits them to Google, and [Notion Uploader](https://apify.com/filip_cicvarek/notion-uploader) logs the results into Notion. ### Step 1: Creating saved Actor tasks An **Actor task** is a saved, reusable configuration of an Actor in [Apify Console](https://console.apify.com/sign-up?fpr=eunit). Tasks let you pre-define all input parameters so they can be run on demand, via API, or on a schedule. 1. **Crawler task**: Open [Fast Sitemap Generator](https://apify.com/eunit/sitemap-generator) in Apify Console, configure your `startUrls`, crawl depth, and regex filters, then click **Save as a new task** in the top-right corner. Name it `sitemap-generator-task`. 2. **Indexer task**: Open [Google Indexer & Instant SEO Submitter](https://apify.com/eunit/google-indexing), fill in your Google Cloud Service Account JSON key, then click **Save as a new task**. Name it `google-indexer-task`. 3. **Uploader task**: Open [Notion Uploader](https://apify.com/filip_cicvarek/notion-uploader), set your Notion MCP connector and target database URL, then click **Save as a new task**. Name it `notion-uploader-task`. All saved tasks are listed under **Saved tasks** in the Apify Console left navigation. ### Step 2: Chaining tasks using Actor-to-Actor integrations The **Integrations** tab on each Actor or task page lets you chain runs together. When one task succeeds, Apify automatically triggers the next one and passes dynamic run data — such as the output dataset ID — via a payload template. ### Chain 1: Sitemap Generator → Google Indexer 1. Open `sitemap-generator-task` in Apify Console and click the **Integrations** tab. 2. Click **Add integration** and select **Run another Actor or task**. 3. Choose `google-indexer-task` as the target. 4. Set the trigger to **Run succeeded**. 5. Set the payload template so the indexer receives the crawler's output dataset: ```json { "datasetId": "{{resource.defaultDatasetId}}", "action": "URL_UPDATED" } ``` ### Chain 2: Google Indexer → Notion Uploader 1. Open `google-indexer-task` in Apify Console and click the **Integrations** tab. 2. Click **Add integration** and select **Run another Actor or task**. 3. Choose `notion-uploader-task` as the target. 4. Set the trigger to **Run succeeded**. 5. Leave `datasetId` out of the payload template — the Notion Uploader automatically uses the triggering run's dataset when no explicit ID is provided: ```json { "dedupeMode": "upsert", "keyProperty": "URL" } ``` Now the full chain fires end-to-end on its own. When `sitemap-generator-task` completes, Apify triggers `google-indexer-task` with the crawler's dataset. When the indexer finishes, Apify triggers `notion-uploader-task`, which reads the indexer's output dataset and writes each result row into your Notion database. ### Step 3: Scheduling automated pipeline execution Configure a recurring schedule so the entire chain runs automatically. 1. In the Apify Console left sidebar, click **Schedules**. 2. Click **Create new schedule**. 3. Select `sitemap-generator-task` as the target. The full chain — crawl, index, upload to Notion — fires automatically when this task runs. 4. Set the frequency using a Cron expression or one of the presets. - **For high-frequency publishing (news, job boards)**: Schedule daily at 2:00 AM UTC. - **For standard marketing blogs and commercial sites**: Schedule weekly, every Monday morning. ```txt Cron expression example (daily at 2:00 AM UTC): 0 2 * * * ``` --- ## Best practices for automated SEO indexing To maintain a healthy indexing strategy while avoiding Google API flags, keep these best practices in mind: ### 1. Submit only modified or new content Do not resubmit your entire site inventory every day. The Google Indexing API is intended for pages that have new content (`URL_UPDATED`) or pages that have been deleted (`URL_DELETED`). Resubmitting unchanged pages wastes daily quotas and can result in rate limiting. ### 2. Match canonical URLs strictly Google requires strict URL matching. If your canonical structure uses `https://example.com/blog/post-name/` with a trailing slash, ensure your crawler filters and API submissions match that format exactly. Submitting `http://` or non-trailing-slash variations can trigger redirect loops and indexing failures. ### 3. Use Test Mode during initial setup Before running full-scale API operations, turn on the **Test Mode** toggle in the [Google Indexer & Instant SEO Submitter](https://apify.com/eunit/google-indexing) Actor. Test Mode executes the full crawl, verifies JSON authentication keys, and simulates Notion MCP logging without consuming your daily Google API quota. ### 4. Monitor quota usage in Google Cloud Console Standard Google Cloud projects receive an initial quota of 200 Indexing API requests per day. If your site publishes hundreds of new pages daily, submit a quota increase request directly through the **Quotas** tab in Google Cloud Console. --- ## Wrapping up Building an automated technical SEO pipeline transforms indexing from a passive waiting game into a predictable, programmatic system. Combining the cloud crawling capabilities of the [Fast Sitemap Generator](https://apify.com/eunit/sitemap-generator), the push notifications of the [Google Indexer & Instant SEO Submitter](https://apify.com/eunit/google-indexing), and real-time database logging through [Notion MCP Connectors](https://apify.com/store/collections/mcp-connectors), you establish an automated SEO engine that operates reliably in the background. Instead of waiting weeks for Google to discover your newest content organically, your site updates are crawled, verified, submitted, and logged in Notion within minutes. ### Read more - Explore the [Fast Sitemap Generator](https://apify.com/eunit/sitemap-generator) on the Apify Store. - Set up the [Google Indexer & Instant SEO Submitter](https://apify.com/eunit/google-indexing) Actor with your Google Cloud Service Account. - Connect your Notion workspace via MCP Connectors to start tracking real-time indexing status today.