Custom HTTP Connector

Connect a knowledge base to any REST API or HTTP-accessible data source. The Custom HTTP Connector connects a knowledge base to any REST API or HTTP-accessible data source and ingests its content for search and retrieval. Use it when no pre-built connector exists for your source system. The connector supports:

• Full sync and delta sync (incremental updates)
• Multiple document objects from one connector
• Permission-aware indexing (ACL crawling)
• Custom scripting to parse non-standard responses

​

How It Works

1. The connector builds a pre-authenticated HTTP client from the attached auth profile.
2. It paginates the list endpoint and collects item identifiers.
3. It calls the content endpoint once per item to retrieve the document body.
4. It chunks, embeds, and indexes each document with its metadata and ACL permissions.

​

Setting Up the Connector

Configuring the Custom HTTP Connector involves six steps: selecting a mode, connecting to your API, defining listing behavior, mapping content and document structure, mapping fields, and reviewing the configuration.

​

Step 1: Mode

Select the mode that matches your API structure. This determines how the connector fetches content.

​

Step 2: Connect

Set the API base URL and configure authentication. These settings apply to every request the connector makes for listing, fetching, and connection testing. Testing the connection: Click Test Connection to verify that the connector can reach your API using the configured Base URL and auth profile. A successful test confirms connectivity only — it doesn’t validate the listing or content endpoints configured in later steps.

​

Step 3: Listing

Define one or more object types to sync from your API. Each object syncs independently and is indexed as a separate searchable document type.

​

Objects

You can configure multiple object types from the same API. Each one appears as a separate tab in the Listing step. Click Add object to configure another. For each object, configure the following:

• Object name: A URL-safe identifier for the object type. Letters, numbers, hyphens (-), and underscores (_) are allowed — for example, articles or posts.
• Listing Mode: The mode selected in Step 1, shown here as read-only. To use a different mode, return to Step 1.

Listing API request: Configure the endpoint used to retrieve the list of records. Click Send to test the request and inspect the response in the preview panel. The object: Defines how a record from the listing response maps to a searchable document.

• Items array dot-path: The dot-path to the array in the API response that contains the individual records. For example, if the response is:
  {
    "data": {
      "results": {
        "items": []
      }
    }
  }
  
  enter data.results.items. Leave this field blank if the response root is the array itself. You can click the array in the response preview to populate the path automatically, or enter it manually.
• Attributes: The fields to extract from each record. Click Add attribute to map fields from the response.
• Pagination: Enable this if the listing endpoint returns results across multiple pages. When enabled, values from the response are stored as ctx variables and referenced in subsequent requests. For each pagination field, specify:
  • Field name
  • Source path in the response (for example, meta.next_cursor)
  • Default value (optional), used for the first request
  Additional setting:
  • Stop paging when: Controls when pagination stops. Defaults to Auto (no more results), which stops when the API returns no new records.
• Max pages per sync: Limits the number of pages processed during a sync. Defaults to 100000. Enable No limit to process all pages.

Sync and freshness: Select how the connector detects changes and performs delta synchronization. Enable scheduled delta: Available only when a delta synchronization mode is selected. When enabled, the connector runs delta syncs automatically at the interval specified in the Interval field (in minutes).

​

Step 4: APIs

Configure the per-item content request — the API call the connector makes to fetch the full content of each record discovered during the Listing step — and define how the response maps to a searchable document. Each object configured in Step 3 appears as its own tab.

​

Content API

The endpoint the connector calls once per item to retrieve its full content. Click Send to execute the request with the test values configured below, then inspect the response in the preview panel. Testing with a sample document: Set test values for any document field variables used in the URL. Click Load sample from List to populate the variables automatically using a real item from the listing response.

​

The Document

Controls how the connector treats each fetched item and which fields it extracts. How should we treat each item? Choose how the connector handles items returned by the Content API. Object attributes: The attributes mapped in the Listing step. These are available as variables in the content request URL and carry through to the final document. To add or change attributes, return to the Listing step, or click a field in the response preview and select Add as attribute. Document body: The dot-path within the content response that holds the main document content — for example, data.body or content.html. Leave empty to use the entire response as the document body. Content type: Defines how the document body should be interpreted. Defaults to Auto-detect from response.

​

Step 5: Fields

Map the attributes defined in the Listing and Content steps to your knowledge base’s canonical fields. Mapped fields become filterable metadata. You can adjust these mappings after the first sync.

​

Step 6: Review

Review the configuration and save it. Click Start Sync to begin ingestion.

​

Connector Overview

Once configured, the connector’s overview page shows sync status, a configuration summary, indexed objects, and recent activity. Sync: Click Sync now to trigger a sync manually.

• Full sync: Fetches all records from the API and re-indexes them, regardless of what was previously indexed.
• Delta sync: Fetches only new, changed, or deleted records since the last sync. Available only when a delta mode is configured in the Listing step.

Reconfigure: Return to the setup wizard to modify any connector setting. Health: Shows the result of the most recent sync, including:

• Status (Completed or Failed)
• Sync type (Full or Delta)
• Number of documents processed
• Number of failed documents
• Timestamp

Configuration: A summary of the connector setup, including strategy, mode, and routes. Objects: Each object type configured in the connector, with its sync state, document count, last sync time, and any failures. Click View documents to inspect the indexed documents for an object. Recent Activity: The latest sync run’s status, including the number of documents processed, failed, and skipped.