Custom HTTP Connector

Connect a knowledge base to any REST API or HTTP-accessible data source. The Custom HTTP Connector connects a knowledge base to any REST API or HTTP-accessible data source and ingests its content for search and retrieval. Use it when no pre-built connector exists for your source system. The connector supports:

Full sync and delta sync (incremental updates)
Multiple document objects from one connector
Permission-aware indexing (ACL crawling)
Custom scripting to parse non-standard responses

How It Works

The connector builds a pre-authenticated HTTP client from the attached auth profile.
It paginates the list endpoint and collects item identifiers.
It calls the content endpoint once per item to retrieve the document body.
It chunks, embeds, and indexes each document with its metadata and ACL permissions.

Setting Up the Connector

Configuring the Custom HTTP Connector involves six steps: selecting a mode, connecting to your API, defining listing behavior, mapping content and document structure, mapping fields, and reviewing the configuration.

Step 1: Mode

Select the mode that matches your API structure. This determines how the connector fetches content.

Mode	Use when	How it works
Most APIs	Your API has separate list and detail endpoints (for example, `GET /articles` followed by `GET /articles/{id}`).	The connector calls the list endpoint to discover new or updated items, fetches each item individually from the content endpoint, then maps, chunks, embeds, and indexes the retrieved content.
A Single API	One paginated endpoint returns the complete content for each item, with no separate content request required.	The connector pages through the API response and maps each item directly to a document. The per-item content fetch stage is skipped.
A Set of IDs	You have a predefined list of record IDs to index.	You specify the IDs in the connector configuration. The connector fetches and indexes each record individually.

Step 2: Connect

Set the API base URL and configure authentication. These settings apply to every request the connector makes for listing, fetching, and connection testing.

Field	Description
Base URL	The root URL for your API. Endpoint paths configured in later steps are appended to this URL. Enter the full URL, including the protocol — for example, `https://jsonplaceholder.typicode.com/`.
Auth profile	The authentication profile the connector uses to call your API. Defaults to No auth for public APIs. Select a saved profile from the dropdown if your API requires authentication.
mTLS client certificate (optional)	A client certificate for mutual TLS authentication, required if your API expects the connector to present a certificate during the TLS handshake. Defaults to None.
Connection test path (optional)	A path appended to the Base URL, used only to verify connectivity. Defaults to `/`. If the root URL returns an error, point this to a real endpoint, such as `/health` or `/v1/items`. This setting affects only the connectivity test, not connector behavior.

Testing the connection: Click Test Connection to verify that the connector can reach your API using the configured Base URL and auth profile. A successful test confirms connectivity only — it doesn’t validate the listing or content endpoints configured in later steps.

Step 3: Listing

Define one or more object types to sync from your API. Each object syncs independently and is indexed as a separate searchable document type.

Objects

You can configure multiple object types from the same API. Each one appears as a separate tab in the Listing step. Click Add object to configure another. For each object, configure the following:

Object name: A URL-safe identifier for the object type. Letters, numbers, hyphens (-), and underscores (_) are allowed — for example, articles or posts.
Listing Mode: The mode selected in Step 1, shown here as read-only. To use a different mode, return to Step 1.

Listing API request: Configure the endpoint used to retrieve the list of records.

Field	Description
Method	The HTTP method for the request.
URL	The endpoint path appended to the Base URL from Step 2 — for example, `/items`.
Params	Query parameters to include in the request. Click Add param to add a key-value pair. Parameters can reference variables using the variable chip selector.
Headers	Request headers required by the listing endpoint.

Click Send to test the request and inspect the response in the preview panel. The object: Defines how a record from the listing response maps to a searchable document.

Items array dot-path: The dot-path to the array in the API response that contains the individual records. For example, if the response is:
{ "data": { "results": { "items": [] } } }
enter data.results.items. Leave this field blank if the response root is the array itself. You can click the array in the response preview to populate the path automatically, or enter it manually.
Attributes: The fields to extract from each record. Click Add attribute to map fields from the response.
Pagination: Enable this if the listing endpoint returns results across multiple pages. When enabled, values from the response are stored as ctx variables and referenced in subsequent requests. For each pagination field, specify:
- Field name
- Source path in the response (for example, meta.next_cursor)
- Default value (optional), used for the first request
Additional setting:
- Stop paging when: Controls when pagination stops. Defaults to Auto (no more results), which stops when the API returns no new records.
Max pages per sync: Limits the number of pages processed during a sync. Defaults to 100000. Enable No limit to process all pages.

Sync and freshness: Select how the connector detects changes and performs delta synchronization.

Option	Description
Read everything every run	Fetches all records on every sync and compares them locally to detect changes. Disables scheduled delta sync.
API filters by changed time	Uses a date or time filter to retrieve only records modified since the previous sync.
API returns a change cursor	Uses a change cursor returned by the API to retrieve only new or updated records.
API sorts the newest changes first	Retrieves records in descending modification order and stops once it reaches older records.
List all and compare the modified field	Fetches the full list on each run and compares each record’s modified field to determine changes.
Compare fetched content hash	Computes a content hash for each record and reindexes only records whose content has changed.

Enable scheduled delta: Available only when a delta synchronization mode is selected. When enabled, the connector runs delta syncs automatically at the interval specified in the Interval field (in minutes).

Step 4: APIs

Configure the per-item content request — the API call the connector makes to fetch the full content of each record discovered during the Listing step — and define how the response maps to a searchable document. Each object configured in Step 3 appears as its own tab.

Content API

The endpoint the connector calls once per item to retrieve its full content.

Field	Description
Method	The HTTP method for the request.
URL	The endpoint path appended to the Base URL — for example, `/posts/{id}`. Use document field variables in curly braces to inject values from the list response, for example, `{id}` inserts the `id` attribute of each item. Click a variable chip to insert a field reference into the URL at the cursor position.
Params	Query parameters to include in the content request. Click Add param to add a key-value pair, or click a variable chip to insert a reference. No parameters are added by default.
Headers	Request headers required by the content endpoint.

Click Send to execute the request with the test values configured below, then inspect the response in the preview panel. Testing with a sample document: Set test values for any document field variables used in the URL. Click Load sample from List to populate the variables automatically using a real item from the listing response.

The Document

Controls how the connector treats each fetched item and which fields it extracts. How should we treat each item? Choose how the connector handles items returned by the Content API.

Option	Description
Document	Each list item becomes one document. The content request enriches it: attributes from the list response are read into the request, and fields are then picked from the content response. Use this for articles, pages, posts, or any item whose full content lives in a single detail endpoint.
Document + fields	Each list item becomes one document and also exposes individual fields for selective extraction. Use this when you need both the full document body and specific structured fields — such as title, author, or date — indexed as separate facets.
Record	Each list item is treated as a structured record rather than a content document. Fields are extracted and indexed individually, with no document body. Use this for structured data such as product catalogs, user records, or datasets with no long-form content to index.

Object attributes: The attributes mapped in the Listing step. These are available as variables in the content request URL and carry through to the final document. To add or change attributes, return to the Listing step, or click a field in the response preview and select Add as attribute. Document body: The dot-path within the content response that holds the main document content — for example, data.body or content.html. Leave empty to use the entire response as the document body. Content type: Defines how the document body should be interpreted. Defaults to Auto-detect from response.

Option	Description
Auto-detect from response	Infers the content type from the response headers and body. Use this when the API sets the correct `Content-Type` header. Default.
Plain text	Treats the document body as unformatted plain text. Use this when the content contains no markup or formatting.
Markdown	Treats the document body as Markdown and parses the formatting before indexing.
HTML	Treats the document body as HTML, stripping tags and extracting readable content before indexing.
JSON	Treats the document body as a JSON string to parse, rather than indexing it as a literal string.

Step 5: Fields

Map the attributes defined in the Listing and Content steps to your knowledge base’s canonical fields. Mapped fields become filterable metadata. You can adjust these mappings after the first sync.

Step 6: Review

Review the configuration and save it. Click Start Sync to begin ingestion.

Connector Overview

Once configured, the connector’s overview page shows sync status, a configuration summary, indexed objects, and recent activity. Sync: Click Sync now to trigger a sync manually.

Full sync: Fetches all records from the API and re-indexes them, regardless of what was previously indexed.
Delta sync: Fetches only new, changed, or deleted records since the last sync. Available only when a delta mode is configured in the Listing step.

Reconfigure: Return to the setup wizard to modify any connector setting. Health: Shows the result of the most recent sync, including:

Status (Completed or Failed)
Sync type (Full or Delta)
Number of documents processed
Number of failed documents
Timestamp

Configuration: A summary of the connector setup, including strategy, mode, and routes. Objects: Each object type configured in the connector, with its sync state, document count, last sync time, and any failures. Click View documents to inspect the indexed documents for an object. Recent Activity: The latest sync run’s status, including the number of documents processed, failed, and skipped.

Get Started

Build

Test and Evaluate

Deploy

Analytics and Insights

Administration

References

How It Works

Setting Up the Connector

Step 1: Mode

Step 2: Connect

Step 3: Listing

Objects

Step 4: APIs

Content API

The Document

Step 5: Fields

Step 6: Review

Connector Overview

​How It Works

​Setting Up the Connector

​Step 1: Mode

​Step 2: Connect

​Step 3: Listing

​Objects

​Step 4: APIs

​Content API

​The Document

​Step 5: Fields

​Step 6: Review

​Connector Overview

How It Works

Setting Up the Connector

Step 1: Mode

Step 2: Connect

Step 3: Listing

Objects

Step 4: APIs

Content API

The Document

Step 5: Fields

Step 6: Review

Connector Overview