Web Crawl API - Kore.ai Docs

Back to API List The Web Crawl APIs enable you to create and manage website sources, configure sitemap-based crawling, and trigger crawl operations programmatically.

Create a Web Source

Creates a new website source for crawling.

Field	Value
Method	POST
Endpoint	`<host_url>/api/public/bot/:appId/sources`
Content-Type	`application/json`
Authorization	`auth: <JWT Token>`
API Scope	Data Ingestion

Query Parameters

Parameter	Description	Mandatory
appId	Provide your application ID here.	Yes

Request Parameters

Field	Type	Required	Description
`name`	String	Yes	Name of the web source.
`desc`	String	No	Description of the web source.
`sourceType`	String	Yes	Source type. Supported values: `url`, `uploadSitemap`.
`url`	String	Required for `url`	Website URL to crawl.
`advanceOpts`	Object	No	Advanced crawl settings.
`advanceOpts.maxUrlLimit`	Integer	No	Maximum number of URLs to crawl.
`advanceOpts.crawlDepth`	Integer	No	Maximum crawl depth.
`authorizationEnabled`	Boolean	No	Enables authorization for the source.

Sample Response

{
    "_id": "fs-52d84393-769d-5707-b413-a60726a8b385",
    "jobId": "fj-23fbd6e1-db5f-5a96-b1ff-acfdcd3791b3",
    "extractionType": "web",
    "contentSource": "web",
    "autoGen": false,
    "recentStatus": "configured",
    "name": "Example source",
    "desc": "Optional source description",
    "searchIndexId": "sidx-403e99ff-4e42-5980-b4ce-dd69db50a5d9",
    "createdBy": "u-0555dabf-d7a4-57c4-ae45-8995f25710f9",
    "lModBy": "u-0555dabf-d7a4-57c4-ae45-8995f25710f9",
    "createdOn": "2026-08-31T11:12:05.483Z",
    "lMod": "2026-08-31T11:12:05.000Z",
    "display": true,
    "rAcl": [
        "*"
    ],
    "numOfDocs": 0,
    "totalSize": 0,
    "url": "https://example.com",
    "settings": {
        "blockHttpsMsgs": false,
        "isJavaScriptRendered": false,
        "crawlBeyondSitemaps": false,
        "respectRobotTxtDirectives": true,
        "useCookies": true,
        "crawlDepth": 5,
        "maxUrlLimit": 10
    },
    "advanceSettings": {
        "scheduleOpt": false,
        "repeatInterval": "",
        "crawlEverything": true,
        "allowedOpt": false,
        "allowedURLs": [],
        "blockedOpt": false,
        "blockedURLs": [],
        "blockHttpsMsgs": false,
        "isJavaScriptRendered": false,
        "crawlBeyondSitemaps": false,
        "respectRobotTxtDirectives": true,
        "useCookies": true,
        "crawlDepth": 1,
        "maxUrlLimit": 5,
        "crawlDelay": 0,
        "processingOpts": true
    },
    "sitemaps": [],
    "validations": {
        "url": {
            "validated": true,
            "msg": "Initial URL validation successful",
            "description": ""
        },
        "networkConnectivity": {
            "validated": true,
            "msg": "Network Connectivity test successful",
            "description": ""
        }
    },
    "authorizationEnabled": false,
    "authorizationProfileId": null,
    "isURLValid": true,
    "customCrawler": {
        "enabled": false,
        "authDetails": {
            "reqHeaders": []
        },
        "useProfile": false
    },
    "filesMetaData": [],
    "sourceType": "url",
    "isDeleted": false,
    "__v": 0
}

Configure an uploaded sitemap for web crawl

To configure a web source using a sitemap, complete the following steps. Step 1: Upload the Sitemap Upload the sitemap file using the public upload API. Use multipart/form-data with the following fields:

Field	Description
`file`	Sitemap file to upload.
`fileContext`	Set to `findly`.
`fileExtension`	Extension of the uploaded sitemap file.

Save the returned fileId from the response. Step 2: Attach the Sitemap Update the web source with the uploaded sitemap using the Create Web Crawl endpoint. Set the sourceType as uploadSitemap, as shown below.

{
  "sourceType": "uploadSitemap",
  "fileId": "<uploaded-file-id>"
}

Initiate a web crawl

This API triggers web crawl.

Field	Value
Method	POST
Endpoint	`<host_url>/api/public/bot/:appId/sources/:sourceId/crawl`
Content-Type	`application/json`
Authorization	`auth: <JWT Token>`
API Scope	Data Ingestion

Query Parameters

Parameter	Description	Mandatory
appId	Provide your application ID here.	Yes
sourceId	Provide the unique identifier for the web source.	Yes

Request Parameters

Field	Type	Required	Description
`forceRecrawl`	Boolean	Yes	When forceRecrawl is true, the API triggers the existing recrawl flow. Otherwise it triggers the existing extract flow.

{
  "forceRecrawl": false
}

​Create a Web Source

​Query Parameters

​Request Parameters

​Sample Response

​Configure an uploaded sitemap for web crawl

​Initiate a web crawl

​Query Parameters

​Request Parameters

Create a Web Source

Query Parameters

Request Parameters

Sample Response

Configure an uploaded sitemap for web crawl

Initiate a web crawl

Query Parameters

Request Parameters