> ## Documentation Index
> Fetch the complete documentation index at: https://koreai.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Web Crawl API

<Badge icon="arrow-left" color="gray">[Back to API List](/ai-for-service/apis/searchai/api-list)</Badge>

The Web Crawl APIs enable you to create and manage website sources, configure sitemap-based crawling, and trigger crawl operations programmatically.

## Create a Web Source

Creates a new website source for crawling.

| Field             | Value                                      |
| ----------------- | ------------------------------------------ |
| **Method**        | POST                                       |
| **Endpoint**      | `<host_url>/api/public/bot/:appId/sources` |
| **Content-Type**  | `application/json`                         |
| **Authorization** | `auth: <JWT Token>`                        |
| **API Scope**     | Data Ingestion                             |

### Query Parameters

| Parameter | Description                       | Mandatory |
| --------- | --------------------------------- | --------- |
| appId     | Provide your application ID here. | Yes       |

### Request Parameters

| Field                     | Type    | Required           | Description                                            |
| ------------------------- | ------- | ------------------ | ------------------------------------------------------ |
| `name`                    | String  | Yes                | Name of the web source.                                |
| `desc`                    | String  | No                 | Description of the web source.                         |
| `sourceType`              | String  | Yes                | Source type. Supported values: `url`, `uploadSitemap`. |
| `url`                     | String  | Required for `url` | Website URL to crawl.                                  |
| `advanceOpts`             | Object  | No                 | Advanced crawl settings.                               |
| `advanceOpts.maxUrlLimit` | Integer | No                 | Maximum number of URLs to crawl.                       |
| `advanceOpts.crawlDepth`  | Integer | No                 | Maximum crawl depth.                                   |
| `authorizationEnabled`    | Boolean | No                 | Enables authorization for the source.                  |

### Sample Response

```json expandable=true theme={null}
{
    "_id": "fs-52d84393-769d-5707-b413-a60726a8b385",
    "jobId": "fj-23fbd6e1-db5f-5a96-b1ff-acfdcd3791b3",
    "extractionType": "web",
    "contentSource": "web",
    "autoGen": false,
    "recentStatus": "configured",
    "name": "Example source",
    "desc": "Optional source description",
    "searchIndexId": "sidx-403e99ff-4e42-5980-b4ce-dd69db50a5d9",
    "createdBy": "u-0555dabf-d7a4-57c4-ae45-8995f25710f9",
    "lModBy": "u-0555dabf-d7a4-57c4-ae45-8995f25710f9",
    "createdOn": "2026-08-31T11:12:05.483Z",
    "lMod": "2026-08-31T11:12:05.000Z",
    "display": true,
    "rAcl": [
        "*"
    ],
    "numOfDocs": 0,
    "totalSize": 0,
    "url": "https://example.com",
    "settings": {
        "blockHttpsMsgs": false,
        "isJavaScriptRendered": false,
        "crawlBeyondSitemaps": false,
        "respectRobotTxtDirectives": true,
        "useCookies": true,
        "crawlDepth": 5,
        "maxUrlLimit": 10
    },
    "advanceSettings": {
        "scheduleOpt": false,
        "repeatInterval": "",
        "crawlEverything": true,
        "allowedOpt": false,
        "allowedURLs": [],
        "blockedOpt": false,
        "blockedURLs": [],
        "blockHttpsMsgs": false,
        "isJavaScriptRendered": false,
        "crawlBeyondSitemaps": false,
        "respectRobotTxtDirectives": true,
        "useCookies": true,
        "crawlDepth": 1,
        "maxUrlLimit": 5,
        "crawlDelay": 0,
        "processingOpts": true
    },
    "sitemaps": [],
    "validations": {
        "url": {
            "validated": true,
            "msg": "Initial URL validation successful",
            "description": ""
        },
        "networkConnectivity": {
            "validated": true,
            "msg": "Network Connectivity test successful",
            "description": ""
        }
    },
    "authorizationEnabled": false,
    "authorizationProfileId": null,
    "isURLValid": true,
    "customCrawler": {
        "enabled": false,
        "authDetails": {
            "reqHeaders": []
        },
        "useProfile": false
    },
    "filesMetaData": [],
    "sourceType": "url",
    "isDeleted": false,
    "__v": 0
}
```

### Configure an uploaded sitemap for web crawl

To configure a web source using a sitemap, complete the following steps.

**Step 1: Upload the Sitemap**

Upload the sitemap file using the public upload API. Use multipart/form-data with the following fields:

| Field           | Description                             |
| --------------- | --------------------------------------- |
| `file`          | Sitemap file to upload.                 |
| `fileContext`   | Set to `findly`.                        |
| `fileExtension` | Extension of the uploaded sitemap file. |

Save the returned `fileId` from the response.

**Step 2: Attach the Sitemap**

Update the web source with the uploaded sitemap using the Create Web Crawl endpoint. Set the sourceType as `uploadSitemap`, as shown below.

```json expandable=true theme={null}
{
  "sourceType": "uploadSitemap",
  "fileId": "<uploaded-file-id>"
}

```

***

## Initiate a web crawl

This API triggers web crawl.

| Field             | Value                                                      |
| ----------------- | ---------------------------------------------------------- |
| **Method**        | POST                                                       |
| **Endpoint**      | `<host_url>/api/public/bot/:appId/sources/:sourceId/crawl` |
| **Content-Type**  | `application/json`                                         |
| **Authorization** | `auth: <JWT Token>`                                        |
| **API Scope**     | Data Ingestion                                             |

### Query Parameters

| Parameter | Description                                       | Mandatory |
| --------- | ------------------------------------------------- | --------- |
| appId     | Provide your application ID here.                 | Yes       |
| sourceId  | Provide the unique identifier for the web source. | Yes       |

### Request Parameters

| Field          | Type    | Required | Description                                                                                                             |
| -------------- | ------- | -------- | ----------------------------------------------------------------------------------------------------------------------- |
| `forceRecrawl` | Boolean | Yes      | When forceRecrawl is true, the API triggers the existing recrawl flow. Otherwise it triggers the existing extract flow. |

```json expandable=true theme={null}
{
  "forceRecrawl": false
}
```