Ingest Data from Content Sources

​

Websites

​

Adding a Web Crawler

​

Advanced Crawl Options

​

URL Filtering

• Crawl everything — All discovered URLs
• Crawl everything except — Block URLs matching specified conditions
• Crawl only specific URLs — Allow only URLs matching specified conditions

​

Authentication

• Username/email and password
• Optional authorization fields (headers, payload, query string, path parameters)
• Authentication URL (may differ from source URL)
• Test type: Text Presence, Redirection, or Status Code

• Form fields with key, type, and value
• Session maintained after initial authentication
• Same test options as Basic Auth

​

Managing Web Sources

​

Troubleshooting

​

Documents

​

Supported Formats

​

Upload Options

​

Upload Limits

​

Managing Documents

• View files: Click any directory to see its files with type and page count
• View file details: Click a file for preview, metadata, and JSON view
• Delete file: Actions > Delete (from directory or file details page)
• Delete directory: Actions > Delete (removes directory and all files)
• Search: Use search bar on Directory page or within directory details

​

Connectors

• Data from the third-party application is ingested, indexed, and made available for search within Search AI.
• The access privileges of the content are maintained as per the privileges assigned to the user in the third-party repository. This ensures that only the files accessible to the user in the third-party application are visible to the user via Search AI as well.

​

Connector Setup Workflow

• Step 1: Authentication: Provide the necessary authentication details (OAuth credentials, API keys, tokens, etc) to establish a connection with the external application.
• Step 2: Manage Content: Choose the type of content to be ingested. Apply filters as needed and optionally customize field mappings to align source fields with the Search AI schema.
• Step 3: Permissions: Select the permission level for the users to access content.
• Step 4: Content: Once synchronization is complete, view the ingested content in the application. You can also configure a scheduler to enable automatic periodic synchronization.
• Step 5: Schedule Sync: View the Sync log and optionally enable automatic periodic synchronization.

​

Configuration Options

​

Sync Operations

​

Managing Connectors

• Enable/Disable: Toggle connector without deleting data (disables sync temporarily). Disabling a connector doesn’t delete the ingested content. It disables any future data synchronization operation with the third-party application. The sync is resumed based on the configuration after the connector is enabled again.
• View content: Content tab lists ingested items with URLs and timestamps.
• Remove connector: Connection Setup > Remove Source. Deletes this content source. This will also delete any data in Search AI indexed from the content source.
• View Content Sync Logs: Connector Settings > Schedule Sync. Capture the full history of connector runs, including start/finish times, document counts, ingestion errors, and status. Sync logs are retained indefinitely.
• View Content Training Logs: Index Configuration > Train > View Training Logs. Use training logs to track training outcomes, identify errors, and ensure your content is ready for search. Training logs are retained for 30 days. For logs older than 30 days, only failed or skipped document details are available. Each log entry includes the training type (Full or Incremental), the time training was initiated, the number of documents successfully processed, the number of documents that encountered errors, the number of documents skipped due to no content found, and an overall status, marked as failed if any document fails. At the document level, each training job also records the document name, extraction strategy, transformation strategy applied, processing status, and the reason for any failure or skipping.

​

Custom Connector

1. Download the reference service implementation from Search AI.
2. Configure config.json with application details and content fields. Set up authentication.
3. Make required changes to the service.
4. Host the service.
5. Configure the service endpoint in Search AI.

​

JSON Connector

• Upload up to 10 files at once
• Maximum 15 MB per file
• Add to existing source or create new

​

Supported Connectors

​

Role-based Access Control (RACL)

​

RACL in Search AI

Ingest Content → Store Permissions in sys_racl → User Submits Query → Match User Identity to sys_racl → Return Answers from Accessible Content Only

​

How RACL Works

• john.doe@example.com (individual)
• jane.doe@example.com (individual)
• testteam@example.com (group)

"sys_racl": [
  "john.doe@example.com",
  "jane.doe@example.com",
  "testteam@example.com"
]

1. Go to Content and open the JSON view for any file.
2. Check the sys_racl field to confirm what identities or entities are stored.
3. The same information is available per chunk in the Content Browser.

curl --request POST \
  'https://<domain>/api/public/stream/<streamId>/advancedSearch?useremail=user@example.com' \
  --header 'Auth: <token>' \
  --data-raw '{"query": "leave policy"}'

​

Enabling RACL

1. Go to the Permissions page of the connector.
2. Select Restricted Access.

​

Updating Permissions After Configuration Changes

​

RACL Scheduler

1. Open the Permissions section of the connector.
2. Enable Permissions Sync Scheduled.
3. Set the desired time and frequency.

​

RACL Limitations

• Supported for connector content only — not for websites or uploaded documents.
• Switching from Restricted to Public Access automatically disables RACL.
• For connectors without automatic resolution, manual Permission Entity management via API is required.

​

Email Notifications

• Choose whether to receive notifications for failures only or all events.
• Select the Search AI modules for which you want to receive notifications.
• Add one or more team members as email recipients.
• Receive alerts for both manually triggered and scheduled jobs.

1. On the Websites or Connectors page, click the email icon to open the Notification Settings in Search AI.
2. Toggle the Enable switch on.
3. Under Select Event Types, choose one of the following:

• Failures only — Receive email notifications only when a sync or training job fails or partially fails. Successful job notifications are suppressed.
• All Events — Receive email notifications for initiation, success, and failure events for the selected modules.

4. Under Select Modules, choose one or more modules for which notifications are required.
5. In the Send to users field, type an email address and press Enter. Repeat for each recipient.
6. Click Save.

​

Quick Reference