Index Configuration Guide

​

Overview

1. Extraction - Breaking down source content into manageable chunks using various extraction strategies
2. Transformation - Enriching and refining extracted content through configurable processing stages
3. Indexing - Storing processed chunks with vector embeddings for efficient retrieval

​

Content Extraction

​

Extraction Strategies

​

Text Extraction Model

• Chunk Size (Pages): Treats every page as a single chunk
• Chunk Size (Tokens): Prepares chunks using:
  • Tokens: Maximum tokens per chunk (up to 5000). Smaller chunks suit granular tasks; larger chunks better preserve context.
  • Chunk Overlap: Number of tokens overlapping between consecutive chunks

​

Layout Aware Extraction

• General Template: Extracts content from complex PDFs and DOCX files including tables and images

​

Advanced HTML Extraction

• General: Identifies different components and classes within HTML documents. Tables and images present between the content are also extracted and stored as chunks.
• Token-Based: Generates chunks based on token size. Images within the content are extracted and stored in the chunk.

  Field	Description	Range	Default
  Tokens	Max tokens per chunk	100 - 1000	300
  Chunk Overlap	Overlapping tokens between adjacent chunks	10 - 100	—

​

Markdown Extraction

​

Image-Based Document Extraction

• Each page in the PDF document is converted to an image, preserving visual structure and layout.
• Pages are processed using a VDR embedding model that captures both textual and visual semantics.
• Page contents are also extracted in standard chunk format alongside the visual embedding.
• Each extracted chunk includes a page_image_url field referencing its corresponding page image.

​

Custom Extraction

• Ingested content is sent to an external service for processing.
• The service structures the data into chunks and returns them to Search AI via a callback API.
• Search AI indexes the returned chunks for search and retrieval.

• Endpoint: URL for sending content (POST endpoint).
• Concurrency: Maximum API calls per second.
• Headers: Additional request headers. Default headers will be added automatically and can’t be edited.
• Request Body: Content fields to send to the service.

• Generated Response — displays the response returned via the callback URL.
• Response Path — provide the JSON path to locate extracted chunks within the response.
• Response Comparison — validates the actual response against the expected structure. An error is shown if they don’t match.

​

Default Extraction Strategies

​

Managing Extraction Strategies

​

Adding a Strategy

1. Navigate to Index > Extract.
2. Click +Add Strategy.
3. Configure:
   • Strategy Name: Unique identifier
   • Define Source: Select data sources and content types using filters
   • Extraction Model: Choose the extraction method
   • Configure model-specific settings

​

Multiple Strategies:

​

Managing Strategies:

• Enable/Disable: Toggle from the strategy page. Use it for testing purposes to evaluate alternative extraction strategies or when it’s no longer needed.
• Delete: Use the Delete button (doesn’t affect existing chunks).
• Reorder: Drag to change sequence.

​

Content Transformation

​

Benefits

• Improved Data Quality: Fix errors, standardize fields, add contextual information.
• Enhanced Control: Adapt enrichment to specific business needs.
• Bulk Transformation: Apply rules to all applicable content simultaneously.

​

Transformation Stages

​

Field Mapping Stage

• Name: Unique stage identifier
• Condition: Rules for selecting content
  • Field Name, Operator, Value
• Outcome: Actions to perform:
  • Set: Sets value for target field
  • Delete: Removes target field
  • Copy: Copies value between fields

​

Custom Script Stage

• Outcome: JavaScript defining transformations.
• View Fields: Lists all the fields of the schema.

int temp_total_pages = 0;
if(ctx.file_content_obj != null){
    for (def item: ctx.file_content_obj) {
        if (item!="") {
            temp_total_pages = temp_total_pages+1;
        }
    }
}
context.total_pages = temp_total_pages;

​

Exclude Documents Stage

• Field: Document field for condition (for example, creation date and file type).
• Operator: Comparison operator (greater than, less than, equals).
• Value: Comparison value.

​

API Stage

• Endpoint: POST endpoint URL (supports dynamic fields with {{field_name}}). Example: https://api.example.com/meta/{{chunkTitle}}?type={{cfs1}}
• Headers: Key-value pairs for authentication
• Request Body: Content to send (use {{field_name}} for dynamic values). Example: "content": {{content}}.

​

LLM Stage

1. Set up required LLM in Models Library.
2. Create custom prompt for ‘Transform Documents with LLM’.
3. Enable the feature in Gen AI features page.

• LLM: Select the language model.
• Prompt: Select the custom prompt.
• Target Field: Field for storing enriched content.

​

Stage Availability by Extraction Strategy

​

Managing Transformation Stages

​

Adding a Stage

1. Navigate to Index > Enrich (or Transform page)
2. Click +New Stage
3. Select stage type and configure
4. Click Save

​

Stage Operations

• Testing a Stage: Click Simulate on the Transform page, select the stage and number of documents to test with. Results appear in the Viewer as a JSON object. Any transformation errors are listed under simulate_errors.
• Enable/Disable: Use ellipsis menu or status toggle. Disabling a stage doesn’t delete or remove it permanently. When a stage is disabled, the output of the previous stage in the sequence is directly sent as input to the next stage.
• Delete: Permanently removes the stage.
• Sequencing: Stages execute top-down. The output of each stage becomes the input of the next — so order directly affects outcomes. Use the drag handle to reorder stages.

​

Content Enrichment Using Workbench

​

Key Features

• Pipeline Processing: Sequential stage execution; the output of each stage feeds into the next.
• Custom Transformation: Customizable stages per business needs.
• Simulation Capabilities: Built-in Built-in simulator to test individual stages or the cumulative effect of multiple stages before deployment.
• Flexible Sequencing: Design efficient processing workflows.

​

Supported Stages

​

Content Transformation vs Enrichment

​

Adding Workbench Stages

1. Navigate to Index > Enrich.
2. Click +New Stage.
3. Select stage type.
4. Configure:
   • Unique stage name
   • Conditions for content selection
   • Outcomes defining transformations
5. Click Save.

​

Stage Management

​

Workbench Simulator

• Test and verify individual stage outputs
• Test cumulative effects of multiple stages
• Works with any data source type

1. Click Simulate option on the Workbench page.
2. Select the stages to test.
3. Choose number of documents for testing.
4. View results in JSON format in the Chunk Viewer.

• Simulator shows changes from all stages in sequence order
• Testing from a specific stage shows cumulative transformations up to that point
• Temporarily disable other stages to test individual stage behavior
• Errors during transformation appear in the simulate_errors object

​

Content Browser

​

Key Capabilities

• Observation and Verification: Inspect and verify extracted chunks for accuracy
• Editing of Chunks: Modify chunk information directly within the browser

​

Viewing Chunks

• Preview of any images or tables present in the chunk. Click a preview to enlarge it to full size. If a chunk contains multiple images, a carousel allows navigation through them.
• Summary information

• Document Title: Source document/page title
• Chunk Title: Title assigned to the chunk
• Chunk Text: Content of the chunk
• Chunk ID: Unique identifier
• Page Number: Source page (if applicable)
• Source Title: Name of the source in the application
• Source: Content type (web pages, files, connectors)
• Extraction Strategy: Strategy used for extraction
• Edited on: Last update date (if applicable)
• Source URL: Source URL (if applicable)

​

Editing Chunks

1. Click the Details icon on a chunk
2. Modify title and/or text
3. Click Save

​

Search and Filter

• Source types
• Extraction strategy
• Content keywords
• And more

​

Vector Configuration

• Select from out-of-the-box XO GPT embedding models or bring your own custom model.
• Choose which chunk fields are used to generate embeddings.
• Generate up to three vectors per chunk, each capturing different semantic aspects via distinct field combinations (multi-vector search).

​

Glossary

​

Prerequisites

1. Navigate to Generative AI Tools > Model Library.
2. Confirm that vector generation is enabled for the desired model.

​

Vector Configuration for Textual Data

​

Multi-Vector Search

1. Configure vectors — Define up to three vectors, each targeting specific sources, file types, and fields.
2. Assign weights — Set a percentage weight per vector to control its influence on the final relevance score.
3. Retrieval — At query time, the user query is converted to a vector and matched against all configured vectors, calculating a similarity score corresponding to each vector. Each similarity score is multiplied by its weight, and scores are summed to produce a final relevance ranking. Chunks are ranked based on these final scores, and the most relevant results are returned.

​

Configuring Vectors

• Source — Select the content source (e.g., uploaded files, connectors). Use the AND operator to narrow the scope further.
• File Types — Select supported file types (e.g., PDF, HTML).
• Fields — Select one or more chunk fields (e.g., Chunk Text, Chunk Title, Record Title) for embedding generation.

• Source: Files
• File Type: PDF
• Fields: cfs1

​

Field Combinations Precedence

​

Vector Coverage

​

Assigning Weights to Vectors

​

Automatic Weight Rebalancing

• Vector 1 — all sources, fields a, b, weight 50%
• Vector 2 — web pages only, fields c, d, weight 20%
• Vector 3 — all sources, fields e, f, weight 30%

Adjusted Weight = Original Weight + ((Unavailable Weight × Original Weight) / Sum of Available Weights)

• Vector 1 → 62.5%
• Vector 3 → 37.5%

​

Points to Note

• The default Vector 1 cannot be disabled or deleted.
• Avoid overlapping conditions across vector definitions.
• After changing the embedding model, updating vector configurations, or reassigning weights — always Train the application to regenerate embeddings with the new settings.

​

Vector Configuration for Image-Based Data

​

Batch Processing of Vector Generation

​

Best Practices

​

Extraction Strategy Selection

• Match strategy to content type and structure
• Consider expected query complexity when setting chunk sizes
• Use Layout Aware for documents with tables and charts

​

Transformation Pipeline Design

• Order stages logically (e.g., exclude before enrichment)
• Use simulation frequently during development
• Keep transformations focused and modular

​

Content Quality

• Review chunks in Content Browser after processing
• Edit chunks only when necessary (edits lost on content updates)
• Use filters to identify problematic chunks

​

Testing and Validation

• Simulate all stages before training
• Test with representative sample documents
• Verify cumulative effects of multiple stages

​

Processing Workflow Summary

1. Content Ingestion
   └── Sources: Websites, Documents, Connectors

2. Extraction Phase
   └── Apply extraction strategy based on content type
   └── Generate initial chunks

3. Transformation Phase (Document Workbench)
   └── Stage 1 → Stage 2 → Stage N
   └── Each stage processes and passes to next

4. Enrichment Phase (Workbench)
   └── Chunks processing stages
   └── Field mapping, scripts, API calls, LLM enrichment

5. Indexing Phase
   └── Vector embedding generation
   └── Storage in vector database

6. Verification
   └── Content Browser review
   └── Manual edits if needed

​

Related Resources

• Content Sources Documentation
• Answer Generation
• Retrieval Strategies
• GenAI Tools - Models Library