Once you add content to the application, it needs to be updated as the content from websites may not be static. You can manage (schedule periodic web crawling and edit crawling) and ensure that the content is in sync with the data on the website.

Schedule Web Crawling

The scheduler allows you to schedule a job to re-crawl the configured website periodically. To schedule a web crawling job, follow the below steps:

1. On the Indices page, click Content on the left pane.
2. On the Content list view page, select the respective source from the list.
3. On the source dialog box, click the Configuration tab.
4. On the Configuration tab, turn on the Schedule toggle.
5. Set the Date, Time, and Frequency.
6. Turn on the Crawl Everything toggle to crawl all the domains. 
7. If you wish to crawl only selected domains, then turn off the Crawl Everything toggle.
8. After you turn off the Crawl Everything toggle, the Allow List toggle is turned on automatically. You can enter the allowed list of URLs in the Allow URLs field.
9. If you wish to block URLs, then turn off the Allow List toggle.
10. After you turn off the Allow List toggle, the Block List toggle is turned on automatically. You can enter URLs to block in the Block URLs field. 
11. Select Crawl Settings: 

• JavaScript-rendered
• Use Cookies
• Respect robots.txt

1. Click Save.

Edit Crawler Configuration

To edit a web crawling source, follow the below steps:

1. On the Indices page, click Content on the left pane.
2. On the Content list view page, select the respective source from the list.
3. On the source dialog box, make the required changes.
4. Click Save.