Skip to main content

Script Types and Data Fetching Process

Different Types of Scripts​

1. Bulk Fetching Scripts​

  • Purpose: Used to fetch large volumes of data, typically covering an entire year or the complete dataset available on a website.
  • Use Case: Initial data ingestion when a data source is first introduced into the system.

2. Incremental Scripts​

  • Purpose: Designed to fetch data within a specific date range.
  • Use Case: Used for monthly or periodic updates after the initial bulk data has been fetched.

3. DB Insertion Script​

  • Purpose: Responsible for inserting the fetched data into the database.
  • Use Case: Run after the data has been crawled to ensure it is properly stored and accessible for reporting and analysis.

4. Updation Script​

  • Purpose: Updates records of pending cases by crawling the website and retrieving the latest case status.
  • Use Case: Regular maintenance to keep case data up-to-date.

Process to Fetch and Update the Data​

  1. Initial Data Collection:

    • When a new data source is integrated, use the Bulk Fetching Script to download all available data.

  2. Ongoing Updates:

    • Perform monthly or periodic updates using the Incremental Script to capture new or changed data.

  3. Database Insertion:

    • After fetching, run the DB Insertion Script to store the crawled data into the database.

  4. Pending Case Updates:

    • Periodically execute the Updation Script to track and update any pending cases using live data from the respective websites.

This structured process ensures comprehensive and up-to-date data collection, supporting robust analysis and reporting capabilities.