Exporting Data

The Export section provides settings to define how extracted data is managed and processed once the extraction is complete.

Export Configuration

The configuration screen for the Export command has two tabs viz. Options, and Text. Use the Options tab to set the configuration for the agent. Use the Text tab to navigate to the text of the agent where the agent configuration text part is present.

Export Properties

On the Options tab of the command, we have four options:

Export
Duplicates
Change Tracking
Schema

Export → Export settings

The Export settings section defines how extracted data is exported, including options for backups and naming the export file.

Export extracted data: Determines whether the extracted data will be exported. By default, this option is enabled.
Backup extracted data: Automatically saves a backup of the extracted data to a database. By default, this option is enabled.
Export name: This property specifies the name of the exported file or table (such as a spreadsheet, CSV file, Parquet file, or XML node), or the top-level key that will contain all extracted data in the output JSON file, if the container generates a new table. If this property is left empty, the agent name or command name will be used by default.
Key name: Defines the name of the primary key column in the exported table.
Sort order: Specifies the sort order of the primary key column in the exported table (database table, spreadsheet, or CSV file). The default value is 0.

Export → Empty data handling

The Empty data handling section specifies how empty rows and empty data tables are managed during export.

Empty row handling
Specifies the action to take when an exported data row is empty. The available options are:
i) Keep: Retains empty rows. This is the default setting.
ii) Remove: Deletes empty rows.
iii) Remove With Error: Deletes empty rows and increments the error count.
iv) Remove With Failure: Causes the export to fail if empty rows are found.
Empty table handling: Specifies the action to take when the exported data table is empty. The available options are:
i) Keep: Retains the empty table. This is the default setting.
ii) Remove: Deletes the empty table.
iii) Remove With Error: Deletes the empty table and triggers an error message.
iv) Remove With Failure: Causes the export to fail if the table is empty.

Export → Data merge method

Data merge method determines how data is exported from the container. The default value is 'Add Rows,' which allows users to append rows only, while columns must be added separately. Different methods for data extraction are available, including:

i) Add Rows: Add new rows.

ii) Add Columns: Add new columns.

Export → API settings

API settings configure the communication mode for exports to Start Agent API endpoint.

i) Checked (Synchronous): Starts the agent run and waits until it finishes. The response includes the extracted content, providing the final result in a single call.

ii) Unchecked (Asynchronous): Starts the agent run and returns immediately. The response includes details of the initiated run.

Once data is captured, it needs to be exported. Export commands manage the output of scraped data into various formats and destinations, such as CSV files, cloud storage services (S3, Google Drive), or even complex data warehouses like Snowflake. These commands define how and where your data is stored after extraction.

Duplicates

Please refer to the “Remove Duplicate” section for more insight.

Change Tracking

Please refer to the “Tracking Changes” section for more insight.

Schema

Please refer to the “Export Schema” section for more insight.