Using Input Data
Sequentum Cloud takes input data from different data sources, such as a CSV file. Though you can use input data for many different purposes, its most common use case involves loading URLs or inputting values for web forms.
We will explain each type of input data in this section:
Data Type
Data Source
Data Type
The Data Type option in the Data List command provides flexibility in selecting the type of data or files you want to process. The Default value of the Data Type is “CSV”. Once you've selected the input source type, you can choose the specific source of that data or files.
In Sequentum Cloud, there are five data types available, namely CSV, List, Selection, Excel and Parquet. We can use these data types to get input data.
CSV - A CSV or a comma separated value file is a file type where values are separated by a delimiter such as a comma. Comma-Separated Values files are commonly used for storing tabular data.
List - In List data type, there is a simple list of values separated by delimiters (e.g., commas, spaces) with no header.
Selection - In selection data type, a user can use input based on XPath selections.
Excel - Microsoft Excel spreadsheets, often used for complex data analysis and reporting can be used as input data by an external excel file.
Parquet - A parquet is a read only file type. It is a columnar file format optimized for large datasets, frequently used in big data applications. In parquet data type, input data can be accessed by an external parquet file.
Data Source
The Data Source option allows the user to specify the source of the selected data type. The available sources vary depending on the chosen data type. The default data source is "Manual," where you manually enter the data.
In Sequentum Cloud, there are five Data Sources available, namely Manual, Agent Parameter, Private File, Private File, Shared File and Script.
Manual - When the data source is set to manual, the user can input data manually for later use. This option is available for CSV and List data types and allows you to manually enter data.
Agent Parameter - The Agent Parameter data source lets the user access input data file(s) from the agent parameter. It is available for CSV and List data types.
Private File - Private File data source permits the user to access input data from a file present within the agent.
Shared File - Shared File data source permits the user to access input data from a file present within the agent’s deployed space.
Script - A script data source lets the user read a file via. Scripting. This option specifies the scripting language to use (e.g., C#, Python, JavaScript, or Regular Expressions) for accessing or manipulating data.
Consuming Data
Consumer commands use the data that comes from different Data Types. The consumer must be a sub-command of the command providing the data. Data consumers include the following types of commands:
URL
Input Action
Data Value
Consuming from CSV Data Type
Choosing CSV as the Data Type uses an external CSV file. We recommend that you choose the CSV Data Provider for large CSV files since the CSV data provider will perform much better for large quantities of data.
We can upload a CSV file either within the agent or in the shared directory for the file to be accessed later.
Data Source → Manual
We can manually pass input by entering comma separated values. Also, we can check or uncheck the ‘Data includes headers’ checkbox to define if the input data has headers or not.
Agent Parameter
Input parameters are named values that get their value assignments when an agent starts. Any data consumer command within the agent can use these parameters.
When running an agent from the Sequentum Cloud editor that has any input parameters, then the Runtime Input Parameters window will automatically appear.
The user running the agent can enter runtime values for the input parameters and can even add new parameters that are only defined during the current execution of the agent. An agent will always use the default input parameters when debugging. The Runtime Input Parameters window will not appear when starting a debugging session, so you will need to change the default values if you want to debug the agent with different input parameters.
Private File
A Private file is the term coined for files uploaded to an individual agent to be processed as input in the agent.
We can access this private file by selecting the ‘Private File’ option from the Data Source followed by the file to be used as an input (refer the below snapshot)
A user can define if the text file includes headers. The default value is checked for ‘Data includes headers’.
Shared File
The files stored in any Space are called shared Files that are accessible to all users who have access to the same space as the agent. This makes them freely accessible and editable. You can use shared files across multiple agents within the same space. The Files inside any Space can’t be accessed by other Space. They are specific to that space.
We can upload a shared file into a space from the Control Center.
Upon selecting the ‘New Shared File’ option from the drop down, it will open a ‘Create Shared File’ window where you can either drag and drop the file or browse the file by selecting the ‘Upload File’ option.
We can then access this shared file in the agent by selecting the ‘Shared File’ option from the Data Source followed by the file to be used as an input (refer the below snapshot)
A user can define if the text file includes headers. The default value is checked for ‘Data includes headers’.
Script
We can also provide input data based on some operations, this can be achieved using the Script option as the Data Type.
We have to select the ‘Language’ the script is written in and the timeout to execute the script.
Following are some simple scripts to read input file from agent parameters using script:
C# Code
#r System.IO.dll
using System.IO
public string GetData (RunContext context){
var fileName = context.GlobalData.GetString("InputFileName");
var filePath = context.PrivateFiles.GetFile(fileName);
var inputFile = File.ReadAllText(filePath);
return inputFile;
}Python Code
from se_scripting_utils import *
import os
def get_data(context: RunContext):
file_name = context.global_data.get_string("InputFileName")
file_path = context.private_files.get_file(file_name)
with open(file_path, 'r') as input_file:
return input_file.read()Consuming from List Data Type
A list command will iterate through all the data rows provided and execute each of the sub-commands once for each row.
It is similar to passing input data as other Data Types, but the difference is in the headers. A List Data Type does not accept headers and rather treats each input as a data row.
Consuming from Selection Data Type
Choosing Selection for the Data Type will extract data from elements on the web page. This data type uses the selection from the XPath of the command and then adds a relative XPath to find the web elements that you include in the selection. You can choose which HTML attributes of the web elements to include as data.
An input action command selects the drop-down on the web page, and a relative XPath selects the option elements i.e. elements of the selected XPath within that drop-down. The drop-down options then become available to the form field command, which can iterate through the options and ensure that the web form submission is done for each item in the drop-down.
Consuming from Parquet Data Type
Similar to CSV Data Type, you can choose Parquet Data Provider. Parquet, an open-source file format for Hadoop, stores nested data structures in a flat columnar format. Compared to a traditional approach where data is stored in a row-oriented approach, parquet is more efficient in terms of storage and performance.
A parquet data type lets the user take inputs either using a Private File or a Shared File similar to CSV data type. By default, the ‘Data includes headers’ checkbox is checked.
Consuming from Excel Data Type
Similar to a parquet data type, an excel data type takes input data either using a private file or a shared file.
By default, the ‘Data includes headers’ checkbox is checked.