The Advanced Settings section is available under the Summary tab of the Integration Bridge page. It provides additional configuration options for fine-tuning the execution of data streaming jobs.
Advance Setting Feature has the following characteristics:
-
Select Worker Capacity: Users can select the number of records workers will process.
-
Number of Workers: This option allows users to select how many workers will work on their data processing. More the number of workers, lesser the time of processing.
-
Source Response Wait Time: Users can set how long the system should wait for source data to arrive before moving forward.
-
Log Stream Size per Line (in KB): This option allows the user to specify the amount of data that should be displayed in each log line.
-
Log Rotation Size (in MB): This option allows users to configure the maximum log size after which the data is reflected in the Log Analytics Dashboard.
Example:
If you have a large dataset, say, 1 million records, you can speed up processing by enabling Advanced Settings from the Summary tab in the Integration Bridge.
In this example, the chunk size is 50,000 records, meaning the source system sends 50,000 records per processing cycle.
You can configure the following settings based on your requirements:
-
Streaming Worker Capacity: 10000
-
Number of Workers: 4
-
Source Response Time: 1000 (in milliseconds)
-
Log Stream Size per Line: 300 KB
-
Log Rotation Size: 50 MB
With this configuration:
-
The 50,000-record chunk will be split evenly across 4 workers.
-
Each worker will process 10,000 records in parallel.
-
Log lines are capped at 300 KB each.
-
Once logs reach 50 MB, the data is reflected in the Log Analytics Dashboard.
-
This leads to faster data processing and optimized storage.
Activate Parallel Processing
The Activate Parallel Processing allows data to be processed simultaneously for faster and more efficient execution.
Activate Parallel Processing feature has the following characteristics:
-
Improves Performance for Large Datasets: Efficiently handles high volumes of data by distributing workload across multiple CPU cores.
-
Minimizes Processing Bottlenecks: Reduces delays caused by single-threaded data operations.
-
Ideal for Batch Jobs or Heavy Transformations: Especially useful in scenarios where complex data transformations or bulk operations are involved.
Log Stream Size per Line
This feature allows users to set the maximum size of each log line directly in the eZintegrations UI. This helps control how much information each log line holds, reducing delays caused by very large log entries during data pipeline execution.
Log Stream Size per Line Feature has the following characteristics:
-
Helps manage logs efficiently in high-volume data integration environments.
-
Prevents logging from becoming a bottleneck when large data sets pass through the pipeline.
-
Allows users to configure and limit the size of each log line via the UI.
-
Reduces delays in file writing caused by overly large log entries.
-
Helps prevent excessive memory usage during logging.
-
Improves readability and maintainability of log files for easier debugging.
Note: Log Stream Size per Line should not be less than 20KB.
Log Rotation Size (in MB)
This feature allows users to configure the maximum size of logs that will be reflected in the Log Analytics Dashboard. Once the logs reach the defined size, the data will appear in the dashboard for monitoring.
Log Rotation Size Feature has the following characteristics:
-
Defines the maximum log size after which data is reflected in the Log Analytics Dashboard.
-
Ensures users always see the most recent log activity in the dashboard.
-
Helps manage and optimize performance in high-volume logging scenarios.
-
Provides better visibility by controlling how much data is shown in the dashboard at a time.
Example:
If the Log Rotation Size is set to 50 MB, once logs reach this size, the corresponding data will be available in the Log Analytics Dashboard for review.
