Advanced-etl-processor
Data Buffer
The Data Buffer Object acts as an intermediary in the data transformation process, ensuring controlled data flow. It temporarily holds data and does not pass it to the subsequent object until the execution of the preceding object is successfully completed.
Use Cases
-
Buffer Functionality
- Serves as a buffer to manage data flow between objects.
- Ensures downstream processes are executed only after the successful completion of prior steps.
-
Common Applications
- File Validation:
Ideal for scenarios where large files need to be validated before loading. Data is only passed if the file passes validation. - Dependent Data Loading:
Useful for loading data into multiple tables, ensuring dependent tables (e.g., fact tables) are populated last.
- File Validation:
Properties
To change Data Buffer Object properties double click on the object.
Example
Note: Validator aborted execution and no records were written by the writer.
Log
Information 26/01/2025 12:24:05 PM Preparing Writers...
Information 26/01/2025 12:24:05 PM Writer: {Writer3} Connection Type is: Text
Information 26/01/2025 12:24:05 PM Writer: {Writer3} Writer { Writer3 } Is Ready
Information 26/01/2025 12:24:05 PM All Writers are Ready
Information 26/01/2025 12:24:05 PM Target file: C:\DEMO\Buffer\result.csv
Information 26/01/2025 12:24:05 PM Processing Data...
Information 26/01/2025 12:24:05 PM Reader:{Reader} Connection Type is: Text
Information 26/01/2025 12:24:05 PM Reader:{Reader} Path: C:\DEMO\Text Files
Information 26/01/2025 12:24:05 PM Reader:{Reader} Mask: Orders.csv
Information 26/01/2025 12:24:05 PM Reader:{Reader} Found: 1 files to read
Information 26/01/2025 12:24:05 PM Reader:{Reader} Reading all files
Information 26/01/2025 12:24:05 PM Reader:{Reader} Source File: C:\DEMO\Text Files\Orders.csv
Information 26/01/2025 12:24:05 PM 1 Data Buffer: Validation Reading Data Into the buffer
Information 26/01/2025 12:24:05 PM 1 Data Buffer: Validation Buffer size: 20,003
**Error 26/01/2025 12:24:05 PM 1997 Is Equal To:{} Execution Aborted: Value of Field [ORDERNO] is 2000. Execution Result of Function 'Is Equal To' is Success**By controlling data flow, the Data Buffer Object adds reliability and flexibility to complex data transformation processes.
For more technologies supported by our ETL Software see Advanced ETL Processor Versions
Confused? Ask question on our ETL Forum