User Guide : Map Connectors : Source and Target Map Connectors : Process and Transformation LogFile Parser
 
Share this page             
Process and Transformation LogFile Parser
This connector uses a .cxl script as the default extract source schema to parse log files. You can parse log files that contain transformation or process errors and messages. The log files can contain a single run of a transformation or process or multiple runs (of transformations, processes, or a combination of both).
Users are often interested in manipulating the log file data in various ways. The following are examples of possible use cases:
Load the contents of the log file into a database
Create reports with portions of the log file
Get a different view of the log
Create custom logs from the main log
Mine the log file for specific values, and then use those values in a transformation or a process step
Connectivity Pointers
Here are some things to remember when using the LogFile Parser connector:
The Extract Schema is already selected for you. This default extract script file is called ProcessTransformationLogFile.cxl.
Records do not necessarily appear in the order read by the integration platform. For a complete list of the record types and fields, see Record Types.
Since the log file parser parses multiple logs that are appended to one another, there may or may not be steps of a certain type in a run of a different process. This means that logic may be needed to write out record types at the proper places in the target. If not, then you could have records that are by themselves or you could even associate records that do not belong together.
In a process, the last thing to be read is the Process_Execution_Stats. The log file format for the LogFile Parser map example is shown below:
Example: Log File Format for the LogFile Parser Map
Process Start1
  SQL Step1 Begin
  SQL Step1 End
Process Stop1
Process Start2
  SQL Step2 Begin
  SQL Step2 End
Process Stop2
Process Start3
  SQL Step3 End
Process Stop3
Here is how it is read into the map:
  SQL Step1 Begin
  SQL Step2 End
Process1 Information
  SQL Step2 Begin
  SQL Step2 End
Process2 Information
  SQL Step3 End
Process3 Information
Here is the XML structure we want to create for the example:
<Information>
  <Process_Information>
    <SQL_Step>
    <SQL_Step>
  </Process_Information>
  <Process_Information>
    <SQL_Step>
    <SQL_Step>
  </Process_Information>
  <Process_Information>
    <SQL_Step>
  </Process_Information>
</Information>
Connector-Specific Notes
No known limitations at this time.
Property Options
You can set the following source (S) and target (T) properties.
Property
S/T
Description
ByteOrder
S
Allows you to specify the byte order of Unicode (wide) characters. The default is Auto and is determined by the architecture of your computer. The list box options are Auto (default), Little Endian and Big Endian. Little Endian byte order is generally used by Intel and DEC Alpha machines and places the least significant portion of a byte value in the left portion of the memory used to store the value. Big Endian byte order is used by IBM 370 computers, Motorola microprocessors and most RISC-based systems and stores the values in the same order as the binary representation.
ProgramVariables
S
Allows you to set or override program variable values. There is no default.
ReportReadingScriptFile
S
The script file, usually with a .cxl extension. It only appears here for verification purposes. The default .cxl file is ProcessTransformationLogFile.cxl.
StartOffset
S
If your source data file starts with characters that need to be excluded from the transformation, set the StartOffset option to specify at which byte of the file to begin. The default value is zero. The correct value may be determined by using the Hex Browser. For details, search for "hex values" in the documentation.
Note:  This property is set in number of bytes, not characters.
Encoding
S
Sets the character encoding for source and target files.
Note:  Shift-JIS encoding is used only in Japanese operating systems. UCS-2 is no longer a valid encoding name, but you may use UCS2 by changing the string value in the XML file.
Record Types
The 29 record types that are supported in this connector represent data collected from the log for the following process and transformation elements:
Processes
Process steps
Process sessions
Transformations
Transformation totals
Record Types for Process Steps
The following steps are specific to running a process. The listed record types are NOT part of a transformation-only log file.
Record Type Name
Field Name for Record Type
Description
Start_Step_End
 
Represents the data for a successful or unsuccessful Start Step in a process log file
 
Start_Step_Status
Status of the step after the run (for example, complete, failed)
 
Start_Step_Begin_Time
Time the Start Step begins
 
Start_Step_End_Time
Time the Start Step ends
 
Error_Description
Description of the error (if an error occurred)
 
Error_Time
The time the error occurred (if an error occurred)
 
Start_Step_Last_Error
Return code from the step
 
Error_Number
Error number associated with the error (if an error occurred)
Stop_Step_End
 
Represents the data for a successful or unsuccessful Stop Step in a process log file
 
Stop_Step_Status
Status of the step after the run (complete, failed, etc)
 
Stop_Step_Begin_Time
Time at the start of the Stop step
 
Stop_Step_End_Time
Time at the end of the Stop step
 
Error_Description
Description of the error (if an error occurred)
 
Error_Time
Time the error occurred (if an error occurred)
 
Stop_Step_Last_Return
Return Code from the step
SQL_Step
 
Represents a SQL Step in the process starting and stopping in the log file
 
SQL_Step_Action
Begin (Start) or End (Stop) for the step
 
Error_Time
Time error occurred (if an error occurred)
 
SQL_Step_Name
Name of the step in the process
 
Error_Number
Error number associated with the error (if an error occurred)
 
SQL_Step_Time
Time the SQL_Step_Action occurred
 
Error_Description
Description of error (if an error occurred)
 
SQL_Step_Status
Failed, successful, etc.
 
SQL_Step_Last_Return
Last return code returned from the Step
DTS_Step_End
 
Represents a DTS step run in the process log file
 
DTS_Step_Status
Status of the step after the run (complete, failed, etc)
 
DTS_Step_Name
Name of the step in the process
 
DTS_Step_End_Time
Time at the end of the DTS step
 
DTS_Step_Begin_Time
Time at the start of the DTS step
 
DTS_Step_Last_Return
Last Return code from the step
Decision_Step_End
 
Represents a Decision Step completing in the process log file
 
Decision_Step_Result
True or False
 
Decision_Step_Name
Name of the Decision step
 
Decision_Step_End_Time
Step end time
 
Decision_Step_Begin_Time
Step begin time
Application_Step_End
 
Represents an Executable (Exe) Step running in the process log file
 
Application_Step_Status
Status of the step after the run (for example, complete, failed)
 
Application_Step_Name
Name of the step in the process
 
Application_Step_End_Time
Time at the end of the Application step
 
Application_Step_Begin_Time
Time at the start of the Application step
 
Application_Step_Last_Return
The last return code from the Step
SubProject_Step_End
 
Represents a Sub-Process step running in a process log file
 
SubProject_Step_Status
Status of the step after the run (for example, complete, failed)
 
SubProject_Step_Name
Name of the step in the process
 
SubProject_Step_End_Time
Time at the end of the step
 
SubProject_Step_Begin_Time
Time at the start of the step
 
SubProject_Step_Last_Return
Last return code from the step
Queue_Step_End
 
Represents the execution of a Queue Step in the process log file
 
Queue_Step_Status
Status of the step after the run (for example, complete, failed)
 
Queue_Step_Name
Name of the step in the process
 
Error_Description
Description of the error (if an error occurred)
 
Queue_Step_End_Time
Time at the end of the Queue step
 
Queue_Step_Start_Time
Time at the start of the Queue step
 
Error_Time
Time error occurred (if an error occurred)
 
Queue_Step_Last_Error
Last error code returned from the step
 
Error_Number
Error number associated with the error (if an error occurred)
Scripting_Step_End
 
Represents the execution of a Scripting step in a process log file
 
Scripting_Step_Status
Status of the step after the run (for example, complete, failed)
 
Scripting_Step_Name
Name of the step in the process
 
Error_Description
Description of the error (if an error occurred)
 
Scripting_Step_End_Time
Time at the end of the Scripting step
 
Scripting_Step_Begin_Time
Time at the start of the Scripting step
 
Error_Time
Time error occurred (if an error occurred)
 
Scripting_Step_Last_Error
Last error code returned from the step
 
Error_Number
Error number associated with the error (if an error occurred)
Invoker_Step_End
 
Represents running the Invoker Step in the process log file
 
Invoker_Step_Status
Status of the step after the run (for example, complete, failed)
 
Invoker_Step_Name
Name of the step in the process
 
Error_Time
Time the error occurred (if an error occurred)
 
Invoker_Step_End_Time
Time at the end of the Invoker step
 
Invoker_Step_Begin_Time
Time at the start of the Invoker step
 
Error_Number
Error number associated with the error (if an error occurred)
 
Error_Description
Description of the error (if an error occurred)
DocValidator_Step_End
 
Represents a Document validator Step running in the process log file
 
DocValidator_Step_Status
Status of the step after the run (for example, complete, failed)
 
Error_Time
Time the error occurred (if an error occurred)
 
DocValidator_Step_Info
Information about the step
 
DocValidator_Step_Error_Description
Error Description Text in the log about the Step
 
DocValidator_Step_Name
Name of the step in the process
 
DocValidator_Step_End_Time
Time at the end of the DocValidator step
 
Error_Number
Error number associated with the error (if an error occurred)
 
DocValidator_Step_Error_Number
Error number associated with the DocValidator step
 
DocValidator_Step_Begin_Time
Time the DocValidator step starts
 
Error_Description
Description of the error (if an error occurred). In this case, possibly more error description
XSLTTransformer_Step_End
 
Represents an XSLT Step running in the process log file
 
XSLTTransformer_Step_Status
Status of the step after the run (complete, failed, Error Descriptions, etc)
 
XSLTTransformer_Step_Name
Name of the step in the process
 
XSLTTransformer_Step_End_Time
Time at the end of the XSLT Transformer step
 
XSLTTransformer_Step_Begin_Time
Time at the start of the XSLT Transformer step
 
XSLTTransformer_Step_Last_Error
Last Error code from the Step
Project_Execution_Stats
 
Represents the statistics from a process running in the log file. In a process, this is the last record type to be read
 
Project_Execution_Time
Amount of time it took to execute the process
 
Serial_Number
Serial Number for the Product
 
Project_File
Process file that was run
 
Project_Execution_Status
Status of the step after the run (complete, failed, etc)
 
Project_Execution_Begin_Time
Time at the start of the process execution
 
Product
Product that ran the process
 
Prepare_Map_Time
Time to prepare any maps that were executed in the process run
 
Owner
Owner of the Product
 
Organization
Company that owns the product
 
Build_Version
Build version of the product (for example, 8.12.1.14 2006061416)
 
Project_Execution_Finish_Time
Time at the end of the execution of the process
 
Project_Init_Start_Time
Initialization time
 
Project_Execution_End_Time
Time at the end of the execution of the process
Transformer_Step_End
 
Represents the running of a Transformer Step in the process log file
 
Transformer_Step_End_Time
Time at the end of the Transformer step
 
Transformer_Step_Last_Error
Last error code from the step
 
Transformer_Step_Status
Status of the step after the run (for example, complete, failed)
 
Transformer_Step_Name
Name of the transformer step in the process
 
Transformer_Step_Begin_Time
Time at the start of the Transformer step
Skipped
 
Represents any steps in the process that are disabled and are therefore skipped over during the run
 
Skipped_Time
Time to skip the step
 
Skipped_Step
Type of step skipped
 
Skipped_Step_Name
Name of the skipped step
Iterator_Step_End
 
Represents an Iterator Step run in the process log file
 
Iterator_Step_Status
Status of the step after the run (for example, complete, failed)
 
Iterator_Step_Name
Name of the step in the process
 
Error_Description
Description of the error (if an error occurred)
 
Iterator_Step_End_Time
Time at the end of the Iterator step
 
Iterator_Step_Begin_Time
Time at the start of the Iterator step
 
Error_Time
Time the error occurred (if an error occurred)
 
Iterator_Step_Last_Error
Last error code from the Step
 
Error_Number
Error number associated with the error (if an error occurred)
Aggregator_Step_End
 
Represents the running of an Aggregator Step in the process log file
 
Aggregator_Step_Status
Status of the step after the run (for example, complete, failed, etc)
 
Error_Description
Description of the error (if an error occurred)
 
Aggregator_Step_Name
Name of the step in the process
 
Aggregator_Step_End_Time
Time at the end of the Aggregator step
 
Error_Time
Time the error occurred (if an error occurred)
 
Aggregator_Step_Begin_Time
Step End Time
 
Aggregator_Step_Last_Error
Last error code from the Step
 
Error_Number
The error number associated with the error (if an error occurred)
Transformation_Step_End
 
Represents the running of a transformation step in the process log file
 
Transformation_Step_Name
Name of the step in the process
 
Transfrmation_Step_Begin_Time
Start Time
 
Transformation_Step_End_Status
Status of the step after the run (for example, complete, failed)
 
Transformation_Step_End_Time
Time at the end of the transformation step
 
Error_Description
Description of the error (if an error occurred)
 
Error_Time
Time the error occurred (if an error occurred)
 
Transformation_Step_Last_Error
Last error returned from the step
 
Error_Number
Error number associated with the error (if an error occurred)
Record Types for Process Sessions
These are sessions that are specific to running a process. They are not be part of a tranformation-only log file or a process log file that does not use process sessions.
Record Type Name
Field Name
Description
Queue_Session
 
Represents a Queue session that was used in the process
 
Queue_Session_Action
Begin (Start) or End (Stop) for the session
 
Error_Number
Error number associated with the error (if an error occurred)
 
Queue_Session_Name
Name of the session in the process
 
Error_Description
Description of the error (if there was an error
 
Queue_Session_Time
Time the Queue_Session_Action occurred
 
Error_Time
Time the error occurred (if an error occurred)
 
Queue_Session_Last_Return
Last return code from the session.
SQL_Session
 
Represents a SQL Session used in the process
 
SQL_Session_Action
Begin (Start) or End (Stop) for the session
 
Error_Time
Time the error occurred (if an error occurred)
 
SQL_Session_Name
Name of the Session in the process
 
Error_Number
Error number associated with the error (if an error occurred)
 
SQL_Session_Time
Time the SQL_Session_Action occurred
 
Error_Description
Description of the error (if an error occurred)
 
SQL_Session_Last_Return
Last return code from the session
Invoker_Session
 
Represents an Invoker Session used in the process
 
Invoker_Session_Action
Begin (Start) or End (Stop) for the session
 
Invoker_Session_Name
Name of the session in the process
 
Invoker_Session_Time
Time the Invoker_Session_Action occurred
 
Invoker_Session_Last_Return
Last return code from the Session
Aggregator_Session
 
Represents an Aggregator session used in the process
 
Aggregator_Session_Action
Begin (Start) or End (Stop) for the session
 
Aggregator_Session_Name
Name of the session in the process
 
Aggregator_Session_Time
Time the Aggregator_Session_Action occurred
 
Aggregator_Session_Last_Return
Last return code from the session
Iterator_Session
 
Represents an Iterator session that was used in the process
 
Iterator_Session_Action
Begin (Start) or End (Stop) for the session
 
Iterator_Session_Name
Name of the session in the process
 
Iterator_Session_Time
Time the Iterator_Session_Action occurred
 
Iterator_Session_Last_Return
Last return code from the session
Transformer_Session
 
Represents a Transformer session used in the process
 
Transformer_Session_Action
Begin (Start) or End (Stop) for the session
 
Transformer_Session_Name
Name of the session in the process
 
Transformer_Session_Time
Time the Transformer_Session_Action occurred
 
Transformer_Session_Last_Return
Last return code from the session
Record Types for Process and Transformation Messages
Messages are collected separately from other steps and sessions. This is because they may or may not have a correlation to the step that is precedes or follows.
The following errors are collected with the step or session that caused the error because the error can reliably be associated with that step or session.
Record Type Name
Field Name
Description
Debug_Messages
 
Represents a Debug message in the log file
 
Debug_Message
Text of the debug message
 
Debug_Message_Time
Time the message was logged
Information_Messages
 
Represents an Informational message logged
 
Information_Time
Time the informational message was logged
 
Information_Message
Text of the message that was logged
Warning_Message
 
Represents a warning message written to the log file
 
Warning_Message
Text written to the log
 
Warning_Message_Last_Return
Last return code from the logged message
 
Warning_Message_Time
Time the message was logged
 
Warning_Message_Info
More information about the warning
Record Types for Transformation-Only Runs
This information will be logged for a stand-alone transformation, a transformation in a process, or both. It is the collected information about the transformation-specific run.
Record Type Name
Field Name
Description
Transformation_Totals
 
Represents the information about a transformation that was run stand-alone or as a transformation in a process step.
 
Total_records_Buffered
Total number of records buffered.
 
Execution_Time
Time to run the transformation.
 
Execution_Begin_Time
Time at the beginning of the transformation.
 
Execution_End_Status
Time at the end of the transformation.
 
Map
Map and location of the transformation being run. xmldb:ref:/// .tf.xml
 
Error_Description
Description of the error (if an error occurred)
 
Prepare_Map_Time
Time to compile and prepare the map for running
 
Total_Error_Count
Total number of errors in the transformation
 
Total_records_Deleted
Number of records deleted in the transformation
 
Total_records_Discarded
Number of records discarded in the transformation
 
Total_Records_Inserted
Number of records inserted in the transformation
 
Records_Read
Number of records read in the transformation
 
Total_Rejected
Number of records rejected in the transformation
 
Total_records_Updated
Number of records updated in the transformation
 
Total_Records_Written
Number of records written in the transformation
 
Total_Records_Rejected
Number of records rejected in the transformation
 
Map_Version
Version number of the map
 
Execution_End_Time
Time at the end of the execution of the transformation.
 
Error_Time
Time the error occurred (if an error occurred)
 
Execution_Init_Time
Time spent in initialization of transformation
 
Transformation_Last_Error
Last error message
 
Error_Number
The error number associated with the error (if an error occurred)
 
Organization
Organization that owns the product
 
Owner
Owner of the product
 
Product
Product that ran the process
 
Serial_Number
Serial number of the product
 
Build_Version
Build version of the product (example, 8.12.1.29 2006020116)
Transformation_Last_Error_Final
 
Represents an error that has occurred in a stand-alone transformation (not a transformation called in a process step). See the related note below this table.
 
Transformation_Last_Error_Final_Time
Last time the error was logged
 
Last_Error_Code
Next to last error code that was logged
 
Transformation_Last_Error_Final_Info
Last error message
 
Transformation_Last_Error_Time
Next to last error occurred and was logged
 
Transformation_Last_Error_Info
Next to Last information about the last error message
Transformation_Last_Error_Final record type
The Transformation_Last_Error_Final record type picks up the last two errors that are logged for a transformation only execution. This record type is logged only for transformations that contain errors.
Two more error messages are logged at the end of the transformation run if a transformation is run stand-alone and there is an error. This information appears in the Transformation_Totals layout if the transformation is run in a process. To find out if there is any data in this layout, do one of the following:
Check the Error_Number in the Transformation_Totals
Use the events similar to any other multirecord layout that may or may not have records of a particular type