DC 12.3 | Process and Transformation LogFile Parser

User Guide > Map Connectors > Source and Target Map Connectors > Process and Transformation LogFile Parser

Was this helpful?

Process and Transformation LogFile Parser

This connector uses a .cxl script as the default extract source schema to parse log files. You can parse log files that contain transformation or process errors and messages. The log files can contain a single run of a transformation or process or multiple runs (of transformations, processes, or a combination of both).

Users are often interested in manipulating the log file data in various ways. The following are examples of possible use cases:

• Load the contents of the log file into a database

• Create reports with portions of the log file

• Get a different view of the log

• Create custom logs from the main log

• Mine the log file for specific values, and then use those values in a transformation or a process step

Connectivity Pointers

Here are some things to remember when using the LogFile Parser connector:

• The Extract Schema is already selected for you. This default extract script file is called ProcessTransformationLogFile.cxl.

• Records do not necessarily appear in the order read by the integration platform. For a complete list of the record types and fields, see Record Types.

• Since the log file parser parses multiple logs that are appended to one another, there may or may not be steps of a certain type in a run of a different process. This means that logic may be needed to write out record types at the proper places in the target. If not, then you could have records that are by themselves or you could even associate records that do not belong together.

• In a process, the last thing to be read is the Process_Execution_Stats. The log file format for the LogFile Parser map example is shown below:

Example: Log File Format for the LogFile Parser Map

Process Start1

SQL Step1 Begin

SQL Step1 End

Process Stop1

Process Start2

SQL Step2 Begin

SQL Step2 End

Process Stop2

Process Start3

SQL Step3 End

Process Stop3

Here is how it is read into the map:

SQL Step1 Begin

SQL Step2 End

Process1 Information

SQL Step2 Begin

SQL Step2 End

Process2 Information

SQL Step3 End

Process3 Information

Here is the XML structure we want to create for the example:

<Process_Information>

<SQL_Step>

</Process_Information>

<Process_Information>

<SQL_Step>

</Process_Information>

<Process_Information>

<SQL_Step>

</Process_Information>

</Information>

Connector Parts

Connector parts are the fields you configure to connect with a data source or target. The fields that are available depend on the connector you select. This is a source only connector. For a list of all parts for source connectors, see Specifying Source Connector, Parts, and Properties.

Property Options

You can specify the following source properties:

Property	Description
ByteOrder	Allows you to specify the byte order of Unicode (wide) characters. The default is Auto and is determined by the architecture of your computer. The list box options are Auto (default), Little Endian and Big Endian. Little Endian byte order is generally used by Intel and DEC Alpha machines and places the least significant portion of a byte value in the left portion of the memory used to store the value. Big Endian byte order is used by IBM 370 computers, Motorola microprocessors and most RISC-based systems and stores the values in the same order as the binary representation.
ProgramVariables	Allows you to set or override program variable values. There is no default.
ReportReadingScriptFile	The script file, usually with a .cxl extension. It only appears here for verification purposes. The default .cxl file is ProcessTransformationLogFile.cxl.
StartOffset	If your source data file starts with characters that need to be excluded from the transformation, set the StartOffset option to specify at which byte of the file to begin. The default value is zero. The correct value may be determined by using the Hex Browser. For details, search for "hex values" in the documentation. Note: This property is set in number of bytes, not characters.
Encoding	Sets the character encoding for source and target files. Note: Shift-JIS encoding is used only in Japanese operating systems. UCS-2 is no longer a valid encoding name, but you may use UCS2 by changing the string value in the XML file.

Record Types

The 29 record types that are supported in this connector represent data collected from the log for the following process and transformation elements:

• Processes

• Process steps

• Process sessions

• Transformations

• Transformation totals

Record Types for Process Steps

The following steps are specific to running a process. The listed record types are NOT part of a transformation-only log file.

Record Type Name	Field Name for Record Type	Description
Start_Step_End		Represents the data for a successful or unsuccessful Start Step in a process log file
	Start_Step_Status	Status of the step after the run (for example, complete, failed)
	Start_Step_Begin_Time	Time the Start Step begins
	Start_Step_End_Time	Time the Start Step ends
	Error_Description	Description of the error (if an error occurred)
	Error_Time	The time the error occurred (if an error occurred)
	Start_Step_Last_Error	Return code from the step
	Error_Number	Error number associated with the error (if an error occurred)
Stop_Step_End		Represents the data for a successful or unsuccessful Stop Step in a process log file
	Stop_Step_Status	Status of the step after the run (complete, failed, etc)
	Stop_Step_Begin_Time	Time at the start of the Stop step
	Stop_Step_End_Time	Time at the end of the Stop step
	Error_Description	Description of the error (if an error occurred)
	Error_Time	Time the error occurred (if an error occurred)
	Stop_Step_Last_Return	Return Code from the step
SQL_Step		Represents a SQL Step in the process starting and stopping in the log file
	SQL_Step_Action	Begin (Start) or End (Stop) for the step
	Error_Time	Time error occurred (if an error occurred)
	SQL_Step_Name	Name of the step in the process
	Error_Number	Error number associated with the error (if an error occurred)
	SQL_Step_Time	Time the SQL_Step_Action occurred
	Error_Description	Description of error (if an error occurred)
	SQL_Step_Status	Failed, successful, etc.
	SQL_Step_Last_Return	Last return code returned from the Step
DTS_Step_End		Represents a DTS step run in the process log file
	DTS_Step_Status	Status of the step after the run (complete, failed, etc)
	DTS_Step_Name	Name of the step in the process
	DTS_Step_End_Time	Time at the end of the DTS step
	DTS_Step_Begin_Time	Time at the start of the DTS step
	DTS_Step_Last_Return	Last Return code from the step
Decision_Step_End		Represents a Decision Step completing in the process log file
	Decision_Step_Result	True or False
	Decision_Step_Name	Name of the Decision step
	Decision_Step_End_Time	Step end time
	Decision_Step_Begin_Time	Step begin time
Application_Step_End		Represents an Executable (Exe) Step running in the process log file
	Application_Step_Status	Status of the step after the run (for example, complete, failed)
	Application_Step_Name	Name of the step in the process
	Application_Step_End_Time	Time at the end of the Application step
	Application_Step_Begin_Time	Time at the start of the Application step
	Application_Step_Last_Return	The last return code from the Step
SubProject_Step_End		Represents a Sub-Process step running in a process log file
	SubProject_Step_Status	Status of the step after the run (for example, complete, failed)
	SubProject_Step_Name	Name of the step in the process
	SubProject_Step_End_Time	Time at the end of the step
	SubProject_Step_Begin_Time	Time at the start of the step
	SubProject_Step_Last_Return	Last return code from the step
Queue_Step_End		Represents the execution of a Queue Step in the process log file
	Queue_Step_Status	Status of the step after the run (for example, complete, failed)
	Queue_Step_Name	Name of the step in the process
	Error_Description	Description of the error (if an error occurred)
	Queue_Step_End_Time	Time at the end of the Queue step
	Queue_Step_Start_Time	Time at the start of the Queue step
	Error_Time	Time error occurred (if an error occurred)
	Queue_Step_Last_Error	Last error code returned from the step
	Error_Number	Error number associated with the error (if an error occurred)
Scripting_Step_End		Represents the execution of a Scripting step in a process log file
	Scripting_Step_Status	Status of the step after the run (for example, complete, failed)
	Scripting_Step_Name	Name of the step in the process
	Error_Description	Description of the error (if an error occurred)
	Scripting_Step_End_Time	Time at the end of the Scripting step
	Scripting_Step_Begin_Time	Time at the start of the Scripting step
	Error_Time	Time error occurred (if an error occurred)
	Scripting_Step_Last_Error	Last error code returned from the step
	Error_Number	Error number associated with the error (if an error occurred)
Invoker_Step_End		Represents running the Invoker Step in the process log file
	Invoker_Step_Status	Status of the step after the run (for example, complete, failed)
	Invoker_Step_Name	Name of the step in the process
	Error_Time	Time the error occurred (if an error occurred)
	Invoker_Step_End_Time	Time at the end of the Invoker step
	Invoker_Step_Begin_Time	Time at the start of the Invoker step
	Error_Number	Error number associated with the error (if an error occurred)
	Error_Description	Description of the error (if an error occurred)
DocValidator_Step_End		Represents a Document validator Step running in the process log file
	DocValidator_Step_Status	Status of the step after the run (for example, complete, failed)
	Error_Time	Time the error occurred (if an error occurred)
	DocValidator_Step_Info	Information about the step
	DocValidator_Step_Error_Description	Error Description Text in the log about the Step
	DocValidator_Step_Name	Name of the step in the process
	DocValidator_Step_End_Time	Time at the end of the DocValidator step
	Error_Number	Error number associated with the error (if an error occurred)
	DocValidator_Step_Error_Number	Error number associated with the DocValidator step
	DocValidator_Step_Begin_Time	Time the DocValidator step starts
	Error_Description	Description of the error (if an error occurred). In this case, possibly more error description
XSLTTransformer_Step_End		Represents an XSLT Step running in the process log file
	XSLTTransformer_Step_Status	Status of the step after the run (complete, failed, Error Descriptions, etc)
	XSLTTransformer_Step_Name	Name of the step in the process
	XSLTTransformer_Step_End_Time	Time at the end of the XSLT Transformer step
	XSLTTransformer_Step_Begin_Time	Time at the start of the XSLT Transformer step
	XSLTTransformer_Step_Last_Error	Last Error code from the Step
Project_Execution_Stats		Represents the statistics from a process running in the log file. In a process, this is the last record type to be read
	Project_Execution_Time	Amount of time it took to execute the process
	Serial_Number	Serial Number for the Product
	Project_File	Process file that was run
	Project_Execution_Status	Status of the step after the run (complete, failed, etc)
	Project_Execution_Begin_Time	Time at the start of the process execution
	Product	Product that ran the process
	Prepare_Map_Time	Time to prepare any maps that were executed in the process run
	Owner	Owner of the Product
	Organization	Company that owns the product
	Build_Version	Build version of the product (for example, 8.12.1.14 2006061416)
	Project_Execution_Finish_Time	Time at the end of the execution of the process
	Project_Init_Start_Time	Initialization time
	Project_Execution_End_Time	Time at the end of the execution of the process
Transformer_Step_End		Represents the running of a Transformer Step in the process log file
	Transformer_Step_End_Time	Time at the end of the Transformer step
	Transformer_Step_Last_Error	Last error code from the step
	Transformer_Step_Status	Status of the step after the run (for example, complete, failed)
	Transformer_Step_Name	Name of the transformer step in the process
	Transformer_Step_Begin_Time	Time at the start of the Transformer step
Skipped		Represents any steps in the process that are disabled and are therefore skipped over during the run
	Skipped_Time	Time to skip the step
	Skipped_Step	Type of step skipped
	Skipped_Step_Name	Name of the skipped step
Iterator_Step_End		Represents an Iterator Step run in the process log file
	Iterator_Step_Status	Status of the step after the run (for example, complete, failed)
	Iterator_Step_Name	Name of the step in the process
	Error_Description	Description of the error (if an error occurred)
	Iterator_Step_End_Time	Time at the end of the Iterator step
	Iterator_Step_Begin_Time	Time at the start of the Iterator step
	Error_Time	Time the error occurred (if an error occurred)
	Iterator_Step_Last_Error	Last error code from the Step
	Error_Number	Error number associated with the error (if an error occurred)
Aggregator_Step_End		Represents the running of an Aggregator Step in the process log file
	Aggregator_Step_Status	Status of the step after the run (for example, complete, failed, etc)
	Error_Description	Description of the error (if an error occurred)
	Aggregator_Step_Name	Name of the step in the process
	Aggregator_Step_End_Time	Time at the end of the Aggregator step
	Error_Time	Time the error occurred (if an error occurred)
	Aggregator_Step_Begin_Time	Step End Time
	Aggregator_Step_Last_Error	Last error code from the Step
	Error_Number	The error number associated with the error (if an error occurred)
Transformation_Step_End		Represents the running of a transformation step in the process log file
	Transformation_Step_Name	Name of the step in the process
	Transfrmation_Step_Begin_Time	Start Time
	Transformation_Step_End_Status	Status of the step after the run (for example, complete, failed)
	Transformation_Step_End_Time	Time at the end of the transformation step
	Error_Description	Description of the error (if an error occurred)
	Error_Time	Time the error occurred (if an error occurred)
	Transformation_Step_Last_Error	Last error returned from the step
	Error_Number	Error number associated with the error (if an error occurred)

Record Types for Process Sessions

These are sessions that are specific to running a process. They are not be part of a tranformation-only log file or a process log file that does not use process sessions.

Record Type Name	Field Name	Description
Queue_Session		Represents a Queue session that was used in the process
	Queue_Session_Action	Begin (Start) or End (Stop) for the session
	Error_Number	Error number associated with the error (if an error occurred)
	Queue_Session_Name	Name of the session in the process
	Error_Description	Description of the error (if there was an error
	Queue_Session_Time	Time the Queue_Session_Action occurred
	Error_Time	Time the error occurred (if an error occurred)
	Queue_Session_Last_Return	Last return code from the session.
SQL_Session		Represents a SQL Session used in the process
	SQL_Session_Action	Begin (Start) or End (Stop) for the session
	Error_Time	Time the error occurred (if an error occurred)
	SQL_Session_Name	Name of the Session in the process
	Error_Number	Error number associated with the error (if an error occurred)
	SQL_Session_Time	Time the SQL_Session_Action occurred
	Error_Description	Description of the error (if an error occurred)
	SQL_Session_Last_Return	Last return code from the session
Invoker_Session		Represents an Invoker Session used in the process
	Invoker_Session_Action	Begin (Start) or End (Stop) for the session
	Invoker_Session_Name	Name of the session in the process
	Invoker_Session_Time	Time the Invoker_Session_Action occurred
	Invoker_Session_Last_Return	Last return code from the Session
Aggregator_Session		Represents an Aggregator session used in the process
	Aggregator_Session_Action	Begin (Start) or End (Stop) for the session
	Aggregator_Session_Name	Name of the session in the process
	Aggregator_Session_Time	Time the Aggregator_Session_Action occurred
	Aggregator_Session_Last_Return	Last return code from the session
Iterator_Session		Represents an Iterator session that was used in the process
	Iterator_Session_Action	Begin (Start) or End (Stop) for the session
	Iterator_Session_Name	Name of the session in the process
	Iterator_Session_Time	Time the Iterator_Session_Action occurred
	Iterator_Session_Last_Return	Last return code from the session
Transformer_Session		Represents a Transformer session used in the process
	Transformer_Session_Action	Begin (Start) or End (Stop) for the session
	Transformer_Session_Name	Name of the session in the process
	Transformer_Session_Time	Time the Transformer_Session_Action occurred
	Transformer_Session_Last_Return	Last return code from the session

Record Types for Process and Transformation Messages

Messages are collected separately from other steps and sessions. This is because they may or may not have a correlation to the step that is precedes or follows.

The following errors are collected with the step or session that caused the error because the error can reliably be associated with that step or session.

Record Type Name	Field Name	Description
Debug_Messages		Represents a Debug message in the log file
	Debug_Message	Text of the debug message
	Debug_Message_Time	Time the message was logged
Information_Messages		Represents an Informational message logged
	Information_Time	Time the informational message was logged
	Information_Message	Text of the message that was logged
Warning_Message		Represents a warning message written to the log file
	Warning_Message	Text written to the log
	Warning_Message_Last_Return	Last return code from the logged message
	Warning_Message_Time	Time the message was logged
	Warning_Message_Info	More information about the warning

Record Types for Transformation-Only Runs

This information will be logged for a stand-alone transformation, a transformation in a process, or both. It is the collected information about the transformation-specific run.

Record Type Name	Field Name	Description
Transformation_Totals		Represents the information about a transformation that was run stand-alone or as a transformation in a process step.
	Total_records_Buffered	Total number of records buffered.
	Execution_Time	Time to run the transformation.
	Execution_Begin_Time	Time at the beginning of the transformation.
	Execution_End_Status	Time at the end of the transformation.
	Map	Map and location of the transformation being run. xmldb:ref:/// .tf.xml
	Error_Description	Description of the error (if an error occurred)
	Prepare_Map_Time	Time to compile and prepare the map for running
	Total_Error_Count	Total number of errors in the transformation
	Total_records_Deleted	Number of records deleted in the transformation
	Total_records_Discarded	Number of records discarded in the transformation
	Total_Records_Inserted	Number of records inserted in the transformation
	Records_Read	Number of records read in the transformation
	Total_Rejected	Number of records rejected in the transformation
	Total_records_Updated	Number of records updated in the transformation
	Total_Records_Written	Number of records written in the transformation
	Total_Records_Rejected	Number of records rejected in the transformation
	Map_Version	Version number of the map
	Execution_End_Time	Time at the end of the execution of the transformation.
	Error_Time	Time the error occurred (if an error occurred)
	Execution_Init_Time	Time spent in initialization of transformation
	Transformation_Last_Error	Last error message
	Error_Number	The error number associated with the error (if an error occurred)
	Organization	Organization that owns the product
	Owner	Owner of the product
	Product	Product that ran the process
	Serial_Number	Serial number of the product
	Build_Version	Build version of the product (example, 8.12.1.29 2006020116)
Transformation_Last_Error_Final		Represents an error that has occurred in a stand-alone transformation (not a transformation called in a process step). See the related note below this table.
	Transformation_Last_Error_Final_Time	Last time the error was logged
	Last_Error_Code	Next to last error code that was logged
	Transformation_Last_Error_Final_Info	Last error message
	Transformation_Last_Error_Time	Next to last error occurred and was logged
	Transformation_Last_Error_Info	Next to Last information about the last error message

Transformation_Last_Error_Final record type

The Transformation_Last_Error_Final record type picks up the last two errors that are logged for a transformation only execution. This record type is logged only for transformations that contain errors.

Two more error messages are logged at the end of the transformation run if a transformation is run stand-alone and there is an error. This information appears in the Transformation_Totals layout if the transformation is run in a process. To find out if there is any data in this layout, do one of the following:

• Check the Error_Number in the Transformation_Totals

• Use the events similar to any other multirecord layout that may or may not have records of a particular type

Last modified date: 10/22/2024