Designing and Executing Processes : Process Steps and Associated Components : Queue Step
 
Share this page                  
Queue Step
Queues provide access to applications or middleware that produce and consume data in the form of messages. Queues encapsulate the information necessary to establish a connection with the messaging system. Queue components can access data and services provided by message queuing systems. This includes the ability to read and write messages to and from a queue as well as transactional support. Queue components can provide access to common message queuing systems such as FTP Queue, File Folder Queue, Email Queue, JMS Queue, Oracle AQ, WebSphere MQ, and MSMQ. They can also be used to provide a queuing interface to a nonmessage-based system, like a file system. Typical queuing systems perform a destructive read. In other words, once a message is read, it is removed from the queue.
For each type of queue component, you create a corresponding Queue step. The Queue step allows you to select a specific instance of a component and to specify the Action for the component to execute.
Queue Step Properties
After adding a /download/attachments/24976145/Queue_step.png?version=1&modificationDate=1490033775548&api=v2 Queue step from the Toolbox Palette to the canvas, you can double-click the step on the canvas and specify the following step properties in the Properties tab displayed at the bottom.
Name
Description
Name
Unique name for the step.
Description
Description of the step (optional).
Enabled
Select this option to enable the step for execution. If it is not enabled, then the step on the canvas is greyed-out.
This option is useful when troubleshooting a process, especially when there are numerous steps in the process.
Error Handling
Select either of the following:
• Ignore Error - The remaining steps in the current process are executed even if this step aborts due to an error. This is selected by default.
• Abort Process - All the steps after the current step are skipped or the entire process is aborted (depending on how you have the error logging options set) if this step aborts due to an error.
Select Queue
Select a Queue component instance. These instances are displayed when you add message components in the Configuration tab. For more information, see Adding Message Components in Configuring Processes.
Action
Select the action for the queue instance.
Parameter
Parameters for the selected action. Specify values for the parameters.
Property
Properties for the selected action. Specify values for the properties.
Note:  The Actions, Parameters, and Properties are specific to the component type. For information about this, based on the component instance selected in the Select Queue drop-down list, click the component type in the Queue Step Components section below.
Queue Step Components
• POP3 Email Queue
• IMAP Email Queue
• File Folder Queue
• FTP Queue
• JMS Queue
• Kafka Writer Queue
• Websphere MQ Queue
• MSMQ Queue
POP3 Email Queue
POP3 Email Queue supports receiving text and binary attachments. DataConnect supports POP3 Email Queue versions 1.0.2 and 1.3.0.
The following table provides the supported content types.
Content Type
Details
text/*
• text
• HTML
image/*
• .gif
• .jpg
application/*
• All applicable applications, such as .xls, .zip, and .doc
The email components support encoding in all character sets when an email is sent and received. The message header and body are encoded in the character set specified by the MIME. In addition, the email components support only Base64 and other MIME encoding. Enriched encoding is not supported.
Note:  The POP3 Email Queue component implicitly trust the certificates presented by the email servers that require SSL or TLS connections, regardless of the issuer.
POP3 Email Queue Properties
 
Property
Default Value
Description
User Name
 
User name to connect to the POP3 email server.
Password
 
Password to connect to the POP3 email server.
Server Name
 
Name of the POP3 email server
Port
110
Port to be used to connect to the server.
Note:  If the default port is not used, then the component assumes a secure (SSL or TLS) connection is required.
ConnectionTimeout
90
Connection time out value in seconds.
Retry Count
3
Number of attempts the server tries to establish connection when the connection times out or fails.
From Contains
 
From email address
To Contains
 
To email address
Subject Contains
 
Subject value
Browse Mode
false
Select true or false. If true is selected, the email message remains on the server even after downloading it.
Download
UNREAD
Download the read and/or unread messages from the server:
• ALL - Download both read and unread messages from the server.
• READ - Download only read messages from the server.
• UNREAD - Download only unread messages from the server.
Note:  This property is displayed only for POP3 Email Queue version 1.0.2 only.
Encoding String
ISO-8859-1
Encoding string required for the email from the list.
Mark Failed Messages as Seen
False
Select True or False:
• True - Marks unprocessed messages as seen. This value provides a method to bypass messages that the component cannot handle by marking them as read.
• False - Leaves unprocessed messages as unseen.
This property is displayed only for POP3 Email Queue version 1.3.0.
Always Create Attachment Messages
False
Select True or False:
• True - Component encodes any primary content that may be an email attachment, and then includes it into a new DJMessage attachment. The original content normally has a MIME type indicating that it is binary data.
• False - Component places any primary content that may be an email attachment in the main DJMessage. The original content normally has a MIME type indicating that it is text data.
Note:  If the email is not multi-part MIME- formatted, then the content is included in the main DJMessage regardless of the MIME type.This property is displayed only for POP3 Email Queue version 1.3.0.
Tip:  You can set filters or flags using the From Contains, To Contains, and Subject Contains properties. However, the filters do not work if the server does not support the same.
Supported Actions
 
Action
Description
Connect
Opens a persistent connection to the database.
GetMessage
Retrieves message into a message object.
Disconnect
Closes an existing database connection and performs any clean up actions if required.
Supported Action Parameters
 
Action
Parameter
Description
GetMessage
Message
Name for the incoming message
Supported Action Properties
There are no supported action properties.
Errors
Errors are generated when email messages cannot be sent or received due to various reasons. Typical errors include the following:
• Connection timeout
• Unable to disconnect from the email server
• Errors while sending email messages
• Errors while Receiving email messages
• Miscellaneous or unknown errors
All errors are written to an error log, which you can read to diagnose and troubleshoot the problem. The error messages specify whether a problem occurred while receiving or sending.
The error log also contains error codes that help you to identify and understand the cause of the error.
The following table describes error codes generated while sending or receiving email.
Code
Description
19
Unable to connect to the mail server
27
Unable to disconnect from the mail server
8
Unable to send email
4
Unable to receive email
50
Unknown error
1
End of file
12
No more messages to download
0
Messages sent or received successfully
IMAP Email Queue
The IMAP Email Queue supports receiving text and binary attachments.
The supported content types are outlined here.
Content Type
Details
text/*
• text
• HTML
image/*
• .gif
• .jpg
application/*
• All applicable applications, such as .xls, .zip, and .doc
The email components support encoding in all character sets when an email is sent and received. The message header and body are encoded in the character set specified by the MIME. In addition, the email components support only Base64 and other MIME encoding. Enriched encoding is not supported.
Note:  The IMAP Email Queue component implicitly trust the certificates presented by the email servers that require SSL or TLS connections, regardless of the issuer.
Instance Properties
IMAP Email Queue has the following instance properties.
Property
Default Value
Description
User Name
 
User name to connect to the IMAP email server.
Password
 
Password to connect to the IMAP email server
Server Name
 
Name of the IMAP email server
Port
143
Port to be used to connect to the server.
Note:  If the default port is not used, then the component assumes a secure (SSL or TLS) connection is required.
ConnectionTimeout
90
Connection time out value in seconds.
Retry Count
3
Number of attempts the server tries to establish connection when the connection times out or fails.
From Contains
 
From email address
To Contains
 
To email address
Subject Contains
 
Subject value
Folder Name
 
Folder that resides in the IMAP server. If you specify a folder name, all the emails are delivered to the specified folder.
Typically, an email is downloaded to the INBOX folder unless you specify another location. Examples of specific folders are INBOX.myemail, INBOX.general, and so on.
Browse Mode
TRUE
If TRUE is selected, the email message remains on the server even after downloading it.
Download
UNREAD
Download the read and/or unread messages from the server:
• ALL - Download both read and unread messages from the server.
• READ - Download only read messages from the server.
• UNREAD - Download only unread messages from the server.
Encoding String
ISO-8859-1
Encoding string required for the email from the list.
Mark Failed Messages as Seen
False
Select True or False:
• True - Marks unprocessed messages as seen. This value provides a method to bypass messages that the component cannot handle by marking them as read.
• False - Leaves unprocessed messages as unseen.
Always Create Attachment Messages
False
Select True or False:
• True - Component encodes any primary content that may be an email attachment, and then includes it into a new DJMessage attachment. The original content normally has a MIME type indicating that it is binary data.
• False - Component places any primary content that may be an email attachment in the main DJMessage. The original content normally has a MIME type indicating that it is text data.
Note:  If the email is not multi-part MIME- formatted, then the content is included in the main DJMessage regardless of the MIME type.
Tip:  You can set filters or flags using the From Contains, To Contains, Subject Contains, or Download properties. It is recommended to download only unread messages. However, filters or flags do not work if the server does not support the same.
Supported Actions
 
Action
Description
Connect
Opens a persistent connection to the database.
Disconnect
Closes an existing database connection and performs any clean up actions if required.
GetMessage
Retrieves message into a message object.
Supported Action Parameters
 
Action
Parameter
Description
GetMessage
Message
Name for the incoming message
Supported Action Properties
There are no supported action properties.
Errors
Errors are generated when email messages cannot be sent or received due to various reasons. Typical errors include the following:
• Connection timeout
• Unable to disconnect from the email server
• Errors while sending email messages
• Errors while Receiving email messages
• Miscellaneous or unknown errors
All errors are written to an error log, which you can read to diagnose and troubleshoot the problem. The error messages specify whether a problem occurred while receiving or sending. The error log also contains error codes that help you to identify and understand the cause of the error.
This table describes error codes generated while sending or receiving email.
Code
Description
19
Unable to connect to the mail server
27
Unable to disconnect from the mail server
8
Unable to send email
4
Unable to receive email
50
Unknown error
1
End of File
12
No more messages to download
0
Messages sent or received successfully
File Folder Queue
File Folder Queue allows a file folder to be treated like a message queue. The files in the folder can be retrieved into message objects by using the GetMessage action in a Queue step. Message objects can also be written to the folder as files using the PutMessage action. The properties of the session control the files that are read and the names of the files that are written, as well as whether to preserve the message properties.
File Folder Queue Properties
 
Property
Default Value
Description
User name
-
User name to log in to the system where the file folder is available.
Password
-
Password to log in to the system.
Server Name
-
Name of the system where the file folder resides.
Directory
 
Folder for reading and writing files.
Working Directory
-
Directory name used to store temporary files that are created when retrieving files with the GetMessage action. If this property is not specified, the connector creates a directory called DJFTFile Folder Queue in the directory specified for the Directory property. This new directory is used to store temporary files.
Deserialize
False
If a message object was written to a folder with the Serialize property set, the message properties are stored in a separate file with an .hdr extension. When reading a file from a folder using the GetMessage action, the Deserialize property indicates that the message properties must be read from the .hdr file.
Serialize
False
When a message object is written to a file with the PutMessage action, only the message body is written. The Serialize property indicates whether to write the message properties to a separate file with an .hdr extension.
Pattern
*
Pattern used to match files to be read using the GetMessage action.
GetMsgClass
dat
The file extension that is appended to the file name when reading a file with the GetMessage Action.
PutMsgClass
dat
File extension that is appended to the file name generated by the FileFormat property when writing a message object to a file with the PutMessage action.
MessageTimeOut
5
Indicates the time to wait (in seconds) to retrieve the first file from the folder when retrieving files with the GetMessage action. The files to be retrieved are determined by the Pattern property.
The MessageTimeOut property works with the SleepInterval property . The File Folder Queue tries to read the first file then sleep for the specified sleep interval. This continues based on the value in the MessageTimeOut option.
PollingTimeOut
5
Indicates the time to wait to retrieve all of the files from the folder when retrieving files with the GetMessage action. The files to be retrieved are determined by the Pattern property. The PollingTimeOut property works with the PollingTimeOutType property.
PollingTimeOutType
Minutes
Type of the polling timeout value. The options available are:
• Years
• Months
• Days
• Hours
• Minutes
• Seconds
SleepInterval
1000
Indicates the time to sleep when no messages are available to process. The sleep interval is in milliseconds.
BrowseMode
False
If this property is set to true, the messages read from the folder are not removed. Else, the files are deleted after processing.
FilenameFormat
MessageFile%n
When writing a message object to a file using the PutMessage action, this format is used to create the file name. The following format specifications can be used to build the format. Each specification begins with a percent sign (%). The specifiers represent properties on the message object that are created by the File Folder Iterator. Any or all of the specifications can be included in any order.
Note:  The format must generate unique names for the files.
The supported formats are:
• MessageFile%n
• %i TransferId
• %p PieceNumber
• %s SourceName
• %h TargetHostName
• %t TargetName
• %d Transport Start Date/Time
• %n Number starting with 0 (zero)
• %c Current date/time stamp displaying number of seconds elapsed since midnight GMT January 1, 1970 (value always contains 10 digits)
Examples
TargetName: TargetFFQFile
PieceNumber: 5
FileFormat: %t%p
MessageClass: dat
Filename: TargetFFQFile5.dat
 
 
Best Practice — The %n and %c format specifications enable you to create unique file names for generic messages. These should be used together to ensure greatest efficiency.
If only %n is used in the file name specification, for example, MessageFile%n, the connector tries to assign the file name "MessageFile0" when a message is created. If this is unique it is used, but if it already exists in the specified directory, %n is incremented by 1 and this is checked against existing files. This continues until a unique file name is found. This means that the more messages that are created, the longer it takes for a unique name to be found.
If only %c is used, the process is limited to writing one file per second as the file name it tries to assign for the new message is not unique until the timestamp has advanced. Therefore, use a combination of %n and %c. For example, MessageFile%c%n creates as many messages each second as required and %n is reset to 0 each time the timestamp changes.
Source Encoding
ENC_OEM or 0
Character set of the source data. This indicates using the default character set for the system.
The ENC_ names can be used in expressions for setting the encoding value. On the integration engine command line, the number must be used.
Binary Indicator
False
Indicates whether the files being read must be opened in binary mode.
Supported Actions
 
Action
Description
Connect
Opens a persistent connection to the database.
Disconnect
Closes an existing database connection and performs any clean up actions if required.
GetMessage
Retrieves message into a message object.
PutMessage
Writes a message object into the queue.
Supported Action Parameters
 
Action
Parameter
Description
GetMessage
PutMessage
Message
Name for the incoming message
Supported Action Properties
There are no supported properties for the actions.
FTP Queue
The FTP Queue enables you to send and receive files using a secure or non-secure FTP connection. It allows a folder on an FTP site to be considered like a message queue. The files in the folder can be retrieved into message objects by using the GetMessage action in a Queue step.
Message objects can also be written to the folder as files using the PutMessage action. The properties control the files that are read and the names of the files that are written.
The primary functions of FTP Queue are:
• Connect to and disconnect from a secure or unsecure FTP server
• Retrieve a file from an FTP server and insert it in a message object
• Put a message object into the selected file on an FTP server
This component has the following versions:
• FTP Queue 1.0.0 - Provides an non-secure connection. While version 2.0.0 is recommended, 1.0.0 is supported for backward compatibility with existing processes.
• FTP Queue 2.0.0 and 4.1.0 - Provides either a secure or non-secure connection using one of the three FTP types:
– FTP (non-secure)
– SFTP (secure FTP over SSH)
– FTPS (secure FTP over SSL (certificate-based authentication))
Active and Passive Modes
FTP Queue 1.0.0 tries passive mode first. If it fails to connect, then it tries active mode.
FTP Queue 2.0.0 and 4.1.0 uses active mode for processing.
Encoding
In secure connections, FTP Queue 2.0.0 and 4.1.0 uses ISO-8859-1 to encode binary data.
FTP Queue 1.0.0 Properties
 
Property
Default Value
Description
User Name
 
User name to connect to the server.
Password
 
Password to connect to the server.
FTPSite
 
Name of the server to connect to.
Type the FTP server name or IP address, port number, and directory in the following format:
FTP_server:port/directory_name
The port number and directory name are optional. If you specify the port number, the component uses the default port number for the selected FTP server type.
Serialize
False
When a message object is written to a file with the PutMessage action, only the message body is written. The Serialize property indicates whether to write the message properties to a separate file with an .hdr extension.
Deserialize
False
If a message object was written to a folder with the Serialize property set, the message properties are stored in a separate file with an .hdr extension. When reading a file from a folder using the GetMessage action, the Deserialize property indicates that the message properties must be read from the .hdr file.
RetryCount
0
Number of times to retry if there is a failure to connect to the FTP site.
FilenameFormat
MessageFile%n
When writing a message object to a file using the PutMessage action, this format is used to create the file name. The following format specifications can be used to build the format. Each specification begins with a percent sign (%). The specifiers represent properties on the message object that are created by the File Folder Iterator. Any or all of the specifications can be included in any order.
Note:  The format must generate unique names for the files.
The supported formats are:
• MessageFile%n
• %i TransferId
• %p PieceNumber
• %s SourceName
• %h TargetHostName
• %t TargetName
• %d Transport Start Date/Time
• %n Number starting with 0 (zero)
• %c Current date/time stamp displaying number of seconds elapsed since midnight GMT January 1, 1970 (value always contains 10 digits)
Examples
TargetName: TargetFFQFile
PieceNumber: 5
FileFormat: %t%p
MessageClass: dat
Filename: TargetFFQFile5.dat
 
 
Best Practice — The %n and %c format specifications enable you to create unique file names for generic messages. These should be used together to ensure greatest efficiency.
If only %n is used in the file name specification, for example, MessageFile%n, the connector tries to assign the file name "MessageFile0" when a message is created. If this is unique it is used, but if it already exists in the specified directory, %n is incremented by 1 and this is checked against existing files. This continues until a unique file name is found. This means that the more messages that are created, the longer it takes for a unique name to be found.
If only %c is used, the process is limited to writing one file per second as the file name it tries to assign for the new message is not unique until the timestamp has advanced. Therefore, use a combination of %n and %c. For example, MessageFile%c%n creates as many messages each second as required and %n is reset to 0 each time the timestamp changes.
GetMsgClass
dat
The file extension that is appended to the file name when reading a file with the GetMessage Action.
PutMsgClass
dat
File extension that is appended to the file name generated by the FileFormat property when writing a message object to a file with the PutMessage action.
Working Directory
 
Directory name used to store temporary files that are created when retrieving files with the GetMessage action.
Pattern
*
Pattern used to detect files to be read, such as File* or *95. To retrieve a specific file, type the exact file name without the extension. The extension is read from the File Extension option.
MessageTimeOut
5
Indicates the number of times to query the FTP folder to get the desired message. Default is 5. This option works with the SleepInterval option.
PollingTimeOut
5
Indicates the maximum total time to take to retrieve all files from the folder. Use this option with the Polling TimeOut Type option.
PollingTimeOutType
Minutes
Type of the PollingTimeOut value. The available options are:
• Years
• Months
• Days
• Hours
• Minutes
• Seconds
SleepInterval
1000
Time interval, in milliseconds, to wait between queries if there are no messages in the queue to process. This option works with the Message TimeOut option.
BrowseMode
False
Indicates whether the file on the FTP server from which the message is read is deleted. If Browse Mode is true, the file is not deleted.
SourceEncoding
0
Character set of the source data. This indicates using the default character set for the system.
The ENC_ names can be used in expressions for setting the encoding value. On the integration engine command line, the number must be used.
Binary Indicator
False
Indicates whether the files being read should be opened in binary mode.
Supported Actions for FTP Queue 1.0.0
 
Action
Description
Connect
Establishes a connection to an FTP server, which is useful when you are testing your configuration.
Disconnect
Breaks the connection with an FTP server, which is useful when you need to disconnect before a process is complete.
GetMessage
Connects to an FTP server, retrieves the contents of a file into a message object, then disconnects. The GetMessage action sets the SourceName message property to the name of the file from which it retrieved the message.
PutMessage
Connects to an FTP server, writes a message object to the specified file, then disconnects.
Supported Action Parameters for FTP Queue 1.0.0
 
Action
Parameter
Description
GetMessage, PutMessage
Message
Name of the message.
FTP Queue 2.0.0 Properties
 
Property
Default Value
Description
User Name
 
User name to connect to the server.
Password
 
Password to connect to the server.
Server Name
localhost
Name of the server to connect to.
Type the FTP server name or IP address, port number, and directory in the following format:
FTP_server:port/directory_name
The port number and directory name are optional. If you do not specify the port number, the component uses the default port number for the selected FTP server type.
FTP Type
FTP
Specify the type of FTP:
• FTP
• SFTP
• FTPS
Retry Count
0
Number of times to retry if there is a failure to connect to the FTP site.
Binary Indicator
False
Indicates whether the files being read should be opened in binary mode. Default is false.
Supported Actions for FTP Queue 2.0.0
 
Action
Description
Connect
Establishes a connection to an FTP server, which is useful when you are testing your configuration.
Disconnect
Breaks the connection with an FTP server, which is useful when you need to disconnect before a process is complete.
GetMessage
Connects to an FTP server, retrieves the contents of a file into a message object, then disconnects. The GetMessage action sets the SourceName message property to the name of the file from which it retrieved the message.
PutMessage
Connects to an FTP server, writes a message object to the specified file, then disconnects.
Supported Action Parameters for FTP Queue 2.0.0
 
Action
Parameter
Description
GetMessage, PutMessage
Message
Name of the message.
Supported Action Properties for FTP Queue 2.0.0
 
Action
Property
Default Value
Description
GetMessage
Pattern
*
Pattern used to detect files to be read, such as File* or *95. To retrieve a specific file, type the exact file name without the extension. The extension is read from the File Extension option.
 
File Extension
 
File extension to append to the file name when retrieving a file into a message object. Use values such as txt and asc. Do not enter the period, only the extension string.
 
Browse Mode
False
Indicates whether the file on the FTP server from which the message is read is deleted. If Browse Mode is true, the file is not deleted.
 
Message Timeout
5
Indicates the number of times to query the FTP folder to get the desired message. Default is 5. This option works with the SleepInterval option.
 
Sleep Interval
1000
Time interval, in milliseconds, to wait between queries if there are no messages in the queue to process. This option works with the Message TimeOut option.
 
Polling TimeOut
5
Indicates the maximum total time to take to retrieve all files from the folder. Use this option with the Polling TimeOut Type option.
 
Polling TimeOut Type
Minutes
Type of the PollingTimeOut value. The available options are:
• Years
• Months
• Days
• Hours
• Minutes
• Seconds
PutMessage
Filename Format
 
Format for the file name written to the message object.
Various format specifications are supported. Each specification begins with a %. Any or all specifications can be used in any order. Make sure that the format generates a unique name for each file. The default format is MessageFile%n, which produces the file name MessageFile0.
The supported format specifications are:
• %n - Inserts a number, starting with 0 (zero).
• %c - Inserts current date/time stamp, displaying number of seconds elapsed since midnight GMT January 1, 1970. The value has 10 digits.
• %s - Inserts the source file name, which can be specified as a message property.
• %t - Inserts the target file name, which can be specified as a message property.
 
File Extension
 
File extension to append to the file name when writing a message object to a file. Use values such as txt and asc. Do not enter the period, only the extension string.
FTP Queue 4.1 Properties
 
Property
Default Value
Description
User Name
 
User name to connect to the server.
Password
 
Password to connect to the server.
Server Name
localhost
Name of the server to connect to.
Type the FTP server name or IP address, port number, and directory in the following format:
FTP_server:port/directory_name
The port number and directory name are optional. If you do not specify the port number, the component uses the default port number for the selected FTP server type.
FTP Type
FTP
Specify the type of FTP:
• FTP
• SFTP
• FTPS
FTP Connection Mode
 
Connection mode is either Active (the default) or
Passive.
Note:  This is not applicable for SFTP.
Message Payload Type
 
Select File Contents (the default) or File Location:
• File Contents: If the actions are:
– If GetMessage is the , contents of the retrieved file are stored in the DJMessage.
– If PutMessage, contents of the DJMessage are sent to the remote FTP site and stored as a file.
• File Location: If actions are:
– GetMessage - File retrieved from the remote FTP site is stored directly as a file.
– PutMessage - Local file contents are sent to the remote FTP location.
Encoding Type
ISO8859-1
Select ISO8859-1 or UTF-8.
Note:  This is not applicable for SFTP.
Binary Indicator
 
False
Indicates whether the files being read must be opened in binary mode.
Supported Actions for FTP Queue 4.1.0
 
Action
Description
Connect
Establishes a connection to an FTP server, which is useful when you are testing your configuration.
Disconnect
Breaks the connection with an FTP server, which is useful when you need to disconnect before a process is complete.
GetMessage
Connects to an FTP server, retrieves the contents of a file into a message object, then disconnects. The GetMessage action sets the SourceName message property to the name of the file from which it retrieved the message.
PutMessage
Connects to an FTP server, writes a message object to the specified file, then disconnects.
Supported Action Parameters for FTP Queue 4.1.0
 
Action
Parameter
Description
GetMessage, PutMessage
Message
Name of the message.
Supported Action Properties for FTP Queue 4.1.0
 
Action
Property
Default Value
Description
GetMessage
Target Directory
 
Target directory for storing the retrieved file on the local machine.
 
Pattern
*
Pattern used to detect files to be read. To retrieve a specific file, type the exact file name with or without the extension. The extension is read from the File Extension option. The pattern is case sensitive. The Pattern properties supports the use of regular expressions. Default is *.
For example, File*, *95, File?, ?95('*' matches any character sequence and '?' matches any character).
 
File Extension
 
File extension to append to the file name when retrieving a file and inserting into a message object. Use values such as txt and asc. Do not enter the dot (.), the extension string.
 
Browse Mode
False
Indicates whether the file on the FTP server from which the message is read is deleted. If Browse Mode is true, the file is not deleted.
Message Timeout
5
Indicates the number of times to query the FTP folder to get the desired message. Default is 5. This option works with the SleepInterval option.
Sleep Interval
1000
Time interval, in milliseconds, to wait between queries if there are no messages in the queue to process. This option works with the Message TimeOut option.
Polling TimeOut
5
Indicates the maximum total time to take to retrieve all files from the folder. Use this option with the Polling TimeOut Type option.
Polling TimeOut Type
Minutes
Type of the PollingTimeOut value. The available options are:
• Years
• Months
• Days
• Hours
• Minutes
• Seconds
PutMessage
Remote Directory
 
Remote location for storing the file on the FTP Server.
 
Filename Format
 
Format for the file name written to the message object.
Various format specifications are supported. Each specification begins with a %. Any or all specifications can be placed consecutive to each other. However, you may not place text after a format specification. For example, MessageFile%n%s is valid, but MessageFile%nTest is invalid.
Make sure that the format generates a unique name for each file. The default format is MessageFile%n, which produces the file name MessageFile0.
The supported format specifications are:
• %n - Inserts a number, starting with 0 (zero).
• %c - Inserts current date/time stamp, displaying number of seconds elapsed since midnight GMT January 1, 1970. The value has 10 digits.
• %s - Inserts the source file name, which can be specified as a message property.
• %t - Inserts the target file name, which can be specified as a message property.
 
File Extension
 
File extension to append to the file name when writing a message object to a file. Use values such as txt and asc. Do not enter the period, only the extension string.
 
Use Temporary Files
 
 
 
Local File
 
Local path to the location where the file exists.
Errors
The following table provides possible FTP Queue error codes.
Error Code
Description
4
Error occurs while trying to get a message from the FTP server.
8
Error occurs while trying to add a message on the FTP server.
12
Error occurs because the GetMessage or PutMessage actions cannot find the specified file.
19
Error occurs while trying to connect to the FTP server.
27
Error occurs while trying to disconnect from the FTP server.
The process log may also contain messages related to FTP Queue.
Error Code
Description
28047
An error occurred executing a queue step.
This run-time error can result from a change in the server certificate after you deploy your integration application. View the process log for the following message:
"Error connecting to FTP Server: Could not validate server certificate. Please make sure you run TestConnect and save the process."
To resolve this error, do one of the following.
• Obtain a new certificate and add it manually to the client trust store file.
• In the process file, use the Test Connect option to connect to the FTP server so that it prompts you to accept the new certificate.
Limitations
• The file transfer is bound by the amount of physical memory and heap size.
• If you are transferring binary files, you must encode the data in the message object in Base64. When you receive files with GetMessage in binary mode, the files will be binary, and will not be encoded in Base64.
JMS Queue
JMS queues are components based on the Java Message Service. The components allows you to connect to JMS-compliant queues. The available JMS queue component is JMS Queue 2.0.0. If you want to use client jars with this component, you must use JMS 2.0.0. The JMS Queue 2.0.0 is a Java component that directly implements the JMS API.
Note:  The JMS 1.0.0 component is deprecated.
Note:  Make sure to set the Java ClassPath option in the cosmos.ini file when using JMS Queue.
Use the following information to migrate from JMS 1.0.0 to JMS 2.0.0.
 
Component Version
Display Name
Actual Option Name Used in Process
Option Type
1.0.0 and 2.0.0 (component configuration)
 
User Name
com.pervasive.di.queue.connection.userid
text
Password
com.pervasive.di.queue.connection.password
password
java.naming.factory.initial
contextfactory
text
javax.jms.connectionfactory
connectionfactory
text
java.naming.provider.url
providerurl
text
PropertiesFile
propertiesfile
file
OutputMessageType
msgtype
enum
MessageSelector
msgselector
text
Timeout
timeout
number
1.0.0 (component configuration)
 
Server Name
com.pervasive.di.queue.connection.server
text
JMSCorrelationID
jmscorrelationid
text
JMSReplyTo
jmsreplyto
text
JMSType
jmstype
text
2.0.0
(component configuration)
JarLocation
jarlocation
text
2.0.0
(configured with PutMessage action in step)
JMSCorrelationID
jmscorrelationid
text
JMSReplyTo
jmsreplyto
text
JMSType
jmstype
text
Note:  Possible values for msgtype are-javax.jms.TextMessage, javax.jms.BygtesMessage, javax.jms.StreamMessage
JarLocation is a single directory that contains all client jars.
JMS Queue Property Descriptions
 
Property
Default Value
Description
User Name
 
User name used to connect with host.
Password
 
Password used to connect with host.
Messaging Styles
PTP
Select one:
• PTP (Point-to-Point)
• Publish-Subscribe
java.naming.factory.initial
 
Fully qualified class name of the initial context factory for the JNDI service provider.
javax.jms.connectionfactory
 
JNDI lookup name of the JMS Queue connection factory.
java.naming.provider.url
 
URL for the JNDI provider-specific configuration information.
PropertiesFile
 
Specifies the name of a properties file that contains the JMS Queue configuration information. You can either specify the three properties above separately, or place them in a properties file.
OutputMessageType
javax.jms.TextMessage
Type of JMS Queue message to create when writing messages.
• javax.jms.TextMessage (default)
• javax.jms.BytesMessage
• javax.jms.StreamMessage
MessageSelector
 
JMS Queue message selector to use for reading messages.
JarLocation
 
Location of the .jar file.
Timeout
0
Maximum number of seconds to wait for a message to arrive on a queue when retrieving messages.
Supported Actions
 
Action
Description
Connect
Establishes a connection to the server.
Disconnect
Disconnects from the server.
GetMessage
Retrieves message into a message object.
PutMessage
Writes a message object into the queue.
BeginTransaction
Starts a JMS Queue transaction.
CommitTransaction
Commits a JMS Queue transaction.
RollbackTransaction
Rolls back a JMS Queue transaction.
Supported Action Parameters
 
Action
Parameter
Description
Connect
Queue
Name of the queue used to connect to.
Supported Action Properties
There are no action properties displayed for JMS Queue component.
Note:  For more information on JMS, see the JMS API documentation.
Kafka Writer Queue
Kafka is a distributed, partitioned, and replicated commit log service. Kafka maintains feeds of messages in categories called topics. The processes that publish messages to a Kafka topic are called producers. The processes that subscribe to topics and process the feed of published messages are called consumers. Kafka is run as a cluster comprised of one or more servers. Each server is called a broker. The producers send messages over the network to the Kafka cluster, which in turn sends them up to consumers.
The Kafka writer provides the ability to send data to Kafka from batch or real-time invoked processes without the expectation of handling big data volumes. Kafka provides a client Producer API for the user to write data to Kafka. The Producer API version used is 0.8.2.0.
The following features are supported:
• All Kafka producer configuration properties except for some properties that are not related to data write properties such as metric.reporters, metrics.num.samples, metrics.sample.window.ms.
• Separate actions for producer connect, data write, and producer disconnect. Therefore, you can use one connection for multiple data write actions.
• Allows you to change topics for each data write action.
Kafka Writer Queue Properties
 
Property
Default Value
Description
Bootstrap Servers
 
List of host or port pairs to use for establishing the initial connection to the Kafka cluster.
The format is host1:port1,host2:port2.
Acks
1
Number of acknowledgments the producer requires the leader to have received before considering a request complete.
The allowed values are:
• 0 - Producer does not wait for any acknowledgment from the server.
• 1 - Leader writes the record to its local log but responds without awaiting full acknowledgement from all followers.
• 2 - Producer requires the given number of acknowledgements but this is generally less useful.
• all - Leader waits for the full set of in-sync replicas to acknowledge the record.
Buffer Memory
33554432
Total bytes of memory the producer can use to buffer records waiting to be sent to the server.
Compression Type
none
Compression type for all data generated by the producer.
The allowed values are:
• none
• gzip
• snappy
Retries
3
If the value is greater than zero, the client resends any record when the previous send has failed with a potentially transient error.
Batch Size
16384
Producer attempts to batch records together into fewer requests whenever multiple records are being sent to the same partition.
Client Id
 
User-specified string sent in each request to help trace calls.
Linger
0
Producer groups together any records that arrive in between request transmissions into a single batched request. The value is in milliseconds.
Max Request Size
1048576
Maximum size (records) of a request.
Receive Buffer
32768
Size of the TCP receive buffer to use when reading data.
Send Buffer
131072
Size of the TCP send buffer to use when sending data.
Timeout
30000
Configuration controls the maximum amount of time the server waits for acknowledgments from followers to meet the acknowledgment requirements the producer has specified with the acks configuration. The value is in ms.
Block On Buffer Full
True
When the memory buffer is exhausted, it should either stop accepting new records (block) or throw errors.
Metadata Fetch Timeout
60000
When data is sent to a topic first time, the metadata about that topic is required to know the servers that host the topic's partitions. This configuration controls the maximum amount of time to wait for the metadata fetch to succeed before throwing an exception back to the client. The value is in ms.
Metadata Max Age
300000
Amount of time in milliseconds after which force a refresh of metadata even if any partition leadership changes are not seen to proactively discover any new brokers or partitions. The default value is in ms.
Reconnect Backoff
10
Amount of time to wait before attempting to reconnect to a host when a connection fails. The value is in ms.
Retry Backoff
100
Amount of time to wait before attempting to retry a failed produce request to a particular topic partition. The value is in ms.
Supported Actions
 
Action
Description
Connect
Establishes a connection to the server.
Disconnect
Disconnects from the server.
PutMessage
Writes a message object into the queue.
Supported Action Parameters
 
Action
Parameter
Description
PutMessage
Message
The message that must be written.
Supported Action Properties
 
Action
Property
Description
PutMessage
Topic
Kafka message topic
Partition Key
Message partition key
Error Conditions
The following table describes error codes generated while sending a message to Kafka.
 
Error Condition
Code
Description
No error
0
The status is OK. There is no error. The information is returned successfully.
Missing license
46
Component Kafka writter is not licensed. The component requires DataConnect professional license. Contact your Actian sales representative to obtain the licensing information.
Missing broker list
4
Error detected when establishing a connection. The Bootstrap Servers property is required.
Error during connect
4
Exception 'xxx' occurred while attempting to connect to Kafka server 'xxx'.
Missing topic
4
Error detected when sending a message to a topic. A topic is required.
Error when sending a message
8
Exception 'xxx' occurred while attempting to put message to a topic 'xxx'.
Note:  For more informaiton about Kafka, see the Kafka documentation available at http://kafka.apache.org.
Websphere MQ Queue
WebSphere MQ Queue allows a message to be put into the queue or retrieved from a queue to a message object. Use the PutMessage action to write to the queue. Use the GetMessage action in a Queue step to retrieve messages. Session properties control which message objects are read and the names of the messages that are written, as well as whether to preserve the message properties.
Note:  WebSphere MQ Queue type supports WebSphere MQ 5.x, 6.x, and 7.x on 64-bit systems.
The section provides information for the following:
Websphere MQ Queue Properties
 
Property
Default Value
Description
Format
MQFMT_NONE
WebSphere MQ message format.
Timeout
0
Maximum number of seconds to wait for a message to arrive on a queue when retrieving messages.
Browse
False
Retrieve messages from a queue without removing them.
Convert
False
Converts the message data to a specific character set or numeric encoding. Requires CCSID or Encoding value.
CCSID
MQCSSI_DEFAULT
Character set identifier of the message data.
Encoding
 
Encoding for numeric data. The representation used for binary integers, packed-decimal integers, and floating-point numbers.
Supported Actions
 
Action
Description
Connect
Establishes a connection to the WebSphere MQ server.
Disconnect
Disconnects from the WebSphere MQ server.
GetMessage
Retrieves message into a message object.
PutMessage
Writes a message object into the queue.
Begin Transaction
Starts a WebSphere MQ transaction.
Commit Transaction
Commits a WebSphere MQ transaction.
Rollback Transaction
Rolls back a WebSphere MQ transaction.
Supported Action Properties
 
Action
Property
Description
PutMessage
GetMessage
Format
WebSphere MQ message format, with allowable values defined by WebSphere MQ
PutMessage
GetMessage
Msgid
Message ID of a specific message put in or retrieve from a queue
PutMessage
GetMessage
Correlationid
Correlation ID of messages to put in or retrieve from a queue
PutMessage
GetMessage
Groupid
Group ID of message to put in or retrieve from a queue
PutMessage
GetMessage
CCSID
Character set that you can convert to. Used with the Convert option
PutMessage
GetMessage
Encoding
Numeric encoding you can convert to. Used with the Convert option
GetMessage
SequenceNumber
Sequence number of a specific message to retrieve from a queue
PutMessage
Offset
Segment offset
PutMessage
ReplyToQ
Reply queue
PutMessage
ReplyToQMgr
Reply queue manage
PutMessage
UserID
User identifier
PutMessage
AccountingToken
Accounting token
PutMessage
ApplIdData
Application identity data
PutMessage
PutAppIName
Application that put the message
PutMessage
PutDate
Date the message was put
PutMessage
PutTime
Time the message was put
PutMessage
ApplOriginData
Application origin data
PutMessage
MsgType
Message type
PutMessage
Expiry
Message lifetime
PutMessage
Feedback
Feedback code
PutMessage
Priority
Message priority
PutMessage
Persistence
Message persistence
PutMessage
BackoutCount
Backout counter
PutMessage
PutApplType
Put application type
PutMessage
MsgFlags
Message flags
PutMessage
OriginalLength
Length of original message
WebSphere MQ Environment Variables
If you use WebSphere MQ environment variables, to ensure compatibility with queue sessions, use the following format:
MQServer=ChannelName/TransportType/IPAddress
Example
MQServer=mymq1/TCP/123.123.123.123
Queue Manager Parameter
Specify the Queue Manager as the Server parameter in the queue session as shown in this example:
Server=<servername> (123.123.123.123) Port=default QueueManager=<queuemanagername> queue=tester1 ServerChannel=serverch1
Server Parameter
The MQServer, MQCHLLIB, and MQCHLTAB environment variables can be changed at runtime or before the GUI tools are launched to test writing to different servers.
In the queue session, there is a Server parameter where you specify a Queue Manager (if you need one).
• If you use the MQCHLLIB or MQCHLTAB variable, you must specify the Server parameter.
• If you just use the MQServer environment variable, it is not necessary to specify the Server parameter. Instead, just change the MQServer environment variable at run time to specify the new Queue Manager to which you are writing. The MQServer environment variable has the channel, protocol, server, and port specified, which is all that is needed to connect to a different queue manager.
Macro Values for the Server Parameter
You can use macro values for the Server parameter of the session. If you use the MQCHLLIB and MQCHLTAB environment variables, you can create a macro to specify the Queue Manager. The Queue Name cannot use a macro value. Change this before deployment, or use a variable that drives Decision steps to do a Get or a Put message to the proper queue.
It is recommended that you change the MQServer environment variable at runtime, since no Queue Manager is needed in the Server parameter of the queue session. You only need to change the queue name.
Key Points About Websphere MQ and the DJComponent Object
IBM Websphere MQ requires a separate queue manager connection for each thread, and does not permit cross-thread operations.
For processes, three scenarios are possible:
• Set MaxConcurrentThreads = 1. All process steps are executed on the same thread, so queue operations can be spread across multiple steps.
• Complete all queue operations in a single step. This includes connecting to a queue, getting or putting messages, and completing a transaction. This guarantees that all queue operations are executed in a single thread. Multiple steps can still contains queue operations, but each step is a separate transactional unit of work.
• Confine all queue operations to the Start and Stop steps. In multithreaded processes, the Start and Stop steps always execute in the main thread. The Start step can be used to create a connection, begin a transaction and retrieve messages from the queue. The Stop step can be used to write messages to the queue and complete the transaction.
Note:  These threading restrictions apply only to DJComponent objects, not DJMessage objects. The DJMessage objects are completely independent from the messaging systems. So, in scenario 3 above, you can pull a message from a queue in the Start step. Use that message object in multiple concurrent Script or Transformation steps, and complete the transaction in the Stop step. Likewise, you can create DJMessage objects in multiple steps and write all of the message objects to a queue in the Stop step. The maximum size limit for DJMessage is 512 MB.
MSMQ Queue
MSMQ Queue allows a file folder to be treated like a message queue. The files in the folder can be retrieved into message objects by using the GetMessage action in a queue step. Message objects can also be written to the folder as files using the PutMessage action. The properties of the session control which files are read and the names of the files that are written, as well as whether to preserve the message properties.
MSMQ Queue Properties
 
Property
Default Value
Description
User Name
 
User name to connect to the MSMQ server.
Password
 
Password to connect to the MSMQ server.
Server Name
 
Name of the MSMQ server.
Timeout
0
Maximum number of seconds to wait for a message to arrive on a queue when retrieving messages
BrowseMode
False
Retrieve messages from a queue without removing them
Supported Actions
 
Action
Description
Connect
Establishes a connection to the MSMQ server.
Disconnect
Disconnects from the MSMQ server.
GetMessage
Retrieves message into a message object.
PutMessage
Writes a message object into the queue.
BeginTransaction
Starts a MSMQ transaction.
PrepareTransaction
Prepares a MSMQ transaction.
CommitTransaction
Commits a MSMQ transaction.
RollbackTransaction
Rolls back a MSMQ transaction.
Supported Action Parameters
 
Action
Parameter
Description
Connect
Server
Name of the MSMQ server.
 
Queue
Name of the queue.
Supported Action Properties
There are no action properties for MSMQ Queue component.