User Guide : Map Connectors : Source and Target Map Connectors : ODBC 3.5 Mass Insert
 
Share this page                  
ODBC 3.5 Mass Insert
The term ODBC stands for "Open Database Connectivity". This is a standard to allow client-side Windows applications to access various back-end databases through a single common interface. The ODBC 3.5 connector allows you to access data from any application that has a Windows or Unix\AS400-compliant ODBC driver.
Note:  This connector only supports 64-bit ODBC drivers.
ODBC 3.5 Mass Insert is a rapid way of inserting records into a relational database. It bypasses the transactional layer of the database and adds information directly to the storage tables. ODBC 3.5 Mass Insert is a useful option if you are dealing with large tables and performance is of paramount importance, but you do not have the option of a native connection.
In most cases, the integration platform connects to the file formats directly through knowledge of their physical file formats (for example, ASCII, Lotus, xBASE...), or through the native interfaces supplied by vendors who define a particular data format such as Oracle. ODBC provides another method (and sometimes the only method) for connecting to a particular file format.
Tip:  This connector is designed for Unicode ODBC 3.5x-compliant data sources. If your data source is not a Unicode-based, ODBC 3.5x-compliant data source, use the ODBC 3.x connector. See ODBC 3.5.
For information on installing ODBC drivers, see Installing an ODBC Driver.
Connectivity Pointers
If your database has many tables, selecting the table from a list may take more time than typing the UserID.Tablename value in the Source Table/View box. If you enter the table name without the user ID and then connect to the database, you can reselect the table in the source data browser to view it.
If possible, create query statements using the source database application, so that syntax and validity can be checked by the native application. You can then copy the query statement and paste it in the Query Statement Box.
Connector-Specific Notes
Connecting with the Client AS400 ODBC driver can result in slow run times. Other ODBC driver vendors may offer better performance.
Although Excel has a Windows-compliant ODBC driver and "Excel Files" may appear as a data source option when you use the ODBC connector, the integration platform cannot connect with Excel files because they comprise worksheets rather than tables.
SQL Server: The use of Regional settings in ODBC DSN connections to SQL Server databases is not supported in this release. In order for currency, numbers, dates and times to be interpreted correctly, ensure that the Use regional settings when outputting currency, numbers, dates and times checkbox in the SQL Server DSN settings is not checked. Then in the ODBC connector DriverOptions property, type Regional=No.
Switching Between iODBC and unixODBC on Linux
The ODBC 3.5 and ODBC 3.x connectors are configured during installation to work with the iODBC driver manager. Users with the unixODBC driver manager must switch to the unixODBC versions.
To switch to the unixODBC connectors, run the following commands as root.
cd <installdir>/runtime/di9rm -f djodbc3.so djodbc35.soln -s djodbc3.unixodbc.so djodbc3.soln -s djodbc35.unixodbc.so djodbc35.sochown diuser.diusers djodbc3.so djodbc35.so
To revert to the iODBC connectors, run the following commands as root.
cd <installdir>/runtime/di9rm -f djodbc3.so djodbc35.soln -s djodbc3.iodbc.so djodbc3.soln -s djodbc35.iodbc.so djodbc35.so
Property Options
You can set the following source (S) and target (T) properties.
Property
S/T
Description
Encoding
T
Sets the character encoding for ODBC 3.x Mass Insert source and target files.
Note:  Shift-JIS encoding is used only in Japanese operating systems. UCS-2 is no longer a valid encoding name, but you may use UCS2 by changing the string value in the XML file.
ConstraintDDL
T
Allows you to specify some additional SQL data definition language statements to be executed after their target table is created. This is similar to the support we provide for SQL pass-through in the SQL import connectors. Each line must be a valid ODBC DDL statement.
For example, you could have the statements:
CREATE UNIQUE INDEX index1 ON mytable (Field1 ASC)
CREATE INDEX index2 ON mytable (Field2, Field3)
These statements would create two indices on the table "mytable". The first one does not allow duplicates and the index values are stored in ascending order. The second index is a compound index on fields Field2 and Field3.
The ConstraintDDL is executed only if the replace mode is used for the target. If there are any errors, the errors are written to the error and event log file. During the transformation, the transformation error dialog box appears. If you want to ignore the DDL errors, they can continue the transformation.
ConstraintDDL also supports an escaping mechanism that allows users to specify DDL in the native SQL of the DBMS. Any statement that is preceded by an '@' is sent straight to the DBMS.
The statement:
@CREATE INDEX pk_mytable ON mytable (Field1, Field2) WITH PRIMARY
is a DDL statement for creating a primary key for the table mytable. Some ODBC drivers do not support the SQL extensions needed to create a primary key with the ODBC variant of the SQL CREATE statement. So to create primary keys, use native SQL.
Note:  This option works only in REPLACE mode.
CommitFrequency
T
Controls how often data is committed to the database. Default is zero, meaning that data is committed at the end of the transformation, allowing rollback on error. This is the slowest setting. When doing large transformations, this is not practical as it may produce too many transaction log entries. Setting the CommitFrequency to some nonzero value tells the connector to do a database commit after inserting or updating the specified number of records. This keeps the transaction log from getting too large but limits the restartability of the transformation.
SystemTables
T
The SystemTables property is only applicable if you are logged onto the database as the database administrator. Only the DBA has access to SystemTables. If set to true, this property allows you to see all the tables created by the DBA. The system table names appear in the table list. The default is false.
Views
T
If set to true, this property allows you to see views. The view names appear in the table list along with the tables. Default is true.
Synonyms
T
If set to true, this property allows you to see synonyms. The alias names appear in the table list along with the tables. Default is false.
DSNType
T
You can specify data sources or drivers to connect to in the Data Source Name list. DSNs are listed in the ODBC Data Source Administrator of Windows Administrative Tools in the Control Panel. The options are User & System, System, User, Driver, and File. Default is User & System.
AutoCommit
T
Allows you to automatically commit changes as they are made by each SQL statement, instead of waiting until the end of the transaction. If AutoCommit is set to true, there is no way to roll back changes once they have been made. The default is false.
BulkOperations
T
Use bulk operations for faster insert. This property determines if an insert statement is executed for each record or a bulk add is executed for each record. The default is true. To cause an insert statement to be executed for each record, this should be set to false, the slower setting.
DriverCompletion
T
Allows you to control whether or not the driver prompts you for information. The options are Prompt, Complete (default), Complete Required, and No Prompt. The Prompt option prompts you for every individual bit of information. Complete prompts you for any information they forgot to enter. The Complete Required option prompts you only for information that is essential to complete the connection. The No Prompt option does not prompt you for any information.
IdentifierQuotes
T
All databases have what are called quoted identifiers. You use quoted identifiers to make the SQL statement parseable and distinguish between columns and character data in SQL statements. For example, Oracle uses double quotes for column and table names in SQL statements and uses single quotes for character data. In a SQL statement, you should enclose identifiers containing special characters or match keywords in identifier quote characters; (also known as delimited identifiers in SQL-92). For example, the Accounts Receivable identifier is quoted in the following SELECT statement:
SELECT * FROM "Accounts Receivable"
If you do not use identifier quotes, the parser assumes there are two tables, Accounts and Receivable and return a syntax error that they are not separated by a comma. Options for IdentifierQuotes are Default, None, " and '.
ModifyDriverOptions
T
Allows you to store the ODBC connection. The default is true. If you set this to false, you are prompted for your connection information each time that you run your transformation.
DriverOptions
T
Enter any valid ODBC connect string options here.
MaxDataLength
T
The maximum data length for long data types. Default is 1 MB. You can reset this number as you choose based on your available memory capabilities and target requirements.
Some ODBC drivers have limitations concerning the maximum data length they can handle. If you choose an Application (ODBC) Source or target connector and the default setting is not 1 MB, then the integration platform sets the default for the capabilities of that particular ODBC driver. It is not recommended that the MaxDataLength property be set any higher under those conditions.
PrimaryKey
T
The PrimaryKey property allows you to specify a list of field names that are used to make the primary key. The field names are delimited by commas. If the PrimaryKey property contains one or more field names, these names are included in the SQL CREATE statement when the connector is in replace mode.
There is one additional requirement for using the PrimaryKey property. The ODBC driver must support integrity enhancement facility (IEF). Only the more advanced ODBC drivers support this.
TransactionIsolation
T
The Translation Isolation option allows you to specify any one of five different isolation levels when reading from or writing to a database table with ODBC. The default is Serializable.
The ANSI SQL 2 standard defines three specific ways in which serializability of a transaction may be violated: P1 (Dirty Read), P2 (Nonrepeatable Read), and P3 (Phantoms).
The following lists the four supported isolation levels.
READ_UNCOMMITTED - Permits P1, P2, and P3.
READ_COMMITTED - Permits P2 and P3. Does not permit P1.
REPEATABLE_READ - Permits P3. Does not permit P1 and P2.
SERIALIZABLE - Does not permit any of P1, P2, and P3.
For more information, see the Microsoft ODBC SDK documentation.
UseCursors
T
The UseCursors property allows you to turn cursor support on and off. The default is false. If you set the property to true and the specified ODBC driver does not support cursor inserts, the integration platform uses the SQL INSERT mode of adding records.
For exports, cursor support is supposed to enhance the performance of inserting records. This appears to be the case for desktop databases. For database servers, there is no noticeable change in insert speed. They seem to execute prepared queries about as quickly as they handle cursor inserts
Another complication of the cursor inserts is that some drivers require that the target table be indexed, otherwise positioned updates (cursors) are not allowed. Two additional properties in the ODBC export connector address this issue: PrimaryKey and ConstraintDDL (see details above).
ArraySize
T
Determines the number of rows to be sent to the server at one time. The default value is 1, meaning each row is individually sent to the server. Larger values will buffer multiple rows and send them all at once. While this improves the speed, it affects error reporting (a server error won't be detected/reported until the next batch of records is sent to the server).
The maximum value allowed for this property is 100000. While the connector allows the value to be set that high, many drivers have lower limits. The connector will log a message indicating if the driver is forcing a lower value for the array size. In addition, the connector does not support arrays when there is a LOB-type field in the table, or when the (maximum) length of a character-type field is longer than 32767 characters. In these cases, a message will be logged indicating the array size has been reduced to 1.
Due to the way the connector attempts to support older drivers, the array support requires either BulkOperations be true (which means UseCursors must also be true), or both must be false. The condition where BulkOperations is false and UseCursors is true will cause the array size to be reset to 1, with a message logged indicating this condition occurred.
Data Types
The following data types are available:
Binary
Bit
Byte
Char
Counter
Currency
Datetime
Double
Guid
Integer
LongBinary
LongChar
Real
SmallInt
Varbinary
VarChar
If you are appending data to an existing table, the data type of each field uses the data type in the selected table by default.
Length
These are field lengths in your data. If you need to change field lengths, reset them in the schema.