COPY VWLOAD (X100 Only)
Valid in: SQL, ESQL, OpenAPI
The COPY...VWLOAD statement copies the contents of a file to an X100 table using the VWLOAD operator. This operation is like using vwload -c on the command line, except that the load can be done from within a multi-statement transaction instead of in a separate session.
The Load Query Statistics and Result Query Statistics can be viewed after executing the COPY VWLOAD query. The Load Query Statistics can be viewed while running the query (in a different session for the same user) and the Result Query Statistics can be viewed after running the query (in the same session for the same user).
See iivwprof_vwload_load_status Catalog and iivwprof_vwload_result_status Catalog in the System Administrator guide.
Note: COPY VWLOAD has the following limitations at this time: The WITH option ATTRIBUTES does not work. If you specify an invalid or mistyped parameter, it is silently ignored and the load executes (to the extent possible as if that parameter was not there) instead of returning an error.
This statement has the following format:
COPY tablename() VWLOAD FROM 'datafile', ... [WITH option {,option}]
tablename()
Specifies the name of the table to be loaded. You can also use table synonym to load data into the table. The column list is empty and not supported.
'datafile'
Specifies the names of the files, using full or relative paths, that contain the data being loaded. File names are specified as a comma-separated list from one to many. An entry can be a file name, a file name with an asterisk as a wildcard, or a directory. If you specify a directory, all files present in the directory are loaded. Valid use of wildcards in names is as follows: *, abc*, *abc, abc*def, *abc*.
File systems supported: abfs, hdfs, http, https, s3a
Note: File names with an asterisk as a wildcard are not supported when using the http and https file system paths.
WITH option {,option}
Specifies optional WITH clause options:
AUTO_DETECT_COMPRESSION
Detects GZ and ZIP files. Compressed files are automatically detected by the file contents, not by the file suffix. If you are loading from a directory or using a file name wildcard, you can mix regular text files, GZ, and ZIP.
Note: VWLOAD supports compression method 8 (Deflate) only.
AWS_ACCESS_KEY='my_access_key'
Specifies the access key ID to access your S3 bucket on Amazon Web Services.
AWS_ENDPOINT='aws_endpoint'
Specifies the URL of the entry point to access AWS buckets.
For Example:
s3.us-east-2.amazonaws.com
AWS_REGION='aws_region_name'
Specifies the region where the source data resides in your S3 bucket on Amazon Web Services.
Note: ' AWS_REGION' and ' AWS_ENDPOINT' are optional components and applicable to both types of AWS credentials.
AWS_SECRET_KEY='my_secret_key'
Specifies the secret access key used to access your S3 bucket on Amazon Web Services.
AWS_SESSION_TOKEN='my_session_token'
Specifies the session token used to access your S3 bucket on Amazon Web Services.
For AWS, not all credential types are required in the credential file, rather a combination of the types using the with session token and without session token as shown below:
with session token:
AWS_ACCESS_KEY='my_access_key'
AWS_SECRET_KEY='my_secret_key'
AWS_SESSION_TOKEN='session_token'
AWS_REGION='us-east-2'
AWS_ENDPOINT='s3.us-east-2.amazonaws.com'
without session token:
AWS_ACCESS_KEY='my_access_key'
AWS_SECRET_KEY='my_secret_key'
AWS_REGION='us-east-2'
AWS_ENDPOINT='s3.us-east-2.amazonaws.com'
AZURE_CLIENT_ENDPOINT='client_endpoint'
Specifies the Directory (tenant) ID used to access your Azure Storage account.
For Example:
https://login.microsoftonline.com/tenant_id/oauth2/token'
AZURE_CLIENT_ID = 'client_id'
Specifies the Application (client) ID used to access your Azure Storage account.
AZURE_CLIENT_SECRET = 'client_secret'
Specifies the Client Secret (password) used to access your Azure Storage account.
INSERTMODE=
Specifies the mode to use for inserts and merges. Valid modes are:
ROW
Inserts through the PDT (allows concurrent inserts)
BULK
Appends directly to disk (does not allow concurrent inserts)
ATTRIBUTES='col1, col2, col3,...'
Specifies list of columns to load. An empty column name indicates an input field that is to be ignored.
Default: All columns are loaded if no attributes argument is specified.
ROLLBACK=
Turns “roll back on error” on or off
ON
(Default) Rolls back on error
OFF
Does not roll back the load transaction when errors are encountered, causing partial data to be loaded.
CHARSET='charset'
Specifies the input character set
Default: no conversion
DATEFORMAT='col1=format1,col2=format2,...'
Sets date format for the column.
When no column is indicated, the format applies to all columns that are not otherwise set. That format must be defined last, without any assignment operation.
Alternatively, a custom format string can be specified by starting the format with a '+' sign followed by the format string. The format string can consist of any of the format specifier characters used on the DATE_FORMAT function, described in the SQL Language Guide.
To be able to use any character in the format string (including for example, "," and "="), you must quote the format string in the same way that you would quote a delimited identifier on the command line (for example: 'ColX="+%M,%Y,%d",ColY=US'). For more information on delimited identifiers, see
Regular and Delimited Identifiers.
Default: US
Examples:
WITH DATEFORMAT='col1=+'%d-%b-%y'
WITH DATEFORMAT='Col1=US,Col2=+%M %Y %d'
Define US date format for columns a2_us, a4_us and GERMAN for all other columns (a1_multi, a3_de, a5_de):
WITH ATTRIBUTES='a1_multi,a2_us,a3_de,a4_us,a5_de', FDELIM=',',
DATEFORMAT='a2_us=US,a4_us=US,GERMAN';
or, which accomplishes the same:
WITH ATTRIBUTES='a1_multi,a2_us,a3_de,a4_us,a5_de', FDELIM=',',
DATEFORMAT='a1_multi=GERMAN,a3_de=GERMAN,a5_de=GERMAN,US';
ERRCOUNT=n
Terminates after the first n errors in all input files. Default: 0 (do not terminate).
ESCAPE='escapechar'
Specifies the escape character to use. This allows escaping of single characters to allow a delimiter character to be part of a field (for example: \) or to allow a quote character to be part of a quoted string.
The argument must be a single ASCII character or an empty string. If the argument is an empty string, this functionality is disabled. To specify a control character, use an escape sequence.
Default: none
ESCAPES
Interprets data escape sequences
FDELIM='fielddelim'
Specifies field delimiter to use. The delimiter must be a single character. To specify a control character, use an escape sequence.
Default: "|"
GCS_EMAIL='name@mail.com'
Specifies the email used for the service account used for Google Cloud Storage access.
GCS_PRIVATE_KEY_ID='private_key_id'
Specifies the ID of the private key for the service account used for Google Cloud Storage access. For information about keeping your credentials safe, see the Security Guide.
GCS_PRIVATE_KEY='-----BEGIN PRIVATE KEY-----\n<long_key>\n-----END PRIVATE KEY-----\n'
Specifies the private key for the service account used for Google Cloud Storage access in PKCS#8 format. The key must include the starting and ending characters: -----BEGIN PRIVATE KEY-----\n and -----END PRIVATE KEY-----\n. For example:
-----BEGIN PRIVATE KEY-----\n
MIIEvAIBADANBgkqhkiG9w0BgSiAgEAAoIBAQDHd1HNub/lfF41 ... jm+177ZXFg+QIyXFCqbMDTAnjYY3
-----END PRIVATE KEY-----\n
HEADER
Skips header line in files
IGNFIRST
Ignores the first field
IGNLAST
Ignores the last field
LOG='dir'
Logs rejected rows and corresponding errors to the specified file. The file is created by the operation.
NULLVALUE='nullvalue'
Defines the string that identifies NULL values. Default: ' '
By default, an empty string is treated as a null unless -n option is used. For example, suppose the following contents in data file a1.csv:
1|\n|
2|a|b
3||c
If we use COPY VWLOAD (t1) FROM a1.csv WITH NULLVALUE='\n', then 3rd row, 2nd column will have an empty string and not null; 1st row, 2nd column \n will be loaded as null as defined in NULLVALUE.
If we use COPY VWLOAD (t1) FROM a1.csv, then 3rd row, 2nd column will be loaded as null; 1st row, 2nd column will be loaded as '\n'.
QUOTE='quotechar'
Specifies the quote character(s) to use. This allows the input to contain quoted fields (for example "Doe, John"), which may contain field or record delimiter characters. To include a quote character inside a quoted field, enter it twice in the input, for example: "The ""BIG"" Boss". When using distinct open and close quote characters, enter only the close quote character twice--for example: [The [BIG]] Boss].
The argument must be one or two ASCII characters or an empty string.
If the argument is an empty string, this functionality is disabled. If the argument is two characters, the first character is used as the opening quote character, and the second as the closing quote character. For example, specify "[]" to allow [quoted string].
To specify a control character, use an escape sequence.
Default: none
RDELIM='recorddelim'
Specifies record delimiter to use. The delimiter must be a single character. To specify a control character, use an escape sequence.
Default: "\n"
STRICTNULLS
Uses strict NULL value checking. This distinguishes between plain and quoted or escaped occurrences of a NULL representation. For example, NULL is a NULL value, whereas "NULL" is a 4-character string value. By default, VWLOAD does not make this distinction. This option allows proper loading of some data generated by MySQL and PostgreSQL.
SUBSTITUTE='substitutechar'
Substitutes the specified character for any invalid input character during character set conversion. When no substitute character is provided, an invalid input character is considered an error condition. Specifying a substitute character allows records that contain invalid characters to be loaded successfully.
The argument must be a single Unicode character or an empty string. If the argument is an empty string, the functionality is disabled.
To specify a control character or Unicode code point that cannot easily be typed on the keyboard, use an escape sequence.
TEXTMODE
Opens input files in text mode and does not perform newline conversion. On Linux, this may gain a 5 to 10 percent performance improvement. On Windows, there is no known advantage to using this option. (By default, VWLOAD opens files in binary mode and performs automatic newline conversion, transparently handling text formats of various operating systems.)
Note: When using -T on Windows, any ^Z (ASCII SUB) character in the input is interpreted as End-Of-File by the Windows library, and causes VWLOAD not to see any data following the first such character.
WORK_DIR
Specifies a directory that will be used for relative paths in filenames. Default is /tmp, so specifying non-absolute paths without a WORK_DIR tells X100 to look for the files in /tmp.