Converting Data Files
Zen includes tools that convert Btrieve files to later versions to change file settings or take advantage of features in newer Zen engines, such as system keys. The following topics cover the concepts and tools for doing these conversions:
The Zen v16 release includes a standalone command line version of the existing Rebuild tool, called rbldcli_offline. It is meant for use in converting Zen data files on systems where Zen is not installed. Rebuilding a large number of files, or very large files, can slow Zen database engine performance for other users. Rebuilding the files on another system avoids this problem. In addition, Zen v16 cannot read files in the 5.x format, so they must be converted to a later version. Only rbldcli_offline can read and convert 5.x files.
Note: The Rebuild tool is not intended for use with corrupted files. To repair corrupted files by cloning or by exporting, importing, or copying data, see Manipulating Btrieve Data Files with the Maintenance Tool. To compress files to improve engine performance, see Monitoring Data File Fragmentation.
Rebuild Tool Concepts
The Rebuild tool allows you to perform the following operations on MicroKernel data files and dictionary files:
• Convert older Zen file formats to a newer one.
• Convert newer file formats to a previous one, provided the previous one is not pre-6.x.
• Rebuild a file using the same file format, provided it is not pre-6.x.
• Rebuild files to retain, add, or drop page or record compression.
• Change the page size in the file.
• Rebuild files to retain, add, or drop system data and system keys.
• Specify a location and name of the log file used by Rebuild.
If your database uses dictionary files (DDFs), you must rebuild them as well as the data files.
Read the following topics to understand the concepts behind rebuilding data files.
• Log File
For information on using the Rebuild tools, see one of these topics:
Platforms Supported
Rebuild comes in two forms: a 32-bit GUI version for Windows, and command line versions for Linux, Raspbian, and Windows. See Rebuild Tool GUI Reference and CLI Rebuild Tasks.
Linux and Raspbian CLI Rebuild
Rebuild runs as a program, rbldcli, on Linux and Raspbian. By default, the program is located in /usr/local/actianzen/bin.
Windows CLI Rebuild
Rebuild runs as a program, rbldcli.exe, on Windows. By default, the program is installed in the Program Files (x86) directory.
File Formats
The current database engines remain compatible with some older data and dictionary file formats, but you may want to convert files to take advantage of newer features. The following table lists common reasons for converting to a higher format.
Converting to Version | Adds Support for File Features |
|---|---|
16.0 | Maximum key lengths widened to 1024 bytes to enable indexing of longer data, such as VARCHAR, without having to use PARTIAL INDEX |
13.0 | • Maximum file sizes in the terabytes • System data v2 for tracking and querying insert and last update time stamps |
9.5 | • More than 119 segment keys and files sizes up to 256 GB • Long owner names with up to 20 characters instead of 8 • AES-256 file encryption of files with long owner names • 16K page size • Page compression |
9.0 | Disabling 2GB file segmentation to create one continuous file |
8.0 | Insert, update, and delete performance improvements offered by Turbo Write Accelerator |
The file format that results from using the command line Rebuild depends on the -f parameter. If you omit the -f parameter, then Rebuild uses the value for the Create File Version setting in the MicroKernel Engine. For example, if the Create File Version value is 9.5, then running the Rebuild tool on version 9.0 files converts them to 9.5 format.
It is suggested that you back up all the data files you plan to convert before running Rebuild. This is particularly true if you are rebuilding files to the same location as the source files (in which case the rebuilt files replace the source files). Having backup copies allows you to restore the original files if you so desire. To ensure that the backup is successful, you may perform one or more of the following operations:
• Close all data files before running a backup.
• Use continuous operations (only during the backup).
Note: You cannot run Rebuild on a file that is in continuous operation mode.
Temporary Files
On Windows, Rebuild creates temporary files in the directory specified by the TMP system environment variable. By default on Linux and Raspbian, Rebuild creates temporary files in the output directory (or in the source directory if the -b option is not used). Therefore, you need enough disk space in the temporary file directory (while Rebuild is running) to potentially accommodate both the original file and the new file. You can specify a different directory for storing these files by using the Output Directory option in the Rebuild GUI version or by using the -b option with the CLI versions.
Normally, Rebuild deletes temporary files when the conversion is complete. However, if a power failure or other serious interruption occurs, Rebuild may not delete the temporary files. If this occurs, you should manually delete the temporary files as described in the following table.
Platform | Temporary File Names |
|---|---|
Linux and Raspbian | _rbldxxxxxx, where xxxxxx is 6 random characters. Caution: Be sure that you do not delete the Rebuild executable, rbldcli. |
Windows | _rbldx, where x is a number. |
Optimizing the Rebuild Process
Rebuild makes Btrieve calls to the database engine. Therefore, the database engine configuration settings and the amount of processing memory affect the performance of the rebuild process, particularly with respect to the amount of time required to rebuild large data files.
In general, building indexes requires much more time than building data pages. If you have a data file with many indexes, it takes more time to rebuild than the same file with fewer indexes.
The following items can affect the rebuild processing time:
CPU Speed and Disk Speed
The speed of the central processing unit (CPU) and access speed of the physical storage disk can affect processing time during a rebuild. In general, the faster the speed for both of these, the faster the rebuild process. Disk speed is more critical for rebuilding files that are too large to fit entirely in memory.
Tip... Large files in the gigabyte range may take several hours to convert. If you have more than one database engine available, you may wish to share the rebuild processing among a number of machine CPUs. For example, you could copy some of your files to each machine that has a database engine installed, then after the rebuild process is finished, copy the files back to their original locations.
Amount of Memory
Rebuild is capable of rebuilding a file using two different methods, a default method and an alternative method. See -m<0 | 2> parameter. For the default method (-m2), Rebuild takes the following steps. The more memory available, the faster the execution.
1. Create a new, empty data file with the same record structure and indexes as the source file.
2. Drop all the indexes from the new file.
3. Copy all the data into the new file, without indexes.
4. Add the indexes, using the following process:
a. For a particular key in the source file, read as many key values as possible into a memory buffer using the Extended Step operation.
b. Sort the values in the memory buffer and write the sorted values to a temporary file.
The temporary file now contains several key value sets, each individually sorted.
5. Merge the sets into index pages, filling each page to capacity. Each index page is added to the data file at the end, extending the file length.
If any failure occurs during this process, such as a failure to open or write the temporary file, Rebuild starts over and uses the alternative method to build the file.
Rebuild uses an alternative method (-m0) when memory is insufficient to use the default method or if the default method encounters processing errors:
1. Create a new, empty data file with the same record structure and indexes as the source file.
2. Copy all the data into the new file.
The alternative method is typically much slower than the default method. If you have large data files with many indexes, the difference between the two methods can amount to many hours or even days. The only way to ensure that Rebuild uses the default method is to have enough available memory. Several configuration settings can affect the amount of available memory.
Sort Buffer Size
This setting specifies the maximum amount of memory that the MicroKernel dynamically allocates and deallocates for sorting purposes during run-time creation of indexes.
If the setting is zero (the default), Rebuild calculates a value for optimal memory bytes needed to rebuild indexes in memory.
If the setting is a nonzero value, and the value is larger than the calculated minimum memory that Rebuild needs, then Rebuild uses the smaller value.
Finally, Rebuild compares the amount of memory that it should allocate with 60% of the amount that is actually available. It then attempts to allocate the smaller of the two. If the memory allocation fails, Rebuild keeps attempting to allocate 80% of the last attempted amount. If the memory allocation fails completely (which means the amount of memory is less than the minimum memory bytes), Rebuild uses the alternative method to rebuild the indexes.
Max MicroKernel Memory Usage
This setting specifies the maximum proportion of total physical memory that the MicroKernel is allowed to consume. L1, L2, and all miscellaneous memory usage by the MicroKernel are included. Usage by the Relational Engine is not included.
If you have large files to rebuild, temporarily set Max MicroKernel Memory Usage to a lower percentage than its default setting. Reset it to your preferred percentage after you complete your rebuilding.
Cache Allocation Size
This setting specifies the size of the Level 1 cache that the MicroKernel allocates. The MicroKernel uses this cache when accessing any data files.
This setting determines how much memory is available to the database engine for accessing data files, not for use when indexes are built.
Increasing Cache Allocation to a high value does not help indexes build faster. In fact, it may slow the process by taking up crucial memory that is now unavailable to Rebuild. When rebuilding large files, decrease the cache value to a low value, such as 20% of your current value but not less than 5 MB. This leaves as much memory as possible available for index rebuilding.
Index Page Size
The page size in your file also affects the speed of index building. If Rebuild uses the alternative method, smaller key pages dramatically increase the time required to build indexes. Key page size has a lesser effect on building indexes if Rebuild uses the default method.
Rebuild can optimize page size for application performance or for disk storage.
To optimize page size for performance in terms of speed of data access, Rebuild uses a default page size of 4096 bytes. This results in larger page sizes in physical storage and slower rebuilding times.
For a discussion of optimizing page size for disk storage, see Choosing a Page Size in Zen Programmer’s Guide.
Log File
Rebuild execution information is appended to a text log file. Its default location varies:
• For Tools > Rebuild in ZenCC, it is C:\ProgramData\Actian\Zen\logs\rebuild.log
• For rbldcli, it is <current working directory>\rbldcli.log.
• For rbldcli_offline, it is <current working directory>\rbldcli_offline.log.
For command line execution, you can set another location and name for the log file instead of using the defaults by using the -lfile parameter.
The information logged includes the following:
• Start time of the rebuild process
• Parameters specified on the command line
• Status code and error description (if an error occurs)
• File being processed
• Information about the processing (such as page size changes)
• Total records processed
• Total indexes rebuilt (if the -m2 processing method is used)
• End time of the rebuild process
• Status of the process (for example, if the file rebuilt successfully)
Rebuild Tool GUI Reference
This topic describes the panels of the Rebuild tool.
File Options
In ZenCC, when you select Tools > Rebuild, the Rebuild wizard displays a panel where you can add data or dictionary files to a list. 
After creating the list of files to be rebuilt, click Next to continue to the Rebuild Options panel.
Rebuild Options
Click an area of the image to learn more about configuring how files can be rebuilt. After you select options, click Next to finish rebuilding the selected files.
GUI | Description | Related Information |
|---|---|---|
System Data | Specifies whether Rebuild is to retain, add, or drop system data and keys. System data is used for transaction durability logging when no user-defined unique data key exists. System data v2 can be used to track record updates. | |
Page Compression | Specifies page compression for the file. The choices are Keep Existing, On, and Off. Keep Existing retains whatever page compression the file contains, if any. | Available only for file format 9.5 or newer. |
Record Compression | Specifies record compression for the file. The choices are Keep Existing, On, and Off. Keep Existing retains whatever record compression the file contains, if any. | |
Continue on Error | Determines whether Rebuild continues if it encounters an error during the rebuild process. If you select Yes, the tool continues with the next file even if an error occurs. The tool alerts you to non-MicroKernel data files or other errors but continues rebuilding data files. If you select No, the tool halts the rebuild if it encounters an error. This option is useful if you use wildcard characters for the rebuilt files. | |
Save Settings on Exit | Saves the current values in this dialog box for use in subsequent Rebuild sessions. | |
Key Number | Specifies the key by which the tool reads when rebuilding a file. If you select NONE, then Rebuild reads the source file in physical order and creates the rebuilt file in physical order with rebuilt indexes. Because NONE is faster and creates smaller files, use it whenever possible. | |
File Format | You can select a version other than the default set in the Create File Version of the engine Compatibility property. | |
Page Size | Specifies the page size (in bytes) of the new files. Choose either EXISTING, Optimal (disk space), Optimal (data access), or a size in bytes. If you select EXISTING, the tool uses the existing page size. The tool changes the page size if the original size is no longer available or would not work for the other options selected. For example, assume you have a v5.x file with a page size of 1024 and 24 keys. Because Btrieve 6.0 and later supports only 23 keys for a page size of 1024, the tool automatically selects a new page size for the file and writes an informative message to the status file. | • For optimizing for data access, see Optimizing the Rebuild Process |
Output Path | Specifies an existing directory for rebuilt files, which can be other than the default of the current directory. This option lets you rebuild large files on a different server. The MicroKernel and its communications components must be loaded on the server that contains the rebuilt files. Do not use wildcard characters in the path. If the output directory is not the original file location, the original file is not deleted during the rebuild. If the output directory is the same as the original file, the original file is deleted and replaced with the rebuilt file. When DefaultDB is set to use database security, you can rebuild files only in locations listed in the Directories property for DefaultDB. | |
Log File | Specifies a rebuild log file directory, which can be other than the default location of the current working directory. Do not use wildcard characters in the path. | • Log File |
Rebuild Tool CLI Reference
Use rbldcli if you prefer a command line interface to the Rebuild GUI. Its executable program is rbldcli.exe on Windows and rbldcli on Linux-based systems. You can manually execute rbldcli commands at a command prompt or as an executable batch script. Before you run rbldcli, we recommend you understand the concepts and settings covered under Rebuild Tool Concepts.
The rest of this section covers the following topics:
Command Syntax
The syntax for a rbldcli command is as follows:
rbldcli [-parameter ...] file [file] | @cmdFile
parameter | A parameter specifies a setting used with the tool. You may use parameters in any order. Precede each parameter with a hyphen (-). Do not place a space after the hyphen or after the single-letter parameter and the parameter value. Note: On Linux and Raspbian, the parameters are case-sensitive. |
file | The data and dictionary files to convert. If the source files are not in the current working directory, you must include the location for each file, either as a fully qualified or a relative path. You may use the asterisk (*) wildcard character in the file name to select multiple files in a location. Note: If the original file contains an owner name, Rebuild applies the same owner name and level to the rebuilt file. |
@commandFile | Fully qualified file name of a command file for Rebuild to execute. A command file can have multiple entries, with each entry containing command line parameters (if any) and the set of files to convert, followed by <end> or [end]. When specifying files to rebuild, use full directory path names. You may use the asterisk (*) wildcard character in the file names. The following is an example of an rbldcli command file: –c d:\mydir\*.* <end> –c –p1024 e:\dir\*.* <end> –m0 –k0 d:\db\*.* <end> |
Parameters
The following table lists rbldcli parameters. Their behavior is the same as their counterparts described under Rebuild Tool GUI Reference, which may provide additional inforamation.
-c | Continues with the next data or dictionary file if an error occurs. The tool notifies you of non-MicroKernel data files or errors with MicroKernel files, but continues rebuilding data files. The error is written to the log file. See Log File. Tip... This parameter is helpful when you use wildcard characters (*.*) for a mixed set of files, which can have a combination of MicroKernel files and non-MicroKernel files. Rebuild reports an error for each non-MicroKernel file, along with errors on MicroKernel files, but continues processing. |
-d | Converts pre-6.0 supplemental indexes, which allow duplicates, to 6.x, 7.x, or 8.x indexes with linked-duplicate keys. Without -d, Rebuild preserves the indexes as repeating-duplicate keys. If you access your data files only through the MicroKernel Engine and your files have a relatively large number of duplicate keys, you can use -d to enhance the performance of the Get Next and Get Previous operations. |
-m<0 | 2> | The "m" stands for "method." By default, Rebuild selects -m2 as the processing method. If you select -m0, then Rebuild uses the alternative method -m0 if memory is insufficient. See Amount of Memory for how it affects the method chosen. |
0 | Rebuilds the data or dictionary file without dropping and replacing indexes. The -m0 method is typically slower than -m2. A file created with -m0 has key pages approximately 55% to 65% full, making it more optimized for writing than for reading. If you can afford the extra rebuild time, which can be considerable depending on the situation, you may choose this option based on the needs of your application. See also Optimizing the Rebuild Process. |
2 | Rebuilds the data or dictionary file by dropping the indexes, copying records to the new file, and rebuilding the indexes. The -m2 method is typically faster and creates smaller files than the -m0 method. It creates a file in which the physical order of records may differ from the original file. A file built using -m2 has key pages 100% full and optimized for reading. |
-p<D | P | bytes> | Optimizes page size for disk storage or processing, or specifies a specific page size to use for the rebuilt file. If -p is omitted, Rebuild uses the page size from the source file. If the source page size is not possible for the new file version, Rebuild changes the page size and displays an informative message explaining the change. For example, older file formats, such as 5.x, supported a page size of 1024 with 24 keys. File format 8.x supports only 23 keys for a page size of 1024, so Rebuild would select a new page size to create an 8.x file. The database engine may ignore the page size specified and automatically upgrade the page size to a multiple of 512 bytes, not exceeding the size allowed for the selected file version. For example, for the 9.5 file format, odd page sizes such as 1536 and 3072 are not supported. The database engine automatically upgrades to the next valid page size because that page size is more efficient. For older file formats, the database engine may upgrade the page size based on additional conditions. See also Index Page Size. |
D | Optimizes page size for disk space efficiency. See Choosing a Page Size in Zen Programmer’s Guide. |
P | Optimizes data access for application performance. For -pP, Rebuild uses a default page size of 4096 bytes. |
bytes | Specifies the page size (in bytes) for the new file. For file versions before 9.0, valid values are 512, 1024, 1536, 2048, 2560, 3072, 3584, and 4096. For file version 9.0, the values are the same, with the addition of 8192. For file version 9.5, valid values are 1024, 2048, 4096, 8192, and 16384. For file versions 13.0 and 16.0, valid values are 4096, 8192, and 16384. |
-bdirectoryname | The default output location is the current directory of the source files to be rebuilt. Use ‑b to select another location for the rebuilt file. You may use either a fully qualified or a relative path. Do not use wildcard characters in directoryname. Directoryname must already exist. Rebuild does not create it. Without ‑b, the rebuilt file replaces the original data file. If you use ‑b to select an alternate directory, the rebuilt file is placed there and the original file is retained. However, if the location already contains data files with the same names, then Rebuild fails and returns an error message to check the log file. You must have create file permission for the directoryname location, as well as for the source file location if you omit ‑b. In order to work, rbldcli requires that the local database engine be running. If directoryname is a location on a remote server, then a database engine must be running there as well. Note: The standalone tool rebldcli_offline.exe does not require a database engine. Other than this, its capabilities are identical to the original rbldcli, except that it cannot output rebuilt files to a location on a remote server. |
-knumber | Selects the key number by which Rebuild reads when rebuilding the source file and sorting the rebuilt file. If you omit this parameter, Rebuild reads the source file in physical order and creates the rebuilt file in physical order with rebuilt indexes. Because omitting the parameter is faster and creates smaller files, use it whenever possible. See also Optimizing the Rebuild Process. |
-s[D | K | 2D | 2K] | Rebuilds the file retaining existing system data and keys from the source file, or adding them if not present. If you omit this parameter, Rebuild does not retain any system data or key in the rebuilt file. |
D | Rebuilds the file with new system data. System data will not be indexed. |
K | Rebuilds the file with new system data. System data will be indexed. |
2D | Rebuilds the file with new system data v2. System data will not be indexed. For 13.0 and 16.0 format files only. |
2K | Rebuilds the file with new system data v2. System data will be indexed. For 13.0 and 16.0 format files only. |
-lfile | Specifies a log file name, and optionally a path location, for the Rebuild log file. The default file name is rbldcli.log on Windows, Linux and Raspbian. The default location is the current working directory. The following conditions apply: • The path location must already exist. Rebuild does not create it. • If you specify a path location without a file name, Rebuild ignores this parameter and uses the default file name and location. • If you specify a file name without a path location, Rebuild uses the default location. • You must have read and write file permission for the location you specify. Rebuild uses the default location if it cannot create the log file because of file permissions. See also Log File. |
-pagecompresson | Turns on page compression for the file provided the following conditions are true: • The version of the database engine is 9.5 or later. • The setting for Create File Version is 0950 (9.5) or later. |
-pagecompressoff | Turns off page compression for the file. This parameter has no effect if the file does not use page compression. |
-recordcompresson | Turns on record compression for the file. |
-recordcompressoff | Turns off record compression for the file. This parameter has no effect the file does not use record compression. |
-f<6 | 7 | 8 | 9 | 95 | 13 | 16> | Specifies a file format for the rebuilt data or dictionary file. File formats supported are versions 6.x, 7.x, 8.x, 9.x, 9.5, 13.0, and 16.0. The following example rebuilds a file in 9.0 format: rbldcli -f9 file_path\class.mkd The following example rebuilds a file in 13.0 format: rbldcli -f13 file_path\class.mkd If omitted, Rebuild defaults to the value selected in ZenCC for the MicroKernel Create File Version setting in the Compatibility properties for the engine. Note: If you specify a file format newer than the version supported by the database engine, Rebuild uses the highest supported file format for that engine. It reports no error or message that this has been done. Note: Rebuild does not convert data types in indexes. If you rebuild a file to an older file format for use with an older database engine, confirm what data types are supported by the database engine and the application. Example: Your data file contains index fields that use the WZSTRING data type. If you rebuild the data file to a 6.x file format, the WZSTRING data type is not converted. You would be unable to use the data file with a Btrieve 6.15 engine. That engine does not support the WZSTRING data type. Example: Your data file contains true NULLs. You rebuild the data file to a 7.x file format. The true NULLs are preserved. You would be unable to use the data file with the Zen 7 engine because it does not support true NULLs. |
-uiduname | Sets the name of the user authorized to access a database with security enabled. |
-pwdpword | Sets the password for the user identified by uname. Pword must be supplied if uname is specified. |
-dbdbname | Selects the name of the secured database. |
Using the Rebuild Tool
The following topics cover the GUI and command line versions of the Rebuild tool:
GUI Rebuild Tasks
To start the GUI Rebuild wizard
Select Tools > Rebuild in the Zen Control Center menu or access Rebuild from the operating system Start menu or Apps screen.
To rebuild a file or files
1. After you click Next at the Rebuild welcome screen, the Select Files screen appears.
2. Click Add and in the Selected Files dialog box, select the data or dictionary file you want to rebuild. You can select more than one file to rebuild at a time.
Rebuild deletes the original file after rebuilding it if the file is being rebuilt in the same directory. If the new file is in a different directory, the original file is not deleted.
3. Click Next after you have added the desired file or files.
4. Specify the rebuild options. See Rebuild Options.
5. Click Next to begin the rebuild process.
Rebuild reports the processing information. When the rebuild process finishes, its success or failure is displayed and View Log File is enabled.
6. To display the results, click View Log File. The contents of the log file display in the default text editor for the operating system.
Rebuild writes to the log file for every file it attempts to convert. If you disabled the Continue on Error setting, the log file contains the information up to the point of the error. If the rebuild was not successful, the status file contains error messages explaining why the rebuild failed.
7. Click Finish when you have finished rebuilding files and viewing the log file.
CLI Rebuild Tasks
The Rebuild command line tool is named rbldcli.exe on Windows and rbldcli on Linux and Raspbian. The following topics cover command line syntax and typical Rebuild tasks:
To run Rebuild on Windows
1. Open a command prompt on the system where Zen is installed.
2. Optionally, change to the \bin directory where you installed the program files. This is not required if the location is in the Path system variable.
3. Enter one of the following commands at the prompt:
rbldcli [–parameter ...] file
or
rbldcli @command_file
Example Usage
The following example continues on error, sets a page size of 4096 bytes, and places the rebuilt files in a different directory on the server.
rbldcli -c -p4096 -bc:\dbtemp c:\datafiles\*.mkd
To run Rebuild on Linux and Raspbian
1. Ensure that the account under which you are logged in has permission to run Zen utilities.
By default, you must be logged in as user zen-svc to run utilities. User zen-svc has no password and can be accessed only through the root account by using the su command. To use utilities from accounts other than zen-svc, you must first make modifications to the .bash_profile. See Zen Account Management on Linux and Raspbian in Getting Started with Zen.
2. Change directory to /usr/local/actianzen/bin directory.
3. Type one of the following commands at the prompt:
rbldcli [–parameter ...] file
or
rbldcli @command_file
Example Usage
The following example continues on error, sets a page size of 4096 bytes, and places the rebuilt files in a different directory on the server.
rbldcli -c -p4096 -b/usr/local/actianzen/tmp /usr/local/actianzen/data/Demodata/*.mkd
To see your progress while rebuilding files
Rebuild reports on the screen the number of records processed per file, incrementing 50 records at a time. In addition, Rebuild writes information to the log file. See Log File.
Last modified date: 07/24/2024