Troubleshooting is a process of defining and correcting a problem that occurs in an otherwise functioning Vector installation and includes narrowing down the problem to a well-defined point, identifying the cause of failure, and eliminating it. When troubleshooting, you:

You can perform system administrator functions—including configuration, performance monitoring, backup and recovery, and remote database optimization—using a variety of tools:

The first step in the process of troubleshooting is to determine the problem area. The following troubleshooting flow chart shows the major problem categories:

The error log files are located in the directory indicated by the Vector environment variable/logical II_CONFIG. The log files are as follows:

The names of optional log files can vary, but all log files end with the extension LOG. Optional log files include:

To display the value for II_CONFIG, type the following command at the operating system prompt:

For a list of log files, type the following command at the operating system prompt from the directory indicated by $II_CONFIG or II_LOG:

Checking for error logs is the first step for determining the nature of the problem. To check for error logs, follow these steps:

To find the problem category, follow these steps:

Use the following flow chart to isolate a problem with startup, shutdown or configuration of your Vector installation:

 

To check if the Vector installation is working fine, follow these steps:

The executable script ingbuild performs all the steps necessary to set up an installation. It checks system resources, installs shared memory and semaphores, configures DBMS server parameters, configures the logging and locking system, and starts all the required processes.

The ingbuild program is located in $II_SYSTEM/ingres/utility. It makes use of numerous shell commands as well as the following Vector binary and shell executables:

One of the last things ingbuild does is call the ingstart script to start installation processes. When ingstart is called, it displays the message “Starting the Name Server process (iigcn).” If there are startup problems after this message has displayed, see Detect Vector Startup Problems.

Before you can diagnose a problem with ingbuild, you must identify which subroutine is failing. If you know which routine is failing and it is ingstart or one of the main installation processes (iigcn, iigcc, II_IUSV_nnn, dmfacp or iidbms), see the section below that addresses that executable.

Details on tracing are described in Bourne Shell -x Option.

To diagnose Vector problems, use the following procedure.

To detect Vector startup problems on system reboot on Linux, follow these steps:

If Vector has problems during shutdown on Linux, follow these steps:

The major Vector processes are as follows:

The Name Server process (iigcn) is not running if either of the following occurs:

You can verify this by attempting to start the Name Server manually.

If the Name Server does not start, follow these steps:

If the Communications Server process (iigcc) did not start, follow this procedure:

If the Bridge Server process (iigcb) did not start, verify that you have installed the Ingres Net component and that you have installed the correct protocol drivers.

If you are still having problems, set the following trace to capture additional diagnostic data before calling technical support:

The recovery process (dmfrcp) must be running before a DBMS server can be started. Failure of the dmfrcp starting process indicates one of the following:

If the recovery process does not start, perform the following procedure:

You can check for the presence of the optional remote command process by entering the following command at the operating system prompt:

The archiver process (dmfacp) does not start unless the recovery (dmfrcp) process is running. However, an installation runs without an archiver process until the log file fills up. User programs are suspended as outstanding transactions in the log file are backed out. For information about the Vector recovery state, see Recovery Process Monitoring.

Use the following procedure to check archiver process startup problems. Some of these checks are the same as for the recovery process:

The command to start the DBMS Server process (iidbms) from the installation owner login is ingstart -iidbms. If the DBMS Server did not start:

The command to start the Data Access Server process (iigcd) from the installation owner login is ingstart -iigcd. If the Data Access Server did not start:

The following flow chart helps you isolate a problem when starting a Vector tool:

Once started with ingstart, the DBMS Server process must continue running until the ingstop or iimonitor command is issued to stop it. If the DBMS Server stops running (“dies”) for any other reason, report it to technical support along with the associated error log messages and, if possible, the cause of the DBMS Server stopping.

Database connection problems occur in the following scenarios:

If you cannot connect to any database, follow these steps:

If you can connect to some databases but not others:

An inconsistent database occurs when administrative changes to a database in the transaction log do not agree with information maintained in the database’s configuration file.

The main causes of inconsistent database errors are improper system administration procedures. These include:

Inconsistent database errors can also be caused by hardware problems or software problems. For example, inconsistent database errors can be caused when the transaction log file or the configuration file has been corrupted by a hardware failure. Inconsistent database errors can also be caused by software bugs. In either case, contact technical support.

Vector automatically handles the transaction failures that cause most database inconsistencies.

If a user program exits or a transaction is aborted for some other reason, the DBMS Server automatically handles transaction rollback. This does not cause an inconsistent database.

At shutdown, all users must have exited their sessions; therefore, all transactions are committed. If users exited their sessions abnormally, the DBMS Server aborts any open transactions associated with the aborted sessions. Very long transactions take time to roll back and cause ingstop to seem to hang. The DBMS Server process cannot exit normally until it finishes recovering the aborted transactions.

If transactions are being rolled back on shutdown, allow the DBMS Server to finish this task before shutting down. If you do not, longer delays occur at startup time while the recovery process is performing rollback.

If transactions have been aborted and were not recovered by a normal shutdown, upon restart the recovery process performs recovery. This occurs, for example, if:

The recovery process performs the following steps upon startup:

Recovery actions are logged in the file $II_SYSTEM/ingres/files/iircp.log.

If you are monitoring Vector startup after a machine reboot, the following messages are displayed:

If the transaction log contained uncommitted transactions when the machine failure occurred, the startup script pauses while the recovery process recovers transactions from the transaction log file. No messages are printed to the screen.

If you are in doubt as to whether recovery is taking place during startup, or to monitor the recovery process, use the following procedure.

Display the recovery process log file by typing the following command at the operating system prompt:

If the system is recovering, the recovery actions are logged to the IIRCP.LOG file. This indicates that Vector is automatically recovering from possible inconsistencies.

Messages are printed to the log file during recovery:

You can also use an operating system command to determine whether the recovery process is recovering transactions by checking to see if it is accumulating CPU time.

On Linux, you can also monitor the files in the database directory of the database you suspect of being the target of the updates that are being backed out. The following command entered at the operating system prompt shows whether data files are being updated:

The file most recently updated is listed first along with the time of last update.

If any of the monitoring techniques above indicate that transaction recovery is taking place, continue to monitor the recovery process until recovery has completed. When the recovery process is complete, CPU time is not accumulated.

After the recovery process has finished, restart the installation with the ingstart command. The ingstart utility first shuts down and brings up all required installation processes. Programs can connect to the databases.

If you receive an “inconsistent database” error after recovery is complete, it means updates and modifications were not properly completed or rolled back, and the database is therefore in an inconsistent state.

Following are examples of “inconsistent database” errors that indicate your database has become inconsistent:

Diagnose the cause and extent of an inconsistent database problem before you attempt to recover your database. Knowing the cause of the problem is essential to choosing the proper recovery procedures. Once a database has been rolled forward from a checkpoint, recovered from an operating system backup, or forced consistent, you cannot determine the cause of inconsistency.

To diagnose the cause and extent of an inconsistent database problem:

Common causes of inconsistent databases are:

To recover a database from an operating system backup that was made while the installation was running, see Recover an Inconsistent Database.

Changing Vector installation variables (such as II_SYSTEM, II_DATABASE, II_CHECKPOINT, II_JOURNAL, II_DUMP, or II_WORK) without using proper procedures, causes inconsistency between the information stored in the installation variables and those stored in the database configuration file “aaaaaaaa.cnf”.

Database inconsistency can occur if you move a database, table, application or some other object by using operating system commands rather than the supported Vector utilities. If the inconsistency is the result of moving a database from another location or installation without using unloaddb, you must remove the database using destroydb, recreate the database using createdb, and repopulate the database using the unloaddb utility.

A database file can become corrupted from hardware or software failures of various kinds. A data file can be inadvertently deleted by hand, but this is rare because only the user who owns the installation can write to the database directories.

If you are in doubt about whether transactions are being recovered, run the logstat utility and examine the “Status” field. It is marked RECOVER if in the recovery state. While a recovery is taking place, for example when restarting after a system failure, the recovery process requires time to read through the transaction log file to back out uncommitted transactions and complete fast-commit transactions. To users, the system appears to hang.

To examine the configuration file of your database, enter the infodb command at the command prompt. You can also use the Database menu in Actian Director or VDBA.

The following are rules that you should keep in mind about the recovery of transactions:

After a system failure, monitor recovery and always allow it to proceed until the “Recovery Complete” message appears in iircp.log.

By disabling the logging or recovery system, the DBA can temporarily turn off logging for the database to speed bulk loading of data. If logging has been turned off for this database, a NOLOGGING error message appears in the error log file. Typically this message is:

E_DM9050_TRANSACTION_NOLOGGING Database dbname has been updated by a session running with SET NOLOGGING defined.

If the database has become inconsistent, you can check for this error message by typing the following command:

If the NOLOGGING error message appears, logging was disabled on this database. If the NOLOGGING message in the error log was written later than the most recent checkpoint of this database, the database must be restored from the checkpoint. To determine if this is the case, compare the timestamp on the error message in errlog.log with the timestamp in the “checkpoint history” field of the output from the command infodb dbname.

For details on set nologging, see the User Guide.

Database inconsistencies can be caused by using unsupported hardware configurations on NFS. In systems that include Network File System (NFS) mounts, be aware that Vector:

To check your configuration, type mount at the operating system prompt. Make sure that the data directories (II_DATABASE, II_CHECKPOINT, II_JOURNAL and II_LOG_FILE) are not NFS-mounted from a remote node.

The recommended method of recovering an inconsistent database is to use the Restore operation in Actian Director or rollforwarddb operation in VDBA. You can also enter the rollforwarddb command from the command line.

If no Vector checkpoint exists, you can recover from an operating system backup.

The recommended way to make your inconsistent database consistent is to use rollforwarddb. It recovers the database from a previous checkpoint and, if journaling was enabled, applies the associated journals. For the full procedure, see the User Guide.

We do not support nor recommend the use of operating system backups as your primary means of ensuring database recoverability. The Vector checkpoint and journaling programs provide the secure way to ensure that your data is recoverable.

If you have an inconsistent database for which no checkpoints or operating system backups exist, you can still gain access to that database and attempt to salvage the data using the verifydb utility.

The verifydb utility can be used to unset the “inconsistent database” flag in the configuration file “aaaaaaaa.cnf”. This permits access to the database; it does not, however, make the data consistent. If verifydb is used to force access to a database that is inconsistent, the state of the database remains unknown. Such a database becomes unsupportable by technical support. Data can be lost, and problems occur weeks or months later. Technical support cannot diagnose the state of such a database because the built-in consistency checks have been overridden.

The format “verifydb -oforce_consistent” does not recover a database. It merely allows access and continued operation to a database that is in an inconsistent state.

If you must use verifydb to gain emergency access to data in an inconsistent database, do so as follows:

Most performance problems stem from multiple causes. A complete performance analysis must include each item in the Diagnostic Hierarchy section below. But if you need to resolve a specific performance problem quickly, focus your attention on that area. The information in this troubleshooting section is designed to assist you in resolving a performance issue after first isolating which factors are influencing performance.

Use the procedures in this section if a Vector tool “hangs,” that is, your program seems to start but nothing happens. Other programs such as SQL display header information, but when you issue a command, nothing happens.

Use the following flow chart to identify and isolate a system performance problem:

Use the following procedure to diagnose a DBMS server that is not responding due to logging system problems.

The problem states and resolutions of logstat fields are described in the following table:

Many other normal states appear on the status line. For additional information on reading logstat output, see the Logstat section in the Command Reference.

To avoid logfull aborts, examine the transaction processing strategy errors that caused the log file to fill. Consider the following points:

Resizing transaction logs involves these steps:

To determine whether a log requires resizing, use one of the following procedures.

Using VDBA’s Performance Monitor, start a monitoring session on the desired node:

Start the Interactive Performance Monitor utility (ipm).

Find the current peak transaction load by using the logstat utility to monitor the amount of log file that is in use during your peak hours. This allows you to choose the correct size for your log file. Because requirements change, you must monitor the log file regularly.

To resize the transaction file, follow these steps:

A failure in one of the two dual log files is most often symptomatic of hardware problems. In the event of such a failure, you must reestablish dual logging at the first opportunity.

To re-establish dual logging after a failure of one of the log files, follow these steps:

Good performance requires planning and regular maintenance. Make sure your operating system is configured with sufficient resources for Vector. Insufficient system resources cause deficient performance or prevent Vector from starting.

The following tools can help you identify operating system resource problems:

If Vector seems slow or unresponsive for no apparent reason, system resources may be insufficient. Follow these steps to diagnose the problem. Write down any error messages you receive when performing these steps:

Before contacting Actian Support, gather as much information as possible and have the following information available.

Save any relevant errors in each of the following logs:

Have the following information available: