ObjectStore Management

Chapter 6

High Availability of Data

Object Design provides a number of facilities for increased levels of reliability. In descending order of reliability the tools and capabilities are

This chapter discusses

Warm failover addresses the issue of local availability of data in the face of a system hardware or software failure. It consists of two sets of failover features - Server failover features and ObjectStore APIs useful to an ObjectStore client application writer.

Asynchronous replication addresses the issue of wide-area availability by providing a method for replicating data to another system, potentially at another geographic location.

Warm Failover

ObjectStore failover features help solve the local availability problem in cases of system hardware or software failure. The ObjectStore Release 5.1 failover system provides a two-part set of failover features. The Server-side failover system enables a database administrator to set up a configuration that provides the Server failover system. Client-side failover capabilities consist of a new set of ObjectStore APIs that enable client application writers to take advantage of this facility.

Failover provides nonstop service of client applications so that a single Server failure does not affect a running application. It does this by shifting processing to a secondary Server system in the event of a failure of the primary system.

A failover system is made up of two ObjectStore Servers, running on different machines, that share a disk. If the primary Objectstore Server fails, the client process continues operation by connecting to the secondary Server. Service is uninterrupted and you can continue to access and modify the database.

The primary Server process accepts connections from an ObjectStore client. The secondary Server does not accept connections from clients, but waits offline, ready to take over should the primary Server fail.

Restrictions
Failover can be used with rawfs partition databases. File databases are not accessible by a failover Server (nor are two-phase commits allowed).

Warm failover alone cannot guarantee 100% fault tolerance as the shared disk can crash, the network can fail, and so on.

Each of the ObjectStore Servers that implements a failover Server must have a rawfs and log file on a disk that is shared by the two machines upon which the Servers are running. They must also be running on the same software architecture.

The ObjectStore client run time, through the use of the new APIs, locates the on-line Server of the failover Server using a locator file that is local to the client. The locator file declares configuration information for a failover Server.

Configuration of the Shared Disk

The configuration depends on a shared, reliable disk (such as RAID 5) connected to both the primary and secondary Servers. The connection might be either by a dual-ported reliable disk (for Sun machines, for example), or by the disk, with the primary and secondary Servers being on the same bus (for Digital or HP systems, for example).

Server Configuration

Each Server in the failover Server pair has a parameter file. The parameter file must specify two things, the heartbeat time and the physical rawfs partitions. The Server parameter files for a failover Server pair must share the same physical partition and log file. The log file must be in the rawfs, not in a file.

You must set the Failover Heartbeat Time parameter in the Server parameter file and restart the Server with the -upgradeRAWFS switch set. To reconfigure without failover, remove the parameter from the Server parameter file and restart the Server with the -upgradeRAWFS switch. The next paragraph describes this parameter.

Failover Heartbeat Time
The Failover Heartbeat Time parameter must be specified if you are using a failover Server. This parameter can be set to between 2 and 60 seconds. The parameter defines how often a heartbeat message is written to disk. In the event of a failure, it takes five times Failover Heartbeat Time for the secondary Server to recognize the failure, initialize, and take over.

Example
Partition0: partition /dev/rdsk/c0t0d0s3
Failover Heartbeat Time: 2

How a Failover Database Is Located by the Client Application

The locator file provides a way of defining pairs of ObjectStore Servers that compose a failover Server. If a Server is not available and the locator file has an alternative Server, the client tries this Server to see if it can provide access to the rawfs database.

Logical Server Names

There are two physical Server processes responsible for serving one logical rawfs. Only one of the Server hosts should ever be recorded in the database pathnames referring to databases maintained on a failover Server. The Server host that is recorded as part of the rawfs database path is referred to as the logical Server host name.

The ObjectStore client application should always use the logical Server host name when manipulating a database on a failover Server. This is because, when an application creates a database on the failover Server and the secondary Server is on line, the pathname of the database includes the logical Server host name, not the name of the secondary Server that created the database. If the application uses the nonlogical Server name in a pathname, the exception err_not_supported is raised.

Declaring a Failover Server in a Locator File

See Chapter 5, Using Locator Files to Set Up Server-Remote Databases, for general information about locator files.

When using locator files to implement failover Servers, be certain that the locator file is in a file that is local to the client application so NFS is not involved in a failover situation.

Do not add failover Server declarations to pre-5.0 ObjectStore client system locator files. ObjectStore clients will fail when trying to parse a locator file containing failover Server declarations.

Locator file syntax has been extended to support failover Servers, as shown in the following failover Server clause:

FAILOVER_SERVER server_host_1 
      ALTERNATIVE_SERVER server_host_2 
      RECONNECT_TIMEOUT integer # in seconds
      RECONNECT_RETRY_INTERVAL integer # in seconds
END
server_host_1 is also the logical Server host name that is recorded in all database pathnames to databases of this failover Server. server_host_2 specifies the backup Server of a failover Server pair. Both values are required.

RECONNECT_TIMEOUT is the maximum amount of time that a client will attempt to reconnect to a failover Server before raising the exception err_broken_replicated_server_connection.

RECONNECT_RETRY_INTERVAL specifies how often the two failover Servers are pinged during the RECONNECT_TIMEOUT. RECONNECT_RETRY_INTERVAL should be less than or equal to RECONNECT_RETRY_TIMEOUT.

Also, RECONNECT_RETRY_INTERVAL cannot be zero if RECONNECT_TIMEOUT is nonzero. The exception err_locator_syntax is raised if these constraints are violated.

The locator file is read the first time it is needed by the ObjectStore run time. The locator file is not read again by the application unless the method objectstore::set_locator_file() is called. Then, the locator file is reread the next time the ObjectStore run time uses its contents.

Notes for Writing Client Applications

If an application is written using lexical transactions (that is, the transaction macros), the retry mechanism in the client is done automatically.

This means that client programs written using lexical transactions do not need to be modified to take advantage of the failover feature.

The Failover API

Failover features available to client application writers include a set of APIs that can be incorporated into client applications. These allow applications to access environment information for failover.

Use of these interfaces is not required. They include the os_failover_server class and functions in other classes, all described here.

objectstore::get_locator_file()

static char* get_locator_file() ;
Returns a string representing the locator file. If the first character of the string is a white-space character or #, the string is the contents of the file rather than a file name.

The caller should delete the returned value.

os_server::get_host_name()

char* get_host_name();
For failover Servers, this function returns the logical failover Server host name. Note that the logical Server name is not always identical to the Server name for the machine providing access to the database. The caller should delete the returned value. See os_failover_server::get_online_server().

os_server::is_failover()

os_boolean is_failover() const;
Returns true if and only if the Server is also a failover Server.

This method is used to identify the os_failover_server in the list of Servers returned by objectstore::get_all_servers().

The os_failover_server Class

class os_failover_server : public os_server
This class is derived from os_server.

The types os_int32 and os_boolean, used throughout this manual, are each defined as a signed 32-bit integer type. The type os_unsigned_int32 is defined as an unsigned 32-bit integer type.

Programs using this class must include <ostore/ostore.hh>, followed by <ostore/coll.hh> (if ObjectStore collections are used).

os_failover_server::get_logical_server_hostname()

char* get_logical_server_hostname() const;
Returns the logical name of a failover Server. A failover Server should always be referred to by its logical Server name.

The caller should delete the returned value.

os_failover_server::get_online_server_hostname()

char* get_online_server_hostname() const;
Returns the Server that the client is currently connected to, either the logical Server, alternative Server, or the empty string if there is no connection.

The caller should delete the returned value.

os_failover_server::get_reconnect_retry_interval()

os_unsigned_int32 get_reconnect_retry_interval() const;
Returns the frequency with which to ping both Servers composing a failover Server pair while attempting to reconnect to them.

os_failover_server::get_reconnect_timeout()

os_unsigned_int32 get_reconnect_timeout() const;
Returns the maximum amount of time that a client application will attempt to reconnect to a broken failover Server connection.

After this amount of time passes, the following exception is raised:

err_broken_failover_server_connection 

os_failover_server::set_reconnect_timeout_and_interval()

os_boolean set_reconnect_timeout_and_interval(
      os_unsigned_int32 total_timeout_secs, 
      os_unsigned_int32 interval_secs);
Sets the total amount of time to try to reconnect a broken connection to a failover Server. The interval_secs argument is used to control how frequently the Servers of a failover Server pair are pinged to see if they are available.

Returns true if the reconnect timeout has been reset with the specified parameters.

If the parameters are invalid, the function returns the value false and does not change the reconnect_timeout or reconnect_retry_interval.

Invalid parameters are those for which

Exceptions and Error Messages for Failover

This section presents run-time exception cases that can occur with failover.

err_failover_server_refused_connection
Raised when the initial connection to a failover Server pair cannot be made.

err_broken_failover_server_connection 
Raised when neither the logical nor alternative Servers comes back up in some predetermined maximum amount of time that a client process should wait for either Server to come up on its own.

err_server_restarted 
Raised when a failover Server connection is discovered to be lost, and then one of the logical or alternative Servers comes back up before RECONNECT_TIMEOUT. Lexical transactions are restarted when the exception err_server_restarted is raised within them.

err_not_supported 
Raised when the alternative Server name is used directly to reference a database. An Objectstore application should only reference the logical Server name of a failover Server pair.

Also raised when os_dbutil::ping_failover_server is called and the locator file does not declare hostname as a failover Server.

err_conflicting_failover_configuration 
Raised by os_dbutil::ping_failover_server() if both Servers composing the failover Server pair are alive or if a server stat of the on-line Server indicates that it is not a failover Server.

Asynchronous Replication

ObjectStore provides the osreplic utility, which produces a continuously updated copy (or replica) of one or more user databases. The utility works by coordinating the actions of a source ObjectStore Server running archive logger, and of a target ObjectStore Server running recover, providing a read-only (MVCC) copy of a database that is dynamically updated from the master database.

ObjectStore Release 4 and later databases and rawfs directories can be replicated, as well as all ObjectStore Release 4 file databases. Native file system directories cannot be replicated.

See osreplic: Replicating Databases for further information.



[previous] [next]

Copyright © 1997 Object Design, Inc. All rights reserved.

Updated: 03/26/98 20:44:34