2
Administration Concepts

This chapter describes the concepts necessary for understanding ConText administration.

The following concepts are discussed in this chapter:

• Administrator Responsibilities
• ConText Roles
• Predefined ConText Users
• ConText Data Dictionary
• ConText Servers
• Personalities
• Text Request Queue

Administrator Responsibilities

Administrative responsibilities for ConText can be divided into two functional areas:

• System Administrator
• Database Administrator (DBA)

System Administrator

The system administrator's main responsibility is managing ConText servers. This includes:

• starting and stopping ConText servers to balance the processing loads of the machines on which the servers are running
• changing the personality masks of ConText servers as needed

Database Administrator (DBA)

The DBA's main responsibilities include:

• creating and maintaining Oracle users for ConText
• monitoring and administering the ConText queues
• monitoring the ConText data dictionary

ConText Roles

ConText provides database roles for performing the following tasks with ConText:

• managing ConText servers, including shut down, monitoring ConText server status, and changing the personality masks for ConText servers
• administering ConText queues, including removing entries from the queues changing the status of the queues, and setting the buffer size of the Query pipes
• managing the ConText data dictionary, including creating/deleting preferences, creating/deleting/modifying policies, creating/deleting ConText indexes, creating/deleting/updating thesauri
• managing linguistics, including creating custom setting configurations for linguistics, enabling setting configurations for linguistics, generating linguistic output
• performing ConText queries

Three database roles are provided for performing these tasks:

• CTXADMIN Role
• CTXAPP Role
• CTXUSER Role

Each ConText user should be assigned one of the ConText roles. The role assigned to a ConText users depends on the tasks performed by the user.
 

Note:  It is not necessary to assign more than one ConText role to a user because the roles are defined hierarchically in the following descending order: CTXADMIN, CTXAPP, CTXUSER.  Each higher-level role provides all the privileges of the lower-level roles, as well as additional privileges.

    

See Also:  For an example of assigning ConText roles to users, see "Managing Users" in Chapter 3, "Administering ConText".  For more information about Oracle users and database roles, see Oracle8 SQL Reference.

   

CTXADMIN Role

The CTXADMIN role provides users with the ability to perform all ConText tasks.

CTXADMIN should be assigned to Oracle users who perform system and database administration for ConText.

CTXAPP Role

The CTXAPP role users with the ability to perform the following tasks/actions:

• managing the ConText data dictionary
• managing linguistics
• performing queries

CTXAPP should be assigned to Oracle users who develop ConText applications.

CTXUSER Role

The CTXUSER role provides users with the ability to perform ConText queries (text and theme). It should be assigned to Oracle users of a ConText application.

Predefined ConText Users

ConText provides two predefined Oracle users:

• CTXSYS User
• CTXDEMO User

CTXSYS User

CTXSYS is created automatically during installation and is used primarily to perform administration tasks, such as starting ConText servers and administering the ConText queues.

CTXSYS has the following functions and privileges:

• Oracle DBA with all privileges
• owner of the ConText data dictionary
• CTXADMIN role

 

Note:  Only the CTXSYS user can start ConText servers.

   

CTXDEMO User

CTXDEMO is created automatically if you choose to enable the ConText demos during installation. It is used primarily to run the sample applications provided with ConText.

CTXDEMO has the following functions and privileges:

• owner of the ConText database schema (tables, views, etc.) and data dictionary objects (policies, preferences) used in the sample applications
• CTXAPP role

The sample applications, which illustrate some typical uses of ConText, can be divided into three groups:

• Sample Populated Database Tables
• Sample SQL Scripts
• Sample Oracle Forms Application

Sample Populated Database Tables

This sample consists of two tables, EMP and DEPT, owned by CTXDEMO. Each table has a text column containing sample plain (i.e. ASCII) English text and a text index associated with the column. These tables can be used to perform ad-hoc text queries to familiarize yourself with ConText's basic features.

The tables and their associated ConText objects are created automatically if you choose to enable the ConText demos when prompted during ConText installation.
 

See Also:  For more information about text columns and text indexes, see Chapter 6, "Text Concepts".  For more information about text queries, see Oracle8 ConText Cartridge Application Developer's Guidee.

   

Sample SQL Scripts

This sample consists of a populated table, ARTICLES, and two sets of SQL scripts, CTXPLUS and CTXLING, for performing the following tasks:

• creating a text index for ARTICLES
• creating result/output tables for highlighting and ConText Linguistics
• generating Linguistic output for documents in ARTICLES
• performing text queries using different query methods
• viewing documents and highlighted documents
• viewing Linguistic output for documents

The table and its associated ConText objects are not created automatically during ConText installation. They must be created after ConText installation using setup scripts provided with the sample scripts.
 

See Also:  For more information about text indexes, see Chapter 6, "Text Concepts".  For instructions on setting up the sample scripts, as well as information about text queries, highlighting, and ConText Linguistics, see Oracle8 ConText Cartridge Application Developer's Guide.

   

Sample Oracle Forms Application

This sample consists of an Oracle Forms application that uses the ARTICLES table (from the SQL scripts sample) to illustrate how to incorporate text queries and document highlighting/viewing into a ConText application.

The Oracle Forms sample is distributed with the ConText Workbench and requires the same setup as the SQL scripts for CTXQUERY.
 

Note:  The Oracle Forms sample uses the same ARTICLES table and ConText objects as the CTXQUERY sample; once setup has been performed for CTXQUERY, you do not need to perform any setup for the Oracle Forms sample.

    

See Also:  For instructions on setting up the CTXQUERY and Oracle Forms samples, as well as information about text queries and highlighting, see Oracle8 ConText Cartridge Application Developer's Guide.  For a description of the Oracle Forms sample, including the source code for the sample, see Oracle8 ConText Cartridge Workbench User's Guide

   

ConText Data Dictionary

ConText utilizes a data dictionary, separate from the Oracle data dictionary, to store the ConText objects required for index creation, Linguistic output generation, and automated text loading:

• template policies
• column policies and the table/column to which each column policy is assigned
• indexing preferences (predefined and user-defined) and the Tiles used to create them
• sources for automated batch-loading of text into database columns
• text loading preferences (predefined and user-defined) and the Tiles used to create them

The ConText data dictionary also stores resource limits and the status of all ConText servers that are currently running.

The ConText data dictionary is owned by the Oracle user CTXSYS. CTXSYS and the data dictionary tables and views are created during installation of ConText.
 

See Also:  For more information about policies and indexing preferences, see Chapter 8, "ConText Indexing".  For more information about sources and text loading preferences, see Chapter 7, "Automated Text Loading".

   

ConText Servers

ConText servers are shared processes that handle text operations in SQL requests. ConText server processes mirror their shared Oracle server counterparts, but process only text-related operations. ConText servers can be started using the ctxsrv executable or the ctxctl utility.

ConText servers can be started only from the command-line of the machine on which the ctxsrv executable is installed. In addition, only the CTXSYS user can start ConText servers.

If the server machine can support multiple ConText servers, multiple ConText servers can run at the same time to help balance the processing load. In addition, ConText servers can work with databases installed on either the server machine or on remote machines.
 

See Also:  For more information about ctxsrv and ctxctl, see Chapter 4, "ConText Server Executable and Utility".

   

Text Operations

ConText servers can process five types of text operations:

• text loading
• DDL operations (creating, dropping, and updating text indexes)
• DML operations (updating the text index for a text column when inserts, updates, and deletes are performed on the table)
• Linguistics requests (generating linguistic output for documents)
• text/theme queries

The type of text operations processed by each ConText server is determined by the personalities assigned to the server.

Requests for text operations are stored in the Text Request Queue. Available ConText servers monitor the queue for text requests. As text operations are submitted, available ConText servers with the appropriate personalities pick up the operations and process them.
 

See Also:  For more information about each of the text operations that can be processed by ConText servers, see "Text Operations" in Chapter 6, "Text Concepts".  For more information about personality masks, see "Personalities" in this chapter.  For more information about the database tables and pipes that comprise the Text Request Queue, see "Text Request Queue" in this chapter.

   

Server Log

ConText provides a timestamped report for each action performed by a ConText server. These reports are written to the standard output on which the server was started; however, the reports can be directed to a log file if one is specified when the ConText server is started.
 

See Also:  For examples of starting ConText servers, see "Starting ConText Servers" in Chapter 3, "Administering ConText".

   

Server Shutdown

A ConText server performs the following tasks before shutting down:

• releases all currently held resources
• cleans up and closes the server's mailboxes
• updates the server process table by removing the corresponding entry for the server from the table

Personalities

A personality for a ConText server indicates the type of text operation that the server can process. A ConText server can be assigned one or more of the following personalities (corresponding to the five text operations supported by ConText):

• Loader (R) Personality
• DDL (D) Personality
• DML (M) Personality
• Query (Q) Personality
• Linguistic (L) Personality

In addition to the user-specified personalities, all ConText servers automatically take on a DBA Personality.
 

See Also:  For more information about the text operations supported by ConText, see "Text Operations" in Chapter 6, "Text Concepts".

   

Personality Masks

The collection of all the personalities for a server is called the personality mask. When a ConText server is started, it is assigned a default personality mask consisting of the DDL (D), DML (M), and Query (Q) personalities. The DBA personality does not appear as part of the personality mask because it is automatically assigned to all ConText servers.

Loader (R) Personality

The Loader personality enables a ConText server to scan directories at regular intervals for files to be loaded into columns in the database. When the ConText server detects a new file in a specified directory, it uses the ctxload utility to load the contents of the file into a specified column.

The directories scanned by ConText servers running with the Loader personality, as well as the scanning intervals and the columns to which the text is loaded, are specified by a ConText object, called a source, which can be attached to a column.
 

See Also:  For more information about automated text loading, see "Overview of Automated Loading" in Chapter 7, "Automated Text Loading".  For an example of setting up ConText servers to load text, see "Using ConText Servers for Automated Text Loading" in Chapter 9, "Setting Up and Managing Text".

   

DDL (D) Personality

The DDL personality enables a ConText server to process requests for creating, optimizing, and dropping text indexes. The text index on a column is what allows users to query the text stored in the column.

The DDL personality also enables a ConText server to process DML requests when DML operations are processed in batch mode.
 

See Also:  For more information about batch DML processing, see "DML" in Chapter 6, "Text Concepts".

   

DML (M) Personality

The DML personality enables a ConText server to automatically update the text index for a table when changes to the table are made that require the text index to be update. Such changes include inserting new documents, updating existing documents, and deleting existing documents.
 

Suggestion:  When systems have a high volume of text inserts, updates, and deletes, assign the DML personality to multiple servers to better distribute the system load.

   

Query (Q) Personality

The Query personality enables a ConText server to process queries for text stored either internally or externally in a text column in a database table. If no running ConText servers have the Query personality, queries submitted to ConText will fail.
 

Note:  The Query personality is not required to use the output generated by the Linguistics. Linguistic output is stored as structured data and, as such, no ConText servers are not required to process queries for this type of information.

   

Linguistic (L) Personality

The Linguistic personality allows a ConText server to process requests for the Linguistics and generate linguistic output. Linguistic output includes:

• document themes
• document Gists and/or document theme summaries

 

Note:  The Linguistic personality is only required to process requests for Linguistics. Once the requests have been processed, Linguistic servers can be shut down or the Linguistic personality can be removed from the personality mask of a running ConText server.  For more information about the ConText Linguistics, see Oracle8 ConText Cartridge Application Developer's Guide.

   

DBA Personality

The DBA personality allows a ConText server to detect and correct client/server failures and perform system cleanup (recovery).

The DBA personality is automatically assigned to each ConText server during start up, which prevents a single point of failure in a multi-server configuration.

ConText Server Monitoring

When a working ConText server detects a failed ConText server, it performs the following DBA actions:

• the working server cleans up the failed server's mailbox and logs the event
• if the failed server was processing a Query request, the working server releases any allocated resources
• if the failed server was processing a DDL request, the working server drops any tables that are not needed for recovery of the requested action
• if the failed server was processing a DML request, the working server places any uncompleted DML requests back in the queue
• if the failed server was processing a Query or DDL request, the working server notifies the waiting client

System Recovery

When a table has a text column with an existing ConText index and the table is dropped without first dropping the index and policy for the column, the index tables and policy for the column remain in the system until recovery is performed.

System recover is performed automatically by ConText servers approximately every fifteen minutes.

System cleanup can also be performed manually using CTX_ADM.RECOVER.

Text Request Queue

Figure 2-1

 

The Text Request Queue is the logical mechanism ConText servers use to process all text operations, except for automated text loading.

When a client submits a request for a text operation, such as a query, the request is sent to the Text Request Queue. All available ConText servers regularly scan the Text Request Queue, retrieve pending requests if they have the appropriate personality, and perform the requested operations. Figure 2-1 illustrates how ConText servers with different personalities access the Text Request Queue.

When a ConText server finishes processing a request of any type, it checks the pipes and queues for the next pending request.

The Text Request Queue is made up of the following database pipes and tables:

• Query Pipe
• DDL Pipe
• DML Queue
• Services Queue

In addition, each ConText server has a dedicated administration pipe for processing the administrative tasks that control its operation (e.g. server shutdown).

Query Pipe

When a SQL statement includes a ConText query (text or theme), the system dispatches the query to the query pipe.

To help regulate the flow of query requests, ConText provides a stored procedure, CTX_ADM.SET_QUERY_BUFFER_SIZE, which allocates the amount of memory used by the query pipe. In addition, the pipe can be disabled/enabled using CTX_ADM.UPDATE_QUEUE_STATUS.

DDL Pipe

ConText dispatches all requests for DDL operations (e.g. CREATE_INDEX, DROP_INDEX, and OPTIMIZE_INDEX) to the DDL pipe. The processing of DDL requests is controlled through CTX_ADM.UPDATE_QUEUE_STATUS, which can be used to disable/enable the DDL pipe.

If a ConText server encounters a problem with a request in the DDL pipe, the error does not affect the pipe or the server processing the pipe. The errored request is recorded as a row in the Services Queue and the server continues processing the remaining requests in the pipe.

The CTX_INDEX_ERRORS view can be used to display errored DDL requests.

DML Queue

The DML Queue stores index update requests when changes are made to a table containing a text column with a ConText index.

When a DML operation is performed (i.e. data in a text table is modified, deleted, or added), an index update request is automatically recorded in the DML Queue. Requests are placed in the DML Queue by internal database triggers that are created during the initial creation of a ConText index for a text column.

If DML processing is running in immediate mode, the next available ConText server with the DML (M) personality picks up the requests and updates the index as soon as resources and system load allow.

If DML processing is running in batch mode, the requests are stored in the queue until a user explicitly requests DML processing. Then, available ConText servers with the DDL (D) personality pick up all the pending requests and process the requests as one or more batches.

The DML Queue consists of the following internal tables:

• Pending Table (DRQ_PENDING)
• In-Progress Table (DRQ_INPROG)
• Waiting Table (DRQ_WAITING)
• Batches Table (DRQ_BATCHES)

 

Note:  The DML tables are internal tables and should not be accessed directly. To view the queue, use the queue views or the GUI administration tools provided with the ConText Workbench. To administer the queue, use the CTX_DML package.  For more information about the DML queue views, see "ConText Queue Views" in Appendix B, "ConText Views".  For more information about the CTX_DML package, see "CTX_DML: ConText Index Update" in Chapter 11, "PL/SQL Packages - Text Management".

    

See Also:  For more information about DML operations, see "DML" in Chapter 6, "Text Concepts".

   

Pending Table (DRQ_PENDING)

This table contains one row for each DML operation (request for index update) that has not yet been picked up by a ConText server. The row indicates the specific cell that has changed in the text table.
 

Note:  Requests are stored in the pending table only after the insert, update, or delete has been committed.

   

When a request has been picked up by a ConText server, the corresponding row is deleted from the pending table and the server beginsto update the ConText index. In addition, a new row is written to the in-progress table to indicate that the request is being processed.

In-Progress Table (DRQ_INPROG)

This table contains one row for each DML request being processed by a ConText server.

Once the ConText server finishes processing the request, the row is deleted from the in-progress table, indicating that the appropriate index has been updated to reflect the document modification that generated the request.

Waiting Table (DRQ_WAITING)

This table contains a row for each DML request for which a duplicate request exists in the pending table. This condition occurs when a DML request for a row has not been picked up by a ConText server and additional requests for the row are issued. This table ensures that DML requests for a row in a text table are processed in order.

Batches Table (DRQ_BATCHES)

This table contains one row for each batch of DML requests. A batch consist s of up to 10,000 DML requests for an indexed column. If the queue contains more than 10,000 DML requests for a column or requests for different columns, ConText uses multiple batches to process the requests.

For each batch, the table stores information such as the number of rows in the batch, the batch ID, and the ID of the server processing the batch.

If immediate DML is enabled, batches are created automatically as ConText servers with the DML personality pick up requests from the queue. If batch DML is enabled, batches are created by users calling CTX_DML.SYNC.
 

See Also:  For more information about immediate and batch DML, see "DML" in Chapter 6, "Text Concepts"

   

Timestamps

Requests in the DML Queue are processed in the order they are received, based on a timestamp. Rows are inserted into the pending table without a timestamp. At a specified time interval, all unmarked rows within the pending table are marked with a timestamp. The timestamp is based on the time each change was committed.

Error Handling

If a ConText server encounters a problem with a request in the DML Queue, the error does not affect the queue or the server processing the queue. The errored request is recorded as a row in the Services Queue and the server continues processing the remaining requests in the queue.

The CTX_INDEX_ERRORS view of the Services Queue can be used to display errored DML requests.

Queue Management

To control the processing of DML requests, CTX_ADM.UPDATE_QUEUE_STATUS can be used to disable/enable the DML Queue. When the DML Queue is disabled, the queue continues to accept requests; however, new requests and any pending requests in the disabled DML Queue are not picked up by ConText servers until the queue is enabled manually.

Services Queue

The Services Queue is used for processing requests for all ConText services. The Services Queue is designed to be extensible. As additional services are provided by ConText, the Services Queue is the mechanism by which the services will be managed. Currently, the Services Queue supports the following services:

• requests for the ConText Linguistics (theme, Gist, and theme summary generation)
• error handling for index creation and optimization (DDL) and index updating (DML)

When a request is submitted for the Linguistics, the request is stored in the Services Queue. A request is picked up by the first available ConText server with the Linguistic personality and the server generates linguistic output for the specified request.

ConText servers with the Linguistic personality pick requests out of the queue based on the request priority and creation timestamp. Clients may queue a request and continue to work while the request is being processed.

The Services Queue is asynchronous. Clients that place a request in the queue do not automatically block their subsequent requests while waiting for a reply. However, clients can choose to block their subsequent requests for a specified time when they submit requests.

Services Queue Table (CTX_SVCQ)

The Services Queue consists of the CTX_SVCQ internal table. This table stores a row for each request for the ConText Linguistics, as well as the request status.
 

Note:  CTX_SVCQ is an internal table and should not be accessed directly. To view the queue, use the queue views or the GUI administration tools provided with the ConText Workbench.  To administer the Services Queue (e.g. cleaning up entries), use the CTX_SVC package or the GUI administration tools.  For more information about the CTX_SVC package, see "CTX_DDL: Text Setup and Management" in Chapter 11, "PL/SQL Packages - Text Management".

   

Error Handling

If a ConText server encounters a problem with a request in the Services Queue, the error does not affect the queue or the server processing the queue. The errored request is recorded as a row in the Services Queue and the server continues processing the remaining requests in the queue.

The CTX_LING_ERRORS view of the Services Queue can be used to display errored requests for Linguistics.

Queue Management

To control the processing of Linguistic requests, CTX_ADM.UPDATE_QUEUE_STATUS can be used to disable/enable the Services Queue. When the Services Queue is disabled, requests are still accepted into the queue; however, new requests and any pending requests in the disabled Services Queue are not picked up by ConText servers until the queue is enabled manually.

---