HP OpenVMS Systems Documentation

Content starts here

OpenVMS Guide to System Security

Previous Contents Index

Chapter 13
Using Protected Subsystems

For the most part, the OpenVMS operating system bases its security controls on user identity. Protected objects, such as files and devices, are accessible to individual users or groups of users. If an object's ACL or protection code allows a user the necessary access, then the user can use that object by using any available software. (See Chapter 4 for a description of OpenVMS object protection.)

In a protected subsystem, an application protected by normal access controls serves as a gatekeeper to objects belonging to the subsystem. Users have no access to the subsystem's objects unless they execute the application serving as gatekeeper. Once users run the application, their process rights list acquires identifiers giving them access to objects owned by the subsystem. As soon as they exit from the application, these identifiers and, therefore, the users' access rights to objects are taken away.

This chapter describes protected subsystems and explains how to build them.

13.1 Advantages of Protected Subsystems

Using protected subsystems offers several advantages:

  • With protected subsystems, you have a mechanism to provide conditional access to data that is not available with traditional OpenVMS access controls. Traditionally, you give users privileges to bypass protection codes or access control lists (ACLs). In giving these privileges, however, you grant users a wide class of access. (Refer to Appendix A for information on the power different privileges carry.) Protected subsystems avoid extensive privilege use by individual users.
  • Protected subsystems give you an alternative to installing images with privilege. Writing a secure privileged image requires skill, and failures can compromise system security.
  • Protected subsystems give you an alternative to creating protected shareable images (also called user-written system services).
  • Protected subsystems make system management easier because unprivileged users can manage them without much assistance from you. See Section 13.5 for details on system management requirements.

13.2 Applications for Protected Subsystems

Protected subsystems have many applications, from databases to common system management situations.

One use for a protected subsystem might be a group membership list that you want to make available to all group members. The list contains the names, addresses, personnel numbers, and interests of group members. When the membership list is set up as a protected subsystem, all members of the group can read selected information and update specific types of information.

A protected subsystem might also solve the problem of confidential information being sent to printers in public areas. You could write an application to filter data for sensitive information. Confidential files would be sent to printers in restricted areas, while public files would be sent to any available printer. Any user with execute access to the application could use the restricted printers, but only through the protected subsystem.

13.3 How Protected Subsystems Work

A protected subsystem is an application that, when run, causes the process running the application to be granted one or more identifiers. For as long as a user runs the subsystem, the user's process rights list carries these additional identifiers. Figure 13-1 shows how a protected subsystem adds a second level of access control to traditional controls.

Figure 13-1 How Protected Subsystems Differ from Normal Access Control

Users with execute access to the application gain access to the subsystem. Once in the subsystem, users can work with the data files and other resources of the subsystem.

A subsystem can have several identifiers because the resources consumed by the subsystem (the files, printers, and so forth) can be protected differently.

Possession of subsystem identifiers is limited to the period users are executing the application. Once the users exit from the application, the identifiers are removed from their process rights lists. Subsystem identifiers are also removed from the rights list whenever users enter a Ctrl/Y sequence or attempt to create a subprocess with the DCL command SPAWN. (In this respect, use of the subsystem identifiers is identical to the operation of images installed with privileges.)

The following identifiers are reserved for use in the security subsystem and should not be granted to any user:


13.4 Design Considerations

Someone developing an application for a protected subsystem must link the application images without the /DEBUG or /TRACEBACK qualifiers.

Although this kind of subsystem often precludes the need for privilege, applications can be installed with privilege. For example, some applications may need the PRMGBL privilege to create permanent global sections, or they may need the AUDIT privilege to send security audit records to the system security audit log file. Compaq does discourage the installation of a protected subsystem application with privileges in the All category. This category includes such privileges as BYPASS, CMKRNL, and SYSPRV---privileges that allow a user to subvert OpenVMS access controls. See Table 8-2 for a list of OpenVMS privileges and Appendix A for a description of the privileges.

Subsystem designers need to generate a list of identifiers that are necessary for it to operate as intended. Then the designers approach you, as the security administrator, to make the preparations described in Section 13.5.

13.5 System Management Requirements

Although an unprivileged user can build and manage a protected subsystem, you need to be involved at two points in the process: at the beginning to create the necessary identifiers for the subsystem and at the end to mount the volume with the protected subsystem.

You need to perform the following tasks:

  1. Create identifiers for the subsystem, each with the Subsystem attribute. The Subsystem attribute empowers the identifier's holder to manage the subsystem.
  2. Grant these subsystem identifiers with Subsystem attributes to the people who will serve as managers of the subsystem. This enables them to assign the subsystem identifier to the images that make up the subsystem.
  3. Give the subsystem managers control access to application images. They need control access so they can add Subsystem ACEs to the image ACLs.
  4. Give the subsystem managers control access to existing resources that are to be managed by the protected subsystem.
    Although subsystem managers may need control access to key system resources, the ACL on the objects limits their access rights to only those resources. This may not be as dangerous as installing an image with SYSPRV.

The following example shows how you can set up identifiers and the necessary application access so that users can manage a membership list:


  1. Use AUTHORIZE to create a subsystem identifier called MEMBERS_SUBSYSTEM. Notice that this identifier carries the Subsystem attribute.
  2. Make Louis and Wu holders of the identifier so they can manage the subsystem.
  3. Give Louis and Wu control access to the subsystem image MEMBER_LIST.EXE.

Note that you create the subsystem identifier MEMBERS_SUBSYSTEM with the Resource attribute. This allows disk space to be charged to the identifier MEMBERS_SUBSYSTEM and not the individuals accessing the subsystem. (When using the Resource attribute, be careful to set the appropriate ACLs on directories [see Section].)

13.6 Building the Subsystem

Once managers of the subsystem have the appropriate identifiers and access rights as described in Section 13.5, they can add the necessary ACEs to a subsystem image. Two kinds of ACEs are necessary to construct a subsystem: the application image receives a Subsystem ACE, and the objects managed by the subsystem receive Identifier ACEs. Therefore, building a subsystem requires the following steps:

  1. Create a Subsystem ACE containing the subsystem identifier in the ACLs of the application images. A Subsystem ACE has the following format:


  2. Grant access to the objects managed by the subsystem. You need to add an Identifier ACE to the ACL of the various objects belonging to the subsystem. Each Identifier ACE contains one of the subsystem identifiers in the following format:

    (IDENTIFIER=identifier, ACCESS=access-type[+...])

In the following example, the subsystem manager uses the DCL command SET SECURITY to associate the subsystem identifier with the images that make up the subsystem. First, the subsystem manager adds a Subsystem ACE with the identifier MEMBERS_SUBSYSTEM to the ACL of the application image MEMBER_LIST.EXE:


Then the subsystem manager adds an Identifier ACE with the subsystem identifier MEMBERS_SUBSYSTEM to the data files managed by the subsystem:


The DCL command SHOW SECURITY displays the security attributes of the files. For example:

MEMBER_LIST.EXE object of class FILE
     Owner: [STAFF]
     Protection: (System: RWED, Owner: RWED, Group, World: RE)

MEMBER_DATA_1.DAT object of class FILE
     Protection: (System: RWED, Owner: RWED, Group, World)
MEMBER_DATA_2.DAT object of class FILE
     Protection: (System: RWED, Owner: RWED, Group, World)

13.7 Enabling Protected Subsystems on a Trusted Volume

A person with the SECURITY privilege can enable subsystems on a volume by using the /SUBSYSTEM qualifier on the MOUNT command. By default, subsystems are enabled only on the system disk. For other disks, you need to enable subsystems every time a volume is mounted.

In the following example, a security administrator uses the MOUNT command with the /SUBSYSTEM qualifier to enable the processing of Subsystem ACEs on device DUA0. Assume that this disk contains the subsystem with the identifier MEMBERS_SUBSYSTEM.


You can turn the processing of Subsystem ACEs on and off dynamically with the DCL command SET VOLUME /SUBSYSTEM. This command is especially useful for the system disk, which is not mounted using the MOUNT command.

Any person mounting a subsystem is responsible for knowing what is on the volume being mounted. Without this knowledge, an operator or system manager can inadvertently subvert system security. For example, it is easy for a user with privileges on one cluster to put an application holding a subsystem identifier on a volume and then take the volume to a naive operator on another cluster and request that it be mounted. Because the application holds an appropriate subsystem identifier, it feigns membership in a subsystem for which it is unauthorized. Therefore, mount volumes of only those users whom you trust, or thoroughly search a volume for Subsystem ACEs before you mount it with subsystems enabled.

13.8 Giving Users Access

All users with execute access to the main application image of the subsystem can use the data files and other objects under control of the subsystem if the subsystem allows the access. However, managers of the subsystem can restrict access to objects of the subsystem in the following ways:

  • They can create special identifiers for resources belonging to the subsystem that they do not want all members to access and add ACEs to these resources.
  • They can use compound expressions in ACEs and thus grant access conditionally. For example, the following ACE grants access to MEMBERS_ADMIN when running MEMBERS_SUBSYSTEM but not to MEMBERS_ADMIN alone nor to other users holding the MEMBERS_SUBSYSTEM identifier:


Remember that as long as users are executing the application image for the subsystem, their process rights list contains the subsystem identifier as well as their normal identifiers. However, as soon as users interrupt or exit from the application, their process rights list loses the subsystem identifier, and they lose access rights to the objects in the subsystem. Subsystem identifiers are not propagated by default when subprocesses are spawned.

13.9 Example of a Protected Subsystem

R. D. Taylor Inc., a company specializing in building supplies, decides to set up a protected subsystem for its purchasing and accounts payable departments. Although the departments are in different parts of the company, they share a common database for recording purchases from suppliers.

When the company's inventory drops below the desired level, the purchasing department is directed to order required supplies. Purchasing personnel find suppliers (if necessary), assign purchase order numbers, and issue a purchase orders.

When the goods arrive, the receiving and quality control departments check the contents against what was ordered, ensure the goods meet quality standards, and put the goods into inventory. Once the shipment is processed, the information goes to the accounts payable department, which settles the invoices.

Administrators in the accounts payable department check the invoices against purchase orders and run a payments program to calculate the monies due to suppliers each week. Payments are recorded in a database, and checks are printed on a printer loaded with company checks.

Using the subsystem lets the company meet two objectives:

  • It gives purchasing personnel the right to reference or record purchase orders in the company database, and it gives personnel in the accounts payable department the right to verify suppliers' invoices. Purchasing personnel with these tasks hold the SUPPLIERS_ORDERS identifier. Accounts payable personnel hold the ACCOUNTS_PAYABLE identifier. These employees run ORDERS.EXE to update the supplier information. The program stores data in ORDERS.DAT.
  • It gives trusted administrators in the accounts payable department the right to update databases, calculate payments due, and print checks. (One printer, loaded with company checks, is used for this purpose.) These administrators hold the ACCOUNTS_PAYABLE identifier.
    The administrators run PAYMENTS.EXE to perform these tasks. The program records payments made in the data file PAYMENTS.DAT.

The company appoints one employee, McGrey, to design and manage the subsystem. Figure 13-2 illustrates the directory structure of the Taylor subsystem, and Example 13-1 shows the command procedure McGrey wrote to implement it.

Figure 13-2 Directory Structure of the Taylor Company's Subsystem

13.9.1 Protecting the Top-Level Directory

McGrey implements a directory structure in which users can gain access to the subsystem only by holding an appropriate identifier: purchasing personnel hold the identifier SUPPLIERS_ORDERS, and the accounts payable administrators hold the identifier ACCOUNTS_PAYABLE. As subsystem manager, McGrey holds the identifier SUPPLIERS_SUBSYSTEM.

The top-level directory SUPPLIERS_SUBSYSTEM.DIR has the following protection:


Directory SYS$SYSDEVICE:[000000]

                     SUPPLIERS_SUBSYSTEM   (RWE,RWE,,)  (1)
          (CREATOR,ACCESS=NONE)  (2)
          (IDENTIFIER=*,ACCESS=NONE)  (7)


Total of 1 file.
  1. The directory's protection code gives read, write, and execute access to users in the system and owner categories but no access to group or world users. Therefore, group and world users have to gain access through the ACL.
  2. A Creator ACE ensures that users creating files in this directory have no special access to them. (See Section for information on Creator ACEs.)
  3. A Default Protection ACE denies group and world users access to files created in directory.
  4. McGrey holds the subsystem identifier SUPPLIERS_SUBSYSTEM. This ACE gives McGrey read, write, and control access so McGrey can manage the subsystem directories and images.
  5. Holders of the SUPPIERS_ORDERS identifier have execute access so they can access files in subdirectories.
  6. Holders of the ACCOUNTS_PAYABLE identifier have execute access so they can access files in subdirectories.
  7. Users holding any other identifiers have no access.
  8. McGrey added the Default attribute to all Identifier ACEs and includes them here so all Identifier ACEs are propagated to subdirectory ACLs.

13.9.2 Protecting Subsystem Directories

The directory EXE.DIR has the same protection as the top-level directory because subsystem users need to access the subsystem images: ORDERS.EXE and PAYMENTS.EXE. The other directory, LIB.DIR, is more restricted because only the subsystem images and McGrey need access.



EXE.DIR;1            SUPPLIERS_SUBSYSTEM   (RWE,RWE,,)              (1)
LIB.DIR;1            SUPPLIERS_SUBSYSTEM   (RWE,RWE,,)             (2)

Total of 2 files.
  1. [SUPPLIERS_SUBSYSTEM.EXE] has the same protection code and ACL as the parent directory shown in Section 13.9.1. Subsystem users need to run programs stored in this directory.
  2. [SUPPLIERS_SUBSYSTEM.LIB] has the same protection code but a more restrictive ACL because only the subsystem manager and the subsystem images need access.

13.9.3 Protecting the Images and Data Files

As the following listing shows, the necessary company personnel can access the subsystem's images, ORDERS.EXE and PAYMENTS.EXE, but only the images can update the data files:



Total of 2 files.

Directory SYS$SYSDEVICE:[SUPPLIERS_SUBSYSTEM.LIB]                 (3)


Total of 2 files.

Grand total of 3 directories, 6 files.

Previous Next Contents Index