OpenVMS Guide to System Security
188.8.131.52 Losing the Link to a Remote Log File
If you are writing auditing messages to a remote log file, as described
in Section 184.108.40.206, the link between the local and remote node can fail.
Should this happen, the audit server broadcasts a warning message to
all operator terminals and attempts to reestablish the link every
minute until the connection is made.
System Security Breaches
Along with developing a security policy and selecting appropriate
security measures to implement that policy, a site needs to establish
and test procedures for handling system, site, or network compromises.
The procedure should address two areas:
- Appropriate responses once a breach is suspected or confirmed.
Site guidelines should help determine whether to increase site security
(eliminating all possibility of further compromise), put proactive
measures in place to apprehend the offender, or collect evidence to
initiate a criminal or civil suit. Each decision has its own set of
rules and guidelines.
- Appropriate contacts and resources outside of the site that may be
needed should such an event occur. For example, a company might want to
become familiar with local, state, and federal authorities (as
applicable), local phone carriers (security division), and the Compaq
This chapter describes how to recognize when an attack on the system is
in progress or has taken place and what countermeasures can be taken.
1 Compaq support groups include the
Software Security Response Team (SSRT) in the United States and the
European Security Program Office (ESPO).
10.1 Forms of System Attacks
As security administrator, you must monitor the system on a regular
basis for possible security breaches. Following are the most common
forms of system attacks:
- Hunting for access lines
- Hunting for passwords
- Attempting a break-in
- Changing or creating user authorization file (UAF) records
- Granting/stealing extra privileges
- Introducing apparently innocent software (Trojan horse software)
that is intended to steal user passwords or do other damage to the
- Introducing viruses in command procedures and programs to gain
access to privileged accounts
- Scavenging disks
- Using a node as a gateway to other nodes
10.2 Indications of Trouble
When your system is vulnerable and possibly under attack, your first
indications may come from the following sources:
- Reports from users
- System monitoring, for example:
- Unexplained changes or behavior in applications or normal processes
- Unexplained messages from OPCOM or the audit server
- Unexplained changes to user accounts in the system authorization
database (privilege changes, protections, priorities, quotas)
10.2.1 Reports from Users
User observations frequently point to system security problems. A user
may contact you with the following situations:
- Files are missing.
- There are unexplained forms of last login messages, such as
successful logins the user did not perform or unexplained login
- A user cannot log in, suggesting the user password might have been
changed since the last successful login or some other form of tampering
- Break-in evasion appears to be in effect, and the user cannot log
- Reports from the SHOW USERS command indicate that the user is
logged in on another terminal when the user did not do so.
- A disconnected job message appears during a login for a process the
user never initiated.
- Files exist in the user's directories that the user did not create.
- Unexplained changes have been found in the protection or ownership
of user files.
- Listings appear that are generated under the user name without the
user requesting the listing.
- A sudden reduction occurs in the availability of resources, such as
Follow up promptly when one of these items is reported to you. You must
confirm or deny that the condition exists. If you find the complaint is
valid, seek a cause and solution.
10.2.2 Monitoring the System
Section 6.7 lists those tasks that can help you detect potential
security breaches on your system. The following list details possible
warning signs you may uncover while performing the recommended tasks:
- A user appears on the SHOW USERS report that you know could not be
currently logged in.
- You observe an unexplained change in the system load or performance.
- You discover media or program listings are missing or notice other
indications that physical security has degraded.
- Your locked file cabinet has been tampered with, and the list of
authorized users has disappeared.
- You find unfamiliar software in the system executable image library
[SYSEXE] or in [SYSLIB].
- You observe unfamiliar images running when you examine the MONITOR
- You observe unauthorized user names when you enter the DCL command
SHOW USER. When you examine the listing that the Authorize utility
(AUTHORIZE) produces with the SHOW command, you find that those users
have been given system access.
- You discover proxy users that you never authorized.
- The accounting report reveals unusual amounts of processing time
expended recently, suggesting outside access.
- You observe unexplained batch jobs on the batch queues.
- You observe unexpected device allocations when you enter the SHOW
- You observe a high level of processing activity at unusual hours.
- The protection codes or the access control lists (ACLs) change on
critical files. Identifiers are added, or holders of identifiers are
added to the rights list.
- There is high personnel turnover or low morale.
All these conditions warrant further investigation. Some indicate that
you already have a problem, and some may have simple explanations,
while others may indicate serious potential problems.
10.3 Routine System Surveillance
The operating system provides a number of mechanisms that allow
systematic surveillance of the activity in your system. There are many
mechanisms available for monitoring the system either manually or by
user-written command procedures, for example:
- Accounting utility (ACCOUNTING)
- Authorize utility (AUTHORIZE)
- Install utility (INSTALL)
- System Management utility (SYSMAN)
Proper use of such mechanisms should help you verify settings, alert
you to problems, and allow you to intervene. This section describes the
most important system surveillance mechanisms--ACCOUNTING and
10.3.1 System Accounting
You can learn what the normal pattern of resource use is by studying
reports of the Accounting utility (ACCOUNTING). To obtain a report, you
run the utility image SYS$SYSTEM:ACC.EXE. The resulting data file is
SYS$MANAGER:ACCOUNTNG.DAT. Review ACCOUNTING reports because they can
provide early indications of problems. Check for the following:
- Unfamiliar user names
- Unfamiliar patterns of use, such as unusual activity for a
particular time of day or day of week
- Use of an unusual amount of resources
- Unfamiliar sources of login, such as network nodes or remote
10.3.2 Security Auditing
As the security administrator, you can have the operating system report
on security-related activity by enabling categories of events for
auditing using the DCL command SET AUDIT. Using the Audit Analysis
utility (ANALYZE/AUDIT), you can periodically review event messages
collected in the security audit log file. (See Chapter 9 for a full
description of the process.)
The operating system can send event messages to an audit log file or to
an operator terminal. You define whether events are reported as audits
or alarms in the following way:
- Ordinarily, enable audits rather than alarms for security-related
events because the audit records are written to the system security
audit log where you can study them in volume and archive log files for
future reference. While an isolated auditing message may offer little
insight, numerous audit records produce a pattern of security
violations. For example, with auditing of object access, you can see a
pattern of time, types of objects being accessed, and other system
information that, in total, paint a picture of how the system is being
used at different times of day.
To enable audits for unsuccessful
access to files, devices, and volumes, enter the following command:
$ SET AUDIT/AUDIT/ENABLE=ACCESS=FAILURE/CLASS=(FILE,DEVICE,VOLUME)
This command records unsuccessful access events in the security
audit log file but sends no alarms to the operator terminal.
- Enable security alarms for real-time events or events that should
be reviewed immediately, for example, intrusion attempts or changes to
the system user authorization file (SYSUAF.DAT). For example, to enable
alarms for modification to the known file list and changes to system
time, enter the following command:
$ SET AUDIT/ALARM/ENABLE=(INSTALL,TIME)
This command sends event messages to the operator terminal. To keep
a hardcopy record of these alarms, use a hardcopy operator terminal, or
enable the events as both alarms and audits.
Because security auditing affects system performance, enable auditing
only for the most important events. The following security-auditing
actions are presented in order of decreasing priority and increasing
- Enable security auditing for login failures and break-ins. This is
the best way to detect probing by outsiders (and insiders looking for
accounts). All sites needing security should enable alarms for these
- Enable security auditing for logins. Auditing successful logins
from the more suspicious sources like remote and dialup users provides
the best way to
track which accounts are being used. An audit record is written before
users logging in to a privileged account can disguise their identity.
- Enable security auditing for unsuccessful file access
(ACCESS=FAILURE). This technique audits all file-protection violations
and is an excellent method of catching probers.
- Apply ACL-based file access auditing to detect write access to
critical system files. The most important files to audit are shown in
Table 10-1. (Table 9-2 presents an example of how to establish
security entries in ACLs.)
You may want to audit only successful access to these files to detect
penetration, or you may want to audit access failures to detect probing
Note that some of the files in Table 10-1 are written
during normal system operation. For example, SYSUAF.DAT is written
during each login, and SYSMGR.DIR is written when the system boots.
Table 10-1 System Files Benefiting from ACL-Based Auditing
|Device and Directory
- Enable security auditing for modifications to system parameters or
the known file list (/ENABLE=(SYSGEN,INSTALL)).
- Audit use of privilege to access files (either write access or all
forms of access). Implement the security audit with the keywords
ACCESS=(SYSPRV,BYPASS,READALL,GRPPRV). Note that this class of auditing
can produce a large volume of output because privileges are often used
in normal system operation for such tasks as mail delivery and operator
Section 9.3 provides further discussion of recommended sets of
security events to audit.
10.4 Handling a Security Breach
There are four phases that security administrators experience while
handling a security breach, whether the breach actually occurred or was
- Detection of a problem
- Identification of the perpetrator
- Prevention of further security violations
- Repair of damage
The following sections describe these phases for both attempted and
In all phases, train personnel to retain information and data as
evidence, should there be a need to apprehend and prosecute the
10.4.1 Unsuccessful Intrusion Attempts
Unsuccessful intrusion attempts include situations where someone has
attempted to guess passwords or browse through files.
10.4.1.1 Detecting Intrusion Attempts
You usually detect intrusion attempts through the following sources:
- Reports from users about unexplained login failures
- Unusual system activity or unavailability of dialup lines
- Security alarms for login failures, break-in attempts, and
- Examination of the intrusion database
10.4.1.2 Identifying the Perpetrator
Enabling file auditing simplifies identification of file browsers. If,
however, browsing is being initiated from another node in the network,
you must inspect the network server log file (NETSERVER.LOG) that
corresponds to the times of the protection violations. Coordinate your
investigation with the security administrator at the remote node.
Identifying a perpetrator who is guessing passwords is considerably
more difficult, especially when the source is anonymous, as from a
dialup line. Usually, you must trade identification for prevention.
Often the only way to positively identify an outsider attempting to
enter the system requires that you permit further attempts while
establishing the perpetrator's identity.
10.4.1.3 Preventing Intrusion Attempts
The prevention phase for this kind of attack involves preventing the
would-be intruder from actually gaining access to the system and making
future attempts more difficult.
To reduce the opportunities for successful password guessing:
- Make certain your users choose appropriate passwords. Consider use
of the password generator (see Section 220.127.116.11).
- Enable system passwords at the points of entry. While a minor
inconvenience to your users, system passwords are the best protection
against further probing. If you already had a system password enabled,
change it (see Section 18.104.22.168).
- Enable auditing of successful logins to catch the event if the
intruder succeeds in getting in (see Section 10.3.2).
To reduce the opportunities for successful file browsing:
- If you can identify the perpetrator, take action as established at
- Warn your users about the importance of adequate protection of
their files, and consider inspecting the protection of user files.
- If file browsing from other nodes in the network becomes a
persistent problem, eliminate the default FAL account and authorize
individual users through proxy login accounts (see Section 12.3.2).
10.4.2 Successful Intrusions
A successful security breach can include a successful password guessing
scheme, theft or modification of either information or system
resources, and placement of damaging software on the system. An
intrusion may require a considerable amount of time to repair,
depending upon the skill and intent of the perpetrator.
10.4.2.1 Identifying the Successful Perpetrator
Identification is often the most difficult part of handling an
intrusion. First, you must establish whether the perpetrator is an
authorized user or not. This determines the nature of the preventive
measures that you will take. However, the distinction between insiders
and outsiders may be difficult to achieve.
Tradeoff Between Identification and Prevention
You may have to make a tradeoff between a positive identification of
the intruder and preventing future attacks. Often, the data available
initially does not allow complete identification. If it is important to
identify the perpetrator, you will often find it necessary to permit
continued intrusions while you analyze the intrusion activity. Increase
your auditing. Consider planting traps in system procedures that are
under your control (such as SYLOGIN.COM) to obtain additional
information. Increase your system backup efforts to permit easier
recovery if files become damaged.
Identification of Outsiders
Identifying external intruders is particularly difficult, especially if
they use any switched forms of communication (such as dialup lines or
public data networks). DECnet for OpenVMS software provides many
features to help you trace the activity through the network back to the
source node. If a local terminal is involved, physical surveillance may
When a switched connection is involved, one of the major computer
security problems is the telephone system itself. Tracing a telephone
or public data network connection is time-consuming. Chasing an
intruder through the telephone system is likely to take months and will
require the assistance of law enforcement authorities. The existence of
multiple long-distance telephone services compounds the problem by
increasing the number of organizations with whom you must deal.
As a result, identifying an outside intruder is usually worthwhile only
when you have sustained substantial financial damage. In many cases, it
may be more useful if you concentrate on preventing recurrences of the
10.4.2.2 Securing the System
The actions you must take to secure your system after an intrusion
depend on the nature and source of that intrusion. This section
describes these actions in order of priority.
- Restore SYSUAF.DAT, NETPROXY.DAT, NET$PROXY.DAT and RIGHTSLIST.DAT
(if damaged) from backups. Alternatively, generate listings of the
files and inspect them closely, looking for improper entries,
additional privileges, and changed UICs. If you are unsure of when
SYSUAF.DAT might first have been modified, inspect it carefully
regardless of whether you are using a backup copy or proceeding with
the existing one. Be sure all authorization files are secure.
- The perpetrator may have discovered passwords by browsing either
through files or from other nodes in the network and may be using
seldom accessed accounts for personal use. Change passwords for
accounts, and have your users appear in person to learn their new
passwords. At a minimum, change passwords on all privileged accounts.
Do not use the same new password for all accounts.
- A sophisticated penetrator may have planted ways to provide future
access to the system even though you have taken the obvious steps of
securing your system. Therefore, you may have to restore selected
components of the OpenVMS software from backups or from your OpenVMS
distribution kit. If the intruder was an outsider, the two critical
components are LOGINOUT.EXE and NETACP.EXE, which validate all entries
to the system.
However, if the intruder was an authorized user,
restore all system files from backup copies. Authorized users can make
use of a wide variety of illicit software patches (called trap
doors) that they insert in the executive (SYS.EXE), the file
system (F11BXQP.EXE), DCL, and other system files. The penetrator may
have planted damaging software in any piece of software or command
procedure likely to be used by a privileged user. Thus, complete
assurance of a secure system requires a wholesale restoration of files
from backups. Also reinstall any image (even from layered products)
installed with privileges because it can also be used for a trap door.
An alternate strategy is to restore trustworthy copies of the obvious
targets of attack and to rely on increased auditing for a period of
time to catch suspicious events.
- Consider implementing additional security features, such as system
passwords, password generation, increased auditing, and more stringent
file protection to prevent a recurrence.
10.4.2.3 Repair After a Successful Intrusion
After an intrusion, restore corrupted files. Decide whether it is
appropriate either to do a wholesale restoration of your system's data
or to repair problems as they are discovered. Look for modifications to
file protection that would have created paths for viruses and for
Trojan horses that were introduced into the system and may still reside