HP OpenVMS Programming Concepts Manual
- The item list for the Get Device/Volume
Information (SYS$GETDVI) system service specifies that the unit number
of the mailbox is to be returned.
- The Create Mailbox and Assign Channel
(SYS$CREMBX) system service creates the mailbox and returns the channel
number at EXCHAN.
- The Create Process (SYS$CREPRC) system
service creates a process to execute the image LYRA.EXE and returns the
process identification at LYRAPID. The mbxunt argument
refers to the unit number of the mailbox, obtained from the Get
Device/Volume Information (SYS$GETDVI) system service.
- The Queue I/O Request (SYS$QIO) system
service queues a read request to the mailbox, specifying both an AST
service routine to receive control when the mailbox receives a message
and the address of a buffer to receive the message. The information in
the message can be accessed by the symbolic offsets defined in the
$ACCDEF macro. The process continues executing.
- When the mailbox receives a message, the AST
service routine EXITAST receives control. Because this mailbox can be
used for other interprocess communication, the AST routine does the
- Checks for successful completion of the I/O operation by examining
the first word in the IOSB
- Checks that the message received is a termination message by
examining the message type field in the termination message at the
- Checks for the process identification of the process that has been
deleted by examining the second longword of the IOSB
- Checks for the completion status of the process by examining the
status field in the termination message at the offset ACC$L_FINALSTS
In this example, the AST service routine performs special action
when the subprocess is deleted.
The Create Mailbox and Assign Channel (SYS$CREMBX), Get Device/Volume
Information (SYS$GETDVI), and Queue I/O Request (SYS$QIO) system
services are described in greater detail in Chapter 23.
Symmetric Multiprocessing (SMP) Systems
5.1 Introduction to Symmetric Multiprocessing
OpenVMS Alpha and OpenVMS I64 support tightly coupled symmetric
multiprocessing (SMP). This chapter presents a brief overview of
symmetric multiprocessing terms and characteristics. For more
information about SMP concepts and hardware configurations, refer to
VMS for Alpha Platforms Internals and Data Structures.
A multiprocessing system consists of two or more CPUs that address
common memory and that can execute instructions simultaneously. If all
CPUs in the system execute the same copy of the operating system, the
multiprocessing system is said to be tightly coupled. If all CPUs have
equal access to memory, interrupts, and I/O devices, the system is said
to be symmetric.
In most respects the members of an OpenVMS SMP system are symmetric.
Each member can perform the following tasks:
- Initiate an I/O request
- Service exceptions
- Service software interrupts
- Service hardware interrupts, such as interprocessor and interval
- Execute process context code in any access mode
5.2 CPU Characteristics of an SMP System
The members of an SMP system are characterized in several ways. One
important characteristic is that of primary CPU. During system
operation the primary CPU has several unique responsibilities for
system timekeeping, writing messages to the console terminal, and
accessing any other I/O devices that are not accessible to all members.
Although the hardware and software permit device interrupts to be
serviced by any processor, in practice all device interrupts are
serviced on the primary CPU. An SMP configuration may include some
devices that are not accessible from all SMP members. The console
terminal, for example, is accessible only from the primary processor.
5.2.1 Booting an SMP System
Booting the system is initiated on a CPU with full access to the
console subsystem and terminal, called the BOOT CPU. The BOOT CPU
controls the bootstrap sequence and boots the other available CPUs. On
OpenVMS Alpha and OpenVMS I64 systems, the BOOT CPU and the primary CPU
are always the same; the others are called secondary processors.
The booted primary and all currently booted secondary processors are
called members of the active set. These processors actively participate
in system operations and respond to interprocessor interrupts, which
coordinate systemwide events.
5.2.2 Interrupt Requests on SMP System
In an SMP system, each processor services its own software interrupt
requests, of which the most significant are the following:
- When a current Kernel thread is preempted by a higher priority
computable resident thread, the IPL 3 rescheduling interrupt service
routine, running on that processor, takes the current thread out of
execution and switches to the higher priority Kernel thread.
- When a device driver completes an I/O request, an IPL 4 I/O
postprocessing interrupt is requested: some completed requests are
queued to a CPU-specific postprocessing queue and are serviced on that
CPU; others are queued to a systemwide queue and serviced on the
- When the current Kernel thread has used its quantum of CPU time,
the software timer interrupt service routine, running on that CPU,
performs quantum-end processing.
- Software interrupts at IPLs 6 and 8 through 11 are requested to
execute fork processes. Each processor services its own set of fork
queues. A fork process generally executes on the same CPU from which it
was requested. However, since many fork processes are requested from
device interrupt service routines, which currently execute only on the
primary CPU, more fork processes execute on the primary than on other
5.3 Symmetric Multiprocessing Goals
SMP supports the following goals:
- One version of the operating system. As part of the standard
OpenVMS Alpha and OpenVMS I64 product, SMP support does not require its
own version. The synchronization methodology and the interface to
synchronization routines are the same on all systems. However, as
described in VMS for Alpha Platforms Internals and Data
Structures, there are different versions of the synchronization
routines themselves in different versions of the OpenVMS Alpha
executive image that implement synchronization. Partly for that reason,
SMP support imposes relatively little additional overhead on a
- Parallelism in kernel mode. SMP support might have been implemented
such that any single processor, but not more than one at a time, could
execute kernel mode code. However, more parallelism was required for a
solution that would support configurations with more CPUs. The members
of an SMP system can be executing different portions of the Executive
The executive has been divided into different
critical regions, each with its own lock, called a spinlock. A spinlock
is one type of system synchronization element that guarantees atomic
access to the functional divisions of the Executive using instructions
specifically designed for multi-processor configurations. Two sections
in Chapter 6, Section 6.6 and Section 6.7 describe both the
underlying architecture and software elements that provide this level
of SMP synchronization.
The spinlock is the heart of the SMP model,
allowing system concurrency at all levels of the operating system. All
components that want to benefit from multiple-CPU configurations must
incorporate these elements to guarantee consistency and correctness.
Device drivers, in particular, use a variant of the static system
spinlock (a devicelock) to ensure its own degree of synchronization and
ownership within the system.
- Symmetric scheduling mechanisms. The standard, default behavior of
the operating system is to impose as little binding between system
executable entities and specific CPUs in the active set as possible.
That is, in general, each CPU is equally able to execute any Kernel
thread. The multi-processor scheduling algorithm is an extension of the
single-CPU behavior, providing consistent preemption and real-time
behavior in all cases.
However, there are circumstances when an
executable Kernel thread needs system resources and services possessed
only by certain CPUs in the configuration. In those non-symmetric
cases, OpenVMS provides a series of privileged, system-level CPU
scheduling routines that supersedes the standard scheduling mechanisms
and binds a Kernel thread to one or more specific CPUs. System
components that are tied to the primary CPU, such as system timekeeping
and console processing, use these mechanisms to guarantee that their
functions are performed in the correct context. Also, because the Alpha
hardware architecture shows significant performance benefits for Kernel
threads run on CPUs where the hardware context has been preserved from
earlier execution, the CPU scheduling mechanisms have been introduced
as a series of system services and user commands. Through the use of
explicit CPU affinity and user capabilities, an application can be
placed throughout the active set to take advantage of the hardware
context. Chapter 4 in Section 4.4 describes these features in
Synchronizing Data Access and Program Operations
This chapter describes the operating system's synchronization features.
It focuses on referencing memory and the techniques used to synchronize
memory access. These techniques are the basis for mechanisms OpenVMS
itself uses and for mechanisms OpenVMS provides for applications to use.
This chapter contains the following sections:
Section 6.1 describes synchronization, execution of threads, and
Section 6.2 describes alignment, granularity, ordering of read and
write operations, and performance of memory read and write operations
on VAX and Alpha systems in uniprocessor and multiprocessor
Section 6.3 describes how alignment and granularity affect the access
of shared data on I64 systems. It also discusses the importance of the
order of reads and writes completed on I64 systems, and how I64 systems
perform memory reads and writes.
Section 6.4 describes memory read-modify-write operations on VAX and
Alpha systems in uniprocessor and multiprocessor environments.
Section 6.5 describes memory read-modify-write operations on I64
Section 6.6 describes hardware-level synchronization methods, such as
interrupt priority level, load-locked/store-conditional and interlocked
instructions, memory barriers, and PALcode routines.
Section 6.7 describes software-level synchronization methods, such as
process-private synchronization techniques, process priority, and spin
locks. It also describes how to write applications for a multiprocessor
environment using higher-level synchronization methods and how to write
to global sections.
Section 6.8 describes how to use local and common event flags for
Section 6.9 describes how to use SYS$SYNCH system service for
6.1 Overview of Synchronization
Software synchronization refers to the coordination of events in such a
way that only one event happens at a time. This kind of synchronization
is a serialization or sequencing of events. Serialized events are
assigned an order and processed one at a time in that order. While a
serialized event is being processed, no other event in the series is
allowed to disrupt it.
By imposing order on events, synchronization allows reading and writing
of several data items indivisibly, or atomically, in order to obtain a
consistent set of data. For example, all of process A's writes to
shared data must happen before or after process B's writes or reads,
but not during process B's writes or reads. In this case, all of
process A's writes must happen indivisibly for the operation to be
correct. This includes process A's updates---reading of a data item,
modifying it, and writing it back (read-modify-write sequence). Other
synchronization techniques are used to ensure the completion of an
asynchronous system service before the caller tries to use the results
of the service.
6.1.1 Threads of Execution
Code threads that can execute within a process include the following:
- Mainline code in an image being executed by a kernel thread, or
- User-mode application threads managed and scheduled through the
POSIX threads library thread manager
- Asynchronous system traps (ASTs) that interrupt a kernel thread
- Condition handlers established by the process, which run after
- Inner access-mode threads of execution that run as a result of
system service, OpenVMS Record Management Services (RMS), and command
language interpreter (CLI) callback requests
Process-based threads of execution can share any data in the
per-process address space and must synchronize access to any data they
share. A thread of execution can incur an exception, which results in
passing of control to a condition handler. Alternatively, the thread
can receive an AST, which results in passing of control to an AST
procedure. Further, an AST procedure can incur an exception, and a
condition handler's execution can be interrupted by an AST delivery. If
a thread of execution requests a system service or RMS service, control
passes to an inner access-mode thread of execution. Code that executes
in the inner mode can also incur exceptions, receive ASTs, and request
Multiple processes, each with its own set of threads of execution, can
execute concurrently. Although each process has private address space,
processes can share data in a global section mapped into each process's
address spaces. You need to synchronize access to global section data
because a thread of execution accessing the data in one process can be
rescheduled, allowing a thread of execution in another process to
access the same data before the first process completes its work.
Although processes access the same system address space, the protection
on system space pages usually prevents outer mode access. However,
process-based code threads running in inner access modes can access
data concurrently in system space and must synchronize access to it.
Interrupt service routines access only system space. They must
synchronize access to shared system space data among themselves and
with process-based threads of execution.
A CPU-based thread of execution and an I/O processor must synchronize
access to shared data structures, such as structures that contain
descriptions of I/O operations to be performed.
Multiprocessor execution increases synchronization requirements when
the threads that must synchronize can run concurrently on different
processors. Because a process with only one kernel thread executes on
only one processor at a time, synchronization of threads of execution
within such a process is unaffected by whether the process runs on a
uniprocessor or on an SMP system. However, a process with multiple
kernel threads can be executing on multiple processors at the same time
on an SMP system. The threads of such a process must synchronize their
access to writable per-process address space.
Also, multiple processes execute simultaneously on different
processors. Because of this, processes sharing data in a global section
can require additional synchronization for SMP system execution.
Further, process-based inner mode and interrupt-based threads can
execute simultaneously on different processors and can require
synchronization of access to system space beyond what is sufficient on
Atomicity is a type of serialization that refers to
the indivisibility of a small number of actions, such as those
occurring during the execution of a single instruction or a small
number of instructions. With more than one action, no single action can
occur by itself. If one action occurs, then all the actions occur.
Atomicity must be qualified by the viewpoint from which the actions
appear indivisible: an operation that is atomic for threads running on
the same processor can appear as multiple actions to a thread of
execution running on a different processor.
An atomic memory reference results in one indivisible read or write of
a data item in memory. No other access to any part of that data can
occur during the course of the atomic reference. Atomic memory
references are important for synchronizing access to a data item that
is shared by multiple writers or by one writer and multiple readers.
References need not be atomic to a data item that is not shared or to
one that is shared but is only read.
6.2 Memory Read and Memory Write Operations for VAX and Alpha
This section presents the important concepts of
alignment and granularity and how
they affect the access of shared data on VAX and Alpha systems. It also
discusses the importance of the order of reads and writes completed on
VAX and Alpha systems, and how VAX and Alpha systems perform memory
reads and writes.
6.2.1 Accessing Memory
The term alignment refers to the placement of a data
item in memory. For a data item to be naturally aligned, its
lowest-addressed byte must reside at an address that is a multiple of
the size of the data item in bytes. For example, a naturally aligned
longword has an address that is a multiple of 4. The term
naturally aligned is usually shortened to
On VAX systems, a thread on a VAX uniprocessor or multiprocessor can
read and write aligned byte, word, and longword data atomically with
respect to other threads of execution accessing the same data.
In contrast to the variety of memory accesses allowed on VAX systems,
an Alpha processor may allow atomic access only to an aligned longword
or an aligned quadword. Reading or writing an aligned longword or
quadword of memory is atomic with respect to any other thread of
execution on the same processor or on other processors. Newer Alpha
processors with the byte-word extension also provide atomic access to
bytes and aligned words.
VAX and Alpha systems differ in granularity of data access. The phrase
granularity of data access refers to the size of
neighboring units of memory that can be written independently and
atomically by multiple processors. Regardless of the order in which the
two units are written, the results must be identical.
VAX systems have byte granularity: individual adjacent or neighboring
bytes within the same longword can be written by multiple threads of
execution on one or more processors, as can aligned words and longwords.
VAX systems provide instructions that can manipulate byte-sized and
aligned word-sized memory data in a single, noninterruptible operation.
On VAX systems, a byte-sized or word-sized data item that is shared can
be manipulated individually.
Alpha systems guarantee longword and quadword granularity. That is,
adjacent aligned longwords or quadwords can be written independently.
Because earlier Alpha systems support instructions that load or store
only longword-sized and quadword-sized memory data, the manipulation of
byte-sized and word-sized data on such Alpha systems may require that
the entire longword or quadword containing the byte- or word-sized item
be manipulated. Thus, simply because of its proximity to an explicitly
shared data item, neighboring data might become shared unintentionally.
Manipulation of byte-sized and word-sized data on such Alpha systems
requires multiple instructions that:
- Fetch the longword or quadword that contains the byte or word
- Mask the nontargeted bytes
- Manipulate the target byte or word
- Store the entire longword or quadword
On such Alpha systems, because this sequence is interruptible,
operations on byte and word data are not atomic. Also, this change in
the granularity of memory access can affect the determination of which
data is actually shared when a byte or word is accessed.
On such Alpha systems, the absence of byte and word granularity has
important implications for access to shared data. In effect, any memory
write of a data item other than an aligned longword or quadword must be
done as a multiple-instruction read-modify-write sequence. Also,
because the amount of data read and written is an entire longword or
quadword, programmers must ensure that all accesses to fields within
the longword or quadword are synchronized with each other.
Alpha systems with the byte-word extension provide instructions that
can read or write byte-size and aligned word-sized memory data in a
single noninterruptible operation.
6.2.2 Ordering of Read and Write Operations
On VAX uniprocessor and multiprocessor systems, write operations and
read operations appear to occur in the same order in which you specify
them from the viewpoint of all types of external threads of execution.
Alpha uniprocessor systems also guarantee that read and write
operations appear ordered for multiple threads of execution running
within a single process or within multiple processes running on a
On Alpha multiprocessor systems, you must order reads and writes
explicitly to ensure that they occur in a specific order from the
viewpoint of threads of execution on other processors. To provide the
necessary operating system primitives and compatibility with VAX
systems, Alpha systems provide instructions that impose an order on
read and write operations.
6.2.3 Memory Reads and Memory Writes
On VAX systems, most instructions that read or write memory are
noninterruptible. A memory write done with a noninterruptible
instruction is atomic from the viewpoint of other threads on the same
On VAX systems, on a uniprocessor system, reads and writes of bytes,
words, longwords, and quadwords are atomic with respect to any thread
on the processor. On a multiprocessor, not all of those accesses are
atomic with respect to any thread on any processor; only reads and
writes of bytes, aligned words, and aligned longwords are atomic.
Accessing unaligned data can require multiple operations. As a result,
even though an unaligned longword is written with a noninterruptible
instruction, if it requires multiple memory accesses, a thread on
another CPU might see memory in an intermediate state. VAX systems do
not guarantee multiprocessor atomic access to quadwords.
On Alpha systems, there is no instruction that performs multiple memory
accesses. Each load or store instruction performs a maximum of one load
from or one store to memory. On an Alpha processor without the
byte-word extension, a load can occur only from an aligned longword or
quadword; a store can occur only to an aligned longword or quadword. On
an Alpha processor with the byte-word extension, a load can also occur
from a byte or an aligned word; a store can also occur to a byte or an
On Alpha systems, although reads and writes from one thread appear to
occur ordered from the viewpoint of other threads on the same
processor, there is no implicit ordering of reads and writes as seen by
threads on other processors.