OpenVMS Programming Concepts Manual
- The item list for the Get Device/Volume
Information (SYS$GETDVI) system service specifies that the unit number
of the mailbox is to be returned.
- The Create Mailbox and Assign Channel
(SYS$CREMBX) system service creates the mailbox and returns the channel
number at EXCHAN.
- The Create Process (SYS$CREPRC) system
service creates a process to execute the image LYRA.EXE and returns the
process identification at LYRAPID. The mbxunt argument
refers to the unit number of the mailbox, obtained from the Get
Device/Volume Information (SYS$GETDVI) system service.
- The Queue I/O Request (SYS$QIO) system
service queues a read request to the mailbox, specifying both an AST
service routine to receive control when the mailbox receives a message
and the address of a buffer to receive the message. The information in
the message can be accessed by the symbolic offsets defined in the
$ACCDEF macro. The process continues executing.
- When the mailbox receives a message, the AST
service routine EXITAST receives control. Because this mailbox can be
used for other interprocess communication, the AST routine does the
- Checks for successful completion of the I/O operation by examining
the first word in the IOSB
- Checks that the message received is a termination message by
examining the message type field in the termination message at the
- Checks for the process identification of the process that has been
deleted by examining the second longword of the IOSB
- Checks for the completion status of the process by examining the
status field in the termination message at the offset ACC$L_FINALSTS
In this example, the AST service routine performs special action
when the subprocess is deleted.
The Create Mailbox and Assign Channel (SYS$CREMBX), Get Device/Volume
Information (SYS$GETDVI), and Queue I/O Request (SYS$QIO) system
services are described in greater detail in Chapter 23.
Symmetric Multiprocessing (SMP) Systems
5.1 Introduction to Symmetric Multiprocessing
OpenVMS Alpha supports tightly coupled symmetric multiprocessing (SMP).
This chapter presents a brief overview of symmetric multiprocessing
terms and characteristics. For more information about SMP concepts and
hardware configurations, refer to VMS for Alpha Platforms Internals
and Data Structures.
A multiprocessing system consists of two or more CPUs that address
common memory and that can execute instructions simultaneously. If all
CPUs in the system execute the same copy of the operating system, the
multiprocessing system is said to be tightly coupled. If all CPUs have
equal access to memory, interrupts, and I/O devices, the system is said
to be symmetric.
In most respects the members of an OpenVMS SMP system are symmetric.
Each member can perform the following tasks:
- Initiate an I/O request
- Service exceptions
- Service software interrupts
- Service hardware interrupts, such as interprocessor and interval
- Execute process context code in any access mode
5.2 CPU Characteristics of an SMP System
The members of an SMP system are characterized in several ways. One
important characteristic is that of primary CPU. During system
operation the primary CPU has several unique responsibilities for
system timekeeping, writing messages to the console terminal, and
accessing any other I/O devices that are not accessible to all members.
Although the hardware and software permit device interrupts to be
serviced by any processor, in practice all device interrupts are
serviced on the primary CPU. An SMP configuration may include some
devices that are not accessible from all SMP members. The console
terminal, for example, is accessible only from the primary processor.
5.2.1 Booting an SMP System
Booting the system is initiated on a CPU with full access to the
console subsystem and terminal, called the BOOT CPU. The BOOT CPU
controls the bootstrap sequence and boots the other available CPUs. On
OpenVMS Alpha systems, the BOOT CPU and the primary CPU are always the
same; the others are called secondary processors.
The booted primary and all currently booted secondary processors are
called members of the active set. These processors actively participate
in system operations and respond to interprocessor interrupts, which
coordinate systemwide events.
5.2.2 Interrupt Requests on SMP System
In an SMP system, each processor services its own software interrupt
requests, of which the most significant are the following:
- When a current Kernel thread is preempted by a higher priority
computable resident thread, the IPL 3 rescheduling interrupt service
routine, running on that processor, takes the current thread out of
execution and switches to the higher priority Kernel thread.
- When a device driver completes an I/O request, an IPL 4 I/O
postprocessing interrupt is requested: some completed requests are
queued to a CPU-specific postprocessing queue and are serviced on that
CPU; others are queued to a systemwide queue and serviced on the
- When the current Kernel thread has used its quantum of CPU time,
the software timer interrupt service routine, running on that CPU,
performs quantum-end processing.
- Software interrupts at IPLs 6 and 8 through 11 are requested to
execute fork processes. Each processor services its own set of fork
queues. A fork process generally executes on the same CPU from which it
was requested. However, since many fork processes are requested from
device interrupt service routines, which currently execute only on the
primary CPU, more fork processes execute on the primary than on other
5.3 Symmetric Multiprocessing Goals
SMP supports the following goals:
- One version of the operating system. As part of the standard
OpenVMS Alpha product, SMP support does not require its own version.
The same OpenVMS version runs on all Alpha processors. The
synchronization methodology and the interface to synchronization
routines are the same on all systems. However, as described in VMS
for Alpha Platforms Internals and Data Structures, there are
different versions of the synchronization routines themselves in
different versions of the executive image that implement
synchronization. Partly for that reason, SMP support imposes relatively
little additional overhead on a uniprocessor system.
- Parallelism in kernel mode. SMP support might have been implemented
such that any single processor, but not more than one at a time, could
execute kernel mode code. However, more parallelism was required for a
solution that would support configurations with more CPUs. The members
of an SMP system can be executing different portions of the Executive
The executive has been divided into different
critical regions, each with its own lock, called a spinlock. A spinlock
is one type of system synchronization element that guarantees atomic
access to the functional divisions of the Executive using Alpha
architecture instructions specifically designed for multi-processor
configurations. Two sections in Chapter 6, Section 6.4 and
Section 6.5 describe both the underlying architecture and software
elements that provide this level of SMP synchronization.
spinlock is the heart of the SMP model, allowing system concurrency at
all levels of the operating system. All components that want to benefit
from multiple-CPU configurations must incorporate these elements to
guarantee consistency and correctness. Device drivers, in particular,
use a variant of the static system spinlock (a devicelock) to ensure
its own degree of synchronization and ownership within the system.
- Symmetric scheduling mechanisms. The standard, default behavior of
the operating system is to impose as little binding between system
executable entities and specific CPUs in the active set as possible.
That is, in general, each CPU is equally able to execute any Kernel
thread. The multi-processor scheduling algorithm is an extension of the
single-CPU behavior, providing consistent preemption and real-time
behavior in all cases.
However, there are circumstances when an
executable Kernel thread needs system resources and services possessed
only by certain CPUs in the configuration. In those non-symmetric
cases, OpenVMS provides a series of privileged, system-level CPU
scheduling routines that supersedes the standard scheduling mechanisms
and binds a Kernel thread to one or more specific CPUs. System
components that are tied to the primary CPU, such as system timekeeping
and console processing, use these mechanisms to guarantee that their
functions are performed in the correct context. Also, because the Alpha
hardware architecture shows significant performance benefits for Kernel
threads run on CPUs where the hardware context has been preserved from
earlier execution, the CPU scheduling mechanisms have been introduced
in OpenVMS Alpha Version7.0 as a series of system services and user
commands. Through the use of explicit CPU affinity and user
capabilities, an application can be placed throughout the active set to
take advantage of the hardware context. Chapter 4 in Section 4.4
describes these features in greater detail.
Synchronizing Data Access and Program Operations
This chapter describes the operating system's synchronization features.
It focuses on how the operating system synchronizes the sequencing of
events to perform memory operations. Memory access synchronization
techniques are based on synchronization techniques that other types of
storage use, whether to share hardware resources or data in files.
This chapter contains the following sections:
Section 6.1 describes synchronization, execution of threads, and
Section 6.2 describes alignment, granularity, ordering of read and
write operations, and performance of memory read and write operations
on VAX and Alpha systems in uniprocessor and multiprocessor
Section 6.3 describes how to perform memory read-modify-write
operations on VAX and Alpha systems in uniprocessor and multiprocessor
Section 6.4 describes hardware-level synchronization methods, such as
interrupt priority level, load-locked/store-conditional and interlocked
instructions, memory barriers, and PALcode routines.
Section 6.5 describes software-level synchronization methods, such as
process-private synchronization techniques, process priority, and spin
locks. It also describes how to write applications for a multiprocessor
environment using higher-level synchronization methods and how to write
to global sections.
Section 6.6 describes how to use local and common event flags for
Section 6.7 describes how to use SYS$SYNCH system service for
6.1 Overview of Synchronization
Software synchronization refers to the coordination of events in such a
way that only one event happens at a time. This kind of synchronization
is a serialization or sequencing of events. Serialized events are
assigned an order and processed one at a time in that order. While a
serialized event is being processed, no other event in the series is
allowed to disrupt it.
By imposing order on events, synchronization allows reading and writing
of several data items indivisibly, or atomically, in order to obtain a
consistent set of data. For example, all of process A's writes to
shared data must happen before or after process B's writes or reads,
but not during process B's writes or reads. In this case, all of
process A's writes must happen indivisibly for the operation to be
correct. This includes process A's updates---reading of a data item,
modifying it, and writing it back (read-modify-write sequence). Other
synchronization techniques are used to ensure the completion of an
asynchronous system service before the caller tries to use the results
of the service.
6.1.1 Threads of Execution
Code threads that can execute within a process include the following:
- Mainline code in an image being executed by a kernel thread, or
- Asynchronous system traps (ASTs) that interrupt the image
- Condition handlers established by the image and that run after
- Inner access-mode threads of execution that run as a result of
system service, OpenVMS Record Management Services (RMS), and command
language interpreter (CLI) callback requests
Process-based threads of execution can share any data in the P0 and P1
address space and must synchronize access to any data they share. A
thread of execution can incur an exception, which results in passing of
control to a condition handler. Alternatively, the thread can receive
an AST, which results in passing of control to an AST procedure.
Further, an AST procedure can incur an exception, and a condition
handler's execution can be interrupted by an AST delivery. If a thread
of execution requests a system service or RMS service, control passes
to an inner access-mode thread of execution. Code that executes in the
inner mode can also incur exceptions, receive ASTs, and request
Multiple processes, each with its own set of threads of execution, can
execute concurrently. Although each process has private P0 and P1
address space, processes can share data in a global section mapped into
each process's address spaces. You need to synchronize access to global
section data because a thread of execution accessing the data in one
process can be rescheduled, allowing a thread of execution in another
process to access the same data before the first process completes its
work. Although processes access the same system address space, the
protection on system space pages usually prevents outer mode access.
However, process-based code threads running in inner access modes can
access data concurrently in system space and must synchronize access to
Interrupt service routines access only system space. They must
synchronize access to shared system space data among themselves and
with process-based threads of execution.
A CPU-based thread of execution and an I/O processor must synchronize
access to shared data structures, such as structures that contain
descriptions of I/O operations to be performed.
Multiprocessor execution increases synchronization requirements when
the threads that must synchronize can run concurrently on different
processors. Because a process executes on only one processor at a time,
synchronization of threads of execution within a process is unaffected
by whether the process runs on a uniprocessor or on a symmetric
multiprocessing (SMP) system. However, multiple processes can execute
simultaneously on different processors. Because of this, processes
sharing data in a global section can require additional synchronization
for SMP system execution. Further, process-based inner mode and
interrupt-based threads can execute simultaneously on different
processors and can require synchronization of access to system space
beyond what is sufficient on a uniprocessor.
Atomicity is a type of serialization that refers to
the indivisibility of a small number of actions, such as those
occurring during the execution of a single instruction or a small
number of instructions. With more than one action, no single action can
occur by itself. If one action occurs, then all the actions occur.
Atomicity must be qualified by the viewpoint from which the actions
appear indivisible: an operation that is atomic for threads running on
the same processor can appear as multiple actions to a thread of
execution running on a different processor.
An atomic memory reference results in one indivisible read or write of
a data item in memory. No other access to any part of that data can
occur during the course of the atomic reference. Atomic memory
references are important for synchronizing access to a data item that
is shared by multiple writers or by one writer and multiple readers.
References need not be atomic to a data item that is not shared or to
one that is shared but is only read.
6.2 Memory Read and Memory Write Operations
This section presents the important concepts of
alignment and granularity and how
they affect the access of shared data on VAX and Alpha systems. It also
discusses the importance of the order of reads and writes completed on
VAX and Alpha systems, and how VAX and Alpha systems perform memory
reads and writes.
The term alignment refers to the placement of a data
item in memory. For a data item to be naturally aligned, its
lowest-addressed byte must reside at an address that is a multiple of
the size of the data item in bytes. For example, a naturally aligned
longword has an address that is a multiple of 4. The term
naturally aligned is usually shortened to
On VAX systems, a thread on a VAX uniprocessor or multiprocessor can
read and write aligned byte, word, and longword data atomically with
respect to other threads of execution accessing the same data.
On Alpha systems, in contrast to the variety of memory accesses allowed
on VAX systems, an Alpha processor allows atomic access only to an
aligned longword or an aligned quadword. Reading or writing an aligned
longword or quadword of memory is atomic with respect to any other
thread of execution on the same processor or on other processors.
VAX and Alpha systems differ in granularity of data access. The phrase
granularity of data access refers to the size of
neighboring units of memory that can be written independently and
atomically by multiple processors. Regardless of the order in which the
two units are written, the results must be identical.
VAX systems have byte granularity: individual adjacent or neighboring
bytes within the same longword can be written by multiple threads of
execution on one or more processors, as can aligned words and longwords.
VAX systems provide instructions that can manipulate byte-sized and
aligned word-sized memory data in a single, noninterruptible operation.
On VAX systems, a byte-sized or word-sized data item that is shared can
be manipulated individually.
Alpha systems have longword and quadword granularity. That is, only
adjacent aligned longwords or quadwords can be written independently.
Because Alpha systems support only instructions that load or store
longword-sized and quadword-sized memory data, the manipulation of
byte-sized and word-sized data on Alpha systems requires that the
entire longword or quadword contain the byte- or word-sized item to be
manipulated. Thus, simply because of its proximity to an explicitly
shared data item, neighboring data might become shared unintentionally.
Manipulation of byte-sized and word-sized data on Alpha systems
requires multiple instructions that:
- Fetch the longword or quadword that contains the byte or word
- Mask the nontargeted bytes
- Manipulate the target byte or word
- Store the entire longword or quadword
On Alpha systems, because this sequence is interruptible, operations on
byte and word data are not atomic. Also, this change in the granularity
of memory access can affect the determination of which data is actually
shared when a byte or word is accessed.
On Alpha systems, the absence of byte and word granularity has
important implications for access to shared data. In effect, any memory
write of a data item other than an aligned longword or quadword must be
done as a multiple-instruction read-modify-write sequence. Also,
because the amount of data read and written is an entire longword or
quadword, programmers must ensure that all accesses to fields within
the longword or quadword are synchronized with each other.
6.2.3 Ordering of Read and Write Operations
On VAX uniprocessor and multiprocessor systems, write operations and
read operations appear to occur in the same order in which you specify
them from the viewpoint of all types of external threads of execution.
Alpha uniprocessor systems also guarantee that read and write
operations appear ordered for multiple threads of execution running
within a single process or within multiple processes running on a
On Alpha multiprocessor systems, you must order reads and writes
explicitly to ensure that they occur in a specific order from the
viewpoint of threads of execution on other processors. To provide the
necessary operating system primitives and compatibility with VAX
systems, Alpha systems support instructions that impose an order on
read and write operations.
6.2.4 Memory Reads and Memory Writes
On VAX systems, most instructions that read or write memory are
noninterruptible. A memory write done with a noninterruptible
instruction is atomic from the viewpoint of other threads on the same
On VAX systems, on a uniprocessor system, reads and writes of bytes,
words, longwords, and quadwords are atomic with respect to any thread
on the processor. On a multiprocessor, not all of those accesses are
atomic with respect to any thread on any processor; only reads and
writes of bytes, aligned words, and aligned longwords are atomic.
Accessing unaligned data can require multiple operations. As a result,
even though an unaligned longword is written with a noninterruptible
instruction, if it requires multiple memory accesses, a thread on
another CPU might see memory in an intermediate state. VAX systems do
not guarantee multiprocessor atomic access to quadwords.
On Alpha systems, there is no instruction that performs multiple memory
accesses. Each load or store instruction performs a maximum of one load
from or one store to memory. A load can occur only from an aligned
longword or quadword. A store can occur only to an aligned longword or
On Alpha systems, although reads and writes from one thread appear to
occur ordered from the viewpoint of other threads on the same
processor, there is no implicit ordering of reads and writes as seen by
threads on other processors.
6.3 Memory Read-Modify-Write Operations
A fundamental synchronization primitive for accessing shared data is an
atomic read-modify-write operation. This operation consists of reading
the contents of a memory location and replacing them with new contents
based on the old. Any intermediate memory state is not visible to other
threads. Both VAX systems and Alpha systems provide this
synchronization primitive, but they implement it in significantly
6.3.1 Uniprocessor Operations
On VAX systems, many instructions are capable of performing a
read-modify-write operation in a single, noninterruptible (atomic)
sequence from the viewpoint of multiple application threads executing
on a single processor. VAX systems provide synchronization among
multiple threads of execution running on a uniprocessor system.
On VAX systems, the implicit dependence on the atomicity of VAX
instructions is not recommended. Because of the optimizations they
perform, the VAX compilers do not guarantee that a certain type of
program statement, such as an increment operation (x=x+1), is
implemented using a VAX atomic instruction, even if one exists.
On Alpha systems, there is no single instruction that performs an
atomic read-modify-write operation. As a result, even uniprocessing
applications in which processes access shared data must provide
explicit synchronization of these accesses, usually through compiler
On Alpha systems, read-modify-write operations that can be performed
atomically on VAX systems require a sequence of instructions on Alpha
systems. Because this sequence can be interrupted, the data may be left
in an unstable state. For example, the VAX increment long (INCL)
instruction fetches the contents of a specified longword, increments
its value, and stores the value back in the longword, performing the
operations without interruption. On Alpha systems, each
step---fetching, incrementing, storing---must be explicitly performed
by a separate instruction. Therefore, another thread in the process
(for example, an AST routine) could execute before the sequence
completes. However, because atomic updates are the basis of
synchronization, and to provide compatibility with VAX systems, Alpha
systems provide the following mechanisms to enable atomic
- Privileged architecture library (PALcode) routines perform queue
insertions and removals.
- Load-locked and store-conditional instructions create a sequence of
instructions that implement an atomic update.
The load-locked and store-conditional instructions also create a
sequence of instructions that are atomic in a multiprocessor system. In
contrast, a VAX INCL instruction is atomic only in a uniprocessor