The Question is:
Is there a malloc bug in DEC C V5.2-003 C compiler? Or any other know problem
that might cause strange crashes?
The Answer is :
The C malloc and free calls are very heavily used within OpenVMS code and
within layered products and customer application code. Accordingly, any
problems that might be lurking within the run-time library (RTL) code that
implements these calls would typically be expected to have very obvious and
very widespread effects.
The Compaq C RTL ships with OpenVMS and not the compiler, and OpenVMS ECO
kits for the RTL are available. That said, the OpenVMS Wizard would NOT
assume that these ECOs will repair the (unspecified) behaviour reported
here. The OpenVMS Wizard WOULD assume that there is an application bug here.
Your Compaq C compiler is also very old, and should be upgraded to a more
recent version. But paralleling the previous paragraph, the OpenVMS Wizard
would again not assume that a compiler upgrade would resolve what is most
often an application bug.
Discussions in topics (1661), (2624), (2630), (3115), (3257), (4808),
(5455), (5640), and likely a few others will be of interest when working
with memory management, and particularly when debugging memory management
problems. The first of these topics includes a discussion of debugging
various typical types of memory allocation problems, and topic (3257)
provides a detailed description of a debugging a technique occasionally
known as "fenceposts".