Ghost in the DCL shell: OpenVMS, touted as ultra reliable, had a local root hole for 30 years

Forget Meltdown and Spectre. Someone's found a local privilege escalation in the operating system world's elderly statesman OpenVMS when running it on VAX and Alpha processors.

On Itanium CPUs, the same bug can be exploited to crash a process.

More details on the flaw, which has been given the designation CVE-2017-17482, are due to be made public in March. The weeks between now and then are intended to give sysadmins time to test and deploy patches.

Software running on seemingly bulletproof OpenVMS systems tends to be rather business critical – the sort of code you deploy and keep running forever – so updates that may disrupt operations are treated with utmost care by administrators. Introduced to the world as VMS in 1977, OpenVMS today still powers a good chunk of billing systems, stock exchanges, semiconductor factories, and similar setups. It is touted as a reliable and secure OS for mission-critical applications.

By the way, the "open" part refers to its support of open standards, such as POSIX; its source code is closed, although copies of the listings can, apparently, be purchased.

The privilege-escalation vulnerability stems from a cockup in the command processing code within the VMS shell, called DCL. The hole can be exploited by normal users to run code in the privileged supervisor mode. That much is now public, though quite how it can be exploited has been held over until next month. Since OpenVMS installations tend to only grant access to trusted staff, the main threat here is dodgy or bribed employees seeking to commandeer systems, rather than outside hackers.

Crucially, the vulnerability dates back decades – as far back as version 4.0 when the OS used to be known simply as VMS – giving miscreants or spies plenty of time to stumble upon the same bug, and abuse it for corporate espionage or similar.

Simon Clubley, the security researcher who discovered the programming blunder, told El Reg this month that the flaw “allows anyone with shell access to totally compromise any version of OpenVMS released for the VAX or Alpha architectures in the last 30 years.”

"This situation is not helped by the very mistaken belief in some parts of the VMS user community that somehow VMS systems are not vulnerable to the same kinds of security issues that other operating systems have,” Clubley said. “These people are very wrong.”

Ask nicely for fixes

People running VSI's OpenVMS V8.4-2L1 or V8.4-2L2 on Alpha hardware should contact VSI for patches to kill the exploitable bug. VSI is a company that was formed in Massachusetts, USA, to port OpenVMS to modern x86-64 processors in a joint agreement with HPE, and it includes ex-DEC engineers and other folks from the OpenVMS world. Sysadmins managing VSI's OpenVMS V8.4-1H1, V8.4-2, or V8.4-2L1 on IA64, or any form of HPE OpenVMS, should contact HPE.

Everyone else, well, try hitting up VSI for help. VMS on VAX and early Alpha releases are unsupported, it appears, so on that hardware, you may be out of luck. Patches started appearing at the end of last year, and were announced on the comp.os.vms newsgroup at the end of January.

"There is a very distinct possibility that there are a significant number of unsupported and vulnerable VMS systems still in production use," Clubley warned.

For what it's worth, today's operating systems usually run applications on processors in what's called user mode, and the core OS and drivers in a privileged kernel mode. VMS uses four modes: user mode; supervisor mode where the DCL shell runs; executive mode for privileged services; and kernel mode, which has power over the system.

VMS runs its shell in supervisor mode. A program can pass malformed command line data to DCL to process, which overflows a buffer and clobbers a return pointer in memory. There are some portions of memory with fixed addresses that all programs which run in a process share, and for some reason can hold executable code. Thus, it's possible to stash some malicious code in those shared areas, pass a booby-trapped command line to the shell to parse, and have the shell jump to the evil attacker-controlled code while still in supervisor mode.

It gets worse

Furthermore, Clubley said he discovered in November that the boundary between supervisor and executive modes is not as watertight as folks are led to believe. Thus, it is possible to leverage the escalation from user mode to supervisor mode to jump into the executive and drill deeper into the system. More on this hidden supervisor-to-executive capability will be revealed next month by Clubley, we're told.

"In addition to those who wrongly believe that their old VMS systems are somehow immune to the security issues of other operating systems, there are also some in the VMS user community who would prefer not to make these issues public, and would prefer to keep them hidden," he added. "They believe that if you don't talk about these issues then everything will be OK.

"Well, over a very interesting few weeks in November 2017, I went from not knowing where to look for this hidden capability, to then finding this hidden capability and then on to developing an exploit that compromised every version of VAX and Alpha VMS released over the last 30 years.

"If I can do that, then so can the people who would want to harm you. By making these issues public, more people receive warning that these issues exist so they can protect themselves. Your desire for the illusion of security by obscurity is just that - an illusion."

VMS. Bulletproof. Depending on the bullet. ®