Home / Blogs

Bashbleed - A Nasty Reminder Never to Forget Security 101

Suresh Ramasubramanian

After the botched burglary at the Watergate Apartments, every scam and scandal that hit the headlines became a 'gate' — Irangate, Contragate, you name it.

The Heartbleed bug is possibly the closest thing to Watergate that this generation of computer security had seen till the past few days — an exploit in a component that is "just there" — something you utterly rely on to be there and perform its duties, and give very little thought to how secure (or rather, insecure) it might be. So, fittingly, every such catastrophic bug in an ubiquitous component is now a 'bleed'.

The latest bug to hit the headlines is, therefore, 'Bashbleed', a remotely exploitable vuln in the venerable old Bourne Again Shell (Bash), which has been around since 1989, gradually gaining in popularity over the years and deployed on various Unix/Linux distributions as well as on Apple Macs.

Finding a remotely exploitable vulnerability after years in code that has been around for over two decades is of course a major shock. Various security practitioners with a sense of black humor are calling it 'shellshock' because that is the exact state of mental trauma in which an admin ends up in when faced with a bug that has multiple attack vectors, and requires urgent patching on hundreds or even thousands of systems (s)he might operate, whether or not they let people ssh into their servers at all.

There are even servers that might not be all that easy to patch. At the low cost and ultra high volume end, embedded systems whose typical upgrade path is "throw old box into the trash, drive over to the store and buy a new one", all the way to ICS and SCADA systems.

It is a major relief to every ISP security practitioner that Bashbleed doesn't affect most consumer grade broadband routers and wifi access points, which run busybox, and therefore the ash shell, which is not vulnerable to bashbleed. Other embedded, ICS and SCADA systems, however, continue to be vulnerable. These systems power everything from auto parts factories to dams and nuclear power plants, and yet many just can't be upgraded — something that people like Joe St.Sauver of the University of Oregon have been saying for over a decade now.

So, what makes this bug so nasty? The sheer number of attack vectors — mailservers, webservers (including the administration web interfaces of some seriously expensive gear) — and the fact that once this vector is exploited, ANY system command that can be entered on a bash shell is available for you to execute.

So you could send an unpatched qmail smtp server something like —

MAIL FROM:<() { :; }; nc -e /bin/bash localhost 7777>

and have it spawn a bash shell for you.

This bash shell could then be used to, say, download a kernel exploit, or perhaps a spambot, from a malicious webserver. There are already bots that are doing something of the sort.

Now for a few best practices straight out of security 101:

Lesson 1 – Always Sanitize Input Data in applications that you develop, or run.

A mailserver that validated SMTP commands you fed it according to the SMTP standard RFC 5322 would absolutely refuse to accept and pass on to the underlying system anything that wasn't a syntactically valid email address — which that shell command is most definitely not.

Nor would a webserver that sanitized HTTP header data.

Or a web application that sanitized data input. A few examples of how to write web applications that sanitize input data are available in this older but still excellent article by Philip Tellis). More information is available in this SANS security checklist for web application development.

Lesson 2 – Stay Informed

Run regular scans of your network using something like Nessus or IBM Appscan. Of course, don't ever try to scan others' systems or networks, unless you have an explicit agreement with that system or network to pentest them for vulnerabilities.

Needless to say, unless you're a research organization that does global internet scanning and measurement of vulnerabilities, steer clear of trying to scan the entire Internet for vulns unless and until you have a clear procedure and protocol in place, and are transparent about your efforts.

In both cases, you're more than likely to get taken for a malicious actor if you do that — and almost certainly are going to be in breach of your Internet provider's terms of use policy.

Look out for unsanitized input data that leads to code injection, as part of a comprehensive vulnerability assessment.

Pay attention to your logs — preferably with tools that let you have a 360 degree view of your logs. You can use something like Logentries and New Relic, or Splunk to aggregate logs from across your devices and track them for anomalous events.

Use network analysis tools such as Netflow and Jflow, as well as security systems that leverage them, to mine network traffic for attack information.

Lesson 3 – Make fixes (relatively) easy and pain free

Treat every update equally — whether it is a server update or a code update. Deploy it as a package and leverage your existing distribution's package management network to perform updates. Even customized updates that you roll with patches that you need. This fits perfectly within the laser focus on automating and tracking deployment of builds that characterizes a DevOps environment — which you should strongly consider adopting, for a variety of reasons.

Use something like Spacewalk or Redhat's Satellite, which is derived from Spacewalk, to update Redhat based linux distributions such as RHEL or Centos, SuSE manager for SuSE linux, or other equivalent systems available for your Linux distribution.

For those who run rather more diverse environments where you end up having to patch everything from SSL load balancers to other Unices, you will need custom Chef and Puppet recipes, or possibly a commercial product like IBM's SmartCloud Orchestrator

Lesson 4 – Keep track of the security scene

Read various security blogs, follow known security researchers on facebook and twitter, and do work with your colleagues to track the mailing lists for software that you deploy and care about.

There's a new bug almost every other day in one product or the other, some minor and some highly serious. Being aware of what is going on that affects your environment is essential.

In conclusion — I'll point out that Bruce Schneier, channeling 'This is Spinal Tap', said about Heartbleed — "'Catastrophic' is the right word. On the scale of 1 to 10, this is an 11."

I am not sure he's rated Bashbleed on that same scale, but I personally would rate this just as serious, if not more. Go out there and look for whatever needs to be secured on your systems. Now, if you haven't ever done so before. Do it now too, even if you do this religiously on a regular basis.

By Suresh Ramasubramanian, Antispam Operations
Follow CircleID on
SHARE THIS POST

If you are pressed for time ...

... this is for you. More and more professionals are choosing to publish critical posts on CircleID from all corners of the Internet industry. If you find it hard to keep up daily, consider subscribing to our weekly digest. We will provide you a convenient summary report once a week sent directly to your inbox. It's a quick and easy read.

I make a point of reading CircleID. There is no getting around the utility of knowing what thoughtful people are thinking and saying about our industry.

Vinton Cerf, Co-designer of the TCP/IP Protocols & the Architecture of the Internet

Share your comments

Brace yourself. Patches are coming. Suresh Ramasubramanian  –  Sep 29, 2014 8:22 AM PST

Also - a way for you to test whether or not you're vulnerable to bashbleed or a related bug

http://www.zdnet.com/shellshock-better-bash-patches-now-available-7000034115/

The underlying bug won't get fixed easily Suresh Ramasubramanian  –  Sep 29, 2014 4:15 PM PST

This is a sound recommendation I came across from Paul Vixie - on , Paul wrote as follows:

Re: if you patched over the weekend, you're out of date, and vulnerable, again.

<< These appear to be different bugs, unless I'm missing something? >>

To me the bug is that GNU Bash ever evals the contents of an environment variable. In other words, all of this week's drama comes from a misfeature. Based on the fix now present on FreeBSD systems, I am not the only one thinking this way. However, the maintainers of GNU Bash are doing their darndest to make this feature safe by making terribly fine distinctions about the exact form, syntax, and content of these environment variables. The reason you see five different CVE's (as of this moment) at http://shellshocked.net/ is that people keep finding new ways to fool the latest patch and access the underlying remote execution vulnerability.

I prefer FreeBSD's fix. Don't evaluate the contents of environment variables by default. To those who warn that this will break some existing GNU Bash scripts, I answer: yes, and that's a bitter pill, but since this is actually misfeature, my feeling is that adding logic to make finer and finer distinctions about the content of environment variables is increasing complexity (and therefore danger), and decreasing auditability and provability (and therefore safety).

I also prefer the Debian approach (/bin/sh is "dash" not "bash") over RedHat and Apple's. GNU Bash is a great interactive shell, but it's way too large and too complex to be allowed to be in the execution path for libc's popen() and system() calls, which are used by Apache and QMail to run commands. /bin/sh should be as simple as possible, which is to say, like "dash" on Debian (which comes from "ash" which is used as /bin/sh on FreeBSD, NetBSD, OpenBSD, and DragonflyBSD).

It's very strange after the last couple of years of hair-on-fire bug-of-the-week theater, to have to argue that complexity ought to be avoided wherever possible in control systems.

To post comments, please login or create an account.

Related

Topics

DNS Security

Sponsored byAfilias

IP Addressing

Sponsored byAvenue4 LLC

Cybersecurity

Sponsored byVerisign

Domain Names

Sponsored byVerisign

Cybercrime

Sponsored byThreat Intelligence Platform

Whois

Sponsored byWhoisXML API

New TLDs

Sponsored byAfilias