During the 1999 Black Hat security conference, several security practitioners moaned that there was nothing new happening in security. Not that nothing was happening, just that it was more of the same old stuff—stuff that should have been fixed years ago.

One particular attack, which has become almost ubiquitous in the last several years, is the bufferoverflow. While the attack requires particularly arcane and detailed knowledge of both assembly language and, in the case of Windows, operating system interface details, once someone has coded an exploit and published it, anyone can use it. The results of these exploits provide interactive command shells on Unix systems and the ability to upload and execute arbitrary programs on Windows systems.

The sad thing about buffer overflow exploits is that good programming practices could wipe out even potential exploits, but that simply hasn’t happened.

Your own defense should revolve around controlling access to sensitive systems, installing software updates that replace exploitable software, and being aware of what a buffer overflow attack looks like when your system is the intended victim.

AN INFAMOUS WORM

In November 1988, the Internet Worm shut down over 6,000 systems, just about cutting off all traffic on the Internet. One of the methods used to gain access to these systems was a buffer overflow exploit of a Unix service “finger.” When you finger a user, the finger service returns information about the user, such as the user’s real name and phone number. In the case of the Worm, the buffer overflow attack on finger replaced the server program with a Unix command interpreter, or shell. This shell was then used to copy across a program that uploaded, linked, and then executed a new copy of the Worm.

Buffer overflow attacks remained relatively unheard of for many years following the Worm. One known example came in November of 1994, when one of the first commercial Web servers, running HP-UX, was successfully breached using a buffer overflow attack against the National Center for Supercomputing Applications’ (NCSA) 1.3 Web server. As this Web server sat on the target’s internal network and could be connected to through the firewall, the attackers had unfettered access to the victim’s internal network.

But the event that really fueled the frequency of attacks was the November 1996 publication of a paper entitled “Smashing the Stack for Fun and Profit,” by Aleph One, in the online hacker magazine Phrack. Aleph One’s paper (itself based on a paper written by Mudge of the L0pht, an independent computer security think tank) explains in detail how to write a buffer overflow exploit against a Unix system program. While somewhat mind-bending, the paper is good reading, especially if you have ever tried to understand how buffer overflow exploits work from reading the source code for an exploit.

In 1997 and 1998, buffer overflow exploits became extremely common, mainly targeting Unix systems—in particular the Open Source versions. While the Open Source organizations, like the various Linux distributors or FreeBSD, were quick to release patches, the number of exploits was astounding. To get a feel for the scope of this problem, visit www.rootshell.com and search on “buffer overflow.” You can also visit other sites, such as the L0pht (with a zero, not an “o”), and search for “exploits.”

You will see that exploits continue right into 1999, and now include Microsoft products. One reason why Microsoft products were not initially attacked is that the techniques required are somewhat different.

ANATOMY OF A BUFFER OVERFLOW

Buffer overflows occur mainly because of the C language and partially because of poor programming practices. The C language is not going to change, but programming practices could undergo some improvement.

The C programming language appeared in the early 1970s as a tool for porting the Unix operating systems and utilities to multiple architectures. While Unix was initially written in assembly, porting it to C made Unix the first really portable operating system. C was designed to be tight and fast, just a step above assembly.

The C language is also a structured programming language. Object-oriented languages, such as Java, are organized around data. Structured programming languages use the function call as their unit of organization. While the designers of C made a great leap forward, they also created the framework for the buffer overflows.

Each time a function is called, arguments to the function get copied to an area of memory called the stack. In assembly, you store things on the stack by pushing them and retrieve them by popping them off the stack. All CPU architectures currently in use support the notion of a stack and have a special register (the stack pointer) and operations for pushing and popping. There is also an operator that takes an address off the stack and copies it into the program counter, the register that determines the address of the next instruction to execute. Calling a function always pushes the return address onto the stack.

The problem with this design shows up within the called function. Any variables defined within this function are also stored in space allocated on the stack. For example, if a string, such as the name of a file to open, needs to be defined in the function, a number of bytes will be allocated on the stack. The function can then use this memory, but it will automatically be unallocated after the function returns—quite a neat design. But C does no bounds checking when data is stored in this area, opening a narrow window for an attacker.

C subroutine calls that copy data but do no bounds checking are the culprits (as well as the programmers who use these calls). The strcat(), strcpy(), sprintf(), vsprintf(), bcopy(), gets(), and scanf() calls can be exploited because these functions don’t check to see if the buffer, allocated on the stack, will be large enough for the data copied into the buffer. It is up to the programmer to either use a version that makes the check (such as strncpy() ) or to count the bytes of data before copying them onto the stack.

Given that there is a list of commonly abused subroutine calls, you might think it reasonable that all uses of these calls would be checked, and that the problem would be fixed forever. Actually, it’s not quite as easy as that, and there are other ways of making similar, and just-as-exploitable, mistakes (for example, appending characters in a loop).

In addition to subroutine calls, an attacker must also understand enough assembly (the byte codes used by processors like the Intel Pentium) to code the exploit itself. In a buffer overflow exploit, code gets written on the stack, beyond the return address and function call arguments, and the return address gets modified so that it will point to the beginning (approximately) of the code. Then, when the function call returns, the attacker’s code gets executed instead of normal program execution.