What is shellcode and how is it used?

Shellcode is a set of instructions that executes a command in software to take control of or exploit a compromised machine. Read up on the malware term and how to mitigate the risk.

Shellcode is commonly part of the payload in the exploitation of a software vulnerability to take control of or...

exploit a compromised machine. The word shellcode literally refers to code that starts a command shell -- an instance of a command-line interpreter, such as the shell /bin/sh on Linux or cmd.exe on Windows. However, the term now embraces any bytecode that can be executed once the code is injected into a running application, even if it doesn't spawn a shell.

Shellcodes are typically injected into computer memory by exploiting stack- or heap-based buffer overflow vulnerabilities -- the most common programming errors that are used for this type of exploit -- or by formatting string vulnerabilities. When the exploit code causes what would normally be a critical error in the targeted program, the program jumps to the shellcode and is tricked into executing the attacker's commands -- all with the privileges of the process being exploited. Common shellcode objectives include installing a rootkit or Trojan horse, stopping antimalware programs and uploading or downloading files.

Inside shellcode exploits

Anyone writing shellcode needs to have an in-depth understanding of assembly or machine code, C and C++ languages, processor architecture and the targeted OS. For example, Windows shellcode is quite different from Linux shellcode. Unlike Linux, Windows does not have a direct kernel interface. The addresses of the functions found in Windows' dynamic link libraries (DLLs) vary from version to version, while Linux has a fixed numbering system for all kernel-level actions.

The main reason shellcode exploits are possible is because the application or library doesn't correctly validate the data it is handling. OSes allocate specific and finite amounts of memory to hold data, such as variables, values and arrays. These storage areas, called buffers, are generally created at the time a program is loaded or dynamically during program execution. When data exceeding the buffer's capacity is input, it overflows the buffer, and the excess data spills into other memory areas or buffers, overwriting some or all of the contents held in that memory space.

Software developers need to properly inspect how much data is written into a specific part of a program's code. In higher-level languages, like Java and C#, such coding errors are harder to make. But, because there are so many applications written in lower-level languages, like C and C++, these exploits are likely to be around for some time to come. Statistics from NIST's Common Vulnerabilities and Exposures database show vulnerabilities caused by buffer overflows rose sharply during 2017 and 2018. Also, with many attackers now using self-decrypting, polymorphic and various static but nonstandard encodings, intrusion detection systems cannot detect their shellcode using simple signature matching. There are also various other databases of shellcode exploits, including Exploit Database and 0day Today.

Types of shellcode

As mentioned, stack- and heap-based buffer overflows are the most popular shellcode examples. A classic attack using shellcode is the exploitation of the JpegOfDeath vulnerability in gdiplus.dll, which intentionally causes a buffer overflow condition. Anyone who opens a JPEG image created with JpegOfDeath exploit code invokes a buffer overflow, which takes advantage of this condition to inject shellcode into memory that is executed when the overflow occurs.

An exploit usually consists of two major components: the exploitation technique and the payload -- the component that enables attackers to execute their malicious code. The objective of the exploitation technique is to insert the shellcode and divert the execution path of the vulnerable program to the shellcode so that it can run the code in the payload. For example, shellcode execution can be triggered by overwriting a stack return address with the address of the injected shellcode. As a result, instead of the subroutine returning to the caller, it returns to the shellcode and spawns a shell.

While stack- and heap-based buffer overflows are the most common shellcode exploits, integer overflow, format string, race condition and memory corruption are other vulnerabilities hackers can take advantage of. Payloads can also be made to loop and wait for further commands from the attacker -- in fact, pretty much anything they require to expand the attack. Shellcode can either be local or remote. Local shellcode is used when an attacker has physical access to a machine, while remote shellcode is used to target a vulnerable process running on another machine to gain access to it across a network.

How to protect against shellcode

To protect against exploits that inject shellcode into vulnerable programs, enterprises need a multilayered security strategy. Hackers use automated scanners to look for applications using known vulnerable code, so enterprise firewalls and device controls need to be set to stop unwanted and known malicious connections. Monitoring controls also need to be in place to spot and quarantine unusual activity on the network using static and behavioral AI to stop the attack before it can do any lasting damage.

In addition, all applications need hardening to make them more resilient against exploits by ensuring that security controls, such as advanced memory protection and data execution prevention, are activated. In-house applications should be subject to static and dynamic tests, including direct attempts to inject shellcode directly into running processes using tools such as Metasploit and PowerSploit.

Finally, to check whether these combined defenses are effective, administrators can use penetration tools to test whether they detect and, most importantly, prevent exploits attempting to run shellcode.

This was last published in December 2019

Dig Deeper on Application and platform security