[Practical Binary Analysis] Part.1 (Chapter.1)

CHPTER 1 - ANATOMY OF BINARY

=================

What exactly is a binary?

Modern computers perform their computations using the binary numerical system, which express all numbers as strings of one and zeros.

The machine code that these systems execute is called binary code.

Every program consists of a collection of:

• Binary code the machine instructions

• Data variables, constants and the like

To keep track of all the different programs on a given system is needed a way to store all the code and data belonging to each program in a self-contained file.

These files, containing executable binary programs, are called binary executable files or simply binary

1.1 The C Compilation Process

Binaries are produced through compilation:

Compiling C code involves four phases (modern compilers often merge some or all of these phases):

1. Processing

2. Compilation

3. Assembly

4. Linking

1.1.1 The Processing Phase

–

Below a typical compilation process for C code:

The compilation process starts with a number of source files that have to be compiled ( file-1.c through file-n.c )

It is possible to have one large file but typically, large programs are composed of many files, this for the following reasons:

a) It makes the project easier to manage

b) It speeds up compilation because if one file changes, only that file has to be recompiled instead of all the code.

C source files contain:

• Macros (#define)

• Directives (#include): are used to include header files (with the extension .h), on which the source code depends.

The Processing phase expands any “#define” macros or “#include” directives in the source file, so all that is left is pure C code ready to be compiled.

Let take the following code as an example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

  //compilation_example.c
  
  #include <stdio.h>
  
  #define FORMAT_STRING “%s”
  
  #define MESSAGE “Hello, world!\n”
  
  int
  
  main(int argc, char \*argv\[\])
  
  {
  
  printf(FORMAT_STRING,MESSAGE);
  
  return 0;
  
  }

Let us compile the file “compilation_example.c” using gcc (into x86-64 code) using the following parameters:

• -E : tells gcc to stop after processing

• -P : causes the compiler to omit the debugging information, so that the output is a bit cleaner.

Below the output of the C processor of the compilation_example.c gcc -E -P compilation_example.c

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

  ---
 $ gcc -E -P compilation_example.c 
  
  typedef long unsigned int size_t;
  
  typedef unsigned char __u_char;
  
  typedef unsigned short int __u_short;
  
  typedef unsigned int __u_int;
  
  typedef unsigned long int __u_long;
  
   /*.....*/
  
  extern int sys_nerr;
  
  extern const char *const sys_errlist\[\];
  
  extern int fileno (FILE *__stream) __attribute__ ((__nothrow__ , __leaf__)) ;
  
  extern int fileno_unlocked (FILE *__stream) __attribute__ ((__nothrow__ , __leaf__)) ;
  
  extern FILE *popen (const char *__command, const char *__modes) ;*
  
  extern int pclose (FILE *__stream);*
  
  extern char *ctermid (char *__s) __attribute__ ((__nothrow__ , __leaf__));
  
  extern void flockfile (FILE *__stream) __attribute__ ((__nothrow__ , __leaf__));
  
  extern int ftrylockfile (FILE *__stream) __attribute__ ((__nothrow__ , __leaf__));
  
  extern void funlockfile (FILE *__stream) __attribute__ ((__nothrow__ , __leaf__));
  
  int
  
  main(int argc, char *argv[]) {
  
  printf("%s", "Hello, world!\n");
  
  return 0;
  
  }

  ---

The “stdio.h” header is included in its entirety, with all its type definitions, global variables, and function prototypes “copied in” to the source file.

• Because this happens for every #include directive, preprocessor output can be quite verbose.

• The preprocessor also fully expands all of any #define macros used. In the above example both arguments to “printf”* are evaluated and replaced by the constant strings they represent:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

  -----
  ..
  
  #define FORMAT_STRING <b>“%s”</b>
  
  #define MESSAGE <b>“Hello, world!\n”</b>
  
  ..
  
  printf(FORMAT_STRING,MESSAGE);*..
  -----

to:

1
2
3
4
5
6
7

  -------
  ..
  
  printf(**"%s**", **"Hello, world!\n"**);
  
  ..
  -------

1.1.2 The Compiling Phase

After the processing part is completed, the source is now ready to be compiled-

Most compilers also perform heavy optimization in this phase, typically configurable as an optimization level through command line switches such as option –Oo3 through –O3 in gcc. The degree of optimization during compilation can have a profound effect on disassembly.

The main reason why the compilation phase produces assembly language instead of machine code is that writing a compiler that emits, at the same time, machine code for many different languages would be an extremely demanding and time-consuming task.

It is better to instead emit assembly code, which is an already challenging task, and have a single dedicated assembler that can handle the final translation of assembly to machine code for every language.

Therefore, the output of the compilation phase is:

• Assembly, in a reasonably human-readable form, with symbolic information intact.

As mentioned, gcc normally calls all compilation phases automatically, so:

To see the emitted assembly from the compilation stage, the parameters and options to use in order to tell gcc to stop after stage and store the assembly to the disk are:

• -S : This flag is for the conventional extension for assembly files “*.s”

• -masm=intel : With this option “gcc” emits assembly in Intel syntax rather than the default AT&T syntax

Below the output of the C processor of the compilation_example.c gcc -S -masm=intel compilation_example.c

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

  --------
  $ gcc -S -masm=intel compilation\_example.c
  
 $ ls –la
  
  *total 32*
  
 \*….\*
  
  -rw-r--r-- 1 binary binary 173 dec 8 16:29 compilation_example.c
  
 -rw-rw-r-- 1 binary binary 562 dec 8 19:57 **compilation_example.s*
  
 -rw-r--r-- 1 binary binary 10240 apr 19 2018 hello.exe
  
 -rw-r--r-- 1 binary binary 723 apr 19 2018 Makefile
  --------

Below the output of $ cat compilation_example.s

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65

  
  $gcc -S -masm=intel compilation\_example.c
  
  $cat compilation\_example.s
  
  .file "compilation\_example.c"
  
  .intel\_syntax noprefix
  
  .section .rodata
  
  .LC0: (1)
  
  .string "Hello, world!"
  
  .text
  
  .globl main
  
  .type main, @function
  
  main: (2)
  
  .LFB0:
  
  .cfi_startproc
  
  push rbp
  
  .cfi_def_cfa_offset 16
  
  .cfi_offset 6, -16
  
  mov rbp, rsp
  
  .cfi_def_cfa_register 6
  
  sub rsp, 16
  
  mov DWORD PTR [rbp-4], edi
  
  mov QWORD PTR [rbp-16], rsi
  
  mov edi, OFFSET FLAT:.LC0 (3)
  
  call puts
  
  mov eax, 0
  
  leave
  
  .cfi_def_cfa 7, 8
  
  ret
  
  .cfi_endproc
  
  .LFE0:
  
  .size main, .-main
  
  .ident "GCC: (Ubuntu 5.4.0-6ubuntu1\~16.04.5) 5.4.0 20160609"
  
  .section .note.GNU-stack,"",@progbits
  

The above code looks relatively easy to read because the symbols and functions have been preserved:

• ln.12 - The auto generated name LC0 for the nameless “Hello, world!” string.
• ln.22 - The explicit label for the mainfunction.
• ln.44 - The symbolic reference to code and data to the LC0(the“Hello, world!” string).

Constants and variables have symbolic names rather than just address:

• Unfortunately, with stripped binaries it’s not possible to gain all these extensive information.

1.1.3 The Assembly Phase

The Assembly phase: when some real machine code is finally generated!

The input of the assembly phase is the set of assembly language files generated in the compilation phase, and the output is a set of object files, sometimes also referred to as module. Object files contain machine instructions that are in principle executable by the processor. Typically, each source file corresponds to one assembly file, and each assembly file corresponds to one object file.

Below the process of generating an object file with gcc .The parameter to generate an object file, is -c : Flag to generate an object file.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

  -----
  ***\$ gcc -c compilation\_example.c***
  
  *\$ ls -la*
  
  *total 36*
  
  *drwxr-xr-x 2 binary binary 4096 dec 8 23:44 .*
  
  *drwxrwxr-x 16 binary binary 4096 apr 19 2018 ..*
  
  *-rw-r--r-- 1 binary binary 173 dec 8 16:29 compilation\_example.c*
  
  *-rw-rw-r-- 1 binary binary 1536 dec 8 23:44 compilation\_example.o*
  
  *-rw-rw-r-- 1 binary binary 562 dec 8 19:57 compilation\_example.s*
  
  *-rw-r--r-- 1 binary binary 10240 apr 19 2018 hello.exe*
  
  *-rw-r--r-- 1 binary binary 723 apr 19 2018 Makefile*
  -----

The “file <file.o>” utility to confirm that the produced file is indeed and object file

Let’s analyse the output, the first part of the file output shows that the file conforms to the elf specification for binary executables:

Object files are compiled independently from each other, so the assembler has no way of knowing the memory addresses of other objects files when assembling an object file.

That’s why objects files need to be relocatable; so that you can link them together in any order to form a complete binary executable. (If objects files were not relocatable, that action would be impossible)

1.1.4 The Linking Phase

Let’s run file commmand angainst the generated binary file a.out

Let’s analyse the output generated by the “file” utility ($ file a.out):

▶ Symbols and Stripped Binaries

=============================

High-level source code, such as C code, centers around functions and variables with meaningful, human-readable names.

When compiling a program, compiler emit symbols, which:

• Keep track of such symbolic names.

• Record which binary code and data correspond to each symbol.

For instance, function symbols provide a mapping from symbolic, high-level function names to the first address and the size of each function.

This information is normally used by the linker when combining object files (for instance, to resolve function and variable references between modules) and also aids debugging.

◌ Viewing Symbolic Information

The below example shows some symbolic information for the file “a.out” using “readelf –syms”:

• ‘-s’ or ‘–syms’ or ‘–symbols’ (An alias for ‘–syms’) : Display the symbol table

In the above listing, with the utility readelf is possible to display the symbol table, among many unfamiliar symbols we can find a symbol for:

• The main function: the value 0x400526represents the address at which “main” will reside when the binary is loaded into memory.

• The output also shows the code size of main (32 bytes) and indicates that you are dealing with a function symbol (type FUNC)

Symbolic information can be emitted either

a) As part of the binary (as shown above).

b) Or in the form of a separate symbol file, and it comes on various flavours.

◌ Another Binary Turns to the Dark Side:
   ▹ Stripping a Binary

Apparently, the default behaviour of gcc is not to automatically strip newly compiled binaries.

The below example shows how binary with symbols end up stripped by using the command :
$ strip –strip-all a.out

Using the command “strip –strip-all a.out”, the binary is now stripped.

Only symbols in ‘.dynsym’ table are left, these are used to resolve dynamic dependencies when the binary is loaded into memory, but they are not much use when disassembling.

1.3 Disassembling a Binary

======================

Now that we have seen how to compile a binary, let’s take a look at the contents of the object file produced in the assembly phase of compilation.

1.3.1 Looking inside an Object File.

Let’s use the objdump utility to show how to do all the disassembling

Below the disassembled version of the object file compilation_example.o using options -s and -j

In the above example the options used to show only the ‘.rodata’ section are:

• -s, –full-contents Display the full contents of all sections requested

• -j, –section=NAME Only display information for section NAME

The content of .rodata section consists of :

• An ASCII encoding of the string (on the left side of the output) **0000 48656c6c 6f2c2077 6f726c64 2100 **

• The human-readable representation of those same bytes (on the right side of the output) Hello, world!.

.rodata stands for “read-only”; it’s part of the binary where all constants are stored (including the “Hello, world!” string)

Below the disassembled version of the object file compilation_example.o using option/switch -M and -d

In the above example the options used are:

• -M, –disassembler-options=OPT Pass text OPT on to the disassembler

  • Intel, for usewith the -M switch This option to display instruction in Intel syntax

• -d, –disassemble Display assembler contents of executable sections

The utility objdump here has been used to disassemble all the code in the object file compilation_example.o, in the Intel syntax.

It contains only the code of the main function because that’s the only function defined in the source file.

For the most part, the output conforms pretty closely to the assembly code previously produced by the compilation phase (give or take few assembly level macros)

What is interesting here is that the pointer to the “Hello, world!” string is set to zero.

The subsequent call that should print the string to the screen using puts also points to non sensical location (offset 19, in the middle of the main).

But why does the call, that should reference puts, point instead into the middle main?

Let’s having in mind that data and code references from object files are not yet fully resolved because the compiler does not know at what base address the file will eventually be loaded. That’s why the call to puts is not yet correctly resolved in the object file.

The object file is waiting for the linker to fill in the correct value for this reference.

Let’s use the command readelf to show all the relocation symbols:

The relocation symbol at offset 000000000010 tells the linker that it should resolve the reference to the string to point to whatever address it ends up at in the .rodata section.

Similarly, the relocation symbol at offset 000000000015 tells the linker how to resolve the call to puts.

The value in the offset “column” is the offset in the object file where the resolved reference must be filled in. If we go back and check the offset generated with the utility objdump we can see that the offset is 0x14.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

  --------
  ***\$ objdump -M intel -d compilation_example.o***
  
  */\* …. \*/*
  
    *f:*           *bf 00 00 00 00 *   *mov *    *edi,0x0*
    ----- -  ------
    *14:*          *e8 00 00 00 00 *   *call *   *19 <main+0x19>*
    */\* …. \*/*                                 
    *1f:*          *c3 *               *ret *    
                                                 
  
  --------

We can notice that the relocation symbol points to offset 0x15 instead, this is because only the operand (at position 0x15) of the instruction needs to be overwritten, not the opcode of the instruction at 0x14 (e8).

It is just to happen that for both instructions that need fixing up, the opcode is 1 byte long, so to point to the instruction’s operand, the relocation symbol needs to skip past the opcode byte.

1.3.2 Examining a complete binary executable

It’s now time to examine a complete binary executable.

Let’s start with a binary with symbols (a.out) and then move on to the stripped equivalent (a.out.stripped) to see the difference in disassembly output.

Let’s disassemble the non-stripped version of the binary file “a.out”

We can see that the binary has a lot more code than the object file.

It’s no longer just the main function or even just a single code section.

There are multiple sections now, with names like :

1. .init

2. .plt

3. .text

These sections contain code serving different function, such as program initialization or stubs for calling shared libraries.

The .text section (3) is the main code section, and it contains:

• The main (4) function.

• Other functions, such, as _start, that are responsible for tasks such as setting up the command line arguments and runtime environment for main and cleaning up after main. (These extra functions are standards functions, present in any ELF file produced by gcc).

We can also see that the previously incomplete code and data references have now been resolved by the linker.

For instance, the call to “puts” now points to the proper stub (in the .plt section) for the shared library that contains puts.

Let’s try now to disassemble the stripped filed version a.out.stripped

• While the difference sections are still clearly distinguishable ( 1.init, 2.plt ,3.tex t) BUT the functions are not!

Now all the functions have been merged into one big blob of code:

• The <_start> function begins at address 400430 and ends at 40045a

• The **<*deregister_tm_clones> ***function begins at address 40046

• The <main> function begins at address 400526 and ends at 400545

As we can see above, there is not anything special to indicate that the instructions at these markers represent function starts.

The only exception are the functions in the .plt section, which still have their names as before.

1.4 Loading and Executing a Binary

==============================

What really happens when a binary file is loaded and executed?

The figure below shows how a loaded ELF binary is represented in memory on a Linux-based platform.

(At a high level, loading a PE binary on Windows is quite similar).

Loading a binary is a complicated process that involves a lot of work by the operating system. It is also important to note a binary’s representation in memory does not necessarily correspond one to one with its on-disk representation.

For instance, large amount of zero-initialized data may be collapsed in the on-disk binary (to save disk space), while all those zeros will be expanded in memory or not loaded into memory at all.

When we decide to run a binary, the operating system starts by setting up a new process for the program to run in, including a virtual address space.

Subsequently, the operating system maps an interpreter into the process’s virtual memory

This is a user space program that knows how to load the binary and perform the necessary relocations:

• On Linux the interpreter is typically a shared library called ld-linux.so.

• On Windows the interpreter functionality is implemented as part of ***ntdll.dll. ***

After loading the interpreter, the kernel transfers control to it, and the interpreter starts its work in user space.

Linux ELF binaries comes with a special section called .interp that specifies the path to the interpreter that is to be used to load the binary.

The command below shows the .interp section of the file a.out:

• -p –string-dump=<number|name> Dump the contents of section <number|name> as strings.

• .interp Is the section name.

1. Here the interpreter loads the binary into its virtual address space (the same space in which the interpreter is loaded).

2. It then parses the binary to find put (among other things) which dynamic libraries the binary uses.

3. The interpreter maps these into the virtual address space ( using mmap or equivalent function) and then performs any necessary last-minute relocations in the binary’s code sections to fill in the correct address for references to the dynamic libraries.

1. In reality, instead of resolving these references immediately at load time, the interpreter resolves references only when they are invoked for the first time. This is known as lazy bindings**.**

2. After relocation is complete, the interpreter looks up the entry point of the binary and transfers control to it, beginning normal execution of the binary.

1.5 Summary

When we decide to run a binary, the operating system starts by setting up a new process for the program to run in, including a virtual address space.

• Here the interpreter loads the binary into its virtual address space (the same space in which the interpreter is loaded).

• It then parses the binary to find put (among other things) which dynamic libraries the binary uses.

• The interpreter maps these into the virtual address space ( using mmap or equivalent function) and then performs any necessary last-minute relocations in the binary’s code sections to fill in the correct address for references to the dynamic libraries. In reality, instead of resolving these references immediately at load time, the interpreter resolves references only when they are invoked for the first time. This is known as lazy bindings**.**

• After relocation is complete, the interpreter looks up the entry point of the binary and transfers control to it, beginning normal execution of the binary.

No. The division of the binary into sections is a convenient organization set up to be parseable by the linker. Some sections contain only data intended for the linker at link time, such as symbolic or relocation information. Much linking occurs before execution time, so some such sections do not need to be loaded into the process’s virtual memory for execution - although with lazy binding, many relocations can occur during execution (the .plt and .got.plt sections are used for this, and do need to be loaded). Sections are organized into groups called segments. Segments of type PT_LOAD are intended to be loaded into memory. The executable’s program headers describe the segments.