Assembly for Security Analysis — x86-64 and ARM64 — DedSec Project

Purpose: Learn enough assembly to follow control flow, calling conventions, data movement, and compiler output during debugging and reverse engineering. This is analysis-oriented, not exploit construction.

Learning objectives

• Read basic x86-64 and AArch64/ARM64 instructions.
• Understand registers, stack frames, calls, returns, condition flags, and branches.
• Connect C/Python-extension behavior to compiled machine code.
• Recognize function prologues, epilogues, loops, comparisons, and common compiler transformations.
• Use local disassembly to explain code you compiled yourself.

Machine code, assembly, and architecture

A CPU executes encoded instructions. Machine code is the bytes; assembly is a human-readable representation of those instructions. The mapping depends on the instruction-set architecture. x86-64 uses variable-length instructions and a large historical instruction set. AArch64 uses fixed-width 32-bit instructions and a different register/calling model.

Security analysis does not require memorizing every opcode. You need a small vocabulary plus the ability to consult architecture references.

Registers

Registers are tiny CPU storage locations used for operands, addresses, return values, arguments, flags, and control state.

On x86-64, general-purpose registers include rax, rbx, rcx, rdx, rsi, rdi, rsp, rbp, and r8–r15. rip tracks the instruction pointer. On AArch64, general-purpose registers are x0–x30 (with 32-bit views w0–w30), plus sp and program counter behavior managed by branch instructions. x30 commonly carries the link register for return addresses.

Register roles depend partly on the ABI/calling convention rather than hardware alone.

Calling conventions

A calling convention specifies how functions exchange arguments, return values, and register ownership. On common System V x86-64 environments, early integer/pointer arguments use registers such as rdi, rsi, rdx, rcx, r8, and r9, while return values commonly use rax. Windows x64 uses a different argument-register sequence. AArch64 commonly uses x0–x7 for early arguments and x0 for the return value.

When reversing, identify the platform ABI first. Otherwise you may mislabel arguments and data flow.

Stack concepts

The stack stores function-local state, saved registers, spill slots, and sometimes arguments. rsp on x86-64 and sp on AArch64 identify current stack position. Compilers may omit a traditional frame pointer, especially under optimization.

Do not assume source variables map one-to-one to fixed stack locations. Optimizers can keep values only in registers, merge variables, eliminate code, or transform loops completely.

Core x86-64 vocabulary

Common instructions you will see include:

• mov — copy data between registers/memory/immediates;
• lea — compute an address-like expression without dereferencing it;
• push / pop — adjust stack while storing/loading a value;
• call / ret — function call and return;
• cmp / test — update condition flags from a comparison;
• je/jz, jne/jnz, jg, jl, and related jumps — conditional control flow;
• jmp — unconditional branch;
• add, sub, and, or, xor, shifts — arithmetic/bit operations.

xor eax,eax is often compiler shorthand for setting eax to zero. lea is frequently arithmetic, not necessarily a pointer dereference.

Core AArch64 vocabulary

Common instructions include:

• mov or aliases for register/immediate movement;
• ldr / str — load/store from memory;
• add / sub — arithmetic and stack adjustments;
• cmp — comparison alias affecting flags;
• b, b.eq, b.ne, and other conditional branches;
• bl — branch with link (function call);
• ret — return, commonly using x30;
• adrp plus add/ldr — common position-independent address construction;
• stp / ldp — store/load register pairs, often in prologues and epilogues.

ARM64 devices also increasingly use pointer-authentication and branch-target protections, which may add instructions around calls/returns.

Endianness and integer width

Most x86-64 and mainstream AArch64 systems are little-endian. Multi-byte integer bytes therefore appear least-significant byte first in memory. Analysts routinely make mistakes by mixing the numeric representation with byte order.

Track widths carefully. Writing a 32-bit x86 register such as eax zero-extends into the full rax. AArch64 wN accesses the low 32 bits of xN. Signed versus unsigned comparisons also choose different branch conditions.

Pointers versus values

Assembly syntax does not magically label a number as a pointer. Context tells you whether a register contains an address. On x86 Intel-style syntax, brackets such as [rax] indicate memory at the address in rax. On AArch64, loads/stores use bracketed base addressing such as [x0].

During analysis, annotate each register with your current hypothesis: integer, pointer, length, file descriptor, flags, object pointer, return code. Revise the annotation when evidence changes.

Control-flow reconstruction

To recover a function's logic:

1. identify entry and return points;
2. mark calls;
3. mark conditional branches and their join points;
4. identify loops from backward branches;
5. label comparisons with the values being tested;
6. trace inputs to outputs;
7. ignore optimization noise until the main structure is clear.

A control-flow graph is often more useful than a line-by-line translation.

Data-flow reconstruction

Security bugs frequently concern where an untrusted value flows. Track the origin, transformations, bounds checks, type conversions, and final sensitive use. This is conceptually similar to taint analysis.

If a length is converted from signed to unsigned, multiplied, truncated, or used for both allocation and copy with different widths, note the transformation. You do not need to build an exploit to identify a dangerous mismatch.

Compiler optimizations

Optimization can inline functions, reorder operations, remove checks it proves redundant, use vector instructions, and turn switch statements into jump tables. Compare -O0 and -O2 output for code you own to learn how source structure changes.

Security research benefits from recognizing semantic operations rather than expecting source-like assembly.

Safe local disassembly lab

Create this benign program:

#include <stdio.h>
int classify(int x) {
    if (x < 0) return -1;
    if (x == 0) return 0;
    return 1;
}
int main(void) {
    printf("%d\n", classify(7));
    return 0;
}

Compile on a Linux lab if a C compiler is available:

cc -O0 -g demo.c -o demo
objdump -d demo | less

Then compare:

cc -O2 -g demo.c -o demo-o2
objdump -d demo-o2 | less

Find the comparisons, return values, call to output, and differences introduced by optimization. The program has no vulnerability; the point is learning representation.

ARM64 and Termux

Most modern Android devices use ARM64. If Termux provides a compiler/toolchain on your device, compiling tiny programs locally can make AArch64 assembly concrete. Android binaries also use ELF, dynamic linking, ASLR, SELinux, and platform-specific hardening. Access to debugging other apps is restricted by the Android sandbox and should remain so.

Recognizing defensive mitigations in code

Disassembly may show stack-protector checks, indirect-branch protection, pointer-authentication instructions, fortified library calls, or sanitizer instrumentation. Learn to identify these as compiler/platform evidence. Their presence does not prove the program is bug-free; their absence may increase the impact of certain bug classes.

Guided study workflow

Before you begin

Complete Modules 20, 53, and 62. Basic C syntax helps.

Practice task

Compile three tiny programs you own: a branch, a loop, and a function with two arguments. Compare debug and optimized disassembly. Annotate registers, calls, and branches.

Evidence to keep

Keep source code, compiler command, architecture, selected disassembly snippets, and your reconstructed pseudocode.

Common mistakes to avoid

• assuming one ABI applies everywhere;
• reading an address as a value or vice versa;
• ignoring signed/unsigned width;
• expecting optimized code to mirror source lines;
• copying exploit-oriented assembly snippets from the internet instead of learning on benign code.

Mastery check

Given a small function, identify its arguments, return path, conditions, and major data flow without relying on source code.

Continue with

Modules 64, 65, 66, 67, and 82.