Reverse Engineering and Program Analysis — DedSec Project

Purpose: Develop a disciplined workflow for understanding binaries, libraries, and mobile/native components you are authorized to analyze.

Learning objectives

• Combine static and dynamic analysis rather than relying on one view.
• Recover function boundaries, strings, data structures, control flow, and high-level behavior.
• Recognize compiler artifacts, indirect calls, virtual dispatch, and stripped symbols.
• Maintain an evidence-backed hypothesis notebook.
• Avoid over-interpreting decompiler output.

Static versus dynamic analysis

Static analysis examines code and metadata without executing the sample. It can reveal strings, imports, functions, control flow, constants, embedded resources, and cross-references. Dynamic analysis observes behavior during execution: files, sockets, syscalls, memory, logs, and state transitions.

Static analysis offers breadth; dynamic analysis offers concrete runtime evidence. Deep reverse engineering alternates between them.

Start with provenance

Before opening a binary, record:

• source and authorization;
• hash and size;
• file type and architecture;
• signature information if applicable;
• timestamps as metadata, not proof;
• surrounding package/container;
• analysis environment and tool versions.

For untrusted samples, use an isolated environment. For this guide's labs, prefer binaries you compile yourself or recognized training samples.

Triage before decompiling everything

A fast first pass asks:

• What format and architecture is it?
• Is it stripped?
• What libraries does it import?
• What strings expose configuration, paths, URLs, error messages, or protocol names?
• What sections/resources exist?
• Is there packing or compression?
• Does the binary contain debug/build identifiers?

This creates hypotheses about where to focus.

Strings as clues, not conclusions

Strings can reveal user-visible text, logs, function names, URLs, SQL fragments, file paths, or format strings. But they can be unused, obfuscated, compressed, dynamically built, or included by libraries. Always follow cross-references to see where and how a string is used.

Function identification

With symbols, functions may be named. Without them, tools infer boundaries from call targets, control flow, unwind metadata, signatures, and heuristics. Compiler optimizations can inline, split, merge, or tail-call functions, so “one decompiler function equals one source function” is not always true.

Rename functions based on evidence and confidence: parse_header_candidate is better than prematurely naming it decrypt_password.

Decompiler output

A decompiler reconstructs high-level pseudocode from machine instructions. Variable names and types are inferred. Loops and conditionals may look source-like while hiding low-level details.

Always confirm security-sensitive behavior in disassembly when types, signedness, exact bounds, pointer arithmetic, or indirect calls matter.

Cross-references

Cross-references answer “who calls this?” and “where is this data used?” They are among the most valuable reverse-engineering tools. Starting from an interesting string, imported API, error message, or comparison constant, follow inbound and outbound references until the surrounding state machine becomes clear.

Recovering data structures

Repeated memory accesses at consistent offsets often suggest fields in a structure/object. Track offset, width, use, and inferred meaning. A field read at offset +0x10 and compared to a length may be a size; a pointer dereferenced from another offset may reference a buffer.

Treat the layout as a hypothesis and test it against multiple functions.

Indirect calls and virtual dispatch

Object-oriented code frequently invokes methods through function tables/vtables. Callback-heavy code, event loops, plugin systems, and dynamically resolved APIs also produce indirect calls. Recovering possible targets requires type/context reasoning, not just following direct call instructions.

Control-flow protections may insert validation around indirect branches; recognize this as compiler/runtime infrastructure.

State-machine reconstruction

For parsers, protocols, and authentication logic, reconstruct states rather than isolated functions. Identify input ingestion, validation, parsing, transformation, authorization, side effects, and error handling.

A useful technique is to create a table: state, accepted input, transition condition, next state, security check, observable output.

Dynamic observation

Run owned/training code with benign inputs and observe:

• file access;
• process/thread creation;
• DNS/socket activity to lab endpoints only;
• log messages;
• loaded modules;
• configuration reads;
• error behavior;
• timing and retries.

Then map observations back to static locations. A log message can become a cross-reference anchor; a file path can lead to parsing code.

Patching as a learning tool

Binary patching can teach control flow, but use only binaries you own or training artifacts. Prefer harmless changes such as modifying a displayed string or changing a toy feature flag. Do not patch security controls in third-party software.

Document original hash, modified bytes, purpose, and rollback.

Obfuscation and packing

Obfuscation aims to make analysis harder by hiding names/control flow/data. Packing compresses/encrypts code that is reconstructed at runtime. These techniques can appear in legitimate protectors as well as malware.

Do not infer maliciousness from packing alone. Dynamic behavior, signature provenance, context, and observed actions matter.

Safe local reverse-engineering lab

Build a small program containing:

• three functions;
• one lookup table;
• one command-line option;
• one error string;
• one dynamically allocated structure;
• one checksum using a standard library.

Compile once with symbols and once stripped. Analyze the stripped copy and create pseudocode. Then compare with source to measure where your inferences were right or wrong.

Reverse-engineering notebook

For each renamed function record:

• original address/identifier;
• proposed name;
• confidence: low/medium/high;
• evidence: callers, strings, imports, fields;
• inputs/outputs;
• side effects;
• unanswered questions.

This prevents guesses from becoming “facts.”

Guided study workflow

Before you begin

Complete Modules 63–66.

Practice task

Reverse a stripped binary you compiled yourself without looking at source until the end. Recover program purpose, major functions, input handling, and at least one data structure.

Evidence to keep

Keep the exact binary hash, annotated function map, pseudocode, confidence notes, and comparison against source.

Common mistakes to avoid

• trusting decompiler types blindly;
• naming behavior from one string;
• ignoring optimization artifacts;
• running untrusted binaries outside isolation;
• patching software you do not own.

Mastery check

Explain one recovered function using both static evidence and a matching runtime observation.

Continue with

Modules 68, 79, 82, 83, and 84.