CPU Privilege, Syscalls and Process Internals — DedSec Project

Purpose: Build the operating-system mental model needed for deeper vulnerability research, reverse engineering, sandbox analysis, and endpoint defense.

Learning objectives

• Explain user mode, kernel mode, virtual memory, page permissions, and process isolation.
• Understand a system call as a controlled transition into the kernel.
• Trace the lifecycle of a process from executable loading through threads, files, memory mappings, signals, and exit.
• Recognize why privilege-boundary bugs are especially important.
• Use read-only local tools to inspect your own processes.

Why internals matter

Security tools eventually reduce to operating-system primitives. A process opens files, maps memory, creates sockets, spawns threads, communicates through IPC, and requests privileged kernel operations through system calls. If you understand those primitives, logs and detections become easier to interpret and reverse-engineered behavior stops looking mysterious.

CPU privilege model

Modern operating systems separate ordinary application execution from privileged kernel execution. On x86, this is commonly described using privilege rings even though mainstream kernels mostly use ring 3 for user applications and ring 0 for kernel code. ARM systems use exception levels with similar separation goals.

The important security idea is not the numeric ring. It is that code executing with ordinary application privilege cannot directly perform operations reserved for the kernel. It must cross a controlled interface. A vulnerability that crosses this boundary can transform a local application bug into a system-level compromise, which is why kernel attack surface receives intense scrutiny.

Virtual memory

Each process sees a virtual address space. The operating system and hardware memory-management unit translate virtual addresses into physical memory mappings while enforcing permissions. Pages can be readable, writable, executable, or combinations allowed by the architecture and operating system.

Virtual memory provides isolation, relocation, demand paging, shared mappings, copy-on-write, guard regions, and memory-mapped files. Security mitigations such as ASLR and non-executable memory depend on this machinery.

Do not assume an address observed in one run is meaningful in another. ASLR deliberately changes selected locations. Also distinguish an address being mapped from the mapped content being valid for the operation you intend.

User-kernel transition

A system call is a defined interface by which a user process asks the kernel to perform an operation. Examples include opening a file, reading from a descriptor, creating a socket, changing memory mappings, or creating another process/thread. Architectures use specialized instructions to transition safely to kernel code.

From a security perspective, every syscall handler is part of the kernel's reachable interface. The kernel must validate pointers, lengths, flags, object lifetimes, permissions, namespaces, and concurrent state because callers are not trusted merely because they are local.

File descriptors and object handles

Unix-like systems expose many resources as file descriptors: files, pipes, sockets, event interfaces, devices, and more. Windows uses object handles with a different object model but a similar lesson: possession of a reference can carry authority.

A common security design question is whether authority is checked when the reference is created, every time it is used, or both. Descriptor/handle inheritance also matters. A child process may unintentionally inherit access to sensitive resources unless they are explicitly closed or marked non-inheritable.

Process creation

On Unix-like systems, process creation often involves fork()-style duplication followed by exec() replacement, although modern implementations and APIs have optimizations and alternatives. Windows commonly creates a new process from an executable image with an explicit creation API.

Security-relevant properties include environment variables, current directory, inherited descriptors/handles, credentials, namespaces, working token, executable search path, loaded libraries, and parent-child relationships.

Threads and concurrency

Threads share much of a process's address space. Concurrency creates security problems when checks and uses are separated in time, objects can change ownership, or references remain after a lifetime ends. Race conditions, time-of-check/time-of-use errors, deadlocks, and reference-count bugs are all easier to understand once you model concurrent actors explicitly.

Memory mappings

Executable code, shared libraries, stacks, heaps, anonymous mappings, and memory-mapped files occupy regions with different properties. On Linux, /proc/<pid>/maps exposes mapping metadata for processes you are allowed to inspect. Use it only for your own processes in this lab.

Try:

python -c 'import os,time; print(os.getpid()); time.sleep(60)'

In another local shell, inspect the printed PID:

cat /proc/PID/maps | head

Observe categories and permissions; do not treat the exact addresses as stable.

Credentials and security context

A process runs with a security context. On Unix-like systems this includes real/effective/saved user and group IDs, supplementary groups, capabilities, namespaces, and security-module labels where applicable. On Windows, access tokens contain identity, groups, privileges, integrity information, and other claims.

The deep lesson is that “administrator” or “root” is too coarse a mental model. Real authorization decisions depend on the current subject, object, requested operation, policy, namespace, label, and sometimes impersonation/delegation state.

IPC and local trust boundaries

Processes communicate through pipes, Unix-domain sockets, shared memory, RPC mechanisms, message buses, loopback TCP, named pipes, files, and framework-specific brokers. Local IPC is still a trust boundary when peers have different privileges.

Review IPC designs for peer authentication, object ownership, message framing, replay, confused deputy behavior, and inherited authority.

Signals, exceptions, and asynchronous control flow

Signals on Unix and exceptions/events on other platforms interrupt normal control flow. Handlers have constraints; unsafe operations inside signal contexts can cause subtle bugs. From a security-analysis standpoint, asynchronous events also complicate crash reproduction because state can change between observations.

/proc as a learning surface

On Linux/Termux, /proc exposes kernel-generated process and system views. Availability differs on Android because SELinux, app sandboxing, kernel configuration, and platform restrictions limit access. Useful read-only observations for your own process include:

cat /proc/self/status
cat /proc/self/maps | head
ls -l /proc/self/fd

Record what exists on your device rather than assuming desktop Linux behavior.

Syscall tracing as explanation

Where available in a disposable Linux lab, syscall tracing can explain what a program asks the OS to do. The objective is not bypassing controls; it is correlating application behavior with OS events. Trace a program you wrote that opens one file and prints it. Identify the open/read/write/close lifecycle and note library/runtime noise around the core operations.

On Android/Termux, tracing support and ptrace restrictions vary. Treat inability to trace as a platform fact, not an error to defeat.

Privilege-boundary review checklist

When reviewing a privileged service, ask:

• Which unprivileged inputs reach it?
• How are message lengths and types validated?
• How is the peer identity established?
• Does it resolve file paths after checking them?
• Does it operate on caller-controlled file descriptors?
• Can objects change between authorization and use?
• Are environment variables or search paths trusted?
• Are inherited handles/descriptors restricted?
• Is the privileged portion minimal?

Safe lab: process anatomy

Write a small Python program that opens a temporary file, creates a socket bound only to 127.0.0.1, starts a thread, waits for input, and exits. While it runs, inspect its /proc status, descriptors, and mappings. Draw a diagram connecting language-level operations to OS resources.

The exercise develops the ability to reason from high-level code to kernel-visible artifacts without attacking anything.

Guided study workflow

Before you begin

Complete Modules 28, 33, 51, 53, and 61. A Linux VM is ideal; Termux can cover many observations but Android deliberately restricts some interfaces.

Practice task

Map one program you own: executable/interpreter, process, threads, descriptors, mappings, sockets, and credentials. Identify which operations require kernel mediation.

Evidence to keep

Keep sanitized /proc excerpts, your diagram, and a table mapping application actions to OS resources.

Common mistakes to avoid

• assuming Linux desktop behavior is identical to Android;
• treating virtual and physical addresses as the same concept;
• assuming all local IPC is trusted;
• describing root/admin as the only meaningful privilege boundary;
• attempting to bypass platform restrictions for a learning exercise.

Mastery check

Explain why a syscall is a security boundary, why descriptor inheritance matters, and how virtual-memory permissions support exploit mitigations.

Continue with

Modules 63, 64, 65, 66, 73, and 74.