path: root/src/posts/primer-on-memory-management/index.md

---
title: A Primer on Memory Management
description: All programing languages have variables in them but have you imagined on where are they stored in memory and how it is managed. Also what is this memory safety and memory leaks which get mentioned around? Lets go to the deep dive into this concept

---

## Back to Basics

OK, so what is a variable?

A variable stores some data. More precisely it is an abstraction to refer to
some data that is stored in memory. It can be an integer, a string or anything
else. This data can be anything such as an integer, a character, a
floating-point number, a string or even a more complex structure. It is composed
of 2 attributes:

1.  Data: The actual value or content stored in memory.

2.  Data Type: Information about what kind of data the variable holds and how it
    should be handled.

### Where are the variable are stored?

*[RAM]: Random Access Memory

Some variables are stored in CPU registers and some are only used during compile
time. But CPU register are limited and have a small size and compile time
variable are constants that the compiler can optimise out. The rest of the
variables are stored in the devices memory, its primary memory (RAM).

The primary memory can be imagined as a massive array of bytes. On 32-bit device
the array size would be 4 GiB and 64-bit systems are unrealistically big. Now
the type of the variable contains 2 attributes to start with: size and
alignment.

#### Size

Size is how much space the variable uses in memory. In out array analogy it the
number of continuous array elements it will consume.

#### Alignment

*[ISA]: Instruction Set Architecture

On several ISA like x86_64, this does not matter but in several ISAs it does
such as MIPS. Now what is alignment? Alignment is the constraint that the memory
address of a variable must be a multiple of some power of two, typically its
size (or natural alignment). For primitive types, the variable is aligned such a
way that the memory address is divisible to its size. An example is if a 16-bit
`int` will be aligned 2 bytes and therefore can only start if the memory address
is divisible by 2 such as 0x100 or 0x102 but not 0x101 or 0x103. This has some
interesting ramification when we deal with composite types such as structs or
tuples. As alignment is usually only relevant when we talk about primitive types
such as `int`, `char` or `double`, a struct aligns it member according to its
alignment and therefore will need to insert padding in between members.

```c
struct data {
    char character;
    uint64_t id;
};
```

Which creates an object of the following layout in memory:

Offset | Content
-------|-----------------------
0x0000 | char character (1B)
0x0001 | padding (7B)
0x0008 | uint64_t id (8B)

The above example is an extreme but shows how is padding. Now in some compilers
the packed struct which disable padding but this is not portable as not all CPUs
can handle such code (but most mainstream ones can). Padding also means you may
want to be careful on order structure field so you do not waste space in
padding.

### Assignment

Another important operation all variables can do (Ignoring if you should) is
assignment. Its can be thought as an universal operation. In the case of
assigning, the value is copying the data. C's `memcpy` is basically the lowest
assignment operator at its raw form (for Plain Old Data).

Now there are caveats to this but we will bring them up later in this post when
we talk about lifetimes. In fact this especially obvious you this with big
objects where some compiler (like gcc or clang) just convert it to a call to
`memcpy`.

``` c
// A 64 kiB struct, too big for register
struct a {
    char a[1<<16];
};

static struct a b;

void a(struct a a) {
// assignment
    b = a; 
}
```

The assembly output on x86_64:

``` nasm
a:
    push    rbp
    mov     rbp, rsp
    lea     rsi, [rbp + 16]
    lea     rdi, [rip + b]
    mov     edx, 65536 ;; also this is assignment in x86
    call    memcpy@PLT ;; notice the memcpy
    pop     rbp
    ret
```

## Refer and Point

As mention above you copy the data on each assignment. This means in places like
function calls, the function has a copy of the original and its modification is
not sent outside. So there is special type of variable call pointers. Pointer
are simply variables that reference another variable. In most cases, this can be
the memory address and in a way the index number of the array analogy earlier.
This is not full accurate as modern OS uses virtual memory so it is the virtual
memory seen by the program is also seen by the assembly code, so this
abstraction doesn't affect low-level reasoning.

As mentioned, a pointers are simply variables whose data are being referenced
and they simply memory addresses which are unsigned integers. So this means you
can do a set of operations on it.

### Load and Store operations

As the data is referencing another variable and if you want read or write to the
referenced variable, you simply load or store. Load is reading from memory and
store is the write. Now to make life easier this is combined to a single
operator call dereferencing operator. You can see the usage in C and Zig bellow.

In C

``` c
*x = 10; // Store
y = *x; // Load
```

In Zig:

``` zig
x.* = 10; // Store
y = x.*; // Load
```

### Pointer Arithmetic

As a pointer is effectively an integer and therefore with some caveats. Addition
and subtraction of a pointer works. When adding as alignment is important, it
treats it as an array of the type and we going to the next index. Similarly when
subtracting from a regular integer. When 2 pointer subtract each other, we get
the offset in memory. Note the caveat about as mentioned above, this is only
true inside an object and not generally.

This can be seen in C's arrays it simply a syntax sugar when both pointer
arithmetic and dereferencing is done.

``` c
a[1] = 10;
// is equivalent
*(a + 1) = 10;
```

### References

A reference is more abstract concept than pointers. They reference another
object like pointers but do not necessary contain the memory address and are
more abstract. They can do load and store but not pointer arithmetic. Some
languages uses this as a safer version to pointers such as C++. I will also
mention about Reference Counting and Handles which are a common way to do this
in several languages.

### Why do you need references??

Well to understand, we need to remember how assignment operator works. It simply
copies data. In fact the parameters of a function are assignments. So when one
passes a parameter, the original variable is not modified.

``` c
#include <assert.h>
#include <stdlib.h>

struct foo {
    char var;
};

void bar(struct foo self) {
    self.var = 'a'; // Will only modify local copy
}

int main(void) {
    auto foo = (struct foo){ .var='z' };
    bar(foo); //send a copy
    assert(foo.var == 'a'); //Will fail
    return EXIT_SUCCESS;
}
```

Now if you do the same for any reference, you are not copies the variable but
the just the reference. For a pointer that will be just the memory address being
copied. Now since the only reference is copied, this has interesting implication
when doing store. When you dereference and write to the variable, it will update
the original variable. Even in places like functions.

So lets repeat the above an example with references instead.

``` c
#include <assert.h>
#include <stdlib.h>

struct foo {
    char var;
};

void bar(struct foo* self){ // passing a pointer
    (*self).var = 'a'; // Will modify the original
}

int main(void){
    auto foo = (struct foo){.var='z'};
    bar(&foo); //send a reference
    assert(foo.var == 'a'); //Will pass
    return EXIT_SUCCESS;
}
```

Also if you are working on certain algorithms or data structures, this allows
you store the reference for later processing. One point to add is that assigning
pointers is often call shallow copy as the pointer is copied but not the data.

Another thing to note is the time of copying. If the variable is less that the
word size of the CPU or in other words the biggest register, there is not much
performance penalty. But if you are transferring a bigger object such as a big
data structure, there is a visible cost in the copy. This is negligible unless
you are copying a multi kilobyte struct but its something to keep in mind.

*[CISC]: Complex Instruction Set Architecture

This also leads to an interesting trivia. As most variable are not allocated to
the registers but most CPU operation must work on registers which means in
machine code the variable is almost always a pointer and all operation are
surrounded by a load or store when doing anything. CISC hide this in the
microcode but look at any MIPS assembly output and the code will be littered
with `LW` and `SW` as shown bellow for adding 2 variables.

``` asm
; int c = a + b;
    lw      $1, 12($fp) ; load
    lw      $2, 8($fp)  ; load
    addu    $1, $1, $2  ; the actual action
    sw      $1, 4($fp)  ; store
```

Functions in compiled code are invoked via their memory address which is
referred to as function pointers in C-like languages.

Another interesting thing which is possible although unsafe is that if you have
pointer to a member of the struct and have its offset in the struct, you can
obtain the reference to its entire struct. This has several use cases such as
how subtyping polymorphism can be implemented but that is off topic for now.

## Advanced Hardware Memory Models

The above model is simplistic view. There are few memory models in the wild and
some nuances that then to be overlooked which is important to take note

### Basic Units of Memory

This seems simple at first as most CPU have few variations here but this is
still important to note.

**Bits** is a simply represent a single base 2 digit (0 and 1).

**Bytes** is the basic addressable memory of a system. On almost all modern
systems it is 8 bits. In fact this is mandated by POSIX and WIN 32 so the on
systems are legacy systems or embedded system. Most languages also make this
assumption (not C but most libraries make this assumption anyway).

**Words** is the unit of data that that a CPU processes natively in one
operation. Usually this represent the register size or the size of pointer. This
is usually `uintptr_t` in C. 32-bit systems have 3- bit word size while 64-bit
have 64-bit word size.

### Memory Models

The flat array model of memory is an oversimplification which is good enough for
most use case. But it still good to know underneath the hood too.

#### Von Neumann Architecture

Von Neuman Architecture is where code and data use the same memory bus and
therefore the same address space. This is basically the same as the model
mentioned earlier. Most modern non-embedded CPUs are using this architecture
with virtual memory. The biggest limitation of the model is there is a bottle
neck due to fetching code and data is the same bus (not parallel).

#### Harvard Architecture

*[JIT]: Just In Time Compilation
*[ROM]: Read only Memory

The data and code use different memory bus and therefore different addressing
space. This is common in embedded where the code is in ROM. Self-modifying and
JIT code is hard or impossible to implement. Probably the most famous one is AVR
used by Arduino.

#### Segmented Memory

Mostly obsolete but was popular in the past. In this the memory was divided into
segments. Each segment has the addressable memory space (this is the pointer). A
way to think of this is the RAM is a 2D array with segment being the new
dimension. Was used as a hack to deal with small word sizes in the 16 bit era.

#### Virtual Memory

*[MMU]: Memory Management Unit

This is an abstraction found on modern system. In this case the userspace gets
unique view of the memory (per process normally). This is managed by the MMU
which the kernel controls. The RAM is divided in pages and each page is mapped
to a process. This effectively sandboxes the processes so they do not access
other processes memory.

#### Cache

Modern CPUs need to multiple loads and stores so they have their own internal
memory called the cache. There are usually multiple layers. This is mostly
transparent but there are caveats especially on performance. As cache gets a
copy of a section of memory that is used, which mean that memory locality is
important as objects that are close together in RAM are faster than one father
apart (fragmented memory). This is relevant for the heap mentioned below.

## Lifetimes and Memory Storage

Well a variable does not exist from the start till the end of the
application(mostly). Each variable has a lifetime. One thing must be clear is
that any memory that needs to allocated at the start of the lifetime and
deallocated when they are no longer needed. Trying to access variables is to
deallocated pointer is invalid and this is called dangling pointer. The memory
being referenced can be used for something else and therefore the behaviour is
undefined.

### Static Memory

Now global variables have a lifetime entire program so it can be allocated with
the entire program and can deallocated when the program end. As long these
objects need to allocate the safe size these variable can always (if compiler
allows) to be allocated in static memory. This is stored in binary in compiled
languages (Sometimes in a way to be auto allocated).

``` c

#include <assert.h>
#include <stdlib.h>

struct foo {
    char var;
};

static struct foo foo; // Global variable that is allocated in static

void bar(){
    foo.var = 'a'; // Will modify the global variable
}

int main(void){
    foo = (struct foo){.var='z'};
    bar(); // Call the function.
    assert(foo.var == 'a'); // Will pass as foo is global is shared in all functions
    return EXIT_SUCCESS;
}
```

There may be copies per thread which have a lifetime equivalent to threads.

### Stack Memory

*[LIFO]: Last in First Out *[OS]: Operating System

Now you have local variables to a particular programming block such as function
or in some case constructs like if-else or loops. As these can be nested or
repeated due to recursion you need multiple copies of them unlike static
variables. But the lifetimes of these variables are equal to the runtime of the
block. To make life easier, the OS allocates a stack which can be used for this.
There are some important things to know about the stack. The stack is LIFO
structure and its top is stored as the stack pointer position. So once
deallocation occurs, the stack point reset to the value before the block. In a
sense the following pseudocode gives an idea how it is done.

``` cpp
static void *sp = &stack; // The entire stack can be though as a giant static memory

T* alloc<typename T>(){
    align(&sp, alignof T); // Make sure the new pointer are in the correct alignment
    T* new = sp; // set the new pointer to current stack pointer
    sp += sizeof T; // Update to stack pointer to length of the pointer
    return new;
}

void dealloc<typename T>(T* old){
    sp = old; // reset
}
```

Observe the `dealloc` function or method. Its really fast but it frees any thing
allocated after it. This means that while it's good for local variables, it also
implies that the return variable of a function must be at the top of the stack
after the function returns. This means returning pointer to local variable is
impossible. And any variable with lifetime that that is unknown and if variable
has an unknown size at compile time can not use stack.

Another major limitation is stack has limit size of most OSes in the range of
1-16 MiB. If you exceed this, you will face a stack overfull and can not store
more data in the stack (safely without crashing or ending up somewhere you
should not).

### The Heap

The last place is the heap. Now when static and stack are unsuitable, there is
the heap. Now heap unlike in stack or static, is not given freely to the user by
the environment but the user must manually allocate the OS from memory and
deallocate. This has few major advantages, mainly as the user is control over
size and lifetimes. In fact the object size is unknown unless the language does
the allocation for the use.

This is especially used for when lifetimes are very different and custom.
Example are network sockets or widgets.

``` c
#include <assert.h>
#include <stdlib.h>

[[nodiscard]] char *bar(){
    char* a = malloc(10); // request 10 bytes of heap from runtime
    if (!a){
        return nullptr; // malloc fails discuss later
    }
    strcpy(a, "hello");
    return a; // Will not free a
}

int main(void){
    auto str = bar(); // Call the function.
    if (!str){
        return EXIT_FAILURE;
    }
    puts(str); // print the string
    free(str); // release the memory
    return EXIT_SUCCESS;
}
```

## Memory Safety

Memory safety is about avoiding bugs where you violate type or lifetime
guarantees. A few common issues:

### Unsafe Pointer Arithmetic

Doing raw arithmetic can take you out of bounds, either into another variable or
into unmapped memory leading to corruption or segmentation faults.

``` c
char a[16] = {0};

// Can give data from another variable or segmentation fault
char const b = *(&a + 17); // out of bounds
char const b = *(&a - 1); // out of bounds
```

### Missing Metadata

In low-level languages like C, heap allocations don't retain type metadata,
which makes safety checks impossible without external info. A good example of
this size which is a source of out of bound error. Most modern languages handle
this as either you have some form of a fat pointer (a pointer with data attached
to it) like slices, span (in C++) or just arrays.

If not available the solution is generally to store the metadata out of bound by
making your own fat pointer type or request in your function.

``` c
// example pattern
void memzero(void* data, size_t size);
```

### Null Pointer

This is is a generic invalid pointer found in many languages. It cannot be
dereference and attempt to do so in most language is a runtime error
(segmentation fault in C like languages, exception/panic is languages that have
them).

The usual semantic is this pointer/reference is not pointing to any variable. If
it is possible to get NULL (the type system does not provide a method to check
at compile time), check it before any for of dereferencing. A lot of languages
have nullish operators (?.) which are useful shortcuts when dealing with null
pointers.

### Out of Memory Errors

Most Programs assume `malloc` will not error which is mostly true as
operating systems overcommit memory. But still either due lack of overcommit so
out of memory or other reason (such as internal fragmentation). This must be
handled (`malloc` returns null and exception in certain languages). Be very
careful and never assume `malloc` will always return memory.

### Lifetime Errors

There are three types of lifetime errors:

- **Use after free**: using memory after it's been deallocated. Can corrupt
future allocations.

- **Use before initialised**: using memory before it's been initialised. Can get
garbage data.

- **Memory leak**: memory is never deallocated after use, slowly consuming RAM.

In C, manual memory management means you have to:

- track lifetimes

- track the ownership

- store sizes

- Be careful with pointer arithmetic.

## Memory Management Approaches

Languages and runtimes use different strategies to manage memory safely and
efficiently.

### Garbage Collection

Used in Python, JavaScript, Java, etc. The runtime tracks which variables are
"still in use" and frees unreferenced ones. This avoids most manual bugs but
comes with costs:

- Garbage Collection adds runtime overhead and pauses.

- May not detect cycles without extra work (some Garbage Collectors leak cyclic
data).

- Performance is unpredictable in tight loops or latency-sensitive systems.

But this is automatic which can simplify code especially since in some languages
(dynamic and prototypical) an type/object is more complicated in memory then a
simple size+data mentioned earlier.

A few languages like Go has escape analysis which allows them to determine
whether a variable escapes the current function scope. If it does not, it can be
safely allocated on the stack even when the syntax appears heap-bound.

### Reference Counting

A simpler alternative to Garbage Collection (Sort of the simplest and naive
possible implementation). Each object tracks how many references point to it.
When reference count hits zero, it's freed.

- Easier to reason about.

- Handles shared ownership.

- Doesn't clean up cycles (unless using weak references or hybrid collectors).

A classic example is C++ smart pointers or Rust's Arc.

The bellow pseudocode shows this

``` cpp
struct ref<typename T>{
    T* ptr;
    unsigned ref_count = 1;
}

void assign(ref<T> *self){
    self->ref_count++;
}

void drop(ref<T> *self){
    self->ref_count--;
    if(0 == self->ref_count){
        free(self->ptr); // Lifetime of the pointer is over
    }
}
```

### Ownership + Lifetimes in Type System

Seen in Rust. The compiler encodes lifetimes and ownership rules directly in
types.

- Compiler ensures memory safety statically.

- You can't use freed memory or double-free.

- But this comes at a cost: increased mental overhead, especially in complex
cases (e.g. double-linked lists, graphs).

### Arenas / Region Allocators

*[AST]: Abstract Syntax Tree

Multiple objects are allocated in a chunk and freed together. Great for when
lifetimes can be grouped (e.g. AST nodes, temporary buffers).

- Reduces malloc/free churn.

- Simplifies lifetime logic.

- But causes waste if lifetimes are mixed (over-retention).

This is a great technique and is popular in modern C and Zig.

A method which I personally use in the case of trees is to create an object pool
(which is similar to arena but only for a single type). This simplifies clean up
of tree when done.

*[RAII]: Resource acquisition is initialization
*[OOP]: Object Oriented Programming

### RAII

RAII is a method that can be considered a very simplified of automatic version
of resource management(which heap is a resource). There are 2 variants of this,
defer and destructor.

In some OOP languages, destructor is called when the pointer in the stack is
freed. This allows cleaning other stuff including the memory holded by the
object. This is a common method in c++ and rust especially when combined with
reference counting.

Similar pattern for defer but this works by inserting code hooks to the end of
scope so you can call cleanup after you are done. This is in Zig.

``` zig 
var list = ArrayList(u8).init(allocator);
defer list.deinit();
```

Now this method is not memory management specific and ownership and lifetimes
are technically also for system resources too like files, devices, windows, etc.
So other language have these mechanism although not relevant in this case.

Although C lacks this, defer can be emulated by goto/longjump and is probably
one the few legitimate uses of that keyword.

### Free on Exit

This is a valid method although not recommended. In many runtimes and OS,
exiting the process will free all heap too. So you can avoid freeing the heap.
This can be used if your program is short. Personally this is an anti-pattern
unless you are exiting due to an unrecoverable error/exception. The reason it is
an anti-pattern is that you risk creating a memory leak once the code expand
(the same logic against global variables or singletons).

### Handles

This is a mix of Arenas and the abstract reference before. The idea is instead
of providing pointer to an object, pass a handle(simplest is an id). The actual
object is stored in a subsystem's internal array or hash map. This is powerful
as the ownership belong to the subsystem. And depending on the mechanism, use
after free of the handle is detectable if the correct metadata is handled.

This is powerful but is more complicated and works with more long term systems.

## Implicit References in High-Level Constructs

Just in case you are wondering in the scenario when your language does not have
pointers, how it is working?

Well most language implicitly use pointers for stuff you may not notice except
the pass by reference at times. I will explain why.

Couple of language features need pointer for implicitly carrying context where
the context of the function is smuggled in. The common features are:

- Subtyping Polymorphism: In the form of inheritance or interfaces, the method
needs to get the correct type (this or self in many languages). This usual works
with methods having a virtual table and a mechanism (either a pointer in the
virtual table or via some pointer arithmetic) to get the actual type. In case of
prototypical inheritance (in JavaScript and Lua), the object are hash maps that
need to store in RAM due to the nature of mutable hash maps.

``` go
func sum(x int) func(b int) int {
    return func (b int){
        return x+b // x is capture and needs to be stored somewhere
    }
}
```

- Lambdas: Lambdas or nested function need to capture the local variable. But as
the function need a copy of all the variables since it is often passed as
callbacks or returned.

- Coroutines: The backbone of any asynchronous system. The coroutine must be
able to suspend it self and therefore save its stack somewhere else while the
rest of the program runs. This is again stored in the heap.

## Closing Thoughts

A few general principles hold across all strategies:

- Prefer **single ownership** if possible. It's easiest to reason about.

- Use **reference counting** for shared ownership.

- **Encode data size** somewhere, either via the type system or with fat
pointers. This helps with bounds checks and memory layout.

- **Respect lifetimes**. Use stack, RAII, or arenas when you can.

*[GUI]: Graphical User Interface

- If you're not working on a performance critical system (e.g. simple GUI or
script), garbage collectors are fine. The mental overhead saved is often worth
it.

Even in high-level systems, understanding the low-level trade-offs helps avoid
performance cliffs and debug nasty bugs faster.