Skip to content

Linux Toolchains

Short notes about native compilers, cross compilers, libc, dynamic linking, and static linking.

Terms

term meaning
Compiler Translate source code to object files or a binary.
Linker Connect object files with libraries and create the final executable.
Host compiler Compiler that builds a binary for the same machine you are using.
Cross compiler Compiler that runs on one machine but builds a binary for another target.
libc The C runtime library used by Linux programs, for example glibc or musl.
Dynamic binary Loads shared libraries at runtime.
Static binary Copies the needed libraries into the executable at link time.

Compile only:

1
g++ -c code/hello.cpp -o hello.o

Link:

1
g++ hello.o -o hello

Compile and link in one command:

1
g++ code/hello.cpp -o hello

g++ calls the compiler and then the linker. For C++ it also links the C++ standard library.

Why linker is used when the binary runs

For a dynamic binary, linking is not fully finished when you build the program.

The binary contains:

  • the program code
  • the shared libraries it needs
  • the runtime loader path, for example /lib64/ld-linux-x86-64.so.2

When you run the binary, Linux starts the loader first. The loader finds the shared libraries, maps them into memory, resolves symbols, and then starts main().

Check it:

1
2
readelf -l hello | grep interpreter
ldd hello

For a static binary, there is no runtime loader dependency for normal libc libraries:

1
2
file hello-static
ldd hello-static

What in the package

A compiler or cross-compiler package usually contains several programs. Cross tools use a target prefix such as aarch64-linux-gnu- or aarch64-linux-musl-; native tools normally have no prefix.

Binary Short description Role in the build process
*-gcc / *-cc C compiler driver Preprocesses and compiles C, then calls the assembler and linker when needed.
*-g++ / *-c++ C++ compiler driver Compiles C++ and automatically links the C++ standard library.
*-cpp C preprocessor Expands headers, macros, and conditional compilation directives.
*-as Assembler Converts assembly source into object files.
*-ld Linker Combines object files and libraries into an executable or shared library.
*-ar Archive manager Creates and updates static libraries such as libexample.a.
*-ranlib Archive indexer Adds or refreshes the symbol index in a static library.
*-nm Symbol viewer Lists symbols defined or referenced by object files and binaries.
*-objcopy Object-file converter Copies or converts object formats and can produce raw binary or Intel HEX images.
*-objdump Object-file inspector Displays headers, symbols, sections, and disassembled machine code.
*-readelf ELF inspector Displays ELF headers, sections, segments, symbols, and dependencies.
*-strip Debug-symbol remover Reduces the size of the final binary by removing symbols and debug data.
*-size Section-size reporter Shows the code, data, and BSS sizes of an object file or executable.
*-strings Printable-text finder Extracts readable strings from object files and binaries.
*-addr2line Address lookup tool Maps machine-code addresses back to source files and line numbers.
*-c++filt C++ symbol demangler Converts encoded C++ symbol names into readable names.

The * represents the target prefix. For example, aarch64-linux-musl-g++ creates AArch64 Linux programs that use musl.

glibc vs musl

The C standard library (libc) provides programs with common functions for memory allocation, file and network I/O, string handling, and process management. It also acts as the main interface between most C programs and the Linux kernel, so the libc used by a toolchain affects compatibility, binary size, and deployment requirements.

C++ programs normally depend on libc as well. The C++ standard library (libstdc++ with GCC or libc++ with Clang) provides C++ features such as containers, streams, and exceptions, while libc provides lower-level services such as memory allocation, file I/O, threads, and system-call wrappers. A dynamically linked C++ executable therefore usually requires both a C++ standard library and the target system's libc. Static linking can include these libraries in the executable, but the selected libc still affects compatibility and behavior.

libc common use
glibc Default libc on most Debian, Ubuntu, Fedora, and many desktop/server Linux systems.
musl Small libc used by Alpine Linux and useful for static embedded deployment.

glibc is common and very compatible with normal Linux distributions.

musl is designed to be small, simple, and friendly to static linking. It is useful when you want one binary that is easy to copy to an embedded target.

Static compiler

"Static compiler" usually means a compiler toolchain or build command that creates a statically linked binary.

It is not a different C++ language. The main difference is the link step:

1
2
g++ code/hello.cpp -o hello-dynamic
g++ code/hello.cpp -static -o hello-static

Install glibc host tools

Install native build tools on Ubuntu/Debian:

1
2
sudo apt update
sudo apt install build-essential file binutils

Build with the host glibc compiler:

1
2
3
g++ code/hello.cpp -o hello-glibc-dynamic
g++ code/hello.cpp -static -o hello-glibc-static
file hello-glibc-dynamic hello-glibc-static

Install glibc cross compiler

For AArch64 target:

1
2
sudo apt update
sudo apt install gcc-aarch64-linux-gnu g++-aarch64-linux-gnu

Verify:

1
2
which aarch64-linux-gnu-g++
aarch64-linux-gnu-g++ --version

Build AArch64 binaries with glibc:

1
2
3
aarch64-linux-gnu-g++ code/hello.cpp -o hello-aarch64-glibc-dynamic
aarch64-linux-gnu-g++ code/hello.cpp -static -o hello-aarch64-glibc-static
file hello-aarch64-glibc-dynamic hello-aarch64-glibc-static

Demo: hello cross compiler

Build cpp app using cmake and cross compiler

1
2
3
4
5
6
7
project/
├── CMakeLists.txt
├── cmake/
│   └── toolchain-aarch64.cmake
└── build/
    ├── native/
    └── arm64/
toolchain-aarch64.cmake
1
2
3
4
5
6
7
# cmake/toolchain-aarch64.cmake

set(CMAKE_SYSTEM_NAME Linux)
set(CMAKE_SYSTEM_PROCESSOR aarch64)

set(CMAKE_C_COMPILER aarch64-linux-gnu-gcc)
set(CMAKE_CXX_COMPILER aarch64-linux-gnu-g++)
CMakeLists.txt
1
2
3
4
5
cmake_minimum_required(VERSION 3.20)

project(hello LANGUAGES CXX)

add_executable(hello main.cpp)
main.cpp
1
2
3
4
5
6
7
#include <iostream>

int main()
{
    std::cout << "Hello ARM64\n";
    return 0;
}

1
2
cmake -S . -B build-arm \
    -DCMAKE_TOOLCHAIN_FILE=cmake/toolchain-aarch64.cmake

1
cmake --build build-arm

modern cmake

Modern CMake describes targets and their usage requirements instead of setting global compiler flags or include paths. Prefer commands such as add_library, add_executable, target_link_libraries, and target_include_directories, with PRIVATE, PUBLIC, or INTERFACE to control which requirements are propagated to dependent targets.

Use CMake's separate workflow commands for each stage:

1
2
3
4
cmake -S . -B build                         # Configure and generate
cmake --build build                         # Compile
cmake --install build --prefix ./install    # Install
ctest --test-dir build --output-on-failure  # Run tests

Useful compilers for other architecture targets:

target packages
AArch64 gcc-aarch64-linux-gnu g++-aarch64-linux-gnu
ARM hard-float gcc-arm-linux-gnueabihf g++-arm-linux-gnueabihf
ARM soft-float gcc-arm-linux-gnueabi g++-arm-linux-gnueabi
x86_64 gcc-x86-64-linux-gnu g++-x86-64-linux-gnu

Cross compiler and ROOTFS

Debug remote with gdb

Install musl cross compiler from musl.cc web site

The site musl.cc provides prebuilt musl toolchains. It is a community source, not the official musl project.

Choose the archive by the target CPU, not by the host CPU.

For an x86_64 Ubuntu host building for AArch64 Linux, use:

1
aarch64-linux-musl-cross.tgz

Do not use this unless the target is x86_64 Linux with musl:

1
x86_64-linux-musl-cross.tgz

Download and install in /opt:

1
2
3
4
cd /tmp
wget https://musl.cc/aarch64-linux-musl-cross.tgz
tar -xf aarch64-linux-musl-cross.tgz
sudo mv aarch64-linux-musl-cross /opt/

Add it to the shell path:

1
2
echo 'export PATH=/opt/aarch64-linux-musl-cross/bin:$PATH' >> ~/.bashrc
source ~/.bashrc

Verify:

1
2
which aarch64-linux-musl-g++
aarch64-linux-musl-g++ --version

Build with musl cross compiler

Dynamic musl build:

1
2
aarch64-linux-musl-g++ code/hello.cpp -o hello-aarch64-musl-dynamic
file hello-aarch64-musl-dynamic

Static musl build:

1
2
aarch64-linux-musl-g++ -static code/hello.cpp -o hello-aarch64-musl-static
file hello-aarch64-musl-static

Static musl binaries are useful for embedded targets because they do not require the target root filesystem to provide matching shared libc files.

Demo: using musl with cmake

1
2
3
4
5
├── build-musl
├── cmake
│   └── musl-toolchain.cmake
├── CMakeLists.txt
└── main.cpp
main.cpp
1
2
3
4
5
6
#include <iostream>

int main() {
    std::cout << "Hello, musl!" << std::endl;
    return 0;
}
CMakeLists.txt
1
2
3
4
cmake_minimum_required(VERSION 3.16)
project(my_app LANGUAGES CXX)

add_executable(my_app main.cpp)
cmake/musl-toolchain.cmake
1
2
3
4
5
6
7
8
set(CMAKE_SYSTEM_NAME Linux)
set(CMAKE_SYSTEM_PROCESSOR aarch64)

find_program(MUSL_CC aarch64-linux-musl-gcc REQUIRED)
find_program(MUSL_CXX aarch64-linux-musl-g++ REQUIRED)
set(CMAKE_C_COMPILER ${MUSL_CC})
set(CMAKE_CXX_COMPILER ${MUSL_CXX}
)

find_program

find_program() searches the system’s PATH for an executable and stores its full path in a CMake variable.

find_program(MUSL_CC aarch64-linux-musl-gcc REQUIRED)

  • MUSL_CC: variable receiving the executable path.
  • aarch64-linux-musl-gcc: program to find.
  • REQUIRED: stops configuration if it is not found.
1
2
3
4
5
cmake -S . -B build-musl \
    -DCMAKE_TOOLCHAIN_FILE=cmake/musl-toolchain.cmake

# build
cmake --build build-musl

Test AArch64 binary on x86_64 host

Install QEMU user emulation:

1
2
sudo apt update
sudo apt install qemu-user

Run the static binary:

1
qemu-aarch64 ./hello-aarch64-musl-static

Expected output:

1
hello toolchain