Running Ubuntu 22.04 on QEMU with ATF

Introduction

This repository contains the scripts and configurations to run a QEMU virtual machine with an ARM64 architecture.

Qemu boot in aarch64, with ATF(arm trusted firmware) and EDK2 firmware.

QEMU virt Armv8-A

Trusted Firmware-A (TF-A) implements the EL3 firmware layer for QEMU virt Armv8-A. BL1 is used as the BootROM, supplied with the -bios argument. When QEMU starts all CPUs are released simultaneously, BL1 selects a primary CPU to handle the boot and the secondaries are placed in a polling loop to be released by normal world via PSCI. BL2 edits the Flattened Device Tree, FDT, generated by QEMU at run-time to add a node describing PSCI and also enable methods for the CPUs. If ARM_LINUX_KERNEL_AS_BL33 is set to 1 then this FDT will be passed to BL33 via register x0, as expected by a Linux kernel. This allows a Linux kernel image to be booted directly as BL33 rather than using a bootloader. An ARM64 defconfig v5.5 Linux kernel is known to boot, FDT doesn't need to be provided as it's generated by QEMU. Current limitations: Only cold boot is supported

Installation

QEMU

Download QEMU

# Install dependencies
sudo apt-get install build-essential gcc pkg-config glib-2.0 libglib2.0-dev libsdl1.2-dev libaio-dev libcap-dev libattr1-dev libpixman-1-dev
# Install required packages
pip install sphinx sphinx_rtd_theme
# Download qemu source code
wget https://download.qemu.org/qemu-8.2.6.tar.xz
tar -xvf qemu-8.2.6.tar.xz
# Build qemu
cd qemu-9.1.0-rc1
./configure
make -j$(nproc)

EDK2

Using edk2 in order to create the QEMU_EFI.fd firmware file for the aarch64 emulation in QEMU.

prerequisites:

binutils-aarch64-linux-gnu,
gcc-10-aarch64-linux-gnu-base,
gcc-aarch64-linux-gnu (for aarch-linux-gnu-gcc),
acpica-tools,
uuid-dev,

QEMU_EFI.fd can be built as follows:

git clone https://github.com/tianocore/edk2.git
cd edk
git submodule update --init
make -C BaseTools (* in case of compilation error, if uuid.h is missing do a apt-get install uuid-dev)
source edksetup.sh
export GCC5_AARCH64_PREFIX=aarch64-linux-gnu-
build -a AARCH64 -t GCC5 -p ArmVirtPkg/ArmVirtQemuKernel.dsc

Then, you will get Build/ArmVirtQemuKernel-AARCH64/DEBUG_GCC5/FV/QEMU_EFI.fd. Also, we copy the generated QEMU_EFI.fd into the arm-trusted-firmware directory as QEMU_edk2_EFI.fd.

# Download the QEMU_EFI image, extract and gunzip it
$ wget https://releases.linaro.org/components/kernel/uefi-linaro/16.02/release/qemu64/QEMU_EFI.img.gz
$ tar -xvf QEMU_EFI.img.gz
$ gunzip QEMU_EFI.img.gz

buildroot

For the rootfs we are using Buildroot. The rootfs can be built by using Buildroot as follows:

git clone git://git.buildroot.net/buildroot.git -b 2024.05
cd buildroot
make qemu_aarch64_virt_defconfig
utils/config -e BR2_TARGET_ROOTFS_CPIO
utils/config -e BR2_TARGET_ROOTFS_CPIO_GZIP
make olddefconfig
make

$ ls output/images/
Image  rootfs.cpio  rootfs.cpio.gz  rootfs.ext2  rootfs.ext4  start-qemu.sh

arm-trusted-firmware

For the ATF compilation:

prerequisites:

libssl-dev

ln -s ~/cca/edk2/Build/ArmVirtQemuKernel-AARCH64/DEBUG_GCC5/FV/QEMU_EFI.fd
make BL33=QEMU_EFI.fd CROSS_COMPILE=aarch64-linux-gnu- PLAT=qemu all fip
dd if=build/qemu/release/bl1.bin of=flash.bin bs=4096 conv=notrunc
dd if=build/qemu/release/fip.bin of=flash.bin seek=64 bs=4096 conv=notrunc

As BL33 we use the QEMU_EFI.fd generated by the EDK2.

# Download the ARM Trusted Firmware and compile it with the QEMU_EFI image
$ cd ~/cca/arm-trusted-firmware
$ make CROSS_COMPILE=aarch64-linux-gnu- PLAT=qemu all fip DEBUG=0 BL33=~/cca/QEMU_EFI.fd

Kernel

# Download the linux kernel and extract it
$ wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.15.115.tar.xz
$ tar -xvf linux-5.15.115.tar.xz

cd linux
make ARCH=arm64 mrproper
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- defconfig
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- dtbs
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- -j128

Install kernel modules to rootfs

mkdir rootfs
sudo qemu-nbd --connect=/dev/nbd0 rootfs.qcow2
sudo mount /dev/nbd0p1 rootfs
sudo make modules_install INSTALL_MOD_PATH=/home/benshan/cca/rootfs
sudo umount rootfs
sudo qemu-nbd --disconnect /dev/nbd0

busybox

$ test -f busybox-1.36.1.tar.bz2 || wget https://busybox.net/downloads/busybox-1.36.1.tar.bz2
$ tar -xvf busybox-1.36.1.tar.bz2
$ cd busybox-1.36.1
$ export ARCH=arm64
$ export CROSS_COMPILE=aarch64-linux-gnu-
$ make menuconfig
Settings  --->
[*] Build static binary (no shared libs)    //静态编译
[*] Build with debug information       //可选，带调试信息，方便后续调试
$ make; make install

busybox 根目录下_install/ 就是根文件系统了

qemu 搭建 arm64 linux kernel 调试环境

用busybox制作initramfs并启动

Build RootFS

Buildroot Image

In this initial script, we use the rootfs.cpio.gz and the Image built previously with the Buildroot.

qemu-system-aarch64 -nographic \
    -machine virt,secure=on \
    -cpu cortex-a57 \
    -kernel buildroot/output/images/Image \
    -append 'console=ttyAMA0,38400 keep_bootcon' \
    -initrd buildroot/output/images/rootfs.cpio.gz \
    -smp 2 \
    -m 1024 \
    -bios arm-trusted-firmware/flash.bin \
    -d unimp \
    -no-acpi

Ubuntu Cloud Image

Prepare cloudimg.

# Create a new image with 32G of space and 64M for the varstore and efi
$ qemu-img create -f qcow2 ubuntu-arm64.qcow2 32G
$ truncate -s 64m varstore.img
$ truncate -s 64m efi.img
$ dd if=/usr/share/qemu-efi-aarch64/QEMU_EFI.fd of=efi.img conv=notrunc
# Download the image from the cloud images
$ axel -n 10 https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-arm64.img
# Resize the image to 32G
$ dd if=/dev/zero of=flash0.img bs=1M count=64

# Download the cloudimg rootfs
$ wget https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-arm64-root.tar.xz
$ wget https://cloud-images.ubuntu.com/releases/22.04/release/ubuntu-22.04-server-cloudimg-arm64-root.tar.xz
$ sudo virt-customize -a rootfs.qcow2 --root-password password:coolpass
[   0.0] Examining the guest ...
[   2.6] Setting a random seed
[   2.6] Setting passwords
[   3.7] Finishing off
$ sudo virt-customize -a rootfs.qcow2 --password ubuntu:password:coolpass
[   0.0] Examining the guest ...
[   2.5] Setting a random seed
[   2.6] Setting passwords
[   3.6] Finishing off

# 屏蔽 snapd, snapd.socket, systemd-remount-fs 和 multipathd 服务
systemctl --failed
sudo systemctl mask systemd-remount-fs.service
sudo systemctl mask snapd.service
sudo systemctl mask snapd.socket
sudo systemctl mask multipathd.service
sudo systemctl mask multipathd.socket

docker build -t rootfs-to-qcow2 .

Prepare user-data for cloud-init.

# Install the cloud-image-utils
sudo apt-get install cloud-image-utils
# Create a user-data file
cat >user-data <<EOF
#cloud-config
password: asdfqwer
chpasswd: { expire: False }
ssh_pwauth: True
EOF
# Set the user-data to the image
cloud-localds user-data.img user-data

Modify password

$ sudo apt install guestfs-tools
# Resize
$ qemu-img resize jammy-server-cloudimg-arm64.img 20GB
# Change the root password
$ sudo virt-customize -a jammy-server-cloudimg-arm64.img --root-password password:coolpass
[   0.0] Examining the guest ...
[  18.8] Setting a random seed
virt-customize: warning: random seed could not be set for this type of 
guest
[  18.9] Setting the machine ID in /etc/machine-id
[  18.9] Setting passwords
[  20.8] SELinux relabelling
[  20.9] Finishing off
# Change the ubuntu user password
$ sudo virt-customize -a jammy-server-cloudimg-arm64.img --password ubuntu:password:coolpass
[   0.0] Examining the guest ...
[  13.7] Setting a random seed
[  13.8] Setting passwords
[  15.6] SELinux relabelling
[  15.6] Finishing off
# Remove the cloud-init package
$ sudo virt-customize -a jammy-server-cloudimg-arm64.img --uninstall cloud-init
[   0.0] Examining the guest ...
[  18.8] Setting a random seed
[  18.9] Running: apt-get remove -y cloud-init
# Show the image information
$ qemu-img info jammy-server-cloudimg-arm64.img 
image: jammy-server-cloudimg-arm64.img
file format: qcow2
virtual size: 20 GiB (21474836480 bytes)
disk size: 598 MiB
cluster_size: 65536
Format specific information:
    compat: 0.10
    compression type: zlib
    refcount bits: 16
Child node '/file':
    filename: jammy-server-cloudimg-arm64.img
    protocol type: file
    file length: 598 MiB (626720768 bytes)
    disk size: 598 MiB

Launch QEMU

# CCA root directory
CCA_ROOT=$(pwd)
HOME_DIR="${HOME}"

NUM_CPUS=8
MAX_RAM=4096

CPU_FEATURES="pmu=on,sve=on,sve128=on,sve256=on,neon=on"
CPU="${CPU},$CPU_FEATURES"

# QEMU command with parameters
sudo qemu-system-aarch64 \
    -smp ${NUM_CPUS} \
    -m ${MAX_RAM} \
    -cpu cortex-a57 \
    -M virt,secure=on,mte=off \
    -nographic \
    -bios arm-trusted-firmware/flash.bin  \
    -drive if=none,file="${CCA_ROOT}"/jammy-server-cloudimg-arm64.img,id=hd0 \
    -device virtio-blk-device,drive=cloud \
    -drive if=none,id=cloud,file=cloud.img,format=raw \
    -device virtio-blk-device,drive=hd0 \
    -virtfs local,path="${HOME_DIR}",mount_tag=host0,security_model=mapped,id=host0 \
    -object rng-random,filename=/dev/urandom,id=rng0 \
    -device virtio-rng-pci,rng=rng0,max-bytes=1024,period=1000 \
    -device virtio-net-device,netdev=net0 \
    -netdev user,id=net0,hostfwd=tcp::2222-:22

Ubuntu Kernel

sudo apt build-dep linux linux-image-unsigned-$(uname -r)
sudo apt install libncurses-dev gawk flex bison openssl libssl-dev dkms libelf-dev libudev-dev libpci-dev libiberty-dev autoconf llvm
sudo apt install git
apt source linux-image-unsigned-$(uname -r)

export $(dpkg-architecture -aarm64)
export CROSS_COMPILE=aarch64-linux-gnu-
export EXTRAVERSION=cca
apt source linux-image-unsigned-$(uname -r)
cd linux-5.15.0
chmod a+x debian/rules
chmod a+x debian/scripts/*
chmod a+x debian/scripts/misc/*
fakeroot debian/rules clean
fakeroot debian/rules editconfigs # you need to go through each (Y, Exit, Y, Exit..) or get a complaint about config later
fakeroot debian/rules clean
fakeroot debian/rules binary-headers binary-generic binary-perarch

virtio-fs

cd virtiofsd
cargo build --release

On host

cd linux
./scripts/config --module VIRTIO_FS

host$ ./target/release/virtiofsd --socket-path=/tmp/vhostqemu -o source=$HOME -o cache=always

On guest

guest# mount -t virtiofs myfs /mnt

References

Using Cloud Images With KVM

qemu 对 ARMv8的支持

qemu: boot aarch64, with ATF(arm trusted firmware) and EDK2 firmware

how to build ubuntu for arm64? (how to give ARCH and CROSS_COMPILE variable to debian/rules command)

QEMU_BOOT_aarch64_atf_edk2_firmware

Kernel/BuildYourOwnKernel - Ubuntu Wiki

KernelTeam/ARMKernelCrossCompile - Ubuntu Wiki

MainlineBuilds

在x64linux下编译Qemu

基于 Ubuntu Base 制作 rootfs

Building an RME stack for QEMU

Standalone virtiofs usage

Name		Name	Last commit message	Last commit date
Latest commit History 13 Commits
arm-trusted-firmware @ 4bcf5b8		arm-trusted-firmware @ 4bcf5b8
buildroot @ 6a1f9b3		buildroot @ 6a1f9b3
edk2 @ 5ff99e0		edk2 @ 5ff99e0
linux @ f443e37		linux @ f443e37
u-boot @ 6becf9b		u-boot @ 6becf9b
virtiofsd @ 2e35f12		virtiofsd @ 2e35f12
.gitignore		.gitignore
.gitmodules		.gitmodules
Dockerfile		Dockerfile
README.md		README.md
build.sh		build.sh
create_rootfs.sh		create_rootfs.sh
docker_setup.yaml		docker_setup.yaml
hosts		hosts
start-qemu.sh		start-qemu.sh
start-rootfs.sh		start-rootfs.sh
start-ubuntu.sh		start-ubuntu.sh

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

Running Ubuntu 22.04 on QEMU with ATF

Introduction

QEMU virt Armv8-A

Installation

QEMU

EDK2

buildroot

arm-trusted-firmware

Kernel

busybox

Build RootFS

Buildroot Image

Ubuntu Cloud Image

Ubuntu Kernel

virtio-fs

References

About

Releases

Packages

Languages

mbs0221-sys/cca

Folders and files

Latest commit

History

Repository files navigation

Running Ubuntu 22.04 on QEMU with ATF

Introduction

QEMU virt Armv8-A

Installation

QEMU

EDK2

buildroot

arm-trusted-firmware

Kernel

busybox

Build RootFS

Buildroot Image

Ubuntu Cloud Image

Ubuntu Kernel

virtio-fs

References

About

Resources

Stars

Watchers

Forks

Releases

Packages 0

Languages

Packages