Development environment setup using QEMU

Usually, we use QEMU with simulated NVMe device to perform development work. We use malloc0 instead of actual NVMe device in bdev.conf to run SPDK applications. Doing so allows multiple users to share the same physical server to perform development task.

Setup a QEMU with customize kernel

• Create f100M using dd if=/dev/urandom of=f100M bs=1048576 count=500

• Build kernel and make sure the following parameters are enabled in .config:

  • CONFIG_UIO_PCI_GENERIC=y (after manually set CONFIG_UIO=y, hit make will trigger the setup for CONFIG_UIO_PCI_GENERIC)
  • CONFIG_VIRTIO_PCI=y

• We start QEMU as follows:

sudo qemu-system-x86_64 -enable-kvm -curses -m 4096 -smp 4 -redir tcp:4444::22 -hda my-disk.img -hdb my-seed.img -drive file=f100M,if=none,id=D22 -device nvme,drive=D22,serial=foo -cpu host -kernel kbuild2/arch/x86_64/boot/bzImage -append "root=/dev/sda1"

• Install necessary packages in QEMU:

sudo apt-get update
sudo apt install make
sudo apt install build-essential
sudo apt install libnuma-dev
sudo apt-get install uuid-dev
sudo apt-get install libssl-dev
sudo apt-get install libaio-dev
sudo apt-get install libcunit1-dev
sudo apt install clang

• Setup SPDK as follows:

git clone https://github.com/spdk/spdk.git
cd spdk
git checkout v18.07
git submodule update --init
sudo scripts/pkgdep.sh
./configure --prefix=/home/ubuntu/spdk_install
make -j8
make install

• sudo scripts/setup.sh in spdk and run sudo ./hello_bdev under spdk/examples/bdev/hello_world/

• Optionally, you can put rustfs on the physical server and share the directory with your QEMU so that you can run your code in QEMU to make sure your change doesn't crash and run on the physical server to gather performance metrics. See ref1 and ref2 on setup directory sharing between host and guest.

We can use mutagen to synchronize files if qemu native sharing is hard to setup. The command, for example, looks like: ./mutagen create /home/zeyuanhu/share/rustfs ubuntu@localhost:4444:~/rustfs --ignore target --sync-mode=one-way-safe

• Install Rust

curl https://sh.rustup.rs -sSf | sh

Note: rustup is needed to set the environment correctly. To verify you have rustup in your environment, type rustup --version. If you already installed rust without rustup, reinstall with curl https://sh.rustup.rs -sSf | sh -s -- -y. You might need to restart the terminal to update $PATH.