diff --git a/README.md b/README.md index 5ba31ce..203aade 100644 --- a/README.md +++ b/README.md @@ -2,73 +2,160 @@ ## Setup -### Libvirt +```sh +git clone https://github.com/ultravioletrs/manager +cd manager +``` + +NB: all relative paths in this document are relative to `manager` repository directory. + +### QEMU-KVM + +[QEMU-KVM](https://www.qemu.org/) is a virtualization platform that allows you to run multiple operating systems on the same physical machine. It is a combination of two technologies: QEMU and KVM. + +- QEMU is an emulator that can run a variety of operating systems, including Linux, Windows, and macOS. +- [KVM](https://wiki.qemu.org/Features/KVM) is a Linux kernel module that allows QEMU to run virtual machines. + +To install QEMU-KVM on a Debian based machine, run ```sh sudo apt update -sudo apt install qemu-kvm libvirt-daemon-system +sudo apt install qemu-kvm ``` -After installing `libvirt-daemon-system`, the user that will be used to manage virtual machines needs to be added to the `libvirt` group. This is done automatically for members of the sudo group, otherwise +Create `img` directory in `cmd/manager`. Create `tmp` directory in `cmd/manager`. + +### focal-server-cloudimg-amd64.img + +First, we will download *focal-server-cloudimg-amd64*. It is a `qcow2` file with Ubuntu server preinstalled, ready to use with the QEMU virtual machine. ```sh -sudo adduser $USER libvirt +cd cmd/manager/img +wget https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.img +# focal-server-cloudimg-amd64 comes without the root password. +sudo apt-get install libguestfs-tools +sudo virt-customize -a focal-server-cloudimg-amd64.img --root-password password:coolpass ``` -### CD iso & hard drive img for virtual machine (VM) +### OVMF -Create `img` directory in `cmd/manager`. Create `iso` directory in `cmd/manager`. Save [alpine-virt-3.18.0-x86_64.iso](https://dl-cdn.alpinelinux.org/alpine/v3.18/releases/x86_64/alpine-virt-3.18.0-x86_64.iso) in `cmd/manager/iso` directory: +We need [Open Virtual Machine Firmware](https://wiki.ubuntu.com/UEFI/OVMF). OVMF is a port of Intel's tianocore firmware - an open source implementation of the Unified Extensible Firmware Interface (UEFI) - to the qemu virtual machine. We need OVMF in order to run virtual machine with *focal-server-cloudimg-amd64*. When we install QEMU, we get two files that we need to start a VM: `OVMF_VARS.fd` and `OVMF_CODE.fd`. We will make a local copy of `OVMF_VARS.fd` since a VM will modify this file. On the other hand, `OVMF_CODE.fd` is only used as a reference, so we only record its path in an environment variable. ```sh -cd cmd/manager/iso -wget https://dl-cdn.alpinelinux.org/alpine/v3.18/releases/x86_64/alpine-virt-3.18.0-x86_64.iso +cd cmd/manager/img + +sudo find / -name OVMF_CODE.fd +# => /usr/share/OVMF/OVMF_CODE.fd +MANAGER_QEMU_OVMF_CODE_FILE=/usr/share/OVMF/OVMF_CODE.fd + +sudo find / -name OVMF_VARS.fd +# => /usr/share/OVMF/OVMF_VARS.fd +MANAGER_QEMU_OVMF_VARS_FILE=/usr/share/OVMF/OVMF_VARS.fd ``` ## Run -We need to run `manager` in the directory where `img`, `iso` and `xml` directories are. `cd` to `cmd/manager` and run +We need to run `manager` in the directory containing `img` directory: ```sh -MANAGER_LOG_LEVEL=info MANAGER_AGENT_GRPC_URL=192.168.122.251:7002 go run main.go +cd cmd/manager +MANAGER_LOG_LEVEL=info MANAGER_AGENT_GRPC_URL=192.168.122.251:7002 MANAGER_QEMU_USE_SUDO=false MANAGER_QEMU_ENABLE_SEV=false go run main.go ``` -This will start an HTTP server on port `9021`, a gRPC server on port `7001` and will establish a connection to [libvirtd](https://libvirt.org/manpages/libvirtd.html). +To enable [AMD SEV](https://www.amd.com/en/developer/sev.html) support, start manager like this -## Domain +```sh +cd cmd/manager +MANAGER_LOG_LEVEL=info MANAGER_AGENT_GRPC_URL=192.168.122.251:7002 MANAGER_QEMU_USE_SUDO=true MANAGER_QEMU_ENABLE_SEV=true MANAGER_QEMU_SEV_CBITPOS=51 go run main.go +``` + +Manager will start an HTTP server on port `9021`, and a gRPC server on port `7001`. -### Domain creation -To create a `libvirt` domain - basically a QEMU instance or a virtual machine (VM) - run +### Create QEMU virtual machine (VM) + +To create an instance of VM, run ```sh -curl -i -X POST -H "Content-Type: application/json" localhost:9021/domain -d '{"pool":"", "volume":"", "domain":""}' +curl -sSi -X GET http://localhost:9021/qemu ``` -You can also use configuration `.xml` files in `cmd/manager/xml`. You can either specify a path to the files on your computer, or you can just leave the `pool`, `volume` and `domain` fields of the request body empty, like this +You should be able to create multiple instances by reruning the command. + +### Verifying VM launch + +NB: To verify that the manager successfully launched the VM, you need to open two terminals on the same machine. In one terminal, you need to launch `go run main.go` (with the environment variables of choice) and in the other, you can run the verification commands. + +To verify that the manager launched the VM successfully, run the following command: ```sh -curl -i -X POST -H "Content-Type: application/json" localhost:9021/domain -d '{"pool":"", "volume":"", "domain":""}' +ps aux | grep qemu-system-x86_64 ``` -### Domain destruction +You should get something similar to this +``` +darko 324763 95.3 6.0 6398136 981044 ? Sl 16:17 0:15 /usr/bin/qemu-system-x86_64 -enable-kvm -machine q35 -cpu EPYC -smp 4,maxcpus=64 -m 4096M,slots=5,maxmem=30G -drive if=pflash,format=raw,unit=0,file=/usr/share/OVMF/OVMF_CODE.fd,readonly=on -drive if=pflash,format=raw,unit=1,file=img/OVMF_VARS.fd -device virtio-scsi-pci,id=scsi,disable-legacy=on,iommu_platform=true -drive file=img/focal-server-cloudimg-amd64.img,if=none,id=disk0,format=qcow2 -device scsi-hd,drive=disk0 -netdev user,id=vmnic,hostfwd=tcp::2222-:22,hostfwd=tcp::9301-:9031,hostfwd=tcp::7020-:7002 -device virtio-net-pci,disable-legacy=on,iommu_platform=true,netdev=vmnic,romfile= -nographic -monitor pty +``` -Normally, you don't need to do anything manually. However, if need be, if you have already created a domain, you can remove it with +If you run a command as `sudo`, you should get the output similar to this one + +``` +root 37982 0.0 0.0 9444 4572 pts/0 S+ 16:18 0:00 sudo /usr/local/bin/qemu-system-x86_64 -enable-kvm -machine q35 -cpu EPYC -smp 4,maxcpus=64 -m 4096M,slots=5,maxmem=30G -drive if=pflash,format=raw,unit=0,file=/usr/share/OVMF/OVMF_CODE.fd,readonly=on -drive if=pflash,format=raw,unit=1,file=img/OVMF_VARS.fd -device virtio-scsi-pci,id=scsi,disable-legacy=on,iommu_platform=true -drive file=img/focal-server-cloudimg-amd64.img,if=none,id=disk0,format=qcow2 -device scsi-hd,drive=disk0 -netdev user,id=vmnic,hostfwd=tcp::2222-:22,hostfwd=tcp::9301-:9031,hostfwd=tcp::7020-:7002 -device virtio-net-pci,disable-legacy=on,iommu_platform=true,netdev=vmnic,romfile= -object sev-guest,id=sev0,cbitpos=51,reduced-phys-bits=1 -machine memory-encryption=sev0 -nographic -monitor pty +root 37989 122 13.1 5345816 4252312 pts/0 Sl+ 16:19 0:04 /usr/local/bin/qemu-system-x86_64 -enable-kvm -machine q35 -cpu EPYC -smp 4,maxcpus=64 -m 4096M,slots=5,maxmem=30G -drive if=pflash,format=raw,unit=0,file=/usr/share/OVMF/OVMF_CODE.fd,readonly=on -drive if=pflash,format=raw,unit=1,file=img/OVMF_VARS.fd -device virtio-scsi-pci,id=scsi,disable-legacy=on,iommu_platform=true -drive file=img/focal-server-cloudimg-amd64.img,if=none,id=disk0,format=qcow2 -device scsi-hd,drive=disk0 -netdev user,id=vmnic,hostfwd=tcp::2222-:22,hostfwd=tcp::9301-:9031,hostfwd=tcp::7020-:7002 -device virtio-net-pci,disable-legacy=on,iommu_platform=true,netdev=vmnic,romfile= -object sev-guest,id=sev0,cbitpos=51,reduced-phys-bits=1 -machine memory-encryption=sev0 -nographic -monitor pty +``` + +The two processes are due to the fact that we run the command `/usr/bin/qemu-system-x86_64` as `sudo`, so there is one process for `sudo` command and the other for `/usr/bin/qemu-system-x86_64`. + +### Troubleshooting + +If the `ps aux | grep qemu-system-x86_64` give you something like this + +``` +darko 13913 0.0 0.0 0 0 pts/2 Z+ 20:17 0:00 [qemu-system-x86] +``` + +means that the a QEMU virtual machine that is currently defunct, meaning that it is no longer running. More precisely, the defunct process in the output is also known as a ["zombie" process](https://en.wikipedia.org/wiki/Zombie_process). + +You can troubleshoot the VM launch procedure by running directly `qemu-system-x86_64` command. When you run `manager` with `MANAGER_LOG_LEVEL=info` env var set, it prints out the entire command used to launch a VM. The relevant part of the log might look like this + +``` +{"level":"info","message":"/usr/bin/qemu-system-x86_64 -enable-kvm -machine q35 -cpu EPYC -smp 4,maxcpus=64 -m 4096M,slots=5,maxmem=30G -drive if=pflash,format=raw,unit=0,file=/usr/share/OVMF/OVMF_CODE.fd,readonly=on -drive if=pflash,format=raw,unit=1,file=img/OVMF_VARS.fd -device virtio-scsi-pci,id=scsi,disable-legacy=on,iommu_platform=true -drive file=img/focal-server-cloudimg-amd64.img,if=none,id=disk0,format=qcow2 -device scsi-hd,drive=disk0 -netdev user,id=vmnic,hostfwd=tcp::2222-:22,hostfwd=tcp::9301-:9031,hostfwd=tcp::7020-:7002 -device virtio-net-pci,disable-legacy=on,iommu_platform=true,netdev=vmnic,romfile= -nographic -monitor pty","ts":"2023-08-14T18:29:19.2653908Z"} +``` + +You can run the command - the value of the `"message"` key - directly in the terminal: ```sh -virsh undefine QEmu-alpine-standard-x86_64; \ -virsh shutdown QEmu-alpine-standard-x86_64; \ -virsh destroy QEmu-alpine-standard-x86_64; \ -rm -rf ~/go/src/github.com/ultravioletrs/manager/cmd/manager/img/boot.img; \ -virsh pool-destroy --pool virtimages +/usr/bin/qemu-system-x86_64 -enable-kvm -machine q35 -cpu EPYC -smp 4,maxcpus=64 -m 4096M,slots=5,maxmem=30G -drive if=pflash,format=raw,unit=0,file=/usr/share/OVMF/OVMF_CODE.fd,readonly=on -drive if=pflash,format=raw,unit=1,file=img/OVMF_VARS.fd -device virtio-scsi-pci,id=scsi,disable-legacy=on,iommu_platform=true -drive file=img/focal-server-cloudimg-amd64.img,if=none,id=disk0,format=qcow2 -device scsi-hd,drive=disk0 -netdev user,id=vmnic,hostfwd=tcp::2222-:22,hostfwd=tcp::9301-:9031,hostfwd=tcp::7020-:7002 -device virtio-net-pci,disable-legacy=on,iommu_platform=true,netdev=vmnic,romfile= -nographic -monitor pty ``` -This will destroy the domain together with volumes and a pool where the volume was logically stored. It is not necessary to remove a domain, since the manager will reuse the existing VM. +and look for the possible problems. This problems can usually be solved by using the adequate env var assignments. Look in the `manager/qemu/config.go` file to see the recognized env vars. Don't forget to prepend `MANAGER_QEMU_` to the name of the env vars. + +#### Kill `qemu-system-x86_64` processes -### Domain management +To kill any leftover `qemu-system-x86_64` processes, use ```sh -sudo apt-get install virt-manager +pkill -f qemu-system-x86_64 ``` -Start virtual manager. Open `QEmu-alpine-standard-x86_64` virtual machine. Log in as root, no password needed, and follow instructions to install and set up Alpine Linux on virtual drive. Basically, you need to run `setup-alpine` script. Use `vda` as a virtual disk drive and `sys` to install Alpine Linux on the virtual disk drive. +The pkill command is used to kill processes by name or by pattern. The -f flag to specify that we want to kill processes that match the pattern `qemu-system-x86_64`. It sends the SIGKILL signal to all processes that are running `qemu-system-x86_64`. -Once you have installed and set up Alpine Linux, follow the instructions in `Agent` [README.md](https://github.com/ultravioletrs/agent) in order to see how to set up `Cocos.ai` `Agent` in the virtual machine. \ No newline at end of file +If this does not work, i.e. if `ps aux | grep qemu-system-x86_64` still outputs `qemu-system-x86_64` related process(es), you can kill the unwanted process with `kill -9 `, which also sends a SIGKILL signal to the process. + +#### Ports in use + +The [NetDevConfig struct](manager/qemu/config.go) defines the network configuration for a virtual machine. The HostFwd* and GuestFwd* fields specify the host and guest ports that are forwarded between the virtual machine and the host machine. By default, these ports are allocated 2222, 9301, and 7020 for HostFwd1, HostFwd2, and HostFwd3, respectively, and 22, 9031, and 7002 for GuestFwd1, GuestFwd2, and GuestFwd3, respectively. However, if these ports are in use, you can configure your own ports by setting the corresponding environment variables. For example, to set the HostFwd1 port to 8080, you would set the MANAGER_QEMU_HOST_FWD_1 environment variable to 8080. For example, + +```sh +export MANAGER_LOG_LEVEL=info +export MANAGER_AGENT_GRPC_URL=192.168.122.251:7002 +export MANAGER_QEMU_USE_SUDO=false +export MANAGER_QEMU_ENABLE_SEV=false +export MANAGER_QEMU_HOST_FWD_1=8080 +export MANAGER_QEMU_GUEST_FWD_1=22 +export MANAGER_QEMU_HOST_FWD_2=9301 +export MANAGER_QEMU_GUEST_FWD_2=9031 +export MANAGER_QEMU_HOST_FWD_3=7020 +export MANAGER_QEMU_GUEST_FWD_3=7002 + +go run main.go +``` diff --git a/cmd/manager/main.go b/cmd/manager/main.go index 56dec7a..5ee2f8d 100644 --- a/cmd/manager/main.go +++ b/cmd/manager/main.go @@ -30,6 +30,7 @@ import ( "github.com/ultravioletrs/manager/manager/api" managergrpc "github.com/ultravioletrs/manager/manager/api/grpc" httpapi "github.com/ultravioletrs/manager/manager/api/http" + "github.com/ultravioletrs/manager/manager/qemu" "github.com/ultravioletrs/manager/manager/tracing" "go.opentelemetry.io/otel/trace" "golang.org/x/sync/errgroup" @@ -42,6 +43,7 @@ const ( envPrefixHTTP = "MANAGER_HTTP_" envPrefixGRPC = "MANAGER_GRPC_" envPrefixAgentGRPC = "AGENT_GRPC_" + envPrefixQemu = "MANAGER_QEMU_" defSvcGRPCPort = "7001" defSvcHTTPPort = "9021" ) @@ -103,7 +105,12 @@ func main() { logger.Info("Successfully connected to agent grpc server " + agentGRPCClient.Secure()) - svc := newService(libvirtConn, agentClient, logger, tracer) + qemuCfg := qemu.Config{} + if err := env.Parse(&qemuCfg, env.Options{Prefix: envPrefixQemu}); err != nil { + logger.Fatal(fmt.Sprintf("failed to load %s QEMU configuration : %s", svcName, err)) + } + + svc := newService(libvirtConn, agentClient, logger, tracer, qemuCfg) var httpServerConfig = server.Config{Port: defSvcHTTPPort} if err := env.Parse(&httpServerConfig, env.Options{Prefix: envPrefixHTTP}); err != nil { @@ -136,10 +143,15 @@ func main() { if err := g.Wait(); err != nil { logger.Error(fmt.Sprintf("%s service terminated: %s", svcName, err)) } + + err = internal.DeleteFilesInDir(qemuCfg.TmpFileLoc) + if err != nil { + logger.Error(err.Error()) + } } -func newService(libvirtConn *libvirt.Libvirt, agent agent.AgentServiceClient, logger logger.Logger, tracer trace.Tracer) manager.Service { - svc := manager.New(libvirtConn, agent) +func newService(libvirtConn *libvirt.Libvirt, agent agent.AgentServiceClient, logger logger.Logger, tracer trace.Tracer, qemuCfg qemu.Config) manager.Service { + svc := manager.New(libvirtConn, agent, qemuCfg) svc = api.LoggingMiddleware(svc, logger) counter, latency := internal.MakeMetrics(svcName, "api") diff --git a/cmd/manager/script/launch-qemu.sh b/cmd/manager/script/launch-qemu.sh new file mode 100755 index 0000000..06b83b8 --- /dev/null +++ b/cmd/manager/script/launch-qemu.sh @@ -0,0 +1,270 @@ +#!/bin/bash + +# +# user changeable parameters +# + +HDA_FILE="cmd/manager/img/focal-server-cloudimg-amd64.qcow2" +GUEST_SIZE_IN_MB="4096" +SEV_GUEST="1" +SMP_NCPUS="4" +CONSOLE="serial" +VNC_PORT="" +USE_VIRTIO="1" + +UEFI_BIOS_CODE="/usr/share/OVMF/OVMF_CODE.fd" +UEFI_BIOS_VARS_ORIG="/usr/share/OVMF/OVMF_VARS.fd" +UEFI_BIOS_VARS_COPY="cmd/manager/img/OVMF_VARS.fd" + +CBITPOS=51 +HOST_HTTP_PORT=9301 +GUEST_HTTP_PORT=9031 +HOST_GRPC_PORT=7020 +GUEST_GRPC_PORT=7002 + +ENABLE_FILE_LOG="0" +EXEC_QEMU_CMDLINE="0" + +usage() { + echo "$0 [options]" + echo "Available :" + echo " -hda hard disk ($HDA_FILE)" + echo " -nosev disable sev support" + echo " -mem guest memory" + echo " -smp number of cpus" + echo " -console display console to use (serial or gxl)" + echo " -vnc VNC port to use" + echo " -bios bios to use (default $UEFI_BIOS_CODE)" + echo " -kernel kernel to use" + echo " -initrd initrd to use" + echo " -cdrom CDROM image" + echo " -virtio use virtio devices" + echo " -cbitpos location of the C-bit" + echo " -hosthttp host http port" + echo " -guesthttp guest http port" + echo " -hostgrpc host grpc port" + echo " -guestgrpc guest grpc port" + echo " -origuefivars UEFI BIOS vars original file (default $UEFI_BIOS_VARS_ORIG)" + echo " -copyuefivars UEFI BIOS vars copy file (default $UEFI_BIOS_VARS_COPY)" + echo " -exec execute the QEMU command (default $EXEC_QEMU_CMDLINE)" + echo " -filelog enable/disable QEMU cmd line file log (default: $ENABLE_FILE_LOG)" + exit 1 +} + +while [[ $1 != "" ]]; do + case "$1" in + -hda) + HDA_FILE=${2} + shift + ;; + -nosev) + SEV_GUEST="0" + ;; + -mem) + GUEST_SIZE_IN_MB=${2} + shift + ;; + -console) + CONSOLE=${2} + shift + ;; + -smp) + SMP_NCPUS=$2 + shift + ;; + -vnc) + VNC_PORT=$2 + shift + ;; + -bios) + UEFI_BIOS_CODE=$2 + shift + ;; + -initrd) + INITRD_FILE=$2 + shift + ;; + -kernel) + KERNEL_FILE=$2 + shift + ;; + -cdrom) + CDROM_FILE=$2 + shift + ;; + -virtio) + USE_VIRTIO="1" + ;; + -cbitpos) + CBITPOS=$2 + shift + ;; + -hosthttp) + HOST_HTTP_PORT=$2 + shift + ;; + -guesthttp) + GUEST_HTTP_PORT=$2 + shift + ;; + -guestgrpc) + GUEST_GRPC_PORT=$2 + shift + ;; + -hostgrpc) + HOST_GRPC_PORT=$2 + shift + ;; + -origuefivars) + UEFI_BIOS_VARS_ORIG=$2 + shift + ;; + -copyuefivars) + UEFI_BIOS_VARS_COPY=$2 + shift + ;; + -exec) + EXEC_QEMU_CMDLINE="1" + ;; + -filelog) + ENABLE_FILE_LOG="1" + ;; + *) + usage;; + esac + shift +done + +# +# func definitions +# + +add_opts() { + echo -n "$* " >> ${QEMU_CMDLINE} +} + +run_cmd() { + if ! "$@"; then + echo "Command '$*' failed" + exit 1 + fi +} + +# copy BIOS variables to new dest for VM use without modifying the original ones +cp "$UEFI_BIOS_VARS_ORIG" "$UEFI_BIOS_VARS_COPY" + +# +# Qemu cmd line construction +# + +# we add all the qemu command line options into a file +QEMU_CMDLINE=/tmp/cmdline.$$ +rm -rf ${QEMU_CMDLINE} + +add_opts "$(which qemu-system-x86_64)" + +# Basic virtual machine property +add_opts "-enable-kvm -cpu EPYC -machine q35" + +# add number of VCPUs +[ -n "$SMP_NCPUS" ] && add_opts "-smp ${SMP_NCPUS},maxcpus=64" + +# define guest memory +add_opts "-m ${GUEST_SIZE_IN_MB}M,slots=5,maxmem=30G" + +# The OVMF binary, including the non-volatile variable store, appears as a +# "normal" qemu drive on the host side, and it is exposed to the guest as a +# persistent flash device. +add_opts "-drive if=pflash,format=raw,unit=0,file=${UEFI_BIOS_CODE},readonly=on" +add_opts "-drive if=pflash,format=raw,unit=1,file=${UEFI_BIOS_VARS_COPY}" + +# add CDROM if specified +[ -n "$CDROM_FILE" ] && add_opts "-drive file=${CDROM_FILE},media=cdrom -boot d" + +add_opts "-netdev user,id=vmnic,hostfwd=tcp::2222-:22,hostfwd=tcp::$HOST_HTTP_PORT-:$GUEST_HTTP_PORT,hostfwd=tcp::$HOST_GRPC_PORT-:$GUEST_GRPC_PORT" +add_opts "-device virtio-net-pci,disable-legacy=on,iommu_platform=true,netdev=vmnic,romfile=" + +# If harddisk file is specified then add the HDD drive +if [ -n "$HDA_FILE" ]; then + if [ "$USE_VIRTIO" = "1" ]; then + if [[ ${HDA_FILE} = *"qcow2" ]]; then + add_opts "-drive file=${HDA_FILE},if=none,id=disk0,format=qcow2" + else + add_opts "-drive file=${HDA_FILE},if=none,id=disk0,format=raw" + fi + add_opts "-device virtio-scsi-pci,id=scsi,disable-legacy=on,iommu_platform=true" + add_opts "-device scsi-hd,drive=disk0" + else + if [[ ${HDA_FILE} = *"qcow2" ]]; then + add_opts "-drive file=${HDA_FILE},format=qcow2" + else + add_opts "-drive file=${HDA_FILE},format=raw" + fi + fi +fi + +# If this is SEV guest then add the encryption device objects to enable support +if [ ${SEV_GUEST} = "1" ]; then + add_opts "-object sev-guest,id=sev0,cbitpos=${CBITPOS},reduced-phys-bits=1" + add_opts "-machine memory-encryption=sev0" +fi + +# if console is serial then disable graphical interface +if [ "${CONSOLE}" = "serial" ]; then + add_opts "-nographic" +else + add_opts "-vga ${CONSOLE}" +fi + +# if -kernel arg is specified then use the kernel provided in command line for boot +if [ "${KERNEL_FILE}" != "" ]; then + add_opts "-kernel $KERNEL_FILE" + add_opts "-append \"console=ttyS0 earlyprintk=serial root=/dev/sda2\"" + [ -n "$INITRD_FILE" ] && add_opts "-initrd ${INITRD_FILE}" +fi + +# start vnc server +[ -n "$VNC_PORT" ] && add_opts "-vnc :${VNC_PORT}" && echo "Starting VNC on port ${VNC_PORT}" + +# start monitor on pty +add_opts "-monitor pty" + +# +# Qemu cmd line log +# + +# Set the log file path if ENABLE_FILE_LOG is 1 +if [ "$ENABLE_FILE_LOG" = "1" ]; then + LOG_FILE=$(pwd)/stdout.log + + # Save the command line args into log file + cat "$QEMU_CMDLINE" > "$LOG_FILE" + echo >> "$LOG_FILE" +fi + + # Log the command line to the console +cat "$QEMU_CMDLINE" + +# +# Qemu cmd line execution +# + +if [[ "${EXEC_QEMU_CMDLINE}" = "0" ]]; then + exit 0 +fi + +# map CTRL-C to CTRL ] +echo "Mapping CTRL-C to CTRL-]" +stty intr ^] + +echo "Launching VM ..." +if [ "$ENABLE_FILE_LOG" = "1" ]; then + bash ${QEMU_CMDLINE} 2>&1 | tee -a "${LOG_FILE}" +else + bash ${QEMU_CMDLINE} 2>&1 +fi + +# restore the mapping +stty intr ^c + +rm -rf ${QEMU_CMDLINE} diff --git a/cmd/manager/xml/dom.xml b/cmd/manager/xml/dom.xml index 8fd5546..2cccad4 100644 --- a/cmd/manager/xml/dom.xml +++ b/cmd/manager/xml/dom.xml @@ -6,13 +6,15 @@ - 786432 - 786432 + 4194304 + 4194304 1 - hvm - + hvm + /usr/share/OVMF/OVMF_CODE.fd + ./img/OVMF_VARS.fd + @@ -36,20 +38,11 @@ /usr/bin/qemu-system-x86_64 - +
- - - - - - -
- - diff --git a/go.mod b/go.mod index 5de5cdf..e5f3ce9 100644 --- a/go.mod +++ b/go.mod @@ -7,6 +7,7 @@ require ( github.com/digitalocean/go-libvirt v0.0.0-20221205150000-2939327a8519 github.com/go-kit/kit v0.12.0 github.com/go-zoo/bone v1.3.0 + github.com/google/uuid v1.3.0 github.com/mainflux/mainflux v0.0.0-20230726142711-2b78902e0170 github.com/prometheus/client_golang v1.16.0 github.com/ultravioletrs/agent v0.0.0-20230727102942-c2240066f943 diff --git a/go.sum b/go.sum index b53ebbf..d8f160a 100644 --- a/go.sum +++ b/go.sum @@ -37,6 +37,8 @@ github.com/golang/protobuf v1.5.3 h1:KhyjKVUg7Usr/dYsdSqoFveMYd5ko72D+zANwlG1mmg github.com/golang/protobuf v1.5.3/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY= github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= github.com/google/go-cmp v0.5.9 h1:O2Tfq5qg4qc4AmwVlvv0oLiVAGB7enBSJ2x2DqQFi38= +github.com/google/uuid v1.3.0 h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I= +github.com/google/uuid v1.3.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/mainflux/mainflux v0.0.0-20230726142711-2b78902e0170 h1:lMvLFbtNJdUufGGYaC/MrNH+B1uEYs4LWstTjphWnbc= github.com/mainflux/mainflux v0.0.0-20230726142711-2b78902e0170/go.mod h1:FoeJ13mrfikrsFDW6bOb3C44D5gZ5m9Jt249G1sLKq0= github.com/matttproud/golang_protobuf_extensions v1.0.4 h1:mmDVorXM7PCGKw94cs5zkfA9PSy5pEvNWRP0ET0TIVo= @@ -63,6 +65,7 @@ github.com/subosito/gotenv v1.4.2 h1:X1TuBLAMDFbaTAChgCBLu3DU3UPyELpnF2jjJ2cz/S8 github.com/subosito/gotenv v1.4.2/go.mod h1:ayKnFf/c6rvx/2iiLrJUk1e6plDbT3edrFNGqEflhK0= github.com/ultravioletrs/agent v0.0.0-20230727102942-c2240066f943 h1:t7TZWtdH5PCMZydP8s/6W877fT5QtUF8C8kTayOOHac= github.com/ultravioletrs/agent v0.0.0-20230727102942-c2240066f943/go.mod h1:AnzhybLxmQwV3ZfVTgM6W+rvIZLfHeNVmwt7tAvAQhw= +github.com/ultravioletrs/agent v0.0.0-20230808151319-fd68c9712b82/go.mod h1:AnzhybLxmQwV3ZfVTgM6W+rvIZLfHeNVmwt7tAvAQhw= github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY= go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.42.0 h1:ZOLJc06r4CB42laIXg/7udr0pbZyuAihN10A/XuiQRY= go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.42.0/go.mod h1:5z+/ZWJQKXa9YT34fQNx5K8Hd1EoIhvtUygUQPqEOgQ= diff --git a/internal/cmd.go b/internal/cmd.go new file mode 100644 index 0000000..1f88528 --- /dev/null +++ b/internal/cmd.go @@ -0,0 +1,74 @@ +package internal + +import ( + "bytes" + "fmt" + "io" + "os" + "os/exec" + "strings" +) + +// ExeShCmdStdout executes a shell command capturing the standard output. +func ExeShCmdStdout(command string, args ...string) (string, error) { + var stdoutBuf, stderrBuf bytes.Buffer + + cmd := exec.Command(command, args...) + + // Capture stdout and stderr using buffers + cmd.Stdout = io.MultiWriter(&stdoutBuf, os.Stdout) + cmd.Stderr = io.MultiWriter(&stderrBuf, os.Stderr) + + err := cmd.Run() + if err != nil { + return "", fmt.Errorf("error executing command '%s': %s", cmd.String(), err) + } + + return stdoutBuf.String(), nil +} + +// ExtractCmdAndArgs extracts the command and its arguments from the output string. +func ExtractCmdAndArgs(cmdLine string, sudo bool) (string, []string) { + lines := strings.Split(cmdLine, "\n") + if len(lines) == 0 { + return "", nil + } + + parts := strings.Fields(lines[0]) + if len(parts) == 0 { + return "", nil + } + + if sudo { + parts = append([]string{"sudo"}, parts...) + } + + cmd := parts[0] + args := parts[1:] + + return cmd, args +} + +// RunCmdOutput runs the specified command and returns its standard output as a string. +func RunCmdOutput(command string, args ...string) (string, error) { + cmd := exec.Command(command, args...) + + output, err := cmd.Output() + if err != nil { + return "", fmt.Errorf("error executing command '%s': %s", cmd.String(), err) + } + + return string(output), nil +} + +// RunCmdStart starts the specified command and returns the *exec.Cmd for the running process. +func RunCmdStart(command string, args ...string) (*exec.Cmd, error) { + cmd := exec.Command(command, args...) + + err := cmd.Start() + if err != nil { + return nil, fmt.Errorf("error starting command '%s': %s", cmd.String(), err) + } + + return cmd, nil +} diff --git a/internal/file.go b/internal/file.go new file mode 100644 index 0000000..ec70524 --- /dev/null +++ b/internal/file.go @@ -0,0 +1,46 @@ +package internal + +import ( + "io" + "os" + "path/filepath" +) + +// CopyFile copies a file from srcPath to dstPath. +func CopyFile(srcPath, dstPath string) error { + src, err := os.Open(srcPath) + if err != nil { + return err + } + defer src.Close() + + dst, err := os.Create(dstPath) + if err != nil { + return err + } + defer dst.Close() + + _, err = io.Copy(dst, src) + if err != nil { + return err + } + + return nil +} + +// DeleteFilesInDir deletes all files in the directory dirPath. +func DeleteFilesInDir(dirPath string) error { + files, err := filepath.Glob(filepath.Join(dirPath, "*")) + if err != nil { + return err + } + + for _, file := range files { + err := os.Remove(file) + if err != nil { + return err + } + } + + return nil +} diff --git a/manager/api/grpc/endpoint.go b/manager/api/grpc/endpoint.go index 020857b..fa12bd8 100644 --- a/manager/api/grpc/endpoint.go +++ b/manager/api/grpc/endpoint.go @@ -15,7 +15,7 @@ func createDomainEndpoint(svc manager.Service) endpoint.Endpoint { return createDomainRes{}, err } - name, err := svc.CreateDomain(ctx, req.Pool, req.Volume, req.Domain) + name, err := svc.CreateLibvirtDomain(ctx, req.Pool, req.Volume, req.Domain) if err != nil { return createDomainRes{}, err } diff --git a/manager/api/http/endpoint.go b/manager/api/http/endpoint.go index 7492d67..e56cd92 100644 --- a/manager/api/http/endpoint.go +++ b/manager/api/http/endpoint.go @@ -5,25 +5,26 @@ package http import ( "context" + "strings" "github.com/go-kit/kit/endpoint" "github.com/ultravioletrs/manager/manager" ) -func createDomainEndpoint(svc manager.Service) endpoint.Endpoint { +func createLibvirtDomainEndpoint(svc manager.Service) endpoint.Endpoint { return func(ctx context.Context, request interface{}) (interface{}, error) { - req := request.(createDomainReq) + req := request.(createLibvirtDomainReq) if err := req.validate(); err != nil { return nil, err } - name, err := svc.CreateDomain(ctx, req.Pool, req.Volume, req.Domain) + name, err := svc.CreateLibvirtDomain(ctx, req.Pool, req.Volume, req.Domain) if err != nil { return nil, err } - res := createDomainRes{ + res := createLibvirtDomainRes{ Name: name, } @@ -31,6 +32,25 @@ func createDomainEndpoint(svc manager.Service) endpoint.Endpoint { } } +func createQemuVMEndpoint(svc manager.Service) endpoint.Endpoint { + return func(ctx context.Context, request interface{}) (interface{}, error) { + req := request.(createQemuVMReq) + + if err := req.validate(); err != nil { + return createQemuVMReq{}, err + } + + cmd, err := svc.CreateQemuVM(ctx) + if err != nil { + return createQemuVMRes{}, err + } + + return createQemuVMRes{ + Path: cmd.Path, + Args: strings.Join(cmd.Args, " "), + }, nil + } +} func runEndpoint(svc manager.Service) endpoint.Endpoint { return func(ctx context.Context, request interface{}) (interface{}, error) { req := request.(runReq) diff --git a/manager/api/http/requests.go b/manager/api/http/requests.go index adbd615..a4cae21 100644 --- a/manager/api/http/requests.go +++ b/manager/api/http/requests.go @@ -7,23 +7,20 @@ import "github.com/ultravioletrs/manager/manager" var ( _ apiReq = (*runReq)(nil) - _ apiReq = (*createDomainReq)(nil) + _ apiReq = (*createLibvirtDomainReq)(nil) ) type apiReq interface { validate() error } -type createDomainReq struct { + +type createLibvirtDomainReq struct { Pool string `json:"pool"` Volume string `json:"volume"` Domain string `json:"domain"` } -func (req createDomainReq) validate() error { - // if req.Pool == "" || req.Volume == "" || req.Domain == "" { - // return manager.ErrMalformedEntity - // } - +func (req createLibvirtDomainReq) validate() error { return nil } @@ -37,3 +34,10 @@ func (req runReq) validate() error { } return nil } + +type createQemuVMReq struct { +} + +func (req createQemuVMReq) validate() error { + return nil +} diff --git a/manager/api/http/responses.go b/manager/api/http/responses.go index d37eace..b4f8427 100644 --- a/manager/api/http/responses.go +++ b/manager/api/http/responses.go @@ -10,23 +10,40 @@ import ( ) var ( - _ mainflux.Response = (*createDomainRes)(nil) + _ mainflux.Response = (*createLibvirtDomainRes)(nil) _ mainflux.Response = (*runRes)(nil) ) -type createDomainRes struct { +type createLibvirtDomainRes struct { Name string `json:"name"` } -func (res createDomainRes) Code() int { +func (res createLibvirtDomainRes) Code() int { return http.StatusOK } -func (res createDomainRes) Headers() map[string]string { +func (res createLibvirtDomainRes) Headers() map[string]string { return map[string]string{} } -func (res createDomainRes) Empty() bool { +func (res createLibvirtDomainRes) Empty() bool { + return false +} + +type createQemuVMRes struct { + Path string `json:"path"` + Args string `json:"args"` +} + +func (res createQemuVMRes) Code() int { + return http.StatusOK +} + +func (res createQemuVMRes) Headers() map[string]string { + return map[string]string{} +} + +func (res createQemuVMRes) Empty() bool { return false } diff --git a/manager/api/http/transport.go b/manager/api/http/transport.go index 1202445..bdfbd1d 100644 --- a/manager/api/http/transport.go +++ b/manager/api/http/transport.go @@ -35,12 +35,19 @@ func MakeHandler(svc manager.Service, instanceID string) http.Handler { r := bone.New() r.Post("/domain", otelhttp.NewHandler(kithttp.NewServer( - createDomainEndpoint(svc), - decodeCreateDomain, + createLibvirtDomainEndpoint(svc), + decodeCreateLibvirtDomain, encodeResponse, opts..., ), "create_domain")) + r.Get("/qemu", otelhttp.NewHandler(kithttp.NewServer( + createQemuVMEndpoint(svc), + decodeCreateQemuVMRequest, + encodeResponse, + opts..., + ), "create_qemu_vm")) + r.Post("/run", otelhttp.NewHandler(kithttp.NewServer( runEndpoint(svc), decodeRun, @@ -54,12 +61,12 @@ func MakeHandler(svc manager.Service, instanceID string) http.Handler { return r } -func decodeCreateDomain(_ context.Context, r *http.Request) (interface{}, error) { +func decodeCreateLibvirtDomain(_ context.Context, r *http.Request) (interface{}, error) { if !strings.Contains(r.Header.Get("Content-Type"), contentType) { return nil, errUnsupportedContentType } - req := createDomainReq{} + req := createLibvirtDomainReq{} if err := json.NewDecoder(r.Body).Decode(&req); err != nil { return nil, err } @@ -67,6 +74,10 @@ func decodeCreateDomain(_ context.Context, r *http.Request) (interface{}, error) return req, nil } +func decodeCreateQemuVMRequest(_ context.Context, r *http.Request) (interface{}, error) { + return createQemuVMReq{}, nil +} + func decodeRun(_ context.Context, r *http.Request) (interface{}, error) { if !strings.Contains(r.Header.Get("Content-Type"), contentType) { return nil, errUnsupportedContentType diff --git a/manager/api/logging.go b/manager/api/logging.go index bf3fec8..c957e8b 100644 --- a/manager/api/logging.go +++ b/manager/api/logging.go @@ -9,6 +9,7 @@ package api import ( "context" "fmt" + "os/exec" "time" log "github.com/mainflux/mainflux/logger" @@ -27,7 +28,7 @@ func LoggingMiddleware(svc manager.Service, logger log.Logger) manager.Service { return &loggingMiddleware{logger, svc} } -func (lm *loggingMiddleware) CreateDomain(ctx context.Context, pool, volume, domain string) (response string, err error) { +func (lm *loggingMiddleware) CreateLibvirtDomain(ctx context.Context, pool, volume, domain string) (response string, err error) { defer func(begin time.Time) { message := fmt.Sprintf("Method CreateDomain for pool %s, volume %s, and domain %s took %s to complete", pool, volume, domain, time.Since(begin)) @@ -38,7 +39,20 @@ func (lm *loggingMiddleware) CreateDomain(ctx context.Context, pool, volume, dom lm.logger.Info(fmt.Sprintf("%s without errors.", message)) }(time.Now()) - return lm.svc.CreateDomain(ctx, pool, volume, domain) + return lm.svc.CreateLibvirtDomain(ctx, pool, volume, domain) +} + +func (lm *loggingMiddleware) CreateQemuVM(ctx context.Context) (cmd *exec.Cmd, err error) { + defer func(begin time.Time) { + message := fmt.Sprintf("Method CreateQemuVM took %s to complete", time.Since(begin)) + if err != nil { + lm.logger.Warn(fmt.Sprintf("%s with error: %s.", message, err)) + return + } + lm.logger.Info(fmt.Sprintf("%s without errors.", message)) + }(time.Now()) + + return lm.svc.CreateQemuVM(ctx) } func (lm *loggingMiddleware) Run(ctx context.Context, computation []byte) (id string, err error) { diff --git a/manager/api/metrics.go b/manager/api/metrics.go index afd8272..76528d2 100644 --- a/manager/api/metrics.go +++ b/manager/api/metrics.go @@ -8,6 +8,7 @@ package api import ( "context" + "os/exec" "time" "github.com/go-kit/kit/metrics" @@ -32,13 +33,22 @@ func MetricsMiddleware(svc manager.Service, counter metrics.Counter, latency met } } -func (ms *metricsMiddleware) CreateDomain(ctx context.Context, pool, volume, domain string) (response string, err error) { +func (ms *metricsMiddleware) CreateLibvirtDomain(ctx context.Context, pool, volume, domain string) (response string, err error) { defer func(begin time.Time) { ms.counter.With("method", "CreateDomain").Add(1) ms.latency.With("method", "CreateDomain").Observe(time.Since(begin).Seconds()) }(time.Now()) - return ms.svc.CreateDomain(ctx, pool, volume, domain) + return ms.svc.CreateLibvirtDomain(ctx, pool, volume, domain) +} + +func (ms *metricsMiddleware) CreateQemuVM(ctx context.Context) (*exec.Cmd, error) { + defer func(begin time.Time) { + ms.counter.With("method", "CreateQemuVM").Add(1) + ms.latency.With("method", "CreateQemuVM").Observe(time.Since(begin).Seconds()) + }(time.Now()) + + return ms.svc.CreateQemuVM(ctx) } func (ms *metricsMiddleware) Run(ctx context.Context, computation []byte) (string, error) { diff --git a/manager/libvirt.go b/manager/libvirt.go index 576ee59..86c8eb5 100644 --- a/manager/libvirt.go +++ b/manager/libvirt.go @@ -9,7 +9,7 @@ import ( var re = regexp.MustCompile(`'([^']*)'`) -const bootTime = 12 * time.Second +const bootTime = 5 * time.Second func entityName(msg string) (string, error) { match := re.FindStringSubmatch(msg) @@ -22,6 +22,7 @@ func entityName(msg string) (string, error) { func createDomain(libvirtConn *libvirt.Libvirt, poolXML string, volXML string, domXML string) (libvirt.Domain, error) { pool, err := libvirtConn.StoragePoolCreateXML(poolXML, 0) + _ = pool if err != nil { lvErr := err.(libvirt.Error) if lvErr.Code == 9 { diff --git a/manager/qemu/config.go b/manager/qemu/config.go new file mode 100644 index 0000000..7737969 --- /dev/null +++ b/manager/qemu/config.go @@ -0,0 +1,188 @@ +package qemu + +import "fmt" + +type MemoryConfig struct { + Size string `env:"MEMORY_SIZE" envDefault:"2048M"` + Slots int `env:"MEMORY_SLOTS" envDefault:"5"` + Max string `env:"MAX_MEMORY" envDefault:"30G"` +} + +type OVMFCodeConfig struct { + If string `env:"OVMF_CODE_IF" envDefault:"pflash"` + Format string `env:"OVMF_CODE_FORMAT" envDefault:"raw"` + Unit int `env:"OVMF_CODE_UNIT" envDefault:"0"` + File string `env:"OVMF_CODE_FILE" envDefault:"/usr/share/OVMF/OVMF_CODE.fd"` + ReadOnly string `env:"OVMF_CODE_READONLY" envDefault:"on"` +} + +type OVMFVarsConfig struct { + If string `env:"OVMF_VARS_IF" envDefault:"pflash"` + Format string `env:"OVMF_VARS_FORMAT" envDefault:"raw"` + Unit int `env:"OVMF_VARS_UNIT" envDefault:"1"` + File string `env:"OVMF_VARS_FILE" envDefault:"/usr/share/OVMF/OVMF_VARS.fd"` +} + +type NetDevConfig struct { + ID string `env:"NETDEV_ID" envDefault:"vmnic"` + HostFwd1 int `env:"HOST_FWD_1" envDefault:"2222"` + GuestFwd1 int `env:"GUEST_FWD_1" envDefault:"22"` + HostFwd2 int `env:"HOST_FWD_2" envDefault:"9301"` + GuestFwd2 int `env:"GUEST_FWD_2" envDefault:"9031"` + HostFwd3 int `env:"HOST_FWD_3" envDefault:"7020"` + GuestFwd3 int `env:"GUEST_FWD_3" envDefault:"7002"` +} + +type VirtioNetPciConfig struct { + DisableLegacy string `env:"VIRTIO_NET_PCI_DISABLE_LEGACY" envDefault:"on"` + IOMMUPlatform bool `env:"VIRTIO_NET_PCI_IOMMU_PLATFORM" envDefault:"true"` + ROMFile string `env:"VIRTIO_NET_PCI_ROMFILE"` +} + +type DiskImgConfig struct { + File string `env:"DISK_IMG_FILE" envDefault:"img/focal-server-cloudimg-amd64.img"` + If string `env:"DISK_IMG_IF" envDefault:"none"` + ID string `env:"DISK_IMG_ID" envDefault:"disk0"` + Format string `env:"DISK_IMG_FORMAT" envDefault:"qcow2"` +} + +type VirtioScsiPciConfig struct { + ID string `env:"VIRTIO_SCSI_PCI_ID" envDefault:"scsi"` + DisableLegacy string `env:"VIRTIO_SCSI_PCI_DISABLE_LEGACY" envDefault:"on"` + IOMMUPlatform bool `env:"VIRTIO_SCSI_PCI_IOMMU_PLATFORM" envDefault:"true"` +} + +type SevConfig struct { + ID string `env:"SEV_ID" envDefault:"sev0"` + CBitPos int `env:"SEV_CBITPOS" envDefault:"51"` + ReducedPhysBits int `env:"SEV_REDUCED_PHYS_BITS" envDefault:"1"` +} + +type Config struct { + TmpFileLoc string `env:"TMP_FILE_LOC" envDefault:"tmp"` + UseSudo bool `env:"USE_SUDO" envDefault:"false"` + EnableSEV bool `env:"ENABLE_SEV" envDefault:"true"` + + EnableKVM bool `env:"ENABLE_KVM" envDefault:"true"` + + // machine, CPU, RAM + Machine string `env:"MACHINE" envDefault:"q35"` + CPU string `env:"CPU" envDefault:"EPYC"` + SmpCount int `env:"SMP_COUNT" envDefault:"4"` + MaxCpus int `env:"SMP_MAXCPUS" envDefault:"64"` + MemoryConfig + + // OVMF + OVMFCodeConfig + OVMFVarsConfig + + // network + NetDevConfig + VirtioNetPciConfig + + // disk + VirtioScsiPciConfig + DiskImgConfig + + // SEV + SevConfig + + // display + NoGraphic bool `env:"NO_GRAPHIC" envDefault:"true"` + Monitor string `env:"MONITOR" envDefault:"pty"` +} + +func constructQemuArgs(config Config) []string { + args := []string{} + + // virtualization + if config.EnableKVM { + args = append(args, "-enable-kvm") + } + + // machine, CPU, RAM + if config.Machine != "" { + args = append(args, "-machine", config.Machine) + } + + if config.CPU != "" { + args = append(args, "-cpu", config.CPU) + } + + args = append(args, "-smp", fmt.Sprintf("%d,maxcpus=%d", config.SmpCount, config.MaxCpus)) + + args = append(args, "-m", fmt.Sprintf("%s,slots=%d,maxmem=%s", + config.MemoryConfig.Size, + config.MemoryConfig.Slots, + config.MemoryConfig.Max)) + + // OVMF + args = append(args, "-drive", + fmt.Sprintf("if=%s,format=%s,unit=%d,file=%s,readonly=%s", + config.OVMFCodeConfig.If, + config.OVMFCodeConfig.Format, + config.OVMFCodeConfig.Unit, + config.OVMFCodeConfig.File, + config.OVMFCodeConfig.ReadOnly)) + + args = append(args, "-drive", + fmt.Sprintf("if=%s,format=%s,unit=%d,file=%s", + config.OVMFVarsConfig.If, + config.OVMFVarsConfig.Format, + config.OVMFVarsConfig.Unit, + config.OVMFVarsConfig.File)) + + // disk + args = append(args, "-device", + fmt.Sprintf("virtio-scsi-pci,id=%s,disable-legacy=%s,iommu_platform=%t", + config.VirtioScsiPciConfig.ID, + config.VirtioScsiPciConfig.DisableLegacy, + config.VirtioScsiPciConfig.IOMMUPlatform)) + + args = append(args, "-drive", + fmt.Sprintf("file=%s,if=%s,id=%s,format=%s", + config.DiskImgConfig.File, + config.DiskImgConfig.If, + config.DiskImgConfig.ID, + config.DiskImgConfig.Format)) + + args = append(args, "-device", + fmt.Sprintf("scsi-hd,drive=%s", config.DiskImgConfig.ID)) + + // network + args = append(args, "-netdev", + fmt.Sprintf("user,id=%s,hostfwd=tcp::%d-:%d,hostfwd=tcp::%d-:%d,hostfwd=tcp::%d-:%d", + config.NetDevConfig.ID, + config.NetDevConfig.HostFwd1, config.NetDevConfig.GuestFwd1, + config.NetDevConfig.HostFwd2, config.NetDevConfig.GuestFwd2, + config.NetDevConfig.HostFwd3, config.NetDevConfig.GuestFwd3)) + + args = append(args, "-device", + fmt.Sprintf("virtio-net-pci,disable-legacy=%s,iommu_platform=%v,netdev=%s,romfile=%s", + config.VirtioNetPciConfig.DisableLegacy, + config.VirtioNetPciConfig.IOMMUPlatform, + config.NetDevConfig.ID, + config.VirtioNetPciConfig.ROMFile)) + + // SEV + if config.EnableSEV { + args = append(args, "-object", + fmt.Sprintf("sev-guest,id=%s,cbitpos=%d,reduced-phys-bits=%d", + config.SevConfig.ID, + config.SevConfig.CBitPos, + config.SevConfig.ReducedPhysBits)) + + args = append(args, "-machine", + fmt.Sprintf("memory-encryption=%s", config.SevConfig.ID)) + } + + // display + if config.NoGraphic { + args = append(args, "-nographic") + } + + args = append(args, "-monitor", config.Monitor) + + return args + +} diff --git a/manager/qemu/qemu.go b/manager/qemu/qemu.go new file mode 100644 index 0000000..cd67fc6 --- /dev/null +++ b/manager/qemu/qemu.go @@ -0,0 +1,35 @@ +package qemu + +import ( + "os/exec" + + "github.com/ultravioletrs/manager/internal" +) + +const qemuRelPath = "qemu-system-x86_64" + +// RunQemuVM runs a QEMU virtual machine: constructs the QEMU command line and starts the QEMU process +func RunQemuVM(exe string, args []string) (*exec.Cmd, error) { + cmd, err := internal.RunCmdStart(exe, args...) + if err != nil { + return nil, err + } + + return cmd, nil +} + +func ExecutableAndArgs(cfg Config) (string, []string, error) { + exe, err := exec.LookPath(qemuRelPath) + if err != nil { + return "", nil, err + } + + args := constructQemuArgs(cfg) + + if cfg.UseSudo { + args = append([]string{exe}, args...) + exe = "sudo" + } + + return exe, args, nil +} diff --git a/manager/service.go b/manager/service.go index 1d04ffe..818156f 100644 --- a/manager/service.go +++ b/manager/service.go @@ -6,13 +6,22 @@ package manager import ( "context" "errors" + "fmt" "os" + "os/exec" "strings" "github.com/digitalocean/go-libvirt" "github.com/ultravioletrs/agent/agent" + "github.com/ultravioletrs/manager/internal" + "github.com/ultravioletrs/manager/manager/qemu" + + "github.com/gofrs/uuid" ) +const firmwareVars = "OVMF_VARS" +const qcow2Img = "focal-server-cloudimg-amd64" + var ( // ErrMalformedEntity indicates malformed entity specification (e.g. // invalid username or password). @@ -26,29 +35,32 @@ var ( ErrNotFound = errors.New("entity not found") ) -// Service specifies an API that must be fullfiled by the domain service +// Service specifies an API that must be fulfilled by the domain service // implementation, and all of its decorators (e.g. logging & metrics). type Service interface { - CreateDomain(ctx context.Context, pool, volume, domain string) (string, error) + CreateLibvirtDomain(ctx context.Context, pool, volume, domain string) (string, error) + CreateQemuVM(ctx context.Context) (*exec.Cmd, error) Run(ctx context.Context, computation []byte) (string, error) } type managerService struct { libvirt *libvirt.Libvirt agent agent.AgentServiceClient + qemuCfg qemu.Config } var _ Service = (*managerService)(nil) // New instantiates the manager service implementation. -func New(libvirtConn *libvirt.Libvirt, agent agent.AgentServiceClient) Service { +func New(libvirtConn *libvirt.Libvirt, agent agent.AgentServiceClient, qemuCfg qemu.Config) Service { return &managerService{ libvirt: libvirtConn, agent: agent, + qemuCfg: qemuCfg, } } -func (ms *managerService) CreateDomain(ctx context.Context, poolXML, volXML, domXML string) (string, error) { +func (ms *managerService) CreateLibvirtDomain(ctx context.Context, poolXML, volXML, domXML string) (string, error) { wd, err := os.Getwd() if err != nil { return "", err @@ -80,6 +92,53 @@ func (ms *managerService) CreateDomain(ctx context.Context, poolXML, volXML, dom return dom.Name, nil } +func (ms *managerService) CreateQemuVM(ctx context.Context) (*exec.Cmd, error) { + // create unique emu device identifiers + id, err := uuid.NewV4() + if err != nil { + return &exec.Cmd{}, err + } + qemuCfg := ms.qemuCfg + qemuCfg.NetDevConfig.ID = fmt.Sprintf("%s-%s", qemuCfg.NetDevConfig.ID, id) + qemuCfg.DiskImgConfig.ID = fmt.Sprintf("%s-%s", qemuCfg.DiskImgConfig.ID, id) + qemuCfg.VirtioScsiPciConfig.ID = fmt.Sprintf("%s-%s", qemuCfg.VirtioScsiPciConfig.ID, id) + qemuCfg.SevConfig.ID = fmt.Sprintf("%s-%s", qemuCfg.SevConfig.ID, id) + + // copy firmware vars file + srcFile := qemuCfg.OVMFVarsConfig.File + dstFile := fmt.Sprintf("%s/%s-%s.fd", ms.qemuCfg.TmpFileLoc, firmwareVars, id) + err = internal.CopyFile(srcFile, dstFile) + if err != nil { + return &exec.Cmd{}, err + } + qemuCfg.OVMFVarsConfig.File = dstFile + + // copy qcow2 img file + srcFile = qemuCfg.DiskImgConfig.File + dstFile = fmt.Sprintf("%s/%s-%s.img", ms.qemuCfg.TmpFileLoc, qcow2Img, id) + err = internal.CopyFile(srcFile, dstFile) + if err != nil { + return &exec.Cmd{}, err + } + qemuCfg.DiskImgConfig.File = dstFile + + exe, args, err := qemu.ExecutableAndArgs(qemuCfg) + if err != nil { + return &exec.Cmd{}, err + } + cmd, err := qemu.RunQemuVM(exe, args) + if err != nil { + return cmd, err + } + + // different VM guests can't forward ports to the same ports on the same host + ms.qemuCfg.NetDevConfig.HostFwd1++ + ms.qemuCfg.NetDevConfig.HostFwd2++ + ms.qemuCfg.NetDevConfig.HostFwd3++ + + return cmd, nil +} + func (ms *managerService) Run(ctx context.Context, computation []byte) (string, error) { res, err := ms.agent.Run(ctx, &agent.RunRequest{Computation: computation}) if err != nil { diff --git a/manager/tracing/tracing.go b/manager/tracing/tracing.go index db3b4be..f85c829 100644 --- a/manager/tracing/tracing.go +++ b/manager/tracing/tracing.go @@ -2,6 +2,7 @@ package tracing import ( "context" + "os/exec" "github.com/ultravioletrs/manager/manager" "go.opentelemetry.io/otel/attribute" @@ -20,7 +21,7 @@ func New(svc manager.Service, tracer trace.Tracer) manager.Service { return &tracingMiddleware{tracer, svc} } -func (tm *tracingMiddleware) CreateDomain(ctx context.Context, pool, volume, domain string) (string, error) { +func (tm *tracingMiddleware) CreateLibvirtDomain(ctx context.Context, pool, volume, domain string) (string, error) { ctx, span := tm.tracer.Start(ctx, "create", trace.WithAttributes( attribute.String("name", pool), attribute.String("volume", volume), @@ -28,7 +29,14 @@ func (tm *tracingMiddleware) CreateDomain(ctx context.Context, pool, volume, dom )) defer span.End() - return tm.svc.CreateDomain(ctx, pool, volume, domain) + return tm.svc.CreateLibvirtDomain(ctx, pool, volume, domain) +} + +func (tm *tracingMiddleware) CreateQemuVM(ctx context.Context) (*exec.Cmd, error) { + ctx, span := tm.tracer.Start(ctx, "createQemuVM") + defer span.End() + + return tm.svc.CreateQemuVM(ctx) } func (tm *tracingMiddleware) Run(ctx context.Context, computation []byte) (string, error) { diff --git a/vendor/modules.txt b/vendor/modules.txt index ca0e3e6..019739f 100644 --- a/vendor/modules.txt +++ b/vendor/modules.txt @@ -54,6 +54,8 @@ github.com/golang/protobuf/ptypes github.com/golang/protobuf/ptypes/any github.com/golang/protobuf/ptypes/duration github.com/golang/protobuf/ptypes/timestamp +# github.com/google/uuid v1.3.0 +## explicit # github.com/mainflux/mainflux v0.0.0-20230726142711-2b78902e0170 ## explicit; go 1.20 github.com/mainflux/mainflux