From 9a3121f14bc75083cd20c02073158ca75b098159 Mon Sep 17 00:00:00 2001 From: James Hackett Date: Sat, 18 Mar 2023 14:05:18 +0000 Subject: [PATCH] add README for user VMs --- jobs/user-vms/README.md | 91 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 91 insertions(+) create mode 100644 jobs/user-vms/README.md diff --git a/jobs/user-vms/README.md b/jobs/user-vms/README.md new file mode 100644 index 0000000..5dbc6a2 --- /dev/null +++ b/jobs/user-vms/README.md @@ -0,0 +1,91 @@ +# User VMs + +This directory contains the configuration files for the user VMs. + +Each VM is configured with cloud-init. Those configuration files are served by wheatley, but they can +be served by any HTTP server. + +## Setting up networking on the host + +The host needs to be configured to allow the VMs to communicate with each other. This is done by creating +a bridge and adding the VMs to it. + +### Create a bridge + +To create a bridge that qemu can use to place the guest (vm) onto the same network as the host, follow +the instructions listed [here](https://wiki.archlinux.org/title/Network_bridge#With_iproute2) for +iproute2, summarised below. + +We need to create a bridge interface on the host. + +```bash +$ sudo ip link add name br0 type bridge +$ sudo ip link set dev br0 up +``` + +We'll be adding a physical interface to this bridge to allow it to communicate with the external (UDM) +network. + +```bash +$ sudo ip link set eno1 master br0 +``` + +You'll need to assign an IP address to the bridge interface. This will be used as the default address +for the host. You can do this with DHCP or by assigning a static IP address. The best way to do this +is to create a DHCP static lease on the UDM for the bridge interface MAC address. + +::: note +TODO: Find out why connectivity seems to be lost when the bridge interface receives an address before +the physical interface. + +If connectivity is lost, release the addresses from both the bridge and the physical interface (in +that order) with `sudo dhclient -v -r ` and then run `sudo dhclient -v ` to assign the +bridge interface an address. +::: + +### Add the VMs to the bridge + +The configuration of the qemu network options in the job file will create a new tap interface and add +it to the bridge and the VM. I advise you for your own sanity to never touch the network options, they +will only cause you pain. + +For others looking, this configuration is specific to QEMU only. + +```bash +qemu-system-x86_64 ... -netdev bridge,id=hn0 -device virtio-net-pci,netdev=hn0,id=nic1 +``` + +You must also add `allow br0` to `/etc/qemu/bridge.conf` to allow qemu to add the tap interfaces to +the bridge. [Source](https://wiki.qemu.org/Features/HelperNetworking) + +The VMs, once connected to the bridge, will be assigned an address via DHCP. You can assign a static +IP address to the VMs by adding a DHCP static lease on the UDM for the VMs MAC address. You can get +the address of a VM by checking the nomad alloc logs for that VM and searching for `ens3`. + +```bash +$ nomad job status distro-vm | grep "Node ID" -A 1 | tail -n 1 | cut -d " " -f 1 +# +$ nomad alloc logs | grep -E "ens3.*global" +# cloud init... ens3: global +``` + +## Configuring the VMs + +The VMs are configured with cloud-init. Their docs are pretty good, so I won't repeat them here. The +files can be served by any HTTP server, and the address is placed into the job file in the QEMU options. + +```hcl +... + args = [ + ... + "-smbios", + "type=1,serial=ds=nocloud-net;s=http://136.206.16.5:8000/", + ] +... +``` + +## Creating a new VM + +To create a new VM, you'll need to create a new job file and a cloud-init configuration file. Copy +any of the existing job files and modify them to suit your needs. The cloud-init configuration files +can be copied and changed based on the user also.