Skip to content

Latest commit

 

History

History
142 lines (114 loc) · 7.22 KB

1.setup-new-proxmox-host.md

File metadata and controls

142 lines (114 loc) · 7.22 KB

Setting up a new proxmox host and including it in the raw-iron proxmox cluster

Pre-requisites:

  • Make sure you have gone through the raw-iron-docs to help you select your hardware and know the purpose of setting up a new host.
  • A dedicated instance in any number of cloud providers, a dedicated instance is needed since we will install Proxmox as our own virtualization layer
  • SSH access to the host, ideally with debian pre-installed

Instructions:

WARNING! The below commands are for a debian system. We are installing proxmox on top of debian to ensure easy installation on a wide range of hardware.

  1. Provision the host with your SSH keys, set hostname, disable root SSH, etc. Playbook for the same can be found here
  2. Add the APT source: echo "deb http://download.proxmox.com/debian/pve buster pve-no-subscription" > /etc/apt/sources.list.d/pve-install-repo.list
  3. Add the key: wget http://download.proxmox.com/debian/proxmox-ve-release-6.x.gpg -O /etc/apt/trusted.gpg.d/proxmox-ve-release-6.x.gpg
  4. Update the packages with: apt-get update && apt-get upgrade
  5. Upgrade the dist: apt-get dist-upgrade
  6. Install the dependencies: aptitude -q -y purge firmware-bnx2x firmware-realtek firmware-linux firmware-linux-free firmware-linux-nonfree
  7. Install proxmox with: apt-get install proxmox-ve
  8. Reboot the host and check if proxmox kernel has been loaded with: uname -rv
  9. Ensure the kvm module has been loaded with: lsmod | grep kvm

At this point, we have proxmox installed and "only" have to setup networking.

The dedicated instances I have seen so far do not come with a DHCP server or are not inside a private network. This networking setup assumes your dedicated instance is the same.

My approach for networking is to use 2 virtual interfaces: 1 for public internet access and one subnet for all the VMs(VM subnet). In order to help with peering/external access, One can set up a manual port forward between the two NICs. There is a script to achieve this, I will link that in the end. We use NAT and masquerading to allow traffic to flow from the VM subnet to the external internet. In order to SSH into the VMs, we then need to use the proxmox host as a jumphost - Information on how to do this is shared later on.

  1. Make a backup of the /etc/network/interfaces in case of an error.

WARNING!!!!!: Inline comments with "#" lead to an error in the /etc/network/interfaces file. Do not have inline comments.

  1. Open /etc/network/interfaces using nano or any other editor, edit the file following this template:
# /etc/network/interfaces
# Loopback device:
auto lo
iface lo inet loopback

# device: eth0
auto enp0s31f6 <OR THE DEFAULT NIC NAME>
iface enp0s31f6 <OR THE DEFAULT NIC NAME> inet static
  # This is the public IP used to SSH into the server
  address <ENTER PUBLIC IP OF SERVER> 
  # The interface and IP is limited to just this one IP, its a /32 subnet and therefore we use a .255 netmas
  netmask 255.255.255.255
  # Pintopoint allows us to configure traffic forwarding from the VM interface
  pointopoint <ENTER GATEWAY IP>
  gateway <ENTER GATEWAY IP >

iface enp0s31f6 inet6 static
  address <ENTER PUBLIC IPv6 OF SERVER> 
  netmask 128
  gateway <ENTER GATEWAY IPv6 PROVIDED>
  up sysctl -p

# for a Subnet
auto vmbr0
iface vmbr0 inet static
  address 10.10.10.1 <OR A DIFFERENT SUBNET IP>
  # ENTER A NETMASK OF /24, ALLOWING FOR 256 ADDRESSES AND VMS IN THIS SUBNET
  netmask 255.255.255.0 
  # THERE IS NO BRIDGE PORT AS SUCH, WE WILL USE MASQUERADING INSTEAD
  bridge_ports none 
  bridge_stp off
  bridge_fd 0
  # IP FORWARDING NEEDS TO BE ENABLED FOR THIS TO WORK
  post-up   echo 1 > /proc/sys/net/ipv4/ip_forward 
  # Setup NAT and Masquerading between the interfaces after the interface is active
  post-up   iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o enp0s31f6 -j MASQUERADE <OR A DIFFERENT SUBNET IP>
  # Delete it once the interface is down
  post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o enp0s31f6 -j MASQUERADE <OR A DIFFERENT SUBNET IP>

WARNING!!! Dragons be here!!! Warning again, the networking template shown above is just that, a template. Please do not use that as the final version without any changes.

  1. Once you are sure (!!), Restart the networking stack with systemctl restart networking. Enter ip addr list to verify the interface IPs and status.

  2. Now we should have networking up, however we do not have DHCP yet (unless there is an upstream DHCP server). We will setup the DHCP server to listen on vmbr0. Install DHCP server with apt install isc-dhcp-server -y.

  3. Edit the nano /etc/dhcp/dhcpd.conf with your DHCP config as shown here:

option domain-name "proxmox.whatever" <OR NAME>;
option domain-name-servers 1.1.1.1,8.8.8.8;

authoritative;

subnet 10.10.10.0 netmask 255.255.255.0 {
  range 10.10.10.20 10.10.10.200; <OR A DESIRED RANGE>
  <FIXED IPs CAN BE SETUP HERE>
  option routers 10.10.10.1; <THIS IS THE SAME IP ADDRESS SET IN THE vmbr0 CONFIGURATION>
}

  1. Since we want the DHCP server to purely listen on the vm subnet, we edit the /etc/default/isc-dhcp-server file and set INTERFACESv4="vmbr0".

  2. Once setup, restart the service so it picks up the config with systemctl restart isc-dhcp-server.

  3. Now reboot the system and visit the proxmox UI at http://PROXMOX-PUBLIC-IP:8006/ and login with your linux username:password.

  4. Remove rpcbind which is not needed by proxmox for most use cases and is a security hole:

    sudo service stop rpcbind.target && sudo service disable rpcbind.target
    sudo service stop rpcbind.socket && sudo service disable rpcbind.socket
    sudo service stop rpcbind.service && sudo service disable rpcbind.service

Create a VM

  1. Login to the proxmox UI, Choose one of the hosts and choose the storage (usually called local).
  2. Switch to the tab called ISO Images, choose Upload and select an VM ISO file (get one from the Ubuntu website)
  3. Choose Create VM on the top right and go through the installer
  4. Once the VM has started, you can click on the VM under the host and choose Console to get access to the visual output
  5. Test internet functionality to ensure everything works as expected
  6. Delete the VM to enable it to join the proxmox cluster. You cannot join a cluster when there are any resources on the host

Joining a Proxmox cluster

  1. Ensure your host is provisioned EXACTLY how you want it. It is very hard to change a proxmox host once it joins a cluster.
  2. Go to your existing proxmox cluster, choose Datacenter > Cluster > Join Information (Create Cluster if one doesn't exist at all)
  3. Copy the join information. Go to your NEW proxmox instance, choose Datacenter > Cluster > Join Cluster. Paste the join information.
  4. Wait for the join to complete. Switch back to the old proxmox cluster GUI, the new host should be present in the cluster.

LXC containers vs VMs

LXC stands for Linux Containers and KVM is an acronym for Kernel-Based Virtual Machine. The main difference here is that virtual machines require their own kernel instance to run while containers share the same kernel.

The rest of this guide assumes that we want to run VMs.