Rebuilding the Hypervisor with new hardware

Good news!
I’ve spend the day rebuilding the server. I’ve completely overhauled the system, replacing what was an aging AMD octacore with a new Intel server.

New specs are:
CPU: Intel E3-1245v5 3.5GHz 4 Core, 8 threads.
Motherboard: INTEL S1200SPL
RAM: 32GB (4x8GB) Crucial 2400Mhz ECC
Other components: Nvidia NVS300 gfx card, quad DVB-T tuner, additional SATA raid card for the 11 hard disks in the two raid arrays.
All houses in a Cooler Master Cosmos II Full Tower.

All in all, the migration has been very smooth. I’ve been able to get all the VMs up and running again without much fuss. I didn’t realise that remote console through the Intel BMC web console was not possible without an additional component, so I’ll be ordering an Intel remote management component (AXXRMM4LITE2) very soon.

PCI and File System Pass-through on KVM

Putting aside Landscape for a moment. That, for the record, I was able to get up and running by following the documentation. The server generated snake-oil SSL certificates and enabled SSL by default which would mean quite a lot of re-configuring to make it work behind the reverse proxy. More problematic was that the other machines, when trying to connect to the landscape server, would reject the connection due to the self-signed certificate. The mechanisms for Landscape aren’t clear, so at this point I’m unsure if this would be a problem if I disabled SSL on apache only (thereby allowing the reverse proxy to handle SSL – and have all the landscape client connect via it) or if the landscape service itself also needed the SSL certificates. If that’s the case then the challenge will be to have the current SSL certs copied to the landscape server when they’re renewed (or nightly via rsync and cron).

So as I said, I’m putting that aside for the moment to focus on changing how my VMs on my KVM server access local files and migrating the last few services on the KVM host itself to a VM. Currently that includes SAMBA/SMB file shares. MythTV and Plex. 

The biggest hurdle is to move mythtv to a VM as it will require PCI passthrough for the TV tuner card. This is possible and the documentation makes it clear how to achieve this, however when I initialled passed the TV tuner card to the VM, the VM refuses to start. Similarly, USB devices are not being passed through.

After researching, there appears to be a bug in apparmor that stops USB from being passed through. Solution available here.

The PCI problem was a little more complex. After checking the output of the error logs for KVM and dmesg and googling what I could. the problem ended up being that PCI cards and devices that share a bus, therefore share the interrupts and have to all be added to the virtual machine. The system cannot differentiate between them. After checking the output of lspci and comparing that to the list of devices in /sys/kernel/iommu_groups/11 (group 11 was the where all the devices were that I needed to pass through). I added all the components of the TV tuner card and a IEEE1394 port on the mother board (that I have never used) to the VM. To make my life easy and ensure I didn’t make mistakes I wrote it out as a script, based on the documentation here.

#!/bin/bash

echo “14f1 8800” > /sys/bus/pci/drivers/pci-stub/new_id
echo “0000:05:06.0” > /sys/bus/pci/devices/0000:05:06.0/driver/unbind
echo “0000:05:06.0” > /sys/bus/pci/drivers/pci-stub/bind

echo “14f1 8802” > /sys/bus/pci/drivers/pci-stub/new_id
echo “0000:05:06.2” > /sys/bus/pci/devices/0000:05:06.2/driver/unbind
echo “0000:05:06.2” > /sys/bus/pci/drivers/pci-stub/bind

echo “14f1 8804” > /sys/bus/pci/drivers/pci-stub/new_id
echo “0000:05:06.4” > /sys/bus/pci/devices/0000:05:06.2/driver/unbind
echo “0000:05:06.4” > /sys/bus/pci/drivers/pci-stub/bind

echo “1106 3044” > /sys/bus/pci/drivers/pci-stub/new_id
echo “0000:05:0e.0” > /sys/bus/pci/devices/0000:05:0e.0/driver/unbind
echo “0000:05:0e.0” > /sys/bus/pci/drivers/pci-stub/bind

executing the script and then adding all the above PCI devices did the trick. The VM now starts and lists all the PCI devices:

wargus@media:~$ lspci

00:08.0 Multimedia controller: Conexant Systems, Inc. CX23880/1/2/3 PCI Video and Audio Decoder [MPEG Port] (rev 05)
00:09.0 Multimedia video controller: Conexant Systems, Inc. CX23880/1/2/3 PCI Video and Audio Decoder (rev 05)
00:0a.0 Multimedia controller: Conexant Systems, Inc. CX23880/1/2/3 PCI Video and Audio Decoder [IR Port] (rev 05)
00:0b.0 FireWire (IEEE 1394): VIA Technologies, Inc. VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller (rev c0)

A quick probe of lsmod also shows that the v4l2 drivers are loaded as are the drivers for the TV tuner card (cx8800). Changes are persistent after a restart of the VM host, too.

Setting up Plex media server, mythTV and samba shouldn’t be a challenge from this point.

This leaves the last challenge – setting up file system pass-through on the VMs. The documentation, here and here, perhaps wasn’t as helpful as it could be. I tested out FS pass-through on my mail server first, as it also hosted my nextcloud installation. I wanted to move the data files that constitute my nextcloud storage to the much roomier RAID+LVM on the KVM host itself.

There was suffice to say, a lot of flaffing about before I managed to get it to work. The screenshot below shows the configuration in virt-manager.

What it does not show are the file permissions on the KVM host. The directory has the permissions of:
ls -l
drwxr-xr-x 3 libvirt-qemu kvm 18 Jan 1 09:44 nextcloud

This is because in ‘mapped’ mode “files are created with Qemu user credentials and the client-user’s credentials are saved in extended attributes.” Whereby client-user is referring to users on the VM. Once mounted on the guest OS with:

sudo mount -t 9p -o trans=virtio,version=9p2000.L /nextcloud /nextcloud/

I was about to copy in the data directory (when apache2 was off) preserving the ownership and permissions of the files.

On the host OS, the files all appear to be owned by libvert-qemu and kvm, on the guest OS they all appear to be owned by the www-data user. The final step is of course to make the changes persistant by editing the /ect/fstab file and adding the in the line:
/nextcloud /nextcloud 9p trans=virtio,version=9p2000.L 0 0

Fixing little things

This was a weekend of Christmas/fixing niggling problems on my systems.

  1. Migrate to NextCloud from Owncloud.
    1. Easy – thanks to documentation and a blog post.
  2. Fix accessibility problems with the calendar plugin in nextcloud.
    1. This was primarly caused by my ignorance – when adding calendars in, say, Thunderbird you need to be very specific on the URL – nextcloud does not make it clear what the url is for specific calendar unless you go looking for it. Simply adding the primary address (https://www.warbel.net/owncloud/remote.php/dav/) will not work.
  3. Fix the zabbix server, as it wasn’t starting with the system.
    1. Checking the service status show the issue here. It wasn’t set to start automatically. Fixed with sudo systemctl enable zabbix-server.service.

As far as getting landscape up and running – this seems a little problematic. After spinning up an VM and using the documentation I can’t seem to add machines to the landscape server. This is because it uses self-signed ssl certificates. The solution is easy enough – provide the letsencrypt certificates. However because it sits behind a reverse proxy the website will need configured to not use SSL but as far as I can tell so far, the landscape service itself needs the ssl certificate. This can be fixed by using rsync and cron to move the necessary files, but it’s going to be a pain. We’ll see.

Configuring Mattermost on Ubuntu 16.04 with Apache, Mysql and Let’s Encrypt

First some background:

I’ve become increasingly aware of ‘free’ services like Slack (but facebook and Google also fit into this category). While they do offer free convenient services, the true cost is to your privacy and security. Having some technical knowledge means that I can have the convenience and features of their platforms without having to sacrifice any of the above – it’s the best of both worlds!

My notes below will only cover some of the more difficult aspects of configuring apache and how I circumvented let’s encrypt’s process of creating and accessing hidden directories. The Mattermost documentation is your friend too; their setup guides were accurate and effective. To anyone reading this: I strongly suggest setting up a test server first before attempting to create a production system from scratch.

Setting up Mattermost:

As stated above, documentation is your friend. I set up the system on a test VM and was able to get it running with minimal fuss. I created a mysql user and database for the installation on my production web server using phpmyadmin and from there, the rest of the configuration was from within Mattermost itself. Mattermost encourage you to use Nginx and PostgreSQL. To configure MySQL in Mattermost, no changes to the SQLSetting are needed except the  DataSource directive which you will need to modify to suit your username/password/database that you setup.

Once I was satisfied with the setup I migrated the directory structure from the testbed to the production server and setup the init scripts as per the mattermost documentation so it runs as a service under it’s own account. I will stress that if you do that, be sure to check the data directory directive so that the right locations are accessible. If you have any trouble with Mattermost the logs are a good place to start looking for the problem. 😉

Setting up Let’s Encrypt:

This is really a two stage process. The first stage is to setup a sub-domain using your DNS provider. I use NoIP, as I can use their client to update my dynamic IP address if and when my internet connection drops out.

As Mattermost runs on a high port number and apache has not been configured as a reverse proxy just yet. I needed to run lets encrypt in standalone mode. In this mode, letsencrypt acts as its own http server in order to verify you have control over the domains you’re trying to create ssl certificates for. The commands I ran looked like this:

sudo service apache2 stop

sudo letsencrypt certonly –standalone -d www.warbel.net -d bel.warbel.net -d blog.warbel.net -d mattermost.warbel.net

sudo service apache2 start

Let’s encrypt recognized that I needed to add the new domain mattermost to my list of sites and updated the certificate accordingly.

Configuring Apache:

Originally I had intended to setup Mattermost as a subdirectory on my primary domain which was in keeping with my previous projects. Unfortunately that seemed impossible. In the end it was easier to setup a sub-domain and then configure apache with a new site. I had to do some serious googling to find a semi-working config. Mattermost uses web-sockets and application program interfaces (APIs) which do not play well with reverse proxies out of the box. Furthermore, as Let’s Encrypt had already reconfigured components of Apache, I had to modify what I found to match with my pre-existing sites.

I created a new site in /etc/apache2/sites-available/ called mattermost.warbel.net.conf and working off this configuration file as an example created the below:

<VirtualHost mattermost.warbel.net:80>
ServerName mattermost.warbel.net
ServerAdmin xxxx@warbel.net

ErrorLog ${APACHE_LOG_DIR}/mattermost-error.log
CustomLog ${APACHE_LOG_DIR}/mattermost-access.log combined

# Enforce HTTPS:
RewriteEngine On
RewriteCond %{HTTPS} !=on
RewriteRule ^/?(.*) https://%{SERVER_NAME}/$1 [R,L]
</VirtualHost>
<IfModule mod_ssl.c>
<VirtualHost mattermost.warbel.net:443>
SSLEngine on
ServerName mattermost.warbel.net
ServerAdmin xxx@warbel.net

ErrorLog ${APACHE_LOG_DIR}/mattermost-error.log
CustomLog ${APACHE_LOG_DIR}mattermost-access.log combined

RewriteEngine On
RewriteCond %{REQUEST_URI} ^/api/v1/websocket [NC,OR]
RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC,OR]
RewriteCond %{HTTP:CONNECTION} ^Upgrade$ [NC]
RewriteRule .* ws://127.0.0.1:8065%{REQUEST_URI} [P,QSA,L]
RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
RewriteRule .* http://127.0.0.1:8065%{REQUEST_URI} [P,QSA,L]
RequestHeader set X-Forwarded-Proto “https”

<Location /api/v1/websocket>
Require all granted
ProxyPassReverse ws://127.0.0.1:8065/api/v1/websocket
ProxyPassReverseCookieDomain 127.0.0.1 mattermost.warbel.net
</Location>
<Location />
Require all granted
ProxyPassReverse https://127.0.0.1:8065/
ProxyPassReverseCookieDomain 127.0.0.1 mattermost.warbel.net
</Location>

ProxyPreserveHost On
ProxyRequests Off
SSLCertificateFile /etc/letsencrypt/live/www.warbel.net/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/www.warbel.net/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf
</VirtualHost>
</IfModule>

The main difference being that the ssl virtualhost needed to be contained within the ssl module configuration. I’ll also point out that there was a typo in the last proxypassreverse directive- the url was missing the https that stopped the websites from pushing new chat messages automagically to clients connected to the server.

Enable the site with:

sudo a2ensite mattermost.warbel.net

and reload or restart the server – you should have a working Mattermost server.

 

MotionEye Cameras Failing

It seems that after a period of time the webcams attached to my raspberry pi fail after about 2 days of usage. While I haven’t had the time to delve into the reasons why, a fix or workaround is to have the device restart every day at 5am.

Log into the raspberry pi as the administrator and edit the crontab file with: crontab -e

Then add the following line:
0 5 * * * shutdown -hr now >/dev/null 2>&1

This will restart the device every day at 5am. It should keep working until I have the time to look through the system logs and find what the problem is. I suspect it may be power related, or just bad coding somewhere?

Migrating from VirtualBox To KVM

vbox_logo2_gradientAs written previously: There are performance benefits to be had by switching from VirtualBox to KVM. And now, after making the switch I can firmly say that not only are the performance benefits noticeable, the configuration of automatic startup and, prima facie, backups, seems to be much easier to establish and use.

I’ve grown very fond of Oracle’s Virtualbox but given that it’s
more of a prosumer product rather than an enterprise one, it’s only  fair that I learn how to use it’s bigger brother KVM.

The process of switching to KVM itself was very simple, all things considered. The process I followed, after troubleshooting the various stages worked like this:

  1. Stop and backup all the virtualbox VMs.
  2. Convert the virtual disks from a virtualbox to kvm format.
  3. Create the virtual machines using virt-kvmbanner-logo3manager.
  4. Removed vboxtool and configs
  5. Create a bridge interface on the KVM Host
  6. Set each of the virtual machines to use the new bridge interface to connect to the internet and local network.
  7. Configure each VM’s network interface to use the new network interface.

In Detail:

Stopping the machines was easy, simply ssh into them and run shutdown -h now.

Backup the machines using the clone option in VirtualBox.

On the Hypervisor, navigate to the virtual machine directory (usually /home/user/VirtualBox VMs/ and create a new disk image from the vdi files like this:

qemu-img convert -f vdi -O qcow2 VIRTUALBOX.vdi KVM.qcow2

Thanks to this website for the useful tip. At this point I moved each of my VM disks to a new separate directory. This wasn’t strictly necessary, it’s just neater!

Use virt-manager to then create the virtual machines. The process is intuitive. Be sure to enable bringing up at boot. It was at this point that I ran into trouble. By default the virtual machines cannot talk to the host which is a problem if the host is also a file server. To get around this I had to modify the network config on the host. The KVM network page provided information on how to achieve this. Ultimately, you create network bridge then set each of the VMs to use that bridge. Below is my modified interfaces file on my Ubuntu 16.04 VM host:

# The primary network interface
#bridge to allow the VMs and the host to communicate
auto br0
iface br0 inet static
address 10.60.204.130
netmask 255.255.255.128
broadcast 10.60.204.255
gateway 10.60.204.129
dns-nameservers 10.60.204.133 8.8.8.8
dns-search warbelnet.local
bridge_ports enp6s0
bridge_stp off
bridge_maxwait 0
bridge_fd 0

Below is the configuration in virt-manager for the network in one of my VMs:

bridge_settings2

This was testing and working.

As my virtual machines are all running Ubuntu 16.04 the network interfaces file needed to be updated as the interface name changes after a hardware change.

Finally, I uninstalled Virtualbox, removed vboxtool (which I had been using to automatically start the Virtualbox VMs), removed vboxtool’s config from /etc/ and restarted everything to test.

Very happy to say it’s been quite a success!

Adding a New Domain and Securing it with SSL

This week my wife asked me to create for her a blog. As such I’ve had to rejig the www server to make space for her new domain.

The process is quite simple:

  1. Create the new user on the www server so she has sftp access.
  2. Create a mailbox, and mysql database for the new user.
  3. Create the directory structure, copy the latest wordpress to it and set file permissions.
  4. Create the DNS entries in my DNS provider, and locally on my home DNS
  5. Copy and edit my blog’s apache config files.
  6. Enable the new site -without SSL
  7. Update the Let’s Encrypt certificate files with the new domain
  8. Enable the SSL website.
  9. Configure wordpress.

In detail:

Create a new user on the web server with adduser -D

Depending on your setup, create a new mailbox, if you like and create a new database and user. I use phpmyadmin and postfixadmin for these tasks. Remember to note down the passwords and make them secure! Use a random password generator if needs must.

I created the directory structure in /var/www/bel.warbel.net/ moving forward, I think it would be more secure to have user’s websites stored in their home directories and then have the users jailed to stop access to the wider system. It would also make sense to have the sub domain match their username for simplicity’s sake. Be sure to change ownership once you’ve copied in the latest word press: chown USER:www-data /var/www/bel.warbel.net -R

WordPress (as www-data) will need write permissions on the sub directories particularly in the data directories to allow for downloading plugins and themes. Be sure to chmod g+w those directories.

At this point, if you have not done so already, create the DNS entries for your site. For me this meant updating my internal DNS records with a CNAME for bel.warbel.net to point to www.warbel.net, which I replicated on my own DNS hosts: https://www.noip.com/ who I recommend. As I do not have a static IP address, I use their dyndns services on my router.

Next, I copied the /etc/apache2/sites-available/blog.warbel.net.conf and blog.warbel.net-le-ssl.conf and renamed them to bel.warbel.net.conf and bel.warbel.net-le-ssl.conf respectively. The let’s encrypt program will, initially, not expect to see a SSL site, so I commented out the redirects in the non-ssl file and updated the config file for all the references to the hostname and root directories.

Enable the new site: a2ensite bel.warbel.net; service apache2 reload

Run the ssl certificate generator with all the domains you need:

sudo letsencrypt certonly –webroot -w /var/www/html -d www.warbel.net -w /var/www/bel.warbel.net -d bel.warbel.net -w /var/www/blog.warbel.net -d blog.warbel.net

If successful, it will show you a screen, prompting you to agree to update the certificate with the new domain:

Lets Encypt

At this point, it is safe and appropriate to enable the ssl site with: a2ensite bel.warbel.net-le-ssl.conf; service apache2 reload

Be sure to edit the non-ssl site’s config and re-enable forced ssl.

Finally, configure the new wordpress site. I found that to enable uploading files (updates etc) I needed to add a line to wp-config.php:

define(‘FS_METHOD’, ‘direct’);

Migrating from Virtualbox to KVM

After doing some much needed research into virtualisation on Linux, it’s become apparent that I should migrate my virtual machines from Virtualbox to KVM. KVM has significant performance benefits and it is a solid ‘production’ system. It’s also clear that if I want to advance my technical skills in the enterprise Linux space, then I need to learn more about KVM and implement it on my systems.

I love Virtualbox because it is cross platform- I can create a VM on a Linux host, and move it to a windows host if needed. The remote desktop server built into the program, too, is a very handy feature. However I will admit, that I very rarely will spin up a VM on Linux and move it to another OS (if ever) and since discovering MobaXTerm on windows, I can now easily, from any windows machine (read: my laptops) access the virt-manager X window session of a running VM on KVM. As an aside, MobaXTerm is an amazing program and compliments putty quite nicely!

My concerns so far about the migration are 3 fold:

  1. I need to convert the disk images into a native format for KVM and virt-manager to use.
  2. I currently automate my VM startup and shutdown with VBoxTool so I will need to either find a preexisting automation solution, or create my own init scripts.
  3. Virtualised hardware: Clearly Virtualbox and KVM will virtualise hardware in their own ways, so I need to be sure that the machines can migrate to the new environment and still work. I’m mostly concerned with networking as experience has taught me that Linux is very forgiving of hardware changes, however with the new naming conventions of Ethernet devices, my network configs will need to be updated.

Using a Raspberry Pi as a cheap security system

A small project this weekend. I used my hitherto untouched Raspberry Pi 2 as a security system. The process is reasonably straight forward to anyone who is already familiar with the Raspberry Pi.

I have two web cams which are attached to the Pi via an external powered usb hub. This is necessary as the device does not have enough power to run itself and the cameras. It also has a USB 2.4G wireless dongle.

I’ve installed MotionEye onto the Pi’s SD card. Again, simply using:
sudo dd if=MotionEyesIMGFile of=/dev/sdX
did the job.

Once the device was setup using the wired network, it could be secured with an admin password, by default it has no password and it can be added to the wireless network. All of the settings can be accessed by clicking the menu icon in the top left hand corner, and the process is intuitive, as is adding the cameras.

The only real difficulty encountered was allowing it to function behind the reverse proxy. To do so relied on having to edit the /etc/motioneye.conf file to include the line:
base_path /security

I had tried to ssh into the device to make the changes, however the file system is set to RO by default, so I ended up removing the microSD card and editing the files on my desktop.

That then needed to be mirrored in my apache reverse config files:
ProxyPass /security http://10.60.204.xxx
ProxyPassReverse /security http://10.60.204.xxx

And done! The new security system is accessible via ssl at: https://www.warbel.net/security/