Virtualbox remote desktop

I just had a frustrating couple of hours trying to get remote desktop working for a Virtualbox guest.

First, even though the settings screen implies that you merely need to turn remote support on you must also have the virtualbox extension pack installed.

Secondly, if you get "connection refused" or similar errors from the rdesktop client then you are probably using virtualbox 5.0.40

It wasn't until I upgraded virtualbox to 5.1.16 that it just magically started working. grrr

Booting Windows as either dual-boot physical machine or as a Virtual machine

How to boot Windows as either dual-boot physical machine or as a Virtual machine

Here is a scenario.

You have a Linux PC that you want to also dual boot to Windows. That's not hard to do right?

Unfortunately though, we all know dual booting is a bit of a pain when you just want to do something quick in Windows. You need to close everything you are doing in Linux just to boot into Windows.

One solution to that is to run Windows in a VM, that's been possible for years now right? Well the problem with that is that sometimes you need to run native Windows to say, play a 3D game.

You could maintain 2 Windows systems, one via dual-booting and the other as a VM but who wants to maintain two copies of Windows?

This is what I do.

Our starting point is a Linux system on one disk (/dev/sda). We will be installing Windows on to a separate disk that is currently empty (/dev/sdb)

WARNING: If you muck this up you can destroy all the data on your Linux system. Make sure you have backups of everything and you are abso-fricking-lutely sure you have identified the correct drive devices

You can check your disks using this command:

sudo fdisk -l

For the remainder of this tutorial I will be using /dev/sdb for the drive that will host Windows 7.

Let's get started!

In your Linux installation you want to add your user to the "disk" group:

sudo usermod -a -G disk brettg

This allows your user to access the physical drive that Windows 7 is installed on.

Note: You will need to logoff and log back in again for this change to take effect.

Install Virtualbox:

sudo apt install virtualbox

Create a place for your Virtualbox disk images:

mkdir -p $HOME/VirtualboxImages/

Create a new virtual disk that references the physical drive that Windows 7 will be installed on:

VBoxManage internalcommands createrawvmdk -filename $HOME/VirtualboxImages/Windows7.vmdk" -rawdisk /dev/sdb

Open up Virtualbox and create a new virtual machine selecting "Use an existing virtual harddisk file" when you are setting it up.

Insert your Windows CD into the virtual machine (either as a physical CD or an ISO image).

Start the VM and go through the normal Windows installation process, then log into Windows.

Open the registry editor. Edit the following keys and set them all to 0:

HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\services\atapi\Start

HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\services\intelide\Start

HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\services\pciide\Start

This forces Windows to install and load three types of disk drivers at boot. (I don't know why you need to set these to zero). By default it just installs the driver that was needed at install time. This is requireed because when we try and boot Windows later it will not be able to read the disk because it doesn't have the proper driver for it.

Shutdown the VM

You will need to add your Windows disk to grub:

sudo update-grub

Now, hopefully if everything went according to plan you should be able to reboot and find Windows listed in your grub menu. Select that and boot into Windows.

Once you are in Windows, you will have to do all the usual driver installs etc but you already knew that right?

Have fun!

Note: Tested with Ubuntu 16.04.1 host and Windows 7/10 guests

Using H.265 (HEVC) on Ubuntu

If you google search how to install H.265 on ubuntu you get a a bunch of posts that describe how to add a PPA for the necessary files.

However the repository hasn't been updated since 2015 (vivid vervet)

If you try to use the vivid repo then things fail because of dependency issues.

But not to worry as it seems that H.265 is now included in the standard xenial repository.

apt-cache search x265
libx265-79 - H.265/HEVC video stream encoder (shared library)
libx265-dev - H.265/HEVC video stream encoder (development files)
libx265-doc - H.265/HEVC video stream encoder (documentation)
x265 - H.265/HEVC video stream encoder

So, all you need to do is;

sudo apt-get install x265

and you are good to go.

Descreen images in Gimp

I needed to remove the moire patterns of a scan. After a quick search on google I found this descreen plugin.

Unfortunately this plugin also requires the FFT gimp plugin, which is old and requires compiling and that page describes installing it for Windows.

This in turn was a bit of a problem because the FFT plugin has a few dependencies that aren't adequately documented.

To successfully build the plugin you need to first install libgimp-dev and libfftw3-lib packages from the standard repositories (debian/ubuntu)

# sudo apt-get install libfftw3-dev libgimp2.0-dev

Follow this by extracting the tar archive somewhere and doing

# make && make install 

Once this is done, grab the descreen.scm file from the first link and plop it into ~/.gimp2.8/scripts

After starting gimp up there should be a "descreen" option under Filters->Enhance

Wordpress Warning

Just a warning about Wordpress. If you create a Wordpress account you cannot delete it.

Their "delete your account" webpage only tells you how to remove a blog. The best advice they can give you is to make your account inactive. You can't delete it.

Find open files on a filesystem in FreeBSD

I had a problem on an archaic FreeBSD machine whose /var filesystem had tipped over the 80% threshold of our monitoring system.

A brief investigation showed that there was a vast discrepancy between the output of the du and df  commands.

The df command showed that the file system was 81% full where the du command showed it was around 5% full.

This situation is caused when a file is deleted but a process keeps it open. The difference is due to the different ways that the df and du commands calculate free space vs used space. This means that the disk space will still be reserved and therefore seen by df, but since du will no longer see a reference to that file in the directory tree, it cannot see those reserved blocks resulting in a difference between reported disk usage.

So, the next step is to find what process is holding a deleted open. Unfortunately, I would normally use something like lsof to do this but on this ancient version of freebsd that is not an option, so a new option is required.

# fstat -f /var 

USER     CMD          PID   FD MOUNT      INUM MODE         SZ|DV R/W

root     fstat      13839   wd /var      94208 drwxr-xr-x     512  r

root     csh        13582   wd /var      94208 drwxr-xr-x     512  r

operator csh         6713   wd /var     188489 dr-xr-xr-x     512  r

nobody   snmpd      83626    3 /var     259076 -rw-r--r--  2766247084  w

root     cron        1287   wd /var     376832 drwxr-x---     512  r

root     cron        1287    3 /var     164874 -rw-------       4  w

_bgpd    bgpd        1251 root /var      47104 dr-xr-xr-x     512  r

_bgpd    bgpd        1251   wd /var      47104 dr-xr-xr-x     512  r

_bgpd    bgpd        1251 jail /var      47104 dr-xr-xr-x     512  r

_bgpd    bgpd        1250 root /var      47104 dr-xr-xr-x     512  r

_bgpd    bgpd        1250   wd /var      47104 dr-xr-xr-x     512  r

_bgpd    bgpd        1250 jail /var      47104 dr-xr-xr-x     512  r

dhcpd    dhcpd       1213    6 /var     400402 -rw-r--r--   45998  w

root     syslogd      930    3 /var     164883 -rw-------       3  w

root     syslogd      930   12 /var     259338 -rw-r--r--   12844  w

root     syslogd      930   13 /var     259324 -rw-------      78  w

root     syslogd      930   14 /var     259278 -rw-------   40072  w

root     syslogd      930   15 /var     259083 -rw-------      62  w

root     syslogd      930   16 /var     259301 -rw-------   83687  w

root     syslogd      930   17 /var     259077 -rw-r-----    1953  w

root     syslogd      930   18 /var     259079 -rw-r--r--      62  w

root     syslogd      930   19 /var     259085 -rw-------      62  w

root     syslogd      930   20 /var     259087 -rw-------  103292  w

root     syslogd      930   21 /var     259284 -rw-------      78  w

root     syslogd      930   22 /var     259084 -rw-r-----      62  w

root     syslogd      930   23 /var     259082 -rw-r-----      62  w

We can see from the output of the fstat command that the snmp daemon has a 2.7Gb file open.

I killed and restarted snmpd and file system usage dropped down to 3% 


Note: You could limit that output to only show files that are deleted (unowned) by grepping for "nobody"

# fstat -f /var 

nobody   snmpd      83626    3 /var     259076 -rw-r--r--  2766247084  w

Configuration management: Salt vs Puppet

I've spent some time experimenting with puppet and I have come to the conclusion that I don't much like it.

There are several reasons why this is the case but the main ones are the module system that it uses and the way it abstracts away the configuration details on the client.

 Note: I gave up on puppet after only a couple of days after reaching a point where there was an error somewhere which was impossible to understand because it had no logical behaviour. Therefore I don't profess to be an expert on puppet to any degree at all so any comments from here on are merely my opinion and will definitely include liberal amounts of my misunderstanding of how Puppet works. That is another one of the problems I had with Puppet. The errors it produces are cryptic and there are so many things going on in the box of smoke and mirrors that you can't see the workings of that it is extremely hard to comprehend and debug things. Especially as a n00b. 

From what I have managed to partially understand, Puppet ships with a bunch of pre-installed modules for common applications (such as apache) so you don't need to find and download the appropriate module.

This is all good of course until you need to do something that is not included by default. In that case you need to get on the Internet and find the module you need and install it. Sometimes there will be an "official" puppetlabs module and sometimes you need to trawl through many community provided modules of varying degrees of quality to find one that looks like it might do what you want. This in itself is not terrible but it does involve a lot more mucking about than I think should be required.

Once you have all your modules in order you need to start configuring stuff and this is where the frustration really kicks in.

Puppet uses a bunch of "manifest" files that must be created for each service, host, or groups of hosts. Manifests can include other manifests and everything must be carefully declared in the right places or else things just don't work, and most of the time you will get an obtuse error message that doesn't help a great deal in tracking things down.

Circular dependencies can be common without due care being taken.

For example, the straw that broke the proverbial camels back for me and turned me away from Puppet was where I had two similar hosts with two identical manifests that despite that would produce different results when trying to update them.

One would fail with "Error: Failed to apply catalog: Could not find dependency File[/etc/postfix/main.cf]" while the other one worked fine. This was strange to me because

    a) the file it was complaining about existed on both hosts and

    b) nowhere was I even trying to manage postfix.

I even went to the length of creating two completely empty manifests for the two hosts that still produced the same dissimilar results.

Somewhere in the "box of smoke and mirrors" something was going on but I could not for the life of me figure it out. I even posted a question on serverfault (I rarely need to go to that extreme) but the kind folks there were unable to help either.

So I just gave up because life is too short for that crap.

There are other reasons that I dislike puppet.

A major one is how it abstracts out configuration details for the hosts it manages.

To explain what I mean I will return to the example of apache from before. Now, I am fairly familiar with apache, and I know my way around the apache config files reasonably well. Consider this snippet:

ServerName webserver01.tuxnetworks.com

<VirtualHost first.example.com:80>
    ServerAdmin brett@mydomain.com
    DocumentRoot /var/www/first
</VirtualHost>

That is a pretty simple config for a global server name and a virtual host in apache.

Now, let's look at how you configure that in puppet.

class { 'apache':
default_mods   => true,
}

apache:servername { 'webserver01.tuxnetworks.com' }

apache::vhost { 'first.example.com':
  port    => '80',
  docroot => '/var/www/first',
}

Now this is just a really simple example, but you will note the syntax that you are required to learn and use is totally different to how the target application would normally be configured. Things get even more excitingly confusing when you are trying to configure a service that you are not familiar with. In such cases you are not only required to first figure out what the native settings you need to put in the config file should be but then you must also figure out the correct Puppet syntax for getting there.

I had to do that just yesterday to add another domain to a haproxy server.

You will also note that the ServerAdmin variable has not been set in the puppet example because the 5 minutes that I spent searching the Internet on how to do it came up with nothing. I'm sure with more time I could find the answer (and there is a good chance just adding "serveradmin => brett@mydomain.com'" in there would work) but the point is that I really don't want to have to deal with this level of abstraction for something that I already know how to do.

So, given that you already think that the concept of configuration management systems is great but you are not prepared to devote the rest of your life wrestling with puppet then what do you do?

 Well, another popular system is Chef, but after doing some research I concluded that Chef had almost as many idiocyncracies as Puppet so I have decided to explore the possibilities of saltstack.

So far I've managed to do most of what I want to do with salt and it has been far easier to get my head around it than I found with puppet, although there are still some conceptual things to understand as well as some gotchas that can stall things early on.

Come back later for a post on getting things up and running if you are interested.

The first good thing is that Salt does not require modules.

Another good thing is that Salt does not abstract away configuration details for most use cases.

Note: Salt has something called "pillars" that I have no understanding of. From what I have read these can get a bit more complex but for my purposes (installing packages & pushing configuration files) I have not yet needed to use those. If my requirements get more complex then I may have to visit them but the point is that as a new salt user you don't need to use them) 

Anyway, on a salt server, you would simply copy a working apache.conf over to a directory on the salt-master and the minions will just use it as is. Simples!

But what if you need to set some of those details on a per host basis I hear you ask? For example maybe I want to set the ServerName directive to be whatever the hostname of the server is?

In that case all I need to do is modify it like so:

ServerName {{servername}}

<VirtualHost first.example.com:80>
    ServerAdmin brett@mydomain.com
    DocumentRoot /var/www/first
</VirtualHost>


What I have done there is simply replaced the server name string to become a variable place holder that can be used by the jinja templating engine that salt uses to insert the proper data whenever the minion is updated.

I don't need to learn a whole new syntax because I can just use the standard Apache conf layout that I am already familiar with and make certain parts of it "variable" as required.

Salt really is in my humble opinion much simpler to get up and running (but that may just be because I'm a bit thick of course). Also no offence intended to the folks who love Puppet, I know it is widely used out there and I still have an inherited puppet system here at work. The bottom line is that OSS is all about choice after all.

That is not to say it is perfect however. The documentation leaves a lot to be desired for instance, which is a pretty big problem in itself.

Anyway, stay tuned for a quick start guide to SaltStack at a later date.