Thursday, 31 December 2015

Descreen images in Gimp

I needed to remove the moire patterns of a scan. After a quick search on google I found this descreen plugin.

Unfortunately this plugin also requires the FFT gimp plugin, which is old and requires compiling and that page describes installing it for Windows.

This in turn was a bit of a problem because the FFT plugin has a few dependencies that aren't adequately documented.

To successfully build the plugin you need to first install libgimp-dev and libfftw3-lib packages from the standard repositories (debian/ubuntu)

# sudo apt-get install libfftw3-dev libgimp2.0-dev

Follow this by extracting the tar archive somewhere and doing

# make && make install 

Once this is done, grab the descreen.scm file from the first link and plop it into ~/.gimp2.8/scripts

After starting gimp up there should be a "descreen" option under Filters->Enhance

Monday, 29 June 2015

Wordpress Warning

Just a warning about Wordpress. If you create a Wordpress account you cannot delete it.

Their "delete your account" webpage only tells you how to remove a blog. The best advice they can give you is to make your account inactive. You can't delete it.

Find open files on a filesystem in FreeBSD

I had a problem on an archaic FreeBSD machine whose /var filesystem had tipped over the 80% threshold of our monitoring system.

A brief investigation showed that there was a vast discrepancy between the output of the du and df  commands.

The df command showed that the file system was 81% full where the du command showed it was around 5% full.

This situation is caused when a file is deleted but a process keeps it open. The difference is due to the different ways that the df and du commands calculate free space vs used space. This means that the disk space will still be reserved and therefore seen by df, but since du will no longer see a reference to that file in the directory tree, it cannot see those reserved blocks resulting in a difference between reported disk usage.

So, the next step is to find what process is holding a deleted open. Unfortunately, I would normally use something like lsof to do this but on this ancient version of freebsd that is not an option, so a new option is required.


# fstat -f /var 

USER     CMD          PID   FD MOUNT      INUM MODE         SZ|DV R/W
root     fstat      13839   wd /var      94208 drwxr-xr-x     512  r
root     csh        13582   wd /var      94208 drwxr-xr-x     512  r
operator csh         6713   wd /var     188489 dr-xr-xr-x     512  r
nobody   snmpd      83626    3 /var     259076 -rw-r--r--  2766247084  w
root     cron        1287   wd /var     376832 drwxr-x---     512  r
root     cron        1287    3 /var     164874 -rw-------       4  w
_bgpd    bgpd        1251 root /var      47104 dr-xr-xr-x     512  r
_bgpd    bgpd        1251   wd /var      47104 dr-xr-xr-x     512  r
_bgpd    bgpd        1251 jail /var      47104 dr-xr-xr-x     512  r
_bgpd    bgpd        1250 root /var      47104 dr-xr-xr-x     512  r
_bgpd    bgpd        1250   wd /var      47104 dr-xr-xr-x     512  r
_bgpd    bgpd        1250 jail /var      47104 dr-xr-xr-x     512  r
dhcpd    dhcpd       1213    6 /var     400402 -rw-r--r--   45998  w
root     syslogd      930    3 /var     164883 -rw-------       3  w
root     syslogd      930   12 /var     259338 -rw-r--r--   12844  w
root     syslogd      930   13 /var     259324 -rw-------      78  w
root     syslogd      930   14 /var     259278 -rw-------   40072  w
root     syslogd      930   15 /var     259083 -rw-------      62  w
root     syslogd      930   16 /var     259301 -rw-------   83687  w
root     syslogd      930   17 /var     259077 -rw-r-----    1953  w
root     syslogd      930   18 /var     259079 -rw-r--r--      62  w
root     syslogd      930   19 /var     259085 -rw-------      62  w
root     syslogd      930   20 /var     259087 -rw-------  103292  w
root     syslogd      930   21 /var     259284 -rw-------      78  w
root     syslogd      930   22 /var     259084 -rw-r-----      62  w
root     syslogd      930   23 /var     259082 -rw-r-----      62  w


We can see from the output of the fstat command that the snmp daemon has a 2.7Gb file open.

I killed and restarted snmpd and file system usage dropped down to 3% 


Note: You could limit that output to only show files that are deleted (unowned) by grepping for "nobody"

# fstat -f /var 
nobody   snmpd      83626    3 /var     259076 -rw-r--r--  2766247084  w

Thursday, 9 April 2015

Configuration management: Salt vs Puppet

I've spent some time experimenting with puppet and I have come to the conclusion that I don't much like it.

There are several reasons why this is the case but the main ones are the module system that it uses and the way it abstracts away the configuration details on the client.

 Note: I gave up on puppet after only a couple of days after reaching a point where there was an error somewhere which was impossible to understand because it had no logical behaviour. Therefore I don't profess to be an expert on puppet to any degree at all so any comments from here on are merely my opinion and will definitely include liberal amounts of my misunderstanding of how Puppet works. That is another one of the problems I had with Puppet. The errors it produces are cryptic and there are so many things going on in the box of smoke and mirrors that you can't see the workings of that it is extremely hard to comprehend and debug things. Especially as a n00b. 

From what I have managed to partially understand, Puppet ships with a bunch of pre-installed modules for common applications (such as apache) so you don't need to find and download the appropriate module.

This is all good of course until you need to do something that is not included by default. In that case you need to get on the Internet and find the module you need and install it. Sometimes there will be an "official" puppetlabs module and sometimes you need to trawl through many community provided modules of varying degrees of quality to find one that looks like it might do what you want. This in itself is not terrible but it does involve a lot more mucking about than I think should be required.

Once you have all your modules in order you need to start configuring stuff and this is where the frustration really kicks in.

Puppet uses a bunch of "manifest" files that must be created for each service, host, or groups of hosts. Manifests can include other manifests and everything must be carefully declared in the right places or else things just don't work, and most of the time you will get an obtuse error message that doesn't help a great deal in tracking things down.

Circular dependencies can be common without due care being taken.

For example, the straw that broke the proverbial camels back for me and turned me away from Puppet was where I had two similar hosts with two identical manifests that despite that would produce different results when trying to update them.

One would fail with "Error: Failed to apply catalog: Could not find dependency File[/etc/postfix/main.cf]" while the other one worked fine. This was strange to me because

    a) the file it was complaining about existed on both hosts and

    b) nowhere was I even trying to manage postfix.

I even went to the length of creating two completely empty manifests for the two hosts that still produced the same dissimilar results.

Somewhere in the "box of smoke and mirrors" something was going on but I could not for the life of me figure it out. I even posted a question on serverfault (I rarely need to go to that extreme) but the kind folks there were unable to help either.

So I just gave up because life is too short for that crap.

There are other reasons that I dislike puppet.

A major one is how it abstracts out configuration details for the hosts it manages.

To explain what I mean I will return to the example of apache from before. Now, I am fairly familiar with apache, and I know my way around the apache config files reasonably well. Consider this snippet:

ServerName webserver01.tuxnetworks.com

<VirtualHost first.example.com:80>
    ServerAdmin brett@mydomain.com
    DocumentRoot /var/www/first
</VirtualHost>

That is a pretty simple config for a global server name and a virtual host in apache.

Now, let's look at how you configure that in puppet.

class { 'apache':
default_mods   => true,
}

apache:servername { 'webserver01.tuxnetworks.com' }

apache::vhost { 'first.example.com':
  port    => '80',
  docroot => '/var/www/first',
}

Now this is just a really simple example, but you will note the syntax that you are required to learn and use is totally different to how the target application would normally be configured. Things get even more excitingly confusing when you are trying to configure a service that you are not familiar with. In such cases you are not only required to first figure out what the native settings you need to put in the config file should be but then you must also figure out the correct Puppet syntax for getting there.

I had to do that just yesterday to add another domain to a haproxy server.

You will also note that the ServerAdmin variable has not been set in the puppet example because the 5 minutes that I spent searching the Internet on how to do it came up with nothing. I'm sure with more time I could find the answer (and there is a good chance just adding "serveradmin => brett@mydomain.com'" in there would work) but the point is that I really don't want to have to deal with this level of abstraction for something that I already know how to do.

So, given that you already think that the concept of configuration management systems is great but you are not prepared to devote the rest of your life wrestling with puppet then what do you do?

 Well, another popular system is Chef, but after doing some research I concluded that Chef had almost as many idiocyncracies as Puppet so I have decided to explore the possibilities of saltstack.

So far I've managed to do most of what I want to do with salt and it has been far easier to get my head around it than I found with puppet, although there are still some conceptual things to understand as well as some gotchas that can stall things early on.

Come back later for a post on getting things up and running if you are interested.

The first good thing is that Salt does not require modules.

Another good thing is that Salt does not abstract away configuration details for most use cases.

Note: Salt has something called "pillars" that I have no understanding of. From what I have read these can get a bit more complex but for my purposes (installing packages & pushing configuration files) I have not yet needed to use those. If my requirements get more complex then I may have to visit them but the point is that as a new salt user you don't need to use them) 
Anyway, on a salt server, you would simply copy a working apache.conf over to a directory on the salt-master and the minions will just use it as is. Simples!

But what if you need to set some of those details on a per host basis I hear you ask? For example maybe I want to set the ServerName directive to be whatever the hostname of the server is?

In that case all I need to do is modify it like so:

ServerName {{servername}}

<VirtualHost first.example.com:80>
    ServerAdmin brett@mydomain.com
    DocumentRoot /var/www/first
</VirtualHost>


What I have done there is simply replaced the server name string to become a variable place holder that can be used by the jinja templating engine that salt uses to insert the proper data whenever the minion is updated.

I don't need to learn a whole new syntax because I can just use the standard Apache conf layout that I am already familiar with and make certain parts of it "variable" as required.

Salt really is in my humble opinion much simpler to get up and running (but that may just be because I'm a bit thick of course). Also no offence intended to the folks who love Puppet, I know it is widely used out there and I still have an inherited puppet system here at work. The bottom line is that OSS is all about choice after all.

That is not to say it is perfect however. The documentation leaves a lot to be desired for instance, which is a pretty big problem in itself.

Anyway, stay tuned for a quick start guide to SaltStack at a later date.

Friday, 12 December 2014

FIX: master and slave have equal MySQL server UUIDs

Newer versions of mysql use a unique UUID to do what the server variable server-id used to do.

That means if you need to do master/slave db replication each server will need its own UUID.

If the two servers have the same UUID, you will receive an error when attempting to sync the slave to the master:

master and slave have equal MySQL server UUIDs


If you are creating servers those servers from a virtual machine template then you will need to clear the UUID on each copy you make.

To do this, stop the mysql server and simply remove the file that contains the UUID

rm /var/lib/mysql/auto.cnf  


When you start the server again it will generate a new one automatically.

Tuesday, 7 October 2014

nvidia optimus

Trying to setup Mint Debian on a laptop with nvidia "optimus" is a bit different than the normal routine.

If you just install the nvidia-driver package and do nvidia-xconfig then when you try and reboot X will fail to start with;

"(EE) No Screens Found(EE)"

The reason for this is that the dual video card arrangement that Optimus provides is not supported "out of the box". You need to do a bit of extra work to get it working.

I am assuming that you have already installed the nvidia-drivers and linux kernel headers at this stage.

Next thing to do is make sure you remove /etc/X11/xorg.conf if you already have created that by using nvidia-xconfig. In fact at this point I recommend you remove nvidia-xconfig from your system entirely.

Next install some packages;

apt-get install bumblebee bumblebee-nvidia primus

Now, it would be nice to think that would be it but we do need to edit a file to get things working;


vi /etc/bumblebee/bumblebee.conf

Towards the bottom of the file in the [driver-nvidia] section change this line to look like this;


KernelDriver=nvidia-current

Reboot and you should be good to go.

You can test this by running this command;

optirun glxgears




Thursday, 2 October 2014

Windows 7 refuses to reconnect to Samba after reboot

Faaaaaark! I have been fighting this off and on now for weeks.

The scenario is this:

The network is a Windows domain. Let's call the domain "WINDOZESUX"

I have a Linux host running Samba that is not a domain member. Let's call it "sambaserver"

sambaserver is configured with:

workgroup = WINDOZESUX

Result: Windows 7 client can successfully mount \\sambaserver\share

That's cool.

Move forward until one of the Windows 7 clients reboots.

Result: Windows 7 client cannot reconnect to the samba share "The user name is not found"

Solution: Delete the saved credentials on the Windows client

Result: Windows 7 client can successfully mount \\sambaserver\share

Rinse, lather repeat.

Much troubleshooting follows. Eventually I had a dim recollection that I had "fixed" this once before by using an IP address instead of DNS

Solution: Use an IP address in the UNC path

Result: Windows 7 client can successfully mount \\1.2.3.4\share

Reboot and the share connects automatically without trouble. yay

Now, I was going to leave it at that (and I have for a while) but I really don't want to hard configure IP addresses all over the place because it can become a management disaster for the future.

Now, I have spent hours googling, found thousands of posts for basic errors such as noobs who just didn't have their DNS configured properly at, also a bunch of other posts suggesting that the share was trying to reconnect before the network adapter had come up which plainly was not the case because it worked fine with an IP address. No it was something do with DNS or WINS resolution. Or so I thought.

Suffice to say I spent a lot of time mucking about with WINS, DNS and SAMBA configurations and nothing made a lick of difference.

Eventually while clutching at straws I changed the sambaserver configuration to:

workgroup = LINUXROX

Went through the rinse, lather and repeat process and still it behaved in the same infuriating fashion.

Then while I was poking around in the Windows credential manager, about to delete the credential for the hundredth time, I clicked on "Edit credential"

Lo and behold, the username in the credential manager was shown as WINDOZESUX\username where I am sure the last time I had connected to the share it was using LINUXROX\username

Hmmm. So I changed the entry in the credential manager to the correct LINUXROX\username and rebooted again.

Result: Windows 7 client successfully remounts \\sambaserver\share

FFS. So it appears Windows has a bug (which knowing Microsoft may in fact be a feature) where when you connect to a non domain share from a domain member it does it correctly but when it saves the credentials it goes and changes the goddamned domain from the one you had specified to the domain that the client is on. But it only does that when you use a DNS name. When you use an IP address it works fine. Who knows why? Will Microsoft fix it? Why would they. They might however put up a technet article one day suggesting the solution is to buy a windows server you smelly Linux user.

God I hate Microsoft crap.

In other news, Bordlerlands 2 was just released on SteamOS/Linux!