Icon Rufen Sie uns an
+49 441.309197-69 +49 441.309197-69

Puppet 4 the win

Posted by Hauke Behrens on Friday, May 20, 2016

Puppet4 has been out there for quite some time as such it was about time to upgrade our puppet infrastructure from version 3 to 4. As puppetlabs made some pretty big changes, this isn't just done by updating the relevant packages.

In fact it took me about two weeks, to make sure the upgrade process went smoothly. Most of the time was needed to make sure our puppet custom modules are compatible with Puppet4.

We have a lot of modules and some of them didn't need a change for years. Funny to see what you and your colleagues coded couple of years ago, back when we weren't that experienced with puppet. Let me just say, I saw some funny code, which worked OK nevertheless.

Some of the most obvious changes introduced in puppet4:

  • reponames, packagenames and version numbers have changed. For example: the puppetmaster (e.g. version 3.8.2) now is called puppetserver (e.g. version 2.4.0)
  • api endpoints for puppetserver and puppetdb have changed
  • puppet agent now bundles facter, mcollective, pxp-agent and comes with a bundled ruby
  • paths to package binaries, configs and certificates were changed

As you can imagine already these changes can cause a lot of trouble, especially the certpath location change.

My goal was to get a smooth upgrade path and let the puppet agents manage the update themselves. This is what puppet is made for.

Among other things this involved moving the existing certificates to the new location. If you don't do this your agents won't be able to communicate with the master/server anymore.

Another way would be to adjust your configs, but I like to use standard paths. This can break your neck at a later point, if you don't.

We also make heavy use of puppet's api endpoints, for example for our IDB, so we had to change a lot in there too.

Another tool which we use a lot is theforeman which isn't yet fully puppet4 compatible. But with some minor changes, I could ensure the functionalities we use kept working.

In the end I also had to fix some agents by hand, but when dealing with > 200 agents, something had to go wrong,right ?

Migrating code

One of the biggest changes and new feautures is the way fact variables are handled in puppet.

In puppet3 everything was more or less a string to the puppetmaster. In puppet4 there are now real variables like booleans, strings, integers and so on. This can lead to a completely different behaviour if your modules rely on fact values.

false is the new true. at least sometimes..

There is also a big difference when it comes to how empty string variables are handled. In puppet3 an empty string was interpreted as false, while in puppet4 an empty string is now interpreted as true.

Users should never have used empty strings in the first place, but it was possible and they did. So did we. Remember the paragraph on the funny but working code ?

Here's an example:

class badcodeexample {


  if $file {
    file { [$badcodeexample::config_file]:
      ensure => file,
      owner  => 'root',
      group  => 'root',
      mode   => '0644',
      source => "puppet:///modules/badcodeexample/custom/${file}",
      notify => Exec['reload_badcodeservice'];
  } else {
    $template = 'badcodeexample/anotherfile'
      file { [$badcodeexample::config_file]:
      ensure  => file,
      owner   => 'root',
      group   => 'root',
      mode    => '0644',
      content => template($template),
      notify  => Exec['reload_badcodeservice'];

Basically this would put a config file to the puppet-agent and the service related to that file got reloaded. The $file variable was passed to the module by the nodes manifest.

In puppet3 the "if condition" would not have been executed if the file variable was empty because as it was interpreted as false.

In puppet4 this has been flipped upside down, the empty string now means true and the if condition now gets executed. Problem is: there is no file existent to be deployed to the server, leading to really strange behaviours and a lot of confusing errors. In our case this caused puppet to create a directory instead of a file, without complaining about it at all.

The correct way to handle this is to set file=undef. This was the correct way in puppet3 too, but a lot people misused the way puppet3 interpreted empty strings.

So before you upgrade to puppet4 make sure your modules still work, as expected.

Puppetlabs provides some pretty good instructions, which are very helpful.


One of my biggest helpers was puppet-lint, it offers a lot of checks for depreacated feautures. It for example helped me discover all the empty strings variables.

New feautures

There are a lot ! A good overview of the new feautures puppet4 offers is available here: http://www.slideshare.net/tuxmea/power-of-puppet-4

One of the reasons we wanted to upgrade to puppet4 is our currently ongoing process to make our icinga2 monitoring managed by puppet. There are already puppet modules out there, but they mostly rely on exported resources.

Our approach to this is to use puppetdb-query. Which makes it possible to query your puppetdb for nodes and facts directly from puppet manifests.

We started using it with puppet3, which was really frustrating, as every approach needed some workarounds and hacks. Now with puppet4 it's a lot easier to access facts hash values and make use of them.

By now puppetlabs also introduced the so called Puppet query language (PQL), which is heavily inspired by the puppetdb-query module, so I guess my work on our icinga2 module extensions needs some overhaul too.

Let me just say that being able to query puppetdb directly from your modules opens up a whole new way of writing puppet modules.

Worth it ?

You read the heading,right ? It is. A lot of work, but it makes dealing with puppet in general a lot more comfortable.

Still using puppet3 ? Need help when upgrading ? Let us know.