Company Blog

Compiling a Rich Configuration Format into JSON – Inline!

I was a little dismayed when I discovered that packer uses JSON as its config file format

From a development perspective, JSON is a great format to work with since it is very simple, has nearly ubiquitous support, and has low overhead. From an operations perspective, JSON is not a fun format to work with for configuration since it lacks support for things like comments and includes.

Turns out this isn’t really a big deal. With a shell idiom known as process substitution, when an app is expecting a filename, we can execute something and provide its output in place of a normal file.

For example, where the typical packer invocation looks like this:

packer build example.json

A richer option might look something like this:

packer build <(./example.rb-packerconfig)

where example.rb-packerconfig is an executable ruby script that writes valid JSON to standard output. How does this work? It’s not input redirection – the packer build command expects a filename as a paramater, and that’s what it’s getting. The shell takes the standard output from the expression evaluated in the parens and provides a path to the file descriptor. How does this work? Lets take a look by trying a simple ruby script.

if __FILE__ == $0
  puts "*** FILENAME ***"
  puts ARGV[0]
  puts "****************"
  puts "*** CONTENTS ***"
  puts File.read(ARGV[0])
  puts "****************"
end

Running this with

ruby read_file.rb <(echo "fake file")

results in the following output on my linux system with zsh:

*** FILENAME ***
/proc/self/fd/11
****************
*** CONTENTS ***
fake file
****************

I asked some colleagues to try this with bash and OS X and the filename sometimes differed, but the outcome was the same. We can pipe the output of an executable into a program expecting a filename for its configuration options.

I particularly like this method of invocation because compiling-to-JSON portion of our configuration leaves behind no artifacts and is truly ephemeral. By being dilligent about using this technique, we never have to worry about forgetting the “build step” that regenerates the temporary JSON configuration.

The sky is now the limit with your config files. My ruby script was simply a ruby hash with some fancy interpolation and insertion of common properties from other files, wrapped up with a

puts JSON.pretty_generate(config)

at the end of the file. But that’s a very naive and slapdash solution, suitable for experimenting with the technique. Later on, it might make sense to use a thin wrapper to convert appropriately-formatted YAML files into JSON. But the beauty of this techinque is that it is very flexible and you can choose a configuration language and conversion process that fits best into your workflow and comfort zone.

For further reading about how this works in bash, look here: http://tldp.org/LDP/abs/html/process-sub.html