Packaging a Haskell library for artefact evaluation using nix September 19, 2018

This year I packaged two artefacts for the ICFP artefact evaluation process. This post explains the system I used to make it easy to produce the docker images using nix. I hope this documentation will be useful for anyone else submitting a Haskell library for evaluation.

The end result will be an artefact.nix file which is used to build a docker image to submit. It will be an entirely reproducible process as we will fix the versions of all the dependencies we use.

Read more

Using funflow to cache a nix based workflow September 12, 2018

My latest project has been to plot a map of orienteering maps in the UK. This post explains the technical aspects behind the project and primarily the use of funflow to turn my assortment of scripts into into a resumable workflow.

There was nothing wrong with my ad-hoc python and bash scripts but they downloaded and regenerated the whole output every time. The whole generation takes about 2 hours so it’s desirable to only recompute the necessary portions. This is where funflow comes in, by stringing together these scripts in their DSL, you get caching for free. The workflow is also highly parallelisable so in the future I could distribute the work across multiple machines if necessary.

The code for the project can be found here.

Read more

Specifying how a plugin affects recompilation August 10, 2018

Plugins have existed for a long time in GHC. The first plugins were implemented in 2008 by Max Bolingbroke. They enabled users to modify the optimisation pipeline. They ran after desugaring and hence were called “core” plugins. Later, Adam Gundry implemented what I shall refer to as “constraint solver” plugins which allow users to provide custom solver logic to solve additional constraints. Recently, Boldizsár Németh has extended the number of extension points again with a set of plugins which can inspect and modify the syntax AST. Plugins can run after parsing, renaming or type checking and hence are called “source” plugins.

The idea behind plugins was great - a user can extend the compiler in their own specific way without having to modify the source tree and rebuild the compiler from scratch. It is far more convenient to write a plugin than to use a custom compiler. However, if a user wants to use a plugin, they will find that every module where the plugin is enabled is always recompiled, even if the source code didn’t change at all. Why is this? Well, a plugin can do anything, it could read the value from a temperature sensor and insert the room temperature into the program. Thus, we would always need to recompile a module if the temperature reading changed as it would affect what our program did.

However, there are also “pure” plugins, whose output is only affected by the program which is passed as in input. For these plugins, if the source code doesn’t change then we don’t need to do any recompilation.

This post is about a new metadata field which I added to the Plugin data type which specifies how a plugin should affect recompilation. This feature will be present in GHC 8.6.

Read more

Reimplementing graphmod as a source plugin: graphmod-plugin August 9, 2018

You may have heard about source plugins by now. They allow you to modify and inspect the compiler’s intermediate representation. This is useful for extending GHC and performing static analysis of Haskell programs.

In order to test them out, I reimplemented the graphmod tool as a source plugin. graphmod generates a graph of the module structure of your package. Reimplementing it as a source plugin makes the implementation more robust. I implemented it as as a type checker plugin which runs after type checking has finished. The result: graphmod-plugin

An example of the structure of the aeson package
An example of the structure of the aeson package
Read more

Nix scaffolding for running Haskell plugins June 24, 2018

I’ve been all about writing source plugins recently but have been dissatisfied with how rough it is to use them practically. In particular, I am writing plugins which don’t change the semantics of my programs but are useful for debugging. I only sometimes want to run them and don’t want them to appear as dependencies at all on Hackage. It needs to be easy to apply them to my own and other people’s packages.

Of course, the solution was to leverage my knowledge of nix to wrap up everything into a nice interface. The key to the interface is a new function haskell.lib.addPlugin which augments an existing package with a plugin.

This invocation will now also output an HTML representation of the core program of the either package by using the dump-core plugin.

This post will explain this interface in more detail but will not get into the gritty implementation details. The implementation can be found in the haskell-nix-plugin repo.

Read more