Complete overkill or exactly right? Deploying a static site using nix

Posted on June 24, 2019

eventlog2html is my new library for visualising Haskell heap profiles as an interactive webpage.

For the documentation, I thought it was important to provide some interactive examples which is why I decided to host my own static webpage rather than rely on a GitHub README. This led to two constraints:

The documentation is a static web page containing up-to-date examples of the tool’s output.
The page should be automatically deployed using CI.

This post is a question about whether the combination of nix, Cachix, Travis CI, haskell.nix and Hakyll was the perfect solution to these constraints or an exercise in overkill.

Generating the static site

The static site is generated using Hakyll. The content is written using markdown and rendered using pandoc. Inline charts are specified using special code blocks.

```{.eventlog traces=False }
examples/ghc.eventlog --bands 10
```

A pandoc filter identifiers a code block which has the eventlog class and replaces it with the suitable visualisation. Options can be specified as attributes or using normal command line arguments.

Using a site generator implemented in Haskell meant that I could import eventlog2html as a library and use it directly without having to modify the external interface. This ended up being about 40 lines for the filter which inserts eventlogs. There is also a simpler filter which inserts the result of calling --help.

Using Hakyll has already proved to be a good idea when I wanted to add the examples gallery. It was trivial to generate this page from a folder of eventlogs so that all I have to do to add a new eventlog is commit it to the repo.

So far, I haven’t broken the complexity budget. In order to satisfy the first constraint and keep the generated documentation up to date I created a package for the site. In the cabal.project file I then added the site’s folder as a subdirectory. Now, hakyll-eventlog will use the local version of eventlog2html as a dependency when it builds the site.

packages: .
          hakyll-eventlog

The site can be built and run using cabal new-build hakyll-eventlog. Now we move onto how to perform deployment of the generated site.

Deploying using Travis

CircleCI and Travis are both popular CI providers and they can both to deploy to GitHub Pages. However, the Travis integration was far simpler to set up. There is built-in support for GitHub pages as a deployment target so a single stanza is necessary to perform the deployment.

deploy:
  provider: pages
  skip_cleanup: true
  github_token: $GITHUB_TOKEN
  keep_history: true
  target_branch: gh-pages
  local_dir: site
  on:
    tags: true

The stanza says, deploy to GitHub pages by pushing the contents of the site directory to the gh-pages branch of the current repository. GitHub then serves the contents of the gh-pages branch on https://mpickering.github.io/eventlog2html.

Now all we need to do is generate the site directory. I found it quite daunting to modify the Travis script generated by haskell-ci so at this point I decided to convert all the CI infrastructure to use nix instead.

Building using nix

An obvious question at this stage is why is nix necessary at all? Wouldn’t a CI configuration which uses cabal have worked equally as well? On reflection, I could think of four reasons why I considered this to be a good idea.

Much more concise than the haskell-ci generated travis file.
Easier to run the same script locally
Easier for other nix users to use the project
Easy caching with Cachix

`haskell.nix`

A key part in the decision was the new haskell.nix tooling to build Haskell packages. If you use the normal Haskell infrastructure which is built into nixpkgs then any collaborator has to know about nix in order to fix CI when it breaks. On the other hand, haskell.nix creates its derivations from the result of cabal new-configure so it matches up with using a new-build workflow locally.

Purity is retained by explicitly passing the --index-state flag to new-configure so anyone can update the CI configuration by changing the index state parameter in the default.nix file.

How does this look in practice? The default.nix is a very concise script which calls haskell.nix.

let
  pin = import ((import ./nix/sources.nix).nixpkgs) {} ;

  # Import the Haskell.nix library,
  haskell = import (builtins.fetchTarball https://github.com/input-output-hk/haskell.nix/archive/master.tar.gz) { pkgs = pin; };

  # Generate the pkgs.nix file using callCabalProjectToNix IFD
  pkgPlan = haskell.callCabalProjectToNix
              { index-state = "2019-05-10T00:00:00Z"
              ; src = pin.lib.cleanSource ./.;};

  # Instantiate a package set using the generated file.
  pkgSet = haskell.mkCabalProjectPkgSet {
    plan-pkgs = import pkgPlan;
    pkg-def-extras = [];
    modules = [];
  };

  site = import ./nix/site.nix { nixpkgs = pin; hspkgs = pkgSet.config.hsPkgs; };

in
  { eventlog2html = pkgSet.config.hsPkgs.eventlog2html.components.exes.eventlog2html ;
    site = site; }

The callCabalProjectToNix function is the key. That is the function which calls new-configure to create the build plan directly using cabal. It produces the same result as calling plan-to-json manually, as the documentation explains how you should use haskell.nix. Therefore, the rest of the documentation can be followed but with the difference that the result of callCabalProjectToNix is passed as an argument to mkCabalProjectPkgSet rather than an explicit pkgs.nix file.

A derivation which generates the documentation site is also created. The definition is simple because haskell.nix takes care of building the site generator for us. All the derivation does it apply the site generator to the contents of the docs/ subdirectory.

{ nixpkgs, hspkgs }:
nixpkgs.stdenv.mkDerivation {
  name = "docs-0.1";

  src = nixpkgs.lib.cleanSource ../docs;
  LANG = "en_US.UTF-8";
  LOCALE_ARCHIVE = "${nixpkgs.glibcLocales}/lib/locale/locale-archive";

  buildInputs = [ hspkgs.hakyll-eventlog.components.exes.site ];

  preConfigure = ''export LANG="en_US.UTF-8";'';

  buildPhase = ''site build'';

  installPhase = ''cp -r _site $out'';
}

Evaluating default.nix results in the a set containing the two outputs of the project. The executable eventlog2html and the documentation site. You can build each attribute locally

cachix use mpickering
nix build -f . eventlog2html
nix build -f . site

but also by passing a link to the generated github tarball.

nix run -f https://github.com/mpickering/eventlog2html/archive/master.tar.gz eventlog2html -c eventlog2html my-leaky-program.eventlog

Updated Travis configuration

The build job now calls nix to build these scripts and uses the -o flag to place the output into the site directory. The precise location where Travis expected to find the generated site so the deployment step can now find the files.

- stage: build documentation
    script:
      - nix-env -iA cachix -f https://cachix.org/api/v1/install
      - cachix use mpickering
      - cachix push mpickering --watch-store&
      - nix-build -A site -o site

We use Cachix to cache the result of building the individual derivations. This makes a huge difference to the total time that CI takes to run.

You can greatly speed up the initial CI runs by pushing local build artifacts to travis.

nix-store -qR --include-outputs $(nix-instantiate default.nix) | cachix push mpickering

Conclusion

That’s basically it. Despite a complicated amalgamation of tools, everything worked out nicely together without any horrible hacks. All I had to do was to work out how to fix the pieces together. When using bleeding edge technology such as haskell.nix, this isn’t always straightforward but now I’ve documented my struggles the next person should find it easier.

Addendum: Using secure env vars in Travis

We need to set two env vars for CI to work. You have to encrypt these so you can place them into the public .travis.yml file without exposing secrets.

GITHUB_TOKEN - To allow travis to push to the repo
CACHIX_SIGNING_KEY - To allow Cachix to push to a cache

To generate the GITHUB_TOKEN go to GitHub settings and generate a token with the public_repo permissions.

The CACHIX_SIGNING_KEY can be found in ~/.config/cachix/cachix.dhall in the secreyKey field for the corresponding binary cache.

Once you have the keys you have to encrypt them using the travis command line application.

nix-shell -p travis
travis encrypt GITHUB_TOKEN=token
travis encrypt CACHIX_SIGNING_KEY=token

Then copy and paste the result into your .travis.yml file. Make sure you add the - so the field is treated as a list. Otherwise Travis will ignore one of your keys.

env:
  global:
    # github
    - secure: <enrypted-key-1>

    # cachix
    - secure: <encrypted-key-2>