Intro to Nix and NixOS

Philip Potter / @philandstuff

philandstuff.github.io/intro-to-nix/

UK Government Digital Service

What is Nix? What is NixOS?

Nix is a package manager

NixOS is a distro based on Nix

What is a package manager?

Which of these tools are package managers?

rpm, dpkg, rubygems, bundler, zip, docker, rsync?

Package manager (n.): a tool that deploys software correctly

(under certain assumptions)

rsync: the null package manager

Let's not kid ourselves: "deployment" is just "copying some files"

rsync can just as easily deploy incorrectly as correctly

What is correct deployment?

given identical inputs, the software should behave the same on an end-user machine as on the developer machine

-- from http://nixos.org/~eelco/pubs/phd-thesis.pdf, section 1.1

What goes wrong with deployments?

Missing dependencies

Wrong versions of dependencies

Dependency conflicts

Missing dependencies

Outside scope of package manager (libyaml, libxml for ruby or python)

Accidentally missed by package (don't all machines have zlib? openssl?)

Wrong versions of dependencies

"It works on my machine"

CI and production don't have same version of library

Conflicting dependencies

My app was working fine, until someone deployed something that needed a later version of libfoo

Now my app is broken because of the libfoo change

DLL hell, Cabal hell

Installing or upgrading one package shouldn't break other packages

Basic principles

Use cases

Arbitrary software (not limited by language)

Allow multiple package versions

Allow unprivileged users to install software

Dependencies

Identify packages by cryptographic hash

Hardcode the specific hash of the package version in the built artefact

The nix store

A content-addressable store, where packages have paths based on cryptographic hash

Once a package is created, it can't be changed (because the hash would have to change)

Structure of a nix path

/nix/store/n0a1y0yd54sh10p7rdi0alysscvac5x5-certificate-transparency-2016-01-14/bin/ct

Locating dependencies

Where other binaries might search for libjson in /usr/lib or /usr/local/lib, we search in /nix/store/6hcccvdx29r4j8n0wxl85b6mlmq0gvs1-libjson-7.6.1, and nowhere else

Only the exact same version is acceptable

Showing dependencies

$ nix-store --query --references /nix/store/n0a1y0yd54sh10p7rdi0alysscvac5x5-certificate-transparency-2016-01-14

$ nix-store --query --requisites /nix/store/n0a1y0yd54sh10p7rdi0alysscvac5x5-certificate-transparency-2016-01-14

How to deploy a package:

  • Identify the package's dependencies (by hash)
  • Correctly deploy each of the package's dependencies (recursively if necessary)
  • Copy the package itself

This works for any target machine (with matching arch & kernel): Ubuntu, RHEL, CentOS...

Command: nix-copy-closure

Avoids conflicts like an omnibus package

Shares common dependencies like a regular package manager

Aside: FHS considered harmful

FHS is about guessing

Whatever is at /usr/lib/libjson.so.1 is fine

v1.0.0? v1.52.65? v1.1.3-random-fork?

This is totally against the goal of correct deployment!

Making a nix package

A worked example: sassc

An example of ./configure ; make ; make install

{ stdenv, fetchurl, autoreconfHook, libsass }:

stdenv.mkDerivation rec {
  name = "sassc-${version}";
  version = "3.3.2";

  src = fetchurl {
    url = "https://github.com/sass/sassc/archive/${version}.tar.gz";
    sha256 = "15a2b2698639dfdc7bd6a5ba7a9ecdaf8ebb9f15503fb04dea1be3133308e41d";
  };

  patchPhase = ''
    export SASSC_VERSION=${version}
  '';

  nativeBuildInputs = [ autoreconfHook ];

  buildInputs = [ libsass ];

  meta = with stdenv.lib; {
    description = "A front-end for libsass";
    homepage = https://github.com/sass/sassc/;
    license = licenses.mit;
    maintainers = with maintainers; [ codyopel pjones ];
    platforms = platforms.unix;
  };
}

{ stdenv, fetchurl, autoreconfHook, libsass }:

stdenv.mkDerivation rec {
  # ...
}

{ stdenv, fetchurl, autoreconfHook, libsass }:

stdenv.mkDerivation rec {
  name = "sassc-${version}";
  version = "3.3.2";

  src = fetchurl {
    url = "https://github.com/sass/sassc/archive/${version}.tar.gz";
    sha256 = "15a2b2698639dfdc7bd6a5ba7a9ecdaf8ebb9f15503fb04dea1be3133308e41d";
  };

  # ...
}

{ stdenv, fetchurl, autoreconfHook, libsass }:

stdenv.mkDerivation rec {
  name = "sassc-${version}";
  version = "3.3.2";

  src = fetchurl # ...;

  patchPhase = ''
    export SASSC_VERSION=${version}
  '';

  nativeBuildInputs = [ autoreconfHook ];

  buildInputs = [ libsass ];

  # ...;
}

{ stdenv, fetchurl, autoreconfHook, libsass }:

stdenv.mkDerivation rec {
  name = "sassc-${version}";
  version = "3.3.2";

  src = fetchurl # ...;

  # ...;

  meta = with stdenv.lib; {
    description = "A front-end for libsass";
    homepage = https://github.com/sass/sassc/;
    license = licenses.mit;
    maintainers = with maintainers; [ codyopel pjones ];
    platforms = platforms.unix;
  };
}

demo

nix-shell --pure -A pkgs.sassc

env | grep libsass

A non-autoconf package: ponysay

{ stdenv, fetchurl, python3, texinfo, makeWrapper }:
stdenv.mkDerivation rec {
  buildInputs = [ python3 texinfo makeWrapper ];

  phases = "unpackPhase installPhase fixupPhase";

  installPhase = ''
    find -type f -name "*.py" | xargs sed -i "s@/usr/bin/env python3@$python3/bin/python3@g"
    substituteInPlace setup.py --replace \
        "fileout.write(('#!/usr/bin/env %s\n' % env).encode('utf-8'))" \
        "fileout.write(('#!%s/bin/%s\n' % (os.environ['python3'], env)).encode('utf-8'))"
    python3 setup.py --prefix=$out --freedom=partial install \
        --with-shared-cache=$out/share/ponysay \
        --with-bash
  '';
}

evilvte: compiled-in configuration

{ stdenv, fetchgit, makeWrapper, ... configH}:

stdenv.mkDerivation rec {
  # ...;

  buildPhase = ''
    cat >src/config.h <<EOF
    ${configH}
    EOF
    make
  '';
}

Glossary so far

Nix store
A read-only content-addressable filesystem
Store path
A path in the nix store, identified by the hash of its contents (a "package")
Derivation
A recipe for constructing a store path
Building
Creating a store path from a derivation
Closure
A store path and all of its dependencies needed to function correctly

Nixpkgs and Hydra

nixpkgs: the nix package collection

https://github.com/NixOS/nixpkgs

76k commits, 696 contributors

21-28 Jan: 59 PRs merged by 38 people

Substitutions

Nix derivations are defined in terms of source code

Building a derivation can be skipped by substitution from a prebuilt binary cache

hydra: the nix continuous build server

every new commit to nixpkgs is built on a build farm

binaries made available at cache.nixos.org substitution server

Profiles and environments

An environment is a set of installed packages

With corresponding environment variables PATH, LD_LIBRARY_PATH, PKG_CONFIG_PATH...

Multiple environments can coexist (because multiple packages are allowed)

A profile is a set of environments over time

Each environment is called a "generation"

Each operation on a profile creates a new generation

Each user can have their own profile

Profile operations

  • Install packages
  • Upgrade packages
  • Remove packages
  • Rollback to a previous generation

Aside: mass upgrades

Ever upgraded major distro release (eg 14.10 to 15.04)?

How do you recover?

Rollbacks save you here!

Development environments

An environment can also be created for a particular project context

Minimum necessary dependencies for project

We already saw this with nix-shell for building sassc

NixOS

Nix vs NixOS

Nix is just a package manager

Can target linux, OS X, freebsd...

NixOS is a linux distribution built on Nix

Store paths revisited

A store path doesn't have to be a directory

It can just be a single static file

/nix/store/mi14q2q22c1fvj4ck79q51j315yb9fr6-etc-hosts

Store paths revisited

A derivation can take store paths as dependencies and create symlinks to them

Store paths revisited

Imagine this directory:

$ ls -l
lrwxrwxrwx bashrc -> /nix/store/r6i1gl85qch03qn7w5w0h5by3nz654c1-etc-bashrc
lrwxrwxrwx fstab -> /nix/store/9rf80psk56v90r23kgii9nmbhdxiq86r-etc-fstab
lrwxrwxrwx hosts -> /nix/store/mi14q2q22c1fvj4ck79q51j315yb9fr6-etc-hosts
lrwxrwxrwx issue -> /nix/store/i233z50axwxgbkf2n5hrqrmpq2ndjhi1-issue
lrwxrwxrwx locale.conf -> /nix/store/ldnwp9g7x6v6g5ms69n8zjy5j39lb6vj-locale.conf
lrwxrwxrwx localtime -> /nix/store/j7hg0289hdcmw7fby9yhcklf50791c8z-tzdata-2015g/share/zoneinfo/Europe/London
lrwxrwxrwx os-release -> /nix/store/yn0p31bvgra01wnm2gnrali6dbh9k0gj-etc-os-release
lrwxrwxrwx profile -> /nix/store/9mj0i0psq7b27vvf7inw87ahgnfw1aq8-etc-profile
lrwxrwxrwx protocols -> /nix/store/4dqzkdz6699zh8bck82jl19mv759n37v-iana-etc-2.30/etc/protocols
lrwxrwxrwx resolvconf.conf -> /nix/store/qjmv1x2j1sp9m61k08xpb7ppqa92qxgs-etc-resolvconf.conf
lrwxrwxrwx services -> /nix/store/4dqzkdz6699zh8bck82jl19mv759n37v-iana-etc-2.30/etc/services
lrwxrwxrwx shells -> /nix/store/4b6n2yyx69ba0km3jajyjbms4cggjjc6-etc-shells
lrwxrwxrwx sudoers -> /nix/store/5656vj85srl1vhkywbhsmcpxpki77mpn-sudoers
lrwxrwxrwx vconsole.conf -> /nix/store/z18ipnvjp88r6iy8gydvxxlakd272vyh-vconsole.conf
lrwxrwxrwx zoneinfo -> /nix/store/j7hg0289hdcmw7fby9yhcklf50791c8z-tzdata-2015g/share/zoneinfo

NixOS: whole system configuration as a nix expression

Populate /etc with links to the nix store

A single toplevel nix expression defines the system

The system profile

/etc is a profile (if you squint)

You can update the toplevel configuration and rebuild the system

You can also rollback the system

(from the GRUB prompt if necessary)

Warning labels

Security updates

Existing package managers

Summary

Package management is about deployment

Deployment is about dependency management

Avoid conflicts, version mismatches by design

Thanks!

Philip Potter / @philandstuff

philandstuff.github.io/intro-to-nix/

UK Government Digital Service