Invoking guix challenge (GNU Guix Reference Manual)

Do the binaries provided by this server really correspond to the source code it claims to build? Is a package build process deterministic? These are the questions the guix challenge command attempts to answer.

The former is obviously an important question: Before using a substitute server (see Substitutes), one had better verify that it provides the right binaries, and thus challenge it. The latter is what enables the former: If package builds are deterministic, then independent builds of the package should yield the exact same result, bit for bit; if a server provides a binary different from the one obtained locally, it may be either corrupt or malicious.

We know that the hash that shows up in /gnu/store file names is the hash of all the inputs of the process that built the file or directory—compilers, libraries, build scripts, etc. (see Introduction). Assuming deterministic build processes, one store file name should map to exactly one build output. guix challenge checks whether there is, indeed, a single mapping by comparing the build outputs of several independent builds of any given store item.

$ guix challenge --substitute-urls="https://ci.guix.gnu.org https://guix.example.org"
updating list of substitutes from 'https://ci.guix.gnu.org'... 100.0%
updating list of substitutes from 'https://guix.example.org'... 100.0%
/gnu/store/…-openssl-1.0.2d contents differ:
  local hash: 0725l22r5jnzazaacncwsvp9kgf42266ayyp814v7djxs7nk963q
  https://ci.guix.gnu.org/nar/…-openssl-1.0.2d: 0725l22r5jnzazaacncwsvp9kgf42266ayyp814v7djxs7nk963q
  https://guix.example.org/nar/…-openssl-1.0.2d: 1zy4fmaaqcnjrzzajkdn3f5gmjk754b43qkq47llbyak9z0qjyim
  differing files:
    /lib/libcrypto.so.1.1
    /lib/libssl.so.1.1

/gnu/store/…-git-2.5.0 contents differ:
  local hash: 00p3bmryhjxrhpn2gxs2fy0a15lnip05l97205pgbk5ra395hyha
  https://ci.guix.gnu.org/nar/…-git-2.5.0: 069nb85bv4d4a6slrwjdy8v1cn4cwspm3kdbmyb81d6zckj3nq9f
  https://guix.example.org/nar/…-git-2.5.0: 0mdqa9w1p6cmli6976v4wi0sw9r4p5prkj7lzfd1877wk11c9c73
  differing file:
    /libexec/git-core/git-fsck

/gnu/store/…-pius-2.1.1 contents differ:
  local hash: 0k4v3m9z1zp8xzzizb7d8kjj72f9172xv078sq4wl73vnq9ig3ax
  https://ci.guix.gnu.org/nar/…-pius-2.1.1: 0k4v3m9z1zp8xzzizb7d8kjj72f9172xv078sq4wl73vnq9ig3ax
  https://guix.example.org/nar/…-pius-2.1.1: 1cy25x1a4fzq5rk0pmvc8xhwyffnqz95h2bpvqsz2mpvlbccy0gs
  differing file:
    /share/man/man1/pius.1.gz

…

6,406 store items were analyzed:
  - 4,749 (74.1%) were identical
  - 525 (8.2%) differed
  - 1,132 (17.7%) were inconclusive

In this example, guix challenge first scans the store to determine the set of locally-built derivations—as opposed to store items that were downloaded from a substitute server—and then queries all the substitute servers. It then reports those store items for which the servers obtained a result different from the local build.

As an example, guix.example.org always gets a different answer. Conversely, ci.guix.gnu.org agrees with local builds, except in the case of Git. This might indicate that the build process of Git is non-deterministic, meaning that its output varies as a function of various things that Guix does not fully control, in spite of building packages in isolated environments (see Features). Most common sources of non-determinism include the addition of timestamps in build results, the inclusion of random numbers, and directory listings sorted by inode number. See https://reproducible-builds.org/docs/, for more information.

guix challenge git \
  --diff=diffoscope \
  --substitute-urls="https://ci.guix.gnu.org https://guix.example.org"

This automatically invokes diffoscope, which displays detailed information about files that differ.

$ wget -q -O - https://ci.guix.gnu.org/nar/lzip/…-git-2.5.0 \
   | lzip -d | guix archive -x /tmp/git
$ diff -ur --no-dereference /gnu/store/…-git.2.5.0 /tmp/git

This command shows the difference between the files resulting from the local build, and the files resulting from the build on ci.guix.gnu.org (see Comparing and Merging Files in Comparing and Merging Files). The diff command works great for text files. When binary files differ, a better option is Diffoscope, a tool that helps visualize differences for all kinds of files.

Once you have done that work, you can tell whether the differences are due to a non-deterministic build process or to a malicious server. We try hard to remove sources of non-determinism in packages to make it easier to verify substitutes, but of course, this is a process that involves not just Guix, but a large part of the free software community. In the meantime, guix challenge is one tool to help address the problem.

If you are writing packages for Guix, you are encouraged to check whether ci.guix.gnu.org and other substitute servers obtain the same build result as you did with:

$ guix challenge package

where package is a package specification such as guile@2.0 or glibc:debug.

guix challenge options [packages…]

When a difference is found between the hash of a locally-built item and that of a server-provided substitute, or among substitutes provided by different servers, the command displays it as in the example above and its exit code is 2 (other non-zero exit codes denote other kinds of errors).

9.11 Invoking guix challenge

9.11 Invoking `guix challenge`