guix.texi

(eq? (service-kind s) nginx-service-type)
@result{} #t
@end lisp

The @code{modify-services} form provides a handy way to change the
parameters of some of the services of a list such as
@code{%base-services} (@pxref{Base Services, @code{%base-services}}).  It
evaluates to a list of services.  Of course, you could always use
standard list combinators such as @code{map} and @code{fold} to do that
(@pxref{SRFI-1, List Library,, guile, GNU Guile Reference Manual});
@code{modify-services} simply provides a more concise form for this
common pattern.

@deffn {Scheme Syntax} modify-services @var{services} @
  (@var{type} @var{variable} => @var{body}) @dots{}

Modify the services listed in @var{services} according to the given
clauses.  Each clause has the form:

@example
(@var{type} @var{variable} => @var{body})
@end example

where @var{type} is a service type---e.g.,
@code{guix-service-type}---and @var{variable} is an identifier that is
bound within the @var{body} to the service parameters---e.g., a
@code{guix-configuration} instance---of the original service of that
@var{type}.

The @var{body} should evaluate to the new service parameters, which will
be used to configure the new service.  This new service will replace the
original in the resulting list.  Because a service's service parameters
are created using @code{define-record-type*}, you can write a succinct
@var{body} that evaluates to the new service parameters by using the
@code{inherit} feature that @code{define-record-type*} provides.

@xref{Using the Configuration System}, for example usage.

@end deffn

Next comes the programming interface for service types.  This is
something you want to know when writing new service definitions, but not
necessarily when simply looking for ways to customize your
@code{operating-system} declaration.

@deftp {Data Type} service-type
@cindex service type
This is the representation of a @dfn{service type} (@pxref{Service Types
and Services}).

@table @asis
@item @code{name}
This is a symbol, used only to simplify inspection and debugging.

@item @code{extensions}
A non-empty list of @code{<service-extension>} objects (see below).

@item @code{compose} (default: @code{#f})
If this is @code{#f}, then the service type denotes services that cannot
be extended---i.e., services that do not receive ``values'' from other
services.

Otherwise, it must be a one-argument procedure.  The procedure is called
by @code{fold-services} and is passed a list of values collected from
extensions.  It may return any single value.

@item @code{extend} (default: @code{#f})
If this is @code{#f}, services of this type cannot be extended.

Otherwise, it must be a two-argument procedure: @code{fold-services}
calls it, passing it the initial value of the service as the first
argument and the result of applying @code{compose} to the extension
values as the second argument.  It must return a value that is a valid
parameter value for the service instance.

@item @code{description}
This is a string, possibly using Texinfo markup, describing in a couple
of sentences what the service is about.  This string allows users to
find about the service through @command{guix system search}
(@pxref{Invoking guix system}).

@item @code{default-value} (default: @code{&no-default-value})
The default value associated for instances of this service type.  This
allows users to use the @code{service} form without its second argument:

@lisp
(service @var{type})
@end lisp

The returned service in this case has the default value specified by
@var{type}.
@end table

@xref{Service Types and Services}, for examples.
@end deftp

@deffn {Scheme Procedure} service-extension @var{target-type} @
                              @var{compute}
Return a new extension for services of type @var{target-type}.
@var{compute} must be a one-argument procedure: @code{fold-services}
calls it, passing it the value associated with the service that provides
the extension; it must return a valid value for the target service.
@end deffn

@deffn {Scheme Procedure} service-extension? @var{obj}
Return true if @var{obj} is a service extension.
@end deffn

Occasionally, you might want to simply extend an existing service.  This
involves creating a new service type and specifying the extension of
interest, which can be verbose; the @code{simple-service} procedure
provides a shorthand for this.

@deffn {Scheme Procedure} simple-service @var{name} @var{target} @var{value}
Return a service that extends @var{target} with @var{value}.  This works
by creating a singleton service type @var{name}, of which the returned
service is an instance.

For example, this extends mcron (@pxref{Scheduled Job Execution}) with
an additional job:

@lisp
(simple-service 'my-mcron-job mcron-service-type
                #~(job '(next-hour (3)) "guix gc -F 2G"))
@end lisp
@end deffn

At the core of the service abstraction lies the @code{fold-services}
procedure, which is responsible for ``compiling'' a list of services
down to a single directory that contains everything needed to boot and
run the system---the directory shown by the @command{guix system build}
command (@pxref{Invoking guix system}).  In essence, it propagates
service extensions down the service graph, updating each node parameters
on the way, until it reaches the root node.

@deffn {Scheme Procedure} fold-services @var{services} @
                            [#:target-type @var{system-service-type}]
Fold @var{services} by propagating their extensions down to the root of
type @var{target-type}; return the root service adjusted accordingly.
@end deffn

Lastly, the @code{(gnu services)} module also defines several essential
service types, some of which are listed below.

@defvr {Scheme Variable} system-service-type
This is the root of the service graph.  It produces the system directory
as returned by the @command{guix system build} command.
@end defvr

@defvr {Scheme Variable} boot-service-type
The type of the ``boot service'', which produces the @dfn{boot script}.
The boot script is what the initial RAM disk runs when booting.
@end defvr

@defvr {Scheme Variable} etc-service-type
The type of the @file{/etc} service.  This service is used to create
files under @file{/etc} and can be extended by
passing it name/file tuples such as:

@lisp
(list `("issue" ,(plain-file "issue" "Welcome!\n")))
@end lisp

In this example, the effect would be to add an @file{/etc/issue} file
pointing to the given file.
@end defvr

@defvr {Scheme Variable} setuid-program-service-type
Type for the ``setuid-program service''.  This service collects lists of
executable file names, passed as gexps, and adds them to the set of
setuid-root programs on the system (@pxref{Setuid Programs}).
@end defvr

@defvr {Scheme Variable} profile-service-type
Type of the service that populates the @dfn{system profile}---i.e., the
programs under @file{/run/current-system/profile}.  Other services can
extend it by passing it lists of packages to add to the system profile.
@end defvr

@cindex provenance tracking, of the operating system
@anchor{provenance-service-type}
@defvr {Scheme Variable} provenance-service-type
This is the type of the service that records @dfn{provenance meta-data}
in the system itself.  It creates several files under
@file{/run/current-system}:

@table @file
@item channels.scm
This is a ``channel file'' that can be passed to @command{guix pull -C}
or @command{guix time-machine -C}, and which describes the channels used
to build the system, if that information was available
(@pxref{Channels}).

@item configuration.scm
This is the file that was passed as the value for this
@code{provenance-service-type} service.  By default, @command{guix
system reconfigure} automatically passes the OS configuration file it
received on the command line.

@item provenance
This contains the same information as the two other files but in a
format that is more readily processable.
@end table

In general, these two pieces of information (channels and configuration
file) are enough to reproduce the operating system ``from source''.

@quotation Caveats
This information is necessary to rebuild your operating system, but it
is not always sufficient.  In particular, @file{configuration.scm}
itself is insufficient if it is not self-contained---if it refers to
external Guile modules or to extra files.  If you want
@file{configuration.scm} to be self-contained, we recommend that modules
or files it refers to be part of a channel.

Besides, provenance meta-data is ``silent'' in the sense that it does
not change the bits contained in your system, @emph{except for the
meta-data bits themselves}.  Two different OS configurations or sets of
channels can lead to the same system, bit-for-bit; when
@code{provenance-service-type} is used, these two systems will have
different meta-data and thus different store file names, which makes
comparison less trivial.
@end quotation

This service is automatically added to your operating system
configuration when you use @command{guix system reconfigure},
@command{guix system init}, or @command{guix deploy}.
@end defvr

@node Shepherd Services
@subsection Shepherd Services

@cindex shepherd services
@cindex PID 1
@cindex init system
The @code{(gnu services shepherd)} module provides a way to define
services managed by the GNU@tie{}Shepherd, which is the
initialization system---the first process that is started when the
system boots, also known as PID@tie{}1
(@pxref{Introduction,,, shepherd, The GNU Shepherd Manual}).

Services in the Shepherd can depend on each other.  For instance, the
SSH daemon may need to be started after the syslog daemon has been
started, which in turn can only happen once all the file systems have
been mounted.  The simple operating system defined earlier (@pxref{Using
the Configuration System}) results in a service graph like this:

@image{images/shepherd-graph,,5in,Typical shepherd service graph.}

You can actually generate such a graph for any operating system
definition using the @command{guix system shepherd-graph} command
(@pxref{system-shepherd-graph, @command{guix system shepherd-graph}}).

The @code{%shepherd-root-service} is a service object representing
PID@tie{}1, of type @code{shepherd-root-service-type}; it can be extended
by passing it lists of @code{<shepherd-service>} objects.

@deftp {Data Type} shepherd-service
The data type representing a service managed by the Shepherd.

@table @asis
@item @code{provision}
This is a list of symbols denoting what the service provides.

These are the names that may be passed to @command{herd start},
@command{herd status}, and similar commands (@pxref{Invoking herd,,,
shepherd, The GNU Shepherd Manual}).  @xref{Slots of services, the
@code{provides} slot,, shepherd, The GNU Shepherd Manual}, for details.

@item @code{requirement} (default: @code{'()})
List of symbols denoting the Shepherd services this one depends on.

@cindex one-shot services, for the Shepherd
@item @code{one-shot?} (default: @code{#f})
Whether this service is @dfn{one-shot}.  One-shot services stop immediately
after their @code{start} action has completed.  @xref{Slots of services,,,
shepherd, The GNU Shepherd Manual}, for more info.

@item @code{respawn?} (default: @code{#t})
Whether to restart the service when it stops, for instance when the
underlying process dies.

@item @code{start}
@itemx @code{stop} (default: @code{#~(const #f)})
The @code{start} and @code{stop} fields refer to the Shepherd's
facilities to start and stop processes (@pxref{Service De- and
Constructors,,, shepherd, The GNU Shepherd Manual}).  They are given as
G-expressions that get expanded in the Shepherd configuration file
(@pxref{G-Expressions}).

@item @code{actions} (default: @code{'()})
@cindex actions, of Shepherd services
This is a list of @code{shepherd-action} objects (see below) defining
@dfn{actions} supported by the service, in addition to the standard
@code{start} and @code{stop} actions.  Actions listed here become available as
@command{herd} sub-commands:

@example
herd @var{action} @var{service} [@var{arguments}@dots{}]
@end example

@item @code{auto-start?} (default: @code{#t})
Whether this service should be started automatically by the Shepherd.  If it
is @code{#f} the service has to be started manually with @code{herd start}.

@item @code{documentation}
A documentation string, as shown when running:

@example
herd doc @var{service-name}
@end example

where @var{service-name} is one of the symbols in @code{provision}
(@pxref{Invoking herd,,, shepherd, The GNU Shepherd Manual}).

@item @code{modules} (default: @code{%default-modules})
This is the list of modules that must be in scope when @code{start} and
@code{stop} are evaluated.

@end table
@end deftp

@deftp {Data Type} shepherd-action
This is the data type that defines additional actions implemented by a
Shepherd service (see above).

@table @code
@item name
Symbol naming the action.

@item documentation
This is a documentation string for the action.  It can be viewed by running:

@example
herd doc @var{service} action @var{action}
@end example

@item procedure
This should be a gexp that evaluates to a procedure of at least one argument,
which is the ``running value'' of the service (@pxref{Slots of services,,,
shepherd, The GNU Shepherd Manual}).
@end table

The following example defines an action called @code{say-hello} that kindly
greets the user:

@lisp
(shepherd-action
  (name 'say-hello)
  (documentation "Say hi!")
  (procedure #~(lambda (running . args)
                 (format #t "Hello, friend! arguments: ~s\n"
                         args)
                 #t)))
@end lisp

Assuming this action is added to the @code{example} service, then you can do:

@example
# herd say-hello example
Hello, friend! arguments: ()
# herd say-hello example a b c
Hello, friend! arguments: ("a" "b" "c")
@end example

This, as you can see, is a fairly sophisticated way to say hello.
@xref{Service Convenience,,, shepherd, The GNU Shepherd Manual}, for more
info on actions.
@end deftp

@defvr {Scheme Variable} shepherd-root-service-type
The service type for the Shepherd ``root service''---i.e., PID@tie{}1.

This is the service type that extensions target when they want to create
shepherd services (@pxref{Service Types and Services}, for an example).
Each extension must pass a list of @code{<shepherd-service>}.
@end defvr

@defvr {Scheme Variable} %shepherd-root-service
This service represents PID@tie{}1.
@end defvr


@node Documentation
@chapter Documentation

@cindex documentation, searching for
@cindex searching for documentation
@cindex Info, documentation format
@cindex man pages
@cindex manual pages
In most cases packages installed with Guix come with documentation.
There are two main documentation formats: ``Info'', a browseable
hypertext format used for GNU software, and ``manual pages'' (or ``man
pages''), the linear documentation format traditionally found on Unix.
Info manuals are accessed with the @command{info} command or with Emacs,
and man pages are accessed using @command{man}.

You can look for documentation of software installed on your system by
keyword.  For example, the following command searches for information
about ``TLS'' in Info manuals:

@example
$ info -k TLS
"(emacs)Network Security" -- STARTTLS
"(emacs)Network Security" -- TLS
"(gnutls)Core TLS API" -- gnutls_certificate_set_verify_flags
"(gnutls)Core TLS API" -- gnutls_certificate_set_verify_function
@dots{}
@end example

@noindent
The command below searches for the same keyword in man pages:

@example
$ man -k TLS
SSL (7)              - OpenSSL SSL/TLS library
certtool (1)         - GnuTLS certificate tool
@dots {}
@end example

These searches are purely local to your computer so you have the
guarantee that documentation you find corresponds to what you have
actually installed, you can access it off-line, and your privacy is
respected.

Once you have these results, you can view the relevant documentation by
running, say:

@example
$ info "(gnutls)Core TLS API"
@end example

@noindent
or:

@example
$ man certtool
@end example

Info manuals contain sections and indices as well as hyperlinks like
those found in Web pages.  The @command{info} reader (@pxref{Top, Info
reader,, info-stnd, Stand-alone GNU Info}) and its Emacs counterpart
(@pxref{Misc Help,,, emacs, The GNU Emacs Manual}) provide intuitive key
bindings to navigate manuals.  @xref{Getting Started,,, info, Info: An
Introduction}, for an introduction to Info navigation.

@node Installing Debugging Files
@chapter Installing Debugging Files

@cindex debugging files
Program binaries, as produced by the GCC compilers for instance, are
typically written in the ELF format, with a section containing
@dfn{debugging information}.  Debugging information is what allows the
debugger, GDB, to map binary code to source code; it is required to
debug a compiled program in good conditions.

The problem with debugging information is that is takes up a fair amount
of disk space.  For example, debugging information for the GNU C Library
weighs in at more than 60 MiB.  Thus, as a user, keeping all the
debugging info of all the installed programs is usually not an option.
Yet, space savings should not come at the cost of an impediment to
debugging---especially in the GNU system, which should make it easier
for users to exert their computing freedom (@pxref{GNU Distribution}).

Thankfully, the GNU Binary Utilities (Binutils) and GDB provide a
mechanism that allows users to get the best of both worlds: debugging
information can be stripped from the binaries and stored in separate
files.  GDB is then able to load debugging information from those files,
when they are available (@pxref{Separate Debug Files,,, gdb, Debugging
with GDB}).

The GNU distribution takes advantage of this by storing debugging
information in the @code{lib/debug} sub-directory of a separate package
output unimaginatively called @code{debug} (@pxref{Packages with
Multiple Outputs}).  Users can choose to install the @code{debug} output
of a package when they need it.  For instance, the following command
installs the debugging information for the GNU C Library and for GNU
Guile:

@example
guix install glibc:debug guile:debug
@end example

GDB must then be told to look for debug files in the user's profile, by
setting the @code{debug-file-directory} variable (consider setting it
from the @file{~/.gdbinit} file, @pxref{Startup,,, gdb, Debugging with
GDB}):

@example
(gdb) set debug-file-directory ~/.guix-profile/lib/debug
@end example

From there on, GDB will pick up debugging information from the
@file{.debug} files under @file{~/.guix-profile/lib/debug}.

In addition, you will most likely want GDB to be able to show the source
code being debugged.  To do that, you will have to unpack the source
code of the package of interest (obtained with @code{guix build
--source}, @pxref{Invoking guix build}), and to point GDB to that source
directory using the @code{directory} command (@pxref{Source Path,
@code{directory},, gdb, Debugging with GDB}).

@c XXX: keep me up-to-date
The @code{debug} output mechanism in Guix is implemented by the
@code{gnu-build-system} (@pxref{Build Systems}).  Currently, it is
opt-in---debugging information is available only for the packages
with definitions explicitly declaring a @code{debug} output.  This may be
changed to opt-out in the future if our build farm servers can handle
the load.  To check whether a package has a @code{debug} output, use
@command{guix package --list-available} (@pxref{Invoking guix package}).


@node Security Updates
@chapter Security Updates

@cindex security updates
@cindex security vulnerabilities
Occasionally, important security vulnerabilities are discovered in software
packages and must be patched.  Guix developers try hard to keep track of
known vulnerabilities and to apply fixes as soon as possible in the
@code{master} branch of Guix (we do not yet provide a ``stable'' branch
containing only security updates).  The @command{guix lint} tool helps
developers find out about vulnerable versions of software packages in the
distribution:

@smallexample
$ guix lint -c cve
gnu/packages/base.scm:652:2: glibc@@2.21: probably vulnerable to CVE-2015-1781, CVE-2015-7547
gnu/packages/gcc.scm:334:2: gcc@@4.9.3: probably vulnerable to CVE-2015-5276
gnu/packages/image.scm:312:2: openjpeg@@2.1.0: probably vulnerable to CVE-2016-1923, CVE-2016-1924
@dots{}
@end smallexample

@xref{Invoking guix lint}, for more information.

Guix follows a functional
package management discipline (@pxref{Introduction}), which implies
that, when a package is changed, @emph{every package that depends on it}
must be rebuilt.  This can significantly slow down the deployment of
fixes in core packages such as libc or Bash, since basically the whole
distribution would need to be rebuilt.  Using pre-built binaries helps
(@pxref{Substitutes}), but deployment may still take more time than
desired.

@cindex grafts
To address this, Guix implements @dfn{grafts}, a mechanism that allows
for fast deployment of critical updates without the costs associated
with a whole-distribution rebuild.  The idea is to rebuild only the
package that needs to be patched, and then to ``graft'' it onto packages
explicitly installed by the user and that were previously referring to
the original package.  The cost of grafting is typically very low, and
order of magnitudes lower than a full rebuild of the dependency chain.

@cindex replacements of packages, for grafts
For instance, suppose a security update needs to be applied to Bash.
Guix developers will provide a package definition for the ``fixed''
Bash, say @code{bash-fixed}, in the usual way (@pxref{Defining
Packages}).  Then, the original package definition is augmented with a
@code{replacement} field pointing to the package containing the bug fix:

@lisp
(define bash
  (package
    (name "bash")
    ;; @dots{}
    (replacement bash-fixed)))
@end lisp

From there on, any package depending directly or indirectly on Bash---as
reported by @command{guix gc --requisites} (@pxref{Invoking guix
gc})---that is installed is automatically ``rewritten'' to refer to
@code{bash-fixed} instead of @code{bash}.  This grafting process takes
time proportional to the size of the package, usually less than a
minute for an ``average'' package on a recent machine.  Grafting is
recursive: when an indirect dependency requires grafting, then grafting
``propagates'' up to the package that the user is installing.

Currently, the length of the name and version of the graft and that of
the package it replaces (@code{bash-fixed} and @code{bash} in the example
above) must be equal.  This restriction mostly comes from the fact that
grafting works by patching files, including binary files, directly.
Other restrictions may apply: for instance, when adding a graft to a
package providing a shared library, the original shared library and its
replacement must have the same @code{SONAME} and be binary-compatible.

The @option{--no-grafts} command-line option allows you to forcefully
avoid grafting (@pxref{Common Build Options, @option{--no-grafts}}).
Thus, the command:

@example
guix build bash --no-grafts
@end example

@noindent
returns the store file name of the original Bash, whereas:

@example
guix build bash
@end example

@noindent
returns the store file name of the ``fixed'', replacement Bash.  This
allows you to distinguish between the two variants of Bash.

To verify which Bash your whole profile refers to, you can run
(@pxref{Invoking guix gc}):

@example
guix gc -R `readlink -f ~/.guix-profile` | grep bash
@end example

@noindent
@dots{} and compare the store file names that you get with those above.
Likewise for a complete Guix system generation:

@example
guix gc -R `guix system build my-config.scm` | grep bash
@end example

Lastly, to check which Bash running processes are using, you can use the
@command{lsof} command:

@example
lsof | grep /gnu/store/.*bash
@end example


@node Bootstrapping
@chapter Bootstrapping

@c Adapted from the ELS 2013 paper.

@cindex bootstrapping

Bootstrapping in our context refers to how the distribution gets built
``from nothing''.  Remember that the build environment of a derivation
contains nothing but its declared inputs (@pxref{Introduction}).  So
there's an obvious chicken-and-egg problem: how does the first package
get built?  How does the first compiler get compiled?  Note that this is
a question of interest only to the curious hacker, not to the regular
user, so you can shamelessly skip this section if you consider yourself
a ``regular user''.

@cindex bootstrap binaries
The GNU system is primarily made of C code, with libc at its core.  The
GNU build system itself assumes the availability of a Bourne shell and
command-line tools provided by GNU Coreutils, Awk, Findutils, `sed', and
`grep'.  Furthermore, build programs---programs that run
@code{./configure}, @code{make}, etc.---are written in Guile Scheme
(@pxref{Derivations}).  Consequently, to be able to build anything at
all, from scratch, Guix relies on pre-built binaries of Guile, GCC,
Binutils, libc, and the other packages mentioned above---the
@dfn{bootstrap binaries}.

These bootstrap binaries are ``taken for granted'', though we can also
re-create them if needed (@pxref{Preparing to Use the Bootstrap
Binaries}).

@menu
* Reduced Binary Seed Bootstrap::  A Bootstrap worthy of GNU.
* Preparing to Use the Bootstrap Binaries:: Building that what matters most.
@end menu

@node Reduced Binary Seed Bootstrap
@section The Reduced Binary Seed Bootstrap

Guix---like other GNU/Linux distributions---is traditionally bootstrapped from
a set of bootstrap binaries: Bourne shell, command-line tools provided by GNU
Coreutils, Awk, Findutils, `sed', and `grep' and Guile, GCC, Binutils, and the
GNU C Library (@pxref{Bootstrapping}).  Usually, these bootstrap binaries are
``taken for granted.''

Taking the bootstrap binaries for granted means that we consider them to
be a correct and trustworthy ``seed'' for building the complete system.
Therein lies a problem: the combined size of these bootstrap binaries is
about 250MB (@pxref{Bootstrappable Builds,,, mes, GNU Mes}).  Auditing
or even inspecting these is next to impossible.

For @code{i686-linux} and @code{x86_64-linux}, Guix now features a
``Reduced Binary Seed'' bootstrap @footnote{We would like to say: ``Full
Source Bootstrap'' and while we are working towards that goal it would
be hyperbole to use that term for what we do now.}.

The Reduced Binary Seed bootstrap removes the most critical tools---from a
trust perspective---from the bootstrap binaries: GCC, Binutils and the GNU C
Library are replaced by: @code{bootstrap-mescc-tools} (a tiny assembler and
linker) and @code{bootstrap-mes} (a small Scheme Interpreter and a C compiler
written in Scheme and the Mes C Library, built for TinyCC and for GCC).

Using these new binary seeds the ``missing'' Binutils, GCC, and the GNU
C Library are built from source.  From here on the more traditional
bootstrap process resumes.  This approach has reduced the bootstrap
binaries in size to about 145MB in Guix v1.1.

The next step that Guix has taken is to replace the shell and all its
utilities with implementations in Guile Scheme, the @emph{Scheme-only
bootstrap}.  Gash (@pxref{Gash,,, gash, The Gash manual}) is a
POSIX-compatible shell that replaces Bash, and it comes with Gash Utils
which has minimalist replacements for Awk, the GNU Core Utilities, Grep,
Gzip, Sed, and Tar.  The rest of the bootstrap binary seeds that were
removed are now built from source.

Building the GNU System from source is currently only possibly by adding
some historical GNU packages as intermediate steps@footnote{Packages
such as @code{gcc-2.95.3}, @code{binutils-2.14}, @code{glibc-2.2.5},
@code{gzip-1.2.4}, @code{tar-1.22}, and some others.  For details, see
@file{gnu/packages/commencement.scm}.}.  As Gash and Gash Utils mature,
and GNU packages become more bootstrappable again (e.g., new releases of
GNU Sed will also ship as gzipped tarballs again, as alternative to the
hard to bootstrap @code{xz}-compression), this set of added packages can
hopefully be reduced again.

The graph below shows the resulting dependency graph for
@code{gcc-core-mesboot0}, the bootstrap compiler used for the
traditional bootstrap of the rest of the Guix System.

@c ./pre-inst-env guix graph -e '(@@ (gnu packages commencement) gcc-core-mesboot0)' | sed -re 's,((bootstrap-mescc-tools|bootstrap-mes|guile-bootstrap).*shape =) box,\1 ellipse,' > doc/images/gcc-core-mesboot0-graph.dot
@image{images/gcc-core-mesboot0-graph,6in,,Dependency graph of gcc-core-mesboot0}

The only significant binary bootstrap seeds that remain@footnote{
Ignoring the 68KB @code{mescc-tools}; that will be removed later,
together with @code{mes}.} are a Scheme intepreter and a Scheme
compiler: GNU Mes and GNU Guile@footnote{Not shown in this graph are the
static binaries for @file{bash}, @code{tar}, and @code{xz} that are used
to get Guile running.}.

This further reduction has brought down the size of the binary seed to
about 60MB for @code{i686-linux} and @code{x86_64-linux}.

Work is ongoing to remove all binary blobs from our free software
bootstrap stack, working towards a Full Source Bootstrap.  Also ongoing
is work to bring these bootstraps to the @code{arm-linux} and
@code{aarch64-linux} architectures and to the Hurd.

If you are interested, join us on @samp{#bootstrappable} on the Freenode
IRC network or discuss on @email{bug-mes@@gnu.org} or
@email{gash-devel@@nongnu.org}.

@node Preparing to Use the Bootstrap Binaries
@section Preparing to Use the Bootstrap Binaries

@c As of Emacs 24.3, Info-mode displays the image, but since it's a
@c large image, it's hard to scroll.  Oh well.
@image{images/bootstrap-graph,6in,,Dependency graph of the early bootstrap derivations}

The figure above shows the very beginning of the dependency graph of the
distribution, corresponding to the package definitions of the @code{(gnu
packages bootstrap)} module.  A similar figure can be generated with
@command{guix graph} (@pxref{Invoking guix graph}), along the lines of:

@example
guix graph -t derivation \
  -e '(@@@@ (gnu packages bootstrap) %bootstrap-gcc)' \
  | dot -Tps > gcc.ps
@end example

or, for the further Reduced Binary Seed bootstrap

@example
guix graph -t derivation \
  -e '(@@@@ (gnu packages bootstrap) %bootstrap-mes)' \
  | dot -Tps > mes.ps
@end example

At this level of detail, things are
slightly complex.  First, Guile itself consists of an ELF executable,
along with many source and compiled Scheme files that are dynamically
loaded when it runs.  This gets stored in the @file{guile-2.0.7.tar.xz}
tarball shown in this graph.  This tarball is part of Guix's ``source''
distribution, and gets inserted into the store with @code{add-to-store}
(@pxref{The Store}).

But how do we write a derivation that unpacks this tarball and adds it
to the store?  To solve this problem, the @code{guile-bootstrap-2.0.drv}
derivation---the first one that gets built---uses @code{bash} as its
builder, which runs @code{build-bootstrap-guile.sh}, which in turn calls
@code{tar} to unpack the tarball.  Thus, @file{bash}, @file{tar},
@file{xz}, and @file{mkdir} are statically-linked binaries, also part of
the Guix source distribution, whose sole purpose is to allow the Guile
tarball to be unpacked.

Once @code{guile-bootstrap-2.0.drv} is built, we have a functioning
Guile that can be used to run subsequent build programs.  Its first task
is to download tarballs containing the other pre-built binaries---this
is what the @file{.tar.xz.drv} derivations do.  Guix modules such as
@code{ftp-client.scm} are used for this purpose.  The
@code{module-import.drv} derivations import those modules in a directory
in the store, using the original layout.  The
@code{module-import-compiled.drv} derivations compile those modules, and
write them in an output directory with the right layout.  This
corresponds to the @code{#:modules} argument of
@code{build-expression->derivation} (@pxref{Derivations}).

Finally, the various tarballs are unpacked by the derivations
@code{gcc-bootstrap-0.drv}, @code{glibc-bootstrap-0.drv}, or
@code{bootstrap-mes-0.drv} and @code{bootstrap-mescc-tools-0.drv}, at which
point we have a working C tool chain.

@unnumberedsec Building the Build Tools

Bootstrapping is complete when we have a full tool chain that does not
depend on the pre-built bootstrap tools discussed above.  This
no-dependency requirement is verified by checking whether the files of
the final tool chain contain references to the @file{/gnu/store}
directories of the bootstrap inputs.  The process that leads to this
``final'' tool chain is described by the package definitions found in
the @code{(gnu packages commencement)} module.

The @command{guix graph} command allows us to ``zoom out'' compared to
the graph above, by looking at the level of package objects instead of
individual derivations---remember that a package may translate to
several derivations, typically one derivation to download its source,
one to build the Guile modules it needs, and one to actually build the
package from source.  The command:

@example
guix graph -t bag \
  -e '(@@@@ (gnu packages commencement)
          glibc-final-with-bootstrap-bash)' | xdot -
@end example

@noindent
displays the dependency graph leading to the ``final'' C
library@footnote{You may notice the @code{glibc-intermediate} label,
suggesting that it is not @emph{quite} final, but as a good
approximation, we will consider it final.}, depicted below.

@image{images/bootstrap-packages,6in,,Dependency graph of the early packages}

@c See <https://lists.gnu.org/archive/html/gnu-system-discuss/2012-10/msg00000.html>.
The first tool that gets built with the bootstrap binaries is
GNU@tie{}Make---noted @code{make-boot0} above---which is a prerequisite
for all the following packages.  From there Findutils and Diffutils get
built.

Then come the first-stage Binutils and GCC, built as pseudo cross
tools---i.e., with @option{--target} equal to @option{--host}.  They are
used to build libc.  Thanks to this cross-build trick, this libc is
guaranteed not to hold any reference to the initial tool chain.

From there the final Binutils and GCC (not shown above) are built.  GCC
uses @command{ld} from the final Binutils, and links programs against
the just-built libc.  This tool chain is used to build the other
packages used by Guix and by the GNU Build System: Guile, Bash,
Coreutils, etc.

And voilà!  At this point we have the complete set of build tools that
the GNU Build System expects.  These are in the @code{%final-inputs}
variable of the @code{(gnu packages commencement)} module, and are
implicitly used by any package that uses @code{gnu-build-system}
(@pxref{Build Systems, @code{gnu-build-system}}).


@unnumberedsec Building the Bootstrap Binaries

@cindex bootstrap binaries
Because the final tool chain does not depend on the bootstrap binaries,
those rarely need to be updated.  Nevertheless, it is useful to have an
automated way to produce them, should an update occur, and this is what
the @code{(gnu packages make-bootstrap)} module provides.

The following command builds the tarballs containing the bootstrap binaries
(Binutils, GCC, glibc, for the traditional bootstrap and linux-libre-headers,
bootstrap-mescc-tools, bootstrap-mes for the Reduced Binary Seed bootstrap,
and Guile, and a tarball containing a mixture of Coreutils and other basic
command-line tools):

@example
guix build bootstrap-tarballs
@end example

The generated tarballs are those that should be referred to in the
@code{(gnu packages bootstrap)} module mentioned at the beginning of
this section.

Still here?  Then perhaps by now you've started to wonder: when do we
reach a fixed point?  That is an interesting question!  The answer is
unknown, but if you would like to investigate further (and have
significant computational and storage resources to do so), then let us
know.

@unnumberedsec Reducing the Set of Bootstrap Binaries

Our traditional bootstrap includes GCC, GNU Libc, Guile, etc.  That's a lot of
binary code!  Why is that a problem?  It's a problem because these big chunks
of binary code are practically non-auditable, which makes it hard to establish
what source code produced them.  Every unauditable binary also leaves us
vulnerable to compiler backdoors as described by Ken Thompson in the 1984
paper @emph{Reflections on Trusting Trust}.

This is mitigated by the fact that our bootstrap binaries were generated
from an earlier Guix revision.  Nevertheless it lacks the level of
transparency that we get in the rest of the package dependency graph,
where Guix always gives us a source-to-binary mapping.  Thus, our goal
is to reduce the set of bootstrap binaries to the bare minimum.

The @uref{https://bootstrappable.org, Bootstrappable.org web site} lists
on-going projects to do that.  One of these is about replacing the
bootstrap GCC with a sequence of assemblers, interpreters, and compilers
of increasing complexity, which could be built from source starting from
a simple and auditable assembler.

Our first major achievement is the replacement of of GCC, the GNU C Library
and Binutils by MesCC-Tools (a simple hex linker and macro assembler) and Mes
(@pxref{Top, GNU Mes Reference Manual,, mes, GNU Mes}, a Scheme interpreter
and C compiler in Scheme).  Neither MesCC-Tools nor Mes can be fully
bootstrapped yet and thus we inject them as binary seeds.  We call this the
Reduced Binary Seed bootstrap, as it has halved the size of our bootstrap
binaries!  Also, it has eliminated the C compiler binary; i686-linux and
x86_64-linux Guix packages are now bootstrapped without any binary C compiler.

Work is ongoing to make MesCC-Tools and Mes fully bootstrappable and we are
also looking at any other bootstrap binaries.  Your help is welcome!

@node Porting
@chapter Porting to a New Platform

As discussed above, the GNU distribution is self-contained, and
self-containment is achieved by relying on pre-built ``bootstrap
binaries'' (@pxref{Bootstrapping}).  These binaries are specific to an
operating system kernel, CPU architecture, and application binary
interface (ABI).  Thus, to port the distribution to a platform that is
not yet supported, one must build those bootstrap binaries, and update
the @code{(gnu packages bootstrap)} module to use them on that platform.

Fortunately, Guix can @emph{cross compile} those bootstrap binaries.
When everything goes well, and assuming the GNU tool chain supports the
target platform, this can be as simple as running a command like this
one:

@example
guix build --target=armv5tel-linux-gnueabi bootstrap-tarballs
@end example

For this to work, the @code{glibc-dynamic-linker} procedure in
@code{(gnu packages bootstrap)} must be augmented to return the right
file name for libc's dynamic linker on that platform; likewise,
@code{system->linux-architecture} in @code{(gnu packages linux)} must be
taught about the new platform.

Once these are built, the @code{(gnu packages bootstrap)} module needs
to be updated to refer to these binaries on the target platform.  That
is, the hashes and URLs of the bootstrap tarballs for the new platform
must be added alongside those of the currently supported platforms.  The
bootstrap Guile tarball is treated specially: it is expected to be
available locally, and @file{gnu/local.mk} has rules to download it for
the supported architectures; a rule for the new platform must be added
as well.

In practice, there may be some complications.  First, it may be that the
extended GNU triplet that specifies an ABI (like the @code{eabi} suffix
above) is not recognized by all the GNU tools.  Typically, glibc
recognizes some of these, whereas GCC uses an extra @option{--with-abi}
configure flag (see @code{gcc.scm} for examples of how to handle this).
Second, some of the required packages could fail to build for that
platform.  Lastly, the generated binaries could be broken for some
reason.

@c *********************************************************************
@include contributing.texi

@c *********************************************************************
@node Acknowledgments
@chapter Acknowledgments

Guix is based on the @uref{https://nixos.org/nix/, Nix package manager},
which was designed and
implemented by Eelco Dolstra, with contributions from other people (see
the @file{nix/AUTHORS} file in Guix).  Nix pioneered functional package
management, and promoted unprecedented features, such as transactional
package upgrades and rollbacks, per-user profiles, and referentially
transparent build processes.  Without this work, Guix would not exist.

The Nix-based software distributions, Nixpkgs and NixOS, have also been
an inspiration for Guix.

GNU@tie{}Guix itself is a collective work with contributions from a
number of people.  See the @file{AUTHORS} file in Guix for more
information on these fine people.  The @file{THANKS} file lists people
who have helped by reporting bugs, taking care of the infrastructure,
providing artwork and themes, making suggestions, and more---thank you!


@c *********************************************************************
@node GNU Free Documentation License
@appendix GNU Free Documentation License
@cindex license, GNU Free Documentation License
@include fdl-1.3.texi

@c *********************************************************************
@node Concept Index
@unnumbered Concept Index
@printindex cp

@node Programming Index
@unnumbered Programming Index
@syncodeindex tp fn
@syncodeindex vr fn