guix.texi

@cindex substituter
Users can obtain the new package definition simply by running
@command{guix pull} (@pxref{Invoking guix pull}).  When
@code{hydra.gnu.org} is done building the package, installing the
package automatically downloads binaries from there
(@pxref{Substitutes}).  The only place where human intervention is
needed is to review and apply the patch.


@menu
* Software Freedom::            What may go into the distribution.
* Package Naming::              What's in a name?
* Version Numbers::             When the name is not enough.
* Synopses and Descriptions::   Helping users find the right package.
* Python Modules::              Taming the snake.
* Perl Modules::                Little pearls.
* Java Packages::               Coffee break.
* Fonts::                       Fond of fonts.
@end menu

@node Software Freedom
@subsection Software Freedom

@c Adapted from http://www.gnu.org/philosophy/philosophy.html.

The GNU operating system has been developed so that users can have
freedom in their computing.  GNU is @dfn{free software}, meaning that
users have the @url{http://www.gnu.org/philosophy/free-sw.html,four
essential freedoms}: to run the program, to study and change the program
in source code form, to redistribute exact copies, and to distribute
modified versions.  Packages found in the GNU distribution provide only
software that conveys these four freedoms.

In addition, the GNU distribution follow the
@url{http://www.gnu.org/distros/free-system-distribution-guidelines.html,free
software distribution guidelines}.  Among other things, these guidelines
reject non-free firmware, recommendations of non-free software, and
discuss ways to deal with trademarks and patents.

Some otherwise free upstream package sources contain a small and optional
subset that violates the above guidelines, for instance because this subset
is itself non-free code.  When that happens, the offending items are removed
with appropriate patches or code snippets in the @code{origin} form of the
package (@pxref{Defining Packages}).  This way, @code{guix
build --source} returns the ``freed'' source rather than the unmodified
upstream source.


@node Package Naming
@subsection Package Naming

A package has actually two names associated with it:
First, there is the name of the @emph{Scheme variable}, the one following
@code{define-public}.  By this name, the package can be made known in the
Scheme code, for instance as input to another package.  Second, there is
the string in the @code{name} field of a package definition.  This name
is used by package management commands such as
@command{guix package} and @command{guix build}.

Both are usually the same and correspond to the lowercase conversion of
the project name chosen upstream, with underscores replaced with
hyphens.  For instance, GNUnet is available as @code{gnunet}, and
SDL_net as @code{sdl-net}.

We do not add @code{lib} prefixes for library packages, unless these are
already part of the official project name.  But @pxref{Python
Modules} and @ref{Perl Modules} for special rules concerning modules for
the Python and Perl languages.

Font package names are handled differently, @pxref{Fonts}.


@node Version Numbers
@subsection Version Numbers

We usually package only the latest version of a given free software
project.  But sometimes, for instance for incompatible library versions,
two (or more) versions of the same package are needed.  These require
different Scheme variable names.  We use the name as defined
in @ref{Package Naming}
for the most recent version; previous versions use the same name, suffixed
by @code{-} and the smallest prefix of the version number that may
distinguish the two versions.

The name inside the package definition is the same for all versions of a
package and does not contain any version number.

For instance, the versions 2.24.20 and 3.9.12 of GTK+ may be packaged as follows:

@example
(define-public gtk+
  (package
    (name "gtk+")
    (version "3.9.12")
    ...))
(define-public gtk+-2
  (package
    (name "gtk+")
    (version "2.24.20")
    ...))
@end example
If we also wanted GTK+ 3.8.2, this would be packaged as
@example
(define-public gtk+-3.8
  (package
    (name "gtk+")
    (version "3.8.2")
    ...))
@end example

@c See <https://lists.gnu.org/archive/html/guix-devel/2016-01/msg00425.html>,
@c for a discussion of what follows.
@cindex version number, for VCS snapshots
Occasionally, we package snapshots of upstream's version control system
(VCS) instead of formal releases.  This should remain exceptional,
because it is up to upstream developers to clarify what the stable
release is.  Yet, it is sometimes necessary.  So, what should we put in
the @code{version} field?

Clearly, we need to make the commit identifier of the VCS snapshot
visible in the version string, but we also need to make sure that the
version string is monotonically increasing so that @command{guix package
--upgrade} can determine which version is newer.  Since commit
identifiers, notably with Git, are not monotonically increasing, we add
a revision number that we increase each time we upgrade to a newer
snapshot.  The resulting version string looks like this:

@example
2.0.11-3.cabba9e
  ^    ^    ^
  |    |    `-- upstream commit ID
  |    |
  |    `--- Guix package revision
  |
latest upstream version
@end example

It is a good idea to strip commit identifiers in the @code{version}
field to, say, 7 digits.  It avoids an aesthetic annoyance (assuming
aesthetics have a role to play here) as well as problems related to OS
limits such as the maximum shebang length (127 bytes for the Linux
kernel.)  It is best to use the full commit identifiers in
@code{origin}s, though, to avoid ambiguities.  A typical package
definition may look like this:

@example
(define my-package
  (let ((commit "c3f29bc928d5900971f65965feaae59e1272a3f7")
        (revision "1"))          ;Guix package revision
    (package
      (version (string-append "0.9-" revision "."
                              (string-take commit 7)))
      (source (origin
                (method git-fetch)
                (uri (git-reference
                      (url "git://example.org/my-package.git")
                      (commit commit)))
                (sha256 (base32 "1mbikn@dots{}"))
                (file-name (string-append "my-package-" version
                                          "-checkout"))))
      ;; @dots{}
      )))
@end example

@node Synopses and Descriptions
@subsection Synopses and Descriptions

As we have seen before, each package in GNU@tie{}Guix includes a
synopsis and a description (@pxref{Defining Packages}).  Synopses and
descriptions are important: They are what @command{guix package
--search} searches, and a crucial piece of information to help users
determine whether a given package suits their needs.  Consequently,
packagers should pay attention to what goes into them.

Synopses must start with a capital letter and must not end with a
period.  They must not start with ``a'' or ``the'', which usually does
not bring anything; for instance, prefer ``File-frobbing tool'' over ``A
tool that frobs files''.  The synopsis should say what the package
is---e.g., ``Core GNU utilities (file, text, shell)''---or what it is
used for---e.g., the synopsis for GNU@tie{}grep is ``Print lines
matching a pattern''.

Keep in mind that the synopsis must be meaningful for a very wide
audience.  For example, ``Manipulate alignments in the SAM format''
might make sense for a seasoned bioinformatics researcher, but might be
fairly unhelpful or even misleading to a non-specialized audience.  It
is a good idea to come up with a synopsis that gives an idea of the
application domain of the package.  In this example, this might give
something like ``Manipulate nucleotide sequence alignments'', which
hopefully gives the user a better idea of whether this is what they are
looking for.

Descriptions should take between five and ten lines.  Use full
sentences, and avoid using acronyms without first introducing them.
Please avoid marketing phrases such as ``world-leading'',
``industrial-strength'', and ``next-generation'', and avoid superlatives
like ``the most advanced''---they are not helpful to users looking for a
package and may even sound suspicious.  Instead, try to be factual,
mentioning use cases and features.

@cindex Texinfo markup, in package descriptions
Descriptions can include Texinfo markup, which is useful to introduce
ornaments such as @code{@@code} or @code{@@dfn}, bullet lists, or
hyperlinks (@pxref{Overview,,, texinfo, GNU Texinfo}).  However you
should be careful when using some characters for example @samp{@@} and
curly braces which are the basic special characters in Texinfo
(@pxref{Special Characters,,, texinfo, GNU Texinfo}).  User interfaces
such as @command{guix package --show} take care of rendering it
appropriately.

Synopses and descriptions are translated by volunteers
@uref{http://translationproject.org/domain/guix-packages.html, at the
Translation Project} so that as many users as possible can read them in
their native language.  User interfaces search them and display them in
the language specified by the current locale.

Translation is a lot of work so, as a packager, please pay even more
attention to your synopses and descriptions as every change may entail
additional work for translators.  In order to help them, it is possible
to make recommendations or instructions visible to them by inserting
special comments like this (@pxref{xgettext Invocation,,, gettext, GNU
Gettext}):

@example
;; TRANSLATORS: "X11 resize-and-rotate" should not be translated.
(description "ARandR is designed to provide a simple visual front end
for the X11 resize-and-rotate (RandR) extension. @dots{}")
@end example


@node Python Modules
@subsection Python Modules

We currently package Python 2 and Python 3, under the Scheme variable names
@code{python-2} and @code{python} as explained in @ref{Version Numbers}.
To avoid confusion and naming clashes with other programming languages, it
seems desirable that the name of a package for a Python module contains
the word @code{python}.

Some modules are compatible with only one version of Python, others with both.
If the package Foo compiles only with Python 3, we name it
@code{python-foo}; if it compiles only with Python 2, we name it
@code{python2-foo}. If it is compatible with both versions, we create two
packages with the corresponding names.

If a project already contains the word @code{python}, we drop this;
for instance, the module python-dateutil is packaged under the names
@code{python-dateutil} and @code{python2-dateutil}.  If the project name
starts with @code{py} (e.g. @code{pytz}), we keep it and prefix it as
described above.


@node Perl Modules
@subsection Perl Modules

Perl programs standing for themselves are named as any other package,
using the lowercase upstream name.
For Perl packages containing a single class, we use the lowercase class name,
replace all occurrences of @code{::} by dashes and prepend the prefix
@code{perl-}.
So the class @code{XML::Parser} becomes @code{perl-xml-parser}.
Modules containing several classes keep their lowercase upstream name and
are also prepended by @code{perl-}.  Such modules tend to have the word
@code{perl} somewhere in their name, which gets dropped in favor of the
prefix.  For instance, @code{libwww-perl} becomes @code{perl-libwww}.


@node Java Packages
@subsection Java Packages

Java programs standing for themselves are named as any other package,
using the lowercase upstream name.

To avoid confusion and naming clashes with other programming languages,
it is desirable that the name of a package for a Java package is
prefixed with @code{java-}.  If a project already contains the word
@code{java}, we drop this; for instance, the package @code{ngsjava} is
packaged under the name @code{java-ngs}.

For Java packages containing a single class or a small class hierarchy,
we use the lowercase class name, replace all occurrences of @code{.} by
dashes and prepend the prefix @code{java-}.  So the class
@code{apache.commons.cli} becomes package
@code{java-apache-commons-cli}.


@node Fonts
@subsection Fonts

For fonts that are in general not installed by a user for typesetting
purposes, or that are distributed as part of a larger software package,
we rely on the general packaging rules for software; for instance, this
applies to the fonts delivered as part of the X.Org system or fonts that
are part of TeX Live.

To make it easier for a user to search for fonts, names for other packages
containing only fonts are constructed as follows, independently of the
upstream package name.

The name of a package containing only one font family starts with
@code{font-}; it is followed by the foundry name and a dash @code{-}
if the foundry is known, and the font family name, in which spaces are
replaced by dashes (and as usual, all upper case letters are transformed
to lower case).
For example, the Gentium font family by SIL is packaged under the name
@code{font-sil-gentium}.

For a package containing several font families, the name of the collection
is used in the place of the font family name.
For instance, the Liberation fonts consist of three families,
Liberation Sans, Liberation Serif and Liberation Mono.
These could be packaged separately under the names
@code{font-liberation-sans} and so on; but as they are distributed together
under a common name, we prefer to package them together as
@code{font-liberation}.

In the case where several formats of the same font family or font collection
are packaged separately, a short form of the format, prepended by a dash,
is added to the package name.  We use @code{-ttf} for TrueType fonts,
@code{-otf} for OpenType fonts and @code{-type1} for PostScript Type 1
fonts.



@node Bootstrapping
@section Bootstrapping

@c Adapted from the ELS 2013 paper.

@cindex bootstrapping

Bootstrapping in our context refers to how the distribution gets built
``from nothing''.  Remember that the build environment of a derivation
contains nothing but its declared inputs (@pxref{Introduction}).  So
there's an obvious chicken-and-egg problem: how does the first package
get built?  How does the first compiler get compiled?  Note that this is
a question of interest only to the curious hacker, not to the regular
user, so you can shamelessly skip this section if you consider yourself
a ``regular user''.

@cindex bootstrap binaries
The GNU system is primarily made of C code, with libc at its core.  The
GNU build system itself assumes the availability of a Bourne shell and
command-line tools provided by GNU Coreutils, Awk, Findutils, `sed', and
`grep'.  Furthermore, build programs---programs that run
@code{./configure}, @code{make}, etc.---are written in Guile Scheme
(@pxref{Derivations}).  Consequently, to be able to build anything at
all, from scratch, Guix relies on pre-built binaries of Guile, GCC,
Binutils, libc, and the other packages mentioned above---the
@dfn{bootstrap binaries}.

These bootstrap binaries are ``taken for granted'', though we can also
re-create them if needed (more on that later).

@unnumberedsubsec Preparing to Use the Bootstrap Binaries

@c As of Emacs 24.3, Info-mode displays the image, but since it's a
@c large image, it's hard to scroll.  Oh well.
@image{images/bootstrap-graph,6in,,Dependency graph of the early bootstrap derivations}

The figure above shows the very beginning of the dependency graph of the
distribution, corresponding to the package definitions of the @code{(gnu
packages bootstrap)} module.  A similar figure can be generated with
@command{guix graph} (@pxref{Invoking guix graph}), along the lines of:

@example
guix graph -t derivation \
  -e '(@@@@ (gnu packages bootstrap) %bootstrap-gcc)' \
  | dot -Tps > t.ps
@end example

At this level of detail, things are
slightly complex.  First, Guile itself consists of an ELF executable,
along with many source and compiled Scheme files that are dynamically
loaded when it runs.  This gets stored in the @file{guile-2.0.7.tar.xz}
tarball shown in this graph.  This tarball is part of Guix's ``source''
distribution, and gets inserted into the store with @code{add-to-store}
(@pxref{The Store}).

But how do we write a derivation that unpacks this tarball and adds it
to the store?  To solve this problem, the @code{guile-bootstrap-2.0.drv}
derivation---the first one that gets built---uses @code{bash} as its
builder, which runs @code{build-bootstrap-guile.sh}, which in turn calls
@code{tar} to unpack the tarball.  Thus, @file{bash}, @file{tar},
@file{xz}, and @file{mkdir} are statically-linked binaries, also part of
the Guix source distribution, whose sole purpose is to allow the Guile
tarball to be unpacked.

Once @code{guile-bootstrap-2.0.drv} is built, we have a functioning
Guile that can be used to run subsequent build programs.  Its first task
is to download tarballs containing the other pre-built binaries---this
is what the @code{.tar.xz.drv} derivations do.  Guix modules such as
@code{ftp-client.scm} are used for this purpose.  The
@code{module-import.drv} derivations import those modules in a directory
in the store, using the original layout.  The
@code{module-import-compiled.drv} derivations compile those modules, and
write them in an output directory with the right layout.  This
corresponds to the @code{#:modules} argument of
@code{build-expression->derivation} (@pxref{Derivations}).

Finally, the various tarballs are unpacked by the
derivations @code{gcc-bootstrap-0.drv}, @code{glibc-bootstrap-0.drv},
etc., at which point we have a working C tool chain.


@unnumberedsubsec Building the Build Tools

Bootstrapping is complete when we have a full tool chain that does not
depend on the pre-built bootstrap tools discussed above.  This
no-dependency requirement is verified by checking whether the files of
the final tool chain contain references to the @file{/gnu/store}
directories of the bootstrap inputs.  The process that leads to this
``final'' tool chain is described by the package definitions found in
the @code{(gnu packages commencement)} module.

The @command{guix graph} command allows us to ``zoom out'' compared to
the graph above, by looking at the level of package objects instead of
individual derivations---remember that a package may translate to
several derivations, typically one derivation to download its source,
one to build the Guile modules it needs, and one to actually build the
package from source.  The command:

@example
guix graph -t bag \
  -e '(@@@@ (gnu packages commencement)
          glibc-final-with-bootstrap-bash)' | dot -Tps > t.ps
@end example

@noindent
produces the dependency graph leading to the ``final'' C
library@footnote{You may notice the @code{glibc-intermediate} label,
suggesting that it is not @emph{quite} final, but as a good
approximation, we will consider it final.}, depicted below.

@image{images/bootstrap-packages,6in,,Dependency graph of the early packages}

@c See <http://lists.gnu.org/archive/html/gnu-system-discuss/2012-10/msg00000.html>.
The first tool that gets built with the bootstrap binaries is
GNU@tie{}Make---noted @code{make-boot0} above---which is a prerequisite
for all the following packages.  From there Findutils and Diffutils get
built.

Then come the first-stage Binutils and GCC, built as pseudo cross
tools---i.e., with @code{--target} equal to @code{--host}.  They are
used to build libc.  Thanks to this cross-build trick, this libc is
guaranteed not to hold any reference to the initial tool chain.

From there the final Binutils and GCC (not shown above) are built.
GCC uses @code{ld}
from the final Binutils, and links programs against the just-built libc.
This tool chain is used to build the other packages used by Guix and by
the GNU Build System: Guile, Bash, Coreutils, etc.

And voilà!  At this point we have the complete set of build tools that
the GNU Build System expects.  These are in the @code{%final-inputs}
variable of the @code{(gnu packages commencement)} module, and are
implicitly used by any package that uses @code{gnu-build-system}
(@pxref{Build Systems, @code{gnu-build-system}}).


@unnumberedsubsec Building the Bootstrap Binaries

Because the final tool chain does not depend on the bootstrap binaries,
those rarely need to be updated.  Nevertheless, it is useful to have an
automated way to produce them, should an update occur, and this is what
the @code{(gnu packages make-bootstrap)} module provides.

The following command builds the tarballs containing the bootstrap
binaries (Guile, Binutils, GCC, libc, and a tarball containing a mixture
of Coreutils and other basic command-line tools):

@example
guix build bootstrap-tarballs
@end example

The generated tarballs are those that should be referred to in the
@code{(gnu packages bootstrap)} module mentioned at the beginning of
this section.

Still here?  Then perhaps by now you've started to wonder: when do we
reach a fixed point?  That is an interesting question!  The answer is
unknown, but if you would like to investigate further (and have
significant computational and storage resources to do so), then let us
know.

@node Porting
@section Porting to a New Platform

As discussed above, the GNU distribution is self-contained, and
self-containment is achieved by relying on pre-built ``bootstrap
binaries'' (@pxref{Bootstrapping}).  These binaries are specific to an
operating system kernel, CPU architecture, and application binary
interface (ABI).  Thus, to port the distribution to a platform that is
not yet supported, one must build those bootstrap binaries, and update
the @code{(gnu packages bootstrap)} module to use them on that platform.

Fortunately, Guix can @emph{cross compile} those bootstrap binaries.
When everything goes well, and assuming the GNU tool chain supports the
target platform, this can be as simple as running a command like this
one:

@example
guix build --target=armv5tel-linux-gnueabi bootstrap-tarballs
@end example

For this to work, the @code{glibc-dynamic-linker} procedure in
@code{(gnu packages bootstrap)} must be augmented to return the right
file name for libc's dynamic linker on that platform; likewise,
@code{system->linux-architecture} in @code{(gnu packages linux)} must be
taught about the new platform.

Once these are built, the @code{(gnu packages bootstrap)} module needs
to be updated to refer to these binaries on the target platform.  That
is, the hashes and URLs of the bootstrap tarballs for the new platform
must be added alongside those of the currently supported platforms.  The
bootstrap Guile tarball is treated specially: it is expected to be
available locally, and @file{gnu/local.mk} has rules do download it for
the supported architectures; a rule for the new platform must be added
as well.

In practice, there may be some complications.  First, it may be that the
extended GNU triplet that specifies an ABI (like the @code{eabi} suffix
above) is not recognized by all the GNU tools.  Typically, glibc
recognizes some of these, whereas GCC uses an extra @code{--with-abi}
configure flag (see @code{gcc.scm} for examples of how to handle this).
Second, some of the required packages could fail to build for that
platform.  Lastly, the generated binaries could be broken for some
reason.

@c *********************************************************************
@include contributing.texi

@c *********************************************************************
@node Acknowledgments
@chapter Acknowledgments

Guix is based on the @uref{http://nixos.org/nix/, Nix package manager},
which was designed and
implemented by Eelco Dolstra, with contributions from other people (see
the @file{nix/AUTHORS} file in Guix.)  Nix pioneered functional package
management, and promoted unprecedented features, such as transactional
package upgrades and rollbacks, per-user profiles, and referentially
transparent build processes.  Without this work, Guix would not exist.

The Nix-based software distributions, Nixpkgs and NixOS, have also been
an inspiration for Guix.

GNU@tie{}Guix itself is a collective work with contributions from a
number of people.  See the @file{AUTHORS} file in Guix for more
information on these fine people.  The @file{THANKS} file lists people
who have helped by reporting bugs, taking care of the infrastructure,
providing artwork and themes, making suggestions, and more---thank you!


@c *********************************************************************
@node GNU Free Documentation License
@appendix GNU Free Documentation License

@include fdl-1.3.texi

@c *********************************************************************
@node Concept Index
@unnumbered Concept Index
@printindex cp

@node Programming Index
@unnumbered Programming Index
@syncodeindex tp fn
@syncodeindex vr fn
@printindex fn

@bye

@c Local Variables:
@c ispell-local-dictionary: "american";
@c End: