guix.texi

package, leading to the installed byproducts in the store.  A notable
exception is the ``bare-bones'' @code{trivial-build-system}
(@pxref{Build Systems}).

As discussed in the previous section, those build systems provide a
standard list of phases.  For @code{gnu-build-system}, the standard
phases include an @code{unpack} phase to unpack the source code tarball,
a @command{configure} phase to run @code{./configure}, a @code{build}
phase to run @command{make}, and (among others) an @code{install} phase
to run @command{make install}; @pxref{Build Systems}, for a more
detailed view of these phases.  Likewise, @code{cmake-build-system}
inherits these phases, but its @code{configure} phase runs
@command{cmake} instead of @command{./configure}.  Other build systems,
such as @code{python-build-system}, have a wholly different list of
standard phases.  All this code runs on the @dfn{build side}: it is
evaluated when you actually build the package, in a dedicated build
process spawned by the build daemon (@pxref{Invoking guix-daemon}).

Build phases are represented as association lists or ``alists''
(@pxref{Association Lists,,, guile, GNU Guile Reference Manual}) where
each key is a symbol for the name of the phase and the associated value
is a procedure that accepts an arbitrary number of arguments.  By
convention, those procedures receive information about the build in the
form of @dfn{keyword parameters}, which they can use or ignore.

For example, here is how @code{(guix build gnu-build-system)} defines
@code{%standard-phases}, the variable holding its alist of build
phases@footnote{We present a simplified view of those build phases, but
do take a look at @code{(guix build gnu-build-system)} to see all the
details!}:

@lisp
;; The build phases of 'gnu-build-system'.

(define* (unpack #:key source #:allow-other-keys)
  ;; Extract the source tarball.
  (invoke "tar" "xvf" source))

(define* (configure #:key outputs #:allow-other-keys)
  ;; Run the 'configure' script.  Install to output "out".
  (let ((out (assoc-ref outputs "out")))
    (invoke "./configure"
            (string-append "--prefix=" out))))

(define* (build #:allow-other-keys)
  ;; Compile.
  (invoke "make"))

(define* (check #:key (test-target "check") (tests? #true)
                #:allow-other-keys)
  ;; Run the test suite.
  (if tests?
      (invoke "make" test-target)
      (display "test suite not run\n")))

(define* (install #:allow-other-keys)
  ;; Install files to the prefix 'configure' specified.
  (invoke "make" "install"))

(define %standard-phases
  ;; The list of standard phases (quite a few are omitted
  ;; for brevity).  Each element is a symbol/procedure pair.
  (list (cons 'unpack unpack)
        (cons 'configure configure)
        (cons 'build build)
        (cons 'check check)
        (cons 'install install)))
@end lisp

This shows how @code{%standard-phases} is defined as a list of
symbol/procedure pairs (@pxref{Pairs,,, guile, GNU Guile Reference
Manual}).  The first pair associates the @code{unpack} procedure with
the @code{unpack} symbol---a name; the second pair defines the
@code{configure} phase similarly, and so on.  When building a package
that uses @code{gnu-build-system} with its default list of phases, those
phases are executed sequentially.  You can see the name of each phase
started and completed in the build log of packages that you build.

Let's now look at the procedures themselves.  Each one is defined with
@code{define*}: @code{#:key} lists keyword parameters the procedure
accepts, possibly with a default value, and @code{#:allow-other-keys}
specifies that other keyword parameters are ignored (@pxref{Optional
Arguments,,, guile, GNU Guile Reference Manual}).

The @code{unpack} procedure honors the @code{source} parameter, which
the build system uses to pass the file name of the source tarball (or
version control checkout), and it ignores other parameters.  The
@code{configure} phase only cares about the @code{outputs} parameter, an
alist mapping package output names to their store file name
(@pxref{Packages with Multiple Outputs}).  It extracts the file name of
for @code{out}, the default output, and passes it to
@command{./configure} as the installation prefix, meaning that
@command{make install} will eventually copy all the files in that
directory (@pxref{Configuration, configuration and makefile
conventions,, standards, GNU Coding Standards}).  @code{build} and
@code{install} ignore all their arguments.  @code{check} honors the
@code{test-target} argument, which specifies the name of the Makefile
target to run tests; it prints a message and skips tests when
@code{tests?} is false.

@cindex build phases, customizing
The list of phases used for a particular package can be changed with the
@code{#:phases} parameter of the build system.  Changing the set of
build phases boils down to building a new alist of phases based on the
@code{%standard-phases} alist described above.  This can be done with
standard alist procedures such as @code{alist-delete} (@pxref{SRFI-1
Association Lists,,, guile, GNU Guile Reference Manual}); however, it is
more convenient to do so with @code{modify-phases} (@pxref{Build
Utilities, @code{modify-phases}}).

Here is an example of a package definition that removes the
@code{configure} phase of @code{%standard-phases} and inserts a new
phase before the @code{build} phase, called
@code{set-prefix-in-makefile}:

@lisp
(define-public example
  (package
    (name "example")
    ;; other fields omitted
    (build-system gnu-build-system)
    (arguments
     '(#:phases (modify-phases %standard-phases
                  (delete 'configure)
                  (add-before 'build 'set-prefix-in-makefile
                    (lambda* (#:key outputs #:allow-other-keys)
                      ;; Modify the makefile so that its
                      ;; 'PREFIX' variable points to "out".
                      (let ((out (assoc-ref outputs "out")))
                        (substitute* "Makefile"
                          (("PREFIX =.*")
                           (string-append "PREFIX = "
                                          out "\n")))
                        #true))))))))
@end lisp

The new phase that is inserted is written as an anonymous procedure,
introduced with @code{lambda*}; it honors the @code{outputs} parameter
we have seen before.  @xref{Build Utilities}, for more about the helpers
used by this phase, and for more examples of @code{modify-phases}.

@cindex code staging
@cindex staging, of code
Keep in mind that build phases are code evaluated at the time the
package is actually built.  This explains why the whole
@code{modify-phases} expression above is quoted (it comes after the
@code{'} or apostrophe): it is @dfn{staged} for later execution.
@xref{G-Expressions}, for an explanation of code staging and the
@dfn{code strata} involved.

@node Build Utilities
@section Build Utilities

As soon as you start writing non-trivial package definitions
(@pxref{Defining Packages}) or other build actions
(@pxref{G-Expressions}), you will likely start looking for helpers for
``shell-like'' actions---creating directories, copying and deleting
files recursively, manipulating build phases, and so on.  The
@code{(guix build utils)} module provides such utility procedures.

Most build systems load @code{(guix build utils)} (@pxref{Build
Systems}).  Thus, when writing custom build phases for your package
definitions, you can usually assume those procedures are in scope.

When writing G-expressions, you can import @code{(guix build utils)} on
the ``build side'' using @code{with-imported-modules} and then put it in
scope with the @code{use-modules} form (@pxref{Using Guile Modules,,,
guile, GNU Guile Reference Manual}):

@lisp
(with-imported-modules '((guix build utils))  ;import it
  (computed-file "empty-tree"
                 #~(begin
                     ;; Put it in scope.
                     (use-modules (guix build utils))

                     ;; Happily use its 'mkdir-p' procedure.
                     (mkdir-p (string-append #$output "/a/b/c")))))
@end lisp

The remainder of this section is the reference for most of the utility
procedures provided by @code{(guix build utils)}.

@c TODO Document what's missing.

@subsection Dealing with Store File Names

This section documents procedures that deal with store file names.

@deffn {Scheme Procedure} %store-directory
Return the directory name of the store.
@end deffn

@deffn {Scheme Procedure} store-file-name? @var{file}
Return true if @var{file} is in the store.
@end deffn

@deffn {Scheme Procedure} strip-store-file-name @var{file}
Strip the @file{/gnu/store} and hash from @var{file}, a store file name.
The result is typically a @code{"@var{package}-@var{version}"} string.
@end deffn

@deffn {Scheme Procedure} package-name->name+version @var{name}
Given @var{name}, a package name like @code{"foo-0.9.1b"}, return two
values: @code{"foo"} and @code{"0.9.1b"}.  When the version part is
unavailable, @var{name} and @code{#f} are returned.  The first hyphen
followed by a digit is considered to introduce the version part.
@end deffn

@subsection File Types

The procedures below deal with files and file types.

@deffn {Scheme Procedure} directory-exists? @var{dir}
Return @code{#t} if @var{dir} exists and is a directory.
@end deffn

@deffn {Scheme Procedure} executable-file? @var{file}
Return @code{#t} if @var{file} exists and is executable.
@end deffn

@deffn {Scheme Procedure} symbolic-link? @var{file}
Return @code{#t} if @var{file} is a symbolic link (aka. a ``symlink'').
@end deffn

@deffn {Scheme Procedure} elf-file? @var{file}
@deffnx {Scheme Procedure} ar-file? @var{file}
@deffnx {Scheme Procedure} gzip-file? @var{file}
Return @code{#t} if @var{file} is, respectively, an ELF file, an
@code{ar} archive (such as a @file{.a} static library), or a gzip file.
@end deffn

@deffn {Scheme Procedure} reset-gzip-timestamp @var{file} [#:keep-mtime? #t]
If @var{file} is a gzip file, reset its embedded timestamp (as with
@command{gzip --no-name}) and return true.  Otherwise return @code{#f}.
When @var{keep-mtime?} is true, preserve @var{file}'s modification time.
@end deffn

@subsection File Manipulation

The following procedures and macros help create, modify, and delete
files.  They provide functionality comparable to common shell utilities
such as @command{mkdir -p}, @command{cp -r}, @command{rm -r}, and
@command{sed}.  They complement Guile's extensive, but low-level, file
system interface (@pxref{POSIX,,, guile, GNU Guile Reference Manual}).

@deffn {Scheme Syntax} with-directory-excursion @var{directory} @var{body}@dots{}
Run @var{body} with @var{directory} as the process's current directory.

Essentially, this macro changes the current directory to @var{directory}
before evaluating @var{body}, using @code{chdir} (@pxref{Processes,,,
guile, GNU Guile Reference Manual}).  It changes back to the initial
directory when the dynamic extent of @var{body} is left, be it @i{via}
normal procedure return or @i{via} a non-local exit such as an
exception.
@end deffn

@deffn {Scheme Procedure} mkdir-p @var{dir}
Create directory @var{dir} and all its ancestors.
@end deffn

@deffn {Scheme Procedure} install-file @var{file} @var{directory}
Create @var{directory} if it does not exist and copy @var{file} in there
under the same name.
@end deffn

@deffn {Scheme Procedure} make-file-writable @var{file}
Make @var{file} writable for its owner.
@end deffn

@deffn {Scheme Procedure} copy-recursively @var{source} @var{destination} @
  [#:log (current-output-port)] [#:follow-symlinks? #f] [#:keep-mtime? #f]
Copy @var{source} directory to @var{destination}.  Follow symlinks if
@var{follow-symlinks?}  is true; otherwise, just preserve them.  When
@var{keep-mtime?} is true, keep the modification time of the files in
@var{source} on those of @var{destination}.  Write verbose output to the
@var{log} port.
@end deffn

@deffn {Scheme Procedure} delete-file-recursively @var{dir} @
  [#:follow-mounts? #f]
Delete @var{dir} recursively, like @command{rm -rf}, without following
symlinks.  Don't follow mount points either, unless @var{follow-mounts?}
is true.  Report but ignore errors.
@end deffn

@deffn {Scheme Syntax} substitute* @var{file} @
  ((@var{regexp} @var{match-var}@dots{}) @var{body}@dots{}) @dots{}
Substitute @var{regexp} in @var{file} by the string returned by
@var{body}.  @var{body} is evaluated with each @var{match-var} bound to
the corresponding positional regexp sub-expression.  For example:

@lisp
(substitute* file
  (("hello")
   "good morning\n")
  (("foo([a-z]+)bar(.*)$" all letters end)
   (string-append "baz" letter end)))
@end lisp

Here, anytime a line of @var{file} contains @code{hello}, it is replaced
by @code{good morning}.  Anytime a line of @var{file} matches the second
regexp, @code{all} is bound to the complete match, @code{letters} is bound
to the first sub-expression, and @code{end} is bound to the last one.

When one of the @var{match-var} is @code{_}, no variable is bound to the
corresponding match substring.

Alternatively, @var{file} may be a list of file names, in which case
they are all subject to the substitutions.

Be careful about using @code{$} to match the end of a line; by itself it
won't match the terminating newline of a line.
@end deffn

@subsection File Search

@cindex file, searching
This section documents procedures to search and filter files.

@deffn {Scheme Procedure} file-name-predicate @var{regexp}
Return a predicate that returns true when passed a file name whose base
name matches @var{regexp}.
@end deffn

@deffn {Scheme Procedure} find-files @var{dir} [@var{pred}] @
  [#:stat lstat] [#:directories? #f] [#:fail-on-error? #f]
Return the lexicographically sorted list of files under @var{dir} for
which @var{pred} returns true.  @var{pred} is passed two arguments: the
absolute file name, and its stat buffer; the default predicate always
returns true.  @var{pred} can also be a regular expression, in which
case it is equivalent to @code{(file-name-predicate @var{pred})}.
@var{stat} is used to obtain file information; using @code{lstat} means
that symlinks are not followed.  If @var{directories?} is true, then
directories will also be included.  If @var{fail-on-error?} is true,
raise an exception upon error.
@end deffn

Here are a few examples where we assume that the current directory is
the root of the Guix source tree:

@lisp
;; List all the regular files in the current directory.
(find-files ".")
@result{} ("./.dir-locals.el" "./.gitignore" @dots{})

;; List all the .scm files under gnu/services.
(find-files "gnu/services" "\\.scm$")
@result{} ("gnu/services/admin.scm" "gnu/services/audio.scm" @dots{})

;; List ar files in the current directory.
(find-files "." (lambda (file stat) (ar-file? file)))
@result{} ("./libformat.a" "./libstore.a" @dots{})
@end lisp

@deffn {Scheme Procedure} which @var{program}
Return the complete file name for @var{program} as found in
@code{$PATH}, or @code{#f} if @var{program} could not be found.
@end deffn

@subsection Build Phases

@cindex build phases
The @code{(guix build utils)} also contains tools to manipulate build
phases as used by build systems (@pxref{Build Systems}).  Build phases
are represented as association lists or ``alists'' (@pxref{Association
Lists,,, guile, GNU Guile Reference Manual}) where each key is a symbol
naming the phase and the associated value is a procedure (@pxref{Build
Phases}).

Guile core and the @code{(srfi srfi-1)} module both provide tools to
manipulate alists.  The @code{(guix build utils)} module complements
those with tools written with build phases in mind.

@cindex build phases, modifying
@deffn {Scheme Syntax} modify-phases @var{phases} @var{clause}@dots{}
Modify @var{phases} sequentially as per each @var{clause}, which may
have one of the following forms:

@lisp
(delete @var{old-phase-name})
(replace @var{old-phase-name} @var{new-phase})
(add-before @var{old-phase-name} @var{new-phase-name} @var{new-phase})
(add-after @var{old-phase-name} @var{new-phase-name} @var{new-phase})
@end lisp

Where every @var{phase-name} above is an expression evaluating to a
symbol, and @var{new-phase} an expression evaluating to a procedure.
@end deffn

The example below is taken from the definition of the @code{grep}
package.  It adds a phase to run after the @code{install} phase, called
@code{fix-egrep-and-fgrep}.  That phase is a procedure (@code{lambda*}
is for anonymous procedures) that takes a @code{#:outputs} keyword
argument and ignores extra keyword arguments (@pxref{Optional
Arguments,,, guile, GNU Guile Reference Manual}, for more on
@code{lambda*} and optional and keyword arguments.)  The phase uses
@code{substitute*} to modify the installed @file{egrep} and @file{fgrep}
scripts so that they refer to @code{grep} by its absolute file name:

@lisp
(modify-phases %standard-phases
  (add-after 'install 'fix-egrep-and-fgrep
    ;; Patch 'egrep' and 'fgrep' to execute 'grep' via its
    ;; absolute file name instead of searching for it in $PATH.
    (lambda* (#:key outputs #:allow-other-keys)
      (let* ((out (assoc-ref outputs "out"))
             (bin (string-append out "/bin")))
        (substitute* (list (string-append bin "/egrep")
                           (string-append bin "/fgrep"))
          (("^exec grep")
           (string-append "exec " bin "/grep")))
        #t))))
@end lisp

In the example below, phases are modified in two ways: the standard
@code{configure} phase is deleted, presumably because the package does
not have a @file{configure} script or anything similar, and the default
@code{install} phase is replaced by one that manually copies the
executable files to be installed:

@lisp
(modify-phases %standard-phases
  (delete 'configure)      ;no 'configure' script
  (replace 'install
    (lambda* (#:key outputs #:allow-other-keys)
      ;; The package's Makefile doesn't provide an "install"
      ;; rule so do it by ourselves.
      (let ((bin (string-append (assoc-ref outputs "out")
                                "/bin")))
        (install-file "footswitch" bin)
        (install-file "scythe" bin)
        #t))))
@end lisp

@c TODO: Add more examples.

@node The Store
@section The Store

@cindex store
@cindex store items
@cindex store paths

Conceptually, the @dfn{store} is the place where derivations that have
been built successfully are stored---by default, @file{/gnu/store}.
Sub-directories in the store are referred to as @dfn{store items} or
sometimes @dfn{store paths}.  The store has an associated database that
contains information such as the store paths referred to by each store
path, and the list of @emph{valid} store items---results of successful
builds.  This database resides in @file{@var{localstatedir}/guix/db},
where @var{localstatedir} is the state directory specified @i{via}
@option{--localstatedir} at configure time, usually @file{/var}.

The store is @emph{always} accessed by the daemon on behalf of its clients
(@pxref{Invoking guix-daemon}).  To manipulate the store, clients
connect to the daemon over a Unix-domain socket, send requests to it,
and read the result---these are remote procedure calls, or RPCs.

@quotation Note
Users must @emph{never} modify files under @file{/gnu/store} directly.
This would lead to inconsistencies and break the immutability
assumptions of Guix's functional model (@pxref{Introduction}).

@xref{Invoking guix gc, @command{guix gc --verify}}, for information on
how to check the integrity of the store and attempt recovery from
accidental modifications.
@end quotation

The @code{(guix store)} module provides procedures to connect to the
daemon, and to perform RPCs.  These are described below.  By default,
@code{open-connection}, and thus all the @command{guix} commands,
connect to the local daemon or to the URI specified by the
@env{GUIX_DAEMON_SOCKET} environment variable.

@defvr {Environment Variable} GUIX_DAEMON_SOCKET
When set, the value of this variable should be a file name or a URI
designating the daemon endpoint.  When it is a file name, it denotes a
Unix-domain socket to connect to.  In addition to file names, the
supported URI schemes are:

@table @code
@item file
@itemx unix
These are for Unix-domain sockets.
@code{file:///var/guix/daemon-socket/socket} is equivalent to
@file{/var/guix/daemon-socket/socket}.

@item guix
@cindex daemon, remote access
@cindex remote access to the daemon
@cindex daemon, cluster setup
@cindex clusters, daemon setup
These URIs denote connections over TCP/IP, without encryption nor
authentication of the remote host.  The URI must specify the host name
and optionally a port number (by default port 44146 is used):

@example
guix://master.guix.example.org:1234
@end example

This setup is suitable on local networks, such as clusters, where only
trusted nodes may connect to the build daemon at
@code{master.guix.example.org}.

The @option{--listen} option of @command{guix-daemon} can be used to
instruct it to listen for TCP connections (@pxref{Invoking guix-daemon,
@option{--listen}}).

@item ssh
@cindex SSH access to build daemons
These URIs allow you to connect to a remote daemon over SSH.  This
feature requires Guile-SSH (@pxref{Requirements}) and a working
@command{guile} binary in @env{PATH} on the destination machine.  It
supports public key and GSSAPI authentication.  A typical URL might look
like this:

@example
ssh://charlie@@guix.example.org:22
@end example

As for @command{guix copy}, the usual OpenSSH client configuration files
are honored (@pxref{Invoking guix copy}).
@end table

Additional URI schemes may be supported in the future.

@c XXX: Remove this note when the protocol incurs fewer round trips
@c and when (guix derivations) no longer relies on file system access.
@quotation Note
The ability to connect to remote build daemons is considered
experimental as of @value{VERSION}.  Please get in touch with us to
share any problems or suggestions you may have (@pxref{Contributing}).
@end quotation
@end defvr

@deffn {Scheme Procedure} open-connection [@var{uri}] [#:reserve-space? #t]
Connect to the daemon over the Unix-domain socket at @var{uri} (a string).  When
@var{reserve-space?} is true, instruct it to reserve a little bit of
extra space on the file system so that the garbage collector can still
operate should the disk become full.  Return a server object.

@var{file} defaults to @code{%default-socket-path}, which is the normal
location given the options that were passed to @command{configure}.
@end deffn

@deffn {Scheme Procedure} close-connection @var{server}
Close the connection to @var{server}.
@end deffn

@defvr {Scheme Variable} current-build-output-port
This variable is bound to a SRFI-39 parameter, which refers to the port
where build and error logs sent by the daemon should be written.
@end defvr

Procedures that make RPCs all take a server object as their first
argument.

@deffn {Scheme Procedure} valid-path? @var{server} @var{path}
@cindex invalid store items
Return @code{#t} when @var{path} designates a valid store item and
@code{#f} otherwise (an invalid item may exist on disk but still be
invalid, for instance because it is the result of an aborted or failed
build).

A @code{&store-protocol-error} condition is raised if @var{path} is not
prefixed by the store directory (@file{/gnu/store}).
@end deffn

@deffn {Scheme Procedure} add-text-to-store @var{server} @var{name} @var{text} [@var{references}]
Add @var{text} under file @var{name} in the store, and return its store
path.  @var{references} is the list of store paths referred to by the
resulting store path.
@end deffn

@deffn {Scheme Procedure} build-derivations @var{store} @var{derivations} @
  [@var{mode}]
Build @var{derivations}, a list of @code{<derivation>} objects, @file{.drv}
file names, or derivation/output pairs, using the specified
@var{mode}---@code{(build-mode normal)} by default.
@end deffn

Note that the @code{(guix monads)} module provides a monad as well as
monadic versions of the above procedures, with the goal of making it
more convenient to work with code that accesses the store (@pxref{The
Store Monad}).

@c FIXME
@i{This section is currently incomplete.}

@node Derivations
@section Derivations

@cindex derivations
Low-level build actions and the environment in which they are performed
are represented by @dfn{derivations}.  A derivation contains the
following pieces of information:

@itemize
@item
The outputs of the derivation---derivations produce at least one file or
directory in the store, but may produce more.

@item
@cindex build-time dependencies
@cindex dependencies, build-time
The inputs of the derivations---i.e., its build-time dependencies---which may
be other derivations or plain files in the store (patches, build scripts,
etc.).

@item
The system type targeted by the derivation---e.g., @code{x86_64-linux}.

@item
The file name of a build script in the store, along with the arguments
to be passed.

@item
A list of environment variables to be defined.

@end itemize

@cindex derivation path
Derivations allow clients of the daemon to communicate build actions to
the store.  They exist in two forms: as an in-memory representation,
both on the client- and daemon-side, and as files in the store whose
name end in @file{.drv}---these files are referred to as @dfn{derivation
paths}.  Derivations paths can be passed to the @code{build-derivations}
procedure to perform the build actions they prescribe (@pxref{The
Store}).

@cindex fixed-output derivations
Operations such as file downloads and version-control checkouts for
which the expected content hash is known in advance are modeled as
@dfn{fixed-output derivations}.  Unlike regular derivations, the outputs
of a fixed-output derivation are independent of its inputs---e.g., a
source code download produces the same result regardless of the download
method and tools being used.

@cindex references
@cindex run-time dependencies
@cindex dependencies, run-time
The outputs of derivations---i.e., the build results---have a set of
@dfn{references}, as reported by the @code{references} RPC or the
@command{guix gc --references} command (@pxref{Invoking guix gc}).  References
are the set of run-time dependencies of the build results.  References are a
subset of the inputs of the derivation; this subset is automatically computed
by the build daemon by scanning all the files in the outputs.

The @code{(guix derivations)} module provides a representation of
derivations as Scheme objects, along with procedures to create and
otherwise manipulate derivations.  The lowest-level primitive to create
a derivation is the @code{derivation} procedure:

@deffn {Scheme Procedure} derivation @var{store} @var{name} @var{builder} @
  @var{args} [#:outputs '("out")] [#:hash #f] [#:hash-algo #f] @
  [#:recursive? #f] [#:inputs '()] [#:env-vars '()] @
  [#:system (%current-system)] [#:references-graphs #f] @
  [#:allowed-references #f] [#:disallowed-references #f] @
  [#:leaked-env-vars #f] [#:local-build? #f] @
  [#:substitutable? #t] [#:properties '()]
Build a derivation with the given arguments, and return the resulting
@code{<derivation>} object.

When @var{hash} and @var{hash-algo} are given, a
@dfn{fixed-output derivation} is created---i.e., one whose result is
known in advance, such as a file download.  If, in addition,
@var{recursive?} is true, then that fixed output may be an executable
file or a directory and @var{hash} must be the hash of an archive
containing this output.

When @var{references-graphs} is true, it must be a list of file
name/store path pairs.  In that case, the reference graph of each store
path is exported in the build environment in the corresponding file, in
a simple text format.

When @var{allowed-references} is true, it must be a list of store items
or outputs that the derivation's output may refer to.  Likewise,
@var{disallowed-references}, if true, must be a list of things the
outputs may @emph{not} refer to.

When @var{leaked-env-vars} is true, it must be a list of strings
denoting environment variables that are allowed to ``leak'' from the
daemon's environment to the build environment.  This is only applicable
to fixed-output derivations---i.e., when @var{hash} is true.  The main
use is to allow variables such as @code{http_proxy} to be passed to
derivations that download files.

When @var{local-build?} is true, declare that the derivation is not a
good candidate for offloading and should rather be built locally
(@pxref{Daemon Offload Setup}).  This is the case for small derivations
where the costs of data transfers would outweigh the benefits.

When @var{substitutable?} is false, declare that substitutes of the
derivation's output should not be used (@pxref{Substitutes}).  This is
useful, for instance, when building packages that capture details of the
host CPU instruction set.

@var{properties} must be an association list describing ``properties'' of the
derivation.  It is kept as-is, uninterpreted, in the derivation.
@end deffn

@noindent
Here's an example with a shell script as its builder, assuming
@var{store} is an open connection to the daemon, and @var{bash} points
to a Bash executable in the store:

@lisp
(use-modules (guix utils)
             (guix store)
             (guix derivations))

(let ((builder   ; add the Bash script to the store
        (add-text-to-store store "my-builder.sh"
                           "echo hello world > $out\n" '())))
  (derivation store "foo"
              bash `("-e" ,builder)
              #:inputs `((,bash) (,builder))
              #:env-vars '(("HOME" . "/homeless"))))
@result{} #<derivation /gnu/store/@dots{}-foo.drv => /gnu/store/@dots{}-foo>
@end lisp

As can be guessed, this primitive is cumbersome to use directly.  A
better approach is to write build scripts in Scheme, of course!  The
best course of action for that is to write the build code as a
``G-expression'', and to pass it to @code{gexp->derivation}.  For more
information, @pxref{G-Expressions}.

Once upon a time, @code{gexp->derivation} did not exist and constructing
derivations with build code written in Scheme was achieved with
@code{build-expression->derivation}, documented below.  This procedure
is now deprecated in favor of the much nicer @code{gexp->derivation}.

@deffn {Scheme Procedure} build-expression->derivation @var{store} @
       @var{name} @var{exp} @
       [#:system (%current-system)] [#:inputs '()] @
       [#:outputs '("out")] [#:hash #f] [#:hash-algo #f] @
       [#:recursive? #f] [#:env-vars '()] [#:modules '()] @
       [#:references-graphs #f] [#:allowed-references #f] @
       [#:disallowed-references #f] @
       [#:local-build? #f] [#:substitutable? #t] [#:guile-for-build #f]
Return a derivation that executes Scheme expression @var{exp} as a
builder for derivation @var{name}.  @var{inputs} must be a list of
@code{(name drv-path sub-drv)} tuples; when @var{sub-drv} is omitted,
@code{"out"} is assumed.  @var{modules} is a list of names of Guile
modules from the current search path to be copied in the store,
compiled, and made available in the load path during the execution of
@var{exp}---e.g., @code{((guix build utils) (guix build
gnu-build-system))}.

@var{exp} is evaluated in an environment where @code{%outputs} is bound
to a list of output/path pairs, and where @code{%build-inputs} is bound
to a list of string/output-path pairs made from @var{inputs}.
Optionally, @var{env-vars} is a list of string pairs specifying the name
and value of environment variables visible to the builder.  The builder
terminates by passing the result of @var{exp} to @code{exit}; thus, when
@var{exp} returns @code{#f}, the build is considered to have failed.

@var{exp} is built using @var{guile-for-build} (a derivation).  When
@var{guile-for-build} is omitted or is @code{#f}, the value of the
@code{%guile-for-build} fluid is used instead.

See the @code{derivation} procedure for the meaning of
@var{references-graphs}, @var{allowed-references},
@var{disallowed-references}, @var{local-build?}, and
@var{substitutable?}.
@end deffn

@noindent
Here's an example of a single-output derivation that creates a directory
containing one file:

@lisp
(let ((builder '(let ((out (assoc-ref %outputs "out")))
                  (mkdir out)    ; create /gnu/store/@dots{}-goo
                  (call-with-output-file (string-append out "/test")
                    (lambda (p)
                      (display '(hello guix) p))))))
  (build-expression->derivation store "goo" builder))

@result{} #<derivation /gnu/store/@dots{}-goo.drv => @dots{}>
@end lisp


@node The Store Monad
@section The Store Monad

@cindex monad

The procedures that operate on the store described in the previous
sections all take an open connection to the build daemon as their first
argument.  Although the underlying model is functional, they either have
side effects or depend on the current state of the store.

The former is inconvenient: the connection to the build daemon has to be
carried around in all those functions, making it impossible to compose
functions that do not take that parameter with functions that do.  The
latter can be problematic: since store operations have side effects
and/or depend on external state, they have to be properly sequenced.

@cindex monadic values
@cindex monadic functions
This is where the @code{(guix monads)} module comes in.  This module
provides a framework for working with @dfn{monads}, and a particularly
useful monad for our uses, the @dfn{store monad}.  Monads are a
construct that allows two things: associating ``context'' with values
(in our case, the context is the store), and building sequences of
computations (here computations include accesses to the store).  Values
in a monad---values that carry this additional context---are called
@dfn{monadic values}; procedures that return such values are called
@dfn{monadic procedures}.

Consider this ``normal'' procedure:

@lisp
(define (sh-symlink store)
  ;; Return a derivation that symlinks the 'bash' executable.
  (let* ((drv (package-derivation store bash))
         (out (derivation->output-path drv))
         (sh  (string-append out "/bin/bash")))
    (build-expression->derivation store "sh"
                                  `(symlink ,sh %output))))
@end lisp

Using @code{(guix monads)} and @code{(guix gexp)}, it may be rewritten
as a monadic function:

@lisp
(define (sh-symlink)
  ;; Same, but return a monadic value.
  (mlet %store-monad ((drv (package->derivation bash)))
    (gexp->derivation "sh"
                      #~(symlink (string-append #$drv "/bin/bash")
                                 #$output))))
@end lisp

There are several things to note in the second version: the @code{store}
parameter is now implicit and is ``threaded'' in the calls to the
@code{package->derivation} and @code{gexp->derivation} monadic
procedures, and the monadic value returned by @code{package->derivation}
is @dfn{bound} using @code{mlet} instead of plain @code{let}.

As it turns out, the call to @code{package->derivation} can even be
omitted since it will take place implicitly, as we will see later
(@pxref{G-Expressions}):

@lisp
(define (sh-symlink)
  (gexp->derivation "sh"
                    #~(symlink (string-append #$bash "/bin/bash")
                               #$output)))
@end lisp

@c See
@c <https://syntaxexclamation.wordpress.com/2014/06/26/escaping-continuations/>
@c for the funny quote.
Calling the monadic @code{sh-symlink} has no effect.  As someone once
said, ``you exit a monad like you exit a building on fire: by running''.
So, to exit the monad and get the desired effect, one must use
@code{run-with-store}:

@lisp
(run-with-store (open-connection) (sh-symlink))
@result{} /gnu/store/...-sh-symlink
@end lisp

Note that the @code{(guix monad-repl)} module extends the Guile REPL with
new ``meta-commands'' to make it easier to deal with monadic procedures:
@code{run-in-store}, and @code{enter-store-monad}.  The former is used
to ``run'' a single monadic value through the store:

@example
scheme@@(guile-user)> ,run-in-store (package->derivation hello)
$1 = #<derivation /gnu/store/@dots{}-hello-2.9.drv => @dots{}>
@end example

The latter enters a recursive REPL, where all the return values are
automatically run through the store:

@example
scheme@@(guile-user)> ,enter-store-monad
store-monad@@(guile-user) [1]> (package->derivation hello)
$2 = #<derivation /gnu/store/@dots{}-hello-2.9.drv => @dots{}>
store-monad@@(guile-user) [1]> (text-file "foo" "Hello!")
$3 = "/gnu/store/@dots{}-foo"
store-monad@@(guile-user) [1]> ,q
scheme@@(guile-user)>
@end example

@noindent
Note that non-monadic values cannot be returned in the
@code{store-monad} REPL.

The main syntactic forms to deal with monads in general are provided by
the @code{(guix monads)} module and are described below.

@deffn {Scheme Syntax} with-monad @var{monad} @var{body} ...
Evaluate any @code{>>=} or @code{return} forms in @var{body} as being
in @var{monad}.
@end deffn

@deffn {Scheme Syntax} return @var{val}
Return a monadic value that encapsulates @var{val}.
@end deffn

@deffn {Scheme Syntax} >>= @var{mval} @var{mproc} ...
@dfn{Bind} monadic value @var{mval}, passing its ``contents'' to monadic
procedures @var{mproc}@dots{}@footnote{This operation is commonly
referred to as ``bind'', but that name denotes an unrelated procedure in
Guile.  Thus we use this somewhat cryptic symbol inherited from the
Haskell language.}.  There can be one @var{mproc} or several of them, as
in this example:

@lisp
(run-with-state
    (with-monad %state-monad
      (>>= (return 1)
           (lambda (x) (return (+ 1 x)))
           (lambda (x) (return (* 2 x)))))
  'some-state)

@result{} 4
@result{} some-state
@end lisp
@end deffn

@deffn {Scheme Syntax} mlet @var{monad} ((@var{var} @var{mval}) ...) @
       @var{body} ...
@deffnx {Scheme Syntax} mlet* @var{monad} ((@var{var} @var{mval}) ...) @
       @var{body} ...
Bind the variables @var{var} to the monadic values @var{mval} in
@var{body}, which is a sequence of expressions.  As with the bind
operator, this can be thought of as ``unpacking'' the raw, non-monadic
value ``contained'' in @var{mval} and making @var{var} refer to that
raw, non-monadic value within the scope of the @var{body}.  The form
(@var{var} -> @var{val}) binds @var{var} to the ``normal'' value
@var{val}, as per @code{let}.  The binding operations occur in sequence
from left to right.  The last expression of @var{body} must be a monadic
expression, and its result will become the result of the @code{mlet} or
@code{mlet*} when run in the @var{monad}.

@code{mlet*} is to @code{mlet} what @code{let*} is to @code{let}
(@pxref{Local Bindings,,, guile, GNU Guile Reference Manual}).
@end deffn

@deffn {Scheme System} mbegin @var{monad} @var{mexp} ...
Bind @var{mexp} and the following monadic expressions in sequence,
returning the result of the last expression.  Every expression in the
sequence must be a monadic expression.

This is akin to @code{mlet}, except that the return values of the
monadic expressions are ignored.  In that sense, it is analogous to
@code{begin}, but applied to monadic expressions.
@end deffn

@deffn {Scheme System} mwhen @var{condition} @var{mexp0} @var{mexp*} ...
When @var{condition} is true, evaluate the sequence of monadic
expressions @var{mexp0}..@var{mexp*} as in an @code{mbegin}.  When
@var{condition} is false, return @code{*unspecified*} in the current
monad.  Every expression in the sequence must be a monadic expression.
@end deffn

@deffn {Scheme System} munless @var{condition} @var{mexp0} @var{mexp*} ...
When @var{condition} is false, evaluate the sequence of monadic
expressions @var{mexp0}..@var{mexp*} as in an @code{mbegin}.  When
@var{condition} is true, return @code{*unspecified*} in the current
monad.  Every expression in the sequence must be a monadic expression.
@end deffn

@cindex state monad
The @code{(guix monads)} module provides the @dfn{state monad}, which
allows an additional value---the state---to be @emph{threaded} through
monadic procedure calls.

@defvr {Scheme Variable} %state-monad
The state monad.  Procedures in the state monad can access and change
the state that is threaded.

Consider the example below.  The @code{square} procedure returns a value
in the state monad.  It returns the square of its argument, but also
increments the current state value:

@lisp
(define (square x)
  (mlet %state-monad ((count (current-state)))
    (mbegin %state-monad
      (set-current-state (+ 1 count))
      (return (* x x)))))

(run-with-state (sequence %state-monad (map square (iota 3))) 0)
@result{} (0 1 4)
@result{} 3
@end lisp

When ``run'' through @code{%state-monad}, we obtain that additional state
value, which is the number of @code{square} calls.
@end defvr

@deffn {Monadic Procedure} current-state
Return the current state as a monadic value.