asdf.texinfo

\input texinfo          @c -*- mode: texinfo; coding: utf-8 -*-
@c %**start of header
@documentencoding UTF-8
@setfilename asdf.info
@settitle ASDF Manual
@c %**end of header

@c We use @&key, etc to escape & from TeX in lambda lists --
@c so we need to define them for info as well.
@macro AallowOtherKeys
&allow-other-keys
@end macro
@macro Aoptional
&optional
@end macro
@macro Arest
&rest
@end macro
@macro Akey
&key
@end macro
@macro Abody
&body
@end macro

@c for install-info
@dircategory Software development
@direntry
* asdf: (asdf).           Another System Definition Facility (for Common Lisp)
@end direntry

@copying
This manual describes ASDF, a system definition facility
for Common Lisp programs and libraries.

You can find the latest version of this manual at
@url{https://common-lisp.net/project/asdf/asdf.html}.

ASDF Copyright @copyright{} 2001-2019 Daniel Barlow and contributors.

This manual Copyright @copyright{} 2001-2019 Daniel Barlow and contributors.

This manual revised @copyright{} 2009-2019 Robert P. Goldman and Francois-Rene Rideau.

Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
``Software''), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:

The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED ``AS IS'', WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

@end copying

@titlepage
@title ASDF: Another System Definition Facility
@subtitle Manual for Version 3.3.6.2
@c The following two commands start the copyright page.
@page
@vskip 0pt plus 1filll
@insertcopying
@end titlepage

@c Output the table of contents at the beginning.
@contents

@c -------------------

@ifnottex

@node Top, Introduction, (dir), (dir)
@top ASDF: Another System Definition Facility
@ifnottex
Manual for Version 3.3.6.2
@end ifnottex


@insertcopying

@menu
* Introduction::
* Quick start summary::
* Loading ASDF::
* Configuring ASDF::
* Using ASDF::
* Defining systems with defsystem::
* The object model of ASDF::
* Controlling where ASDF searches for systems::
* Controlling where ASDF saves compiled files::
* Error handling::
* Miscellaneous additional functionality::
* Getting the latest version::
* FAQ::
* Ongoing Work::
* Bibliography::
* Concept Index::
* Function and Macro Index::
* Variable Index::              @c @detailmenu
* Class and Type Index::        @c

@detailmenu
 --- The Detailed Node Listing ---

Loading ASDF

* Loading a pre-installed ASDF::
* Checking whether ASDF is loaded::
* Upgrading ASDF::
* Replacing your implementation's ASDF::
* Loading ASDF from source::

Configuring ASDF

* Configuring ASDF to find your systems::
* Configuring ASDF to find your systems --- old style::
* Configuring where ASDF stores object files::
* Resetting the ASDF configuration::

Using ASDF

* Loading a system::
* Convenience Functions::
* Moving on::

Defining systems with defsystem

* The defsystem form::
* A more involved example::
* The defsystem grammar::
* Other code in .asd files::
* The package-inferred-system extension::

The Object model of ASDF

* Operations::
* Components::
* Dependencies::
* Functions::
* Parsing system definitions::

Operations

* Predefined operations of ASDF::
* Creating new operations::

Components

* Common attributes of components::
* Pre-defined subclasses of component::
* Creating new component types::

Properties

* Pre-defined subclasses of component::
* Creating new component types::

Controlling where ASDF searches for systems

* Configurations::
* Truenames and other dangers::
* XDG base directory::
* Backward Compatibility::
* Configuration DSL::
* Configuration Directories::
* Shell-friendly syntax for configuration::
* Search Algorithm::
* Caching Results::
* Configuration API::
* Introspection::
* Status::
* Rejected ideas::
* TODO::
* Credits for the source-registry::

Configuration Directories

* The here directive::

Introspection

* *source-registry-parameter* variable::
* Information about system dependencies::

Controlling where ASDF saves compiled files

* Output Configurations::
* Output Backward Compatibility::
* Output Configuration DSL::
* Output Configuration Directories::
* Output Shell-friendly syntax for configuration::
* Semantics of Output Translations::
* Output Caching Results::
* Output location API::
* Credits for output translations::

Miscellaneous additional functionality

* Controlling file compilation::
* Controlling source file character encoding::
* Miscellaneous Functions::
* Some Utility Functions::

FAQ

* Where do I report a bug?::
* Mailing list::
* What has changed between ASDF 1 ASDF 2 and ASDF 3?::
* Issues with installing the proper version of ASDF::
* Issues with configuring ASDF::
* Issues with using and extending ASDF to define systems::
* ASDF development FAQs::

``What has changed between ASDF 1, ASDF 2, and ASDF 3?''

* What are ASDF 1 2 3?::
* How do I detect the ASDF version?::
* ASDF can portably name files in subdirectories::
* Output translations::
* Source Registry Configuration::
* Usual operations are made easier to the user::
* Many bugs have been fixed::
* ASDF itself is versioned::
* ASDF can be upgraded::
* Decoupled release cycle::
* Pitfalls of the transition to ASDF 2::
* Pitfalls of the upgrade to ASDF 3::
* What happened to the bundle operations::

Issues with installing the proper version of ASDF

* My Common Lisp implementation comes with an outdated version of ASDF. What to do?::
* I'm a Common Lisp implementation vendor. When and how should I upgrade ASDF?::
* After upgrading ASDF, ASDF (and Quicklisp) can't find my systems: After upgrading ASDF.

Issues with configuring ASDF

* How can I customize where fasl files are stored?::
* How can I wholly disable the compiler output cache?::
* How can I debug problems finding ASDF systems::

Issues with using and extending ASDF to define systems

* How can I cater for unit-testing in my system?::
* How can I cater for documentation generation in my system?::
* How can I maintain non-Lisp (e.g. C) source files?::
* I want to put my module's files at the top level.  How do I do this?::
* How do I create a system definition where all the source files have a .cl extension?::
* How do I mark a source file to be loaded only and not compiled?::
* How do I work with readtables?::
* How can I capture ASDF's output?::
* LOAD-PATHNAME has a weird value::
* How can I produce a binary at a specific path from sources at a specific path::

ASDF development FAQs

* How do I run the tests interactively in a REPL?::

@end detailmenu
@end menu

@end ifnottex

@c -------------------

@node Introduction, Quick start summary, Top, Top
@comment  node-name,  next,  previous,  up
@chapter Introduction
@cindex ASDF-related features
@vindex *features*
@cindex Testing for ASDF
@cindex ASDF versions
@cindex :asdf
@cindex :asdf2
@cindex :asdf3

ASDF, or Another System Definition Facility, is a @emph{build system}:
a tool for specifying how systems of Common Lisp software
are made up of components (sub-systems and files),
and how to operate on these components in the right order
so that they can be compiled, loaded, tested, etc.
If you are new to ASDF, @pxref{Quick start summary,,the quick start
guide}.

ASDF presents three faces:
one for users of Common Lisp software who want to reuse other people's code,
one for writers of Common Lisp software who want to specify how to build their systems,
and one for implementers of Common Lisp extensions who want to extend
the build system.
For more specifics,
@pxref{Using ASDF},
to learn how to use ASDF to load a system.
@xref{Defining systems with defsystem},
to learn how to define a system of your own.
@xref{The object model of ASDF}, for a description of
the ASDF internals and how to extend ASDF.

Note that
ASDF is @emph{not} a tool for library and system @emph{installation};
it plays a role like @code{make} or @code{ant}, not like a package manager.
In particular, ASDF should not to be confused with Quicklisp or ASDF-Install,
that attempt to find and download ASDF systems for you.
Despite what the name might suggest,
ASDF-Install was never a part of ASDF; it was always a separate piece of software.
ASDF-Install has also been unmaintained and obsolete for a very long time.
We recommend you use Quicklisp
(@uref{http://www.quicklisp.org/}) instead,
a Common Lisp package manager which works well and is being actively maintained.
If you want to download software from version control instead of tarballs,
so you may more easily modify it,
we recommend clbuild (@uref{http://common-lisp.net/project/clbuild/}).
As for where on your filesystem to install Common Lisp software,
we recommend subdirectories of @file{~/common-lisp/}:
starting with ASDF 3.1.2 (2014), this hierarchy is included
in the default source-registry configuration.

Finally, note that this manual is incomplete.
All the bases are covered,
but many advanced topics are only barely alluded to,
and there is not much in terms of examples.
The source code remains the ultimate source of information,
free software systems in Quicklisp remain the best source of examples,
and the mailing-list the best place to ask for help.


@node  Quick start summary, Loading ASDF, Introduction, Top
@chapter Quick start summary

@itemize

@item To load an ASDF system:

@itemize
@item
Load ASDF itself into your Lisp image, using
@code{(require "asdf")}.
Check that you have a recent version using @code{(asdf:asdf-version)}.
For more details, or if any of the above fails, @pxref{Loading ASDF}.

@item
Make sure software is installed where ASDF can find it.
The simplest way is to put all your Lisp code in subdirectories of
@file{~/common-lisp/} (starting with ASDF 3.1.2),
or @file{~/.local/share/common-lisp/source/}
(for ASDF 2 and later, or if you want to keep source in a hidden directory).
For more details, @pxref{Configuring ASDF to find your systems}.

@item
Load your system with @code{(asdf:load-system "@var{my-system}")}.
@xref{Using ASDF}.

@end itemize

@item To make your own ASDF system:

@itemize
@item
As above, load and configure ASDF.

@item
Make a new directory for your system, @code{@var{my-system}/},
again in a location where ASDF can find it.
All else being equal, the easiest location is probably
@file{~/common-lisp/my-system/}.
@xref{Configuring ASDF to find your systems}.


@item
Create an ASDF system definition listing the dependencies of
your system, its components, and their interdependencies,
and put it in @file{@var{my-system}.asd}.
This file must have the same name as your system, all lowercase.
@xref{Defining systems with defsystem}.

@item
Use @code{(asdf:load-system "@var{my-system}")}
to make sure it's all working properly. @xref{Using ASDF}.

@end itemize
@end itemize

@c FIXME: (1) add a sample project that the user can cut and paste to
@c get started.  (2) discuss the option of starting with Quicklisp.





@node Loading ASDF, Configuring ASDF, Quick start summary, Top
@comment  node-name,  next,  previous,  up
@chapter Loading ASDF

@menu
* Loading a pre-installed ASDF::
* Checking whether ASDF is loaded::
* Upgrading ASDF::
* Replacing your implementation's ASDF::
* Loading ASDF from source::
@end menu

@node  Loading a pre-installed ASDF, Checking whether ASDF is loaded, Loading ASDF, Loading ASDF
@section Loading a pre-installed ASDF

The recommended way to load ASDF is via:
@lisp
(require "asdf")
@end lisp

All actively maintained Lisp implementations now include a copy of ASDF 3
that you can load this way using Common Lisp's @code{require} function.@footnote{
NB: all implementations except GNU CLISP also accept
@code{(require "ASDF")}, @code{(require 'asdf)} and @code{(require :asdf)}.
For portability's sake, you should use @code{(require "asdf")}.
}

If the implementation you are using doesn't provide a recent ASDF 3,
we recommend you upgrade it.
If for some reason you would rather not upgrade it,
we recommend you replace your implementation's ASDF.
@xref{Replacing your implementation's ASDF}.
If all else fails, see @pxref{Loading ASDF from source} below.

If you use an actively maintained implementation that fails to provide
an up-to-date enough stable release of ASDF,
you may also send a bug report to your Lisp vendor and complain about it
--- or you may fix the issue yourself if it's free software.

As of the writing of this manual,
the following implementations provide ASDF 3 this way:
ABCL, Allegro CL, CLASP, Clozure CL, CMUCL, ECL, GNU CLISP, LispWorks, MKCL, SBCL.
The following implementations only provide ASDF 2:
MOCL, XCL.
The following implementations don't provide ASDF:
Corman CL, GCL, Genera, MCL, SCL.
The latter implementations are not actively maintained (except maybe GCL);
if some of them are ever released again, they probably will include ASDF 3.

For maximum convenience you might want to have ASDF loaded
whenever you start your Lisp implementation,
for example by loading it from the startup script or dumping a custom core
--- check your Lisp implementation's manual for details.
SLIME notably sports a @code{slime-asdf} contrib that makes life easier with ASDF.


@node Checking whether ASDF is loaded, Upgrading ASDF, Loading a pre-installed ASDF, Loading ASDF
@section Checking whether ASDF is loaded

To check that ASDF is properly loaded, you can run this form:

@lisp
(asdf:asdf-version)
@end lisp

If it returns a string,
that is the version of ASDF that is currently installed.
If that version is suitably recent (say, 3.1.2 or later),
then you can skip directly to next chapter: @xref{Configuring ASDF}.

If it raises an error,
then either ASDF is not loaded, or
you are using a very old version of ASDF,
and need to install ASDF 3.

For more precision in detecting versions old and new,
@pxref{How do I detect the ASDF version?}.

If you are experiencing problems with ASDF,
please try upgrading to the latest released version,
using the method below,
before you contact us and raise an issue.

@node  Upgrading ASDF, Replacing your implementation's ASDF, Checking whether ASDF is loaded, Loading ASDF
@section Upgrading ASDF
@c FIXME: tighten this up a bit -- there's a lot of stuff here that
@c doesn't matter to almost anyone.  Move discussion of updating antique
@c versions of ASDF down, or encapsulate it.

If your implementation already provides ASDF 3 or later (and it should),
but you want a more recent ASDF version than your implementation provides, then
you just need to ensure the more recent ASDF is installed in a configured path,
like any other system.
We recommend you download an official tarball or checkout a release from git into
@file{~/common-lisp/asdf/}.
(@pxref{Configuring ASDF to find your systems}).

Once the source code for ASDF is installed,
you don't need any extra step to load it beyond the usual @code{(require "asdf")}:
ASDF 3 will automatically look whether an updated version of itself is available
amongst the regularly configured systems, before it compiles anything else.

If your implementation fails to provide ASDF 3 or later,
@pxref{Replacing your implementation's ASDF}.


@node Replacing your implementation's ASDF, Loading ASDF from source, Upgrading ASDF, Loading ASDF
@section Replacing your implementation's ASDF

All maintained implementations now provide ASDF 3 in their latest release.
If yours doesn't, we recommend you upgrade it.

Now, if you insist on using an old implementation
that didn't provide ASDF or provided an old version,
we recommend installing a recent ASDF, as explained below,
into your implementation's installation directory.
Thus your modified implementation will now provide ASDF 3.
This requires proper write permissions and
may necessitate execution as a system administrator.

The ASDF source repository contains a tool to
help you upgrade your implementation's ASDF.
You can invoke it from the shell command-line as
@code{tools/asdf-tools install-asdf lispworks}
(where you can replace @code{lispworks} by the name of the relevant implementation),
or you can @code{(load "tools/install-asdf.lisp")} from your Lisp REPL.

This script works on
Allegro CL, Clozure CL, CMU CL, ECL, GCL, GNU CLISP, LispWorks, MKCL, SBCL, SCL, XCL.
It doesn't work on ABCL, Corman CL, Genera, MCL, MOCL.
Happily, ABCL is usually pretty up to date and shouldn't need that script.
GCL requires a very recent version, and hasn't been tested much.
Corman CL, Genera, MCL are obsolete anyway.
MOCL is incomplete.


@node Loading ASDF from source,  , Replacing your implementation's ASDF, Loading ASDF
@section Loading ASDF from source

If you write build scripts that must remain portable to old machines with old implementations
that you cannot ensure have been upgraded or modified to provide a recent ASDF,
you may have to install the file @file{asdf.lisp}
somewhere and load it with:

@lisp
(load "/path/to/your/installed/asdf.lisp")
@end lisp

The single file @file{asdf.lisp} is all you normally need to use ASDF.

You can extract this file from latest release tarball on the
@url{https://common-lisp.net/project/asdf/,ASDF website}.
If you are daring and willing to report bugs, you can get
the latest and greatest version of ASDF from its git repository.
@xref{Getting the latest version}.

For scripts that try to use ASDF simply via @code{require} at first, and
make heroic attempts to load it the hard way if at first they don't succeed,
see @file{tools/load-asdf.lisp} distributed with the ASDF source repository,
or the code of @url{https://cliki.net/cl-launch,@code{cl-launch}}.


@node Configuring ASDF, Using ASDF, Loading ASDF, Top
@comment  node-name,  next,  previous,  up
@chapter Configuring ASDF

For standard use cases, ASDF should work pretty much out of the box.
We recommend you skim the sections on configuring ASDF to find your systems
and choose the method of installing Lisp software that works best for you.
Then skip directly to @xref{Using ASDF}. That will probably be enough.
You are unlikely to have to worry about the way ASDF stores object files,
and resetting the ASDF configuration is usually only needed in corner cases.


@menu
* Configuring ASDF to find your systems::
* Configuring ASDF to find your systems --- old style::
* Configuring where ASDF stores object files::
* Resetting the ASDF configuration::
@end menu

@node Configuring ASDF to find your systems, Configuring ASDF to find your systems --- old style, Configuring ASDF, Configuring ASDF
@section Configuring ASDF to find your systems

In order to compile and load your systems, ASDF must be configured to find
the @file{.asd} files that contain system definitions.

There are a number of different techniques for setting yourself up with
ASDF, starting from easiest to the most complex:

@itemize @bullet

@item
Put all of your systems in one of the standard locations,
subdirectories of
@itemize
@item
@file{~/common-lisp/} or
@item
@file{~/.local/share/common-lisp/source/}.
@end itemize
If you install software there, you don't need further
configuration.@footnote{
  @file{~/common-lisp/} is only included in
  the default configuration
  starting with ASDF 3.1.2 or later.
  If your implementation provides an earlier variant of ASDF,
  you may need to explicitly configure it to use this path,
  as further explained.
}
You can then skip to the next section. @xref{Loading a system}.

@item
If you're using some tool to install software (e.g. Quicklisp),
the authors of that tool should already have configured ASDF.

@item
If you have more specific desires about how to lay out your software on
disk, the preferred way to configure where ASDF finds your systems is
the @code{source-registry} facility,
fully described in its own chapter of this manual.
@xref{Controlling where ASDF searches for systems}.
Here is a quick recipe for getting started.

First create the directory
@file{~/.config/common-lisp/source-registry.conf.d/}@footnote{
  For Windows users, and starting with ASDF 3.1.5, start from your
  @file{%LOCALAPPDATA%}, which is usually @file{~/AppData/Local/}
  (but you can ask in a @code{CMD.EXE} terminal
  @code{echo %LOCALAPPDATA%} to make sure)
  and underneath create a subpath
  @file{config/common-lisp/source-registry.conf.d/}.
};
there create a file with any name of your choice
but with the type @file{conf}@footnote{
  By requiring the @file{.conf}
  extension, and ignoring other files, ASDF allows you to have disabled files,
  editor backups, etc. in the same directory with your active
  configuration files.

  ASDF will also ignore files whose names start with a @file{.} character.

  It is customary to start the filename with two digits, to control the
  sorting of the @code{conf} files in the source registry directory, and
  thus the order in which the directories will be scanned.
},
for instance @file{50-luser-lisp.conf};
in this file, add the following line
to tell ASDF to recursively scan all the subdirectories under @file{/home/luser/lisp/}
for @file{.asd} files:
@kbd{(:tree "/home/luser/lisp/")}

That's enough. You may replace @file{/home/luser/lisp/} by wherever you want to install your source code.
You don't actually need to specify anything if you use the default @file{~/common-lisp/} as above
and your implementation provides ASDF 3.1.2 or later.
If your implementation provides an earlier variant of ASDF 3,
you might want to specify @kbd{(:tree (:home "common-lisp/"))} for bootstrap purposes,
then install a recent source tree of ASDF under @file{~/common-lisp/asdf/}.

If you prefer to use a ``link farm'', which is faster to use but costlier to manage than a recursive traversal,
say at @file{/home/luser/.asd-link-farm/}, then
you may instead (or additionally) create a file @file{42-asd-link-farm.conf}, containing the line:
@kbd{(:directory "/home/luser/.asd-link-farm/")}

ASDF will automatically read your configuration
the first time you try to find a system.
If necessary, you can reset the source-registry configuration with:

@lisp
(asdf:clear-source-registry)
@end lisp

@item
In earlier versions of ASDF, the system source registry was configured
using a global variable, @code{asdf:*central-registry*}.
For more details about this, see the following section,
@ref{Configuring ASDF to find your systems --- old style}.
Unless you need to understand this,
skip directly to @ref{Configuring where ASDF stores object files}.

@end itemize

Note that your Operating System distribution or your system administrator
may already have configured system-managed libraries for you.



@node Configuring ASDF to find your systems --- old style, Configuring where ASDF stores object files, Configuring ASDF to find your systems, Configuring ASDF
@section Configuring ASDF to find your systems --- old style


@c FIXME: this section should be moved elsewhere.  The novice user
@c should not be burdened with it. [2014/02/27:rpg]

Novices may skip this section.
Please @emph{do not} use the central-registry if you are a novice,
and @emph{do not} instruct novices to use the central-registry.
@c ``Experts may read it then proceed to ...''
@c some better section explaining
@c *central-registry* vs source-registry vs *system-definition-search-functions*,
@c and .../asdf/tools/cl-source-registry-cache.lisp

The old way to configure ASDF to find your systems is by
@code{push}ing directory pathnames onto the variable
@code{asdf:*central-registry*}.

You @emph{must} configure this variable @emph{after} you load ASDF 3 or later,
yet @emph{before} the first time you try to use it.
This loading and configuring of ASDF must happen
as part of some initialization script:
typically, either a script you maintain that builds your project,
or your implementation's initialization script
(e.g. @file{~/.sbclrc} for SBCL).

Also, if you are using an ancient ASDF 2 or earlier to load ASDF 3 or later,
then after it loads the ancient ASDF, your script @emph{must} configure
the central-registry a first time to tell ASDF 1 or 2 where to find ASDF 3,
then load ASDF 3 with e.g. @code{(asdf:operate 'asdf:load-op "asdf")},
then configure the central-registry again, because
ASDF 3 will not preserve the central-registry from ASDF 2 when upgrading.
You should probably be using the source-registry instead, which will be preserved
(unless you manually called @code{asdf:initialize-source-registry} with an argument,
in which case you will have to do it again indeed).
However, if you are using an ancient ASDF 2 or earlier,
we @emph{strongly} recommend that you should instead upgrade your implementation,
or overwrite the ancient ASDF installation with a more recent one:
@xref{Replacing your implementation's ASDF}.

The @code{asdf:*central-registry*} is empty by default in ASDF 2 or ASDF 3,
but is still supported for compatibility with ASDF 1.
When used, it takes precedence over the above source-registry.@footnote{
It is possible to further customize
the system definition file search.
That's considered advanced use, and covered later:
search forward for
@code{*system-definition-search-functions*}.
@xref{Defining systems with defsystem}.}

For example, let's say you want ASDF to find the @file{.asd} file
@file{/home/me/src/foo/foo.asd}.
In your Lisp initialization file, you could have the following:

@lisp
(require "asdf")
(push "/home/me/src/foo/" asdf:*central-registry*)
@end lisp

Note the trailing slash: when searching for a system,
ASDF will evaluate each entry of the central registry
and coerce the result to a pathname.@footnote{
ASDF will indeed call @code{eval} on each entry.
It will skip entries that evaluate to @code{nil}.

Strings and pathname objects are self-evaluating,
in which case the @code{eval} step does nothing;
but you may push arbitrary s-expressions onto the central registry.
These s-expressions may be evaluated to compute context-dependent
entries, e.g. things that depend
on the value of shell variables or the identity of the user.

The variable @code{asdf:*central-registry*} is thus a list of
``system directory designators''.
A @dfn{system directory designator} is a form
which will be evaluated whenever a system is to be found,
and must evaluate to a directory to look in (or @code{nil}).
By ``directory'', we mean
``designator for a pathname with a non-empty DIRECTORY component''.
}
The trailing directory name separator
is necessary to tell Lisp that you're discussing a directory
rather than a file.  If you leave it out, ASDF is likely to look in
@code{/home/me/src/} instead of @code{/home/me/src/foo/} as you
intended, and fail to find your system definition.
Modern versions of ASDF will issue an error and offer you to
remove such entries from the central-registry.

Typically there are a lot of @file{.asd} files, and
a common idiom was to put
@emph{symbolic links} to all of one's @file{.asd} files
in a common directory
and push @emph{that} directory (the ``link farm'')
onto
@code{asdf:*central-registry*},
instead of pushing each individual system directory.

ASDF knows to follow @emph{symlinks}
to the actual location of the systems.@footnote{
  On Windows, you can use Windows shortcuts instead of POSIX symlinks.
  if you try aliases under MacOS, we are curious to hear about your experience.
}

For example, if @code{#p"/home/me/cl/systems/"}
is an element of @code{*central-registry*}, you could set up the
system @var{foo} as follows:

@example
$ cd /home/me/cl/systems/
$ ln -s ~/src/foo/foo.asd .
@end example

This old style for configuring ASDF is not recommended for new users,
but it is supported for old users, and for users who want a simple way to
programmatically control what directories are added to the ASDF search path.


@node Configuring where ASDF stores object files, Resetting the ASDF configuration, Configuring ASDF to find your systems --- old style, Configuring ASDF
@section Configuring where ASDF stores object files
@findex clear-output-translations

ASDF lets you configure where object files will be stored.
Sensible defaults are provided and
you shouldn't normally have to worry about it.

This allows the same source code repository to be shared
between several versions of several Common Lisp implementations,
between several users using different compilation options,
with users who lack write privileges on shared source directories, etc.
This also keeps source directories from being cluttered
with object/fasl files.

Starting with ASDF 2, the @code{asdf-output-translations} facility
was added to ASDF itself.  This facility controls where object files will be stored.
This facility is fully described in a chapter of this manual,
@ref{Controlling where ASDF saves compiled files}.

@c FIXME: possibly this should be moved elsewhere.  It's redundant here,
@c and makes this section of the manual too long and daunting for the
@c new user. [2014/02/27:rpg]
@c The simplest way to add a translation to your search path,
@c say from @file{/foo/bar/baz/quux/}
@c to @file{/where/i/want/my/fasls/}
@c is to create the directory
@c @file{~/.config/common-lisp/asdf-output-translations.conf.d/}
@c and there create a file with any name of your choice and the type @file{conf},
@c for instance @file{42-bazquux.conf}
@c containing the line:

@c @kbd{("/foo/bar/baz/quux/" "/where/i/want/my/fasls/")}

@c To disable output translations for source under a given directory,
@c say @file{/toto/tata/}
@c you can create a file @file{40-disable-toto.conf}
@c with the line:

@c @kbd{("/toto/tata/")}

@c To wholly disable output translations for all directories,
@c you can create a file @file{00-disable.conf}
@c with the line:

@c @kbd{(t t)}

@c Note that your Operating System distribution or your system administrator
@c may already have configured translations for you.
@c In absence of any configuration, the default is to redirect everything
@c under an implementation-dependent subdirectory of @file{~/.cache/common-lisp/}.
@c @xref{Controlling where ASDF searches for systems}, for full details.

@c The required @file{.conf} extension allows you to have disabled files
@c or editor backups (ending in @file{~}), and works portably
@c (for instance, it is a pain to allow both empty and non-empty extension on CLISP).
@c Excluded are files the name of which start with a @file{.} character.
@c It is customary to start the filename with two digits
@c that specify the order in which the directories will be scanned.

@c ASDF will automatically read your configuration
@c the first time you try to find a system.
@c You can reset the source-registry configuration with:

@c @lisp
@c (asdf:clear-output-translations)
@c @end lisp

@c And you probably should do so before you dump your Lisp image,
@c if the configuration may change
@c between the machine where you save it at the time you save it
@c and the machine you resume it at the time you resume it.
@c (Once again, you should use @code{(asdf:clear-configuration)}
@c before you dump your Lisp image, which includes the above.)

Note that before ASDF 2,
other ASDF add-ons offered the same functionality,
each in subtly different and incompatible ways:
ASDF-Binary-Locations, cl-launch, common-lisp-controller.
ASDF-Binary-Locations is now not needed anymore and should not be used.
cl-launch 3.000 and common-lisp-controller 7.2 have been updated
to delegate object file placement to ASDF.

@node Resetting the ASDF configuration,  , Configuring where ASDF stores object files, Configuring ASDF
@section Resetting the ASDF configuration

@c FIXME: this should probably be moved out of the "quickstart" part of
@c the manual. [2014/02/27:rpg]


When you dump and restore an image, or when you tweak your configuration,
you may want to reset the ASDF configuration.
For that you may use the following function:

@defun clear-configuration
   Undoes any ASDF configuration
   regarding source-registry or output-translations.
@end defun

@vindex *image-dump-hook*
This function is pushed onto the @code{uiop:*image-dump-hook*} by default,
which means that if you save an image using @code{uiop:dump-image},
or via @code{asdf:image-op} and @code{asdf:program-op},
it will be automatically called to clear your configuration.
If for some reason you prefer to call your implementation's underlying functionality,
be sure to call @code{clear-configuration} manually,
or push it into your implementation's equivalent of @code{uiop:*image-dump-hook*},
e.g. @code{sb-ext:*save-hooks*} on SBCL, or @code{ext:*before-save-initializations*}
on CMUCL and SCL, etc.

@node  Using ASDF, Defining systems with defsystem, Configuring ASDF, Top
@chapter Using ASDF

@menu
* Loading a system::
* Convenience Functions::
* Moving on::
@end menu

@node Loading a system, Convenience Functions, Using ASDF, Using ASDF
@section Loading a system

The system @var{foo} is loaded (and compiled, if necessary)
by evaluating the following Lisp form:

@example
(asdf:load-system :@var{foo})
@end example

On some implementations (@pxref{Convenience Functions}),
ASDF hooks into the @code{cl:require} facility and you can just use:

@example
(require :@var{foo})
@end example

Note that the canonical name of a system is a string, in @emph{lowercase}.
System names can also be specified as symbols (including keyword
symbols).
If a symbol is given as argument, its package is ignored,
its @code{symbol-name} is taken, and converted to lowercase.
The name must be a suitable value for the @code{:name} initarg
to @code{make-pathname} in whatever filesystem the system is to be found.

Using lowercase as canonical is unconventional,
but was selected after some consideration.
The type of file systems we  support
either have lowercase as customary case (Unix, Mac, Windows)
or silently convert lowercase to uppercase (lpns).
@c so this makes more sense than attempting to use @code{:case :common},
@c which is reported not to work on some implementations


@node Convenience Functions, Moving on, Loading a system, Using ASDF
@section Convenience Functions

@c I believe thes are all unnecessary because of the function macros
@c below [2016/01/30:rpg]
@c @findex load-system
@c @findex compile-system
@c @findex test-system
@c @findex require-system
@c @findex make

ASDF provides three commands for the most common system operations:
@code{load-system}, @code{compile-system}, and @code{test-system}.

ASDF also provides @code{require-system}, a variant of @code{load-system}
that skips loading systems that are already loaded.  This is sometimes
useful, for example, in order to avoid re-loading libraries that come
pre-loaded into your  lisp implementation.

ASDF also provides @code{make}, a way of allowing system developers to
choose a default operation for their systems.  For example, a developer
who has created a system intended to format a specific document, might
make document-formatting the default operation invoked by @code{make},
instead of loading.  If the system developer doesn't specify in the
system definition, the default operation will be loading.

@c FIXME: We seem to export @findex bundle-system also, that some ECL users seem to rely on.
@c But it's probably better that bundle operations have their own manual chapter at some point.


@c FIXME: There should be a @defun for OPERATE, but there isn't.  Not
@c sure where it belongs...  The discussion here is just confusing if
@c the reader doesn't understand how ASDF works. [2016/01/30:rpg]
@findex operate
@findex oos

Because ASDF is an extensible system
for defining @emph{operations} on @emph{components},
it also provides a generic function @code{operate},
so you may arbitrarily operate on your systems beyond the default operations.
(At the interactive REPL, users often use its shorter alias @code{oos},
which stands for @code{operate-on-system}, a name inherited from @code{mk-defsystem}.)
You'll use @code{operate} whenever you want to do something beyond
compiling, loading and testing.

@c Reminder: before ASDF can operate on a system, however,
@c it must be able to find and load that system's definition.
@c @xref{Configuring ASDF to find your systems}.

@c FIXME: the following is too complicated for here, especially since
@c :force hasn't been defined yet.  Move it. [2014/02/27:rpg]

@defun load-system system @Arest{} keys @Akey{} force force-not verbose version @AallowOtherKeys{}
Apply @code{operate} with the operation @code{load-op}, the
@var{system}, and any provided keyword arguments.  Calling
@code{load-system} is the regular, recommended way to load a system
into the current image.
@end defun

@defun compile-system system @Arest{} keys @Akey{} force force-not verbose version @AallowOtherKeys{}
Apply @code{operate} with the operation @code{compile-op},
the @var{system}, and any provided keyword arguments.
This will make sure all the files in the system are compiled,
but not necessarily load any of them in the current image;
on most systems, it will @emph{not} load all compiled files in the current image.
This function exists for symmetry with @code{load-system} but is not recommended
unless you are writing build scripts and know what you're doing.
But then, you might be interested in @code{program-op} rather than @code{compile-op}.
@end defun

@defun test-system system @Arest{} keys @Akey{} force force-not verbose version @AallowOtherKeys{}
Apply @code{operate} with the operation @code{test-op},
the @var{system}, and any provided keyword arguments.
@xref{test-op}.
@end defun

@defun make system @Arest{} keys @Akey{} @AallowOtherKeys{}
Do ``The Right Thing'' with your system.
Starting with ASDF 3.1, this function @code{make} is also available.
The default behaviour is to load the system as if by @code{load-system};
but system authors can override this default in their system definition
they may specify an alternate operation as the intended use of their system,
with a @code{:build-operation} option in the @code{defsystem} form
(@pxref{Build-operation}),
and an intended output pathname for that operation with
@code{:build-pathname}.
@c Document :build-operation in the defsystem section.
@c Document in the extension section that for richer programmatic access, you may instead use an overriding
@c @code{(defmethod component-depends-on ((o build-op) (s system))
@c ...)}.
This function is experimental and largely untested.  Use at your own risk.
@end defun
@cindex build-operation

@defun require-system system @Arest{} keys @Akey{} @AallowOtherKeys{}
@code{require-system} skips any update to systems that have already been loaded,
in the spirit of @code{cl:require}.
It does it by calling @code{load-system} with a keyword option
excluding already loaded systems.@footnote{
  For the curious, the option is @code{:force-not (already-loaded-systems)}.
}.
On actively maintained free software implementations
(namely recent versions of ABCL, Clozure CL, CMUCL, ECL, GNU CLISP, MKCL and SBCL),
once ASDF itself is loaded, @code{cl:require} too can load ASDF systems,
by falling back on @code{require-system}
for module names not recognized by the implementation.
(Note however that @code{require-system} does @emph{not} fall back on @code{cl:require};
that would introduce an ``interesting'' potential infinite loop to break somehow.)

@code{cl:require} and @code{require-system} are appropriate to load code
that is not being modified during the current programming session.
@code{cl:require} will notably load the implementation-provided extension modules;
@code{require-system} won't, unless they are also defined as systems somehow,
which SBCL and MKCL do.
@code{require-system} may also be used to load any number of ASDF systems
that the user isn't either developing or debugging,
for which a previously installed version is deemed to be satisfactory;
@code{cl:require} on the above-mentioned implementations will delegate to @code{require-system}
and may load them as well.
But for code that you are actively developing, debugging, or otherwise modifying,
you should use @code{load-system}, so ASDF will pick on your modifications
and transitively re-build the modified files and everything that depends on them
(that the requested @var{system} itself depends on ---
ASDF itself never builds anything unless
it's an explicitly requested system or the dependencies thereof).
@end defun

@defun already-loaded-systems
Returns a list of names of the systems that have been successfully
loaded so far.
@end defun




@node Moving on,  , Convenience Functions, Using ASDF
@section Moving on

That's all you need to know to use ASDF to load systems written by others.
The rest of this manual deals with writing system definitions
for Common Lisp software you write yourself,
including how to extend ASDF to define new operation and component types.


@node Defining systems with defsystem, The object model of ASDF, Using ASDF, Top
@comment  node-name,  next,  previous,  up
@chapter Defining systems with defsystem

This chapter describes how to use ASDF to define systems and develop
software.


@menu
* The defsystem form::
* A more involved example::
* The defsystem grammar::
* Other code in .asd files::
* The package-inferred-system extension::
@end menu

@node  The defsystem form, A more involved example, Defining systems with defsystem, Defining systems with defsystem
@comment  node-name,  next,  previous,  up
@section The defsystem form
@findex defsystem
@cindex asdf-user
@findex load-asd

This section begins with an example of a system definition,
then gives the full grammar of @code{defsystem}.

Let's look at a simple system.
This is a complete file that should be saved as @file{hello-lisp.asd}
(in order that ASDF can find it
when ordered to operate on the system named @code{"hello-lisp"}).

@lisp
;; Usual Lisp comments are allowed here

(defsystem "hello-lisp"
  :description "hello-lisp: a sample Lisp system."
  :version "0.0.1"
  :author "Joe User <joe@@example.com>"
  :licence "Public Domain"
  :depends-on ("optima.ppcre" "command-line-arguments")
  :components ((:file "packages")
               (:file "macros" :depends-on ("packages"))
               (:file "hello" :depends-on ("macros"))))
@end lisp

Some notes about this example:

@itemize

@item
The @code{defsystem} form defines a system named @code{hello-lisp}
that contains three source files:
@file{packages.lisp}, @file{macros.lisp} and @file{hello.lisp}.

@item
The @file{.lisp} suffix is implicit for Lisp source files.
The source files are located in the same directory
as the @code{.asd} file with the system definition.
@c FIXME: the following should live somewhere, but not in the quickstart
@c page. [2014/05/03:rpg]
@c ASDF resolves symbolic links (or Windows shortcuts)
@c before loading the system definition file and
@c stores its location in the resulting system@footnote{
@c It is possible, though almost never necessary, to override this behaviour.}.
@c This is a good thing because the user can move the system sources
@c without having to edit the system definition.

@c FIXME: The first example system should probably use just :serial T.
@item
The file @file{macros} depends on @file{packages}
(presumably because the package it's in is defined in @file{packages}),
and the file @file{hello} depends on @file{macros}
(and hence, transitively on @file{packages}).
This means that ASDF will compile and load @file{packages} then @file{macros}
before starting the compilation of file @file{hello}.

@item
This example system has external dependencies on two other systems,
@code{optima.ppcre} (that provides a friendly interface to matching regular expressions),
and @code{command-line-arguments} (that provides a way to parse arguments passed from the shell command line).
To use this system, ASDF must be configured to find installed copies of these systems;
it will load them before it tries to compile and load @code{hello-lisp}.

@item
This system also defines a bunch of metadata.
While it is optional to define these fields
(and other fields like @code{:bug-tracker}, @code{:mailto}, @code{:long-name},
@code{:long-description}, @code{:source-control}),
it is strongly recommended to define the fields @code{:description}, @code{:version}, @code{:author}, and @code{:licence},
especially if you intend your software to be eventually included in Quicklisp.

@c FIXME: Should have cross-reference to "Version specifiers" in the
@c defsystem grammar, but the cross-referencing is so broken by
@c insufficient node breakdown that I have not put one in.
@c FIXME: this is way too detailed for the first example!
@c move it!
@item
Make sure you know how the @code{:version} numbers will be parsed!
Only period-separated non-negative integers are accepted at present.
@xref{Version specifiers}.

@item
This file contains a single form, the @code{defsystem} declaration.
No @code{in-package} form, no @code{asdf:} package prefix, no nothing.
Just the one naked @code{defsystem} form.
This is what we recommend.
More complex system definition files are possible with arbitrary Lisp code,
but we recommend that you keep it simple if you can.
This will make your system definitions more robust and more future-proof.

@cindex :version

@end itemize

This is all you need to know to define simple systems.
The next example is much more involved, to give you a glimpse of how you can do more complex things.
However, since it's ultimately arbitrary Lisp code, there is no bottom to the rabbit hole.
@c FIXME: divide the next example into many examples, to introduce fewer concepts at once.


@node  A more involved example, The defsystem grammar, The defsystem form, Defining systems with defsystem
@comment  node-name,  next,  previous,  up
@section A more involved example
@findex defsystem

Let's illustrate some more involved uses of @code{defsystem} via a
slightly convoluted example:

@lisp
(in-package :asdf-user)

(defsystem "foo"
  :version (:read-file-form "variables" :at (3 2))
  :components
  ((:file "package")
   (:file "variables" :depends-on ("package"))
   (:module "mod"
     :depends-on ("package")
     :serial t
     :components ((:file "utils")
                  (:file "reader")
                  (:file "cooker")
                  (:static-file "data.raw"))
     :output-files (compile-op (o c) (list "data.cooked"))
     :perform (compile-op :after (o c)
        (cook-data
         :in (component-pathname (find-component c "data.raw"))
         :out (first (output-files o c)))))
   (:file "foo" :depends-on ("mod"))))

(defmethod action-description
    ((o compile-op) (c (eql (find-component "foo" "mod"))))
  "cooking data")
@end lisp

Here are some notes about this example:

@itemize

@item
The main thing this file does is define a system @code{foo}.
It also contains other Lisp forms, which we'll examine below.

@item
Besides Lisp source files, this system contains a @code{:module} component
named @code{"mod"}, which is a collection of three Lisp source files
@file{utils.lisp}, @file{reader.lisp}, @file{cooker.lisp} and @file{data.raw}

@item
Note that the @code{:static-file} does not have an implicit file type,
unlike the Lisp source files.

@item
This files will be located in a subdirectory of the main code directory named
@file{mod/} (this location could have been overridden to be
in the same directory, or in a different subdirectory;
see the discussion of the @code{:pathname} option in @ref{The defsystem grammar}).

@item
The @code{:serial t} says that each sub-component of @code{mod} depends on the previous components,
so that @file{cooker.lisp} depends-on @file{reader.lisp}, which depends-on @file{utils.lisp}.
Also @file{data.raw} depends on all of them, but that doesn't matter since it's a static file;
on the other hand, if it appeared first, then all the Lisp files would be recompiled
when the data is modified, which is probably not what is desired in this case.

@item
The method-form tokens provide a shorthand for defining methods on
particular components.  This part

@lisp
     :output-files (compile-op (o c) (list "data.cooked"))
     :perform (compile-op :after (o c)
        (cook-data
         :in (component-pathname (find-component c "data.raw"))
         :out (first (output-files o c))))
@end lisp

has the effect of

@lisp
(defmethod output-files ((o compile-op) (c (eql ...)))
  (list "data.cooked"))
(defmethod perform :after ((o compile-op) (c (eql ...)))
  (cook-data
   :in (component-pathname (find-component c "data.raw"))
   :out (first (output-files o c))))
@end lisp

where @code{...} is the component in question.
In this case @code{...} would expand to something like

@lisp
(find-component "foo" "mod")
@end lisp

For more details on the syntax of such forms,
@pxref{The defsystem grammar}.
For more details on what these methods do, @pxref{Operations} in
@ref{The object model of ASDF}.

@item
There is an additional @code{defmethod} with a similar effect,
because ASDF (as of ASDF 3.1.5)
fails to accept inline-methods as above for @code{action-description},
instead only supporting the deprecated @code{explain} interface.

@c FIXME: The following plunge into detail weeds is not appropriate in this
@c location. [2010/10/03:rpg]
@c note that although this also supports @code{:before} methods,
@c they may not do what you want them to ---
@c a @code{:before} method on perform @code{((op compile-op) (c (eql ...)))}
@c will run after all the dependencies and sub-components have been processed,
@c but before the component in question has been compiled.

@item
In this case, these methods describe how this module defines code
that it then uses to cook some data.

@item
Importantly, ASDF is told about the input and output files
used by the data cooker,
and to make sure everyone agrees,
the cooking function explicitly uses ASDF to access pathnames
to the input and output data.

@c FIXME: move most of this package discussion to its own section,
@c and leave only a reference here.

@item
The file starts with a form @code{(in-package :asdf-user)},
but it is actually redundant, not necessary and not recommended.
But yet more complex cases (also not recommended) may usefully use an @code{in-package} form.

@item
Indeed, ASDF does not load @file{.asd} files simply with @code{cl:load},
and neither should you.
You should let ASDF find and load them when you operate on systems.
If you somehow @emph{must} load a @file{.asd} file,
use the same function @code{asdf:load-asd} that ASDF uses.
Among other things, it already binds the @code{*package*} to @code{asdf-user}.
Recent versions of SLIME (2013-02 and later) know to do that when you @kbd{C-c C-k}
when you use the @code{slime-asdf} contrib.

@item
You shouldn't use an @code{in-package} form
if you're keeping things simple.
You should only use @code{in-package} (and before it, a @code{defpackage})
when you're going to define new classes, functions, variables, macros, etc.,
in the @code{.asd} file, and want to thereby avoid name clashes.
Manuals for old versions of ASDF recommended use of such an idiom in @file{.asd} files,
but as of ASDF 3, we recommend that you don't do that anymore,
and instead define any ASDF extensions in their own system,
on which you can then declare a dependency using @code{:defsystem-depends-on}.
@xref{The defsystem grammar}.

@item
More generally, you can always rely on symbols
from packages @code{asdf}, @code{common-lisp} and @code{uiop}
being available in @code{.asd} files ---
most importantly including @code{defsystem}.
It is therefore redundant and in bad taste to use a package-prefixed @code{asdf:defsystem} symbol
in a @file{.asd} file.
Just use @code{(defsystem ...)}.
Only package-prefix it when somehow dynamically generating system definitions
from a package that doesn't already use the ASDF package.

@item
@code{asdf-user} is actually only available starting since ASDF 3, but then again,
ASDF 1 and 2 did crazy things with packages that ASDF 3 has stopped doing@footnote{
  ASDF 1 and 2 (up until 2.26)
  used to dynamically create and delete temporary packages @code{asdf@emph{N}},
  one for each @file{.asd} file, in a misguided attempt to thereby reduce name clashes;
  but it failed at that goal and only made things more complex.
  ASDF 3 just uses a shared package @code{asdf-user} instead,
  and relies on the usual Common Lisp conventions to avoid clashes.
  As far as package oddities go, you may just notice that
  the @code{asdf-user} package also uses @code{uiop/common-lisp},
  a variant of the @code{common-lisp} package that papers over
  deficiencies in more obscure Common Lisp implementations;
  but unless you care about Corman Lisp, GCL, Genera or MCL, you shouldn't be concerned.
},
and since all implementations provide ASDF 3, you shouldn't care about compatibility with ASDF 2.
We do not support ASDF 2 anymore, and we recommend that neither should you.

@item
Starting with ASDF 3.1, @code{asdf-user} uses @code{uiop},
whereas in earlier variants of ASDF 3 it only used @code{uiop/package}.
We recommend you either prefix use of UIOP functions with the package prefix @code{uiop:},
or make sure your system @code{:depends-on ((:version "asdf" "3.1.2"))}
or has a @code{#-asdf3.1 (error "MY-SYSTEM requires ASDF 3.1.2")}.

@item
Finally, we elided most metadata, but showed how you can have ASDF automatically extract
the system's version from a source file. In this case, the 3rd subform of the 4th form
(note that Lisp uses 0-based indexing, English uses 1-based indexing).
Presumably, the 4th form looks like @code{(defparameter *foo-version* "5.6.7")}.

@end itemize


@node  The defsystem grammar, Other code in .asd files, A more involved example, Defining systems with defsystem
@comment  node-name,  next,  previous,  up
@section The defsystem grammar
@findex defsystem
@cindex DEFSYSTEM grammar

@macro defrule {NAME}
@anchor{rule-\NAME\}\NAME\ :=
@end macro

@ifhtml
@macro refrule {NAME}
@ref{rule-\NAME\,\NAME\}
@end macro
@end ifhtml

@ifinfo
@macro refrule {NAME}
@ref{rule-\NAME\,\NAME\,\NAME\,\NAME\}
@end macro
@end ifinfo

@ifnothtml
@ifnotinfo
@macro refrule {NAME}
\NAME\
@end macro
@end ifnotinfo
@end ifnothtml

@example
@defrule{system-definition} ( defsystem @refrule{system-designator} @refrule{system-option}* )

@defrule{system-designator} @refrule{simple-component-name}
                   | @refrule{complex-component-name}

# NOTE: Underscores are not permitted.
# @pxref{Simple component names,Simple component names,Simple component names}
@defrule{simple-component-name} @var{lower-case string} | @var{symbol}

# @pxref{Complex component names,Complex component names,Complex component names}
@defrule{complex-component-name} @var{string} | @var{symbol}

@defrule{system-option} :defsystem-depends-on @refrule{dependency-def}
               | :weakly-depends-on @refrule{system-list}
               | :class @var{class-name} # @pxref{System class names}
               | :build-pathname @refrule{pathname-specifier}
               | :build-operation @refrule{operation-name}
               | @refrule{system-option/asdf3}
               | @refrule{module-option}
               | @refrule{option}

# These are only available since ASDF 3 (actually its alpha release
# 2.27)
@defrule{system-option/asdf3} :homepage @var{string}
                     | :bug-tracker @var{string}
                     | :mailto @var{string}
                     | :long-name @var{string}
                     | :source-control @refrule{source-control}
                     | :version @refrule{version-specifier}
                     | :entry-point @var{object} # @pxref{Entry point}

@defrule{source-control} ( @var{keyword} @var{string} )

@defrule{module-option} :components @refrule{component-list}
               | :serial [ t | nil ]

@defrule{option} :description @var{string}
        | :long-description @var{string}
        | :author @refrule{person-or-persons}
        | :maintainer @refrule{person-or-persons}
        | :pathname @refrule{pathname-specifier}
        | :default-component-class @var{class-name}
        | :perform @refrule{method-form}
        | :explain @refrule{method-form}
        | :output-files @refrule{method-form}
        | :operation-done-p @refrule{method-form}
        | :if-feature @refrule{feature-expression}
        | :depends-on ( @refrule{dependency-def}* )
        | :in-order-to ( @refrule{dependency}+ )

@defrule{person-or-persons} @var{string} | ( @var{string}+ )

@defrule{system-list} ( @refrule{simple-component-name}* )

@defrule{component-list} ( @refrule{component-def}* )

@defrule{component-def} ( @refrule{component-type} @refrule{simple-component-name} @refrule{option}* )

@defrule{component-type} :module | :file | :static-file | @refrule{other-component-type}

@defrule{other-component-type} symbol-by-name # @pxref{Component types}

# This is used in :depends-on, as opposed to "dependency", which is used
# in :in-order-to
@defrule{dependency-def} @refrule{simple-component-name}
                | ( :feature @refrule{feature-expression} @refrule{dependency-def} ) # @pxref{Feature dependencies}
                | ( :version @refrule{simple-component-name} @refrule{version-specifier} )
                | ( :require @var{module-name} )

# "dependency" is used in :in-order-to, as opposed to "dependency-def"
@defrule{dependency} ( @refrule{dependent-op} @refrule{requirement}+ )
@defrule{requirement} ( @refrule{required-op} @var{required-component}+ )
@defrule{dependent-op} @refrule{operation-name}
@defrule{required-op} @refrule{operation-name}

# NOTE: pathnames should be all lower case, and have no underscores,
# although hyphens are permitted.
@defrule{pathname-specifier} @var{pathname} | @var{string} | @var{symbol}

@defrule{version-specifier} @var{string}
                   | ( :read-file-form @refrule{pathname-specifier} @refrule{form-specifier}? )
                   | ( :read-file-line @refrule{pathname-specifier} @refrule{line-specifier}? )
@defrule{line-specifier} :at @var{integer} # base zero
@defrule{form-specifier} :at [ @var{integer} | ( @var{integer}+ ) ]

@defrule{method-form} ( @refrule{operation-name} @refrule{qual} @var{lambda-list} @Arest{} @var{body} )
@defrule{qual} @refrule{method-qualifier}?
@defrule{method-qualifier} :before | :after | :around

@defrule{feature-expression} @var{keyword}
                    | ( :and @refrule{feature-expression}* )
                    | ( :or @refrule{feature-expression}* )
                    | ( :not @refrule{feature-expression} )

@defrule{operation-name} @var{symbol}
@end example

@subsection System designators

System designators are either simple component names, or
complex (``slashy'') component names.

@subsection Simple component names (@code{simple-component-name})
@anchor{Simple component names}

Simple component names may be written as either strings or symbols.

When using strings, use lower case exclusively.

Symbols will be interpreted as convenient shorthand for the string
that is their @code{symbol-name}, converted to lower case.  Put
differently, a symbol may be a simple component name @emph{designator},
but the simple component name itself is the string.

@strong{Never}  use underscores in  component names, whether written as
strings or symbols.

@strong{Never} use slashes (``/'') in simple component names.  A slash
indicates a @emph{complex} component name; see below.  Using a slash
improperly will cause ASDF to issue a warning.

Violating these constraints by mixing case, or including underscores in
component names, may lead to systems or components being impossible to
find, because component names are interpreted as file names.  These
problems will @emph{definitely} occur for users who have configured ASDF
using logical pathnames.

@subsection Complex component names
@anchor{Complex component names}

A complex component name is a master name followed by a slash, followed
by a subsidiary name.  This allows programmers to put multiple system
definitions in a single @code{.asd} file, while still allowing ASDF to
find these systems.

The master name of a complex system @strong{must} be the same as the
name of the @code{.asd} file.

The file @code{foo.asd} will contain the definition of the system
@code{"foo"}.  However, it may @emph{also} contain definitions of
subsidiary systems, such as @code{"foo/test"}, @code{"foo/docs"}, and so
forth.  ASDF will ``know'' that if you ask it to load system
@code{"foo/test"} it should look for that system's definition in @code{foo.asd}.

@subsection Component types
@anchor{Component types}

Component type names, even if expressed as keywords, will be looked up
by name in the current package and in the asdf package, if not found in
the current package.  So a component type @code{my-component-type}, in
the current package @code{my-system-asd} can be specified as
@code{:my-component-type}, or @code{my-component-type}.

@code{system} and its subclasses are @emph{not}
allowed as component types for such children components.

@subsection System class names
@anchor{System class names}

A system class name will be looked up
in the same way as a Component type (see above),
except that only @code{system} and its subclasses are allowed.
Typically, one will not need to specify a system
class name, unless using a non-standard system class defined in some
ASDF extension, typically loaded through @code{DEFSYSTEM-DEPENDS-ON},
see below.  For such class names in the ASDF package, we recommend that
the @code{:class} option be specified using a keyword symbol, such as

@example
:class :MY-NEW-SYSTEM-SUBCLASS
@end example

This practice will ensure that package name conflicts are avoided.
Otherwise, the symbol @code{MY-NEW-SYSTEM-SUBCLASS} will be read into
the current package @emph{before} it has been exported from the ASDF
extension loaded by @code{:defsystem-depends-on}, causing a name
conflict in the current package.

@subsection Defsystem depends on
@cindex :defsystem-depends-on

The @code{:defsystem-depends-on} option to @code{defsystem} allows the
programmer to specify another ASDF-defined system or set of systems that
must be loaded @emph{before} the system definition is processed.
Typically this is used to load an ASDF extension that is used in the
system definition.

@subsection Build-operation
@cindex :build-operation
@anchor{Build-operation}

The @code{:build-operation} option to @code{defsystem} allows the
programmer to specify an operation that will be applied, in place of
@code{load-op} when @code{make} (@pxref{Convenience Functions, make})
is run on the system.    The option
value should be the name of an operation.  E.g., @code{:build-operation doc-op}

This feature is
experimental and largely untested.  Use at your own risk.

@subsection Weakly depends on
@cindex :weakly-depends-on

We do @emph{NOT} recommend you use this feature.
If you are tempted to write a system @var{foo}
that weakly-depends-on a system @var{bar},
we recommend that you should instead
write system @var{foo} in a parametric way,
and offer some special variable and/or some hook to specialize its behaviour;
then you should write a system @var{foo+bar}
that does the hooking of things together.

The (deprecated) @code{:weakly-depends-on} option to @code{defsystem}
allows the programmer to specify another ASDF-defined system or set of systems
that ASDF should @emph{try} to load,
but need not load in order to be successful.
Typically this is used if there are a number of systems
that, if present, could provide additional functionality,
but which are not necessary for basic function.

Currently, although it is specified to be an option only to @code{defsystem},
this option is accepted at any component, but it probably
only makes sense at the @code{defsystem} level.
Programmers are cautioned not
to use this component option except at the @code{defsystem} level, as
this anomalous behaviour may be removed without warning.

@c Finally, you might look into the @code{asdf-system-connections} extension,
@c that will let you define additional code to be loaded
@c when two systems are simultaneously loaded.
@c It may or may not be considered good style, but at least it can be used
@c in a way that has deterministic behaviour independent of load order,
@c unlike @code{weakly-depends-on}.


@subsection Pathname specifiers
@cindex pathname specifiers
@anchor{Pathname specifiers}

A pathname specifier (@code{pathname-specifier})
may be a pathname, a string or a symbol.
When no pathname specifier is given for a component,
which is the usual case, the component name itself is used.

If a string is given, which is the usual case,
the string will be interpreted as a Unix-style pathname
where @code{/} characters will be interpreted as directory separators.
Usually, Unix-style relative pathnames are used
(i.e. not starting with @code{/}, as opposed to absolute pathnames);
they are relative to the path of the parent component.
Finally, depending on the @code{component-type},
the pathname may be interpreted as either a file or a directory,
and if it's a file,
a file type may be added corresponding to the @code{component-type},
or else it will be extracted from the string itself (if applicable).

For instance, the @code{component-type} @code{:module}
wants a directory pathname, and so a string @code{"foo/bar"}
will be interpreted as the pathname @file{#p"foo/bar/"}.
On the other hand, the @code{component-type} @code{:file}
wants a file of type @code{lisp}, and so a string @code{"foo/bar"}
will be interpreted as the pathname @file{#p"foo/bar.lisp"},
and a string @code{"foo/bar.quux"}
will be interpreted as the pathname @file{#p"foo/bar.quux.lisp"}.
Finally, the @code{component-type} @code{:static-file}
wants a file without specifying a type, and so a string @code{"foo/bar"}
will be interpreted as the pathname @file{#p"foo/bar"},
and a string @code{"foo/bar.quux"}
will be interpreted as the pathname @file{#p"foo/bar.quux"}.

ASDF interprets the string @code{".."}
as the pathname directory component word @code{:back},
which when merged, goes back one level in the directory hierarchy.

If a symbol is given, it will be translated into a string,
and downcased in the process.
The downcasing of symbols is unconventional,
but was selected after some consideration.
The file systems we  support
either have lowercase as customary case (Unix, Mac, Windows)
or silently convert lowercase to uppercase (lpns),
so this makes more sense than attempting to use @code{:case :common}
as argument to @code{make-pathname},
which is reported not to work on some implementations.

Please avoid using underscores in system names, or component (module or
file) names,  since underscores are not
compatible with logical pathnames (@pxref{Using logical pathnames}).

Pathname objects may be given to override the path for a component.
Such objects are typically specified using reader macros such as @code{#p}
or @code{#.(make-pathname ...)}.
Note however, that @code{#p...} is
a shorthand for @code{#.(parse-namestring ...)}
and that the behaviour of @code{parse-namestring} is completely non-portable,
unless you are using Common Lisp @code{logical-pathname}s,
which themselves involve other non-portable behaviour
(@pxref{Using logical pathnames}).
Pathnames made with @code{#.(make-pathname ...)}
can usually be done more easily with the string syntax above.
The only case that you really need a pathname object is to override
the component-type default file type for a given component.
Therefore, pathname objects should only rarely be used.
Unhappily, ASDF 1 used not to properly support
parsing component names as strings specifying paths with directories,
and the cumbersome @code{#.(make-pathname ...)} syntax had to be used.
An alternative to @code{#.} read-time evaluation is to use
@code{(eval `(defsystem ... ,pathname ...))}.

Note that when specifying pathname objects,
ASDF does not do any special interpretation of the pathname
influenced by the component type, unlike the procedure for
pathname-specifying strings.
On the one hand, you have to be careful to provide a pathname that correctly
fulfills whatever constraints are required from that component type
(e.g. naming a directory or a file with appropriate type);
on the other hand, you can circumvent the file type that would otherwise
be forced upon you if you were specifying a string.

@subsection Version specifiers
@cindex version specifiers
@cindex :version
@anchor{Version specifiers}

Version specifiers are strings to be parsed as period-separated lists of integers.
I.e., in the example, @code{"0.2.1"} is to be interpreted,
roughly speaking, as @code{(0 2 1)}.
In particular, version @code{"0.2.1"} is interpreted the same as @code{"0.0002.1"},
though the latter is not canonical and may lead to a warning being issued.
Also, @code{"1.3"} and @code{"1.4"} are both strictly @code{uiop:version<} to @code{"1.30"},
quite unlike what would have happened
had the version strings been interpreted as decimal fractions.

Instead of a string representing the version,
the @code{:version} argument can be an expression that is resolved to
such a string using the following trivial domain-specific language:
in addition to being a literal string, it can be an expression of the form
@code{(:read-file-form <pathname-or-string> [:at <access-at-specifier>])},
or @code{(:read-file-line <pathname-or-string> [:at <access-at-specifier>])}.
As the name suggests, the former will be resolved by reading a form in the specified pathname
(read as a subpathname of the current system if relative or a
unix-namestring), and the latter by reading a line.
You may use a @code{uiop:access-at} specifier
with the @code{:at} keyword,
by default the specifier is @code{0}, meaning the first form/line is
returned.
For @code{:read-file-form},
subforms can also be specified, with e.g. @code{(1 2 2)} specifying
``the third subform (index 2) of the third subform (index 2) of the second form (index 1)''
in the file (mind the off-by-one error in the English language).

System definers are encouraged to use version identifiers of the form
@var{x}.@var{y}.@var{z} for
major version, minor version and patch level,
where significant API incompatibilities are signaled by an increased major number.

@xref{Common attributes of components}.

@subsection Require
@cindex :require dependencies

Use the implementation's own @code{require} to load the @var{module-name}.

It is good taste to use @code{(:feature @emph{:implementation-name} (:require @var{module-name}))}
rather than @code{#+@emph{implementation-name} (:require @var{module-name})}
to only depend on the specified module on the specific implementation that provides it.
@xref{Feature dependencies}.

@subsection Feature dependencies
@cindex :feature dependencies
@anchor{Feature dependencies}

A feature dependency is of the form
@code{(:feature @var{feature-expression} @var{dependency})}
If the @var{feature-expression} is satisfied by the running lisp at the
time the system definition is parsed, then the @var{dependency} will be
added to the system's dependencies.  If the @var{feature-expression} is
@emph{not} satisfied, then the feature dependency form is ignored.

Note that this means that @code{:feature} @strong{cannot} be used to
enforce a feature dependency for the system in question.  I.e., it
cannot be used to require that a feature hold in order for the system
definition to be loaded.  E.g., one cannot use @code{(:feature :sbcl)}
to require that a system only be used on SBCL.

Feature dependencies are not to be confused with the obsolete
feature requirement (@pxref{feature requirement}), or
with @code{if-feature}.

@subsection Using logical pathnames
@cindex logical pathnames
@anchor{Using logical pathnames}

We do not generally recommend the use of logical pathnames,
especially not so to newcomers to Common Lisp.
However, we do support the use of logical pathnames by old timers,
when such is their preference.

To use logical pathnames,
you will have to provide a pathname object as a @code{:pathname} specifier
to components that use it, using such syntax as
@code{#p"LOGICAL-HOST:absolute;path;to;component.lisp"}.

You only have to specify such logical pathname
for your system or some top-level component.
Sub-components' relative pathnames,
specified using the string syntax for names,
will be properly merged with the pathnames of their parents.
The specification of a logical pathname host however is @emph{not}
otherwise directly supported in the ASDF syntax
for pathname specifiers as strings.

The @code{asdf-output-translation} layer will
avoid trying to resolve and translate logical pathnames.
The advantage of this is that
you can define yourself what translations you want to use
with the logical pathname facility.
The disadvantage is that if you do not define such translations,
any system that uses logical pathnames will behave differently under
asdf-output-translations than other systems you use.

If you wish to use logical pathnames you will have to configure the
translations yourself before they may be used.
ASDF currently provides no specific support
for defining logical pathname translations.

Note that the reasons we do not recommend logical pathnames are that
(1) there is no portable way to set up logical pathnames @emph{before} they are used,
(2) logical pathnames are limited to only portably use
a single character case, digits and hyphens.
While you can solve the first issue on your own,
describing how to do it on each of fifteen implementations supported by ASDF
is more than we can document.
As for the second issue, mind that the limitation is notably enforced on SBCL,
and that you therefore can't portably violate the limitations
but must instead define some encoding of your own and add individual mappings
to name physical pathnames that do not fit the restrictions.
This can notably be a problem when your Lisp files are part of a larger project
in which it is common to name files or directories in a way that
includes the version numbers of supported protocols,
or in which files are shared with software written
in different programming languages where conventions include the use of
underscores, dots or CamelCase in pathnames.


@subsection Serial dependencies
@cindex serial dependencies

If the @code{:serial t} option is specified for a module,
ASDF will add dependencies for each child component,
on all the children textually preceding it.
This is done as if by @code{:depends-on}.

@lisp
:serial t
:components ((:file "a") (:file "b") (:file "c"))
@end lisp

is equivalent to

@lisp
:components ((:file "a")
             (:file "b" :depends-on ("a"))
             (:file "c" :depends-on ("a" "b")))
@end lisp


@subsection Source location (@code{:pathname})

The @code{:pathname} option is optional in all cases for systems
defined via @code{defsystem}, and generally is unnecessary.  In the
simple case, source files will be found in the same directory as the
system or, in the case of modules, in a subdirectory with the same name
as the module.

@c FIXME: This should be moved elsewhere -- it's too much detail for the
@c grammar section.

More specifically, ASDF follows a hairy set of rules that are designed so that
@enumerate
@item
@code{find-system}
will load a system from disk
and have its pathname default to the right place.

@item
This pathname information will not be overwritten with
@code{*default-pathname-defaults*}
(which could be somewhere else altogether)
if the user loads up the @file{.asd} file into his editor
and interactively re-evaluates that form.
@end enumerate

If a system is being loaded for the first time,
its top-level pathname will be set to:

@itemize
@item
The host/device/directory parts of @code{*load-truename*},
if it is bound.
@item
@code{*default-pathname-defaults*}, otherwise.
@end itemize

If a system is being redefined, the top-level pathname will be

@itemize
@item
changed, if explicitly supplied or obtained from @code{*load-truename*}
(so that an updated source location is reflected in the system definition)

@item
changed if it had previously been set from @code{*default-pathname-defaults*}

@item
left as before, if it had previously been set from @code{*load-truename*}
and @code{*load-truename*} is currently unbound
(so that a developer can evaluate a @code{defsystem} form
from within an editor without clobbering its source location)
@end itemize

@subsection if-feature option
@cindex :if-feature component option
@anchor{if-feature option}      @c redo if this ever becomes a node in
@c its own right...

This option allows you to specify a feature expression to be evaluated
as if by @code{#+} to conditionally include a component in your build.
If the expression is false, the component is dropped
as well as any dependency pointing to it.
As compared to using @code{#+} which is expanded at read-time,
this allows you to have an object in your component hierarchy
that can be used for manipulations beside building your project, and
that is accessible to outside code that wishes to reason about system
structure.

Programmers should be careful to consider @strong{when} the
@code{:if-feature} is evaluated.  Recall that ASDF first computes a
build plan, and then executes that plan.  ASDF will check to see whether
or not a feature is present @strong{at planning time}, not during the
build.  It follows that one cannot use @code{:if-feature} to check
features that are set during the course of the build.  It can only be
used to check the state of features before any build operations have
been performed.

This option was added in ASDF 3.  For more information,
@xref{required-features, Required features}.

@subsection Entry point
@cindex :entry-point
@anchor{Entry point}
The @code{:entry-point} option allows a developer to specify the entry point of an executable program created by @code{program-op}.

When @code{program-op} is invoked, the form passed to this option is converted to a function by @code{uiop:ensure-function} and bound to @code{uiop:*image-entry-point*}. Typically one will specify a string, e.g. @code{"foo:main"}, so that the executable starts with the @code{foo:main} function. Note that using the symbol @code{foo:main} instead might not work because the @code{foo} package doesn't necessarily exist when ASDF reads the @code{defsystem} form. For more information on @code{program-op}, @pxref{Predefined operations of ASDF}.

@subsection feature requirement
@anchor{feature requirement}
This requirement was removed in ASDF 3.1.  Please do not use it.  In
most cases, @code{:if-feature} (@pxref{if-feature option}) will provide
an adequate substitute.

The @code{feature} requirement used to ensure that a chain of component
dependencies would fail when a key feature was absent.
Used in conjunction with @code{:if-component-dep-fails}
this provided
a roundabout way to express conditional compilation.


@node Other code in .asd files, The package-inferred-system extension, The defsystem grammar, Defining systems with defsystem
@section Other code in .asd files

Files containing @code{defsystem} forms
are regular Lisp files that are executed by @code{load}.
Consequently, you can put whatever Lisp code you like into these files.
However, it is recommended to keep such forms to a minimal,
and to instead define @code{defsystem} extensions
that you use with @code{:defsystem-depends-on}.

If however, you might insist on including code in the @file{.asd} file itself,
e.g., to examine and adjust the compile-time environment,
possibly adding appropriate features to @code{*features*}.
If so, here are some conventions we recommend you follow,
so that users can control certain details of execution
of the Lisp in @file{.asd} files:

@itemize
@item
Any informative output
(other than warnings and errors,
which are the condition system's to dispose of)
should be sent to the standard CL stream @code{*standard-output*},
so that users can easily control the disposition
of output from ASDF operations.
@end itemize


@node The package-inferred-system extension,  , Other code in .asd files, Defining systems with defsystem
@section The package-inferred-system extension
@codequoteundirected on
@cindex Package inferred systems
@cindex Packages, inferring dependencies from
@findex package-inferred-system
@cindex One package per file systems

Starting with release 3.1.2,
ASDF supports a one-package-per-file style of programming,
in which each file is its own system,
and dependencies are deduced from the @code{defpackage} form
or its variant, @code{uiop:define-package}.

In this style of system definition, package names map to systems with
the same name (in lower case letters),
and if a system is defined with @code{:class package-inferred-system},
then system names that start with that name
(using the slash @code{/} separator)
refer to files under the filesystem hierarchy where the system is defined.
For instance, if system @code{my-lib} is defined in
@file{/foo/bar/my-lib/my-lib.asd}, then system @code{my-lib/src/utility}
will be found in file @file{/foo/bar/my-lib/src/utility.lisp}.

One package per file style was made popular by @code{faslpath} and @code{quick-build},
and at the cost of stricter package discipline,
may yield more maintainable code.
This style is used in ASDF itself (starting with ASDF 3), by @code{lisp-interface-library},
and a few other libraries.

To use this style, choose a toplevel system name, e.g. @code{my-lib},
and create a file @file{my-lib.asd}.
Define @code{my-lib}
using the @code{:class :package-inferred-system} option in its @code{defsystem}.
For instance:
@example
;; This example is based on lil.asd of LISP-INTERFACE-LIBRARY.

#-asdf3.1 (error "MY-LIB requires ASDF 3.1 or later.")
(defsystem "my-lib"
  :class :package-inferred-system
  :depends-on ("my-lib/interface/all"
                        "my-lib/src/all"
                        "my-lib/extras/all")
  :in-order-to ((test-op (load-op "my-lib/test/all")))
  :perform (test-op (o c) (symbol-call :my-lib/test/all :test-suite)))

(defsystem "my-lib/test" :depends-on ("my-lib/test/all"))

(register-system-packages "my-lib/interface/all" '(:my-lib-interface))
(register-system-packages "my-lib/src/all" '(:my-lib-implementation))
(register-system-packages "my-lib/test/all" '(:my-lib-test))

(register-system-packages
 "closer-mop"
 '(:c2mop
   :closer-common-lisp
   :c2cl
   :closer-common-lisp-user
   :c2cl-user))
@end example

@noindent
In the code above, the first form checks that we are using ASDF 3.1 or
later, which provides @code{package-inferred-system}.  This is probably
no longer necessary, since none of the major lisp implementations
provides an older version of ASDF.

@findex register-system-packages
The function @code{register-system-packages} must be called to register
packages used or provided by your system
when the name of the system/file that provides the package
is not the same as the package name (converted to lower case).

Each file under the @code{my-lib} hierarchy will start with a
package definition.
@findex define-package
@findex uiop:define-package
The form @code{uiop:define-package} is supported as well as
@code{defpackage}.
ASDF will compute dependencies from the
@code{:use}, @code{:mix}, and other importation clauses of this package definition. Take the file
@file{interface/order.lisp} as an example:

@example
(uiop:define-package :my-lib/interface/order
  (:use :closer-common-lisp
   :my-lib/interface/definition
   :my-lib/interface/base)
  (:mix :fare-utils :uiop :alexandria)
  (:export ...))
@end example

ASDF can tell that this file/system depends on system @code{closer-mop} (registered above),
@code{my-lib/interface/definition}, and @code{my-lib/interface/base}.

How can ASDF find the file @file{interface/order.lisp} from the
toplevel system @code{my-lib}, however? In the example above,
@file{interface/all.lisp} (and other @file{all.lisp}) reexport
all the symbols exported from the packages at the same or lower levels
of the hierarchy. This can be easily done with
@code{uiop:define-package}, which has many options that prove useful in this
context.  For example:

@example
(uiop:define-package :my-lib/interface/all
  (:nicknames :my-lib-interface)
  (:use :closer-common-lisp)
  (:mix :fare-utils :uiop :alexandria)
  (:use-reexport
   :my-lib/interface/definition
   :my-lib/interface/base
   :my-lib/interface/order
   :my-lib/interface/monad/continuation))
@end example

Thus the top level system need only depend on the @code{my-lib/.../all} systems
because ASDF detects
@file{interface/order.lisp} and all other dependencies from @code{all}
systems' @code{:use-reexport} clauses, which effectively
allow for ``inheritance'' of symbols being exported.

ASDF also detects dependencies from @code{:import-from} clauses.
You may thus import a well-defined set of symbols from an existing
package, and ASDF will know to load the system that provides that
package.  In the following example, ASDF will infer that the current
system depends on @code{foo/baz} from the first @code{:import-from}.
If you prefer to use any such symbol fully qualified by a package prefix,
you may declare a dependency on such a package and its corresponding system
via an @code{:import-from} clause with an empty list of symbols.  For
example, if we preferred to use the name `foo/quux:bletch`, the second,
empty, @code{:import-from} form would cause ASDF to load
@code{foo/quux}.

@example
(defpackage :foo/bar
  (:use :cl)
  (:import-from :foo/baz #:sym1 #:sym2)
  (:import-from :foo/quux)
  (:export ...))
@end example

Note that starting with ASDF 3.1.5.6 only, ASDF will look for source files under
the @code{component-pathname} (specified via the @code{:pathname} option),
whereas earlier versions ignore this option and use the @code{system-source-directory}
where the @file{.asd} file resides.

@c See this blog post about it:
@c @url{http://davazp.net/2014/11/26/modern-library-with-asdf-and-package-inferred-system.html}

@codequoteundirected off

@node The object model of ASDF, Controlling where ASDF searches for systems, Defining systems with defsystem, Top
@comment  node-name,  next,  previous,  up
@chapter The Object model of ASDF
@tindex component
@tindex operation

ASDF is designed in an object-oriented way from the ground up.
Both a system's structure and the operations that can be performed on systems
follow an extensible protocol, allowing programmers to add new behaviours to ASDF.
For example, @code{cffi} adds support for special FFI description files
that interface with C libraries and for wrapper files that embed C code in Lisp.
@code{asdf-jar} supports creating Java JAR archives in ABCL.
@code{poiu} supports compiling code in parallel using background processes.

The key classes in ASDF are @code{component} and @code{operation}.
A @code{component} represents an individual source file or a group of source files,
and the products (e.g., fasl files) produced from it.
An @code{operation} represents a transformation that can be performed on a component,
turning them from source files to intermediate results to final outputs.
Components are related by @emph{dependencies}, specified in system
definitions.

When ordered to @code{operate} with some operation on a component (usually a system),
ASDF will first compute a @emph{plan}
by traversing the dependency graph using function @code{make-plan}.@footnote{
  Historically, the function that built a plan was
  called @code{traverse}, and returned a list of actions;
  it was deprecated in favor of @code{make-plan} (that returns a plan object)
  when the @code{plan} objects were introduced with ASDF 3;
  the old function is kept for backward compatibility and debugging purposes only,
  and may be removed in the near future.
}
The resulting plan object contains an ordered list of @emph{actions}.
An action is a pair of an @code{operation} and a @code{component},
representing  a particular build step to be @code{perform}ed.
The ordering of the plan ensures that no action is performed before
all its dependencies have been fulfilled.@footnote{
  The term @emph{action}
  was used by Kent Pitman in his article, ``The Description of Large Systems,''
  (@pxref{Bibliography}),
  and we suspect might be traced to @code{make}.
  Although the term was only used by ASDF hackers starting with ASDF 2,
  the concept was there since the very beginning of ASDF 1,
  just not clearly articulated.
}

In this chapter, we describe ASDF's object-oriented protocol,
the classes that make it up, and the generic functions on those classes.
These generic functions often take
both an operation and a component as arguments:
much of the power and configurability of ASDF is provided by
this use of CLOS's multiple dispatch.
We will describe the built-in component and operation classes, and
explain how to extend the ASDF protocol by defining new classes and
methods for ASDF's generic functions.
We will also describe the many @emph{hooks} that can be configured to
customize the behaviour of existing @emph{functions}.

@c FIXME: Swap operations and components.
@c FIXME: Possibly add a description of the PLAN object.
@c Not critical, since the user isn't expected to interact with it.
@menu
* Operations::
* Components::
* Dependencies::
* Functions::
* Parsing system definitions::
@end menu

@node  Operations, Components, The object model of ASDF, The object model of ASDF
@comment  node-name,  next,  previous,  up
@section Operations
@cindex operation

An @dfn{operation} object of the appropriate type is instantiated
whenever the user wants to do something with a system like

@itemize
@item compile all its files
@item load the files into a running lisp environment
@item copy its source files somewhere else
@end itemize

Operations can be invoked directly, or examined
to see what their effects would be without performing them.
There are a bunch of methods specialised on operation and component type
that actually do the grunt work.
Operations are invoked on systems via @code{operate} (@pxref{operate}).

ASDF contains a number of pre-defined @t{operation} classes for common,
and even fairly uncommon tasks that you might want to do with it.
In addition, ASDF contains ``abstract'' @t{operation} classes that
programmers can use as building blocks to define ASDF extensions.  We
discuss these in turn below.

@c The operation object contains whatever state is relevant for this purpose
@c (perhaps a list of visited nodes, for example)
@c but primarily is a nice thing to specialise operation methods on
@c and easier than having them all be @code{EQL} methods.

@menu
* Predefined operations of ASDF::
* Creating new operations::
@end menu

Operations are invoked on systems via @code{operate}.
@anchor{operate}
@deffn {Generic function} operate @var{operation} @var{component} @Arest{} @var{initargs} @Akey{} @code{force} @code{force-not} @code{verbose} @AallowOtherKeys
@deffnx {Generic function} oos @var{operation} @var{component} @Arest{} @var{initargs} @Akey{} @AallowOtherKeys{}
@code{operate} invokes @var{operation} on @var{system}.
@code{oos} is a synonym for @code{operate} (it stands for operate-on-system).

@var{operation} is an operation designator:
it can be an operation object itself, or, typically,
a symbol that is passed to @code{make-operation} (which will call @code{make-instance}),
to create the operation object.
@var{component} is a component designator:
it can be a component object itself, or, typically,
a string or symbol (to be @code{string-downcase}d) that names a system,
more rarely a list of strings or symbols that designate a subcomponent of a system.

The ability to pass @var{initargs} to @code{make-operation} is now deprecated, and will be removed.
For more details, @pxref{make-operation}.
Note that dependencies may cause the operation
to invoke other operations on the system or its components:
the new operations may or may not be created
with the same @var{initargs} as the original one (for the moment).

If @var{force} is @code{:all}, then all systems
are forced to be recompiled even if not modified since last compilation.
If @var{force} is @code{t}, then only the system being loaded
is forced to be recompiled even if not modified since last compilation,
but other systems are not affected.
If @var{force} is a list, then it specifies a list of systems that
are forced to be recompiled even if not modified since last compilation.
If @var{force-not} is @code{:all}, then all systems
are forced not to be recompiled even if modified since last compilation.
If @var{force-not} is @code{t}, then all systems but the system being loaded
are forced not to be recompiled even if modified since last compilation
(note: this was changed in ASDF 3.1.2).
If @var{force-not} is a list, then it specifies a list of systems that
are forced not to be recompiled even if modified since last compilation.

@findex register-immutable-system
@cindex immutable systems
Both @var{force} and @var{force-not} apply to systems that are dependencies and were already compiled.
@var{force-not} takes precedences over @var{force},
as it should, really, but unhappily only since ASDF 3.1.2.
Moreover, systems which have been registered as immutable by @code{register-immutable-system} (since ASDF 3.1.5)
are always considered @var{forced-not}, and even their @file{.asd} are not refreshed from the filesystem.
@xref{Miscellaneous Functions}.

@findex traverse
To see what @code{operate} would do, you can use:
@example
(asdf:traverse operation-class system-name)
@end example

@end deffn

@defun make-operation @var{operation-class} @Arest{} @var{initargs}
@anchor{make-operation}

The @var{initargs} are passed to @code{make-instance} call
when creating the operation object.

@strong{Note:}@var{initargs} for @code{operation}s are now deprecated,
and will be removed from ASDF in the near future.

@strong{Note:} @code{operation} instances must @strong{never} be created
using @code{make-instance} directly: only through
@code{make-operation}. Attempts to directly make @code{operation}
instances will cause a run-time error.
@end defun


@node Predefined operations of ASDF, Creating new operations, Operations, Operations
@comment  node-name,  next,  previous,  up
@subsection Predefined operations of ASDF

All the operations described in this section are in the @code{asdf} package.
They are invoked via the @code{operate} generic function.

@lisp
(asdf:operate 'asdf:@var{operation-name} :@var{system-name} @{@var{operation-options ...}@})
@end lisp

@deftp Operation compile-op

This operation compiles the specified component.
A @code{cl-source-file} will be @code{compile-file}'d.
All the children and dependencies of a system or module
will be recursively compiled by @code{compile-op}.

@code{compile-op} depends on @code{prepare-op} which
itself depends on a @code{load-op} of all of a component's dependencies,
as well as of its parent's dependencies.
When @code{operate} is called on @code{compile-op},
all these dependencies will be loaded as well as compiled;
yet, some parts of the system main remain unloaded,
because nothing depends on them.
Use @code{load-op} to load a system.
@end deftp

@deftp Operation load-op

This operation loads the compiled code for a specified component.
A @code{cl-source-file} will have its compiled fasl @code{load}ed,
which fasl is the output of @code{compile-op} that @code{load-op} depends on.

@code{load-op} will recursively load all the children of a system or module.

@code{load-op} also depends on @code{prepare-op} which
itself depends on a @code{load-op} of all of a component's dependencies,
as well as of its parent's dependencies.
@end deftp

@deftp Operation prepare-op

This operation ensures that the dependencies of a component
and its recursive parents are loaded (as per @code{load-op}),
as a prerequisite before @code{compile-op} and @code{load-op} operations
may be performed on a given component.
@end deftp

@deftp Operation load-source-op
@deftpx Operation prepare-source-op

@code{load-source-op} will load the source for the files in a module
rather than the compiled fasl output.
It has a @code{prepare-source-op} analog to @code{prepare-op},
that ensures the dependencies are themselves loaded via @code{load-source-op}.

@end deftp

@anchor{test-op}
@deftp Operation test-op

This operation will perform some tests on the module.
The default method will do nothing.
The default dependency is to require
@code{load-op} to be performed on the module first.
Its default @code{operation-done-p} method returns @code{nil},
which means that the operation is @emph{never} done
--
we assume that if you invoke the @code{test-op},
you want to test the system, even if you have already done so.

The results of this operation are not defined by ASDF.
It has proven difficult to define how the test operation
should signal its results to the user
in a way that is compatible with all of the various test libraries
and test techniques in use in the community, and
given the fact that ASDF operations do not return a value indicating
success or failure.
For those willing to go to the effort, we suggest defining conditions to
signal when a @code{test-op} fails, and storing in those conditions
information that describes which tests fail.

People typically define a separate test @emph{system} to hold the tests.
Doing this avoids unnecessarily adding a test framework as a dependency
on a library.  For example, one might have
@lisp
(defsystem "foo"
  :in-order-to ((test-op (test-op "foo/test")))
 ...)

(defsystem "foo/test"
  :depends-on ("foo" "fiveam") ; fiveam is a test framework library
  ...)
@end lisp

Then one defines @code{perform} methods on
@code{test-op} such as the following:
@lisp
(defsystem "foo/test"
  :depends-on ("foo" "fiveam") ; fiveam is a test framework library
  :perform (test-op (o s)
                    (uiop:symbol-call :fiveam '#:run!
                       (uiop:find-symbol* '#:foo-test-suite
                                            :foo-tests)))
  ...)
@end lisp

@end deftp


@cindex bundle operations
@deftp Operation compile-bundle-op
@deftpx Operation monolithic-compile-bundle-op
@deftpx Operation load-bundle-op
@deftpx Operation monolithic-load-bundle-op
@deftpx Operation deliver-asd-op
@deftpx Operation monolithic-deliver-asd-op
@deftpx Operation lib-op
@deftpx Operation monolithic-lib-op
@deftpx Operation dll-op
@deftpx Operation monolithic-dll-op
@deftpx Operation image-op
@deftpx Operation program-op

These are ``bundle'' operations, that can create a single-file ``bundle''
for all the contents of each system in an application,
or for the entire application.

@code{compile-bundle-op} will create a single fasl file for each of the systems needed,
grouping all its many fasls in one,
so you can deliver each system as a single fasl.
@code{monolithic-compile-bundle-op} will create a single fasl file for the target system
and all its dependencies,
so you can deliver your entire application as a single fasl.
@code{load-bundle-op} will load the output of @code{compile-bundle-op}.
Note that if the output is not up-to-date,
@code{compile-bundle-op} may load the intermediate fasls as a side-effect.
Bundling fasls together matters a lot on ECL,
where the dynamic linking involved in loading tens of individual fasls
can be noticeably more expensive than loading a single one.

NB: @code{compile-bundle-op}, @code{monolithic-compile-bundle-op}, @code{load-bundle-op}, @code{monolithic-load-bundle-op}, @code{deliver-asd-op}, @code{monolithic-deliver-asd-op} were respectively called
@code{fasl-op}, @code{monolithic-fasl-op}, @code{load-fasl-op}, @code{monolithic-load-fasl-op}, @code{binary-op}, @code{monolithic-binary-op} before ASDF 3.1.
The old names still exist for backward compatibility,
though they poorly label what is going on.

Once you have created a fasl with @code{compile-bundle-op},
you can use @code{precompiled-system} to deliver it in a way
that is compatible with clients having dependencies on your system,
whether it is distributed as source or as a single binary;
the @file{.asd} file to be delivered with the fasl will look like this:
@example
(defsystem :mysystem :class :precompiled-system
  :fasl (some expression that will evaluate to a pathname))
@end example
Or you can use @code{deliver-asd-op} to let ASDF create such a system for you
as well as the @code{compile-bundle-op} output,
or @code{monolithic-deliver-asd-op}.
This allows you to deliver code for your systems or applications
as a single file.
Of course, if you want to test the result in the current image,
@emph{before} you try to use any newly created @file{.asd} files,
you should not forget to @code{(asdf:clear-configuration)}
or at least @code{(asdf:clear-source-registry)},
so it re-populates the source-registry from the filesystem.

The @code{program-op} operation will create an executable program
from the specified system and its dependencies.
You can use UIOP for its pre-image-dump hooks, its post-image-restore hooks,
and its access to command-line arguments.
And you can specify an entry point @code{my-app:main}
by specifying in your @code{defsystem}
the option @code{:entry-point "my-app:main"}.
Depending on your implementation,
running @code{(asdf:operate 'asdf:program-op :my-app)}
may quit the current Lisp image upon completion.
See the example in
@file{test/hello-world-example.asd} and @file{test/hello.lisp},
as built and tested by
@file{test/test-program.script} and @file{test/make-hello-world.lisp}.
@code{image-op} will dump an image that may not be standalone
and does not start its own function,
but follows the usual execution convention of the underlying Lisp,
just with more code pre-loaded,
for use as an intermediate build result or with a wrapper invocation script.

There is also @code{lib-op}
for building a linkable @file{.a} file (Windows: @file{.lib})
from all linkable object dependencies (FFI files, and on ECL, Lisp files too),
and its monolithic equivalent @code{monolithic-lib-op}.
And there is also @code{dll-op}
(respectively its monolithic equivalent @code{monolithic-dll-op})
for building a linkable @file{.so} file
(Windows: @file{.dll}, MacOS X: @file{.dynlib})
to create a single dynamic library
for all the extra FFI code to be linked into each of your systems
(respectively your entire application).

All these ``bundle'' operations are available since ASDF 3
on all actively supported Lisp implementations,
but may be unavailable on unmaintained legacy implementations.
This functionality was previously available for select implementations,
as part of a separate system @code{asdf-bundle},
itself descended from the ECL-only @code{asdf-ecl}.

The pathname of the output of bundle operations
is subject to output-translation as usual,
unless the operation is equal to
the @code{:build-operation} argument to @code{defsystem}.
This behaviour is not very satisfactory and may change in the future.
Maybe you have suggestions on how to better configure it?
@end deftp

@deftp Operation concatenate-source-op
@deftpx Operation monolithic-concatenate-source-op
@deftpx Operation load-concatenated-source-op
@deftpx Operation compile-concatenated-source-op
@deftpx Operation load-compiled-concatenated-source-op
@deftpx Operation monolithic-load-concatenated-source-op
@deftpx Operation monolithic-compile-concatenated-source-op
@deftpx Operation monolithic-load-compiled-concatenated-source-op

These operations, as their respective names indicate,
will concatenate all the @code{cl-source-file} source files in a system
(or in a system and all its dependencies, if monolithic),
in the order defined by dependencies,
then load the result, or compile and then load the result.

These operations are useful to deliver a system or application
as a single source file,
and for testing that said file loads properly, or compiles and then loads properly.

ASDF itself is delivered as a single source file this way,
using @code{monolithic-concatenate-source-op},
prepending a prelude and the @code{uiop} library
before the @code{asdf/defsystem} system itself.
@end deftp

See also FAQ entries @pxref{What happened to the bundle operations} and @pxref{How can I produce a binary at a specific path from sources at a specific path}.


@node  Creating new operations,  , Predefined operations of ASDF, Operations
@comment  node-name,  next,  previous,  up
@subsection Creating new operations

ASDF was designed to be extensible in an object-oriented fashion.
To teach ASDF new tricks, a programmer can implement the behaviour he wants
by creating a subclass of @code{operation}.

ASDF's pre-defined operations are in no way ``privileged'',
but it is requested that developers never use the @code{asdf} package
for operations they develop themselves.
The rationale for this rule is that we don't want to establish a
``global asdf operation name registry'',
but also want to avoid name clashes.

Your operation @emph{must} usually provide methods
for one or more of the following generic functions:

@itemize

@findex perform
@item @code{perform}
Unless your operation, like @code{prepare-op},
is for dependency propagation only,
the most important function for which to define a method
is usually @code{perform},
which will be called to perform the operation on a specified component,
after all dependencies have been performed.

The @code{perform} method must call @code{input-files} and @code{output-files} (see below)
to locate its inputs and outputs,
because the user is allowed to override the method
or tweak the output-translation mechanism.
Perform should only use the primary value returned by @code{output-files}.
If one and only one output file is expected,
it can call @code{output-file} that checks that this is the case
and returns the first and only list element.

@findex output-files
@item @code{output-files}
If your perform method has any output,
you must define a method for this function.
for ASDF to determine where the outputs of performing operation lie.

Your method may return two values, a list of pathnames, and a boolean.
If the boolean is @code{nil} (or you fail to return multiple values),
then enclosing @code{:around} methods may translate these pathnames,
e.g. to ensure object files are somehow stored
in some implementation-dependent cache.
If the boolean is @code{t} then the pathnames are marked
not be translated by the enclosing @code{:around} method.

@findex component-depends-on
@item @code{component-depends-on}
If the action of performing the operation on a component has dependencies,
you must define a method on @code{component-depends-on}.

Your method will take as specialized arguments
an operation and a component which together identify an action,
and return a list of entries describing actions that this action depends on.
The format of entries is described below.

It is @emph{strongly} advised that
you should always append the results of @code{(call-next-method)}
to the results of your method,
or ``interesting'' failures will likely occur,
unless you're a true specialist of ASDF internals.
It is unhappily too late to compatibly use the @code{append} method combination,
but conceptually that's the protocol that is being manually implemented.

Each entry returned by @code{component-depends-on} is itself a list.

The first element of an entry is an operation designator:
either an operation object designating itself, or
a symbol that names an operation class
(that ASDF will instantiate using @code{make-operation}).
For instance, @code{load-op}, @code{compile-op} and @code{prepare-op}
are common such names, denoting the respective operations.

@c FIXME COERCE-NAME is referenced, but not defined.
@findex coerce-name
@findex find-component
The rest of each entry is a list of component designators:
either a component object designating itself,
or an identifier to be used with @code{find-component}.
@code{find-component} will be called with the current component's parent as parent,
and the identifier as second argument.
The identifier is typically a string,
a symbol (to be downcased as per @code{coerce-name}),
or a list of strings or symbols.
In particular, the empty list @code{nil} denotes the parent itself.

@end itemize

An operation @emph{may} provide methods for the following generic functions:

@itemize

@item @code{input-files}
@findex input-files
A method for this function is often not needed,
since ASDF has a pretty clever default @code{input-files} mechanism.
You only need create a method if there are multiple ultimate input
files.
Most operations inherit from @code{selfward-operation}, which
appropriately sets the input-files to include the source file itself.

@c FIXME: Add documentation of built-in operation types.

@defun input-files operation component
Return a list of pathnames that represent the input to @var{operation}
performed on @var{component}.
@end defun

@item @code{operation-done-p}
@findex operation-done-p
You only need to define a method on that function
if you can detect conditions that invalidate previous runs of the operation,
even though no filesystem timestamp has changed,
in which case you return @code{nil} (the default is @code{t}).

For instance, the method for @code{test-op} always returns @code{nil},
so that tests are always run afresh.
Of course, the @code{test-op} for your system could depend
on a deterministically repeatable @code{test-report-op},
and just read the results from the report files,
in which case you could have this method return @code{t}.

@end itemize

Operations that print output should send that output to the standard
CL stream @code{*standard-output*}, as the Lisp compiler and loader do.

@node Components, Dependencies, Operations, The object model of ASDF
@comment  node-name,  next,  previous,  up
@section Components
@cindex component
@cindex system
@cindex system designator
@cindex component designator
@vindex *system-definition-search-functions*

A @code{component} represents an individual source file or a group of source files,
and the things that get transformed into.
A @code{system} is a component at the top level of the component hierarchy,
that can be found via @code{find-system}.
A @code{source-file} is a component representing a single source-file
and the successive output files into which it is transformed.
A @code{module} is an intermediate component itself grouping several other components,
themselves source-files or further modules.

A @dfn{system designator} is a system itself,
or a string or symbol that behaves just like any other component name
(including with regard to the case conversion rules for component names).

A @dfn{component designator}, relative to a base component,
is either a component itself,
or a string or symbol,
or a list of designators.

@defun find-system system-designator @Aoptional{} (error-p t)

Given a system designator, @code{find-system} finds and returns a system.
If no system is found, an error of type
@code{missing-component} is thrown,
or @code{nil} is returned if @code{error-p} is false.

To find and update systems, @code{find-system} funcalls each element
in the @code{*system-definition-search-functions*} list,
expecting a pathname to be returned, or a system object,
from which a pathname may be extracted, and that will be registered.
The resulting pathname (if any) is loaded
if one of the following conditions is true:

@itemize
@item
there is no system of that name in memory
@item
the pathname is different from that which was previously loaded
@item
the file's @code{last-modified} time exceeds the @code{last-modified} time
of the system in memory
@end itemize

@cindex ASDF-USER package
When system definitions are loaded from @file{.asd} files,
they are implicitly loaded into the @code{ASDF-USER} package,
which uses @code{ASDF}, @code{UIOP} and @code{UIOP/COMMON-LISP}@footnote{
Note that between releases 2.27 and 3.0.3, only @code{UIOP/PACKAGE},
not all of @code{UIOP}, was used; if you want your code to work
with releases earlier than 3.1.2, you may have to explicitly define a package
that uses @code{UIOP}, or use proper package prefix to your symbols, as in
@code{uiop:version<}.}
Programmers who do anything non-trivial in a @file{.asd} file,
such as defining new variables, functions or classes,
should include @code{defpackage} and @code{in-package} forms in this file,
so they will not overwrite each others' extensions.
Such forms might also help the files behave identically
if loaded manually with @code{cl:load} for development or debugging,
though we recommend you use the function @code{asdf::load-asd} instead,
which the @code{slime-asdf} contrib knows about.

The default value of @code{*system-definition-search-functions*}
is a list of three functions.
The first function looks in each of the directories given
by evaluating members of @code{*central-registry*}
for a file whose name is the name of the system and whose type is @file{asd};
the first such file is returned,
whether or not it turns out to actually define the appropriate system.
The second function does something similar,
for the directories specified in the @code{source-registry},
but searches the filesystem only once and caches its results.
The third function makes the @code{package-inferred-system} extension work,
@pxref{The package-inferred-system extension}.

Because of the way these search functions are defined,
you should put the definition for a system
@var{foo} in a file named @file{foo.asd},
in a directory that is
in the central registry or
which can be found using the
source registry configuration.

@c FIXME: Move this discussion to the system definition grammar, or somewhere else.
@anchor{System names}
@cindex System names
@cindex Primary system name
@findex primary-system-name
It is often useful to define multiple systems in a same file,
but ASDF can only locate a system's definition file based on the system
name.
For this reason,
ASDF 3's system search algorithm has been extended to
allow a file @file{foo.asd} to contain
secondary systems named @var{foo/bar}, @var{foo/baz}, @var{foo/quux}, etc.,
in addition to the primary system named @var{foo}.
The first component of a system name,
separated by the slash character, @code{/},
is called the primary name of a system.
The primary name may be
extracted by function @code{asdf::primary-system-name};
when ASDF 3 is told to find a system whose name has a slash,
it will first attempt to load the corresponding primary system,
and will thus see any such definitions, and/or any
definition of a @code{package-inferred-system}.@footnote{
ASDF 2.26 and earlier versions
do not support this primary system name convention.
With these versions of ASDF
you must explicitly load @file{foo.asd}
before you can use system @var{foo/bar} defined therein,
e.g. using @code{(asdf:find-system "foo")}.
We do not support ASDF 2, and recommend that you should upgrade to ASDF 3.
}
If your file @file{foo.asd} also defines systems
that do not follow this convention, e.g., a system named @var{foo-test},
ASDF will not be able to automatically locate a definition for these systems,
and will only see their definition
if you explicitly find or load the primary system
using e.g. @code{(asdf:find-system "foo")} before you try to use them.
We strongly recommend against this practice,
though it is currently supported for backward compatibility.

@end defun

@defun primary-system-name name

Internal (not exported) function, @code{asdf::primary-system-name}.
Returns the primary system name (the portion before
the slash, @code{/}, in a secondary system name) from @var{name}.

@end defun

@defun locate-system name

This function should typically @emph{not} be invoked directly.  It is
exported as part of the API only for programmers who wish to provide
their own @code{*system-definition-search-functions*}.

Given a system @var{name} designator,
try to locate where to load the system definition from.
@c (This does not include the loading of the system definition,
@c which is done by @code{find-system},
@c or the loading of the system itself, which is done by @code{load-system};
@c however, for systems the definition of which has already been loaded,
@c @code{locate-system} may return an object of class @code{system}.)
Returns five values: @var{foundp}, @var{found-system}, @var{pathname},
@var{previous}, and @var{previous-time}.
@var{foundp} is true when a system was found,
either a new as yet unregistered one, or a previously registered one.
The @var{found-system} return value
will be a @code{system} object, if a system definition is found in your
source registry.
@c This system may be registered (by @code{register-system}) or may not, if
@c it's preloaded code.  Fare writes:
@c In the case of preloaded code, as for "asdf", "uiop", etc.,
@c themselves, the system objects are not registered until after they are
@c initially located by sysdef-preloaded-system-search as a fallback when
@c no source code was found.
The system definition will @emph{not} be
loaded if it hasn't been loaded already.
@var{pathname} when not null is a path from which to load the system,
either associated with @var{found-system}, or with the @var{previous} system.
If @var{previous} is not null, it will be a @emph{previously loaded}
@code{system} object of the same name (note that the system
@emph{definition} is previously-loaded: the system itself may or may not be).
@var{previous-time} when not null is
the timestamp of the previous system definition file, at the
time when the @var{previous} system definition was loaded.

For example, if your current registry has @file{foo.asd} in
@file{/current/path/to/foo.asd},
but system @code{foo} was previously loaded from @file{/previous/path/to/foo.asd}
then @var{locate-system} will return the following values:
@enumerate
@item
@var{foundp} will be @code{t},
@item
@var{found-system} will be @code{nil},
@item
@var{pathname} will be @code{#p"/current/path/to/foo.asd"},
@item
@var{previous} will be an object of type @code{SYSTEM} with
@code{system-source-file} slot value of
@code{#p"/previous/path/to/foo.asd"}
@item
@var{previous-time} will be the timestamp of
@code{#p"/previous/path/to/foo.asd"} at the time it was loaded.
@end enumerate
@end defun

@defun find-component base path

Given a @var{base} component (or designator for such),
and a @var{path}, find the component designated by the @var{path}
starting from the @var{base}.

If @var{path} is a component object, it designates itself,
independently from the base.

@findex coerce-name
If @var{path} is a string, or symbol denoting a string via @code{coerce-name},
then @var{base} is resolved to a component object,
which must be a system or module,
and the designated component is the child named by the @var{path}.

If @var{path} is a @code{cons} cell,
@code{find-component} with the base and the @code{car} of the @var{path},
and the resulting object is used as the base for a tail call
to @code{find-component} with the @code{car} of the @var{path}.

If @var{base} is a component object, it designates itself.

If @var{base} is null, then @var{path} is used as the base, with @code{nil} as the path.

If @var{base} is a string, or symbol denoting a string via @code{coerce-name},
it designates a system as per @code{find-system}.

If @var{base} is a @code{cons} cell, it designates the component found by
@code{find-component} with its @code{car} as base and @code{cdr} as path.
@end defun


@menu
* Common attributes of components::
* Pre-defined subclasses of component::
* Creating new component types::
@end menu

@node  Common attributes of components, Pre-defined subclasses of component, Components, Components
@comment  node-name,  next,  previous,  up
@subsection Common attributes of components

All components, regardless of type, have the following attributes.
All attributes except @code{name} are optional.

@subsubsection Name
@findex coerce-name
A component name is a string or a symbol.
If a symbol, its name is taken and lowercased.  This translation is
performed by the exported function @code{coerce-name}.

Unless overridden by a @code{:pathname} attribute,
the name will be interpreted as a pathname specifier according
to a Unix-style syntax.
@xref{Pathname specifiers}.

@subsubsection Version identifier
@findex version-satisfies
@cindex :version

This optional attribute specifies a version for the current component.
The version should typically be a string of integers separated by dots,
for example @samp{1.0.11}. @xref{Version specifiers}.

A version may then be queried by the generic function @code{version-satisfies},
to see if @code{:version} dependencies are satisfied,
and when specifying dependencies, a constraint of minimal version to satisfy
can be specified using e.g. @code{(:version "mydepname" "1.0.11")}.

Note that in the wild, we typically see version numbering
only on components of type @code{system}.
Presumably it is much less useful within a given system,
wherein the library author is responsible to keep the various files in synch.

@subsubsection Required features
@anchor{required-features}

Traditionally defsystem users have used @code{#+} reader conditionals
to include or exclude specific per-implementation files.
For example, CFFI, the portable C foreign function interface contained
lines like:
@lisp
     #+sbcl       (:file "cffi-sbcl")
@end lisp
An unfortunate side effect of this approach is that no single
implementation can read the entire system.
This causes problems if, for example, one wished to design an @code{archive-op}
that would create an archive file containing all the sources, since
for example the file @code{cffi-sbcl.lisp} above would be invisible when
running the @code{archive-op} on any implementation other than SBCL.

Starting with ASDF 3,
components may therefore have an @code{:if-feature} option.
The value of this option should be
a feature expression using the same syntax as @code{#+} does.
If that feature expression evaluates to false, any reference to the component will be ignored
during compilation, loading and/or linking.
Since the expression is read by the normal reader,
you must explicitly prefix your symbols with @code{:} so they be read as keywords;
this is as contrasted with the @code{#+} syntax
that implicitly reads symbols in the keyword package by default.

For instance, @code{:if-feature (:and :x86 (:or :sbcl :cmu :scl))} specifies that
the given component is only to be compiled and loaded
when the implementation is SBCL, CMUCL or Scieneer CL on an x86 machine.
You cannot write it as @code{:if-feature (and x86 (or sbcl cmu scl))}
since the symbols would not be read as keywords.

@xref{if-feature option}.

@subsubsection Dependencies

This attribute specifies dependencies of the component on its siblings.
It is optional but often necessary.

There is an excitingly complicated relationship between the initarg
and the method that you use to ask about dependencies

Dependencies are between (operation component) pairs.
In your initargs for the component, you can say

@lisp
:in-order-to ((compile-op (load-op "a" "b") (compile-op "c"))
              (load-op (load-op "foo")))
@end lisp

This means the following things:
@itemize
@item
before performing compile-op on this component, we must perform
load-op on @var{a} and @var{b}, and compile-op on @var{c},
@item
before performing @code{load-op}, we have to load @var{foo}
@end itemize

The syntax is approximately

@verbatim
(this-op @{(other-op required-components)@}+)

simple-component-name := string
                      |  symbol

required-components := simple-component-name
                     | (required-components required-components)

component-name := simple-component-name
                | (:version simple-component-name minimum-version-object)
@end verbatim

Side note:

This is on a par with what ACL defsystem does.
mk-defsystem is less general: it has an implied dependency

@verbatim
  for all source file x, (load x) depends on (compile x)
@end verbatim

and using a @code{:depends-on} argument to say that @var{b} depends on
@var{a} @emph{actually} means that

@verbatim
  (compile b) depends on (load a)
@end verbatim

This is insufficient for e.g. the McCLIM system, which requires that
all the files are loaded before any of them can be compiled ]

End side note

In ASDF, the dependency information for a given component and operation
can be queried using @code{(component-depends-on operation component)},
which returns a list

@lisp
((load-op "a") (load-op "b") (compile-op "c") ...)
@end lisp

@code{component-depends-on} can be subclassed for more specific
component/operation types: these need to @code{(call-next-method)}
and append the answer to their dependency, unless
they have a good reason for completely overriding the default dependencies.

If it weren't for CLISP, we'd be using @code{LIST} method
combination to do this transparently.
But, we need to support CLISP.
If you have the time for some CLISP hacking,
I'm sure they'd welcome your fixes.
@c Doesn't CLISP now support LIST method combination?

A minimal version can be specified for a component you depend on
(typically another system), by specifying @code{(:version "other-system" "1.2.3")}
instead of simply @code{"other-system"} as the dependency.
See the discussion of the semantics of @code{:version}
in the defsystem grammar.

@c FIXME: Should have cross-reference to "Version specifiers" in the
@c defsystem grammar, but the cross-referencing is so broken by
@c insufficient node breakdown that I have not put one in.


@subsubsection pathname

This attribute is optional and if absent (which is the usual case),
the component name will be used.

@xref{Pathname specifiers},
for an explanation of how this attribute is interpreted.

Note that the @code{defsystem} macro (used to create a ``top-level'' system)
does additional processing to set the filesystem location of
the top component in that system.
This is detailed elsewhere. @xref{Defining systems with defsystem}.

To find the CL pathname corresponding to a component, use

@defun component-pathname component
Returns the pathname corresponding to @var{component}.  For components
such as source files, this will be a filename pathname.  For example:

@lisp
CL-USER> (asdf:component-pathname (asdf:find-system "xmls"))
#P"/Users/rpg/lisp/xmls/"
@end lisp

and

@lisp
CL-USER> (asdf:component-pathname
           (asdf:find-component
              (asdf:find-system "xmls") "xmls"))
#P"/Users/rpg/lisp/xmls/xmls.lisp"
@end lisp
@end defun




@subsubsection Properties

This attribute is optional.

Packaging systems often require information about files or systems
in addition to that specified by ASDF's pre-defined component attributes.
Programs that create vendor packages out of ASDF systems therefore
have to create ``placeholder'' information to satisfy these systems.
Sometimes the creator of an ASDF system may know the additional
information and wish to provide it directly.

@code{(component-property component property-name)} and
associated @code{setf} method will allow
the programmatic update of this information.
Property names are compared as if by @code{EQL},
so use symbols or keywords or something.

@menu
* Pre-defined subclasses of component::
* Creating new component types::
@end menu

@node Pre-defined subclasses of component, Creating new component types, Common attributes of components, Components
@comment  node-name,  next,  previous,  up
@subsection Pre-defined subclasses of component

@deftp Component source-file

A source file is any file that the system does not know how to
generate from other components of the system.

Note that this is not necessarily the same thing as
``a file containing data that is typically fed to a compiler''.
If a file is generated by some pre-processor stage
(e.g. a @file{.h} file from @file{.h.in} by autoconf)
then it is not, by this definition, a source file.
Conversely, we might have a graphic file
that cannot be automatically regenerated,
or a proprietary shared library that we received as a binary:
these do count as source files for our purposes.

Subclasses of source-file exist for various languages.
@emph{FIXME: describe these.}
@end deftp

@deftp Component module

A module is a collection of sub-components.

A module component has the following extra initargs:

@itemize
@item
@code{:components} the components contained in this module

@item
@code{:default-component-class}
All children components which don't specify their class explicitly
are inferred to be of this type.

@item
@code{:if-component-dep-fails}
This attribute was removed in ASDF 3. Do not use it.
Use @code{:if-feature} instead (@pxref{required-features}, and @pxref{if-feature option}).

@item
@code{:serial} When this attribute is set,
each subcomponent of this component is assumed to depend on all subcomponents
before it in the list given to @code{:components}, i.e.
all of them are loaded before a compile or load operation is performed on it.

@end itemize

The default operation knows how to traverse a module, so
most operations will not need to provide methods specialised on modules.

@code{module} may be subclassed to represent components such as
foreign-language linked libraries or archive files.
@end deftp

@deftp Component system

@code{system} is a subclass of @code{module}.

A system is a module with a few extra attributes for documentation
purposes; these are given elsewhere.
@xref{The defsystem grammar}.

Users can create new classes for their systems:
the default @code{defsystem} macro takes a @code{:class} keyword argument.
@end deftp

@node  Creating new component types,  , Pre-defined subclasses of component, Components
@comment  node-name,  next,  previous,  up
@subsection Creating new component types

New component types are defined by subclassing one of the existing
component classes and specializing methods on the new component class.

@c FIXME: this should perhaps be explained more thoroughly,
@c not only by example ...

As an example, suppose we have some implementation-dependent
functionality that we want to isolate
in one subdirectory per Lisp implementation our system supports.
We create a subclass of
@code{cl-source-file}:

@lisp
(defclass unportable-cl-source-file (cl-source-file)
  ())
@end lisp

Function @code{asdf:implementation-type} (exported since 2.014.14)
gives us the name of the subdirectory.
All that's left is to define how to calculate the pathname
of an @code{unportable-cl-source-file}.

@lisp
(defmethod component-pathname ((component unportable-cl-source-file))
  (merge-pathnames*
   (parse-unix-namestring (format nil "~(~A~)/" (asdf:implementation-type)))
   (call-next-method)))
@end lisp

The new component type is used in a @code{defsystem} form in this way:

@lisp
(defsystem :foo
    :components
    ((:file "packages")
     ...
     (:unportable-cl-source-file "threads"
      :depends-on ("packages" ...))
     ...
    )
@end lisp

@node Dependencies, Functions, Components, The object model of ASDF
@section Dependencies
@c FIXME: Moved this material here, but it isn't very comfortable
@c here....  Also needs to be revised to be coherent.

To be successfully build-able, this graph of actions must be acyclic.
If, as a user, extender or implementer of ASDF, you introduce
a cycle into the dependency graph,
ASDF will fail loudly.
To clearly distinguish the direction of dependencies,
ASDF 3 uses the words @emph{requiring} and @emph{required}
as applied to an action depending on the other:
the requiring action @code{depends-on} the completion of all required actions
before it may itself be @code{perform}ed.

Using the @code{defsystem} syntax, users may easily express
direct dependencies along the graph of the object hierarchy:
between a component and its parent, its children, and its siblings.
By defining custom CLOS methods, you can express more elaborate dependencies as you wish.
Most common operations, such as @code{load-op}, @code{compile-op} or @code{load-source-op}
are automatically propagate ``downward'' the component hierarchy and are ``covariant'' with it:
to act the operation on the parent module, you must first act it on all the children components,
with the action on the parent being parent of the action on each child.
Other operations, such as @code{prepare-op} and @code{prepare-source-op}
(introduced in ASDF 3) are automatically propagated ``upward'' the component hierarchy
and are ``contravariant'' with it:
to perform the operation of preparing for compilation of a child component,
you must perform the operation of preparing for compilation of its parent component, and so on,
ensuring that all the parent's dependencies are (compiled and) loaded
before the child component may be compiled and loaded.
Yet other operations, such as @code{test-op} or @code{load-bundle-op}
remain at the system level, and are not propagated along the hierarchy,
but instead do something global on the system.


@node Functions, Parsing system definitions, Dependencies, The object model of ASDF
@comment  node-name,  next,  previous,  up
@section Functions

@c FIXME: this does not belong here....
@defun version-satisfies @var{version} @var{version-spec}
Does @var{version} satisfy the @var{version-spec}.  A generic function.
ASDF provides built-in methods for @var{version} being a @code{component} or @code{string}.
@var{version-spec} should be a string.
If it's a component, its version is extracted as a string before further processing.

A version string satisfies the version-spec if after parsing,
the former is no older than the latter.
Therefore @code{"1.9.1"}, @code{"1.9.2"} and @code{"1.10"} all satisfy @code{"1.9.1"},
but @code{"1.8.4"} or @code{"1.9"} do not.
For more information about how @code{version-satisfies} parses and interprets
version strings and specifications,
@pxref{Version specifiers} and
@ref{Common attributes of components}.

Note that in versions of ASDF prior to 3.0.1,
including the entire ASDF 1 and ASDF 2 series,
@code{version-satisfies} would also require that the version and the version-spec
have the same major version number (the first integer in the list);
if the major version differed, the version would be considered as not matching the spec.
But that feature was not documented, therefore presumably not relied upon,
whereas it was a nuisance to several users.
Starting with ASDF 3.0.1,
@code{version-satisfies} does not treat the major version number specially,
and returns T simply if the first argument designates a version that isn't older
than the one specified as a second argument.
If needs be, the @code{(:version ...)} syntax for specifying dependencies
could be in the future extended to specify an exclusive upper bound for compatible versions
as well as an inclusive lower bound.
@end defun

@node Parsing system definitions,  , Functions, The object model of ASDF
@section Parsing system definitions
@cindex Parsing system definitions
@cindex Extending ASDF's defsystem parser

Thanks to Eric Timmons, ASDF now provides hooks to extend how it parses
@code{defsystem} forms.

@strong{Warning!} These interfaces are experimental, and as such are not
exported from the ASDF package yet. We plan to export them in ASDF
3.4.0. If you use them before they are exported, please subscribe to
@url{https://gitlab.common-lisp.net/asdf/asdf/-/issues/76} so you are
made aware of any changes.

@defun parse-component-form @var{parent} @var{options} &key @var{previous-serial-components}

This is the core function for parsing a system definition. At the
moment, we do not expect ASDF extenders to modify this function.

When being called on a @code{component} of type @code{system} (i.e.,
inside the @code{defsystem} macro), @var{parent} will be @code{NIL}.
@end defun

@deffn {Generic function} compute-component-children @var{component} @var{components} @var{serial-p}

This generic function provides standard means for computing component
children, but can be extended with additional methods by a programmer.

Returns a list of children (of type @code{component}) for @var{component}.
@var{components} is a list of the explicitly defined children descriptions.
@var{serial-p} is non-@code{NIL} if each child in @var{components} should depend on the previous
children.


@end deffn

@deffn {Generic function} class-for-type @var{parent} @var{type-designator}

Return a @code{class} designator to be used to instantiate a component
whose type is specified by @var{type-designator} in the context of
@var{parent}, which should be either a @code{parent-component} -- or subclass
thereof -- or @code{nil} (if the type designator is some class of system).

This generic function provides a means for changing how ASDF translates
type-designators (like @code{:file}) into CLOS classes.  It is intended
for programmers to extend by adding new methods.

@strong{Warning!} Adding new methods for @code{class-for-type} is
typically @emph{not} necessary: much can already be done by using
@code{:default-component-class} and defining (and explicitly calling
for) new component types.

@end deffn



@node Controlling where ASDF searches for systems, Controlling where ASDF saves compiled files, The object model of ASDF, Top
@comment  node-name,  next,  previous,  up
@chapter Controlling where ASDF searches for systems



@menu
* Configurations::
* Truenames and other dangers::
* XDG base directory::
* Backward Compatibility::
* Configuration DSL::
* Configuration Directories::
* Shell-friendly syntax for configuration::
* Search Algorithm::
* Caching Results::
* Configuration API::
* Introspection::
* Status::
* Rejected ideas::
* TODO::
* Credits for the source-registry::
@end menu

@node  Configurations, Truenames and other dangers, Controlling where ASDF searches for systems, Controlling where ASDF searches for systems
@section Configurations

Configurations specify paths where to find system files.

@enumerate

@item
The search registry may use some hardcoded wrapping registry specification.
This allows some implementations (notably SBCL) to specify where to find
some special implementation-provided systems that
need to precisely match the version of the implementation itself.

@item
An application may explicitly initialize the source-registry configuration
using the configuration API
(@pxref{Controlling where ASDF searches for systems,Configuration API,Configuration API}, below)
in which case this takes precedence.
It may itself compute this configuration from the command-line,
from a script, from its own configuration file, etc.

@item
The source registry will be configured from
the environment variable @code{CL_SOURCE_REGISTRY} if it exists.

@item
The source registry will be configured from
user configuration file
@file{$XDG_CONFIG_DIRS/common-lisp/source-registry.conf}
(which defaults to
@file{~/.config/common-lisp/source-registry.conf})
if it exists.

@item
The source registry will be configured from
user configuration directory
@file{$XDG_CONFIG_DIRS/common-lisp/source-registry.conf.d/}
(which defaults to
@file{~/.config/common-lisp/source-registry.conf.d/})
if it exists.

@item
The source registry will be configured from
default user configuration trees
@file{~/common-lisp/} (since ASDF 3.1.2 only),
@file{~/.sbcl/systems/} (on SBCL only),
@file{$XDG_DATA_HOME/common-lisp/systems/} (no recursion, link farm)
@file{$XDG_DATA_HOME/common-lisp/source/}.
The @code{XDG_DATA_HOME} directory defaults to @file{~/.local/share/}.
On Windows, the @code{local-appdata} and @code{appdata} directories are used instead.

@item
The source registry will be configured from
system configuration file
@file{/etc/common-lisp/source-registry.conf}
if it exists.

@item
The source registry will be configured from
system configuration directory
@file{/etc/common-lisp/source-registry.conf.d/}
if it exists.

@item
The source registry will be configured from a default configuration.
This configuration may allow for implementation-specific systems
to be found, for systems to be found the current directory
(at the time that the configuration is initialized) as well as
@code{:directory} entries for @file{$XDG_DATA_DIRS/common-lisp/systems/} and
@code{:tree} entries for @file{$XDG_DATA_DIRS/common-lisp/source/},
where @code{XDG_DATA_DIRS} defaults to @file{/usr/local/share} and @file{/usr/share} on Unix,
and the @code{common-appdata} directory on Windows.

@item
The source registry may include implementation-dependent directories
that correspond to implementation-provided extensions.

@end enumerate

Each of these configurations is specified as an s-expression
in a trivial domain-specific language (defined below).
Additionally, a more shell-friendly syntax is available
for the environment variable (defined yet below).

Each of these configurations is only used if the previous
configuration explicitly or implicitly specifies that it
includes its inherited configuration.

Additionally, some implementation-specific directories
may be automatically prepended to whatever directories are specified
in configuration files, no matter if the last one inherits or not.

@node Truenames and other dangers, XDG base directory, Configurations, Controlling where ASDF searches for systems
@section Truenames and other dangers

One great innovation of the original ASDF was its ability to leverage
@code{CL:TRUENAME} to locate where your source code was and where to build it,
allowing for symlink farms as a simple but effective configuration mechanism
that is easy to control programmatically.
ASDF 3 still supports this configuration style, and it is enabled by default;
however we recommend you instead use
our source-registry configuration mechanism described below,
because it is easier to setup in a portable way across users and implementations.

Additionally, some people dislike truename,
either because it is very slow on their system, or
because they are using content-addressed storage where the truename of a file
is related to a digest of its individual contents,
and not to other files in the same intended project.
For these people, ASDF 3 allows to eschew the @code{TRUENAME} mechanism,
by setting the variable @var{asdf:*resolve-symlinks*} to @code{nil}.

PS: Yes, if you haven't read Vernor Vinge's short but great classic
``True Names... and Other Dangers'' then you're in for a treat.

@node XDG base directory, Backward Compatibility, Truenames and other dangers, Controlling where ASDF searches for systems
@section XDG base directory

Note that we purport to respect the XDG base directory specification
as to where configuration files are located,
where data files are located,
where output file caches are located.
Mentions of XDG variables refer to that document.

@url{http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html}

This specification allows the user to specify some environment variables
to customize how applications behave to his preferences.

On Windows platforms, even when not using Cygwin, and starting with ASDF 3.1.5,
we still do a best effort at following the XDG base directory specification,
even though it doesn't exactly fit common practice for Windows applications.
However, we replace the fixed Unix paths @file{~/.local}, @file{/usr/local} and @file{/usr}
with their rough Windows equivalent @file{Local AppData}, @file{AppData}, @file{Common AppData}, etc.
Since support for querying the Windows registry
is not possible to do in reasonable amounts of portable Common Lisp code,
ASDF 3 relies on the environment variables that Windows usually exports,
and are hopefully in synch with the Windows registry.
If you care about the details, see @file{uiop/configuration.lisp} and don't hesitate to suggest improvements.

@node Backward Compatibility, Configuration DSL, XDG base directory, Controlling where ASDF searches for systems
@section Backward Compatibility

For backward compatibility as well as to provide a practical backdoor for hackers,
ASDF will first search for @file{.asd} files in the directories specified in
@code{asdf:*central-registry*}
before it searches in the source registry above.

@xref{Configuring ASDF,,Configuring ASDF to find your systems --- old style}.

By default, @code{asdf:*central-registry*} will be empty.

This old mechanism will therefore not affect you if you don't use it,
but will take precedence over the new mechanism if you do use it.

@node Configuration DSL, Configuration Directories, Backward Compatibility, Controlling where ASDF searches for systems
@section Configuration DSL
@cindex :inherit-configuration source config directive
@cindex inherit-configuration source config directive
@cindex :ignore-invalid-entries source config directive
@cindex ignore-invalid-entries source config directive
@cindex :directory source config directive
@cindex directory source config directive
@cindex :tree source config directive
@cindex tree source config directive
@cindex :exclude source config directive
@cindex exclude source config directive
@cindex :also-exclude source config directive
@cindex also-exclude source config directive
@cindex :include source config directive
@cindex include source config directive
@cindex :default-registry source config directive
@cindex default-registry source config directive

Here is the grammar of the s-expression (SEXP) DSL for source-registry
configuration:

@c FIXME: This is too wide for happy compilation into pdf.

@example
;; A configuration is a single SEXP starting with the keyword
;; :source-registry followed by a list of directives.
CONFIGURATION := (:source-registry DIRECTIVE ...)

;; A directive is one of the following:
DIRECTIVE :=
    ;; INHERITANCE DIRECTIVE:
    ;; Your configuration expression MUST contain
    ;; exactly one of the following:
    :inherit-configuration |
    ;; splices inherited configuration (often specified last) or
    :ignore-inherited-configuration |
    ;; drop inherited configuration (specified anywhere)

    ;; forward compatibility directive (since ASDF 2.011.4), useful when
    ;; you want to use new configuration features but have to bootstrap
    ;; the newer required ASDF from an older release that doesn't
    ;; support said features:
    :ignore-invalid-entries |

    ;; add a single directory to be scanned (no recursion)
    (:directory DIRECTORY-PATHNAME-DESIGNATOR) |

    ;; add a directory hierarchy, recursing but
    ;; excluding specified patterns
    (:tree DIRECTORY-PATHNAME-DESIGNATOR) |

    ;; override the defaults for exclusion patterns
    (:exclude EXCLUSION-PATTERN ...) |
    ;; augment the defaults for exclusion patterns
    (:also-exclude EXCLUSION-PATTERN ...) |
    ;; Note that the scope of a an exclude pattern specification is
    ;; the rest of the current configuration expression or file.

    ;; splice the parsed contents of another config file
    (:include REGULAR-FILE-PATHNAME-DESIGNATOR) |

    ;; This directive specifies that some default must be spliced.
    :default-registry

REGULAR-FILE-PATHNAME-DESIGNATOR
    := PATHNAME-DESIGNATOR ; interpreted as a file
DIRECTORY-PATHNAME-DESIGNATOR
    := PATHNAME-DESIGNATOR ; interpreted as a directory

PATHNAME-DESIGNATOR :=
    NIL | ;; Special: skip this entry.
    ABSOLUTE-COMPONENT-DESIGNATOR ;; see pathname DSL

EXCLUSION-PATTERN := a string without wildcards, that will be matched
    exactly against the name of a any subdirectory in the directory
    component of a path. e.g. @code{"_darcs"} will match
    @file{#p"/foo/bar/_darcs/src/bar.asd"}
@end example

Pathnames are designated using another DSL,
shared with the output-translations configuration DSL below.
The DSL is resolved by the function @code{asdf::resolve-location},
to be documented and exported at some point in the future.

@example
ABSOLUTE-COMPONENT-DESIGNATOR :=
    (ABSOLUTE-COMPONENT-DESIGNATOR RELATIVE-COMPONENT-DESIGNATOR ...) |
    STRING |
    ;; namestring (better be absolute or bust, directory assumed where
    ;; applicable).  In output-translations, directory is assumed and
    ;; **/*.*.* added if it's last.  On MCL, a MacOSX-style POSIX
    ;; namestring (for MacOS9 style, use #p"..."); Note that none of the
    ;; above applies to strings used in *central-registry*, which
    ;; doesn't use this DSL: they are processed as normal namestrings.
    ;; however, you can compute what you put in the *central-registry*
    ;; based on the results of say
    ;; (asdf::resolve-location "/Users/fare/cl/cl-foo/")
    PATHNAME |
    ;; pathname (better be an absolute path, or bust)
    ;; In output-translations, unless followed by relative components,
    ;; it better have appropriate wildcards, as in **/*.*.*
    :HOME | ; designates the user-homedir-pathname ~/
    :USER-CACHE | ; designates the default location for the user cache
    :HERE |
    ;; designates the location of the configuration file
    ;; (or *default-pathname-defaults*, if invoked interactively)
    :ROOT
    ;; magic, for output-translations source only: paths that are relative
    ;; to the root of the source host and device

They keyword :SYSTEM-CACHE is not accepted in ASDF 3.1 and beyond: it
was a security hazard.

RELATIVE-COMPONENT-DESIGNATOR :=
    (RELATIVE-COMPONENT-DESIGNATOR RELATIVE-COMPONENT-DESIGNATOR ...) |
    STRING |
      ;; relative directory pathname as interpreted by
      ;; parse-unix-namestring.
      ;; In output translations, if last component, **/*.*.* is added
    PATHNAME | ; pathname; unless last component, directory is assumed.
    :IMPLEMENTATION |
       ;; directory based on implementation, e.g. sbcl-1.0.45-linux-x64
    :IMPLEMENTATION-TYPE |
       ;; a directory based on lisp-implementation-type only, e.g. sbcl
    :DEFAULT-DIRECTORY |
       ;; a relativized version of the default directory
    :*/ | ;; any direct subdirectory (since ASDF 2.011.4)
    :**/ | ;; any recursively inferior subdirectory (since ASDF 2.011.4)
    :*.*.* | ;; any file (since ASDF 2.011.4)

The keywords :UID and :USERNAME are no longer supported.
@end example

For instance, as a simple case, my @file{~/.config/common-lisp/source-registry.conf},
which is the default place ASDF looks for this configuration, once contained:
@example
(:source-registry
  (:tree (:home "cl")) ;; will expand to e.g. "/home/joeluser/cl/"
  :inherit-configuration)
@end example

@node Configuration Directories, Shell-friendly syntax for configuration, Configuration DSL, Controlling where ASDF searches for systems
@section Configuration Directories

Configuration directories consist in files each containing
a list of directives without any enclosing @code{(:source-registry ...)} form.
The files will be sorted by namestring as if by @code{string<} and
the lists of directives of these files with be concatenated in order.
An implicit @code{:inherit-configuration} will be included
at the @emph{end} of the list.

System-wide or per-user Common Lisp software distributions
such as Debian packages or some future version of @code{clbuild}
may then include files such as
@file{/etc/common-lisp/source-registry.conf.d/10-foo.conf} or
@file{~/.config/common-lisp/source-registry.conf.d/10-foo.conf}
to easily and modularly register configuration information
about software being distributed.

The convention is that, for sorting purposes,
the names of files in such a directory begin with two digits
that determine the order in which these entries will be read.
Also, the type of these files must be @file{.conf},
which not only simplifies the implementation by allowing
for more portable techniques in finding those files,
but also makes it trivial to disable a file, by renaming it to a different file type.

Directories may be included by specifying a directory pathname
or namestring in an @code{:include} directive, e.g.:

@example
	(:include "/foo/bar/")
@end example

Hence, to achieve the same effect as
my example @file{~/.config/common-lisp/source-registry.conf} above,
I could simply create a file
@file{~/.config/common-lisp/source-registry.conf.d/33-home-fare-cl.conf}
alone in its directory with the following contents:
@example
(:tree "/home/fare/cl/")
@end example

@menu
* The here directive::
@end menu

@node The here directive,  , Configuration Directories, Configuration Directories
@subsection The :here directive

The @code{:here} directive is an absolute pathname designator that
refers to the directory containing the configuration file currently
being processed.

The @code{:here} directive is intended to simplify the delivery of
complex CL systems, and for easy configuration of projects shared through
revision control systems, in accordance with our design principle that
each participant should be able to provide all and only the information
available to him or her.

Consider a person X who has set up the source code repository for a
complex project with a master directory @file{dir/}.  Ordinarily, one
might simply have the user add a directive that would look something
like this:
@example
   (:tree "path/to/dir")
@end example
But what if X knows that there are very large subtrees
under dir that are filled with, e.g., Java source code, image files for
icons, etc.?  All of the asdf system definitions are contained in the
subdirectories @file{dir/src/lisp/} and @file{dir/extlib/lisp/}, and
these are the only directories that should be searched.

In this case, X can put into @file{dir/} a file @file{asdf.conf} that
contains the following:
@example
(:source-registry
   (:tree (:here "src/lisp/"))
   (:tree (:here "extlib/lisp"))
   (:directory (:here "outlier/")))
@end example

Then when someone else (call her Y) checks out a copy of this
repository, she need only add
@example
(:include "/path/to/my/checkout/directory/asdf.conf")
@end example
to one of her previously-existing asdf source location configuration
files, or invoke @code{initialize-source-registry} with a configuration
form containing that s-expression.  ASDF will find the .conf file that X
has provided, and then set up source locations within the working
directory according to X's (relative) instructions.

@node Shell-friendly syntax for configuration, Search Algorithm, Configuration Directories, Controlling where ASDF searches for systems
@section Shell-friendly syntax for configuration

When considering environment variable @code{CL_SOURCE_REGISTRY}
ASDF will skip to next configuration if it's an empty string.
It will @code{READ} the string as a SEXP in the DSL
if it begins with a paren @code{(},
otherwise it will be interpreted much like @code{TEXINPUTS},
as a list of paths, where

@itemize
@item paths are separated by a @code{:} (colon) on Unix platforms
   (including cygwin), by a @code{;} (semicolon) on other platforms
   (mainly, Windows).

@item each entry is a directory to add to the search path.

@item if the entry ends with a double slash @code{//} then it instead
   indicates a tree in the subdirectories of which to recurse.

@item if the entry is the empty string (which may only appear once),
   then it indicates that the inherited configuration should be spliced
   there.
@end itemize

@node Search Algorithm, Caching Results, Shell-friendly syntax for configuration, Controlling where ASDF searches for systems
@section Search Algorithm
@vindex *default-source-registry-exclusions*

In case that isn't clear, the semantics of the configuration is that
when searching for a system of a given name,
directives are processed in order.

When looking in a directory, if the system is found, the search succeeds,
otherwise it continues.

When looking in a tree, if one system is found, the search succeeds.
If multiple systems are found, the consequences are unspecified:
the search may succeed with any of the found systems,
or an error may be raised.
ASDF 3.2.1 or later returns the pathname whose normalized directory component
has the shortest length (as a list), and breaks ties by choosing the system
with the smallest @code{unix-namestring} when compared with @code{string<}.
Earlier versions of ASDF return ASDF return the first system found,
which is implementation-dependent, and may or may not be the pathname
with the smallest @code{unix-namestring} when compared with @code{string<}.
XCVB raises an error.
If none is found, the search continues.

Exclude statements specify patterns of subdirectories
the systems from which to ignore.
Typically you don't want to use copies of files kept by such
version control systems as Darcs.
Exclude statements are not propagated to further included or inherited
configuration files or expressions;
instead the defaults are reset around every configuration statement
to the default defaults from @code{asdf::*default-source-registry-exclusions*}.

Include statements cause the search to recurse with the path specifications
from the file specified.

An inherit-configuration statement cause the search to recurse with the path
specifications from the next configuration
(@pxref{Configurations} above).


@node Caching Results, Configuration API, Search Algorithm, Controlling where ASDF searches for systems
@section Caching Results

The implementation is allowed to either eagerly compute the information
from the configurations and file system, or to lazily re-compute it
every time, or to cache any part of it as it goes.
In practice, the recommended @code{source-registry} eagerly collects and caches results
and you need to explicitly flush the cache for change to be taken into account,
whereas the old-style @code{*central-registry*} mechanism queries the filesystem every time.

To explicitly flush any information cached by the system
after a change was made in the filesystem, @xref{Configuration API},
and e.g. call @code{asdf:clear-source-registry}.

Starting with ASDF 3.1.4, you can also explicitly build a persistent cache
of the @file{.asd} files found under a tree:
when recursing into a directory declared by @code{:tree} and its transitive subdirectories,
if a file @file{.cl-source-registry.cache} exists containing a form
that is a list starting with @code{:source-registry-cache} followed by a list of strings,
as in @code{(:source-registry-cache @emph{"foo/bar.asd" "path/to/more.asd" ...})},
then the strings are assumed to be @code{unix-namestring}s designating
the available asd files under that tree, and the recursion otherwise stops.
The list can also be empty, allowing to stop a costly recursion in a huge directory tree.

To update such a cache after you install, update or remove source repositories,
you can run a script distributed with ASDF:
@code{tools/cl-source-registry-cache.lisp @emph{/path/to/directory}}.
To wholly invalidate the cache, you can
delete the file @file{.cl-source-registry.cache} in that directory.
In either case, for an existing Lisp process to see this change,
it needs to clear its own cache with e.g. @code{(asdf:clear-source-registry)}.

Developers may safely create a cache in their development tree,
and we recommend they do it at the top of their source tree if
it contains more than a small number of files and directories;
they only need update it when they create, remove or move @file{.asd} files.
Software distribution managers may also safely create such a cache,
but they must be careful to update it every time they install, update or remove
a software source repository or installation package.
Finally, advanced developers who juggle with a lot of code
in their @code{source-registry} may manually manage such a cache,
to allow for faster startup of Lisp programs.

This persistence cache can help you reduce startup latency.
For instance, on one machine with hundreds of source repositories,
such a cache shaves half a second at the startup
of every @code{#!/usr/bin/cl} script using SBCL, more on other implementations;
this makes a notable difference as to
their subjective interactivity and usability.
The speedup will only happen if the implementation-provided ASDF is recent enough
(3.1.3.7 or later); it is not enough for a recent ASDF upgrade to be present,
since the upgrade will itself be found but
after the old version has scanned the directories without heeding such a cache.
To upgrade the implementation-provided ASDF,
@pxref{Replacing your implementation's ASDF}.


@node Configuration API, Introspection, Caching Results, Controlling where ASDF searches for systems
@section Configuration API

The specified functions are exported from your build system's package.
Thus for ASDF the corresponding functions are in package ASDF,
and for XCVB the corresponding functions are in package XCVB.

@defun initialize-source-registry @Aoptional{} PARAMETER
   will read the configuration and initialize all internal variables.
   You may extend or override configuration
   from the environment and configuration files
   with the given @var{PARAMETER}, which can be
   @code{nil} (no configuration override),
   or a SEXP (in the SEXP DSL),
   a string (as in the string DSL),
   a pathname (of a file or directory with configuration),
   or a symbol (fbound to function that when called returns one of the above).
@end defun

@defun clear-source-registry
   undoes any source registry configuration
   and clears any cache for the search algorithm.
   You might want to call this function
   (or better, @code{clear-configuration})
   before you dump an image that would be resumed
   with a different configuration,
   and return an empty configuration.
   Note that this does not include clearing information about
   systems defined in the current image, only about
   where to look for systems not yet defined.
@end defun

@defun ensure-source-registry @Aoptional{} PARAMETER
   checks whether a source registry has been initialized.
   If not, initialize it with the given @var{PARAMETER}.
@end defun

Every time you use ASDF's @code{find-system}, or
anything that uses it (such as @code{operate}, @code{load-system}, etc.),
@code{ensure-source-registry} is called with parameter @code{nil},
which the first time around causes your configuration to be read.
If you change a configuration file,
you need to explicitly @code{initialize-source-registry} again,
or maybe simply to @code{clear-source-registry} (or @code{clear-configuration})
which will cause the initialization to happen next time around.

@node Introspection, Status, Configuration API, Controlling where ASDF searches for systems
@section Introspection

@menu
* *source-registry-parameter* variable::
* Information about system dependencies::
@end menu

@node *source-registry-parameter* variable, Information about system dependencies, Introspection, Introspection
@subsection *source-registry-parameter* variable
@vindex *source-registry-parameter*

We have made available the variable @code{*source-registry-parameter*}
that can be used by code that wishes to introspect about the (past)
configuration of ASDF's source registry.  @strong{This variable should
never be set!}  It will be set as a side-effect of calling
@code{initialize-source-registry}; user code should treat it as
read-only.

@node Information about system dependencies,  , *source-registry-parameter* variable, Introspection
@subsection Information about system dependencies

ASDF makes available three functions to read system interdependencies.
These are intended to aid programmers who wish to perform dependency
analyses.

@defun system-defsystem-depends-on system
@end defun

@defun system-depends-on system
@end defun

@defun system-weakly-depends-on system
Returns a list of names of systems that are weakly depended on by
@var{system}.  Weakly depended on systems are optionally loaded only if
ASDF can find them; failure to find such systems does @emph{not} cause an
error in loading.

Note that the return value for @code{system-weakly-depends-on} is simpler
than the return values of the other two system dependency introspection
functions.
@end defun

@node Status, Rejected ideas, Introspection, Controlling where ASDF searches for systems
@section Status

This mechanism is vastly successful, and we have declared
that @code{asdf:*central-registry*} is not recommended anymore,
though we will continue to support it.
All hooks into implementation-specific search mechanisms
have been integrated in the @code{wrapping-source-registry}
that everyone uses implicitly.

@node Rejected ideas, TODO, Status, Controlling where ASDF searches for systems
@section Rejected ideas

Alternatives I (FRR) considered and rejected while developing ASDF 2 included:

@enumerate
@item Keep @code{asdf:*central-registry*} as the master with its current semantics,
   and somehow the configuration parser expands the new configuration
   language into a expanded series of directories of subdirectories to
   lookup, pre-recursing through specified hierarchies. This is kludgy,
   and leaves little space of future cleanups and extensions.

@item Keep @code{asdf:*central-registry*} as the master but extend its semantics
   in completely new ways, so that new kinds of entries may be implemented
   as a recursive search, etc. This seems somewhat backwards.

@item Completely remove @code{asdf:*central-registry*}
   and break backwards compatibility.
   Hopefully this will happen in a few years after everyone migrate to
   a better ASDF and/or to XCVB, but it would be very bad to do it now.

@item Replace @code{asdf:*central-registry*} by a symbol-macro with appropriate magic
   when you dereference it or setf it. Only the new variable with new
   semantics is handled by the new search procedure.
   Complex and still introduces subtle semantic issues.
@end enumerate


I've been suggested the below features, but have rejected them,
for the sake of keeping ASDF no more complex than strictly necessary.

@itemize
@item
  More syntactic sugar: synonyms for the configuration directives, such as
  @code{(:add-directory X)} for @code{(:directory X)}, or @code{(:add-directory-hierarchy X)}
  or @code{(:add-directory X :recurse t)} for @code{(:tree X)}.

@item
  The possibility to register individual files instead of directories.

@item
  Integrate Xach Beane's tilde expander into the parser,
  or something similar that is shell-friendly or shell-compatible.
  I'd rather keep ASDF minimal. But maybe this precisely keeps it
  minimal by removing the need for evaluated entries that ASDF has?
  i.e. uses of @code{USER-HOMEDIR-PATHNAME} and @code{$SBCL_HOME}
  Hopefully, these are already superseded by the @code{:default-registry}

@item
  Using the shell-unfriendly syntax @code{/**} instead of TEXINPUTS-like @code{//} to specify recursion
  down a filesystem tree in the environment variable.
  It isn't that Lisp friendly either.
@end itemize

@node TODO, Credits for the source-registry, Rejected ideas, Controlling where ASDF searches for systems
@section TODO

@itemize
@item Add examples
@end itemize

@node Credits for the source-registry,  , TODO, Controlling where ASDF searches for systems
@section Credits for the source-registry

Thanks a lot to Stelian Ionescu for the initial idea.

Thanks to Rommel Martinez for the initial implementation attempt.

All bad design ideas and implementation bugs are mine, not theirs.
But so are good design ideas and elegant implementation tricks.

 --- Francois-Rene Rideau @email{fare@@tunes.org}, Mon, 22 Feb 2010 00:07:33 -0500



@node Controlling where ASDF saves compiled files, Error handling, Controlling where ASDF searches for systems, Top
@comment  node-name,  next,  previous,  up
@chapter Controlling where ASDF saves compiled files
@cindex asdf-output-translations
@vindex ASDF_OUTPUT_TRANSLATIONS

Each Common Lisp implementation has its own format
for compiled files or fasls.@footnote{``FASL'' is short for ``FASt Loading.''}
If you use multiple implementations
(or multiple versions of the same implementation),
you'll soon find your source directories
littered with various @file{fasl}s, @file{dfsl}s, @file{cfsl}s and so
on.
Worse yet, multiple implementations use the same file extension and
some implementations maintain the same file extension
while changing formats from version to version (or platform to
platform).
This can lead to many errors and much confusion
as you switch from one implementation to the next.
Finally, this requires write access to the source directory,
and therefore precludes sharing of a same source code directory between multiple users.

Since ASDF 2, ASDF includes the @code{asdf-output-translations} facility
to mitigate the problem.

@menu
* Output Configurations::
* Output Backward Compatibility::
* Output Configuration DSL::
* Output Configuration Directories::
* Output Shell-friendly syntax for configuration::
* Semantics of Output Translations::
* Output Caching Results::
* Output location API::
* Credits for output translations::
@end menu

@node Output Configurations, Output Backward Compatibility, Controlling where ASDF saves compiled files, Controlling where ASDF saves compiled files
@section Configurations

@c FIXME: Explain how configurations work: can't expect reader will have
@c looked at previous chapter. Probably cut and paste will do.


Configurations specify mappings from input locations to output locations.
Once again we rely on the XDG base directory specification for configuration.
@xref{Controlling where ASDF searches for systems,,XDG base directory}.

@enumerate

@item
Some hardcoded wrapping output translations configuration may be used.
This allows special output translations (or usually, invariant directories)
to be specified corresponding to the similar special entries in the source registry.

@item
An application may explicitly initialize the output-translations
configuration using the Configuration API
in which case this takes precedence.
(@pxref{Controlling where ASDF saves compiled files,,Configuration API}.)
It may itself compute this configuration from the command-line,
from a script, from its own configuration file, etc.

@item
The source registry will be configured from
the environment variable @code{ASDF_OUTPUT_TRANSLATIONS} if it exists.

@item
The source registry will be configured from
user configuration file
@file{$XDG_CONFIG_DIRS/common-lisp/asdf-output-translations.conf}
(which defaults to
@file{~/.config/common-lisp/asdf-output-translations.conf})
if it exists.

@item
The source registry will be configured from
user configuration directory
@file{$XDG_CONFIG_DIRS/common-lisp/asdf-output-translations.conf.d/}
(which defaults to
@file{~/.config/common-lisp/asdf-output-translations.conf.d/})
if it exists.

@item
The source registry will be configured from
system configuration file
@file{/etc/common-lisp/asdf-output-translations.conf}
if it exists.

@item
The source registry will be configured from
system configuration directory
@file{/etc/common-lisp/asdf-output-translations.conf.d/}
if it exists.

@end enumerate

Each of these configurations is specified as a SEXP
in a trivial domain-specific language (@pxref{Configuration DSL}).
Additionally, a more shell-friendly syntax is available
for the environment variable (@pxref{Shell-friendly syntax for configuration}).

When processing an entry in the above list of configuration methods,
ASDF will stop unless that entry
explicitly or implicitly specifies that it
includes its inherited configuration.

Note that by default, a per-user cache is used for output files.
This allows the seamless use of shared installations of software
between several users, and takes files out of the way of the developers
when they browse source code,
at the expense of taking a small toll when developers have to clean up
output files and find they need to get familiar with output-translations
first.@footnote{A @code{CLEAN-OP} would be a partial solution to this problem.}


@node Output Backward Compatibility, Output Configuration DSL, Output Configurations, Controlling where ASDF saves compiled files
@section Backward Compatibility
@cindex ASDF-BINARY-LOCATIONS compatibility
@c FIXME: Demote this section -- the typical reader doesn't care about
@c backwards compatibility.


We purposely do @emph{not} provide backward compatibility with earlier versions of
@code{ASDF-Binary-Locations} (8 Sept 2009),
@code{common-lisp-controller} (7.0) or
@code{cl-launch} (2.35),
each of which had similar general capabilities.
The APIs of these programs were not designed
for easy user configuration
through configuration files.
Recent versions of @code{common-lisp-controller} (7.2) and @code{cl-launch} (3.000)
use the new @code{asdf-output-translations} API as defined below.
@code{ASDF-Binary-Locations} is fully superseded and not to be used anymore.

This incompatibility shouldn't inconvenience many people.
Indeed, few people use and customize these packages;
these few people are experts who can trivially adapt to the new configuration.
Most people are not experts, could not properly configure these features
(except inasmuch as the default configuration of
@code{common-lisp-controller} and/or @code{cl-launch}
might have been doing the right thing for some users),
and yet will experience software that ``just works'',
as configured by the system distributor, or by default.

Nevertheless, if you are a fan of @code{ASDF-Binary-Locations},
we provide a limited emulation mode:

@defun enable-asdf-binary-locations-compatibility @Akey{} centralize-lisp-binaries default-toplevel-directory include-per-user-information map-all-source-files source-to-target-mappings
This function will initialize the new @code{asdf-output-translations} facility in a way
that emulates the behaviour of the old @code{ASDF-Binary-Locations} facility.
Where you would previously set global variables
@var{*centralize-lisp-binaries*},
@var{*default-toplevel-directory*},
@var{*include-per-user-information*},
@var{*map-all-source-files*} or @var{*source-to-target-mappings*}
you will now have to pass the same values as keyword arguments to this function.
Note however that as an extension the @code{:source-to-target-mappings} keyword argument
will accept any valid pathname designator for @code{asdf-output-translations}
instead of just strings and pathnames.
@end defun

If you insist, you can also keep using the old @code{ASDF-Binary-Locations}
(the one available as an extension to load of top of ASDF,
not the one built into a few old versions of ASDF),
but first you must disable @code{asdf-output-translations}
with @code{(asdf:disable-output-translations)},
or you might experience ``interesting'' issues.

Also, note that output translation is enabled by default.
To disable it, use @code{(asdf:disable-output-translations)}.

@node Output Configuration DSL, Output Configuration Directories, Output Backward Compatibility, Controlling where ASDF saves compiled files
@section Configuration DSL

Here is the grammar of the SEXP DSL
for @code{asdf-output-translations} configuration:

@verbatim
;; A configuration is single SEXP starting with keyword :output-translations
;; followed by a list of directives.
CONFIGURATION := (:output-translations DIRECTIVE ...)

;; A directive is one of the following:
DIRECTIVE :=
    ;; INHERITANCE DIRECTIVE:
    ;; Your configuration expression MUST contain
    ;; exactly one of either of these:
    :inherit-configuration |
      ;; splices inherited configuration (often specified last)
    :ignore-inherited-configuration |
      ;; drop inherited configuration (specified anywhere)

    ;; forward compatibility directive (since ASDF 2.011.4), useful when
    ;; you want to use new configuration features but have to bootstrap a
    ;; the newer required ASDF from an older release that doesn't have
    ;; said features:
    :ignore-invalid-entries |

    ;; include a configuration file or directory
    (:include PATHNAME-DESIGNATOR) |

    ;; enable global cache in ~/.common-lisp/cache/sbcl-1.0.45-linux-amd64/
    ;; or something.
    :enable-user-cache |
    ;; Disable global cache. Map / to /
    :disable-cache |

    ;; add a single directory to be scanned (no recursion)
    (DIRECTORY-DESIGNATOR DIRECTORY-DESIGNATOR)

    ;; use a function to return the translation of a directory designator
    (DIRECTORY-DESIGNATOR (:function TRANSLATION-FUNCTION))

DIRECTORY-DESIGNATOR :=
    NIL | ; As source: skip this entry. As destination: same as source
    T | ; as source matches anything, as destination
        ; maps pathname to itself.
    ABSOLUTE-COMPONENT-DESIGNATOR ; same as in the source-registry language

TRANSLATION-FUNCTION :=
    SYMBOL | ;; symbol naming a function that takes two arguments:
             ;; the pathname to be translated and the matching
             ;; DIRECTORY-DESIGNATOR
    LAMBDA   ;; A form which evaluates to a function taking two arguments:
             ;; the pathname to be translated and the matching
             ;; DIRECTORY-DESIGNATOR

@end verbatim

Relative components better be either relative
or subdirectories of the path before them, or bust.

@c FIXME: the following assumes that the reader is familiar with the use
@c of this pattern in logical pathnames, which may not be a reasonable
@c assumption.  Expand.
The last component, if not a pathname, is notionally completed by @file{/**/*.*}.
You can specify more fine-grained patterns
by using a pathname object as the last component
e.g. @file{#p"some/path/**/foo*/bar-*.fasl"}

You may use @code{#+features} to customize the configuration file.

The second designator of a mapping may be @code{nil}, indicating that files are not mapped
to anything but themselves (same as if the second designator was the same as the first).

When the first designator is @code{t},
the mapping always matches.
When the first designator starts with @code{:root},
the mapping matches any host and device.
In either of these cases, if the second designator
isn't @code{t} and doesn't start with @code{:root},
then strings indicating the host and pathname are somehow copied
in the beginning of the directory component of the source pathname
before it is translated.

When the second designator is @code{t}, the mapping is the identity.
When the second designator starts with @code{:root},
the mapping preserves the host and device of the original pathname.
Notably, this allows you to map files
to a subdirectory of the whichever directory the file is in.
Though the syntax is not quite as easy to use as we'd like,
you can have an (source destination) mapping entry such as follows
in your configuration file,
or you may use @code{enable-asdf-binary-locations-compatibility}
with @code{:centralize-lisp-binaries nil}
which will do the same thing internally for you:
@lisp
#.(let ((wild-subdir
          (make-pathname :directory '(:relative :wild-inferiors)))
        (wild-file
          (make-pathname :name :wild :version :wild :type :wild)))
   `((:root ,wild-subdir ,wild-file)
     (:root ,wild-subdir :implementation ,wild-file)))
@end lisp
Starting with ASDF 2.011.4, you can use the simpler:
	@code{`(:root (:root :**/ :implementation :*.*.*))}



@code{:include} statements cause the search to recurse with the path specifications
from the file specified.

If the @code{translate-pathname} mechanism cannot achieve a desired
translation, the user may provide a function which provides the
required algorithm.  Such a translation function is specified by
supplying a list as the second @code{directory-designator}
the first element of which is the keyword @code{:function},
and the second element of which is
either a symbol which designates a function or a lambda expression.
The function designated by the second argument must take two arguments,
the first being the pathname of the source file,
the second being the wildcard that was matched.
When invoked, the function should return the translated pathname.

An @code{:inherit-configuration} statement causes the search to recurse with the path
specifications from the next configuration in the bulleted list.
@xref{Controlling where ASDF saves compiled files,,Configurations}, above.

@vindex asdf::*user-cache*
@itemize
@item
@code{:enable-user-cache} is the same as @code{(t :user-cache)}.
@item
@code{:disable-cache} is the same as @code{(t t)}.
@item
@code{:user-cache} uses the contents of variable @code{asdf::*user-cache*}
which by default is the same as using
@code{(:home ".cache" "common-lisp" :implementation)}.
@end itemize


@node Output Configuration Directories, Output Shell-friendly syntax for configuration, Output Configuration DSL, Controlling where ASDF saves compiled files
@section Configuration Directories

Configuration directories consist of files, each of which contains
a list of directives without any enclosing
@code{(:output-translations ...)} form.
The files will be sorted by namestring as if by @code{string<} and
the lists of directives of these files with be concatenated in order.
An implicit @code{:inherit-configuration} will be included
at the @emph{end} of the list.

System-wide or per-user Common Lisp software distributions
such as Debian packages or some future version of @code{clbuild}
may then include files such as
@file{/etc/common-lisp/asdf-output-translations.conf.d/10-foo.conf} or
@file{~/.config/common-lisp/asdf-output-translations.conf.d/10-foo.conf}
to easily and modularly register configuration information
about software being distributed.

The convention is that, for sorting purposes,
the names of files in such a directory begin with two digits
that determine the order in which these entries will be read.
Also, the type of these files must be @file{.conf},
which not only simplifies the implementation by allowing
for more portable techniques in finding those files,
but also makes it trivial to disable a file, by renaming it to a different file type.

Directories may be included by specifying a directory pathname
or namestring in an @code{:include} directive, e.g.:

@verbatim
	(:include "/foo/bar/")
@end verbatim

@node Output Shell-friendly syntax for configuration, Semantics of Output Translations, Output Configuration Directories, Controlling where ASDF saves compiled files
@section Shell-friendly syntax for configuration

When processing the environment variable
@code{ASDF_OUTPUT_TRANSLATIONS}:
@itemize
@item ASDF will skip to the next configuration if it's an empty string.
@item ASDF will @code{READ} the string as an SEXP in the DSL, if it
begins with a parenthesis @code{(}.
@item Otherwise ASDF will interpret the value as a list of directories
(see below).
@end itemize

In the directory list format,
directories should come in pairs, each pair indicating a mapping directive.
Entries are separated
by a @code{:} (colon) on Unix platforms (including Mac and cygwin), and
by a @code{;} (semicolon) on other platforms (mainly, Windows).

The magic empty entry,
if it comes in what would otherwise be the first entry in a pair,
indicates the splicing of inherited configuration;
the next entry (if any) then starts a new pair.
If the second entry in a pair is empty,
it indicates that the directory in the first entry is to be left untranslated
(which has the same effect as if the directory had been repeated).

For example, @code{"/foo:/bar::/baz:"} means:
specify that outputs for things under directory @file{/foo/}
are translated to be under @file{/bar/};
then include the inherited configuration;
then specify that outputs for things under directory @file{/baz/} are not translated.


@node Semantics of Output Translations, Output Caching Results, Output Shell-friendly syntax for configuration, Controlling where ASDF saves compiled files
@section Semantics of Output Translations

From the specified configuration,
a list of mappings is extracted in a straightforward way:
mappings are collected in order, recursing through
included or inherited configuration as specified.
To this list is prepended some implementation-specific mappings,
and is appended a global default.

The list is then compiled to a mapping table as follows:
for each entry, in order, resolve the first designated directory
into an actual directory pathname for source locations.
If no mapping was specified yet for that location,
resolve the second designated directory to an output location directory
add a mapping to the table mapping the source location to the output location,
and add another mapping from the output location to itself
(unless a mapping already exists for the output location).

Based on the table, a mapping function is defined,
mapping source pathnames to output pathnames:
given a source pathname, locate the longest matching prefix
in the source column of the mapping table.
Replace that prefix by the corresponding output column
in the same row of the table, and return the result.
If no match is found, return the source pathname.
(A global default mapping the filesystem root to itself
may ensure that there will always be a match,
with same fall-through semantics).

@node Output Caching Results, Output location API, Semantics of Output Translations, Controlling where ASDF saves compiled files
@section Caching Results

The implementation is allowed to either eagerly compute the information
from the configurations and file system, or to lazily re-compute it
every time, or to cache any part of it as it goes.
To explicitly flush any information cached by the system, use the API below.


@node Output location API, Credits for output translations, Output Caching Results, Controlling where ASDF saves compiled files
@section Output location API

The specified functions are exported from package ASDF.

@defun initialize-output-translations @Aoptional{} PARAMETER
   will read the configuration and initialize all internal variables.
   You may extend or override configuration
   from the environment and configuration files
   with the given @var{PARAMETER}, which can be
   @code{nil} (no configuration override),
   or a SEXP (in the SEXP DSL),
   a string (as in the string DSL),
   a pathname (of a file or directory with configuration),
   or a symbol (fbound to function that when called returns one of the above).
@end defun

@defun disable-output-translations
   will initialize output translations in a way
   that maps every pathname to itself,
   effectively disabling the output translation facility.
@end defun

@defun clear-output-translations
   undoes any output translation configuration
   and clears any cache for the mapping algorithm.
   You might want to call this function
   (or better, @code{clear-configuration})
   before you dump an image that would be resumed
   with a different configuration,
   and return an empty configuration.
   Note that this does not include clearing information about
   systems defined in the current image, only about
   where to look for systems not yet defined.
@end defun

@defun ensure-output-translations @Aoptional{} PARAMETER
   checks whether output translations have been initialized.
   If not, initialize them with the given @var{PARAMETER}.
   This function will be called before any attempt to operate on a system.
@end defun

@defun apply-output-translations PATHNAME
   Applies the configured output location translations to @var{PATHNAME}
   (calls @code{ensure-output-translations} for the translations).
@end defun

Every time you use ASDF's @code{output-files}, or
anything that uses it (that may compile, such as @code{operate}, @code{perform}, etc.),
@code{ensure-output-translations} is called with parameter @code{nil},
which the first time around causes your configuration to be read.
If you change a configuration file,
you need to explicitly @code{initialize-output-translations} again,
or maybe @code{clear-output-translations} (or @code{clear-configuration}),
which will cause the initialization to happen next time around.


@node Credits for output translations,  , Output location API, Controlling where ASDF saves compiled files
@section Credits for output translations

Thanks a lot to Peter van Eynde for @code{Common Lisp Controller}
and to Bjorn Lindberg and Gary King for @code{ASDF-Binary-Locations}.

All bad design ideas and implementation bugs are to mine, not theirs.
But so are good design ideas and elegant implementation tricks.

 --- Francois-Rene Rideau @email{fare@@tunes.org}

@c @section Default locations
@c @findex output-files-for-system-and-operation

@c The default binary location for each Lisp implementation
@c is a subdirectory of each source directory.
@c To account for different Lisps, Operating Systems, Implementation versions,
@c and so on, ASDF borrows code from SLIME
@c to create reasonable custom directory names.
@c Here are some examples:

@c @itemize
@c @item
@c SBCL, version 1.0.45 on Mac OS X for Intel: @code{sbcl-1.0.45-darwin-x86}

@c @item
@c Franz Allegro, version 8.0, ANSI Common Lisp: @code{allegro-8.0a-macosx-x86}

@c @item
@c Franz Allegro, version 8.1, Modern (case sensitive) Common Lisp: @code{allegro-8.1m-macosx-x86}
@c @end itemize

@c By default, all output file pathnames will be relocated
@c to some thus-named subdirectory of @file{~/.cache/common-lisp/}.

@c See the document @file{README.asdf-output-translations}
@c for a full specification on how to configure @code{asdf-output-translations}.

@node  Error handling, Miscellaneous additional functionality, Controlling where ASDF saves compiled files, Top
@comment  node-name,  next,  previous,  up
@chapter Error handling
@tindex system-definition-error
@tindex operation-error

@section ASDF errors

If ASDF detects an incorrect system definition, it will signal a generalised instance of
@code{system-definition-error}.

Operations may go wrong (for example when source files contain errors).
These are signalled using generalised instances of
@code{operation-error}.

@section Compilation error and warning handling
@vindex *compile-file-warnings-behaviour*
@vindex *compile-file-failure-behaviour*

ASDF checks for warnings and errors when a file is compiled.
The variables @var{*compile-file-warnings-behaviour*} and
@var{*compile-file-failure-behaviour*}
control the handling of any such events.
The valid values for these variables are
@code{:error}, @code{:warn}, and @code{:ignore}.

@node  Miscellaneous additional functionality, Getting the latest version, Error handling, Top
@comment  node-name,  next,  previous,  up
@chapter Miscellaneous additional functionality

ASDF includes several additional features that are generally
useful for system definition and development.

@menu
* Controlling file compilation::
* Controlling source file character encoding::
* Miscellaneous Functions::
* Some Utility Functions::
@end menu

@node Controlling file compilation, Controlling source file character encoding, Miscellaneous additional functionality, Miscellaneous additional functionality
@section Controlling file compilation
@cindex :around-compile
@cindex around-compile keyword
@cindex compile-check keyword
@cindex :compile-check
@findex compile-file*

@c FIXME: Needs rewrite.  Start with motivation -- why are we doing
@c this?  (there is some, but it's buried).  Also, all of a sudden in
@c the middle of the discussion we start talking about a "hook," which
@c is confusing.

When declaring a component (system, module, file),
you can specify a keyword argument @code{:around-compile function}.
If left unspecified (and therefore unbound),
the value will be inherited from the parent component if any,
or with a default of @code{nil}
if no value is specified in any transitive parent.

The argument must be either @code{nil}, an fbound symbol,
a lambda-expression (e.g. @code{(lambda (thunk) ...(funcall thunk ...) ...)})
a function object (e.g. using @code{#.#'} but that's discouraged
because it prevents the introspection done by e.g. asdf-dependency-grovel),
or a string that when @code{read} yields a symbol or a lambda-expression.
@code{nil} means the normal compile-file function will be called.
A non-nil value designates a function of one argument
that will be called with a function that will
invoke @code{compile-file*} with various arguments;
the around-compile hook may supply additional keyword arguments
to pass to that call to @code{compile-file*}.

One notable argument that is heeded by @code{compile-file*} is
@code{:compile-check},
a function called when the compilation was otherwise a success,
with the same arguments as @code{compile-file};
the function shall return true if the compilation
and its resulting compiled file respected all system-specific invariants,
and false (@code{nil}) if it broke any of those invariants;
it may issue warnings or errors before it returns @code{nil}.
(NB: The ability to pass such extra flags
is only available starting with ASDF 2.22.3.)
This feature is notably exercised by asdf-finalizers.

By using a string, you may reference
a function, symbol and/or package
that will only be created later during the build, but
isn't yet present at the time the defsystem form is evaluated.
However, if your entire system is using such a hook, you may have to
explicitly override the hook with @code{nil} for all the modules and files
that are compiled before the hook is defined.

Using this hook, you may achieve such effects as:
locally renaming packages,
binding @var{*readtables*} and other syntax-controlling variables,
handling warnings and other conditions,
proclaiming consistent optimization settings,
saving code coverage information,
maintaining meta-data about compilation timings,
setting gensym counters and PRNG seeds and other sources of non-determinism,
overriding the source-location and/or timestamping systems,
checking that some compile-time side-effects were properly balanced,
etc.

Note that there is no around-load hook. This is on purpose.
Some implementations such as ECL, GCL or MKCL link object files,
which allows for no such hook.
Other implementations allow for concatenating FASL files,
which doesn't allow for such a hook either.
We aim to discourage something that's not portable,
and has some dubious impact on performance and semantics
even when it is possible.
Things you might want to do with an around-load hook
are better done around-compile,
though it may at times require some creativity
(see e.g. the @code{package-renaming} system).


@node Controlling source file character encoding, Miscellaneous Functions, Controlling file compilation, Miscellaneous additional functionality
@section Controlling source file character encoding

Starting with ASDF 2.21, components accept a @code{:encoding} option
so authors may specify which character encoding should be used
to read and evaluate their source code.
When left unspecified, the encoding is inherited
from the parent module or system;
if no encoding is specified at any point,
or if @code{nil} is explicitly specified,
an extensible protocol described below is followed,
that ultimately defaults to @code{:utf-8} since ASDF 3.

The protocol to determine the encoding is
to call the function @code{detect-encoding},
which itself, if provided a valid file,
calls the function specified by @var{*encoding-detection-hook*},
or else defaults to the @var{*default-encoding*}.
The @var{*encoding-detection-hook*} is by default bound
to function @code{always-default-encoding},
that always returns the contents of @var{*default-encoding*}.
@var{*default-encoding*} is bound to @code{:utf-8} by default
(before ASDF 3, the default was @code{:default}).

Whichever encoding is returned must be a portable keyword,
that will be translated to an implementation-specific external-format designator
by function @code{encoding-external-format},
which itself simply calls the function specified @var{*encoding-external-format-hook*};
that function by default is @code{default-encoding-external-format},
that only recognizes @code{:utf-8} and @code{:default},
and translates the former to the implementation-dependent @var{*utf-8-external-format*},
and the latter to itself (that itself is portable but has an implementation-dependent meaning).

In other words, there now are plenty of extension hooks, but
by default ASDF enforces the previous @emph{de facto} standard behaviour
of using @code{:utf-8}, independently from
whatever configuration the user may be using.
Thus, system authors can now rely on @code{:utf-8}
being used while compiling their files,
even if the user is currently using @code{:koi8-r} or @code{:euc-jp}
as their interactive encoding.
(Before ASDF 3, there was no such guarantee, @code{:default} was used,
and only plain ASCII was safe to include in source code.)

Some legacy implementations only support 8-bit characters,
and some implementations provide 8-bit only variants.
On these implementations, the @var{*utf-8-external-format*}
gracefully falls back to @code{:default},
and Unicode characters will be read as multi-character mojibake.
To detect such situations, UIOP will push the @code{:asdf-unicode} feature
on implementations that support Unicode, and you can use reader-conditionalization
to protect any @code{:encoding @emph{encoding}} statement, as in
@code{#+asdf-unicode :encoding #+asdf-unicode :utf-8}.
We recommend that you avoid using unprotected @code{:encoding} specifications
until after ASDF 2.21 or later becomes widespread.
As of May 2016, all maintained implementations provide ASDF 3.1,
so you may prudently start using this and other features without such protection.

While it offers plenty of hooks for extension,
and one such extension is available (see @code{asdf-encodings} below),
ASDF itself only recognizes one encoding beside @code{:default},
and that is @code{:utf-8}, which is the @emph{de facto} standard,
already used by the vast majority of libraries that use more than ASCII.
On implementations that do not support unicode,
the feature @code{:asdf-unicode} is absent, and
the @code{:default} external-format is used
to read even source files declared as @code{:utf-8}.
On these implementations, non-ASCII characters
intended to be read as one CL character
may thus end up being read as multiple CL characters.
In most cases, this shouldn't affect the software's semantics:
comments will be skipped just the same, strings with be read and printed
with slightly different lengths, symbol names will be accordingly longer,
but none of it should matter.
But a few systems that actually depend on unicode characters
may fail to work properly, or may work in a subtly different way.
See for instance @code{lambda-reader}.

We invite you to embrace UTF-8
as the encoding for non-ASCII characters starting today,
even without any explicit specification in your @file{.asd} files.
Indeed, on some implementations and configurations,
UTF-8 is already the @code{:default},
and loading your code may cause errors if it is encoded in anything but UTF-8.
Therefore, even with the legacy behaviour,
non-UTF-8 is guaranteed to break for some users,
whereas UTF-8 is pretty much guaranteed not to break anywhere
(provided you do @emph{not} use a BOM),
although it might be read incorrectly on some implementations.
@code{:utf-8} has been the default value of @code{*default-encoding*} since ASDF 3.

If you need non-standard character encodings for your source code,
use the extension system @code{asdf-encodings}, by specifying
@code{:defsystem-depends-on ("asdf-encodings")} in your @code{defsystem}.
This extension system will register support for more encodings using the
@code{*encoding-external-format-hook*} facility,
so you can explicitly specify @code{:encoding :latin1}
in your @file{.asd} file.
Using the @code{*encoding-detection-hook*} it will also
eventually implement some autodetection of a file's encoding
from an emacs-style @code{-*- mode: lisp ; coding: latin1 -*-} declaration,
or otherwise based on an analysis of octet patterns in the file.
At this point, @code{asdf-encoding} only supports the encodings
that are supported as part of your implementation.
Since the list varies depending on implementations,
we still recommend you use @code{:utf-8} everywhere,
which is the most portable (next to it is @code{:latin1}).

Recent versions of Quicklisp include @code{asdf-encodings};
if you're not using it, you may get this extension using git:
@kbd{git clone https://gitlab.common-lisp.net/asdf/asdf-encodings.git}
or
@kbd{git clone git@@gitlab.common-lisp.net:asdf/asdf-encodings.git}.
You can also browse the repository on
@url{https://gitlab.common-lisp.net/asdf/asdf-encodings}.

When you use @code{asdf-encodings},
any @file{.asd} file loaded
will use the autodetection algorithm to determine its encoding.
If you depend on this detection happening,
you should explicitly load @code{asdf-encodings} early in your build.
Note that @code{:defsystem-depends-on} cannot be used here: by the time
the @code{:defsystem-depends-on} is loaded, the enclosing
@code{defsystem} form has already been read.

In practice, this means that the @code{*default-encoding*}
is usually used for @file{.asd} files.
Currently, this defaults to @code{:utf-8}, and
you should be safe using Unicode characters in those files.
This might matter, for instance, in meta-data about author's names.
Otherwise, the main data in these files is component (path)names,
and we don't recommend using non-ASCII characters for these,
for the result probably isn't very portable.

@node Miscellaneous Functions, Some Utility Functions, Controlling source file character encoding, Miscellaneous additional functionality
@section Miscellaneous Functions

These functions are exported by ASDF for your convenience.

@anchor{system-relative-pathname}
@defun system-relative-pathname system name @Akey{} type

It's often handy to locate a file relative to some system.
The @code{system-relative-pathname} function meets this need.

It takes two mandatory arguments @var{system} and @var{name}
and a keyword argument @var{type}:
@var{system} is name of a system, whereas @var{name} and optionally @var{type}
specify a relative pathname, interpreted like a component pathname specifier
by @code{coerce-pathname}. @xref{Pathname specifiers}.

It returns a pathname built from the location of the system's
source directory and the relative pathname. For example:

@lisp
> (asdf:system-relative-pathname 'cl-ppcre "regex.data")
#P"/repository/other/cl-ppcre/regex.data"
@end lisp

@end defun

@defun system-source-directory system-designator

ASDF does not provide a turnkey solution for locating
data (or other miscellaneous) files
that are distributed together with the source code of a system.
Programmers can use @code{system-source-directory} to find such files.
Returns a pathname object.
The @var{system-designator} may be a string, symbol, or ASDF system object.
@end defun

@defun clear-system system-designator

It is sometimes useful to force recompilation of a previously loaded system.
For these cases, @code{(asdf:clear-system :foo)}
will remove the system from the table of currently loaded systems:
the next time the system @code{foo} or one that depends on it is re-loaded,
@code{foo} will be loaded again.@footnote{Alternatively, you could touch @code{foo.asd} or
remove the corresponding fasls from the output file cache.}

Note that this does not and cannot undo
the previous loading of the system.
Common Lisp has no provision for such an operation,
and its reliance on irreversible side-effects to global data structures
makes such a thing impossible in the general case.
If the software being re-loaded is not conceived with hot upgrade in mind,
re-loading may cause many errors, warnings or subtle silent problems,
as packages, generic function signatures, structures, types, macros, constants, etc.
are being redefined incompatibly.
It is up to the user to make sure that reloading is possible and has the desired effect.
In some cases, extreme measures such as recursively deleting packages,
unregistering symbols, defining methods on @code{update-instance-for-redefined-class}
and much more are necessary for reloading to happen smoothly.
ASDF itself goes to extensive effort to make a hot upgrade possible
with respect to its own code.
If you want, you can reuse some of its utilities such as
@code{uiop:define-package} and @code{uiop:with-upgradability},
and get inspiration (or disinspiration)
from what it does in @file{header.lisp} and @file{upgrade.lisp}.
@end defun

@defun register-preloaded-system name @Arest{} keys @Akey{} version @AallowOtherKeys{}
A system with name @var{name},
created by @code{make-instance} with extra keys @var{keys}
(e.g. @code{:version}),
is registered as @emph{preloaded}.
If @var{version} is @code{t} (default), then the version is copied from the defined system
of the same name (if registered) or else is @code{nil}
(this automatic copy of version is only available starting since ASDF 3.1.8).

A preloaded system is considered as having already been loaded into the current image,
and if at some point some other system @code{:depends-on} it yet no source code is found,
it is considered as already provided,
and ASDF will not raise a @code{missing-component} error.

This function is particularly useful if you distribute your code
as fasls with either @code{compile-bundle-op} or @code{monolithic-compile-bundle-op},
and want to register systems so that dependencies will work uniformly
whether you're using your software from source or from fasl.

Note that if the system was already defined or loaded from source code,
its build information will remain active until you call @code{clear-system} on it,
at which point a system without build information will be registered in its place.
@end defun

@defun register-immutable-system name @Arest{} keys
A system with name @var{name} is registered as preloaded,
and additionally is marked as @emph{immutable}:
that is, attempts to compile or load it will be succeed
without actually reading, creating or loading any file,
as if the system was passed as a @code{force-not} argument
to all calls to @code{plan} or @code{operate}.
There will be no search for an updated @file{.asd} file
to override the loaded version,
whether from the source-register or any other method.

If a @var{version} keyword argument is specified as @code{t} or left unspecified,
then the version is copied from the defined system
of the same name (if registered) or else is @code{nil}.
This automatic copy of version is available starting
since immutable systems have been available in ASDF 3.1.5.

This function, available since ASDF 3.1.5, is particularly useful
if you distribute a large body of code as a precompiled image,
and want to allow users to extend the image with further extension systems,
but without making thousands of filesystem requests looking for inexistent
(or worse, out of date) source code
for all the systems that came bundled with the image but aren't
distributed as source code to regular users.
@cindex immutable systems
@end defun

@defun run-shell-command control-string @Arest{} args

This function is obsolete and present only for the sake of backwards-compatibility:
``If it's not backwards, it's not compatible''. We @emph{strongly} discourage its use.
Its current behaviour is only well-defined on Unix platforms
(which include MacOS X and cygwin). On Windows, anything goes.
The following documentation is only for the purpose of your migrating away from it
in a way that preserves semantics.

Instead we recommend the use @code{run-program}, described in the next section, and
available as part of ASDF since ASDF 3.

@code{run-shell-command} takes as arguments a format @code{control-string}
and arguments to be passed to @code{format} after this control-string
to produce a string.
This string is a command that will be evaluated with a POSIX shell if possible;
yet, on Windows, some implementations will use CMD.EXE,
while others (like SBCL) will make an attempt at invoking a POSIX shell
(and fail if it is not present).
@end defun

@node Some Utility Functions,  , Miscellaneous Functions, Miscellaneous additional functionality
@section Some Utility Functions

The below functions are not exported by ASDF itself, but by UIOP, available since ASDF 3.
Some of them have precursors in ASDF 2, but we recommend that for active developments,
you should rely on the package UIOP as included in ASDF 3.
UIOP provides many, many more utility functions, and we recommend
you read its @file{README.md} and sources for more information.


@defun parse-unix-namestring name @Akey{} type defaults dot-dot ensure-directory @AallowOtherKeys
Coerce @var{name} into a @var{pathname} using standard Unix syntax.

Unix syntax is used whether or not the underlying system is Unix;
on non-Unix systems it is only usable for relative pathnames.
In order to manipulate relative pathnames portably, it is crucial
to possess a portable pathname syntax independent of the underlying OS.
This is what @code{parse-unix-namestring} provides, and why we use it in ASDF.

When given a @code{pathname} object, just return it untouched.
When given @code{nil}, just return @code{nil}.
When given a non-null @code{symbol}, first downcase its name and treat it as a string.
When given a @code{string}, portably decompose it into a pathname as below.

@code{#\/} separates directory components.

The last @code{#\/}-separated substring is interpreted as follows:
1- If @var{type} is @code{:directory} or @var{ensure-directory} is true,
 the string is made the last directory component, and its @code{name} and @code{type} are @code{nil}.
 if the string is empty, it's the empty pathname with all slots @code{nil}.
2- If @var{type} is @code{nil}, the substring is a file-namestring,
 and its @code{name} and @code{type} are separated by @code{split-name-type}.
3- If @var{type} is a string, it is the given @code{type}, and the whole string is the @code{name}.

Directory components with an empty name the name @code{.} are removed.
Any directory named @code{..} is read as @var{dot-dot},
which must be one of @code{:back} or @code{:up} and defaults to @code{:back}.

@vindex *nil-pathname*
@code{host}, @code{device} and @code{version} components are taken from @var{defaults},
which itself defaults to @code{*nil-pathname*}.
@code{*nil-pathname*} is also used if @var{defaults} is @code{nil}.
No host or device can be specified in the string itself,
which makes it unsuitable for absolute pathnames outside Unix.

For relative pathnames, these components (and hence the defaults) won't matter
if you use @code{merge-pathnames*} but will matter if you use @code{merge-pathnames},
which is an important reason to always use @code{merge-pathnames*}.

Arbitrary keys are accepted, and the parse result is passed to @code{ensure-pathname}
with those keys, removing @var{type}, @var{defaults} and @var{dot-dot}.
When you're manipulating pathnames that are supposed to make sense portably
even though the OS may not be Unixish, we recommend you use @code{:want-relative t}
so that @code{parse-unix-namestring} will throw an error if the pathname is absolute.
@end defun

@defun merge-pathnames* specified @Aoptional{} defaults

This function is a replacement for @code{merge-pathnames} that uses the host and device
from the @var{defaults} rather than the @var{specified} pathname when the latter
is a relative pathname. This allows ASDF and its users to create and use relative pathnames
without having to know beforehand what are the host and device
of the absolute pathnames they are relative to.

@end defun

@defun subpathname pathname subpath @Akey{} type

This function takes a @var{pathname} and a @var{subpath} and a @var{type}.
If @var{subpath} is already a @code{pathname} object (not namestring),
and is an absolute pathname at that, it is returned unchanged;
otherwise, @var{subpath} is turned into a relative pathname with given @var{type}
as per @code{parse-unix-namestring} with @code{:want-relative t :type }@var{type},
then it is merged with the @code{pathname-directory-pathname} of @var{pathname},
as per @code{merge-pathnames*}.

We strongly encourage the use of this function
for portably resolving relative pathnames in your code base.
@end defun

@defun subpathname* pathname subpath @Akey{} type

This function returns @code{nil} if the base @var{pathname} is @code{nil},
otherwise acts like @code{subpathname}.
@end defun

@defun run-program command @Akey{} ignore-error-status force-shell input output @
error-output if-input-does-not-exist if-output-exists if-error-output-exists @
element-type external-format @AallowOtherKeys

@code{run-program} takes a @var{command} argument that is either
a list of a program name or path and its arguments,
or a string to be executed by a shell.
It spawns the command, waits for it to return,
verifies that it exited cleanly (unless told not too below),
and optionally captures and processes its output.
It accepts many keyword arguments to configure its behaviour.

@code{run-program} returns three values: the first for the output,
the second for the error-output, and the third for the return value.
(Beware that before ASDF 3.0.2.11, it didn't handle input or error-output,
and returned only one value,
the one for the output if any handler was specified, or else the exit code;
please upgrade ASDF, or at least UIOP, to rely on the new enhanced behaviour.)

@var{output} is its most important argument;
it specifies how the output is captured and processed.
If it is @code{nil}, then the output is redirected to the null device,
that will discard it.
If it is @code{:interactive}, then it is inherited from the current process
(beware: this may be different from your @var{*standard-output*},
and under SLIME will be on your @code{*inferior-lisp*} buffer).
If it is @code{t}, output goes to your current @var{*standard-output*} stream.
Otherwise, @var{output} should be a value that is a suitable first argument to
@code{slurp-input-stream} (see below), or
a list of such a value and keyword arguments.
In this case, @code{run-program} will
create a temporary stream for the program output;
the program output, in that stream,
will be processed by a call to @code{slurp-input-stream},
using @var{output} as the first argument
(or if it's a list the first element of @var{output} and the rest as keywords).
The primary value resulting from that call
(or @code{nil} if no call was needed)
will be the first value returned by @code{run-program}.
E.g., using @code{:output :string}
will have it return the entire output stream as a string.
And using @code{:output '(:string :stripped t)}
will have it return the same string stripped of any ending newline.

@var{error-output} is similar to @var{output}, except that
the resulting value is returned as the second value of @code{run-program}.
@code{t} designates the @var{*error-output*}.
Also @code{:output} means redirecting the error output to the output stream,
in which case @code{nil} is returned.

@var{input} is similar to @var{output}, except that
@code{vomit-output-stream} is used, no value is returned,
and @code{t} designates the @var{*standard-input*}.

@code{element-type} and @code{external-format} are passed on
to your Lisp implementation, when applicable, for creation of the output stream.

One and only one of the stream slurping or vomiting may or may not happen
in parallel in parallel with the subprocess,
depending on options and implementation,
and with priority being given to output processing.
Other streams are completely produced or consumed
before or after the subprocess is spawned, using temporary files.

@code{force-shell} forces evaluation of the command through a shell,
even if it was passed as a list rather than a string.
If a shell is used, it is @file{/bin/sh} on Unix or @file{CMD.EXE} on Windows,
except on implementations that (erroneously, IMNSHO)
insist on consulting @code{$SHELL} like clisp.

@code{ignore-error-status} causes @code{run-program}
to not raise an error if the spawned program exits in error.
Following POSIX convention, an error is anything but
a normal exit with status code zero.
By default, an error of type @code{subprocess-error} is raised in this case.

@code{run-program} works on all platforms supported by ASDF, except Genera.
See the source code for more documentation.

@end defun

@defun slurp-input-stream processor input-stream @Akey{}

@code{slurp-input-stream} is a generic function of two arguments, a target object and an input stream,
and accepting keyword arguments.
Predefined methods based on the target object are as follows:

@itemize
@item
If the object is a function, the function is called with the stream as argument.

@item If the object is a cons, its first element is applied to its rest appended by
a list of the input stream.

@item If the object is an output stream, the contents of the input stream are copied to it.
If the @var{linewise} keyword argument is provided, copying happens line by line,
and an optional @var{prefix} is printed before each line.
Otherwise, copying happen based on a buffer of size @var{buffer-size},
using the specified @var{element-type}.

@item If the object is @code{'string} or @code{:string}, the content is captured into a string.
Accepted keywords include the @var{element-type} and a flag @var{stripped},
which when true causes any single line ending to be removed as per @code{uiop:stripln}.

@item If the object is @code{:lines}, the content is captured as a list of strings,
one per line, without line ending. If the @var{count} keyword argument is provided,
it is a maximum count of lines to be read.

@item If the object is @code{:line}, the content is captured as with @code{:lines} above,
and then its sub-object is extracted with the @var{at} argument,
which defaults to @code{0}, extracting the first line.
A number will extract the corresponding line.
See the documentation for @code{uiop:access-at}.

@item If the object is @code{:forms}, the content is captured as a list of s-expressions,
as read by the Lisp reader.
If the @var{count} argument is provided,
it is a maximum count of lines to be read.
We recommend you control the syntax with such macro as
@code{uiop:with-safe-io-syntax}.

@item If the object is @code{:form}, the content is captured as with @code{:forms} above,
and then its sub-object is extracted with the @var{at} argument,
which defaults to @code{0}, extracting the first form.
A number will extract the corresponding form.
See the documentation for @code{uiop:access-at}.
We recommend you control the syntax with such macro as
@code{uiop:with-safe-io-syntax}.
@end itemize
@end defun


@node Getting the latest version, FAQ, Miscellaneous additional functionality, Top
@comment  node-name,  next,  previous,  up
@chapter Getting the latest version


Decide which version you want.
The @code{master} branch is where development happens;
its @code{HEAD} is usually OK, including the latest fixes and portability tweaks,
but an occasional regression may happen despite our (limited) test suite.

The @code{release} branch is what cautious people should be using;
it has usually been tested more, and releases are cut at a point
where there isn't any known unresolved issue.

You may get the ASDF source repository using git:
@kbd{git clone https://gitlab.common-lisp.net/asdf/asdf.git}

You will find the above referenced tags in this repository.
You can also browse the repository on
@url{https://gitlab.common-lisp.net/asdf/asdf}.

Discussion of ASDF development is conducted on the
mailing list (@pxref{Mailing list}).

@node FAQ, Ongoing Work, Getting the latest version, Top
@comment  node-name,  next,  previous,  up
@chapter FAQ

@menu
* Where do I report a bug?::
* Mailing list::
* What has changed between ASDF 1 ASDF 2 and ASDF 3?::
* Issues with installing the proper version of ASDF::
* Issues with configuring ASDF::
* Issues with using and extending ASDF to define systems::
* ASDF development FAQs::
@end menu

@node Where do I report a bug?, Mailing list, FAQ, FAQ
@section  ``Where do I report a bug?''
@cindex bug tracker
@cindex gitlab
@cindex launchpad
ASDF bugs are tracked on common-lisp.net's gitlab:: @url{https://gitlab.common-lisp.net/asdf/asdf/issues}.
Previously, we had done bug-tracking on @url{https://launchpad.net/asdf},
but we are now consolidating project management on @code{common-lisp.net}.

If you're unsure about whether something is a bug, or for general discussion,
use the asdf-devel mailing list (@pxref{Mailing list}).

@node Mailing list, What has changed between ASDF 1 ASDF 2 and ASDF 3?, Where do I report a bug?, FAQ
@section Mailing list
@cindex mailing list

Discussion of ASDF development is conducted on the
mailing list
@kbd{asdf-devel@@common-lisp.net}.
@url{http://common-lisp.net/cgi-bin/mailman/listinfo/asdf-devel}


@node What has changed between ASDF 1 ASDF 2 and ASDF 3?, Issues with installing the proper version of ASDF, Mailing list, FAQ
@section ``What has changed between ASDF 1, ASDF 2, and ASDF 3?''

We released ASDF 2.000 on May 31st 2010,
ASDF 3.0.0 on May 15th 2013,
ASDF 3.1.2 on May 6th 2014.
Releases of ASDF 2 and now ASDF 3 have since then been included
in all actively maintained CL implementations that used to bundle ASDF 1,
plus many implementations that previously did not.
ASDF has been made to work with all actively maintained CL
implementations and even a few implementations that are @emph{not}
actively maintained.

Furthermore, it is possible to upgrade from ASDF 1 to ASDF 2 or ASDF 3 on the fly
(though we recommend instead upgrading your implementation or replacing its ASDF module).
For this reason, we have stopped supporting ASDF 1 and ASDF 2.
If you are using ASDF 1 or ASDF 2 and are experiencing any kind of issues or limitations,
we recommend you upgrade to ASDF 3
--- and we explain how to do that. @xref{Loading ASDF}.

Note that in the context of compatibility requirements,
ASDF 2.27, released on Feb 1st 2013, and further releases up to 2.33,
count as pre-releases of ASDF 3, and define the @code{:asdf3} feature,
though the first stable release of ASDF 3 was release 3.0.1.
Significant new or improved functionality were added in ASDF 3.1;
the @code{:asdf3.1} feature is present in recent enough versions to detect this functionality;
the first stable release since then was ASDF 3.1.2.
New @code{*features*} are only added at major milestones,
and the next one will probably be @code{:asdf3.2}.


@menu
* What are ASDF 1 2 3?::
* How do I detect the ASDF version?::
* ASDF can portably name files in subdirectories::
* Output translations::
* Source Registry Configuration::
* Usual operations are made easier to the user::
* Many bugs have been fixed::
* ASDF itself is versioned::
* ASDF can be upgraded::
* Decoupled release cycle::
* Pitfalls of the transition to ASDF 2::
* Pitfalls of the upgrade to ASDF 3::
* What happened to the bundle operations::
@end menu

@node What are ASDF 1 2 3?, How do I detect the ASDF version?, What has changed between ASDF 1 ASDF 2 and ASDF 3?, What has changed between ASDF 1 ASDF 2 and ASDF 3?
@subsection What are ASDF 1, ASDF 2, and ASDF 3?

ASDF 1 refers to any release earlier than 1.369 or so (from August 2001 to October 2009),
and to any development revision earlier than 2.000 (May 2010).
If your copy of ASDF doesn't even contain version information, it's an old ASDF 1.
Revisions between 1.656 and 1.728 may count as development releases for ASDF 2.

ASDF 2 refers to releases from 2.000 (May 31st 2010) to 2.26 (Oct 30th 2012),
and any development revision newer than ASDF 1 and older than 2.27 (Feb 1st 2013).

ASDF 3 refers to releases from 2.27 (Feb 1st 2013) to 2.33 and 3.0.0 onward (May 15th 2013).
2.27 to 2.33 count as pre-releases to ASDF 3.

ASDF 3.1 refers to releases from 3.1.2 (May 6th 2014) onward.
These releases are also considered part of ASDF 3.

@node How do I detect the ASDF version?, ASDF can portably name files in subdirectories, What are ASDF 1 2 3?, What has changed between ASDF 1 ASDF 2 and ASDF 3?
@subsection How do I detect the ASDF version?
@findex asdf-version
@cindex *features*

All releases of ASDF
push @code{:asdf} onto @code{*features*}.
Releases starting with ASDF 2
push @code{:asdf2} onto @code{*features*}.
Releases starting with ASDF 3 (including 2.27 and later pre-releases)
push @code{:asdf3} onto @code{*features*}.
Furthermore, releases starting with ASDF 3.1.2 (May 2014),
though they count as ASDF 3, include enough progress that they
also push @code{:asdf3.1} onto @code{*features*}.
You may depend on the presence or absence of these features
to write code that takes advantage of recent ASDF functionality
but still works on older versions, or at least detects the old version and signals an error.

Additionally, all releases starting with ASDF 2
define a function @code{(asdf:asdf-version)} you may use to query the version.
All releases starting with 2.013 display the version number prominently
on the second line of the @file{asdf.lisp} source file.

If you are experiencing problems or limitations of any sort with ASDF 1 or ASDF 2,
we recommend that you should upgrade to the latest release, be it ASDF 3 or other.

Finally, here is a code snippet to programmatically determine what version of ASDF is loaded, if any,
that works on all versions including very old ones:

@lisp
(when (find-package :asdf)
  (let ((ver (symbol-value
                   (or (find-symbol (string :*asdf-version*) :asdf)
                       (find-symbol (string :*asdf-revision*) :asdf)))))
    (etypecase ver
      (string ver)
      (cons (with-output-to-string (s)
              (loop for (n . m) on ver
                    do (princ n s)
                       (when m (princ "." s)))))
      (null "1.0"))))
@end lisp

If it returns @code{nil} then ASDF is not installed.
Otherwise it should return a string.
If it returns @code{"1.0"}, then it can actually be
any version before 1.77 or so, or some buggy variant of 1.x.
If it returns anything older than @code{"3.0.1"},
you really need to upgrade your implementation or at least upgrade its ASDF.
@xref{Replacing your implementation's ASDF}.



@node ASDF can portably name files in subdirectories, Output translations, How do I detect the ASDF version?, What has changed between ASDF 1 ASDF 2 and ASDF 3?
@subsection ASDF can portably name files in subdirectories

Common Lisp namestrings are not portable,
except maybe for logical pathname namestrings,
that themselves have various limitations and require a lot of setup
that is itself ultimately non-portable.

In ASDF 1, the only portable ways to refer to pathnames inside systems and components
were very awkward, using @code{#.(make-pathname ...)} and
@code{#.(merge-pathnames ...)}.
Even the above were themselves were inadequate in the general case
due to host and device issues, unless horribly complex patterns were used.
Plenty of simple cases that looked portable actually weren't,
leading to much confusion and greavance.

ASDF 2 implements its own portable syntax for strings as pathname specifiers.
Naming files within a system definition becomes easy and portable again.
@xref{Miscellaneous additional functionality,system-relative-pathname},
@code{merge-pathnames*},
@code{coerce-pathname}.

On the other hand, there are places where systems used to accept namestrings
where you must now use an explicit pathname object:
@code{(defsystem ... :pathname "LOGICAL-HOST:PATH;TO;SYSTEM;" ...)}
must now be written with the @code{#p} syntax:
@code{(defsystem ... :pathname #p"LOGICAL-HOST:PATH;TO;SYSTEM;" ...)}.
We recommend against using pathname objects in general and logical pathnames in particular.
Your code will be much more portable using ASDF's pathname specifiers.

@xref{Pathname specifiers}.


@node Output translations, Source Registry Configuration, ASDF can portably name files in subdirectories, What has changed between ASDF 1 ASDF 2 and ASDF 3?
@subsection Output translations

A popular feature added to ASDF was output pathname translation:
@code{asdf-binary-locations}, @code{common-lisp-controller},
@code{cl-launch} and other hacks were all implementing it in ways
both mutually incompatible and difficult to configure.

Output pathname translation is essential to share
source directories of portable systems across multiple implementations
or variants thereof,
or source directories of shared installations of systems across multiple users,
or combinations of the above.

In ASDF 2, a standard mechanism is provided for that,
@code{asdf-output-translations},
with sensible defaults, adequate configuration languages,
a coherent set of configuration files and hooks,
and support for non-Unix platforms.

@xref{Controlling where ASDF saves compiled files}.

@node Source Registry Configuration, Usual operations are made easier to the user, Output translations, What has changed between ASDF 1 ASDF 2 and ASDF 3?
@subsection Source Registry Configuration

Configuring ASDF used to require special magic
to be applied just at the right moment,
between the moment ASDF is loaded and the moment it is used,
in a way that is specific to the user,
the implementation he is using and the application he is building.

This made for awkward configuration files and startup scripts
that could not be shared between users, managed by administrators
or packaged by distributions.

ASDF 2 provides a well-documented way to configure ASDF,
with sensible defaults, adequate configuration languages,
and a coherent set of configuration files and hooks.

We believe it's a vast improvement because it decouples
application distribution from library distribution.
The application writer can avoid thinking where the libraries are,
and the library distributor (dpkg, clbuild, advanced user, etc.)
can configure them once and for every application.
Yet settings can be easily overridden where needed,
so whoever needs control has exactly as much as required.

At the same time, ASDF 2 remains compatible
with the old magic you may have in your build scripts
(using @code{*central-registry*} and
@code{*system-definition-search-functions*})
to tailor the ASDF configuration to your build automation needs,
and also allows for new magic, simpler and more powerful magic.

@xref{Controlling where ASDF searches for systems}.


@node Usual operations are made easier to the user, Many bugs have been fixed, Source Registry Configuration, What has changed between ASDF 1 ASDF 2 and ASDF 3?
@subsection Usual operations are made easier to the user

In ASDF 1, you had to use the awkward syntax
@code{(asdf:oos 'asdf:load-op :foo)}
to load a system,
and similarly for @code{compile-op}, @code{test-op}.

In ASDF 2 and later, you can use shortcuts for the usual operations:
@code{(asdf:load-system :foo)}, and
similarly for @code{compile-system}, @code{test-system}.


@node Many bugs have been fixed, ASDF itself is versioned, Usual operations are made easier to the user, What has changed between ASDF 1 ASDF 2 and ASDF 3?
@subsection Many bugs have been fixed

The following issues and many others have been fixed:

@itemize
@item
The infamous TRAVERSE function has been revamped completely
between ASDF 1 and ASDF 2, with many bugs squashed.
In particular, dependencies were not correctly propagated
across modules but now are.
It has been completely rewritten many times over
between ASDF 2.000 and ASDF 3,
with fundamental issues in the original model being fixed.
Timestamps were not propagated at all, and now are.
The internal model of how actions depend on each other
is now both consistent and complete.
As of ASDF 3.3, multiple phases of loading are well supported,
wherein correctly interpreting `defsystem` statements in some `.asd` files
itself depends on loading other systems, e.g. via `:defsystem-depends-on`.
The @code{:version} and
the @code{:force (system1 .. systemN)} features have been fixed.

@item
Performance has been notably improved for large systems
(say with thousands of components) by using
hash-tables instead of linear search,
and linear-time list accumulation instead of cubic time recursive append,
for an overall @emph{O(n)} complexity vs @emph{O(n^4)}.

@item
Many features used to not be portable,
especially where pathnames were involved.
Windows support was notably quirky because of such non-portability.

@item
The internal test suite used to massively fail on many implementations.
While still incomplete, it now fully passes
on all implementations supported by the test suite,
though some tests are commented out on a few implementations.

@item
Support was lacking for some implementations.
ABCL and GCL were notably wholly broken.
ECL extensions were not integrated with ASDF release.

@item
The documentation was grossly out of date.

@end itemize


@node ASDF itself is versioned, ASDF can be upgraded, Many bugs have been fixed, What has changed between ASDF 1 ASDF 2 and ASDF 3?
@subsection ASDF itself is versioned

Between new features, old bugs fixed, and new bugs introduced,
there were various releases of ASDF in the wild,
and no simple way to check which release had which feature set.
People using or writing systems had to either make worst-case assumptions
as to what features were available and worked,
or take great pains to have the correct version of ASDF installed.

With ASDF 2 and later, we provide a new stable set of working features
that everyone can rely on from now on.
Use @code{#+asdf2}, @code{#+asdf3}, @code{#+asdf3.1} or @code{#+asdf3.3}
to detect presence of relevant versions of ASDF and their features, or
@code{(asdf:version-satisfies (asdf:asdf-version) "2.345.67")}
to check the availability of a version no earlier than required.


@node ASDF can be upgraded, Decoupled release cycle, ASDF itself is versioned, What has changed between ASDF 1 ASDF 2 and ASDF 3?
@subsection ASDF can be upgraded

When an old version of ASDF was loaded,
it was very hard to upgrade ASDF in your current image
without breaking everything.
Instead you had to exit the Lisp process and
somehow arrange to start a new one from a simpler image.
Something that can't be done from within Lisp,
making automation of it difficult,
which compounded with difficulty in configuration,
made the task quite hard.
Yet as we saw before, the task would have been required
to not have to live with the worst case or non-portable
subset of ASDF features.

With ASDF 2, it is easy to upgrade
from ASDF 2 to later versions from within Lisp,
and not too hard to upgrade from ASDF 1 to ASDF 2 from within Lisp.
We support hot upgrade of ASDF and any breakage is a bug
that we will do our best to fix.
There are still limitations on upgrade, though,
most notably the fact that after you upgrade ASDF,
you must also reload or upgrade all ASDF extensions.

@node Decoupled release cycle, Pitfalls of the transition to ASDF 2, ASDF can be upgraded, What has changed between ASDF 1 ASDF 2 and ASDF 3?
@subsection Decoupled release cycle

When vendors were releasing their Lisp implementations with ASDF,
they had to basically never change version
because neither upgrade nor downgrade was possible
without breaking something for someone,
and no obvious upgrade path was visible and recommendable.

With ASDF 2, upgrade is possible, easy and can be recommended.
This means that vendors can safely ship a recent version of ASDF,
confident that if a user isn't fully satisfied,
he can easily upgrade ASDF and deal
with a supported recent version of it.
This means that release cycles will be causally decoupled,
the practical consequence of which will mean faster convergence
towards the latest version for everyone.


@node Pitfalls of the transition to ASDF 2, Pitfalls of the upgrade to ASDF 3, Decoupled release cycle, What has changed between ASDF 1 ASDF 2 and ASDF 3?
@subsection Pitfalls of the transition to ASDF 2

The main pitfalls in upgrading to ASDF 2 seem to be related
to the output translation mechanism.

@itemize

@item
Output translations is enabled by default. This may surprise some users,
most of them in pleasant way (we hope), a few of them in an unpleasant way.
It is trivial to disable output translations.
@xref{FAQ,,``How can I wholly disable the compiler output cache?''}.

@item
Some systems in the large have been known
not to play well with output translations.
They were relatively easy to fix.
Once again, it is also easy to disable output translations,
or to override its configuration.

@item
The new ASDF output translations are incompatible with ASDF-Binary-Locations.
They replace A-B-L, and there is compatibility mode to emulate
your previous A-B-L configuration.
See @code{enable-asdf-binary-locations-compatibility} in
@pxref{Controlling where ASDF saves compiled files,,Backward Compatibility}.
But thou shalt not load ABL on top of ASDF 2.

@end itemize

Other issues include the following:

@itemize

@item
ASDF pathname designators are now specified
in places where they were unspecified,
and a few small adjustments have to be made to some non-portable defsystems.
Notably, in the @code{:pathname} argument
to a @code{defsystem} and its components,
a logical pathname (or implementation-dependent hierarchical pathname)
must now be specified with @code{#p} syntax
where the namestring might have previously sufficed;
moreover when evaluation is desired @code{#.} must be used,
where it wasn't necessary in the toplevel @code{:pathname} argument
(but necessary in other @code{:pathname} arguments).

@item
There is a slight performance bug, notably on SBCL,
when initially searching for @file{asd} files,
the implicit @code{(directory "/configured/path/**/*.asd")}
for every configured path @code{(:tree "/configured/path/")}
in your @code{source-registry} configuration can cause a slight pause.
Try to @code{(time (asdf:initialize-source-registry))}
to see how bad it is or isn't on your system.
If you insist on not having this pause,
you can avoid the pause by overriding the default source-registry configuration
and not use any deep @code{:tree} entry but only @code{:directory} entries
or shallow @code{:tree} entries.
Or you can fix your implementation to not be quite that slow
when recursing through directories.
@emph{Update}: This performance bug fixed the hard way in 2.010.

@item
On Windows, only LispWorks supports proper default configuration pathnames
based on the Windows registry.
Other implementations make do with environment variables,
that you may have to define yourself
if you're using an older version of Windows.
Windows support is somewhat less tested than Unix support.
Please help report and fix bugs.
@emph{Update}: As of ASDF 2.21, all implementations
should now use the same proper default configuration pathnames
and they should actually work, though they haven't all been tested.

@item
The mechanism by which one customizes a system so that Lisp files
may use a different extension from the default @file{.lisp} has changed.
Previously, the pathname for a component
was lazily computed when operating on a system,
and you would
@code{(defmethod source-file-type ((component cl-source-file) (system (eql (find-system 'foo))))
  (declare (ignorable component system)) "lis")}.
Now, the pathname for a component is eagerly computed when defining the system,
and instead you will @code{(defclass cl-source-file.lis (cl-source-file) ((type :initform "lis")))}
and use @code{:default-component-class cl-source-file.lis}
as argument to @code{defsystem},
as detailed in a @pxref{FAQ,How do I create a system definition where all the source files have a .cl extension?} below.
@code{source-file-type} is deprecated.  To access a component's
file-type, use @code{file-type}, instead.  @code{source-file-type} will
be removed.

@findex source-file-type
@findex file-type

@end itemize


@node Pitfalls of the upgrade to ASDF 3, What happened to the bundle operations, Pitfalls of the transition to ASDF 2, What has changed between ASDF 1 ASDF 2 and ASDF 3?
@subsection Pitfalls of the upgrade to ASDF 3

While ASDF 3 is largely compatible with ASDF 2,
there are a few pitfalls when upgrading from ASDF 2,
due to limitations in ASDF 2.

@itemize

@item
ASDF 2 was designed so it could be upgraded;
but upgrading it required a special setup at the beginning of your build files.
Failure to upgrade it early could result in catastrophic attempt to self-upgrade in mid-build.

@item
Starting with ASDF 3 (2.27 or later),
ASDF will automatically attempt to upgrade itself
as the first step before any system operation,
to avoid any possibility of such catastrophic mid-build self-upgrade.
But that doesn't help if your old implementation still provides ASDF 2.

@item
It was unsafe in ASDF 2 for a system definition to declare a dependency on ASDF,
since it could trigger such catastrophe for users who were not carefully configured.
If you declare a dependency on a recent enough ASDF,
yet want to be nice with these potentially misconfigured users,
we recommend that you not only specify a recent ASDF in your dependencies with
@code{:depends-on ((:version "asdf" "3.1.2"))},
but that you @emph{also} check that ASDF 3 is installed,
or else the upgrade catastrophe might happen before that specification is checked,
by starting your @file{.asd} file with a version check as follows:
@example
#-asdf3 (error "@var{MY-SYSTEM} requires ASDF 3.1.2")
@end example

@item
When you upgrade from too old a version of ASDF,
previously loaded ASDF extensions become invalid, and will need to be reloaded.
Example extensions include CFFI-Grovel, hacks used by ironclad, etc.
Since it isn't possible to automatically detect what extensions
need to be invalidated and what systems use them,
ASDF will invalidate @emph{all} previously loaded systems
when it is loaded on top of a forward-incompatible ASDF version.
@footnote{
@vindex *oldest-forward-compatible-asdf-version*
Forward incompatibility can be determined using the variable
@code{asdf/upgrade::*oldest-forward-compatible-asdf-version*},
which is 2.33 at the time of this writing.}

@item
To write a portable build script, you need to rely on a recent version of UIOP,
but until you have ensured a recent ASDF is loaded,
you can't rely on UIOP being present,
and thus must manually avoid all the pathname pitfalls when loading ASDF itself.

@item
Bugs in CMUCL and XCL prevent upgrade of ASDF from an old forward-incompatible version.
Happily, CMUCL comes with a recent ASDF,
and XCL is more of a working demo than something you'd use seriously anyway.

@item
For the above reasons, your build and startup scripts
should load ASDF 3, configure it, and upgrade it,
among the very first things they do.
They should ensure that only ASDF 3 or later is used indeed,
and error out if ASDF 2 or earlier was used.

@item
Now that (since May 2016) all maintained implementations
(i.e. having had at least one release since 2014,
or a commit on their public source code repository)
provide ASDF 3.1 or later,
the simple solution is just to use code as below in your setup,
and when it fails, upgrade your implementation or replace its ASDF.
(@pxref{Replacing your implementation's ASDF}):
@example
(require "asdf")
#-asdf3.1 (error "ASDF 3.1 or bust")
@end example

@item
For scripts that try to use ASDF simply via @code{require} at first, and
make heroic attempts to load it the hard way if at first they don't succeed,
see @file{tools/load-asdf.lisp} distributed with the ASDF source repository,
or the code of @url{https://cliki.net/cl-launch,@code{cl-launch}}.

@item @anchor{reinitializeASDFAfterUpgrade}
Note that in addition to the pitfalls and constraints above,
these heroic scripts (should you wish to write or modify one),
must take care to configure ASDF @emph{twice}.
A first time, right after you load the old ASDF 2 (or 1!)
and before you upgrade to the new ASDF 3,
so it may find where you put ASDF 3.
A second time, because most implementations can't handle a smooth upgrade from ASDF 2 to ASDF 3,
so ASDF 3 doesn't try (anymore) and loses any configuration from ASDF 2.
@lisp
(ignore-errors (funcall 'require "asdf")) ;; <--- try real hard
;; <--- insert heroics here, if that failed to provide ASDF 2 or 3
;; <--- insert configuration here, if that succeeded
(asdf:load-system "asdf")
;; <--- re-configure here, too, in case at first you got ASDF 2
@end lisp

@end itemize


@node What happened to the bundle operations,  , Pitfalls of the upgrade to ASDF 3, What has changed between ASDF 1 ASDF 2 and ASDF 3?
@subsection What happened to the bundle operations?

@tindex fasl-op (obsolete)
@tindex load-fasl-op (obsolete)
@tindex binary-op (obsolete)
@tindex monolithic-fasl-op (obsolete)
@tindex monolithic-load-fasl-op (obsolete)
@tindex monolithic-binary-op (obsolete)
@tindex compile-bundle-op
@tindex load-bundle-op
@tindex deliver-asd-op
@tindex monolithic-compile-bundle-op
@tindex monolithic-load-bundle-op
@tindex monolithic-deliver-asd-op
@cindex bundle operations
@tindex program-op

@code{asdf-ecl} and its short-lived successor @code{asdf-bundle} are no more,
having been replaced by code now built into ASDF 3.
Moreover, the name of the bundle operations has changed since ASDF 3.1.3.
Starting with ASDF 3.2.0, @code{load-system}
will once again use @code{load-bundle-op} instead of @code{load-op} on ECL,
as originally intended by @code{asdf-ecl} authors, but disabled for a long time
due to bugs in both ECL and ASDF.

Note that some of the bundle operations were renamed after ASDF 3.1.3,
and the old names have been removed.
Old bundle operations, and their modern equivalents are:

@itemize
@item
@code{fasl-op} is now @code{compile-bundle-op}
@item
@code{load-fasl-op} is now @code{load-bundle-op}
@item
@code{binary-op} is now @code{deliver-asd-op}
@item
@code{monolithic-fasl-op} is now @code{monolithic-compile-bundle-op}
@item
@code{monolithic-load-fasl-op} is now @code{monolithic-load-bundle-op}
@item
@code{monolithic-binary-op} is now @code{monolithic-deliver-asd-op}
@end itemize



@node Issues with installing the proper version of ASDF, Issues with configuring ASDF, What has changed between ASDF 1 ASDF 2 and ASDF 3?, FAQ
@section Issues with installing the proper version of ASDF

@menu
* My Common Lisp implementation comes with an outdated version of ASDF. What to do?::
* I'm a Common Lisp implementation vendor. When and how should I upgrade ASDF?::
* After upgrading ASDF, ASDF (and Quicklisp) can't find my systems: After upgrading ASDF.
@end menu

@node My Common Lisp implementation comes with an outdated version of ASDF. What to do?, I'm a Common Lisp implementation vendor. When and how should I upgrade ASDF?, Issues with installing the proper version of ASDF, Issues with installing the proper version of ASDF
@subsection ``My Common Lisp implementation comes with an outdated version of ASDF. What to do?''

If you have a recent implementation, it should already come with ASDF 3 or later.
If you need a more recent version than is provided,
we recommend you simply upgrade ASDF by installing a recent version
in a path configured in your source-registry.
@xref{Upgrading ASDF}.

If you have an old implementation that does not provide ASDF 3,
we recommend you replace your implementation's ASDF.
@xref{Replacing your implementation's ASDF}.


@node I'm a Common Lisp implementation vendor. When and how should I upgrade ASDF?, After upgrading ASDF, My Common Lisp implementation comes with an outdated version of ASDF. What to do?, Issues with installing the proper version of ASDF
@subsection ``I'm a Common Lisp implementation vendor. When and how should I upgrade ASDF?''

Since ASDF 2,
it should always be a good time to upgrade to a recent version of ASDF.
You may consult with the maintainer for which specific version they recommend,
but the latest @code{release} should be correct.
Though we do try to test ASDF releases against all implementations that we can,
we may not be testing against all variants of your implementation,
and we may not be running enough tests;
we trust you to thoroughly test it with your own implementation
before you release it.
If there are any issues with the current release,
it's a bug that you should report upstream and that we will fix ASAP.

As to how to include ASDF, we recommend the following:

@itemize
@item
If ASDF isn't loaded yet, then @code{(require "asdf")}
should load the version of ASDF that is bundled with your system.
If possible so should @code{(require "ASDF")}.
You may have it load some other version configured by the user,
if you allow such configuration.

@item
If your system provides a mechanism to hook into @code{cl:require},
then it would be nice to add ASDF to this hook the same way that
ABCL, CCL, CLISP, CMUCL, ECL, SBCL and SCL do it.
Please send us appropriate code to this end.

@item
You may, like SBCL since 1.1.13 or MKCL since 1.1.9,
have ASDF create bundle FASLs
that are provided as modules by your Lisp distribution.
You may also, but we don't recommend that anymore,
as in SBCL up until 1.1.12, have ASDF be implicitly used
to @code{cl:require} these modules that are provided by your Lisp distribution;
if you do, you should add these modules in the beginning of both
@code{wrapping-source-registry} and @code{wrapping-output-translations}.

@item
If you have magic systems as above, like SBCL used to do,
then we explicitly ask you to @emph{NOT} distribute
@file{asdf.asd} as part of those magic systems.
You should still include the file @file{asdf.lisp} in your source distribution
and precompile it in your binary distribution,
but @file{asdf.asd} if included at all,
should be secluded from the magic systems,
in a separate file hierarchy.
Alternatively, you may provide the system
after renaming it and its @file{.asd} file to e.g.
@code{asdf-ecl} and @file{asdf-ecl.asd}, or
@code{sb-asdf} and @file{sb-asdf.asd}.
Indeed, if you made @file{asdf.asd} a magic system,
then users would no longer be able to upgrade ASDF using ASDF itself
to some version of their preference that
they maintain independently from your Lisp distribution.

@item
If you do not have any such magic systems, or have other non-magic systems
that you want to bundle with your implementation,
then you may add them to the @code{wrapping-source-registry},
and you are welcome to include @file{asdf.asd} amongst them.
Non-magic systems should be at the back of the @code{wrapping-source-registry}
while magic systems are at the front.
If they are precompiled,
they should also be in the @code{wrapping-output-translations}.

@item
Since ASDF 3, the library UIOP comes transcluded in ASDF.
But if you want to be nice to users who care for UIOP but not for ASDF,
you may package UIOP separately,
so that one may @code{(require "uiop")} and not load ASDF,
or one may @code{(require "asdf")}
which would implicitly require and load the former.

@item
Please send us upstream any patches you make to ASDF itself,
so we can merge them back in for the benefit of your users
when they upgrade to the upstream version.

@end itemize

@node After upgrading ASDF,  , I'm a Common Lisp implementation vendor. When and how should I upgrade ASDF?, Issues with installing the proper version of ASDF
@subsection After upgrading ASDF, ASDF (and Quicklisp) can't find my systems
@vindex *central-registry*
@cindex Quicklisp

When you upgrade the ASDF running in your Lisp image
from an ancient ASDF 2 or older to ASDF 3 or newer,
then you may have to re-configure ASDF.
If your configuration only consists in
using the source-registry and output-translations (as it should),
and if you are not explicitly calling @code{asdf:initialize-source-registry}
or @code{asdf:initialize-output-translations} with a non-nil argument,
then ASDF will reconfigure itself.
Otherwise, you will have to configure ASDF 2 (or older) to find ASDF 3,
then configure ASDF 3.
Notably, @var{*central-registry*} is not maintained across upgrades from ASDF 2.
@xref{reinitializeASDFAfterUpgrade,note about ASDF reconfiguration after upgrade}.

Problems like this may be experienced if one loads Quicklisp
(which as of this writing bundles an obsolete ASDF version 2.26),
upgrades ASDF, and then tries to load new systems.
The correct solution is to load the most up-to-date ASDF you can,
@emph{then} configure it, @emph{then} load Quicklisp and any other extension.
Do @emph{not} try to upgrade from ASDF 2 @emph{after} loading Quicklisp,
for it will leave both ASDF and Quicklisp badly misconfigured.
For details see the discussion at the above cross-reference.

Also, if you are experiencing such failures due to Quicklisp shipping an ancient ASDF,
please complain to Zach Beane about it.


@node Issues with configuring ASDF, Issues with using and extending ASDF to define systems, Issues with installing the proper version of ASDF, FAQ
@section Issues with configuring ASDF

@menu
* How can I customize where fasl files are stored?::
* How can I wholly disable the compiler output cache?::
* How can I debug problems finding ASDF systems::
@end menu

@node How can I customize where fasl files are stored?, How can I wholly disable the compiler output cache?, Issues with configuring ASDF, Issues with configuring ASDF
@subsection ``How can I customize where fasl files are stored?''

@xref{Controlling where ASDF saves compiled files}.

Note that in the past there was an add-on to ASDF called
@code{ASDF-binary-locations}, developed by Gary King.
That add-on has been merged into ASDF proper,
then superseded by the @code{asdf-output-translations} facility.

Note that use of @code{asdf-output-translations}
can interfere with one aspect of your systems
--- if your system uses @code{*load-truename*} to find files
(e.g., if you have some data files stored with your program),
then the relocation that this ASDF customization performs
is likely to interfere.
Use @code{asdf:system-relative-pathname} to locate a file
in the source directory of some system, and
use @code{asdf:apply-output-translations} to locate a file
whose pathname has been translated by the facility.

@node How can I wholly disable the compiler output cache?, How can I debug problems finding ASDF systems, How can I customize where fasl files are stored?, Issues with configuring ASDF
@subsection ``How can I wholly disable the compiler output cache?''

To permanently disable the compiler output cache
for all future runs of ASDF, you can:

@example
mkdir -p ~/.config/common-lisp/asdf-output-translations.conf.d/
echo ':disable-cache' > \
~/.config/common-lisp/asdf-output-translations.conf.d/99-disable-cache.conf
@end example

This assumes that you didn't otherwise configure the ASDF files
(if you did, edit them again),
and don't somehow override the configuration at runtime
with a shell variable (see below) or some other runtime command
(e.g. some call to @code{asdf:initialize-output-translations}).

To disable the compiler output cache in Lisp processes
run by your current shell, try (assuming @code{bash} or @code{zsh})
(on Unix and cygwin only):

@example
export ASDF_OUTPUT_TRANSLATIONS=/:
@end example

To disable the compiler output cache just in the current Lisp process,
use (after loading ASDF but before using it):

@example
(asdf:disable-output-translations)
@end example

Note that this does @emph{NOT} belong in a @file{.asd} file.
Please do not tamper with ASDF configuration from a @file{.asd} file,
and only do this from your personal configuration or build scripts.

@node How can I debug problems finding ASDF systems,  , How can I wholly disable the compiler output cache?, Issues with configuring ASDF
@comment  node-name,  next,  previous,  up
@subsection How can I debug problems finding ASDF systems?

Sometimes ASDF will be unable to find and load your systems, although
you believe that it should be able to.  There are a number of things you
can do to debug such issues.

@vindex *central-registry*
If you are using @code{asdf:*central-registry*} (@pxref{Configuring ASDF to find your systems --- old style}),
you can
simply look at the pathnames and namestrings in this variable, and use
conventional tools such as @code{cl:probe-file} and @code{cl:directory}
to poke around and see why your systems are not being found.

If you are using one of the newer methods for configuring ASDF's system
finding (@pxref{Controlling where ASDF searches for systems}), you can try:

@vindex *source-registry*
@example
(alexandria:hash-table-alist asdf/source-registry::*source-registry*)
@end example

(alphabetizing the results here may be helpful). Or for a higher-level
view:

@findex flatten-source-registry
@example
(asdf/source-registry:flatten-source-registry)
@end example

Finally, if you use the source registry cache (@pxref{Caching Results}), you can:
@example
find ~/common-lisp -name .cl-source-registry.cache
@end example
at the shell.

It is still, unfortunately, an open question how to monitor ASDF's
interpretation of its source configuration as it happens.


@node Issues with using and extending ASDF to define systems, ASDF development FAQs, Issues with configuring ASDF, FAQ
@section Issues with using and extending ASDF to define systems

@menu
* How can I cater for unit-testing in my system?::
* How can I cater for documentation generation in my system?::
* How can I maintain non-Lisp (e.g. C) source files?::
* I want to put my module's files at the top level.  How do I do this?::
* How do I create a system definition where all the source files have a .cl extension?::
* How do I mark a source file to be loaded only and not compiled?::
* How do I work with readtables?::
* How can I capture ASDF's output?::
* LOAD-PATHNAME has a weird value::
* How can I produce a binary at a specific path from sources at a specific path::
@end menu

@node How can I cater for unit-testing in my system?, How can I cater for documentation generation in my system?, Issues with using and extending ASDF to define systems, Issues with using and extending ASDF to define systems
@subsection ``How can I cater for unit-testing in my system?''

ASDF provides a predefined test operation, @code{test-op}.
@xref{Predefined operations of ASDF, test-op}.
The test operation, however, is largely left to the system definer to specify.
@code{test-op} has been
a topic of considerable discussion on the
@url{http://common-lisp.net/cgi-bin/mailman/listinfo/asdf-devel,asdf-devel mailing list}
(@pxref{Mailing list}),
and on the
@url{https://launchpad.net/asdf,launchpad bug-tracker} (@pxref{Where do I report a bug?}).
We provide some guidelines in the discussion of @code{test-op}.

@c cut the following because it's discussed in the discussion of test-op.
@c Here are some guidelines:

@c @itemize
@c @item
@c For a given system, @var{foo}, you will want to define a corresponding
@c test system, such as @var{foo-test}.  The reason that you will want this
@c separate system is that ASDF does not out of the box supply components
@c that are conditionally loaded.  So if you want to have source files
@c (with the test definitions) that will not be loaded except when testing,
@c they should be put elsewhere.

@c @item
@c The @var{foo-test} system can be defined in an asd file of its own or
@c together with @var{foo}.  An aesthetic preference against cluttering up
@c the filesystem with extra asd files should be balanced against the
@c question of whether one might want to directly load @var{foo-test}.
@c Typically one would not want to do this except in early stages of
@c debugging.

@c @item
@c Record that testing is implemented by @var{foo-test}.  For example:
@c @example
@c (defsystem @var{foo}
@c    :in-order-to ((test-op (test-op @var{foo-test})))
@c    ....)

@c (defsystem @var{foo-test}
@c    :depends-on (@var{foo} @var{my-test-library} ...)
@c    ....)
@c @end example
@c @end itemize

@c This procedure will allow you to support users who do not wish to
@c install your test framework.

@c One oddity of ASDF is that @code{operate} (@pxref{Operations,operate})
@c does not return a value.  So in current versions of ASDF there is no
@c reliable programmatic means of determining whether or not a set of tests
@c has passed, or which tests have failed.  The user must simply read the
@c console output.  This limitation has been the subject of much
@c discussion.

@node How can I cater for documentation generation in my system?, How can I maintain non-Lisp (e.g. C) source files?, How can I cater for unit-testing in my system?, Issues with using and extending ASDF to define systems
@subsection ``How can I cater for documentation generation in my system?''

Various ASDF extensions provide some kind of @code{doc-op} operation.
See also @url{https://bugs.launchpad.net/asdf/+bug/479470}.


@node How can I maintain non-Lisp (e.g. C) source files?, I want to put my module's files at the top level.  How do I do this?, How can I cater for documentation generation in my system?, Issues with using and extending ASDF to define systems
@subsection ``How can I maintain non-Lisp (e.g. C) source files?''

See @code{cffi}'s @code{cffi-grovel}.

@anchor{report-bugs}


@node I want to put my module's files at the top level.  How do I do this?, How do I create a system definition where all the source files have a .cl extension?, How can I maintain non-Lisp (e.g. C) source files?, Issues with using and extending ASDF to define systems
@subsection ``I want to put my module's files at the top level.  How do I do this?''

By default, the files contained in an asdf module go
in a subdirectory with the same name as the module.
However, this can be overridden by adding a @code{:pathname ""} argument
to the module description.
For example, here is how it could be done
in the spatial-trees ASDF system definition for ASDF 2 or later:

@example
(asdf:defsystem "spatial-trees"
  :components
  ((:module "base"
            :pathname ""
            :components
            ((:file "package")
             (:file "basedefs" :depends-on ("package"))
             (:file "rectangles" :depends-on ("package"))))
   (:module tree-impls
            :depends-on ("base")
            :pathname ""
            :components
            ((:file "r-trees")
             (:file "greene-trees" :depends-on ("r-trees"))
             (:file "rstar-trees" :depends-on ("r-trees"))
             (:file "rplus-trees" :depends-on ("r-trees"))
             (:file "x-trees" :depends-on ("r-trees" "rstar-trees"))))
   (:module viz
            :depends-on ("base")
            :pathname ""
            :components
            ((:static-file "spatial-tree-viz.lisp")))
   (:module tests
            :depends-on ("base")
            :pathname ""
            :components
            ((:static-file "spatial-tree-test.lisp")))
   (:static-file "LICENCE")
   (:static-file "TODO")))
@end example

All of the files in the @code{tree-impls} module are at the top level,
instead of in a @file{tree-impls/} subdirectory.

Note that the argument to @code{:pathname} can be either a pathname object or a string.
A pathname object can be constructed with the @file{#p"foo/bar/"} syntax,
but this is discouraged because the results of parsing a namestring are not portable.
A pathname can only be portably constructed with such syntax as
@code{#.(make-pathname :directory '(:relative "foo" "bar"))},
and similarly the current directory can only be portably specified as
@code{#.(make-pathname :directory '(:relative))}.
However, as of ASDF 2, you can portably use a string to denote a pathname.
The string will be parsed as a @code{/}-separated path from the current directory,
such that the empty string @code{""} denotes the current directory, and
@code{"foo/bar"} (no trailing @code{/} required in the case of modules)
portably denotes the same subdirectory as above.
When files are specified, the last @code{/}-separated component is interpreted
either as the name component of a pathname
(if the component class specifies a pathname type),
or as a name component plus optional dot-separated type component
(if the component class doesn't specifies a pathname type).

@node How do I create a system definition where all the source files have a .cl extension?, How do I mark a source file to be loaded only and not compiled?, I want to put my module's files at the top level.  How do I do this?, Issues with using and extending ASDF to define systems
@subsection How do I create a system definition where all the source files have a .cl extension?

Starting with ASDF 2.014.14, you may just pass
the builtin class @code{cl-source-file.cl} as
the @code{:default-component-class} argument to @code{defsystem}:

@lisp
(defsystem my-cl-system
  :default-component-class cl-source-file.cl
  ...)
@end lisp

Another builtin class @code{cl-source-file.lsp} is offered
for files ending in @file{.lsp}.

If you want to use a different extension
for which ASDF doesn't provide builtin support,
or want to support versions of ASDF
earlier than 2.014.14 (but later than 2.000),
you can define a class as follows:

@lisp
;; Prologue: make sure we're using a sane package.
(defpackage :my-asdf-extension
   (:use :asdf :common-lisp)
   (:export #:cl-source-file.lis))
(in-package :my-asdf-extension)

(defclass cl-source-file.lis (cl-source-file)
  ((type :initform "lis")))
@end lisp

Then you can use it as follows:
@lisp
(defsystem my-cl-system
  :default-component-class my-asdf-extension:cl-source-file.lis
  ...)
@end lisp

Of course, if you're in the same package, e.g. in the same file,
you won't need to use the package qualifier before @code{cl-source-file.lis}.
Actually, if all you're doing is defining this class
and using it in the same file without other fancy definitions,
you might skip package complications:

@lisp
(in-package :asdf)
(defclass cl-source-file.lis (cl-source-file)
   ((type :initform "lis")))
(defsystem my-cl-system
  :default-component-class cl-source-file.lis
  ...)
@end lisp

@node How do I mark a source file to be loaded only and not compiled?, How do I work with readtables?, How do I create a system definition where all the source files have a .cl extension?, Issues with using and extending ASDF to define systems
@subsection How do I mark a source file to be loaded only and not compiled?

There is no provision in ASDF for ensuring that
some components are always loaded as source, while others are always
compiled.
There is @code{load-source-op} (@pxref{Predefined operations of
ASDF,load-source-op}), but that is an operation to be applied to a
system as a whole, not to one or another specific source files.
While this idea often comes up in discussions,
it doesn't play well with either the linking model of ECL
or with various bundle operations.
In addition, the dependency model of ASDF would have to be modified incompatibly
to allow for such a trick.
@c If your code doesn't compile cleanly, fix it.
@c If compilation makes it slow, use @code{declaim} or @code{eval-when}
@c to adjust your compiler settings,
@c or eschew compilation by @code{eval}uating a quoted source form at load-time.

@node How do I work with readtables?, How can I capture ASDF's output?, How do I mark a source file to be loaded only and not compiled?, Issues with using and extending ASDF to define systems
@subsection How do I work with readtables?

@cindex readtables

It is possible to configure the lisp syntax by modifying the currently-active readtable.
However, this same readtable is shared globally by all software being compiled by ASDF,
especially since @code{load} and @code{compile-file} both bind @var{*readtable*},
so that its value is the same across the build at the start of every file
(unless overridden by some @code{perform :around} method),
even if a file locally binds it to a different readtable during the build.

Therefore, the following hygiene restrictions apply. If you don't abide by these restrictions,
there will be situations where your output files will be corrupted during an incremental build.
We are not trying to prescribe new restrictions for the sake of good style:
these restrictions have always applied implicitly, and
we are simply describing what they have always been.

@itemize
@item It is forbidden to modifying any standard character or standard macro dispatch defined in the CLHS.
@item No two dependencies may assign different meanings to the same non-standard character.
@item Using any non-standard character while expecting the implementation to treat some way
    counts as such an assignment of meaning.
@item libraries need to document these assignments of meaning to non-standard characters.
@item free software libraries will register these changes on:
        @url{http://www.cliki.net/Macro%20Characters}
@end itemize

If you want to use readtable modifications that cannot abide by those restrictions,
you @emph{must} create a different readtable object and set @var{*readtable*}
to temporarily bind it to your new readtable (which will be undone after processing the file).

For that, we recommend you use system @code{named-readtables}
to define or combine such readtables using @code{named-readtables:defreadtable}
and use them using @code{named-readtables:in-readtable}.
Equivalently, you can use system @code{cl-syntax},
that itself uses @code{named-readtables},
but may someday do more with, e.g. @var{*print-pprint-dispatch*}.

For even more advanced syntax modification beyond what a readtable can express,
you may consider either:
@itemize
@item a @code{perform} method that compiles a constant file that contains a single form
  @code{#.*code-read-with-alternate-reader*} in an environment where this special variable
  was bound to the code read by your alternate reader, or
@item using the system @code{reader-interception}.
@end itemize

Beware that @c unless and until the @code{syntax-control} branch is merged,
it is unsafe to use ASDF from the REPL to compile or load systems
while the readtable isn't the shared readtable previously used to build software.
You @emph{must} manually undo any binding of @var{*readtable*} at the REPL
and restore its initial value whenever you call @code{operate}
(via e.g. @code{load-system}, @code{test-system} or @code{require})
from a REPL that is using a different readtable.

@subsubsection How should my system use a readtable exported by another system?

Use from the @code{named-readtables} system the macro @code{named-readtables:in-readtable}.

If the other system fails to use @code{named-readtables}, fix it and send a patch upstream.
In the day and age of Quicklisp and clbuild, there is little reason
to eschew using such an important library anymore.

@subsubsection How should my library make a readtable available to other systems?

Use from the @code{named-readtables} system the macro @code{named-readtables:defreadtable}.

@node How can I capture ASDF's output?, LOAD-PATHNAME has a weird value, How do I work with readtables?, Issues with using and extending ASDF to define systems
@subsection How can I capture ASDF's output?

@cindex ASDF output
@cindex Capturing ASDF output
@vindex *standard-output*

Output from ASDF and ASDF extensions are sent to the CL stream
@code{*standard-output*}, so rebinding that stream around calls to
@code{asdf:operate} should redirect all output from ASDF operations.

@node LOAD-PATHNAME has a weird value, How can I produce a binary at a specific path from sources at a specific path, How can I capture ASDF's output?, Issues with using and extending ASDF to define systems
@subsection *LOAD-PATHNAME* and *LOAD-TRUENAME* have weird values, help!
@vindex *LOAD-PATHNAME*
@vindex *LOAD-TRUENAME*

Conventional Common Lisp code may use @code{*LOAD-TRUENAME*} or @code{*LOAD-PATHNAME*} to find
files adjacent to source files.  This will generally @emph{not} work in
ASDF-loaded systems.  Recall that ASDF relocates the FASL files it
builds, typically to a special cache directory.  Thus the value of
@code{*LOAD-PATHNAME*} and @code{*LOAD-TRUENAME*}  at load time, when ASDF is loading your system,
will typically be a pathname in that cache directory, and useless to you
for finding other system components.

There are two ways to work around this problem:
@enumerate
@findex system-relative-pathname
@item
Use the @code{system-relative-pathname} function.  This can readily be
used from outside the system, but it is probably not good software
engineering to require a source file @emph{of} a system to know what
system it is going to be part of.  Contained objects should not have to
know their containers.
@item
Store the pathname at compile time, so that you get the pathname of the
source file, which is presumably what you want.  To do this, you can
capture the value of @code{(or *compile-file-pathname* *load-truename*)}
(or @code{*LOAD-PATHNAME*}, if you prefer)
in a macro expansion or other compile-time evaluated context.

@end enumerate

@node How can I produce a binary at a specific path from sources at a specific path,  , LOAD-PATHNAME has a weird value, Issues with using and extending ASDF to define systems
@subsection How can I produce a binary at a specific path from sources at a specific path?
@cindex bundle operations
@tindex program-op

Especially when integrating with outside build systems, it's useful to have
fine-grained control over where ASDF puts output files. The following is an
example of a build script that takes the directory to find the source and the
path to put the resulting binary at, and compiles a system there.

@example
(in-package :cl-user)

;; Tell ASDF where to find your source files. This may not
;; be necessary if you've already configured this elsewhere.
(asdf:initialize-source-registry
  `(:source-registry
     :inherit-configuration
     (:directory ,(uiop:getenv-absolute-directory "SRC"))))

;; Set the output pathname when building the system.
(defmethod asdf:output-files ((o asdf:image-op)
                              (system (eql (asdf:find-system @var{foo}))))
  (declare (ignorable system))
  (values (list (uiop:getenv-pathname "OUT")) t))

;; Build the system.
(asdf:operate-on-system 'asdf:program-op @var{foo})
(uiop:quit)
@end example

@node ASDF development FAQs,  , Issues with using and extending ASDF to define systems, FAQ
@section ASDF development FAQs

@menu
* How do I run the tests interactively in a REPL?::
@end menu

@node How do I run the tests interactively in a REPL?,  , ASDF development FAQs, ASDF development FAQs
@subsection How do I run the tests interactively in a REPL?

This not-so-frequently asked question is primarily for ASDF developers,
but those who encounter an unexpected error in some test may be
interested, too.

Here's the procedure for experimenting with tests in a REPL:
@example
;; BEWARE! Some tests expect you to be in the .../asdf/test directory
;; If your REPL is not there yet, change your current directory:
;; under SLIME, you may: ,change-directory ~/common-lisp/asdf/test/
;; otherwise you may evaluate something like:
(require "asdf") (asdf:upgrade-asdf) ;load UIOP & update asdf.lisp
(uiop:chdir (asdf:system-relative-pathname :asdf "test/"))
(setf *default-pathname-defaults* (uiop:getcwd))

;; Load the test script support.
(load "script-support.lisp")

;; Initialize the script support for interaction.
;; This will also change your *package* to asdf-test
;; after frobbing the asdf-test package to make it usable.
;; NB: this function is also available from package cl-user,
;; and also available with the shorter name da in both packages.
(asdf-test:debug-asdf)

;; Now, you may experiment with test code from a .script file.
;; See the instructions given at the end of your failing test
;; to identify which form is needed, e.g.
(run-test-script "test-utilities.script")
@end example


@comment FIXME: Add a FAQ about how to use a new system class...

@comment  node-name,  next,  previous,  up
@node  Ongoing Work, Bibliography, FAQ, Top
@unnumbered Ongoing Work
For an active list of things to be done,
see the @file{TODO} file in the source repository.

Also, bugs are currently tracked on launchpad:
@url{https://launchpad.net/asdf}.

@node Bibliography, Concept Index, Ongoing Work, Top
@unnumbered Bibliography

@itemize
@item Andrey Mokhov, Neil Mitchell and Simon Peyton Jones:
  ``Build Systems à la Carte'',
  International Conference on Functional Programming, 2018.
  @url{https://www.microsoft.com/en-us/research/uploads/prod/2018/03/build-systems-final.pdf}
  This influential article provides axes along which to describe build systems in general;
  ASDF, in addition to being in-image (an axis not considered by these authors),
  has the following characteristics:
  ASDF's persistent build information is file modification times
  (the way ASDF is written, it should be easy enough to write an extension that modifies it
  to use a ``cloud cache'' à la Bazel,
  but that would involve using some database, network and cryptographic libraries,
  which cannot reasonably be included in the base ASDF,
  that must remain a minimal bootstrappable system with no external dependencies).
  The object model of ASDF was initially designed for ``static'' dependencies
  with a ``topological'' scheduler, but its @code{defsystem-depends-on} mechanism
  (and more generally, the ability to call ASDF from within an @code{.asd} file)
  allows for multiple @emph{phases} of execution resulting
  in ``dynamic'' dependencies with a ``suspending'' scheduler.
  The rebuilder essentially uses a ``dirty bit'', except that the in-image model
  and the multiple phase support mean that's actually more than a bit:
  instead it's three bits plus the timestamp plus a phase depth level.
  The build is guaranteed ``minimal'' in number of steps computed.
  It is local. It assumes but does not enforce determinism.
  It does not assume early cutoff of the build when rebuild dependencies didn't change.
@item Robert Goldman, Elias Pipping, and François-René Rideau:
  ``Delivering Common Lisp Applications with ASDF 3.3'',
  European Lisp Symposium, 2017.
  @url{https://github.com/fare/asdf2017}
  This short article gives an overview of the changes in ASDF 3.2 and 3.3, including
  improved application delivery, asynchronous subprocess management,
  correct support for multi-phase builds, and enhanced source location configuration.
@item Francois-Rene Rideau:
  ``ASDF 3, or Why Lisp is Now an Acceptable Scripting Language'',
  European Lisp Symposium, 2014.
  @url{https://github.com/fare/asdf3-2013}
  This article describes the innovations in ASDF 3 and 3.1,
  as well as historical information on previous versions.
@item Alastair Bridgewater:
  ``Quick-build'' (private communication), 2012.
  @code{quick-build} is a simple and robust one file, one package build system,
  similar to @code{faslpath}, in 182 lines of code
  (117 of which are neither blank nor comments nor docstrings).
  Unhappily, it remains unpublished and its IP status is unclear as of April 2014.
  @code{asdf/package-system} is mostly compatible with it,
  modulo a different setup for toplevel hierarchies.
@item Zach Beane:
  ``Quicklisp'', 2011.
  The Quicklisp blog and Xach's personal blogs contain information on Quicklisp.
  @url{http://blog.quicklisp.org/}
  @url{http://lispblog.xach.com/} (new)
  @url{http://xach.livejournal.com/} (old)
@item Francois-Rene Rideau and Robert Goldman:
  ``Evolving ASDF: More Cooperation, Less Coordination'',
  International Lisp Conference, 2010.
  This article describes the main issues solved by ASDF 2,
  and exposes its design principles.
  @url{https://common-lisp.net/project/asdf/ilc2010draft.pdf}
  @url{http://rpgoldman.goldman-tribe.org/papers/ilc2010-asdf.pdf}
@item Francois-Rene Rideau and Spencer Brody:
  ``XCVB: an eXtensible Component Verifier and Builder for Common Lisp'',
  International Lisp Conference, 2009.
  This article describes XCVB, a proposed competitor for ASDF;
  many of its ideas have been incorporated into ASDF 2 and 3,
  though many other ideas still haven't.
  @url{https://common-lisp.net/project/xcvb/}
@item Peter von Etter:
  ``faslpath'', 2009.
  @code{faslpath} is similar to the latter @code{quick-build}
  and our yet latter @code{asdf/package-system} extension,
  except that it uses dot @code{.} rather than slash @code{/} as a separator.
  @url{https://code.google.com/p/faslpath/}
@item Drew McDermott:
  ``A Framework for Maintaining the Coherence of a Running Lisp,''
  International Lisp Conference, 2005.
  @url{http://www.cs.yale.edu/homes/dvm/papers/lisp05.pdf}
@item Dan Barlow: ``ASDF Manual'', 2004.
  Older versions of this document from the days of ASDF 1;
  they include ideas laid down by Dan Barlow,
  and comparisons with older defsystems (@code{mk-defsystem})
  and defsystem (@code{defsystem-4}, kmp's Memo 801).
@item Marco Antoniotti and Peter Van Eynde:
 ``@code{DEFSYSTEM}: A @code{make} for Common Lisp, A Thoughtful Re-Implementation of an Old Idea'', 2002.
  The @file{defsystem-4} proposal available in the CLOCC repository.
@item Mark Kantrovitz:
  ``Defsystem: A Portable Make Facility for Common Lisp'', 1990.
  The classic @file{mk-defsystem}, later variants of which
  are available in the CLOCC repository as @code{defsystem-3.x}.
@item Richard Elliot Robbins:
  ``BUILD: A Tool for Maintaining Consistency in Modular Systems'', MIT AI TR 874, 1985.
  @url{http://www.dtic.mil/dtic/tr/fulltext/u2/a162744.pdf}
@item Kent M. Pitman (kmp): ``The Description of Large Systems'', MIT AI Memo 801, 1984.
  Available in updated-for-CL form on the web at
  @url{http://nhplace.com/kent/Papers/Large-Systems.html}
@item Dan Weinreb and David Moon:
  ``Lisp Machine Manual'', 3rd Edition MIT, March 1981.
  The famous CHINE NUAL describes one of the earliest variants of DEFSYSTEM.
  (NB: Not present in the second preliminary version of January 1979)
  @url{http://bitsavers.org/pdf/mit/cadr/chinual_3rdEd_Mar81.pdf}
@end itemize


@node Concept Index, Function and Macro Index, Bibliography, Top
@unnumbered Concept Index

@printindex cp

@node Function and Macro Index, Variable Index, Concept Index, Top
@unnumbered Function and Macro Index

@printindex fn

@node Variable Index, Class and Type Index, Function and Macro Index, Top
@unnumbered Variable Index

@printindex vr

@node  Class and Type Index,  , Variable Index, Top
@unnumbered Class and Type Index

@printindex tp

@bye

@c  LocalWords:  clbuild tarballs defsystem Quicklisp initarg uiop fasl
@c  LocalWords:  namestring initargs fasls