autodepsYabs is a build system that takes the form of a Python library.
Build systems that use Yabs are specified as a Python programme that makes calls to Yabs functions to register rules for object files, executables and other build targets. A primary aim of Yabs is to provide high-level functionality so that these programmes are shorter and easier to write than makefiles or similar.
Yabs consists of the module yabs which contains
the core functionality, the module yabs2 containing a
command-line interface and support for Make-style pattern rules,
and the module yabs3 which implements a particular way
of specifying and building executables and libraries.
The primary documentation for Yabs is the documentation strings in the above Python files. This readme is only intended to give an overview of Yabs.
The core of Yabs, defined in the yabs module, is a
dependency tree, allowing rules to be defined that make target files
from prerequisite files. These rules are expressed as Python functions
which take a target filename and, if the rule can generate this target,
return a Python tuple containing the command(s) that should be run (or
a python function to be called) to create the target and a list of the
target's prerequisites required by these commands. This design means
that the full power of the Python language is available to the user when
writing rules.
For each target that it is asked to build, Yabs calls each rule-function that it knows about, and if the rule-function succeeds and returns such a tuple, it calls itself recursively for each prerequisite. Finally, if the target doesn't already exist, or any of these prerequisites are newer than the target, it calls the command(s) returned in the tuple.
A simple rule could look like this:
def myrule( target, state):
if target!='foo': return
return 'echo hello world > ' + target,
yabs.add_rule( myrule)
This basic functionality can be used by wrapper libraries to support
making various builds of object files and executables; see yabs3
for an example of this.
Yabs supports automatic remaking of targets if the command that generates
the target is changed, even if the existing target file is newer than all of
the prerequisites. See yabs.add_rule()'s autocmds
parameter.
On Unix platforms, Yabs can automatically detect hidden dependencies
- it detects which files are opened by commands while the commands are
running, and uses this
information in subsequent builds to force rebuilds of targets when any
of these files have been modified. For example, this can provide automatic
recompilation of C source files when headers that they directly or
indirectly include, are modified. See yabs.add_rule()'s autodeps parameter.
A Yabs programme can be run on its own or imported, without modification, by a different Yabs programme. In the latter case, the rules in the imported programme will become part of the master programme's rules and will work correctly even if the different Yabs programmes are in different directories.
Yabs requires python-2.2 or later.
Here's a Yabs script that uses yabs3 to make a rule for building
an executable from main.c and other.c:
import yabs.yabs2, yabs.yabs3
yabs3.add_exe( 'myprog', 'main.c other.c')
yabs2.appmakeexit()
If this script is called make.py, then one can build
the executable with:
./make.py myprog.exe
There are few useful things that yabs3 supports:
./make.py other.c.o
./make.py other.c.c
./make.py other.c.s
All the above commands will default to using debug flags. One can
change this with the -b flag: -b gcc,release will
default to release flags.
Alternatively, one can specify the build-type as part of the target:
./make.py myprog,release.exe
./make.py other.c,release.o
./make.py myprog,release,-DFOO=42.exe
If you are reading this on the Yabs website, then this release is available as the file: yabs-24.tar.gz.
Version: 24.
Date: 2012 December 01.
Yabs is Copyright © 2004-2010 Julian Smith.
License: Yabs is released under the GNU Public License. Please see http://www.gnu.org/copyleft/gpl.html, or write to the Free Software Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
Contact: Julian Smith: jules@op59.net
The main files in this release are:
pydoc:
(For more details, see yabs.html and yabs.py)
The yabs module has a global object,
yabs.default_state, which contains a list of all registered
rule-functions, plus caches for information such as file modification times.
Most yabs functions require this object to be passed as a
parameter, though the main functions default to
yabs.default_state if it is not specified.
Each rule-function has to be registered by passing it to
yabs.add_rule().
The function yabs.make() takes a single target and calls each
of the available rule-functions until it finds one that claims to be able to
make the target. It then calls itself recursively for each of the
prerequisites that this rule-function specified. Finally, if any of these
prerequisites are now newer than the target, it runs the command(s) return by
the rule-function.
If one of the prerequisites cannot be made, yabs.make()
carries on calling the available rule-functions until one succeeds. This
means that a failed rule may have caused some of its prerequisites to have
been re-made successfully. This behaviour is different from traditional Make,
which generates a complete tree of rules before running them. The advantage
of Yab's system is that it enables rules to use files that are not explicitly
represented in the dependency tree, for example one rule could extract an
initially unknown number of files from a tar archives and a later rule can
use these files.
There is a special last-resort rule, yabs._file_rule()
that is always registered, which simply looks for the existence of the
target file on disc. This rule is only attempted if no other rules have
failed to create the target due to failed prerequisites. This avoids
problems when an old version of a target exists, but the standard rules
for creating the target fail, e.g. because a prerequisite is incorrectly
specified.
The yabs module uses several caches to avoid repeating work.
These caches are python dictionaries; they exist for the lifetime of the
python programme that calls Yabs.
There is a cache of file modification dates, so that finding this information repeatedly for the same file doesn't involve calling the filesystem each time.
Whenever yabs investigates how to remake a particular target,
the results of this search are also cached.
Whenever yabs actually remakes a target, it remembers whether
the target was updated by the rule. This information can then be used if the
same target is required later on - e.g. if a target is a prerequisite of more
than one other target.
Rules can be specified as phony. A phony rule isn't required to
remake or even create its target, and any file of the same name is ignored..
Phony rules can be useful to give a simple target-name to a real target, such
as test. For non-phony rules, Yabs will give an error if a
target file doesn't exist after the command has returned.
Rules return a command to run, optionally followed by a list of prerequisites, optionally followed by a list of semi-prerequisites. Other information can also be returned - see the doc strings in the Yabs source for details.
Semi-prerequisites are useful to model things like header files. Yabs attempts to make semi-prerequisites in the same way that is attempts to make prerequisites, but a failure to make a semi-prerequisite is not regarded as an error - it just causes the target to be re-made.
The lists of prerequisites and semi-prerequisites can instead each be a function that returns a list, which can save calculating the list in some cases where it is not needed.
Similarly, the returned command can be either a string or a function that
is called to remake the target. In the latter case, the function can also
return a string that is run after the function returns. However the string is
generated, each line in the string is taken as a separate command and any
line that starts with a `-' results in any error from that line
being ignored (this is similar to same scheme used by GNU Make (see References).
By default, all targets passed to rules are absolute pathnames, and the prerequisites and semi-prerequisites returned from rules are required to be absolute pathnames also, so that they work regardless of the current directory in which the Yabs programme is run.
However, a root directory can be specified when a rule-function is
registered with yabs.add_rule(). When a target is within such a
rule-function's root directory, only the relative path is passed to the
rule-function, and the rule-function can return relative prerequisite and
semi-prerequisite filenames - they are immediately converted into absolute
filenames using the root directory. The rule's command(s) are run with the
working directory set to the specified root by prefixing
each command with cd <root> && (this works on
both Windows and Unix).
Some high-level Yabs rules (e.g. yabs3.add_exe() and
yabs2.add_patternrule()) automatically look at their caller's
module's containing directory and use this as a root directory. This allows
high-level rules, targets and prerequisites to be specified as relative to
their caller's module's directory (which is useful for things like
executables and phony targets), while still allowing a master Yabs programme
to import the modules that specify these targets and use their
rules without having to worry about absolute/relative pathname issues.
An example of this can be seen in the make.py Yabs programme, which imports the example-1/make.py Yabs programme, and can be told to make the executable specified in the imported file with (for example):
./make.py example-1/foo.exeOne can use the -W flag to mark certain files as being new,
and this will effect how targets in example-1 are built:
./make.py example-1/foo.exe -W example-1/main.cpp- will do the same thing as:
cd example-1
./make.py foo.exe -W main.cppYabs contains support for forcing targets to be re-made when the commands for these targets are changed (because the build script was changed), even if the existing target is newer than all of its prerequisites.
To use this support, specify an autocmds
parameter when calling yabs.add_rule() or
yabs2.add_patternrule(). The commands used to
create target will then be written to the file
target<autocmds>, and Yabs will ensure that this
information is used to rebuild targets when the rule is changed.
Under Unix, Yabs can detect all the files are used by the commands that rebuild a target, and force a rebuild of the target if any of these files are changed. This is done by specifying the autodeps parameter to yabs.add_rule(). The extra dependencies are saved to a filename formed by appending the autodeps parameter to the target filename, and the contents are used as semi-prerequisites in subsequent builds.
For example, header files opened by a compiler will be picked up by the autodeps system, removing the need to use things like gcc -MM. Any similar use of files by other tools will also be picked up, without Yabs having to know anything about these tools.
Commands from an autodeps rule are run with $LD_PRELOAD set to use a special shared-library in the Yabs directory. This shared-library is added to list of prerequisites returned by autodeps rules, and is built from the file autodeps.c by the yabs._autodeps_rule().
At the time of writing, this system is only tested on OpenBSD and Linux. Other Unix systems will probably work fine with some tweaking, but I don't know how to do a Win32 implementation.
Failed attempts to open files are also logged. This enables Yabs to detect a need to rebuild files in certain subtle situations - see the docs for yabs.add_rule(). This works on OpenBSD - see test14. Unfortunately, on Linux the $LD_PRELOAD technique with gcc doesn't seem to log failed file opens, possibly because of glib making direct system calls.
Yabs can run multiple commands at the same time The simple way
of doing this is to use -j <N>, which will run up
to N commands concurrently. Yabs takes care to only run
independent rules concurrently. For example, this can be used to take
advantage of multiple CPU cores.
A more advanced form of concurrency is available: rules
can return a resource to Yabs; the resource must
provide tryacquire() and release() methods. Yabs
only runs the rule's command after tryacquire() returns
True. If multiple rules return the same resource, the resource
can restrict how many rule commands are run. For example, this can be
useful when writing test frameworks that run test on multiple machines -
it is easy to write a resource that allows only one test to run on each
test machine.
(For more details, see yabs2.html and yabs2.py)
The yabs2 module contains various utility functions for generating string representations of the build environment, manipulating filenames etc.
The function yabs2.add_patternrule() allows rules to be
specified in a similar way to GNU make rules.
There is also the function yabs2.appmakeexit(), which can
be used as the final call in a Yabs programme. It parses the command line
parameters for various flags and targets (using similar options to GNU make),
and then attempts to remake each target. Finally it exits with an appropriate
error code. The parameters are modelled on GNU Make's interface:
Yabs options are:
-b <build>
Sets default build string.
-B Prints default build strings.
--changed <file>
Assume <file> has been changed by a yabs rule.
--cwd Print filenames relative to current directory.
-d Increment debugging level.
--d<level>
Set debug level.
-e <TRCOE>
Detailed control of diagnostics when -s and -S are not enough.
The five sub-params are:
T: target
R: rule
C: command
O: output
E: exception
Each of the sub-params should be 0, 1, 2 or 3:
0: never show
1: show with: --summary auto
2: show if rule fails
3: show when running rule.
Thus `-e 31222' will always display targets when they are made;
if an error occurs, the command, output and any exception will be
shown. If `--summary auto' is specified, the rule is displayed.
Using E=3 is not useful because exceptions are errors.
--echo-prefix <prefix>
Sets prefix used for all command output.
-h Show this help.
--help Show this help.
-j <mt> Control concurrency. If <mt> is 0, no concurrency.
Otherwise build targets as allowed by the resources returned
by rules. <mt> also governs the maximum concurrency of the
default resource.
-k Keep going as much as possible after errors.
-K Equivalent to: -k -e 30222. Useful when one wants to carry on
after errors, but see information only about errors.
-l <load-average>
Only used with -j. Only start making new targets if the system's
load average is less than <load-average> (interpreted as a
floating-point number).
-n Don't run commands.
--new <file>
Assume <file> is infinitely new. Same as -W.
-o <file>
Assume <file> is infinitely old.
-O <file>
Assume <file> doesn't exist.
--oldprefix <prefix>
Assume that all filesnames starting with <prefix> are infinitely
old.
--periodic-debug <secs>
Output brief status information every <secs> seconds.
--prefix <text>
Sets prefix used for all yabs diagnostics.
--prefix-id
Sets prefix to <user>@<hostname>:
--psyco
Attempt to use psyco.full() to optimise Yabs execution.
--pty Use pty.fork and os.exec to run commands. This appears to
work fine on linux, working with commands that read from
stdin, while still allowing control of echoing and capture
of the output.
-s Quiet operation: don't show commands unless they fail. Equivalent
to `-e 30233'.
-S Very quiet operation: don't show commands or output unless the
command fails. Equivalent to `-e 30222'.
--setbuf <stdout> <stderr>
Sets buffering to use for stdout and stderr. -ve sets to
system default, 0 is no buffering, 1 is line buffering,
other values are buffer size.
--statefile <file>
Writes current blocking operation to <file>.
--summary <format>
--summaryf <format> [+]<filename>
Adds a summary of failures. <format> is comma-separated list
of items:
'rule': show the rule that failed.
'command': show command that failed.
'output': show output from rule that failed.
'except': show exception from failed rule.
'': ignored
'.': ignored
'auto':
show just the information that was not output while
running the commands - (e.g. if -s is specified, the
summary shows the output from commands). This is the
default if --summary is not specified.
--summaryf writes the summary information to the specified
file. If <filename> is prefixed with a `+', the data is
appended to the file.
--system
Use os.system to run commands. This works better than
the default when commands use stdin, but doesn't support
non-echoing or capture of the output.
--targets
List info about available targets. Usually this is a list of
regexes for targets that are defined in terms of a regex.
--test-flock
Runs a test that Yabs' flock abstraction works.
--unchanged <file>
Assume <file> has been left unchanged by a yabs rule.
-v Print version.
-W <file>
Assume <file> is infinitely new. Same as --new.
--xtermtitle
Writes <user>@<host>:<current target> into xterm titlebar.
Multiple single-hypen options can be grouped together, e.g. `-dds' is
equivalent to `-d -d -s'.
Option values can also be specified with '=', e.g. '-j=7' is equivalent to
'-j 7'.
Anything that doesn't start with `-' is taken as a target.
One useful feature is the -s flag. This turns off display of the command that is being run, but the command is displayed if it fails. Thus you can get very clean output for the common case where things work, but still get detailed information about any build failures.
The -S flag extends this to also hide the output from commands, storing the output internally and displaying it only if the command fails.
yabs2.appmakeexit() also sets things up so that Yabs
will output its current state (the list of prerequisite targets that are
currently being considered), if it receives a SIGHUP.
(For more details, see yabs3.html and yabs3.py)
The yabs3 module supports a particular way of building
projects, in which all generated files such as object file are placed near to
the source files from which they were compiled, and their filenames encode
the parameters and the environment that were used to create them.
It should be understood that yabs3 is not the only way of
supporting high-level build targets. Yabs is flexible enough to support
many different approaches.
Currently, yabs3 supports building with gcc and, less well
tested, Microsoft VC++ under Cygwin.
A Yabs programme that uses yabs3 can specify executables in a
very simple way, while allowing different builds of these executables to be
built from the same source code without a separate configuration command. For
example, an executable named foo, built from source files
main.cpp and bar.c could be specified with a
programme build.py:
#!/usr/bin/env python import yabs2, yabs3 yabs3.add_exe( 'foo', 'main.cpp somedir/bar.c ../shared/foo.c') yabs2.appmakeexit()
All object files and executables generated by yabs3 are
placed in a _yabs sub-directory.
Building the executable can be done with the following command (assuming
build.py is in the current directory and is executable):
./build.py foo.exeThis will create a default-build of the executable. The default build
defaults to ,gcc,debug, so the above command will create an
executable called something like:
_yabs/foo,debug,gcc2.95.3,os=OpenBSD,cpu=i386,osv=3.3.exeThe default build can be modified using the -b flag or,
perhaps better, by inserting flags into the target name:
./build.py foo.exe -b gcc,release,threads
./build.py foo,release,threads.exe- will create an executable using gcc with optimisation turned on and with support for threading, called something like:
_yabs/foo,gcc2.95.3,release,threads,os=OpenBSD,cpu=i386,osv=3.3.exeThe actual parameters used when calling gcc/g++ are specified in the
yabs3.py file itself - the function compile_rule()
handles compilation/preprocessing of C and C++ files, while the nested
function add_exe.add_exe_rule() handles linking. They both look
at the flags implied in the filename that they have been asked to build, and
modify the command they run accordingly. The extra OS information embedded in
the filename is also used to tune the parameters - for example OpenBSD's gcc
uses -pthread to turn on support for threading, but other
operating systems' gcc compilers use -D_REENTRANT.
yabs3 contains a particular choice of build parameters (for
example, gcc,release is translated to gcc -O2). It
is intended that a customised version of yabs3 should be used
for a particular site.
All files generated by yabs3, such as object files, are
placed in a _yabs/ subdirectory. For object files, this
subdirectory is next to the relevant source file. For executables, the
subdirectory is in the directory that contains the Yabs programme. In the
above example the object files will be called:
./_yabs/main.cpp,debug,gcc2.95,debug,os=OpenBSD,cpu=i386,osv=3.3.o
./somedir/_yabs/bar.c,gcc2.95,debug,os=OpenBSD,cpu=i386,osv=3.3.o
../shared/_yabs/foo.c,gcc2.95,debug,os=OpenBSD,cpu=i386,osv=3.3.oPutting generated files near to their source files in the
_yabs/ subdirectory has the advantage that different Yabs
programmes can refer to the same source files, and share the resulting
generated files, as long as they import the same yabs3 module so
that they all run the same build commands.
It would be straightforward to write an alternative to yabs3
that put generated file in a top-level build directory.
One can get a preprocessed version of a source file by appending the build and then repeating the suffix:
./build.py main.cpp,debug,gcc.cppThe preprocessing rule uses some post-processing calls to sed
to work around problems with various versions of gcc, and ensure that the
resulting file could be compiled without errors.
The handling of default builds, described earlier for building
executables, is done by yabs3 adding a generic phony rule that
takes targets and makes a prerequisite by inserting the default build string
plus a representation of the environment into the target.
As well as building executables, this phony rule can also be used to easily preprocess and compile source files, using the default build.
To preprocess a file, simply repeat the suffix. For example, to preprocess
foo.cpp, do:
./make.py foo.cpp.cppThis will generate a preprocessed file called something like:
_yabs/foo.cpp,debug,gcc3.3.1,os=CYGWIN_NT-5.1,cpu=i686,osv=1.5.9{0.112-4-2}.cpp
Similarly, appending .o to a filename will compile it:
./make.py foo.cpp.o- will generate something like
_yabs/foo.cpp,debug,gcc3.3.1,os=CYGWIN_NT-5.1,cpu=i686,osv=1.5.9{0.112-4-2}.o.
One can preprocess and compile a file by concatenating the suffixes:
./make.py foo.cpp.cpp.o- will generate something like
_yabs/_yabs/foo.cpp,debug,gcc3.3.1,os=CYGWIN_NT-5.1,cpu=i686,osv=1.5.9{0.112-4-2}.cpp,debug,gcc3.3.1,os=CYGWIN_NT-5.1,cpu=i686,osv=1.5.9{0.112-4-2}.o.
In theory, one can preprocess a file more than once, but the resulting filenames can be too long for some file systems.
yabs3's compilation rule uses a modification of
Yabs's autocmds technique for ensuring that
object files are rebuilt when their compilation commands change.
The autocmds technique would create an extra
foo<build>.o.cmd file for every object file,
each of which would have to be read, which could take significant time in
large projects, so instead, for each directory, yabs3 writes a
generic compilation command into the file
foo/bar/_yabs/.cpp.<build>.o, only updating this
file when its contents would change. The file is then specified as a
prerequisite of all foo/bar/_yabs/*.cpp.<build>.o
files.
This is a compromise between storing the command that builds each
generated file alongside each generated files (which ensures that no files
are recompiled unnecessarily, but could be slow), and storing the generic
command in some central place, such as /var/yabs/. It has the
useful property that restoring a rule to its original form after making a
temporary change, will only force a rebuild of targets that are in the same
directory as any targets built with the temporary rule.
A simple Yabs build file with hand-written rules could look like:
#! /usr/bin/env python
import sys, os, string
# Add yabs's location to Python's import path:
sys.path.insert( 0, '~/yabs-7')
import yabs, yabs2
# Use a pattern rule for compiling:
yabs2.add_patternrule( '%.o', '%1.c', 'gcc -c -o $@ $<', autocmds='.cmds')
# Use a hand-written rule for linking:
def linkrule( target, state):
if target != 'myprog': return None
prereqs = [ 'foo.o', 'bar.o']
command = 'gcc -o ' + target + ' ' + string.join( prereqs, ' ')
semiprereqs = [ 'foo.h']
return command, prereqs, semiprereqs
yabs.add_rule( linkrule, autocmds='.cmds')
yabs2.appmakeexit()
If the above file is called make.py, then one can build
myprof by running: ./make.py myprog. This will
output the commands as it runs them, which can be a little noisy. It
can be more convenient to use ./make.py -s myprog, which
supresses the display of commands.
The above rules use Yab's autocmds support to ensure that
targets are rebuilt if the link or compile commands are changed in the make.py
file itself.
One can use yabs3 to specify a project more simply:
#! /usr/bin/env python
import sys, os
# Add yabs's location to Python's import path:
sys.path.insert( 0, '~/yabs-7')
import yabs2, yabs3
yabs3.add_exe( 'foo', 'main.cpp foo.cpp')
yabs2.appmakeexit()
The directory example-1 contains a simple Yabs
programme example-1/make.py that uses
yabs3 to build an executable from two source files. Different
versions of the executable can be build with the commands like the
following:
cd example-1
./make.py foo.exe
./make.py foo.exe -b gcc,release
./make.py foo.exe -b gcc,threads,release
./make.py foo.exe -b gcc,threads,debug
In the main yabs directory, make.py is
a Yabs programme with targets that constitute some regression tests, including
checking that the Yabs build inside example-1 deals correctly
with a modified header file or a modified command-file. It also has targets
that use Yabs to build releases of Yabs itself.
In particular, one can run all the regression tests with the following command:
./make.py testYabs requires Python 2.2 or later. Various versions of Yabs have been tested on various combinations of openbsd-3.6..4.0, a dozen or so Linux distributions, Windows XP, Windows XP/cygwin, python-2.2..2.4, gcc-2.95.3, gcc-3.x, gcc-4.x etc.
Yabs' design draws from various sources. Peter Miller's
essay Recursive Make Considered Harmful (see References) makes the case for specifying a
project in one place, rather than using recursive invocation of a build
tool for each sub-directory. Paul D. Smith's extensive website on
using GNU make (see References) is required
reading for anyone interested in getting GNU Make to do the right thing,
particularly the details of auto-dependency generation. The Scons
project (see References) is a build system
written in python. yabs3's use of filenames to encode build
information originates in a GNU Make build system that I wrote a few
years ago.
Much has been written about how easy yet powerful Python is to use. Yabs was my first Python project, but even so I think I spent a lot more time working on the design of Yabs than I spent struggling with the language itself. The availability of closures was a delight to someone who had previously only really worked in depth in the C and C++ worlds, and they serve a vital role in simplifying Yabs' code.
IMHO, all build systems are (or should be) basically dependency
tree engines. Dependency trees are pretty simple things really, so
build systems should in turn be simple. The core function in Yabs is
yabs.start_make(); this function scans all available rules,
calls itself to build prerequisites, and runs the commands required to
build a particular target. It is about 400 lines long.
Getting GNU Make to provide the same functionality as Yabs is extremely
hard. Much of what Yabs does can just about be done using Make 2.80's
$(eval ...) macro, but even then only with a lot of work.
There are also some fundamental limitations with GNU Make that I cannot
find a way round. For example, using .d dependency files in a
large system with many different available builds, is very slow, because Make
has to load all dependency files, not just the ones that are needed for a
particular target. Yabs's semi-prerequisites are a much more direct way of
getting proper dependency handling without this overhead.
Another example is that I have never been able to tell Make to compile
foo/bar/main.c into a file foo/bar/_build/main.o -
GNU Make's % pattern matching is simply not powerful enough.
Using a full programming language like Python removes this sort of problem at
a stroke.
Compared with Scons, I think that Yabs is much smaller and simpler, with no attempt to provide a complete set of rules for all occasions. Yabs is likely to be much faster than Scons on large projects because it uses gcc to find implicit dependencies when it compiles files, rather than using a separate tool; this also guarantees that the implicit dependencies are correct - e.g. there is no risk of the include path being subtley different. Also, Yabs will only read dependency files when necessary. Yabs uses timestamps rather than MD5 checksums when comparing files, although optionally using checksums is on the to-do list.
With Yabs, decisions about what flags to pass to a command are made by
Python code in the rule-function itself; there is no use of variables like
$CCFLAGS, which are inherently less flexible than code.
It also enables Yabs to build different targets using different compilers in
the same build, which could be useful when building for a different machine
requires some tools to be built and run on the build machine. Similarly, no
variable substitution is performed on commands - rules are expected to insert
the target/prerequisite filenames in the appropriate places themselves. Yabs
does not impose any fixed interpretation such as
<dir>/<basename>.<suffix> on filenames -
rules-functions can analyse filenames in any way they want, for example using
regular expressions. I think requiring the user to write code to make
decisions is better than getting them to set particular variables, because it
is more general, and because the user is probably a programmer anyway.
Additionally, python makes writing small functions very easy.
Yabs only deals with dependency trees and deliberately has no special
handling of compilers, or environment variables such $CFLAGS.
Similarly, it has no special knowledge about version control systems. The
presumption is that most build problems can be expressed as a dependency
tree, and don't need the build system to have hard-coded information about
source code repositories etc.
There are some issues of style that are questionable, such as the use of
yabs.default_state to encode all rules and state and the
partitioning of Yabs into the yabs, yabs2 and
yabs3 modules. Í'm sure that many won't like
yabs3's approach of embedding build information in filenames
(e.g. this information could be put in the name of a top-level directory),
although remember that yabs3 is just one way of extending the
core Yabs functionality - it would be trivial to write a different extension
that behaved differently.
At the moment, some of the Yabs scripts modify sys.path to
allow a subsequent import to find a particular file, which seems
rather crude. This allows Yabs to be used without it being installed into the
standard Python module directory (e.g. it could be included in the release of
a project) - and also allows one Yabs programme to import another. There is
some discussion on the python newsgroups about allowing absolute imports in
future, which may help in this area.
Some things I think are pretty robustly defensible: the use of functions
to represent rules, the use of yabs.appmakeexit() to provide an
easy command-line interface to build scripts, the definition and use of
semi-prerequisites and the ability to build targets using text commands or a
python function. The support for rebuilding targets when their rules change
is very simple yet appears to work well.
One fundamental issue in yabs3 is the complete absence
of any build-flags being generated internally and being passed as
Python parameters to rule-functions. Instead, rules always look at a
target's filename and extract build flags from this filename - they are
never passed build information in any other way. This is important,
because it means that the filenames convey all the required information,
which makes the whole system much more robust because otherwise it
would be easy to let filenames and internal build information become
inconsistent. Also, it means that intermediate targets can be built
independently (e.g. ./make.py foo.c,gcc,release.o) and will be
handled identically as when they are built as part of a higher-level
target.
The Scons build system does this. On first glance, I'd imagine it would
makes things terribly slow for a large project because of the need to run
something like md5 on every source file in a project, but I'm
willing to be shown otherwise.
It would be straightforward to support rules that generate more than one target - this would require changing rule functions to return an extra list of subsidiary targets. I don't know whether this useful though.
If a rule fails due to Yabs not being able to build its prerequisites,
Yabs will carry on looking for a different matching rule. This can result in
many generic rules being reported as failing to make a target. It might be
better to be able to tell Yabs that if a rule returns non-None,
and one of the prerequisites cannot be made, then no other rule should be
tried and the build should fail immediately.
autodepsIt would be straightforward to intercept calls to getenv(),
and store the name/returned-value in the autodeps output file (which
currently only stores the filenames that were opened). Then Yabs
could compare the current enviroment with the values in the autodeps file,
forcing a rebuild if there is a difference.
There could be a problem if commands modify environmental variables before calling other commands - Yabs would end up always re-running these commands.
autodeps could be made to also output information
on files that the command tried to open, but which failed. For
example, if a C source file has #include "foo.h",
and is compiled with gcc -I /home/me/includes -I
/home/you/includes/, and the initial build is done with the
file called /home/me/includes not existing, but a file
/home/you/includes/foo.h existing, then adding a file
/home/me/includes/foo.h should force a rebuild.
Reverse how commands/prerequisites are handled: rules would not list prerequisites. Instead, their commands are run immediately, but any file operations are trapped, and blocked while Yabs makes the requested file. When the requested file is up to date, the original command is resumed.
Export yabs2's % wildcard matching system for user to use.
Find a generic way of storing yabs3-style per-directory autocmds information.
GNU make: http://www.gnu.org/software/make/make.html
Recursive make considered harmful: http://www.tip.net.au/~millerp/rmch/recu-make-cons-harm.html
Paul D. Smith's GNU make page: http://make.paulandlesley.org
Scons - a Software Construction tool: http://www.scons.org