A utility for automating multi-host, multi-environment software buildsand deployments.
Project description
batou
batou is a multi-(component|host|environment|…) deployment utility.
Deployments are described in a model using “components” and can be deployed locally and remotely. Models are written as Python code.
See the batou documentation for details on how to use it.
Changelog
1.0b20 (2014-05-13)
Start using python-requests for HTTP/S downloads to support authentication (and other goodies).
Improve error logging.
1.0b19 (2014-04-07)
Allow specifying alternate find-links in requirements.txt
Remove remnants of the buildout component’s extends option.
1.0b18 (2014-03-31)
Introduce environment option connect_mode, either ‘ssh+sudo’ (default) or plain ‘ssh’.
Improve update code: existing installations would not update cleanly to a new version due to wrong use of pkg_resources API. Existing installations need to be cleaned manually. :/
Archive: recognize .tar.xz files and extract them with tar.
Fix bug in CMMI verify that would always raise UpdateNeeded due to a path mismatch (#13145).
Component.cmd() now performs expand() on the given command by default. Saves typing:
self.cmd(‘rm -rf {{component.workdir}}’)
instead of
self.cmd(self.expand(‘rm -rf {{component.workdir}}’))
File component: apply character encoding when reading and writing files. (#13165)
Mercurial clone: changed update behaviour yet again: unless configured differently, always achieve target state, even by losing uncommitted changes.
Allow unnamed access to the last added/ored component using the ‘self._’ attribute:
- def configure(self):
self += Download(…) self += Extract(self._.target)
1.0b17 (2013-12-06)
Mercurial clone: refuse to update if there are uncommitted changes or outgoing changesets.
Mercurial clone: added an option to avoid updating existing clones, which is often desirable during development.
1.0b16 (2013-11-29)
Improve the bootstrap/upgrade fix for #13049: the bootstrap template needed a second improvement.
1.0b15 (2013-11-29)
Try to fix #13049 - will only work once the dev version is already at least at b15. Not sure whether it really does the job, though. Hard to test.
Make environment and host arguments for running batou local optional. dev and localhost are the defaults now. Fixes #13034
Allow specifying secrets with [component:xy] in addition to [xy] to make it easier to discover when used to the environment overrides. Both options are supported now. Fixes #13003
Refactor Python virtualenv and package management. Provide explicit knowledge about Python versions 2.4, 2.5, 2.6, 2.7, 3.1, 3.2, and 3.3 to use compatible virtualenv/pip/setuptools versions and commands. Fixes #12941
NOTE: The API for batou.lib.python.Package has changed in backwards-incompatible ways:
Package() needs to be added to a VirtualEnv component. Before:
self += VirtualEnv(...) self += Package(...)
Now:
venv = VirtualEnv(...) self += venv venv += Package(...)
Package parameter pip_install_options has been renamed to install_options.
Use urllib module instead of wget command for cross platform compatibility.
1.0b14 (2013-11-26)
Make Buildout component install zc.buildout without dependencies (which is just setuptools), since that can break the version pinning of setuptools.
Check whether packages required via a -e expression in the local requirements.txt are already installed.
#12946: Make full path to python executable configurable for virtualenv (and buildout) component.
Remove cleanup code that removes batou-0.2-style virtualenv directories (bin, lib, include etc.) from remote deployments.
Add mailto setting to crontab component.
Added some documentation.
1.0b13 (2013-11-19)
Introduce a plugin mechanism: if a local requirements.txt file is present, install its contents into the .batou virtualenv.
Load components in alphabetical order. This enables importing from other components (when naming the directories appropriately).
#12986: Only try to unbundle if there actually are changes to unbundle.
Raise an error if a cron job is configured without a timing value.
Add platform support for Debian (crontab, logrotate, supervisor).
Added some documentation.
1.0b12 (2013-11-04)
Added branch argument to mercurial.Clone. Setting a branch automatically updates to the branch head on deploy. This is mostly useful for development environments.
Create the ‘secrets’ directory if it doesn’t exist, yet. Also, disallow editing secret files for non-existing environments.
Support continuing remote bootstrapping if we failed after creating the initial remote directory but were unable to use Mercurial.
#12898: build.Configure component was broken when using the default prefix.
1.0b11 (2013-10-17)
#12897: Use non-SSL pypi mirror for downloading virtualenv to fix tests failing randomly on machines that (for some reason) can’t validate PyPI’s certificate.
#12911: Ensure that we can configure file owners when they don’t exist during configure phase yet.
#12912: Fix untested and broken file ownership management.
#12847: Clean up unicode handling for File and Content components and templating.
#12910: Remote deployments failed when using bundles for transfers if no changes needed bundling.
#12766: Allow bootstrapping a batou project in an existing directory to support migration from 0.2.
#12283: Recognize files as ‘is_template’ by default. Auto-detect source files in the definition directory if they have the same basename. This is what you want in 99% of all cases. Explicitly stating either the ‘content’ or ‘source’ parameter disables auto-detection.
Now you can write this:
File(‘foo’)
and have components/x/foo recognized as the source file and handled as a template.
Use ConfigParser instead of configobj which is effectively unmaintained (see https://twitter.com/voidspace/status/380699699042213889) and support lists separated by newlines in addition to commas.
1.0b10 (2013-09-27)
Package our own virtualenv instead of depending on the system-installed one. This should alleviate troubles due to old virtualenv versions that package distribute, which causes conflicts with recent setuptools versions (#12874).
Update supervisor version to 3.0.
1.0b9 (2013-08-22)
Update Package component so it ignores installed packages when installing. This way, we actually install setuptools even when distribute is installed. (Otherwise it’s a no-op since distribute tells pip that setuptools is already satisfied).
Fix update process: wrong call to old ‘.batou/bin/batou’ failed and early bootstrapping would downgrade temporarily which is confusing and superfluous. Fixes #12739.
1.0b8 (2013-08-17)
Remove superfluous mkdir call during remote bootstrap.
Make batou init print that it’s working. Bootstrapping can take a while, so at least signal that something’s going on.
1.0b7 (2013-08-17)
Depend on Python2.7 to be available on the PATH during early bootstrap. Otherwise our chances to get a 2.7 virtualenv are pretty small, too.
Improve project template: ignore the work/ directory by default.
1.0b6 (2013-08-17)
More MANIFEST inclusions: bootstrap-template.
1.0b5 (2013-08-17)
Improve MANIFEST so we actually package the init template and other generated files, like version.txt and requirements.txt.
1.0b4 (2013-08-17)
Provide a simple project-creation command, both for pip-installed batou’s as well as spawning new projects from existing ones. Fixes #12730
Fix #12679: make timeouts configurable.
Removed re-imports from batou main module to support light-weight self-installation and bootstrapping. I.e. ‘from batou import Component’ no longer works.
Provide a single main command together with a ‘bootstrap’ wrapper that you can check into your project and that is maintained during updates automatically. It also provides fully automatic bootstrapping, installation, upgrading and other maintenance.
Fix Python package installation version check.
Don’t use bin/buildout bootstrap command anymore. PIP installs a sufficient bin/buildout so buildout can do the rest internally.
Install zc.buildout during bootstrapping phase using PIP to avoid bootstrap.py problems.
Shorten URLs in the Build component to their basename.
Add ‘assert_cmd’ API to support simpler assertions for verify when needing to check the result of an external command.
Switch to asking pip installing eggs instead of flat installations as namespaces seem to collide otherwise.
Remove non-functional deprecated ‘md5sum’ attribute.
Components are context managers now. If you provide __enter__ it will be called before verify() and if you provide __exit__ this will be called after update (always - even if update isn’t actually called). This allows you to manage temporary state on the target system more gracefully. See the DMGExtractor for an example.
Major refactoring of internal data structures to simplify and improve test coverage. Some breakage to be expected:
Components do not have a ‘service’ attribute any longer. If you need access to the service base directory use ‘environment.base_dir’.
Another attempt at getting convergency for package installation: actually leverage pkg_resources to check what version of a package is installed. PIP does not necessarily list or freeze setuptools, even if forced to reinstall.
Remove superfluous ‘–update’ from pip calls that would temporarily install the incorrect package version.
Improve convenience of the ‘secrets-edit’ wrapper script to allow re-encrypting without re-entering the editor.
Consistently switch to using setuptools.
Fix #12399: incorrect stat attributes for Owner and Group
Add exclude parameter to Directory component.
Add env parameter to Component.cmd() (and corresponding build_environment parameter to the Build component) to allow adding/overriding environment variables.
1.0b3 (2013-07-09)
Enable logging in the remote core to see what’s going on on the remote side.
Try to better format exceptions from the remote side.
Try harder to get virtualenv back into a working state.
Allow remote deployments from root of repository.
Make PIP management more robust.
1.0b2 (2013-07-09)
Add component to manage PIP within a virtual env.
Add component to manage packages with PIP within a virtual env.
Restructure buildout component to make it more robust regarding setuptools/distribute preparation. Also remove usage of bootstrap completely as we rely on virtualenv anyway.
1.0b1 (2013-07-09)
Apply semantic versioning: initial development is over, so this is 1.0 now.
Major revamp of secrets management:
switch to GPG (instead of aespipe)
turn secrets into a core feature, removing the need for a special component
Add ‘–single’ to suppress parallel bootstrapping.
Developing batou
- Issue tracker:
- Source code:
