././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1743675672.4112492 zc_buildout-4.1.6/0000755000076500000240000000000014773460430013464 5ustar00mauritsstaff././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675670.0 zc_buildout-4.1.6/.coveragerc0000644000076500000240000000022714773460426015613 0ustar00mauritsstaff[run] source = zc.buildout parallel = true [report] exclude_lines = pragma: no cover if __name__ == '__main__': raise NotImplementedError ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675670.0 zc_buildout-4.1.6/CHANGES.rst0000644000076500000240000006373214773460426015306 0ustar00mauritsstaffChange History ************** .. You should *NOT* be adding new change log entries to this file. You should create a file in the news directory instead. For helpful instructions, please see: https://github.com/buildout/buildout/blob/master/doc/ADD-A-NEWS-ITEM.rst .. towncrier release notes start 4.1.6 (2025-04-03) ------------------ Tests - While creating sample packages for testing, mostly create wheels instead of eggs. For the sample source distributions, create ``tar.gz`` instead of ``zip`` files. Then our package index for testing is more like the actual PyPI. [maurits] (#675) 4.1.5 (2025-03-31) ------------------ Bug fixes: - Implement PEP 503: request normalized package url on PyPI servers. [andreclimaco] (#634) - Install ``wheel`` before ``setuptools`` when checking if an upgrade and restart are needed. [maurits] (#691) 4.1.4 (2025-03-07) ------------------ Bug fixes: - If needed, copy and rename wheels before making an egg out of them. This helps for wheels of namespace packages created with ``setuptools`` 75.8.1 or higher. For namespace package we need a dot instead of an underscore in the resulting egg name. [maurits] (#686) 4.1.3 (2025-03-05) ------------------ Bug fixes: - Patch the ``find`` method from ``pkg_resources.WorkingSet``. Let this use the code from ``setuptools`` 75.8.2, if the currently used version is older. This is better at finding installed distributions. But don't patch ``setuptools`` versions older than 61: the new version of the method would give an error there. [maurits] (#682) 4.1.2 (2025-03-05) ------------------ Bug fixes: - Fix error finding the ``zc.buildout`` distribution when checking if we need to upgrade/restart. This depends on your ``setuptools`` version. [maurits] (#681) 4.1.1 (2025-03-04) ------------------ Bug fixes: - Fix error adding minimum ``zc.buildout`` version as requirement. [maurits] (#679) 4.1 (2025-03-04) ---------------- New features: - In the ``ls`` testing method, add keyword argument ``lowercase_and_sort_output``. The default is False, so no change. When true, as the name says, it sorts the output by lowercase, and prints it lowercase. We need this in one test because with ``setuptools`` 75.8.1 we no longer have a filename ``MIXEDCASE-0.5-pyN.N.egg``, but ``mixedcase-0.5-pyN.N.egg``. [maurits] (#7581) Bug fixes: - When trying to find a distribution for ``package.name``, first try the normalized name (``package_name``). This fixes an error finding entry points for namespace packages. The error is: ``TypeError: ('Expected str, Requirement, or Distribution', None)``. [maurits] (#7581) Development: - Test with latest ``setuptools`` 75.8.2 and with ``pip`` 25.0.1. Note that ``setuptools`` 75.8.1 can be troublesome and should be avoided. [maurits] (#7581) 4.0 (2025-01-30) ---------------- Breaking changes: - Drop Python 3.8 support. Require 3.9 as minimum. (#38) Development: - Test against `setuptools == 75.6.0`. (#671) 4.0.0a1 (2024-10-22) -------------------- Breaking changes: - Add dependency on ``packaging``. This gets rid of ugly compatibility code. [maurits] (#38) - Require ``setuptools >= 49.0.0``. This is the first version that supports PEP 496 environment markers, for example ``demo ==0.1; python_version < '3.9'``. An earlier change had ``setuptools >= 42.0.2``, otherwise we got ImportErrors. Also, since this is higher than 38.2.3, we are sure to have support for wheels. Remove support for ``distribute``, which was probably already broken. [maurits] (#38) - Drop support for Python 2. Require Python 3.8 as minimum. [maurits] (#38) New features: - Support Python 3.12 and 3.13. This only needed a few test fixes. [maurits] (#38) 3.3 (2024-10-17) ---------------- New features: - Allow the ``-I`` option in the Python interpreter wrapper installed by buildout when using the ``zc.recipe.egg`` recipe's `interpreter =` directive. This solves the issue when VSCode calls the designated Python interpreter for a workspace with this option to determine the Python version etc. (`#627 `_) 3.2.0 (2024-09-26) ------------------ New features: - Add config option: ``optional-extends``. This is the same as the ``extends`` option, but then for optional files. The names must be file paths, not URLs. If the path does not exist, it is silently ignored. This is useful for optionally loading a ``local.cfg`` or ``custom.cfg`` with options specific for the developer or the server. [maurits] (`#665 `_) 3.1.1 (2024-09-20) ------------------ Bug fixes: - Fix: a variable defined with initial ``+=`` was undefined and would lead to a corrupted ``.installed.cfg``. Fixes `issue 641 `_. [distributist] - Fix: extends with increments could result in missing values. Buildout processes them in the correct order now and combines them correctly. Fixes `issue 176 `_ and `issue 629 `_. [distributist] (#644) - Fix: Multiple ``+=`` or ``/-=`` in one file would lose assignment in a previous file. Fixes `issue 656 `_. [distributist] 3.1.0 (2024-08-29) ------------------ Breaking changes: - Drop support for Python 3.5. It is unsupported, and testing it is too hard. [maurits] (#35) Bug fixes: - Normalize package names when gathering packages. This should help find all distributions for package ``name.space``, whether they are called ``name.space-1.0.tar.gz`` with a dot or ``name_space-1.0.tar.gz`` with an underscore (created with ``setuptools`` 69.3 or higher). [maurits] (#647) - Fix ImportError: cannot import name ``packaging`` from ``pkg_resources`` with setuptools 70. Done by adding a compatibility module that tries to import `packaging` from several places. Fixes `issue 648 `_. [maurits] (#648) 3.0.1 (2022-11-08) ------------------ Bug fixes: - Fixed import of packaging.markers. [maurits] (#621) 3.0.0 (2022-11-07) ------------------ New features: - Add support for PEP 508 markers in section condition expressions. For example: ``[versions:python_version <= "3.9"]``. [maurits] (#621) Bug fixes: - Command-line 'extends' now works with dirs in file names [gotcha] (cli-extends) - Add support for python311-315 in conditional section expressions. (#311) - Make compatible with pip 22.2+, restoring Requires-Python functionality there. Fixes `issue 613 `_. [maurits] (#613) 3.0.0rc3 (2022-04-07) --------------------- Bug fixes: - Fix `TypeError: dist must be a Distribution instance` due to issue between `setuptools` and `pip`. (#600) 3.0.0rc2 (2022-03-04) --------------------- New features: - add support for PEP496 environment markers (pep496) Bug fixes: - Fix TypeError for missing required `use_deprecated_html5lib` with pip 22. Keep compatible with earlier pip versions. (#598) 3.0.0rc1 (2021-12-16) --------------------- Bug fixes: - Call pip via `python -m pip`. (#569) 3.0.0b5 (2021-11-29) -------------------- Bug fixes: - Fix when c extension implements namespace packages without the corresponding directories. (#589) - Honor command-line buildout:extends (#592) 3.0.0b4 (2021-11-25) -------------------- New features: - Allow to run buildout in FIPS enabled environments. (#570) - Proper error message if extends-cache tries to expand ${section:variable} (#585) Bug fixes: - Forward verbose option to pip (#576) - Check that file top_level.txt exists before opening. Add check for other files as well. (#582) - Return code of pip install subprocess is now properly returned to buildout. (#586) 3.0.0b3 (2021-10-08) -------------------- New features: - Improve warning message when a section contains unused options. (#483) Bug fixes: - Fix support of ``pip>=21.1`` (#567) - Fix confusion when using multiple Python versions and installing packages with C extensions without proper binary wheel available. (#574) Development: - Avoid broken jobs on Travis because of security on PRs (travis-pr) 3.0.0b2 (2021-03-09) -------------------- New features: - Improve error message when a package version is not pinned and `allow-picked-versions = false`. (#481) Bug fixes: - Fix FileNotFoundError when installing eggs with top-level directory without code (like doc). (#556) Development: - Login to docker hub to avoid pull limits (travis) - Initialize towncrier (#519) 3.0.0b1 (2021-03-07) ==================== - Fix issue with combination of `>` specs and `extras` and recent `setuptools`. - Fix issue with incrementing options from `.buildout/default.cfg`. - Support python37, python38 and python39 in conditional section expressions. - Fix bootstrapping for python27 and python35. 3.0.0a2 (2020-05-25) ==================== - Ignore `.git` when computing signature of a recipe develop egg - Warn when the name passed to `zc.recipe.egg:scripts` is not defined in egg entry points. - Show pip warning about Python version only once. - Better patch for ``pkg_resources.Distribution.hashcmp`` performance. 3.0.0a1 (2020-05-17) ==================== - Scripts: ensure eggs are inserted before ``site-packages`` in ``sys.path``. - Fix forever loop when changing ``zc.buildout`` version via ``buildout``. - Add support for ``Requires-Python`` metadata. Fragile monkeypatch that relies on ``pip._internal``. Emits a warning when support is disabled due to changes in ``pip``. - Use ``pip install`` instead of deprecated ``setuptools.easy_install``. - Patch ``pkg_resources.Distribution`` to make install of unpinned versions quicker. Most obvious with ``setuptools``. 2.13.3 (2020-02-11) =================== - Fix DeprecationWarning about MutableMapping. (`#484 `_) 2.13.2 (2019-07-03) =================== - Fixed DeprecationWarning on python 3.7: "'U' mode is deprecated". 2.13.1 (2019-01-29) =================== - Documentation update for the new ``buildout query`` command. 2.13.0 (2019-01-17) =================== - Get information about the configuration with new command ``buildout query``. 2.12.2 (2018-09-04) =================== - Upon an error, buildout exits with a non-zero exit code. This now also works when running with ``-D``. - Fixed most 'Deprecation' and 'Resource' warnings. 2.12.1 (2018-07-02) =================== - zc.buildout now explicitly requests zc.recipe.egg >=2.0.6 now. 2.12.0 (2018-07-02) =================== - Add a new buildout option ``allow-unknown-extras`` to enable installing requirements that specify extras that do not exist. This needs a corresponding update to zc.recipe.egg. See `issue 457 `_. zc.recipe.egg has been updated to 2.0.6 for this change. 2.11.5 (2018-06-19) =================== - Fix for `issue 295 `_. On windows, deletion of temporary egg files is more robust now. 2.11.4 (2018-05-14) =================== - Fix for `issue 451 `_: distributions with a version number that normalizes to a shorter version number (3.3.0 to 3.3, for instance) can be installed now. 2.11.3 (2018-04-13) =================== - Update to use the new PyPI at https://pypi.org/. 2.11.2 (2018-03-19) =================== - Fix for the #442 issue: AttributeError on ``pkg_resources.SetuptoolsVersion``. 2.11.1 (2018-03-01) =================== - Made upgrade check more robust. When using extensions, the improvement introduced in 2.11 could prevent buildout from restarting itself when it upgraded setuptools. 2.11.0 (2018-01-21) =================== - Installed packages are added to the working set immediately. This helps in some corner cases that occur when system packages have versions that conflict with our specified versions. 2.10.0 (2017-12-04) =================== - Setuptools 38.2.0 started supporting wheels. Through setuptools, buildout now also supports wheels! You need at least version 38.2.3 to get proper namespace support. This setuptools change interfered with buildout's recent support for `buildout.wheel `_, resulting in a sudden "Wheels are not supported" error message (see `issue 435 `_). Fixed by making setuptools the default, though you can still use the buildout.wheel if you want. 2.9.6 (2017-12-01) ================== - Fixed: could not install eggs when sdist file name and package name had different case. 2.9.5 (2017-09-22) ================== - Use HTTPS for PyPI's index. PyPI redirects HTTP to HTTPS by default now so using HTTPS directly avoids the potential for that redirect being modified in flight. 2.9.4 (2017-06-20) ================== - Sort the distributions used to compute ``__buildout_signature__`` to ensure reproducibility under Python 3 or under Python 2 when ``-R`` is used on ``PYTHONHASHSEED`` is set to ``random``. Fixes `issue 392 `_. **NOTE**: This may cause existing ``.installed.cfg`` to be considered outdated and lead to parts being reinstalled spuriously under Python 2. - Add support code for doctests to be able to easily measure code coverage. See `issue 397 `_. 2.9.3 (2017-03-30) ================== - Add more verbosity to ``annotate`` results with ``-v`` - Select one or more sections with arguments after ``buildout annotate``. 2.9.2 (2017-03-06) ================== - Fixed: We unnecessarily used a function from newer versions of setuptools that caused problems when older setuptools or pkg_resources installs were present (as in travis.ci). 2.9.1 (2017-03-06) ================== - Fixed a minor packaging bug that broke the PyPI page. 2.9.0 (2017-03-06) ================== - Added new syntax to explicitly declare that a part depends on other part. See http://docs.buildout.org/en/latest/topics/implicit-parts.html - Internal refactoring to work with `buildout.wheel `_. - Fixed a bugs in ``zc.buildout.testing.Buildout``. It was loading user-default configuration. It didn't support calling the ``created`` method on its sections. - Fixed a bug (windows, py 3.4) When processing metadata on "old-style" distutils scripts, .exe stubs appeared in ``metadata_listdir``, in turn reading those burped with ``UnicodeDecodeError``. Skipping .exe stubs now. 2.8.0 (2017-02-13) ================== - Added a hook to enable a soon-to-be-released buildout extension to provide wheel support. 2.7.1 (2017-01-31) ================== - Fixed a bug introduced in 2.6.0: zc.buildout and its dependeoncies were reported as picked even when their versions were fixed in a ``versions`` section. Worse, when the ``update-versions-file`` option was used, the ``versions`` section was updated needlessly on every run. 2.7.0 (2017-01-30) ================== - Added a buildout option, ``abi-tag-eggs`` that, when true, causes the `ABI tag `_ for the buildout environment to be added to the eggs directory name. This is useful when switching Python implementations (e.g. CPython vs PyPI or debug builds vs regular builds), especially when environment differences aren't reflected in egg names. It also has the side benefit of making eggs directories smaller, because eggs for different Python versions are in different directories. 2.6.0 (2017-01-29) ================== - Updated to work with the latest setuptools. - Added (verified) Python 3.6 support. 2.5.3 (2016-09-05) ================== - After a dist is fetched and put into its final place, compile its python files. No longer wait with compiling until all dists are in place. This is related to the change below about not removing an existing egg. [maurits] - Do not remove an existing egg. When installing an egg to a location that already exists, keep the current location (directory or file). This can only happen when the location at first did not exist and this changed during the buildout run. We used to remove the previous location, but this could cause problems when running two buildouts at the same time, when they try to install the same new egg. Fixes #307. [maurits] - In ``zc.buildout.testing.system``, set ``TERM=dumb`` in the environment. This avoids invisible control characters popping up in some terminals, like ``xterm``. Note that this may affect tests by buildout recipes. [maurits] - Removed Python 2.6 and 3.2 support. [do3cc] 2.5.2 (2016-06-07) ================== - Fixed ``-=`` and ``+=`` when extending sections. See #161. [puittenbroek] 2.5.1 (2016-04-06) ================== - Fix python 2 for downloading external config files with basic auth in the URL. Fixes #257. 2.5.0 (2015-11-16) ================== - Added more elaborate version and requirement information when there's a version conflict. Previously, you could get a report of a version conflict without information about which dependency requested the conflicing requirement. Now all this information is logged and displayed in case of an error. [reinout] - Dropped 3.2 support (at least in the automatic tests) as setuptools will soon stop supporting it. Added python 3.5 to the automatic tests. [reinout] 2.4.7 (2015-10-29) ================== - Fix for #279. Distutils script detection previously broke on windows with python 3 because it errored on ``.exe`` files. [reinout] 2.4.6 (2015-10-28) ================== - Relative paths are now also correctly generated for the current directory ("develop = ."). [youngking] 2.4.5 (2015-10-14) ================== - More complete fix for #24. Distutils scripts are now also generated for develop eggs. [reinout] 2.4.4 (2015-10-02) ================== - zc.buildout is now also released as a wheel. (Note: buildout itself doesn't support installing wheels yet.) [graingert] 2.4.3 (2015-09-03) ================== - Added nested directory creation support [guyzmo] 2.4.2 (2015-08-26) ================== - If a downloaded config file in the "extends-cache" gets corrupted, buildout now tells you the filename in the cache. Handy for troubleshooting. [reinout] 2.4.1 (2015-08-08) ================== - Check the ``use-dependency-links`` option earlier. This can give a small speed increase. [maurits] - When using python 2, urllib2 is used to work around Python issue 24599, which affects downloading from behind a proxy. [stefano-m] 2.4.0 (2015-07-01) ================== - Buildout no longer breaks on packages that contain a file with a non-ascii filename. Fixes #89 and #148. [reinout] - Undo breakage on Windows machines where ``sys.prefix`` can also be a ``site-packages`` directory: don't remove it from ``sys.path``. See https://github.com/buildout/buildout/issues/217 . - Remove assumption that ``pkg_resources`` is a module (untrue since release of `setuptools 8.3``). See https://github.com/buildout/buildout/issues/227 . - Fix for #212. For certain kinds of conflict errors you'd get an UnpackError when rendering the error message. Instead of a nicely formatted version conflict message. [reinout] - Making sure we use the correct easy_install when setuptools is installed globally. See https://github.com/buildout/buildout/pull/232 and https://github.com/buildout/buildout/pull/222 . [lrowe] - Updated buildout's `travis-ci `_ configuration so that tests run much quicker so that buildout is easier and quicker to develop. [reinout] - Note: zc.recipe.egg has also been updated to 2.0.2 together with this zc.buildout release. Fixed: In ``zc.recipe.egg#custom`` recipe's ``rpath`` support, don't assume path elements are buildout-relative if they start with one of the "special" tokens (e.g., ``$ORIGIN``). See: https://github.com/buildout/buildout/issues/225. [tseaver] - ``download-cache``, ``eggs-directory`` and ``extends-cache`` are now automatically created if their parent directory exists. Also they can be relative directories (relative to the location of the buildout config file that defines them). Also they can now be in the form ``~/subdir``, with the usual convention that the ``~`` char means the home directory of the user running buildout. [lelit] - A new bootstrap.py file is released (version 2015-07-01). - When bootstrapping, the ``develop-eggs/`` directory is first removed. This prevents old left-over ``.egg-link`` files from breaking buildout's careful package collection mechanism. [reinout] - The bootstrap script now accepts ``--to-dir``. Setuptools is installed there. If already available there, it is reused. This can be used to bootstrap buildout without internet access. Similarly, a local ``ez_setup.py`` is used when available instead of it being downloaded. You need setuptools 14.0 or higher for this functionality. [lrowe] - The bootstrap script now uses ``--buildout-version`` instead of ``--version`` to pick a specific buildout version. [reinout] - The bootstrap script now accepts ``--version`` which prints the bootstrap version. This version is the date the bootstrap.py was last changed. A date is handier or less confusing than either tracking zc.buildout's version or having a separate bootstrap version number. [reinout] 2.3.1 (2014-12-16) ================== - Fixed: Buildout merged single-version requirements with version-range requirements in a way that caused it to think there wasn't a single-version requirement. IOW, buildout through that versions were being picked when they weren't. - Suppress spurious (and possibly non-spurious) version-parsing warnings. 2.3.0 (2014-12-14) ================== - Buildout is now compatible with (and requires) setuptools 8. 2.2.5 (2014-11-04) ================== - Improved fix for #198: when bootstrapping with an extension, buildout was too strict on itself, resulting in an inability to upgrade or downgrade its own version. [reinout] - Setuptools must be at 3.3 or higher now. If you use the latest bootstrap from http://downloads.buildout.org/2/bootstrap.py you're all set. [reinout] - Installing *recipes* that themselves have dependencies used to fail with a VersionConflict if such a dependency was installed globally with a lower version. Buildout now ignores the version conflict in those cases and simply installs the correct version. [reinout] 2.2.4 (2014-11-01) ================== - Fix for #198: buildout 2.2.3 caused a version conflict when bootstrapping a buildout with a version pinned to an earlier one. Same version conflict could occur with system-wide installed packages that were newer than the pinned version. [reinout] 2.2.3 (2014-10-30) ================== - Fix #197, Python 3 regression [aclark4life] 2.2.2 (2014-10-30) ================== - Open files for ``exec()`` in universal newlines mode. See https://github.com/buildout/buildout/issues/130 - Add ``BUILDOUT_HOME`` as an alternate way to control how the user default configuration is found. - Close various files when finished writing to them. This avoids ResourceWarnings on Python 3, and better supports doctests under PyPy. - Introduce improved easy_install Install.install function. This is present in 1.5.X and 1.7X but was never merged into 2.X somehow. 2.2.1 (2013-09-05) ================== - ``distutils`` scripts: correct order of operations on ``from ... import`` lines (see https://github.com/buildout/buildout/issues/134). - Add an ``--allow-site-packges`` option to ``bootstrap.py``, defaulting to False. If the value is false, strip any "site packages" (as defined by the ``site`` module) from ``sys.path`` before attempting to import ``setuptools`` / ``pkg_resources``. - Updated the URL used to fetch ``ez_setup.py`` to the official, non-version- pinned version. 2.2.0 (2013-07-05) ================== - Handle both addition and subtraction of elements (+= and -=) on the same key in the same section. Forward-ported from buildout 1.6. - Suppress the useless ``Link to ***BLOCKED*** by --allow-hosts`` error message being emitted by distribute / setuptools. - Extend distutils script generation to support module docstrings and __future__ imports. - Refactored picked versions logic to make it easier to use for plugins. - Use ``get_win_launcher`` API to find Windows launcher (falling back to ``resource_string`` for ``cli.exe``). - Remove ``data_files`` from ``setup.py``: it was installing ``README.txt`` in current directory during installation (merged from 1.x branch). - Switch dependency from ``distribute 0.6.x`` to ``setuptools 0.7.x``. 2.1.0 (2013-03-23) ================== - Meta-recipe support - Conditional sections - Buildout now accepts a ``--version`` command-line option to print its version. Fixed: Builout didn't exit with a non-zero exit status if there was a failure in combination with an upgrade. Fixed: We now fail with an informative error when an old bootstrap script causes buildout 2 to be used with setuptools. Fixed: An error incorrectly suggested that buildout 2 implemented all of the functionality of dumppickedversions. Fixed: Buildout generated bad scripts when no eggs needed to be added to ``sys.path``. Fixed: Buildout didn't honour Unix umask when generating scripts. https://bugs.launchpad.net/zc.buildout/+bug/180705 Fixed: ``update-versions-file`` didn't work unless ``show-picked-versions`` was also set. https://github.com/buildout/buildout/issues/71 2.0.1 (2013-02-16) ================== - Fixed: buildout didn't honor umask settings when creating scripts. - Fix for distutils scripts installation on Python 3, related to ``__pycache__`` directories. - Fixed: encoding data in non-entry-point-based scripts was lost. ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675670.0 zc_buildout-4.1.6/CONTRIBUTING.rst0000644000076500000240000000741114773460426016135 0ustar00mauritsstaffHow To Contribute ***************** Thank you for considering contributing to ``buildout``! Workflow ======== - No contribution is too small! - Please make sure to create one pull request for one change. - Please try to add tests for your code. - Make sure your changes pass **continuous integration**. When CI fails, please try to fix it or ask for help. Developing buildout itself and running the test suite ===================================================== When you're developing buildout itself, you need to know a few things: - The presence of ``~/.buildout/default.cfg`` may interfere with the tests. If you suspect this, you may want to temporarily rename it so that it does not get in the way. - Similarly, if you manually run ``bin/buildout`` you can run it with ``-U``. This ignores user (default) settings which can interfere with using the development version. For your convenience we provide a Makefile to create a Python virtual environment in the ``venvs`` subdirectory of the buildout checkout. You can have several venvs next to each other, but easiest is to use only one: the venvs are quick to recreate. To start, simply run ``make``. This uses ``python3``, so you get a venv with whatever your default ``python3`` version is. * It creates a venv named after this Python, for example ``venvs/python3.10``. * This has our dependencies: ``pip``, ``setuptools``, ``wheel``, ``packaging``. Plus ``build``, which may become a dependency, and that we need for the next step. * Using this venv, it creates a ``bin/buildout`` script in the main repository directory. * It then calls ``bin/buildout`` and this creates a ``bin/test`` script. * Then it calls ``bin/test`` to run the tests. You can call ``make clean`` to remove everything that was created. You can use environment variables to influence used versions: * Python, for example: ``PYTHON_VERSION=3.12`` * pip, for example: ``PIP_VERSION=24.2`` * setuptools, for example: ``SETUPTOOLS_VERSION=70.0.0`` * Use ``PIP_ARGS`` for extra arguments. The default is ``PIP_ARGS=-U``, so upgrade packages to the latest available version. So you can use this on the command line:: PYTHON_VERSION=3.12 PIP_VERSION=24.2 SETUPTOOLS_VERSION=70.0.0 make Releases: zc.buildout, zc.recipe.egg ==================================== Buildout consists of two Python packages that are released separately: zc.buildout and zc.recipe.egg. zc.recipe.egg is changed much less often than zc.buildout. zc.buildout's setup.py and changelog is in the same directory as this ``CONTRIBUTING.txt`` and the code is in ``src/zc/buildout``. zc.recipe.egg, including setup.py and a separate changelog, is in the ``zc.recipe.egg_`` subdirectory. When releasing, make sure you also build a (universal) wheel in addition to the regular .tar.gz:: $ ./venvs/python3.10/bin/python -m build . You can also use zest.releaser to release it. If you've installed it as ``zest.releaser[recommended]`` it builds the wheel for you and uploads using ``twine``. Roadmap ======= Currently, there are two active branches: - master (development branch for the upcoming version 4) - 3.x (development branch for the current version 3) Active feature development and bug fixes only happen on the **master** branch. Supported Python Versions ========================= We align the support of Python versions with `Zope `_ and `Plone `_ development. This means, the upcoming version 4 will support Python 3.8-3.13. If you need support for Python 2.7 or older Python 3 versions, use version 3. Licensing ========= This project is licensed under the Zope Public License. Unlike contributing to the Zope and Plone projects, you do not need to sign a contributor agreement to contribute to **buildout**. ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675670.0 zc_buildout-4.1.6/COPYRIGHT.txt0000644000076500000240000000004014773460426015574 0ustar00mauritsstaffZope Foundation and Contributors././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675670.0 zc_buildout-4.1.6/HISTORY.rst0000644000076500000240000007530614773460426015377 0ustar00mauritsstaffChange History -- Old changes ***************************** 2.0.0 (2013-02-10) ================== This is a backward incompatible release of buildout that attempts to correct mistakes made in buildout 1. - Buildout no-longer tries to provide full or partial isolation from system Python installations. If you want isolation, use buildout with virtualenv, or use a clean build of Python to begin with. Providing isolation was a noble goal, but it's implementation complicated buildout's implementation too much. - Buildout no-longer supports using multiple versions of Python in a single buildout. This too was a noble goal, but added too much complexity to the implementation. - Changed the configuration file format: - Relative indentation in option values is retained if the first line is blank. (IOW, if the non-blank text is on the continuation lines.) As in:: [mysection] tree = /root branch In such cases, internal blank lines are also retained. - The configuration syntax is more tightly defined, allowing fewer syntax definitions. Buildout 1 configuration files were parsed with the Python ConfigParser module. The ConfigParser module's format is poorly documented and wildly flexible. For example: - Any characters other than left square brackets were allowed in section names. - Arbitrary text was allowed and ignored after the closing bracket on section header lines. - Any characters other than equal signs or colons were allowed in an option name. - Configuration options could be spelled as RFC 822 mail headers (using a colon, rather than an equal sign). - Comments could begin with "rem". - Semicolons could be used to start inline comments, but only if preceded by a whitespace character. See `Configuration file syntax`_. - Buildout now prefers final releases by default (buildout:prefer-final now defaults to true, rather than false.) However, if buildout is bootstrapped with a non-final release, it won't downgrade itself to a final release. - Buildout no-longer installs zipped eggs. (Distribute may still install a zipped egg of itself during the bootstrapping process.) The ``buildout:unzip`` option has been removed. - Buildout no-longer supports setuptools. It now uses distribute exclusively. - Integrated the `buildout-versions `_ extension into buildout itself. For this, a few options were added to buildout: - If ``show-picked-versions`` is set to true, all picked versions are printed at the end of the buildout run. This saves you from running buildout in verbose mode and extracting the picked versions from the output. - If ``update-versions-file`` is set to a filename (relative to the buildout directory), the ``show-picked-versions`` output is appended to that file. - Buildout options can be given on the command line using the form:: option_name=value as a short-hand for:: buildout:option_name=value - The ``versions`` option now defaults to ``versions``, so you no longer need to include:: versions = versions in a ``buildout`` section when pinning versions. A ``versions`` section is added, if necessary, if a ``versions`` option isn't used. - Buildout-defined default versions are included in the versions section, if there is one. - The ``buildout:zc.buildout-version`` and ``buildout:distribute-version`` options have been removed in favor of providing version constraints in a versions section. - Error if install-from-cache and offline are used together, because offline largely means "don't install". - Provide better error messages when distributions can't be installed because buildout is run in offline mode. - Versions in versions sections can now be simple constraints, like >=2.0dev in addition to being simple versions. Buildout 2 leverages this to make sure it uses zc.recipe.egg>=2.0.0a3, which mainly matters for Python 3. - The buildout init command now accepts distribution requirements and paths to set up a custom interpreter part that has the distributions or parts in the path. For example:: python bootstrap.py init BeautifulSoup - Added buildout:socket-timeout option so that socket timeout can be configured both from command line and from config files. (gotcha) - Distutils-style scripts are also installed now (for instance pyflakes' and docutils' scripts). https://bugs.launchpad.net/zc.buildout/+bug/422724 - Avoid sorting the working set and requirements when it won't be logged. When profiling a simple buildout with 10 parts with identical and large working sets, this resulted in a decrease of run time from 93.411 to 15.068 seconds, about a 6 fold improvement. To see the benefit be sure to run without any increase in verbosity ("-v" option). (rossp) - Introduce a cache for the expensive `buildout._dir_hash` function. - Remove duplicate path from script's sys.path setup. - Make sure to download extended configuration files only once per buildout run even if they are referenced multiple times (patch by Rafael Monnerat). - Removed any traces of the implementation of ``extended-by``. Raise a UserError if the option is encountered instead of ignoring it, though. Fixed: relative-paths weren't honored when bootstrapping or upgrading (which is how the buildout script gets generated). Fixed: initialization code wasn't included in interpreter scripts. Fixed: macro inheritance bug, https://github.com/buildout/buildout/pull/37 Fixed: In the download module, fixed the handling of directories that are pointed to by file-system paths and ``file:`` URLs. Fixed if you have a configuration with an extends entry in the [buildout] section which points to a non-existing URL the result is not very user friendly. https://bugs.launchpad.net/zc.buildout/+bug/566167 Fixed: https://bugs.launchpad.net/bugs/697913 : Buildout doesn't honor exit code from scripts. Fixed. 1.4.4 (2010-08-20) ================== The 1.4.4 release is a release for people who encounter trouble with the 1.5 line. By switching to `the associated bootstrap script `_ you can stay on 1.4.4 until you are ready to migrate. 1.4.3 (2009-12-10) ================== Bugs fixed: - Using pre-detected setuptools version for easy_installing tgz files. This prevents a recursion error when easy_installing an upgraded "distribute" tgz. Note that setuptools did not have this recursion problem solely because it was packaged as an ``.egg``, which does not have to go through the easy_install step. 1.4.2 (2009-11-01) ================== New Feature: - Added a --distribute option to the bootstrap script, in order to use Distribute rather than Setuptools. By default, Setuptools is used. Bugs fixed: - While checking for new versions of setuptools and buildout itself, compare requirement locations instead of requirement objects. - Incrementing didn't work properly when extending multiple files. https://bugs.launchpad.net/zc.buildout/+bug/421022 - The download API computed MD5 checksums of text files wrong on Windows. 1.4.1 (2009-08-27) ================== New Feature: - Added a debug built-in recipe to make writing some tests easier. Bugs fixed: - (introduced in 1.4.0) option incrementing (-=) and decrementing (-=) didn't work in the buildout section. https://bugs.launchpad.net/zc.buildout/+bug/420463 - Option incrementing and decrementing didn't work for options specified on the command line. - Scripts generated with relative-paths enabled couldn't be symbolically linked to other locations and still work. - Scripts run using generated interpreters didn't have __file__ set correctly. - The standard Python -m option didn't work for custom interpreters. 1.4.0 (2009-08-26) ================== - When doing variable substitutions, you can omit the section name to refer to a variable in the same section (e.g. ${:foo}). - When doing variable substitution, you can use the special option, ``_buildout_section_name_`` to get the section name. This is most handy for getting the current section name (e.g. ${:_buildout_section_name_}). - A new special option, ``<`` allows sections to be used as macros. - Added annotate command for annotated sections. Displays sections key-value pairs along with the value origin. - Added a download API that handles the download cache, offline mode etc and is meant to be reused by recipes. - Used the download API to allow caching of base configurations (specified by the buildout section's 'extends' option). 1.3.1 (2009-08-12) ================== - Bug fixed: extras were ignored in some cases when versions were specified. 1.3.0 (2009-06-22) ================== - Better Windows compatibility in test infrastructure. - Now the bootstrap.py has an optional --version argument, that can be used to force buildout version to use. - ``zc.buildout.testing.buildoutSetUp`` installs a new handler in the python root logging facility. This handler is now removed during tear down as it might disturb other packages reusing buildout's testing infrastructure. - fixed usage of 'relative_paths' keyword parameter on Windows - Added an unload entry point for extensions. - Fixed bug: when the relative paths option was used, relative paths could be inserted into sys.path if a relative path was used to run the generated script. 1.2.1 (2009-03-18) ================== - Refactored generation of relative egg paths to generate simpler code. 1.2.0 (2009-03-17) ================== - Added a relative_paths option to zc.buildout.easy_install.script to generate egg paths relative to the script they're used in. 1.1.2 (2009-03-16) ================== - Added Python 2.6 support. Removed Python 2.3 support. - Fixed remaining deprecation warnings under Python 2.6, both when running our tests and when using the package. - Switched from using os.popen* to subprocess.Popen, to avoid a deprecation warning in Python 2.6. See: http://docs.python.org/library/subprocess.html#replacing-os-popen-os-popen2-os-popen3 - Made sure the 'redo_pyc' function and the doctest checkers work with Python executable paths containing spaces. - Expand shell patterns when processing the list of paths in `develop`, e.g:: [buildout] develop = ./local-checkouts/* - Conditionally import and use hashlib.md5 when it's available instead of md5 module, which is deprecated in Python 2.6. - Added Jython support for bootstrap, development bootstrap and buildout support on Jython - Fixed a bug that would cause buildout to break while computing a directory hash if it found a broken symlink (Launchpad #250573) 1.1.1 (2008-07-28) ================== - Fixed a bug that caused buildouts to fail when variable substitutions are used to name standard directories, as in:: [buildout] eggs-directory = ${buildout:directory}/develop-eggs 1.1.0 (2008-07-19) ================== - Added a buildout-level unzip option to change the default policy for unzipping zip-safe eggs. - Tracebacks are now printed for internal errors (as opposed to user errors) even without the -D option. - pyc and pyo files are regenerated for installed eggs so that the stored path in code objects matches the install location. 1.0.6 (2008-06-13) ================== - Manually reverted the changeset for the fix for https://bugs.launchpad.net/zc.buildout/+bug/239212 to verify thet the test actually fails with the changeset: http://svn.zope.org/zc.buildout/trunk/src/zc/buildout/buildout.py?rev=87309&r1=87277&r2=87309 Thanks tarek for pointing this out. (seletz) - fixed the test for the += -= syntax in buildout.txt as the test was actually wrong. The original implementation did a split/join on whitespace, and later on that was corrected to respect the original EOL setting, the test was not updated, though. (seletz) - added a test to verify against https://bugs.launchpad.net/zc.buildout/+bug/239212 in allowhosts.txt (seletz) - further fixes for """AttributeError: Buildout instance has no attribute '_logger'""" by providing reasonable defaults within the Buildout constructor (related to the new 'allow-hosts' option) (patch by Gottfried Ganssauge) (ajung) 1.0.5 (2008-06-10) ================== - Fixed wrong split when using the += and -= syntax (mustapha) 1.0.4 (2008-06-10) ================== - Added the `allow-hosts` option (tarek) - Quote the 'executable' argument when trying to detect the python version using popen4. (sidnei) - Quote the 'spec' argument, as in the case of installing an egg from the buildout-cache, if the filename contains spaces it would fail (sidnei) - Extended configuration syntax to allow -= and += operators (malthe, mustapha). 1.0.3 (2008-06-01) ================== - fix for """AttributeError: Buildout instance has no attribute '_logger'""" by providing reasonable defaults within the Buildout constructor. (patch by Gottfried Ganssauge) (ajung) 1.0.2 (2008-05-13) ================== - More fixes for Windows. A quoted sh-bang is now used on Windows to make the .exe files work with a Python executable in 'program files'. - Added "-t " option for specifying the socket timeout. (ajung) 1.0.1 (2008-04-02) ================== - Made easy_install.py's _get_version accept non-final releases of Python, like 2.4.4c0. (hannosch) - Applied various patches for Windows (patch by Gottfried Ganssauge). (ajung) - Applied patch fixing rmtree issues on Windows (patch by Gottfried Ganssauge). (ajung) 1.0.0 (2008-01-13) ================== - Added a French translation of the buildout tutorial. 1.0.0b31 (2007-11-01) ===================== Feature Changes --------------- - Added a configuration option that allows buildouts to ignore dependency_links metadata specified in setup. By default dependency_links in setup are used in addition to buildout specified find-links. This can make it hard to control where eggs come from. Here's how to tell buildout to ignore URLs in dependency_links:: [buildout] use-dependency-links = false By default use-dependency-links is true, which matches the behavior of previous versions of buildout. - Added a configuration option that causes buildout to error if a version is picked. This is a nice safety belt when fixing all versions is intended, especially when creating releases. Bugs Fixed ---------- - 151820: Develop failed if the setup.py script imported modules in the distribution directory. - Verbose logging of the develop command was omitting detailed output. - The setup command wasn't documented. - The setup command failed if run in a directory without specifying a configuration file. - The setup command raised a stupid exception if run without arguments. - When using a local find links or index, distributions weren't copied to the download cache. - When installing from source releases, a version specification (via a buildout versions section) for setuptools was ignored when deciding which setuptools to use to build an egg from the source release. 1.0.0b30 (2007-08-20) ===================== Feature Changes --------------- - Changed the default policy back to what it was to avoid breakage in existing buildouts. Use:: [buildout] prefer-final = true to get the new policy. The new policy will go into effect in buildout 2. 1.0.0b29 (2007-08-20) ===================== Feature Changes --------------- - Now, final distributions are preferred over non-final versions. If both final and non-final versions satisfy a requirement, then the final version will be used even if it is older. The normal way to override this for specific packages is to specifically require a non-final version, either specifically or via a lower bound. - There is a buildout prefer-final version that can be used with a value of "false":: prefer-final = false To prefer newer versions, regardless of whether or not they are final, buildout-wide. - The new simple Python index, http://cheeseshop.python.org/simple, is used as the default index. This will provide better performance than the human package index interface, http://pypi.python.org/pypi. More importantly, it lists hidden distributions, so buildouts with fixed distribution versions will be able to find old distributions even if the distributions have been hidden in the human PyPI interface. Bugs Fixed ---------- - 126441: Look for default.cfg in the right place on Windows. 1.0.0b28 (2007-07-05) ===================== Bugs Fixed ---------- - When requiring a specific version, buildout looked for new versions even if that single version was already installed. 1.0.0b27 (2007-06-20) ===================== Bugs Fixed ---------- - Scripts were generated incorrectly on Windows. This included the buildout script itself, making buildout completely unusable. 1.0.0b26 (2007-06-19) ===================== Feature Changes --------------- - Thanks to recent fixes in setuptools, I was able to change buildout to use find-link and index information when searching extensions. Sadly, this work, especially the timing, was motivated my the need to use alternate indexes due to performance problems in the cheese shop (http://www.python.org/pypi/). I really home we can address these performance problems soon. 1.0.0b25 (2007-05-31) ===================== Feature Changes --------------- - buildout now changes to the buildout directory before running recipe install and update methods. - Added a new init command for creating a new buildout. This creates an empty configuration file and then bootstraps. - Except when using the new init command, it is now an error to run buildout without a configuration file. - In verbose mode, when adding distributions to fulfil requirements of already-added distributions, we now show why the new distributions are being added. - Changed the logging format to exclude the logger name for the buildout logger. This reduces noise in the output. - Clean up lots of messages, adding missing periods and adding quotes around requirement strings and file paths. Bugs Fixed ---------- - 114614: Buildouts could take a very long time if there were dependency problems in large sets of pathologically interdependent packages. - 59270: Buggy recipes can cause failures in later recipes via chdir - 61890: file:// urls don't seem to work in find-links setuptools requires that file urls that point to directories must end in a "/". Added a workaround. - 75607: buildout should not run if it creates an empty buildout.cfg 1.0.0b24 (2007-05-09) ===================== Feature Changes --------------- - Improved error reporting by showing which packages require other packages that can't be found or that cause version conflicts. - Added an API for use by recipe writers to clean up created files when recipe errors occur. - Log installed scripts. Bugs Fixed ---------- - 92891: bootstrap crashes with recipe option in buildout section. - 113085: Buildout exited with a zero exist status when internal errors occurred. 1.0.0b23 (2007-03-19) ===================== Feature Changes --------------- - Added support for download caches. A buildout can specify a cache for distribution downloads. The cache can be shared among buildouts to reduce network access and to support creating source distributions for applications allowing install without network access. - Log scripts created, as suggested in: https://bugs.launchpad.net/zc.buildout/+bug/71353 Bugs Fixed ---------- - It wasn't possible to give options on the command line for sections not defined in a configuration file. 1.0.0b22 (2007-03-15) ===================== Feature Changes --------------- - Improved error reporting and debugging support: - Added "logical tracebacks" that show functionally what the buildout was doing when an error occurs. Don't show a Python traceback unless the -D option is used. - Added a -D option that causes the buildout to print a traceback and start the pdb post-mortem debugger when an error occurs. - Warnings are printed for unused options in the buildout section and installed-part sections. This should make it easier to catch option misspellings. - Changed the way the installed database (.installed.cfg) is handled to avoid database corruption when a user breaks out of a buildout with control-c. - Don't save an installed database if there are no installed parts or develop egg links. 1.0.0b21 (2007-03-06) ===================== Feature Changes --------------- - Added support for repeatable buildouts by allowing egg versions to be specified in a versions section. - The easy_install module install and build functions now accept a versions argument that supplied to mapping from project name to version numbers. This can be used to fix version numbers for required distributions and their dependencies. When a version isn't fixed, using either a versions option or using a fixed version number in a requirement, then a debug log message is emitted indicating the version picked. This is useful for setting versions options. A default_versions function can be used to set a default value for this option. - Adjusted the output for verbosity levels. Using a single -v option no longer causes voluminous setuptools output. Using -vv and -vvv now triggers extra setuptools output. - Added a remove testing helper function that removes files or directories. 1.0.0b20 (2007-02-08) ===================== Feature Changes --------------- - Added a buildout newest option, to control whether the newest distributions should be sought to meet requirements. This might also provide a hint to recipes that don't deal with distributions. For example, a recipe that manages subversion checkouts might not update a checkout if newest is set to "false". - Added a *newest* keyword parameter to the zc.buildout.easy_install.install and zc.buildout.easy_install.build functions to control whether the newest distributions that need given requirements should be sought. If a false value is provided for this parameter and already installed eggs meet the given requirements, then no attempt will be made to search for newer distributions. - The recipe-testing support setUp function now adds the name *buildout* to the test namespace with a value that is the path to the buildout script in the sample buildout. This allows tests to use >>> print system(buildout), rather than: >>> print system(join('bin', 'buildout')), Bugs Fixed ---------- - Paths returned from update methods replaced lists of installed files rather than augmenting them. 1.0.0b19 (2007-01-24) ===================== Bugs Fixed ---------- - Explicitly specifying a Python executable failed if the output of running Python with the -V option included a 2-digit (rather than a 3-digit) version number. 1.0.0b18 (2007-01-22) ===================== Feature Changes --------------- - Added documentation for some previously undocumented features of the easy_install APIs. - By popular demand, added a -o command-line option that is a short hand for the assignment buildout:offline=true. Bugs Fixed ---------- - When deciding whether recipe develop eggs had changed, buildout incorrectly considered files in .svn and CVS directories. 1.0.0b17 (2006-12-07) ===================== Feature Changes --------------- - Configuration files can now be loaded from URLs. Bugs Fixed ---------- - https://bugs.launchpad.net/products/zc.buildout/+bug/71246 Buildout extensions installed as eggs couldn't be loaded in offline mode. 1.0.0b16 (2006-12-07) ===================== Feature Changes --------------- - A new command-line argument, -U, suppresses reading user defaults. - You can now suppress use of an installed-part database (e.g. .installed.cfg) by specifying an empty value for the buildout installed option. Bugs Fixed ---------- - When the install command is used with a list of parts, only those parts are supposed to be installed, but the buildout was also building parts that those parts depended on. 1.0.0b15 (2006-12-06) ===================== Bugs Fixed ---------- - Uninstall recipes weren't loaded correctly in cases where no parts in the (new) configuration used the recipe egg. 1.0.0b14 (2006-12-05) ===================== Feature Changes --------------- - Added uninstall recipes for dealing with complex uninstallation scenarios. Bugs Fixed ---------- - Automatic upgrades weren't performed on Windows due to a bug that caused buildout to incorrectly determine that it wasn't running locally in a buildout. - Fixed some spurious test failures on Windows. 1.0.0b13 (2006-12-04) ===================== Feature Changes --------------- - Variable substitutions now reflect option data written by recipes. - A part referenced by a part in a parts list is now added to the parts list before the referencing part. This means that you can omit parts from the parts list if they are referenced by other parts. - Added a develop function to the easy_install module to aid in creating develop eggs with custom build_ext options. - The build and develop functions in the easy_install module now return the path of the egg or egg link created. - Removed the limitation that parts named in the install command can only name configured parts. - Removed support ConfigParser-style variable substitutions (e.g. %(foo)s). Only the string-template style of variable (e.g. ${section:option}) substitutions will be supported. Supporting both violates "there's only one way to do it". - Deprecated the buildout-section extendedBy option. Bugs Fixed ---------- - We treat setuptools as a dependency of any distribution that (declares that it) uses namespace packages, whether it declares setuptools as a dependency or not. This wasn't working for eggs installed by virtue of being dependencies. 1.0.0b12 (2006-10-24) ===================== Feature Changes --------------- - Added an initialization argument to the zc.buildout.easy_install.scripts function to include initialization code in generated scripts. 1.0.0b11 (2006-10-24) ===================== Bugs Fixed ---------- `67737 `_ Verbose and quite output options caused errors when the develop buildout option was used to create develop eggs. `67871 `_ Installation failed if the source was a (local) unzipped egg. `67873 `_ There was an error in producing an error message when part names passed to the install command weren't included in the configuration. 1.0.0b10 (2006-10-16) ===================== Feature Changes --------------- - Renamed the runsetup command to setup. (The old name still works.) - Added a recipe update method. Now install is only called when a part is installed for the first time, or after an uninstall. Otherwise, update is called. For backward compatibility, recipes that don't define update methods are still supported. - If a distribution defines namespace packages but fails to declare setuptools as one of its dependencies, we now treat setuptools as an implicit dependency. We generate a warning if the distribution is a develop egg. - You can now create develop eggs for setup scripts that don't use setuptools. Bugs Fixed ---------- - Egg links weren't removed when corresponding entries were removed from develop sections. - Running a non-local buildout command (one not installed in the buildout) led to a hang if there were new versions of buildout or setuptools were available. Now we issue a warning and don't upgrade. - When installing zip-safe eggs from local directories, the eggs were moved, rather than copied, removing them from the source directory. 1.0.0b9 (2006-10-02) ==================== Bugs Fixed ---------- Non-zip-safe eggs were not unzipped when they were installed. 1.0.0b8 (2006-10-01) ==================== Bugs Fixed ---------- - Installing source distributions failed when using alternate Python versions (depending on the versions of Python used.) - Installing eggs wasn't handled as efficiently as possible due to a bug in egg URL parsing. - Fixed a bug in runsetup that caused setup scripts that introspected __file__ to fail. 1.0.0b7 ======= Added a documented testing framework for use by recipes. Refactored the buildout tests to use it. Added a runsetup command run a setup script. This is handy if, like me, you don't install setuptools in your system Python. 1.0.0b6 ======= Fixed https://launchpad.net/products/zc.buildout/+bug/60582 Use of extension options caused bootstrapping to fail if the eggs directory didn't already exist. We no longer use extensions for bootstrapping. There really isn't any reason to anyway. 1.0.0b5 ======= Refactored to do more work in buildout and less work in easy_install. This makes things go a little faster, makes errors a little easier to handle, and allows extensions (like the sftp extension) to influence more of the process. This was done to fix a problem in using the sftp support. 1.0.0b4 ======= - Added an **experimental** extensions mechanism, mainly to support adding sftp support to buildouts that need it. - Fixed buildout self-updating on Windows. 1.0.0b3 ======= - Added a help option (-h, --help) - Increased the default level of verbosity. - Buildouts now automatically update themselves to new versions of buildout and setuptools. - Added Windows support. - Added a recipe API for generating user errors. - No-longer generate a py_zc.buildout script. - Fixed some bugs in variable substitutions. The characters "-", "." and " ", weren't allowed in section or option names. Substitutions with invalid names were ignored, which caused misleading failures downstream. - Improved error handling. No longer show tracebacks for user errors. - Now require a recipe option (and therefore a section) for every part. - Expanded the easy_install module API to: - Allow extra paths to be provided - Specify explicit entry points - Specify entry-point arguments 1.0.0b2 ======= Added support for specifying some build_ext options when installing eggs from source distributions. 1.0.0b1 ======= - Changed the bootstrapping code to only install setuptools and buildout. The bootstrap code no-longer runs the buildout itself. This was to fix a bug that caused parts to be recreated unnecessarily because the recipe signature in the initial buildout reflected temporary locations for setuptools and buildout. - Now create a minimal setup.py if it doesn't exist and issue a warning that it is being created. - Fixed bug in saving installed configuration data. %'s and extra spaces weren't quoted. 1.0.0a1 ======= Initial public version ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675670.0 zc_buildout-4.1.6/LICENSE.txt0000644000076500000240000000646014773460426015322 0ustar00mauritsstaffZope Public License (ZPL) Version 2.1 A copyright notice accompanies this license document that identifies the copyright holders. This license has been certified as open source. It has also been designated as GPL compatible by the Free Software Foundation (FSF). Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions in source code must retain the accompanying copyright notice, this list of conditions, and the following disclaimer. 2. Redistributions in binary form must reproduce the accompanying copyright notice, this list of conditions, and the following disclaimer in the documentation and/or other materials provided with the distribution. 3. Names of the copyright holders must not be used to endorse or promote products derived from this software without prior written permission from the copyright holders. 4. The right to distribute this software or to use it for any purpose does not give you the right to use Servicemarks (sm) or Trademarks (tm) of the copyright holders. Use of them is covered by separate agreement with the copyright holders. 5. If any files are modified, you must cause the modified files to carry prominent notices stating that you changed the files and the date of any change. Disclaimer THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. =============================================================================== The module, zc.buildout.pep425tags, is copied from the wheel package, which uses the MIT license: "wheel" copyright (c) 2012-2014 Daniel Holth and contributors. The MIT License Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675670.0 zc_buildout-4.1.6/MANIFEST.in0000644000076500000240000000051114773460426015224 0ustar00mauritsstaffinclude *.py include *.rst include *.txt include .coveragerc include pyproject.toml recursive-include specifications *.txt recursive-include src/zc *.test recursive-include src/zc *.txt recursive-include src/zc *.py prune doc prune old-tutorial prune zc.recipe.egg_ recursive-exclude news * exclude .deepsource.toml exclude news ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1743675672.4110773 zc_buildout-4.1.6/PKG-INFO0000644000076500000240000007331214773460430014567 0ustar00mauritsstaffMetadata-Version: 2.4 Name: zc.buildout Version: 4.1.6 Summary: System for managing development buildouts Home-page: http://buildout.org Author: Jim Fulton Author-email: jim@zope.com License: ZPL 2.1 Keywords: development build Classifier: Development Status :: 6 - Mature Classifier: Intended Audience :: Developers Classifier: License :: OSI Approved :: Zope Public License Classifier: Programming Language :: Python Classifier: Programming Language :: Python :: 3.9 Classifier: Programming Language :: Python :: 3.10 Classifier: Programming Language :: Python :: 3.11 Classifier: Programming Language :: Python :: 3.12 Classifier: Programming Language :: Python :: 3.13 Classifier: Topic :: Software Development :: Build Tools Classifier: Topic :: Software Development :: Libraries :: Python Modules Requires-Python: >=3.9 License-File: LICENSE.txt Requires-Dist: setuptools>=49.0.0 Requires-Dist: packaging Requires-Dist: pip Requires-Dist: wheel Provides-Extra: test Requires-Dist: zope.testing; extra == "test" Requires-Dist: manuel; extra == "test" Requires-Dist: bobo==2.3.0; extra == "test" Requires-Dist: zdaemon; extra == "test" Requires-Dist: zc.zdaemonrecipe; extra == "test" Requires-Dist: zc.recipe.deployment; extra == "test" Dynamic: author Dynamic: author-email Dynamic: classifier Dynamic: description Dynamic: home-page Dynamic: keywords Dynamic: license Dynamic: license-file Dynamic: provides-extra Dynamic: requires-dist Dynamic: requires-python Dynamic: summary ******** Buildout ******** .. image:: https://github.com/buildout/buildout/actions/workflows/run-tests.yml/badge.svg :alt: GHA tests report :target: https://github.com/buildout/buildout/actions/workflows/run-tests.yml Buildout is a project designed to solve 2 problems: 1. Application-centric assembly and deployment *Assembly* runs the gamut from stitching together libraries to create a running program, to production deployment configuration of applications, and associated systems and tools (e.g. run-control scripts, cron jobs, logs, service registration, etc.). Buildout might be confused with build tools like make or ant, but it is a little higher level and might invoke systems like make or ant to get its work done. Buildout might be confused with systems like puppet or chef, but it is more application focused. Systems like puppet or chef might use buildout to get their work done. Buildout is also somewhat Python-centric, even though it can be used to assemble and deploy non-python applications. It has some special features for assembling Python programs. It's scripted with Python, unlike, say puppet or chef, which are scripted with Ruby. 2. Repeatable assembly of programs from Python software distributions Buildout puts great effort toward making program assembly a highly repeatable process, whether in a very open-ended development mode, where dependency versions aren't locked down, or in a deployment environment where dependency versions are fully specified. You should be able to check buildout into a VCS and later check it out. Two checkouts built at the same time in the same environment should always give the same result, regardless of their history. Among other things, after a buildout, all dependencies should be at the most recent version consistent with any version specifications expressed in the buildout. Buildout supports applications consisting of multiple programs, with different programs in an application free to use different versions of Python distributions. This is in contrast with a Python installation (real or virtual), where, for any given distribution, there can only be one installed. To learn more about buildout, including how to use it, see https://www.buildout.org/. Change History ************** .. You should *NOT* be adding new change log entries to this file. You should create a file in the news directory instead. For helpful instructions, please see: https://github.com/buildout/buildout/blob/master/doc/ADD-A-NEWS-ITEM.rst .. towncrier release notes start 4.1.6 (2025-04-03) ------------------ Tests - While creating sample packages for testing, mostly create wheels instead of eggs. For the sample source distributions, create ``tar.gz`` instead of ``zip`` files. Then our package index for testing is more like the actual PyPI. [maurits] (#675) 4.1.5 (2025-03-31) ------------------ Bug fixes: - Implement PEP 503: request normalized package url on PyPI servers. [andreclimaco] (#634) - Install ``wheel`` before ``setuptools`` when checking if an upgrade and restart are needed. [maurits] (#691) 4.1.4 (2025-03-07) ------------------ Bug fixes: - If needed, copy and rename wheels before making an egg out of them. This helps for wheels of namespace packages created with ``setuptools`` 75.8.1 or higher. For namespace package we need a dot instead of an underscore in the resulting egg name. [maurits] (#686) 4.1.3 (2025-03-05) ------------------ Bug fixes: - Patch the ``find`` method from ``pkg_resources.WorkingSet``. Let this use the code from ``setuptools`` 75.8.2, if the currently used version is older. This is better at finding installed distributions. But don't patch ``setuptools`` versions older than 61: the new version of the method would give an error there. [maurits] (#682) 4.1.2 (2025-03-05) ------------------ Bug fixes: - Fix error finding the ``zc.buildout`` distribution when checking if we need to upgrade/restart. This depends on your ``setuptools`` version. [maurits] (#681) 4.1.1 (2025-03-04) ------------------ Bug fixes: - Fix error adding minimum ``zc.buildout`` version as requirement. [maurits] (#679) 4.1 (2025-03-04) ---------------- New features: - In the ``ls`` testing method, add keyword argument ``lowercase_and_sort_output``. The default is False, so no change. When true, as the name says, it sorts the output by lowercase, and prints it lowercase. We need this in one test because with ``setuptools`` 75.8.1 we no longer have a filename ``MIXEDCASE-0.5-pyN.N.egg``, but ``mixedcase-0.5-pyN.N.egg``. [maurits] (#7581) Bug fixes: - When trying to find a distribution for ``package.name``, first try the normalized name (``package_name``). This fixes an error finding entry points for namespace packages. The error is: ``TypeError: ('Expected str, Requirement, or Distribution', None)``. [maurits] (#7581) Development: - Test with latest ``setuptools`` 75.8.2 and with ``pip`` 25.0.1. Note that ``setuptools`` 75.8.1 can be troublesome and should be avoided. [maurits] (#7581) 4.0 (2025-01-30) ---------------- Breaking changes: - Drop Python 3.8 support. Require 3.9 as minimum. (#38) Development: - Test against `setuptools == 75.6.0`. (#671) 4.0.0a1 (2024-10-22) -------------------- Breaking changes: - Add dependency on ``packaging``. This gets rid of ugly compatibility code. [maurits] (#38) - Require ``setuptools >= 49.0.0``. This is the first version that supports PEP 496 environment markers, for example ``demo ==0.1; python_version < '3.9'``. An earlier change had ``setuptools >= 42.0.2``, otherwise we got ImportErrors. Also, since this is higher than 38.2.3, we are sure to have support for wheels. Remove support for ``distribute``, which was probably already broken. [maurits] (#38) - Drop support for Python 2. Require Python 3.8 as minimum. [maurits] (#38) New features: - Support Python 3.12 and 3.13. This only needed a few test fixes. [maurits] (#38) 3.3 (2024-10-17) ---------------- New features: - Allow the ``-I`` option in the Python interpreter wrapper installed by buildout when using the ``zc.recipe.egg`` recipe's `interpreter =` directive. This solves the issue when VSCode calls the designated Python interpreter for a workspace with this option to determine the Python version etc. (`#627 `_) 3.2.0 (2024-09-26) ------------------ New features: - Add config option: ``optional-extends``. This is the same as the ``extends`` option, but then for optional files. The names must be file paths, not URLs. If the path does not exist, it is silently ignored. This is useful for optionally loading a ``local.cfg`` or ``custom.cfg`` with options specific for the developer or the server. [maurits] (`#665 `_) 3.1.1 (2024-09-20) ------------------ Bug fixes: - Fix: a variable defined with initial ``+=`` was undefined and would lead to a corrupted ``.installed.cfg``. Fixes `issue 641 `_. [distributist] - Fix: extends with increments could result in missing values. Buildout processes them in the correct order now and combines them correctly. Fixes `issue 176 `_ and `issue 629 `_. [distributist] (#644) - Fix: Multiple ``+=`` or ``/-=`` in one file would lose assignment in a previous file. Fixes `issue 656 `_. [distributist] 3.1.0 (2024-08-29) ------------------ Breaking changes: - Drop support for Python 3.5. It is unsupported, and testing it is too hard. [maurits] (#35) Bug fixes: - Normalize package names when gathering packages. This should help find all distributions for package ``name.space``, whether they are called ``name.space-1.0.tar.gz`` with a dot or ``name_space-1.0.tar.gz`` with an underscore (created with ``setuptools`` 69.3 or higher). [maurits] (#647) - Fix ImportError: cannot import name ``packaging`` from ``pkg_resources`` with setuptools 70. Done by adding a compatibility module that tries to import `packaging` from several places. Fixes `issue 648 `_. [maurits] (#648) 3.0.1 (2022-11-08) ------------------ Bug fixes: - Fixed import of packaging.markers. [maurits] (#621) 3.0.0 (2022-11-07) ------------------ New features: - Add support for PEP 508 markers in section condition expressions. For example: ``[versions:python_version <= "3.9"]``. [maurits] (#621) Bug fixes: - Command-line 'extends' now works with dirs in file names [gotcha] (cli-extends) - Add support for python311-315 in conditional section expressions. (#311) - Make compatible with pip 22.2+, restoring Requires-Python functionality there. Fixes `issue 613 `_. [maurits] (#613) 3.0.0rc3 (2022-04-07) --------------------- Bug fixes: - Fix `TypeError: dist must be a Distribution instance` due to issue between `setuptools` and `pip`. (#600) 3.0.0rc2 (2022-03-04) --------------------- New features: - add support for PEP496 environment markers (pep496) Bug fixes: - Fix TypeError for missing required `use_deprecated_html5lib` with pip 22. Keep compatible with earlier pip versions. (#598) 3.0.0rc1 (2021-12-16) --------------------- Bug fixes: - Call pip via `python -m pip`. (#569) 3.0.0b5 (2021-11-29) -------------------- Bug fixes: - Fix when c extension implements namespace packages without the corresponding directories. (#589) - Honor command-line buildout:extends (#592) 3.0.0b4 (2021-11-25) -------------------- New features: - Allow to run buildout in FIPS enabled environments. (#570) - Proper error message if extends-cache tries to expand ${section:variable} (#585) Bug fixes: - Forward verbose option to pip (#576) - Check that file top_level.txt exists before opening. Add check for other files as well. (#582) - Return code of pip install subprocess is now properly returned to buildout. (#586) 3.0.0b3 (2021-10-08) -------------------- New features: - Improve warning message when a section contains unused options. (#483) Bug fixes: - Fix support of ``pip>=21.1`` (#567) - Fix confusion when using multiple Python versions and installing packages with C extensions without proper binary wheel available. (#574) Development: - Avoid broken jobs on Travis because of security on PRs (travis-pr) 3.0.0b2 (2021-03-09) -------------------- New features: - Improve error message when a package version is not pinned and `allow-picked-versions = false`. (#481) Bug fixes: - Fix FileNotFoundError when installing eggs with top-level directory without code (like doc). (#556) Development: - Login to docker hub to avoid pull limits (travis) - Initialize towncrier (#519) 3.0.0b1 (2021-03-07) ==================== - Fix issue with combination of `>` specs and `extras` and recent `setuptools`. - Fix issue with incrementing options from `.buildout/default.cfg`. - Support python37, python38 and python39 in conditional section expressions. - Fix bootstrapping for python27 and python35. 3.0.0a2 (2020-05-25) ==================== - Ignore `.git` when computing signature of a recipe develop egg - Warn when the name passed to `zc.recipe.egg:scripts` is not defined in egg entry points. - Show pip warning about Python version only once. - Better patch for ``pkg_resources.Distribution.hashcmp`` performance. 3.0.0a1 (2020-05-17) ==================== - Scripts: ensure eggs are inserted before ``site-packages`` in ``sys.path``. - Fix forever loop when changing ``zc.buildout`` version via ``buildout``. - Add support for ``Requires-Python`` metadata. Fragile monkeypatch that relies on ``pip._internal``. Emits a warning when support is disabled due to changes in ``pip``. - Use ``pip install`` instead of deprecated ``setuptools.easy_install``. - Patch ``pkg_resources.Distribution`` to make install of unpinned versions quicker. Most obvious with ``setuptools``. 2.13.3 (2020-02-11) =================== - Fix DeprecationWarning about MutableMapping. (`#484 `_) 2.13.2 (2019-07-03) =================== - Fixed DeprecationWarning on python 3.7: "'U' mode is deprecated". 2.13.1 (2019-01-29) =================== - Documentation update for the new ``buildout query`` command. 2.13.0 (2019-01-17) =================== - Get information about the configuration with new command ``buildout query``. 2.12.2 (2018-09-04) =================== - Upon an error, buildout exits with a non-zero exit code. This now also works when running with ``-D``. - Fixed most 'Deprecation' and 'Resource' warnings. 2.12.1 (2018-07-02) =================== - zc.buildout now explicitly requests zc.recipe.egg >=2.0.6 now. 2.12.0 (2018-07-02) =================== - Add a new buildout option ``allow-unknown-extras`` to enable installing requirements that specify extras that do not exist. This needs a corresponding update to zc.recipe.egg. See `issue 457 `_. zc.recipe.egg has been updated to 2.0.6 for this change. 2.11.5 (2018-06-19) =================== - Fix for `issue 295 `_. On windows, deletion of temporary egg files is more robust now. 2.11.4 (2018-05-14) =================== - Fix for `issue 451 `_: distributions with a version number that normalizes to a shorter version number (3.3.0 to 3.3, for instance) can be installed now. 2.11.3 (2018-04-13) =================== - Update to use the new PyPI at https://pypi.org/. 2.11.2 (2018-03-19) =================== - Fix for the #442 issue: AttributeError on ``pkg_resources.SetuptoolsVersion``. 2.11.1 (2018-03-01) =================== - Made upgrade check more robust. When using extensions, the improvement introduced in 2.11 could prevent buildout from restarting itself when it upgraded setuptools. 2.11.0 (2018-01-21) =================== - Installed packages are added to the working set immediately. This helps in some corner cases that occur when system packages have versions that conflict with our specified versions. 2.10.0 (2017-12-04) =================== - Setuptools 38.2.0 started supporting wheels. Through setuptools, buildout now also supports wheels! You need at least version 38.2.3 to get proper namespace support. This setuptools change interfered with buildout's recent support for `buildout.wheel `_, resulting in a sudden "Wheels are not supported" error message (see `issue 435 `_). Fixed by making setuptools the default, though you can still use the buildout.wheel if you want. 2.9.6 (2017-12-01) ================== - Fixed: could not install eggs when sdist file name and package name had different case. 2.9.5 (2017-09-22) ================== - Use HTTPS for PyPI's index. PyPI redirects HTTP to HTTPS by default now so using HTTPS directly avoids the potential for that redirect being modified in flight. 2.9.4 (2017-06-20) ================== - Sort the distributions used to compute ``__buildout_signature__`` to ensure reproducibility under Python 3 or under Python 2 when ``-R`` is used on ``PYTHONHASHSEED`` is set to ``random``. Fixes `issue 392 `_. **NOTE**: This may cause existing ``.installed.cfg`` to be considered outdated and lead to parts being reinstalled spuriously under Python 2. - Add support code for doctests to be able to easily measure code coverage. See `issue 397 `_. 2.9.3 (2017-03-30) ================== - Add more verbosity to ``annotate`` results with ``-v`` - Select one or more sections with arguments after ``buildout annotate``. 2.9.2 (2017-03-06) ================== - Fixed: We unnecessarily used a function from newer versions of setuptools that caused problems when older setuptools or pkg_resources installs were present (as in travis.ci). 2.9.1 (2017-03-06) ================== - Fixed a minor packaging bug that broke the PyPI page. 2.9.0 (2017-03-06) ================== - Added new syntax to explicitly declare that a part depends on other part. See http://docs.buildout.org/en/latest/topics/implicit-parts.html - Internal refactoring to work with `buildout.wheel `_. - Fixed a bugs in ``zc.buildout.testing.Buildout``. It was loading user-default configuration. It didn't support calling the ``created`` method on its sections. - Fixed a bug (windows, py 3.4) When processing metadata on "old-style" distutils scripts, .exe stubs appeared in ``metadata_listdir``, in turn reading those burped with ``UnicodeDecodeError``. Skipping .exe stubs now. 2.8.0 (2017-02-13) ================== - Added a hook to enable a soon-to-be-released buildout extension to provide wheel support. 2.7.1 (2017-01-31) ================== - Fixed a bug introduced in 2.6.0: zc.buildout and its dependeoncies were reported as picked even when their versions were fixed in a ``versions`` section. Worse, when the ``update-versions-file`` option was used, the ``versions`` section was updated needlessly on every run. 2.7.0 (2017-01-30) ================== - Added a buildout option, ``abi-tag-eggs`` that, when true, causes the `ABI tag `_ for the buildout environment to be added to the eggs directory name. This is useful when switching Python implementations (e.g. CPython vs PyPI or debug builds vs regular builds), especially when environment differences aren't reflected in egg names. It also has the side benefit of making eggs directories smaller, because eggs for different Python versions are in different directories. 2.6.0 (2017-01-29) ================== - Updated to work with the latest setuptools. - Added (verified) Python 3.6 support. 2.5.3 (2016-09-05) ================== - After a dist is fetched and put into its final place, compile its python files. No longer wait with compiling until all dists are in place. This is related to the change below about not removing an existing egg. [maurits] - Do not remove an existing egg. When installing an egg to a location that already exists, keep the current location (directory or file). This can only happen when the location at first did not exist and this changed during the buildout run. We used to remove the previous location, but this could cause problems when running two buildouts at the same time, when they try to install the same new egg. Fixes #307. [maurits] - In ``zc.buildout.testing.system``, set ``TERM=dumb`` in the environment. This avoids invisible control characters popping up in some terminals, like ``xterm``. Note that this may affect tests by buildout recipes. [maurits] - Removed Python 2.6 and 3.2 support. [do3cc] 2.5.2 (2016-06-07) ================== - Fixed ``-=`` and ``+=`` when extending sections. See #161. [puittenbroek] 2.5.1 (2016-04-06) ================== - Fix python 2 for downloading external config files with basic auth in the URL. Fixes #257. 2.5.0 (2015-11-16) ================== - Added more elaborate version and requirement information when there's a version conflict. Previously, you could get a report of a version conflict without information about which dependency requested the conflicing requirement. Now all this information is logged and displayed in case of an error. [reinout] - Dropped 3.2 support (at least in the automatic tests) as setuptools will soon stop supporting it. Added python 3.5 to the automatic tests. [reinout] 2.4.7 (2015-10-29) ================== - Fix for #279. Distutils script detection previously broke on windows with python 3 because it errored on ``.exe`` files. [reinout] 2.4.6 (2015-10-28) ================== - Relative paths are now also correctly generated for the current directory ("develop = ."). [youngking] 2.4.5 (2015-10-14) ================== - More complete fix for #24. Distutils scripts are now also generated for develop eggs. [reinout] 2.4.4 (2015-10-02) ================== - zc.buildout is now also released as a wheel. (Note: buildout itself doesn't support installing wheels yet.) [graingert] 2.4.3 (2015-09-03) ================== - Added nested directory creation support [guyzmo] 2.4.2 (2015-08-26) ================== - If a downloaded config file in the "extends-cache" gets corrupted, buildout now tells you the filename in the cache. Handy for troubleshooting. [reinout] 2.4.1 (2015-08-08) ================== - Check the ``use-dependency-links`` option earlier. This can give a small speed increase. [maurits] - When using python 2, urllib2 is used to work around Python issue 24599, which affects downloading from behind a proxy. [stefano-m] 2.4.0 (2015-07-01) ================== - Buildout no longer breaks on packages that contain a file with a non-ascii filename. Fixes #89 and #148. [reinout] - Undo breakage on Windows machines where ``sys.prefix`` can also be a ``site-packages`` directory: don't remove it from ``sys.path``. See https://github.com/buildout/buildout/issues/217 . - Remove assumption that ``pkg_resources`` is a module (untrue since release of `setuptools 8.3``). See https://github.com/buildout/buildout/issues/227 . - Fix for #212. For certain kinds of conflict errors you'd get an UnpackError when rendering the error message. Instead of a nicely formatted version conflict message. [reinout] - Making sure we use the correct easy_install when setuptools is installed globally. See https://github.com/buildout/buildout/pull/232 and https://github.com/buildout/buildout/pull/222 . [lrowe] - Updated buildout's `travis-ci `_ configuration so that tests run much quicker so that buildout is easier and quicker to develop. [reinout] - Note: zc.recipe.egg has also been updated to 2.0.2 together with this zc.buildout release. Fixed: In ``zc.recipe.egg#custom`` recipe's ``rpath`` support, don't assume path elements are buildout-relative if they start with one of the "special" tokens (e.g., ``$ORIGIN``). See: https://github.com/buildout/buildout/issues/225. [tseaver] - ``download-cache``, ``eggs-directory`` and ``extends-cache`` are now automatically created if their parent directory exists. Also they can be relative directories (relative to the location of the buildout config file that defines them). Also they can now be in the form ``~/subdir``, with the usual convention that the ``~`` char means the home directory of the user running buildout. [lelit] - A new bootstrap.py file is released (version 2015-07-01). - When bootstrapping, the ``develop-eggs/`` directory is first removed. This prevents old left-over ``.egg-link`` files from breaking buildout's careful package collection mechanism. [reinout] - The bootstrap script now accepts ``--to-dir``. Setuptools is installed there. If already available there, it is reused. This can be used to bootstrap buildout without internet access. Similarly, a local ``ez_setup.py`` is used when available instead of it being downloaded. You need setuptools 14.0 or higher for this functionality. [lrowe] - The bootstrap script now uses ``--buildout-version`` instead of ``--version`` to pick a specific buildout version. [reinout] - The bootstrap script now accepts ``--version`` which prints the bootstrap version. This version is the date the bootstrap.py was last changed. A date is handier or less confusing than either tracking zc.buildout's version or having a separate bootstrap version number. [reinout] 2.3.1 (2014-12-16) ================== - Fixed: Buildout merged single-version requirements with version-range requirements in a way that caused it to think there wasn't a single-version requirement. IOW, buildout through that versions were being picked when they weren't. - Suppress spurious (and possibly non-spurious) version-parsing warnings. 2.3.0 (2014-12-14) ================== - Buildout is now compatible with (and requires) setuptools 8. 2.2.5 (2014-11-04) ================== - Improved fix for #198: when bootstrapping with an extension, buildout was too strict on itself, resulting in an inability to upgrade or downgrade its own version. [reinout] - Setuptools must be at 3.3 or higher now. If you use the latest bootstrap from http://downloads.buildout.org/2/bootstrap.py you're all set. [reinout] - Installing *recipes* that themselves have dependencies used to fail with a VersionConflict if such a dependency was installed globally with a lower version. Buildout now ignores the version conflict in those cases and simply installs the correct version. [reinout] 2.2.4 (2014-11-01) ================== - Fix for #198: buildout 2.2.3 caused a version conflict when bootstrapping a buildout with a version pinned to an earlier one. Same version conflict could occur with system-wide installed packages that were newer than the pinned version. [reinout] 2.2.3 (2014-10-30) ================== - Fix #197, Python 3 regression [aclark4life] 2.2.2 (2014-10-30) ================== - Open files for ``exec()`` in universal newlines mode. See https://github.com/buildout/buildout/issues/130 - Add ``BUILDOUT_HOME`` as an alternate way to control how the user default configuration is found. - Close various files when finished writing to them. This avoids ResourceWarnings on Python 3, and better supports doctests under PyPy. - Introduce improved easy_install Install.install function. This is present in 1.5.X and 1.7X but was never merged into 2.X somehow. 2.2.1 (2013-09-05) ================== - ``distutils`` scripts: correct order of operations on ``from ... import`` lines (see https://github.com/buildout/buildout/issues/134). - Add an ``--allow-site-packges`` option to ``bootstrap.py``, defaulting to False. If the value is false, strip any "site packages" (as defined by the ``site`` module) from ``sys.path`` before attempting to import ``setuptools`` / ``pkg_resources``. - Updated the URL used to fetch ``ez_setup.py`` to the official, non-version- pinned version. 2.2.0 (2013-07-05) ================== - Handle both addition and subtraction of elements (+= and -=) on the same key in the same section. Forward-ported from buildout 1.6. - Suppress the useless ``Link to ***BLOCKED*** by --allow-hosts`` error message being emitted by distribute / setuptools. - Extend distutils script generation to support module docstrings and __future__ imports. - Refactored picked versions logic to make it easier to use for plugins. - Use ``get_win_launcher`` API to find Windows launcher (falling back to ``resource_string`` for ``cli.exe``). - Remove ``data_files`` from ``setup.py``: it was installing ``README.txt`` in current directory during installation (merged from 1.x branch). - Switch dependency from ``distribute 0.6.x`` to ``setuptools 0.7.x``. 2.1.0 (2013-03-23) ================== - Meta-recipe support - Conditional sections - Buildout now accepts a ``--version`` command-line option to print its version. Fixed: Builout didn't exit with a non-zero exit status if there was a failure in combination with an upgrade. Fixed: We now fail with an informative error when an old bootstrap script causes buildout 2 to be used with setuptools. Fixed: An error incorrectly suggested that buildout 2 implemented all of the functionality of dumppickedversions. Fixed: Buildout generated bad scripts when no eggs needed to be added to ``sys.path``. Fixed: Buildout didn't honour Unix umask when generating scripts. https://bugs.launchpad.net/zc.buildout/+bug/180705 Fixed: ``update-versions-file`` didn't work unless ``show-picked-versions`` was also set. https://github.com/buildout/buildout/issues/71 2.0.1 (2013-02-16) ================== - Fixed: buildout didn't honor umask settings when creating scripts. - Fix for distutils scripts installation on Python 3, related to ``__pycache__`` directories. - Fixed: encoding data in non-entry-point-based scripts was lost. ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675670.0 zc_buildout-4.1.6/README.rst0000644000076500000240000000444014773460426015162 0ustar00mauritsstaff******** Buildout ******** .. image:: https://github.com/buildout/buildout/actions/workflows/run-tests.yml/badge.svg :alt: GHA tests report :target: https://github.com/buildout/buildout/actions/workflows/run-tests.yml Buildout is a project designed to solve 2 problems: 1. Application-centric assembly and deployment *Assembly* runs the gamut from stitching together libraries to create a running program, to production deployment configuration of applications, and associated systems and tools (e.g. run-control scripts, cron jobs, logs, service registration, etc.). Buildout might be confused with build tools like make or ant, but it is a little higher level and might invoke systems like make or ant to get its work done. Buildout might be confused with systems like puppet or chef, but it is more application focused. Systems like puppet or chef might use buildout to get their work done. Buildout is also somewhat Python-centric, even though it can be used to assemble and deploy non-python applications. It has some special features for assembling Python programs. It's scripted with Python, unlike, say puppet or chef, which are scripted with Ruby. 2. Repeatable assembly of programs from Python software distributions Buildout puts great effort toward making program assembly a highly repeatable process, whether in a very open-ended development mode, where dependency versions aren't locked down, or in a deployment environment where dependency versions are fully specified. You should be able to check buildout into a VCS and later check it out. Two checkouts built at the same time in the same environment should always give the same result, regardless of their history. Among other things, after a buildout, all dependencies should be at the most recent version consistent with any version specifications expressed in the buildout. Buildout supports applications consisting of multiple programs, with different programs in an application free to use different versions of Python distributions. This is in contrast with a Python installation (real or virtual), where, for any given distribution, there can only be one installed. To learn more about buildout, including how to use it, see https://www.buildout.org/. ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675670.0 zc_buildout-4.1.6/dev.py0000644000076500000240000000253114773460426014622 0ustar00mauritsstaff# Create bin/buildout script. # This should be called with python from a virtualenv that has all our # dependencies already installed. from pathlib import Path import build import os import pkg_resources import platform import sys EGG_INFO_PATH = "src/zc.buildout.egg-info" if not os.path.exists(EGG_INFO_PATH): print(f"ERROR: {EGG_INFO_PATH} does not exist.") print("You should run 'python -m build --sdist' (or 'python setup.py egg_info').") sys.exit(1) # The 'bin' directory must exist. os.makedirs('bin', exist_ok=True) # zc.buildout must be importable in the current session. pkg_resources.working_set.add_entry('src') # Now this import should work. # Important note: isort must NOT move this line. import zc.buildout.easy_install # And then Buildout can install its own script. zc.buildout.easy_install.scripts( ['zc.buildout'], pkg_resources.working_set, sys.executable, 'bin' ) if platform.system() == "Windows": buildout_script = Path("bin/buildout.exe") else: buildout_script = Path("bin/buildout") if buildout_script.exists(): print(f"SUCCESS: Generated {buildout_script} script.") if platform.system() != "Windows": # On Windows you get a UnicodeDecodeError that I don't want to debug. print(buildout_script.read_text()) else: print(f"ERROR: Generating {buildout_script} failed.") sys.exit(1) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675670.0 zc_buildout-4.1.6/pyproject.toml0000644000076500000240000000106414773460426016406 0ustar00mauritsstaff[tool.towncrier] filename = "CHANGES.rst" directory = "news/" title_format = "{version} ({project_date})" underlines = ["-", ""] [[tool.towncrier.type]] directory = "breaking" name = "Breaking changes:" showcontent = true [[tool.towncrier.type]] directory = "feature" name = "New features:" showcontent = true [[tool.towncrier.type]] directory = "bugfix" name = "Bug fixes:" showcontent = true [[tool.towncrier.type]] directory = "develop" name = "Development:" showcontent = true [[tool.towncrier.type]] directory = "tests" name = "Tests" showcontent = true ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1743675672.4116666 zc_buildout-4.1.6/setup.cfg0000644000076500000240000000037714773460430015314 0ustar00mauritsstaff[bdist_wheel] universal = 0 [check-manifest] ignore = *.cfg .dockerignore Makefile prepare.sh zc.recipe.egg_/* zc.recipe.egg_ .github/* .github [zest.releaser] create-wheel = yes extra-message = [ci skip] [egg_info] tag_build = tag_date = 0 ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675670.0 zc_buildout-4.1.6/setup.py0000644000076500000240000000461114773460426015205 0ustar00mauritsstaff############################################################################## # # Copyright (c) 2006-2009 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## name = "zc.buildout" version = '4.1.6' import os from setuptools import setup def read(*rnames): with open(os.path.join(os.path.dirname(__file__), *rnames)) as f: return f.read() long_description= read('README.rst') + '\n' + read('CHANGES.rst') entry_points = """ [console_scripts] buildout = %(name)s.buildout:main [zc.buildout] debug = %(name)s.testrecipes:Debug """ % dict(name=name) setup( name = name, version = version, author = "Jim Fulton", author_email = "jim@zope.com", description = "System for managing development buildouts", long_description=long_description, license = "ZPL 2.1", keywords = "development build", url='http://buildout.org', packages = ['zc', 'zc.buildout'], package_dir = {'': 'src'}, python_requires = '>=3.9', namespace_packages = ['zc'], install_requires = [ 'setuptools>=49.0.0', 'packaging', 'pip', 'wheel', ], include_package_data = True, entry_points = entry_points, extras_require = dict( test=['zope.testing', 'manuel', 'bobo ==2.3.0', 'zdaemon', 'zc.zdaemonrecipe', 'zc.recipe.deployment']), zip_safe=False, classifiers = [ 'Development Status :: 6 - Mature', 'Intended Audience :: Developers', 'License :: OSI Approved :: Zope Public License', 'Programming Language :: Python', 'Programming Language :: Python :: 3.9', 'Programming Language :: Python :: 3.10', 'Programming Language :: Python :: 3.11', 'Programming Language :: Python :: 3.12', 'Programming Language :: Python :: 3.13', 'Topic :: Software Development :: Build Tools', 'Topic :: Software Development :: Libraries :: Python Modules', ], ) ././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1743675672.394036 zc_buildout-4.1.6/specifications/0000755000076500000240000000000014773460430016467 5ustar00mauritsstaff././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675670.0 zc_buildout-4.1.6/specifications/README.txt0000644000076500000240000000036314773460426020174 0ustar00mauritsstaffThis directory is (an experimental) place to manage Launchpad specification bodies. Files here are referenced (via svn.zope.org URLs) from the buildout project specifications in Launchpad, https://features.launchpad.net/products/zc.buildout. ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675670.0 zc_buildout-4.1.6/specifications/repeatable.txt0000644000076500000240000000562314773460426021347 0ustar00mauritsstaffRepeatable (taggable) buildouts =============================== It's important to be able to tag a buildout in a software repository in such a way that, months, or even years later, the buildout tag can be checked out and used to construct the same collection of parts, with the same versions. (Note that parts could still behave differently due to changes in parts of the environment, such as system libraries, not controlled by the buildout.) A feature of the buildout is it's use of eggs and the automatic resolution of dependencies. The latest versions of dependencies are automatically downloaded and installed. This is great during development or when using the buildout for casual software development, but it doesn't work very well for reproducing an old buildout. What's needed is some way to, when needed, record information about the versions of eggs (and any other bits) who's versions are determined dynamically. Proposal -------- We'll add a buildout option, create-repeatable. The option will specify a file into which option information should be saved to create a repeatable buildout. The data will be saved in a form that can be used by the buildout or recipes in a later run. To make a tagged buildout, a user would run the buildout with the create-repeatable option set to a file name and then modify the buildout to be extended-by this file. Consider the following example buildout.cfg:: [buildout] parts = foo [foo] recipe = zc.recipe.eggs eggs = foo eek Now assume that: - The current version of foo is 1.1 - Foo depends on bar =, which depends on baz. The current versions of bar and bas are 1.1 and 2.1. - The current version of eek is 1.5 - eek depends on ook, which is as version 1.3. - zc.recipe.egg is at version 1.0b5 If we run the buildout with the command-line option:: buildout:create-repeatable=reapeatable.cfg we'll get a repeatable.cfg file that looks something like:: [foo] recipe = zc.recipe.eggs ==1.0b5 static = true eggs = foo ==1.1 bar ==1.1 baz ==2.1 eek ==1.5 ook ==1.3 The file contains options for the foo part. The buildout software itself added an entry for the recipe that fixes the recipe version at the version used by the buildout. The zc.recipe.eggs recipe added the eggs option that lists the specific releases that were assembled. Finally the buildout.cfg file can be modified to use the repeatable.cfg file:: [buildout] parts = foo extended-by: repeatable.cfg [foo] recipe = zc.recipe.eggs eggs = foo eek When the buildout is run, the options in repeatable.cfg will override the ones in buildout.cfg, providing a repeatable buildout Python API ---------- The recipe API will grow a repeatable method that is called after the install method and is passed a dictionary that a recipe can store option data in. A recipe instance will only be able to provide repeatable data for it's part. ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1743675672.3889039 zc_buildout-4.1.6/src/0000755000076500000240000000000014773460430014253 5ustar00mauritsstaff././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1743675672.394422 zc_buildout-4.1.6/src/zc/0000755000076500000240000000000014773460430014667 5ustar00mauritsstaff././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675670.0 zc_buildout-4.1.6/src/zc/__init__.py0000644000076500000240000000014614773460426017006 0ustar00mauritsstafftry: __import__('pkg_resources').declare_namespace(__name__) except: # bootstrapping pass ././@PaxHeader0000000000000000000000000000003200000000000010210 xustar0026 mtime=1743675672.40138 zc_buildout-4.1.6/src/zc/buildout/0000755000076500000240000000000014773460430016516 5ustar00mauritsstaff././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675670.0 zc_buildout-4.1.6/src/zc/buildout/__init__.py0000644000076500000240000000307514773460426020641 0ustar00mauritsstaff############################################################################## # # Copyright (c) 2006 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Buildout package """ # do not change the import order # deleting the spec_for_pip hack needs to be done before importing pip # see https://github.com/pypa/pip/issues/8761 to understand # the reason for the hack. # I think it is reasonable to assume we will not run into the race. import setuptools try: from _distutils_hack import DistutilsMetaFinder if hasattr(DistutilsMetaFinder, 'spec_for_pip'): del DistutilsMetaFinder.spec_for_pip except ImportError: pass import pip # NOQA import warnings from pkg_resources import PkgResourcesDeprecationWarning warnings.filterwarnings('ignore', category=PkgResourcesDeprecationWarning) warnings.filterwarnings('ignore', message='Setuptools is replacing distutils.') import sys import zc.buildout.patches # NOQA WINDOWS = sys.platform.startswith('win') class UserError(Exception): """Errors made by a user """ def __str__(self): return " ".join(map(str, self.args)) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675670.0 zc_buildout-4.1.6/src/zc/buildout/buildout.py0000644000076500000240000025330214773460426020731 0ustar00mauritsstaff############################################################################## # # Copyright (c) 2005-2009 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Buildout main script """ from collections.abc import MutableMapping as DictMixin from functools import partial from hashlib import md5 as md5_original from packaging import utils as packaging_utils from zc.buildout.rmtree import rmtree import zc.buildout.easy_install import zc.buildout.configparser import copy import datetime import distutils.errors import glob import importlib import inspect import itertools import logging import os import pkg_resources import re import shutil import subprocess import sys import tempfile import zc.buildout import zc.buildout.download try: hashed = md5_original(b'test') md5 = md5_original except ValueError: md5 = partial(md5_original, usedforsecurity=False) def command(method): method.buildout_command = True return method def commands(cls): for name, method in cls.__dict__.items(): if hasattr(method, "buildout_command"): cls.COMMANDS.add(name) return cls def _print_options(sep=' ', end='\n', file=None): return sep, end, file def print_(*args, **kw): sep, end, file = _print_options(**kw) if file is None: file = sys.stdout file.write(sep.join(map(str, args))+end) realpath = zc.buildout.easy_install.realpath _isurl = re.compile('([a-zA-Z0-9+.-]+)://').match class MissingOption(zc.buildout.UserError, KeyError): """A required option was missing. """ class MissingSection(zc.buildout.UserError, KeyError): """A required section is missing. """ def __str__(self): return "The referenced section, %r, was not defined." % self.args[0] def _annotate_section(section, source): for key in section: section[key] = SectionKey(section[key], source) return section class SectionKey(object): def __init__(self, value, source): self.history = [] self.value = value self.addToHistory("SET", value, source) @property def source(self): return self.history[-1].source def overrideValue(self, sectionkey): self.value = sectionkey.value if sectionkey.history[-1].operation not in ['ADD', 'REMOVE']: self.addToHistory("OVERRIDE", sectionkey.value, sectionkey.source) else: self.history = copy.deepcopy(sectionkey.history) def setDirectory(self, value): self.value = value self.addToHistory("DIRECTORY", value, self.source) def addToValue(self, added, source): subvalues = self.value.split('\n') + added.split('\n') self.value = "\n".join(subvalues) self.addToHistory("ADD", added, source) def removeFromValue(self, removed, source): subvalues = [ v for v in self.value.split('\n') if v not in removed.split('\n') ] self.value = "\n".join(subvalues) self.addToHistory("REMOVE", removed, source) def addToHistory(self, operation, value, source): item = HistoryItem(operation, value, source) self.history.append(item) def printAll(self, key, basedir, verbose): self.printKeyAndValue(key) if verbose: self.printVerbose(basedir) else: self.printTerse(basedir) def printKeyAndValue(self, key): lines = self.value.splitlines() if len(lines) <= 1: args = [key, "="] if self.value: args.append(" ") args.append(self.value) print_(*args, sep='') else: print_(key, "= ", lines[0], sep='') for line in lines[1:]: print_(line) def printVerbose(self, basedir): print_() for item in reversed(self.history): item.printAll(basedir) print_() def printTerse(self, basedir): toprint = [] history = copy.deepcopy(self.history) while history: next = history.pop() if next.operation in ["ADD", "REMOVE"]: next.printShort(toprint, basedir) else: next.printShort(toprint, basedir) break for line in reversed(toprint): if line.strip(): print_(line) def __repr__(self): return "" % ( " ".join(self.value.split('\n')), self.source) class HistoryItem(object): def __init__(self, operation, value, source): self.operation = operation self.value = value self.source = source def printShort(self, toprint, basedir): source = self.source_for_human(basedir) if self.operation in ["OVERRIDE", "SET", "DIRECTORY"]: toprint.append(" " + source) elif self.operation == "ADD": toprint.append("+= " + source) elif self.operation == "REMOVE": toprint.append("-= " + source) def printOperation(self): lines = self.value.splitlines() if len(lines) <= 1: print_(" ", self.operation, "VALUE =", self.value) else: print_(" ", self.operation, "VALUE =") for line in lines: print_(" ", " ", line) def printSource(self, basedir): if self.source in ( 'DEFAULT_VALUE', 'COMPUTED_VALUE', 'COMMAND_LINE_VALUE' ): prefix = "AS" else: prefix = "IN" print_(" ", prefix, self.source_for_human(basedir)) def source_for_human(self, basedir): if self.source.startswith(basedir): return os.path.relpath(self.source, basedir) else: return self.source def printAll(self, basedir): self.printSource(basedir) self.printOperation() def __repr__(self): return "" % ( self.operation, " ".join(self.value.split('\n')), self.source) def _annotate(data, note): for key in data: data[key] = _annotate_section(data[key], note) return data def _print_annotate(data, verbose, chosen_sections, basedir): sections = list(data.keys()) sections.sort() print_() print_("Annotated sections") print_("="*len("Annotated sections")) for section in sections: if (not chosen_sections) or (section in chosen_sections): print_() print_('[%s]' % section) keys = list(data[section].keys()) keys.sort() for key in keys: sectionkey = data[section][key] sectionkey.printAll(key, basedir, verbose) def _unannotate_section(section): return {key: entry.value for key, entry in section.items()} def _unannotate(data): return {key: _unannotate_section(section) for key, section in data.items()} def _format_picked_versions(picked_versions, required_by): output = ['[versions]'] required_output = [] for dist_, version in picked_versions: if dist_ in required_by: required_output.append('') required_output.append('# Required by:') for req_ in sorted(required_by[dist_]): required_output.append('# '+req_) target = required_output else: target = output target.append("%s = %s" % (dist_, version)) output.extend(required_output) return output _buildout_default_options = _annotate_section({ 'allow-hosts': '*', 'allow-picked-versions': 'true', 'bin-directory': 'bin', 'develop-eggs-directory': 'develop-eggs', 'eggs-directory': 'eggs', 'executable': sys.executable, 'find-links': '', 'install-from-cache': 'false', 'installed': '.installed.cfg', 'log-format': '', 'log-level': 'INFO', 'newest': 'true', 'offline': 'false', 'parts-directory': 'parts', 'prefer-final': 'true', 'python': 'buildout', 'show-picked-versions': 'false', 'socket-timeout': '', 'update-versions-file': '', 'use-dependency-links': 'true', 'allow-unknown-extras': 'false', }, 'DEFAULT_VALUE') def _get_user_config(): buildout_home = os.path.join(os.path.expanduser('~'), '.buildout') buildout_home = os.environ.get('BUILDOUT_HOME', buildout_home) return os.path.join(buildout_home, 'default.cfg') @commands class Buildout(DictMixin): COMMANDS = set() def __init__(self, config_file, cloptions, use_user_defaults=True, command=None, args=()): __doing__ = 'Initializing.' # default options _buildout_default_options_copy = copy.deepcopy( _buildout_default_options) data = dict(buildout=_buildout_default_options_copy) self._buildout_dir = os.getcwd() if config_file and not _isurl(config_file): config_file = os.path.abspath(config_file) if not os.path.exists(config_file): if command == 'init': self._init_config(config_file, args) elif command == 'setup': # Sigh. This model of a buildout instance # with methods is breaking down. :( config_file = None data['buildout']['directory'] = SectionKey( '.', 'COMPUTED_VALUE') else: raise zc.buildout.UserError( "Couldn't open %s" % config_file) elif command == 'init': raise zc.buildout.UserError( "%r already exists." % config_file) if config_file: data['buildout']['directory'] = SectionKey( os.path.dirname(config_file), 'COMPUTED_VALUE') cloptions = dict( (section, dict((option, SectionKey(value, 'COMMAND_LINE_VALUE')) for (_, option, value) in v)) for (section, v) in itertools.groupby(sorted(cloptions), lambda v: v[0]) ) override = copy.deepcopy(cloptions.get('buildout', {})) # load user defaults, which override defaults user_config = _get_user_config() if use_user_defaults and os.path.exists(user_config): download_options = data['buildout'] user_defaults, _ = _open( os.path.dirname(user_config), user_config, [], download_options, override, set(), {} ) for_download_options = _update(data, user_defaults) else: user_defaults = {} for_download_options = copy.deepcopy(data) # load configuration files if config_file: download_options = for_download_options['buildout'] cfg_data, _ = _open( os.path.dirname(config_file), config_file, [], download_options, override, set(), user_defaults ) data = _update(data, cfg_data) # extends from command-line if 'buildout' in cloptions: cl_extends = cloptions['buildout'].pop('extends', None) if cl_extends: for extends in cl_extends.value.split(): download_options = for_download_options['buildout'] cfg_data, _ = _open( os.path.dirname(extends), os.path.basename(extends), [], download_options, override, set(), user_defaults ) data = _update(data, cfg_data) # apply command-line options data = _update(data, cloptions) # Set up versions section, if necessary if 'versions' not in data['buildout']: data['buildout']['versions'] = SectionKey( 'versions', 'DEFAULT_VALUE') if 'versions' not in data: data['versions'] = {} # Default versions: versions_section_name = data['buildout']['versions'].value if versions_section_name: versions = data[versions_section_name] else: versions = {} if 'zc.buildout' not in versions: # Prevent downgrading of zc.buildout itself due to prefer-final. ws = pkg_resources.working_set dist = ws.find( pkg_resources.Requirement.parse('zc-buildout') ) if dist is None: # older setuptools dist = ws.find( pkg_resources.Requirement.parse('zc.buildout') ) if dist is None: # This would be really strange, but I prefer an explicit # failure here over an unclear error later. raise ValueError( "Could not find distribution for zc.buildout in working set." ) minimum = dist.version versions['zc.buildout'] = SectionKey(f'>={minimum}', 'DEFAULT_VALUE') if 'zc.recipe.egg' not in versions: # zc.buildout and zc.recipe egg are closely linked, but zc.buildout # does NOT depend on it: we do not want to add it to our # install_requires. (One could debate why, although one answer # would be to avoid a circular dependency. Maybe we could merge them, # as I see no use case for Buildout without recipes. But we would # need to update the zc.recipe.egg test setup first.) # # Anyway: we use a different way to set a minimum version. # Originally (in 2013, zc.buildout 2.0.0b1) we made sure # zc.recipe.egg>=2.0.0a3 was pinned, mostly to avoid problems # when prefer-final is true. # Later (in 2018, zc.buildout 2.12.1) we updated the minimum version # to 2.0.6, to avoid a KeyError: 'allow-unknown-extras'. # See https://github.com/buildout/buildout/pull/461 # I wonder if we really need a minimum version, as older versions # are unlikely to even be installable by supported Python versions. # But if we ever really need a more recent minimum version, # it is easy to update a version here. versions['zc.recipe.egg'] = SectionKey('>=2.0.6', 'DEFAULT_VALUE') # Absolutize some particular directory, handling also the ~/foo form, # and considering the location of the configuration file that generated # the setting as the base path, falling back to the main configuration # file location for name in ('download-cache', 'eggs-directory', 'extends-cache'): if name in data['buildout']: sectionkey = data['buildout'][name] origdir = sectionkey.value src = sectionkey.source if '${' in origdir: continue if not os.path.isabs(origdir): if src in ('DEFAULT_VALUE', 'COMPUTED_VALUE', 'COMMAND_LINE_VALUE'): if 'directory' in data['buildout']: basedir = data['buildout']['directory'].value else: basedir = self._buildout_dir else: if _isurl(src): raise zc.buildout.UserError( 'Setting "%s" to a non absolute location ("%s") ' 'within a\n' 'remote configuration file ("%s") is ambiguous.' % ( name, origdir, src)) basedir = os.path.dirname(src) absdir = os.path.expanduser(origdir) if not os.path.isabs(absdir): absdir = os.path.join(basedir, absdir) absdir = os.path.abspath(absdir) sectionkey.setDirectory(absdir) self._annotated = copy.deepcopy(data) self._raw = _unannotate(data) self._data = {} self._parts = [] # provide some defaults before options are parsed # because while parsing options those attributes might be # used already (Gottfried Ganssauge) buildout_section = self._raw['buildout'] # Try to make sure we have absolute paths for standard # directories. We do this before doing substitutions, in case # a one of these gets read by another section. If any # variable references are used though, we leave it as is in # _buildout_path. if 'directory' in buildout_section: self._buildout_dir = buildout_section['directory'] for name in ('bin', 'parts', 'eggs', 'develop-eggs'): d = self._buildout_path(buildout_section[name+'-directory']) buildout_section[name+'-directory'] = d # Attributes on this buildout object shouldn't be used by # recipes in their __init__. It can cause bugs, because the # recipes will be instantiated below (``options = self['buildout']``) # before this has completed initializing. These attributes are # left behind for legacy support but recipe authors should # beware of using them. A better practice is for a recipe to # use the buildout['buildout'] options. links = buildout_section['find-links'] self._links = links and links.split() or () allow_hosts = buildout_section['allow-hosts'].split('\n') self._allow_hosts = tuple([host.strip() for host in allow_hosts if host.strip() != '']) self._logger = logging.getLogger('zc.buildout') self.offline = bool_option(buildout_section, 'offline') self.newest = ((not self.offline) and bool_option(buildout_section, 'newest') ) ################################################################## ## WARNING!!! ## ALL ATTRIBUTES MUST HAVE REASONABLE DEFAULTS AT THIS POINT ## OTHERWISE ATTRIBUTEERRORS MIGHT HAPPEN ANY TIME FROM RECIPES. ## RECIPES SHOULD GENERALLY USE buildout['buildout'] OPTIONS, NOT ## BUILDOUT ATTRIBUTES. ################################################################## # initialize some attrs and buildout directories. options = self['buildout'] # now reinitialize links = options.get('find-links', '') self._links = links and links.split() or () allow_hosts = options['allow-hosts'].split('\n') self._allow_hosts = tuple([host.strip() for host in allow_hosts if host.strip() != '']) self._buildout_dir = options['directory'] # Make sure we have absolute paths for standard directories. We do this # a second time here in case someone overrode these in their configs. for name in ('bin', 'parts', 'eggs', 'develop-eggs'): d = self._buildout_path(options[name+'-directory']) options[name+'-directory'] = d if options['installed']: options['installed'] = os.path.join(options['directory'], options['installed']) self._setup_logging() self._setup_socket_timeout() # finish w versions if versions_section_name: # refetching section name just to avoid a warning versions = self[versions_section_name] else: # remove annotations versions = dict((k, v.value) for (k, v) in versions.items()) options['versions'] # refetching section name just to avoid a warning self.versions = versions zc.buildout.easy_install.default_versions(versions) zc.buildout.easy_install.prefer_final( bool_option(options, 'prefer-final')) zc.buildout.easy_install.use_dependency_links( bool_option(options, 'use-dependency-links')) zc.buildout.easy_install.allow_picked_versions( bool_option(options, 'allow-picked-versions')) self.show_picked_versions = bool_option(options, 'show-picked-versions') self.update_versions_file = options['update-versions-file'] zc.buildout.easy_install.store_required_by(self.show_picked_versions or self.update_versions_file) download_cache = options.get('download-cache') extends_cache = options.get('extends-cache') if bool_option(options, 'abi-tag-eggs', 'false'): from zc.buildout.pep425tags import get_abi_tag options['eggs-directory'] = os.path.join( options['eggs-directory'], get_abi_tag()) eggs_cache = options.get('eggs-directory') for cache in [download_cache, extends_cache, eggs_cache]: if cache: cache = os.path.join(options['directory'], cache) if not os.path.exists(cache): self._logger.info('Creating directory %r.', cache) os.makedirs(cache) if download_cache: # Actually, we want to use a subdirectory in there called 'dist'. download_cache = os.path.join(download_cache, 'dist') if not os.path.exists(download_cache): os.mkdir(download_cache) zc.buildout.easy_install.download_cache(download_cache) if bool_option(options, 'install-from-cache'): if self.offline: raise zc.buildout.UserError( "install-from-cache can't be used with offline mode.\n" "Nothing is installed, even from cache, in offline\n" "mode, which might better be called 'no-install mode'.\n" ) zc.buildout.easy_install.install_from_cache(True) # "Use" each of the defaults so they aren't reported as unused options. for name in _buildout_default_options: options[name] os.chdir(options['directory']) def _buildout_path(self, name): if '${' in name: return name return os.path.join(self._buildout_dir, name) @command def bootstrap(self, args): __doing__ = 'Bootstrapping.' if os.path.exists(self['buildout']['develop-eggs-directory']): if os.path.isdir(self['buildout']['develop-eggs-directory']): rmtree(self['buildout']['develop-eggs-directory']) self._logger.debug( "Removed existing develop-eggs directory") self._setup_directories() # Now copy buildout and setuptools eggs, and record destination eggs: entries = [] for dist in zc.buildout.easy_install.buildout_and_setuptools_dists: if dist.precedence == pkg_resources.DEVELOP_DIST: dest = os.path.join(self['buildout']['develop-eggs-directory'], dist.key + '.egg-link') with open(dest, 'w') as fh: fh.write(dist.location) entries.append(dist.location) else: dest = os.path.join(self['buildout']['eggs-directory'], os.path.basename(dist.location)) entries.append(dest) if not os.path.exists(dest): if os.path.isdir(dist.location): shutil.copytree(dist.location, dest) else: shutil.copy2(dist.location, dest) # Create buildout script ws = pkg_resources.WorkingSet(entries) ws.require('zc.buildout') options = self['buildout'] eggs_dir = options['eggs-directory'] develop_eggs_dir = options['develop-eggs-directory'] ws = zc.buildout.easy_install.sort_working_set( ws, eggs_dir=eggs_dir, develop_eggs_dir=develop_eggs_dir ) zc.buildout.easy_install.scripts( ['zc.buildout'], ws, sys.executable, options['bin-directory'], relative_paths = ( bool_option(options, 'relative-paths', False) and options['directory'] or ''), ) def _init_config(self, config_file, args): print_('Creating %r.' % config_file) f = open(config_file, 'w') sep = re.compile(r'[\\/]') if args: eggs = '\n '.join(a for a in args if not sep.search(a)) sepsub = os.path.sep == '/' and '/' or re.escape(os.path.sep) paths = '\n '.join( sep.sub(sepsub, a) for a in args if sep.search(a)) f.write('[buildout]\n' 'parts = py\n' '\n' '[py]\n' 'recipe = zc.recipe.egg\n' 'interpreter = py\n' 'eggs =\n' ) if eggs: f.write(' %s\n' % eggs) if paths: f.write('extra-paths =\n %s\n' % paths) for p in [a for a in args if sep.search(a)]: if not os.path.exists(p): os.mkdir(p) else: f.write('[buildout]\nparts =\n') f.close() @command def init(self, args): self.bootstrap(()) if args: self.install(()) @command def install(self, install_args): __doing__ = 'Installing.' self._load_extensions() self._setup_directories() # Add develop-eggs directory to path so that it gets searched # for eggs: sys.path.insert(0, self['buildout']['develop-eggs-directory']) # Check for updates. This could cause the process to be restarted self._maybe_upgrade() # load installed data (installed_part_options, installed_exists )= self._read_installed_part_options() # Remove old develop eggs self._uninstall( installed_part_options['buildout'].get( 'installed_develop_eggs', '') ) # Build develop eggs installed_develop_eggs = self._develop() installed_part_options['buildout']['installed_develop_eggs' ] = installed_develop_eggs if installed_exists: self._update_installed( installed_develop_eggs=installed_develop_eggs) # get configured and installed part lists conf_parts = self['buildout']['parts'] conf_parts = conf_parts and conf_parts.split() or [] installed_parts = installed_part_options['buildout']['parts'] installed_parts = installed_parts and installed_parts.split() or [] if install_args: install_parts = install_args uninstall_missing = False else: install_parts = conf_parts uninstall_missing = True # load and initialize recipes [self[part]['recipe'] for part in install_parts] if not install_args: install_parts = self._parts if self._log_level < logging.DEBUG: sections = list(self) sections.sort() print_() print_('Configuration data:') for section in sorted(self._data): _save_options(section, self[section], sys.stdout) print_() # compute new part recipe signatures self._compute_part_signatures(install_parts) # uninstall parts that are no-longer used or who's configs # have changed for part in reversed(installed_parts): if part in install_parts: old_options = installed_part_options[part].copy() installed_files = old_options.pop('__buildout_installed__') new_options = self.get(part) if old_options == new_options: # The options are the same, but are all of the # installed files still there? If not, we should # reinstall. if not installed_files: continue for f in installed_files.split('\n'): if not os.path.exists(self._buildout_path(f)): break else: continue # output debugging info if self._logger.getEffectiveLevel() < logging.DEBUG: for k in old_options: if k not in new_options: self._logger.debug("Part %s, dropped option %s.", part, k) elif old_options[k] != new_options[k]: self._logger.debug( "Part %s, option %s changed:\n%r != %r", part, k, new_options[k], old_options[k], ) for k in new_options: if k not in old_options: self._logger.debug("Part %s, new option %s.", part, k) elif not uninstall_missing: continue self._uninstall_part(part, installed_part_options) installed_parts = [p for p in installed_parts if p != part] if installed_exists: self._update_installed(parts=' '.join(installed_parts)) # Check for unused buildout options: _check_for_unused_options_in_section(self, 'buildout') # install new parts for part in install_parts: signature = self[part].pop('__buildout_signature__') saved_options = self[part].copy() recipe = self[part].recipe if part in installed_parts: # update need_to_save_installed = False __doing__ = 'Updating %s.', part self._logger.info(*__doing__) old_options = installed_part_options[part] old_installed_files = old_options['__buildout_installed__'] try: update = recipe.update except AttributeError: update = recipe.install self._logger.warning( "The recipe for %s doesn't define an update " "method. Using its install method.", part) try: installed_files = self[part]._call(update) except: installed_parts.remove(part) self._uninstall(old_installed_files) if installed_exists: self._update_installed( parts=' '.join(installed_parts)) raise old_installed_files = old_installed_files.split('\n') if installed_files is None: installed_files = old_installed_files else: if isinstance(installed_files, str): installed_files = [installed_files] else: installed_files = list(installed_files) need_to_save_installed = [ p for p in installed_files if p not in old_installed_files] if need_to_save_installed: installed_files = (old_installed_files + need_to_save_installed) else: # install need_to_save_installed = True __doing__ = 'Installing %s.', part self._logger.info(*__doing__) installed_files = self[part]._call(recipe.install) if installed_files is None: self._logger.warning( "The %s install returned None. A path or " "iterable os paths should be returned.", part) installed_files = () elif isinstance(installed_files, str): installed_files = [installed_files] else: installed_files = list(installed_files) installed_part_options[part] = saved_options saved_options['__buildout_installed__' ] = '\n'.join(installed_files) saved_options['__buildout_signature__'] = signature installed_parts = [p for p in installed_parts if p != part] installed_parts.append(part) _check_for_unused_options_in_section(self, part) if need_to_save_installed: installed_part_options['buildout']['parts'] = ( ' '.join(installed_parts)) self._save_installed_options(installed_part_options) installed_exists = True else: assert installed_exists self._update_installed(parts=' '.join(installed_parts)) if installed_develop_eggs: if not installed_exists: self._save_installed_options(installed_part_options) elif (not installed_parts) and installed_exists: os.remove(self['buildout']['installed']) if self.show_picked_versions or self.update_versions_file: self._print_picked_versions() self._unload_extensions() def _update_installed(self, **buildout_options): installed = self['buildout']['installed'] f = open(installed, 'a') f.write('\n[buildout]\n') for option, value in list(buildout_options.items()): _save_option(option, value, f) f.close() def _uninstall_part(self, part, installed_part_options): # uninstall part __doing__ = 'Uninstalling %s.', part self._logger.info(*__doing__) # run uninstall recipe recipe, entry = _recipe(installed_part_options[part]) try: uninstaller = _install_and_load( recipe, 'zc.buildout.uninstall', entry, self) self._logger.info('Running uninstall recipe.') uninstaller(part, installed_part_options[part]) except (ImportError, pkg_resources.DistributionNotFound): pass # remove created files and directories self._uninstall( installed_part_options[part]['__buildout_installed__']) def _setup_directories(self): __doing__ = 'Setting up buildout directories' # Create buildout directories for name in ('bin', 'parts', 'develop-eggs'): d = self['buildout'][name+'-directory'] if not os.path.exists(d): self._logger.info('Creating directory %r.', d) os.mkdir(d) def _develop(self): """Install sources by running setup.py develop on them """ __doing__ = 'Processing directories listed in the develop option' develop = self['buildout'].get('develop') if not develop: return '' dest = self['buildout']['develop-eggs-directory'] old_files = os.listdir(dest) env = dict(os.environ, PYTHONPATH=zc.buildout.easy_install.setuptools_pythonpath) here = os.getcwd() try: try: for setup in develop.split(): setup = self._buildout_path(setup) files = glob.glob(setup) if not files: self._logger.warning("Couldn't develop %r (not found)", setup) else: files.sort() for setup in files: self._logger.info("Develop: %r", setup) __doing__ = 'Processing develop directory %r.', setup zc.buildout.easy_install.develop(setup, dest) except: # if we had an error, we need to roll back changes, by # removing any files we created. self._sanity_check_develop_eggs_files(dest, old_files) self._uninstall('\n'.join( [os.path.join(dest, f) for f in os.listdir(dest) if f not in old_files ])) raise else: self._sanity_check_develop_eggs_files(dest, old_files) return '\n'.join([os.path.join(dest, f) for f in os.listdir(dest) if f not in old_files ]) finally: os.chdir(here) def _sanity_check_develop_eggs_files(self, dest, old_files): for f in os.listdir(dest): if f in old_files: continue if not (os.path.isfile(os.path.join(dest, f)) and f.endswith('.egg-link')): self._logger.warning( "Unexpected entry, %r, in develop-eggs directory.", f) def _compute_part_signatures(self, parts): # Compute recipe signature and add to options for part in parts: options = self.get(part) if options is None: options = self[part] = {} recipe, entry = _recipe(options) req = pkg_resources.Requirement.parse(recipe) sig = _dists_sig(pkg_resources.working_set.resolve([req])) options['__buildout_signature__'] = ' '.join(sig) def _read_installed_part_options(self): old = self['buildout']['installed'] if old and os.path.isfile(old): fp = open(old) sections = zc.buildout.configparser.parse(fp, old) fp.close() result = {} for section, options in sections.items(): for option, value in options.items(): if '%(' in value: for k, v in _spacey_defaults: value = value.replace(k, v) options[option] = value result[section] = self.Options(self, section, options) return result, True else: return ({'buildout': self.Options(self, 'buildout', {'parts': ''})}, False, ) def _uninstall(self, installed): for f in installed.split('\n'): if not f: continue f = self._buildout_path(f) if os.path.isdir(f): rmtree(f) elif os.path.isfile(f): try: os.remove(f) except OSError: if not ( sys.platform == 'win32' and (realpath(os.path.join(os.path.dirname(sys.argv[0]), 'buildout.exe')) == realpath(f) ) # Sigh. This is the executable used to run the buildout # and, of course, it's in use. Leave it. ): raise def _install(self, part): options = self[part] recipe, entry = _recipe(options) recipe_class = pkg_resources.load_entry_point( recipe, 'zc.buildout', entry) installed = recipe_class(self, part, options).install() if installed is None: installed = [] elif isinstance(installed, str): installed = [installed] base = self._buildout_path('') installed = [d.startswith(base) and d[len(base):] or d for d in installed] return ' '.join(installed) def _save_installed_options(self, installed_options): installed = self['buildout']['installed'] if not installed: return f = open(installed, 'w') _save_options('buildout', installed_options['buildout'], f) for part in installed_options['buildout']['parts'].split(): print_(file=f) _save_options(part, installed_options[part], f) f.close() def _error(self, message, *args): raise zc.buildout.UserError(message % args) def _setup_socket_timeout(self): timeout = self['buildout']['socket-timeout'] if timeout != '': try: timeout = int(timeout) import socket self._logger.info( 'Setting socket time out to %d seconds.', timeout) socket.setdefaulttimeout(timeout) except ValueError: self._logger.warning("Default socket timeout is used !\n" "Value in configuration is not numeric: [%s].\n", timeout) def _setup_logging(self): root_logger = logging.getLogger() self._logger = logging.getLogger('zc.buildout') handler = logging.StreamHandler(sys.stdout) log_format = self['buildout']['log-format'] if not log_format: # No format specified. Use different formatter for buildout # and other modules, showing logger name except for buildout log_format = '%(name)s: %(message)s' buildout_handler = logging.StreamHandler(sys.stdout) buildout_handler.setFormatter(logging.Formatter('%(message)s')) self._logger.propagate = False self._logger.addHandler(buildout_handler) handler.setFormatter(logging.Formatter(log_format)) root_logger.addHandler(handler) level = self['buildout']['log-level'] if level in ('DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL'): level = getattr(logging, level) else: try: level = int(level) except ValueError: self._error("Invalid logging level %s", level) verbosity = self['buildout'].get('verbosity', 0) try: verbosity = int(verbosity) except ValueError: self._error("Invalid verbosity %s", verbosity) level -= verbosity root_logger.setLevel(level) self._log_level = level def _maybe_upgrade(self): # See if buildout or setuptools or other dependencies need to be upgraded. # If they do, do the upgrade and restart the buildout process. __doing__ = 'Checking for upgrades.' if 'BUILDOUT_RESTART_AFTER_UPGRADE' in os.environ: return if not self.newest: return # We must install `wheel` before `setuptools`` to avoid confusion between # the true `wheel` package and the one vendorized by `setuptools`. # See https://github.com/buildout/buildout/issues/691 projects = ('zc.buildout', 'wheel', 'pip', 'setuptools') ws = zc.buildout.easy_install.install( projects, self['buildout']['eggs-directory'], links = self['buildout'].get('find-links', '').split(), index = self['buildout'].get('index'), path = [self['buildout']['develop-eggs-directory']], allow_hosts = self._allow_hosts ) upgraded = [] for project in projects: canonicalized_name = packaging_utils.canonicalize_name(project) req = pkg_resources.Requirement.parse(canonicalized_name) dist = ws.find(req) if dist is None and canonicalized_name != project: # Try with the original project name. Depending on which setuptools # version is used, this is either useless or a life saver. req = pkg_resources.Requirement.parse(project) dist = ws.find(req) importlib.import_module(project) if dist is None: # This is unexpected. This must be some problem with how we use # setuptools/pkg_resources. But since the import worked, it feels # safe to ignore. self._logger.warning( "Could not find %s in working set during upgrade check. Ignoring.", project, ) continue if not inspect.getfile(sys.modules[project]).startswith(dist.location): upgraded.append(dist) if not upgraded: return __doing__ = 'Upgrading.' should_run = realpath( os.path.join(os.path.abspath(self['buildout']['bin-directory']), 'buildout') ) if sys.platform == 'win32': should_run += '-script.py' if (realpath(os.path.abspath(sys.argv[0])) != should_run): self._logger.debug("Running %r.", realpath(sys.argv[0])) self._logger.debug("Local buildout is %r.", should_run) self._logger.warning("Not upgrading because not running a local " "buildout command.") return self._logger.info("Upgraded:\n %s;\nRestarting.", ",\n ".join([("%s version %s" % (dist.project_name, dist.version) ) for dist in upgraded ] ), ) # the new dist is different, so we've upgraded. # Update the scripts and return True options = self['buildout'] eggs_dir = options['eggs-directory'] develop_eggs_dir = options['develop-eggs-directory'] ws = zc.buildout.easy_install.sort_working_set( ws, eggs_dir=eggs_dir, develop_eggs_dir=develop_eggs_dir ) zc.buildout.easy_install.scripts( ['zc.buildout'], ws, sys.executable, options['bin-directory'], relative_paths = ( bool_option(options, 'relative-paths', False) and options['directory'] or ''), ) # Restart args = sys.argv[:] if not __debug__: args.insert(0, '-O') args.insert(0, sys.executable) env=dict(os.environ, BUILDOUT_RESTART_AFTER_UPGRADE='1') sys.exit(subprocess.call(args, env=env)) def _load_extensions(self): __doing__ = 'Loading extensions.' specs = self['buildout'].get('extensions', '').split() for superceded_extension in ['buildout-versions', 'buildout.dumppickedversions']: if superceded_extension in specs: msg = ("Buildout now includes 'buildout-versions' (and part " "of the older 'buildout.dumppickedversions').\n" "Remove the extension from your configuration and " "look at the 'show-picked-versions' option in " "buildout's documentation.") raise zc.buildout.UserError(msg) if specs: path = [self['buildout']['develop-eggs-directory']] if self.offline: dest = None path.append(self['buildout']['eggs-directory']) else: dest = self['buildout']['eggs-directory'] zc.buildout.easy_install.install( specs, dest, path=path, working_set=pkg_resources.working_set, links = self['buildout'].get('find-links', '').split(), index = self['buildout'].get('index'), newest=self.newest, allow_hosts=self._allow_hosts) # Clear cache because extensions might now let us read pages we # couldn't read before. zc.buildout.easy_install.clear_index_cache() for ep in pkg_resources.iter_entry_points('zc.buildout.extension'): ep.load()(self) def _unload_extensions(self): __doing__ = 'Unloading extensions.' specs = self['buildout'].get('extensions', '').split() if specs: for ep in pkg_resources.iter_entry_points( 'zc.buildout.unloadextension'): ep.load()(self) def _print_picked_versions(self): picked_versions, required_by = (zc.buildout.easy_install .get_picked_versions()) if not picked_versions: # Don't print empty output. return output = _format_picked_versions(picked_versions, required_by) if self.show_picked_versions: print_("Versions had to be automatically picked.") print_("The following part definition lists the versions picked:") print_('\n'.join(output)) if self.update_versions_file: # Write to the versions file. if os.path.exists(self.update_versions_file): output[:1] = [ '', '# Added by buildout at %s' % datetime.datetime.now() ] output.append('') f = open(self.update_versions_file, 'a') f.write(('\n'.join(output))) f.close() print_("Picked versions have been written to " + self.update_versions_file) @command def setup(self, args): if not args: raise zc.buildout.UserError( "The setup command requires the path to a setup script or \n" "directory containing a setup script, and its arguments." ) setup = args.pop(0) if os.path.isdir(setup): setup = os.path.join(setup, 'setup.py') self._logger.info("Running setup script %r.", setup) setup = os.path.abspath(setup) fd, tsetup = tempfile.mkstemp() try: os.write(fd, (zc.buildout.easy_install.runsetup_template % dict( setupdir=os.path.dirname(setup), setup=setup, __file__ = setup, )).encode()) args = [sys.executable, tsetup] + args zc.buildout.easy_install.call_subprocess(args) finally: os.close(fd) os.remove(tsetup) @command def runsetup(self, args): self.setup(args) @command def query(self, args=None): if args is None or len(args) != 1: _error('The query command requires a single argument.') option = args[0] option = option.split(':') if len(option) == 1: option = 'buildout', option[0] elif len(option) != 2: _error('Invalid option:', args[0]) section, option = option verbose = self['buildout'].get('verbosity', 0) != 0 if verbose: print_('${%s:%s}' % (section, option)) try: print_(self._raw[section][option]) except KeyError: if section in self._raw: _error('Key not found:', option) else: _error('Section not found:', section) @command def annotate(self, args=None): verbose = self['buildout'].get('verbosity', 0) != 0 section = None if args is None: sections = [] else: sections = args _print_annotate(self._annotated, verbose, sections, self._buildout_dir) def print_options(self, base_path=None): for section in sorted(self._data): if section == 'buildout' or section == self['buildout']['versions']: continue print_('['+section+']') for k, v in sorted(self._data[section].items()): if '\n' in v: v = '\n ' + v.replace('\n', '\n ') else: v = ' '+v if base_path: v = v.replace(os.getcwd(), base_path) print_("%s =%s" % (k, v)) def __getitem__(self, section): __doing__ = 'Getting section %s.', section try: return self._data[section] except KeyError: pass try: data = self._raw[section] except KeyError: raise MissingSection(section) options = self.Options(self, section, data) self._data[section] = options options._initialize() return options def __setitem__(self, name, data): if name in self._raw: raise KeyError("Section already exists", name) self._raw[name] = dict((k, str(v)) for (k, v) in data.items()) self[name] # Add to parts def parse(self, data): from io import StringIO import textwrap sections = zc.buildout.configparser.parse( StringIO(textwrap.dedent(data)), '', _default_globals) for name in sections: if name in self._raw: raise KeyError("Section already exists", name) self._raw[name] = dict((k, str(v)) for (k, v) in sections[name].items()) for name in sections: self[name] # Add to parts def __delitem__(self, key): raise NotImplementedError('__delitem__') def keys(self): return list(self._raw.keys()) def __iter__(self): return iter(self._raw) def __len__(self): return len(self._raw) def _install_and_load(spec, group, entry, buildout): __doing__ = 'Loading recipe %r.', spec try: req = pkg_resources.Requirement.parse(spec) buildout_options = buildout['buildout'] if pkg_resources.working_set.find(req) is None: __doing__ = 'Installing recipe %s.', spec if buildout.offline: dest = None path = [buildout_options['develop-eggs-directory'], buildout_options['eggs-directory'], ] else: dest = buildout_options['eggs-directory'] path = [buildout_options['develop-eggs-directory']] # Pin versions when processing the buildout section versions_section_name = buildout['buildout'].get('versions', 'versions') versions = buildout.get(versions_section_name, {}) zc.buildout.easy_install.allow_picked_versions( bool_option(buildout['buildout'], 'allow-picked-versions') ) zc.buildout.easy_install.install( [spec], dest, links=buildout._links, index=buildout_options.get('index'), path=path, working_set=pkg_resources.working_set, newest=buildout.newest, allow_hosts=buildout._allow_hosts, versions=versions, ) __doing__ = 'Loading %s recipe entry %s:%s.', group, spec, entry return pkg_resources.load_entry_point( req.project_name, group, entry) except Exception: v = sys.exc_info()[1] buildout._logger.log( 1, "Couldn't load %s entry point %s\nfrom %s:\n%s.", group, entry, spec, v) raise class Options(DictMixin): def __init__(self, buildout, section, data): self.buildout = buildout self.name = section self._raw = data self._cooked = {} self._data = {} def _initialize(self): name = self.name __doing__ = 'Initializing section %s.', name if '<' in self._raw: self._raw = self._do_extend_raw(name, self._raw, []) # force substitutions for k, v in sorted(self._raw.items()): if '${' in v: self._dosub(k, v) if name == 'buildout': return # buildout section can never be a part for dname in self.get('', '').split(): # force use of dependencies in buildout: self.buildout[dname] if self.get('recipe'): self.initialize() self.buildout._parts.append(name) def initialize(self): reqs, entry = _recipe(self._data) buildout = self.buildout recipe_class = _install_and_load(reqs, 'zc.buildout', entry, buildout) name = self.name self.recipe = recipe_class(buildout, name, self) def _do_extend_raw(self, name, data, doing): if name == 'buildout': return data if name in doing: raise zc.buildout.UserError("Infinite extending loop %r" % name) doing.append(name) try: to_do = data.get('<', None) if to_do is None: return data __doing__ = 'Loading input sections for %r', name result = {} for iname in to_do.split('\n'): iname = iname.strip() if not iname: continue raw = self.buildout._raw.get(iname) if raw is None: raise zc.buildout.UserError("No section named %r" % iname) result.update(self._do_extend_raw(iname, raw, doing)) result = _annotate_section(result, "") data = _annotate_section(copy.deepcopy(data), "") result = _update_section(result, data) result = _unannotate_section(result) result.pop('<', None) return result finally: assert doing.pop() == name def _dosub(self, option, v): __doing__ = 'Getting option %s:%s.', self.name, option seen = [(self.name, option)] v = '$$'.join([self._sub(s, seen) for s in v.split('$$')]) self._cooked[option] = v def get(self, option, default=None, seen=None): try: return self._data[option] except KeyError: pass v = self._cooked.get(option) if v is None: v = self._raw.get(option) if v is None: return default __doing__ = 'Getting option %s:%s.', self.name, option if '${' in v: key = self.name, option if seen is None: seen = [key] elif key in seen: raise zc.buildout.UserError( "Circular reference in substitutions.\n" ) else: seen.append(key) v = '$$'.join([self._sub(s, seen) for s in v.split('$$')]) seen.pop() self._data[option] = v return v _template_split = re.compile('([$]{[^}]*})').split _simple = re.compile('[-a-zA-Z0-9 ._]+$').match _valid = re.compile(r'\${[-a-zA-Z0-9 ._]*:[-a-zA-Z0-9 ._]+}$').match def _sub(self, template, seen): value = self._template_split(template) subs = [] for ref in value[1::2]: s = tuple(ref[2:-1].split(':')) if not self._valid(ref): if len(s) < 2: raise zc.buildout.UserError("The substitution, %s,\n" "doesn't contain a colon." % ref) if len(s) > 2: raise zc.buildout.UserError("The substitution, %s,\n" "has too many colons." % ref) if not self._simple(s[0]): raise zc.buildout.UserError( "The section name in substitution, %s,\n" "has invalid characters." % ref) if not self._simple(s[1]): raise zc.buildout.UserError( "The option name in substitution, %s,\n" "has invalid characters." % ref) section, option = s if not section: section = self.name v = self.buildout[section].get(option, None, seen) if v is None: if option == '_buildout_section_name_': v = self.name else: raise MissingOption("Referenced option does not exist:", section, option) subs.append(v) subs.append('') return ''.join([''.join(v) for v in zip(value[::2], subs)]) def __getitem__(self, key): try: return self._data[key] except KeyError: pass v = self.get(key) if v is None: raise MissingOption("Missing option: %s:%s" % (self.name, key)) return v def __setitem__(self, option, value): if not isinstance(value, str): raise TypeError('Option values must be strings', value) self._data[option] = value def __delitem__(self, key): if key in self._raw: del self._raw[key] if key in self._data: del self._data[key] if key in self._cooked: del self._cooked[key] elif key in self._data: del self._data[key] else: raise KeyError(key) def keys(self): raw = self._raw return list(self._raw) + [k for k in self._data if k not in raw] def __iter__(self): return iter(self.keys()) def __len__(self): return len(self.keys()) def copy(self): result = copy.deepcopy(self._raw) result.update(self._cooked) result.update(self._data) return result def _call(self, f): buildout_directory = self.buildout['buildout']['directory'] self._created = [] try: try: os.chdir(buildout_directory) return f() except: for p in self._created: if os.path.isdir(p): rmtree(p) elif os.path.isfile(p): os.remove(p) else: self.buildout._logger.warning("Couldn't clean up %r.", p) raise finally: self._created = None os.chdir(buildout_directory) def created(self, *paths): try: self._created.extend(paths) except AttributeError: raise TypeError( "Attempt to register a created path while not installing", self.name) return self._created def __repr__(self): return repr(dict(self)) Buildout.Options = Options _spacey_nl = re.compile('[ \t\r\f\v]*\n[ \t\r\f\v\n]*' '|' '^[ \t\r\f\v]+' '|' '[ \t\r\f\v]+$' ) _spacey_defaults = [ ('%(__buildout_space__)s', ' '), ('%(__buildout_space_n__)s', '\n'), ('%(__buildout_space_r__)s', '\r'), ('%(__buildout_space_f__)s', '\f'), ('%(__buildout_space_v__)s', '\v'), ] def _quote_spacey_nl(match): match = match.group(0).split('\n', 1) result = '\n\t'.join( [(s .replace(' ', '%(__buildout_space__)s') .replace('\r', '%(__buildout_space_r__)s') .replace('\f', '%(__buildout_space_f__)s') .replace('\v', '%(__buildout_space_v__)s') .replace('\n', '%(__buildout_space_n__)s') ) for s in match] ) return result def _save_option(option, value, f): value = _spacey_nl.sub(_quote_spacey_nl, value) if value.startswith('\n\t'): value = '%(__buildout_space_n__)s' + value[2:] if value.endswith('\n\t'): value = value[:-2] + '%(__buildout_space_n__)s' print_(option, '=', value, file=f) def _save_options(section, options, f): print_('[%s]' % section, file=f) items = list(options.items()) items.sort() for option, value in items: _save_option(option, value, f) def _default_globals(): """Return a mapping of default and precomputed expressions. These default expressions are convenience defaults available when eveluating section headers expressions. NB: this is wrapped in a function so that the computing of these expressions is lazy and done only if needed (ie if there is at least one section with an expression) because the computing of some of these expressions can be expensive. """ # partially derived or inspired from its.py # Copyright (c) 2012, Kenneth Reitz All rights reserved. # Redistribution and use in source and binary forms, with or without modification, # are permitted provided that the following conditions are met: # Redistributions of source code must retain the above copyright notice, this list # of conditions and the following disclaimer. Redistributions in binary form must # reproduce the above copyright notice, this list of conditions and the following # disclaimer in the documentation and/or other materials provided with the # distribution. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND # CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A # PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR # CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, # OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF # SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS # INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN # CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING # IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY # OF SUCH DAMAGE. # default available modules, explicitly re-imported locally here on purpose import sys import os import platform import re globals_defs = {'sys': sys, 'os': os, 'platform': platform, 're': re,} # major python major_python_versions as python2 and python3 major_python_versions = tuple(map(str, platform.python_version_tuple())) globals_defs.update({'python2': major_python_versions[0] == '2', 'python3': major_python_versions[0] == '3'}) # minor python major_python_versions as python24, python25 ... python39 minor_python_versions = ('24', '25', '26', '27', '30', '31', '32', '33', '34', '35', '36', '37', '38', '39', '310', '311', '312', '313', '314', '315') for v in minor_python_versions: globals_defs['python' + v] = ''.join(major_python_versions[:2]) == v # interpreter type sys_version = sys.version.lower() pypy = 'pypy' in sys_version jython = 'java' in sys_version ironpython ='iron' in sys_version # assume CPython, if nothing else. cpython = not any((pypy, jython, ironpython,)) globals_defs.update({'cpython': cpython, 'pypy': pypy, 'jython': jython, 'ironpython': ironpython}) # operating system sys_platform = str(sys.platform).lower() globals_defs.update({'linux': 'linux' in sys_platform, 'windows': 'win32' in sys_platform, 'cygwin': 'cygwin' in sys_platform, 'solaris': 'sunos' in sys_platform, 'macosx': 'darwin' in sys_platform, 'posix': 'posix' in os.name.lower()}) #bits and endianness import struct void_ptr_size = struct.calcsize('P') * 8 globals_defs.update({'bits32': void_ptr_size == 32, 'bits64': void_ptr_size == 64, 'little_endian': sys.byteorder == 'little', 'big_endian': sys.byteorder == 'big'}) return globals_defs variable_template_split = re.compile('([$]{[^}]*})').split def _open( base, filename, seen, download_options, override, downloaded, user_defaults ): """Open a configuration file and return the result as a dictionary, Recursively open other files based on buildout options found. """ download_options = _update_section(download_options, override) raw_download_options = _unannotate_section(download_options) newest = bool_option(raw_download_options, 'newest', 'false') fallback = newest and not (filename in downloaded) extends_cache = raw_download_options.get('extends-cache') if extends_cache and variable_template_split(extends_cache)[1::2]: raise ValueError( "extends-cache '%s' may not contain ${section:variable} to expand." % extends_cache ) download = zc.buildout.download.Download( raw_download_options, cache=extends_cache, fallback=fallback, hash_name=True) is_temp = False downloaded_filename = None if _isurl(filename): downloaded_filename, is_temp = download(filename) fp = open(downloaded_filename) base = filename[:filename.rfind('/')] elif _isurl(base): if os.path.isabs(filename): fp = open(filename) base = os.path.dirname(filename) else: filename = base + '/' + filename downloaded_filename, is_temp = download(filename) fp = open(downloaded_filename) base = filename[:filename.rfind('/')] else: filename = os.path.join(base, filename) fp = open(filename) base = os.path.dirname(filename) downloaded.add(filename) if filename in seen: if is_temp: fp.close() os.remove(downloaded_filename) raise zc.buildout.UserError("Recursive file include", seen, filename) root_config_file = not seen seen.append(filename) filename_for_logging = filename if downloaded_filename: filename_for_logging = '%s (downloaded as %s)' % ( filename, downloaded_filename) result = zc.buildout.configparser.parse( fp, filename_for_logging, _default_globals) fp.close() if is_temp: os.remove(downloaded_filename) options = result.get('buildout', {}) extends = options.pop('extends', None) if 'extended-by' in options: raise zc.buildout.UserError( 'No-longer supported "extended-by" option found in %s.' % filename) result = _annotate(result, filename) if root_config_file and 'buildout' in result: download_options = _update_section( download_options, result['buildout'] ) # Process extends to handle nested += and -= eresults = [] if extends: extends = extends.split() for fname in extends: next_extend, user_defaults = _open( base, fname, seen, download_options, override, downloaded, user_defaults) eresults.extend(next_extend) else: if user_defaults: result = _update(user_defaults, result) user_defaults = {} optional_extends = options.pop('optional-extends', None) if optional_extends: optional_extends = optional_extends.value.split() for fname in optional_extends: if not os.path.exists(fname): print("optional-extends file not found: %s" % fname) continue next_extend, user_defaults = _open( base, fname, seen, download_options, override, downloaded, user_defaults) eresults.extend(next_extend) eresults.append(result) seen.pop() if root_config_file: final_result = {} for eresult in eresults: final_result = _update(final_result, eresult) return final_result, user_defaults else: return eresults, user_defaults ignore_directories = '.svn', 'CVS', '__pycache__', '.git' _dir_hashes = {} def _dir_hash(dir): dir_hash = _dir_hashes.get(dir, None) if dir_hash is not None: return dir_hash hash = md5() for (dirpath, dirnames, filenames) in os.walk(dir): dirnames[:] = sorted(n for n in dirnames if n not in ignore_directories) filenames[:] = sorted(f for f in filenames if (not (f.endswith('pyc') or f.endswith('pyo')) and os.path.exists(os.path.join(dirpath, f))) ) for_hash = ' '.join(dirnames + filenames) if isinstance(for_hash, str): for_hash = for_hash.encode() hash.update(for_hash) for name in filenames: path = os.path.join(dirpath, name) if name == 'entry_points.txt': f = open(path) # Entry points aren't written in stable order. :( try: sections = zc.buildout.configparser.parse(f, path) data = repr([(sname, sorted(sections[sname].items())) for sname in sorted(sections)]).encode('utf-8') except Exception: f.close() f = open(path, 'rb') data = f.read() else: f = open(path, 'rb') data = f.read() f.close() hash.update(data) _dir_hashes[dir] = dir_hash = hash.hexdigest() return dir_hash def _dists_sig(dists): seen = set() result = [] for dist in sorted(dists): if dist in seen: continue seen.add(dist) location = dist.location if dist.precedence == pkg_resources.DEVELOP_DIST: result.append(dist.project_name + '-' + _dir_hash(location)) else: result.append(os.path.basename(location)) return result def _update_section(in1, s2): s1 = copy.deepcopy(in1) # Base section 2 on section 1; section 1 is copied, with key-value pairs # in section 2 overriding those in section 1. If there are += or -= # operators in section 2, process these to add or subtract items (delimited # by newlines) from the preexisting values. s2 = copy.deepcopy(s2) # avoid mutating the second argument, which is unexpected # Sort on key, then on the addition or subtraction operator (+ comes first) for k, v in sorted(s2.items(), key=lambda x: (x[0].rstrip(' +'), x[0][-1])): if k.endswith('+'): key = k.rstrip(' +') implicit_value = SectionKey("", "IMPLICIT_VALUE") # Find v1 in s2 first; it may have been defined locally too. section_key = s2.get(key, s1.get(key, implicit_value)) section_key = copy.deepcopy(section_key) section_key.addToValue(v.value, v.source) s2[key] = section_key del s2[k] elif k.endswith('-'): key = k.rstrip(' -') implicit_value = SectionKey("", "IMPLICIT_VALUE") # Find v1 in s2 first; it may have been set by a += operation first section_key = s2.get(key, s1.get(key, implicit_value)) section_key = copy.deepcopy(section_key) section_key.removeFromValue(v.value, v.source) s2[key] = section_key del s2[k] _update_verbose(s1, s2) return s1 def _update_verbose(s1, s2): for key, v2 in s2.items(): if key in s1: v1 = s1[key] v1.overrideValue(v2) else: s1[key] = copy.deepcopy(v2) def _update(in1, d2): d1 = copy.deepcopy(in1) for section in d2: if section in d1: d1[section] = _update_section(d1[section], d2[section]) elif '<' not in d2[section].keys(): # Skip sections that extend in other sections (macros), as we don't # have all the data (these will be processed when the section is # extended) temp = copy.deepcopy(d2[section]) # 641 - Process base definitions done with += and -= for k, v in sorted(temp.items(), key=lambda item: item[0]): # Process + before -, configparser resolves conflicts if k[-1] == '+' and k[:-2] not in temp: # Turn += without a preceding = into an assignment temp[k[:-2]] = temp[k] del temp[k] elif k[-1] == '-' and k[:-2] not in temp: # Turn -= without a preceding = into an empty assignment temp[k[:-2]] = temp[k] temp[k[:-2]].removeFromValue( temp[k[:-2]].value, "IMPLICIT_VALUE" ) del temp[k] # 656 - Handle multiple option assignments/extensions/removals # in the same file, which can happen with conditional sections d1[section] = _update_section({}, temp) else: d1[section] = copy.deepcopy(d2[section]) return d1 def _recipe(options): recipe = options['recipe'] if ':' in recipe: recipe, entry = recipe.split(':') else: entry = 'default' return recipe, entry def _doing(): _, v, tb = sys.exc_info() message = str(v) doing = [] while tb is not None: d = tb.tb_frame.f_locals.get('__doing__') if d: doing.append(d) tb = tb.tb_next if doing: sys.stderr.write('While:\n') for d in doing: if not isinstance(d, str): d = d[0] % d[1:] sys.stderr.write(' %s\n' % d) def _error(*message): sys.stderr.write('Error: ' + ' '.join(message) +'\n') sys.exit(1) _internal_error_template = """ An internal error occurred due to a bug in either zc.buildout or in a recipe being used: """ def _check_for_unused_options_in_section(buildout, section): options = buildout[section] unused = [option for option in sorted(options._raw) if option not in options._data] if unused: buildout._logger.warning( "Section `%s` contains unused option(s): %s.\n" "This may be an indication for either a typo in the option's name " "or a bug in the used recipe." % (section, ' '.join(map(repr, unused))) ) _usage = """\ Usage: buildout [options] [assignments] [command [command arguments]] Options: -c config_file Specify the path to the buildout configuration file to be used. This defaults to the file named "buildout.cfg" in the current working directory. -D Debug errors. If an error occurs, then the post-mortem debugger will be started. This is especially useful for debugging recipe problems. -h, --help Print this message and exit. -N Run in non-newest mode. This is equivalent to the assignment buildout:newest=false. With this setting, buildout will not seek new distributions if installed distributions satisfy it's requirements. -q Decrease the level of verbosity. This option can be used multiple times. -t socket_timeout Specify the socket timeout in seconds. -U Don't read user defaults. -v Increase the level of verbosity. This option can be used multiple times. --version Print buildout version number and exit. Assignments are of the form: section:option=value and are used to provide configuration options that override those given in the configuration file. For example, to run the buildout in offline mode, use buildout:offline=true. Options and assignments can be interspersed. Commands: install Install the parts specified in the buildout configuration. This is the default command if no command is specified. bootstrap Create a new buildout in the current working directory, copying the buildout and setuptools eggs and, creating a basic directory structure and a buildout-local buildout script. init [requirements] Initialize a buildout, creating a minimal buildout.cfg file if it doesn't exist and then performing the same actions as for the bootstrap command. If requirements are supplied, then the generated configuration will include an interpreter script that requires them. This provides an easy way to quickly set up a buildout to experiment with some packages. setup script [setup command and options] Run a given setup script arranging that setuptools is in the script's path and and that it has been imported so that setuptools-provided commands (like bdist_egg) can be used even if the setup script doesn't import setuptools. The script can be given either as a script path or a path to a directory containing a setup.py script. annotate Display annotated sections. All sections are displayed, sorted alphabetically. For each section, all key-value pairs are displayed, sorted alphabetically, along with the origin of the value (file name or COMPUTED_VALUE, DEFAULT_VALUE, COMMAND_LINE_VALUE). query section:key Display value of given section key pair. """ def _help(): print_(_usage) sys.exit(0) def _version(): version = pkg_resources.working_set.find( pkg_resources.Requirement.parse('zc.buildout')).version print_("buildout version %s" % version) sys.exit(0) def main(args=None): if args is None: args = sys.argv[1:] config_file = 'buildout.cfg' verbosity = 0 options = [] use_user_defaults = True debug = False while args: if args[0][0] == '-': op = orig_op = args.pop(0) op = op[1:] while op and op[0] in 'vqhWUoOnNDA': if op[0] == 'v': verbosity += 10 elif op[0] == 'q': verbosity -= 10 elif op[0] == 'U': use_user_defaults = False elif op[0] == 'o': options.append(('buildout', 'offline', 'true')) elif op[0] == 'O': options.append(('buildout', 'offline', 'false')) elif op[0] == 'n': options.append(('buildout', 'newest', 'true')) elif op[0] == 'N': options.append(('buildout', 'newest', 'false')) elif op[0] == 'D': debug = True else: _help() op = op[1:] if op[:1] in ('c', 't'): op_ = op[:1] op = op[1:] if op_ == 'c': if op: config_file = op else: if args: config_file = args.pop(0) else: _error("No file name specified for option", orig_op) elif op_ == 't': try: timeout_string = args.pop(0) timeout = int(timeout_string) options.append( ('buildout', 'socket-timeout', timeout_string)) except IndexError: _error("No timeout value specified for option", orig_op) except ValueError: _error("Timeout value must be numeric", orig_op) elif op: if orig_op == '--help': _help() elif orig_op == '--version': _version() else: _error("Invalid option", '-'+op[0]) elif '=' in args[0]: option, value = args.pop(0).split('=', 1) option = option.split(':') if len(option) == 1: option = 'buildout', option[0] elif len(option) != 2: _error('Invalid option:', option) section, option = option options.append((section.strip(), option.strip(), value.strip())) else: # We've run out of command-line options and option assignments # The rest should be commands, so we'll stop here break if verbosity: options.append(('buildout', 'verbosity', str(verbosity))) if args: command = args.pop(0) if command not in Buildout.COMMANDS: _error('invalid command:', command) else: command = 'install' try: try: buildout = Buildout(config_file, options, use_user_defaults, command, args) getattr(buildout, command)(args) except SystemExit: logging.shutdown() # Make sure we properly propagate an exit code from a restarted # buildout process. raise except Exception: v = sys.exc_info()[1] _doing() exc_info = sys.exc_info() import pdb, traceback if debug: traceback.print_exception(*exc_info) sys.stderr.write('\nStarting pdb:\n') pdb.post_mortem(exc_info[2]) else: if isinstance(v, (zc.buildout.UserError, distutils.errors.DistutilsError ) ): _error(str(v)) else: sys.stderr.write(_internal_error_template) traceback.print_exception(*exc_info) sys.exit(1) finally: logging.shutdown() _bool_names = {'true': True, 'false': False, True: True, False: False} def bool_option(options, name, default=None): value = options.get(name, default) if value is None: raise KeyError(name) try: return _bool_names[value] except KeyError: raise zc.buildout.UserError( 'Invalid value for %r option: %r' % (name, value)) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675670.0 zc_buildout-4.1.6/src/zc/buildout/configparser.py0000644000076500000240000002600214773460426021557 0ustar00mauritsstaff############################################################################## # # Copyright Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## # The following copied from Python 2 config parser because: # - The py3 configparser isn't backward compatible # - Both strip option values in undesirable ways # - dict of dicts is a much simpler api import re import textwrap import logging from packaging import markers Marker = markers.Marker InvalidMarker = markers.InvalidMarker logger = logging.getLogger('zc.buildout') class Error(Exception): """Base class for ConfigParser exceptions.""" def _get_message(self): """Getter for 'message'; needed only to override deprecation in BaseException.""" return self.__message def _set_message(self, value): """Setter for 'message'; needed only to override deprecation in BaseException.""" self.__message = value # BaseException.message has been deprecated since Python 2.6. To prevent # DeprecationWarning from popping up over this pre-existing attribute, use # a new property that takes lookup precedence. message = property(_get_message, _set_message) def __init__(self, msg=''): self.message = msg Exception.__init__(self, msg) def __repr__(self): return self.message __str__ = __repr__ class ParsingError(Error): """Raised when a configuration file does not follow legal syntax.""" def __init__(self, filename): Error.__init__(self, 'File contains parsing errors: %s' % filename) self.filename = filename self.errors = [] def append(self, lineno, line): self.errors.append((lineno, line)) self.message += '\n\t[line %2d]: %s' % (lineno, line) class MissingSectionHeaderError(ParsingError): """Raised when a key-value pair is found before any section header.""" def __init__(self, filename, lineno, line): Error.__init__( self, 'File contains no section headers.\nfile: %s, line: %d\n%r' % (filename, lineno, line)) self.filename = filename self.lineno = lineno self.line = line # This regex captures either sections headers with optional trailing comment # separated by a semicolon or a hash. Section headers can have an optional # expression. Expressions and comments can contain brackets but no verbatim '#' # and ';' : these need to be escaped. # A title line with an expression has the general form: # [section_name: some Python expression] #; some comment # This regex leverages the fact that the following is a valid Python expression: # [some Python expression] # some comment # and that section headers are also delimited by [brackets] that are also [list] # delimiters. # So instead of doing complex parsing to balance brackets in an expression, we # capture just enough from a header line to collect then remove the section_name # and colon expression separator keeping only a list-enclosed expression and # optional comments. The parsing and validation of this Python expression can be # entirely delegated to Python's eval. The result of the evaluated expression is # the always returned wrapped in a list with a single item that contains the # original expression section_header = re.compile( r'(?P\[)' r'\s*' r'(?P[^\s#[\]:;{}]+)' r'\s*' r'(:(?P[^#;]*))?' r'\s*' r'(?P]' r'\s*' r'([#;].*)?$)' ).match option_start = re.compile( r'(?P[^\s{}[\]=:]+\s*[-+]?)' r'=' r'(?P.*)$').match leading_blank_lines = re.compile(r"^(\s*\n)+") def parse(fp, fpname, exp_globals=dict): """Parse a sectioned setup file. The sections in setup files contain a title line at the top, indicated by a name in square brackets (`[]'), plus key/value options lines, indicated by `name: value' format lines. Continuations are represented by an embedded newline then leading whitespace. Blank lines, lines beginning with a '#', and just about everything else are ignored. The title line is in the form [name] followed by an optional trailing comment separated by a semicolon `;' or a hash `#' character. Optionally the title line can have the form `[name:expression]' where expression is an arbitrary Python expression. Sections with an expression that evaluates to False are ignored. Semicolon `;' an hash `#' characters must be string-escaped in expression literals. exp_globals is a callable returning a mapping of defaults used as globals during the evaluation of a section conditional expression. """ sections = {} # the current section condition, possibly updated from a section expression section_condition = True context = None cursect = None # None, or a dictionary blockmode = None optname = None lineno = 0 e = None # None, or an exception while True: line = fp.readline() if not line: break # EOF lineno = lineno + 1 if line[0] in '#;': continue # comment if line[0].isspace() and cursect is not None and optname: if not section_condition: #skip section based on its expression condition continue # continuation line if blockmode: line = line.rstrip() else: line = line.strip() if not line: continue cursect[optname] = "%s\n%s" % (cursect[optname], line) else: header = section_header(line) if header: # reset to True when starting a new section section_condition = True sectname = header.group('name') head = header.group('head') # the starting [ expression = header.group('expression') tail = header.group('tail') # closing ]and comment if expression: # normalize tail comments to Python style tail = tail.replace(';', '#') if tail else '' # un-escape literal # and ; . Do not use a # string-escape decode expr = expression.replace(r'\x23','#').replace(r'\x3b', ';') try: # new-style markers as used in pip constraints, e.g.: # 'python_version < "3.11" and platform_system == "Windows"' marker = Marker(expr) section_condition = marker.evaluate() except InvalidMarker: # old style buildout expression # rebuild a valid Python expression wrapped in a list expr = head + expr + tail # lazily populate context only expression if not context: context = exp_globals() # evaluated expression is in list: get first element section_condition = eval(expr, context)[0] # finally, ignore section when an expression # evaluates to false if not section_condition: logger.debug( 'Ignoring section %(sectname)r with [expression]:' ' %(expression)r' % locals()) continue if sectname in sections: cursect = sections[sectname] else: sections[sectname] = cursect = {} # So sections can't start with a continuation line optname = None elif cursect is None: if not line.strip(): continue # no section header in the file? raise MissingSectionHeaderError(fpname, lineno, line) else: if line[:2] == '=>': line = ' = ' + line[2:] mo = option_start(line) if mo: if not section_condition: # filter out options of conditionally ignored section continue # option start line optname, optval = mo.group('name', 'value') optname = optname.rstrip() optval = optval.strip() # Handle multiple extensions of the same value in the # same file. This happens with conditional sections. opt_op = optname[-1] if opt_op not in '+-': opt_op = '=' if optname in cursect and opt_op in '+-': # Strip any trailing \n, which happens when we have multiple # +=/-= in one file cursect[optname] = cursect[optname].rstrip() if optval: cursect[optname] = "%s\n%s" % (cursect[optname], optval) else: # If an assignment (=) comes after an extend (+=) / # remove (-=), it overrides and replaces the preceding # extend / remove if opt_op == '=': for suffix in '+-': tempname = "%s %s" % (optname, suffix) if tempname in cursect: del cursect[tempname] cursect[optname] = optval blockmode = not optval elif not (optname or line.strip()): # blank line after section start continue else: # a non-fatal parsing error occurred. set up the # exception but keep going. the exception will be # raised at the end of the file and will contain a # list of all bogus lines if not e: e = ParsingError(fpname) e.append(lineno, repr(line)) # if any parsing errors occurred, raise an exception if e: raise e for sectname in sections: section = sections[sectname] for name in section: value = section[name] if value[:1].isspace(): section[name] = leading_blank_lines.sub( '', textwrap.dedent(value.rstrip())) return sections ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675670.0 zc_buildout-4.1.6/src/zc/buildout/download.py0000644000076500000240000002070414773460426020707 0ustar00mauritsstaff############################################################################## # # Copyright (c) 2009 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Buildout download infrastructure""" from hashlib import md5 from urllib.request import urlretrieve from urllib.parse import urlparse from zc.buildout.easy_install import realpath import logging import os import os.path import re import shutil import sys import tempfile import zc.buildout class ChecksumError(zc.buildout.UserError): pass class Download(object): """Configurable download utility. Handles the download cache and offline mode. Download(options=None, cache=None, namespace=None, offline=False, fallback=False, hash_name=False, logger=None) options: mapping of buildout options (e.g. a ``buildout`` config section) cache: path to the download cache (excluding namespaces) namespace: namespace directory to use inside the cache offline: whether to operate in offline mode fallback: whether to use the cache as a fallback (try downloading first) hash_name: whether to use a hash of the URL as cache file name logger: an optional logger to receive download-related log messages """ def __init__(self, options=None, cache=-1, namespace=None, offline=-1, fallback=False, hash_name=False, logger=None): if options is None: options = {} self.directory = options.get('directory', '') self.cache = cache if cache == -1: self.cache = options.get('download-cache') self.namespace = namespace self.offline = offline if offline == -1: self.offline = (options.get('offline') == 'true' or options.get('install-from-cache') == 'true') self.fallback = fallback self.hash_name = hash_name self.logger = logger or logging.getLogger('zc.buildout') @property def download_cache(self): if self.cache is not None: return realpath(os.path.join(self.directory, self.cache)) @property def cache_dir(self): if self.download_cache is not None: return os.path.join(self.download_cache, self.namespace or '') def __call__(self, url, md5sum=None, path=None): """Download a file according to the utility's configuration. url: URL to download md5sum: MD5 checksum to match path: where to place the downloaded file Returns the path to the downloaded file. """ if self.cache: local_path, is_temp = self.download_cached(url, md5sum) else: local_path, is_temp = self.download(url, md5sum, path) return locate_at(local_path, path), is_temp def download_cached(self, url, md5sum=None): """Download a file from a URL using the cache. This method assumes that the cache has been configured. Optionally, it raises a ChecksumError if a cached copy of a file has an MD5 mismatch, but will not remove the copy in that case. """ if not os.path.exists(self.download_cache): raise zc.buildout.UserError( 'The directory:\n' '%r\n' "to be used as a download cache doesn't exist.\n" % self.download_cache) cache_dir = self.cache_dir if not os.path.exists(cache_dir): os.mkdir(cache_dir) cache_key = self.filename(url) cached_path = os.path.join(cache_dir, cache_key) self.logger.debug('Searching cache at %s' % cache_dir) if os.path.exists(cached_path): is_temp = False if self.fallback: try: _, is_temp = self.download(url, md5sum, cached_path) except ChecksumError: raise except Exception: pass if not check_md5sum(cached_path, md5sum): raise ChecksumError( 'MD5 checksum mismatch for cached download ' 'from %r at %r' % (url, cached_path)) self.logger.debug('Using cache file %s' % cached_path) else: self.logger.debug('Cache miss; will cache %s as %s' % (url, cached_path)) _, is_temp = self.download(url, md5sum, cached_path) return cached_path, is_temp def download(self, url, md5sum=None, path=None): """Download a file from a URL to a given or temporary path. An online resource is always downloaded to a temporary file and moved to the specified path only after the download is complete and the checksum (if given) matches. If path is None, the temporary file is returned and the client code is responsible for cleaning it up. """ # Make sure the drive letter in windows-style file paths isn't # interpreted as a URL scheme. if re.match(r"^[A-Za-z]:\\", url): url = 'file:' + url parsed_url = urlparse(url, 'file') url_scheme, _, url_path = parsed_url[:3] if url_scheme == 'file': self.logger.debug('Using local resource %s' % url) if not check_md5sum(url_path, md5sum): raise ChecksumError( 'MD5 checksum mismatch for local resource at %r.' % url_path) return locate_at(url_path, path), False if self.offline: raise zc.buildout.UserError( "Couldn't download %r in offline mode." % url) self.logger.info('Downloading %s' % url) handle, tmp_path = tempfile.mkstemp(prefix='buildout-') os.close(handle) try: tmp_path, headers = urlretrieve(url, tmp_path) if not check_md5sum(tmp_path, md5sum): raise ChecksumError( 'MD5 checksum mismatch downloading %r' % url) except IOError: e = sys.exc_info()[1] os.remove(tmp_path) raise zc.buildout.UserError("Error downloading extends for URL " "%s: %s" % (url, e)) except Exception: os.remove(tmp_path) raise if path: shutil.move(tmp_path, path) return path, False else: return tmp_path, True def filename(self, url): """Determine a file name from a URL according to the configuration. """ if self.hash_name: return md5(url.encode()).hexdigest() else: if re.match(r"^[A-Za-z]:\\", url): url = 'file:' + url parsed = urlparse(url, 'file') url_path = parsed[2] if parsed[0] == 'file': while True: url_path, name = os.path.split(url_path) if name: return name if not url_path: break else: for name in reversed(url_path.split('/')): if name: return name url_host, url_port = parsed[-2:] return '%s:%s' % (url_host, url_port) def check_md5sum(path, md5sum): """Tell whether the MD5 checksum of the file at path matches. No checksum being given is considered a match. """ if md5sum is None: return True f = open(path, 'rb') checksum = md5() try: chunk = f.read(2**16) while chunk: checksum.update(chunk) chunk = f.read(2**16) return checksum.hexdigest() == md5sum finally: f.close() def remove(path): if os.path.exists(path): os.remove(path) def locate_at(source, dest): if dest is None or realpath(dest) == realpath(source): return source if os.path.isdir(source): shutil.copytree(source, dest) else: try: os.link(source, dest) except (AttributeError, OSError): shutil.copyfile(source, dest) return dest ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675670.0 zc_buildout-4.1.6/src/zc/buildout/easy_install.py0000644000076500000240000022305414773460426021572 0ustar00mauritsstaff############################################################################# # # Copyright (c) 2005 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Python easy_install API This module provides a high-level Python API for installing packages. It doesn't install scripts. It uses setuptools and requires it to be installed. """ import copy import distutils.errors import errno import glob import logging import operator import os import pkg_resources import py_compile import re import setuptools.archive_util import setuptools.command.easy_install import setuptools.command.setopt import setuptools.package_index import shutil import subprocess import sys import tempfile import zc.buildout import zc.buildout.rmtree from packaging import specifiers from packaging import utils as packaging_utils from pkg_resources import Distribution from setuptools.wheel import Wheel from zc.buildout import WINDOWS from zc.buildout.utils import normalize_name import warnings import csv BIN_SCRIPTS = 'Scripts' if WINDOWS else 'bin' warnings.filterwarnings( 'ignore', '.+is being parsed as a legacy, non PEP 440, version') _oprp = getattr(os.path, 'realpath', lambda path: path) def realpath(path): return os.path.normcase(os.path.abspath(_oprp(path))) default_index_url = os.environ.get( 'buildout_testing_index_url', 'https://pypi.org/simple', ) logger = logging.getLogger('zc.buildout.easy_install') url_match = re.compile('[a-z0-9+.-]+://').match is_source_encoding_line = re.compile(r'coding[:=]\s*([-\w.]+)').search # Source encoding regex from http://www.python.org/dev/peps/pep-0263/ is_win32 = sys.platform == 'win32' is_jython = sys.platform.startswith('java') if is_jython: import java.lang.System jython_os_name = (java.lang.System.getProperties()['os.name']).lower() # Include buildout and setuptools eggs in paths. We get this # initially from the entire working set. Later, we'll use the install # function to narrow to just the buildout and setuptools paths. buildout_and_setuptools_path = sorted({d.location for d in pkg_resources.working_set}) setuptools_path = buildout_and_setuptools_path pip_path = buildout_and_setuptools_path logger.debug('before restricting versions: pip_path %r', pip_path) FILE_SCHEME = re.compile('file://', re.I).match DUNDER_FILE_PATTERN = re.compile(r"__file__ = '(?P.+)'$") class _Monkey(object): def __init__(self, module, **kw): mdict = self._mdict = module.__dict__ self._before = mdict.copy() self._overrides = kw def __enter__(self): self._mdict.update(self._overrides) return self def __exit__(self, exc_type, exc_value, traceback): self._mdict.clear() self._mdict.update(self._before) class _NoWarn(object): def warn(self, *args, **kw): pass _no_warn = _NoWarn() class EnvironmentMixin(object): """Mixin class for Environment and PackageIndex for canonicalized names. * pkg_resources defines the Environment class * setuptools defines a PackageIndex class that inherits from Environment * Buildout needs a few fixes that should be used by both. The fixes are needed for this issue, where distributions created by setuptools 69.3+ get a different name than with older versions: https://github.com/buildout/buildout/issues/647 """ def __getitem__(self, project_name): """Return a newest-to-oldest list of distributions for `project_name` Uses case-insensitive `project_name` comparison, assuming all the project's distributions use their project's name converted to all lowercase as their key. """ distribution_key = normalize_name(project_name) return self._distmap.get(distribution_key, []) def add(self, dist): """Add `dist` if we ``can_add()`` it and it has not already been added """ if self.can_add(dist) and dist.has_version(): # Instead of 'dist.key' we add a normalized version. distribution_key = normalize_name(dist.key) dists = self._distmap.setdefault(distribution_key, []) if dist not in dists: dists.append(dist) dists.sort(key=operator.attrgetter('hashcmp'), reverse=True) class Environment(EnvironmentMixin, pkg_resources.Environment): """Buildout version of Environment with canonicalized names. * pkg_resources defines the Environment class * setuptools defines a PackageIndex class that inherits from Environment * Buildout needs a few fixes that should be used by both. The fixes are needed for this issue, where distributions created by setuptools 69.3+ get a different name than with older versions: https://github.com/buildout/buildout/issues/647 """ class AllowHostsPackageIndex(EnvironmentMixin, setuptools.package_index.PackageIndex): """Will allow urls that are local to the system. No matter what is allow_hosts. """ def url_ok(self, url, fatal=False): if FILE_SCHEME(url): return True # distutils has its own logging, which can't be hooked / suppressed, # so we monkey-patch the 'log' submodule to suppress the stupid # "Link to ***BLOCKED*** by --allow-hosts" message. with _Monkey(setuptools.package_index, log=_no_warn): return setuptools.package_index.PackageIndex.url_ok( self, url, False) _indexes = {} def _get_index(index_url, find_links, allow_hosts=('*',)): key = index_url, tuple(find_links) index = _indexes.get(key) if index is not None: return index if index_url is None: index_url = default_index_url if index_url.startswith('file://'): index_url = index_url[7:] index = AllowHostsPackageIndex(index_url, hosts=allow_hosts) if find_links: index.add_find_links(find_links) _indexes[key] = index return index clear_index_cache = _indexes.clear if is_win32: # work around spawn lamosity on windows # XXX need safe quoting (see the subproces.list2cmdline) and test def _safe_arg(arg): return '"%s"' % arg else: _safe_arg = str def call_subprocess(args, **kw): if subprocess.call(args, **kw) != 0: raise Exception( "Failed to run command:\n%s" % repr(args)[1:-1]) def _execute_permission(): current_umask = os.umask(0o022) # os.umask only returns the current umask if you also give it one, so we # have to give it a dummy one and immediately set it back to the real # value... Distribute does the same. os.umask(current_umask) return 0o777 - current_umask def get_namespace_package_paths(dist): """ Generator of the expected pathname of each __init__.py file of the namespaces of a distribution. """ base = [dist.location] init = ['__init__.py'] for namespace in dist.get_metadata_lines('namespace_packages.txt'): yield os.path.join(*(base + namespace.split('.') + init)) def namespace_packages_need_pkg_resources(dist): if os.path.isfile(dist.location): # Zipped egg, with namespaces, surely needs setuptools return True # If they have `__init__.py` files that use pkg_resources and don't # fallback to using `pkgutil`, then they need setuptools/pkg_resources: for path in get_namespace_package_paths(dist): if os.path.isfile(path): with open(path, 'rb') as f: source = f.read() if (source and b'pkg_resources' in source and not b'pkgutil' in source): return True return False def dist_needs_pkg_resources(dist): """ A distribution needs setuptools/pkg_resources added as requirement if: * It has namespace packages declared with: - `pkg_resources.declare_namespace()` * Those namespace packages don't fall back to `pkgutil` * It doesn't have `setuptools/pkg_resources` as requirement already """ return ( dist.has_metadata('namespace_packages.txt') and # This will need to change when `pkg_resources` gets its own # project: 'setuptools' not in {r.project_name for r in dist.requires()} and namespace_packages_need_pkg_resources(dist) ) class Installer(object): _versions = {} _required_by = {} _picked_versions = {} _download_cache = None _install_from_cache = False _prefer_final = True _use_dependency_links = True _allow_picked_versions = True _store_required_by = False _allow_unknown_extras = False def __init__(self, dest=None, links=(), index=None, executable=sys.executable, always_unzip=None, # Backward compat :/ path=None, newest=True, versions=None, use_dependency_links=None, allow_hosts=('*',), check_picked=True, allow_unknown_extras=False, ): assert executable == sys.executable, (executable, sys.executable) self._dest = dest if dest is None else pkg_resources.normalize_path(dest) self._allow_hosts = allow_hosts self._allow_unknown_extras = allow_unknown_extras if self._install_from_cache: if not self._download_cache: raise ValueError("install_from_cache set to true with no" " download cache") links = () index = 'file://' + self._download_cache if use_dependency_links is not None: self._use_dependency_links = use_dependency_links self._links = links = list(self._fix_file_links(links)) if self._download_cache and (self._download_cache not in links): links.insert(0, self._download_cache) self._index_url = index path = (path and path[:] or []) + buildout_and_setuptools_path self._path = path if self._dest is None: newest = False self._newest = newest self._env = self._make_env() self._index = _get_index(index, links, self._allow_hosts) self._requirements_and_constraints = [] self._check_picked = check_picked if versions is not None: self._versions = normalize_versions(versions) def _make_env(self): full_path = self._get_dest_dist_paths() + self._path env = Environment(full_path) # this needs to be called whenever self._env is modified (or we could # make an Environment subclass): self._eggify_env_dest_dists(env, self._dest) return env def _env_rescan_dest(self): self._env.scan(self._get_dest_dist_paths()) self._eggify_env_dest_dists(self._env, self._dest) def _get_dest_dist_paths(self): dest = self._dest if dest is None: return [] eggs = glob.glob(os.path.join(dest, '*.egg')) dists = [os.path.dirname(dist_info) for dist_info in glob.glob(os.path.join(dest, '*', '*.dist-info'))] return list(set(eggs + dists)) @staticmethod def _eggify_env_dest_dists(env, dest): """ Make sure everything found under `dest` is seen as an egg, even if it's some other kind of dist. """ for project_name in env: for dist in env[project_name]: if os.path.dirname(dist.location) == dest: dist.precedence = pkg_resources.EGG_DIST def _version_conflict_information(self, name): """Return textual requirements/constraint information for debug purposes We do a very simple textual search, as that filters out most extraneous information without missing anything. """ output = [ "Version and requirements information containing %s:" % name] version_constraint = self._versions.get(name) if version_constraint: output.append( "[versions] constraint on %s: %s" % (name, version_constraint)) output += [line for line in self._requirements_and_constraints if name.lower() in line.lower()] return '\n '.join(output) def _satisfied(self, req, source=None): dists = [dist for dist in self._env[req.project_name] if dist in req] if not dists: logger.debug('We have no distributions for %s that satisfies %r.', req.project_name, str(req)) return None, self._obtain(req, source) # Note that dists are sorted from best to worst, as promised by # env.__getitem__ for dist in dists: if (dist.precedence == pkg_resources.DEVELOP_DIST): logger.debug('We have a develop egg: %s', dist) return dist, None # Special common case, we have a specification for a single version: specs = req.specs if len(specs) == 1 and specs[0][0] == '==': logger.debug('We have the distribution that satisfies %r.', str(req)) return dists[0], None if self._prefer_final: fdists = [dist for dist in dists if self._final_version(dist.parsed_version) ] if fdists: # There are final dists, so only use those dists = fdists if not self._newest: # We don't need the newest, so we'll use the newest one we # find, which is the first returned by # Environment.__getitem__. return dists[0], None best_we_have = dists[0] # Because dists are sorted from best to worst # We have some installed distros. There might, theoretically, be # newer ones. Let's find out which ones are available and see if # any are newer. We only do this if we're willing to install # something, which is only true if dest is not None: best_available = self._obtain(req, source) if best_available is None: # That's a bit odd. There aren't any distros available. # We should use the best one we have that meets the requirement. logger.debug( 'There are no distros available that meet %r.\n' 'Using our best, %s.', str(req), best_we_have) return best_we_have, None if self._prefer_final: if self._final_version(best_available.parsed_version): if self._final_version(best_we_have.parsed_version): if (best_we_have.parsed_version < best_available.parsed_version ): return None, best_available else: return None, best_available else: if (not self._final_version(best_we_have.parsed_version) and (best_we_have.parsed_version < best_available.parsed_version ) ): return None, best_available else: if (best_we_have.parsed_version < best_available.parsed_version ): return None, best_available logger.debug( 'We have the best distribution that satisfies %r.', str(req)) return best_we_have, None def _call_pip_install(self, spec, dest, dist): tmp = tempfile.mkdtemp(dir=dest) try: paths = call_pip_install(spec, tmp) dists = [] env = Environment(paths) for project in env: dists.extend(env[project]) if not dists: raise zc.buildout.UserError("Couldn't install: %s" % dist) if len(dists) > 1: logger.warn("Installing %s\n" "caused multiple distributions to be installed:\n" "%s\n", dist, '\n'.join(map(str, dists))) else: d = dists[0] if d.project_name != dist.project_name: logger.warn("Installing %s\n" "Caused installation of a distribution:\n" "%s\n" "with a different project name.", dist, d) if d.version != dist.version: logger.warn("Installing %s\n" "Caused installation of a distribution:\n" "%s\n" "with a different version.", dist, d) result = [] for d in dists: # TODO: maybe pass a project_name. But in my testing I did not reach # this code part, so it is hard to say what exactly to use as # project name and if that would help. result.append(_move_to_eggs_dir_and_compile(d, dest)) return result finally: zc.buildout.rmtree.rmtree(tmp) def _obtain(self, requirement, source=None): # initialize out index for this project: index = self._index if index.obtain(requirement) is None: # Nothing is available. return None # Filter the available dists for the requirement and source flag dists = [dist for dist in index[requirement.project_name] if ((dist in requirement) and ((not source) or (dist.precedence == pkg_resources.SOURCE_DIST) ) ) ] # If we prefer final dists, filter for final and use the # result if it is non empty. if self._prefer_final: fdists = [dist for dist in dists if self._final_version(dist.parsed_version) ] if fdists: # There are final dists, so only use those dists = fdists # Now find the best one: best = [] bestv = None for dist in dists: distv = dist.parsed_version if bestv is None or distv > bestv: best = [dist] bestv = distv elif distv == bestv: best.append(dist) if not best: return None if len(best) == 1: return best[0] if self._download_cache: for dist in best: if (realpath(os.path.dirname(dist.location)) == self._download_cache ): return dist best.sort() return best[-1] def _fetch(self, dist, tmp, download_cache): if (download_cache and (realpath(os.path.dirname(dist.location)) == download_cache) ): logger.debug("Download cache has %s at: %s", dist, dist.location) return dist logger.debug("Fetching %s from: %s", dist, dist.location) new_location = self._index.download(dist.location, tmp) if (download_cache and (realpath(new_location) == realpath(dist.location)) and os.path.isfile(new_location) ): # setuptools avoids making extra copies, but we want to copy # to the download cache shutil.copy2(new_location, tmp) new_location = os.path.join(tmp, os.path.basename(new_location)) return dist.clone(location=new_location) def _get_dist(self, requirement, ws): __doing__ = 'Getting distribution for %r.', str(requirement) # Maybe an existing dist is already the best dist that satisfies the # requirement. If not, get a link to an available distribution that # we could download. The method returns a tuple with an existing # dist or an available dist. Either 'dist' is None, or 'avail' # is None, or both are None. dist, avail = self._satisfied(requirement) if dist is None: if self._dest is None: raise zc.buildout.UserError( "We don't have a distribution for %s\n" "and can't install one in offline (no-install) mode.\n" % requirement) logger.info(*__doing__) if avail is None: # We have no existing dist, and none is available for download. raise MissingDistribution(requirement, ws) # We may overwrite distributions, so clear importer # cache. sys.path_importer_cache.clear() tmp = self._download_cache if tmp is None: tmp = tempfile.mkdtemp('get_dist') try: dist = self._fetch(avail, tmp, self._download_cache) if dist is None: raise zc.buildout.UserError( "Couldn't download distribution %s." % avail) dists = [_move_to_eggs_dir_and_compile( dist, self._dest, project_name=requirement.project_name )] for _d in dists: if _d not in ws: ws.add(_d, replace=True) finally: if tmp != self._download_cache: zc.buildout.rmtree.rmtree(tmp) self._env_rescan_dest() dist = self._env.best_match(requirement, ws) logger.info("Got %s.", dist) else: dists = [dist] if dist not in ws: ws.add(dist) if not self._install_from_cache and self._use_dependency_links: self._add_dependency_links_from_dists(dists) if self._check_picked: self._check_picked_requirement_versions(requirement, dists) return dists def _add_dependency_links_from_dists(self, dists): reindex = False links = self._links for dist in dists: if dist.has_metadata('dependency_links.txt'): for link in dist.get_metadata_lines('dependency_links.txt'): link = link.strip() if link not in links: logger.debug('Adding find link %r from %s', link, dist) links.append(link) reindex = True if reindex: self._index = _get_index(self._index_url, links, self._allow_hosts) def _check_picked_requirement_versions(self, requirement, dists): """ Check whether we picked a version and, if we did, report it """ for dist in dists: if not (dist.precedence == pkg_resources.DEVELOP_DIST or (len(requirement.specs) == 1 and requirement.specs[0][0] == '==') ): logger.debug('Picked: %s = %s', dist.project_name, dist.version) self._picked_versions[dist.project_name] = dist.version if not self._allow_picked_versions: msg = NOT_PICKED_AND_NOT_ALLOWED.format( name=dist.project_name, version=dist.version ) raise zc.buildout.UserError(msg) def _maybe_add_setuptools(self, ws, dist): if dist_needs_pkg_resources(dist): # We have a namespace package but no requirement for setuptools if dist.precedence == pkg_resources.DEVELOP_DIST: logger.warning( "Develop distribution: %s\n" "uses namespace packages but the distribution " "does not require setuptools.", dist) requirement = self._constrain( pkg_resources.Requirement.parse('setuptools') ) if ws.find(requirement) is None: self._get_dist(requirement, ws) def _constrain(self, requirement): """Return requirement with optional [versions] constraint added.""" constraint = self._versions.get(requirement.project_name.lower()) if constraint: try: requirement = _constrained_requirement(constraint, requirement) except IncompatibleConstraintError: logger.info(self._version_conflict_information( requirement.project_name.lower())) raise return requirement def install(self, specs, working_set=None): logger.debug('Installing %s.', repr(specs)[1:-1]) self._requirements_and_constraints.append( "Base installation request: %s" % repr(specs)[1:-1]) for_buildout_run = bool(working_set) requirements = [pkg_resources.Requirement.parse(spec) for spec in specs] requirements = [ self._constrain(requirement) for requirement in requirements if not requirement.marker or requirement.marker.evaluate() ] if working_set is None: ws = pkg_resources.WorkingSet([]) else: ws = working_set for requirement in requirements: for dist in self._get_dist(requirement, ws): self._maybe_add_setuptools(ws, dist) # OK, we have the requested distributions and they're in the working # set, but they may have unmet requirements. We'll resolve these # requirements. This is code modified from # pkg_resources.WorkingSet.resolve. We can't reuse that code directly # because we have to constrain our requirements (see # versions_section_ignored_for_dependency_in_favor_of_site_packages in # zc.buildout.tests). requirements.reverse() # Set up the stack. processed = {} # This is a set of processed requirements. best = {} # This is a mapping of package name -> dist. # Note that we don't use the existing environment, because we want # to look for new eggs unless what we have is the best that # matches the requirement. env = Environment(ws.entries) while requirements: # Process dependencies breadth-first. current_requirement = requirements.pop(0) req = self._constrain(current_requirement) if req in processed: # Ignore cyclic or redundant dependencies. continue dist = best.get(req.key) if dist is None: try: dist = env.best_match(req, ws) except pkg_resources.VersionConflict as err: logger.debug( "Version conflict while processing requirement %s " "(constrained to %s)", current_requirement, req) # Installing buildout itself and its extensions and # recipes requires the global # ``pkg_resources.working_set`` to be active, which also # includes all system packages. So there might be # conflicts, which are fine to ignore. We'll grab the # correct version a few lines down. if not for_buildout_run: raise VersionConflict(err, ws) if dist is None: if self._dest: logger.debug('Getting required %r', str(req)) else: logger.debug('Adding required %r', str(req)) self._log_requirement(ws, req) for dist in self._get_dist(req, ws): self._maybe_add_setuptools(ws, dist) if dist not in req: # Oops, the "best" so far conflicts with a dependency. logger.info(self._version_conflict_information(req.key)) raise VersionConflict( pkg_resources.VersionConflict(dist, req), ws) best[req.key] = dist missing_requested = sorted( set(req.extras) - set(dist.extras) ) for missing in missing_requested: logger.warning( '%s does not provide the extra \'%s\'', dist, missing ) if missing_requested: if not self._allow_unknown_extras: raise zc.buildout.UserError( "Couldn't find the required extra. " "This means the requirement is incorrect. " "If the requirement is itself from software you " "requested, then there might be a bug in " "requested software. You can ignore this by " "using 'allow-unknown-extras=true', however " "that may simply cause needed software to be omitted." ) extra_requirements = sorted( set(dist.extras) & set(req.extras) ) else: extra_requirements = dist.requires(req.extras)[::-1] for extra_requirement in extra_requirements: self._requirements_and_constraints.append( "Requirement of %s: %s" % ( current_requirement, extra_requirement)) requirements.extend(extra_requirements) processed[req] = True return ws def build(self, spec, build_ext): requirement = self._constrain(pkg_resources.Requirement.parse(spec)) dist, avail = self._satisfied(requirement, 1) if dist is not None: return [dist.location] # Retrieve the dist: if avail is None: raise zc.buildout.UserError( "Couldn't find a source distribution for %r." % str(requirement)) if self._dest is None: raise zc.buildout.UserError( "We don't have a distribution for %s\n" "and can't build one in offline (no-install) mode.\n" % requirement ) logger.debug('Building %r', spec) tmp = self._download_cache if tmp is None: tmp = tempfile.mkdtemp('get_dist') try: dist = self._fetch(avail, tmp, self._download_cache) build_tmp = tempfile.mkdtemp('build') try: setuptools.archive_util.unpack_archive(dist.location, build_tmp) if os.path.exists(os.path.join(build_tmp, 'setup.py')): base = build_tmp else: setups = glob.glob( os.path.join(build_tmp, '*', 'setup.py')) if not setups: raise distutils.errors.DistutilsError( "Couldn't find a setup script in %s" % os.path.basename(dist.location) ) if len(setups) > 1: raise distutils.errors.DistutilsError( "Multiple setup scripts in %s" % os.path.basename(dist.location) ) base = os.path.dirname(setups[0]) setup_cfg = os.path.join(base, 'setup.cfg') if not os.path.exists(setup_cfg): f = open(setup_cfg, 'w') f.close() setuptools.command.setopt.edit_config( setup_cfg, dict(build_ext=build_ext)) dists = self._call_pip_install(base, self._dest, dist) return [dist.location for dist in dists] finally: zc.buildout.rmtree.rmtree(build_tmp) finally: if tmp != self._download_cache: zc.buildout.rmtree.rmtree(tmp) def _fix_file_links(self, links): for link in links: if link.startswith('file://') and link[-1] != '/': if os.path.isdir(link[7:]): # work around excessive restriction in setuptools: link += '/' yield link def _log_requirement(self, ws, req): if (not logger.isEnabledFor(logging.DEBUG) and not Installer._store_required_by): # Sorting the working set and iterating over it's requirements # is expensive, so short circuit the work if it won't even be # logged. When profiling a simple buildout with 10 parts with # identical and large working sets, this resulted in a # decrease of run time from 93.411 to 15.068 seconds, about a # 6 fold improvement. return ws = list(ws) ws.sort() for dist in ws: if req in dist.requires(): logger.debug(" required by %s." % dist) req_ = str(req) if req_ not in Installer._required_by: Installer._required_by[req_] = set() Installer._required_by[req_].add(str(dist.as_requirement())) def _final_version(self, parsed_version): return not parsed_version.is_prerelease def normalize_versions(versions): """Return version dict with keys normalized to lowercase. PyPI is case-insensitive and not all distributions are consistent in their own naming. """ return dict([(k.lower(), v) for (k, v) in versions.items()]) def default_versions(versions=None): old = Installer._versions if versions is not None: Installer._versions = normalize_versions(versions) return old def download_cache(path=-1): old = Installer._download_cache if path != -1: if path: path = realpath(path) Installer._download_cache = path return old def install_from_cache(setting=None): old = Installer._install_from_cache if setting is not None: Installer._install_from_cache = bool(setting) return old def prefer_final(setting=None): old = Installer._prefer_final if setting is not None: Installer._prefer_final = bool(setting) return old def use_dependency_links(setting=None): old = Installer._use_dependency_links if setting is not None: Installer._use_dependency_links = bool(setting) return old def allow_picked_versions(setting=None): old = Installer._allow_picked_versions if setting is not None: Installer._allow_picked_versions = bool(setting) return old def store_required_by(setting=None): old = Installer._store_required_by if setting is not None: Installer._store_required_by = bool(setting) return old def get_picked_versions(): picked_versions = sorted(Installer._picked_versions.items()) required_by = Installer._required_by return (picked_versions, required_by) def install(specs, dest, links=(), index=None, executable=sys.executable, always_unzip=None, # Backward compat :/ path=None, working_set=None, newest=True, versions=None, use_dependency_links=None, allow_hosts=('*',), include_site_packages=None, allowed_eggs_from_site_packages=None, check_picked=True, allow_unknown_extras=False, ): assert executable == sys.executable, (executable, sys.executable) assert include_site_packages is None assert allowed_eggs_from_site_packages is None installer = Installer(dest, links, index, sys.executable, always_unzip, path, newest, versions, use_dependency_links, allow_hosts=allow_hosts, check_picked=check_picked, allow_unknown_extras=allow_unknown_extras) return installer.install(specs, working_set) buildout_and_setuptools_dists = list(install(['zc.buildout'], None, check_picked=False)) buildout_and_setuptools_path = sorted({d.location for d in buildout_and_setuptools_dists}) pip_dists = [d for d in buildout_and_setuptools_dists if d.project_name != 'zc.buildout'] pip_path = sorted({d.location for d in pip_dists}) logger.debug('after restricting versions: pip_path %r', pip_path) pip_pythonpath = os.pathsep.join(pip_path) setuptools_path = pip_path setuptools_pythonpath = pip_pythonpath def build(spec, dest, build_ext, links=(), index=None, executable=sys.executable, path=None, newest=True, versions=None, allow_hosts=('*',)): assert executable == sys.executable, (executable, sys.executable) installer = Installer(dest, links, index, executable, True, path, newest, versions, allow_hosts=allow_hosts) return installer.build(spec, build_ext) def _rm(*paths): for path in paths: if os.path.isdir(path): zc.buildout.rmtree.rmtree(path) elif os.path.exists(path): os.remove(path) def _copyeggs(src, dest, suffix, undo): result = [] undo.append(lambda : _rm(*result)) for name in os.listdir(src): if name.endswith(suffix): new = os.path.join(dest, name) _rm(new) os.rename(os.path.join(src, name), new) result.append(new) assert len(result) == 1, str(result) undo.pop() return result[0] _develop_distutils_scripts = {} def _detect_distutils_scripts(directory): """Record detected distutils scripts from develop eggs ``setup.py develop`` doesn't generate metadata on distutils scripts, in contrast to ``setup.py install``. So we have to store the information for later. """ dir_contents = os.listdir(directory) egginfo_filenames = [filename for filename in dir_contents if filename.endswith('.egg-link')] if not egginfo_filenames: return egg_name = egginfo_filenames[0].replace('.egg-link', '') marker = 'EASY-INSTALL-DEV-SCRIPT' scripts_found = [] for filename in dir_contents: if filename.endswith('.exe'): continue filepath = os.path.join(directory, filename) if not os.path.isfile(filepath): continue with open(filepath) as fp: dev_script_content = fp.read() if marker in dev_script_content: # The distutils bin script points at the actual file we need. for line in dev_script_content.splitlines(): match = DUNDER_FILE_PATTERN.search(line) if match: # The ``__file__ =`` line in the generated script points # at the actual distutils script we need. actual_script_filename = match.group('filename') with open(actual_script_filename) as fp: actual_script_content = fp.read() scripts_found.append([filename, actual_script_content]) if scripts_found: logger.debug( "Distutils scripts found for develop egg %s: %s", egg_name, scripts_found) _develop_distutils_scripts[egg_name] = scripts_found def develop(setup, dest, build_ext=None, executable=sys.executable): assert executable == sys.executable, (executable, sys.executable) if os.path.isdir(setup): directory = setup setup = os.path.join(directory, 'setup.py') else: directory = os.path.dirname(setup) undo = [] try: if build_ext: setup_cfg = os.path.join(directory, 'setup.cfg') if os.path.exists(setup_cfg): os.rename(setup_cfg, setup_cfg+'-develop-aside') def restore_old_setup(): if os.path.exists(setup_cfg): os.remove(setup_cfg) os.rename(setup_cfg+'-develop-aside', setup_cfg) undo.append(restore_old_setup) else: f = open(setup_cfg, 'w') f.close() undo.append(lambda: os.remove(setup_cfg)) setuptools.command.setopt.edit_config( setup_cfg, dict(build_ext=build_ext)) fd, tsetup = tempfile.mkstemp() undo.append(lambda: os.remove(tsetup)) undo.append(lambda: os.close(fd)) os.write(fd, (runsetup_template % dict( setupdir=directory, setup=setup, __file__ = setup, )).encode()) tmp3 = tempfile.mkdtemp('build', dir=dest) undo.append(lambda : zc.buildout.rmtree.rmtree(tmp3)) args = [executable, tsetup, '-q', 'develop', '-mN', '-d', tmp3] log_level = logger.getEffectiveLevel() if log_level <= 0: if log_level == 0: del args[2] else: args[2] == '-v' if log_level < logging.DEBUG: logger.debug("in: %r\n%s", directory, ' '.join(args)) call_subprocess(args) _detect_distutils_scripts(tmp3) return _copyeggs(tmp3, dest, '.egg-link', undo) finally: undo.reverse() [f() for f in undo] def working_set(specs, executable, path=None, include_site_packages=None, allowed_eggs_from_site_packages=None): # Backward compat: if path is None: path = executable else: assert executable == sys.executable, (executable, sys.executable) assert include_site_packages is None assert allowed_eggs_from_site_packages is None return install(specs, None, path=path) def scripts(reqs, working_set, executable, dest=None, scripts=None, extra_paths=(), arguments='', interpreter=None, initialization='', relative_paths=False, ): assert executable == sys.executable, (executable, sys.executable) path = [dist.location for dist in working_set] path.extend(extra_paths) # order preserving unique unique_path = [] for p in path: if p not in unique_path: unique_path.append(p) path = [realpath(p) for p in unique_path] generated = [] if isinstance(reqs, str): raise TypeError('Expected iterable of requirements or entry points,' ' got string.') if initialization: initialization = '\n'+initialization+'\n' entry_points = [] distutils_scripts = [] for req in reqs: if isinstance(req, str): orig_req = pkg_resources.Requirement.parse(req) if orig_req.marker and not orig_req.marker.evaluate(): continue dist = None if packaging_utils.is_normalized_name(orig_req.name): dist = working_set.find(orig_req) if dist is None: raise ValueError( f"Could not find requirement '{orig_req.name}' in working set. " ) else: # First try finding the package by its canonical name. canonicalized_name = packaging_utils.canonicalize_name(orig_req.name) canonical_req = pkg_resources.Requirement.parse(canonicalized_name) dist = working_set.find(canonical_req) if dist is None: # Now try to find the package by the original name we got from # the requirements. This may succeed with setuptools versions # older than 75.8.2. dist = working_set.find(orig_req) if dist is None: raise ValueError( f"Could not find requirement '{orig_req.name}' in working " f"set. Could not find it with normalized " f"'{canonicalized_name}' either." ) # regular console_scripts entry points for name in pkg_resources.get_entry_map(dist, 'console_scripts'): entry_point = dist.get_entry_info('console_scripts', name) entry_points.append( (name, entry_point.module_name, '.'.join(entry_point.attrs)) ) # The metadata on "old-style" distutils scripts is not retained by # distutils/setuptools, except by placing the original scripts in # /EGG-INFO/scripts/. if dist.metadata_isdir('scripts'): # egg-info metadata from installed egg. for name in dist.metadata_listdir('scripts'): if dist.metadata_isdir('scripts/' + name): # Probably Python 3 __pycache__ directory. continue if name.lower().endswith('.exe'): # windows: scripts are implemented with 2 files # the .exe gets also into metadata_listdir # get_metadata chokes on the binary continue contents = dist.get_metadata('scripts/' + name) distutils_scripts.append((name, contents)) elif dist.key in _develop_distutils_scripts: # Development eggs don't have metadata about scripts, so we # collected it ourselves in develop()/ and # _detect_distutils_scripts(). for name, contents in _develop_distutils_scripts[dist.key]: distutils_scripts.append((name, contents)) else: entry_points.append(req) entry_points_names = [] for name, module_name, attrs in entry_points: entry_points_names.append(name) if scripts is not None: sname = scripts.get(name) if sname is None: continue else: sname = name sname = os.path.join(dest, sname) spath, rpsetup = _relative_path_and_setup(sname, path, relative_paths) generated.extend( _script(module_name, attrs, spath, sname, arguments, initialization, rpsetup) ) # warn when a script name passed in 'scripts' argument # is not defined in an entry point. if scripts is not None: for name, target in scripts.items(): if name not in entry_points_names: if name == target: logger.warning("Could not generate script '%s' as it is not " "defined in the egg entry points.", name) else: logger.warning("Could not generate script '%s' as script " "'%s' is not defined in the egg entry points.", name, target) for name, contents in distutils_scripts: if scripts is not None: sname = scripts.get(name) if sname is None: continue else: sname = name sname = os.path.join(dest, sname) spath, rpsetup = _relative_path_and_setup(sname, path, relative_paths) generated.extend( _distutils_script(spath, sname, contents, initialization, rpsetup) ) if interpreter: sname = os.path.join(dest, interpreter) spath, rpsetup = _relative_path_and_setup(sname, path, relative_paths) generated.extend(_pyscript(spath, sname, rpsetup, initialization)) return generated def _relative_path_and_setup(sname, path, relative_paths): if relative_paths: relative_paths = os.path.normcase(relative_paths) sname = os.path.normcase(os.path.abspath(sname)) spath = ',\n '.join( [_relativitize(os.path.normcase(path_item), sname, relative_paths) for path_item in path] ) rpsetup = relative_paths_setup for i in range(_relative_depth(relative_paths, sname)): rpsetup += "base = os.path.dirname(base)\n" else: spath = repr(path)[1:-1].replace(', ', ',\n ') rpsetup = '' return spath, rpsetup def _relative_depth(common, path): n = 0 while 1: dirname = os.path.dirname(path) if dirname == path: raise AssertionError("dirname of %s is the same" % dirname) if dirname == common: break n += 1 path = dirname return n def _relative_path(common, path): r = [] while 1: dirname, basename = os.path.split(path) r.append(basename) if dirname == common: break if dirname == path: raise AssertionError("dirname of %s is the same" % dirname) path = dirname r.reverse() return os.path.join(*r) def _relativitize(path, script, relative_paths): if path == script: raise AssertionError("path == script") if path == relative_paths: return "base" common = os.path.dirname(os.path.commonprefix([path, script])) if (common == relative_paths or common.startswith(os.path.join(relative_paths, '')) ): return "join(base, %r)" % _relative_path(common, path) else: return repr(path) relative_paths_setup = """ import os join = os.path.join base = os.path.dirname(os.path.abspath(os.path.realpath(__file__))) """ def _script(module_name, attrs, path, dest, arguments, initialization, rsetup): if is_win32: dest += '-script.py' python = _safe_arg(sys.executable) contents = script_template % dict( python = python, path = path, module_name = module_name, attrs = attrs, arguments = arguments, initialization = initialization, relative_paths_setup = rsetup, ) return _create_script(contents, dest) def _distutils_script(path, dest, script_content, initialization, rsetup): if is_win32: dest += '-script.py' lines = script_content.splitlines(True) if not ('#!' in lines[0]) and ('python' in lines[0]): # The script doesn't follow distutil's rules. Ignore it. return [] lines = lines[1:] # Strip off the first hashbang line. line_with_first_import = len(lines) for line_number, line in enumerate(lines): if not 'import' in line: continue if not (line.startswith('import') or line.startswith('from')): continue if '__future__' in line: continue line_with_first_import = line_number break before = ''.join(lines[:line_with_first_import]) after = ''.join(lines[line_with_first_import:]) python = _safe_arg(sys.executable) contents = distutils_script_template % dict( python = python, path = path, initialization = initialization, relative_paths_setup = rsetup, before = before, after = after ) return _create_script(contents, dest) def _file_changed(filename, old_contents, mode='r'): try: with open(filename, mode) as f: return f.read() != old_contents except EnvironmentError as e: if e.errno == errno.ENOENT: return True else: raise def _create_script(contents, dest): generated = [] script = dest changed = _file_changed(dest, contents) if is_win32: # generate exe file and give the script a magic name: win32_exe = os.path.splitext(dest)[0] # remove ".py" if win32_exe.endswith('-script'): win32_exe = win32_exe[:-7] # remove "-script" win32_exe = win32_exe + '.exe' # add ".exe" new_data = setuptools.command.easy_install.get_win_launcher('cli') if _file_changed(win32_exe, new_data, 'rb'): # Only write it if it's different. with open(win32_exe, 'wb') as f: f.write(new_data) generated.append(win32_exe) if changed: with open(dest, 'w') as f: f.write(contents) logger.info( "Generated script %r.", # Normalize for windows script.endswith('-script.py') and script[:-10] or script) try: os.chmod(dest, _execute_permission()) except (AttributeError, os.error): pass generated.append(dest) return generated if is_jython and jython_os_name == 'linux': script_header = '#!/usr/bin/env %(python)s' else: script_header = '#!%(python)s' script_template = script_header + '''\ %(relative_paths_setup)s import sys sys.path[0:0] = [ %(path)s, ] %(initialization)s import %(module_name)s if __name__ == '__main__': sys.exit(%(module_name)s.%(attrs)s(%(arguments)s)) ''' distutils_script_template = script_header + ''' %(before)s %(relative_paths_setup)s import sys sys.path[0:0] = [ %(path)s, ] %(initialization)s %(after)s''' def _pyscript(path, dest, rsetup, initialization=''): generated = [] script = dest if is_win32: dest += '-script.py' python = _safe_arg(sys.executable) if path: path += ',' # Courtesy comma at the end of the list. contents = py_script_template % dict( python = python, path = path, relative_paths_setup = rsetup, initialization=initialization, ) changed = _file_changed(dest, contents) if is_win32: # generate exe file and give the script a magic name: exe = script + '.exe' with open(exe, 'wb') as f: f.write( pkg_resources.resource_string('setuptools', 'cli.exe') ) generated.append(exe) if changed: with open(dest, 'w') as f: f.write(contents) try: os.chmod(dest, _execute_permission()) except (AttributeError, os.error): pass logger.info("Generated interpreter %r.", script) generated.append(dest) return generated py_script_template = script_header + '''\ %(relative_paths_setup)s import sys sys.path[0:0] = [ %(path)s ] %(initialization)s _interactive = True if len(sys.argv) > 1: # The Python interpreter wrapper allows only some of the options that a # "regular" Python interpreter accepts. _options, _args = __import__("getopt").getopt(sys.argv[1:], 'Iic:m:') _interactive = False for (_opt, _val) in _options: if _opt == '-i': _interactive = True elif _opt == '-c': exec(_val) elif _opt == '-m': sys.argv[1:] = _args _args = [] __import__("runpy").run_module( _val, {}, "__main__", alter_sys=True) elif _opt == '-I': # Allow yet silently ignore the `-I` option. The original behaviour # for this option is to create an isolated Python runtime. It was # deemed acceptable to allow the option here as this Python wrapper # is isolated from the system Python already anyway. # The specific use-case that led to this change is how the Python # language extension for Visual Studio Code calls the Python # interpreter when initializing the extension. pass if _args: sys.argv[:] = _args __file__ = _args[0] del _options, _args with open(__file__) as __file__f: exec(compile(__file__f.read(), __file__, "exec")) if _interactive: del _interactive __import__("code").interact(banner="", local=globals()) ''' runsetup_template = """ import sys sys.path.insert(0, %%(setupdir)r) sys.path[0:0] = %r import os, setuptools __file__ = %%(__file__)r os.chdir(%%(setupdir)r) sys.argv[0] = %%(setup)r with open(%%(setup)r) as f: exec(compile(f.read(), %%(setup)r, 'exec')) """ % setuptools_path class VersionConflict(zc.buildout.UserError): def __init__(self, err, ws): ws = list(ws) ws.sort() self.err, self.ws = err, ws def __str__(self): result = ["There is a version conflict."] if len(self.err.args) == 2: existing_dist, req = self.err.args result.append("We already have: %s" % existing_dist) for dist in self.ws: if req in dist.requires(): result.append("but %s requires %r." % (dist, str(req))) else: # The error argument is already a nice error string. result.append(self.err.args[0]) return '\n'.join(result) class MissingDistribution(zc.buildout.UserError): def __init__(self, req, ws): ws = list(ws) ws.sort() self.data = req, ws def __str__(self): req, ws = self.data return "Couldn't find a distribution for %r." % str(req) def redo_pyc(egg): if not os.path.isdir(egg): return for dirpath, dirnames, filenames in os.walk(egg): for filename in filenames: if not filename.endswith('.py'): continue filepath = os.path.join(dirpath, filename) if not (os.path.exists(filepath+'c') or os.path.exists(filepath+'o')): # If it wasn't compiled, it may not be compilable continue # OK, it looks like we should try to compile. # Remove old files. for suffix in 'co': if os.path.exists(filepath+suffix): os.remove(filepath+suffix) # Compile under current optimization try: py_compile.compile(filepath) except py_compile.PyCompileError: logger.warning("Couldn't compile %s", filepath) else: # Recompile under other optimization. :) args = [sys.executable] if __debug__: args.append('-O') args.extend(['-m', 'py_compile', filepath]) call_subprocess(args) def _constrained_requirement(constraint, requirement): assert isinstance(requirement, pkg_resources.Requirement) if constraint[0] not in '<>': if constraint.startswith('='): assert constraint.startswith('==') version = constraint[2:] else: version = constraint constraint = '==' + constraint if version not in requirement: msg = ("The requirement (%r) is not allowed by your [versions] " "constraint (%s)" % (str(requirement), version)) raise IncompatibleConstraintError(msg) specifier = specifiers.SpecifierSet(constraint) else: specifier = requirement.specifier & constraint constrained = copy.deepcopy(requirement) constrained.specifier = specifier return pkg_resources.Requirement.parse(str(constrained)) class IncompatibleConstraintError(zc.buildout.UserError): """A specified version is incompatible with a given requirement. """ IncompatibleVersionError = IncompatibleConstraintError # Backward compatibility def call_pip_install(spec, dest): """ Call `pip install` from a subprocess to install a distribution specified by `spec` into `dest`. Returns all the paths inside `dest` created by the above. """ args = [sys.executable, '-m', 'pip', 'install', '--no-deps', '-t', dest] level = logger.getEffectiveLevel() if level >= logging.INFO: args.append('-q') else: args.append('-v') args.append(spec) try: from pip._internal.cli.cmdoptions import no_python_version_warning HAS_WARNING_OPTION = True except ImportError: HAS_WARNING_OPTION = False if HAS_WARNING_OPTION: if not hasattr(call_pip_install, 'displayed'): call_pip_install.displayed = True else: args.append('--no-python-version-warning') env = copy.copy(os.environ) python_path = copy.copy(pip_path) python_path.append(env.get('PYTHONPATH', '')) env['PYTHONPATH'] = os.pathsep.join(python_path) if level <= logging.DEBUG: logger.debug('Running pip install:\n"%s"\npath=%s\n', '" "'.join(args), pip_path) sys.stdout.flush() # We want any pending output first exit_code = subprocess.call(list(args), env=env) if exit_code: logger.error( "An error occurred when trying to install %s. " "Look above this message for any errors that " "were output by pip install.", spec) sys.exit(1) split_entries = [os.path.splitext(entry) for entry in os.listdir(dest)] try: distinfo_dir = [ base + ext for base, ext in split_entries if ext == ".dist-info" ][0] except IndexError: logger.error( "No .dist-info directory after successful pip install of %s", spec) raise return make_egg_after_pip_install(dest, distinfo_dir) def make_egg_after_pip_install(dest, distinfo_dir): """build properly named egg directory""" # `pip install` does not build the namespace aware __init__.py files # but they are needed in egg directories. # Add them before moving files setup by pip namespace_packages_file = os.path.join( dest, distinfo_dir, 'namespace_packages.txt' ) if os.path.isfile(namespace_packages_file): with open(namespace_packages_file) as f: namespace_packages = [ line.strip().replace('.', os.path.sep) for line in f.readlines() ] for namespace_package in namespace_packages: namespace_package_dir = os.path.join(dest, namespace_package) if os.path.isdir(namespace_package_dir): init_py_file = os.path.join( namespace_package_dir, '__init__.py') with open(init_py_file, 'w') as f: f.write( "__import__('pkg_resources')." "declare_namespace(__name__)" ) # Remove `bin` directory if needed # as there is no way to avoid script installation # when running `pip install` entry_points_file = os.path.join(dest, distinfo_dir, 'entry_points.txt') if os.path.isfile(entry_points_file): with open(entry_points_file) as f: content = f.read() if "console_scripts" in content or "gui_scripts" in content: bin_dir = os.path.join(dest, BIN_SCRIPTS) if os.path.exists(bin_dir): shutil.rmtree(bin_dir) # Make properly named new egg dir distro = list(pkg_resources.find_distributions(dest))[0] base = "{}-{}".format( distro.egg_name(), pkg_resources.get_supported_platform() ) egg_name = base + '.egg' new_distinfo_dir = base + '.dist-info' egg_dir = os.path.join(dest, egg_name) os.mkdir(egg_dir) # Move ".dist-info" dir into new egg dir os.rename( os.path.join(dest, distinfo_dir), os.path.join(egg_dir, new_distinfo_dir) ) top_level_file = os.path.join(egg_dir, new_distinfo_dir, 'top_level.txt') if os.path.isfile(top_level_file): with open(top_level_file) as f: top_levels = filter( (lambda x: len(x) != 0), [line.strip() for line in f.readlines()] ) else: top_levels = () # Move all top_level modules or packages for top_level in top_levels: # as package top_level_dir = os.path.join(dest, top_level) if os.path.exists(top_level_dir): shutil.move(top_level_dir, egg_dir) continue # as module top_level_py = top_level_dir + '.py' if os.path.exists(top_level_py): shutil.move(top_level_py, egg_dir) top_level_pyc = top_level_dir + '.pyc' if os.path.exists(top_level_pyc): shutil.move(top_level_pyc, egg_dir) continue record_file = os.path.join(egg_dir, new_distinfo_dir, 'RECORD') if os.path.isfile(record_file): with open(record_file, newline='') as f: all_files = [row[0] for row in csv.reader(f)] # There might be some c extensions left over for entry in all_files: if entry.endswith(('.pyc', '.pyo')): continue dest_entry = os.path.join(dest, entry) # work around pip install -t bug that leaves entries in RECORD # that starts with '../../' if not os.path.abspath(dest_entry).startswith(dest): continue egg_entry = os.path.join(egg_dir, entry) if os.path.exists(dest_entry) and not os.path.exists(egg_entry): egg_entry_dir = os.path.dirname(egg_entry) if not os.path.exists(egg_entry_dir): os.makedirs(egg_entry_dir) os.rename(dest_entry, egg_entry) return [egg_dir] def unpack_egg(location, dest): # Buildout 2 no longer installs zipped eggs, # so we always want to unpack it. dest = os.path.join(dest, os.path.basename(location)) setuptools.archive_util.unpack_archive(location, dest) def unpack_wheel(location, dest): wheel = Wheel(location) wheel.install_as_egg(os.path.join(dest, wheel.egg_name())) UNPACKERS = { '.egg': unpack_egg, '.whl': unpack_wheel, } def _get_matching_dist_in_location(dist, location): """ Check if `locations` contain only the one intended dist. Return the dist with metadata in the new location. """ # Getting the dist from the environment causes the distribution # meta data to be read. Cloning isn't good enough. We must compare # dist.parsed_version, not dist.version, because one or the other # may be normalized (e.g., 3.3 becomes 3.3.0 when downloaded from # PyPI.) env = Environment([location]) dists = [ d for project_name in env for d in env[project_name] ] dist_infos = [ (normalize_name(d.project_name), d.parsed_version) for d in dists ] if dist_infos == [(normalize_name(dist.project_name), dist.parsed_version)]: return dists.pop() def _maybe_copy_and_rename_wheel(dist, dest, project_name): """Maybe copy and rename wheel. Then move to eggs dir and compile. We are called by _move_to_eggs_dir_and_compile and call it ourselves when we indeed needed to rename the wheel. Return the new dist or None. So why do we do this? We need to check a special case: - zest_releaser-9.4.0-py3-none-any.whl with an underscore results in: zest_releaser-9.4.0-py3.13.egg In the resulting `bin/fullrease` script the zest.releaser distribution is not found. - So in this function we copy and rename the wheel to: zest.releaser-9.4.0-py3-none-any.whl with a dot, which results in: zest.releaser-9.4.0-py3-none-any.whl The resulting `bin/fullrease` script works fine. See https://github.com/buildout/buildout/issues/686 So check if we should rename the wheel before handling it. Note that source dists do not have this problem. Or not anymore, after some fixes in Buildout last year: - zest_releaser-9.4.0.tar.gz with an underscore results in (in my case): zest_releaser-9.4.0-py3.13-macosx-14.7-x86_64.egg And this works fine, despite having an underscore. The egg has a dist-info directory: zest_releaser-9.4.0-py3.13-macosx-14.7-x86_64.dist-info The egg from any of the two wheels only has an EGG-INFO directory. I guess the dist-info directory somehow helps. """ wheel = Wheel(dist.location) if wheel.project_name == project_name: return filename = os.path.basename(dist.location) new_filename = filename.replace(wheel.project_name, project_name) if filename == new_filename: return tmp_wheeldir = tempfile.mkdtemp() try: new_location = os.path.join(tmp_wheeldir, new_filename) shutil.copy(dist.location, new_location) # Now we create a clone of the original distribution, # but with the new location and the wanted project name. new_dist = Distribution( new_location, project_name=project_name, version=dist.version, py_version=dist.py_version, platform=dist.platform, precedence=dist.precedence, ) # We were called by _move_to_eggs_dir_and_compile. # Now we call it again with the new dist. # Note that here we do not pass a project_name, # so it won't call us again. return _move_to_eggs_dir_and_compile(new_dist, dest) finally: # Remember that temporary directories must be removed zc.buildout.rmtree.rmtree(tmp_wheeldir) def _move_to_eggs_dir_and_compile(dist, dest, project_name=""): """Move distribution to the eggs destination directory. And compile the py files, if we have actually moved the dist. Its new location is expected not to exist there yet, otherwise we would not be calling this function: the egg is already there. But the new location might exist at this point if another buildout is running in parallel. So we copy to a temporary directory first. See discussion at https://github.com/buildout/buildout/issues/307 We return the new distribution with properly loaded metadata. """ # First make sure the destination directory exists. This could suffer from # the same kind of race condition as the rest: if we check that it does not # exist, and we then create it, it will fail when a second buildout is # doing the same thing. try: os.makedirs(dest) except OSError: if not os.path.isdir(dest): # Unknown reason. Reraise original error. raise tmp_dest = tempfile.mkdtemp(dir=dest) try: installed_with_pip = False if (os.path.isdir(dist.location) and dist.precedence >= pkg_resources.BINARY_DIST): # We got a pre-built directory. It must have been obtained locally. # Just copy it. tmp_loc = os.path.join(tmp_dest, os.path.basename(dist.location)) shutil.copytree(dist.location, tmp_loc) else: # It is an archive of some sort. # Figure out how to unpack it, or fall back to easy_install. _, ext = os.path.splitext(dist.location) if ext in UNPACKERS: unpacker = UNPACKERS[ext] if project_name and ext == '.whl': new_dist = _maybe_copy_and_rename_wheel(dist, dest, project_name) if new_dist is not None: return new_dist unpacker(dist.location, tmp_dest) [tmp_loc] = glob.glob(os.path.join(tmp_dest, '*')) else: [tmp_loc] = call_pip_install(dist.location, tmp_dest) installed_with_pip = True # We have installed the dist. Now try to rename/move it. newloc = os.path.join(dest, os.path.basename(tmp_loc)) try: os.rename(tmp_loc, newloc) except OSError: # Might be for various reasons. If it is because newloc already # exists, we can investigate. if not os.path.exists(newloc): # No, it is a different reason. Give up. raise # Try to use it as environment and check if our project is in it. newdist = _get_matching_dist_in_location(dist, newloc) if newdist is None: # Path exists, but is not our package. We could # try something, but it seems safer to bail out # with the original error. raise # newloc looks okay to use. Do print a warning. logger.warn( "Path %s unexpectedly already exists.\n" "Maybe a buildout running in parallel has added it. " "We will accept it.\n" "If this contains a wrong package, please remove it yourself.", newloc) else: # There were no problems during the rename. # Do the compile step. redo_pyc(newloc) newdist = _get_matching_dist_in_location(dist, newloc) assert newdist is not None # newloc above is missing our dist?! finally: # Remember that temporary directories must be removed zc.buildout.rmtree.rmtree(tmp_dest) if installed_with_pip: newdist.precedence = pkg_resources.EGG_DIST return newdist def sort_working_set(ws, eggs_dir, develop_eggs_dir): develop_paths = set() pattern = os.path.join(develop_eggs_dir, '*.egg-link') for egg_link in glob.glob(pattern): with open(egg_link, 'rt') as f: path = f.readline().strip() if path: develop_paths.add(path) sorted_paths = [] egg_paths = [] other_paths = [] for dist in ws: path = dist.location if path in develop_paths: sorted_paths.append(path) elif os.path.commonprefix([path, eggs_dir]) == eggs_dir: egg_paths.append(path) else: other_paths.append(path) sorted_paths.extend(egg_paths) sorted_paths.extend(other_paths) return pkg_resources.WorkingSet(sorted_paths) NOT_PICKED_AND_NOT_ALLOWED = """\ Picked: {name} = {version} The `{name}` egg does not have a version pin and `allow-picked-versions = false`. To resolve this, add {name} = {version} to the [versions] section, OR set `allow-picked-versions = true`.""" ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675670.0 zc_buildout-4.1.6/src/zc/buildout/patches.py0000644000076500000240000003434514773460426020535 0ustar00mauritsstaff############################################################################## # # Copyright (c) 2006 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## def patch_Distribution(): try: from pkg_resources import Distribution from .compat import version except ImportError: return def hashcmp(self): if hasattr(self, '_hashcmp'): return self._hashcmp else: try: parsed_version = self.parsed_version except Exception: # You get here when there is an distribution on PyPI # with a version that is no longer seen as valid. # I want to catch version.InvalidVersion, but it may # come from a different place then I think. parsed_version = version.Version("0.0.0") self._hashcmp = result = ( parsed_version, self.precedence, self.key, self.location, self.py_version or '', self.platform or '', ) return result setattr(Distribution, 'hashcmp', property(hashcmp)) patch_Distribution() def patch_PackageIndex(): """Patch the package index from setuptools. Main goal: check the package urls on an index page to see if they are compatible with the Python version. """ try: import logging logging.getLogger('pip._internal.index.collector').setLevel(logging.ERROR) from setuptools.package_index import PackageIndex from setuptools.package_index import URL_SCHEME from setuptools.package_index import distros_for_url try: # pip 22.2+ from pip._internal.index.collector import IndexContent except ImportError: # pip 22.1- from pip._internal.index.collector import HTMLPage as IndexContent from pip._internal.index.collector import parse_links from pip._internal.index.package_finder import _check_link_requires_python from pip._internal.models.target_python import TargetPython from urllib.error import HTTPError except ImportError: import logging logger = logging.getLogger('zc.buildout.patches') logger.warning( 'Requires-Python support missing and could not be patched into ' 'zc.buildout. \n\n', exc_info=True ) return PY_VERSION_INFO = TargetPython().py_version_info # method copied over from setuptools 46.1.3 # Unchanged in setuptools 70.0.0. def process_url(self, url, retrieve=False): """Evaluate a URL as a possible download, and maybe retrieve it""" if url in self.scanned_urls and not retrieve: return self.scanned_urls[url] = True if not URL_SCHEME(url): self.process_filename(url) return else: dists = list(distros_for_url(url)) if dists: if not self.url_ok(url): return self.debug("Found link: %s", url) if dists or not retrieve or url in self.fetched_urls: list(map(self.add, dists)) return # don't need the actual page if not self.url_ok(url): self.fetched_urls[url] = True return self.info("Reading %s", url) self.fetched_urls[url] = True # prevent multiple fetch attempts tmpl = "Download error on %s: %%s -- Some packages may not be found!" f = self.open_url(url, tmpl % url) if f is None: return if isinstance(f, HTTPError) and f.code == 401: self.info("Authentication error: %s" % f.msg) self.fetched_urls[f.url] = True if 'html' not in f.headers.get('content-type', '').lower(): f.close() # not html, we can't process it return base = f.url # handle redirects page = f.read() # --- LOCAL CHANGES MADE HERE: --- if isinstance(page, str): page = page.encode('utf8') charset = 'utf8' else: if isinstance(f, HTTPError): # Errors have no charset, assume latin1: charset = 'latin-1' else: try: charset = f.headers.get_param('charset') or 'latin-1' except AttributeError: # Python 2 charset = f.headers.getparam('charset') or 'latin-1' try: content_type = f.getheader('content-type') except AttributeError: # On at least Python 2.7: # addinfourl instance has no attribute 'getheader' content_type = "text/html" try: # pip 22.2+ html_page = IndexContent( page, content_type=content_type, encoding=charset, url=base, cache_link_parsing=False, ) except TypeError: try: # pip 20.1-22.1 html_page = IndexContent(page, charset, base, cache_link_parsing=False) except TypeError: # pip 20.0 or older html_page = IndexContent(page, charset, base) # https://github.com/buildout/buildout/issues/598 # use_deprecated_html5lib is a required addition in pip 22.0/22.1 # and it is gone already in 22.2 try: plinks = parse_links(html_page, use_deprecated_html5lib=False) except TypeError: plinks = parse_links(html_page) plinks = list(plinks) # --- END OF LOCAL CHANGES --- if not isinstance(page, str): # In Python 3 and got bytes but want str. page = page.decode(charset, "ignore") f.close() # --- LOCAL CHANGES MADE HERE: --- for link in plinks: if _check_link_requires_python(link, PY_VERSION_INFO): self.process_url(link.url) # --- END OF LOCAL CHANGES --- if url.startswith(self.index_url) and getattr(f, 'code', None) != 404: page = self.process_index(url, page) setattr(PackageIndex, 'process_url', process_url) patch_PackageIndex() def patch_interpret_distro_name(): """Goal: recognize distro names better. interpret_distro_name was changed as part of https://github.com/pypa/setuptools/pull/2822 This landed in setuptools 70. We seem to need this version, to avoid problems recognizing distro names. 'basename' is for example 'mauritstest.namespacepackage-1.0.0' The new code correctly handles this as project name ''mauritstest.namespacepackage', instead of yielding multiple distros. """ try: from packaging import version from pkg_resources import Distribution from pkg_resources import SOURCE_DIST from setuptools import package_index import re import setuptools except ImportError: return if version.parse(setuptools.__version__) >= version.Version("70"): # Patch is not needed. return def interpret_distro_name( location, basename, metadata, py_version=None, precedence=SOURCE_DIST, platform=None ): """Generate the interpretation of a source distro name Note: if `location` is a filesystem filename, you should call ``pkg_resources.normalize_path()`` on it before passing it to this routine! """ parts = basename.split('-') if not py_version and any(re.match(r'py\d\.\d$', p) for p in parts[2:]): # it is a bdist_dumb, not an sdist -- bail out return # find the pivot (p) that splits the name from the version. # infer the version as the first item that has a digit. for p in range(len(parts)): if parts[p][:1].isdigit(): break else: p = len(parts) yield Distribution( location, metadata, '-'.join(parts[:p]), '-'.join(parts[p:]), py_version=py_version, precedence=precedence, platform=platform, ) package_index.interpret_distro_name = interpret_distro_name patch_interpret_distro_name() def patch_pkg_resources_requirement_contains(): """Patch pkg_resources.Requirement contains method. What this hopefully solves, is checking if a Requirement contains a Distribution, without the key needing to be exactly the same. We want to compare normalized names. """ try: from pkg_resources import Distribution from pkg_resources import Requirement from zc.buildout.utils import normalize_name except ImportError: return def __contains__(self, item): if isinstance(item, Distribution): # if item.key != self.key: if normalize_name(item.key) != normalize_name(self.key): return False item = item.version # Allow prereleases always in order to match the previous behavior of # this method. In the future this should be smarter and follow PEP 440 # more accurately. try: return self.specifier.contains(item, prereleases=True) except Exception: # For example on https://pypi.org/simple/zope-exceptions/ # the first distribution is zope.exceptions-3.4dev-r73107.tar.gz # I want to catch version.InvalidVersion, but it may # come from a different place then I think. return False Requirement.__contains__ = __contains__ patch_pkg_resources_requirement_contains() def patch_pkg_resources_working_set_find(): """Patch pkg_resources.WorkingSet find method. setuptools 75.8.1 fixed wheel file naming to follow the binary distribution specification. This broke a lot for us, especially when using editable installs. We fixed several parts of our code. setuptools 75.8.2 fixed some of the breakage by updating the `find` method of working sets. When finding a requirement, it now considers several candidates: different spellings of the requirement name. See https://github.com/pypa/setuptools/pull/4856 A lot of remaining problems in Buildout are fixed if we use this setuptools version. So what we do in this patch, is to check which setuptools version is used, and patch the 'find' method if the version is older than 75.8.2. But: if the version is *much* older, the patch can be applied, but calling the `find` method will raise: AttributeError: 'WorkingSet' object has no attribute 'normalized_to_canonical_keys' The first setuptools version that has this, is 61.0.0. So don't patch versions that are older than that. Alternatively, we could drop support. That is fine with me. """ try: from importlib.metadata import version from packaging.version import parse from packaging.version import Version setuptools_version = parse(version("setuptools")) if setuptools_version >= Version("75.8.2"): return if setuptools_version < Version("61"): return except Exception: return try: from pkg_resources import Distribution from pkg_resources import Requirement from pkg_resources import safe_name from pkg_resources import VersionConflict from pkg_resources import WorkingSet except ImportError: return def find(self, req): """Find a distribution matching requirement `req` Note: I removed the type hints, because they failed on Python 3.9: TypeError: unsupported operand type(s) for |: 'type' and 'NoneType' If there is an active distribution for the requested project, this returns it as long as it meets the version requirement specified by `req`. But, if there is an active distribution for the project and it does *not* meet the `req` requirement, ``VersionConflict`` is raised. If there is no active distribution for the requested project, ``None`` is returned. """ dist = None candidates = ( req.key, self.normalized_to_canonical_keys.get(req.key), safe_name(req.key).replace(".", "-"), ) for candidate in filter(None, candidates): dist = self.by_key.get(candidate) if dist: req.key = candidate break if dist is not None and dist not in req: # XXX add more info raise VersionConflict(dist, req) return dist WorkingSet.find = find patch_pkg_resources_working_set_find() def patch_find_packages(): """ Patch setuptools.package_index.PackageIndex find_packages method. Implements PEP 503 """ try: from setuptools.package_index import PackageIndex import re except ImportError: return # method copied over from setuptools 46.1.3 # Unchanged in setuptools 77.0.1. def find_packages(self, requirement): url_name = re.sub(r"[-_.]+", "-", requirement.unsafe_name).lower() self.scan_url(self.index_url + url_name + '/') if not self.package_pages.get(requirement.key): # Fall back to safe version of the name self.scan_url(self.index_url + requirement.project_name + '/') if not self.package_pages.get(requirement.key): # We couldn't find the target package, so search the index page too self.not_found_in_index(requirement) for url in list(self.package_pages.get(requirement.key, ())): # scan each page that might be related to the desired package self.scan_url(url) setattr(PackageIndex, 'find_packages', find_packages) patch_find_packages() ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675670.0 zc_buildout-4.1.6/src/zc/buildout/pep425tags.py0000644000076500000240000001231514773460426020775 0ustar00mauritsstaff"""Generate and work with PEP 425 Compatibility Tags.""" import sys import warnings import sysconfig import distutils.util def get_config_var(var): try: return sysconfig.get_config_var(var) except IOError as e: # pip Issue #1074 warnings.warn("{0}".format(e), RuntimeWarning) return None def get_abbr_impl(): """Return abbreviated implementation name.""" if hasattr(sys, 'pypy_version_info'): pyimpl = 'pp' elif sys.platform.startswith('java'): pyimpl = 'jy' elif sys.platform == 'cli': pyimpl = 'ip' else: pyimpl = 'cp' return pyimpl def get_impl_ver(): """Return implementation version.""" impl_ver = get_config_var("py_version_nodot") if not impl_ver or get_abbr_impl() == 'pp': impl_ver = ''.join(map(str, get_impl_version_info())) return impl_ver def get_impl_version_info(): """Return sys.version_info-like tuple for use in decrementing the minor version.""" if get_abbr_impl() == 'pp': # as per https://github.com/pypa/pip/issues/2882 return (sys.version_info[0], sys.pypy_version_info.major, sys.pypy_version_info.minor) else: return sys.version_info[0], sys.version_info[1] def get_flag(var, fallback, expected=True, warn=True): """Use a fallback method for determining SOABI flags if the needed config var is unset or unavailable.""" val = get_config_var(var) if val is None: if warn: warnings.warn("Config variable '{0}' is unset, Python ABI tag may " "be incorrect".format(var), RuntimeWarning, 2) return fallback() return val == expected def get_abi_tag(): """Return the ABI tag based on SOABI (if available) or emulate SOABI (CPython 2, PyPy).""" soabi = get_config_var('SOABI') impl = get_abbr_impl() if not soabi and impl in ('cp', 'pp') and hasattr(sys, 'maxunicode'): d = '' m = '' u = '' if get_flag('Py_DEBUG', lambda: hasattr(sys, 'gettotalrefcount'), warn=(impl == 'cp')): d = 'd' if get_flag('WITH_PYMALLOC', lambda: impl == 'cp', warn=(impl == 'cp')): m = 'm' if get_flag('Py_UNICODE_SIZE', lambda: sys.maxunicode == 0x10ffff, expected=4, warn=(impl == 'cp' and sys.version_info < (3, 3))) \ and sys.version_info < (3, 3): u = 'u' abi = '%s%s%s%s%s' % (impl, get_impl_ver(), d, m, u) elif soabi and soabi.startswith('cpython-'): abi = 'cp' + soabi.split('-')[1] elif soabi: abi = soabi.replace('.', '_').replace('-', '_') else: abi = None return abi def get_platform(): """Return our platform name 'win32', 'linux_x86_64'""" # XXX remove distutils dependency result = distutils.util.get_platform().replace('.', '_').replace('-', '_') if result == "linux_x86_64" and sys.maxsize == 2147483647: # pip pull request #3497 result = "linux_i686" return result def get_supported(versions=None, supplied_platform=None): """Return a list of supported tags for each version specified in `versions`. :param versions: a list of string versions, of the form ["33", "32"], or None. The first version will be assumed to support our ABI. """ supported = [] # Versions must be given with respect to the preference if versions is None: versions = [] version_info = get_impl_version_info() major = version_info[:-1] # Support all previous minor Python versions. for minor in range(version_info[-1], -1, -1): versions.append(''.join(map(str, major + (minor,)))) impl = get_abbr_impl() abis = [] abi = get_abi_tag() if abi: abis[0:0] = [abi] abi3s = set() import imp for suffix in imp.get_suffixes(): if suffix[0].startswith('.abi'): abi3s.add(suffix[0].split('.', 2)[1]) abis.extend(sorted(list(abi3s))) abis.append('none') platforms = [] if supplied_platform: platforms.append(supplied_platform) platforms.append(get_platform()) # Current version, current API (built specifically for our Python): for abi in abis: for arch in platforms: supported.append(('%s%s' % (impl, versions[0]), abi, arch)) # No abi / arch, but requires our implementation: for i, version in enumerate(versions): supported.append(('%s%s' % (impl, version), 'none', 'any')) if i == 0: # Tagged specifically as being cross-version compatible # (with just the major version specified) supported.append(('%s%s' % (impl, versions[0][0]), 'none', 'any')) # Major Python version + platform; e.g. binaries not using the Python API supported.append(('py%s' % (versions[0][0]), 'none', arch)) # No abi / arch, generic Python for i, version in enumerate(versions): supported.append(('py%s' % (version,), 'none', 'any')) if i == 0: supported.append(('py%s' % (version[0]), 'none', 'any')) return supported ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675670.0 zc_buildout-4.1.6/src/zc/buildout/rmtree.py0000644000076500000240000000377314773460426020405 0ustar00mauritsstaff############################################################################## # # Copyright (c) 2006 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import shutil import os import doctest import time def rmtree (path): """ A variant of shutil.rmtree which tries hard to be successful. On windows shutil.rmtree aborts when it tries to delete a read only file or a file which is still handled by another process (e.g. antivirus scanner). This tries to chmod the file to writeable and retries 10 times before giving up. >>> from tempfile import mkdtemp Let's make a directory ... >>> d = mkdtemp() and make sure it is actually there >>> os.path.isdir (d) 1 Now create a file ... >>> foo = os.path.join (d, 'foo') >>> with open (foo, 'w') as f: _ = f.write ('huhu') and make it unwriteable >>> os.chmod (foo, 256) # 0400 rmtree should be able to remove it: >>> rmtree (d) and now the directory is gone >>> os.path.isdir (d) 0 """ def retry_writeable (func, path, exc): os.chmod (path, 384) # 0600 for i in range(10): try: func (path) break except OSError: time.sleep(0.1) else: # tried 10 times without success, thus # finally rethrow the last exception raise shutil.rmtree (path, onerror = retry_writeable) def test_suite(): return doctest.DocTestSuite() if "__main__" == __name__: doctest.testmod() ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675670.0 zc_buildout-4.1.6/src/zc/buildout/testing.py0000644000076500000240000005551314773460426020563 0ustar00mauritsstaff############################################################################# # # Copyright (c) 2004-2009 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Various test-support utility functions """ from http.server import HTTPServer, BaseHTTPRequestHandler from urllib.request import urlopen import errno import logging import multiprocessing import operator import os import pkg_resources import random import re import shutil import socket import subprocess import sys import tempfile import threading import time import zc.buildout.buildout import zc.buildout.easy_install from zc.buildout.rmtree import rmtree print_ = zc.buildout.buildout.print_ fsync = getattr(os, 'fsync', lambda fileno: None) is_win32 = sys.platform == 'win32' def read(path='out', *rest): with open(os.path.join(path, *rest)) as f: return f.read() def cat(dir, *names): path = os.path.join(dir, *names) if (not os.path.exists(path) and is_win32 and os.path.exists(path+'-script.py') ): path = path+'-script.py' with open(path) as f: print_(f.read(), end='') def eqs(a, *b): a = set(a); b = set(b) return None if a == b else (a - b, b - a) def clear_here(): for name in os.listdir('.'): if os.path.isfile(name) or os.path.islink(name): os.remove(name) else: shutil.rmtree(name) def ls(dir, *subs, lowercase_and_sort_output=False): if subs: dir = os.path.join(dir, *subs) if lowercase_and_sort_output: # Get the original names, but sorted lowercase. names = sorted(os.listdir(dir), key=operator.methodcaller("lower")) else: names = sorted(os.listdir(dir)) for name in names: # If we're running under coverage, elide coverage files if os.getenv("COVERAGE_PROCESS_START") and name.startswith('.coverage.'): continue if os.path.isdir(os.path.join(dir, name)): print_('d ', end=' ') elif os.path.islink(os.path.join(dir, name)): print_('l ', end=' ') else: print_('- ', end=' ') if lowercase_and_sort_output: name = name.lower() print_(name) def mkdir(*path): os.mkdir(os.path.join(*path)) def remove(*path): path = os.path.join(*path) if os.path.isdir(path): shutil.rmtree(path) else: os.remove(path) def rmdir(*path): shutil.rmtree(os.path.join(*path)) def write(dir, *args): path = os.path.join(dir, *(args[:-1])) f = open(path, 'w') f.write(args[-1]) f.flush() fsync(f.fileno()) f.close() def clean_up_pyc(*path): base, filename = os.path.join(*path[:-1]), path[-1] if filename.endswith('.py'): filename += 'c' # .py -> .pyc for path in ( os.path.join(base, filename), os.path.join(base, '__pycache__'), ): if os.path.isdir(path): rmdir(path) elif os.path.exists(path): remove(path) ## FIXME - check for other platforms MUST_CLOSE_FDS = not sys.platform.startswith('win') def system(command, input='', with_exit_code=False, env=None): # Some TERMinals, especially xterm and its variants, add invisible control # characters, which we do not want as they mess up doctests. See: # https://github.com/buildout/buildout/pull/311 # http://bugs.python.org/issue19884 sub_env = dict(os.environ, TERM='dumb') if env is not None: sub_env.update(env) # We used to pass for example 'buildout annotate' as command, and call Popen # with 'shell=True'. Since October 2024 this no longer works on Windows on GHA. # So we pass the command as a list. But then it breaks on POSIX when we have # 'shell=True', because args[1:] gets passed as options for '/bin/sh' instead # of options for our command. So let's let the value of 'shell' depend on # whether command is a list or a string. # See also https://stackoverflow.com/a/2401128/621201 # Actually, having the command as a string turns out to only be a problem on # Windows if there is a space in the path, like with 'Program Files'. if isinstance(command, list): shell = False else: shell = True p = subprocess.Popen(command, shell=shell, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, close_fds=MUST_CLOSE_FDS, env=sub_env) i, o, e = (p.stdin, p.stdout, p.stderr) if input: i.write(input.encode()) i.close() result = o.read() + e.read() o.close() e.close() output = result.decode() if with_exit_code: # Use the with_exit_code=True parameter when you want to test the exit # code of the command you're running. output += 'EXIT CODE: %s' % p.wait() p.wait() return output def get(url): return str(urlopen(url).read().decode()) def _runsetup(setup, *args): if os.path.isdir(setup): setup = os.path.join(setup, 'setup.py') args = list(args) args.insert(0, '-q') here = os.getcwd() try: os.chdir(os.path.dirname(setup)) zc.buildout.easy_install.call_subprocess( [sys.executable, setup] + args, env=dict(os.environ, PYTHONPATH=zc.buildout.easy_install.pip_pythonpath, ), # Prevent showing several lines of output per created distribution, # especially with older setuptools versions. stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, ) if os.path.exists('build'): rmtree('build') finally: os.chdir(here) def sdist(setup, dest): _runsetup(setup, 'sdist', '-d', dest) def bdist_egg(setup, executable, dest=None): # Backward compat: if dest is None: dest = executable else: assert executable == sys.executable, (executable, sys.executable) _runsetup(setup, 'bdist_egg', '-d', dest) def bdist_wheel(setup, dest): _runsetup(setup, 'bdist_wheel', '-d', dest) def wait_until(label, func, *args, **kw): if 'timeout' in kw: kw = dict(kw) timeout = kw.pop('timeout') else: timeout = 30 deadline = time.time()+timeout while time.time() < deadline: if func(*args, **kw): return time.sleep(0.01) raise ValueError('Timed out waiting for: '+label) class TestOptions(zc.buildout.buildout.Options): def __init__(self, *args): zc.buildout.buildout.Options.__init__(self, *args) self._created = [] def initialize(self): pass class Buildout(zc.buildout.buildout.Buildout): def __init__(self): for name in 'eggs', 'parts': if not os.path.exists(name): os.mkdir(name) zc.buildout.buildout.Buildout.__init__( self, '', [('buildout', 'directory', os.getcwd())], False) Options = TestOptions def buildoutSetUp(test): test.globs['__tear_downs'] = __tear_downs = [] test.globs['register_teardown'] = register_teardown = __tear_downs.append prefer_final = zc.buildout.easy_install.prefer_final() register_teardown( lambda: zc.buildout.easy_install.prefer_final(prefer_final) ) here = os.getcwd() register_teardown(lambda: os.chdir(here)) handlers_before_set_up = logging.getLogger().handlers[:] def restore_root_logger_handlers(): root_logger = logging.getLogger() for handler in root_logger.handlers[:]: root_logger.removeHandler(handler) for handler in handlers_before_set_up: root_logger.addHandler(handler) bo_logger = logging.getLogger('zc.buildout') for handler in bo_logger.handlers[:]: bo_logger.removeHandler(handler) register_teardown(restore_root_logger_handlers) base = tempfile.mkdtemp('buildoutSetUp') base = os.path.realpath(base) register_teardown(lambda base=base: rmtree(base)) old_home = os.environ.get('HOME') os.environ['HOME'] = os.path.join(base, 'bbbBadHome') def restore_home(): if old_home is None: del os.environ['HOME'] else: os.environ['HOME'] = old_home register_teardown(restore_home) base = os.path.join(base, '_TEST_') os.mkdir(base) tmp = tempfile.mkdtemp('buildouttests') register_teardown(lambda: rmtree(tmp)) zc.buildout.easy_install.default_index_url = 'file://'+tmp os.environ['buildout_testing_index_url'] = ( zc.buildout.easy_install.default_index_url) def tmpdir(name): path = os.path.join(base, name) mkdir(path) return path sample = tmpdir('sample-buildout') os.chdir(sample) # Create a basic buildout.cfg to avoid a warning from buildout: with open('buildout.cfg', 'w') as f: f.write("[buildout]\nparts =\n") # Use the buildout bootstrap command to create a buildout zc.buildout.buildout.Buildout( 'buildout.cfg', [('buildout', 'log-level', 'WARNING'), # trick bootstrap into putting the buildout develop egg # in the eggs dir. ('buildout', 'develop-eggs-directory', 'eggs'), ] ).bootstrap([]) # Create the develop-eggs dir, which didn't get created the usual # way due to the trick above: os.mkdir('develop-eggs') path_to_coveragerc = os.getenv("COVERAGE_PROCESS_START", None) if path_to_coveragerc is not None: # Before we return to the current directory and destroy the # temporary working directory, we need to copy all the coverage files # back so that they can be `coverage combine`d. def copy_coverage_files(): coveragedir = os.path.dirname(path_to_coveragerc) import glob for f in glob.glob('.coverage*'): shutil.copy(f, coveragedir) __tear_downs.insert(0, copy_coverage_files) # Now we must modify the newly created bin/buildout to # actually begin coverage. with open('bin/buildout') as f: import textwrap lines = f.read().splitlines() assert lines[1] == '', lines lines[1] = 'import coverage; coverage.process_startup()' with open('bin/buildout', 'w') as f: f.write('\n'.join(lines)) def start_server(path): port, thread = _start_server(path, name=path) url = 'http://localhost:%s/' % port register_teardown(lambda: stop_server(url, thread)) return url cdpaths = [] def cd(*path): path = os.path.join(*path) cdpaths.append(os.path.abspath(os.getcwd())) os.chdir(path) def uncd(): os.chdir(cdpaths.pop()) test.globs.update(dict( sample_buildout = sample, ls = ls, cat = cat, mkdir = mkdir, rmdir = rmdir, remove = remove, tmpdir = tmpdir, write = write, system = system, get = get, cd = cd, uncd = uncd, join = os.path.join, sdist = sdist, bdist_egg = bdist_egg, start_server = start_server, stop_server = stop_server, buildout = os.path.join(sample, 'bin', 'buildout'), wait_until = wait_until, print_ = print_, clean_up_pyc = clean_up_pyc, )) zc.buildout.easy_install.prefer_final(prefer_final) def buildoutTearDown(test): for f in test.globs['__tear_downs']: f() class Server(HTTPServer): def __init__(self, tree, *args): HTTPServer.__init__(self, *args) self.tree = os.path.abspath(tree) __run = True def serve_forever(self): while self.__run: self.handle_request() def handle_error(self, *_): self.__run = False class Handler(BaseHTTPRequestHandler): Server.__log = False def __init__(self, request, address, server): self.__server = server self.tree = server.tree BaseHTTPRequestHandler.__init__(self, request, address, server) def do_GET(self): if '__stop__' in self.path: self.__server.server_close() raise SystemExit def k(): self.send_response(200) out = 'k\n'.encode() self.send_header('Content-Length', str(len(out))) self.send_header('Content-Type', 'text/html') self.end_headers() self.wfile.write(out) if self.path == '/enable_server_logging': self.__server.__log = True return k() if self.path == '/disable_server_logging': self.__server.__log = False return k() path = os.path.abspath(os.path.join(self.tree, *self.path.split('/'))) if not ( ((path == self.tree) or path.startswith(self.tree+os.path.sep)) and os.path.exists(path) ): self.send_response(404, 'Not Found') #self.send_response(200) out = 'Not Found'.encode() #out = '\n'.join(self.tree, self.path, path) self.send_header('Content-Length', str(len(out))) self.send_header('Content-Type', 'text/html') self.end_headers() self.wfile.write(out) return self.send_response(200) if os.path.isdir(path): out = ['\n'] names = sorted(os.listdir(path)) for name in names: if os.path.isdir(os.path.join(path, name)): name += '/' out.append('%s
\n' % (name, name)) out.append('\n') out = ''.join(out).encode() self.send_header('Content-Length', str(len(out))) self.send_header('Content-Type', 'text/html') else: with open(path, 'rb') as f: out = f.read() self.send_header('Content-Length', len(out)) if path.endswith('.egg'): self.send_header('Content-Type', 'application/zip') elif path.endswith('.gz'): self.send_header('Content-Type', 'application/x-gzip') elif path.endswith('.zip'): self.send_header('Content-Type', 'application/x-gzip') elif path.endswith('.whl'): self.send_header('Content-Type', 'application/octet-stream') else: self.send_header('Content-Type', 'text/html') self.end_headers() self.wfile.write(out) def log_request(self, code): if self.__server.__log: print_('%s %s %s' % (self.command, code, self.path)) def _run(tree, port): server_address = ('localhost', port) httpd = Server(tree, server_address, Handler) httpd.serve_forever() httpd.server_close() def get_port(): for i in range(10): port = random.randrange(20000, 30000) s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) try: try: s.connect(('localhost', port)) except socket.error: return port finally: s.close() raise RuntimeError("Can't find port") def _start_server(tree, name=''): port = get_port() thread = threading.Thread(target=_run, args=(tree, port), name=name) thread.setDaemon(True) thread.start() wait(port, up=True) return port, thread def start_server(tree): return _start_server(tree)[0] def stop_server(url, thread=None): try: urlopen(url+'__stop__') except Exception: pass if thread is not None: thread.join() # wait for thread to stop def wait(port, up): addr = 'localhost', port for i in range(120): time.sleep(0.25) try: s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.connect(addr) s.close() if up: break except socket.error: e = sys.exc_info()[1] if e[0] not in (errno.ECONNREFUSED, errno.ECONNRESET): raise s.close() if not up: break else: if up: raise else: raise SystemError("Couldn't stop server") def install(project, destination): if not isinstance(destination, str): destination = os.path.join(destination.globs['sample_buildout'], 'eggs') dist = pkg_resources.working_set.find( pkg_resources.Requirement.parse(project)) if dist.location.endswith('.egg'): destination = os.path.join(destination, os.path.basename(dist.location), ) if os.path.isdir(dist.location): shutil.copytree(dist.location, destination) else: shutil.copyfile(dist.location, destination) else: # copy link with open(os.path.join(destination, project+'.egg-link'), 'w') as f: f.write(dist.location) def install_develop(project, destination): if not isinstance(destination, str): destination = os.path.join(destination.globs['sample_buildout'], 'develop-eggs') dist = pkg_resources.working_set.find( pkg_resources.Requirement.parse(project)) with open(os.path.join(destination, project+'.egg-link'), 'w') as f: f.write(dist.location) def _normalize_path(match): path = match.group(1) if os.path.sep == '\\': path = path.replace('\\\\', '/') if path.startswith('\\'): path = path[1:] return '/' + path.replace(os.path.sep, '/') normalize_path = ( re.compile( r'''[^'" \t\n\r]+\%(sep)s_[Tt][Ee][Ss][Tt]_\%(sep)s([^"' \t\n\r]+)''' % dict(sep=os.path.sep)), _normalize_path, ) normalize_endings = re.compile('\r\n'), '\n' normalize_script = ( re.compile('(\n?)- ([a-zA-Z_.-]+)-script.py\n- \\2.exe\n'), '\\1- \\2\n') normalize___pycache__ = ( re.compile('(\n?)d __pycache__\n'), '\\1') normalize_egg_py = ( re.compile(r'-py\d[.]\d+(-\S+)?\.egg'), '-pyN.N.egg', ) normalize_exception_type_for_python_2_and_3 = ( re.compile(r'^(\w+\.)*([A-Z][A-Za-z0-9]+Error: )'), '\2') normalize_open_in_generated_script = ( re.compile(r"open\(__file__, 'U'\)"), 'open(__file__)') not_found = (re.compile(r'Not found: [^\n]+/(\w|\.|-)+/\r?\n'), '') easyinstall_deprecated = (re.compile(r'.*EasyInstallDeprecationWarning.*\n'),'') setuptools_deprecated = (re.compile(r'.*SetuptoolsDeprecationWarning.*\n'),'') pkg_resources_deprecated = (re.compile(r'.*PkgResourcesDeprecationWarning.*\n'),'') warnings_warn = (re.compile(r'.*warnings\.warn.*\n'),'') # Setuptools now pulls in dependencies when installed. adding_find_link = (re.compile(r"Adding find link '[^']+'" r" from setuptools .*\r?\n"), '') ignore_not_upgrading = ( re.compile( 'Not upgrading because not running a local buildout command.\n' ), '') # The root logger from setuptools prints all kinds of lines. # This might depend on which setuptools version, or something else, # because it did not happen before. Sample lines: # "root: Couldn't retrieve index page for 'zc.recipe.egg'" # "root: Scanning index of all packages. # "root: Found: /sample-buildout/recipe/dist/spam-2-pyN.N.egg" # I keep finding new lines like that, so let's ignore all. ignore_root_logger = (re.compile(r'root:.*'), '') # Now replace a multiline warning about that you should switch to native namespaces. ignore_native_namespace_warning_1 = (re.compile(r'!!'), '') ignore_native_namespace_warning_2 = (re.compile(r'\*' * 80), '') ignore_native_namespace_warning_3 = (re.compile( r'Please replace its usage with implicit namespaces \(PEP 420\).'), '' ) ignore_native_namespace_warning_4 = (re.compile( r'See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages for details.'), '' ) ignore_native_namespace_warning_5 = (re.compile( r'ep.load\(\)\(self, ep.name, value\)'), '' ) def run_buildout(command): # Make sure we don't get .buildout os.environ['HOME'] = os.path.join(os.getcwd(), 'home') args = command.split() buildout = pkg_resources.load_entry_point( 'zc.buildout', 'console_scripts', args[0]) buildout(args[1:]) def run_from_process(target, *args, **kw): sys.stdout = sys.stderr = open('out', 'w') target(*args, **kw) def run_in_process(*args, **kwargs): try: ctx = multiprocessing.get_context('fork') process = ctx.Process(target=run_from_process, args=args, kwargs=kwargs) except AttributeError: process = multiprocessing.Process(target=run_from_process, args=args, kwargs=kwargs) process.daemon = True process.start() process.join(99) if process.is_alive() or process.exitcode: with open('out') as f: print(f.read()) def run_buildout_in_process(command='buildout'): command = command.split(' ', 1) command.insert( 1, " use-dependency-links=false" # Leaving this here so we can uncomment to see what's going on. #" log-format=%(asctime)s____%(levelname)s_%(message)s -vvv" " index=" + __file__ + 'nonexistent' # hide index ) command = ' '.join(command) run_in_process(run_buildout, command) def setup_coverage(path_to_coveragerc): if 'RUN_COVERAGE' not in os.environ: return if not os.path.exists(path_to_coveragerc): raise ValueError('coveragerc file %s does not exist.' % path_to_coveragerc) os.environ['COVERAGE_PROCESS_START'] = path_to_coveragerc rootdir = os.path.dirname(path_to_coveragerc) def combine_report(): subprocess.call( [ sys.executable, '-m', 'coverage', 'combine', ], cwd=rootdir, ) subprocess.call( [ sys.executable, '-m', 'coverage', 'report', ], cwd=rootdir, ) if path_to_coveragerc: try: import coverage print("Coverage configured with %s" % path_to_coveragerc) if 'COVERAGE_REPORT' in os.environ: import atexit atexit.register(combine_report) coverage.process_startup() except ImportError: print( "You try to run coverage " "but coverage is not installed in your environment." ) sys.exit(1) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675670.0 zc_buildout-4.1.6/src/zc/buildout/testing.txt0000644000076500000240000001615414773460426020750 0ustar00mauritsstaffTesting Support =============== The zc.buildout.testing module provides an API that can be used when writing recipe tests. This API is documented below. Many examples of using this API can be found in the zc.buildout, zc.recipe.egg, and zc.recipe.testrunner tests. zc.buildout.testing.buildoutSetUp(test) --------------------------------------- The buildoutSetup function can be used as a doctest setup function. It creates a sample buildout that can be used by tests, changing the current working directory to the sample_buildout. It also adds a number of names to the test namespace: ``sample_buildout`` This is the name of a buildout with a basic configuration. ``buildout`` This is the path of the buildout script in the sample buildout. ``ls(*path)`` List the contents of a directory. The directory path is provided as one or more strings, to be joined with os.path.join. ``cat(*path)`` Display the contents of a file. The file path is provided as one or more strings, to be joined with os.path.join. On Windows, if the file doesn't exist, the function will try adding a '-script.py' suffix. This helps to work around a difference in script generation on windows. ``mkdir(*path)`` Create a directory. The directory path is provided as one or more strings, to be joined with os.path.join. ``rmdir(*path)`` Remove a directory. The directory path is provided as one or more strings, to be joined with os.path.join. ``remove(*path)`` Remove a directory or file. The path is provided as one or more strings, to be joined with os.path.join. ``tmpdir(name)`` Create a temporary directory with the given name. The directory will be automatically removed at the end of the test. The path of the created directory is returned. Further, if the the normalize_path normalizing substitution (see below) is used, then any paths starting with this path will be normalized to:: /name/restofpath No two temporary directories can be created with the same name. A directory created with tmpdir can be removed with rmdir and recreated. Note that the sample_buildout directory is created by calling this function. ``write(*path_and_contents)`` Create a file. The file path is provided as one or more strings, to be joined with os.path.join. The last argument is the file contents. ``system(command, input='')`` Execute a system command with the given input passed to the command's standard input. The output (error and regular output) from the command is returned. ``get(url)`` Get a web page. ``cd(*path)`` Change to the given directory. The directory path is provided as one or more strings, to be joined with os.path.join. The directory will be reset at the end of the test. ``uncd()`` Change to the directory that was current prior to the previous call to ``cd``. You can call ``cd`` multiple times and then ``uncd`` the same number of times to return to the same location. ``join(*path)`` A convenient reference to os.path.join. ``register_teardown(func)`` Register a tear-down function. The function will be called with no arguments at the end of the test. ``start_server(path)`` Start a web server on the given path. The server will be shut down at the end of the test. The server URL is returned. You can cause the server to start and stop logging it's output using: >>> get(server_url+'enable_server_logging') and: >>> get(server_url+'disable_server_logging') This can be useful to see how buildout is interacting with a server. ``sdist(setup, dest)`` Create a source distribution by running the given setup file and placing the result in the given destination directory. If the setup argument is a directory, the setup.py file in that directory is used. ``bdist_egg(setup, dest)`` Create an egg by running the given setup file and placing the result in the given destination directory. If the setup argument is a directory, then the setup.py file in that directory is used. Code Coverage Measurement ~~~~~~~~~~~~~~~~~~~~~~~~~ ``buildoutSetUp`` prepares a new ``bin/buildout`` executable for doctests to run. Because this is will run in a subprocess, there are some extra steps needed to collect code coverage data for buildout and buildout recipes. This function does most of the heavy lifting for you, you only need to do a few things. First, you need to define a coverage configuration ('.coveragerc') which specifies that multiple different runs should be collected:: [run] source = my.egg.name parallel = true Second, when you invoke the test runner, you need to define the ``COVERAGE_PROCESS_START`` environment variable to point to this configuration and invoke it with coverage:: COVERAGE_PROCESS_START=.coveragerc coverage run -m zope.testrunner --test-path=src Finally, after the tests have finished, you need to combine the resulting coverage data files and then you can produce reports:: coverage combine coverage report .. caution:: You must be working in an environment (e.g., a virtual environment) that has coverage already installed. ``zc.buildout.testing.buildoutTearDown(test)`` ---------------------------------------------- Tear down everything set up by zc.buildout.testing.buildoutSetUp. Any functions passed to register_teardown are called as well. ``install(project, destination)`` --------------------------------- Install eggs for a given project into a destination. If the destination is a test object, then the eggs directory of the sample buildout (sample_buildout) defined by the test will be used. Tests will use this to install the distributions for the packages being tested (and their dependencies) into a sample buildout. The egg to be used should already be loaded, by importing one of the modules provided, before calling this function. ``install_develop(project, destination)`` ----------------------------------------- Like install, but a develop egg is installed even if the current egg if not a develop egg. ``Output normalization`` ------------------------ Recipe tests often generate output that is dependent on temporary file locations, operating system conventions or Python versions. To deal with these dependencies, we often use zope.testing.renormalizing.RENormalizing to normalize test output. zope.testing.renormalizing.RENormalizing takes pairs of regular expressions and substitutions. The zc.buildout.testing module provides a few helpful variables that define regular-expression/substitution pairs that you can pass to zope.testing.renormalizing.RENormalizing. ``normalize_path`` Converts tests paths, based on directories created with tmpdir(), to simple paths. ``normalize_script`` On Unix-like systems, scripts are implemented in single files without suffixes. On windows, scripts are implemented with 2 files, a -script.py file and a .exe file. This normalization converts directory listings of Windows scripts to the form generated on UNix-like systems. ``normalize_egg_py`` Normalize Python version and platform indicators, if specified, in egg names. ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675670.0 zc_buildout-4.1.6/src/zc/buildout/testrecipes.py0000644000076500000240000000062514773460426021432 0ustar00mauritsstafffrom zc.buildout.buildout import print_ class Debug: def __init__(self, buildout, name, options): self.buildout = buildout self.name = name self.options = options def install(self): items = list(self.options.items()) items.sort() for option, value in items: print_(" %s=%r" % (option, value)) return () update = install ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1743675672.4093058 zc_buildout-4.1.6/src/zc/buildout/tests/0000755000076500000240000000000014773460430017660 5ustar00mauritsstaff././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675670.0 zc_buildout-4.1.6/src/zc/buildout/tests/__init__.py0000644000076500000240000001737514773460426022013 0ustar00mauritsstaff# -*- coding: utf-8 -*- ############################################################################## # # Copyright (c) 2020 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import re import os import sys import shutil import tempfile import zc.buildout def create_sample_eggs(test, executable=sys.executable): assert executable == sys.executable, (executable, sys.executable) write = test.globs['write'] dest = test.globs['sample_eggs'] tmp = tempfile.mkdtemp() try: write(tmp, 'README.txt', '') for i in (0, 1, 2): write(tmp, 'eggrecipedemoneeded.py', 'y=%s\ndef f():\n pass' % i) rc1 = 'rc1' if i==2 else '' write( tmp, 'setup.py', "from setuptools import setup\n" "setup(name='demoneeded', py_modules=['eggrecipedemoneeded']," " zip_safe=True, version='1.%s%s', author='bob', url='bob', " "author_email='bob')\n" % (i, rc1) ) zc.buildout.testing.sdist(tmp, dest) write( tmp, 'distutilsscript', '#!/usr/bin/python\n' '# -*- coding: utf-8 -*-\n' '"""Module docstring."""\n' 'from __future__ import print_statement\n' 'import os\n' 'import sys; sys.stdout.write("distutils!\\n")\n' ) write( tmp, 'setup.py', "from setuptools import setup\n" "setup(name='other', zip_safe=False, version='1.0', " "scripts=['distutilsscript']," "py_modules=['eggrecipedemoneeded'])\n" ) zc.buildout.testing.bdist_wheel(tmp, dest) write( tmp, 'setup.py', "from setuptools import setup\n" "setup(name='du_zipped', zip_safe=True, version='1.0', " "scripts=['distutilsscript']," "py_modules=['eggrecipedemoneeded'])\n" ) # We still create an egg for this one, as we use it for testing # distutils scripts in a zipped egg. zc.buildout.testing.bdist_egg(tmp, executable, dest) os.remove(os.path.join(tmp, 'distutilsscript')) os.remove(os.path.join(tmp, 'eggrecipedemoneeded.py')) for i in (1, 2, 3, 4): write( tmp, 'eggrecipedemo.py', 'import eggrecipedemoneeded, sys\n' 'def print_(*a):\n' ' sys.stdout.write(" ".join(map(str, a))+"\\n")\n' 'x=%s\n' 'def main():\n' ' print_(x, eggrecipedemoneeded.y)\n' % i) rc1 = 'rc1' if i==4 else '' write( tmp, 'setup.py', "from setuptools import setup\n" "setup(name='demo', py_modules=['eggrecipedemo']," " install_requires = 'demoneeded'," " entry_points={'console_scripts': " "['demo = eggrecipedemo:main']}," " zip_safe=True, version='0.%s%s')\n" % (i, rc1) ) zc.buildout.testing.bdist_wheel(tmp, dest) write(tmp, 'mixedcase.py', 'def f():\n pass') write( tmp, 'setup.py', "from setuptools import setup\n" "setup(name='MIXEDCASE', py_modules=['mixedcase']," " author='bob', url='bob', author_email='bob'," " install_requires = 'demoneeded'," " zip_safe=True, version='0.5')\n" ) zc.buildout.testing.sdist(tmp, dest) # rename file to lower case # to test issues between file and package name curdir = os.getcwd() os.chdir(dest) for file in os.listdir(dest): if "MIXEDCASE" in file: os.rename(file, file.lower()) os.chdir(curdir) write(tmp, 'eggrecipebigdemo.py', 'import eggrecipedemo') write( tmp, 'setup.py', "from setuptools import setup\n" "setup(name='bigdemo', " " install_requires = 'demo'," " py_modules=['eggrecipebigdemo'], " " zip_safe=True, version='0.1')\n" ) zc.buildout.testing.bdist_wheel(tmp, dest) finally: shutil.rmtree(tmp) extdemo_c = """ #include #include static PyMethodDef methods[] = {{NULL}}; #define MOD_DEF(ob, name, doc, methods) \ static struct PyModuleDef moduledef = { \ PyModuleDef_HEAD_INIT, name, doc, -1, methods, }; \ ob = PyModule_Create(&moduledef); #define MOD_INIT(name) PyMODINIT_FUNC PyInit_##name(void) MOD_INIT(extdemo) { PyObject *m; MOD_DEF(m, "extdemo", "", methods); #ifdef TWO PyModule_AddObject(m, "val", PyLong_FromLong(2)); #else PyModule_AddObject(m, "val", PyLong_FromLong(EXTDEMO)); #endif return m; } """ extdemo_setup_py = r""" import os, sys from distutils.core import setup, Extension if os.environ.get('test_environment_variable'): print( "Have environment test_environment_variable: %%s" %% os.environ['test_environment_variable'] ) setup(name = "extdemo", version = "%s", url="http://www.zope.org", author="Demo", author_email="demo@demo.com", ext_modules = [Extension('extdemo', ['extdemo.c'])], ) """ def add_source_dist(test, version=1.4): if 'extdemo' not in test.globs: test.globs['extdemo'] = test.globs['tmpdir']('extdemo') tmp = test.globs['extdemo'] write = test.globs['write'] try: write(tmp, 'extdemo.c', extdemo_c); write(tmp, 'setup.py', extdemo_setup_py % version); write(tmp, 'README', ""); write(tmp, 'MANIFEST.in', "include *.c\n"); test.globs['sdist'](tmp, test.globs['sample_eggs']) except: shutil.rmtree(tmp) def easy_install_SetUp(test): zc.buildout.testing.buildoutSetUp(test) sample_eggs = test.globs['tmpdir']('sample_eggs') test.globs['sample_eggs'] = sample_eggs os.mkdir(os.path.join(sample_eggs, 'index')) create_sample_eggs(test) add_source_dist(test) test.globs['link_server'] = test.globs['start_server']( test.globs['sample_eggs']) test.globs['update_extdemo'] = lambda : add_source_dist(test, 1.5) zc.buildout.testing.install_develop('zc.recipe.egg', test) normalize_bang = ( re.compile(re.escape('#!'+ zc.buildout.easy_install._safe_arg(sys.executable)) + ".*"), '#!/usr/local/bin/python2.7', ) def create_egg(name, version, dest, install_requires=None, dependency_links=None): d = tempfile.mkdtemp() if dest=='available': extras = dict(x=['x']) else: extras = {} if dependency_links: links = 'dependency_links = %s, ' % dependency_links else: links = '' if install_requires: requires = 'install_requires = %s, ' % install_requires else: requires = '' try: with open(os.path.join(d, 'setup.py'), 'w') as f: f.write( 'from setuptools import setup\n' 'setup(name=%r, version=%r, extras_require=%r, zip_safe=True,\n' ' %s %s py_modules=["setup"]\n)' % (name, str(version), extras, requires, links) ) zc.buildout.testing.bdist_egg(d, sys.executable, os.path.abspath(dest)) finally: shutil.rmtree(d) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675670.0 zc_buildout-4.1.6/src/zc/buildout/tests/allow-unknown-extras.txt0000644000076500000240000000351414773460426024550 0ustar00mauritsstaff====================== Allow Unknown Extras ====================== Sometimes we need to allow unknown extras. The ``allow-unknown-extras`` option lets us do that in a buildout configuration, just as we can directly calling ``easy_install`` works exactly like the one provided in ``easy_install``. Let's create a develop egg that requires a bogus extra. >>> mkdir(sample_buildout, 'allowdemo') >>> write(sample_buildout, 'allowdemo', 'dependencydemo.py', ... 'import eggrecipekss.core') >>> write(sample_buildout, 'allowdemo', 'setup.py', ... '''from setuptools import setup; setup( ... name='allowdemo', py_modules=['dependencydemo'], ... zip_safe=True, version='1') ... ''') Now let's configure the buildout to use the develop egg with a bogus extra. >>> write(sample_buildout, 'buildout.cfg', ... ''' ... [buildout] ... develop = allowdemo ... parts = eggs ... ... [eggs] ... recipe = zc.recipe.egg:eggs ... eggs = allowdemo[bad_extra] ... ''') Now we can run the buildout and see that it fails: >>> print_(system(buildout), end='') # doctest: +ELLIPSIS Develop: '/sample-buildout/allowdemo' Installing eggs... ... While: Installing eggs. Error: Couldn't find the required extra... If we flip the option on, the buildout succeeds >>> write(sample_buildout, 'buildout.cfg', ... ''' ... [buildout] ... develop = allowdemo ... parts = eggs ... allow-unknown-extras = true ... ... [eggs] ... recipe = zc.recipe.egg:eggs ... eggs = allowdemo[bad_extra] ... ''') Now we can run the buildout and only get a warning:: >>> print_(system(buildout), end='') # doctest: +ELLIPSIS Develop: '/sample-buildout/allowdemo' Installing eggs... allowdemo 1 does not provide the extra 'bad_extra' ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675670.0 zc_buildout-4.1.6/src/zc/buildout/tests/allowhosts.txt0000644000076500000240000000750214773460426022631 0ustar00mauritsstaffAllow hosts ----------- On some environments the links visited by `zc.buildout` can be forbidden by paranoiac firewalls. These URL might be on the chain of links visited by `zc.buildout` whether they are defined in the `find-links` option or by various eggs in their `url`, `download_url` and `dependency_links` metadata. It is even harder to track that package_index works like a spider and might visit links and go to other location. The `allow-hosts` option provides a way to prevent this, and works exactly like the one provided in `easy_install` (see `easy_install allow-hosts option`_). You can provide a list of allowed host, together with wildcards:: [buildout] ... allow-hosts = *.python.org example.com Let's create a develop egg in our buildout that specifies `dependency_links` which points to a server in the outside world:: >>> mkdir(sample_buildout, 'allowdemo') >>> write(sample_buildout, 'allowdemo', 'dependencydemo.py', ... 'import eggrecipekss.core') >>> write(sample_buildout, 'allowdemo', 'setup.py', ... '''from setuptools import setup; setup( ... name='allowdemo', py_modules=['dependencydemo'], ... install_requires = 'kss.core', ... dependency_links = ['http://dist.plone.org'], ... zip_safe=True, version='1') ... ''') Now let's configure the buildout to use the develop egg, together with some rules that disallow any web site but PyPI and local files:: >>> write(sample_buildout, 'buildout.cfg', ... ''' ... [buildout] ... develop = allowdemo ... parts = eggs ... allow-hosts = ... pypi.org ... ... [eggs] ... recipe = zc.recipe.egg:eggs ... eggs = allowdemo ... ''') Now we can run the buildout and make sure all attempts to dist.plone.org fails:: >>> print_(system(buildout), end='') # doctest: +ELLIPSIS Develop: '/sample-buildout/allowdemo' Installing eggs... ... While: Installing eggs. Getting distribution for 'kss.core'. Error: Couldn't find a distribution for 'kss.core'. That's what we wanted : this will prevent any attempt to access unwanted domains. For instance, some packages are listing in their links `svn://` links. These can lead to error in some cases, and can therefore be protected like this:: XXX (showcase with a svn:// file) >>> write(sample_buildout, 'buildout.cfg', ... ''' ... [buildout] ... develop = allowdemo ... parts = eggs ... allow-hosts = ... ^(!svn://).* ... ... [eggs] ... recipe = zc.recipe.egg:eggs ... eggs = allowdemo ... ''') Now we can run the buildout and make sure all attempts to dist.plone.org fails:: >>> print_(system(buildout), end='') # doctest: +ELLIPSIS Develop: '/sample-buildout/allowdemo' Installing eggs... ... While: Installing eggs. Getting distribution for 'kss.core'. Error: Couldn't find a distribution for 'kss.core'. Test for issues --------------- Test for 1.0.5 breakage as in https://bugs.launchpad.net/zc.buildout/+bug/239212:: >>> write(sample_buildout, 'buildout.cfg', ... ''' ... [buildout] ... parts=python ... foo = ${python:interpreter} ... ... [python] ... recipe=zc.recipe.egg ... eggs=zc.buildout ... interpreter=python ... ''') >>> print_('XX'); print_(system(buildout), end='') # doctest: +ELLIPSIS X... Section `buildout` contains unused option(s): 'foo'. This may be an indication for either a typo in the option's name or a bug in the used recipe. Installing python... Generated interpreter '/sample-buildout/bin/python'. The bug 239212 above would have got us an *AttributeError* on *buildout._allow_hosts*. This was fixed in this changeset: http://svn.zope.org/zc.buildout/trunk/src/zc/buildout/buildout.py?rev=87309&r1=87277&r2=87309 ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675670.0 zc_buildout-4.1.6/src/zc/buildout/tests/bootstrap.txt0000644000076500000240000001320014773460426022437 0ustar00mauritsstaffMake sure the bootstrap script actually works:: >>> import os, sys >>> from os.path import dirname, join >>> import zc.buildout >>> bootstrap_py = join( ... dirname( ... dirname( ... dirname( ... dirname(zc.buildout.__file__) ... ) ... ) ... ), ... 'bootstrap', 'bootstrap.py') >>> sample_buildout = tmpdir('sample') >>> os.chdir(sample_buildout) >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = ... ''') >>> with open(bootstrap_py) as f: ... bootstrap_py_contents = f.read() >>> write('bootstrap.py', bootstrap_py_contents) >>> print_('X'); print_(system( ... zc.buildout.easy_install._safe_arg(sys.executable)+' '+ ... 'bootstrap.py')); print_('X') # doctest: +ELLIPSIS X... Creating directory '/sample/eggs'. Creating directory '/sample/bin'. Creating directory '/sample/parts'. Creating directory '/sample/develop-eggs'. Generated script '/sample/bin/buildout'. ... >>> ls(sample_buildout) d bin - bootstrap.py - buildout.cfg d develop-eggs d eggs d parts >>> ls(sample_buildout, 'bin') - buildout >>> print_('X'); ls(sample_buildout, 'eggs') # doctest: +ELLIPSIS X... d zc.buildout-...egg By default it gets the latest version:: >>> buildout_script = join(sample_buildout, 'bin', 'buildout') >>> if sys.platform.startswith('win'): ... buildout_script += '-script.py' >>> with open(buildout_script) as f: print_(f.read()) # doctest: +ELLIPSIS #... sys.path[0:0] = [ '/sample/eggs/zc.buildout-22.0.0...egg', '/sample/eggs/setuptools-...egg'... ]... Now trying the ``--buildout-version`` option, that let you define a version for ``zc.buildout``. Let's try with an unknown version:: >>> print_('X'); print_(system( ... zc.buildout.easy_install._safe_arg(sys.executable)+' '+ ... 'bootstrap.py --buildout-version UNKNOWN')); print_('X') # doctest: +ELLIPSIS ... X ... No local packages or...download links found for zc.buildout==UNKNOWN ... Now let's try with ``2.0.0``, which happens to exist:: >>> print_('X'); print_(system( ... zc.buildout.easy_install._safe_arg(sys.executable)+' '+ ... 'bootstrap.py --buildout-version 2.0.0')); print_('X') ... # doctest: +ELLIPSIS X...Generated script '/sample/bin/buildout'...X Let's make sure the generated ``buildout`` script uses it:: >>> with open(buildout_script) as f: print_(f.read()) # doctest: +ELLIPSIS #... sys.path[0:0] = [ '/sample/eggs/zc.buildout-2.0.0...egg', '/sample/eggs/setuptools-...egg'... ]... Now trying the ``--setuptools-version`` option, that lets you define a version for ``setuptools``. Now let's try with ``31.0.0``, which happens to exist:: >>> print_('X'); print_(system( ... zc.buildout.easy_install._safe_arg(sys.executable)+' '+ ... 'bootstrap.py --setuptools-version 31.0.0')); print_('X') ... # doctest: +ELLIPSIS X...Generated script '/sample/bin/buildout'...X Let's make sure the generated ``buildout`` script uses it:: >>> with open(buildout_script) as f: print_(f.read()) # doctest: +ELLIPSIS #... sys.path[0:0] = [ '/sample/eggs/zc.buildout-...egg', '/sample/eggs/setuptools-31.0.0...egg'... ]... Now let's try specifying both ``zc.buildout`` and ``setuptools`` to versions which happens to exist:: >>> print_('X'); print_(system( ... zc.buildout.easy_install._safe_arg(sys.executable)+' '+ ... 'bootstrap.py --setuptools-version 31.0.0 --buildout-version 2.0.0')); print_('X') ... # doctest: +ELLIPSIS X...Generated script '/sample/bin/buildout'...X Let's make sure the generated ``buildout`` script uses it:: >>> with open(buildout_script) as f: print_(f.read()) # doctest: +ELLIPSIS #... sys.path[0:0] = [ '/sample/eggs/zc.buildout-2.0.0...egg', '/sample/eggs/setuptools-31.0.0...egg'... ]... For a completely offline install we want to avoid downloading ``ez_setup.py``, specify the setuptools version, and to reuse the setuptools zipfile:: >>> from urllib.request import urlopen >>> ez_setup = urlopen('https://bootstrap.pypa.io/ez_setup.py').read() >>> write('ez_setup.py', ... '''print("Using local ez_setup.py") ... ''' + ez_setup.decode('ascii')) >>> os.path.exists('setuptools-32.1.0.zip') False >>> print_('X'); print_(system( ... zc.buildout.easy_install._safe_arg(sys.executable)+' '+ ... 'bootstrap.py --setuptools-version 32.1.0 --buildout-version 2.0.0 '+ ... '--setuptools-to-dir .')); print_('X') ... # doctest: +ELLIPSIS X...Using local ez_setup.py...Generated script '/sample/bin/buildout'...X >>> os.path.exists('setuptools-32.1.0.zip') True >>> with open(buildout_script) as f: print_(f.read()) # doctest: +ELLIPSIS #... sys.path[0:0] = [ '/sample/eggs/zc.buildout-2.0.0...egg', '/sample/eggs/setuptools-32.1.0...egg'... ]... You can ask ``bootstrap.py`` for its version. This is really the day the last change was made. A date leads to less confusion than a version number separate from buildout's own version number. Similarly, tracking buildout's version number leads to bootstraps with a new version number but without changes or to bootstraps with version number from an already-older buildout. So: a date is best:: >>> print_('X'); print_(system( ... zc.buildout.easy_install._safe_arg(sys.executable)+' '+ ... 'bootstrap.py --version')); print_('X') ... # doctest: +ELLIPSIS X...2015...X ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675670.0 zc_buildout-4.1.6/src/zc/buildout/tests/bootstrap_cl_settings.test0000644000076500000240000000371514773460426025207 0ustar00mauritsstaffSome people pass buildout settings to bootstrap. >>> import os, sys >>> from os.path import dirname, join >>> import zc.buildout >>> bootstrap_py = join( ... dirname( ... dirname( ... dirname( ... dirname(zc.buildout.__file__) ... ) ... ) ... ), ... 'bootstrap', 'bootstrap.py') >>> top = tmpdir('top') >>> mkdir(top, 'buildout') >>> os.chdir(top) >>> write('buildout', 'buildout.cfg', ... ''' ... [buildout] ... parts = ... ''') >>> with open(bootstrap_py) as f: bootstrap_py_contents = f.read() >>> write('bootstrap.py', bootstrap_py_contents) >>> print_('X'); print_(system( ... zc.buildout.easy_install._safe_arg(sys.executable) + ... ' bootstrap.py buildout:directory=' + top + ... ' -c'+join('buildout', 'buildout.cfg') ... )); print_('X') # doctest: +ELLIPSIS X... Creating directory '/top/eggs'. Creating directory '/top/bin'. Creating directory '/top/parts'. Creating directory '/top/develop-eggs'. Generated script '/top/bin/buildout'. ... X They might do it with init, but no worries: >>> os.chdir('..') >>> remove(top) >>> top = tmpdir('top') >>> os.chdir(top) >>> mkdir(top, 'buildout') >>> os.chdir(top) >>> with open(bootstrap_py) as f: bootstrap_py_contents = f.read() >>> write('bootstrap.py', bootstrap_py_contents) >>> print_('X'); print_(system( ... zc.buildout.easy_install._safe_arg(sys.executable) + ... ' bootstrap.py buildout:directory=' + top + ... ' -c'+join('buildout', 'buildout.cfg') + ... ' init' ... )); print_('X') # doctest: +ELLIPSIS X... Creating '/top/buildout/buildout.cfg'. Creating directory '/top/eggs'. Creating directory '/top/bin'. Creating directory '/top/parts'. Creating directory '/top/develop-eggs'. Generated script '/top/bin/buildout'. ... X ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675670.0 zc_buildout-4.1.6/src/zc/buildout/tests/buildout.txt0000644000076500000240000031353714773460426022271 0ustar00mauritsstaffBuildouts ========= The word "buildout" refers to a description of a set of parts and the software to create and assemble them. It is often used informally to refer to an installed system based on a buildout definition. For example, if we are creating an application named "Foo", then "the Foo buildout" is the collection of configuration and application-specific software that allows an instance of the application to be created. We may refer to such an instance of the application informally as "a Foo buildout". This document describes how to define buildouts using buildout configuration files and recipes. There are three ways to set up the buildout software and create a buildout instance: 1. Install the ``zc.buildout`` egg with ``easy_install`` and use the buildout script installed in a Python scripts area. 2. Use the buildout bootstrap script to create a buildout that includes both the ``setuptools`` and ``zc.buildout`` eggs. This allows you to use the buildout software without modifying a Python install. The buildout script is installed into your buildout local scripts area. 3. Use a buildout command from an already installed buildout to bootstrap a new buildout. (See the section on bootstrapping later in this document.) Often, a software project will be managed in a software repository, such as a subversion repository, that includes some software source directories, buildout configuration files, and a copy of the buildout bootstrap script. To work on the project, one would check out the project from the repository and run the bootstrap script which installs ``setuptools`` and ``zc.buildout`` into the checkout as well as any parts defined. We have a sample buildout that we created using the bootstrap command of an existing buildout (method 3 above). It has the absolute minimum information. We have ``bin``, ``develop-eggs``, ``eggs`` and ``parts`` directories, and a configuration file:: >>> ls(sample_buildout) d bin - buildout.cfg d develop-eggs d eggs d parts The ``bin`` directory contains scripts:: >>> ls(sample_buildout, 'bin') - buildout The ``eggs`` directory has installed distributions: >>> ls(sample_buildout, 'eggs') - packaging.egg-link - pip.egg-link - setuptools.egg-link - wheel.egg-link - zc.buildout.egg-link The ``develop-eggs`` and ``parts`` directories are initially empty:: >>> ls(sample_buildout, 'develop-eggs') >>> ls(sample_buildout, 'parts') The ``develop-eggs`` directory holds egg links for software being developed in the buildout. We separate ``develop-eggs`` and other eggs to allow eggs directories to be shared across multiple buildouts. For example, a common developer technique is to define a common eggs directory in their home that all non-develop eggs are stored in. This allows larger buildouts to be set up much more quickly and saves disk space. The ``parts`` directory provides an area where recipes can install part data. For example, if we built a custom Python, we would install it in the ``parts`` directory. Part data is stored in a sub-directory of the parts directory with the same name as the part. Buildouts are defined using configuration files. These are in the format defined by the Python ``ConfigParser`` module, with extensions that we'll describe later. By default, when a buildout is run, it looks for the file ``buildout.cfg`` in the directory where the buildout is run. The minimal configuration file has a ``buildout`` section that defines no parts:: >>> cat(sample_buildout, 'buildout.cfg') [buildout] parts = A part is simply something to be created by a buildout. It can be almost anything, such as a Python package, a program, a directory, or even a configuration file. Recipes ------- A part is created by a recipe. Recipes are always installed as Python eggs. They can be downloaded from a package server, such as the Python Package Index, or they can be developed as part of a project using a "develop" egg. A develop egg is a special kind of egg that gets installed as an "egg link" that contains the name of a source directory. Develop eggs don't have to be packaged for distribution to be used and can be modified in place, which is especially useful while they are being developed. Let's create a recipe as part of the sample project. We'll create a recipe for creating directories. First, we'll create a recipes source directory for our local recipes:: >>> mkdir(sample_buildout, 'recipes') and then we'll create a source file for our ``mkdir`` recipe:: >>> write(sample_buildout, 'recipes', 'mkdir.py', ... """ ... import logging, os, zc.buildout ... ... class Mkdir: ... ... def __init__(self, buildout, name, options): ... self.name, self.options = name, options ... options['path'] = os.path.join( ... buildout['buildout']['directory'], ... options['path'], ... ) ... if not os.path.isdir(os.path.dirname(options['path'])): ... logging.getLogger(self.name).error( ... 'Cannot create %s. %s is not a directory.', ... options['path'], os.path.dirname(options['path'])) ... raise zc.buildout.UserError('Invalid Path') ... ... ... def install(self): ... path = self.options['path'] ... logging.getLogger(self.name).info( ... 'Creating directory %s', os.path.basename(path)) ... os.mkdir(path) ... return path ... ... def update(self): ... pass ... """) Currently, recipes must define 3 methods: - a constructor, - an install method, and - an update method. The constructor is responsible for updating a parts options to reflect data read from other sections. The buildout system keeps track of whether a part specification has changed. A part specification has changed if it's options, after adjusting for data read from other sections, has changed, or if the recipe has changed. Only the options for the part are considered. If data are read from other sections, then that information has to be reflected in the parts options. In the ``mkdir`` example, the given path is interpreted relative to the buildout directory, and data from the buildout directory is read. The path option is updated to reflect this. If the directory option was changed in the buildout sections, we would know to update parts created using the ``mkdir`` recipe using relative path names. When buildout is run, it saves configuration data for installed parts in a file named ``.installed.cfg``. In subsequent runs, it compares part-configuration data stored in the ``.installed.cfg`` file and the part-configuration data loaded from the configuration files as modified by recipe constructors to decide if the configuration of a part has changed. If the configuration has changed, or if the recipe has changed, then the part is uninstalled and reinstalled. The buildout only looks at the part's options, so any data used to configure the part needs to be reflected in the part's options. It is the job of a recipe constructor to make sure that the options include all relevant data. Of course, parts are also uninstalled if they are no longer used. The recipe defines a constructor that takes a buildout object, a part name, and an options dictionary. It saves them in instance attributes. If the path is relative, we'll interpret it as relative to the buildout directory. The buildout object passed in is a mapping from section name to a mapping of options for that section. The buildout directory is available as the directory option of the buildout section. We normalize the path and save it back into the options directory. The install method is responsible for creating the part. In this case, we need the path of the directory to create. We'll use a path option from our options dictionary. The install method logs what it's doing using the Python ``logging`` call. We return the path that we installed. If the part is uninstalled or reinstalled, then the path returned will be removed by the buildout machinery. A recipe install method is expected to return a string, or an iterable of strings containing paths to be removed if a part is uninstalled. For most recipes, this is all of the uninstall support needed. For more complex uninstallation scenarios, use `Uninstall recipes`_. The update method is responsible for updating an already installed part. An empty method is often provided, as in this example, if parts can't be updated. An update method can return None, a string, or an iterable of strings. If a string or iterable of strings is returned, then the saved list of paths to be uninstalled is updated with the new information by adding any new files returned by the update method. We need to provide packaging information so that our recipe can be installed as a develop egg. The minimum information we need to specify is a name. For recipes, we also need to define the names of the recipe classes as entry points. Packaging information is provided via a ``setup.py`` script:: >>> write(sample_buildout, 'recipes', 'setup.py', ... """ ... from setuptools import setup ... ... setup( ... name = "recipes", ... entry_points = {'zc.buildout': ['mkdir = mkdir:Mkdir']}, ... ) ... """) Our setup script defines an *entry point*. Entry points provide a way for an egg to define the services it provides. Here we've said that we define a ``zc.buildout`` entry point named ``mkdir``. Recipe classes must be exposed as entry points in the ``zc.buildout`` group. We give entry points names within the group. We also need a ``README.txt`` for our recipes to avoid an annoying warning from ``distutils`` (which ``setuptools`` and ``zc.buildout`` are based on):: >>> write(sample_buildout, 'recipes', 'README.txt', " ") Now let's update our ``buildout.cfg``:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = recipes ... parts = data-dir ... ... [data-dir] ... recipe = recipes:mkdir ... path = mystuff ... """) Let's go through the changes one by one:: develop = recipes This tells the buildout to install a development egg for our recipes. Any number of paths can be listed. The paths can be relative or absolute. If relative, they are treated as relative to the buildout directory. They can be directory or file paths. If a file path is given, it should point to a Python setup script. If a directory path is given, it should point to a directory containing a ``setup.py`` file. Development eggs are installed before building any parts, as they may provide locally-defined recipes needed by the parts. :: parts = data-dir Here we've named a part to be "built". We can use any name we want, except that part names have to be unique and recipes will often use the part name to decide what to do. :: [data-dir] recipe = recipes:mkdir path = mystuff When we name a part, we also create a section of the same name that contains part data. In this section, we'll define the recipe to be used to install the part. In this case, we also specify the path to be created. Let's run the buildout. We do so by running the build script in the buildout:: >>> import os >>> os.chdir(sample_buildout) >>> buildout = os.path.join(sample_buildout, 'bin', 'buildout') >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Installing data-dir. data-dir: Creating directory mystuff We see that the recipe created the directory, as expected:: >>> ls(sample_buildout) - .installed.cfg d bin - buildout.cfg d develop-eggs d eggs d mystuff d parts d recipes In addition, .installed.cfg has been created containing information about the part we installed:: >>> cat(sample_buildout, '.installed.cfg') [buildout] installed_develop_eggs = /sample-buildout/develop-eggs/recipes.egg-link parts = data-dir [data-dir] __buildout_installed__ = /sample-buildout/mystuff __buildout_signature__ = recipes-c7vHV6ekIDUPy/7fjAaYjg== path = /sample-buildout/mystuff recipe = recipes:mkdir Note that the directory we installed is included in .installed.cfg. In addition, the path option includes the actual destination directory. If we change the name of the directory in the configuration file, we'll see that the directory gets removed and recreated:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = recipes ... parts = data-dir ... ... [data-dir] ... recipe = recipes:mkdir ... path = mydata ... """) >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Uninstalling data-dir. Installing data-dir. data-dir: Creating directory mydata >>> ls(sample_buildout) - .installed.cfg d bin - buildout.cfg d develop-eggs d eggs d mydata d parts d recipes If any of the files or directories created by a recipe are removed, the part will be reinstalled:: >>> rmdir(sample_buildout, 'mydata') >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Uninstalling data-dir. Installing data-dir. data-dir: Creating directory mydata Error reporting --------------- If a user makes an error the error needs to be reported, and work needs to stop. This is accomplished by logging a detailed error message and then raising a ``zc.buildout.UserError`` exception (or a subclass of this exception). Raising an error other than a ``UserError`` still displays the error, but labels it as a bug in the buildout software or recipe. In the sample above, if someone gives a non-existent directory to create the directory in:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = recipes ... parts = data-dir ... ... [data-dir] ... recipe = recipes:mkdir ... path = /xxx/mydata ... """) we'll get a user error, not a traceback:: >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' data-dir: Cannot create .../xxx/mydata. .../xxx is not a directory. While: Installing. Getting section data-dir. Initializing section data-dir. Error: Invalid Path Recipe Error Handling --------------------- If an error occurs during installation, it is up to the recipe to clean up any system side effects, such as files created. Let's update the ``mkdir`` recipe to support multiple paths:: >>> write(sample_buildout, 'recipes', 'mkdir.py', ... """ ... import logging, os, zc.buildout ... ... class Mkdir: ... ... def __init__(self, buildout, name, options): ... self.name, self.options = name, options ... ... # Normalize paths and check that their parent ... # directories exist: ... paths = [] ... for path in options['path'].split(): ... path = os.path.join(buildout['buildout']['directory'], path) ... if not os.path.isdir(os.path.dirname(path)): ... logging.getLogger(self.name).error( ... 'Cannot create %s. %s is not a directory.', ... options['path'], os.path.dirname(options['path'])) ... raise zc.buildout.UserError('Invalid Path') ... paths.append(path) ... options['path'] = ' '.join(paths) ... ... def install(self): ... paths = self.options['path'].split() ... for path in paths: ... logging.getLogger(self.name).info( ... 'Creating directory %s', os.path.basename(path)) ... os.mkdir(path) ... return paths ... ... def update(self): ... pass ... """) .. >>> clean_up_pyc(sample_buildout, 'recipes', 'mkdir.py') If there is an error creating a path, the install method will exit and leave previously created paths in place:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = recipes ... parts = data-dir ... ... [data-dir] ... recipe = recipes:mkdir ... path = foo bin ... """) >>> print_(system(buildout)) # doctest: +ELLIPSIS Develop: '/sample-buildout/recipes' Uninstalling data-dir. Installing data-dir. data-dir: Creating directory foo data-dir: Creating directory bin While: Installing data-dir. An internal error occurred due to a bug in either zc.buildout or in a recipe being used: Traceback (most recent call last): ... exists... We meant to create a directory ``bins``, but typed ``bin``. Now ``foo`` was left behind:: >>> os.path.exists('foo') True If we fix the typo:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = recipes ... parts = data-dir ... ... [data-dir] ... recipe = recipes:mkdir ... path = foo bins ... """) >>> print_(system(buildout)) # doctest: +ELLIPSIS Develop: '/sample-buildout/recipes' Installing data-dir. data-dir: Creating directory foo While: Installing data-dir. An internal error occurred due to a bug in either zc.buildout or in a recipe being used: Traceback (most recent call last): ... exists... Now they fail because ``foo`` exists, because it was left behind:: >>> remove('foo') Let's fix the recipe:: >>> write(sample_buildout, 'recipes', 'mkdir.py', ... """ ... import logging, os, zc.buildout, sys ... ... class Mkdir: ... ... def __init__(self, buildout, name, options): ... self.name, self.options = name, options ... ... # Normalize paths and check that their parent ... # directories exist: ... paths = [] ... for path in options['path'].split(): ... path = os.path.join(buildout['buildout']['directory'], path) ... if not os.path.isdir(os.path.dirname(path)): ... logging.getLogger(self.name).error( ... 'Cannot create %s. %s is not a directory.', ... options['path'], os.path.dirname(options['path'])) ... raise zc.buildout.UserError('Invalid Path') ... paths.append(path) ... options['path'] = ' '.join(paths) ... ... def install(self): ... paths = self.options['path'].split() ... created = [] ... try: ... for path in paths: ... logging.getLogger(self.name).info( ... 'Creating directory %s', os.path.basename(path)) ... os.mkdir(path) ... created.append(path) ... except Exception: ... for d in created: ... os.rmdir(d) ... assert not os.path.exists(d) ... logging.getLogger(self.name).info( ... 'Removed %s due to error', ... os.path.basename(d)) ... sys.stderr.flush() ... sys.stdout.flush() ... raise ... ... return paths ... ... def update(self): ... pass ... """) .. >>> clean_up_pyc(sample_buildout, 'recipes', 'mkdir.py') And put back the typo:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = recipes ... parts = data-dir ... ... [data-dir] ... recipe = recipes:mkdir ... path = foo bin ... """) When we rerun the buildout:: >>> print_(system(buildout)) # doctest: +ELLIPSIS Develop: '/sample-buildout/recipes' Installing data-dir. data-dir: Creating directory foo data-dir: Creating directory bin data-dir: Removed foo due to error While: Installing data-dir. An internal error occurred due to a bug in either zc.buildout or in a recipe being used: Traceback (most recent call last): ... exists... we get the same error, but we don't get the directory left behind:: >>> os.path.exists('foo') False It's critical that recipes clean up partial effects when errors occur. Because recipes most commonly create files and directories, buildout provides a helper API for removing created files when an error occurs. Option objects have a ``created`` method that can be called to record files as they are created. If the ``install`` or ``update`` method returns with an error, then any registered paths are removed automatically. The method returns the files registered and can be used to return the files created. Let's use this API to simplify the recipe:: >>> write(sample_buildout, 'recipes', 'mkdir.py', ... """ ... import logging, os, zc.buildout ... ... class Mkdir: ... ... def __init__(self, buildout, name, options): ... self.name, self.options = name, options ... ... # Normalize paths and check that their parent ... # directories exist: ... paths = [] ... for path in options['path'].split(): ... path = os.path.join(buildout['buildout']['directory'], path) ... if not os.path.isdir(os.path.dirname(path)): ... logging.getLogger(self.name).error( ... 'Cannot create %s. %s is not a directory.', ... options['path'], os.path.dirname(options['path'])) ... raise zc.buildout.UserError('Invalid Path') ... paths.append(path) ... options['path'] = ' '.join(paths) ... ... def install(self): ... paths = self.options['path'].split() ... for path in paths: ... logging.getLogger(self.name).info( ... 'Creating directory %s', os.path.basename(path)) ... os.mkdir(path) ... self.options.created(path) ... ... return self.options.created() ... ... def update(self): ... pass ... """) .. >>> clean_up_pyc(sample_buildout, 'recipes', 'mkdir.py') We returned by calling ``created``, taking advantage of the fact that it returns the registered paths. We did this for illustrative purposes. It would be simpler to just return the paths as before. If we rerun the buildout again, we'll get the error and no directories will be created:: >>> print_(system(buildout)) # doctest: +ELLIPSIS Develop: '/sample-buildout/recipes' Installing data-dir. data-dir: Creating directory foo data-dir: Creating directory bin While: Installing data-dir. An internal error occurred due to a bug in either zc.buildout or in a recipe being used: Traceback (most recent call last): ... exists... >>> os.path.exists('foo') False Now, we'll fix the typo again and we'll get the directories we expect:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = recipes ... parts = data-dir ... ... [data-dir] ... recipe = recipes:mkdir ... path = foo bins ... """) >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Installing data-dir. data-dir: Creating directory foo data-dir: Creating directory bins >>> os.path.exists('foo') True >>> os.path.exists('bins') True Configuration file syntax ------------------------- A buildout configuration file consists of a sequence of sections. A section has a *section header* followed by 0 or more *section options*. (Buildout configuration files may be viewed as a variation on INI files.) A section header consists of a section name enclosed in square braces. A section name consists of one or more non-whitespace characters other than square braces (``[``, ``]``), curly braces (``{``, ``}``), colons (``:``) or equal signs (``=``). Whitespace surrounding section names is ignored. A section header can optionally have a condition expression separated by a colon. See `Conditional sections`_. Options consist of option names, followed by optional space or tab characters, an optional plus or minus sign and an equal sign and values. An option value may be spread over multiple lines as long as the lines after the first start with a whitespace character. An option name consists of one or more non-whitespace characters other than equal signs, square braces (``[``, ``]``), curly braces (``{``, ``}``), plus signs or colons (``:``). The option name ``<`` is reserved. An option's data consists of the characters following the equal sign on the start line, plus the continuation lines. Option values have extra whitespace stripped. How this is done depends on whether the value has non-whitespace characters on the first line. If an option value has non-whitespace characters on the first line, then each line is stripped and blank lines are removed. For example, in:: [foo] bar = 1 baz = a b c .. -> text >>> from io import StringIO >>> import pprint, zc.buildout.configparser >>> pprint.pprint(zc.buildout.configparser.parse(StringIO( ... text), 'test')) {'foo': {'bar': '1', 'baz': 'a\nb\nc'}} The value of of ``bar`` is ``'1'`` and the value of ``baz`` is ``'a\nb\nc'``. If the first line of an option does **not** contain whitespace, then the value is dedented (with ``textwrap.dedent``), trailing spaces in lines are removed, and leading and trailing blank lines are removed. For example, in:: [foo] bar = baz = a b c .. -> text >>> pprint.pprint(zc.buildout.configparser.parse(StringIO( ... text), 'test')) {'foo': {'bar': '', 'baz': 'a\n b\n\nc'}} The value of bar is ``''``, and the value of baz is ``'a\n b\n\nc'``. Lines starting with ``#`` or ``;`` characters are comments. Comments can also be placed after the closing square bracket (``]``) in a section header. Buildout configuration data are Python strings, which are bytes in Python 2 and unicode in Python 3. Sections and options within sections may be repeated. Multiple occurrences of a section are treated as if they were concatenated. The last option value for a given name in a section overrides previous values. In addition to the syntactic details above: - option names are case sensitive - option values can use a substitution syntax, described below, to refer to option values in specific sections. - option values can be appended or removed using the - and + operators. Annotated sections ------------------ When used with the ``annotate`` command, buildout displays annotated sections. All sections are displayed, sorted alphabetically. For each section, all key-value pairs are displayed, sorted alphabetically, along with the origin of the value (file name or ``COMPUTED_VALUE``, ``DEFAULT_VALUE``, ``COMMAND_LINE_VALUE``):: >>> print_(system([buildout, 'annotate']), end='') ... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE Annotated sections ================== [buildout] allow-hosts= * DEFAULT_VALUE allow-picked-versions= true DEFAULT_VALUE allow-unknown-extras= false DEFAULT_VALUE bin-directory= bin DEFAULT_VALUE develop= recipes buildout.cfg develop-eggs-directory= develop-eggs DEFAULT_VALUE directory= /sample-buildout COMPUTED_VALUE eggs-directory= /sample-buildout/eggs DEFAULT_VALUE executable= ... DEFAULT_VALUE find-links= DEFAULT_VALUE install-from-cache= false DEFAULT_VALUE installed= .installed.cfg DEFAULT_VALUE log-format= DEFAULT_VALUE log-level= INFO DEFAULT_VALUE newest= true DEFAULT_VALUE offline= false DEFAULT_VALUE parts= data-dir buildout.cfg parts-directory= parts DEFAULT_VALUE prefer-final= true DEFAULT_VALUE python= buildout DEFAULT_VALUE show-picked-versions= false DEFAULT_VALUE socket-timeout= DEFAULT_VALUE update-versions-file= DEFAULT_VALUE use-dependency-links= true DEFAULT_VALUE versions= versions DEFAULT_VALUE [data-dir] path= foo bins buildout.cfg recipe= recipes:mkdir buildout.cfg [versions] zc.buildout = >=1.99 DEFAULT_VALUE zc.recipe.egg = >=1.99 DEFAULT_VALUE The ``annotate`` command is sensitive to the verbosity flag. You get more information about the way values are computed:: >>> print_(system([buildout, '-v', 'annotate']), end='') ... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE Annotated sections ================== [buildout] allow-hosts= * AS DEFAULT_VALUE SET VALUE = * allow-picked-versions= true AS DEFAULT_VALUE SET VALUE = true allow-unknown-extras= false AS DEFAULT_VALUE SET VALUE = false bin-directory= bin AS DEFAULT_VALUE SET VALUE = bin develop= recipes IN buildout.cfg SET VALUE = recipes develop-eggs-directory= develop-eggs AS DEFAULT_VALUE SET VALUE = develop-eggs directory= /sample-buildout AS COMPUTED_VALUE SET VALUE = /sample-buildout eggs-directory= /sample-buildout/eggs AS DEFAULT_VALUE DIRECTORY VALUE = /sample-buildout/eggs AS DEFAULT_VALUE SET VALUE = eggs executable= ... AS DEFAULT_VALUE SET VALUE = ... find-links= AS DEFAULT_VALUE SET VALUE = install-from-cache= false AS DEFAULT_VALUE SET VALUE = false installed= .installed.cfg AS DEFAULT_VALUE SET VALUE = .installed.cfg log-format= AS DEFAULT_VALUE SET VALUE = log-level= INFO AS DEFAULT_VALUE SET VALUE = INFO newest= true AS DEFAULT_VALUE SET VALUE = true offline= false AS DEFAULT_VALUE SET VALUE = false parts= data-dir IN buildout.cfg SET VALUE = data-dir parts-directory= parts AS DEFAULT_VALUE SET VALUE = parts prefer-final= true AS DEFAULT_VALUE SET VALUE = true python= buildout AS DEFAULT_VALUE SET VALUE = buildout show-picked-versions= false AS DEFAULT_VALUE SET VALUE = false socket-timeout= AS DEFAULT_VALUE SET VALUE = update-versions-file= AS DEFAULT_VALUE SET VALUE = use-dependency-links= true AS DEFAULT_VALUE SET VALUE = true verbosity= 10 AS COMMAND_LINE_VALUE SET VALUE = 10 versions= versions AS DEFAULT_VALUE SET VALUE = versions [data-dir] path= foo bins IN buildout.cfg SET VALUE = foo bins recipe= recipes:mkdir IN buildout.cfg SET VALUE = recipes:mkdir [versions] ... The output of the ``annotate`` command can be very long. You can restrict the output to some sections by passing section names as arguments:: >>> print_(system([buildout, 'annotate', 'versions']), end='') ... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE Annotated sections ================== [versions] zc.buildout= >=1.99 DEFAULT_VALUE zc.recipe.egg= >=1.99 DEFAULT_VALUE Query values ------------ For continuous integration, it might be useful to query the buildout config. >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = . ... ... [values] ... host = buildout.org ... multiline = ... first ... second ... """) >>> print_(system([buildout, 'query', 'buildout:develop']), end='') . >>> print_(system([buildout, 'query', 'values:host']), end='') buildout.org >>> print_(system([buildout, 'query', 'values:multiline']), end='') first second As with assignments, if the section is omitted, 'buildout' section is assumed. >>> print_(system([buildout, 'query', 'develop']), end='') . When used with -v option, the query command also displays section and key. >>> print_(system([buildout, '-v', 'query', 'develop']), end='') ${buildout:develop} . >>> print_(system([buildout, '-v', 'query', 'values:host']), end='') ${values:host} buildout.org The query commands outputs proper error messages. >>> print_(system([buildout, 'query', 'versions', 'parts']), end='') Error: The query command requires a single argument. >>> print_(system([buildout, 'query']), end='') Error: The query command requires a single argument. >>> print_(system([buildout, 'query', 'invalid:section:key']), end='') Error: Invalid option: invalid:section:key >>> print_(system([buildout, '-v', 'query', 'values:port']), end='') ${values:port} Error: Key not found: port >>> print_(system([buildout, '-v', 'query', 'versionx']), end='') ${buildout:versionx} Error: Key not found: versionx >>> print_(system([buildout, '-v', 'query', 'specific:port']), end='') ${specific:port} Error: Section not found: specific Variable substitutions ---------------------- Buildout configuration files support variable substitution. To illustrate this, we'll create an debug recipe to allow us to see interactions with the buildout:: >>> write(sample_buildout, 'recipes', 'debug.py', ... """ ... import sys ... class Debug: ... ... def __init__(self, buildout, name, options): ... self.buildout = buildout ... self.name = name ... self.options = options ... ... def install(self): ... for option, value in sorted(self.options.items()): ... sys.stdout.write('%s %s\\n' % (option, value)) ... return () ... ... update = install ... """) >>> write(sample_buildout, 'recipes', 'environ.py', ... """ ... import sys ... import os ... class Environ: ... ... def __init__(self, buildout, name, options): ... self.buildout = buildout ... self.options = options ... ... def install(self): ... _ = self.options['name'] ... sys.stdout.write('HOME %s\\n' % os.environ['HOME']) ... sys.stdout.write('USERPROFILE %s\\n' % os.environ['USERPROFILE']) ... sys.stdout.write('expanduser %s\\n' % os.path.expanduser('~')) ... return () ... ... update = install ... """) This recipe doesn't actually create anything. The install method doesn't return anything, because it didn't create any files or directories. We also have to update our setup script:: >>> write(sample_buildout, 'recipes', 'setup.py', ... """ ... from setuptools import setup ... entry_points = ( ... ''' ... [zc.buildout] ... mkdir = mkdir:Mkdir ... debug = debug:Debug ... environ = environ:Environ ... ''') ... setup(name="recipes", entry_points=entry_points, py_modules=['debug', 'environ']) ... """) We've rearranged the script a bit to make the entry points easier to edit. Specifically, entry points are now defined as a configuration string, rather than a dictionary. Let's update our configuration to provide variable substitution examples:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = recipes ... parts = data-dir debug ... log-level = INFO ... ... [debug] ... recipe = recipes:debug ... File-1 = ${data-dir:path}/file ... File-2 = ${debug:File-1}/log ... ... [data-dir] ... recipe = recipes:mkdir ... path = mydata ... """) We used a string-template substitution for ``File-1`` and ``File-2``. This type of substitution uses the ``string.Template`` syntax. Names substituted are qualified option names, consisting of a section name and option name joined by a colon. Now, if we run the buildout, we'll see the options with the values substituted:: >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Uninstalling data-dir. Installing data-dir. data-dir: Creating directory mydata Installing debug. File-1 /sample-buildout/mydata/file File-2 /sample-buildout/mydata/file/log recipe recipes:debug Note that the substitution of the ``data-dir`` path option reflects the update to the option performed by the ``mkdir`` recipe. It might seem surprising that ``mydata`` was created again. This is because we changed our ``recipes`` package by adding the ``debug`` module. The buildout system didn't know if this module could affect the ``mkdir`` recipe, so it assumed it could and reinstalled ``mydata``. If we rerun the buildout:: >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Updating data-dir. Updating debug. File-1 /sample-buildout/mydata/file File-2 /sample-buildout/mydata/file/log recipe recipes:debug we can see that ``mydata`` was not recreated. Note that, in this case, we didn't specify a log level, so we didn't get output about what the buildout was doing. Section and option names in variable substitutions are only allowed to contain alphanumeric characters, hyphens, periods and spaces. This restriction might be relaxed in future releases. We can omit the section name in a variable substitution to refer to the current section. We can also use the special option, ``_buildout_section_name_`` to get the current section name:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = recipes ... parts = data-dir debug ... log-level = INFO ... ... [debug] ... recipe = recipes:debug ... File-1 = ${data-dir:path}/file ... File-2 = ${:File-1}/log ... my_name = ${:_buildout_section_name_} ... ... [data-dir] ... recipe = recipes:mkdir ... path = mydata ... """) >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Uninstalling debug. Updating data-dir. Installing debug. File-1 /sample-buildout/mydata/file File-2 /sample-buildout/mydata/file/log my_name debug recipe recipes:debug Automatic part selection and ordering ------------------------------------- When a section with a recipe is referred to, either through variable substitution or by an initializing recipe, the section is treated as a part and added to the part list before the referencing part. For example, we can leave ``data-dir`` out of the parts list:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = recipes ... parts = debug ... log-level = INFO ... ... [debug] ... recipe = recipes:debug ... File-1 = ${data-dir:path}/file ... File-2 = ${debug:File-1}/log ... ... [data-dir] ... recipe = recipes:mkdir ... path = mydata ... """) It will still be treated as a part:: >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Uninstalling debug. Updating data-dir. Installing debug. File-1 /sample-buildout/mydata/file File-2 /sample-buildout/mydata/file/log recipe recipes:debug >>> cat('.installed.cfg') # doctest: +ELLIPSIS [buildout] installed_develop_eggs = /sample-buildout/develop-eggs/recipes.egg-link parts = data-dir debug ... Note that the ``data-dir`` part is included *before* the ``debug`` part, because the ``debug`` part refers to the ``data-dir`` part. Even if we list the ``data-dir`` part after the ``debug`` part, it will be included before:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = recipes ... parts = debug data-dir ... log-level = INFO ... ... [debug] ... recipe = recipes:debug ... File-1 = ${data-dir:path}/file ... File-2 = ${debug:File-1}/log ... ... [data-dir] ... recipe = recipes:mkdir ... path = mydata ... """) It will still be treated as a part:: >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Updating data-dir. Updating debug. File-1 /sample-buildout/mydata/file File-2 /sample-buildout/mydata/file/log recipe recipes:debug >>> cat('.installed.cfg') # doctest: +ELLIPSIS [buildout] installed_develop_eggs = /sample-buildout/develop-eggs/recipes.egg-link parts = data-dir debug ... Extending sections (macros) --------------------------- A section (other than the buildout section) can extend one or more other sections using the ``<`` option. Options from the referenced sections are copied to the referring section *before* variable substitution. This, together with the ability to refer to variables of the current section, allows sections to be used as macros:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = recipes ... parts = myfiles ... log-level = INFO ... ... [debug] ... recipe = recipes:debug ... ... [with_file1] ... <= debug ... file1 = ${:path}/file1 ... color = red ... ... [with_file2] ... <= debug ... file2 = ${:path}/file2 ... color = blue ... ... [myfiles] ... <= with_file1 ... with_file2 ... path = mydata ... """) >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Uninstalling debug. Uninstalling data-dir. Installing myfiles. color blue file1 mydata/file1 file2 mydata/file2 path mydata recipe recipes:debug In this example, the ``debug``, ``with_file1`` and ``with_file2`` sections act as macros. In particular, the variable substitutions are performed relative to the ``myfiles`` section. .. note:: Don't be fooled by the appearance of the ``<= section`` lines --- though ``<=`` may look like a new operator, it's still just the familiar ``key = value`` syntax. .. cleanup buildout >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = recipes ... parts = ... """) >>> _ = system(buildout) Extending sections (macros) - Adding and removing options --------------------------------------------------------- We can also add and remove options in extended sections. This is illustrated below; first we define a base configuration. >>> write(sample_buildout, 'base.cfg', ... """ ... [buildout] ... parts = part1 part2 part3 ... ... [part1] ... recipe = ... option = a1 ... a2 ... ... [part2] ... <= part1 ... option -= a1 ... option += c3 c4 ... ... [part3] ... <= part2 ... option += d2 ... c5 d1 d6 ... option -= a2 ... """) To verify that the options are adjusted correctly, we'll set up an extension that prints out the options. >>> mkdir(sample_buildout, 'demo') >>> write(sample_buildout, 'demo', 'demo.py', ... """ ... import sys ... def ext(buildout): ... sys.stdout.write(str( ... [part['option'] for name, part in sorted(buildout.items()) ... if name.startswith('part')])+'\\n') ... """) >>> write(sample_buildout, 'demo', 'setup.py', ... """ ... from setuptools import setup ... ... setup( ... name="demo", ... entry_points={'zc.buildout.extension': ['ext = demo:ext']}, ... ) ... """) Set up a buildout configuration for this extension. >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = demo ... parts = ... """) >>> os.chdir(sample_buildout) >>> print_(system(os.path.join(sample_buildout, 'bin', 'buildout')), end='') # doctest: +ELLIPSIS Develop: '/sample-buildout/demo' Verify option values. >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = demo ... extensions = demo ... extends = base.cfg ... """) >>> print_(system(os.path.join('bin', 'buildout')), end='') ['a1/na2', 'a2/nc3 c4', 'c3 c4/nd2/nc5 d1 d6'] Develop: '/sample-buildout/demo' Cleanup. >>> os.remove(os.path.join(sample_buildout, 'base.cfg')) >>> rmdir(sample_buildout, 'demo') Adding and removing options --------------------------- We can append and remove values to an option by using the ``+`` and ``-`` operators. This is illustrated below; first we define a base configuration:: >>> write(sample_buildout, 'base.cfg', ... """ ... [buildout] ... parts = part1 part2 part3 ... ... [part1] ... recipe = ... option = a1 a2 ... ... [part2] ... recipe = ... option = b1 b2 b3 b4 ... ... [part3] ... recipe = ... option = c1 c2 ... ... [part4] ... recipe = ... option = d2 ... d3 ... d5 ... ... # Issue #641 - Properly handle options which are initially defined ... # using += / -= ... [part6] ... option += e1 ... ... [part7] ... option -= f1 ... ... """) Extending this configuration, we can "adjust" the values set in the base configuration file:: >>> write(sample_buildout, 'extension1.cfg', ... """ ... [buildout] ... extends = base.cfg ... ... # appending values ... [part1] ... option += a3 a4 ... ... # removing values ... [part2] ... option -= b1 b2 ... ... # alt. spelling ... [part3] ... option+=c3 c4 c5 ... ... # combining both adding and removing ... [part4] ... option += d1 ... d4 ... option -= d5 ... ... # normal assignment ... [part5] ... option = h1 h2 ... ... """) An additional extension:: >>> write(sample_buildout, 'extension2.cfg', ... """ ... [buildout] ... extends = extension1.cfg ... ... # appending values ... [part1] ... option += a5 ... ... # removing values ... [part2] ... option -= b1 b2 b3 ... ... """) To verify that the options are adjusted correctly, we'll set up an extension that prints out the options:: >>> mkdir(sample_buildout, 'demo') >>> write(sample_buildout, 'demo', 'demo.py', ... """ ... import sys ... def ext(buildout): ... sys.stdout.write(str( ... [part.get('option') for name, part in sorted(buildout.items()) ... if name.startswith('part')])+'\\n') ... """) >>> write(sample_buildout, 'demo', 'setup.py', ... """ ... from setuptools import setup ... ... setup( ... name="demo", ... entry_points={'zc.buildout.extension': ['ext = demo:ext']}, ... ) ... """) Set up a buildout configuration for this extension:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = demo ... parts = ... """) >>> os.chdir(sample_buildout) >>> print_(system(os.path.join(sample_buildout, 'bin', 'buildout')), end='') ... # doctest: +ELLIPSIS Develop: '/sample-buildout/demo'... Verify option values:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = demo ... extensions = demo ... extends = extension2.cfg ... """) >>> print_(system(os.path.join('bin', 'buildout')), end='') ['a1 a2/na3 a4/na5', 'b1 b2 b3 b4', 'c1 c2/nc3 c4 c5', 'd2/nd3/nd1/nd4', 'h1 h2', 'e1', ''] Develop: '/sample-buildout/demo' Annotated sections output shows which files are responsible for which operations:: >>> print_(system(os.path.join('bin', 'buildout') + ' annotate'), end='') ... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE Annotated sections ================== ... [part1] option= a1 a2 a3 a4 a5 base.cfg += extension1.cfg += extension2.cfg recipe= base.cfg [part2] option= b1 b2 b3 b4 base.cfg -= extension1.cfg -= extension2.cfg recipe= base.cfg [part3] option= c1 c2 c3 c4 c5 base.cfg += extension1.cfg recipe= base.cfg [part4] option= d2 d3 d1 d4 base.cfg += extension1.cfg -= extension1.cfg recipe= base.cfg [part5] option= h1 h2 extension1.cfg [part6] option= e1 base.cfg [part7] option= base.cfg -= IMPLICIT_VALUE [versions] zc.buildout= >=1.99 DEFAULT_VALUE zc.recipe.egg= >=1.99 DEFAULT_VALUE With more verbosity:: >>> print_(system(os.path.join('bin', 'buildout') + ' -v annotate'), end='') ... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE Annotated sections ================== ... [part1] option= a1 a2 a3 a4 a5 IN extension2.cfg ADD VALUE = a5 IN extension1.cfg ADD VALUE = a3 a4 IN base.cfg SET VALUE = a1 a2 ... [part2] option= b1 b2 b3 b4 IN extension2.cfg REMOVE VALUE = b1 b2 b3 IN extension1.cfg REMOVE VALUE = b1 b2 IN base.cfg SET VALUE = b1 b2 b3 b4 ... [part3] option= c1 c2 c3 c4 c5 IN extension1.cfg ADD VALUE = c3 c4 c5 IN base.cfg SET VALUE = c1 c2 ... [part4] option= d2 d3 d1 d4 IN extension1.cfg REMOVE VALUE = d5 IN extension1.cfg ADD VALUE = d1 d4 IN base.cfg SET VALUE = d2 d3 d5 ... [part5] option= h1 h2 IN extension1.cfg SET VALUE = h1 h2 [part6] option= e1 IN base.cfg SET VALUE = e1 [part7] option= IN IMPLICIT_VALUE REMOVE VALUE = f1 IN base.cfg SET VALUE = f1 ... Issue #656 - If an option was defined in one file, then extended twice in another - once in a regular section and again in a conditonal, the original definition is lost, not extended:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... extends = ... extension1.cfg ... extension2.cfg ... develop = demo ... extensions = demo ... parts = part1 ... """) >>> write(sample_buildout, 'extension1.cfg', ... """ ... [part1] ... recipe = ... option = ... a ... b ... """) >>> write(sample_buildout, 'extension2.cfg', ... """ ... [part1] ... option += ... c ... d ... ... [part1:False] ... option += ... z ... ... [part1:True] ... option += ... e ... """) ... parts = part1 >>> print_(system(os.path.join('bin', 'buildout')), end='') ['a/nb/nc/nd/ne'] Develop: '/sample-buildout/demo' Cleanup:: >>> os.remove(os.path.join(sample_buildout, 'base.cfg')) >>> os.remove(os.path.join(sample_buildout, 'extension1.cfg')) >>> os.remove(os.path.join(sample_buildout, 'extension2.cfg')) Multiple configuration files ---------------------------- A configuration file can *extend* another configuration file. Options are read from the other configuration file if they aren't already defined by your configuration file. The configuration files your file extends can extend other configuration files. The same file may be used more than once although, of course, cycles aren't allowed. To see how this works, we use an example:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... extends = base.cfg ... ... [debug] ... op = buildout ... """) >>> write(sample_buildout, 'base.cfg', ... """ ... [buildout] ... develop = recipes ... parts = debug ... ... [debug] ... recipe = recipes:debug ... op = base ... """) >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Installing debug. op buildout recipe recipes:debug The example is pretty trivial, but the pattern it illustrates is pretty common. In a more practical example, the base buildout might represent a product and the extending buildout might be a customization. Here is a more elaborate example:: >>> other = tmpdir('other') >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... extends = b1.cfg b2.cfg %(b3)s ... ... [debug] ... op = buildout ... """ % dict(b3=os.path.join(other, 'b3.cfg'))) >>> write(sample_buildout, 'b1.cfg', ... """ ... [buildout] ... extends = base.cfg ... ... [debug] ... op1 = b1 1 ... op2 = b1 2 ... """) >>> write(sample_buildout, 'b2.cfg', ... """ ... [buildout] ... extends = base.cfg ... ... [debug] ... op2 = b2 2 ... op3 = b2 3 ... """) >>> write(other, 'b3.cfg', ... """ ... [buildout] ... extends = b3base.cfg ... ... [debug] ... op4 = b3 4 ... """) >>> write(other, 'b3base.cfg', ... """ ... [debug] ... op5 = b3base 5 ... """) >>> write(sample_buildout, 'base.cfg', ... """ ... [buildout] ... develop = recipes ... parts = debug ... ... [debug] ... recipe = recipes:debug ... name = base ... ... [environ] ... recipe = recipes:environ ... name = base ... """) >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Uninstalling debug. Installing debug. name base op buildout op1 b1 1 op2 b2 2 op3 b2 3 op4 b3 4 op5 b3base 5 recipe recipes:debug There are several things to note about this example: - We can name multiple files in an ``extends`` option. - We can reference files recursively. - Relative file names in extended options are interpreted relative to the directory containing the referencing configuration file. >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... extends = b1.cfg ... ... [debug] ... op = buildout ... """) >>> print_(system([buildout, 'buildout:extends=b2.cfg']), end='') Develop: '/sample-buildout/recipes' Uninstalling debug. Installing debug. name base op buildout op1 b1 1 op2 b2 2 op3 b2 3 recipe recipes:debug >>> print_(system([buildout, 'buildout:extends=b2.cfg %(b3)s' ... % dict(b3=os.path.join(other, 'b3.cfg'))]), end='') Develop: '/sample-buildout/recipes' Uninstalling debug. Installing debug. name base op buildout op1 b1 1 op2 b2 2 op3 b2 3 op4 b3 4 op5 b3base 5 recipe recipes:debug >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... extends = b1.cfg b2.cfg %(b3)s ... ... [debug] ... op = buildout ... """ % dict(b3=os.path.join(other, 'b3.cfg'))) Optional extends ---------------- With ``optional-extends`` you can specify an extends file that is only loaded when it exists. The main idea is that a Buildout project in a shared Git repository can specify an ``optional-extends`` line with ``local.cfg`` or ``custom.cfg``. This optional file would be added to the files ignored by Git, and would only exist on individual computers. Any developer who wants this, can then create such a file and for example use this to always add the ``pdbpp`` egg to one of the Buildout parts. Or they can add a part, or other options specific to their system. >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... extends = b1.cfg ... optional-extends = optional.cfg ... ... [debug] ... op = buildout ... """) >>> print_(system(buildout), end='') optional-extends file not found: optional.cfg Develop: '/sample-buildout/recipes' Uninstalling debug. Installing debug. name base op buildout op1 b1 1 op2 b1 2 recipe recipes:debug Now add the optional config: >>> write('optional.cfg', ... """ ... [debug] ... op2 = optional2 2 ... """) >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Uninstalling debug. Installing debug. name base op buildout op1 b1 1 op2 optional2 2 recipe recipes:debug Restore the previous ``buildout.cfg`` and remove the new file, so we do not interfere with existing tests:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... extends = b1.cfg b2.cfg %(b3)s ... ... [debug] ... op = buildout ... """ % dict(b3=os.path.join(other, 'b3.cfg'))) >>> remove(sample_buildout, 'optional.cfg') Loading Configuration from URLs ------------------------------- Configuration files can be loaded from URLs. To see how this works, we'll set up a web server with some configuration files:: >>> server_data = tmpdir('server_data') >>> write(server_data, "r1.cfg", ... """ ... [debug] ... op1 = r1 1 ... op2 = r1 2 ... """) >>> write(server_data, "r2.cfg", ... """ ... [buildout] ... extends = r1.cfg ... ... [debug] ... op2 = r2 2 ... op3 = r2 3 ... """) >>> server_url = start_server(server_data) >>> write('client.cfg', ... """ ... [buildout] ... develop = recipes ... parts = debug ... extends = %(url)s/r2.cfg ... ... [debug] ... recipe = recipes:debug ... name = base ... """ % dict(url=server_url)) >>> print_(system([buildout, '-c', 'client.cfg']), end='') Develop: '/sample-buildout/recipes' Uninstalling debug. Installing debug. name base op1 r1 1 op2 r2 2 op3 r2 3 recipe recipes:debug Here we specified a URL for the file we extended. The file we downloaded itself referred to a file on the server using a relative URL reference. Relative references are interpreted relative to the base URL when they appear in configuration files loaded via URL. We can also specify a URL as the configuration file to be used by a buildout:: >>> os.remove('client.cfg') >>> write(server_data, 'remote.cfg', ... """ ... [buildout] ... develop = recipes ... parts = debug ... extends = r2.cfg ... ... [debug] ... recipe = recipes:debug ... name = remote ... """) >>> print_(system([buildout, '-c', server_url + '/remote.cfg']), end='') While: Initializing. Error: Missing option: buildout:directory Normally, the buildout directory defaults to a directory containing a configuration file. This won't work for configuration files loaded from URLs. In this case, the buildout directory would normally be defined on the command line:: >>> print_(system([buildout, ... '-c', server_url + '/remote.cfg', ... 'buildout:directory=' + sample_buildout, ... ]), end='') Develop: '/sample-buildout/recipes' Uninstalling debug. Installing debug. name remote op1 r1 1 op2 r2 2 op3 r2 3 recipe recipes:debug User defaults ------------- If the file ``$HOME/.buildout/default.cfg`` exists, it is read before reading the configuration file. (``$HOME`` is the value of the ``HOME`` environment variable. The ``/`` is replaced by the operating system file delimiter.):: >>> home = tmpdir('home') >>> mkdir(home, '.buildout') >>> default_cfg = join(home, '.buildout', 'default.cfg') >>> write(default_cfg, ... """ ... [debug] ... op1 = 1 ... op7 = 7 ... """) >>> env = dict(HOME=home, USERPROFILE=home) >>> print_(system(buildout, env=env), end='') Develop: '/sample-buildout/recipes' Uninstalling debug. Installing debug. name base op buildout op1 b1 1 op2 b2 2 op3 b2 3 op4 b3 4 op5 b3base 5 op7 7 recipe recipes:debug A buildout command-line argument, ``-U``, can be used to suppress reading user defaults:: >>> print_(system([buildout, '-U'], env=env), end='') Develop: '/sample-buildout/recipes' Uninstalling debug. Installing debug. name base op buildout op1 b1 1 op2 b2 2 op3 b2 3 op4 b3 4 op5 b3base 5 recipe recipes:debug If the environment variable ``BUILDOUT_HOME`` is non-empty, that is used to locate ``default.cfg`` instead of looking in ``~/.buildout/``. Let's set up a configuration file in an alternate directory and verify that we get the appropriate set of defaults:: >>> alterhome = tmpdir('alterhome') >>> write(alterhome, 'default.cfg', ... """ ... [debug] ... op1 = 1' ... op7 = 7' ... op8 = eight! ... """) >>> env['BUILDOUT_HOME'] = alterhome >>> print_(system(buildout, env=env), end='') Develop: '/sample-buildout/recipes' Uninstalling debug. Installing debug. name base op buildout op1 b1 1 op2 b2 2 op3 b2 3 op4 b3 4 op5 b3base 5 op7 7' op8 eight! recipe recipes:debug The ``-U`` argument still suppresses reading of the ``default.cfg`` file from ``BUILDOUT_HOME``:: >>> print_(system([buildout, '-U'], env=env), end='') Develop: '/sample-buildout/recipes' Uninstalling debug. Installing debug. name base op buildout op1 b1 1 op2 b2 2 op3 b2 3 op4 b3 4 op5 b3base 5 recipe recipes:debug Log level --------- We can control the level of logging by specifying a log level in our configuration file. For example, to suppress info messages, we can set the logging level to *WARNING*:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... log-level = WARNING ... extends = b1.cfg b2.cfg ... """) >>> print_(system(buildout), end='') name base op1 b1 1 op2 b2 2 op3 b2 3 recipe recipes:debug Socket timeout -------------- The timeout of the connections to egg and configuration servers can be configured in the buildout section. Its value is configured in seconds:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... socket-timeout = 5 ... develop = recipes ... parts = debug ... ... [debug] ... recipe = recipes:debug ... op = timeout ... """) >>> print_(system(buildout), end='') Setting socket time out to 5 seconds. Develop: '/sample-buildout/recipes' Uninstalling debug. Installing debug. op timeout recipe recipes:debug If the ``socket-timeout`` is not numeric, a warning is issued and the default timeout of the Python socket module is used:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... socket-timeout = 5s ... develop = recipes ... parts = debug ... ... [debug] ... recipe = recipes:debug ... op = timeout ... """) >>> print_(system(buildout), end='') Default socket timeout is used ! Value in configuration is not numeric: [5s]. Develop: '/sample-buildout/recipes' Updating debug. op timeout recipe recipes:debug Uninstall recipes ----------------- As we've seen, when parts are installed, buildout keeps track of files and directories that they create. When the parts are uninstalled these files and directories are deleted. Sometimes more clean-up is needed. For example, a recipe might add a system service by calling ``chkconfig --add`` during installation. Later during uninstallation, ``chkconfig --del`` will need to be called to remove the system service. In order to deal with these uninstallation issues, you can register uninstall recipes. Uninstall recipes are registered using the ``zc.buildout.uninstall`` entry point. Parts specify uninstall recipes using the ``uninstall`` option. In comparison to regular recipes, uninstall recipes are much simpler. They are simply callable objects that accept the name of the part to be uninstalled and the part's options dictionary. Uninstall recipes don't have access to the part itself since it may be impossible to instantiate at uninstallation time. Here's a recipe that simulates installation of a system service, along with an uninstall recipe that simulates removing the service:: >>> write(sample_buildout, 'recipes', 'service.py', ... """ ... import sys ... class Service: ... ... def __init__(self, buildout, name, options): ... self.buildout = buildout ... self.name = name ... self.options = options ... ... def install(self): ... sys.stdout.write("chkconfig --add %s\\n" ... % self.options['script']) ... return () ... ... def update(self): ... pass ... ... ... def uninstall_service(name, options): ... sys.stdout.write("chkconfig --del %s\\n" % options['script']) ... """) To use these recipes we must register them using entry points. Make sure to use the same name for the recipe and uninstall recipe. This is required to let buildout know which uninstall recipe goes with which recipe:: >>> write(sample_buildout, 'recipes', 'setup.py', ... """ ... from setuptools import setup ... entry_points = ( ... ''' ... [zc.buildout] ... mkdir = mkdir:Mkdir ... debug = debug:Debug ... service = service:Service ... ... [zc.buildout.uninstall] ... service = service:uninstall_service ... ''') ... setup(name="recipes", entry_points=entry_points, py_modules=['debug', 'environ', 'service']) ... """) Here's how these recipes could be used in a buildout:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = recipes ... parts = service ... ... [service] ... recipe = recipes:service ... script = /path/to/script ... """) When the buildout is run the service will be installed:: >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Uninstalling debug. Installing service. chkconfig --add /path/to/script The service has been installed. If the buildout is run again with no changes, the service shouldn't be changed:: >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Updating service. Now we change the service part to trigger uninstallation and re-installation:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = recipes ... parts = service ... ... [service] ... recipe = recipes:service ... script = /path/to/a/different/script ... """) >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Uninstalling service. Running uninstall recipe. chkconfig --del /path/to/script Installing service. chkconfig --add /path/to/a/different/script Now we remove the service part, and add another part:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = recipes ... parts = debug ... ... [debug] ... recipe = recipes:debug ... """) >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Uninstalling service. Running uninstall recipe. chkconfig --del /path/to/a/different/script Installing debug. recipe recipes:debug Uninstall recipes don't have to take care of removing all the files and directories created by the part. This is still done automatically, following the execution of the uninstall recipe. An upshot is that an uninstallation recipe can access files and directories created by a recipe before they are deleted. For example, here's an uninstallation recipe that simulates backing up a directory before it is deleted. It is designed to work with the ``mkdir`` recipe introduced earlier:: >>> write(sample_buildout, 'recipes', 'backup.py', ... """ ... import os, sys ... def backup_directory(name, options): ... path = options['path'] ... size = len(os.listdir(path)) ... sys.stdout.write("backing up directory %s of size %s\\n" ... % (path, size)) ... """) It must be registered with the ``zc.buildout.uninstall`` entry point. Notice how it is given the name ``mkdir`` to associate it with the ``mkdir`` recipe:: >>> write(sample_buildout, 'recipes', 'setup.py', ... """ ... from setuptools import setup ... entry_points = ( ... ''' ... [zc.buildout] ... mkdir = mkdir:Mkdir ... debug = debug:Debug ... service = service:Service ... ... [zc.buildout.uninstall] ... uninstall_service = service:uninstall_service ... mkdir = backup:backup_directory ... ''') ... setup(name="recipes", entry_points=entry_points, py_modules=['debug', 'environ', 'service', 'backup']) ... """) Now we can use it with a ``mkdir`` part:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = recipes ... parts = dir debug ... ... [dir] ... recipe = recipes:mkdir ... path = my_directory ... ... [debug] ... recipe = recipes:debug ... """) Run the buildout to install the part:: >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Uninstalling debug. Installing dir. dir: Creating directory my_directory Installing debug. recipe recipes:debug Now we remove the part from the configuration file:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = recipes ... parts = debug ... ... [debug] ... recipe = recipes:debug ... """) When the buildout is run the part is removed, and the uninstall recipe is run before the directory is deleted:: >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Uninstalling dir. Running uninstall recipe. backing up directory /sample-buildout/my_directory of size 0 Updating debug. recipe recipes:debug Now we will return the registration to normal for the benefit of the rest of the examples:: >>> write(sample_buildout, 'recipes', 'setup.py', ... """ ... from setuptools import setup ... entry_points = ( ... ''' ... [zc.buildout] ... mkdir = mkdir:Mkdir ... debug = debug:Debug ... ''') ... setup(name="recipes", entry_points=entry_points, py_modules=['debug', 'environ', 'service', 'backup']) ... """) Command-line usage ------------------ A number of arguments can be given on the buildout command line. The command usage is:: buildout [options and assignments] [command [command arguments]] The following options are supported: ``-h`` (or ``--help``) Print basic usage information. If this option is used, then all other options are ignored. ``-c`` filename The ``-c`` option can be used to specify a configuration file, rather than ``buildout.cfg`` in the current directory. ``-t`` socket_timeout Specify the socket timeout in seconds. ``-v`` Increment the verbosity by 10. The verbosity is used to adjust the logging level. The verbosity is subtracted from the numeric value of the log-level option specified in the configuration file. ``-q`` Decrement the verbosity by 10. ``-U`` Don't read user-default configuration. ``-o`` Run in off-line mode. This is equivalent to the assignment ``buildout:offline=true``. ``-O`` Run in non-off-line mode. This is equivalent to the assignment ``buildout:offline=false``. This is the default buildout mode. The ``-O`` option would normally be used to override a true offline setting in a configuration file. ``-n`` Run in newest mode. This is equivalent to the assignment ``buildout:newest=true``. With this setting, which is the default, buildout will try to find the newest versions of distributions available that satisfy its requirements. ``-N`` Run in non-newest mode. This is equivalent to the assignment ``buildout:newest=false``. With this setting, buildout will not seek new distributions if installed distributions satisfy its requirements. Assignments are of the form:: section_name:option_name=value or:: option_name=value which is equivalent to:: buildout:option_name=value Options and assignments can be given in any order. Here's an example:: >>> write(sample_buildout, 'other.cfg', ... """ ... [buildout] ... develop = recipes ... parts = debug ... installed = .other.cfg ... log-level = WARNING ... ... [debug] ... name = other ... recipe = recipes:debug ... """) Note that we used the installed buildout option to specify an alternate file to store information about installed parts:: >>> print_(system([buildout, '-c', 'other.cfg', 'debug:op1=foo', '-v']), end='') Develop: '/sample-buildout/recipes' Installing debug. name other op1 foo recipe recipes:debug Here we used the ``-c`` option to specify an alternate configuration file, and the ``-v`` option to increase the level of logging from the default, *WARNING*. Options can also be combined in the usual Unix way, as in:: >>> print_(system([buildout, '-vcother.cfg', 'debug:op1=foo']), end='') Develop: '/sample-buildout/recipes' Updating debug. name other op1 foo recipe recipes:debug Here we combined the ``-v`` and ``-c`` options with the configuration file name. Note that the ``-c`` option has to be last, because it takes an argument:: >>> os.remove(os.path.join(sample_buildout, 'other.cfg')) >>> os.remove(os.path.join(sample_buildout, '.other.cfg')) The most commonly used command is ``install``, and it takes a list of parts to install. If any parts are specified, only those parts are installed. To illustrate this, we'll update our configuration and run the buildout in the usual way:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = recipes ... parts = debug d1 d2 d3 ... ... [d1] ... recipe = recipes:mkdir ... path = d1 ... ... [d2] ... recipe = recipes:mkdir ... path = d2 ... ... [d3] ... recipe = recipes:mkdir ... path = d3 ... ... [debug] ... recipe = recipes:debug ... """) >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Uninstalling debug. Installing debug. recipe recipes:debug Installing d1. d1: Creating directory d1 Installing d2. d2: Creating directory d2 Installing d3. d3: Creating directory d3 >>> ls(sample_buildout) - .installed.cfg - b1.cfg - b2.cfg - base.cfg d bin - buildout.cfg d d1 d d2 d d3 d demo d develop-eggs d eggs d parts d recipes >>> cat(sample_buildout, '.installed.cfg') ... # doctest: +NORMALIZE_WHITESPACE [buildout] installed_develop_eggs = /sample-buildout/develop-eggs/recipes.egg-link parts = debug d1 d2 d3 [debug] __buildout_installed__ = __buildout_signature__ = recipes-PiIFiO8ny5yNZ1S3JfT0xg== recipe = recipes:debug [d1] __buildout_installed__ = /sample-buildout/d1 __buildout_signature__ = recipes-PiIFiO8ny5yNZ1S3JfT0xg== path = /sample-buildout/d1 recipe = recipes:mkdir [d2] __buildout_installed__ = /sample-buildout/d2 __buildout_signature__ = recipes-PiIFiO8ny5yNZ1S3JfT0xg== path = /sample-buildout/d2 recipe = recipes:mkdir [d3] __buildout_installed__ = /sample-buildout/d3 __buildout_signature__ = recipes-PiIFiO8ny5yNZ1S3JfT0xg== path = /sample-buildout/d3 recipe = recipes:mkdir Now we'll update our configuration file:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = recipes ... parts = debug d2 d3 d4 ... ... [d2] ... recipe = recipes:mkdir ... path = data2 ... ... [d3] ... recipe = recipes:mkdir ... path = data3 ... ... [d4] ... recipe = recipes:mkdir ... path = ${d2:path}-extra ... ... [debug] ... recipe = recipes:debug ... x = 1 ... """) and run the buildout specifying just ``d3`` and ``d4``:: >>> print_(system([buildout, 'install', 'd3', 'd4']), end='') Develop: '/sample-buildout/recipes' Uninstalling d3. Installing d3. d3: Creating directory data3 Installing d4. d4: Creating directory data2-extra >>> ls(sample_buildout) - .installed.cfg - b1.cfg - b2.cfg - base.cfg d bin - buildout.cfg d d1 d d2 d data2-extra d data3 d demo d develop-eggs d eggs d parts d recipes Only the ``d3`` and ``d4`` recipes ran. ``d3`` was removed and ``data3`` and ``data2-extra`` were created. The ``.installed.cfg`` is only updated for the recipes that ran:: >>> cat(sample_buildout, '.installed.cfg') ... # doctest: +NORMALIZE_WHITESPACE [buildout] installed_develop_eggs = /sample-buildout/develop-eggs/recipes.egg-link parts = debug d1 d2 d3 d4 [debug] __buildout_installed__ = __buildout_signature__ = recipes-PiIFiO8ny5yNZ1S3JfT0xg== recipe = recipes:debug [d1] __buildout_installed__ = /sample-buildout/d1 __buildout_signature__ = recipes-PiIFiO8ny5yNZ1S3JfT0xg== path = /sample-buildout/d1 recipe = recipes:mkdir [d2] __buildout_installed__ = /sample-buildout/d2 __buildout_signature__ = recipes-PiIFiO8ny5yNZ1S3JfT0xg== path = /sample-buildout/d2 recipe = recipes:mkdir [d3] __buildout_installed__ = /sample-buildout/data3 __buildout_signature__ = recipes-PiIFiO8ny5yNZ1S3JfT0xg== path = /sample-buildout/data3 recipe = recipes:mkdir [d4] __buildout_installed__ = /sample-buildout/data2-extra __buildout_signature__ = recipes-PiIFiO8ny5yNZ1S3JfT0xg== path = /sample-buildout/data2-extra recipe = recipes:mkdir Note that the installed data for ``debug``, ``d1``, and ``d2`` haven't changed, because we didn't install those parts, and that the ``d1`` and ``d2`` directories are still there. Now, if we run the buildout without the install command:: >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Uninstalling d2. Uninstalling d1. Uninstalling debug. Installing debug. recipe recipes:debug x 1 Installing d2. d2: Creating directory data2 Updating d3. Updating d4. We see the output of the debug recipe, and that ``data2`` was created. We also see that ``d1`` and ``d2`` have gone away:: >>> ls(sample_buildout) - .installed.cfg - b1.cfg - b2.cfg - base.cfg d bin - buildout.cfg d data2 d data2-extra d data3 d demo d develop-eggs d eggs d parts d recipes Alternate directory and file locations -------------------------------------- The buildout normally puts the ``bin``, ``eggs``, and ``parts`` directories in the directory in the directory containing the configuration file. You can provide alternate locations, and even names for these directories:: >>> alt = tmpdir('sample-alt') >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = recipes ... parts = ... develop-eggs-directory = %(developbasket)s ... eggs-directory = %(basket)s ... bin-directory = %(scripts)s ... parts-directory = %(work)s ... """ % dict( ... developbasket = os.path.join(alt, 'developbasket'), ... basket = os.path.join(alt, 'basket'), ... scripts = os.path.join(alt, 'scripts'), ... work = os.path.join(alt, 'work'), ... )) >>> print_(system(buildout), end='') Creating directory '/sample-alt/basket'. Creating directory '/sample-alt/scripts'. Creating directory '/sample-alt/work'. Creating directory '/sample-alt/developbasket'. Develop: '/sample-buildout/recipes' Uninstalling d4. Uninstalling d3. Uninstalling d2. Uninstalling debug. >>> ls(alt) d basket d developbasket d scripts d work >>> ls(alt, 'developbasket') - recipes.egg-link You can also specify an alternate buildout directory:: >>> rmdir(alt) >>> alt = tmpdir('sample-alt') >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... directory = %(alt)s ... develop = %(recipes)s ... parts = ... """ % dict( ... alt=alt, ... recipes=os.path.join(sample_buildout, 'recipes'), ... )) >>> print_(system(buildout), end='') Creating directory '/sample-alt/eggs'. Creating directory '/sample-alt/bin'. Creating directory '/sample-alt/parts'. Creating directory '/sample-alt/develop-eggs'. Develop: '/sample-buildout/recipes' >>> ls(alt) - .installed.cfg d bin d develop-eggs d eggs d parts >>> ls(alt, 'develop-eggs') - recipes.egg-link Logging control --------------- Three buildout options are used to control logging: ``log-level`` specifies the log level ``verbosity`` adjusts the log level ``log-format`` allows an alternate logging format to be specified We've already seen the log level and verbosity. Let's look at an example of changing the format:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = recipes ... parts = ... log-level = 25 ... verbosity = 5 ... log-format = %(levelname)s %(message)s ... """) Here, we've changed the format to include the log-level name, rather than the logger name. We've also illustrated, with a contrived example, that the log level can be a numeric value and that the verbosity can be specified in the configuration file. Because the verbosity is subtracted from the log level, we get a final log level of 20, which is the *INFO* level:: >>> print_(system(buildout), end='') INFO Develop: '/sample-buildout/recipes' Predefined buildout options --------------------------- Buildouts have a number of predefined options that recipes can use and that users can override in their configuration files. To see these, we'll run a minimal buildout configuration with a ``debug`` logging level. One of the features of ``debug`` logging is that the configuration database is shown:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... parts = ... """) >>> print_(system([buildout, '-vv']), end='') # doctest: +NORMALIZE_WHITESPACE Installing 'zc.buildout', 'wheel', 'pip', 'setuptools'. ... Configuration data: [buildout] allow-hosts = * allow-picked-versions = true allow-unknown-extras = false bin-directory = /sample-buildout/bin develop-eggs-directory = /sample-buildout/develop-eggs directory = /sample-buildout eggs-directory = /sample-buildout/eggs executable = python find-links = install-from-cache = false installed = /sample-buildout/.installed.cfg log-format = log-level = INFO newest = true offline = false parts = parts-directory = /sample-buildout/parts prefer-final = true python = buildout show-picked-versions = false socket-timeout = update-versions-file = use-dependency-links = true verbosity = 20 versions = versions [versions] zc.buildout = >=1.99 zc.recipe.egg = >=1.99 All of these options can be overridden by configuration files or by command-line assignments. We've discussed most of these options already, but let's review them and touch on some we haven't discussed: ``abi-tag-eggs`` Add an `ABI tag `_ to the directory name given by the ``eggs-directory`` option. This is useful when switching between python implementations when details of the implementation aren't reflected in egg names. It also has the side benefit of making eggs directories smaller, because eggs for different Python versions are in different directories. ``allow-hosts`` On some environments the links visited by ``zc.buildout`` can be forbidden by paranoid firewalls. These URLs might be in the chain of links visited by ``zc.buildout`` as defined by buildout's ``find-links`` option, or as defined by various eggs in their ``url``, ``download_url``, ``dependency_links`` metadata. The fact that ``package_index`` works like a spider and might visit links and go to other locations makes this even harder. The ``allow-hosts`` option provides a way to prevent this, and works exactly like the one provided in ``easy_install``. You can provide a list of allowed hosts, together with wildcards:: [buildout] ... allow-hosts = *.python.org example.com All URLs that do not match these hosts will not be visited. ``allow-picked-versions`` By default, the buildout will choose the best match for a given requirement if the requirement is not specified precisely (for instance, using the ``versions`` option. This behavior corresponds to the ``allow-picked-versions`` being set to its default value, ``true``. If ``allow-picked-versions`` is ``false``, instead of picking the best match, buildout will raise an error. This helps enforce repeatability. ``bin-directory`` The directory path where scripts are written. This can be a relative path, which is interpreted relative to the directory option. ``develop-eggs-directory`` The directory path where development egg links are created for software being created in the local project. This can be a relative path, which is interpreted relative to the directory option. ``directory`` The buildout directory. This is the base for other buildout file and directory locations, when relative locations are used. ``eggs-directory`` The directory path where downloaded eggs are put. It is common to share this directory across buildouts. This can be a relative path, which is interpreted relative to the directory option. .. warning:: Eggs in this directory should *never* be modified. ``find-links`` You can specify more locations to search for distributions using the ``find-links`` option. All locations specified will be searched for distributions along with the package index as described before. Locations can be URLs:: [buildout] ... find-links = http://download.zope.org/distribution/ They can also be directories on disk:: [buildout] ... find-links = /some/path Finally, they can also be direct paths to distributions:: [buildout] ... find-links = /some/path/someegg-1.0.0-py2.3.egg Any number of locations can be specified in the ``find-links`` option:: [buildout] ... find-links = http://download.zope.org/distribution/ /some/otherpath /some/path/someegg-1.0.0-py2.3.egg ``install-from-cache`` A download cache can be used as the basis of application source releases. In an application source release, we want to distribute an application that can be built without making any network accesses. In this case, we distribute a buildout along with a download cache, and tell the buildout to install from the download cache only, without making network accesses. The buildout ``install-from-cache`` option can be used to signal that packages should be installed only from the download cache. ``installed`` The file path where information about the results of the previous buildout run is written. This can be a relative path, which is interpreted relative to the directory option. This file provides an inventory of installed parts with information needed to decide which if any parts need to be uninstalled. ``log-format`` The format used for logging messages. ``log-level`` The log level before verbosity adjustment ``newest`` By default buildout and recipes will try to find the newest versions of distributions needed to satisfy requirements. This can be very time consuming, especially when incrementally working on setting up a buildout or working on a recipe. The buildout ``newest`` option can be used to to suppress this. If the ``newest`` option is set to false, then new distributions won't be sought if an installed distribution meets requirements. The ``newest`` option can also be set to false using the -N command-line option. See also the ``offline`` option. ``offline`` The ``offline`` option goes a bit further than the ``newest`` option. If the buildout ``offline`` option is given a value of ``true``, the buildout and recipes that are aware of the option will avoid doing network access. This is handy when running the buildout when not connected to the internet. It also makes buildouts run much faster. This option is typically set using the buildout ``-o`` option. ``parts`` A whitespace-separated list of parts to be installed. ``parts-directory`` A working directory that parts can used to store data. ``prefer-final`` Currently, when searching for new releases, the newest available release is used. This isn't usually ideal, as you may get a development release or alpha releases not ready to be widely used. You can request that final releases be preferred using the ``prefer-final`` option in the ``buildout`` section:: [buildout] ... prefer-final = true When the ``prefer-final`` option is set to ``true``, then when searching for new releases, final releases are preferred. If there are final releases that satisfy distribution requirements, then those releases are used even if newer non-final releases are available. The buildout ``prefer-final`` option can be used to override this behavior. In buildout version 2, final releases will be preferred by default. You will then need to use a ``false`` value for ``prefer-final`` to get the newest releases. ``use-dependency-links`` By default buildout will obey the setuptools ``dependency_links`` metadata when it looks for dependencies. This behavior can be controlled with the ``use-dependency-links`` buildout option:: [buildout] ... use-dependency-links = false The option defaults to ``true``. If you set it to ``false``, then dependency links are only looked for in the locations specified by ``find-links``. ``verbosity`` A log-level adjustment. Typically, this is set via the ``-q`` and ``-v`` command-line options. Creating new buildouts and bootstrapping ---------------------------------------- If ``zc.buildout`` is installed, you can use it to create a new buildout with its own local copies of ``zc.buildout`` and ``setuptools`` and with local buildout scripts:: >>> sample_bootstrapped = tmpdir('sample-bootstrapped') >>> print_(system([buildout, ... '-c' + os.path.join(sample_bootstrapped, 'setup.cfg'), ... 'init']), end='') Creating '/sample-bootstrapped/setup.cfg'. Creating directory '/sample-bootstrapped/eggs'. Creating directory '/sample-bootstrapped/bin'. Creating directory '/sample-bootstrapped/parts'. Creating directory '/sample-bootstrapped/develop-eggs'. Generated script '/sample-bootstrapped/bin/buildout'. Note that a basic ``setup.cfg`` was created for us. This is because we provided an ``init`` argument. By default, the generated ``setup.cfg`` is as minimal as can be:: >>> cat(sample_bootstrapped, 'setup.cfg') [buildout] parts = We also get other buildout artifacts:: >>> ls(sample_bootstrapped) d bin d develop-eggs d eggs d parts - setup.cfg >>> ls(sample_bootstrapped, 'bin') - buildout >>> _ = (ls(sample_bootstrapped, 'eggs'), ... ls(sample_bootstrapped, 'develop-eggs')) - packaging.egg-link - pip.egg-link - setuptools.egg-link - wheel.egg-link - zc.buildout.egg-link (We list both the ``eggs`` and ``develop-eggs`` directories because the buildout or setuptools egg could be installed in the ``develop-eggs`` directory if the original buildout had develop eggs for either buildout or setuptools.) Note that the buildout script was installed but not run. To run the buildout, we'd have to run the installed buildout script. If we have an existing buildout that already has a ``buildout.cfg``, we'll normally use the ``bootstrap`` command instead of ``init``. It will complain if there isn't a configuration file:: >>> sample_bootstrapped2 = tmpdir('sample-bootstrapped2') >>> print_(system([buildout, ... '-c' + os.path.join(sample_bootstrapped2, 'setup.cfg'), ... 'bootstrap']), end='') While: Initializing. Error: Couldn't open /sample-bootstrapped2/setup.cfg >>> write(sample_bootstrapped2, 'setup.cfg', ... """ ... [buildout] ... parts = ... """) >>> print_(system([buildout, ... '-c' + os.path.join(sample_bootstrapped2, 'setup.cfg'), ... 'bootstrap']), end='') Creating directory '/sample-bootstrapped2/eggs'. Creating directory '/sample-bootstrapped2/bin'. Creating directory '/sample-bootstrapped2/parts'. Creating directory '/sample-bootstrapped2/develop-eggs'. Generated script '/sample-bootstrapped2/bin/buildout'. Similarly, if there is a configuration file and we use the ``init`` command, we'll get an error that the configuration file already exists:: >>> print_(system([buildout, ... '-c' + os.path.join(sample_bootstrapped, 'setup.cfg'), ... 'init']), end='') While: Initializing. Error: '/sample-bootstrapped/setup.cfg' already exists. Initial eggs ------------ When using the ``init`` command, you can specify distribution requirements or paths to use:: >>> cd(sample_bootstrapped) >>> remove('setup.cfg') >>> print_(system([buildout, '-csetup.cfg', 'init', 'demo', 'other', './src']), end='') Creating '/sample-bootstrapped/setup.cfg'. Creating directory '/sample-bootstrapped/develop-eggs'. Getting distribution for 'zc.recipe.egg>=2.0.6'. Got zc.recipe.egg Installing py. Getting distribution for 'demo'. Got demo 0.3. Getting distribution for 'other'. Got other 1.0. Getting distribution for 'demoneeded'. Got demoneeded 1.1. Generated script '/sample-bootstrapped/bin/demo'. Generated script '/sample-bootstrapped/bin/distutilsscript'. Generated interpreter '/sample-bootstrapped/bin/py'. This causes a ``py`` part to be included that sets up a custom Python interpreter with the given requirements or paths:: >>> cat('setup.cfg') [buildout] parts = py [py] recipe = zc.recipe.egg interpreter = py eggs = demo other extra-paths = ./src Passing requirements or paths causes the buildout to be run as part of initialization. In the example above, we got a number of distributions installed and 2 scripts generated. The first, ``demo``, was defined by the ``demo`` project. The second, ``py`` was defined by the generated configuration. It's a *custom interpreter* that behaves like a standard Python interpreter, except that it includes the specified eggs and extra paths in its Python path. We specified a source directory that didn't exist. Buildout created it for us:: >>> ls('.') - .installed.cfg d bin d develop-eggs d eggs d parts - setup.cfg d src >>> uncd() .. Make sure it works if the dir is already there: >>> cd(sample_bootstrapped) >>> _ = system([buildout, '-csetup.cfg', 'buildout:parts=']) >>> remove('setup.cfg') >>> print_(system([buildout, '-csetup.cfg', 'init', 'demo', 'other', './src']), end='') Creating '/sample-bootstrapped/setup.cfg'. Creating directory '/sample-bootstrapped/develop-eggs'. Installing py. Generated script '/sample-bootstrapped/bin/demo'. Generated script '/sample-bootstrapped/bin/distutilsscript'. Generated interpreter '/sample-bootstrapped/bin/py'. .. cleanup >>> _ = system([buildout, '-csetup.cfg', 'buildout:parts=']) >>> uncd() Finding distributions --------------------- By default, buildout searches the Python Package Index when looking for distributions. You can, instead, specify your own index to search using the `index` option:: [buildout] ... index = https://index.example.com/ This index, or the default of https://pypi.org/simple/ if no index is specified, will always be searched for distributions unless running buildout with options that prevent searching for distributions. The latest version of the distribution that meets the requirements of the buildout will always be used. You can also specify more locations to search for distributions using the ``find-links`` option. See its description above. Controlling the installation database ------------------------------------- The buildout ``installed`` option is used to specify the file used to save information on installed parts. This option is initialized to ``.installed.cfg``, but it can be overridden in the configuration file or on the command line:: >>> write('buildout.cfg', ... """ ... [buildout] ... develop = recipes ... parts = debug ... ... [debug] ... recipe = recipes:debug ... """) >>> print_(system([buildout, 'buildout:installed=inst.cfg']), end='') Develop: '/sample-buildout/recipes' Installing debug. recipe recipes:debug >>> ls(sample_buildout) - b1.cfg - b2.cfg - base.cfg d bin - buildout.cfg d demo d develop-eggs d eggs - inst.cfg d parts d recipes The installation database can be disabled by supplying an empty buildout installed option:: >>> os.remove('inst.cfg') >>> print_(system([buildout, 'buildout:installed=']), end='') Develop: '/sample-buildout/recipes' Installing debug. recipe recipes:debug >>> ls(sample_buildout) - b1.cfg - b2.cfg - base.cfg d bin - buildout.cfg d demo d develop-eggs d eggs d parts d recipes Note that there will be no installation database if there are no parts:: >>> write('buildout.cfg', ... """ ... [buildout] ... parts = ... """) >>> print_(system([buildout, 'buildout:installed=inst.cfg']), end='') >>> ls(sample_buildout) - b1.cfg - b2.cfg - base.cfg d bin - buildout.cfg d demo d develop-eggs d eggs d parts d recipes Extensions ---------- A feature allows code to be loaded and run *after* configuration files have been read, but *before* the buildout has begun any processing. The intent is to allow special plugins such as ``urllib2`` request handlers to be loaded. To load an extension we use the ``extensions`` option and list one or more distribution requirements, on separate lines. The distributions named will be loaded and any ``zc.buildout.extension`` entry points found will be called with the buildout as an argument. When buildout finishes processing, any ``zc.buildout.unloadextension`` entry points found will be called with the buildout as an argument. Let's create a sample extension in our sample buildout created in the previous section:: >>> mkdir(sample_bootstrapped, 'demo') >>> write(sample_bootstrapped, 'demo', 'demo.py', ... """ ... import sys ... def ext(buildout): ... sys.stdout.write('%s %s\\n' % ('ext', sorted(buildout))) ... def unload(buildout): ... sys.stdout.write('%s %s\\n' % ('unload', sorted(buildout))) ... """) >>> write(sample_bootstrapped, 'demo', 'setup.py', ... """ ... from setuptools import setup ... ... setup( ... name = "demo", ... entry_points = { ... 'zc.buildout.extension': ['ext = demo:ext'], ... 'zc.buildout.unloadextension': ['ext = demo:unload'], ... }, ... ) ... """) Our extension just prints out the word 'demo', and lists the sections found in the buildout passed to it. We'll update our ``buildout.cfg`` to list the demo directory as a develop egg to be built:: >>> write(sample_bootstrapped, 'buildout.cfg', ... """ ... [buildout] ... develop = demo ... parts = ... """) >>> os.chdir(sample_bootstrapped) >>> print_(system(os.path.join(sample_bootstrapped, 'bin', 'buildout')), ... end='') Develop: '/sample-bootstrapped/demo' Now we can add the ``extensions`` option. We were a bit tricky and ran the buildout once with the demo develop egg defined but without the extension option. This is because extensions are loaded before the buildout creates develop eggs. We needed to use a separate buildout run to create the develop egg. Normally, when eggs are loaded from the network, we wouldn't need to do anything special. :: >>> write(sample_bootstrapped, 'buildout.cfg', ... """ ... [buildout] ... develop = demo ... extensions = demo ... parts = ... """) We see that our extension is loaded and executed:: >>> print_(system(os.path.join(sample_bootstrapped, 'bin', 'buildout')), ... end='') ext ['buildout', 'versions'] Develop: '/sample-bootstrapped/demo' unload ['buildout', 'versions'] .. >>> stop_server(server_url) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675670.0 zc_buildout-4.1.6/src/zc/buildout/tests/configparser.test0000644000076500000240000002640514773460426023257 0ustar00mauritsstaffSome tests of the basic config-file parser: First, an example that illustrates a well-formed configuration:: [s1] a = 1 [ s2 ] # a comment long = a b c l2 = a # not a comment # comment ; also a comment b c empty = c=1 b += 1 [s3]; comment x = a b .. -> text >>> from io import StringIO >>> from pprint import pprint >>> import zc.buildout.configparser >>> def parse(config, *args, **kw): ... return zc.buildout.configparser.parse(StringIO(config), ... 'test', *args, **kw) >>> pprint(parse(text)) {'s1': {'a': '1'}, 's2': {'b +': '1', 'c': '1', 'empty': '', 'l2': 'a\n\n\n# not a comment\n\n\nb\n\n c', 'long': 'a\nb\nc'}, 's3': {'x': 'a b'}} Here's an example with leading blank lines: >>> text = '\n\n[buildout]\nz=1\n\n' >>> pprint(parse(text)) {'buildout': {'z': '1'}} From email: "It fails when the first non-comment line after a section (even an otherwise empty section) is blank. For example:" [buildout] parts = hello versions = versions [versions] # Add any version pins here. [hello] recipe = collective.recipe.cmd on_install = true on_update = true cmds = echo Hello .. -> text >>> pprint(parse(text)) {'buildout': {'parts': 'hello', 'versions': 'versions'}, 'hello': {'cmds': 'echo Hello', 'on_install': 'true', 'on_update': 'true', 'recipe': 'collective.recipe.cmd'}, 'versions': {}} Sections headers can contain an optional arbitrary Python expression. When the expression evaluates to false the whole section is skipped. Several sections can have the same name with different expressions, enabling conditional exclusion of sections:: [s1: 2 + 2 == 4] # this expression is true [therefore "this section" _will_ be NOT skipped a = 1 [ s2 : 2 + 2 == 5 ] # comment: this expression is false, so this section will be ignored] long = a [ s2 : 41 + 1 == 42 ] # a comment: this expression is [true], so this section will be kept long = b [s3:2 in map(lambda i:i*2, [i for i in range(10)])] ;# Complex expressions are [possible!];, though they should not be (abused:) # this section will not be skipped long = c .. -> text >>> pprint(parse(text)) {'s1': {'a': '1'}, 's2': {'long': 'b'}, 's3': {'long': 'c'}} Title line optional trailing comments are separated by a hash '#' or semicolon ';' character. The expression is an arbitrary expression with one restriction: it cannot contain a literal hash '#' or semicolon ';' character: these need to be string-escaped. The comment can contain arbitrary characters, including brackets that are also used to mark the end of a section header and may be ambiguous to recognize in some cases. For example, valid sections lines include:: [ a ] a=1 [ b ] # [] b=1 [ c : True ] # ] c =1 [ d : True] # [] d=1 [ e ] # [] e = 1 [ f ] # ] f = 1 [g:2 in map(lambda i:i*2, ['''\x23\x3b)'''] + [i for i in range(10)] + list('\x23[]][\x3b\x23'))] # Complex #expressions; ][are [possible!] and can us escaped # and ; in literals g = 1 [ h : True ] ; ] h =1 [ i : True] ; [] i=1 [j:2 in map(lambda i:i*2, ['''\x23\x3b)'''] + [i for i in range(10)] + list('\x23[]][\x3b\x23'))] ; Complex #expressions; ][are [possible!] and can us escaped # and ; in literals j = 1 .. -> text >>> pprint(parse(text)) {'a': {'a': '1'}, 'b': {'b': '1'}, 'c': {'c': '1'}, 'd': {'d': '1'}, 'e': {'e': '1'}, 'f': {'f': '1'}, 'g': {'g': '1'}, 'h': {'h': '1'}, 'i': {'i': '1'}, 'j': {'j': '1'}} A title line optional trailing comment be separated by a hash or semicolon character. The following are valid semicolon-separated comments:: [ a ] ;semicolon comment are supported for lines without expressions ] a = 1 [ b ] ; [] b = 1 [ c ] ; ] c = 1 [ d ] ; [ d = 1 [ e: True ] ;semicolon comments are supported for lines with expressions ] e = 1 .. -> text >>> pprint(parse(text)) {'a': {'a': '1'}, 'b': {'b': '1'}, 'c': {'c': '1'}, 'd': {'d': '1'}, 'e': {'e': '1'}} The following sections with hash comment separators are valid too:: [ a ] #hash comment ] are supported for lines without expressions ] a = 1 [ b ] # [] b = 1 [ c ] # ] c = 1 [ d ] # [ d = 1 [ e: True ] #hash comments] are supported for lines with expressions ] e = 1 .. -> text >>> pprint(parse(text)) {'a': {'a': '1'}, 'b': {'b': '1'}, 'c': {'c': '1'}, 'd': {'d': '1'}, 'e': {'e': '1'}} However, explicit semicolon and hash characters are invalid in expressions and must be escaped or this triggers an error. In the rare case where a hash '#' or semicolon ';' would be needed in an expression literal, you can use the string-escaped representation of these characters: use '\x23' for hash '#' and '\x3b' for semicolon ';' to avoid evaluation errors. These expressions are valid and use escaped hash and semicolons in literals:: [a:2 in map(lambda i:i*2, ['''\x23\x3b)'''] + [i for i in range(10)] + list('\x23[]][\x3b\x23'))] # Complex #expressions; ][are [possible!] and can us escaped # and ; in literals a = 1 [b:2 in map(lambda i:i*2, ['''\x23\x3b)'''] + [i for i in range(10)] + list('\x23[]][\x3b\x23'))] ; Complex #expressions; ][are [possible!] and can us escaped # and ; in literals b = 1 .. -> text >>> pprint(parse(text)) {'a': {'a': '1'}, 'b': {'b': '1'}} And using unescaped semicolon and hash characters in expressions triggers an error:: [a:'#' in '#;'] # this is not a supported expression a = 1 .. -> text >>> try: parse(text) ... except zc.buildout.configparser.MissingSectionHeaderError: pass # success One of the typical usage of expression is to have buildout parts that are operating system or platform-specific. The configparser.parse function has an optional exp_globals argument. This is a callable returning a mapping of objects made available to the evaluation context of the expression. Here we add the platform and sys modules to the evaluation context, so we can access platform and sys modules functions and objects in our expressions :: [s1: str(platform.python_version_tuple()[0]) in ('2', '3',)] # this expression is true, the major versions of python are either 2 or 3 a = 1 [s2:sys.version[0] == '0'] # comment: this expression "is false", there no major version 0 of Python so this section will be ignored long = a [s2:len(platform.uname()) > 0] # a comment: this expression is likely always true, so this section will be kept long = b .. -> text >>> import platform, sys >>> globs = lambda: {'platform': platform, 'sys': sys} >>> pprint(parse(text, exp_globals=globs)) {'s1': {'a': '1'}, 's2': {'long': 'b'}} Some limited (but hopefully sane and sufficient) default modules and pre-computed common expressions available to an expression when the parser in called by buildout:: #imported modules [s1: sys and re and os and platform] # this expression is true: these modules are available a = 1 # major and minor python versions, yes even python 3.5 and 3.6 are there , prospectively # comment: this expression "is true" and not that long expression cannot span several lines [s2: any([python2, python3, python24 , python25 , python26 , python27 , python30 , python31 , python32 , python33 , python34 , python35 , python36, python37, python38, python39, python310, python311, python312, python313, python314, python315]) ] b = 1 # common python interpreter types [s3:cpython or pypy or jython or ironpython] # a comment: this expression is likely always true, so this section will be kept c = 1 # common operating systems [s4:linux or windows or cygwin or macosx or solaris or posix or True] d = 1 # common bitness and endianness [s5:bits32 or bits64 or little_endian or big_endian] e = 1 .. -> text >>> import zc.buildout.buildout >>> pprint(parse(text, zc.buildout.buildout._default_globals)) {'s1': {'a': '1'}, 's2': {'b': '1'}, 's3': {'c': '1'}, 's4': {'d': '1'}, 's5': {'e': '1'}} Preprocessing of implication and unicode cuteness:: [foo] => part1 part2 .. -> text >>> pprint(parse(text)) {'foo': {'': 'part1 part2'}} A recent addition is support for PEP 508 markers:: [section] # These are the values when no other section overrides them. a = 1 b = 1 [section: python_version < "2.6"] a = 26 [section: python_version < "4.0"] b = 40 .. -> text >>> pprint(parse(text)) {'section': {'a': '1', 'b': '40'}} You can use the platform. This is hard to test because the tests run on various platforms. But an unknown platform should never match:: [section] # These are the values when no other section overrides them. a = 1 [section: platform_system == "msdos"] a = 2 .. -> text >>> pprint(parse(text)) {'section': {'a': '1'}} You can make combinations:: [section] # These are the values when no other section overrides them. a = 1 b = 1 [section: python_version >= "2.0" and platform_system != "msdos"] a = 2 [section: python_version >= "2.0" or platform_system == "msdos"] b = 3 .. -> text >>> pprint(parse(text)) {'section': {'a': '2', 'b': '3'}} The old and new style conditional expressions can be used in the same file:: [section] # These are the values when no other section overrides them. a = 1 b = 1 [section: python_version >= "2.0"] a = 4 [section:linux or windows or cygwin or macosx or solaris or posix or True] b = 5 .. -> text >>> pprint(parse(text, zc.buildout.buildout._default_globals)) {'section': {'a': '4', 'b': '5'}} Issue 656: Because conditional sections mean we can have the same definition twice or more in a file, the parser needs to process them in the order they occur in the file, and handle conflicts appropriately. For example, a "+=" in a non-conditional section shouldn't be overwritten by a "+=" in a following conditional section. Instead the second "+=" should be appended to the first. Likewise, an "+=" in a section *should* be overwritten by an "=" in a following conditional section:: [section] # =, += a = a b c # =, -= b = a b c # =, = c = a b c # +=, = d += a b c # +=, += e += a b c # +=, -= f += a b c # -=, = g -= a b c # -=, += h -= a b c # -=, -= i -= a b c [section:True] a += d b -= b c = x y z d = x y z e += d e f f -= a g = d e f h += d e f i -= d .. -> text >>> pprint(parse(text)) {'section': {'a': 'a\nb\nc', 'a +': 'd', 'b': 'a\nb\nc', 'b -': 'b', 'c': 'x\ny\nz', 'd': 'x\ny\nz', 'e +': 'a\nb\nc\nd\ne\nf', 'f +': 'a\nb\nc', 'f -': 'a', 'g': 'd\ne\nf', 'h +': 'd\ne\nf', 'h -': 'a\nb\nc', 'i -': 'a\nb\nc\nd'}} ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675670.0 zc_buildout-4.1.6/src/zc/buildout/tests/debugging.txt0000644000076500000240000000556614773460426022375 0ustar00mauritsstaffDebugging buildouts =================== Buildouts can be pretty complex. When things go wrong, it isn't always obvious why. Errors can occur due to problems in user input or due to bugs in zc.buildout or recipes. When an error occurs, Python's post-mortem debugger can be used to inspect the state of the buildout or recipe code were there error occurred. To enable this, use the -D option to the buildout. Let's create a recipe that has a bug: >>> mkdir(sample_buildout, 'recipes') >>> write(sample_buildout, 'recipes', 'mkdir.py', ... """ ... import os, zc.buildout ... ... class Mkdir: ... ... def __init__(self, buildout, name, options): ... self.name, self.options = name, options ... options['path'] = os.path.join( ... buildout['buildout']['directory'], ... options['path'], ... ) ... ... def install(self): ... directory = self.options['directory'] ... os.mkdir(directory) ... return directory ... ... def update(self): ... pass ... """) >>> write(sample_buildout, 'recipes', 'setup.py', ... """ ... from setuptools import setup ... ... setup(name = "recipes", ... entry_points = {'zc.buildout': ['mkdir = mkdir:Mkdir']}, ... ) ... """) And create a buildout that uses it: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = recipes ... parts = data-dir ... ... [data-dir] ... recipe = recipes:mkdir ... path = mystuff ... """) If we run the buildout, we'll get an error: >>> print_(system(buildout, with_exit_code=True), end='') Develop: '/sample-buildout/recipes' Installing data-dir. While: Installing data-dir. Error: Missing option: data-dir:directory EXIT CODE: 1 If we want to debug the error, we can add the -D option. Here's we'll supply some input: >>> print_(system(buildout+" -D", """\ ... up ... p sorted(self.options.keys()) ... q ... """, with_exit_code=True), end='') # doctest: +ELLIPSIS Develop: '/sample-buildout/recipes' Installing data-dir. > /zc/buildout/buildout.py(925)__getitem__() -> raise MissingOption("Missing option: %s:%s" % (self.name, key)) (Pdb) > /sample-buildout/recipes/mkdir.py(14)install() -> directory = self.options['directory'] (Pdb) ['path', 'recipe'] ...While: Installing data-dir. Traceback (most recent call last): File "/zc/buildout/buildout.py", line 1352, in main ... File "/zc/buildout/buildout.py", line 925, in __getitem__ raise MissingOption("Missing option: %s:%s" % (self.name, key)) MissingOption: Missing option: data-dir:directory Starting pdb: EXIT CODE: 1 ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675670.0 zc_buildout-4.1.6/src/zc/buildout/tests/dependencylinks.txt0000644000076500000240000001356414773460426023616 0ustar00mauritsstaffDependency links ---------------- By default buildout will obey the setuptools dependency_links metadata when it looks for dependencies. This behavior can be controlled with the use-dependency-links buildout option. [buildout] ... use-dependency-links = false The option defaults to true. If you set it to false, then dependency links are only looked for in the locations specified by find-links. Let's see this feature in action. To begin, let's create a new egg repository. This repository uses the same sample eggs as the normal testing repository. >>> link_server2 = start_server(sample_eggs) Turn on logging on this server so that we can see when eggs are pulled from it. >>> _ = get(link_server2 + 'enable_server_logging') GET 200 /enable_server_logging Let's create a develop egg in our buildout that specifies dependency_links which point to the new server. >>> mkdir(sample_buildout, 'depdemo') >>> write(sample_buildout, 'depdemo', 'dependencydemo.py', ... 'import eggrecipedemoneeded') >>> write(sample_buildout, 'depdemo', 'setup.py', ... '''from setuptools import setup; setup( ... name='depdemo', py_modules=['dependencydemo'], ... install_requires = 'demoneeded', ... dependency_links = ['%s'], ... zip_safe=True, version='1') ... ''' % link_server2) Now let's configure the buildout to use the develop egg. >>> write(sample_buildout, 'buildout.cfg', ... ''' ... [buildout] ... develop = depdemo ... parts = eggs ... ... [eggs] ... recipe = zc.recipe.egg:eggs ... eggs = depdemo ... ''') Now we can run the buildout. >>> print_(system(buildout), end='') GET 200 / GET 200 /demoneeded-1.1.tar.gz Develop: '/sample-buildout/depdemo' Installing eggs. Getting distribution for 'demoneeded'. Got demoneeded 1.1. Notice that the egg was retrieved from the logging server. Now let's change the egg so that it doesn't specify dependency links. >>> write(sample_buildout, 'depdemo', 'setup.py', ... '''from setuptools import setup; setup( ... name='depdemo', py_modules=['dependencydemo'], ... install_requires = 'demoneeded', ... zip_safe=True, version='1') ... ''') Now we'll remove the existing dependency egg, and rerunning the buildout to see where the egg comes from this time. >>> from glob import glob >>> from os.path import join >>> def remove_demoneeded_egg(): ... for egg in glob(join(sample_buildout, 'eggs', 'demoneeded*.egg')): ... remove(sample_buildout, 'eggs', egg) >>> remove_demoneeded_egg() >>> print_(system(buildout), end='') # doctest: +ELLIPSIS Develop: '/sample-buildout/depdemo' Updating eggs. ... While: Updating eggs. Getting distribution for 'demoneeded'. Error: Couldn't find a distribution for 'demoneeded'. Now it can't find the dependency since neither the buildout configuration nor setup specifies where to look. Let's change things so that the buildout configuration specifies where to look for eggs. >>> write(sample_buildout, 'buildout.cfg', ... ''' ... [buildout] ... develop = depdemo ... parts = eggs ... find-links = %s ... ... [eggs] ... recipe = zc.recipe.egg:eggs ... eggs = depdemo ... ''' % link_server) >>> print_(system(buildout), end='') Develop: '/sample-buildout/depdemo' Installing eggs. Getting distribution for 'demoneeded'. Got demoneeded 1.1. This time the dependency egg was found on the server without logging configured. Now let's change things once again so that both buildout and setup specify different places to look for the dependency egg. >>> write(sample_buildout, 'depdemo', 'setup.py', ... '''from setuptools import setup; setup( ... name='depdemo', py_modules=['dependencydemo'], ... install_requires = 'demoneeded', ... dependency_links = ['%s'], ... zip_safe=True, version='1') ... ''' % link_server2) >>> remove_demoneeded_egg() >>> print_(system(buildout), end='') #doctest: +ELLIPSIS GET 200 /... Develop: '/sample-buildout/depdemo' Updating eggs. Getting distribution for 'demoneeded'. Got demoneeded 1.1. So when both setuptools and buildout specify places to search for eggs, the dependency_links takes precedence over find-links. There is a buildout option that you can specify to change this behavior. It is the use-dependency-links option. This option defaults to true. When you specify false for this option, buildout will ignore dependency_links and only look for eggs using find-links. Here is an example of using this option to disable dependency_links. >>> write(sample_buildout, 'buildout.cfg', ... ''' ... [buildout] ... develop = depdemo ... parts = eggs ... find-links = %s ... use-dependency-links = false ... ... [eggs] ... recipe = zc.recipe.egg:eggs ... eggs = depdemo ... ''' % link_server) >>> remove_demoneeded_egg() >>> print_(system(buildout), end='') Develop: '/sample-buildout/depdemo' Updating eggs. Getting distribution for 'demoneeded'. Got demoneeded 1.1. Notice that this time the egg isn't downloaded from the logging server. If we set the option to true, things return to the way they were before. The dependency's are looked for first in the logging server. >>> write(sample_buildout, 'buildout.cfg', ... ''' ... [buildout] ... develop = depdemo ... parts = eggs ... find-links = %s ... use-dependency-links = true ... ... [eggs] ... recipe = zc.recipe.egg:eggs ... eggs = depdemo ... ''' % link_server) >>> remove_demoneeded_egg() >>> print_(system(buildout), end='') #doctest: +ELLIPSIS GET 200 /... Develop: '/sample-buildout/depdemo' Updating eggs. Getting distribution for 'demoneeded'. Got demoneeded 1.1. ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675670.0 zc_buildout-4.1.6/src/zc/buildout/tests/download.txt0000644000076500000240000004252214773460426022242 0ustar00mauritsstaffUsing the download utility ========================== The ``zc.buildout.download`` module provides a download utility that handles the details of downloading files needed for a buildout run from the internet. It downloads files to the local file system, using the download cache if desired and optionally checking the downloaded files' MD5 checksum. We setup an HTTP server that provides a file we want to download: >>> server_data = tmpdir('sample_files') >>> write(server_data, 'foo.txt', 'This is a foo text.') >>> server_url = start_server(server_data) We also use a fresh directory for temporary files in order to make sure that all temporary files have been cleaned up in the end: >>> import tempfile >>> old_tempdir = tempfile.tempdir >>> tempfile.tempdir = tmpdir('tmp') Downloading without using the cache ----------------------------------- If no download cache should be used, the download utility is instantiated without any arguments: >>> from zc.buildout.download import Download >>> download = Download() >>> print_(download.cache_dir) None Downloading a file is achieved by calling the utility with the URL as an argument. A tuple is returned that consists of the path to the downloaded copy of the file and a boolean value indicating whether this is a temporary file meant to be cleaned up during the same buildout run: >>> path, is_temp = download(server_url+'foo.txt') >>> print_(path) /.../buildout-... >>> cat(path) This is a foo text. As we aren't using the download cache and haven't specified a target path either, the download has ended up in a temporary file: >>> is_temp True >>> import tempfile >>> path.startswith(tempfile.gettempdir()) True We are responsible for cleaning up temporary files behind us: >>> remove(path) When trying to access a file that doesn't exist, we'll get an exception: >>> try: download(server_url+'not-there') # doctest: +ELLIPSIS ... except: print_('download error') ... else: print_('woops') download error Downloading a local file doesn't produce a temporary file but simply returns the local file itself: >>> download(join(server_data, 'foo.txt')) ('/sample_files/foo.txt', False) We can also have the downloaded file's MD5 sum checked: >>> from hashlib import md5 >>> path, is_temp = download(server_url+'foo.txt', ... md5('This is a foo text.'.encode()).hexdigest()) >>> is_temp True >>> remove(path) >>> download(server_url+'foo.txt', ... md5('The wrong text.'.encode()).hexdigest()) Traceback (most recent call last): ChecksumError: MD5 checksum mismatch downloading 'http://localhost/foo.txt' The error message in the event of an MD5 checksum mismatch for a local file reads somewhat differently: >>> download(join(server_data, 'foo.txt'), ... md5('This is a foo text.'.encode()).hexdigest()) ('/sample_files/foo.txt', False) >>> download(join(server_data, 'foo.txt'), ... md5('The wrong text.'.encode()).hexdigest()) Traceback (most recent call last): ChecksumError: MD5 checksum mismatch for local resource at '/sample_files/foo.txt'. Finally, we can download the file to a specified place in the file system: >>> target_dir = tmpdir('download-target') >>> path, is_temp = download(server_url+'foo.txt', ... path=join(target_dir, 'downloaded.txt')) >>> print_(path) /download-target/downloaded.txt >>> cat(path) This is a foo text. >>> is_temp False Trying to download a file in offline mode will result in an error: >>> download = Download(cache=None, offline=True) >>> download(server_url+'foo.txt') Traceback (most recent call last): UserError: Couldn't download 'http://localhost/foo.txt' in offline mode. As an exception to this rule, file system paths and URLs in the ``file`` scheme will still work: >>> cat(download(join(server_data, 'foo.txt'))[0]) This is a foo text. >>> cat(download('file:' + join(server_data, 'foo.txt'))[0]) This is a foo text. >>> remove(path) Downloading using the download cache ------------------------------------ In order to make use of the download cache, we need to configure the download utility differently. To do this, we pass a directory path as the ``cache`` attribute upon instantiation: >>> cache = tmpdir('download-cache') >>> download = Download(cache=cache) >>> print_(download.cache_dir) /download-cache/ Simple usage ~~~~~~~~~~~~ When using the cache, a file will be stored in the cache directory when it is first downloaded. The file system path returned by the download utility points to the cached copy: >>> ls(cache) >>> path, is_temp = download(server_url+'foo.txt') >>> print_(path) /download-cache/foo.txt >>> cat(path) This is a foo text. >>> is_temp False Whenever the file is downloaded again, the cached copy is used. Let's change the file on the server to see this: >>> write(server_data, 'foo.txt', 'The wrong text.') >>> path, is_temp = download(server_url+'foo.txt') >>> print_(path) /download-cache/foo.txt >>> cat(path) This is a foo text. If we specify an MD5 checksum for a file that is already in the cache, the cached copy's checksum will be verified: >>> download(server_url+'foo.txt', md5('The wrong text.'.encode()).hexdigest()) Traceback (most recent call last): ChecksumError: MD5 checksum mismatch for cached download from 'http://localhost/foo.txt' at '/download-cache/foo.txt' Trying to access another file at a different URL which has the same base name will result in the cached copy being used: >>> mkdir(server_data, 'other') >>> write(server_data, 'other', 'foo.txt', 'The wrong text.') >>> path, is_temp = download(server_url+'other/foo.txt') >>> print_(path) /download-cache/foo.txt >>> cat(path) This is a foo text. Given a target path for the download, the utility will provide a copy of the file at that location both when first downloading the file and when using a cached copy: >>> remove(cache, 'foo.txt') >>> ls(cache) >>> write(server_data, 'foo.txt', 'This is a foo text.') >>> path, is_temp = download(server_url+'foo.txt', ... path=join(target_dir, 'downloaded.txt')) >>> print_(path) /download-target/downloaded.txt >>> cat(path) This is a foo text. >>> is_temp False >>> ls(cache) - foo.txt >>> remove(path) >>> write(server_data, 'foo.txt', 'The wrong text.') >>> path, is_temp = download(server_url+'foo.txt', ... path=join(target_dir, 'downloaded.txt')) >>> print_(path) /download-target/downloaded.txt >>> cat(path) This is a foo text. >>> is_temp False In offline mode, downloads from any URL will be successful if the file is found in the cache: >>> download = Download(cache=cache, offline=True) >>> cat(download(server_url+'foo.txt')[0]) This is a foo text. Local resources will be cached just like any others since download caches are sometimes used to create source distributions: >>> remove(cache, 'foo.txt') >>> ls(cache) >>> write(server_data, 'foo.txt', 'This is a foo text.') >>> download = Download(cache=cache) >>> cat(download('file:' + join(server_data, 'foo.txt'), path=path)[0]) This is a foo text. >>> ls(cache) - foo.txt >>> remove(cache, 'foo.txt') >>> cat(download(join(server_data, 'foo.txt'), path=path)[0]) This is a foo text. >>> ls(cache) - foo.txt >>> remove(cache, 'foo.txt') However, resources with checksum mismatches will not be copied to the cache: >>> download(server_url+'foo.txt', md5('The wrong text.'.encode()).hexdigest()) Traceback (most recent call last): ChecksumError: MD5 checksum mismatch downloading 'http://localhost/foo.txt' >>> ls(cache) >>> remove(path) If the file is completely missing it should notify the user of the error: >>> download(server_url+'bar.txt') # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS Traceback (most recent call last): ... UserError: Error downloading extends for URL http://localhost/bar.txt: ...404... >>> ls(cache) Finally, let's see what happens if the download cache to be used doesn't exist as a directory in the file system yet: >>> Download(cache=join(cache, 'non-existent'))(server_url+'foo.txt') Traceback (most recent call last): UserError: The directory: '/download-cache/non-existent' to be used as a download cache doesn't exist. Using namespace sub-directories of the download cache ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ It is common to store cached copies of downloaded files within sub-directories of the download cache to keep some degree of order. For example, zc.buildout stores downloaded distributions in a sub-directory named "dist". Those sub-directories are also known as namespaces. So far, we haven't specified any namespaces to use, so the download utility stored files directly inside the download cache. Let's use a namespace "test" instead: >>> download = Download(cache=cache, namespace='test') >>> print_(download.cache_dir) /download-cache/test The namespace sub-directory hasn't been created yet: >>> ls(cache) Downloading a file now creates the namespace sub-directory and places a copy of the file inside it: >>> path, is_temp = download(server_url+'foo.txt') >>> print_(path) /download-cache/test/foo.txt >>> ls(cache) d test >>> ls(cache, 'test') - foo.txt >>> cat(path) This is a foo text. >>> is_temp False The next time we want to download that file, the copy from inside the cache namespace is used. To see this clearly, we put a file with the same name but different content both on the server and in the cache's root directory: >>> write(server_data, 'foo.txt', 'The wrong text.') >>> write(cache, 'foo.txt', 'The wrong text.') >>> path, is_temp = download(server_url+'foo.txt') >>> print_(path) /download-cache/test/foo.txt >>> cat(path) This is a foo text. >>> rmdir(cache, 'test') >>> remove(cache, 'foo.txt') >>> write(server_data, 'foo.txt', 'This is a foo text.') Using a hash of the URL as the filename in the cache ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ So far, the base name of the downloaded file read from the URL has been used for the name of the cached copy of the file. This may not be desirable in some cases, for example when downloading files from different locations that have the same base name due to some naming convention, or if the file content depends on URL parameters. In such cases, an MD5 hash of the complete URL may be used as the filename in the cache: >>> download = Download(cache=cache, hash_name=True) >>> path, is_temp = download(server_url+'foo.txt') >>> print_(path) /download-cache/09f5793fcdc1716727f72d49519c688d >>> cat(path) This is a foo text. >>> ls(cache) - 09f5793fcdc1716727f72d49519c688d The path was printed just to illustrate matters; we cannot know the real checksum since we don't know which port the server happens to listen at when the test is run, so we don't actually know the full URL of the file. Let's check that the checksum actually belongs to the particular URL used: >>> (path.lower() == ... join(cache, md5((server_url+'foo.txt').encode()).hexdigest()).lower()) True The cached copy is used when downloading the file again: >>> write(server_data, 'foo.txt', 'The wrong text.') >>> (path, is_temp) == download(server_url+'foo.txt') True >>> cat(path) This is a foo text. >>> ls(cache) - 09f5793fcdc1716727f72d49519c688d If we change the URL, even in such a way that it keeps the base name of the file the same, the file will be downloaded again this time and put in the cache under a different name: >>> path2, is_temp = download(server_url+'other/foo.txt') >>> print_(path2) /download-cache/537b6d73267f8f4447586989af8c470e >>> path == path2 False >>> (path2.lower() == ... join(cache, md5((server_url+'other/foo.txt').encode()).hexdigest() ... ).lower()) True >>> cat(path) This is a foo text. >>> cat(path2) The wrong text. >>> ls(cache) - 09f5793fcdc1716727f72d49519c688d - 537b6d73267f8f4447586989af8c470e >>> remove(path) >>> remove(path2) >>> write(server_data, 'foo.txt', 'This is a foo text.') Using the cache purely as a fall-back ------------------------------------- Sometimes it is desirable to try downloading a file from the net if at all possible, and use the cache purely as a fall-back option when a server is down or if we are in offline mode. This mode is only in effect if a download cache is configured in the first place: >>> download = Download(cache=cache, fallback=True) >>> print_(download.cache_dir) /download-cache/ A downloaded file will be cached: >>> ls(cache) >>> path, is_temp = download(server_url+'foo.txt') >>> ls(cache) - foo.txt >>> cat(cache, 'foo.txt') This is a foo text. >>> is_temp False If the file cannot be served, the cached copy will be used: >>> remove(server_data, 'foo.txt') >>> try: Download()(server_url+'foo.txt') # doctest: +ELLIPSIS ... except: print_('download error') ... else: print_('woops') download error >>> path, is_temp = download(server_url+'foo.txt') >>> cat(path) This is a foo text. >>> is_temp False Similarly, if the file is served but we're in offline mode, we'll fall back to using the cache: >>> write(server_data, 'foo.txt', 'The wrong text.') >>> get(server_url+'foo.txt') 'The wrong text.' >>> offline_download = Download(cache=cache, offline=True, fallback=True) >>> path, is_temp = offline_download(server_url+'foo.txt') >>> print_(path) /download-cache/foo.txt >>> cat(path) This is a foo text. >>> is_temp False However, when downloading the file normally with the cache being used in fall-back mode, the file will be downloaded from the net and the cached copy will be replaced with the new content: >>> cat(download(server_url+'foo.txt')[0]) The wrong text. >>> cat(cache, 'foo.txt') The wrong text. When trying to download a resource whose checksum does not match, the cached copy will neither be used nor overwritten: >>> write(server_data, 'foo.txt', 'This is a foo text.') >>> download(server_url+'foo.txt', md5('The wrong text.'.encode()).hexdigest()) Traceback (most recent call last): ChecksumError: MD5 checksum mismatch downloading 'http://localhost/foo.txt' >>> cat(cache, 'foo.txt') The wrong text. Configuring the download utility from buildout options ------------------------------------------------------ The configuration options explained so far derive from the build logic implemented by the calling code. Other options configure the download utility for use in a particular project or buildout run; they are read from the ``buildout`` configuration section. The latter can be passed directly as the first argument to the download utility's constructor. The location of the download cache is specified by the ``download-cache`` option: >>> download = Download({'download-cache': cache}, namespace='cmmi') >>> print_(download.cache_dir) /download-cache/cmmi If the ``download-cache`` option specifies a relative path, it is understood relative to the current working directory, or to the buildout directory if that is given: >>> download = Download({'download-cache': 'relative-cache'}) >>> print_(download.cache_dir) /sample-buildout/relative-cache/ >>> download = Download({'directory': join(sample_buildout, 'root'), ... 'download-cache': 'relative-cache'}) >>> print_(download.cache_dir) /sample-buildout/root/relative-cache/ Keyword parameters take precedence over the corresponding options: >>> download = Download({'download-cache': cache}, cache=None) >>> print_(download.cache_dir) None Whether to assume offline mode can be inferred from either the ``offline`` or the ``install-from-cache`` option. As usual with zc.buildout, these options must assume one of the values 'true' and 'false': >>> download = Download({'offline': 'true'}) >>> download.offline True >>> download = Download({'offline': 'false'}) >>> download.offline False >>> download = Download({'install-from-cache': 'true'}) >>> download.offline True >>> download = Download({'install-from-cache': 'false'}) >>> download.offline False These two options are combined using logical 'or': >>> download = Download({'offline': 'true', 'install-from-cache': 'false'}) >>> download.offline True >>> download = Download({'offline': 'false', 'install-from-cache': 'true'}) >>> download.offline True The ``offline`` keyword parameter takes precedence over both the ``offline`` and ``install-from-cache`` options: >>> download = Download({'offline': 'true'}, offline=False) >>> download.offline False >>> download = Download({'install-from-cache': 'false'}, offline=True) >>> download.offline True Regressions ----------- MD5 checksum calculation needs to be reliable on all supported systems, which requires text files to be treated as binary to avoid implicit line-ending conversions: >>> text = 'First line of text.\r\nSecond line.\r\n' >>> f = open(join(server_data, 'foo.txt'), 'wb') >>> _ = f.write(text.encode()) >>> f.close() >>> path, is_temp = Download()(server_url+'foo.txt', ... md5(text.encode()).hexdigest()) >>> remove(path) When "downloading" a directory given by file-system path or ``file:`` URL and using a download cache at the same time, the cached directory wasn't handled correctly. Consequently, the cache was defeated and an attempt to cache the directory a second time broke. This is how it should work: >>> download = Download(cache=cache) >>> dirpath = join(server_data, 'some_directory') >>> mkdir(dirpath) >>> dest, _ = download(dirpath) If we now modify the source tree, the second download will produce the original one from the cache: >>> mkdir(join(dirpath, 'foo')) >>> ls(dirpath) d foo >>> dest, _ = download(dirpath) >>> ls(dest) Clean up -------- We should have cleaned up all temporary files created by downloading things: >>> ls(tempfile.tempdir) Reset the global temporary directory: >>> tempfile.tempdir = old_tempdir ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675670.0 zc_buildout-4.1.6/src/zc/buildout/tests/downloadcache.txt0000644000076500000240000001636214773460426023231 0ustar00mauritsstaffUsing a download cache ====================== Normally, when distributions are installed, if any processing is needed, they are downloaded from the internet to a temporary directory and then installed from there. A download cache can be used to avoid the download step. This can be useful to reduce network access and to create source distributions of an entire buildout. The buildout download-cache option can be used to specify a directory to be used as a download cache. In this example, we'll create a directory to hold the cache: >>> cache = tmpdir('cache') And set up a buildout that downloads some eggs: >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = eggs ... download-cache = %(cache)s ... find-links = %(link_server)s ... ... [eggs] ... recipe = zc.recipe.egg ... eggs = demo ==0.2 ... ''' % globals()) We specified a link server that has some distributions available for download: >>> print_(get(link_server), end='') bigdemo-0.1-py3-none-any.whl
demo-0.1-py3-none-any.whl
demo-0.2-py3-none-any.whl
demo-0.3-py3-none-any.whl
demo-0.4rc1-py3-none-any.whl
demoneeded-1.0.tar.gz
demoneeded-1.1.tar.gz
demoneeded-1.2rc1.tar.gz
du_zipped-1.0-pyN.N.egg
extdemo-1.4.tar.gz
index/
mixedcase-0.5.tar.gz
other-1.0-py3-none-any.whl
We'll enable logging on the link server so we can see what's going on: >>> _ = get(link_server+'enable_server_logging') GET 200 /enable_server_logging We also specified a download cache. If we run the buildout, we'll see the eggs installed from the link server as usual: >>> print_(system(buildout), end='') GET ... GET 200 /demo-0.2-py3-none-any.whl GET 200 /demoneeded-1.1.tar.gz Installing eggs. Getting distribution for 'demo==0.2'. Got demo 0.2. Getting distribution for 'demoneeded'. Got demoneeded 1.1. Generated script '/sample-buildout/bin/demo'. We'll also get the download cache populated. The buildout doesn't put files in the cache directly. It creates an intermediate directory, dist: >>> ls(cache) d dist >>> ls(cache, 'dist') - demo-0.2-py3-none-any.whl - demoneeded-1.1.tar.gz If we remove the installed eggs from eggs directory and re-run the buildout: >>> import os >>> for f in os.listdir('eggs'): ... if f.startswith('demo'): ... remove('eggs', f) >>> print_(system(buildout), end='') GET ... Updating eggs. Getting distribution for 'demo==0.2'. Got demo 0.2. Getting distribution for 'demoneeded'. Got demoneeded 1.1. We see that the distributions aren't downloaded, because they're downloaded from the cache. Installing solely from a download cache --------------------------------------- A download cache can be used as the basis of application source releases. In an application source release, we want to distribute an application that can be built without making any network accesses. In this case, we distribute a buildout with download cache and tell the buildout to install from the download cache only, without making network accesses. The buildout install-from-cache option can be used to signal that packages should be installed only from the download cache. Let's remove our installed eggs and run the buildout with the install-from-cache option set to true: >>> for f in os.listdir('eggs'): ... if f.startswith('demo'): ... remove('eggs', f) >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = eggs ... download-cache = %(cache)s ... install-from-cache = true ... find-links = %(link_server)s ... ... [eggs] ... recipe = zc.recipe.egg ... eggs = demo ... ''' % globals()) >>> print_(system(buildout), end='') Uninstalling eggs. Installing eggs. Getting distribution for 'demo'. Got demo 0.2. Getting distribution for 'demoneeded'. Got demoneeded 1.1. Generated script '/sample-buildout/bin/demo'. Auto-creation of download cache directory ----------------------------------------- With zc.buildout version 2.2.2 or higher the cache directory is automatically created:: >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = ... download-cache = %(cache)s/newdir ... ''' % globals()) >>> print_(system(buildout), end='') Creating directory '/cache/newdir'. Uninstalling eggs. >>> ls(cache) d dist d newdir Using relative paths -------------------- You can use a relative path for ``download-cache`` (the same logic is applied to ``eggs-directory`` and to ``extends-cache`` too) and in such case it is considered relative to the location of the configuration file that sets its value. As an example, we create a ``base.cfg`` configuration in a different directory:: >>> basedir = tmpdir('basecfg') >>> write(basedir, 'base.cfg', ... ''' ... [buildout] ... download-cache = cache ... ''') and a ``buildout.cfg`` that extends from there:: >>> write('buildout.cfg', ... ''' ... [buildout] ... extends = %(basedir)s/base.cfg ... parts = ... ''' % globals()) >>> dummy = system(buildout) >>> ls(basedir) - base.cfg d cache Of course this cannot be used when the base configuration is not on the local filesystem because it wouldn't make any sense having a remote cache:: >>> server_data = tmpdir('server_data') >>> server_url = start_server(server_data) >>> cd(sample_buildout) >>> write(server_data, 'base.cfg', """\ ... [buildout] ... download-cache = cache ... """) >>> write('buildout.cfg', ... ''' ... [buildout] ... extends = %(server_url)s/base.cfg ... parts = ... ''' % globals()) >>> print_(system(buildout), end='') # doctest: +ELLIPSIS While: Initializing. Error: Setting "download-cache" to a non absolute location ("cache") within a remote configuration file... Though, you can create the ``download-cache`` within a nested directory, so that you can group all your generated directories (like ``eggs-directory`` or ``extends-cache`` too) within a single directory: >>> test_nested = tmpdir('test_nested') >>> cd(test_nested) >>> write('buildout.cfg', ... ''' ... [buildout] ... download-cache = ${buildout:directory}/var/cache ... eggs-directory = ${buildout:directory}/var/eggs ... parts-directory = ${buildout:directory}/var/parts ... develop-eggs-directory = ${buildout:directory}/var/develop-eggs ... ''') >>> dummy = system(buildout) >>> ls(test_nested) d bin - buildout.cfg d var >>> ls(os.path.join(test_nested, 'var')) d cache d develop-eggs d eggs d parts ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675670.0 zc_buildout-4.1.6/src/zc/buildout/tests/easy_install.txt0000644000076500000240000015011714773460426023122 0ustar00mauritsstaffPython API for egg and script installation ========================================== The easy_install module provides some functions to provide support for egg and script installation. It provides functionality at the python level that is similar to easy_install, with a few exceptions: - By default, we look for new packages *and* the packages that they depend on. This is somewhat like (and uses) the --upgrade option of easy_install, except that we also upgrade required packages. - If the highest-revision package satisfying a specification is already present, then we don't try to get another one. This saves a lot of search time in the common case that packages are pegged to specific versions. - If there is a develop egg that satisfies a requirement, we don't look for additional distributions. We always give preference to develop eggs. - Distutils options for building extensions can be passed. Distribution installation ------------------------- The easy_install module provides a function, install, for installing one or more packages and their dependencies. The install function takes 2 positional arguments: - An iterable of setuptools requirement strings for the distributions to be installed, and - A destination directory to install to and to satisfy requirements from. The destination directory can be None, in which case, no new distributions are downloaded and there will be an error if the needed distributions can't be found among those already installed. It supports a number of optional keyword arguments: links A sequence of URLs, file names, or directories to look for links to distributions. index The URL of an index server, or almost any other valid URL. :) If not specified, the Python Package Index, https://pypi.org/simple/, is used. You can specify an alternate index with this option. If you use the links option and if the links point to the needed distributions, then the index can be anything and will be largely ignored. In the examples, here, we'll just point to an empty directory on our link server. This will make our examples run a little bit faster. path A list of additional directories to search for locally-installed distributions. working_set An existing working set to be augmented with additional distributions, if necessary to satisfy requirements. This allows you to call install multiple times, if necessary, to gather multiple sets of requirements. newest A boolean value indicating whether to search for new distributions when already-installed distributions meet the requirement. When this is true, the default, and when the destination directory is not None, then the install function will search for the newest distributions that satisfy the requirements. versions A dictionary mapping project names to version numbers to be used when selecting distributions. This can be used to specify a set of distribution versions independent of other requirements. use_dependency_links A flag indicating whether to search for dependencies using the setup dependency_links metadata or not. If true, links are searched for using dependency_links in preference to other locations. Defaults to true. relative_paths Adjust egg paths so they are relative to the script path. This allows scripts to work when scripts and eggs are moved, as long as they are both moved in the same way. allow_unknown_extras Install the requirements, even if one of them specifies an extra not provided by the distribution. The install method returns a working set containing the distributions needed to meet the given requirements. We have a link server that has a number of eggs: >>> print_(get(link_server), end='') bigdemo-0.1-py3-none-any.whl
demo-0.1-py3-none-any.whl
demo-0.2-py3-none-any.whl
demo-0.3-py3-none-any.whl
demo-0.4rc1-py3-none-any.whl
demoneeded-1.0.tar.gz
demoneeded-1.1.tar.gz
demoneeded-1.2rc1.tar.gz
du_zipped-1.0-pyN.N.egg
extdemo-1.4.tar.gz
index/
mixedcase-0.5.tar.gz
other-1.0-py3-none-any.whl
Let's make a directory and install the demo egg to it, using the demo: >>> dest = tmpdir('sample-install') >>> import zc.buildout.easy_install >>> ws = zc.buildout.easy_install.install( ... ['demo==0.2'], dest, ... links=[link_server], index=link_server+'index/') We requested version 0.2 of the demo distribution to be installed into the destination server. We specified that we should search for links on the link server and that we should use the (empty) link server index directory as a package index. The working set contains the distributions we retrieved. >>> for dist in ws: ... print_(dist) demoneeded 1.1 demo 0.2 We got demoneeded because it was a dependency of demo. And the actual eggs were added to the eggs directory. >>> ls(dest) d demo-0.2-py2.4.egg d demoneeded-1.1-py2.4.egg If we remove the version restriction on demo, but specify a false value for newest, no new distributions will be installed: >>> ws = zc.buildout.easy_install.install( ... ['demo'], dest, links=[link_server], index=link_server+'index/', ... newest=False) >>> ls(dest) d demo-0.2-py2.4.egg d demoneeded-1.1-py2.4.egg If we leave off the newest option, we'll get an update for demo: >>> ws = zc.buildout.easy_install.install( ... ['demo'], dest, links=[link_server], index=link_server+'index/') >>> ls(dest) d demo-0.2-py2.4.egg d demo-0.3-py2.4.egg d demoneeded-1.1-py2.4.egg Note that we didn't get the newest versions available. There were release candidates for newer versions of both packages. By default, final releases are preferred. We can change this behavior using the prefer_final function: >>> zc.buildout.easy_install.prefer_final(False) True The old setting is returned. >>> ws = zc.buildout.easy_install.install( ... ['demo'], dest, links=[link_server], index=link_server+'index/') >>> for dist in ws: ... print_(dist) demoneeded 1.2rc1 demo 0.4rc1 >>> ls(dest) d demo-0.2-py2.4.egg d demo-0.3-py2.4.egg d demo-0.4rc1-py2.4.egg d demoneeded-1.1-py2.4.egg d demoneeded-1.2rc1-py2.4.egg Let's put the setting back to the default. >>> zc.buildout.easy_install.prefer_final(True) False We can supply additional distributions. We can also supply specifications for distributions that would normally be found via dependencies. We might do this to specify a specific version. >>> ws = zc.buildout.easy_install.install( ... ['demo', 'other', 'demoneeded==1.0'], dest, ... links=[link_server], index=link_server+'index/') >>> for dist in ws: ... print_(dist) demoneeded 1.0 other 1.0 demo 0.3 >>> ls(dest) d demo-0.2-py2.4.egg d demo-0.3-py2.4.egg d demo-0.4rc1-py2.4.egg d demoneeded-1.0-py2.4.egg d demoneeded-1.1-py2.4.egg d demoneeded-1.2rc1-py2.4.egg d other-1.0-py2.4.egg >>> rmdir(dest) Unknown extras -------------- Attempting to install a requirement with an extra it doesn't provide is an error. >>> ws = zc.buildout.easy_install.install( ... ['demo[unknown_extra]'], dest, links=[link_server], ... index=link_server+'index/') Traceback (most recent call last): ... UserError: Couldn't find the required extra... We can pass the ``allow_unknown_extras`` argument to force the installation to proceed. >>> ws = zc.buildout.easy_install.install( ... ['demo[unknown_extra]'], dest, links=[link_server], ... index=link_server+'index/', ... allow_unknown_extras=True) >>> ls(dest) d demo-0.3-py2.4.egg >>> rmdir(dest) Case issues ----------- Let's install an egg with case naming issues. Specifically, the sdist file is lower case while the name of the package is uppercase. Let's enable server logging to check that the lower case file is downloaded. >>> _ = get(link_server + 'enable_server_logging') GET 200 /enable_server_logging >>> ws = zc.buildout.easy_install.install( ... ['MIXEDCASE'], dest, ... links=[link_server], index=link_server+'index/') GET 404 /index/mixedcase/ GET 404 /index/MIXEDCASE/ GET 200 /mixedcase-0.5.tar.gz GET 200 /demoneeded-1.1.tar.gz Let's check that the uppercase dist is installed. setuptools 75.8.1+ reports the name in all lowercase, earlier versions showed it in uppercase. So we compare lowercase. >>> for dist in ws: ... print_(str(dist).lower()) demoneeded 1.1 mixedcase 0.5 >>> ls(dest, lowercase_and_sort_output=True) d demoneeded-1.1-py2.4.egg d mixedcase-0.5-pyN.N.egg And cleanup. >>> _ = get(link_server + 'disable_server_logging') >>> rmdir(dest) Specifying version information independent of requirements ---------------------------------------------------------- Sometimes it's useful to specify version information independent of normal requirements specifications. For example, a buildout may need to lock down a set of versions, without having to put put version numbers in setup files or part definitions. If a dictionary is passed to the install function, mapping project names to version numbers, then the versions numbers will be used. >>> ws = zc.buildout.easy_install.install( ... ['demo'], dest, links=[link_server], index=link_server+'index/', ... versions = dict(demo='0.2', demoneeded='1.0')) >>> [d.version for d in ws] ['1.0', '0.2'] In this example, we specified a version for demoneeded, even though we didn't define a requirement for it. The versions specified apply to dependencies as well as the specified requirements. If we specify a version that's incompatible with a requirement, then we'll get an error: >>> from zope.testing.loggingsupport import InstalledHandler >>> handler = InstalledHandler('zc.buildout.easy_install') >>> import logging >>> logging.getLogger('zc.buildout.easy_install').propagate = False >>> ws = zc.buildout.easy_install.install( ... ['demo >0.2'], dest, links=[link_server], ... index=link_server+'index/', ... versions = dict(demo='0.2', demoneeded='1.0')) Traceback (most recent call last): ... IncompatibleConstraintError: The requirement ('demo>0.2') is not allowed by your [versions] constraint (0.2) >>> print_(handler) zc.buildout.easy_install DEBUG Installing 'demo >0.2'. zc.buildout.easy_install INFO Version and requirements information containing demo: [versions] constraint on demo: 0.2 Base installation request: 'demo >0.2' >>> handler.clear() If no versions are specified, a debugging message will be output reporting that a version was picked automatically: >>> ws = zc.buildout.easy_install.install( ... ['demo'], dest, links=[link_server], index=link_server+'index/', ... ) >>> print_(handler) # doctest: +ELLIPSIS zc.buildout.easy_install DEBUG Installing 'demo'. zc.buildout.easy_install INFO Getting distribution for 'demo'. zc.buildout.easy_install DEBUG Fetching demo 0.3 from: http://.../demo-0.3-py3-none-any.whl zc.buildout.easy_install INFO Got demo 0.3. zc.buildout.easy_install DEBUG Picked: demo = 0.3 zc.buildout.easy_install DEBUG Getting required 'demoneeded' zc.buildout.easy_install DEBUG required by demo 0.3. zc.buildout.easy_install INFO Getting distribution for 'demoneeded'. zc.buildout.easy_install DEBUG Fetching demoneeded 1.1 from: http://.../demoneeded-1.1.tar.gz zc.buildout.easy_install DEBUG Running pip install:... zc.buildout.easy_install INFO Got demoneeded 1.1. zc.buildout.easy_install DEBUG Picked: demoneeded = 1.1 zc.buildout.easy_install DEBUG Installing 'demo'. zc.buildout.easy_install DEBUG We have the best distribution that satisfies 'demo'. zc.buildout.easy_install DEBUG Picked: demo = 0.3 zc.buildout.easy_install DEBUG Getting required 'demoneeded' zc.buildout.easy_install DEBUG required by demo 0.3. zc.buildout.easy_install DEBUG We have the best distribution that satisfies 'demoneeded'. zc.buildout.easy_install DEBUG Picked: demoneeded = 1.1 >>> handler.uninstall() >>> logging.getLogger('zc.buildout.easy_install').propagate = True We can request that we get an error if versions are picked: >>> zc.buildout.easy_install.allow_picked_versions(False) True (The old setting is returned.) >>> ws = zc.buildout.easy_install.install( ... ['demo'], dest, links=[link_server], index=link_server+'index/', ... ) Traceback (most recent call last): ... UserError: Picked: demo = 0.3 The `demo` egg does not have a version pin and `allow-picked-versions = false`. To resolve this, add demo = 0.3 to the [versions] section, OR set `allow-picked-versions = true`. >>> zc.buildout.easy_install.allow_picked_versions(True) False The function default_versions can be used to get and set default version information to be used when no version information is passes. If called with an argument, it sets the default versions: >>> zc.buildout.easy_install.default_versions(dict(demoneeded='1')) ... # doctest: +ELLIPSIS {...} It always returns the previous default versions. If called without an argument, it simply returns the default versions without changing them: >>> zc.buildout.easy_install.default_versions() {'demoneeded': '1'} So with the default versions set, we'll get the requested version even if the versions option isn't used: >>> ws = zc.buildout.easy_install.install( ... ['demo'], dest, links=[link_server], index=link_server+'index/', ... ) >>> [d.version for d in ws] ['0.3', '1.0'] Of course, we can unset the default versions by passing an empty dictionary: >>> zc.buildout.easy_install.default_versions({}) {'demoneeded': '1'} >>> ws = zc.buildout.easy_install.install( ... ['demo'], dest, links=[link_server], index=link_server+'index/', ... ) >>> [d.version for d in ws] ['0.3', '1.1'] Dependency links ---------------- Setuptools allows metadata that describes where to search for package dependencies. This option is called dependency_links. Buildout has its own notion of where to look for dependencies, but it also uses the setup tools dependency_links information if it's available. Let's demo this by creating an egg that specifies dependency_links. To begin, let's create a new egg repository. This repository hold a newer version of the 'demoneeded' egg than the sample repository does. >>> repoloc = tmpdir('repo') >>> from zc.buildout.tests import create_egg >>> create_egg('demoneeded', '1.2', repoloc) >>> link_server2 = start_server(repoloc) Turn on logging on this server so that we can see when eggs are pulled from it. >>> _ = get(link_server2 + 'enable_server_logging') GET 200 /enable_server_logging Now we can create an egg that specifies that its dependencies are found on this server. >>> repoloc = tmpdir('repo2') >>> create_egg('hasdeps', '1.0', repoloc, ... install_requires = "'demoneeded'", ... dependency_links = [link_server2]) Let's add the egg to another repository. >>> link_server3 = start_server(repoloc) Now let's install the egg. >>> example_dest = tmpdir('example-install') >>> workingset = zc.buildout.easy_install.install( ... ['hasdeps'], example_dest, ... links=[link_server3], index=link_server3+'index/') GET 200 / GET 200 /demoneeded-1.2-pyN.N.egg The server logs show that the dependency was retrieved from the server specified in the dependency_links. Now let's see what happens if we provide two different ways to retrieve the dependencies. >>> rmdir(example_dest) >>> example_dest = tmpdir('example-install') >>> workingset = zc.buildout.easy_install.install( ... ['hasdeps'], example_dest, index=link_server+'index/', ... links=[link_server, link_server3]) GET 200 / GET 200 /demoneeded-1.2-pyN.N.egg Once again the dependency is fetched from the logging server even though it is also available from the non-logging server. This is because the version on the logging server is newer and buildout normally chooses the newest egg available. If you wish to control where dependencies come from regardless of dependency_links setup metadata use the 'use_dependency_links' option to zc.buildout.easy_install.install(). >>> rmdir(example_dest) >>> example_dest = tmpdir('example-install') >>> workingset = zc.buildout.easy_install.install( ... ['hasdeps'], example_dest, index=link_server+'index/', ... links=[link_server, link_server3], ... use_dependency_links=False) Notice that this time the dependency egg is not fetched from the logging server. When you specify not to use dependency_links, eggs will only be searched for using the links you explicitly provide. Another way to control this option is with the zc.buildout.easy_install.use_dependency_links() function. This function sets the default behavior for the zc.buildout.easy_install() function. >>> zc.buildout.easy_install.use_dependency_links(False) True The function returns its previous setting. >>> rmdir(example_dest) >>> example_dest = tmpdir('example-install') >>> workingset = zc.buildout.easy_install.install( ... ['hasdeps'], example_dest, index=link_server+'index/', ... links=[link_server, link_server3]) It can be overridden by passing a keyword argument to the install function. >>> rmdir(example_dest) >>> example_dest = tmpdir('example-install') >>> workingset = zc.buildout.easy_install.install( ... ['hasdeps'], example_dest, index=link_server+'index/', ... links=[link_server, link_server3], ... use_dependency_links=True) GET 200 /demoneeded-1.2-pyN.N.egg To return the dependency_links behavior to normal call the function again. >>> zc.buildout.easy_install.use_dependency_links(True) False >>> rmdir(example_dest) >>> example_dest = tmpdir('example-install') >>> workingset = zc.buildout.easy_install.install( ... ['hasdeps'], example_dest, index=link_server+'index/', ... links=[link_server, link_server3]) GET 200 /demoneeded-1.2-pyN.N.egg Script generation ----------------- The easy_install module provides support for creating scripts from eggs. It provides a function similar to setuptools except that it provides facilities for baking a script's path into the script. This has two advantages: - The eggs to be used by a script are not chosen at run time, making startup faster and, more importantly, deterministic. - The script doesn't have to import pkg_resources because the logic that pkg_resources would execute at run time is executed at script-creation time. The scripts method can be used to generate scripts. Let's create a destination directory for it to place them in: >>> import tempfile >>> bin = tmpdir('bin') Now, we'll use the scripts method to generate scripts in this directory from the demo egg: >>> import sys >>> scripts = zc.buildout.easy_install.scripts( ... ['demo'], ws, sys.executable, bin) the three arguments we passed were: 1. A sequence of distribution requirements. These are of the same form as setuptools requirements. Here we passed a single requirement, for the version 0.1 demo distribution. 2. A working set, 3. The destination directory. The bin directory now contains a generated script: >>> ls(bin) - demo The return value is a list of the scripts generated: >>> import os, sys >>> if sys.platform == 'win32': ... scripts == [os.path.join(bin, 'demo.exe'), ... os.path.join(bin, 'demo-script.py')] ... else: ... scripts == [os.path.join(bin, 'demo')] True Note that in Windows, 2 files are generated for each script. A script file, ending in '-script.py', and an exe file that allows the script to be invoked directly without having to specify the Python interpreter and without having to provide a '.py' suffix. The demo script run the entry point defined in the demo egg: >>> cat(bin, 'demo') # doctest: +NORMALIZE_WHITESPACE #!/usr/local/bin/python2.7 import sys sys.path[0:0] = [ '/sample-install/demo-0.3-py2.4.egg', '/sample-install/demoneeded-1.1-py2.4.egg', ] import eggrecipedemo if __name__ == '__main__': sys.exit(eggrecipedemo.main()) Some things to note: - The demo and demoneeded eggs are added to the beginning of sys.path. - The module for the script entry point is imported and the entry point, in this case, 'main', is run. Rather than requirement strings, you can pass tuples containing 3 strings: - A script name, - A module, - An attribute expression for an entry point within the module. For example, we could have passed entry point information directly rather than passing a requirement: >>> scripts = zc.buildout.easy_install.scripts( ... [('demo', 'eggrecipedemo', 'main')], ws, ... sys.executable, bin) >>> cat(bin, 'demo') # doctest: +NORMALIZE_WHITESPACE #!/usr/local/bin/python2.7 import sys sys.path[0:0] = [ '/sample-install/demo-0.3-py2.4.egg', '/sample-install/demoneeded-1.1-py2.4.egg', ] import eggrecipedemo if __name__ == '__main__': sys.exit(eggrecipedemo.main()) Passing entry-point information directly is handy when using eggs (or distributions) that don't declare their entry points, such as distributions that aren't based on setuptools. The interpreter keyword argument can be used to generate a script that can be used to invoke the Python interactive interpreter with the path set based on the working set. This generated script can also be used to run other scripts with the path set on the working set: >>> scripts = zc.buildout.easy_install.scripts( ... ['demo'], ws, sys.executable, bin, interpreter='py') >>> ls(bin) - demo - py >>> if sys.platform == 'win32': ... scripts == [os.path.join(bin, 'demo.exe'), ... os.path.join(bin, 'demo-script.py'), ... os.path.join(bin, 'py.exe'), ... os.path.join(bin, 'py-script.py')] ... else: ... scripts == [os.path.join(bin, 'demo'), ... os.path.join(bin, 'py')] True The py script simply runs the Python interactive interpreter with the path set: >>> cat(bin, 'py') # doctest: +NORMALIZE_WHITESPACE +REPORT_NDIFF #!/usr/local/bin/python2.7 import sys sys.path[0:0] = [ '/sample-install/demo-0.3-pyN.N.egg', '/sample-install/demoneeded-1.1-pyN.N.egg', ] _interactive = True if len(sys.argv) > 1: # The Python interpreter wrapper allows only some of the options that a # "regular" Python interpreter accepts. _options, _args = __import__("getopt").getopt(sys.argv[1:], 'Iic:m:') _interactive = False for (_opt, _val) in _options: if _opt == '-i': _interactive = True elif _opt == '-c': exec(_val) elif _opt == '-m': sys.argv[1:] = _args _args = [] __import__("runpy").run_module( _val, {}, "__main__", alter_sys=True) elif _opt == '-I': # Allow yet silently ignore the `-I` option. The original behaviour # for this option is to create an isolated Python runtime. It was # deemed acceptable to allow the option here as this Python wrapper # is isolated from the system Python already anyway. # The specific use-case that led to this change is how the Python # language extension for Visual Studio Code calls the Python # interpreter when initializing the extension. pass if _args: sys.argv[:] = _args __file__ = _args[0] del _options, _args with open(__file__, 'U') as __file__f: exec(compile(__file__f.read(), __file__, "exec")) if _interactive: del _interactive __import__("code").interact(banner="", local=globals()) If invoked with a script name and arguments, it will run that script, instead. >>> write('ascript', r''' ... "demo doc" ... import sys ... print_ = lambda *a: sys.stdout.write(' '.join(map(str, a))+'\n') ... print_(sys.argv) ... print_((__name__, __file__, __doc__)) ... ''') >>> print_(system(join(bin, 'py')+' ascript a b c'), end='') ['ascript', 'a', 'b', 'c'] ('__main__', 'ascript', 'demo doc') You can also use the -m option to run a module: >>> print_(system(join(bin, 'py')+' -m pdb'), end='') ... # doctest: +ELLIPSIS usage: pdb... >>> print_(system(join(bin, 'py')+' -m pdb what'), end='') # if you have pdbpp installed, this may fail. Error: what does not exist An interpreter can also be generated without other eggs: >>> scripts = zc.buildout.easy_install.scripts( ... [], [], sys.executable, bin, interpreter='py') >>> cat(bin, 'py') # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS #!/usr/local/bin/python2.7 import sys sys.path[0:0] = [ ] ... An additional argument can be passed to define which scripts to install and to provide script names. The argument is a dictionary mapping original script names to new script names. >>> bin = tmpdir('bin2') >>> scripts = zc.buildout.easy_install.scripts( ... ['demo'], ws, sys.executable, bin, dict(demo='run')) >>> if sys.platform == 'win32': ... scripts == [os.path.join(bin, 'run.exe'), ... os.path.join(bin, 'run-script.py')] ... else: ... scripts == [os.path.join(bin, 'run')] True >>> ls(bin) - run >>> print_(system(os.path.join(bin, 'run')), end='') 3 1 The scripts that are generated are made executable: >>> if sys.platform == 'win32': ... os.access(os.path.join(bin, 'run.exe'), os.X_OK) ... else: ... os.access(os.path.join(bin, 'run'), os.X_OK) True Including extra paths in scripts -------------------------------- We can pass a keyword argument, extra paths, to cause additional paths to be included in the a generated script: >>> foo = tmpdir('foo') >>> scripts = zc.buildout.easy_install.scripts( ... ['demo'], ws, sys.executable, bin, dict(demo='run'), ... extra_paths=[foo]) >>> cat(bin, 'run') # doctest: +NORMALIZE_WHITESPACE #!/usr/local/bin/python2.7 import sys sys.path[0:0] = [ '/sample-install/demo-0.3-py2.4.egg', '/sample-install/demoneeded-1.1-py2.4.egg', '/foo', ] import eggrecipedemo if __name__ == '__main__': sys.exit(eggrecipedemo.main()) Providing script arguments -------------------------- An "argument" keyword argument can be used to pass arguments to an entry point. The value passed is a source string to be placed between the parentheses in the call: >>> scripts = zc.buildout.easy_install.scripts( ... ['demo'], ws, sys.executable, bin, dict(demo='run'), ... arguments='1, 2') >>> cat(bin, 'run') # doctest: +NORMALIZE_WHITESPACE #!/usr/local/bin/python2.7 import sys sys.path[0:0] = [ '/sample-install/demo-0.3-py2.4.egg', '/sample-install/demoneeded-1.1-py2.4.egg', ] import eggrecipedemo if __name__ == '__main__': sys.exit(eggrecipedemo.main(1, 2)) Passing initialization code --------------------------- You can also pass script initialization code: >>> scripts = zc.buildout.easy_install.scripts( ... ['demo'], ws, sys.executable, bin, dict(demo='run'), ... arguments='1, 2', ... initialization='import os\nos.chdir("foo")', ... interpreter='py') >>> cat(bin, 'run') # doctest: +NORMALIZE_WHITESPACE #!/usr/local/bin/python2.7 import sys sys.path[0:0] = [ '/sample-install/demo-0.3-py2.4.egg', '/sample-install/demoneeded-1.1-py2.4.egg', ] import os os.chdir("foo") import eggrecipedemo if __name__ == '__main__': sys.exit(eggrecipedemo.main(1, 2)) It will be included in interpreters too: >>> cat(bin, 'py') # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS #!/usr/local/bin/python2.7 import sys sys.path[0:0] = [ '/sample-install/demo-0.3-py3.3.egg', '/sample-install/demoneeded-1.1-py3.3.egg', ] import os os.chdir("foo") _interactive = True ... Relative paths -------------- Sometimes, you want to be able to move a buildout directory around and have scripts still work without having to rebuild them. We can control this using the relative_paths option to install. You need to pass a common base directory of the scripts and eggs: >>> bo = tmpdir('bo') >>> ba = tmpdir('ba') >>> mkdir(bo, 'eggs') >>> mkdir(bo, 'bin') >>> mkdir(bo, 'other') >>> ws = zc.buildout.easy_install.install( ... ['demo'], join(bo, 'eggs'), links=[link_server], ... index=link_server+'index/') >>> scripts = zc.buildout.easy_install.scripts( ... ['demo'], ws, sys.executable, join(bo, 'bin'), dict(demo='run'), ... extra_paths=[ba, join(bo, 'bar'), bo], ... interpreter='py', ... relative_paths=bo) >>> cat(bo, 'bin', 'run') #!/usr/local/bin/python2.7 import os join = os.path.join base = os.path.dirname(os.path.abspath(os.path.realpath(__file__))) base = os.path.dirname(base) import sys sys.path[0:0] = [ join(base, 'eggs/demoneeded-1.1-pyN.N.egg'), join(base, 'eggs/demo-0.3-pyN.N.egg'), '/ba', join(base, 'bar'), base, ] import eggrecipedemo if __name__ == '__main__': sys.exit(eggrecipedemo.main()) Note that the extra path we specified that was outside the directory passed as relative_paths wasn't converted to a relative path. Of course, running the script works: >>> print_(system(join(bo, 'bin', 'run')), end='') 3 1 We specified an interpreter and its paths are adjusted too: >>> cat(bo, 'bin', 'py') # doctest: +NORMALIZE_WHITESPACE +REPORT_NDIFF #!/usr/local/bin/python2.7 import os join = os.path.join base = os.path.dirname(os.path.abspath(os.path.realpath(__file__))) base = os.path.dirname(base) import sys sys.path[0:0] = [ join(base, 'eggs/demoneeded-1.1-pyN.N.egg'), join(base, 'eggs/demo-0.3-pyN.N.egg'), '/ba', join(base, 'bar'), base, ] _interactive = True if len(sys.argv) > 1: # The Python interpreter wrapper allows only some of the options that a # "regular" Python interpreter accepts. _options, _args = __import__("getopt").getopt(sys.argv[1:], 'Iic:m:') _interactive = False for (_opt, _val) in _options: if _opt == '-i': _interactive = True elif _opt == '-c': exec(_val) elif _opt == '-m': sys.argv[1:] = _args _args = [] __import__("runpy").run_module( _val, {}, "__main__", alter_sys=True) elif _opt == '-I': # Allow yet silently ignore the `-I` option. The original behaviour # for this option is to create an isolated Python runtime. It was # deemed acceptable to allow the option here as this Python wrapper # is isolated from the system Python already anyway. # The specific use-case that led to this change is how the Python # language extension for Visual Studio Code calls the Python # interpreter when initializing the extension. pass if _args: sys.argv[:] = _args __file__ = _args[0] del _options, _args with open(__file__, 'U') as __file__f: exec(compile(__file__f.read(), __file__, "exec")) if _interactive: del _interactive __import__("code").interact(banner="", local=globals()) Installing distutils-style scripts ---------------------------------- Most python libraries use the console_scripts entry point nowadays. But several still have a ``scripts=['bin/something']`` in their setup() call. Buildout also installs those: >>> distdir = tmpdir('distutilsscriptdir') >>> distbin = tmpdir('distutilsscriptbin') >>> ws = zc.buildout.easy_install.install( ... ['other'], distdir, ... links=[link_server], index=link_server+'index/') >>> scripts = zc.buildout.easy_install.scripts( ... ['other'], ws, sys.executable, distbin) >>> ls(distbin) - distutilsscript Like for console_scripts, the output is a list of the scripts generated. Likewise, on windows two files, an ``.exe`` and a script with ``-script.py`` appended, are generated: >>> import os, sys >>> if sys.platform == 'win32': ... scripts == [os.path.join(distbin, 'distutilsscript.exe'), ... os.path.join(distbin, 'distutilsscript-script.py')] ... else: ... scripts == [os.path.join(distbin, 'distutilsscript')] True It also works for zipped eggs: >>> distdir2 = tmpdir('distutilsscriptdir2') >>> distbin2 = tmpdir('distutilsscriptbin2') >>> ws = zc.buildout.easy_install.install( ... ['du_zipped'], distdir2, ... links=[link_server], index=link_server+'index/') >>> scripts = zc.buildout.easy_install.scripts( ... ['du_zipped'], ws, sys.executable, distbin2) >>> ls(distbin2) - distutilsscript Distutils copies the script files verbatim, apart from a line at the top that looks like ``#!/usr/bin/python``, which gets replaced by the actual python interpreter. Buildout does the same, but additionally also adds the sys.path like for the console_scripts. >>> cat(distbin, 'distutilsscript') #!/usr/local/bin/python2.7 # -*- coding: utf-8 -*- """Module docstring.""" from __future__ import print_statement import sys sys.path[0:0] = [ '/distutilsscriptdir/other-1.0-pyN.N.egg', ] import os import sys; sys.stdout.write("distutils!\n") Note that there are several items that need to come first in such a script *before* buildout's ``sys.path`` statements: a source encoding hint, a module docstring and ``__future__`` imports. Buildout retains them in their proper place by looking at the first non-future import and placing its ``sys.path`` statement before that. Due to the nature of distutils scripts, buildout cannot pass arguments as there's no specific method to pass them to. In some cases, a python 3 ``__pycache__`` directory can end up in an internal ``EGG-INFO`` metadata directory, next to the script information we're looking for. Buildout doesn't crash on that: >>> eggname = [name for name in os.listdir(distdir2) ... if name.endswith('egg')][0] >>> scripts_metadata_dir = os.path.join( ... distdir2, eggname, 'EGG-INFO', 'scripts') >>> os.mkdir(os.path.join(scripts_metadata_dir, '__dummy__')) >>> scripts = zc.buildout.easy_install.scripts( ... ['du_zipped'], ws, sys.executable, distbin2) >>> ls(distbin2) - distutilsscript Installing develop eggs sadly means that setuptools doesn't record distutils scripts in the metadata. We try to detect such scripts anyhow: >>> dev_distutils_dir = tmpdir('dev_distutils_dir') >>> dev_distutils_dest = tmpdir('dev_distutils_dest') >>> dev_eggs_dir = os.path.join(dev_distutils_dest, 'develop-eggs') >>> bin_dir = os.path.join(dev_distutils_dest, 'bin') >>> os.mkdir(dev_eggs_dir) >>> os.mkdir(bin_dir) >>> write(dev_distutils_dir, 'distutilsscript2', ... '#!/usr/bin/python\n' ... '# -*- coding: utf-8 -*-\n' ... '"""Module docstring."""\n' ... 'from __future__ import print_statement\n' ... 'import os\n' ... 'import sys; sys.stdout.write("distutils!\\n")\n' ... ) >>> write(dev_distutils_dir, 'setup.py', ... ''' ... from setuptools import setup ... setup(name="foo2", ... scripts=['distutilsscript2']) ... ''') >>> zc.buildout.easy_install.develop( ... dev_distutils_dir, dev_eggs_dir) '/dev_distutils_dest/develop-eggs/foo2.egg-link' >>> ws = zc.buildout.easy_install.working_set( ... ['foo2'], sys.executable, [dev_eggs_dir]) >>> scripts = zc.buildout.easy_install.scripts( ... ['foo2'], ws, sys.executable, dest=bin_dir) >>> if sys.platform == 'win32': ... scripts == [os.path.join(dev_distutils_dest, 'bin', 'distutilsscript2.exe'), ... os.path.join(dev_distutils_dest, 'bin', 'distutilsscript2-script.py')] ... else: ... scripts == [os.path.join(dev_distutils_dest, 'bin', 'distutilsscript2')] True Handling custom build options for extensions provided in source distributions ----------------------------------------------------------------------------- Sometimes, we need to control how extension modules are built. The build function provides this level of control. It takes a single package specification, downloads a source distribution, and builds it with specified custom build options. The build function takes 3 positional arguments: spec A package specification for a source distribution dest A destination directory build_ext A dictionary of options to be passed to the distutils build_ext command when building extensions. It supports a number of optional keyword arguments: links a sequence of URLs, file names, or directories to look for links to distributions, index The URL of an index server, or almost any other valid URL. :) If not specified, the Python Package Index, https://pypi.org/simple/, is used. You can specify an alternate index with this option. If you use the links option and if the links point to the needed distributions, then the index can be anything and will be largely ignored. In the examples, here, we'll just point to an empty directory on our link server. This will make our examples run a little bit faster. path A list of additional directories to search for locally-installed distributions. newest A boolean value indicating whether to search for new distributions when already-installed distributions meet the requirement. When this is true, the default, and when the destination directory is not None, then the install function will search for the newest distributions that satisfy the requirements. versions A dictionary mapping project names to version numbers to be used when selecting distributions. This can be used to specify a set of distribution versions independent of other requirements. Our link server included a source distribution that includes a simple extension, extdemo.c:: #include #include static PyMethodDef methods[] = {}; PyMODINIT_FUNC initextdemo(void) { PyObject *m; m = Py_InitModule3("extdemo", methods, ""); #ifdef TWO PyModule_AddObject(m, "val", PyInt_FromLong(2)); #else PyModule_AddObject(m, "val", PyInt_FromLong(EXTDEMO)); #endif } The extension depends on a system-dependent include file, extdemo.h, that defines a constant, EXTDEMO, that is exposed by the extension. We'll add an include directory to our sample buildout and add the needed include file to it: >>> mkdir('include') >>> write('include', 'extdemo.h', ... """ ... #define EXTDEMO 42 ... """) Now, we can use the build function to create an egg from the source distribution: >>> zc.buildout.easy_install.build( ... 'extdemo', dest, ... {'include_dirs': os.path.join(sample_buildout, 'include')}, ... links=[link_server], index=link_server+'index/') ['/sample-install/extdemo-1.4-py2.4-unix-i686.egg'] The function returns the list of eggs Now if we look in our destination directory, we see we have an extdemo egg: >>> ls(dest) d demo-0.2-py2.4.egg d demo-0.3-py2.4.egg d demoneeded-1.0-py2.4.egg d demoneeded-1.1-py2.4.egg d extdemo-1.4-py2.4-unix-i686.egg Let's update our link server with a new version of extdemo: >>> update_extdemo() >>> print_(get(link_server), end='') bigdemo-0.1-py3-none-any.whl
demo-0.1-py3-none-any.whl
demo-0.2-py3-none-any.whl
demo-0.3-py3-none-any.whl
demo-0.4rc1-py3-none-any.whl
demoneeded-1.0.tar.gz
demoneeded-1.1.tar.gz
demoneeded-1.2rc1.tar.gz
du_zipped-1.0-pyN.N.egg
extdemo-1.4.tar.gz
extdemo-1.5.tar.gz
index/
mixedcase-0.5.tar.gz
other-1.0-py3-none-any.whl
The easy_install caches information about servers to reduce network access. To see the update, we have to call the clear_index_cache function to clear the index cache: >>> zc.buildout.easy_install.clear_index_cache() If we run build with newest set to False, we won't get an update: >>> zc.buildout.easy_install.build( ... 'extdemo', dest, ... {'include_dirs': os.path.join(sample_buildout, 'include')}, ... links=[link_server], index=link_server+'index/', ... newest=False) ['/sample-install/extdemo-1.4-py2.4-linux-i686.egg'] >>> ls(dest) d demo-0.2-py2.4.egg d demo-0.3-py2.4.egg d demoneeded-1.0-py2.4.egg d demoneeded-1.1-py2.4.egg d extdemo-1.4-py2.4-unix-i686.egg But if we run it with the default True setting for newest, then we'll get an updated egg: >>> zc.buildout.easy_install.build( ... 'extdemo', dest, ... {'include_dirs': os.path.join(sample_buildout, 'include')}, ... links=[link_server], index=link_server+'index/') ['/sample-install/extdemo-1.5-py2.4-unix-i686.egg'] >>> ls(dest) d demo-0.2-py2.4.egg d demo-0.3-py2.4.egg d demoneeded-1.0-py2.4.egg d demoneeded-1.1-py2.4.egg d extdemo-1.4-py2.4-unix-i686.egg d extdemo-1.5-py2.4-unix-i686.egg The versions option also influences the versions used. For example, if we specify a version for extdemo, then that will be used, even though it isn't the newest. Let's clean out the destination directory first: >>> import os >>> for name in os.listdir(dest): ... remove(dest, name) >>> zc.buildout.easy_install.build( ... 'extdemo', dest, ... {'include_dirs': os.path.join(sample_buildout, 'include')}, ... links=[link_server], index=link_server+'index/', ... versions=dict(extdemo='1.4')) ['/sample-install/extdemo-1.4-py2.4-unix-i686.egg'] >>> ls(dest) d extdemo-1.4-py2.4-unix-i686.egg Handling custom build options for extensions in develop eggs ------------------------------------------------------------ The develop function is similar to the build function, except that, rather than building an egg from a source directory containing a setup.py script. The develop function takes 2 positional arguments: setup The path to a setup script, typically named "setup.py", or a directory containing a setup.py script. dest The directory to install the egg link to It supports some optional keyword argument: build_ext A dictionary of options to be passed to the distutils build_ext command when building extensions. We have a local directory containing the extdemo source: >>> contents = os.listdir(extdemo) >>> 'MANIFEST.in' in contents True >>> 'README' in contents True >>> 'extdemo.c' in contents True >>> 'setup.py' in contents True Now, we can use the develop function to create a develop egg from the source distribution: >>> zc.buildout.easy_install.develop( ... extdemo, dest, ... {'include_dirs': os.path.join(sample_buildout, 'include')}) '/sample-install/extdemo.egg-link' The name of the egg link created is returned. Now if we look in our destination directory, we see we have an extdemo egg link: >>> ls(dest) d extdemo-1.4-py2.4-unix-i686.egg - extdemo.egg-link And that the source directory contains the compiled extension: >>> contents = os.listdir(extdemo) >>> bool([f for f in contents if f.endswith('.so') or f.endswith('.pyd')]) True Download cache -------------- Normally, when distributions are installed, if any processing is needed, they are downloaded from the internet to a temporary directory and then installed from there. A download cache can be used to avoid the download step. This can be useful to reduce network access and to create source distributions of an entire buildout. A download cache is specified by calling the download_cache function. The function always returns the previous setting. If no argument is passed, then the setting is unchanged. If an argument is passed, the download cache is set to the given path, which must point to an existing directory. Passing None clears the cache setting. To see this work, we'll create a directory and set it as the cache directory: >>> cache = tmpdir('cache') >>> zc.buildout.easy_install.download_cache(cache) We'll recreate our destination directory: >>> remove(dest) >>> dest = tmpdir('sample-install') We'd like to see what is being fetched from the server, so we'll enable server logging: >>> _ = get(link_server+'enable_server_logging') GET 200 /enable_server_logging Now, if we install demo, and extdemo: >>> ws = zc.buildout.easy_install.install( ... ['demo==0.2'], dest, ... links=[link_server], index=link_server+'index/') GET 200 / GET 404 /index/demo/ GET 200 /index/ GET 200 /demo-0.2-py3-none-any.whl GET 404 /index/demoneeded/ GET 200 /demoneeded-1.1.tar.gz >>> zc.buildout.easy_install.build( ... 'extdemo', dest, ... {'include_dirs': os.path.join(sample_buildout, 'include')}, ... links=[link_server], index=link_server+'index/') GET 404 /index/extdemo/ GET 200 /extdemo-1.5.tar.gz ['/sample-install/extdemo-1.5-py2.4-linux-i686.egg'] Not only will we get eggs in our destination directory: >>> ls(dest) d demo-0.2-py2.4.egg d demoneeded-1.1-py2.4.egg d extdemo-1.5-py2.4-linux-i686.egg But we'll get distributions in the cache directory: >>> ls(cache) - demo-0.2-py3-none-any.whl - demoneeded-1.1.tar.gz - extdemo-1.5.tar.gz The cache directory contains uninstalled distributions, such as zipped eggs or source distributions. Let's recreate our destination directory and clear the index cache: >>> remove(dest) >>> dest = tmpdir('sample-install') >>> zc.buildout.easy_install.clear_index_cache() Now when we install the distributions: >>> ws = zc.buildout.easy_install.install( ... ['demo==0.2'], dest, ... links=[link_server], index=link_server+'index/') GET 200 / GET 404 /index/demo/ GET 200 /index/ GET 404 /index/demoneeded/ >>> zc.buildout.easy_install.build( ... 'extdemo', dest, ... {'include_dirs': os.path.join(sample_buildout, 'include')}, ... links=[link_server], index=link_server+'index/') GET 404 /index/extdemo/ ['/sample-install/extdemo-1.5-py2.4-linux-i686.egg'] >>> ls(dest) d demo-0.2-py2.4.egg d demoneeded-1.1-py2.4.egg d extdemo-1.5-py2.4-linux-i686.egg Note that we didn't download the distributions from the link server. If we remove the restriction on demo, we'll download a newer version from the link server: >>> ws = zc.buildout.easy_install.install( ... ['demo'], dest, ... links=[link_server], index=link_server+'index/') GET 200 /demo-0.3-py3-none-any.whl Normally, the download cache is the preferred source of downloads, but not the only one. Installing solely from a download cache --------------------------------------- A download cache can be used as the basis of application source releases. In an application source release, we want to distribute an application that can be built without making any network accesses. In this case, we distribute a download cache and tell the easy_install module to install from the download cache only, without making network accesses. The install_from_cache function can be used to signal that packages should be installed only from the download cache. The function always returns the previous setting. Calling it with no arguments returns the current setting without changing it: >>> zc.buildout.easy_install.install_from_cache() False Calling it with a boolean value changes the setting and returns the previous setting: >>> zc.buildout.easy_install.install_from_cache(True) False Let's remove demo-0.3-py2.4.egg from the cache, clear the index cache, recreate the destination directory, and reinstall demo: >>> for f in os.listdir(cache): ... if f.startswith('demo-0.3-'): ... remove(cache, f) >>> zc.buildout.easy_install.clear_index_cache() >>> remove(dest) >>> dest = tmpdir('sample-install') >>> ws = zc.buildout.easy_install.install( ... ['demo'], dest, ... links=[link_server], index=link_server+'index/') >>> ls(dest) d demo-0.2-py2.4.egg d demoneeded-1.1-py2.4.egg This time, we didn't download from or even query the link server. .. Disable the download cache: >>> zc.buildout.easy_install.download_cache(None) '/cache' >>> zc.buildout.easy_install.install_from_cache(False) True Environment markers ------------------- Specifications can include PEP 496 environment markers. >>> _ = get(link_server + 'disable_server_logging') >>> spec = ["demo ==0.1; python_version < '3.10'", ... "demo == 0.2; python_version >= '3.10'"] >>> ws = zc.buildout.easy_install.install( ... spec, dest, links=[link_server], index=link_server+'index/') >>> demo_version = None >>> for egg in ws: ... if egg.project_name == 'demo': ... demo_version = egg.version >>> demo_version is not None True >>> if (sys.version_info.minor < 10): ... demo_version ... else: ... '0.1' '0.1' >>> if (sys.version_info.minor >= 10): ... demo_version ... else: ... '0.2' '0.2' Conflicts properly with version spec. >>> ws = zc.buildout.easy_install.install( ... spec, dest, links=[link_server], index=link_server+'index/', ... versions = dict(demo='0.3')) Traceback (most recent call last): ... zc.buildout.easy_install.IncompatibleConstraintError: The requirement ('demo==0... ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675670.0 zc_buildout-4.1.6/src/zc/buildout/tests/extends-cache.txt0000644000076500000240000004405114773460426023145 0ustar00mauritsstaffCaching extended configuration ============================== As mentioned in the general buildout documentation, configuration files can extend each other, including the ability to download configuration being extended from a URL. If desired, zc.buildout caches downloaded configuration in order to be able to use it when run offline. As we're going to talk about downloading things, let's start an HTTP server. Also, all of the following will take place inside the sample buildout. >>> server_data = tmpdir('server_data') >>> server_url = start_server(server_data) >>> cd(sample_buildout) We also use a fresh directory for temporary files in order to make sure that all temporary files have been cleaned up in the end: >>> import tempfile >>> old_tempdir = tempfile.tempdir >>> tempfile.tempdir = tmpdir('tmp') Basic use of the extends cache ------------------------------ We put some base configuration on a server and reference it from a sample buildout: >>> write(server_data, 'base.cfg', """\ ... [buildout] ... parts = ... foo = bar ... """) >>> write('buildout.cfg', """\ ... [buildout] ... extends = %sbase.cfg ... """ % server_url) When trying to run this buildout offline, we'll find that we cannot read all of the required configuration: >>> print_(system(buildout + ' -o')) While: Initializing. Error: Couldn't download 'http://localhost/base.cfg' in offline mode. Trying the same online, we can: >>> print_(system(buildout)) Section `buildout` contains unused option(s): 'foo'. This may be an indication for either a typo in the option's name or a bug in the used recipe. As long as we haven't said anything about caching downloaded configuration, nothing gets cached. Offline mode will still cause the buildout to fail: >>> print_(system(buildout + ' -o')) While: Initializing. Error: Couldn't download 'http://localhost/base.cfg' in offline mode. Let's now specify a cache for base configuration files. This cache is different from the download cache used by recipes for caching distributions and other files; one might, however, use a namespace subdirectory of the download cache for it. The configuration cache we specify will be created when running buildout and the base.cfg file will be put in it (with the file name being a hash of the complete URL): >>> mkdir('cache') >>> write('buildout.cfg', """\ ... [buildout] ... extends = %sbase.cfg ... extends-cache = cache ... """ % server_url) >>> print_(system(buildout)) Section `buildout` contains unused option(s): 'foo'. This may be an indication for either a typo in the option's name or a bug in the used recipe. >>> cache = join(sample_buildout, 'cache') >>> ls(cache) - 5aedc98d7e769290a29d654a591a3a45 >>> import os >>> cat(cache, os.listdir(cache)[0]) [buildout] parts = foo = bar We can now run buildout offline as it will read base.cfg from the cache: >>> print_(system(buildout + ' -o')) Section `buildout` contains unused option(s): 'foo'. This may be an indication for either a typo in the option's name or a bug in the used recipe. The cache is being used purely as a fall-back in case we are offline or don't have access to a configuration file to be downloaded. As long as we are online, buildout attempts to download a fresh copy of each file even if a cached copy of the file exists. To see this, we put different configuration in the same place on the server and run buildout in offline mode so it takes base.cfg from the cache: >>> write(server_data, 'base.cfg', """\ ... [buildout] ... parts = ... bar = baz ... """) >>> print_(system(buildout + ' -o')) Section `buildout` contains unused option(s): 'foo'. This may be an indication for either a typo in the option's name or a bug in the used recipe. .. verify same behavior for install-from-cache: >>> print_(system(buildout + ' install-from-cache=true download-cache=.')) Section `buildout` contains unused option(s): 'foo'. This may be an indication for either a typo in the option's name or a bug in the used recipe. In online mode, buildout will download and use the modified version: >>> print_(system(buildout)) Section `buildout` contains unused option(s): 'bar'. This may be an indication for either a typo in the option's name or a bug in the used recipe. Trying offline mode again, the new version will be used as it has been put in the cache now: >>> print_(system(buildout + ' -o')) Section `buildout` contains unused option(s): 'bar'. This may be an indication for either a typo in the option's name or a bug in the used recipe. Clean up: >>> rmdir(cache) Specifying extends cache and offline mode ----------------------------------------- Normally, the values of buildout options such as the location of a download cache or whether to use offline mode are determined by first reading the user's default configuration, updating it with the project's configuration and finally applying command-line options. User and project configuration are assembled by reading a file such as ``~/.buildout/default.cfg``, ``buildout.cfg`` or a URL given on the command line, recursively (depth-first) downloading any base configuration specified by the ``buildout:extends`` option read from each of those config files, and finally evaluating each config file to provide default values for options not yet read. This works fine for all options that do not influence how configuration is downloaded in the first place. The ``extends-cache`` and ``offline`` options, however, are treated differently from the procedure described in order to make it simple and obvious to see where a particular configuration file came from under any particular circumstances. - Offline and extends-cache settings are read from the two root config files exclusively. Otherwise one could construct configuration files that, when read, imply that they should have been read from a different source than they have. Also, specifying the extends cache within a file that might have to be taken from the cache before being read wouldn't make a lot of sense. - Offline and extends-cache settings given by the user's defaults apply to the process of assembling the project's configuration. If no extends cache has been specified by the user's default configuration, the project's root config file must be available, be it from disk or from the net. - Offline mode turned on by the ``-o`` command line option is honored from the beginning even though command line options are applied to the configuration last. If offline mode is not requested by the command line, it may be switched on by either the user's or the project's config root. Extends cache ~~~~~~~~~~~~~ Let's see the above rules in action. We create a new home directory for our user and write user and project configuration that recursively extends online bases, using different caches: >>> mkdir('home') >>> mkdir('home', '.buildout') >>> mkdir('cache') >>> mkdir('user-cache') >>> home=join(sample_buildout, 'home') >>> env=dict(HOME=home, USERPROFILE=home) >>> import functools >>> system = functools.partial(system, env=env) >>> write('home', '.buildout', 'default.cfg', """\ ... [buildout] ... extends = fancy_default.cfg ... extends-cache = user-cache ... """) >>> write('home', '.buildout', 'fancy_default.cfg', """\ ... [buildout] ... extends = %sbase_default.cfg ... """ % server_url) >>> write(server_data, 'base_default.cfg', """\ ... [buildout] ... foo = bar ... offline = false ... """) >>> write('buildout.cfg', """\ ... [buildout] ... extends = fancy.cfg ... extends-cache = cache ... """) >>> write('fancy.cfg', """\ ... [buildout] ... extends = %sbase.cfg ... """ % server_url) >>> write(server_data, 'base.cfg', """\ ... [buildout] ... parts = ... offline = false ... """) Buildout will now assemble its configuration from all of these 6 files, defaults first. The online resources end up in the respective extends caches: >>> print_(system(buildout)) Section `buildout` contains unused option(s): 'foo'. This may be an indication for either a typo in the option's name or a bug in the used recipe. >>> ls('user-cache') - 10e772cf422123ef6c64ae770f555740 >>> cat('user-cache', os.listdir('user-cache')[0]) [buildout] foo = bar offline = false >>> ls('cache') - c72213127e6eb2208a3e1fc1dba771a7 >>> cat('cache', os.listdir('cache')[0]) [buildout] parts = offline = false If, on the other hand, the extends caches are specified in files that get extended themselves, they won't be used for assembling the configuration they belong to (user's or project's, resp.). The extends cache specified by the user's defaults does, however, apply to downloading project configuration. Let's rewrite the config files, clean out the caches and re-run buildout: >>> write('home', '.buildout', 'default.cfg', """\ ... [buildout] ... extends = fancy_default.cfg ... """) >>> write('home', '.buildout', 'fancy_default.cfg', """\ ... [buildout] ... extends = %sbase_default.cfg ... extends-cache = user-cache ... """ % server_url) >>> write('buildout.cfg', """\ ... [buildout] ... extends = fancy.cfg ... """) >>> write('fancy.cfg', """\ ... [buildout] ... extends = %sbase.cfg ... extends-cache = cache ... """ % server_url) >>> remove('user-cache', os.listdir('user-cache')[0]) >>> remove('cache', os.listdir('cache')[0]) >>> print_(system(buildout)) Section `buildout` contains unused option(s): 'foo'. This may be an indication for either a typo in the option's name or a bug in the used recipe. >>> ls('user-cache') - 0548bad6002359532de37385bb532e26 >>> cat('user-cache', os.listdir('user-cache')[0]) [buildout] parts = offline = false >>> ls('cache') Clean up: >>> rmdir('user-cache') >>> rmdir('cache') Offline mode and installation from cache ---------------------------------------- If we run buildout in offline mode now, it will fail because it cannot get at the remote configuration file needed by the user's defaults: >>> print_(system(buildout + ' -o')) While: Initializing. Error: Couldn't download 'http://localhost/base_default.cfg' in offline mode. Let's now successively turn on offline mode by different parts of the configuration and see when buildout applies this setting in each case: >>> write('home', '.buildout', 'default.cfg', """\ ... [buildout] ... extends = fancy_default.cfg ... offline = true ... """) >>> print_(system(buildout)) While: Initializing. Error: Couldn't download 'http://localhost/base_default.cfg' in offline mode. >>> write('home', '.buildout', 'default.cfg', """\ ... [buildout] ... extends = fancy_default.cfg ... """) >>> write('home', '.buildout', 'fancy_default.cfg', """\ ... [buildout] ... extends = %sbase_default.cfg ... offline = true ... """ % server_url) >>> print_(system(buildout)) While: Initializing. Error: Couldn't download 'http://localhost/base.cfg' in offline mode. >>> write('home', '.buildout', 'fancy_default.cfg', """\ ... [buildout] ... extends = %sbase_default.cfg ... """ % server_url) >>> write('buildout.cfg', """\ ... [buildout] ... extends = fancy.cfg ... offline = true ... """) >>> print_(system(buildout)) While: Initializing. Error: Couldn't download 'http://localhost/base.cfg' in offline mode. >>> write('buildout.cfg', """\ ... [buildout] ... extends = fancy.cfg ... """) >>> write('fancy.cfg', """\ ... [buildout] ... extends = %sbase.cfg ... offline = true ... """ % server_url) >>> print_(system(buildout)) Section `buildout` contains unused option(s): 'foo'. This may be an indication for either a typo in the option's name or a bug in the used recipe. The ``install-from-cache`` option is treated accordingly: >>> write('home', '.buildout', 'default.cfg', """\ ... [buildout] ... extends = fancy_default.cfg ... install-from-cache = true ... """) >>> print_(system(buildout)) While: Initializing. Error: Couldn't download 'http://localhost/base_default.cfg' in offline mode. >>> write('home', '.buildout', 'default.cfg', """\ ... [buildout] ... extends = fancy_default.cfg ... """) >>> write('home', '.buildout', 'fancy_default.cfg', """\ ... [buildout] ... extends = %sbase_default.cfg ... install-from-cache = true ... """ % server_url) >>> print_(system(buildout)) While: Initializing. Error: Couldn't download 'http://localhost/base.cfg' in offline mode. >>> write('home', '.buildout', 'fancy_default.cfg', """\ ... [buildout] ... extends = %sbase_default.cfg ... """ % server_url) >>> write('buildout.cfg', """\ ... [buildout] ... extends = fancy.cfg ... install-from-cache = true ... """) >>> print_(system(buildout)) While: Initializing. Error: Couldn't download 'http://localhost/base.cfg' in offline mode. >>> write('buildout.cfg', """\ ... [buildout] ... extends = fancy.cfg ... """) >>> write('fancy.cfg', """\ ... [buildout] ... extends = %sbase.cfg ... install-from-cache = true ... """ % server_url) >>> print_(system(buildout)) While: Installing. Checking for upgrades. An internal error occurred ... ValueError: install_from_cache set to true with no download cache >>> rmdir('home', '.buildout') Newest and non-newest behavior for extends cache ------------------------------------------------- While offline mode forbids network access completely, 'newest' mode determines whether to look for updated versions of a resource even if some version of it is already present locally. If we run buildout in newest mode (``newest = true``), the configuration files are updated with each run: >>> mkdir("cache") >>> write(server_data, 'base.cfg', """\ ... [buildout] ... parts = ... """) >>> write('buildout.cfg', """\ ... [buildout] ... extends-cache = cache ... extends = %sbase.cfg ... """ % server_url) >>> print_(system(buildout)) >>> ls('cache') - 5aedc98d7e769290a29d654a591a3a45 >>> cat('cache', os.listdir(cache)[0]) [buildout] parts = A change to ``base.cfg`` is picked up on the next buildout run: >>> write(server_data, 'base.cfg', """\ ... [buildout] ... parts = ... foo = bar ... """) >>> print_(system(buildout + " -n")) Section `buildout` contains unused option(s): 'foo'. This may be an indication for either a typo in the option's name or a bug in the used recipe. >>> cat('cache', os.listdir(cache)[0]) [buildout] parts = foo = bar In contrast, when not using ``newest`` mode (``newest = false``), the files already present in the extends cache will not be updated: >>> write(server_data, 'base.cfg', """\ ... [buildout] ... parts = ... """) >>> print_(system(buildout + " -N")) Section `buildout` contains unused option(s): 'foo'. This may be an indication for either a typo in the option's name or a bug in the used recipe. >>> cat('cache', os.listdir(cache)[0]) [buildout] parts = foo = bar Even when updating base configuration files with a buildout run, any given configuration file will be downloaded only once during that particular run. If some base configuration file is extended more than once, its cached copy is used: >>> write(server_data, 'baseA.cfg', """\ ... [buildout] ... extends = %sbase.cfg ... foo = bar ... """ % server_url) >>> write(server_data, 'baseB.cfg', """\ ... [buildout] ... extends-cache = cache ... extends = %sbase.cfg ... bar = foo ... """ % server_url) >>> write('buildout.cfg', """\ ... [buildout] ... extends-cache = cache ... newest = true ... extends = %sbaseA.cfg %sbaseB.cfg ... """ % (server_url, server_url)) >>> print_(system(buildout + " -n")) Section `buildout` contains unused option(s): 'bar' 'foo'. This may be an indication for either a typo in the option's name or a bug in the used recipe. (XXX We patch download utility's API to produce readable output for the test; a better solution would re-use the logging already done by the utility.) >>> import zc.buildout >>> old_download = zc.buildout.download.Download.download >>> def wrapper_download(self, url, md5sum=None, path=None): ... print_("The URL %s was downloaded." % url) ... return old_download(url, md5sum, path) >>> zc.buildout.download.Download.download = wrapper_download >>> zc.buildout.buildout.main([]) The URL http://localhost/baseA.cfg was downloaded. The URL http://localhost/base.cfg was downloaded. The URL http://localhost/baseB.cfg was downloaded. Not upgrading because not running a local buildout command. Section `buildout` contains unused option(s): 'bar' 'foo'. This may be an indication for either a typo in the option's name or a bug in the used recipe. >>> zc.buildout.download.Download.download = old_download The deprecated ``extended-by`` option ------------------------------------- The ``buildout`` section used to recognize an option named ``extended-by`` that was deprecated at some point and removed in the 1.5 line. Since ignoring this option silently was considered harmful as a matter of principle, a UserError is raised if that option is encountered now: >>> write(server_data, 'base.cfg', """\ ... [buildout] ... parts = ... extended-by = foo.cfg ... """) >>> print_(system(buildout)) While: Initializing. Error: No-longer supported "extended-by" option found in http://localhost/base.cfg. Reporting cached locations for downloads of faulty config files --------------------------------------------------------------- A downloaded config file might be invalid. A cancelled buildout run, an accidentally gzip-encoded download and so on. >>> write(server_data, 'faulty.cfg', """\ ... This is definitively not ... a proper() config file. ... """) >>> write('buildout.cfg', """\ ... [buildout] ... extends = %sfaulty.cfg ... """ % server_url) >>> print_(system(buildout)) While: Initializing. ... File contains no section headers. file: http://localhost/faulty.cfg (downloaded as ...), line: 1 'This is definitively not\n' Failing if extends-cache contains ${section:variable} ----------------------------------------------------- Because extends-cache is used to download extends, it may not contain ${section:variable} as the state of all variables cannot be computed before all extends have been loaded. >>> write(server_data, 'proper.cfg', """\ ... [buildout] ... dummy = fjhfj ... """) >>> write('buildout.cfg', """\ ... [buildout] ... extends = %sproper.cfg ... extends-cache = ${buildout:dummy} ... """ % server_url) >>> print_(system(buildout)) While: Initializing. ... ValueError: extends-cache '${buildout:dummy}' may not contain ${section:variable} to expand. Clean up -------- We should have cleaned up all temporary files created by downloading things: >>> ls(tempfile.tempdir) Reset the global temporary directory: >>> tempfile.tempdir = old_tempdir ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675670.0 zc_buildout-4.1.6/src/zc/buildout/tests/repeatable.txt0000644000076500000240000003305714773460426022542 0ustar00mauritsstaffRepeatable buildouts: controlling eggs used =========================================== One of the goals of zc.buildout is to provide enough control to make buildouts repeatable. It should be possible to check the buildout configuration files for a project into a version control system and later use the checked in files to get the same buildout, subject to changes in the environment outside the buildout. An advantage of using Python eggs is that dependencies of eggs used are automatically determined and used. The automatic inclusion of dependent distributions is at odds with the goal of repeatable buildouts. To support repeatable buildouts, a versions section can be created with options for each distribution name who's version is to be fixed. The section can then be specified via the buildout versions option. To see how this works, we'll create two versions of a recipe egg: >>> mkdir('recipe') >>> write('recipe', 'recipe.py', ... ''' ... import sys ... print_ = lambda *a: sys.stdout.write(' '.join(map(str, a))+'\\n') ... class Recipe: ... def __init__(*a): pass ... def install(self): ... print_('recipe v1') ... return () ... update = install ... ''') >>> write('recipe', 'setup.py', ... ''' ... from setuptools import setup ... setup(name='spam', version='1', py_modules=['recipe'], ... entry_points={'zc.buildout': ['default = recipe:Recipe']}, ... ) ... ''') >>> write('recipe', 'README', '') >>> print_(system(buildout+' setup recipe bdist_egg')) # doctest: +ELLIPSIS Running setup script 'recipe/setup.py'. ... >>> rmdir('recipe', 'build') >>> write('recipe', 'recipe.py', ... ''' ... import sys ... print_ = lambda *a: sys.stdout.write(' '.join(map(str, a))+'\\n') ... class Recipe: ... def __init__(*a): pass ... def install(self): ... print_('recipe v2') ... return () ... update = install ... ''') >>> write('recipe', 'setup.py', ... ''' ... from setuptools import setup ... setup(name='spam', version='2', py_modules=['recipe'], ... entry_points={'zc.buildout': ['default = recipe:Recipe']}, ... ) ... ''') >>> print_(system(buildout+' setup recipe bdist_egg')) # doctest: +ELLIPSIS Running setup script 'recipe/setup.py'. ... and we'll configure a buildout to use it: >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = foo ... find-links = %s ... ... [foo] ... recipe = spam ... ''' % join('recipe', 'dist')) If we run the buildout, it will use version 2: >>> print_(system(buildout), end='') Getting distribution for 'spam'. Got spam 2. Installing foo. recipe v2 We can specify a versions section that lists our recipe and name it in the buildout section: >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = foo ... find-links = %s ... ... [versions] ... spam = 1 ... eggs = 2.2 ... ... [foo] ... recipe = spam ... ''' % join('recipe', 'dist')) Here we created a versions section listing the version 1 for the spam distribution. We told the buildout to use it by specifying release-1 as in the versions option. Now, if we run the buildout, we'll use version 1 of the spam recipe: >>> print_(system(buildout), end='') Getting distribution for 'spam==1'. Got spam 1. Uninstalling foo. Installing foo. recipe v1 Running the buildout in verbose mode will help us get information about versions used. If we run the buildout in verbose mode without specifying a versions section: >>> print_(system(buildout+' buildout:versions= -v'), end='') Installing 'zc.buildout', 'wheel', 'pip', 'setuptools'. ... Installing 'spam'. We have the best distribution that satisfies 'spam'. Picked: spam = 2. Uninstalling foo. Installing foo. recipe v2 We'll get output that includes lines that tell us what versions buildout chose a for us, like:: zc.buildout.easy_install.picked: spam = 2 This allows us to discover versions that are picked dynamically, so that we can fix them in a versions section. If we run the buildout with the versions section: >>> print_(system(buildout+' -v'), end='') Installing 'zc.buildout', 'wheel', 'pip', 'setuptools'. ... Installing 'spam'. We have the distribution that satisfies 'spam==1'. Uninstalling foo. Installing foo. recipe v1 We won't get output for the spam distribution, which we didn't pick, but we will get output for setuptools, which we didn't specify versions for. .. Edge case: version applied to range requirement: >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = foo ... find-links = %s ... ... [versions] ... spam = 1 ... eggs = 2.2 ... ... [foo] ... recipe = spam >0 ... ''' % join('recipe', 'dist')) >>> print_(system(buildout+' -v'), end='') Installing 'zc.buildout', 'wheel', 'pip', 'setuptools'. ... Installing 'spam >0'. We have the distribution that satisfies 'spam==1'. Uninstalling foo. Installing foo. recipe v1 Edge case (issue #577) where a substitution inside the buildout section which comes from a section with a recipe, then the versions versions specifications were ignored, and the latest version installed, even if allow-picked-versions is false. >>> write('buildout.cfg', ... ''' ... [buildout] ... allow-picked-versions = false ... #show-picked-versions = true ... parts = foo ... find-links = %s ... test = ${foo:option} ... ... [versions] ... spam = 1 ... ... [foo] ... recipe = spam ... option = TEST ... ''' % join('recipe', 'dist')) >>> print_(system(buildout), end='') # doctest: +ELLIPSIS Uninstalling foo. Section `buildout` contains unused option(s): 'test'. ... Installing foo. recipe v1 You can request buildout to generate an error if it picks any versions: >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = foo ... find-links = %s ... allow-picked-versions = false ... ... [versions] ... eggs = 2.2 ... ... [foo] ... recipe = spam ... ''' % join('recipe', 'dist')) >>> print_(system(buildout), end='') # doctest: +ELLIPSIS While: Installing. Getting section foo. Initializing section foo. Installing recipe spam. Getting distribution for 'spam'. Error: Picked: spam = 2 ... We can name a version something else, if we wish, using the versions option: >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = foo ... find-links = %s ... versions = release1 ... ... [release1] ... spam = 1 ... eggs = 2.2 ... ... [foo] ... recipe = spam ... ''' % join('recipe', 'dist')) >>> print_(system(buildout), end='') # doctest: +ELLIPSIS Uninstalling foo. Installing foo. recipe v1 We can also disable checking versions: >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = foo ... find-links = %s ... versions = ... ... [versions] ... spam = 1 ... eggs = 2.2 ... ... [foo] ... recipe = spam ... ''' % join('recipe', 'dist')) >>> print_(system(buildout), end='') # doctest: +ELLIPSIS Uninstalling foo. Installing foo. recipe v2 Easier reporting and managing of versions (new in buildout 2.0) --------------------------------------------------------------- Since buildout 2.0, the functionality of the `buildout-versions `_ extension is part of buildout itself. This makes reporting and managing versions easier. Buildout picks versions for pip and setuptools and for the tests, we need to grab the version number: >>> import pkg_resources >>> req = pkg_resources.Requirement.parse('setuptools') >>> setuptools_version = pkg_resources.working_set.find(req).version >>> req = pkg_resources.Requirement.parse('pip') >>> pip_version = pkg_resources.working_set.find(req).version If you set the ``show-picked-versions`` option, buildout will print versions it picked at the end of its run: >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = foo ... find-links = %s ... show-picked-versions = true ... ... [versions] ... ... [foo] ... recipe = spam ... ''' % join('recipe', 'dist')) >>> print_(system(buildout), end='') # doctest: +ELLIPSIS Updating foo. recipe v2 Versions had to be automatically picked. The following part definition lists the versions picked: [versions] spam = 2 When everything is pinned, no output is generated: >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = foo ... find-links = %s ... show-picked-versions = true ... ... [versions] ... pip = %s ... setuptools = %s ... spam = 2 ... ... [foo] ... recipe = spam ... ''' % (join('recipe', 'dist'), pip_version, setuptools_version)) >>> print_(system(buildout), end='') # doctest: +ELLIPSIS Updating foo. recipe v2 The Python package index is case-insensitive. Both https://pypi.org/simple/Django/ and https://pypi.org/simple/dJaNgO/ work. And distributions aren't always naming themselves consistently case-wise. So all version names are normalized and case differences won't impact the pinning: >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = foo ... find-links = %s ... show-picked-versions = true ... ... [versions] ... pip = %s ... setuptools = %s ... Spam = 2 ... ... [foo] ... recipe = spam ... ''' % (join('recipe', 'dist'), pip_version, setuptools_version)) >>> print_(system(buildout), end='') # doctest: +ELLIPSIS Updating foo. recipe v2 Sometimes it is handy to have a separate file with versions. This is a regular buildout file with a single ``[versions]`` section. You include it by extending from that versions file: >>> write('my_versions.cfg', ... ''' ... [versions] ... pip = %s ... setuptools = %s ... spam = 2 ... ''' % (pip_version, setuptools_version)) >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = foo ... extends = my_versions.cfg ... find-links = %s ... show-picked-versions = true ... ... [foo] ... recipe = spam ... ''' % join('recipe', 'dist')) >>> print_(system(buildout), end='') # doctest: +ELLIPSIS Updating foo. recipe v2 If not everything is pinned and buildout has to pick versions, you can tell buildout to append the versions to your versions file. It simply appends them at the end. >>> write('my_versions.cfg', ... ''' ... [versions] ... pip = %s ... setuptools = %s ... ''' % (pip_version, setuptools_version)) >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = foo ... extends = my_versions.cfg ... update-versions-file = my_versions.cfg ... find-links = %s ... show-picked-versions = true ... ... [foo] ... recipe = spam ... ''' % join('recipe', 'dist')) >>> print_(system(buildout), end='') # doctest: +ELLIPSIS Updating foo. recipe v2 Versions had to be automatically picked. The following part definition lists the versions picked: [versions] spam = 2 Picked versions have been written to my_versions.cfg The versions file now contains the extra pin: >>> with open('my_versions.cfg') as f: print_(f.read()) # doctest: +ELLIPSIS ... # Added by buildout at YYYY-MM-DD hh:mm:ss.dddddd spam = 2 And re-running buildout doesn't report any picked versions anymore: >>> 'picked' in system(buildout) False If you've enabled ``update-versions-file`` but not ``show-picked-versions``, buildout will append the versions to your versions file anyway (without printing them to the console): >>> write('my_versions.cfg', ... ''' ... [versions] ... pip = %s ... setuptools = %s ... ''' % (pip_version, setuptools_version)) >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = foo ... extends = my_versions.cfg ... update-versions-file = my_versions.cfg ... find-links = %s ... show-picked-versions = false ... ... [foo] ... recipe = spam ... ''' % join('recipe', 'dist')) >>> print_(system(buildout), end='') # doctest: +ELLIPSIS Updating foo. recipe v2 Picked versions have been written to my_versions.cfg The versions file contains the extra pin: >>> with open('my_versions.cfg') as f: print_(f.read()) # doctest: +ELLIPSIS [versions] ... # Added by buildout at YYYY-MM-DD hh:mm:ss.dddddd spam = 2 Because buildout now includes buildout-versions' (and part of the older buildout.dumppickedversions') functionality, it warns if these extensions are configured. >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... parts = foo ... extensions = buildout-versions ... ... [foo] ... recipe = spam ... """) >>> print_(system(buildout), end='') # doctest: +NORMALIZE_WHITESPACE While: Installing. Loading extensions. Error: Buildout now includes 'buildout-versions' (and part of the older 'buildout.dumppickedversions'). Remove the extension from your configuration and look at the 'show-picked-versions' option in buildout's documentation. ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675670.0 zc_buildout-4.1.6/src/zc/buildout/tests/runsetup.txt0000644000076500000240000000326214773460426022316 0ustar00mauritsstaffRunning setup scripts ===================== Buildouts are often used to work on packages that will be distributed as eggs. During development, we use develop eggs. When you've completed a development cycle, you'll need to run your setup script to generate a distribution and, perhaps, uploaded it to the Python package index. If your script uses setuptools, you'll need setuptools in your Python path, which may be an issue if you haven't installed setuptools into your Python installation. The buildout setup command is helpful in a situation like this. It can be used to run a setup script and it does so with the setuptools egg in the Python path and with setuptools already imported. The fact that setuptools is imported means that you can use setuptools-based commands, like bdist_egg even with packages that don't use setuptools. To illustrate this, we'll create a package in a sample buildout: >>> mkdir('hello') >>> write('hello', 'hello.py', ... 'import sys; sys.stdout.write("Hello World!\\n")\n') >>> write('hello', 'README', 'This is hello') >>> write('hello', 'setup.py', ... """ ... from distutils.core import setup ... setup(name="hello", ... version="1.0", ... py_modules=["hello"], ... author="Bob", ... author_email="bob@foo.com", ... ) ... """) We can use the buildout command to generate the hello egg: >>> print_(system(buildout +' setup hello -q bdist_egg'), end='') Running setup script 'hello/setup.py'. zip_safe flag not set; analyzing archive contents... The hello directory now has a hello egg in it's dist directory: >>> ls('hello', 'dist') - hello-1.0-py2.4.egg ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675670.0 zc_buildout-4.1.6/src/zc/buildout/tests/setup.txt0000644000076500000240000000371614773460426021575 0ustar00mauritsstaffUsing zc.buildout to run setup scripts ====================================== zc buildout has a convenience command for running setup scripts. Why? There are two reasons. If a setup script doesn't import setuptools, you can't use any setuptools-provided commands, like bdist_egg. When buildout runs a setup script, it arranges to import setuptools before running the script so setuptools-provided commands are available. If you use a squeaky-clean Python to do your development, the setup script that would import setuptools because setuptools isn't in the path. Because buildout requires setuptools and knows where it has installed a setuptools egg, it adds the setuptools egg to the Python path before running the script. To run a setup script, use the buildout setup command, passing the name of a script or a directory containing a setup script and arguments to the script. Let's look at an example: >>> mkdir('test') >>> cd('test') >>> write('setup.py', ... ''' ... from distutils.core import setup ... setup(name='sample') ... ''') We've created a super simple (stupid) setup script. Note that it doesn't import setuptools. Let's try running it to create an egg. We'll use the buildout script from our sample buildout: >>> print_(system(buildout+' setup'), end='') ... # doctest: +NORMALIZE_WHITESPACE Creating directory '/sample-buildout/test/eggs'. Error: The setup command requires the path to a setup script or directory containing a setup script, and its arguments. Oops, we forgot to give the name of the setup script: >>> print_(system(buildout+' setup setup.py bdist_egg')) ... # doctest: +ELLIPSIS Running setup script 'setup.py'. ... >>> ls('dist') - sample-0.0.0-py2.5.egg Note that we can specify a directory name. This is often shorter and preferred by the lazy :) >>> print_(system(buildout+' setup . bdist_egg')) # doctest: +ELLIPSIS Running setup script './setup.py'. ... ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675670.0 zc_buildout-4.1.6/src/zc/buildout/tests/test_all.py0000644000076500000240000033543314773460426022061 0ustar00mauritsstaff# -*- coding: utf-8 -*- ############################################################################## # # Copyright (c) 2004-2009 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## from __future__ import print_function import unittest from zc.buildout.buildout import print_ from zope.testing import renormalizing, setupstack import doctest import manuel.capture import manuel.doctest import manuel.testing import os import pkg_resources import re import shutil import sys import tempfile import zc.buildout.easy_install import zc.buildout.testing import zipfile from zc.buildout.tests import easy_install_SetUp from zc.buildout.tests import normalize_bang from zc.buildout.tests import create_egg from zc.buildout.tests import create_sample_eggs os_path_sep = os.path.sep if os_path_sep == '\\': os_path_sep *= 2 class TestEasyInstall(unittest.TestCase): # The contents of a zipped egg, created by setuptools: # from setuptools import setup # setup( # name='TheProject', # version='3.3', # ) # # (we can't run setuptools at runtime, it may not be installed) EGG_DATA = ( b'PK\x03\x04\x14\x00\x00\x00\x08\x00q8\xa8Lg0\xb7ix\x00\x00\x00\xb6\x00' b'\x00\x00\x11\x00\x00\x00EGG-INFO/PKG-INFO\xf3M-ILI,I\xd4\rK-*' b'\xce\xcc\xcf\xb3R0\xd43\xe0\xf2K\xccM\xb5R\x08\xc9H\r(\xca\xcfJM' b'.\xe1\x82\xcb\x1a\xeb\x19s\x05\x97\xe6\xe6&\x16UZ)\x84\xfay\xfb\xf9\x87\xfb' b'qy\xe4\xe7\xa6\xea\x16$\xa6\xa7"\x84\x1cKK2\xf2\x8b\xd0\xf9\xba\xa9\xb9\x89' b'\x999\x08Q\x9f\xcc\xe4\xd4\xbcb$m.\xa9\xc5\xc9E\x99\x05%`\xbb`\x82\x019\x89%' b'i\xf9E\xb9\x08\x11\x00PK\x03\x04\x14\x00\x00\x00\x08\x00q8\xa8L61\xa1' b'XL\x00\x00\x00\x87\x00\x00\x00\x14\x00\x00\x00EGG-INFO/SOURCES.txt\x0b\xc9H' b'\r(\xca\xcfJM.\xd1KMO\xd7\xcd\xccK\xcb\xd7\x0f\xf0v\xd7\xf5\xf4s' b'\xf3\xe7\n\xc1"\x19\xec\x1f\x1a\xe4\xec\x1a\xacWRQ\x82U>%\xb5 5/%5/\xb92>\'3' b'/\xbb\x18\xa7\xc2\x92\xfc\x82\xf8\x9c\xd4\xb2\xd4\x1c\x90\n\x00PK\x03' b'\x04\x14\x00\x00\x00\x08\x00q8\xa8L\x93\x06\xd72\x03\x00\x00\x00\x01' b'\x00\x00\x00\x1d\x00\x00\x00EGG-INFO/dependency_links.txt\xe3\x02\x00P' b'K\x03\x04\x14\x00\x00\x00\x08\x00q8\xa8L\x93\x06\xd72\x03\x00\x00' b'\x00\x01\x00\x00\x00\x16\x00\x00\x00EGG-INFO/top_level.txt\xe3\x02\x00PK' b'\x03\x04\x14\x00\x00\x00\x08\x00q8\xa8L\x93\x06\xd72\x03\x00\x00\x00' b'\x01\x00\x00\x00\x11\x00\x00\x00EGG-INFO/zip-safe\xe3\x02\x00PK\x01\x02' b'\x14\x03\x14\x00\x00\x00\x08\x00q8\xa8Lg0\xb7ix\x00\x00\x00\xb6\x00\x00\x00' b'\x11\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xa4\x81\x00\x00\x00\x00EG' b'G-INFO/PKG-INFOPK\x01\x02\x14\x03\x14\x00\x00\x00\x08\x00q8\xa8L61\xa1XL' b'\x00\x00\x00\x87\x00\x00\x00\x14\x00\x00\x00\x00\x00\x00\x00\x00' b'\x00\x00\x00\xa4\x81\xa7\x00\x00\x00EGG-INFO/SOURCES.txtPK\x01' b'\x02\x14\x03\x14\x00\x00\x00\x08\x00q8\xa8L\x93\x06\xd72\x03\x00\x00' b'\x00\x01\x00\x00\x00\x1d\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' b'\x00\xa4\x81%\x01\x00\x00EGG-INFO/dependency_links.txtPK\x01\x02' b'\x14\x03\x14\x00\x00\x00\x08\x00q8\xa8L\x93\x06\xd72\x03\x00\x00\x00' b'\x01\x00\x00\x00\x16\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' b'\xa4\x81c\x01\x00\x00EGG-INFO/top_level.txtPK\x01\x02\x14\x03\x14\x00' b'\x00\x00\x08\x00q8\xa8L\x93\x06\xd72\x03\x00\x00\x00\x01\x00\x00\x00' b'\x11\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xa4\x81\x9a\x01\x00\x00EG' b'G-INFO/zip-safePK\x05\x06\x00\x00\x00\x00\x05\x00\x05\x00O\x01\x00\x00\xcc' b'\x01\x00\x00\x00\x00' ) def setUp(self): self.cwd = os.getcwd() self.temp_dir = tempfile.mkdtemp('.buildouttest') self.project_dir = os.path.join(self.temp_dir, 'TheProject') self.project_dist_dir = os.path.join(self.temp_dir, 'dist') os.mkdir(self.project_dist_dir) self.egg_path = os.path.join(self.project_dist_dir, 'TheProject.egg') os.mkdir(self.project_dir) self.setup_path = os.path.join(self.project_dir, 'setup.py') os.chdir(self.temp_dir) def tearDown(self): os.chdir(self.cwd) shutil.rmtree(self.temp_dir) def _make_egg(self): with open(self.egg_path, 'wb') as f: f.write(self.EGG_DATA) def _get_distro_and_egg_path(self): # Returns a distribution with a version of '3.3.0', # but an egg with a version of '3.3' self._make_egg() from distutils.dist import Distribution dist = Distribution() dist.project_name = 'TheProject' dist.version = '3.3.0' dist.parsed_version = pkg_resources.parse_version(dist.version) return dist, self.egg_path def test_get_matching_dist_in_location_uses_parsed_version(self): # https://github.com/buildout/buildout/pull/452 # An egg built with the version '3.3' should match a distribution # looking for '3.3.0' dist, location = self._get_distro_and_egg_path() result = zc.buildout.easy_install._get_matching_dist_in_location( dist, self.project_dist_dir ) self.assertIsNotNone(result) self.assertEqual(result.version, '3.3') def test_move_to_eggs_dir_and_compile(self): # https://github.com/buildout/buildout/pull/452 # An egg built with the version '3.3' should match a distribution # looking for '3.3.0' dist, location = self._get_distro_and_egg_path() dist.location = location dest = os.path.join(self.temp_dir, 'NewLoc') result = zc.buildout.easy_install._move_to_eggs_dir_and_compile( dist, dest ) self.assertIsNotNone(result) self.assertEqual(result.version, '3.3') if zc.buildout.WINDOWS: self.assertIn(dest.lower(), result.location) else: self.assertIn(dest, result.location) def develop_w_non_setuptools_setup_scripts(): """ We should be able to deal with setup scripts that aren't setuptools based. >>> mkdir('foo') >>> write('foo', 'setup.py', ... ''' ... from distutils.core import setup ... setup(name="foo") ... ''') >>> write('buildout.cfg', ... ''' ... [buildout] ... develop = foo ... parts = ... ''') >>> print_(system(join('bin', 'buildout')), end='') Develop: '/sample-buildout/foo' >>> ls('develop-eggs') - foo.egg-link - zc.recipe.egg.egg-link """ def develop_verbose(): """ We should be able to deal with setup scripts that aren't setuptools based. >>> mkdir('foo') >>> write('foo', 'setup.py', ... ''' ... from setuptools import setup ... setup(name="foo") ... ''') >>> write('buildout.cfg', ... ''' ... [buildout] ... develop = foo ... parts = ... ''') >>> print_(system(join('bin', 'buildout')+' -vv'), end='') ... # doctest: +ELLIPSIS Installing... Develop: '/sample-buildout/foo' ... Installed /sample-buildout/foo ... >>> ls('develop-eggs') - foo.egg-link - zc.recipe.egg.egg-link >>> print_(system(join('bin', 'buildout')+' -vvv'), end='') ... # doctest: +ELLIPSIS Installing... Develop: '/sample-buildout/foo' in: '/sample-buildout/foo' ... -q develop -mN -d /sample-buildout/develop-eggs/... """ def buildout_error_handling(): r"""Buildout error handling Asking for a section that doesn't exist, yields a missing section error: >>> import os >>> os.chdir(sample_buildout) >>> import zc.buildout.buildout >>> buildout = zc.buildout.buildout.Buildout('buildout.cfg', []) >>> buildout['eek'] Traceback (most recent call last): ... MissingSection: The referenced section, 'eek', was not defined. Asking for an option that doesn't exist, a MissingOption error is raised: >>> buildout['buildout']['eek'] Traceback (most recent call last): ... MissingOption: Missing option: buildout:eek It is an error to create a variable-reference cycle: >>> write(sample_buildout, 'buildout.cfg', ... ''' ... [buildout] ... parts = ... x = ${buildout:y} ... y = ${buildout:z} ... z = ${buildout:x} ... ''') >>> print_(system(os.path.join(sample_buildout, 'bin', 'buildout')), ... end='') ... # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS While: Initializing. Getting section buildout. Initializing section buildout. Getting option buildout:x. Getting option buildout:y. Getting option buildout:z. Getting option buildout:x. Error: Circular reference in substitutions. It is an error to use funny characters in variable references: >>> write(sample_buildout, 'buildout.cfg', ... ''' ... [buildout] ... develop = recipes ... parts = data_dir debug ... x = ${bui$ldout:y} ... ''') >>> print_(system(os.path.join(sample_buildout, 'bin', 'buildout')), ... end='') While: Initializing. Getting section buildout. Initializing section buildout. Getting option buildout:x. Error: The section name in substitution, ${bui$ldout:y}, has invalid characters. >>> write(sample_buildout, 'buildout.cfg', ... ''' ... [buildout] ... develop = recipes ... parts = data_dir debug ... x = ${buildout:y{z} ... ''') >>> print_(system(os.path.join(sample_buildout, 'bin', 'buildout')), ... end='') While: Initializing. Getting section buildout. Initializing section buildout. Getting option buildout:x. Error: The option name in substitution, ${buildout:y{z}, has invalid characters. and too have too many or too few colons: >>> write(sample_buildout, 'buildout.cfg', ... ''' ... [buildout] ... develop = recipes ... parts = data_dir debug ... x = ${parts} ... ''') >>> print_(system(os.path.join(sample_buildout, 'bin', 'buildout')), ... end='') While: Initializing. Getting section buildout. Initializing section buildout. Getting option buildout:x. Error: The substitution, ${parts}, doesn't contain a colon. >>> write(sample_buildout, 'buildout.cfg', ... ''' ... [buildout] ... develop = recipes ... parts = data_dir debug ... x = ${buildout:y:z} ... ''') >>> print_(system(os.path.join(sample_buildout, 'bin', 'buildout')), ... end='') While: Initializing. Getting section buildout. Initializing section buildout. Getting option buildout:x. Error: The substitution, ${buildout:y:z}, has too many colons. All parts have to have a section: >>> write(sample_buildout, 'buildout.cfg', ... ''' ... [buildout] ... parts = x ... ''') >>> print_(system(os.path.join(sample_buildout, 'bin', 'buildout')), ... end='') While: Installing. Getting section x. Error: The referenced section, 'x', was not defined. and all parts have to have a specified recipe: >>> write(sample_buildout, 'buildout.cfg', ... ''' ... [buildout] ... parts = x ... ... [x] ... foo = 1 ... ''') >>> print_(system(os.path.join(sample_buildout, 'bin', 'buildout')), ... end='') While: Installing. Error: Missing option: x:recipe """ make_dist_that_requires_setup_py_template = """ from setuptools import setup setup(name=%r, version=%r, install_requires=%r, ) """ def make_dist_that_requires(dest, name, requires=None, version=1, egg=''): if requires is None: requires = [] os.mkdir(os.path.join(dest, name)) with open(os.path.join(dest, name, 'setup.py'), 'w') as f: f.write( make_dist_that_requires_setup_py_template % (name, version, requires) ) def show_who_requires_when_there_is_a_conflict(): """ It's a pain when we require eggs that have requirements that are incompatible. We want the error we get to tell us what is missing. Let's make a few develop distros, some of which have incompatible requirements. >>> make_dist_that_requires(sample_buildout, 'sampley', ... ['demoneeded ==1.0']) >>> make_dist_that_requires(sample_buildout, 'samplez', ... ['demoneeded ==1.1']) Now, let's create a buildout that requires y and z: >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = eggs ... develop = sampley samplez ... find-links = %(link_server)s ... ... [eggs] ... recipe = zc.recipe.egg ... eggs = sampley ... samplez ... ''' % globals()) >>> print_(system(buildout), end='') Develop: '/sample-buildout/sampley' Develop: '/sample-buildout/samplez' Installing eggs. Getting distribution for 'demoneeded==1.1'. Got demoneeded 1.1. Version and requirements information containing demoneeded: Requirement of samplez: demoneeded==1.1 Requirement of sampley: demoneeded==1.0 While: Installing eggs. Error: There is a version conflict. We already have: demoneeded 1.1 but sampley 1 requires 'demoneeded==1.0'. Here, we see that sampley required an older version of demoneeded. What if we hadn't required sampley ourselves: >>> make_dist_that_requires(sample_buildout, 'samplea', ['sampleb']) >>> make_dist_that_requires(sample_buildout, 'sampleb', ... ['sampley', 'samplea']) >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = eggs ... develop = sampley samplez samplea sampleb ... find-links = %(link_server)s ... ... [eggs] ... recipe = zc.recipe.egg ... eggs = samplea ... samplez ... ''' % globals()) If we use the verbose switch, we can see where requirements are coming from: >>> print_(system(buildout+' -v'), end='') # doctest: +ELLIPSIS Installing 'zc.buildout', 'wheel'... Develop: '/sample-buildout/sampley' Develop: '/sample-buildout/samplez' Develop: '/sample-buildout/samplea' Develop: '/sample-buildout/sampleb' ... Installing eggs. Installing 'samplea', 'samplez'. We have a develop egg: samplea 1 We have a develop egg: samplez 1 Getting required 'demoneeded==1.1' required by samplez 1. We have the distribution that satisfies 'demoneeded==1.1'. Getting required 'sampleb' required by samplea 1. We have a develop egg: sampleb 1 Getting required 'sampley' required by sampleb 1. We have a develop egg: sampley 1 Version and requirements information containing demoneeded: Requirement of samplez: demoneeded==1.1 Requirement of sampley: demoneeded==1.0 While: Installing eggs. Error: There is a version conflict. We already have: demoneeded 1.1 but sampley 1 requires 'demoneeded==1.0'. """ def version_conflict_rendering(): """ We use the arguments passed by pkg_resources.VersionConflict to construct a nice error message: >>> error = pkg_resources.VersionConflict('pkg1 2.1', 'pkg1 1.0') >>> ws = [] # Not relevant for this test >>> print_(zc.buildout.easy_install.VersionConflict( ... error, ws)) # doctest: +ELLIPSIS There is a version conflict... But sometimes pkg_resources passes a nicely formatted string itself already. Extracting the original arguments fails in that case, so we just show the string. >>> error = pkg_resources.VersionConflict('pkg1 2.1 is simply wrong') >>> ws = [] # Not relevant for this test >>> print_(zc.buildout.easy_install.VersionConflict( ... error, ws)) # doctest: +ELLIPSIS There is a version conflict. pkg1 2.1 is simply wrong """ def show_who_requires_missing_distributions(): """ When working with a lot of eggs, which require eggs recursively, it can be hard to tell why we're requiring things we can't find. Fortunately, buildout will tell us who's asking for something that we can't find. when run in verbose mode >>> make_dist_that_requires(sample_buildout, 'sampley', ['demoneeded']) >>> make_dist_that_requires(sample_buildout, 'samplea', ['sampleb']) >>> make_dist_that_requires(sample_buildout, 'sampleb', ... ['sampley', 'samplea']) >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = eggs ... develop = sampley samplea sampleb ... ... [eggs] ... recipe = zc.recipe.egg ... eggs = samplea ... ''') >>> print_(system(buildout+' -v'), end='') # doctest: +ELLIPSIS Installing ... Installing 'samplea'. We have a develop egg: samplea 1 Getting required 'sampleb' required by samplea 1. We have a develop egg: sampleb 1 Getting required 'sampley' required by sampleb 1. We have a develop egg: sampley 1 Getting required 'demoneeded' required by sampley 1. We have no distributions for demoneeded that satisfies 'demoneeded'. ... While: Installing eggs. Getting distribution for 'demoneeded'. Error: Couldn't find a distribution for 'demoneeded'. """ def show_who_requires_picked_versions(): """ The show-picked-versions prints the versions, but it also prints who required the picked distributions. We do not need to run in verbose mode for that to work: >>> make_dist_that_requires(sample_buildout, 'sampley', ['demo']) >>> make_dist_that_requires(sample_buildout, 'samplea', ['sampleb']) >>> make_dist_that_requires(sample_buildout, 'sampleb', ... ['sampley', 'samplea']) >>> write('buildout.cfg', ... ''' ... [buildout] ... find-links = %(sample_eggs)s ... parts = eggs ... show-picked-versions = true ... develop = sampley samplea sampleb ... ... [eggs] ... recipe = zc.recipe.egg ... eggs = samplea ... ''' % globals()) >>> print_(system(buildout), end='') # doctest: +ELLIPSIS Develop: ... Versions had to be automatically picked. The following part definition lists the versions picked: [versions] # Required by: # sampley==1 demo = 0.3 # Required by: # demo==0.3 demoneeded = 1.1 """ def test_comparing_saved_options_with_funny_characters(): """ If an option has newlines, extra/odd spaces or a %, we need to make sure the comparison with the saved value works correctly. >>> mkdir(sample_buildout, 'recipes') >>> write(sample_buildout, 'recipes', 'debug.py', ... ''' ... class Debug: ... def __init__(self, buildout, name, options): ... options['debug'] = \"\"\" ... ... ... path foo ... ... ... ... \"\"\" ... options['debug1'] = \"\"\" ... ... ... ... path foo ... ... ... ... \"\"\" ... options['debug2'] = ' x ' ... options['debug3'] = '42' ... options['format'] = '%3d' ... ... def install(self): ... with open('t', 'w') as f: f.write('t') ... return 't' ... ... update = install ... ''') >>> write(sample_buildout, 'recipes', 'setup.py', ... ''' ... from setuptools import setup ... setup( ... name = "recipes", ... entry_points = {'zc.buildout': ['default = debug:Debug']}, ... ) ... ''') >>> write(sample_buildout, 'recipes', 'README.txt', " ") >>> write(sample_buildout, 'buildout.cfg', ... ''' ... [buildout] ... develop = recipes ... parts = debug ... ... [debug] ... recipe = recipes ... ''') >>> os.chdir(sample_buildout) >>> buildout = os.path.join(sample_buildout, 'bin', 'buildout') >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Installing debug. If we run the buildout again, we shouldn't get a message about uninstalling anything because the configuration hasn't changed. >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Updating debug. """ def finding_eggs_as_local_directories(): r""" It is possible to set up find-links so that we could install from a local directory that may contained unzipped eggs. >>> src = tmpdir('src') >>> write(src, 'setup.py', ... ''' ... from setuptools import setup ... setup(name='demo', py_modules=[''], ... zip_safe=False, version='1.0', author='bob', url='bob', ... author_email='bob') ... ''') >>> write(src, 't.py', '#\n') >>> write(src, 'README.txt', '') >>> _ = system(join('bin', 'buildout')+' setup ' + src + ' bdist_egg') Install it so it gets unzipped: >>> d1 = tmpdir('d1') >>> ws = zc.buildout.easy_install.install( ... ['demo'], d1, links=[join(src, 'dist')], ... ) >>> ls(d1) d demo-1.0-py2.4.egg Then try to install it again: >>> d2 = tmpdir('d2') >>> ws = zc.buildout.easy_install.install( ... ['demo'], d2, links=[d1], ... ) >>> ls(d2) d demo-1.0-py2.4.egg """ def create_sections_on_command_line(): """ >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = ... x = ${foo:bar} ... ''') >>> print_(system(buildout + ' foo:bar=1 -vv'), end='') ... # doctest: +ELLIPSIS Installing 'zc.buildout', 'wheel', 'pip', 'setuptools'. ... [foo] bar = 1 ... """ def test_help(): """ >>> print_(system(os.path.join(sample_buildout, 'bin', 'buildout')+' -h')) ... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE Usage: buildout [options] [assignments] [command [command arguments]] Options: -c config_file Specify the path to the buildout configuration file to be used. This defaults to the file named "buildout.cfg" in the current working directory. ... -h, --help ... >>> print_(system(os.path.join(sample_buildout, 'bin', 'buildout') ... +' --help')) ... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE Usage: buildout [options] [assignments] [command [command arguments]] Options: -c config_file Specify the path to the buildout configuration file to be used. This defaults to the file named "buildout.cfg" in the current working directory. ... -h, --help ... """ def test_version(): """ >>> buildout = os.path.join(sample_buildout, 'bin', 'buildout') >>> print_(system(buildout+' --version')) ... # doctest: +ELLIPSIS buildout version ... """ def test_bootstrap_with_extension(): """ We had a problem running a bootstrap with an extension. Let's make sure it is fixed. Basically, we don't load extensions when bootstrapping. >>> d = tmpdir('sample-bootstrap') >>> write(d, 'buildout.cfg', ... ''' ... [buildout] ... extensions = some_awsome_extension ... parts = ... ''') >>> os.chdir(d) >>> print_(system(os.path.join(sample_buildout, 'bin', 'buildout') ... + ' bootstrap'), end='') Creating directory '/sample-bootstrap/eggs'. Creating directory '/sample-bootstrap/bin'. Creating directory '/sample-bootstrap/parts'. Creating directory '/sample-bootstrap/develop-eggs'. Generated script '/sample-bootstrap/bin/buildout'. """ def bug_92891_bootstrap_crashes_with_egg_recipe_in_buildout_section(): """ >>> d = tmpdir('sample-bootstrap') >>> write(d, 'buildout.cfg', ... ''' ... [buildout] ... parts = buildout ... eggs-directory = eggs ... ... [buildout] ... recipe = zc.recipe.egg ... eggs = zc.buildout ... scripts = buildout=buildout ... ''') >>> os.chdir(d) >>> print_(system(os.path.join(sample_buildout, 'bin', 'buildout') ... + ' bootstrap'), end='') Creating directory '/sample-bootstrap/eggs'. Creating directory '/sample-bootstrap/bin'. Creating directory '/sample-bootstrap/parts'. Creating directory '/sample-bootstrap/develop-eggs'. Generated script '/sample-bootstrap/bin/buildout'. >>> print_(system(os.path.join('bin', 'buildout')), end='') Section `buildout` contains unused option(s): 'eggs' 'scripts'. This may be an indication for either a typo in the option's name or a bug in the used recipe. """ def removing_eggs_from_develop_section_causes_egg_link_to_be_removed(): ''' >>> cd(sample_buildout) Create a develop egg: >>> mkdir('foo') >>> write('foo', 'setup.py', ... """ ... from setuptools import setup ... setup(name='foox') ... """) >>> write('buildout.cfg', ... """ ... [buildout] ... develop = foo ... parts = ... """) >>> print_(system(join('bin', 'buildout')), end='') Develop: '/sample-buildout/foo' >>> ls('develop-eggs') - foox.egg-link - zc.recipe.egg.egg-link Create another: >>> mkdir('bar') >>> write('bar', 'setup.py', ... """ ... from setuptools import setup ... setup(name='fooy') ... """) >>> write('buildout.cfg', ... """ ... [buildout] ... develop = foo bar ... parts = ... """) >>> print_(system(join('bin', 'buildout')), end='') Develop: '/sample-buildout/foo' Develop: '/sample-buildout/bar' >>> ls('develop-eggs') - foox.egg-link - fooy.egg-link - zc.recipe.egg.egg-link Remove one: >>> write('buildout.cfg', ... """ ... [buildout] ... develop = bar ... parts = ... """) >>> print_(system(join('bin', 'buildout')), end='') Develop: '/sample-buildout/bar' It is gone >>> ls('develop-eggs') - fooy.egg-link - zc.recipe.egg.egg-link Remove the other: >>> write('buildout.cfg', ... """ ... [buildout] ... parts = ... """) >>> print_(system(join('bin', 'buildout')), end='') All gone >>> ls('develop-eggs') - zc.recipe.egg.egg-link ''' def add_setuptools_to_dependencies_when_namespace_packages(): ''' Often, a package depends on setuptools solely by virtue of using namespace packages. In this situation, package authors often forget to declare setuptools as a dependency. This is a mistake, but, unfortunately, a common one that we need to work around. If an egg uses namespace packages and does not include setuptools as a dependency, we will still include setuptools in the working set. If we see this for a develop egg, we will also generate a warning. >>> mkdir('foo') >>> mkdir('foo', 'src') >>> mkdir('foo', 'src', 'stuff') >>> write('foo', 'src', 'stuff', '__init__.py', ... """__import__('pkg_resources').declare_namespace(__name__) ... """) >>> mkdir('foo', 'src', 'stuff', 'foox') >>> write('foo', 'src', 'stuff', 'foox', '__init__.py', '') >>> write('foo', 'setup.py', ... """ ... from setuptools import setup ... setup(name='foox', ... namespace_packages = ['stuff'], ... package_dir = {'': 'src'}, ... packages = ['stuff', 'stuff.foox'], ... ) ... """) >>> write('foo', 'README.txt', '') >>> write('buildout.cfg', ... """ ... [buildout] ... develop = foo ... parts = ... """) >>> print_(system(join('bin', 'buildout')), end='') Develop: '/sample-buildout/foo' Now, if we generate a working set using the egg link, we will get a warning and we will get setuptools included in the working set. >>> import logging, zope.testing.loggingsupport >>> handler = zope.testing.loggingsupport.InstalledHandler( ... 'zc.buildout.easy_install', level=logging.WARNING) >>> logging.getLogger('zc.buildout.easy_install').propagate = False >>> def get_working_set(*project_names): ... paths = [join(sample_buildout, 'eggs'), ... join(sample_buildout, 'develop-eggs')] ... return [ ... dist.project_name ... for dist in zc.buildout.easy_install.working_set( ... project_names, sys.executable, paths) ... ] >>> get_working_set('foox') ['foox', 'setuptools'] >>> print_(handler) zc.buildout.easy_install WARNING Develop distribution: foox 0.0.0 uses namespace packages but the distribution does not require setuptools. >>> handler.clear() On the other hand, if we have a zipped egg, rather than a develop egg: >>> os.remove(join('develop-eggs', 'foox.egg-link')) >>> _ = system(join('bin', 'buildout') + ' setup foo bdist_egg') >>> foox_dist = join('foo', 'dist') >>> import glob >>> [foox_egg] = glob.glob(join(foox_dist, 'foox-*.egg')) >>> _ = shutil.copy(foox_egg, join(sample_buildout, 'eggs')) >>> ls('develop-eggs') - zc.recipe.egg.egg-link >>> ls('eggs') # doctest: +ELLIPSIS - foox-0.0.0-py2.4.egg - packaging.egg-link - pip.egg-link - setuptools.egg-link - wheel.egg-link - zc.buildout.egg-link We do not get a warning, but we do get setuptools included in the working set: >>> get_working_set('foox') ['foox', 'setuptools'] >>> print_(handler, end='') Likewise for an unzipped egg: >>> foox_egg_basename = os.path.basename(foox_egg) >>> os.remove(join(sample_buildout, 'eggs', foox_egg_basename)) >>> _ = zc.buildout.easy_install.install( ... ['foox'], join(sample_buildout, 'eggs'), links=[foox_dist], ... index='file://' + foox_dist) >>> ls('develop-eggs') - zc.recipe.egg.egg-link >>> get_working_set('foox') ['foox', 'setuptools'] >>> print_(handler, end='') We get the same behavior if it is a dependency that uses a namespace package. >>> mkdir('bar') >>> write('bar', 'setup.py', ... """ ... from setuptools import setup ... setup(name='bar', install_requires = ['foox']) ... """) >>> write('bar', 'README.txt', '') >>> write('buildout.cfg', ... """ ... [buildout] ... develop = foo bar ... parts = ... """) >>> print_(system(join('bin', 'buildout')), end='') Develop: '/sample-buildout/foo' Develop: '/sample-buildout/bar' >>> get_working_set('bar') ['bar', 'foox', 'setuptools'] >>> print_(handler, end='') zc.buildout.easy_install WARNING Develop distribution: foox 0.0.0 uses namespace packages but the distribution does not require setuptools. On the other hand, if the distribution uses ``pkgutil.extend_path()`` to implement its namespaces, even if just as fallback from the absence of ``pkg_resources``, then ``setuptools`` should not be added as requirement to its unzipped egg: >>> foox_installed_egg = join(sample_buildout, 'eggs', foox_egg_basename) >>> namespace_init = join(foox_installed_egg, 'stuff', '__init__.py') >>> write(namespace_init, ... """try: ... __import__('pkg_resources').declare_namespace(__name__) ... except ImportError: ... __path__ = __import__('pkgutil').extend_path(__path__, __name__) ... """) >>> os.remove(join('develop-eggs', 'foox.egg-link')) >>> os.remove(join('develop-eggs', 'bar.egg-link')) >>> get_working_set('foox') ['foox'] The same goes for packages using PEP420 namespaces >>> os.remove(namespace_init) >>> get_working_set('foox') ['foox'] Cleanup: >>> logging.getLogger('zc.buildout.easy_install').propagate = True >>> handler.uninstall() ''' def develop_preserves_existing_setup_cfg(): """ See "Handling custom build options for extensions in develop eggs" in easy_install.txt. This will be very similar except that we'll have an existing setup.cfg: >>> write(extdemo, "setup.cfg", ... ''' ... # sampe cfg file ... ... [foo] ... bar = 1 ... ... [build_ext] ... define = X,Y ... ''') >>> mkdir('include') >>> write('include', 'extdemo.h', ... ''' ... #define EXTDEMO 42 ... ''') >>> dest = tmpdir('dest') >>> zc.buildout.easy_install.develop( ... extdemo, dest, ... {'include-dirs': os.path.join(sample_buildout, 'include')}) '/dest/extdemo.egg-link' >>> ls(dest) - extdemo.egg-link >>> cat(extdemo, "setup.cfg") # sampe cfg file [foo] bar = 1 [build_ext] define = X,Y """ def uninstall_recipes_used_for_removal(): r""" Uninstall recipes need to be called when a part is removed too: >>> mkdir("recipes") >>> write("recipes", "setup.py", ... ''' ... from setuptools import setup ... setup(name='recipes', ... entry_points={ ... 'zc.buildout': ["demo=demo:Install"], ... 'zc.buildout.uninstall': ["demo=demo:uninstall"], ... }) ... ''') >>> write("recipes", "demo.py", ... r''' ... import sys ... class Install: ... def __init__(*args): pass ... def install(self): ... sys.stdout.write('installing\n') ... return () ... def uninstall(name, options): ... sys.stdout.write('uninstalling\n') ... ''') >>> write('buildout.cfg', ''' ... [buildout] ... develop = recipes ... parts = demo ... [demo] ... recipe = recipes:demo ... ''') >>> print_(system(join('bin', 'buildout')), end='') Develop: '/sample-buildout/recipes' Installing demo. installing >>> write('buildout.cfg', ''' ... [buildout] ... develop = recipes ... parts = demo ... [demo] ... recipe = recipes:demo ... x = 1 ... ''') >>> print_(system(join('bin', 'buildout')), end='') Develop: '/sample-buildout/recipes' Uninstalling demo. Running uninstall recipe. uninstalling Installing demo. installing >>> write('buildout.cfg', ''' ... [buildout] ... develop = recipes ... parts = ... ''') >>> print_(system(join('bin', 'buildout')), end='') Develop: '/sample-buildout/recipes' Uninstalling demo. Running uninstall recipe. uninstalling """ def extensions_installed_as_eggs_work_in_offline_mode(): ''' >>> mkdir('demo') >>> write('demo', 'demo.py', ... r""" ... import sys ... def print_(*args): ... sys.stdout.write(' '.join(map(str, args)) + '\\n') ... def ext(buildout): ... print_('ext', sorted(buildout)) ... """) >>> write('demo', 'setup.py', ... """ ... from setuptools import setup ... ... setup( ... name = "demo", ... py_modules=['demo'], ... entry_points = {'zc.buildout.extension': ['ext = demo:ext']}, ... ) ... """) >>> bdist_egg(join(sample_buildout, "demo"), sys.executable, ... join(sample_buildout, "eggs")) >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... extensions = demo ... parts = ... offline = true ... """) >>> print_(system(join(sample_buildout, 'bin', 'buildout')), end='') ext ['buildout', 'versions'] ''' def changes_in_svn_or_git_dont_affect_sig(): """ If we have a develop recipe, it's signature shouldn't be affected to changes in .git, .svn directories. CVS directories used to work as well but do not anymore. >>> mkdir('recipe') >>> write('recipe', 'setup.py', ... ''' ... from setuptools import setup ... setup(name='recipe', ... entry_points={'zc.buildout': ['default=foo:Foo']}) ... ''') >>> write('recipe', 'foo.py', ... ''' ... class Foo: ... def __init__(*args): pass ... def install(*args): return () ... update = install ... ''') >>> write('buildout.cfg', ... ''' ... [buildout] ... develop = recipe ... parts = foo ... ... [foo] ... recipe = recipe ... ''') >>> print_(system(join(sample_buildout, 'bin', 'buildout')), end='') Develop: '/sample-buildout/recipe' Installing foo. >>> mkdir('recipe', '.git') >>> mkdir('recipe', '.svn') >>> # deprecated CVS mkdir('recipe', 'CVS') >>> print_(system(join(sample_buildout, 'bin', 'buildout')), end='') Develop: '/sample-buildout/recipe' Updating foo. >>> write('recipe', '.git', 'x', '1') >>> write('recipe', '.svn', 'x', '1') >>> # deprecated CVS write('recipe', 'CVS', 'x', '1') >>> print_(system(join(sample_buildout, 'bin', 'buildout')), end='') Develop: '/sample-buildout/recipe' Updating foo. """ if hasattr(os, 'symlink'): def bug_250537_broken_symlink_doesnt_affect_sig(): """ If we have a develop recipe, it's signature shouldn't be affected by broken symlinks, and better yet, computing the hash should not break because of the missing target file. >>> mkdir('recipe') >>> write('recipe', 'setup.py', ... ''' ... from setuptools import setup ... setup(name='recipe', ... entry_points={'zc.buildout': ['default=foo:Foo']}) ... ''') >>> write('recipe', 'foo.py', ... ''' ... class Foo: ... def __init__(*args): pass ... def install(*args): return () ... update = install ... ''') >>> write('buildout.cfg', ... ''' ... [buildout] ... develop = recipe ... parts = foo ... ... [foo] ... recipe = recipe ... ''') >>> print_(system(join(sample_buildout, 'bin', 'buildout')), end='') Develop: '/sample-buildout/recipe' Installing foo. >>> write('recipe', 'some-file', '1') >>> os.symlink(join('recipe', 'some-file'), ... join('recipe', 'another-file')) >>> remove('recipe', 'some-file') >>> print_(system(join(sample_buildout, 'bin', 'buildout')), end='') Develop: '/sample-buildout/recipe' Updating foo. """ def unicode_filename_doesnt_break_hash(): """ Buildout's _dir_hash() used to break on non-ascii filenames on python 2. >>> mkdir('héhé') >>> write('héhé', 'héhé.py', ... ''' ... print('Example filename from pyramid tests') ... ''') >>> from zc.buildout.buildout import _dir_hash >>> dont_care = _dir_hash('héhé') """ def o_option_sets_offline(): """ >>> print_(system(join(sample_buildout, 'bin', 'buildout')+' -vvo'), end='') ... # doctest: +ELLIPSIS ... offline = true ... """ def recipe_upgrade(): r""" The buildout will upgrade recipes in newest (and non-offline) mode. Let's create a recipe egg >>> mkdir('recipe') >>> write('recipe', 'recipe.py', ... r''' ... import sys ... class Recipe: ... def __init__(*a): pass ... def install(self): ... sys.stdout.write('recipe v1\n') ... return () ... update = install ... ''') >>> write('recipe', 'setup.py', ... ''' ... from setuptools import setup ... setup(name='recipe', version='1', py_modules=['recipe'], ... entry_points={'zc.buildout': ['default = recipe:Recipe']}, ... ) ... ''') >>> write('recipe', 'README', '') >>> print_(system(buildout+' setup recipe bdist_egg')) # doctest: +ELLIPSIS Running setup script 'recipe/setup.py'. ... >>> rmdir('recipe', 'build') And update our buildout to use it. >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = foo ... find-links = %s ... ... [foo] ... recipe = recipe ... ''' % join('recipe', 'dist')) >>> print_(system(buildout), end='') Getting distribution for 'recipe'. Got recipe 1. Installing foo. recipe v1 Now, if we update the recipe egg: >>> write('recipe', 'recipe.py', ... r''' ... import sys ... class Recipe: ... def __init__(*a): pass ... def install(self): ... sys.stdout.write('recipe v2\n') ... return () ... update = install ... ''') >>> write('recipe', 'setup.py', ... ''' ... from setuptools import setup ... setup(name='recipe', version='2', py_modules=['recipe'], ... entry_points={'zc.buildout': ['default = recipe:Recipe']}, ... ) ... ''') >>> print_(system(buildout+' setup recipe bdist_egg')) # doctest: +ELLIPSIS Running setup script 'recipe/setup.py'. ... We won't get the update if we specify -N: >>> print_(system(buildout+' -N'), end='') Updating foo. recipe v1 or if we use -o: >>> print_(system(buildout+' -o'), end='') Updating foo. recipe v1 But we will if we use neither of these: >>> print_(system(buildout), end='') Getting distribution for 'recipe'. Got recipe 2. Uninstalling foo. Installing foo. recipe v2 We can also select a particular recipe version: >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = foo ... find-links = %s ... ... [foo] ... recipe = recipe ==1 ... ''' % join('recipe', 'dist')) >>> print_(system(buildout), end='') Uninstalling foo. Installing foo. recipe v1 """ def update_adds_to_uninstall_list(): """ Paths returned by the update method are added to the list of paths to uninstall >>> mkdir('recipe') >>> write('recipe', 'setup.py', ... ''' ... from setuptools import setup ... setup(name='recipe', ... entry_points={'zc.buildout': ['default = recipe:Recipe']}, ... ) ... ''') >>> write('recipe', 'recipe.py', ... ''' ... import os ... class Recipe: ... def __init__(*_): pass ... def install(self): ... r = ('a', 'b', 'c') ... for p in r: os.mkdir(p) ... return r ... def update(self): ... r = ('c', 'd', 'e') ... for p in r: ... if not os.path.exists(p): ... os.mkdir(p) ... return r ... ''') >>> write('buildout.cfg', ... ''' ... [buildout] ... develop = recipe ... parts = foo ... ... [foo] ... recipe = recipe ... ''') >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipe' Installing foo. >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipe' Updating foo. >>> cat('.installed.cfg') # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE [buildout] ... [foo] __buildout_installed__ = a b c d e __buildout_signature__ = ... """ def log_when_there_are_not_local_distros(): """ >>> from zope.testing.loggingsupport import InstalledHandler >>> handler = InstalledHandler('zc.buildout.easy_install') >>> import logging >>> logger = logging.getLogger('zc.buildout.easy_install') >>> old_propogate = logger.propagate >>> logger.propagate = False >>> dest = tmpdir('sample-install') >>> import zc.buildout.easy_install >>> ws = zc.buildout.easy_install.install( ... ['demo==0.2'], dest, ... links=[link_server], index=link_server+'index/') >>> print_(handler) # doctest: +ELLIPSIS zc.buildout.easy_install DEBUG Installing 'demo==0.2'. zc.buildout.easy_install DEBUG We have no distributions for demo that satisfies 'demo==0.2'. ... >>> handler.uninstall() >>> logger.propagate = old_propogate """ def internal_errors(): """Internal errors are clearly marked and don't generate tracebacks: >>> mkdir(sample_buildout, 'recipes') >>> write(sample_buildout, 'recipes', 'mkdir.py', ... ''' ... class Mkdir: ... def __init__(self, buildout, name, options): ... self.name, self.options = name, options ... options['path'] = os.path.join( ... buildout['buildout']['directory'], ... options['path'], ... ) ... ''') >>> write(sample_buildout, 'recipes', 'setup.py', ... ''' ... from setuptools import setup ... setup(name = "recipes", ... entry_points = {'zc.buildout': ['mkdir = mkdir:Mkdir']}, ... ) ... ''') >>> write(sample_buildout, 'buildout.cfg', ... ''' ... [buildout] ... develop = recipes ... parts = data-dir ... ... [data-dir] ... recipe = recipes:mkdir ... ''') >>> print_(system(buildout), end='') # doctest: +ELLIPSIS Develop: '/sample-buildout/recipes' While: Installing. Getting section data-dir. Initializing section data-dir. An internal error occurred due to a bug in either zc.buildout or in a recipe being used: Traceback (most recent call last): ... NameError: global name 'os' is not defined... """ def whine_about_unused_options(): ''' >>> write('foo.py', ... """ ... class Foo: ... ... def __init__(self, buildout, name, options): ... self.name, self.options = name, options ... options['x'] ... ... def install(self): ... self.options['y'] ... return () ... """) >>> write('setup.py', ... """ ... from setuptools import setup ... setup(name = "foo", ... py_modules=['foo'], ... entry_points = {'zc.buildout': ['default = foo:Foo']}, ... ) ... """) >>> write('buildout.cfg', ... """ ... [buildout] ... develop = . ... parts = foo ... a = 1 ... ... [foo] ... recipe = foo ... x = 1 ... y = 1 ... z = 1 ... """) >>> print_(system(buildout), end='') Develop: '/sample-buildout/.' Section `buildout` contains unused option(s): 'a'. This may be an indication for either a typo in the option's name or a bug in the used recipe. Installing foo. Section `foo` contains unused option(s): 'z'. This may be an indication for either a typo in the option's name or a bug in the used recipe. ''' def abnormal_exit(): """ People sometimes hit control-c while running a builout. We need to make sure that the installed database Isn't corrupted. To test this, we'll create some evil recipes that exit uncleanly: >>> mkdir('recipes') >>> write('recipes', 'recipes.py', ... ''' ... import os ... ... class Clean: ... def __init__(*_): pass ... def install(_): return () ... def update(_): pass ... ... class EvilInstall(Clean): ... def install(_): os._exit(1) ... ... class EvilUpdate(Clean): ... def update(_): os._exit(1) ... ''') >>> write('recipes', 'setup.py', ... ''' ... import setuptools ... setuptools.setup(name='recipes', ... entry_points = { ... 'zc.buildout': [ ... 'clean = recipes:Clean', ... 'evil_install = recipes:EvilInstall', ... 'evil_update = recipes:EvilUpdate', ... 'evil_uninstall = recipes:Clean', ... ], ... }, ... ) ... ''') Now let's look at 3 cases: 1. We exit during installation after installing some other parts: >>> write('buildout.cfg', ... ''' ... [buildout] ... develop = recipes ... parts = p1 p2 p3 p4 ... ... [p1] ... recipe = recipes:clean ... ... [p2] ... recipe = recipes:clean ... ... [p3] ... recipe = recipes:evil_install ... ... [p4] ... recipe = recipes:clean ... ''') >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Installing p1. Installing p2. Installing p3. >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Updating p1. Updating p2. Installing p3. >>> print_(system(buildout+' buildout:parts='), end='') Develop: '/sample-buildout/recipes' Uninstalling p2. Uninstalling p1. 2. We exit while updating: >>> write('buildout.cfg', ... ''' ... [buildout] ... develop = recipes ... parts = p1 p2 p3 p4 ... ... [p1] ... recipe = recipes:clean ... ... [p2] ... recipe = recipes:clean ... ... [p3] ... recipe = recipes:evil_update ... ... [p4] ... recipe = recipes:clean ... ''') >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Installing p1. Installing p2. Installing p3. Installing p4. >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Updating p1. Updating p2. Updating p3. >>> print_(system(buildout+' buildout:parts='), end='') Develop: '/sample-buildout/recipes' Uninstalling p2. Uninstalling p1. Uninstalling p4. Uninstalling p3. 3. We exit while installing or updating after uninstalling: >>> write('buildout.cfg', ... ''' ... [buildout] ... develop = recipes ... parts = p1 p2 p3 p4 ... ... [p1] ... recipe = recipes:evil_update ... ... [p2] ... recipe = recipes:clean ... ... [p3] ... recipe = recipes:clean ... ... [p4] ... recipe = recipes:clean ... ''') >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Installing p1. Installing p2. Installing p3. Installing p4. >>> write('buildout.cfg', ... ''' ... [buildout] ... develop = recipes ... parts = p1 p2 p3 p4 ... ... [p1] ... recipe = recipes:evil_update ... ... [p2] ... recipe = recipes:clean ... ... [p3] ... recipe = recipes:clean ... ... [p4] ... recipe = recipes:clean ... x = 1 ... ''') >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Uninstalling p4. Updating p1. >>> write('buildout.cfg', ... ''' ... [buildout] ... develop = recipes ... parts = p1 p2 p3 p4 ... ... [p1] ... recipe = recipes:clean ... ... [p2] ... recipe = recipes:clean ... ... [p3] ... recipe = recipes:clean ... ... [p4] ... recipe = recipes:clean ... ''') >>> print_(system(buildout), end='') Develop: '/sample-buildout/recipes' Uninstalling p1. Installing p1. Updating p2. Updating p3. Installing p4. """ def install_source_dist_with_bad_py(): r""" >>> mkdir('badegg') >>> mkdir('badegg', 'badegg') >>> write('badegg', 'badegg', '__init__.py', '#\\n') >>> mkdir('badegg', 'badegg', 'scripts') >>> write('badegg', 'badegg', 'scripts', '__init__.py', '#\\n') >>> write('badegg', 'badegg', 'scripts', 'one.py', ... ''' ... return 1 ... ''') >>> write('badegg', 'setup.py', ... ''' ... from setuptools import setup, find_packages ... setup( ... name='badegg', ... version='1', ... packages = find_packages('.'), ... zip_safe=False) ... ''') >>> print_(system(buildout+' setup badegg sdist'), end='') # doctest: +ELLIPSIS Running setup script 'badegg/setup.py'. ... >>> dist = join('badegg', 'dist') >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = eggs bo ... find-links = %(dist)s ... ... [eggs] ... recipe = zc.recipe.egg ... eggs = badegg ... ... [bo] ... recipe = zc.recipe.egg ... eggs = zc.buildout ... scripts = buildout=bo ... ''' % globals()) >>> print_(system(buildout));print_('X') # doctest: +ELLIPSIS Installing eggs. Getting distribution for 'badegg'. Got badegg 1. Installing bo. Generated script '/sample-buildout/bin/bo'. X >>> ls('eggs') # doctest: +ELLIPSIS d badegg-1-py2.4.egg ... >>> ls('bin') - bo - buildout """ def version_requirements_in_build_honored(): ''' >>> update_extdemo() >>> dest = tmpdir('sample-install') >>> mkdir('include') >>> write('include', 'extdemo.h', ... """ ... #define EXTDEMO 42 ... """) >>> zc.buildout.easy_install.build( ... 'extdemo ==1.4', dest, ... {'include-dirs': os.path.join(sample_buildout, 'include')}, ... links=[link_server], index=link_server+'index/', ... newest=False) ['/sample-install/extdemo-1.4-py2.4-linux-i686.egg'] ''' def bug_105081_Specific_egg_versions_are_ignored_when_newer_eggs_are_around(): """ Buildout might ignore a specific egg requirement for a recipe: - Have a newer version of an egg in your eggs directory - Use 'recipe==olderversion' in your buildout.cfg to request an older version Buildout will go and fetch the older version, but it will *use* the newer version when installing a part with this recipe. >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = x ... find-links = %(sample_eggs)s ... ... [x] ... recipe = zc.recipe.egg ... eggs = demo ... ''' % globals()) >>> print_(system(buildout), end='') Installing x. Getting distribution for 'demo'. Got demo 0.3. Getting distribution for 'demoneeded'. Got demoneeded 1.1. Generated script '/sample-buildout/bin/demo'. >>> print_(system(join('bin', 'demo')), end='') 3 1 >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = x ... find-links = %(sample_eggs)s ... ... [x] ... recipe = zc.recipe.egg ... eggs = demo ==0.1 ... ''' % globals()) >>> print_(system(buildout), end='') Uninstalling x. Installing x. Getting distribution for 'demo==0.1'. Got demo 0.1. Generated script '/sample-buildout/bin/demo'. >>> print_(system(join('bin', 'demo')), end='') 1 1 """ def test_exit_codes(): """ >>> import subprocess >>> def call(s): ... p = subprocess.Popen(s, stdin=subprocess.PIPE, ... stdout=subprocess.PIPE, stderr=subprocess.STDOUT) ... p.stdin.close() ... print_(p.stdout.read().decode()) ... print_('Exit:', bool(p.wait())) ... p.stdout.close() >>> call(buildout) Exit: False >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = x ... ''') >>> call(buildout) # doctest: +NORMALIZE_WHITESPACE While: Installing. Getting section x. Error: The referenced section, 'x', was not defined. Exit: True >>> write('setup.py', ... ''' ... from setuptools import setup ... setup(name='zc.buildout.testexit', ... py_modules=['testexitrecipe'], ... entry_points={'zc.buildout': ['default = testexitrecipe:x']}) ... ''') >>> write('testexitrecipe.py', ... ''' ... x y ... ''') >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = x ... develop = . ... ... [x] ... recipe = zc.buildout.testexit ... ''') >>> call(buildout) # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS Develop: '/sample-buildout/.' While: Installing. Getting section x. Initializing section x. Loading zc.buildout recipe entry zc.buildout.testexit:default. An internal error occurred due to a bug in either zc.buildout or in a recipe being used: Traceback (most recent call last): ... x y ...^... SyntaxError... Exit: True """ def bug_59270_recipes_always_start_in_buildout_dir(): r""" Recipes can rely on running from buildout directory >>> mkdir('bad_start') >>> write('bad_recipe.py', ... r''' ... import os, sys ... def print_(*args): ... sys.stdout.write(' '.join(map(str, args)) + '\n') ... class Bad: ... def __init__(self, *_): ... print_(os.getcwd()) ... def install(self): ... sys.stdout.write(os.getcwd()+'\n') ... os.chdir('bad_start') ... sys.stdout.write(os.getcwd()+'\n') ... return () ... ''') >>> write('setup.py', ... ''' ... from setuptools import setup ... setup(name='bad.test', ... py_modules=['bad_recipe'], ... entry_points={'zc.buildout': ['default=bad_recipe:Bad']},) ... ''') >>> write('buildout.cfg', ... ''' ... [buildout] ... develop = . ... parts = b1 b2 ... [b1] ... recipe = bad.test ... [b2] ... recipe = bad.test ... ''') >>> os.chdir('bad_start') >>> print_(system(join(sample_buildout, 'bin', 'buildout') ... +' -c '+join(sample_buildout, 'buildout.cfg')), end='') Develop: '/sample-buildout/.' /sample-buildout /sample-buildout Installing b1. /sample-buildout /sample-buildout/bad_start Installing b2. /sample-buildout /sample-buildout/bad_start """ def bug_61890_file_urls_dont_seem_to_work_in_find_dash_links(): """ This bug arises from the fact that setuptools is overly restrictive about file urls, requiring that file urls pointing at directories must end in a slash. >>> dest = tmpdir('sample-install') >>> import zc.buildout.easy_install >>> sample_eggs = sample_eggs.replace(os.path.sep, '/') >>> ws = zc.buildout.easy_install.install( ... ['demo==0.2'], dest, ... links=['file://'+sample_eggs], index=link_server+'index/') >>> for dist in ws: ... print_(dist) demoneeded 1.1 demo 0.2 >>> ls(dest) d demo-0.2-py2.4.egg d demoneeded-1.1-py2.4.egg """ def bug_75607_buildout_should_not_run_if_it_creates_an_empty_buildout_cfg(): """ >>> remove('buildout.cfg') >>> print_(system(buildout), end='') While: Initializing. Error: Couldn't open /sample-buildout/buildout.cfg """ def dealing_with_extremely_insane_dependencies(): r""" There was a problem with analysis of dependencies taking a long time, in part because the analysis would get repeated every time a package was encountered in a dependency list. Now, we don't do the analysis any more: >>> import os >>> for i in range(5): ... p = 'pack%s' % i ... deps = [('pack%s' % j) for j in range(5) if j is not i] ... if i == 4: ... deps.append('pack5') ... mkdir(p) ... write(p, 'setup.py', ... 'from setuptools import setup\n' ... 'setup(name=%r, install_requires=%r,\n' ... ' url="u", author="a", author_email="e")\n' ... % (p, deps)) >>> write('buildout.cfg', ... ''' ... [buildout] ... develop = pack0 pack1 pack2 pack3 pack4 ... parts = pack1 ... ... [pack1] ... recipe = zc.recipe.egg:eggs ... eggs = pack0 ... ''') >>> print_(system(buildout), end='') # doctest: +ELLIPSIS Develop: '/sample-buildout/pack0' Develop: '/sample-buildout/pack1' Develop: '/sample-buildout/pack2' Develop: '/sample-buildout/pack3' Develop: '/sample-buildout/pack4' Installing pack1. ... While: Installing pack1. Getting distribution for 'pack5'. Error: Couldn't find a distribution for 'pack5'. However, if we run in verbose mode, we can see why packages were included: >>> print_(system(buildout+' -v'), end='') # doctest: +ELLIPSIS Installing 'zc.buildout', 'wheel', 'pip', 'setuptools'. ... Develop: '/sample-buildout/pack0' Develop: '/sample-buildout/pack1' Develop: '/sample-buildout/pack2' Develop: '/sample-buildout/pack3' Develop: '/sample-buildout/pack4' ... Installing pack1. Installing 'pack0'. We have a develop egg: pack0 0.0.0 Getting required 'pack4' required by pack0 0.0.0. We have a develop egg: pack4 0.0.0 Getting required 'pack3' required by pack0 0.0.0. required by pack4 0.0.0. We have a develop egg: pack3 0.0.0 Getting required 'pack2' required by pack0 0.0.0. required by pack3 0.0.0. required by pack4 0.0.0. We have a develop egg: pack2 0.0.0 Getting required 'pack1' required by pack0 0.0.0. required by pack2 0.0.0. required by pack3 0.0.0. required by pack4 0.0.0. We have a develop egg: pack1 0.0.0 Getting required 'pack5' required by pack4 0.0.0. We have no distributions for pack5 that satisfies 'pack5'. ... While: Installing pack1. Getting distribution for 'pack5'. Error: Couldn't find a distribution for 'pack5'. """ def read_find_links_to_load_extensions(): r""" We'll create a wacky buildout extension that just announces itself when used: >>> src = tmpdir('src') >>> write(src, 'wacky_handler.py', ... ''' ... import sys ... def install(buildout=None): ... sys.stdout.write("I am a wacky extension\\n") ... ''') >>> write(src, 'setup.py', ... ''' ... from setuptools import setup ... setup(name='wackyextension', version='1', ... py_modules=['wacky_handler'], ... entry_points = {'zc.buildout.extension': ... ['default = wacky_handler:install'] ... }, ... ) ... ''') >>> print_(system(buildout+' setup '+src+' bdist_egg'), end='') ... # doctest: +ELLIPSIS Running setup ... creating 'dist/wackyextension-1-... Now we'll create a buildout that uses this extension to load other packages: >>> dist = 'file://' + join(src, 'dist').replace(os.path.sep, '/') >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = ... extensions = wackyextension ... find-links = %(dist)s ... ''' % globals()) When we run the buildout. it will load the extension from the dist directory and then use the wacky extension to load the demo package >>> print_(system(buildout), end='') Getting distribution for 'wackyextension'. Got wackyextension 1. I am a wacky extension """ def distributions_from_local_find_links_make_it_to_download_cache(): """ If we specify a local directory in find links, distros found there need to make it to the download cache. >>> mkdir('test') >>> write('test', 'setup.py', ... ''' ... from setuptools import setup ... setup(name='foo') ... ''') >>> print_(system(buildout+' setup test bdist_egg')) # doctest: +ELLIPSIS Running setup script 'test/setup.py'. ... >>> mkdir('cache') >>> old_cache = zc.buildout.easy_install.download_cache('cache') >>> list(zc.buildout.easy_install.install(['foo'], 'eggs', ... links=[join('test', 'dist')])) # doctest: +ELLIPSIS [foo 0.0.0 ... >>> ls('cache') - foo-0.0.0-py2.4.egg >>> _ = zc.buildout.easy_install.download_cache(old_cache) """ def prefer_final_permutation(existing, available): for d in ('existing', 'available'): if os.path.exists(d): shutil.rmtree(d) os.mkdir(d) for version in existing: create_egg('spam', version, 'existing') for version in available: create_egg('spam', version, 'available') zc.buildout.easy_install.clear_index_cache() [dist] = list( zc.buildout.easy_install.install(['spam'], 'existing', ['available']) ) if dist.extras: print_('downloaded', dist.version) else: print_('had', dist.version) sys.path_importer_cache.clear() def prefer_final(): """ This test tests several permutations: Using different version numbers to work around zip importer cache problems. :( - With prefer final: - no existing and newer dev available >>> prefer_final_permutation((), [1, '2a1']) downloaded 1 - no existing and only dev available >>> prefer_final_permutation((), ['3a1']) downloaded 3a1 - final existing and only dev acailable >>> prefer_final_permutation([4], ['5a1']) had 4 - final existing and newer final available >>> prefer_final_permutation([6], [7]) downloaded 7 - final existing and same final available >>> prefer_final_permutation([8], [8]) had 8 - final existing and older final available >>> prefer_final_permutation([10], [9]) had 10 - only dev existing and final available >>> prefer_final_permutation(['12a1'], [11]) downloaded 11 - only dev existing and no final available newer dev available >>> prefer_final_permutation(['13a1'], ['13a2']) downloaded 13a2 - only dev existing and no final available older dev available >>> prefer_final_permutation(['15a1'], ['14a1']) had 15a1 - only dev existing and no final available same dev available >>> prefer_final_permutation(['16a1'], ['16a1']) had 16a1 - Without prefer final: >>> _ = zc.buildout.easy_install.prefer_final(False) - no existing and newer dev available >>> prefer_final_permutation((), [18, '19a1']) downloaded 19a1 - no existing and only dev available >>> prefer_final_permutation((), ['20a1']) downloaded 20a1 - final existing and only dev acailable >>> prefer_final_permutation([21], ['22a1']) downloaded 22a1 - final existing and newer final available >>> prefer_final_permutation([23], [24]) downloaded 24 - final existing and same final available >>> prefer_final_permutation([25], [25]) had 25 - final existing and older final available >>> prefer_final_permutation([27], [26]) had 27 - only dev existing and final available >>> prefer_final_permutation(['29a1'], [28]) had 29a1 - only dev existing and no final available newer dev available >>> prefer_final_permutation(['30a1'], ['30a2']) downloaded 30a2 - only dev existing and no final available older dev available >>> prefer_final_permutation(['32a1'], ['31a1']) had 32a1 - only dev existing and no final available same dev available >>> prefer_final_permutation(['33a1'], ['33a1']) had 33a1 >>> _ = zc.buildout.easy_install.prefer_final(True) """ def buildout_prefer_final_option(): """ The prefer-final buildout option can be used for override the default preference for newer distributions. The default is prefer-final = true: >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = eggs ... find-links = %(link_server)s ... update-versions-file = versions-picked.cfg ... ... [eggs] ... recipe = zc.recipe.egg:eggs ... eggs = demo ... ''' % globals()) >>> print_(system(buildout), end='') # doctest: +ELLIPSIS Installing ... ... written to versions-picked.cfg >>> cat('versions-picked.cfg') [versions] demo = 0.3 # Required by: # demo==0.3 demoneeded = 1.1 >>> remove('versions-picked.cfg') Here we see that the final versions of demo and demoneeded are used. We get the same behavior if we add prefer-final = true >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = eggs ... find-links = %(link_server)s ... prefer-final = true ... update-versions-file = versions-picked.cfg ... ... [eggs] ... recipe = zc.recipe.egg:eggs ... eggs = demo ... ''' % globals()) >>> print_(system(buildout), end='') # doctest: +ELLIPSIS Updating ... ... written to versions-picked.cfg >>> cat('versions-picked.cfg') [versions] demo = 0.3 # Required by: # demo==0.3 demoneeded = 1.1 >>> remove('versions-picked.cfg') If we specify prefer-final = false, we'll get the newest distributions: >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = eggs ... find-links = %(link_server)s ... prefer-final = false ... update-versions-file = versions-picked.cfg ... ... [eggs] ... recipe = zc.recipe.egg:eggs ... eggs = demo ... ''' % globals()) >>> print_(system(buildout), end='') # doctest: +ELLIPSIS Updating ... ... written to versions-picked.cfg >>> cat('versions-picked.cfg') [versions] demo = 0.4rc1 # Required by: # demo==0.4rc1 demoneeded = 1.2rc1 >>> remove('versions-picked.cfg') We get an error if we specify anything but true or false: >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = eggs ... find-links = %(link_server)s ... prefer-final = no ... ... [eggs] ... recipe = zc.recipe.egg:eggs ... eggs = demo ... ''' % globals()) >>> print_(system(buildout+' -v'), end='') # doctest: +ELLIPSIS While: Initializing. Error: Invalid value for 'prefer-final' option: 'no' """ def wont_downgrade_due_to_prefer_final(): r""" If we install a non-final buildout version, we don't want to downgrade just because we prefer-final. If a buildout version isn't specified using a versions entry, then buildout's version requirement gets set to >=CURRENT_VERSION. >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = ... ''') >>> [v] = [l.split('= >=', 1)[1].strip() ... for l in system(buildout+' -vv').split('\n') ... if l.startswith('zc.buildout = >=')] >>> v == pkg_resources.working_set.find( ... pkg_resources.Requirement.parse('zc.buildout') ... ).version True >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = ... [versions] ... zc.buildout = >0.1 ... ''') >>> [str(l.split('= >', 1)[1].strip()) ... for l in system(buildout+' -vv').split('\n') ... if l.startswith('zc.buildout =')] ['0.1'] >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = ... versions = versions ... [versions] ... zc.buildout = 43 ... ''') >>> print_(system(buildout), end='') # doctest: +ELLIPSIS Getting distribution for 'zc.buildout==43'. ... """ def develop_with_modules(): """ Distribution setup scripts can import modules in the distribution directory: >>> mkdir('foo') >>> write('foo', 'bar.py', ... '''# empty ... ''') >>> write('foo', 'setup.py', ... ''' ... import bar ... from setuptools import setup ... setup(name="foo") ... ''') >>> write('buildout.cfg', ... ''' ... [buildout] ... develop = foo ... parts = ... ''') >>> print_(system(join('bin', 'buildout')), end='') Develop: '/sample-buildout/foo' >>> ls('develop-eggs') - foo.egg-link - zc.recipe.egg.egg-link """ def dont_pick_setuptools_if_version_is_specified_when_required_by_src_dist(): r""" When installing a source distribution, we got setuptools without honoring our version specification. >>> mkdir('dist') >>> write('setup.py', ... ''' ... from setuptools import setup ... setup(name='foo', version='1', py_modules=['foo'], zip_safe=True) ... ''') >>> write('foo.py', '') >>> _ = system(buildout+' setup . sdist') >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = foo ... find-links = dist ... versions = versions ... allow-picked-versions = false ... ... [versions] ... wtf = %s ... foo = 1 ... ... [foo] ... recipe = zc.recipe.egg ... eggs = foo ... ''' % ('\n'.join( ... '%s = %s' % (d.key, d.version) ... for d in zc.buildout.easy_install.buildout_and_setuptools_dists))) >>> print_(system(buildout), end='') Installing foo. Getting distribution for 'foo==1'. Got foo 1. """ def pyc_and_pyo_files_have_correct_paths(): r""" >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = eggs ... find-links = %(link_server)s ... ... [eggs] ... recipe = zc.recipe.egg ... eggs = demo ... interpreter = py ... ''' % globals()) >>> _ = system(buildout) >>> write('t.py', ... r''' ... import eggrecipedemo, eggrecipedemoneeded, sys ... code = lambda f: f.__code__ ... sys.stdout.write(code(eggrecipedemo.main).co_filename+'\n') ... sys.stdout.write(code(eggrecipedemoneeded.f).co_filename+'\n') ... ''') >>> print_(system(join('bin', 'py')+ ' t.py'), end='') /sample-buildout/eggs/demo-0.3-py2.4.egg/eggrecipedemo.py /sample-buildout/eggs/demoneeded-1.1-py2.4.egg/eggrecipedemoneeded.py """ def dont_mess_with_standard_dirs_with_variable_refs(): """ >>> write('buildout.cfg', ... ''' ... [buildout] ... eggs-directory = ${buildout:directory}/develop-eggs ... parts = ... ''' % globals()) >>> print_(system(buildout), end='') """ def expand_shell_patterns_in_develop_paths(): """ Sometimes we want to include a number of eggs in some directory as develop eggs, without explicitly listing all of them in our buildout.cfg >>> make_dist_that_requires(sample_buildout, 'sampley') >>> make_dist_that_requires(sample_buildout, 'samplez') Now, let's create a buildout that has a shell pattern that matches both: >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = eggs ... develop = sample* ... find-links = %(link_server)s ... ... [eggs] ... recipe = zc.recipe.egg ... eggs = sampley ... samplez ... ''' % globals()) We can see that both eggs were found: >>> print_(system(buildout), end='') Develop: '/sample-buildout/sampley' Develop: '/sample-buildout/samplez' Installing eggs. """ def warn_users_when_expanding_shell_patterns_yields_no_results(): """ Sometimes shell patterns do not match anything, so we want to warn our users about it... >>> make_dist_that_requires(sample_buildout, 'samplea') So if we have 2 patterns, one that has a matching directory, and another one that does not >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = eggs ... develop = samplea grumble* ... find-links = %(link_server)s ... ... [eggs] ... recipe = zc.recipe.egg ... eggs = samplea ... ''' % globals()) We should get one of the eggs, and a warning for the pattern that did not match anything. >>> print_(system(buildout), end='') Develop: '/sample-buildout/samplea' Couldn't develop '/sample-buildout/grumble*' (not found) Installing eggs. """ def make_sure_versions_dont_cancel_extras(): """ There was a bug that caused extras in requirements to be lost. >>> with open('setup.py', 'w') as f: ... _ = f.write(''' ... from setuptools import setup ... setup(name='extraversiondemo', version='1.0', ... url='x', author='x', author_email='x', ... extras_require=dict(foo=['demo']), py_modules=['t']) ... ''') >>> open('README', 'w').close() >>> open('t.py', 'w').close() >>> sdist('.', sample_eggs) >>> mkdir('dest') >>> ws = zc.buildout.easy_install.install( ... ['extraversiondemo[foo]'], 'dest', links=[sample_eggs], ... versions = dict(extraversiondemo='1.0') ... ) >>> sorted(dist.key for dist in ws) ['demo', 'demoneeded', 'extraversiondemo'] """ def increment_buildout_options(): r""" >>> write('b1.cfg', ''' ... [buildout] ... parts = p1 ... x = 1 ... y = a ... b ... ... [p1] ... recipe = zc.buildout:debug ... foo = ${buildout:x} ${buildout:y} ... ''') >>> write('buildout.cfg', ''' ... [buildout] ... extends = b1.cfg ... parts += p2 ... x += 2 ... y -= a ... ... [p2] ... <= p1 ... ''') >>> print_(system(buildout), end='') Installing p1. foo='1\n2 b' recipe='zc.buildout:debug' Installing p2. foo='1\n2 b' recipe='zc.buildout:debug' """ def increment_buildout_with_multiple_extended_files_421022(): r""" >>> write('foo.cfg', ''' ... [buildout] ... foo-option = foo ... [other] ... foo-option = foo ... ''') >>> write('bar.cfg', ''' ... [buildout] ... bar-option = bar ... [other] ... bar-option = bar ... ''') >>> write('buildout.cfg', ''' ... [buildout] ... parts = p other ... extends = bar.cfg foo.cfg ... bar-option += baz ... foo-option += ham ... ... [other] ... recipe = zc.buildout:debug ... bar-option += baz ... foo-option += ham ... ... [p] ... recipe = zc.buildout:debug ... x = ${buildout:bar-option} ${buildout:foo-option} ... ''') >>> print_(system(buildout), end='') Installing p. recipe='zc.buildout:debug' x='bar\nbaz foo\nham' Installing other. bar-option='bar\nbaz' foo-option='foo\nham' recipe='zc.buildout:debug' """ def increment_on_command_line(): r""" >>> write('buildout.cfg', ''' ... [buildout] ... parts = p1 ... x = 1 ... y = a ... b ... ... [p1] ... recipe = zc.buildout:debug ... foo = ${buildout:x} ${buildout:y} ... ... [p2] ... <= p1 ... ''') >>> print_(system(buildout+' buildout:parts+=p2 p1:foo+=bar'), end='') Installing p1. foo='1 a\nb\nbar' recipe='zc.buildout:debug' Installing p2. foo='1 a\nb\nbar' recipe='zc.buildout:debug' """ def test_constrained_requirement(): """ zc.buildout.easy_install._constrained_requirement(constraint, requirement) Transforms an environment by applying a constraint. Here's a table of examples: >>> from zc.buildout.easy_install import IncompatibleConstraintError >>> examples = [ ... # original, constraint, transformed ... ('x', '1', 'x==1'), ... ('x>1', '2', 'x==2'), ... ('x>3', '2', IncompatibleConstraintError), ... ('x>1', '>2', 'x>1,>2'), ... ] >>> from zc.buildout.easy_install import _constrained_requirement >>> for o, c, e in examples: ... try: ... o = pkg_resources.Requirement.parse(o) ... if isinstance(e, str): ... e = pkg_resources.Requirement.parse(e) ... g = _constrained_requirement(c, o) ... except IncompatibleConstraintError: ... g = IncompatibleConstraintError ... if str(g) != str(e): ... print_('failed', o, c, g, '!=', e) """ def test_distutils_scripts_using_import_are_properly_parsed(): """ zc.buildout.easy_install._distutils_script(path, dest, script_content, initialization, rsetup): Creates a script for a distutils based project. In this example for a hypothetical code quality checker called 'pyflint' that uses an import statement to import its code. >>> pyflint_script = '''#!/path/to/bin/python ... import pyflint.do_something ... pyflint.do_something() ... ''' >>> import sys >>> original_executable = sys.executable >>> sys.executable = 'python' >>> from zc.buildout.easy_install import _distutils_script >>> generated = _distutils_script('\\'/path/test/\\'', 'bin/pyflint', pyflint_script, '', '') >>> if sys.platform == 'win32': ... generated == ['bin/pyflint.exe', 'bin/pyflint-script.py'] ... else: ... generated == ['bin/pyflint'] True >>> if sys.platform == 'win32': ... cat('bin/pyflint-script.py') ... else: ... cat('bin/pyflint') #!python import sys sys.path[0:0] = [ '/path/test/', ] import pyflint.do_something pyflint.do_something() >>> sys.executable = original_executable """ def test_distutils_scripts_using_from_are_properly_parsed(): """ zc.buildout.easy_install._distutils_script(path, dest, script_content, initialization, rsetup): Creates a script for a distutils based project. In this example for a hypothetical code quality checker called 'pyflint' that uses a from statement to import its code. >>> pyflint_script = '''#!/path/to/bin/python ... from pyflint import do_something ... do_something() ... ''' >>> import sys >>> original_executable = sys.executable >>> sys.executable = 'python' >>> from zc.buildout.easy_install import _distutils_script >>> generated = _distutils_script('\\'/path/test/\\'', 'bin/pyflint', pyflint_script, '', '') >>> if sys.platform == 'win32': ... generated == ['bin/pyflint.exe', 'bin/pyflint-script.py'] ... else: ... generated == ['bin/pyflint'] True >>> if sys.platform == 'win32': ... cat('bin/pyflint-script.py') ... else: ... cat('bin/pyflint') #!python import sys sys.path[0:0] = [ '/path/test/', ] from pyflint import do_something do_something() >>> sys.executable = original_executable """ def want_new_zcrecipeegg(): """ >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = egg ... [egg] ... recipe = zc.recipe.egg <2dev ... eggs = demo ... ''') >>> print_(system(join('bin', 'buildout')), end='') # doctest: +ELLIPSIS Getting distribution for 'zc.recipe.egg<2dev,>=2.0.6'... While: Installing. Getting section egg. Initializing section egg. Installing recipe zc.recipe.egg <2dev. Getting distribution for 'zc.recipe.egg<2dev,>=2.0.6'. Error: Couldn't find a distribution for 'zc.recipe.egg<2dev,>=2.0.6'. """ def macro_inheritance_bug(): """ There was a bug preventing a section from using another section as a macro if that section was extended with macros, and both sections were listed as parts (phew!). The following contrived example demonstrates that this now works. >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = foo bar ... [base] ... recipe = zc.recipe.egg ... [foo] ... <=base ... eggs = zc.buildout ... interpreter = python ... [bar] ... <=foo ... interpreter = py ... ''') >>> print_(system(join('bin', 'buildout')), end='') # doctest: +ELLIPSIS Installing foo. ... Installing bar. ... >>> ls("./bin") - buildout - py - python """ def bootstrap_honors_relative_paths(): """ >>> working = tmpdir('working') >>> cd(working) >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = ... relative-paths = true ... ''') >>> _ = system(buildout+' bootstrap') >>> cat('bin', 'buildout') # doctest: +ELLIPSIS #!/usr/local/bin/python2.7 import os join = os.path.join base = os.path.dirname(os.path.abspath(os.path.realpath(__file__))) base = os.path.dirname(base) import sys sys.path[0:0] = [ ... ] import zc.buildout.buildout if __name__ == '__main__': sys.exit(zc.buildout.buildout.main()) """ def cant_use_install_from_cache_and_offline_together(): r""" >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = ... offline = true ... install-from-cache = true ... ''') >>> print_(system(join('bin', 'buildout')), end='') # doctest: +ELLIPSIS While: Initializing. Error: install-from-cache can't be used with offline mode. Nothing is installed, even from cache, in offline mode, which might better be called 'no-install mode'. """ def error_installing_in_offline_mode_if_dont_have_needed_dist(): r""" >>> import zc.buildout.easy_install >>> ws = zc.buildout.easy_install.install( ... ['demo==0.2'], None, ... links=[link_server], index=link_server+'index/') Traceback (most recent call last): ... UserError: We don't have a distribution for demo==0.2 and can't install one in offline (no-install) mode. """ def error_building_in_offline_mode_if_dont_have_needed_dist(): r""" >>> zc.buildout.easy_install.build( ... 'extdemo', None, ... {}, links=[link_server], index=link_server+'index/') Traceback (most recent call last): ... UserError: We don't have a distribution for extdemo and can't build one in offline (no-install) mode. """ def test_buildout_section_shorthand_for_command_line_assignments(): r""" >>> write('buildout.cfg', '') >>> print_(system(buildout+' parts='), end='') # doctest: +ELLIPSIS """ def buildout_honors_umask(): """ For setting the executable permission, the user's umask is honored: >>> orig_umask = os.umask(0o077) # Only user gets permissions. >>> zc.buildout.easy_install._execute_permission() == 0o700 True >>> tmp = os.umask(0o022) # User can write, the rest not. >>> zc.buildout.easy_install._execute_permission() == 0o755 True >>> tmp = os.umask(orig_umask) # Reset umask to the original value. """ def parse_with_section_expr(): r""" >>> class Recipe: ... def __init__(self, buildout, *_): ... buildout.parse(''' ... [foo : sys.version_info[0] > 0] ... x = 1 ... ''') >>> buildout = zc.buildout.testing.Buildout() >>> buildout.parse(''' ... [foo : sys.version_info[0] > 0] ... x = 1 ... ''') >>> buildout.print_options() [foo] x = 1 """ def test_abi_tag_eggs(): r""" >>> write('buildout.cfg', ... ''' ... [buildout] ... find-links = %(sample_eggs)s ... parts = abi ... abi-tag-eggs = true ... [abi] ... recipe = zc.recipe.egg ... eggs = demo ... ''' % globals()) >>> _ = system(join('bin', 'buildout')) >>> from zc.buildout.pep425tags import get_abi_tag >>> abi_tag = get_abi_tag() >>> abi_tag in os.listdir(join(sample_buildout, 'eggs')) True >>> ls('eggs', abi_tag) d demo-0.3-py3.7.egg d demoneeded-1.1-py3.7.egg """ def test_buildout_doesnt_keep_adding_itself_to_versions(): r""" We were constantly writing to versions.cfg for buildout and setuptools >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = ... extends = versions.cfg ... show-picked-versions = true ... update-versions-file = versions.cfg ... extends = versions.cfg ... ''') >>> write('versions.cfg', ... '''[versions] ... ''') >>> _ = system(join('bin', 'buildout')) >>> with open('versions.cfg') as f: ... versions = f.read() >>> _ = system(join('bin', 'buildout')) On the first run, some pins were added: >>> cat('versions.cfg') # doctest: +ELLIPSIS [versions] >>> _ = system(join('bin', 'buildout')) >>> _ = system(join('bin', 'buildout')) Subsequent runs didn't add additional text: >>> with open('versions.cfg') as f: ... versions == f.read() True """ if sys.platform == 'win32': del buildout_honors_umask # umask on dohs is academic ###################################################################### def buildout_txt_setup(test): zc.buildout.testing.buildoutSetUp(test) mkdir = test.globs['mkdir'] eggs = os.environ['buildout_testing_index_url'][7:] test.globs['sample_eggs'] = eggs create_sample_eggs(test) for name in os.listdir(eggs): if '-' in name: pname = name.split('-')[0] if not os.path.exists(os.path.join(eggs, pname)): mkdir(eggs, pname) shutil.move(os.path.join(eggs, name), os.path.join(eggs, pname, name)) dist = pkg_resources.working_set.find( pkg_resources.Requirement.parse('zc.recipe.egg')) mkdir(eggs, 'zc.recipe.egg') zc.buildout.testing.sdist( os.path.dirname(dist.location), os.path.join(eggs, 'zc.recipe.egg'), ) egg_parse = re.compile(r'([0-9a-zA-Z_.]+)-([0-9a-zA-Z_.]+)-py(\d[.]\d+)$' ).match def makeNewRelease(project, ws, dest, version='99.99'): dist = ws.find(pkg_resources.Requirement.parse(project)) eggname, oldver, pyver = egg_parse(dist.egg_name()).groups() dest = os.path.join(dest, "%s-%s-py%s.egg" % (eggname, version, pyver)) if os.path.isfile(dist.location): shutil.copy(dist.location, dest) zip = zipfile.ZipFile(dest, 'a') zip.writestr( 'EGG-INFO/PKG-INFO', ((zip.read('EGG-INFO/PKG-INFO').decode('ISO-8859-1') ).replace("Version: %s" % oldver, "Version: %s" % version) ).encode('ISO-8859-1') ) zip.close() elif dist.location.endswith('site-packages'): os.mkdir(dest) shutil.copytree( os.path.join(dist.location, project), os.path.join(dest, project), ) distinfo = '%s-%s.dist-info' % (project, oldver) shutil.copytree( os.path.join(dist.location, distinfo), os.path.join(dest, distinfo), ) info_path = os.path.join(dest, distinfo, 'METADATA') with open(info_path) as f: info = f.read().replace("Version: %s" % oldver, "Version: %s" % version) with open(info_path, 'w') as f: f.write(info) new_distinfo = '%s-%s.dist-info' % (project, version) os.rename( os.path.join(dest, distinfo), os.path.join(dest, new_distinfo), ) else: shutil.copytree(dist.location, dest) info_path = os.path.join(dest, 'EGG-INFO', 'PKG-INFO') with open(info_path) as f: info = f.read().replace("Version: %s" % oldver, "Version: %s" % version) with open(info_path, 'w') as f: f.write(info) def getWorkingSetWithBuildoutEgg(test): sample_buildout = test.globs['sample_buildout'] eggs = os.path.join(sample_buildout, 'eggs') # If the zc.buildout dist is a develop dist, convert it to a # regular egg in the sample buildout req = pkg_resources.Requirement.parse('zc.buildout') dist = pkg_resources.working_set.find(req) if dist.precedence == pkg_resources.DEVELOP_DIST: # We have a develop egg, create a real egg for it: here = os.getcwd() os.chdir(os.path.dirname(dist.location)) zc.buildout.easy_install.call_subprocess( [sys.executable, os.path.join(os.path.dirname(dist.location), 'setup.py'), '-q', 'bdist_egg', '-d', eggs], env=dict(os.environ, PYTHONPATH=zc.buildout.easy_install.pip_pythonpath, ), ) os.chdir(here) os.remove(os.path.join(eggs, 'zc.buildout.egg-link')) # Rebuild the buildout script ws = pkg_resources.WorkingSet([eggs]) ws.require('zc.buildout') zc.buildout.easy_install.scripts( ['zc.buildout'], ws, sys.executable, os.path.join(sample_buildout, 'bin')) else: ws = pkg_resources.working_set return ws def updateSetup(test): zc.buildout.testing.buildoutSetUp(test) new_releases = test.globs['tmpdir']('new_releases') test.globs['new_releases'] = new_releases ws = getWorkingSetWithBuildoutEgg(test) # now let's make the new releases # TODO enable new releases of pip wheel setuptools # when eggs enforced makeNewRelease('zc.buildout', ws, new_releases) os.mkdir(os.path.join(new_releases, 'zc.buildout')) def ancestor(path, level): while level > 0: path = os.path.dirname(path) level -= 1 return path bootstrap_py = os.path.join(ancestor(__file__, 4), 'bootstrap', 'bootstrap.py') def bootstrapSetup(test): buildout_txt_setup(test) test.globs['link_server'] = test.globs['start_server']( test.globs['sample_eggs']) sample_eggs = test.globs['sample_eggs'] ws = getWorkingSetWithBuildoutEgg(test) makeNewRelease('zc.buildout', ws, sample_eggs, '2.0.0') makeNewRelease('zc.buildout', ws, sample_eggs, '22.0.0') os.environ['bootstrap-testing-find-links'] = test.globs['link_server'] test.globs['bootstrap_py'] = bootstrap_py def test_suite(): test_suite = [ manuel.testing.TestSuite( manuel.doctest.Manuel() + manuel.capture.Manuel(), 'configparser.test'), manuel.testing.TestSuite( manuel.doctest.Manuel( optionflags=doctest.NORMALIZE_WHITESPACE | doctest.ELLIPSIS, checker=renormalizing.RENormalizing([ zc.buildout.testing.normalize_path, zc.buildout.testing.normalize_endings, zc.buildout.testing.normalize_script, zc.buildout.testing.normalize_egg_py, zc.buildout.testing.not_found, zc.buildout.testing.adding_find_link, zc.buildout.testing.easyinstall_deprecated, zc.buildout.testing.setuptools_deprecated, zc.buildout.testing.pkg_resources_deprecated, zc.buildout.testing.warnings_warn, zc.buildout.testing.ignore_root_logger, # (re.compile(r"Installing 'zc.buildout >=\S+"), ''), (re.compile(r'__buildout_signature__ = recipes-\S+'), '__buildout_signature__ = recipes-SSSSSSSSSSS'), (re.compile(r'executable = [\S ]+python\S*', re.I), 'executable = python'), (re.compile(r'[-d] (setuptools|setuptools)-\S+[.]egg'), 'setuptools.egg'), (re.compile(r'pip-\S+[.]egg'), 'pip.egg'), (re.compile(r'wheel-\S+[.]egg'), 'wheel.egg'), (re.compile(r'zc.buildout(-\S+)?[.]egg(-link)?'), 'zc.buildout.egg'), (re.compile(r'creating \S*setup.cfg'), 'creating setup.cfg'), (re.compile(r'hello\%ssetup' % os.path.sep), 'hello/setup'), (re.compile(r'Picked: (\S+) = \S+'), 'Picked: \\1 = V.V'), (re.compile(r'We have a develop egg: zc.buildout (\S+)'), 'We have a develop egg: zc.buildout X.X.'), (re.compile(r'\\[\\]?'), '/'), (re.compile('WindowsError'), 'OSError'), (re.compile(r'\[Error \d+\] Cannot create a file ' r'when that file already exists: '), '[Errno 17] File exists: ' ), (re.compile('setuptools'), 'setuptools'), (re.compile(r'Got zc.recipe.egg \S+'), 'Got zc.recipe.egg'), (re.compile(r'zc\.(buildout|recipe\.egg)\s*= >=\S+'), 'zc.\\1 = >=1.99'), ]) ) + manuel.capture.Manuel(), 'buildout.txt', setUp=buildout_txt_setup, tearDown=zc.buildout.testing.buildoutTearDown, ), doctest.DocFileSuite( 'runsetup.txt', 'repeatable.txt', 'setup.txt', setUp=zc.buildout.testing.buildoutSetUp, tearDown=zc.buildout.testing.buildoutTearDown, optionflags=doctest.NORMALIZE_WHITESPACE | doctest.ELLIPSIS, checker=renormalizing.RENormalizing([ zc.buildout.testing.normalize_path, zc.buildout.testing.normalize_endings, zc.buildout.testing.normalize_script, zc.buildout.testing.normalize_egg_py, zc.buildout.testing.not_found, zc.buildout.testing.adding_find_link, zc.buildout.testing.easyinstall_deprecated, zc.buildout.testing.setuptools_deprecated, zc.buildout.testing.pkg_resources_deprecated, zc.buildout.testing.warnings_warn, zc.buildout.testing.ignore_root_logger, # (re.compile(r"Installing 'zc.buildout >=\S+"), ''), # (re.compile(r"Getting distribution for 'zc.buildout >=\S+"), # ''), (re.compile(r'__buildout_signature__ = recipes-\S+'), '__buildout_signature__ = recipes-SSSSSSSSSSS'), (re.compile(r'[-d] setuptools-\S+[.]egg'), 'setuptools.egg'), (re.compile(r'zc.buildout(-\S+)?[.]egg(-link)?'), 'zc.buildout.egg'), (re.compile(r'creating \S*setup.cfg'), 'creating setup.cfg'), (re.compile(r'hello\%ssetup' % os.path.sep), 'hello/setup'), (re.compile(r'Picked: (\S+) = \S+'), 'Picked: \\1 = V.V'), (re.compile(r'We have a develop egg: zc.buildout (\S+)'), 'We have a develop egg: zc.buildout X.X.'), (re.compile(r'\\[\\]?'), '/'), (re.compile(r'WindowsError'), 'OSError'), (re.compile(r'pip = \S+'), 'pip = 20.0.1'), (re.compile(r'setuptools = \S+'), 'setuptools = 46.1.3'), (re.compile(r'\[Error 17\] Cannot create a file ' r'when that file already exists: '), '[Errno 17] File exists: ' ), (re.compile('executable = %s' % re.escape(sys.executable)), 'executable = python'), (re.compile(r'\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{6}'), 'YYYY-MM-DD hh:mm:ss.dddddd'), ]), ), doctest.DocFileSuite( 'debugging.txt', setUp=zc.buildout.testing.buildoutSetUp, tearDown=zc.buildout.testing.buildoutTearDown, checker=renormalizing.RENormalizing([ zc.buildout.testing.normalize_path, zc.buildout.testing.normalize_endings, zc.buildout.testing.normalize_exception_type_for_python_2_and_3, zc.buildout.testing.not_found, zc.buildout.testing.adding_find_link, zc.buildout.testing.easyinstall_deprecated, zc.buildout.testing.setuptools_deprecated, zc.buildout.testing.pkg_resources_deprecated, zc.buildout.testing.warnings_warn, zc.buildout.testing.ignore_root_logger, (re.compile('zc.buildout.buildout.MissingOption'), 'MissingOption'), (re.compile(r'\S+buildout.py'), 'buildout.py'), (re.compile(r'line \d+'), 'line NNN'), (re.compile(r'py\(\d+\)'), 'py(NNN)'), ]) ), doctest.DocFileSuite( 'easy_install.txt', 'downloadcache.txt', 'dependencylinks.txt', 'allowhosts.txt', 'allow-unknown-extras.txt', setUp=easy_install_SetUp, tearDown=zc.buildout.testing.buildoutTearDown, optionflags=doctest.NORMALIZE_WHITESPACE | doctest.ELLIPSIS, checker=renormalizing.RENormalizing([ zc.buildout.testing.normalize_script, zc.buildout.testing.normalize_path, zc.buildout.testing.normalize_endings, zc.buildout.testing.normalize_egg_py, zc.buildout.testing.normalize_exception_type_for_python_2_and_3, zc.buildout.testing.normalize_open_in_generated_script, zc.buildout.testing.adding_find_link, zc.buildout.testing.not_found, zc.buildout.testing.easyinstall_deprecated, zc.buildout.testing.setuptools_deprecated, zc.buildout.testing.pkg_resources_deprecated, zc.buildout.testing.warnings_warn, zc.buildout.testing.ignore_root_logger, normalize_bang, (re.compile(r'[-d] setuptools-\S+[.]egg'), 'setuptools.egg'), (re.compile(r'\\[\\]?'), '/'), (re.compile('(\n?)- ([a-zA-Z_.-]+)\n- \\2.exe\n'), '\\1- \\2\n'), ]), ), doctest.DocFileSuite( 'download.txt', 'extends-cache.txt', setUp=easy_install_SetUp, tearDown=zc.buildout.testing.buildoutTearDown, optionflags=doctest.NORMALIZE_WHITESPACE | doctest.ELLIPSIS, checker=renormalizing.RENormalizing([ zc.buildout.testing.normalize_exception_type_for_python_2_and_3, zc.buildout.testing.not_found, zc.buildout.testing.adding_find_link, zc.buildout.testing.easyinstall_deprecated, zc.buildout.testing.setuptools_deprecated, zc.buildout.testing.pkg_resources_deprecated, zc.buildout.testing.warnings_warn, zc.buildout.testing.ignore_root_logger, (re.compile(' at -?0x[^>]+'), ''), (re.compile('http://localhost:[0-9]{4,5}/'), 'http://localhost/'), (re.compile('[0-9a-f]{32}'), ''), zc.buildout.testing.normalize_path, zc.buildout.testing.ignore_not_upgrading, ]), ), doctest.DocTestSuite( setUp=easy_install_SetUp, tearDown=zc.buildout.testing.buildoutTearDown, optionflags=doctest.NORMALIZE_WHITESPACE | doctest.ELLIPSIS, checker=renormalizing.RENormalizing([ zc.buildout.testing.normalize_path, zc.buildout.testing.normalize_endings, zc.buildout.testing.normalize_script, zc.buildout.testing.normalize_egg_py, zc.buildout.testing.normalize___pycache__, zc.buildout.testing.not_found, zc.buildout.testing.normalize_exception_type_for_python_2_and_3, zc.buildout.testing.adding_find_link, zc.buildout.testing.easyinstall_deprecated, zc.buildout.testing.setuptools_deprecated, zc.buildout.testing.pkg_resources_deprecated, zc.buildout.testing.warnings_warn, zc.buildout.testing.ignore_root_logger, zc.buildout.testing.ignore_native_namespace_warning_1, zc.buildout.testing.ignore_native_namespace_warning_2, zc.buildout.testing.ignore_native_namespace_warning_3, zc.buildout.testing.ignore_native_namespace_warning_4, zc.buildout.testing.ignore_native_namespace_warning_5, normalize_bang, (re.compile(r'^(\w+\.)*(Missing\w+: )'), '\2'), (re.compile(r"buildout: Running \S*setup.py"), 'buildout: Running setup.py'), (re.compile(r'pip-\S+-'), 'pip.egg'), (re.compile(r'setuptools-\S+-'), 'setuptools.egg'), (re.compile(r'zc.buildout-\S+-'), 'zc.buildout.egg'), (re.compile(r'pip = \S+'), 'pip = 20.0.0'), (re.compile(r'setuptools = \S+'), 'setuptools = 0.7.99'), (re.compile(r'File "\S+one.py"'), 'File "one.py"'), (re.compile(r'We have a develop egg: (\S+) (\S+)'), r'We have a develop egg: \1 V'), (re.compile(r'Picked: setuptools = \S+'), 'Picked: setuptools = V'), (re.compile('[-d] pip'), '- pip'), (re.compile('[-d] setuptools'), '- setuptools'), (re.compile(r'\\[\\]?'), '/'), (re.compile( '-q develop -mxN -d "/sample-buildout/develop-eggs'), '-q develop -mxN -d /sample-buildout/develop-eggs' ), (re.compile(r'^[*]...'), '...'), # for # bug_92891 # bootstrap_crashes_with_egg_recipe_in_buildout_section (re.compile(r"Unused options for buildout: 'eggs' 'scripts'\."), "Unused options for buildout: 'scripts' 'eggs'."), # Python 3.4 changed the wording of NameErrors (re.compile('NameError: global name'), 'NameError: name'), # fix for test_distutils_scripts_using_import_are_properly_parsed # and test_distutils_scripts_using_from_are_properly_parsed # win32 apparently adds a " around sys.executable (re.compile('#!"python"'), '#!python'), ]), ), zc.buildout.rmtree.test_suite(), doctest.DocFileSuite( 'windows.txt', setUp=zc.buildout.testing.buildoutSetUp, tearDown=zc.buildout.testing.buildoutTearDown, optionflags=doctest.NORMALIZE_WHITESPACE | doctest.ELLIPSIS, checker=renormalizing.RENormalizing([ zc.buildout.testing.normalize_path, zc.buildout.testing.normalize_endings, zc.buildout.testing.normalize_script, zc.buildout.testing.normalize_egg_py, zc.buildout.testing.not_found, zc.buildout.testing.adding_find_link, zc.buildout.testing.easyinstall_deprecated, zc.buildout.testing.setuptools_deprecated, zc.buildout.testing.pkg_resources_deprecated, zc.buildout.testing.warnings_warn, zc.buildout.testing.ignore_root_logger, (re.compile(r'__buildout_signature__ = recipes-\S+'), '__buildout_signature__ = recipes-SSSSSSSSSSS'), (re.compile(r'[-d] setuptools-\S+[.]egg'), 'setuptools.egg'), (re.compile(r'zc.buildout(-\S+)?[.]egg(-link)?'), 'zc.buildout.egg'), (re.compile(r'creating \S*setup.cfg'), 'creating setup.cfg'), (re.compile(r'hello\%ssetup' % os.path.sep), 'hello/setup'), (re.compile(r'Picked: (\S+) = \S+'), 'Picked: \\1 = V.V'), (re.compile(r'We have a develop egg: zc.buildout (\S+)'), 'We have a develop egg: zc.buildout X.X.'), (re.compile(r'\\[\\]?'), '/'), (re.compile('WindowsError'), 'OSError'), (re.compile(r'\[Error 17\] Cannot create a file ' r'when that file already exists: '), '[Errno 17] File exists: ' ), ]) ), doctest.DocFileSuite('testing_bugfix.txt'), ] if not sys.platform.startswith('win'): # In the update.txt tests on Windows, instead of # "Upgraded: zc.buildout version NINETYNINE.NINETYNINE;" # we get: # "Not upgrading because not running a local buildout command." # I don't know why that is only the case on Windows. test_suite.append( doctest.DocFileSuite( 'update.txt', setUp=updateSetup, tearDown=zc.buildout.testing.buildoutTearDown, optionflags=doctest.NORMALIZE_WHITESPACE | doctest.ELLIPSIS, checker=renormalizing.RENormalizing([ (re.compile(r'(zc.buildout|setuptools)-\d+[.]\d+\S*' r'-py\d.\d+.egg'), '\\1.egg'), zc.buildout.testing.normalize_path, zc.buildout.testing.normalize_endings, zc.buildout.testing.normalize_script, zc.buildout.testing.normalize_egg_py, zc.buildout.testing.not_found, zc.buildout.testing.adding_find_link, zc.buildout.testing.easyinstall_deprecated, zc.buildout.testing.setuptools_deprecated, zc.buildout.testing.pkg_resources_deprecated, zc.buildout.testing.warnings_warn, zc.buildout.testing.ignore_root_logger, normalize_bang, (re.compile(r"Getting distribution for 'zc.buildout>=\S+"), ''), (re.compile('99[.]99'), 'NINETYNINE.NINETYNINE'), (re.compile( r'(zc.buildout|setuptools|pip)( version)? \d+[.]\d+\S*'), '\\1 V.V'), (re.compile('[-d] setuptools'), '- setuptools'), (re.compile('[-d] pip'), '- pip'), (re.compile('[-d] wheel'), '- wheel'), (re.compile(re.escape(os.path.sep)+'+'), '/'), ]) ) ) docdir = os.path.join(ancestor(__file__, 5), 'doc') if os.path.exists(docdir) and not sys.platform.startswith('win'): # Note that the purpose of the documentation tests are mainly # to test the documentation, not to test buildout. def docSetUp(test): def write(text, *path): with open(os.path.join(*path), 'w') as f: f.write(text) test.globs.update( run_buildout=zc.buildout.testing.run_buildout_in_process, yup=lambda cond, orelse='Nope': None if cond else orelse, nope=lambda cond, orelse='Nope': orelse if cond else None, eq=lambda a, b: None if a == b else (a, b), eqs=zc.buildout.testing.eqs, read=zc.buildout.testing.read, write=write, ls=lambda d='.', *rest: os.listdir(os.path.join(d, *rest)), join=os.path.join, clear_here=zc.buildout.testing.clear_here, os=os, ) setupstack.setUpDirectory(test) test_suite.append( manuel.testing.TestSuite( manuel.doctest.Manuel( optionflags=doctest.NORMALIZE_WHITESPACE | doctest.ELLIPSIS, checker=renormalizing.RENormalizing([ zc.buildout.testing.easyinstall_deprecated, zc.buildout.testing.setuptools_deprecated, zc.buildout.testing.pkg_resources_deprecated, zc.buildout.testing.warnings_warn, zc.buildout.testing.ignore_root_logger, ]), ) + manuel.capture.Manuel(), os.path.join(docdir, 'getting-started.rst'), os.path.join(docdir, 'reference.rst'), os.path.join(docdir, 'topics', 'bootstrapping.rst'), os.path.join(docdir, 'topics', 'implicit-parts.rst'), os.path.join( docdir, 'topics', 'variables-extending-and-substitutions.rst'), os.path.join(docdir, 'topics', 'writing-recipes.rst'), os.path.join(docdir, 'topics', 'optimizing.rst'), os.path.join(docdir, 'topics', 'meta-recipes.rst'), setUp=docSetUp, tearDown=setupstack.tearDown )) # adding bootstrap.txt doctest to the suite # only if bootstrap.py is present if os.path.exists(bootstrap_py): test_suite.append(doctest.DocFileSuite( 'bootstrap.txt', 'bootstrap_cl_settings.test', setUp=bootstrapSetup, tearDown=zc.buildout.testing.buildoutTearDown, checker=renormalizing.RENormalizing([ zc.buildout.testing.normalize_path, zc.buildout.testing.normalize_endings, zc.buildout.testing.normalize_script, zc.buildout.testing.not_found, normalize_bang, zc.buildout.testing.adding_find_link, (re.compile('Downloading.*setuptools.*egg\n'), ''), ]), )) test_suite.append(unittest.defaultTestLoader.loadTestsFromName(__name__)) return unittest.TestSuite(test_suite) if __name__ == '__main__': unittest.main() ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675670.0 zc_buildout-4.1.6/src/zc/buildout/tests/test_extras.py0000644000076500000240000001144314773460426022607 0ustar00mauritsstaff# -*- coding: utf-8 -*- ############################################################################## # # Copyright (c) 2020 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import doctest import re from zope.testing import renormalizing import zc.buildout.testing from zc.buildout.tests import easy_install_SetUp from zc.buildout.tests import normalize_bang def install_extras_with_greater_than_constrains(): """ There was a bug that caused extras in requirements to be lost. >>> working = tmpdir('working') >>> cd(working) >>> mkdir('dependency') >>> cd('dependency') >>> with open('setup.py', 'w') as f: ... _ = f.write(''' ... from setuptools import setup ... setup(name='dependency', version='1.0', ... url='x', author='x', author_email='x', ... py_modules=['t']) ... ''') >>> open('README', 'w').close() >>> open('t.py', 'w').close() >>> sdist('.', sample_eggs) >>> cd(working) >>> mkdir('extras') >>> cd('extras') >>> with open('setup.py', 'w') as f: ... _ = f.write(''' ... from setuptools import setup ... setup(name='extraversiondemo', version='1.0', ... url='x', author='x', author_email='x', ... extras_require=dict(foo=['dependency']), py_modules=['t']) ... ''') >>> open('README', 'w').close() >>> open('t.py', 'w').close() >>> sdist('.', sample_eggs) >>> mkdir('dest') >>> ws = zc.buildout.easy_install.install( ... ['extraversiondemo[foo]'], 'dest', links=[sample_eggs], ... versions = dict(extraversiondemo='1.0', dependency='>0.9') ... ) >>> sorted(dist.key for dist in ws) ['dependency', 'extraversiondemo'] """ def test_suite(): return doctest.DocTestSuite( setUp=easy_install_SetUp, tearDown=zc.buildout.testing.buildoutTearDown, checker=renormalizing.RENormalizing([ zc.buildout.testing.normalize_path, zc.buildout.testing.normalize_endings, zc.buildout.testing.normalize_script, zc.buildout.testing.normalize_egg_py, zc.buildout.testing.normalize___pycache__, zc.buildout.testing.not_found, zc.buildout.testing.normalize_exception_type_for_python_2_and_3, zc.buildout.testing.adding_find_link, zc.buildout.testing.easyinstall_deprecated, zc.buildout.testing.setuptools_deprecated, zc.buildout.testing.pkg_resources_deprecated, zc.buildout.testing.warnings_warn, normalize_bang, (re.compile(r'^(\w+\.)*(Missing\w+: )'), '\2'), (re.compile(r"buildout: Running \S*setup.py"), 'buildout: Running setup.py'), (re.compile(r'pip-\S+-'), 'pip.egg'), (re.compile(r'setuptools-\S+-'), 'setuptools.egg'), (re.compile(r'zc.buildout-\S+-'), 'zc.buildout.egg'), (re.compile(r'pip = \S+'), 'pip = 20.0.0'), (re.compile(r'setuptools = \S+'), 'setuptools = 0.7.99'), (re.compile(r'File "\S+one.py"'), 'File "one.py"'), (re.compile(r'We have a develop egg: (\S+) (\S+)'), r'We have a develop egg: \1 V'), (re.compile(r'Picked: setuptools = \S+'), 'Picked: setuptools = V'), (re.compile('[-d] pip'), '- pip'), (re.compile('[-d] setuptools'), '- setuptools'), (re.compile(r'\\[\\]?'), '/'), (re.compile( '-q develop -mxN -d "/sample-buildout/develop-eggs'), '-q develop -mxN -d /sample-buildout/develop-eggs' ), (re.compile(r'^[*]...'), '...'), # for # bug_92891 # bootstrap_crashes_with_egg_recipe_in_buildout_section (re.compile(r"Unused options for buildout: 'eggs' 'scripts'\."), "Unused options for buildout: 'scripts' 'eggs'."), # Python 3.4 changed the wording of NameErrors (re.compile('NameError: global name'), 'NameError: name'), # fix for test_distutils_scripts_using_import_are_properly_parsed # and test_distutils_scripts_using_from_are_properly_parsed # win32 apparently adds a " around sys.executable (re.compile('#!"python"'), '#!python'), ]), ) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675670.0 zc_buildout-4.1.6/src/zc/buildout/tests/test_increment.py0000644000076500000240000003203614773460426023266 0ustar00mauritsstaff# -*- coding: utf-8 -*- ############################################################################## # # Copyright (c) 2020 Zope Foundation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import doctest import re from zope.testing import renormalizing import zc.buildout.testing from zc.buildout.tests import easy_install_SetUp from zc.buildout.tests import normalize_bang def default_cfg(): r""" >>> home = tmpdir('home') >>> mkdir(home, '.buildout') >>> default_cfg = join(home, '.buildout', 'default.cfg') >>> write(default_cfg, ''' ... [debug] ... dec = 1 ... 2 ... inc = 1 ... ''') >>> write('buildout.cfg', ''' ... [buildout] ... ... [debug] ... dec -= 2 ... inc += 2 ... ''') >>> env = dict(HOME=home, USERPROFILE=home) >>> print_(system(buildout+' annotate debug', env=env), end='') Annotated sections ================== [debug] dec= 1 /home/.buildout/default.cfg -= buildout.cfg inc= 1 2 /home/.buildout/default.cfg += buildout.cfg """ def default_cfg_extensions(): r""" Add two extensions as develop eggs >>> mkdir('demo') >>> write('demo', 'demo.py', ''' ... import sys ... def ext(buildout): ... sys.stdout.write('demo %s %s\\n' % ('ext', sorted(buildout))) ... def unload(buildout): ... sys.stdout.write('demo %s %s\\n' % ('unload', sorted(buildout))) ... ''') >>> write('demo', 'setup.py', ''' ... from setuptools import setup ... ... setup( ... name = "demo", ... entry_points = { ... 'zc.buildout.extension': ['ext = demo:ext'], ... 'zc.buildout.unloadextension': ['ext = demo:unload'], ... }, ... ) ... ''') >>> mkdir('demo2') >>> write('demo2', 'demo2.py', ''' ... import sys ... def ext(buildout): ... sys.stdout.write('demo2 %s %s\\n' % ('ext', sorted(buildout))) ... def unload(buildout): ... sys.stdout.write('demo2 %s %s\\n' % ('unload', sorted(buildout))) ... ''') >>> write('demo2', 'setup.py', ''' ... from setuptools import setup ... ... setup( ... name = "demo2", ... entry_points = { ... 'zc.buildout.extension': ['ext = demo2:ext'], ... 'zc.buildout.unloadextension': ['ext = demo2:unload'], ... }, ... ) ... ''') >>> write('buildout.cfg', ''' ... [buildout] ... develop = demo demo2 ... parts = ... ''') Run buildout once without extensions to actually develop the eggs. (Develop happens after loading extensions.) >>> print_(system(buildout), end='') Develop: '/sample-buildout/demo' Develop: '/sample-buildout/demo2' >>> ls("develop-eggs") - demo.egg-link - demo2.egg-link - zc.recipe.egg.egg-link extensions in .buildout/default.cfg incremented in buildout.cfg >>> home = tmpdir('home') >>> mkdir(home, '.buildout') >>> default_cfg = join(home, '.buildout', 'default.cfg') >>> write(default_cfg, ''' ... [buildout] ... extensions = demo ... ''') >>> write('buildout.cfg', ''' ... [buildout] ... develop = demo demo2 ... extensions += demo2 ... parts = ... ''') >>> env = dict(HOME=home, USERPROFILE=home) >>> print_(system(buildout+' annotate buildout', env=env), end='') ... # doctest: +ELLIPSIS Annotated sections ================== [buildout] ... extensions= demo demo2 /home/.buildout/default.cfg += buildout.cfg ... versions= versions DEFAULT_VALUE """ def with_extends_increment_in_base(): r""" >>> home = tmpdir('home') >>> mkdir(home, '.buildout') >>> default_cfg = join(home, '.buildout', 'default.cfg') >>> write(default_cfg, ''' ... [buildout] ... extensions = demo ... ''') >>> write('base.cfg', ''' ... [buildout] ... extensions += demo2 ... ''') >>> write('buildout.cfg', ''' ... [buildout] ... extends = base.cfg ... parts = ... ''') >>> env = dict(HOME=home, USERPROFILE=home) >>> print_(system(buildout+' annotate buildout', env=env), end='') ... # doctest: +ELLIPSIS Annotated sections ================== [buildout] ... extensions= demo demo2 /home/.buildout/default.cfg += base.cfg ... versions= versions DEFAULT_VALUE """ def with_extends_increment_in_base2(): r""" >>> home = tmpdir('home') >>> mkdir(home, '.buildout') >>> default_cfg = join(home, '.buildout', 'default.cfg') >>> write(default_cfg, ''' ... [buildout] ... extensions = demo ... ''') >>> write('base.cfg', ''' ... [buildout] ... ''') >>> write('base2.cfg', ''' ... [buildout] ... extensions += demo2 ... ''') >>> write('buildout.cfg', ''' ... [buildout] ... extends = base.cfg ... base2.cfg ... parts = ... ''') >>> env = dict(HOME=home, USERPROFILE=home) >>> print_(system(buildout+' annotate buildout', env=env), end='') ... # doctest: +ELLIPSIS Annotated sections ================== [buildout] ... extensions= demo demo2 /home/.buildout/default.cfg += base2.cfg ... versions= versions DEFAULT_VALUE """ def with_extends_increment_in_base2_and_base3(): r""" >>> home = tmpdir('home') >>> mkdir(home, '.buildout') >>> default_cfg = join(home, '.buildout', 'default.cfg') >>> write(default_cfg, ''' ... [buildout] ... extensions = demo ... ''') >>> write('base.cfg', ''' ... [buildout] ... ''') >>> write('base2.cfg', ''' ... [buildout] ... extensions += demo2 ... ''') >>> write('base3.cfg', ''' ... [buildout] ... extensions += demo3 ... ''') >>> write('buildout.cfg', ''' ... [buildout] ... extends = base.cfg ... base2.cfg ... base3.cfg ... parts = ... ''') >>> env = dict(HOME=home, USERPROFILE=home) >>> print_(system(buildout+' annotate buildout', env=env), end='') ... # doctest: +ELLIPSIS Annotated sections ================== [buildout] ... extensions= demo demo2 demo3 /home/.buildout/default.cfg += base2.cfg += base3.cfg ... versions= versions DEFAULT_VALUE """ def with_extends_increment_in_buildout(): r""" >>> home = tmpdir('home') >>> mkdir(home, '.buildout') >>> default_cfg = join(home, '.buildout', 'default.cfg') >>> write(default_cfg, ''' ... [buildout] ... extensions = demo ... ''') >>> write('base.cfg', ''' ... [buildout] ... ''') >>> write('buildout.cfg', ''' ... [buildout] ... extends = base.cfg ... extensions += demo2 ... parts = ... ''') >>> env = dict(HOME=home, USERPROFILE=home) >>> print_(system(buildout+' annotate buildout', env=env), end='') ... # doctest: +ELLIPSIS Annotated sections ================== [buildout] ... extensions= demo demo2 /home/.buildout/default.cfg += buildout.cfg ... versions= versions DEFAULT_VALUE """ def with_extends_increment_in_buildout_with_base_and_root(): r""" >>> home = tmpdir('home') >>> mkdir(home, '.buildout') >>> default_cfg = join(home, '.buildout', 'default.cfg') >>> write(default_cfg, ''' ... [buildout] ... extensions = demo ... ''') >>> write('root.cfg', ''' ... [buildout] ... ''') >>> write('base.cfg', ''' ... [buildout] ... extends = root.cfg ... ''') >>> write('buildout.cfg', ''' ... [buildout] ... extends = base.cfg ... extensions += demo2 ... parts = ... ''') >>> env = dict(HOME=home, USERPROFILE=home) >>> print_(system(buildout+' annotate buildout', env=env), end='') ... # doctest: +ELLIPSIS Annotated sections ================== [buildout] ... extensions= demo demo2 /home/.buildout/default.cfg += buildout.cfg ... versions= versions DEFAULT_VALUE """ def no_default_with_extends_increment_in_base2_and_base3(): r""" >>> write('base.cfg', ''' ... [buildout] ... ''') >>> write('base2.cfg', ''' ... [buildout] ... extensions += demo2 ... ''') >>> write('base3.cfg', ''' ... [buildout] ... extensions += demo3 ... ''') >>> write('buildout.cfg', ''' ... [buildout] ... extends = base.cfg ... base2.cfg ... base3.cfg ... parts = ... ''') >>> print_(system(buildout+' annotate buildout'), end='') ... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE Annotated sections ================== [buildout] ... extensions= demo2 demo3 IMPLICIT_VALUE += base2.cfg += base3.cfg ... versions= versions DEFAULT_VALUE """ def increment_buildout_with_multiple_extended_without_base_equals(): r""" >>> write('buildout.cfg', ''' ... [buildout] ... extends = base1.cfg base2.cfg ... parts += foo ... [foo] ... recipe = zc.buildout:debug ... [base1] ... recipe = zc.buildout:debug ... [base2] ... recipe = zc.buildout:debug ... ''') >>> write('base1.cfg', ''' ... [buildout] ... extends = base3.cfg ... parts += base1 ... ''') >>> write('base2.cfg', ''' ... [buildout] ... extends = base3.cfg ... parts += base2 ... ''') >>> write('base3.cfg', ''' ... [buildout] ... ''') >>> print_(system(buildout), end='') Installing base1. recipe='zc.buildout:debug' Installing base2. recipe='zc.buildout:debug' Installing foo. recipe='zc.buildout:debug' """ def test_suite(): return doctest.DocTestSuite( setUp=easy_install_SetUp, tearDown=zc.buildout.testing.buildoutTearDown, checker=renormalizing.RENormalizing([ zc.buildout.testing.normalize_path, zc.buildout.testing.normalize_endings, zc.buildout.testing.normalize_script, zc.buildout.testing.normalize_egg_py, zc.buildout.testing.normalize___pycache__, zc.buildout.testing.not_found, zc.buildout.testing.normalize_exception_type_for_python_2_and_3, zc.buildout.testing.adding_find_link, zc.buildout.testing.easyinstall_deprecated, zc.buildout.testing.setuptools_deprecated, zc.buildout.testing.pkg_resources_deprecated, zc.buildout.testing.warnings_warn, normalize_bang, (re.compile(r'^(\w+\.)*(Missing\w+: )'), '\2'), (re.compile(r"buildout: Running \S*setup.py"), 'buildout: Running setup.py'), (re.compile(r'pip-\S+-'), 'pip.egg'), (re.compile(r'setuptools-\S+-'), 'setuptools.egg'), (re.compile(r'zc.buildout-\S+-'), 'zc.buildout.egg'), (re.compile(r'pip = \S+'), 'pip = 20.0.0'), (re.compile(r'setuptools = \S+'), 'setuptools = 0.7.99'), (re.compile(r'File "\S+one.py"'), 'File "one.py"'), (re.compile(r'We have a develop egg: (\S+) (\S+)'), r'We have a develop egg: \1 V'), (re.compile(r'Picked: setuptools = \S+'), 'Picked: setuptools = V'), (re.compile('[-d] pip'), '- pip'), (re.compile('[-d] setuptools'), '- setuptools'), (re.compile(r'\\[\\]?'), '/'), (re.compile( '-q develop -mxN -d "/sample-buildout/develop-eggs'), '-q develop -mxN -d /sample-buildout/develop-eggs' ), (re.compile(r'^[*]...'), '...'), # for # bug_92891 # bootstrap_crashes_with_egg_recipe_in_buildout_section (re.compile(r"Unused options for buildout: 'eggs' 'scripts'\."), "Unused options for buildout: 'scripts' 'eggs'."), # Python 3.4 changed the wording of NameErrors (re.compile('NameError: global name'), 'NameError: name'), # fix for test_distutils_scripts_using_import_are_properly_parsed # and test_distutils_scripts_using_from_are_properly_parsed # win32 apparently adds a " around sys.executable (re.compile('#!"python"'), '#!python'), ]), ) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675670.0 zc_buildout-4.1.6/src/zc/buildout/tests/testing_bugfix.txt0000644000076500000240000000234114773460426023447 0ustar00mauritsstaffBug fixes in zc.buildout.testing ================================ Logging handler which did not get deleted ----------------------------------------- The buildout testing set up runs a buildout which adds a ``logging.StreamHandler`` to the root logger. But tear down did not remove it. This can disturb other tests of packages reusing zc.buildout.testing. The handlers before calling set up are: >>> import logging >>> count = len(logging.getLogger().handlers) >>> logging.getLogger().handlers # doctest: +ELLIPSIS [<...NullHandler...>] After calling it, a ``logging.StreamHandler`` was added: >>> import zc.buildout.testing >>> import doctest >>> test = doctest.DocTestParser().get_doctest( ... '>>> x', {}, 'foo', 'foo.py', 0) >>> zc.buildout.testing.buildoutSetUp(test) >>> len(logging.getLogger().handlers) == count + 1 True >>> logging.getLogger().handlers # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE [<...NullHandler...StreamHandler...>] But tear down removes the new logging handler: >>> zc.buildout.testing.buildoutTearDown(test) >>> len(logging.getLogger().handlers) == count True >>> logging.getLogger().handlers # doctest: +ELLIPSIS [<...NullHandler...>] ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675670.0 zc_buildout-4.1.6/src/zc/buildout/tests/update.txt0000644000076500000240000002001314773460426021704 0ustar00mauritsstaffAutomatic Buildout Updates ========================== When a buildout is run, one of the first steps performed is to check for updates to either zc.buildout or setuptools. To demonstrate this, we've created some "new releases" of buildout and setuptools in a new_releases folder: >>> ls(new_releases) d zc.buildout - zc.buildout-99.99-py2.4.egg Let's update the sample buildout.cfg to look in this area: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... find-links = %(new_releases)s ... index = %(new_releases)s ... parts = show-versions ... develop = showversions ... ... [show-versions] ... recipe = showversions ... """ % dict(new_releases=new_releases)) We'll also include a recipe that echos the versions of setuptools and zc.buildout used: >>> mkdir(sample_buildout, 'showversions') >>> write(sample_buildout, 'showversions', 'showversions.py', ... """ ... import pkg_resources ... import sys ... print_ = lambda *a: sys.stdout.write(' '.join(map(str, a))+'\\n') ... ... class Recipe: ... ... def __init__(self, buildout, name, options): ... pass ... ... def install(self): ... for project in ['zc.buildout']: ... req = pkg_resources.Requirement.parse(project) ... print_(project, pkg_resources.working_set.find(req).version) ... return () ... update = install ... """) >>> write(sample_buildout, 'showversions', 'setup.py', ... """ ... from setuptools import setup ... ... setup( ... name = "showversions", ... entry_points = {'zc.buildout': ['default = showversions:Recipe']}, ... ) ... """) Now if we run the buildout, the buildout will upgrade itself to the new versions found in new releases: >>> print_(system(buildout), end='') Got zc.buildout NINETYNINE.NINETYNINE. Upgraded: zc.buildout version NINETYNINE.NINETYNINE; Restarting. Generated script '/sample-buildout/bin/buildout'. Develop: '/sample-buildout/showversions' Installing show-versions. zc.buildout NINETYNINE.NINETYNINE Our buildout script has been updated to use the new eggs: >>> cat(sample_buildout, 'bin', 'buildout') #!/usr/local/bin/python2.7 import sys sys.path[0:0] = [ '/sample-buildout/eggs/zc.buildout-99.99-py2.4.egg', ... ] import zc.buildout.buildout if __name__ == '__main__': sys.exit(zc.buildout.buildout.main()) Now, let's recreate the sample buildout. If we specify constraints on the versions of zc.buildout and setuptools to use, running the buildout will install earlier versions of these packages: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... find-links = %(new_releases)s ... index = %(new_releases)s ... parts = show-versions ... develop = showversions ... ... [versions] ... zc.buildout = < 99 ... ... [show-versions] ... recipe = showversions ... """ % dict(new_releases=new_releases)) Now we can see that we actually "upgrade" to an earlier version. >>> print_(system(buildout), end='') Upgraded: zc.buildout V.V Restarting. Generated script '/sample-buildout/bin/buildout'. Develop: '/sample-buildout/showversions' Updating show-versions. zc.buildout V.V There are a number of cases, described below, in which the updates don't happen. We won't upgrade in offline mode: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... find-links = %(new_releases)s ... index = %(new_releases)s ... parts = show-versions ... develop = showversions ... ... [show-versions] ... recipe = showversions ... """ % dict(new_releases=new_releases)) >>> print_(system(buildout+' -o'), end='') Develop: '/sample-buildout/showversions' Updating show-versions. zc.buildout 1.0.0 Or in non-newest mode: >>> print_(system(buildout+' -N'), end='') Develop: '/sample-buildout/showversions' Updating show-versions. zc.buildout 1.0.0 We also won't upgrade if the buildout script being run isn't in the buildouts bin directory. To see this we'll create a new buildout directory: >>> sample_buildout2 = tmpdir('sample_buildout2') >>> write(sample_buildout2, 'buildout.cfg', ... """ ... [buildout] ... find-links = %(new_releases)s ... index = %(new_releases)s ... parts = ... """ % dict(new_releases=new_releases)) >>> cd(sample_buildout2) >>> print_(system(buildout), end='') Creating directory '/sample_buildout2/eggs'. Creating directory '/sample_buildout2/bin'. Creating directory '/sample_buildout2/parts'. Creating directory '/sample_buildout2/develop-eggs'. Got zc.buildout NINETYNINE.NINETYNINE. Not upgrading because not running a local buildout command. >>> ls('bin') .. The relative-paths option is honored: >>> cd(sample_buildout) >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... find-links = %(new_releases)s ... index = %(new_releases)s ... parts = show-versions ... develop = showversions ... relative-paths = true ... ... [show-versions] ... recipe = showversions ... """ % dict(new_releases=new_releases)) >>> print_(system(buildout), end='') Upgraded: zc.buildout version NINETYNINE.NINETYNINE; Restarting. Generated script '/sample-buildout/bin/buildout'. Develop: '/sample-buildout/showversions' Section `buildout` contains unused option(s): 'relative-paths'. This may be an indication for either a typo in the option's name or a bug in the used recipe. Updating show-versions. zc.buildout NINETYNINE.NINETYNINE >>> cat('bin', 'buildout') # doctest +ELL #!/usr/local/bin/python2.7 import os join = os.path.join base = os.path.dirname(os.path.abspath(os.path.realpath(__file__))) base = os.path.dirname(base) import sys sys.path[0:0] = [ join(base, 'eggs/zc.buildout-99.99-py3.3.egg'), ... ] import zc.buildout.buildout if __name__ == '__main__': sys.exit(zc.buildout.buildout.main()) When buildout restarts and the restarted buildout exits with an error code, the original buildout that called the second buildout also exits with that error code. Otherwise build scripts can erroneously detect a successful buildout run even if it failed. Make a recipe that fails: >>> mkdir(sample_buildout, 'failrecipe') >>> write(sample_buildout, 'failrecipe', 'failrecipe.py', ... """ ... import pkg_resources ... import sys ... print_ = lambda *a: sys.stdout.write(' '.join(map(str, a))+'\\n') ... ... class Recipe: ... ... def __init__(self, buildout, name, options): ... sys.exit('recipe sys-exits') ... ... def install(self): ... pass ... ... update = install ... """) >>> write(sample_buildout, 'failrecipe', 'setup.py', ... """ ... from setuptools import setup ... ... setup( ... name = "failrecipe", ... entry_points = {'zc.buildout': ['default = failrecipe:Recipe']}, ... ) ... """) Let's downgrade again, triggering a restart. And use the failing recipe that gives us a sys.exit: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... find-links = %(new_releases)s ... index = %(new_releases)s ... parts = fail ... develop = failrecipe ... ... [versions] ... zc.buildout = < 99 ... ... [fail] ... recipe = failrecipe ... """ % dict(new_releases=new_releases)) Run the buildout: >>> print_(system(buildout, with_exit_code=True), end='') Upgraded: zc.buildout V.V Restarting. Generated script '/sample-buildout/bin/buildout'. Develop: '/sample-buildout/failrecipe' recipe sys-exits EXIT CODE: 1 ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675670.0 zc_buildout-4.1.6/src/zc/buildout/tests/windows.txt0000644000076500000240000000375114773460426022126 0ustar00mauritsstaffzc.buildout on MS-Windows ========================= Certain aspects of every software project are dependent on the operating system used. The same - of course - applies to zc.buildout. To test that Windows doesn't get in the way, we'll test some system dependent aspects. The following recipe will create a read-only file which shutil.rmtree can't delete. >>> mkdir('recipe') >>> write('recipe', 'recipe.py', ... ''' ... import os ... import sys ... print_ = lambda *a: sys.stdout.write(' '.join(map(str, a))+'\\n') ... class Recipe: ... def __init__(self, buildout, name, options): ... self.location = os.path.join( ... buildout['buildout']['parts-directory'], ... name) ... ... def install(self): ... print_("can't remove read only files") ... if not os.path.exists (self.location): ... os.makedirs (self.location) ... ... name = os.path.join (self.location, 'readonly.txt') ... with open (name, 'w') as f: f.write ('this is a read only file') ... os.chmod(name, 256) ... return () ... ... update = install ... ''') >>> write('recipe', 'setup.py', ... ''' ... from setuptools import setup ... setup(name='spam', version='1', py_modules=['recipe'], ... entry_points={'zc.buildout': ['default = recipe:Recipe']}, ... ) ... ''') >>> write('recipe', 'README', '') >>> print_(system(buildout+' setup recipe bdist_egg')) # doctest: +ELLIPSIS Running setup script 'recipe/setup.py'. ... and we'll configure a buildout to use it: >>> write('buildout.cfg', ... ''' ... [buildout] ... parts = foo ... find-links = %s ... ... [foo] ... recipe = spam ... ''' % join('recipe', 'dist')) >>> print_(system(buildout), end='') Getting distribution for 'spam'. Got spam 1. Installing foo. can't remove read only files ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675670.0 zc_buildout-4.1.6/src/zc/buildout/utils.py0000644000076500000240000000070714773460426020241 0ustar00mauritsstaffimport re def normalize_name(name): """PEP 503 normalization plus dashes as underscores. Taken over from importlib.metadata. I don't want to think about where to import this from in each Python version, or having it as extra dependency. Note that there is also packaging_utils.canonicalize_name which turns "foo.bar" into "foo-bar", so it is different. """ return re.sub(r"[-_.]+", "-", name).lower().replace('-', '_') ././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1743675672.410005 zc_buildout-4.1.6/src/zc.buildout.egg-info/0000755000076500000240000000000014773460430020207 5ustar00mauritsstaff././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675672.0 zc_buildout-4.1.6/src/zc.buildout.egg-info/PKG-INFO0000644000076500000240000007331214773460430021312 0ustar00mauritsstaffMetadata-Version: 2.4 Name: zc.buildout Version: 4.1.6 Summary: System for managing development buildouts Home-page: http://buildout.org Author: Jim Fulton Author-email: jim@zope.com License: ZPL 2.1 Keywords: development build Classifier: Development Status :: 6 - Mature Classifier: Intended Audience :: Developers Classifier: License :: OSI Approved :: Zope Public License Classifier: Programming Language :: Python Classifier: Programming Language :: Python :: 3.9 Classifier: Programming Language :: Python :: 3.10 Classifier: Programming Language :: Python :: 3.11 Classifier: Programming Language :: Python :: 3.12 Classifier: Programming Language :: Python :: 3.13 Classifier: Topic :: Software Development :: Build Tools Classifier: Topic :: Software Development :: Libraries :: Python Modules Requires-Python: >=3.9 License-File: LICENSE.txt Requires-Dist: setuptools>=49.0.0 Requires-Dist: packaging Requires-Dist: pip Requires-Dist: wheel Provides-Extra: test Requires-Dist: zope.testing; extra == "test" Requires-Dist: manuel; extra == "test" Requires-Dist: bobo==2.3.0; extra == "test" Requires-Dist: zdaemon; extra == "test" Requires-Dist: zc.zdaemonrecipe; extra == "test" Requires-Dist: zc.recipe.deployment; extra == "test" Dynamic: author Dynamic: author-email Dynamic: classifier Dynamic: description Dynamic: home-page Dynamic: keywords Dynamic: license Dynamic: license-file Dynamic: provides-extra Dynamic: requires-dist Dynamic: requires-python Dynamic: summary ******** Buildout ******** .. image:: https://github.com/buildout/buildout/actions/workflows/run-tests.yml/badge.svg :alt: GHA tests report :target: https://github.com/buildout/buildout/actions/workflows/run-tests.yml Buildout is a project designed to solve 2 problems: 1. Application-centric assembly and deployment *Assembly* runs the gamut from stitching together libraries to create a running program, to production deployment configuration of applications, and associated systems and tools (e.g. run-control scripts, cron jobs, logs, service registration, etc.). Buildout might be confused with build tools like make or ant, but it is a little higher level and might invoke systems like make or ant to get its work done. Buildout might be confused with systems like puppet or chef, but it is more application focused. Systems like puppet or chef might use buildout to get their work done. Buildout is also somewhat Python-centric, even though it can be used to assemble and deploy non-python applications. It has some special features for assembling Python programs. It's scripted with Python, unlike, say puppet or chef, which are scripted with Ruby. 2. Repeatable assembly of programs from Python software distributions Buildout puts great effort toward making program assembly a highly repeatable process, whether in a very open-ended development mode, where dependency versions aren't locked down, or in a deployment environment where dependency versions are fully specified. You should be able to check buildout into a VCS and later check it out. Two checkouts built at the same time in the same environment should always give the same result, regardless of their history. Among other things, after a buildout, all dependencies should be at the most recent version consistent with any version specifications expressed in the buildout. Buildout supports applications consisting of multiple programs, with different programs in an application free to use different versions of Python distributions. This is in contrast with a Python installation (real or virtual), where, for any given distribution, there can only be one installed. To learn more about buildout, including how to use it, see https://www.buildout.org/. Change History ************** .. You should *NOT* be adding new change log entries to this file. You should create a file in the news directory instead. For helpful instructions, please see: https://github.com/buildout/buildout/blob/master/doc/ADD-A-NEWS-ITEM.rst .. towncrier release notes start 4.1.6 (2025-04-03) ------------------ Tests - While creating sample packages for testing, mostly create wheels instead of eggs. For the sample source distributions, create ``tar.gz`` instead of ``zip`` files. Then our package index for testing is more like the actual PyPI. [maurits] (#675) 4.1.5 (2025-03-31) ------------------ Bug fixes: - Implement PEP 503: request normalized package url on PyPI servers. [andreclimaco] (#634) - Install ``wheel`` before ``setuptools`` when checking if an upgrade and restart are needed. [maurits] (#691) 4.1.4 (2025-03-07) ------------------ Bug fixes: - If needed, copy and rename wheels before making an egg out of them. This helps for wheels of namespace packages created with ``setuptools`` 75.8.1 or higher. For namespace package we need a dot instead of an underscore in the resulting egg name. [maurits] (#686) 4.1.3 (2025-03-05) ------------------ Bug fixes: - Patch the ``find`` method from ``pkg_resources.WorkingSet``. Let this use the code from ``setuptools`` 75.8.2, if the currently used version is older. This is better at finding installed distributions. But don't patch ``setuptools`` versions older than 61: the new version of the method would give an error there. [maurits] (#682) 4.1.2 (2025-03-05) ------------------ Bug fixes: - Fix error finding the ``zc.buildout`` distribution when checking if we need to upgrade/restart. This depends on your ``setuptools`` version. [maurits] (#681) 4.1.1 (2025-03-04) ------------------ Bug fixes: - Fix error adding minimum ``zc.buildout`` version as requirement. [maurits] (#679) 4.1 (2025-03-04) ---------------- New features: - In the ``ls`` testing method, add keyword argument ``lowercase_and_sort_output``. The default is False, so no change. When true, as the name says, it sorts the output by lowercase, and prints it lowercase. We need this in one test because with ``setuptools`` 75.8.1 we no longer have a filename ``MIXEDCASE-0.5-pyN.N.egg``, but ``mixedcase-0.5-pyN.N.egg``. [maurits] (#7581) Bug fixes: - When trying to find a distribution for ``package.name``, first try the normalized name (``package_name``). This fixes an error finding entry points for namespace packages. The error is: ``TypeError: ('Expected str, Requirement, or Distribution', None)``. [maurits] (#7581) Development: - Test with latest ``setuptools`` 75.8.2 and with ``pip`` 25.0.1. Note that ``setuptools`` 75.8.1 can be troublesome and should be avoided. [maurits] (#7581) 4.0 (2025-01-30) ---------------- Breaking changes: - Drop Python 3.8 support. Require 3.9 as minimum. (#38) Development: - Test against `setuptools == 75.6.0`. (#671) 4.0.0a1 (2024-10-22) -------------------- Breaking changes: - Add dependency on ``packaging``. This gets rid of ugly compatibility code. [maurits] (#38) - Require ``setuptools >= 49.0.0``. This is the first version that supports PEP 496 environment markers, for example ``demo ==0.1; python_version < '3.9'``. An earlier change had ``setuptools >= 42.0.2``, otherwise we got ImportErrors. Also, since this is higher than 38.2.3, we are sure to have support for wheels. Remove support for ``distribute``, which was probably already broken. [maurits] (#38) - Drop support for Python 2. Require Python 3.8 as minimum. [maurits] (#38) New features: - Support Python 3.12 and 3.13. This only needed a few test fixes. [maurits] (#38) 3.3 (2024-10-17) ---------------- New features: - Allow the ``-I`` option in the Python interpreter wrapper installed by buildout when using the ``zc.recipe.egg`` recipe's `interpreter =` directive. This solves the issue when VSCode calls the designated Python interpreter for a workspace with this option to determine the Python version etc. (`#627 `_) 3.2.0 (2024-09-26) ------------------ New features: - Add config option: ``optional-extends``. This is the same as the ``extends`` option, but then for optional files. The names must be file paths, not URLs. If the path does not exist, it is silently ignored. This is useful for optionally loading a ``local.cfg`` or ``custom.cfg`` with options specific for the developer or the server. [maurits] (`#665 `_) 3.1.1 (2024-09-20) ------------------ Bug fixes: - Fix: a variable defined with initial ``+=`` was undefined and would lead to a corrupted ``.installed.cfg``. Fixes `issue 641 `_. [distributist] - Fix: extends with increments could result in missing values. Buildout processes them in the correct order now and combines them correctly. Fixes `issue 176 `_ and `issue 629 `_. [distributist] (#644) - Fix: Multiple ``+=`` or ``/-=`` in one file would lose assignment in a previous file. Fixes `issue 656 `_. [distributist] 3.1.0 (2024-08-29) ------------------ Breaking changes: - Drop support for Python 3.5. It is unsupported, and testing it is too hard. [maurits] (#35) Bug fixes: - Normalize package names when gathering packages. This should help find all distributions for package ``name.space``, whether they are called ``name.space-1.0.tar.gz`` with a dot or ``name_space-1.0.tar.gz`` with an underscore (created with ``setuptools`` 69.3 or higher). [maurits] (#647) - Fix ImportError: cannot import name ``packaging`` from ``pkg_resources`` with setuptools 70. Done by adding a compatibility module that tries to import `packaging` from several places. Fixes `issue 648 `_. [maurits] (#648) 3.0.1 (2022-11-08) ------------------ Bug fixes: - Fixed import of packaging.markers. [maurits] (#621) 3.0.0 (2022-11-07) ------------------ New features: - Add support for PEP 508 markers in section condition expressions. For example: ``[versions:python_version <= "3.9"]``. [maurits] (#621) Bug fixes: - Command-line 'extends' now works with dirs in file names [gotcha] (cli-extends) - Add support for python311-315 in conditional section expressions. (#311) - Make compatible with pip 22.2+, restoring Requires-Python functionality there. Fixes `issue 613 `_. [maurits] (#613) 3.0.0rc3 (2022-04-07) --------------------- Bug fixes: - Fix `TypeError: dist must be a Distribution instance` due to issue between `setuptools` and `pip`. (#600) 3.0.0rc2 (2022-03-04) --------------------- New features: - add support for PEP496 environment markers (pep496) Bug fixes: - Fix TypeError for missing required `use_deprecated_html5lib` with pip 22. Keep compatible with earlier pip versions. (#598) 3.0.0rc1 (2021-12-16) --------------------- Bug fixes: - Call pip via `python -m pip`. (#569) 3.0.0b5 (2021-11-29) -------------------- Bug fixes: - Fix when c extension implements namespace packages without the corresponding directories. (#589) - Honor command-line buildout:extends (#592) 3.0.0b4 (2021-11-25) -------------------- New features: - Allow to run buildout in FIPS enabled environments. (#570) - Proper error message if extends-cache tries to expand ${section:variable} (#585) Bug fixes: - Forward verbose option to pip (#576) - Check that file top_level.txt exists before opening. Add check for other files as well. (#582) - Return code of pip install subprocess is now properly returned to buildout. (#586) 3.0.0b3 (2021-10-08) -------------------- New features: - Improve warning message when a section contains unused options. (#483) Bug fixes: - Fix support of ``pip>=21.1`` (#567) - Fix confusion when using multiple Python versions and installing packages with C extensions without proper binary wheel available. (#574) Development: - Avoid broken jobs on Travis because of security on PRs (travis-pr) 3.0.0b2 (2021-03-09) -------------------- New features: - Improve error message when a package version is not pinned and `allow-picked-versions = false`. (#481) Bug fixes: - Fix FileNotFoundError when installing eggs with top-level directory without code (like doc). (#556) Development: - Login to docker hub to avoid pull limits (travis) - Initialize towncrier (#519) 3.0.0b1 (2021-03-07) ==================== - Fix issue with combination of `>` specs and `extras` and recent `setuptools`. - Fix issue with incrementing options from `.buildout/default.cfg`. - Support python37, python38 and python39 in conditional section expressions. - Fix bootstrapping for python27 and python35. 3.0.0a2 (2020-05-25) ==================== - Ignore `.git` when computing signature of a recipe develop egg - Warn when the name passed to `zc.recipe.egg:scripts` is not defined in egg entry points. - Show pip warning about Python version only once. - Better patch for ``pkg_resources.Distribution.hashcmp`` performance. 3.0.0a1 (2020-05-17) ==================== - Scripts: ensure eggs are inserted before ``site-packages`` in ``sys.path``. - Fix forever loop when changing ``zc.buildout`` version via ``buildout``. - Add support for ``Requires-Python`` metadata. Fragile monkeypatch that relies on ``pip._internal``. Emits a warning when support is disabled due to changes in ``pip``. - Use ``pip install`` instead of deprecated ``setuptools.easy_install``. - Patch ``pkg_resources.Distribution`` to make install of unpinned versions quicker. Most obvious with ``setuptools``. 2.13.3 (2020-02-11) =================== - Fix DeprecationWarning about MutableMapping. (`#484 `_) 2.13.2 (2019-07-03) =================== - Fixed DeprecationWarning on python 3.7: "'U' mode is deprecated". 2.13.1 (2019-01-29) =================== - Documentation update for the new ``buildout query`` command. 2.13.0 (2019-01-17) =================== - Get information about the configuration with new command ``buildout query``. 2.12.2 (2018-09-04) =================== - Upon an error, buildout exits with a non-zero exit code. This now also works when running with ``-D``. - Fixed most 'Deprecation' and 'Resource' warnings. 2.12.1 (2018-07-02) =================== - zc.buildout now explicitly requests zc.recipe.egg >=2.0.6 now. 2.12.0 (2018-07-02) =================== - Add a new buildout option ``allow-unknown-extras`` to enable installing requirements that specify extras that do not exist. This needs a corresponding update to zc.recipe.egg. See `issue 457 `_. zc.recipe.egg has been updated to 2.0.6 for this change. 2.11.5 (2018-06-19) =================== - Fix for `issue 295 `_. On windows, deletion of temporary egg files is more robust now. 2.11.4 (2018-05-14) =================== - Fix for `issue 451 `_: distributions with a version number that normalizes to a shorter version number (3.3.0 to 3.3, for instance) can be installed now. 2.11.3 (2018-04-13) =================== - Update to use the new PyPI at https://pypi.org/. 2.11.2 (2018-03-19) =================== - Fix for the #442 issue: AttributeError on ``pkg_resources.SetuptoolsVersion``. 2.11.1 (2018-03-01) =================== - Made upgrade check more robust. When using extensions, the improvement introduced in 2.11 could prevent buildout from restarting itself when it upgraded setuptools. 2.11.0 (2018-01-21) =================== - Installed packages are added to the working set immediately. This helps in some corner cases that occur when system packages have versions that conflict with our specified versions. 2.10.0 (2017-12-04) =================== - Setuptools 38.2.0 started supporting wheels. Through setuptools, buildout now also supports wheels! You need at least version 38.2.3 to get proper namespace support. This setuptools change interfered with buildout's recent support for `buildout.wheel `_, resulting in a sudden "Wheels are not supported" error message (see `issue 435 `_). Fixed by making setuptools the default, though you can still use the buildout.wheel if you want. 2.9.6 (2017-12-01) ================== - Fixed: could not install eggs when sdist file name and package name had different case. 2.9.5 (2017-09-22) ================== - Use HTTPS for PyPI's index. PyPI redirects HTTP to HTTPS by default now so using HTTPS directly avoids the potential for that redirect being modified in flight. 2.9.4 (2017-06-20) ================== - Sort the distributions used to compute ``__buildout_signature__`` to ensure reproducibility under Python 3 or under Python 2 when ``-R`` is used on ``PYTHONHASHSEED`` is set to ``random``. Fixes `issue 392 `_. **NOTE**: This may cause existing ``.installed.cfg`` to be considered outdated and lead to parts being reinstalled spuriously under Python 2. - Add support code for doctests to be able to easily measure code coverage. See `issue 397 `_. 2.9.3 (2017-03-30) ================== - Add more verbosity to ``annotate`` results with ``-v`` - Select one or more sections with arguments after ``buildout annotate``. 2.9.2 (2017-03-06) ================== - Fixed: We unnecessarily used a function from newer versions of setuptools that caused problems when older setuptools or pkg_resources installs were present (as in travis.ci). 2.9.1 (2017-03-06) ================== - Fixed a minor packaging bug that broke the PyPI page. 2.9.0 (2017-03-06) ================== - Added new syntax to explicitly declare that a part depends on other part. See http://docs.buildout.org/en/latest/topics/implicit-parts.html - Internal refactoring to work with `buildout.wheel `_. - Fixed a bugs in ``zc.buildout.testing.Buildout``. It was loading user-default configuration. It didn't support calling the ``created`` method on its sections. - Fixed a bug (windows, py 3.4) When processing metadata on "old-style" distutils scripts, .exe stubs appeared in ``metadata_listdir``, in turn reading those burped with ``UnicodeDecodeError``. Skipping .exe stubs now. 2.8.0 (2017-02-13) ================== - Added a hook to enable a soon-to-be-released buildout extension to provide wheel support. 2.7.1 (2017-01-31) ================== - Fixed a bug introduced in 2.6.0: zc.buildout and its dependeoncies were reported as picked even when their versions were fixed in a ``versions`` section. Worse, when the ``update-versions-file`` option was used, the ``versions`` section was updated needlessly on every run. 2.7.0 (2017-01-30) ================== - Added a buildout option, ``abi-tag-eggs`` that, when true, causes the `ABI tag `_ for the buildout environment to be added to the eggs directory name. This is useful when switching Python implementations (e.g. CPython vs PyPI or debug builds vs regular builds), especially when environment differences aren't reflected in egg names. It also has the side benefit of making eggs directories smaller, because eggs for different Python versions are in different directories. 2.6.0 (2017-01-29) ================== - Updated to work with the latest setuptools. - Added (verified) Python 3.6 support. 2.5.3 (2016-09-05) ================== - After a dist is fetched and put into its final place, compile its python files. No longer wait with compiling until all dists are in place. This is related to the change below about not removing an existing egg. [maurits] - Do not remove an existing egg. When installing an egg to a location that already exists, keep the current location (directory or file). This can only happen when the location at first did not exist and this changed during the buildout run. We used to remove the previous location, but this could cause problems when running two buildouts at the same time, when they try to install the same new egg. Fixes #307. [maurits] - In ``zc.buildout.testing.system``, set ``TERM=dumb`` in the environment. This avoids invisible control characters popping up in some terminals, like ``xterm``. Note that this may affect tests by buildout recipes. [maurits] - Removed Python 2.6 and 3.2 support. [do3cc] 2.5.2 (2016-06-07) ================== - Fixed ``-=`` and ``+=`` when extending sections. See #161. [puittenbroek] 2.5.1 (2016-04-06) ================== - Fix python 2 for downloading external config files with basic auth in the URL. Fixes #257. 2.5.0 (2015-11-16) ================== - Added more elaborate version and requirement information when there's a version conflict. Previously, you could get a report of a version conflict without information about which dependency requested the conflicing requirement. Now all this information is logged and displayed in case of an error. [reinout] - Dropped 3.2 support (at least in the automatic tests) as setuptools will soon stop supporting it. Added python 3.5 to the automatic tests. [reinout] 2.4.7 (2015-10-29) ================== - Fix for #279. Distutils script detection previously broke on windows with python 3 because it errored on ``.exe`` files. [reinout] 2.4.6 (2015-10-28) ================== - Relative paths are now also correctly generated for the current directory ("develop = ."). [youngking] 2.4.5 (2015-10-14) ================== - More complete fix for #24. Distutils scripts are now also generated for develop eggs. [reinout] 2.4.4 (2015-10-02) ================== - zc.buildout is now also released as a wheel. (Note: buildout itself doesn't support installing wheels yet.) [graingert] 2.4.3 (2015-09-03) ================== - Added nested directory creation support [guyzmo] 2.4.2 (2015-08-26) ================== - If a downloaded config file in the "extends-cache" gets corrupted, buildout now tells you the filename in the cache. Handy for troubleshooting. [reinout] 2.4.1 (2015-08-08) ================== - Check the ``use-dependency-links`` option earlier. This can give a small speed increase. [maurits] - When using python 2, urllib2 is used to work around Python issue 24599, which affects downloading from behind a proxy. [stefano-m] 2.4.0 (2015-07-01) ================== - Buildout no longer breaks on packages that contain a file with a non-ascii filename. Fixes #89 and #148. [reinout] - Undo breakage on Windows machines where ``sys.prefix`` can also be a ``site-packages`` directory: don't remove it from ``sys.path``. See https://github.com/buildout/buildout/issues/217 . - Remove assumption that ``pkg_resources`` is a module (untrue since release of `setuptools 8.3``). See https://github.com/buildout/buildout/issues/227 . - Fix for #212. For certain kinds of conflict errors you'd get an UnpackError when rendering the error message. Instead of a nicely formatted version conflict message. [reinout] - Making sure we use the correct easy_install when setuptools is installed globally. See https://github.com/buildout/buildout/pull/232 and https://github.com/buildout/buildout/pull/222 . [lrowe] - Updated buildout's `travis-ci `_ configuration so that tests run much quicker so that buildout is easier and quicker to develop. [reinout] - Note: zc.recipe.egg has also been updated to 2.0.2 together with this zc.buildout release. Fixed: In ``zc.recipe.egg#custom`` recipe's ``rpath`` support, don't assume path elements are buildout-relative if they start with one of the "special" tokens (e.g., ``$ORIGIN``). See: https://github.com/buildout/buildout/issues/225. [tseaver] - ``download-cache``, ``eggs-directory`` and ``extends-cache`` are now automatically created if their parent directory exists. Also they can be relative directories (relative to the location of the buildout config file that defines them). Also they can now be in the form ``~/subdir``, with the usual convention that the ``~`` char means the home directory of the user running buildout. [lelit] - A new bootstrap.py file is released (version 2015-07-01). - When bootstrapping, the ``develop-eggs/`` directory is first removed. This prevents old left-over ``.egg-link`` files from breaking buildout's careful package collection mechanism. [reinout] - The bootstrap script now accepts ``--to-dir``. Setuptools is installed there. If already available there, it is reused. This can be used to bootstrap buildout without internet access. Similarly, a local ``ez_setup.py`` is used when available instead of it being downloaded. You need setuptools 14.0 or higher for this functionality. [lrowe] - The bootstrap script now uses ``--buildout-version`` instead of ``--version`` to pick a specific buildout version. [reinout] - The bootstrap script now accepts ``--version`` which prints the bootstrap version. This version is the date the bootstrap.py was last changed. A date is handier or less confusing than either tracking zc.buildout's version or having a separate bootstrap version number. [reinout] 2.3.1 (2014-12-16) ================== - Fixed: Buildout merged single-version requirements with version-range requirements in a way that caused it to think there wasn't a single-version requirement. IOW, buildout through that versions were being picked when they weren't. - Suppress spurious (and possibly non-spurious) version-parsing warnings. 2.3.0 (2014-12-14) ================== - Buildout is now compatible with (and requires) setuptools 8. 2.2.5 (2014-11-04) ================== - Improved fix for #198: when bootstrapping with an extension, buildout was too strict on itself, resulting in an inability to upgrade or downgrade its own version. [reinout] - Setuptools must be at 3.3 or higher now. If you use the latest bootstrap from http://downloads.buildout.org/2/bootstrap.py you're all set. [reinout] - Installing *recipes* that themselves have dependencies used to fail with a VersionConflict if such a dependency was installed globally with a lower version. Buildout now ignores the version conflict in those cases and simply installs the correct version. [reinout] 2.2.4 (2014-11-01) ================== - Fix for #198: buildout 2.2.3 caused a version conflict when bootstrapping a buildout with a version pinned to an earlier one. Same version conflict could occur with system-wide installed packages that were newer than the pinned version. [reinout] 2.2.3 (2014-10-30) ================== - Fix #197, Python 3 regression [aclark4life] 2.2.2 (2014-10-30) ================== - Open files for ``exec()`` in universal newlines mode. See https://github.com/buildout/buildout/issues/130 - Add ``BUILDOUT_HOME`` as an alternate way to control how the user default configuration is found. - Close various files when finished writing to them. This avoids ResourceWarnings on Python 3, and better supports doctests under PyPy. - Introduce improved easy_install Install.install function. This is present in 1.5.X and 1.7X but was never merged into 2.X somehow. 2.2.1 (2013-09-05) ================== - ``distutils`` scripts: correct order of operations on ``from ... import`` lines (see https://github.com/buildout/buildout/issues/134). - Add an ``--allow-site-packges`` option to ``bootstrap.py``, defaulting to False. If the value is false, strip any "site packages" (as defined by the ``site`` module) from ``sys.path`` before attempting to import ``setuptools`` / ``pkg_resources``. - Updated the URL used to fetch ``ez_setup.py`` to the official, non-version- pinned version. 2.2.0 (2013-07-05) ================== - Handle both addition and subtraction of elements (+= and -=) on the same key in the same section. Forward-ported from buildout 1.6. - Suppress the useless ``Link to ***BLOCKED*** by --allow-hosts`` error message being emitted by distribute / setuptools. - Extend distutils script generation to support module docstrings and __future__ imports. - Refactored picked versions logic to make it easier to use for plugins. - Use ``get_win_launcher`` API to find Windows launcher (falling back to ``resource_string`` for ``cli.exe``). - Remove ``data_files`` from ``setup.py``: it was installing ``README.txt`` in current directory during installation (merged from 1.x branch). - Switch dependency from ``distribute 0.6.x`` to ``setuptools 0.7.x``. 2.1.0 (2013-03-23) ================== - Meta-recipe support - Conditional sections - Buildout now accepts a ``--version`` command-line option to print its version. Fixed: Builout didn't exit with a non-zero exit status if there was a failure in combination with an upgrade. Fixed: We now fail with an informative error when an old bootstrap script causes buildout 2 to be used with setuptools. Fixed: An error incorrectly suggested that buildout 2 implemented all of the functionality of dumppickedversions. Fixed: Buildout generated bad scripts when no eggs needed to be added to ``sys.path``. Fixed: Buildout didn't honour Unix umask when generating scripts. https://bugs.launchpad.net/zc.buildout/+bug/180705 Fixed: ``update-versions-file`` didn't work unless ``show-picked-versions`` was also set. https://github.com/buildout/buildout/issues/71 2.0.1 (2013-02-16) ================== - Fixed: buildout didn't honor umask settings when creating scripts. - Fix for distutils scripts installation on Python 3, related to ``__pycache__`` directories. - Fixed: encoding data in non-entry-point-based scripts was lost. ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675672.0 zc_buildout-4.1.6/src/zc.buildout.egg-info/SOURCES.txt0000644000076500000240000000326214773460430022076 0ustar00mauritsstaff.coveragerc CHANGES.rst CONTRIBUTING.rst COPYRIGHT.txt HISTORY.rst LICENSE.txt MANIFEST.in README.rst dev.py pyproject.toml setup.cfg setup.py specifications/README.txt specifications/repeatable.txt src/zc/__init__.py src/zc.buildout.egg-info/PKG-INFO src/zc.buildout.egg-info/SOURCES.txt src/zc.buildout.egg-info/dependency_links.txt src/zc.buildout.egg-info/entry_points.txt src/zc.buildout.egg-info/namespace_packages.txt src/zc.buildout.egg-info/not-zip-safe src/zc.buildout.egg-info/requires.txt src/zc.buildout.egg-info/top_level.txt src/zc/buildout/__init__.py src/zc/buildout/buildout.py src/zc/buildout/configparser.py src/zc/buildout/download.py src/zc/buildout/easy_install.py src/zc/buildout/patches.py src/zc/buildout/pep425tags.py src/zc/buildout/rmtree.py src/zc/buildout/testing.py src/zc/buildout/testing.txt src/zc/buildout/testrecipes.py src/zc/buildout/utils.py src/zc/buildout/tests/__init__.py src/zc/buildout/tests/allow-unknown-extras.txt src/zc/buildout/tests/allowhosts.txt src/zc/buildout/tests/bootstrap.txt src/zc/buildout/tests/bootstrap_cl_settings.test src/zc/buildout/tests/buildout.txt src/zc/buildout/tests/configparser.test src/zc/buildout/tests/debugging.txt src/zc/buildout/tests/dependencylinks.txt src/zc/buildout/tests/download.txt src/zc/buildout/tests/downloadcache.txt src/zc/buildout/tests/easy_install.txt src/zc/buildout/tests/extends-cache.txt src/zc/buildout/tests/repeatable.txt src/zc/buildout/tests/runsetup.txt src/zc/buildout/tests/setup.txt src/zc/buildout/tests/test_all.py src/zc/buildout/tests/test_extras.py src/zc/buildout/tests/test_increment.py src/zc/buildout/tests/testing_bugfix.txt src/zc/buildout/tests/update.txt src/zc/buildout/tests/windows.txt././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675672.0 zc_buildout-4.1.6/src/zc.buildout.egg-info/dependency_links.txt0000644000076500000240000000000114773460430024255 0ustar00mauritsstaff ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675672.0 zc_buildout-4.1.6/src/zc.buildout.egg-info/entry_points.txt0000644000076500000240000000015414773460430023505 0ustar00mauritsstaff[console_scripts] buildout = zc.buildout.buildout:main [zc.buildout] debug = zc.buildout.testrecipes:Debug ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675672.0 zc_buildout-4.1.6/src/zc.buildout.egg-info/namespace_packages.txt0000644000076500000240000000000314773460430024533 0ustar00mauritsstaffzc ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675672.0 zc_buildout-4.1.6/src/zc.buildout.egg-info/not-zip-safe0000644000076500000240000000000114773460430022435 0ustar00mauritsstaff ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675672.0 zc_buildout-4.1.6/src/zc.buildout.egg-info/requires.txt0000644000076500000240000000017514773460430022612 0ustar00mauritsstaffsetuptools>=49.0.0 packaging pip wheel [test] zope.testing manuel bobo==2.3.0 zdaemon zc.zdaemonrecipe zc.recipe.deployment ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1743675672.0 zc_buildout-4.1.6/src/zc.buildout.egg-info/top_level.txt0000644000076500000240000000000314773460430022732 0ustar00mauritsstaffzc