pax_global_header00006660000000000000000000000064147650046100014515gustar00rootroot0000000000000052 comment=fb58f40ae348992f3309e11ce878e4902ba6ba4a wikimedia-cumin-36f957f/000077500000000000000000000000001476500461000151445ustar00rootroot00000000000000wikimedia-cumin-36f957f/.coveragerc000066400000000000000000000000511476500461000172610ustar00rootroot00000000000000[run] branch = True omit = cumin/tests/* wikimedia-cumin-36f957f/.gitignore000066400000000000000000000002041476500461000171300ustar00rootroot00000000000000/build/ /doc/build/ /.cache/ /.coverage* /.eggs/ /.pytest_cache/ /.tox/ /.venv/ /dist/ /logs/ /*.egg-info/ __pycache__/ *.pyc *.sw? wikimedia-cumin-36f957f/.gitreview000066400000000000000000000001341476500461000171500ustar00rootroot00000000000000[gerrit] host=gerrit.wikimedia.org port=29418 project=operations/software/cumin.git track=1 wikimedia-cumin-36f957f/.readthedocs.yml000066400000000000000000000003451476500461000202340ustar00rootroot00000000000000version: 2 build: os: "ubuntu-22.04" tools: python: "3" python: install: - method: pip path: . extra_requirements: - tests sphinx: configuration: "doc/source/conf.py" fail_on_warning: false wikimedia-cumin-36f957f/.wmfconfig000066400000000000000000000002221476500461000171200ustar00rootroot00000000000000[github] repository = cumin [pypi] name = cumin wheels_prefix = cumin [debian] distros = bullseye source_package = cumin binary_package = cumin wikimedia-cumin-36f957f/CHANGELOG.rst000066400000000000000000001301771476500461000171760ustar00rootroot00000000000000Cumin Changelog --------------- `v5.1.1`_ (2025-03-14) ^^^^^^^^^^^^^^^^^^^^^^ Bug fixes """"""""" * puppetdb: fix inventory query with quoted part in dot-notation for the ``/inventory`` endpoint. `v5.1.0`_ (2025-03-13) ^^^^^^^^^^^^^^^^^^^^^^ New features """""""""""" * puppetdb: add support for structured facts and more querying the inventory endpoint of PuppetDB, that allows to query structured facts and other resources exposed by this endpoint with the PuppetDB's dot notation. Minor improvements """""""""""""""""" * cli: log an eventual exception to stderr when ``--trace`` is set in addition to the log entry in the logs on file for better visibility (`T384539`_). Bug fixes """"""""" * query: do not error on no match in first subquery. When running queries that have subqueries and the first subquery is empty, cumin is currently raising an ``InvalidQueryError`` while that's still a valid result for the first subquery and should not raise. Miscellanea """"""""""" * docs: removed deprecated call to ``sphinx_rtd_theme``. `v5.0.0`_ (2025-01-16) ^^^^^^^^^^^^^^^^^^^^^^ Dependencies breaking changes """"""""""""""""""""""""""""" * Drop Python support for 3.7, 3.8, add suppoert for 3.11, 3.12 and 3.13. * Drop support for Python 3.7 and 3.8 (up to Debian Buster). * Add support for Python 3.11, 3.12 and 3.13. * Drop testing on Python 3.10 to speed up CI as that's a version not present in any Debian release. * Update minimum dependencies to the version in Debian Bullseye (apart few exceptions). * Move the tests for minimum version from Python 3.7 to 3.9. * Fix .gitignore matching for .coverage* files. * Add temporary pinning of some sphinx-contrib packages to make it work both on bullseye and sid. API breaking changes """""""""""""""""""" * puppetdb: drop support for deprecated API v3. As PuppetDB have since long time deprecated version 3 of the API in favor of version 4 and there is no plan of a version 5, cleaning also the multi-version support and stop reading the version from the configuration file. Minor improvements """""""""""""""""" * puppetdb backend: updated the query so that we only select the certname, which is all cumin needs, it also uses group by to only return unique results. Bug fixes """"""""" * setup.py: force a newer sphinx_rtd_theme. * tests: make it compatible with urllib3 v2.0+. * tox.ini: make it compatible with tox 4.0+. * tox.ini: add compatibility with newer Sphinx 7.1+ using ``sphinx-build`` instead of ``setup.py build_sphinx.``. * doc: update .readthedocs.yml configuration to be compatible with their version 2 of the configuration file schema. * doc: don't fail on warning on readthedocs, we currently have a warning to keep the compatibility between bullseye and sid when generating the documentation. Miscellanea """"""""""" * doc: mention inclusion into Debian upstream. * Use importlib.metadata instead of pkg_resources now that Python 3.7 support has been dropped, remove the need for the pkg_resource library and use importlib.metadata instead. * integration tests: use linuxserver/openssh-server and adapt config to use a dedicated user and not the root user to the container. `v4.2.0`_ (2023-01-12) ^^^^^^^^^^^^^^^^^^^^^^ Minor improvements """""""""""""""""" * backends.puppetdb: add a timeout to all requests calls to PuppetDB APIs. Allow to set the timeout from the configuration file, with a default of 30 seconds if not set. * backends.puppetdb: support using TLS client certificates when communicating with the PuppetDB server. * setup.py: support Python 3.10 and Pyparsing 3 * cumin: set ``__version__`` during Debian build: * In order to better support building in a pure Debian environment where the tests and the man page generation are done without installing the package, add support for reading the ``setuptools_scm`` fake version when exported via ``SETUPTOOLS_SCM_PRETEND_VERSION``. Bug fixes """"""""" * CLI: minor fix on confirmation message. * backends.puppetdb: fix ``urllib3`` import. More recent versions of ``requests`` don't ship ``urllib3`` embedded, import that directly. * setup.py: add max version limit for ``Jinja2`` in the ``tests-min`` environment because it breaks the old Sphinx version with more recent versions. * grammars: remove usage of ``leaveWhitespace``. The calls to pyparsing's ``leaveWhitespace`` in various grammars was not really necessary and breaks with pyparsing 3+. Removing them in order to add support for pyparsing 3. Miscellanea """"""""""" * pylint: fix newly reported pylint issues and removed unnecessary disable comments. * tests: fix typo in test name * prospector: disable pylint's ``consider-using-f-string`` error as the code still uses ``format()``. To be converted to f-string in the future. * doc: set the default language. Latest Sphinx 5.0 release requires language to not be ``None``, it raises a warning otherwise, and we fail on warnings. * Add configuration for the release script. This adds a WMF-specific configuration file to be used by the release script to make new releases (git tag, debian package, release to apt.w.o, PyPI release). * mypy: remove unnecessary ignores now that the upstream bug has been fixed. * doc: remove broken badges from the README. * setup.py: temporary fix for test dependencies * setup.py: add ``python_requires`` metadata. * setup.py: set a temporary upper limit for pylint and prospector. * Refactor tox to simplify the testing matrix testing the minimum version of dependencies only with the oldest supported Python version (3.7). `v4.1.1`_ (2021-06-23) ^^^^^^^^^^^^^^^^^^^^^^ New features """""""""""" * config: add support for Kerberos auth (`T244840`_): * In an environment where when running Cumin the user authenticates the SSH connection to the remote hosts via Kerberos, in case the user doesn't have a valid Kerberos ticket, it would get a cryptic authentication failure message. * In order to present the user a more meaningful error, a new configuration stanza named ``kerberos`` is added, see the example configuration file for all the details. * When configured to do so Cumin will ensure that the running user has a valid Kerberos ticket before trying to SSH to the target hosts, and present the user a nicer error message otherwise. * When using Cumin as a library, the ``cumin.ensure_kerberos_ticket`` function can be used to achieve the same functionality. Miscellanea """"""""""" * setup.py: add types dependencies for mypy for dependencies that don't have yet type hints. * doc: use ``add_css_file()`` instead of ``add_stylesheet()`` as Sphinx version 4 removed the old name. `v4.1.0`_ (2021-05-03) ^^^^^^^^^^^^^^^^^^^^^^ CLI breaking changes """""""""""""""""""" * cli: change confirmation input check * When asking for confirmation to execute a command in interactive mode, instead of ``y/n`` ask the user to enter the exact number of affected hosts to make sure they are aware of the impact of the action to be performed and prevent muscle memory errors. * Inspired by: https://rachelbythebay.com/w/2020/10/26/num/ * cli: in dry-run mode send the list of hosts to stdout (`T212783`_). * To simplify the usage from other tools that want to consume the generated list of hosts when not executing commands (dry-run mode), using cumin just to query matching hosts, send the list to stdout while the rest of the output is sent to stderr. * In conjunction with the new ``-n/--no-color`` option it should allow for an easy piping into additional tools using cumin as a selector of hosts:: cumin -n "QUERY" 2> /dev/null | some_other_tool New features """""""""""" * cli: add a ``-n/--no-colors`` option to suppress any colored output (`T212783`_). * cli/clustershell: allow to disable progress bars (`T212783`_): * Expose in the clustershell module the possibility to disable the show of the progress bars during execution. * Allow to disable the progress bars also in the CLI with a ``--no-progress`` flag. * config: allow using tilde ``~`` to specify config paths. This allows abstracting away the paths and seamlessly allowing per-user configurations. * config: expand user's home directory for logging. Allow to specify a log path relative to the user's home directory in the configuration file for the ``log_file`` entry that will expand ``~`` when present. * clustershell: allow to choose different reporters (`T212783`_): * Make the event handler reporter configuratble between: * A null reporter that doesn't print anything (``NullReporter``) * A Tqdm-compatible reporter that prints just the success/failure results to stderr but doesn't print the actual command outputs (``TqdmQuietReporter``) * A Tqdm-compatible reporter that prints the command outputs to stdout and the success/failure results to stderr, as it was until now (``TqdmReporter``, the default). * setup.py: support more recent PyParsing versions * In order to be able to build Cumin on Debian bullseye, add support for more recent versions of PyParsing that introduced backward incompatible changes. Minor improvements """""""""""""""""" * Add support for Python 3.8 and 3.9 Bug fixes """"""""" * tests: fix dependencies for tests (`T270795`_). * Remove the limitation on prospector as the upstream bug was fixed. * Exclude flake8 from the minimum requirements as we just run the unit tests with the minimum requirements. * This will require the removal of ``python3-flake8`` from the ``Build-Depends`` on ``debian/control`` when doing the next release as ``flake8`` is not needed when building the package. * tests: fix integration tests as the newer versions of the sshd docker container needs a specific environment variable to enable the root user SSH access. * setup.py: Add missing ``long_description_content_type`` parameter. * doc: fix sphinx warning in docstring. Miscellanea """"""""""" * setup.py: revert tqdm upper limit constraint. * As the upstream issue has been fixed in tqdm ``v4.48.0``, remove the upper limit constraint. * Note: cumin will have output issues if used with a tqdm between ``v4.24.0`` and ``v4.48.0`` excluded. * Use ``@abstractmethod`` instead of ``@abstractproperty``. The latter it's actually deprecated in favor of usin ``@abstractmethod`` in conjunction with ``@property`` and ``@example.setter``. * Extracting obvious reporting code to a Reporter class to be able to expose the reporting functionality via the library APIs (`T212783`_). * Introduce an interface for progress bars. * tox: add mypy environment. * In order to start adding type hints to the project, add a mypy environment to tox to ensure those added are correct. * Keep the configuration very light for now until type hints are added to the whole project. * tests: remove unnecessary environmental variables config. As cumin can run as a normal user some configuration to make it think it was run as root is not needed anymore across unit and integration tests. * integration tests: add undeduplicated output test. * The case of undeduplicated output, like when there is only one target host, was not tested by the integration tests. Adding a test to cover that use case too. * tests: fix pip backtracking * With the current setup of minimizing the number of different virtualenvs used by tox we ended up hitting an issue of pip backtracking. As prospector seems to be the most likely culprit here because has a lot of dependencies, and in the past too we had issues between prospector and flake8 dependencies, move prospector to its own virtualenv. * Add also mypy as an explicit dependency. * tests: fix minimum dependency and pytest warning. * Change the behaviour of the -min environment in tox to test with the minimum supported version of only the real dependencies and not the ones used only for the tests, with the only exception of Sphinx-related dependencies that are needed to build the manpage during the Debian build process. * Update pytest's command line options to prevent deprecation warnings. `v4.0.0`_ (2020-09-10) ^^^^^^^^^^^^^^^^^^^^^^ * No changes from the RC1 release. `v4.0.0rc1`_ (2020-06-09) ^^^^^^^^^^^^^^^^^^^^^^^^^ Dependency breaking changes """"""""""""""""""""""""""" * tqdm: limit the compatible versions of tqdm allowed to a small range of versions between ``4.19.4`` and ``4.24.0``) due to an upstream bug, see `tqdm issue #777`_. The ``4.23.4-1~wmf1`` version of tqdm is available as a Debian package for buster in the Wikimedia APT repository in the ``component/spicerack`` component. New features """""""""""" * Replace colorama with custom module (`T217038`_). * In Debian stretch there is a regression in colorama in conjunction with tqdm that leads to a slow down of the progress of the script proportional to the amount of data printed to stdout/err. Colorama starts having very huge stacktraces and the process is stuck at 100% CPU for an increasingly amount of time while more data is printed. * Given the very simple usage of colors that is made in Cumin as of now, it seems much more feasible to replace the colorama library (as all that cross-OS support is not needed) and add a simple module with ANSI escape sequence support. * Use a type (metaclass) to be able to override ``__getattr__`` for the static methods of the classes that use it and to automatically define a method for each color in a DRY way without code duplication. * Define a ``Colored`` class that uses ``ColoredType`` as metaclass to inherit its type with the custom behaviour. * For each color defined in ``ColoredType.COLORS`` a method of ``Colored`` is defined, e.g. ``Colored.red()``. * The ``Colored`` class has a ``disabled`` property that can be set to ``True`` to globally disable coloring. This could for example be integrated later into the CLI as an option to disable colors or allow to add some code to the ``color.py`` module to autodetect when not in a TTY and automatically disable all colors. * Allow running cumin as a regular user (`T218440`_). * backends.puppetdb: make the PuppetDB backend process primitive types for queries (`T207037`_). * Modify the grammar to recognize primitive PuppetDB types, communicate quotedness to the final output as appropriate. * backends.puppetdb: allow to override the URL scheme in the configuration (`T218441`_). * In some environments the PuppetDB hosts might listen only on HTTP on localhost and the Cumin host might connect to it via an SSH tunnel. * Allow to override the default HTTPS scheme of the PuppetDB URL in the configuration. * backends.puppetdb: fix regex matching. * Fix regex matching in PuppetDB queries that requires that all backslashes are escaped according to the PuppetDB API. See PuppetDB documentation on `regexp-match`_. * backends.openstack: add custom parameters for the client (`T201881`_). * The instantiation of the novaclient ``Client`` might require additional parameters based on the specific OpenStack installation, like for example a ``region_name``. * Add a generic ``client_params`` section to the configuration to allow to set arbitrary additional parameters that will be passed to the novalicent's ``Client``. * CLI: improve help message (`T204680`_). * Specify that the ``--debug`` and ``--trace`` options affect the logs and not the output and where to find the logs. Miscellanea """"""""""" * Add official support to Python 3.7, deprecate support for 3.4, 3.5 and 3.6. * setup.py: make it compatible with Debian buster. * Add support for Debian Buster, using its versions as minimum required version for dependencies except tqdm. * For tqdm restrict the possible versions to a specific range, that is the only one that works fine with multiple progress bars and colors. * Remove support for Debian Stretch * transports.clustershell: extract progress bars from clustershell event handling. * tests: fix any newly reported issue by the various linters and static checkers. * tests: refactor some tests taking advantage of pytest functionalities. * tests: refactor tox configuration. * Updated documentation according to external dependency changes. * flake8: enforce import order and adopt ``W504``. * Add ``flake8-import-order`` to enforce the import order using the ``edited`` style that corresponds to our styleguide, see: `Python imports`_. * Fix all out of order imports. * For line breaks around binary operators, adopt ``W504`` (breaking before the operator) and ignore ``W503``, following PEP8 suggestion, see: `PEP8 binary operator`_. * Fix all line breaks around binary operators to follow ``W504``. * test: improve integration tests * Don't hide the output of the setup commands, it's useful to both see that the output is visually correct and allow to debug any error in setting up the integration tests. * Allow to pass arguments to the integrations tests so that the deletion of the test instances and temporarily generated files can be accessed for debugging. * doc: fix and improve documentation. * Adapt Sphinx settings according to the newer version used. * Fix links to the documentation of external libraries. * Add and include the diagram image for the available transitions for the ``cumin.transports.State`` class. * Improve docstrings for a better generated documentation result. * Remove unnecessary Sphinx helper functions, now correctly handled by Sphinx natively. * doc: split HTML and manpage generation. * Add a ``man`` tox environment to build only the manpage. * Add a dedicated ``man-min`` environment to build the manpage with the minimum version of Sphinx, that is the one of Debian Buster and that will be used to generate the manpage when building the Debian package. * Let the sphinx tox environment just build the HTML documentation. `v3.0.2`_ (2018-07-30) ^^^^^^^^^^^^^^^^^^^^^^ Bug Fixes """"""""" * Fix the ``-o/--output`` option (bytes->str conversion) (`T200622`_): * The migration to Python3 left the ``-o/--output`` option of the CLI with some decoding issue from bytes to string. * Uniforming all calls to ``bytes.decode()`` not specifying the encoding as ``utf-8`` is the default in Python 3. * Add integration tests for the ``-o/--output`` option. * CLI: fix ``setup_logging()`` when called without path (`T188627`_): * Fix the ``setup_logging()`` function when it's called with a filename without a path, in order to log directly into the current directory. Thanks goes to aggro for reporting it. * Fix debugging log message conversion. The Command.timeout can also be None in case is not set, converting it to string instead of integer for the debug messages. Miscellanea """"""""""" * Updated PyPI URLs to the new PyPI website `v3.0.1`_ (2018-02-19) ^^^^^^^^^^^^^^^^^^^^^^ Bug Fixes """"""""" * CLI: fix help message `v3.0.0`_ (2018-02-19) ^^^^^^^^^^^^^^^^^^^^^^ API breaking changes """""""""""""""""""" * Migration to Python 3, dropping support of Python 2. Besides the usual Py2 -> Py3 conversions, the main changes are: * Add ``nodeset()`` and ``nodeset_fromlist()`` functions in the ``cumin`` module to instantiate ClusterShell's NodeSet objects with the resolver set to ``RESOLVER_NOGROUP``, due to `ClusterShell issue #368`_. * Bump dependency on ClusterShell library to 1.8. * Adapt callbacks in ClusterShell backend to the new ClusterShell's API signatures of version 1.8. * Use ``threading.Lock()`` calls as context managers for the ``with`` statement. * Use Colorama autoreset feature, simplifying its related calls. New features """""""""""" * Backends: add known hosts files backend: * The ``knownhosts`` backend allow to use Cumin taking advantage of existing SSH known hosts files that are not hashed. It allow to write arbitrarily complex queries with subgroups and boolean operators, but each item must be either the hostname itself, or using host expansion with the powerful ClusterShell's ``NodeSet`` syntax. * See the example configuration on how to configure this backend with the list of known hosts files to be parsed. * The typical use case for the ``knownhosts`` backend is when the known hosts file(s) are generated and kept updated by some external configuration manager or tool that is not yet supported as a backend for Cumin. It can also work as a fallback backend in case the primary backend is unavailable but the known hosts file(s) are still up to date. * Batch size: allow to specify it in percentage (`T187185`_): * Transports: allow to specify a ``batch_size_ratio`` as a float number in the Target constructor to set the ``batch_size`` as a percentage of the hosts list. * CLI: make the ``--batch-size`` option to accept both integers and percentage (i.e. ``50%``) values. `v2.0.0`_ (2018-01-19) ^^^^^^^^^^^^^^^^^^^^^^ API breaking changes """""""""""""""""""" * Logging: uniform loggers (`T179002`_): * Remove optional parameter logger from all classes where it was accepted, the classes instantiate the proper logger based on the current module and class name. * ClusterShell backend: fix ``execute()`` return code: * The return code of the ``execute()`` method was not respecting the parent class contract for its return code when there are no commands set or no hosts to target. * Make the ``Target`` class raise a ``WorkerError`` exception on instantiation if there are no target hosts. * Make the ``execute()`` method raise a ``WorkerError`` exception if there are no commands to execute. New features """""""""""" * Backends: add support to external backends plugins (`T178342`_): * Custom external backends can be developed outside of Cumin and used by Cumin as any other backend. * The external backends must: * Be present in Python ``PATH``. * Define a ``GRAMMAR_PREFIX`` attribute that doesn't conflict with built-in backends prefixes. * Define a ``query_class`` attribute pointing to a class that inherit from ``cumin.backends.BaseQuery``. * The CLI is not anymore able to enforce that the ``--backend`` parameter is valid when parsing the command line arguments, but will fail later on with a clear message. * PuppetDB backend: add support for PuppetDB API v4 (`T182575`_): * Allow to set the API version via configuration. * Default to API v4 as v3 is obsolete. * Use POST for API v4 to overcome GET limits on large queries, fixes `T166397`_. * Bumped minimum version for ``requests-mock`` to ``1.3.0``. Minor improvements """""""""""""""""" * Logging: uniform loggers (`T179002`_): * Use proper hierarchical loggers across the project. * For classes inherited from a base abstract class, the logger is defined only in the base abstract class, with the name of the concrete class that is calling it. * Changed CLI logging format to take advantage of the hirarchical logging. * Logging: use ``%`` syntax for parameters (`T179002`_): * For optimization purposes and to adhere to Python best practices, use ``%s`` syntax in logging messages and pass the replacement parameters to the logging function. Some messages are still pre-formatted before the call to the logging function because used also for other purposes. * pylint: re-enable the check for logging-format-interpolation. `v1.3.0`_ (2017-11-03) ^^^^^^^^^^^^^^^^^^^^^^ New features """""""""""" * PuppetDB backend: Class, Roles and Profiles shortcuts (`T178279`_): * It is becoming common practice to use the role/profile paradigm in Puppet, where each host has only one role named ``Role::Module::Name`` that includes multiple profiles of the type ``Profile::Module::Name``. If this practice is used, queries for those resources in Cumin will be very common and not user-friendly, requiring to write queries of the type ``R:Class = Role::Module::Name``. Add support to Roles and Profiles so that they can be queried via shortcuts with ``O:Module::Name`` for roles and ``P:Module::Name`` for profiles. * Add also a generic class shortcut to quickly query a class resource with ``C:class_name`` or ``C:path::to::class``. * The special syntax for fields ``@field`` and parameters ``%param`` are also supported. When querying for any of the above shortcuts, like ``P:Module::Name%param = value``. The generated query will include two subqueries in ``AND`` between them, one for the class title and the other for the class parameter. Minor improvements """""""""""""""""" * Refactor documentation: * Moved most of the content from the README to the classes, function and documentation pages where it really belongs. * Add documentation files for an introduction to cumin, how to install it, how to develop it and with the release notes. * Add animated GIF to the README and documentation introduction. Bug Fixes """"""""" * Documentation: amend CHANGELOG and TODO for the addition of the manpage in `v1.2.2`_ (`T159308`_). * Documentation: add ReadTheDocs specific configuration. * Documentation: fix ReadTheDocs CSS override `v1.2.2`_ (2017-10-11) ^^^^^^^^^^^^^^^^^^^^^^ Minor improvements """""""""""""""""" * Dependencies: split the OpenStack dependencies into a separate ``extras_require`` in ``setup.py``. This allows to install Cumin without all the dependencies needed for the OpenStack backend, if that is not needed. * Docstrings: use Google Style Python Docstrings to allow to automatically generate documentation with Sphinx. * Documentation: converted ``README``, ``CHANGELOG`` and ``TODO`` from Markdown to reStructuredText. PyPI renders only reStructuredText while GitHub renders both. Moving to reStructuredText to be PyPI friendly and allow to write more powerful documentation. * CLI: extract the ``ArgumentParser`` definition from ``parse_args()`` into a ``get_parser()`` function for easier testability and documentation generation. Uniform help messages in ``ArgumentParser`` options. * setup.py: prepare for PyPi submission. Include the full ``README.rst`` as long description. * Documentation: setup Sphinx to generate the documentation and to auto-document the API and CLI. * Testing: refactored ``tox.ini`` to reduce the number of virtualenv while expanding the available environments for static analysis and tests performed, including running unit tests with the minimum supported versions of all the dependencies. * CLI: add manpage (`T159308`_) `v1.2.1`_ (2017-09-27) ^^^^^^^^^^^^^^^^^^^^^^ New features """""""""""" * OpenStack backend: allow to set default query params in the configuration (`T176314`_): Allow to set arbitrary default query params in the configuration for the OpenStack backend. This is useful for example if Cumin is installed inside an OpenStack project to automatically search only within the instances of the current project. See the example in the provided ``doc/examples/config.yaml`` file. Bug Fixes """"""""" * Configuration: do not raise on empty configuration or aliases. Moved the check of required parameters where needed, in order to raise explicit exceptions with a more meaningful message for the user. * Exceptions: convert remaining spurious exceptions to CuminError or improve their error message. `v1.1.1`_ (2017-09-26) ^^^^^^^^^^^^^^^^^^^^^^ Bug Fixes """"""""" * OpenStack: limit grammar to not overlap with the global one. `v1.1.0`_ (2017-09-21) ^^^^^^^^^^^^^^^^^^^^^^ New features """""""""""" * Backends: add OpenStack backend (`T175711`_). Bug Fixes """"""""" * CLI: fix --version option. * Installation: fix ``data_files`` installation directory (`T174008`_) * Transports: better handling of empty list (`T174911`_): * BaseWorker: accept an empty list in the command setter. It's its default value, there is no point in forbidding a client to set it to the same value. * ClusterShellWorker: return immediately if there are no target hosts. * Clustershell: make call to tqdm.write() explicit where to send the output, not relying on its default. `v1.0.0`_ (2017-08-23) ^^^^^^^^^^^^^^^^^^^^^^ CLI breaking changes """""""""""""""""""" * CLI: migrate to timeout per command (`T164838`_): * the global timeout command line options changes from ``-t/--timeout`` to ``--global-timeout``. * the ``-t/--timeout`` option is now used to set the timeout for each command in each host independently. Configuration breaking changes """""""""""""""""""""""""""""" * Query: add multi-query support (`T170394`_): * Remove the ``backend`` configuration key as it is not anymore used. * Add a new optional ``default_backend`` configuration key. If set the query will be first executed with the default backend, and if failing the parsing it will be executed with the global multi-query grammar. This allow to keep backward compatibility with the query that were executed with previous versions of Cumin. API breaking changes """""""""""""""""""" * PuppetDB backend: consistently use ``InvalidQueryError`` (`T162151`_). * Transports: refactor command handling to support new features (`T164838`_), (`T164833`_) and (`T171679`_): * Transports: move ``BaseWorker`` helper methods to module functions. * Transports: add ``Command`` class. * Transports: use the new ``Command`` class in ``BaseWorker``, moving from a list of strings to a list of ``Command`` objects. * Transports: maintain backward compatibility and easy of usage automatically converting a list of strings to a list of ``Command`` objects when setting the commands property. * Allow to set the ``ok_codes`` property of the ``transports.Command`` class to an empty list to consider any return code as successful. The case in which no return code should be treated successful has no practical use. * ClusterShell: adapt the calls to commands for the new ``Command`` objects. * Configuration: move configuration loader from the ``cli`` module to the main ``cumin`` module (`T169640`_): * add a ``cumin.Config`` class. * move the ``parse_config`` helper to cumin's main module from the ``cli`` one, to allow to easily load the configuration also when it's used as a Python library. * ``QueryBuilder``: move query string to ``build()`` method. The constructor of the ``QueryBuilder`` was changed to not accept anymore a query string directly, but just the configuration and the optional logger. The query string is now a required parameter of the ``build()`` method. This properly split configuration and parameters, allowing to easily ``build()`` multiple queries with the same ``QueryBuilder`` instance. * Transports: convert hosts to ClusterShell's ``NodeSet`` (`T170394`_): * in preparation for the multi-query support, start moving the transports to accept a ClusterShell's ``NodeSet`` instead of a list of nodes. With the new multi-query support the backends too will return only NodeSets. * Query: add multi-query support (`T170394`_): * Aliases are now global and must use the global grammar syntax. * ``Query`` class: the public ``build()`` method has become private and now is sufficient to call the ``execute(query_string)`` method. Example usage:: config = cumin.Config(args.config) hosts = query.Query(config, logger=logger).execute(query_string) * ``Query`` class: the public methods ``open_subgroup()`` and ``close_subgroup()`` have become private, ``_open_subgroup()`` and ``_close_subgroup()`` respectively. * Transports: improve target management (`T171684`_): * Add a ``Target`` class to handle all the target-related configuration. * Let the ``BaseWorker`` require an instance of the ``Target`` class and delegate to it for all the target-related configuration. * This changes the ``BaseWorker`` constructor signature and removes the ``hosts``, ``batch_size`` and ``batch_sleep`` setters/getters. New features """""""""""" * CLI: automatically set dry-run mode when no commands are specified (`T161887`_). * ClusterShell transport: output directly when only a single host is targeted. When the commands are executed against only one host, print the output directly as it comes, to give the user an immediate feedback. There is no advantage to collect the output for de-duplication in this case (`T164827`_). * Transports: allow to specify a timeout per ``Command`` (`T164838`_). * Transports: allow to specify exit codes per ``Command`` (`T164833`_). Allow to specify for each ``Command`` object a list of exit codes to be considered successful when executing its specific command. * ClusterShell backend: allow to specify exit codes per ``Command`` (`T164833`_). * ClusterShell backend: allow to set a timeout per ``Command`` (`T164838`_). * CLI: add ``-i/--interactive`` option (`T165838`_). When set, this option drops into a Python shell (REPL) after the execution, allowing the user to manipulate the results with the full power of Python. In this first iteration it can be used only when one command is specified. * CLI: add ``-o/--output`` to get the output in different formats (`T165842`_). Allow to have ``txt`` and ``json`` output when only one command is specified. In this first iteration the formatted output will be printed after the standard output with a separator, in a next iteration the standard output will be suppressed. * Query and grammar: add support for aliases (`T169640`_): * Allow aliases of the form ``A:alias_name`` into the grammar. * Automatically replace recursively all the aliases directly in the ``QueryBuilder``, to make it completely transparent for the backends. * Configuration: automatically load aliases from file (`T169640`_). When loading the configuration, automatically load also any aliases present in the ``aliases.yaml`` file in the same directory of the configuration file, if present. * Query: add multi-query support (`T170394`_): * Each backend has now its own grammar and parsing rules as they are completely independent from each other. * Add a new global grammar that allows to execute blocks of queries with different backends and aggregate the results. * CLI: add an option to ignore exit codes of commands (`T171679`_). Add the ``-x/--ignore-exit-codes`` option to consider any executed command as successful, ignoring the returned exit codes. This can be useful for a cleaner output and the usage of batches when running troubleshooting commands for which the return code might be ignored (i.e. grep). Minor improvements """""""""""""""""" * CLI: improve configuration error handling (`T158747`_). * Fix Pylint and other validation tools reported errors (`T154588`_). * Package metadata and testing tools improvements (`T154588`_): * Fill ``setup.py`` with all the parameters, suitable for a future submission to PyPI. * Autodetect the version from Git tags and expose it in the module using ``setuptools_scm``. * CLI: add a ``--version`` option to print the current version and exit. * Tests: use ``pytest`` to run the tests. * Tests: convert tests from ``unittest`` to ``pytest``. * Tests: make ``tox`` use the dependencies in ``setup.py``, removing the now unnecessary requirements files. * Tests: add security analyzer ``Bandit`` to ``tox``. * Tests: add ``Prospector`` to ``tox``, that in turns runs multiple additional tools: ``dodgy``, ``mccabe``, ``pep257``, ``pep8``, ``profile-validator``, ``pyflakes``, ``pylint``, ``pyroma``, ``vulture``. * Tests: simplify and improve parametrized tests. Take advantage of ``pytest.mark.parametrize`` to run the same test multiple times with different parameters instead of looping inside the same test. This not only simplifies the code but also will make each parametrized test fail independently allowing an easier debugging. * CLI: simplify imports and introspection. * Logging: add a custom ``trace()`` logging level: * Add an additional custom logging level after ``DEBUG`` called ``TRACE`` mainly for development debugging. * Fail in case the same log level is already set with a different name. This could happen when used as a library. * CLI: add the ``--trace`` option to enable said logging level. * Tests: improved tests fixture usage and removed usage of the example configuration present in the documentation from the tests. * Transports: improve command list validation of the ``transports.Command`` class to not allow an empty list for the commands property (`T171679`_). Bug Fixes """"""""" * PuppetDB backend: do not auto upper case the first character when the query is a regex (`T161730`_). * PuppetDB backend: forbid resource's parameters regex as PuppetDB API v3 do not support regex match for resource's parameters (`T162151`_). * ClusterShell transport: fix set of list options (`T164824`_). * Transports: fix ``success_threshold`` getter when set to ``0`` (`T167392`_). * Transports: fix ``ok_codes`` getter for empty list (`T167394`_). * ``QueryBuilder``: fix subgroup close at the end of query. When a query was having subgroups that were closed at the end of the query, QueryBuilder was not calling the ``close_subgroup()`` method of the related backend as it should have. For example in a query like ``host1* and (R:Class = Foo or R:Class = Bar)``. * Fix test dependency issue. Due to a braking API change in the latest version of ``Vulture``, ``Prospector`` is not working anymore with the installed version of ``Vulture`` due to missing constraint in their ``setup.py``. See `Prospector issue #230`_ for more details. `v0.0.2`_ (2017-03-15) ^^^^^^^^^^^^^^^^^^^^^^ Configuration breaking changes """""""""""""""""""""""""""""" * Add support for batch processing (`T159968`_): * Moved the ``environment`` block in the configuration file to the top level from within a specific transport. API breaking changes """""""""""""""""""" * Add support for batch processing (`T159968`_): * Refactored the ``BaseWorker`` class (and the ``ClusterShellWorker`` accordingly) to avoid passing a lot of parameters to the execute() method, moving them to setters and getters with validation and default values, respectively. * Add state machine for a transport's node state. * Add CuminError exception and make all custom exceptions inherit from it to allow to easily catch only Cumin's exceptions. * ClusterShell transport: always require an event handler (`T159968`_): * Since the addition of the batch capability running without an event handler doesn't really work because only the first batch will be scheduled. * Updated the CLI to work transparently and set the mode to ``sync`` when there is only one command. * Unify the reporting lines format and logic between ``sync`` and ``async`` modes for coherence. New features """""""""""" * Add support for ``not`` in simple hosts selection queries (`T158748`_). * Add support for batch processing (`T159968`_): * It's now possible to specify a ``batch_size`` and a ``batch_sleep`` parameters to define the size of a sliding batch and an optional sleep between hosts executions. * ClusterShell transport: the batches behaves accordingly to the specified mode when multiple commands are specified: * ``sync``: the first command is executed in a sliding batch until executed on all hosts or aborted due unmet success ratio. Then the execution of the second command will start if the success ratio is reached. * ``async``: all the commands are executed in series in the first batch, and then will proceed with the next hosts with a sliding batch, if the success ratio is met. * Improves logging for backends and transport. * CLI: updated to use the batch functionality, use the transport return value as return code on exit. * Improves test coverage. * PuppetDB backend: automatically upper case the first character in resource names (`T159970`_). Minor improvements """""""""""""""""" * Moved ``config.yaml`` to a ``doc/examples/`` directory. It simplify the ship of the example file when packaging. * Allow to ignore selected ``urllib3`` warnings (`T158758`_). * Add codecov and codacy config and badges. * Fixing minor issues reported by codacy (`T158967`_). * Add integration tests for ClusterShell transport using Docker (`T159969`_). Bug Fixes """"""""" * Match the whole string for hosts regex matching (`T158746`_). `v0.0.1`_ (2017-02-17) ^^^^^^^^^^^^^^^^^^^^^^ * First released version (`T154588`_). .. _`Prospector issue #230`: https://github.com/landscapeio/prospector/issues/230 .. _`ClusterShell issue #368`: https://github.com/cea-hpc/clustershell/issues/368 .. _`tqdm issue #777`: https://github.com/tqdm/tqdm/issues/777 .. _`regexp-match`: https://puppet.com/docs/puppetdb/4.4/api/query/v4/ast.html#regexp-match .. _`Python imports`: https://www.mediawiki.org/wiki/Manual:Coding_conventions/Python#Imports .. _`PEP8 binary operator`: https://www.python.org/dev/peps/pep-0008/#should-a-line-break-before-or-after-a-binary-operator .. _`T154588`: https://phabricator.wikimedia.org/T154588 .. _`T158746`: https://phabricator.wikimedia.org/T158746 .. _`T158747`: https://phabricator.wikimedia.org/T158747 .. _`T158748`: https://phabricator.wikimedia.org/T158748 .. _`T158758`: https://phabricator.wikimedia.org/T158758 .. _`T158967`: https://phabricator.wikimedia.org/T158967 .. _`T159308`: https://phabricator.wikimedia.org/T159308 .. _`T159968`: https://phabricator.wikimedia.org/T159968 .. _`T159969`: https://phabricator.wikimedia.org/T159969 .. _`T159970`: https://phabricator.wikimedia.org/T159970 .. _`T161730`: https://phabricator.wikimedia.org/T161730 .. _`T161887`: https://phabricator.wikimedia.org/T161887 .. _`T162151`: https://phabricator.wikimedia.org/T162151 .. _`T164824`: https://phabricator.wikimedia.org/T164824 .. _`T164827`: https://phabricator.wikimedia.org/T164827 .. _`T164833`: https://phabricator.wikimedia.org/T164833 .. _`T164838`: https://phabricator.wikimedia.org/T164838 .. _`T165838`: https://phabricator.wikimedia.org/T165838 .. _`T165842`: https://phabricator.wikimedia.org/T165842 .. _`T166397`: https://phabricator.wikimedia.org/T166397 .. _`T167392`: https://phabricator.wikimedia.org/T167392 .. _`T167394`: https://phabricator.wikimedia.org/T167394 .. _`T169640`: https://phabricator.wikimedia.org/T169640 .. _`T170394`: https://phabricator.wikimedia.org/T170394 .. _`T171679`: https://phabricator.wikimedia.org/T171679 .. _`T171684`: https://phabricator.wikimedia.org/T171684 .. _`T174008`: https://phabricator.wikimedia.org/T174008 .. _`T174911`: https://phabricator.wikimedia.org/T174911 .. _`T175711`: https://phabricator.wikimedia.org/T175711 .. _`T176314`: https://phabricator.wikimedia.org/T176314 .. _`T178279`: https://phabricator.wikimedia.org/T178279 .. _`T178342`: https://phabricator.wikimedia.org/T178342 .. _`T179002`: https://phabricator.wikimedia.org/T179002 .. _`T182575`: https://phabricator.wikimedia.org/T182575 .. _`T187185`: https://phabricator.wikimedia.org/T187185 .. _`T188627`: https://phabricator.wikimedia.org/T188627 .. _`T200622`: https://phabricator.wikimedia.org/T200622 .. _`T201881`: https://phabricator.wikimedia.org/T201881 .. _`T204680`: https://phabricator.wikimedia.org/T204680 .. _`T207037`: https://phabricator.wikimedia.org/T207037 .. _`T212783`: https://phabricator.wikimedia.org/T212783 .. _`T217038`: https://phabricator.wikimedia.org/T217038 .. _`T218440`: https://phabricator.wikimedia.org/T218440 .. _`T218441`: https://phabricator.wikimedia.org/T218441 .. _`T244840`: https://phabricator.wikimedia.org/T244840 .. _`T270795`: https://phabricator.wikimedia.org/T270795 .. _`T384539`: https://phabricator.wikimedia.org/T384539 .. _`v0.0.1`: https://github.com/wikimedia/cumin/releases/tag/v0.0.1 .. _`v0.0.2`: https://github.com/wikimedia/cumin/releases/tag/v0.0.2 .. _`v1.0.0`: https://github.com/wikimedia/cumin/releases/tag/v1.0.0 .. _`v1.1.0`: https://github.com/wikimedia/cumin/releases/tag/v1.1.0 .. _`v1.1.1`: https://github.com/wikimedia/cumin/releases/tag/v1.1.1 .. _`v1.2.1`: https://github.com/wikimedia/cumin/releases/tag/v1.2.1 .. _`v1.2.2`: https://github.com/wikimedia/cumin/releases/tag/v1.2.2 .. _`v1.3.0`: https://github.com/wikimedia/cumin/releases/tag/v1.3.0 .. _`v2.0.0`: https://github.com/wikimedia/cumin/releases/tag/v2.0.0 .. _`v3.0.0`: https://github.com/wikimedia/cumin/releases/tag/v3.0.0 .. _`v3.0.1`: https://github.com/wikimedia/cumin/releases/tag/v3.0.1 .. _`v3.0.2`: https://github.com/wikimedia/cumin/releases/tag/v3.0.2 .. _`v4.0.0rc1`: https://github.com/wikimedia/cumin/releases/tag/v4.0.0rc1 .. _`v4.0.0`: https://github.com/wikimedia/cumin/releases/tag/v4.0.0 .. _`v4.1.0`: https://github.com/wikimedia/cumin/releases/tag/v4.1.0 .. _`v4.1.1`: https://github.com/wikimedia/cumin/releases/tag/v4.1.1 .. _`v4.2.0`: https://github.com/wikimedia/cumin/releases/tag/v4.2.0 .. _`v5.0.0`: https://github.com/wikimedia/cumin/releases/tag/v5.0.0 .. _`v5.1.0`: https://github.com/wikimedia/cumin/releases/tag/v5.1.0 .. _`v5.1.1`: https://github.com/wikimedia/cumin/releases/tag/v5.1.1 wikimedia-cumin-36f957f/COPYRIGHT000066400000000000000000000004311476500461000164350ustar00rootroot00000000000000For all the files in this repository: Copyright (c) 2017-2018 Riccardo Coccioli Copyright (c) 2017-2018 Wikimedia Foundation, Inc. This program is Free Software, you can find details about the license in the LICENSE file in the root of this repository. wikimedia-cumin-36f957f/LICENSE000066400000000000000000001035251476500461000161570ustar00rootroot00000000000000GNU GENERAL PUBLIC LICENSE Version 3, 29 June 2007 Copyright (C) 2007 Free Software Foundation, Inc. Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. Preamble The GNU General Public License is a free, copyleft license for software and other kinds of works. The licenses for most software and other practical works are designed to take away your freedom to share and change the works. By contrast, the GNU General Public License is intended to guarantee your freedom to share and change all versions of a program--to make sure it remains free software for all its users. We, the Free Software Foundation, use the GNU General Public License for most of our software; it applies also to any other work released this way by its authors. You can apply it to your programs, too. When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for them if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs, and that you know you can do these things. To protect your rights, we need to prevent others from denying you these rights or asking you to surrender the rights. Therefore, you have certain responsibilities if you distribute copies of the software, or if you modify it: responsibilities to respect the freedom of others. For example, if you distribute copies of such a program, whether gratis or for a fee, you must pass on to the recipients the same freedoms that you received. You must make sure that they, too, receive or can get the source code. And you must show them these terms so they know their rights. Developers that use the GNU GPL protect your rights with two steps: (1) assert copyright on the software, and (2) offer you this License giving you legal permission to copy, distribute and/or modify it. For the developers' and authors' protection, the GPL clearly explains that there is no warranty for this free software. For both users' and authors' sake, the GPL requires that modified versions be marked as changed, so that their problems will not be attributed erroneously to authors of previous versions. Some devices are designed to deny users access to install or run modified versions of the software inside them, although the manufacturer can do so. This is fundamentally incompatible with the aim of protecting users' freedom to change the software. The systematic pattern of such abuse occurs in the area of products for individuals to use, which is precisely where it is most unacceptable. Therefore, we have designed this version of the GPL to prohibit the practice for those products. If such problems arise substantially in other domains, we stand ready to extend this provision to those domains in future versions of the GPL, as needed to protect the freedom of users. Finally, every program is threatened constantly by software patents. States should not allow patents to restrict development and use of software on general-purpose computers, but in those that do, we wish to avoid the special danger that patents applied to a free program could make it effectively proprietary. To prevent this, the GPL assures that patents cannot be used to render the program non-free. The precise terms and conditions for copying, distribution and modification follow. TERMS AND CONDITIONS 0. Definitions. "This License" refers to version 3 of the GNU General Public License. "Copyright" also means copyright-like laws that apply to other kinds of works, such as semiconductor masks. "The Program" refers to any copyrightable work licensed under this License. Each licensee is addressed as "you". "Licensees" and "recipients" may be individuals or organizations. To "modify" a work means to copy from or adapt all or part of the work in a fashion requiring copyright permission, other than the making of an exact copy. The resulting work is called a "modified version" of the earlier work or a work "based on" the earlier work. A "covered work" means either the unmodified Program or a work based on the Program. To "propagate" a work means to do anything with it that, without permission, would make you directly or secondarily liable for infringement under applicable copyright law, except executing it on a computer or modifying a private copy. Propagation includes copying, distribution (with or without modification), making available to the public, and in some countries other activities as well. To "convey" a work means any kind of propagation that enables other parties to make or receive copies. Mere interaction with a user through a computer network, with no transfer of a copy, is not conveying. An interactive user interface displays "Appropriate Legal Notices" to the extent that it includes a convenient and prominently visible feature that (1) displays an appropriate copyright notice, and (2) tells the user that there is no warranty for the work (except to the extent that warranties are provided), that licensees may convey the work under this License, and how to view a copy of this License. If the interface presents a list of user commands or options, such as a menu, a prominent item in the list meets this criterion. 1. Source Code. The "source code" for a work means the preferred form of the work for making modifications to it. "Object code" means any non-source form of a work. A "Standard Interface" means an interface that either is an official standard defined by a recognized standards body, or, in the case of interfaces specified for a particular programming language, one that is widely used among developers working in that language. The "System Libraries" of an executable work include anything, other than the work as a whole, that (a) is included in the normal form of packaging a Major Component, but which is not part of that Major Component, and (b) serves only to enable use of the work with that Major Component, or to implement a Standard Interface for which an implementation is available to the public in source code form. A "Major Component", in this context, means a major essential component (kernel, window system, and so on) of the specific operating system (if any) on which the executable work runs, or a compiler used to produce the work, or an object code interpreter used to run it. The "Corresponding Source" for a work in object code form means all the source code needed to generate, install, and (for an executable work) run the object code and to modify the work, including scripts to control those activities. However, it does not include the work's System Libraries, or general-purpose tools or generally available free programs which are used unmodified in performing those activities but which are not part of the work. For example, Corresponding Source includes interface definition files associated with source files for the work, and the source code for shared libraries and dynamically linked subprograms that the work is specifically designed to require, such as by intimate data communication or control flow between those subprograms and other parts of the work. The Corresponding Source need not include anything that users can regenerate automatically from other parts of the Corresponding Source. The Corresponding Source for a work in source code form is that same work. 2. Basic Permissions. All rights granted under this License are granted for the term of copyright on the Program, and are irrevocable provided the stated conditions are met. This License explicitly affirms your unlimited permission to run the unmodified Program. The output from running a covered work is covered by this License only if the output, given its content, constitutes a covered work. This License acknowledges your rights of fair use or other equivalent, as provided by copyright law. You may make, run and propagate covered works that you do not convey, without conditions so long as your license otherwise remains in force. You may convey covered works to others for the sole purpose of having them make modifications exclusively for you, or provide you with facilities for running those works, provided that you comply with the terms of this License in conveying all material for which you do not control copyright. Those thus making or running the covered works for you must do so exclusively on your behalf, under your direction and control, on terms that prohibit them from making any copies of your copyrighted material outside their relationship with you. Conveying under any other circumstances is permitted solely under the conditions stated below. Sublicensing is not allowed; section 10 makes it unnecessary. 3. Protecting Users' Legal Rights From Anti-Circumvention Law. No covered work shall be deemed part of an effective technological measure under any applicable law fulfilling obligations under article 11 of the WIPO copyright treaty adopted on 20 December 1996, or similar laws prohibiting or restricting circumvention of such measures. When you convey a covered work, you waive any legal power to forbid circumvention of technological measures to the extent such circumvention is effected by exercising rights under this License with respect to the covered work, and you disclaim any intention to limit operation or modification of the work as a means of enforcing, against the work's users, your or third parties' legal rights to forbid circumvention of technological measures. 4. Conveying Verbatim Copies. You may convey verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice; keep intact all notices stating that this License and any non-permissive terms added in accord with section 7 apply to the code; keep intact all notices of the absence of any warranty; and give all recipients a copy of this License along with the Program. You may charge any price or no price for each copy that you convey, and you may offer support or warranty protection for a fee. 5. Conveying Modified Source Versions. You may convey a work based on the Program, or the modifications to produce it from the Program, in the form of source code under the terms of section 4, provided that you also meet all of these conditions: a) The work must carry prominent notices stating that you modified it, and giving a relevant date. b) The work must carry prominent notices stating that it is released under this License and any conditions added under section 7. This requirement modifies the requirement in section 4 to "keep intact all notices". c) You must license the entire work, as a whole, under this License to anyone who comes into possession of a copy. This License will therefore apply, along with any applicable section 7 additional terms, to the whole of the work, and all its parts, regardless of how they are packaged. This License gives no permission to license the work in any other way, but it does not invalidate such permission if you have separately received it. d) If the work has interactive user interfaces, each must display Appropriate Legal Notices; however, if the Program has interactive interfaces that do not display Appropriate Legal Notices, your work need not make them do so. A compilation of a covered work with other separate and independent works, which are not by their nature extensions of the covered work, and which are not combined with it such as to form a larger program, in or on a volume of a storage or distribution medium, is called an "aggregate" if the compilation and its resulting copyright are not used to limit the access or legal rights of the compilation's users beyond what the individual works permit. Inclusion of a covered work in an aggregate does not cause this License to apply to the other parts of the aggregate. 6. Conveying Non-Source Forms. You may convey a covered work in object code form under the terms of sections 4 and 5, provided that you also convey the machine-readable Corresponding Source under the terms of this License, in one of these ways: a) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by the Corresponding Source fixed on a durable physical medium customarily used for software interchange. b) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by a written offer, valid for at least three years and valid for as long as you offer spare parts or customer support for that product model, to give anyone who possesses the object code either (1) a copy of the Corresponding Source for all the software in the product that is covered by this License, on a durable physical medium customarily used for software interchange, for a price no more than your reasonable cost of physically performing this conveying of source, or (2) access to copy the Corresponding Source from a network server at no charge. c) Convey individual copies of the object code with a copy of the written offer to provide the Corresponding Source. This alternative is allowed only occasionally and noncommercially, and only if you received the object code with such an offer, in accord with subsection 6b. d) Convey the object code by offering access from a designated place (gratis or for a charge), and offer equivalent access to the Corresponding Source in the same way through the same place at no further charge. You need not require recipients to copy the Corresponding Source along with the object code. If the place to copy the object code is a network server, the Corresponding Source may be on a different server (operated by you or a third party) that supports equivalent copying facilities, provided you maintain clear directions next to the object code saying where to find the Corresponding Source. Regardless of what server hosts the Corresponding Source, you remain obligated to ensure that it is available for as long as needed to satisfy these requirements. e) Convey the object code using peer-to-peer transmission, provided you inform other peers where the object code and Corresponding Source of the work are being offered to the general public at no charge under subsection 6d. A separable portion of the object code, whose source code is excluded from the Corresponding Source as a System Library, need not be included in conveying the object code work. A "User Product" is either (1) a "consumer product", which means any tangible personal property which is normally used for personal, family, or household purposes, or (2) anything designed or sold for incorporation into a dwelling. In determining whether a product is a consumer product, doubtful cases shall be resolved in favor of coverage. For a particular product received by a particular user, "normally used" refers to a typical or common use of that class of product, regardless of the status of the particular user or of the way in which the particular user actually uses, or expects or is expected to use, the product. A product is a consumer product regardless of whether the product has substantial commercial, industrial or non-consumer uses, unless such uses represent the only significant mode of use of the product. "Installation Information" for a User Product means any methods, procedures, authorization keys, or other information required to install and execute modified versions of a covered work in that User Product from a modified version of its Corresponding Source. The information must suffice to ensure that the continued functioning of the modified object code is in no case prevented or interfered with solely because modification has been made. If you convey an object code work under this section in, or with, or specifically for use in, a User Product, and the conveying occurs as part of a transaction in which the right of possession and use of the User Product is transferred to the recipient in perpetuity or for a fixed term (regardless of how the transaction is characterized), the Corresponding Source conveyed under this section must be accompanied by the Installation Information. But this requirement does not apply if neither you nor any third party retains the ability to install modified object code on the User Product (for example, the work has been installed in ROM). The requirement to provide Installation Information does not include a requirement to continue to provide support service, warranty, or updates for a work that has been modified or installed by the recipient, or for the User Product in which it has been modified or installed. Access to a network may be denied when the modification itself materially and adversely affects the operation of the network or violates the rules and protocols for communication across the network. Corresponding Source conveyed, and Installation Information provided, in accord with this section must be in a format that is publicly documented (and with an implementation available to the public in source code form), and must require no special password or key for unpacking, reading or copying. 7. Additional Terms. "Additional permissions" are terms that supplement the terms of this License by making exceptions from one or more of its conditions. Additional permissions that are applicable to the entire Program shall be treated as though they were included in this License, to the extent that they are valid under applicable law. If additional permissions apply only to part of the Program, that part may be used separately under those permissions, but the entire Program remains governed by this License without regard to the additional permissions. When you convey a copy of a covered work, you may at your option remove any additional permissions from that copy, or from any part of it. (Additional permissions may be written to require their own removal in certain cases when you modify the work.) You may place additional permissions on material, added by you to a covered work, for which you have or can give appropriate copyright permission. Notwithstanding any other provision of this License, for material you add to a covered work, you may (if authorized by the copyright holders of that material) supplement the terms of this License with terms: a) Disclaiming warranty or limiting liability differently from the terms of sections 15 and 16 of this License; or b) Requiring preservation of specified reasonable legal notices or author attributions in that material or in the Appropriate Legal Notices displayed by works containing it; or c) Prohibiting misrepresentation of the origin of that material, or requiring that modified versions of such material be marked in reasonable ways as different from the original version; or d) Limiting the use for publicity purposes of names of licensors or authors of the material; or e) Declining to grant rights under trademark law for use of some trade names, trademarks, or service marks; or f) Requiring indemnification of licensors and authors of that material by anyone who conveys the material (or modified versions of it) with contractual assumptions of liability to the recipient, for any liability that these contractual assumptions directly impose on those licensors and authors. All other non-permissive additional terms are considered "further restrictions" within the meaning of section 10. If the Program as you received it, or any part of it, contains a notice stating that it is governed by this License along with a term that is a further restriction, you may remove that term. If a license document contains a further restriction but permits relicensing or conveying under this License, you may add to a covered work material governed by the terms of that license document, provided that the further restriction does not survive such relicensing or conveying. If you add terms to a covered work in accord with this section, you must place, in the relevant source files, a statement of the additional terms that apply to those files, or a notice indicating where to find the applicable terms. Additional terms, permissive or non-permissive, may be stated in the form of a separately written license, or stated as exceptions; the above requirements apply either way. 8. Termination. You may not propagate or modify a covered work except as expressly provided under this License. Any attempt otherwise to propagate or modify it is void, and will automatically terminate your rights under this License (including any patent licenses granted under the third paragraph of section 11). However, if you cease all violation of this License, then your license from a particular copyright holder is reinstated (a) provisionally, unless and until the copyright holder explicitly and finally terminates your license, and (b) permanently, if the copyright holder fails to notify you of the violation by some reasonable means prior to 60 days after the cessation. Moreover, your license from a particular copyright holder is reinstated permanently if the copyright holder notifies you of the violation by some reasonable means, this is the first time you have received notice of violation of this License (for any work) from that copyright holder, and you cure the violation prior to 30 days after your receipt of the notice. Termination of your rights under this section does not terminate the licenses of parties who have received copies or rights from you under this License. If your rights have been terminated and not permanently reinstated, you do not qualify to receive new licenses for the same material under section 10. 9. Acceptance Not Required for Having Copies. You are not required to accept this License in order to receive or run a copy of the Program. Ancillary propagation of a covered work occurring solely as a consequence of using peer-to-peer transmission to receive a copy likewise does not require acceptance. However, nothing other than this License grants you permission to propagate or modify any covered work. These actions infringe copyright if you do not accept this License. Therefore, by modifying or propagating a covered work, you indicate your acceptance of this License to do so. 10. Automatic Licensing of Downstream Recipients. Each time you convey a covered work, the recipient automatically receives a license from the original licensors, to run, modify and propagate that work, subject to this License. You are not responsible for enforcing compliance by third parties with this License. An "entity transaction" is a transaction transferring control of an organization, or substantially all assets of one, or subdividing an organization, or merging organizations. If propagation of a covered work results from an entity transaction, each party to that transaction who receives a copy of the work also receives whatever licenses to the work the party's predecessor in interest had or could give under the previous paragraph, plus a right to possession of the Corresponding Source of the work from the predecessor in interest, if the predecessor has it or can get it with reasonable efforts. You may not impose any further restrictions on the exercise of the rights granted or affirmed under this License. For example, you may not impose a license fee, royalty, or other charge for exercise of rights granted under this License, and you may not initiate litigation (including a cross-claim or counterclaim in a lawsuit) alleging that any patent claim is infringed by making, using, selling, offering for sale, or importing the Program or any portion of it. 11. Patents. A "contributor" is a copyright holder who authorizes use under this License of the Program or a work on which the Program is based. The work thus licensed is called the contributor's "contributor version". A contributor's "essential patent claims" are all patent claims owned or controlled by the contributor, whether already acquired or hereafter acquired, that would be infringed by some manner, permitted by this License, of making, using, or selling its contributor version, but do not include claims that would be infringed only as a consequence of further modification of the contributor version. For purposes of this definition, "control" includes the right to grant patent sublicenses in a manner consistent with the requirements of this License. Each contributor grants you a non-exclusive, worldwide, royalty-free patent license under the contributor's essential patent claims, to make, use, sell, offer for sale, import and otherwise run, modify and propagate the contents of its contributor version. In the following three paragraphs, a "patent license" is any express agreement or commitment, however denominated, not to enforce a patent (such as an express permission to practice a patent or covenant not to sue for patent infringement). To "grant" such a patent license to a party means to make such an agreement or commitment not to enforce a patent against the party. If you convey a covered work, knowingly relying on a patent license, and the Corresponding Source of the work is not available for anyone to copy, free of charge and under the terms of this License, through a publicly available network server or other readily accessible means, then you must either (1) cause the Corresponding Source to be so available, or (2) arrange to deprive yourself of the benefit of the patent license for this particular work, or (3) arrange, in a manner consistent with the requirements of this License, to extend the patent license to downstream recipients. "Knowingly relying" means you have actual knowledge that, but for the patent license, your conveying the covered work in a country, or your recipient's use of the covered work in a country, would infringe one or more identifiable patents in that country that you have reason to believe are valid. If, pursuant to or in connection with a single transaction or arrangement, you convey, or propagate by procuring conveyance of, a covered work, and grant a patent license to some of the parties receiving the covered work authorizing them to use, propagate, modify or convey a specific copy of the covered work, then the patent license you grant is automatically extended to all recipients of the covered work and works based on it. A patent license is "discriminatory" if it does not include within the scope of its coverage, prohibits the exercise of, or is conditioned on the non-exercise of one or more of the rights that are specifically granted under this License. You may not convey a covered work if you are a party to an arrangement with a third party that is in the business of distributing software, under which you make payment to the third party based on the extent of your activity of conveying the work, and under which the third party grants, to any of the parties who would receive the covered work from you, a discriminatory patent license (a) in connection with copies of the covered work conveyed by you (or copies made from those copies), or (b) primarily for and in connection with specific products or compilations that contain the covered work, unless you entered into that arrangement, or that patent license was granted, prior to 28 March 2007. Nothing in this License shall be construed as excluding or limiting any implied license or other defenses to infringement that may otherwise be available to you under applicable patent law. 12. No Surrender of Others' Freedom. If conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot convey a covered work so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not convey it at all. For example, if you agree to terms that obligate you to collect a royalty for further conveying from those to whom you convey the Program, the only way you could satisfy both those terms and this License would be to refrain entirely from conveying the Program. 13. Use with the GNU Affero General Public License. Notwithstanding any other provision of this License, you have permission to link or combine any covered work with a work licensed under version 3 of the GNU Affero General Public License into a single combined work, and to convey the resulting work. The terms of this License will continue to apply to the part which is the covered work, but the special requirements of the GNU Affero General Public License, section 13, concerning interaction through a network will apply to the combination as such. 14. Revised Versions of this License. The Free Software Foundation may publish revised and/or new versions of the GNU General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. Each version is given a distinguishing version number. If the Program specifies that a certain numbered version of the GNU General Public License "or any later version" applies to it, you have the option of following the terms and conditions either of that numbered version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of the GNU General Public License, you may choose any version ever published by the Free Software Foundation. If the Program specifies that a proxy can decide which future versions of the GNU General Public License can be used, that proxy's public statement of acceptance of a version permanently authorizes you to choose that version for the Program. Later license versions may give you additional or different permissions. However, no additional obligations are imposed on any author or copyright holder as a result of your choosing to follow a later version. 15. Disclaimer of Warranty. THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 16. Limitation of Liability. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. 17. Interpretation of Sections 15 and 16. If the disclaimer of warranty and limitation of liability provided above cannot be given local legal effect according to their terms, reviewing courts shall apply local law that most closely approximates an absolute waiver of all civil liability in connection with the Program, unless a warranty or assumption of liability accompanies a copy of the Program in return for a fee. END OF TERMS AND CONDITIONS How to Apply These Terms to Your New Programs If you develop a new program, and you want it to be of the greatest possible use to the public, the best way to achieve this is to make it free software which everyone can redistribute and change under these terms. To do so, attach the following notices to the program. It is safest to attach them to the start of each source file to most effectively state the exclusion of warranty; and each file should have at least the "copyright" line and a pointer to where the full notice is found. Cumin - An automation and orchestration framework Copyright (C) 2017-2018 Riccardo Coccioli Wikimedia Foundation, Inc. This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . Also add information on how to contact you by electronic and paper mail. If the program does terminal interaction, make it output a short notice like this when it starts in an interactive mode: cumin Copyright (C) 2017-2018 Riccardo Coccioli Wikimedia Foundation, Inc. This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'. This is free software, and you are welcome to redistribute it under certain conditions; type `show c' for details. The hypothetical commands `show w' and `show c' should show the appropriate parts of the General Public License. Of course, your program's commands might be different; for a GUI interface, you would use an "about box". You should also get your employer (if you work as a programmer) or school, if any, to sign a "copyright disclaimer" for the program, if necessary. For more information on this, and how to apply and follow the GNU GPL, see . The GNU General Public License does not permit incorporating your program into proprietary programs. If your program is a subroutine library, you may consider it more useful to permit linking proprietary applications with the library. If this is what you want to do, use the GNU Lesser General Public License instead of this License. But first, please read . wikimedia-cumin-36f957f/README.rst000066400000000000000000000032111476500461000166300ustar00rootroot00000000000000Cumin - An automation and orchestration framework ------------------------------------------------- |GitHub Release| |PyPI Release| |License| Cumin provides a flexible and scalable automation framework to execute multiple commands on multiple hosts in parallel. It allows to easily perform complex selections of hosts through a user-friendly query language which can interface with different backend modules and combine their results for a fine grained selection. The transport layer can also be selected, and can provide multiple execution strategies. The executed commands outputs are automatically grouped for an easy-to-read result. It can be used both via its command line interface (CLI) `cumin` and as a Python 3 only library. Cumin was Python 2 only before the 3.0.0 release, due to ClusterShell not yet being Python 3 compatible. |Cumin GIF| The documentation is available on `Wikimedia Documentation`_ and `Read the Docs`_. The details on how Cumin it's used at the Wikimedia Foundation are available on `Wikitech`_. .. |GitHub Release| image:: https://img.shields.io/github/release/wikimedia/cumin.svg :target: https://github.com/wikimedia/cumin/releases .. |PyPI Release| image:: https://img.shields.io/pypi/v/cumin.svg :target: https://pypi.org/project/cumin/ .. |License| image:: https://img.shields.io/badge/license-GPLv3%2B-blue.svg :target: https://github.com/wikimedia/cumin/blob/master/LICENSE .. |Cumin GIF| image:: https://people.wikimedia.org/~volans/cumin.gif .. _`Read the Docs`: https://cumin.readthedocs.io .. _`Wikimedia Documentation`: https://doc.wikimedia.org/cumin .. _`Wikitech`: https://wikitech.wikimedia.org/wiki/Cumin wikimedia-cumin-36f957f/TODO.rst000066400000000000000000000062451476500461000164520ustar00rootroot00000000000000########## CUMIN TODO ########## Tracking ideas to improve Cumin. They are in no particular order inside each section and there is no guarantee that any item listed will be implemented in the nearby future. On the masters ============== Internal improvements / bug fixes --------------------------------- * CLI: fix progress bar interaction with ctrl+c and ``sigint_handler()``. * CLI: suppress normal output when ``-o/--output`` is used. * clustershell transport: decouple the output handling from the event handlers. * clustershell transport: improve test coverage for partial/total failures and timeouts. * clustershell transport: improve and extend integration tests. Small improvements ------------------ * global: allow to log the whole output to a specific file, to allow multiple people follow the progress. * global: allow to randomize the list hosts before execution `T164587`_. * CLI: improve the dry-run mode to show what would have been done. * CLI: read commands from a file, one per line. * CLI: add ``--limit`` to randomly select N hosts within a broader selection. * puppetdb backend: improve globbing support, check if fnmatch could be used without conflicting with ClusterShell NodeSet. * puppetdb backend: allow to specify boolean values for resource parameters `T161545`_. New Features ------------ * global: connection timeout/failure should be treated differently than normal failures: * don't consider them for the success threshold by default, add a ``--fail-always`` option for that * if the first command executed on a host fails with exit code 255, try to run ``/bin/true``, if it fails too it should be considered a connection timeout/failure * global: allow to notify the user who launched the execution on failure/termination trough IRC/email. Useful for long running jobs. * global: allow to differentiate the command to execute on a per-host basis, i.e. passing a different parameter for each host. * global: allow to have an external audit log and/or announce commands execution on IRC. * transports: add an output-only transport to nicely print the matching hosts and some related count. * transports: allow to specify a rollback strategy to be executed in each host on failure. * transports: add parallel execution of local commands on the master for each targeted host with the host as a parameter. Needs a new local transport with ExecWorker to shell out in parallel. * backends: generalize backends to allow to return other data too, not only the host certnames. * backends: add a new backend to support conftool. * CLI: when ``-i/--interactive`` is used and no command or query is specified, drop into a REPL session allowing to easily setup them. On the targets ============== Future plans ------------ * Create a single entry point to allow the execution of idempotent modules. * Create a safe and reliable sync up mechanism for the modules. * Allow to handle timeouts and failures locally within the module (local rollback and/or cleanup). * Allow to drop privileges into a different user. .. _`T159308`: https://phabricator.wikimedia.org/T159308 .. _`T161545`: https://phabricator.wikimedia.org/T161545 .. _`T164587`: https://phabricator.wikimedia.org/T164587 wikimedia-cumin-36f957f/cumin/000077500000000000000000000000001476500461000162575ustar00rootroot00000000000000wikimedia-cumin-36f957f/cumin/__init__.py000066400000000000000000000140411476500461000203700ustar00rootroot00000000000000"""Automation and orchestration framework written in Python.""" import logging import os import subprocess from importlib.metadata import PackageNotFoundError, version import yaml from ClusterShell.NodeSet import NodeSet, RESOLVER_NOGROUP KERBEROS_KLIST = '/usr/bin/klist' try: __version__ = version(__name__) """:py:class:`str`: the version of the current Cumin module.""" except PackageNotFoundError: # pragma: no cover - this happens only if the package is not installed # Support the use case of the Debian building system where tests are run without installation if 'SETUPTOOLS_SCM_PRETEND_VERSION' in os.environ: __version__ = os.environ['SETUPTOOLS_SCM_PRETEND_VERSION'] class CuminError(Exception): """Base Exception class for all Cumin's custom Exceptions.""" ############################################################################## # Add a custom log level TRACE to logging for development debugging LOGGING_TRACE_LEVEL_NUMBER = 8 LOGGING_TRACE_LEVEL_NAME = 'TRACE' # Fail if the custom logging slot is already in use with a different name or # Access to a private property of logging was preferred over matching the default string returned by # logging.getLevelName() for unused custom slots. if (LOGGING_TRACE_LEVEL_NUMBER in logging._levelToName # pylint: disable=protected-access and LOGGING_TRACE_LEVEL_NAME not in logging._nameToLevel): # pylint: disable=protected-access raise CuminError("Unable to set custom logging for trace, logging level {level} is alredy set for '{name}'.".format( level=LOGGING_TRACE_LEVEL_NUMBER, name=logging.getLevelName(LOGGING_TRACE_LEVEL_NUMBER))) def trace(self, msg, *args, **kwargs): """Additional logging level for development debugging. :Parameters: according to :py:class:`logging.Logger` interface for log levels. """ if self.isEnabledFor(LOGGING_TRACE_LEVEL_NUMBER): self._log(LOGGING_TRACE_LEVEL_NUMBER, msg, args, **kwargs) # pragma: no cover, pylint: disable=protected-access # Install the trace method and it's logging level if not already present if LOGGING_TRACE_LEVEL_NAME not in logging._nameToLevel: # pylint: disable=protected-access logging.addLevelName(LOGGING_TRACE_LEVEL_NUMBER, LOGGING_TRACE_LEVEL_NAME) if not hasattr(logging.Logger, 'trace'): logging.Logger.trace = trace # type: ignore ############################################################################## class Config(dict): """Singleton-like dictionary class to load the configuration from a given path only once.""" _instances = {} # Keep track of different loaded configurations def __new__(cls, config='/etc/cumin/config.yaml'): """Load the given configuration if not already loaded and return it. Called by Python's data model for each new instantiation of the class. Arguments: config (str, optional): path to the configuration file to load. Returns: dict: the configuration dictionary. Examples: >>> import cumin >>> config = cumin.Config() """ if config not in cls._instances: cls._instances[config] = parse_config(config) alias_file = os.path.join(os.path.dirname(config), 'aliases.yaml') if os.path.isfile(alias_file): # Load the aliases only if present cls._instances[config]['aliases'] = parse_config(alias_file) return cls._instances[config] def parse_config(config_file): """Parse the YAML configuration file. Arguments: config_file (str): the path of the configuration file to load. Returns: dict: the configuration dictionary. Raises: CuminError: if unable to read or parse the configuration. """ try: with open(os.path.expanduser(config_file), 'r', encoding='utf8') as f: config = yaml.safe_load(f) except IOError as e: raise CuminError('Unable to read configuration file: {message}'.format(message=e)) from e except yaml.parser.ParserError as e: raise CuminError("Unable to parse configuration file '{config}':\n{message}".format( config=config_file, message=e)) from e if config is None: config = {} return config def nodeset(nodes=None): """Instantiate a ClusterShell NodeSet with the resolver defaulting to :py:const:`RESOLVER_NOGROUP`. This allow to avoid any conflict with Cumin grammars. Returns: ClusterShell.NodeSet.NodeSet: the instantiated NodeSet. See Also: https://github.com/cea-hpc/clustershell/issues/368 """ return NodeSet(nodes=nodes, resolver=RESOLVER_NOGROUP) def nodeset_fromlist(nodelist): """Instantiate a ClusterShell NodeSet from a list with the resolver defaulting to :py:const:`RESOLVER_NOGROUP`. This allow to avoid any conflict with Cumin grammars. Returns: ClusterShell.NodeSet.NodeSet: the instantiated NodeSet. See Also: https://github.com/cea-hpc/clustershell/issues/368 """ return NodeSet.fromlist(nodelist, resolver=RESOLVER_NOGROUP) def ensure_kerberos_ticket(config: Config) -> None: """Ensure that there is a valid Kerberos ticket for the current user, according to the given configuration. Arguments: config (cumin.Config): the Cumin's configuration dictionary. """ kerberos_config = config.get('kerberos', {}) if not kerberos_config or not kerberos_config.get('ensure_ticket', False): return if not kerberos_config.get('ensure_ticket_root', False) and os.geteuid() == 0: return if not os.access(KERBEROS_KLIST, os.X_OK): raise CuminError('The Kerberos config ensure_ticket is set to true, but {klist} executable was ' 'not found.'.format(klist=KERBEROS_KLIST)) try: subprocess.run([KERBEROS_KLIST, '-s'], check=True) # nosec except subprocess.CalledProcessError as e: raise CuminError('The Kerberos config ensure_ticket is set to true, but no active Kerberos ticket was found, ' "please run 'kinit' and retry.") from e wikimedia-cumin-36f957f/cumin/backends/000077500000000000000000000000001476500461000200315ustar00rootroot00000000000000wikimedia-cumin-36f957f/cumin/backends/__init__.py000066400000000000000000000162571476500461000221550ustar00rootroot00000000000000"""Abstract backend.""" import logging from abc import ABCMeta, abstractmethod import pyparsing from cumin import CuminError, nodeset class InvalidQueryError(CuminError): """Custom exception class for invalid queries.""" class BaseQuery(metaclass=ABCMeta): """Query abstract class. All backends query classes must inherit, directly or indirectly, from this one. """ grammar = pyparsing.NoMatch() # This grammar will never match. """:py:class:`pyparsing.ParserElement`: derived classes must define their own pyparsing grammar and set this class attribute accordingly.""" def __init__(self, config): """Query constructor. Arguments: config (dict): a dictionary with the parsed configuration file. """ self.config = config self.logger = logging.getLogger('.'.join((self.__module__, self.__class__.__name__))) self.logger.trace('Backend %s created with config: %s', type(self).__name__, config) def execute(self, query_string): """Build and execute the query, return the NodeSet of FQDN hostnames that matches. Arguments: query_string (str): the query string to be parsed and executed. Returns: ClusterShell.NodeSet.NodeSet: with the FQDNs of the matching hosts. """ self._build(query_string) return self._execute() @abstractmethod def _execute(self): """Execute the already parsed query and return the NodeSet of FQDN hostnames that matches. Returns: ClusterShell.NodeSet.NodeSet: with the FQDNs of the matching hosts. """ @abstractmethod def _parse_token(self, token): """Recursively interpret the tokens returned by the grammar parsing. Arguments: token (pyparsing.ParseResults): a single token returned by the grammar parsing. """ def _build(self, query_string): """Parse the query string according to the grammar and build the query for later execution. Arguments: query_string (str): the query string to be parsed. """ self.logger.trace('Parsing query: %s', query_string) parsed = self.grammar.parseString(query_string.strip(), parseAll=True) self.logger.trace('Parsed query: %s', parsed) for token in parsed: self._parse_token(token) class BaseQueryAggregator(BaseQuery): """Query aggregator abstract class. Add to :py:class:`cumin.backends.BaseQuery` the capability of aggregating query subgroups and sub tokens into a unified result using common boolean operators for sets: ``and``, ``or``, ``and not`` and ``xor``. The class has a stack-like structure that must be populated by the derived classes while building the query. On execution the stack is traversed and the results are aggreagated together based on subgroups and boolean operators. """ def __init__(self, config): """Query aggregator constructor, initialize the stack. :Parameters: according to parent :py:meth:`cumin.backends.BaseQuery.__init__`. """ super().__init__(config) self.stack = None self.stack_pointer = None def _build(self, query_string): """Override parent method to reset the stack and log it. :Parameters: according to parent :py:meth:`cumin.backends.BaseQuery._build`. """ self.stack = self._get_stack_element() self.stack_pointer = self.stack super()._build(query_string) self.logger.trace('Query stack: %s', self.stack) def _execute(self): """Concrete implementation of parent abstract method. :Parameters: according to parent :py:meth:`cumin.backends.BaseQuery._execute`. """ hosts = nodeset() self._loop_stack(hosts, self.stack) # The hosts NodeSet is updated in place while looping the stack self.logger.debug('Found %d hosts', len(hosts)) return hosts def _open_subgroup(self): """Handle subgroup opening.""" element = self._get_stack_element() element['parent'] = self.stack_pointer self.stack_pointer['children'].append(element) self.stack_pointer = element def _close_subgroup(self): """Handle subgroup closing.""" self.stack_pointer = self.stack_pointer['parent'] @abstractmethod def _parse_token(self, token): """Re-define abstract method from parent abstract class. :Parameters: according to parent :py:meth:`cumin.backends.BaseQuery._parse_token`. """ @staticmethod def _get_stack_element(): """Return an empty stack element. Returns: dict: the dictionary with an empty stack element. """ return {'hosts': None, 'children': [], 'parent': None, 'bool': None} def _loop_stack(self, hosts, stack_element): """Loop the stack generated while parsing the query and aggregate the results. Arguments: hosts (ClusterShell.NodeSet.NodeSet): the hosts to be updated with the current stack element results. This object is updated in place by reference. stack_element (dict): the stack element to iterate. """ if stack_element['hosts'] is None: element_hosts = nodeset() for child in stack_element['children']: self._loop_stack(element_hosts, child) else: element_hosts = stack_element['hosts'] self._aggregate_hosts(hosts, element_hosts, stack_element['bool']) def _aggregate_hosts(self, hosts, element_hosts, bool_operator): """Aggregate hosts according to their boolean operator. Arguments: hosts (ClusterShell.NodeSet.NodeSet): the hosts to update with the results in ``element_hosts`` according to the ``bool_operator``. This object is updated in place by reference. element_hosts (ClusterShell.NodeSet.NodeSet): the additional hosts to aggregate to the results based on the ``bool_operator``. bool_operator (str, None): the boolean operator to apply while aggregating the two NodeSet. It must be :py:data:`None` when adding the first hosts. """ self.logger.trace("Aggregating: %s | %s | %s", hosts, bool_operator, element_hosts) # This should never happen if (bool_operator is None and hosts) or (bool_operator is not None and hosts is None): # pragma: no cover raise InvalidQueryError("Unexpected boolean operator '{boolean}' with hosts '{hosts}'".format( boolean=bool_operator, hosts=hosts)) if bool_operator is None or bool_operator == 'or': hosts |= element_hosts elif bool_operator == 'and': hosts &= element_hosts elif bool_operator == 'and not': hosts -= element_hosts elif bool_operator == 'xor': hosts ^= element_hosts else: # pragma: no cover - this should never happen raise InvalidQueryError( "Invalid bool operator '{boolean}' found, one of or|and|and not|xor expected".format( boolean=bool_operator)) wikimedia-cumin-36f957f/cumin/backends/direct.py000066400000000000000000000104611476500461000216570ustar00rootroot00000000000000"""Direct backend.""" import pyparsing as pp from cumin import nodeset_fromlist from cumin.backends import BaseQueryAggregator, InvalidQueryError def grammar(): """Define the query grammar. Backus-Naur form (BNF) of the grammar:: ::= | ::= | "(" ")" ::= "and not" | "and" | "xor" | "or" Given that the pyparsing library defines the grammar in a BNF-like style, for the details of the tokens not specified above check directly the source code. Returns: pyparsing.ParserElement: the grammar parser. """ # Boolean operators boolean = (pp.CaselessKeyword('and not') | pp.CaselessKeyword('and') | pp.CaselessKeyword('xor') | pp.CaselessKeyword('or'))('bool') # Parentheses lpar = pp.Literal('(')('open_subgroup') rpar = pp.Literal(')')('close_subgroup') # Hosts selection: clustershell (,!&^[]) syntax is allowed: host10[10-42].domain hosts = (~(boolean) + pp.Word(pp.alphanums + '-_.,!&^[]'))('hosts') # Final grammar, see the docstring for its BNF based on the tokens defined above # Groups are used to split the parsed results for an easy access full_grammar = pp.Forward() item = hosts | lpar + full_grammar + rpar full_grammar << pp.Group(item) + pp.ZeroOrMore(pp.Group(boolean + item)) # pylint: disable=expression-not-assigned return full_grammar class DirectQuery(BaseQueryAggregator): """DirectQuery query builder. The `direct` backend allow to use Cumin without any external dependency for the hosts selection. It allow to write arbitrarily complex queries with subgroups and boolean operators, but each item must be either the hostname itself, or the using host expansion using the powerful :py:class:`ClusterShell.NodeSet.NodeSet` syntax. The typical usage for the `direct` backend is as a reliable alternative in cases in which the primary host selection mechanism is not working and also for testing the transports without any external backend dependency. Some query examples: * Simple selection: ``host1.domain`` * ClusterShell syntax for hosts expansion: ``host10[10-42].domain,host2010.other-domain`` * A complex selection: ``host100[1-5].domain or (host10[30-40].domain and (host10[10-42].domain and not host33.domain))`` """ grammar = grammar() """:py:class:`pyparsing.ParserElement`: load the grammar parser only once in a singleton-like way.""" def _parse_token(self, token): """Concrete implementation of parent abstract method. :Parameters: according to parent :py:meth:`cumin.backends.BaseQueryAggregator._parse_token`. """ if not isinstance(token, pp.ParseResults): # pragma: no cover - this should never happen raise InvalidQueryError('Expecting ParseResults object, got {type}: {token}'.format( type=type(token), token=token)) token_dict = token.asDict() self.logger.trace('Token is: %s | %s', token_dict, token) if 'hosts' in token_dict: element = self._get_stack_element() element['hosts'] = nodeset_fromlist(token_dict['hosts']) if 'bool' in token_dict: element['bool'] = token_dict['bool'] self.stack_pointer['children'].append(element) elif 'open_subgroup' in token_dict and 'close_subgroup' in token_dict: self._open_subgroup() if 'bool' in token_dict: self.stack_pointer['bool'] = token_dict['bool'] for subtoken in token: if isinstance(subtoken, str): # Grammar literals, boolean operators and parentheses continue self._parse_token(subtoken) self._close_subgroup() else: # pragma: no cover - this should never happen raise InvalidQueryError('Got unexpected token: {token}'.format(token=token)) GRAMMAR_PREFIX = 'D' """:py:class:`str`: the prefix associate to this grammar, to register this backend into the general grammar. Required by the backend auto-loader in :py:meth:`cumin.grammar.get_registered_backends`.""" query_class = DirectQuery # pylint: disable=invalid-name """Required by the backend auto-loader in :py:meth:`cumin.grammar.get_registered_backends`.""" wikimedia-cumin-36f957f/cumin/backends/knownhosts.py000066400000000000000000000232641476500461000226270ustar00rootroot00000000000000"""Known hosts backend.""" import ipaddress import pyparsing as pp from ClusterShell.NodeSet import NodeSet from ClusterShell.NodeUtils import GroupResolver, GroupSource from cumin.backends import BaseQueryAggregator, InvalidQueryError def grammar(): """Define the query grammar. Some query examples: * Simple selection: ``host1.domain`` * ClusterShell syntax for hosts expansion: ``host10[10-42].domain,host2010.other-domain`` * ClusterShell syntax for hosts globbing: ``host10[10-42]*`` * A complex selection: ``host100[1-5]* or (host10[30-40].domain and (host10[10-42].domain and not host33.domain))`` Backus-Naur form (BNF) of the grammar:: ::= | ::= | "(" ")" ::= "and not" | "and" | "xor" | "or" Given that the pyparsing library defines the grammar in a BNF-like style, for the details of the tokens not specified above check directly the source code. Returns: pyparsing.ParserElement: the grammar parser. """ # Boolean operators boolean = (pp.CaselessKeyword('and not') | pp.CaselessKeyword('and') | pp.CaselessKeyword('xor') | pp.CaselessKeyword('or'))('bool') # Parentheses lpar = pp.Literal('(')('open_subgroup') rpar = pp.Literal(')')('close_subgroup') # Hosts selection: clustershell (,!&^[]) syntax is allowed: host10[10-42].domain hosts = (~(boolean) + pp.Word(pp.alphanums + '-_.,!&^[]*?'))('hosts') # Final grammar, see the docstring for its BNF based on the tokens defined above # Groups are used to split the parsed results for an easy access full_grammar = pp.Forward() item = hosts | lpar + full_grammar + rpar full_grammar << pp.Group(item) + pp.ZeroOrMore(pp.Group(boolean + item)) # pylint: disable=expression-not-assigned return full_grammar class KnownHostsLineError(InvalidQueryError): """Custom exception class for invalid lines in SSH known hosts files.""" class KnownHostsSkippedLineError(InvalidQueryError): """Custom exception class for skipped lines in SSH known hosts files.""" class KnownHostsQuery(BaseQueryAggregator): """KnownHostsQuery query builder. The ``knownhosts`` backend allow to use Cumin taking advantage of existing SSH known hosts files that are not hashed. It allow to write arbitrarily complex queries with subgroups and boolean operators, but each item must be either the hostname itself, or using host expansion with the powerful :py:class:`ClusterShell.NodeSet.NodeSet` syntax. The typical use case for the ``knownhosts`` backend is when the known hosts file(s) are generated and kept updated by some external configuration manager or tool that is not yet supported as a backend for Cumin. It can also work as a fallback backend in case the primary backend is unavailable but the known hosts file(s) are still up to date. """ grammar = grammar() """:py:class:`pyparsing.ParserElement`: load the grammar parser only once in a singleton-like way.""" def __init__(self, config): """Known hosts query constructor, initialize the known hosts. :Parameters: according to parent :py:meth:`cumin.backends.BaseQuery.__init__`. """ super().__init__(config) self.known_hosts = set() self.resolver = None def _build(self, query_string): """Override parent method to lazy-loading the known hosts if needed. :Parameters: according to parent :py:meth:`cumin.backends.BaseQuery._build`. """ if not self.known_hosts: self._load_known_hosts() if self.resolver is None: source = GroupSource('all', allgroups='\n'.join(self.known_hosts)) self.resolver = GroupResolver(default_source=source) super()._build(query_string) def _execute(self): """Override parent method to ensure to return only existing hosts. :Parameters: according to parent :py:meth:`cumin.backends.BaseQuery._execute`. """ hosts = super()._execute() return hosts & NodeSet('*', resolver=self.resolver) def _parse_token(self, token): """Concrete implementation of parent abstract method. :Parameters: according to parent :py:meth:`cumin.backends.BaseQueryAggregator._parse_token`. """ if not isinstance(token, pp.ParseResults): # pragma: no cover - this should never happen raise InvalidQueryError('Expecting ParseResults object, got {type}: {token}'.format( type=type(token), token=token)) token_dict = token.asDict() self.logger.trace('Token is: %s | %s', token_dict, token) if 'hosts' in token_dict: element = self._get_stack_element() element['hosts'] = NodeSet.fromlist(token_dict['hosts'], resolver=self.resolver) if 'bool' in token_dict: element['bool'] = token_dict['bool'] self.stack_pointer['children'].append(element) elif 'open_subgroup' in token_dict and 'close_subgroup' in token_dict: self._open_subgroup() if 'bool' in token_dict: self.stack_pointer['bool'] = token_dict['bool'] for subtoken in token: if isinstance(subtoken, str): # Grammar literals, boolean operators and parentheses continue self._parse_token(subtoken) self._close_subgroup() else: # pragma: no cover - this should never happen raise InvalidQueryError('Got unexpected token: {token}'.format(token=token)) def _load_known_hosts(self): """Load all known hosts file listed in the configuration.""" config = self.config.get('knownhosts', {}) known_hosts_filenames = config.get('files', []) for filename in known_hosts_filenames: hosts = set() with open(filename, 'r', encoding='utf8') as known_hosts_file: for lineno, line in enumerate(known_hosts_file, 1): try: found, skipped = KnownHostsQuery.parse_known_hosts_line(line) if skipped: self.logger.trace("Skipped patterns at line %d in known hosts file '%s': %s", lineno, filename, ', '.join(skipped)) hosts.update(found) except KnownHostsLineError as e: self.logger.warning("Discarded invalid line %d (%s) in known hosts file '%s': %s", lineno, e, filename, line) except KnownHostsSkippedLineError as e: self.logger.trace("Skipped %s line %d in known hosts file '%s': %s", e, lineno, filename, line) self.logger.debug("Loaded %d hosts from '%s'", len(hosts), filename) self.known_hosts.update(hosts) @staticmethod def parse_known_hosts_line(line): """Parse an SSH known hosts formatted line and extract the valid hostnames. See the ``SSH_KNOWN_HOSTS FILE FORMAT` in ``man sshd`` for the details of the file format. Arguments: line (str): the line to parse. Raises: KnownHostsSkippedLineError: if the line is skipped. KnownHostsLineError: if unable to parse the line. Returns: set: a set with the hostnames found in the given line. """ line = line.strip() if not line: raise KnownHostsSkippedLineError('empty line') if line[0] == '#': raise KnownHostsSkippedLineError('comment') if line[0] == '|': raise KnownHostsSkippedLineError('hashed') fields = line.split() if len(fields) < 3: raise KnownHostsLineError('not enough fields') if line[0] == '@': if len(fields) < 4: raise KnownHostsLineError('not enough fields') if fields[0] == '@cert-authority': line_hosts = fields[1] elif fields[0] == '@revoked': raise KnownHostsSkippedLineError('revoked') else: raise KnownHostsLineError('unknown marker') else: line_hosts = fields[0] return KnownHostsQuery.parse_line_hosts(line_hosts) @staticmethod def parse_line_hosts(line_hosts): """Parse a comma-separated hostnamed from an SSH known hosts formatted line and extract the valid hostnames. Arguments: line_hosts (str): the hostnames to parse. Returns: tuple: a tuple with two sets, the hostnames found in the given line and the hostnames skipped. """ hosts = set() skipped = set() for host in line_hosts.split(','): if not host: continue if host[0] == '!': host = host[1:] if host[0] == '[': host = host[1:].split(']')[0] if '*' in host or '?' in host: skipped.add(host) else: try: ipaddress.ip_address(host) skipped.add(host) except ValueError: hosts.add(host) # Add hostnames, skip IP addresses return hosts, skipped GRAMMAR_PREFIX = 'K' """:py:class:`str`: the prefix associate to this grammar, to register this backend into the general grammar. Required by the backend auto-loader in :py:meth:`cumin.grammar.get_registered_backends`.""" query_class = KnownHostsQuery # pylint: disable=invalid-name """Required by the backend auto-loader in :py:meth:`cumin.grammar.get_registered_backends`.""" wikimedia-cumin-36f957f/cumin/backends/openstack.py000066400000000000000000000240761476500461000224030ustar00rootroot00000000000000"""OpenStack backend.""" import pyparsing as pp from keystoneauth1 import session as keystone_session from keystoneauth1.identity import v3 as keystone_identity from keystoneclient.v3 import client as keystone_client from novaclient import client as nova_client from cumin import nodeset, nodeset_fromlist from cumin.backends import BaseQuery, InvalidQueryError def grammar(): """Define the query grammar. Backus-Naur form (BNF) of the grammar:: ::= "*" | ::= | ::= : Given that the pyparsing library defines the grammar in a BNF-like style, for the details of the tokens not specified above check directly the source code. Returns: pyparsing.ParserElement: the grammar parser. """ quoted_string = pp.quotedString.copy().addParseAction(pp.removeQuotes) # Both single and double quotes are allowed # Key-value tokens: key:value # Lowercase key, all printable characters except the parentheses that are part of the global grammar for the value key = pp.Word(pp.srange('[a-z0-9-_.]"'), min=2)('key') all_but_par = ''.join([c for c in pp.printables if c not in ('(', ')', '{', '}')]) value = (quoted_string | pp.Word(all_but_par))('value') item = pp.Combine(key + ':' + value) # Final grammar, see the docstring for its BNF based on the tokens defined above # Groups are used to split the parsed results for an easy access return pp.Group(pp.Literal('*')('all')) | pp.OneOrMore(pp.Group(item)) def _get_keystone_session(config, project=None): """Return a new keystone session based on configuration. Arguments: config (dict): a dictionary with the session configuration keys: ``auth_url``, ``username``, ``password``. project (str, optional): a project to scope the session to. Returns: keystoneauth1.session.Session: the Keystone session scoped for the project if specified. """ auth = keystone_identity.Password( auth_url='{auth_url}/v3'.format(auth_url=config.get('auth_url', 'http://localhost:5000')), username=config.get('username', 'username'), password=config.get('password', 'password'), project_name=project, user_domain_id='default', project_domain_id='default') return keystone_session.Session(auth=auth) def _get_nova_client(config, project): """Return a new nova client tailored to the given project. Arguments: config (dict): a dictionary with the session configuration keys: ``auth_url``, ``username``, ``password``, ``nova_api_version``, ``timeout``. project (str): the project to scope the `novaclient` session to. Returns: novaclient.client.Client: the novaclient Client instance, already authenticated. """ params = config.get('client_params', {}) return nova_client.Client( config.get('nova_api_version', '2'), session=_get_keystone_session(config, project), endpoint_type='public', timeout=config.get('timeout', 10), **params) class OpenStackQuery(BaseQuery): r"""OpenStackQuery query builder. Query VMs deployed in an OpenStack infrastructure using the API. This is an optional backend, its dependencies will not be installed automatically, see the Installation section of the documentation for more details. * Each query can specify multiple parameters to filter the hosts selection in the form ``key:value``. * The special ``project`` key allow to filter by the OpenStack project name: ``project:project_name``. If not specified all the visible and enabled projects will be queried. * Any other ``key:value`` pair will be passed as is to the `OpenStack Compute API list-servers `_. Multiple filters can be added separated by space. The value can be enclosed in single or double quotes: ``name:"host1.*\.domain" image:UUID`` * By default the filters ``status:ACTIVE`` and ``vm_state:ACTIVE`` are also added, but will be overridden if specified in the query. * To mix multiple selections the general grammar must be used with multiple subqueries: ``O{project:project1} or O{project:project2}`` * The special query ``*`` is a shortcut to select all hosts in all OpenStack projects. * See the example configuration in ``doc/examples/config.yaml`` for all the OpenStack-related parameters that can be set. Some query examples: * All hosts in all OpenStack projects: ``*`` * All hosts in a specific OpenStack project: ``project:project_name`` * Filter hosts using any parameter allowed by the OpenStack list-servers API: ``name:host1 image:UUID`` See `OpenStack Compute API list-servers `_ for more details. Multiple filters can be added separated by space. The value can be enclosed in single or double quotes. If the ``project`` key is not specified the hosts will be selected from all projects. * To mix multiple selections the general grammar must be used with multiple subqueries: ``O{project:project1} or O{project:project2}`` """ grammar = grammar() """:py:class:`pyparsing.ParserElement`: load the grammar parser only once in a singleton-like way.""" def __init__(self, config): """Override parent class constructor for specific setup. :Parameters: according to parent :py:meth:`cumin.backends.BaseQuery.__init__`. """ super().__init__(config) self.openstack_config = self.config.get('openstack', {}) self.search_project = None self.search_params = self._get_default_search_params() def _get_default_search_params(self): """Return the default search parameters dictionary and set the project, if configured. Returns: dict: the dictionary with the default search parameters. """ params = {'status': 'ACTIVE', 'vm_state': 'ACTIVE'} config_params = self.openstack_config.get('query_params', {}) if 'project' in config_params: self.search_project = config_params.pop('project') params.update(config_params) return params def _build(self, query_string): """Override parent class _build method to reset the search parameters. :Parameters: according to parent :py:meth:`cumin.backends.BaseQuery._build`. """ self.search_params = self._get_default_search_params() super()._build(query_string) def _execute(self): """Concrete implementation of parent abstract method. :Parameters: according to parent :py:meth:`cumin.backends.BaseQuery._execute`. Returns: ClusterShell.NodeSet.NodeSet: with the FQDNs of the matching hosts. """ if self.search_project is None: hosts = nodeset() for project in self._get_projects(): hosts |= self._get_project_hosts(project) else: hosts = self._get_project_hosts(self.search_project) return hosts def _parse_token(self, token): """Concrete implementation of parent abstract method. :Parameters: according to parent :py:meth:`cumin.backends.BaseQuery._parse_token`. Raises: cumin.backends.InvalidQueryError: on internal parsing error. """ if not isinstance(token, pp.ParseResults): # pragma: no cover - this should never happen raise InvalidQueryError('Expecting ParseResults object, got {type}: {token}'.format( type=type(token), token=token)) token_dict = token.asDict() self.logger.trace('Token is: %s | %s', token_dict, token) if 'key' in token_dict and 'value' in token_dict: if token_dict['key'] == 'project': self.search_project = token_dict['value'] else: self.search_params[token_dict['key']] = token_dict['value'] elif 'all' in token_dict: pass # nothing to do, search_project and search_params have the right defaults else: # pragma: no cover - this should never happen raise InvalidQueryError('Got unexpected token: {token}'.format(token=token)) def _get_projects(self): """Get all the project names from keystone API, filtering out the special `admin` project. Is a `generator`. Yields: str: the project name for all the selected projects. """ client = keystone_client.Client( session=_get_keystone_session(self.openstack_config), timeout=self.openstack_config.get('timeout', 10)) return (project.name for project in client.projects.list(enabled=True) if project.name != 'admin') def _get_project_hosts(self, project): """Return a NodeSet with the list of matching hosts based for the project based on the search parameters. Arguments: project (str): the project name where to get the list of hosts. Returns: ClusterShell.NodeSet.NodeSet: with the FQDNs of the matching hosts. """ client = _get_nova_client(self.openstack_config, project) domain = '' domain_suffix = self.openstack_config.get('domain_suffix', None) if domain_suffix is not None: if domain_suffix[0] != '.': domain = '.{suffix}'.format(suffix=domain_suffix) else: domain = domain_suffix return nodeset_fromlist('{host}.{project}{domain}'.format(host=server.name, project=project, domain=domain) for server in client.servers.list(search_opts=self.search_params)) GRAMMAR_PREFIX = 'O' """:py:class:`str`: the prefix associate to this grammar, to register this backend into the general grammar. Required by the backend auto-loader in :py:meth:`cumin.grammar.get_registered_backends`.""" query_class = OpenStackQuery # pylint: disable=invalid-name """Required by the backend auto-loader in :py:meth:`cumin.grammar.get_registered_backends`.""" wikimedia-cumin-36f957f/cumin/backends/puppetdb.py000066400000000000000000000640441476500461000222360ustar00rootroot00000000000000"""PuppetDB backend.""" from re import IGNORECASE from string import capwords import pyparsing as pp import requests import urllib3 from cumin import nodeset, nodeset_fromlist from cumin.backends import BaseQuery, InvalidQueryError CATEGORIES = ('C', 'F', 'I', 'O', 'P', 'R') """:py:func:`tuple`: available categories in the grammar. * ``C``: shortcut for querying resources of type ``Class``, equivalent of `R:Class = class_path``. * ``F``: for querying legacy facts. * ``I``: for querying structured facts and other supported entities using the PuppetDB's dot notation through the inventory API endpoint. * ``O``: shortcut for querying resources of type ``Class`` that starts with ``Role::``. * ``P``: shortcut for querying resources of type ``Class`` that starts with ``Profile::``. * ``R``: for querying generic resources. """ OPERATORS = ('=', '>=', '<=', '<', '>', '~') """:py:func:`tuple`: available operators in the grammar, the same available in PuppetDB API. The ``~`` one is used for regex matching. """ class ParsedString: """Simple string wrapper which can communicate if a string should be enquoted downstream.""" def __init__(self, string, is_quoted): """Constructor for ParsedString. Arguments: string (str): The string to store in this object. is_quoted (bool): Whether the output should be quoted when this is converted to a string. """ self.string = str(string) self.is_quoted = is_quoted def __str__(self): """Return a string version of this value, enquoted or not based on the is_quoted property.""" if self.is_quoted: return '"{}"'.format(self.string) return self.string def capwords(self, sep): """Perform capwords operation on internal value and return a new ParsedString. :Parameters: according to :py:meth:`string.capwords`. """ return ParsedString(capwords(self.string, sep), self.is_quoted) def replace(self, old, new, count=-1): """Perform replace operation on internal value and return a new ParsedString. :Parameters: according to :py:meth:`str.replace`. """ return ParsedString(self.string.replace(old, new, count), self.is_quoted) def grammar(): # pylint: disable=too-many-locals """Define the query grammar. Backus-Naur form (BNF) of the grammar:: ::= | ::= [] | [] "(" ")" ::= | ::= : [ ] ::= | | | Given that the pyparsing library defines the grammar in a BNF-like style, for the details of the tokens not specified above check directly the source code. Returns: pyparsing.ParserElement: the grammar parser. """ # Boolean operators and_or = (pp.CaselessKeyword('and') | pp.CaselessKeyword('or'))('bool') # 'neg' is used as label to allow the use of dot notation, 'not' is a reserved word in Python neg = pp.CaselessKeyword('not')('neg') operator = pp.oneOf(OPERATORS, caseless=True)('operator') # Comparison operators quoted_string = pp.quotedString.copy().addParseAction(pp.removeQuotes) # Both single and double quotes are allowed # Parentheses lpar = pp.Literal('(')('open_subgroup') rpar = pp.Literal(')')('close_subgroup') # Hosts selection: glob (*) and clustershell (,!&^[]) syntaxes are allowed: # i.e. host10[10-42].*.domain hosts = quoted_string | (~(and_or | neg) + pp.Word(pp.alphanums + '-_.*,!&^[]')) # Key-value token for allowed categories using the available comparison operators # i.e. F:key = value category = pp.oneOf(tuple(c for c in CATEGORIES if c != 'I'), caseless=True)('category') key = pp.Word(pp.alphanums + '-_.%@:')('key') # Key-value token for the inventory category using the available comparison operators # i.e. I:facts.key[0]."sub.key" = value inventory_category = pp.oneOf(('I',), caseless=True)('category') inventory_key = pp.Word(pp.alphanums + '-_.%@:"[]')('key') # Category and key selector selector = (pp.Combine(category + ':' + key) | pp.Combine(inventory_category + ':' + inventory_key)) # All printables characters except the parentheses that are part of this or the global grammar all_but_par = ''.join([c for c in pp.printables if c not in ('(', ')', '{', '}')]) # PuppetDB accepts JSON Atoms bareword = pp.oneOf(('true', 'false')) # octal numbers are bare numerics that lead with 0. octal = pp.Word("0", "01234567", min=2).addParseAction(lambda toks: int(toks[0], 8)) # hex integers are in the format 0x[0-9A-F]+ hexadecimal = pp.Regex(r'0x[0-9A-F]+', flags=IGNORECASE).addParseAction(lambda toks: int(toks[0], 16)) number = pp.pyparsing_common.number # label indicates post-processing needed (value = nonquoted, quoted=quoted) value = (hexadecimal ^ octal ^ number ^ bareword)('value') ^ (quoted_string ^ pp.Word(all_but_par))('quoted') token = selector + pp.Optional(operator + value) # Final grammar, see the docstring for its BNF based on the tokens defined above # Groups are used to split the parsed results for an easy access full_grammar = pp.Forward() item = pp.Group(pp.Optional(neg) + (token | hosts('hosts'))) | pp.Group( pp.Optional(neg) + lpar + full_grammar + rpar) full_grammar << item + pp.ZeroOrMore(pp.Group(and_or) + full_grammar) # pylint: disable=expression-not-assigned return full_grammar class PuppetDBQuery(BaseQuery): """PuppetDB query builder. The `puppetdb` backend allow to use an existing PuppetDB instance for the hosts selection. The supported PuppetDB API version is 4. * Each query part can be composed with the others using boolean operators (``and``, ``or``, ``not``) * Multiple query parts can be grouped together with parentheses (``(``, ``)``). * A query part can be of two different types: * ``Hostname matching``: this is a simple string that be used to match directly the hostname of the hosts in the selected backend. It allows for glob expansion (``*``) and the use of the powerful :py:class:`ClusterShell.NodeSet.NodeSet`. * ``Category matching``: an identifier composed by a category, a colon and a key, followed by a comparison operator and a value, as in ``F:key = value``. * Values may be of various types supported by PuppetDB (numerics, boolean, and strings) for example: * Booleans: ``true``, ``false`` * Strings: ``'a string'``, and unquoted single words that aren't ``true`` or ``false`` and do not start with an integer. * Numeric values: ``15``, ``23.5``, ``0``, ``0xfa`` *Note: hexadecimal and octal numbers are supported by cumin but converted into normal integers. Some fields in PuppetDB may have hex or octal stored as strings, and should be quoted such as* ``'0xfa'``. *Note: PuppetDB may or may not support a particular value type for a particular resource.* Some query examples: * All hosts: ``*`` * Hosts globbing: ``host10*`` * :py:class:`ClusterShell.NodeSet.NodeSet` syntax for hosts expansion: ``host10[10-42].domain`` * Category based key-value selection: * ``R:Resource::Name``: query all the hosts that have a resource of type `Resource::Name`. * ``R:Resource::Name = 'resource-title'``: query all the hosts that have a resource of type `Resource::Name` whose title is ``resource-title``. For example ``R:Class = MyModule::MyClass``. * ``R:Resource::Name@field = 'some-value'``: query all the hosts that have a resource of type ``Resource::Name`` whose field ``field`` has the value ``some-value``. The valid fields are: ``tag``, ``certname``, ``type``, ``title``, ``exported``, ``file``, ``line``. The previous syntax is a shortcut for this one with the field ``title``. * ``R:Resource::Name%param = 'some-value'``: query all the hosts that have a resource of type ``Resource::Name`` whose parameter ``param`` has the value ``some-value``. * ``C:Class::Name``: special shortcut to query all the hosts that have a resource of type ``Class`` whose name is ``Class::Name``. The ``Class::Name`` part is completely arbitrary and depends on the puppet hierarchy chosen. It's equivalent to ``R:Class = Class::Name``, with the addition that the ``param`` and ``field`` selectors described above can be used directly without the need to add another condition. * ``O:Module::Name``: special shortcut to query all the hosts that have a resource of type ``Class`` whose name is ``Role::Module::Name``. The ``Module::Name`` part is completely arbitrary and depends on the puppet hierarchy chosen. It's equivalent to ``R:Class = Role::Module::Name``, with the addition that the ``param`` and ``field`` selectors described above can be used directly without the need to add another condition, although usually roles should not have parameters in the role/profile Puppet paradigm. * ``P:Module::Name``: special shortcut to query all the hosts that have a resource of type ``Class`` whose name is ``Profile::Module::Name``. The ``Module::Name`` part is completely arbitrary and depends on the puppet hierarchy chosen. It's equivalent to ``R:Class = Profile::Module::Name``, with the addition that the ``param`` and ``field`` selectors described above can be used directly without the need to add another condition. * ``F:FactName = value``: query all the hosts that have a fact ``FactName``, as reported by facter, with the value ``value``. * ``I:facts.factname.key.subkey = value``: query all the hosts that have the structured fact value represented in PuppetDB's dot notation with the value ``value`` using PuppetDB's inventory endpoint. The ``facts.`` prefix is required to query facts. It also allows to query other entities returned by the ``inventory`` endpoint. See also: https://www.puppet.com/docs/puppetdb/latest/api/query/v4/ast#dot-notation * Mixed facts/resources queries are not supported, but the same result can be achieved using the global grammar with multiple subqueries for the PuppetDB backend. * All hosts with physicalcorecount fact greater than 2: ``F:physicalcorecount > 2`` or ``I:facts.processors.cores > 2`` * A complex selection for facts: ``host10[10-42].*.domain or (not F:key1 = value1 and host10*) or (F:key2 > value2 and F:key3 ~ '^value[0-9]+')`` """ base_url_template = '{scheme}://{host}:{port}/pdb/query/v4/' """:py:class:`str`: string template in the :py:meth:`str.format` style used to generate the base URL of the PuppetDB server.""" endpoints = {'C': 'resources', 'F': 'nodes', 'I': 'inventory', 'O': 'resources', 'P': 'resources', 'R': 'resources'} """:py:class:`dict`: dictionary with the mapping of the available categories in the grammar to the PuppetDB API endpoints.""" category_prefixes = {'C': '', 'O': 'Role', 'P': 'Profile'} """:py:class:`dict`: dictionary with the mapping of special categories to title prefixes.""" grammar = grammar() """:py:class:`pyparsing.ParserElement`: load the grammar parser only once in a singleton-like way.""" def __init__(self, config): """Query constructor for the PuppetDB backend. :Parameters: according to parent :py:meth:`cumin.backends.BaseQuery.__init__`. """ super().__init__(config) self.grouped_tokens = None self.current_group = self.grouped_tokens self._endpoint = None puppetdb_config = self.config.get('puppetdb', {}) self.url = self.base_url_template.format( scheme=puppetdb_config.get('scheme', 'https'), host=puppetdb_config.get('host', 'localhost'), port=puppetdb_config.get('port', 443)) self.timeout = puppetdb_config.get('timeout', 30) self.ssl_verify = puppetdb_config.get('ssl_verify', True) self.ssl_client_cert = puppetdb_config.get('ssl_client_cert', '') self.ssl_client_key = puppetdb_config.get('ssl_client_key', '') for exception in puppetdb_config.get('urllib3_disable_warnings', []): urllib3.disable_warnings(category=getattr(urllib3.exceptions, exception)) @property def endpoint(self): """Endpoint in the PuppetDB API for the current query. :Getter: Returns the current `endpoint` or a default value if not set. :Setter: :py:class:`str`: the value to set the `endpoint` to. Raises: cumin.backends.InvalidQueryError: if trying to set it to an invalid `endpoint` or mixing endpoints in a single query. """ return self._endpoint or 'nodes' @endpoint.setter def endpoint(self, value): """Setter for the `endpoint` property. The relative documentation is in the getter.""" if value not in self.endpoints.values(): raise InvalidQueryError("Invalid value '{endpoint}' for endpoint property".format(endpoint=value)) if self._endpoint is not None and value != self._endpoint: raise InvalidQueryError('Mixed endpoints are not supported, use the global grammar to mix them.') self._endpoint = value def _open_subgroup(self): """Handle subgroup opening.""" token = PuppetDBQuery._get_grouped_tokens() token['parent'] = self.current_group self.current_group['tokens'].append(token) self.current_group = token def _close_subgroup(self): """Handle subgroup closing.""" self.current_group = self.current_group['parent'] @staticmethod def _get_grouped_tokens(): """Return an empty grouped tokens structure. Returns: dict: the dictionary with the empty grouped tokens structure. """ return {'parent': None, 'bool': None, 'tokens': []} def _build(self, query_string): """Override parent class _build method to reset tokens and add logging. :Parameters: according to parent :py:meth:`cumin.backends.BaseQuery._build`. """ self.grouped_tokens = PuppetDBQuery._get_grouped_tokens() self.current_group = self.grouped_tokens super()._build(query_string) self.logger.trace('Query tokens: %s', self.grouped_tokens) def _execute(self): """Concrete implementation of parent abstract method. :Parameters: according to parent :py:meth:`cumin.backends.BaseQuery._execute`. Returns: ClusterShell.NodeSet.NodeSet: with the FQDNs of the matching hosts. """ query = self._get_query_string(group=self.grouped_tokens) full_query = f'["extract", ["certname"], {query}, ["group_by", "certname"]]' hosts = nodeset_fromlist([host['certname'] for host in self._api_call(full_query)]) self.logger.debug("Queried puppetdb for '%s', got '%d' results.", query, len(hosts)) return hosts def _add_category(self, *, category, key, value=None, operator='=', neg=False): """Add a category token to the query 'F:key = value'. Arguments: category (str): the category of the token, one of :py:const:`CATEGORIES`. key (str): the key for this category. value (str, optional): the value to match, if not specified the key itself will be matched. operator (str, optional): the comparison operator to use, one of :py:const:`OPERATORS`. neg (bool, optional): whether the token must be negated. Raises: cumin.backends.InvalidQueryError: on internal parsing error. """ self.endpoint = self.endpoints[category] if operator == '~': # PuppetDB API requires to escape every backslash # See: https://puppet.com/docs/puppetdb/4.4/api/query/v4/ast.html#regexp-match value = value.replace('\\', '\\\\') if category in ('C', 'O', 'P'): query = self._get_special_resource_query(category, key, value, operator) elif category == 'R': query = self._get_resource_query(key, value, operator) elif category == 'F': query = '["{op}", ["fact", "{key}"], {val}]'.format(op=operator, key=key, val=value) elif category == 'I': # In dot-notation if a part of a key has dots they need to be double-quoted, escaping for the JSON query = '["{op}", "{key}", {val}]'.format(op=operator, key=key.replace('"', r'\"'), val=value) else: # pragma: no cover - this should never happen raise InvalidQueryError( "Got invalid category '{category}', one of F|O|P|R expected".format(category=category)) if neg: query = '["not", {query}]'.format(query=query) self.current_group['tokens'].append(query) def _add_hosts(self, hosts, neg=False): """Add a list of hosts to the query. Arguments: hosts (list): list of :py:class:`ClusterShell.NodeSet.NodeSet` with the list of hosts to search. neg (bool, optional): whether the token must be negated. """ hosts_tokens = [] for hosts_set in hosts: for host in hosts_set: operator = '=' # Convert a glob expansion into a regex if '*' in host: operator = '~' host = r'^' + host.replace('.', r'\\.').replace('*', '.*') + r'$' hosts_tokens.append('["{op}", "certname", "{host}"]'.format(op=operator, host=host)) if not hosts_tokens: return query = '["or", {hosts}]'.format(hosts=', '.join(hosts_tokens)) if neg: query = '["not", {query}]'.format(query=query) self.current_group['tokens'].append(query) def _parse_token(self, token): """Concrete implementation of parent abstract method. :Parameters: according to parent :py:meth:`cumin.backends.BaseQuery._parse_token`. Raises: cumin.backends.InvalidQueryError: on internal parsing error. """ if isinstance(token, str): return token_dict = token.asDict() # post-process types if 'quoted' in token_dict: token_dict['value'] = ParsedString(token_dict['quoted'], True) del token_dict['quoted'] elif 'value' in token_dict: token_dict['value'] = ParsedString(token_dict['value'], False) # Based on the token type build the corresponding query object if 'open_subgroup' in token_dict: self._open_subgroup() for subtoken in token: self._parse_token(subtoken) self._close_subgroup() elif 'bool' in token_dict: self._add_bool(token_dict['bool']) elif 'hosts' in token_dict: if isinstance(token_dict['hosts'], str): # Backward compatibility with PyParsing <2.3.1 token_dict['hosts'] = [token_dict['hosts']] token_dict['hosts'] = [nodeset(token_hosts) for token_hosts in token_dict['hosts']] self._add_hosts(**token_dict) elif 'category' in token_dict: self._add_category(**token_dict) else: # pragma: no cover - this should never happen raise InvalidQueryError( "No valid key found in token, one of bool|hosts|category expected: {token}".format(token=token_dict)) def _get_resource_query(self, key, value=None, operator='='): """Build a resource query based on the parameters, resolving the special cases for ``%params`` and ``@field``. Arguments: key (str): the key of the resource. value (str, optional): the value to match, if not specified the key itself will be matched. operator (str, optional): the comparison operator to use, one of :py:const:`OPERATORS`. Returns: str: the resource query. Raises: cumin.backends.InvalidQueryError: on invalid combinations of parameters. """ if all(char in key for char in ('%', '@')): raise InvalidQueryError(("Resource key cannot contain both '%' (query a resource's parameter) and '@' " "(query a resource's field)")) if '%' in key: # Querying a specific parameter of the resource key, param = key.split('%', 1) query_part = ', ["{op}", ["parameter", "{param}"], {value}]'.format(op=operator, param=param, value=value) elif '@' in key: # Querying a specific field of the resource key, field = key.split('@', 1) query_part = ', ["{op}", "{field}", {value}]'.format(op=operator, field=field, value=value) elif value is None: # Querying a specific resource type query_part = '' else: # Querying a specific resource title if key.lower() == 'class' and operator != '~': value = value.capwords('::') # Auto ucfirst the class title query_part = ', ["{op}", "title", {value}]'.format(op=operator, value=value) query = '["and", ["=", "type", "{type}"]{query_part}]'.format(type=capwords(key, '::'), query_part=query_part) return query def _get_special_resource_query(self, category, key, value, operator): """Build a query for Roles and Profiles, resolving the special cases for ``%params`` and ``@field``. Arguments: category (str): the category of the token, one of :py:data:`category_prefixes` keys. key (str): the key of the resource to use as a suffix for the Class title matching. value (str, optional): the value to match in case ``%params`` or ``@field`` is specified. operator (str, optional): the comparison operator to use if there is a value, one of :py:const:`OPERATORS`. Returns: str: the resource query. Raises: cumin.backends.InvalidQueryError: on invalid combinations of parameters. """ if all(char in key for char in ('%', '@')): raise InvalidQueryError(("Resource key cannot contain both '%' (query a resource's parameter) and '@' " "(query a resource's field)")) param = None if '%' in key: special = '%' key, param = key.split('%') elif '@' in key: special = '@' key, param = key.split('@') else: special = None if value is not None: raise InvalidQueryError(("Invalid query of the form '{category}:key = value'. The matching of a value " "is accepted only when using %param or @field.").format(category=category)) if self.category_prefixes[category]: title = ParsedString('{prefix}::{key}'.format(prefix=self.category_prefixes[category], key=key), True) else: title = ParsedString(key, True) query = self._get_resource_query('Class', title, '=') if special is not None: # pylint: disable-next=possibly-used-before-assignment param_query = self._get_resource_query(''.join(('Class', special, param)), value, operator) query = '["and", {query}, {param_query}]'.format(query=query, param_query=param_query) return query def _get_query_string(self, group): """Recursively build and return the PuppetDB query string. Arguments: group (dict): a dictionary with the grouped tokens. Returns: str: the query string for the PuppetDB API. """ if group['bool']: query = '["{bool}", '.format(bool=group['bool']) else: query = '' last_index = len(group['tokens']) for i, token in enumerate(group['tokens']): if isinstance(token, dict): query += self._get_query_string(group=token) else: query += token if i < last_index - 1: query += ', ' if group['bool']: query += ']' return query def _add_bool(self, bool_op): """Add a boolean AND or OR query block to the query and validate logic. Arguments: bool_op (str): the boolean operator to add to the query: ``and``, ``or``. Raises: cumin.backends.InvalidQueryError: if an invalid boolean operator was found. """ if self.current_group['bool'] is None: self.current_group['bool'] = bool_op elif self.current_group['bool'] == bool_op: return else: raise InvalidQueryError("Got unexpected '{bool}' boolean operator, current operator was '{current}'".format( bool=bool_op, current=self.current_group['bool'])) def _api_call(self, query): """Execute a query to PuppetDB API and return the parsed JSON. Arguments: query (str): the query parameter to send to the PuppetDB API. Raises: requests.HTTPError: if the PuppetDB API call fails. """ params = { 'verify': self.ssl_verify, 'timeout': self.timeout } if self.ssl_client_cert: if self.ssl_client_key: params['cert'] = (self.ssl_client_cert, self.ssl_client_key) else: params['cert'] = self.ssl_client_cert params['json'] = {'query': query} resources = requests.post(self.url + self.endpoint, **params) # nosec resources.raise_for_status() return resources.json() GRAMMAR_PREFIX = 'P' """:py:class:`str`: the prefix associate to this grammar, to register this backend into the general grammar. Required by the backend auto-loader in :py:meth:`cumin.grammar.get_registered_backends`.""" query_class = PuppetDBQuery # pylint: disable=invalid-name """Required by the backend auto-loader in :py:meth:`cumin.grammar.get_registered_backends`.""" wikimedia-cumin-36f957f/cumin/cli.py000066400000000000000000000503071476500461000174050ustar00rootroot00000000000000#!/usr/bin/python3 """Cumin CLI entry point.""" import argparse import code import json import logging import os import pkgutil import signal import sys from logging.handlers import RotatingFileHandler from tqdm import tqdm import cumin from cumin import backends, ensure_kerberos_ticket, query, transport, transports from cumin.color import Colored from cumin.transports.clustershell import TqdmQuietReporter logger = logging.getLogger(__name__) """logging.Logger: The logging instance.""" OUTPUT_FORMATS = ('txt', 'json') """tuple: A tuple with the possible output formats.""" OUTPUT_SEPARATOR = '_____FORMATTED_OUTPUT_____' """str: The output separator used when -o/--out is set.""" INTERACTIVE_BANNER = """===== Cumin Interactive REPL ===== # Press Ctrl+d or type exit() to exit the program. = Available variables = # hosts: the ClusterShell NodeSet of targeted hosts. # worker: the instance of the Transport worker that was used for the execution. # args: the parsed command line arguments, an argparse.Namespace instance. # config: the cofiguration dictionary. # exit_code: the return code of the execution, that will be used as exit code. = Useful functions = # worker.get_results(): generator that yields the tuple (nodes, output) for each grouped result, where: # - nodes: is a ClusterShell.NodeSet.NodeSet instance # - output: is a ClusterShell.MsgTree.MsgTreeElem instance # h(): print this help message. # help(object_name): Python default interactive help and documentation of the given object. = Example usage: for nodes, output in worker.get_results(): print(nodes) print(output.message().decode()) print('-----') """ """str: The message to print when entering the intractive REPL mode.""" class KeyboardInterruptError(cumin.CuminError): """Custom KeyboardInterrupt exception class for the SIGINT signal handler.""" def get_parser(): """Create and return the command line arguments parser. Returns: argparse.ArgumentParser: the parser object. """ sync_mode = 'sync' async_mode = 'async' # Get the list of existing backends and transports backends_names = [name for _, name, ispkg in pkgutil.iter_modules(backends.__path__) if not ispkg] transports_names = [name for _, name, ispkg in pkgutil.iter_modules(transports.__path__) if not ispkg] parser = argparse.ArgumentParser( prog='cumin', description='Cumin CLI - Automation and orchestration framework written in Python', epilog='More details at https://wikitech.wikimedia.org/wiki/Cumin') parser.add_argument('-c', '--config', default='/etc/cumin/config.yaml', help='configuration file. [default: /etc/cumin/config.yaml]') parser.add_argument('--global-timeout', type=int, help='Global timeout in seconds (int) for the whole execution. [default: None (unlimited)]') parser.add_argument('-t', '--timeout', type=int, help=('Timeout in seconds (int) for the the execution of every command in each host. ' '[default: None (unlimited)]')) parser.add_argument('-m', '--mode', choices=(sync_mode, async_mode), help=('Execution mode, required when there are multiple COMMANDS to be executed. In sync mode, ' 'execute the first command on all hosts, then proceed with the next one only if ' '-p/--success-percentage is reached. In async mode, execute on each host independently ' 'from each other, the list of commands, aborting the execution on any given host at the ' 'first command that fails.')) parser.add_argument('-p', '--success-percentage', type=int, choices=range(101), metavar='PCT', default=100, help=(('Percentage threshold to consider an execution unit successful. Required in sync mode, ' 'optional in async mode when -b/--batch-size is used. Accepted values are integers ' 'in the range 0-100. [default: 100]'))) parser.add_argument('-b', '--batch-size', type=target_batch_size, default={'value': None, 'ratio': None}, help=('The commands will be executed with a sliding batch of this size. The batch mode depends ' 'on the -m/--mode option when multiple commands are specified. In sync mode the first ' 'command is executed in batch to all hosts before proceeding with the next one. In async ' 'mode all commands are executed on the first batch of hosts, proceeding with the next ' 'hosts as soon as one host completes all the commands. The -p/--success-percentage is ' 'checked before starting the execution in each host. It accept an absolute integer ' '(i.e. 10) or a percentage (i.e. 50%%). [default: None (# of hosts)]')) parser.add_argument('-s', '--batch-sleep', type=float, help=('Sleep in seconds (float) to wait before starting the execution on the next host when ' '-b/--batch-size is used. [default: None]')) parser.add_argument('-x', '--ignore-exit-codes', action='store_true', help=('USE WITH CAUTION! Treat any executed command as successful, ignoring the exit codes. ' '[default: False]')) parser.add_argument('-o', '--output', choices=OUTPUT_FORMATS, help='Specify a different output format. [default: None]') parser.add_argument('-i', '--interactive', action='store_true', help='Drop into a Python shell with the results. [default: False]') parser.add_argument('-n', '--no-colors', action='store_true', help='Disable colored output. [default: False]') parser.add_argument('--force', action='store_true', help=('USE WITH CAUTION! Force the execution without confirmation of the affected hosts. ' '[default: False]')) parser.add_argument('--backend', help=('Override the default backend selected in the configuration file for this execution. The ' 'backend-specific configuration must be already present in the configuration file. ' 'One of [{backends}] or any external backend listed in the configuration file ' '[default: None]').format(backends=', '.join(backends_names))) parser.add_argument('--transport', choices=transports_names, help=('Override the default transport selected in the configuration file for this execution. ' 'The transport-specific configuration must already be present in the configuration file. ' '[default: None]')) parser.add_argument('--dry-run', action='store_true', help=('Do not execute any command, just return the list of matching hosts and exit. ' '[default: False]')) parser.add_argument('--no-progress', action='store_true', help='Do not show the progress bars during execution.') parser.add_argument('--version', action='version', version='%(prog)s {version}'.format(version=cumin.__version__)) parser.add_argument('-d', '--debug', action='store_true', help='Set log level to DEBUG. See also log_file in the configuration. [default: False]') parser.add_argument('--trace', action='store_true', help=('Set log level to TRACE, a custom logging level intended for development debugging. See ' 'also log_file in the configuration. [default: False]')) parser.add_argument('hosts', metavar='HOSTS_QUERY', help='Hosts selection query') parser.add_argument('commands', metavar='COMMAND', nargs='*', help='Command to be executed. If no commands are specified, --dry-run is set.') return parser def target_batch_size(string): """Validator for the --batch-size command line argument to be used as type in ArgumentParser. Arguments: string: the input string to be validated and parsed. Returns: dict: a dictionary with the batch size absolute value as integer (`value` key) or the ratio value as float (`ratio` key) to be used when instantiating a Target object. """ is_percentage = False orig = string if string[-1] == '%': is_percentage = True string = string[:-1] value = int(string) if is_percentage and not (0 <= value <= 100): # pylint: disable=superfluous-parens raise argparse.ArgumentTypeError( '{size} is not a valid percentage, expected in range 0%-100% or positive integer.'.format(size=orig)) if not is_percentage and value <= 0: raise argparse.ArgumentTypeError( '{size} is not a valid value, expected positive integer or percentage in range 0%-100%'.format(size=orig)) ret = {'value': None, 'ratio': None} if is_percentage: ret['ratio'] = value / 100 else: ret['value'] = value return ret def parse_args(argv): """Parse command line arguments, validate and return them. Arguments: argv: the list of command line arguments to use. Returns: argparse.Namespace: the parsed arguments. """ parser = get_parser() parsed_args = parser.parse_args(argv) if parsed_args.no_colors: Colored.disabled = True # Validation and default values num_commands = len(parsed_args.commands) if num_commands == 0: parsed_args.dry_run = True elif num_commands == 1: parsed_args.mode = 'sync' elif num_commands > 1: if parsed_args.mode is None: parser.error('-m/--mode is required when there are multiple COMMANDS') if parsed_args.interactive: parser.error('-i/--interactive can be used only with one command') if parsed_args.output is not None: parser.error('-o/--output can be used only with one command') if parsed_args.ignore_exit_codes: stderr('IGNORE EXIT CODES mode enabled, all commands executed will be considered successful') return parsed_args def get_running_user(): """Ensure that the original user is detected and return it.""" if os.getenv('USER') == 'root': if os.getenv('SUDO_USER') in (None, 'root'): raise cumin.CuminError('Unable to determine real user, logged in as root?') return os.getenv('SUDO_USER') return os.getenv('USER') def setup_logging(filename, debug=False, trace=False): """Setup the logger instance. Arguments: filename: the filename of the log file debug: whether to set logging level to DEBUG [optional, default: False] trace: whether to set logging level to TRACE [optional, default: False] """ file_path = os.path.dirname(filename) if file_path and not os.path.exists(file_path): os.makedirs(file_path, 0o770) log_formatter = logging.Formatter(fmt='%(asctime)s [%(levelname)s %(process)s %(name)s.%(funcName)s] %(message)s') log_handler = RotatingFileHandler(filename, maxBytes=(5 * (1024**2)), backupCount=30) log_handler.setFormatter(log_formatter) root_logger = logging.getLogger() root_logger.addHandler(log_handler) root_logger.raiseExceptions = False if trace: root_logger.setLevel(cumin.LOGGING_TRACE_LEVEL_NUMBER) elif debug: root_logger.setLevel(logging.DEBUG) else: root_logger.setLevel(logging.INFO) def sigint_handler(*args): # pylint: disable=unused-argument """Signal handler for Ctrl+c / SIGINT, raises KeyboardInterruptError. Arguments (as defined in https://docs.python.org/3/library/signal.html): signum: the signal number frame: the current stack frame """ # pylint: disable-next=no-member; https://github.com/prospector-dev/prospector/issues/677 if not sys.stdout.isatty(): logger.warning('Execution interrupted by Ctrl+c/SIGINT') raise KeyboardInterruptError # TODO: make the below code block to work as expected with ClusterShell # temporarily exit upon Ctrl+c also in interactive mode logger.warning('Execution interrupted by Ctrl+c/SIGINT') raise KeyboardInterruptError # logger.warning('Received Ctrl+c/SIGINT') # for i in xrange(10): # stderr('Ctrl+c pressed, sure to quit [y/n]?\n') # try: # answer = input('\n') # nosec # except RuntimeError: # # Can't re-enter readline when already waiting for input in get_hosts(). Assuming 'y' as answer # stderr('Ctrl+c pressed while waiting for answer. Aborting') # answer = 'y' # # if not answer: # continue # # if answer.lower() == 'y': # logger.warning('Execution interrupted by Ctrl+c/SIGINT') # raise KeyboardInterruptError # elif answer.lower() == 'n': # message = 'Ctrl+c/SIGINT aborted, resuming execution' # logger.warning(message) # stderr(message) # break # else: # logger.warning('Execution interrupted by Ctrl+c/SIGINT: got invalid answer for %d times', i) # raise KeyboardInterruptError def stderr(message, end='\n'): r"""Print a message to stderr and flush. Arguments: message: the message to print to sys.stderr end: the character to use at the end of the message. [optional, default: \n] """ tqdm.write(Colored.yellow(message), file=sys.stderr, end=end) def get_hosts(args, config): """Resolve the hosts selection into a list of hosts and return it. Raises KeyboardInterruptError. Arguments: args: ArgumentParser instance with parsed command line arguments config: a dictionary with the parsed configuration file """ hosts = query.Query(config).execute(args.hosts) if not hosts: stderr('No hosts found that matches the query') return hosts stderr('{num} hosts will be targeted:'.format(num=len(hosts))) # The list is sent to stdout or stderr based on the dry_run mode if args.dry_run: tqdm.write(Colored.cyan(cumin.nodeset_fromlist(hosts))) stderr('DRY-RUN mode enabled, aborting') return [] stderr(Colored.cyan(cumin.nodeset_fromlist(hosts))) if args.force: stderr('FORCE mode enabled, continuing without confirmation') return hosts if not sys.stdout.isatty(): message = 'Not in a TTY but neither DRY-RUN nor FORCE mode were specified.' stderr(message) raise cumin.CuminError(message) for i in range(10): stderr(('OK to proceed on {num} hosts? Enter the number of affected hosts to confirm ' 'or "q" to quit:'.format(num=len(hosts))), end=' ') answer = input() # nosec if not answer: continue if answer == str(len(hosts)): break if answer in 'qQ': raise KeyboardInterruptError else: stderr('Got invalid answer for {i} times'.format(i=i)) raise KeyboardInterruptError return hosts def print_output(output_format, worker): """Print the execution results in a specific format. Arguments: output_format: the output format to use, one of: 'txt', 'json'. worker: the Transport worker instance to retrieve the results from. """ if output_format not in OUTPUT_FORMATS: raise cumin.CuminError("Got invalid output format '{fmt}', expected one of {allowed}".format( fmt=output_format, allowed=OUTPUT_FORMATS)) out = {} for nodeset, output in worker.get_results(): for node in nodeset: if output_format == 'txt': out[node] = '\n'.join('{node}: {line}'.format(node=node, line=line.decode()) for line in output.lines()) elif output_format == 'json': out[node] = output.message().decode() if output_format == 'txt': for node in sorted(out.keys()): tqdm.write(out[node]) elif output_format == 'json': tqdm.write(json.dumps(out, indent=4, sort_keys=True)) def run(args, config): """Execute the commands on the selected hosts and print the results. Arguments: args: ArgumentParser instance with parsed command line arguments config: a dictionary with the parsed configuration file """ hosts = get_hosts(args, config) if not hosts: return 0 ensure_kerberos_ticket(config) target = transports.Target(hosts, batch_size=args.batch_size['value'], batch_size_ratio=args.batch_size['ratio'], batch_sleep=args.batch_sleep) worker = transport.Transport.new(config, target) ok_codes = None if args.ignore_exit_codes: ok_codes = [] worker.commands = [transports.Command(command, timeout=args.timeout, ok_codes=ok_codes) for command in args.commands] worker.timeout = args.global_timeout worker.handler = args.mode worker.progress_bars = not args.no_progress if args.output is not None: # TODO: set the reporter to tqdm when releasing v5.0.0 worker.reporter = TqdmQuietReporter worker.success_threshold = args.success_percentage / 100 exit_code = worker.execute() if args.interactive: # Define a help function h() that will be available in the interactive shell to print the help message. # The name is to not shadow the Python built-in help() that might be usefult too to inspect objects. def h(): # pylint: disable=possibly-unused-variable,invalid-name """Print the help message in interactive shell.""" tqdm.write(INTERACTIVE_BANNER) code.interact(banner=INTERACTIVE_BANNER, local=locals()) elif args.output is not None: tqdm.write(OUTPUT_SEPARATOR) # TODO: to be removed when releasing v5.0.0 print_output(args.output, worker) return exit_code def validate_config(config): """Perform validation of mandatory keys in the configuration. Arguments: config (dict): the loaded configuration to validate. Raises: cumin.CuminError: if a mandatory configuration key is missing. """ if 'log_file' not in config: raise cumin.CuminError("Missing required parameter 'log_file' in the configuration file '{config}'".format( config=config)) def main(argv=None): # noqa: MC0001 """CLI entry point. Execute commands on hosts according to arguments. Arguments: argv: the list of command line arguments to use. If not specified it will be automatically taken from sys.argv [optional, default: None] """ if argv is None: argv = sys.argv[1:] signal.signal(signal.SIGINT, sigint_handler) # Setup try: args = parse_args(argv) user = get_running_user() config = cumin.Config(args.config) validate_config(config) setup_logging(os.path.expanduser(config['log_file']), debug=args.debug, trace=args.trace) except cumin.CuminError as e: stderr(e) return 2 except Exception as e: # pylint: disable=broad-except stderr('Caught {name} exception: {msg}'.format(name=e.__class__.__name__, msg=e)) return 3 # Override config with command line arguments if args.backend is not None: config['default_backend'] = args.backend if args.transport is not None: config['transport'] = args.transport logger.info("Cumin called by user '%s' with args: %s", user, args) # Execution try: exit_code = run(args, config) except KeyboardInterruptError: stderr('Execution interrupted by Ctrl+c/SIGINT/Aborted') exit_code = 98 except Exception as e: # pylint: disable=broad-except stderr('Caught {name} exception: {msg}'.format(name=e.__class__.__name__, msg=e)) if args.trace: logger.addHandler(logging.lastResort) # Add a stream handler to log to stderr the exception logger.exception('Failed to execute') exit_code = 99 return exit_code if __name__ == '__main__': sys.exit(main()) wikimedia-cumin-36f957f/cumin/color.py000066400000000000000000000037601476500461000177550ustar00rootroot00000000000000"""Colors module.""" from abc import ABCMeta class ColoredType(ABCMeta): """Metaclass to define a new type that dynamically adds static methods to its classes.""" COLORS = { 'red': 31, 'green': 32, 'yellow': 33, 'blue': 34, 'cyan': 36, } """:py:class:`dict`: a mapping of colors to the ANSI foreground color code.""" def __getattr__(cls, name): # noqa: N805 (Prospector reqires an older version of pep8-naming) """Dynamically check access to members of classes of this type. :Parameters: according to Python's Data model :py:meth:`object.__getattr__`. """ color_code = ColoredType.COLORS.get(name, None) if color_code is None: raise AttributeError("'{cls}' object has no attribute '{attr}'".format(cls=cls.__name__, attr=name)) return lambda obj: cls._color(color_code, obj) class Colored(metaclass=ColoredType): """Class to manage colored output. Available methods are dynamically added based on the keys of the :py:const:`ColoredType.COLORS` dictionary. For each color a method with the color name is available to color any object with that specific color code. Examples:: Colored.green(object) """ disabled = False """:py:class:`bool`: switch to globally control the coloring. Set it to :py:const`True` to disable all coloring.""" @staticmethod def _color(color_code, obj): """Color the given object, unless coloring is globally disabled. Arguments: color_code (int): a valid ANSI escape sequence color code. obj (mixed): the object to color. Return: str: the string representation of the object encapsulated in the red ANSI escape sequence. """ message = str(obj) if not message: return '' if Colored.disabled: return message return '\x1b[{code}m{message}\x1b[39m'.format(code=color_code, message=message) wikimedia-cumin-36f957f/cumin/grammar.py000066400000000000000000000137401476500461000202640ustar00rootroot00000000000000"""Query grammar definition.""" import importlib import pkgutil from collections import namedtuple import pyparsing as pp from cumin import backends, CuminError INTERNAL_BACKEND_PREFIX = 'cumin.backends' """:py:class:`str` with the prefix for built-in backends.""" Backend = namedtuple('Backend', ['keyword', 'name', 'cls']) """:py:func:`collections.namedtuple` that define a Backend object. Keyword Arguments: keyword (str): The backend keyword to be used in the grammar. name (str): The backend name. cls (BaseQuery): The backend class object. """ def get_registered_backends(external=()): """Get a mapping of all the registered backends with their keyword. Arguments: external (list, tuple, optional): external backend modules to register. Returns: dict: A dictionary with a ``{keyword: Backend object}`` mapping for each available backend. Raises: cumin.CuminError: If unable to register a backend. """ available_backends = {} backend_names = ['{prefix}.{backend}'.format(prefix=INTERNAL_BACKEND_PREFIX, backend=name) for _, name, ispkg in pkgutil.iter_modules(backends.__path__) if not ispkg] for name in backend_names + list(external): keyword, backend = _import_backend(name, available_backends) if keyword is not None and backend is not None: available_backends[keyword] = backend return available_backends def grammar(backend_keys): """Define the main multi-query grammar. Cumin provides a user-friendly generic query language that allows to combine the results of subqueries for multiple backends: * Each query part can be composed with the others using boolean operators ``and``, ``or``, ``and not``, ``xor``. * Multiple query parts can be grouped together with parentheses ``(``, ``)``. * Specific backend query ``I{backend-specific query syntax}``, where ``I`` is an identifier for the specific backend. * Alias replacement, according to aliases defined in the configuration file ``A:group1``. * The identifier ``A`` is reserved for the aliases replacement and cannot be used to identify a backend. * A complex query example: ``(D{host1 or host2} and (P{R:Class = Role::MyClass} and not A:group1)) or D{host3}`` Backus-Naur form (BNF) of the grammar:: ::= | ::= | | "(" ")" ::= "{" "}" ::= A: ::= "and not" | "and" | "xor" | "or" Given that the pyparsing library defines the grammar in a BNF-like style, for the details of the tokens not specified above check directly the source code. Arguments: backend_keys (list): list of the GRAMMAR_PREFIX for each registered backend. Returns: pyparsing.ParserElement: the grammar parser. """ # Boolean operators boolean = (pp.CaselessKeyword('and not') | pp.CaselessKeyword('and') | pp.CaselessKeyword('xor') | pp.CaselessKeyword('or'))('bool') # Parentheses lpar = pp.Literal('(')('open_subgroup') rpar = pp.Literal(')')('close_subgroup') # Backend query: P{PuppetDB specific query} query_start = pp.Combine(pp.oneOf(backend_keys, caseless=True)('backend') + pp.Literal('{')) query_end = pp.Literal('}') # Allow the backend specific query to use the end_query token as well, as long as it's in a quoted string # and fail if there is a query_start token before the first query_end is reached query = pp.SkipTo(query_end, ignore=pp.quotedString, failOn=query_start)('query') backend_query = pp.Combine(query_start + query + query_end) # Alias alias = pp.Combine(pp.CaselessKeyword('A') + ':' + pp.Word(pp.alphanums + '-_.+')('alias')) # Final grammar, see the docstring for its BNF based on the tokens defined above # Group are used to have an easy dictionary access to the parsed results full_grammar = pp.Forward() item = backend_query | alias | lpar + full_grammar + rpar full_grammar << pp.Group(item) + pp.ZeroOrMore(pp.Group(boolean + item)) # pylint: disable=expression-not-assigned return full_grammar def _import_backend(module, available_backends): """Dynamically import a backend for Cumin and validate it. Arguments: module (str): the full module name of the backend to register. Must be importable from Python ``PATH``. available_backends (dict): dictionary with a ``{keyword: Backend object}`` mapping for all registered backends. Returns: tuple: with two elements: ``(keyword, Backend object)`` of the imported backend. """ try: backend = importlib.import_module(module) except ImportError as e: if not module.startswith(INTERNAL_BACKEND_PREFIX): raise CuminError("Unable to import backend '{module}': {e}".format(module=module, e=e)) from e return (None, None) # Internal backend not available, are all the dependencies installed? name = module.split('.')[-1] message = "Unable to register backend '{name}' in module '{module}'".format(name=name, module=module) try: keyword = backend.GRAMMAR_PREFIX except AttributeError as e: raise CuminError('{message}: GRAMMAR_PREFIX module attribute not found'.format(message=message)) from e if keyword in available_backends: raise CuminError(("{message}: keyword '{key}' already registered: {backends}").format( message=message, key=keyword, backends=available_backends)) try: class_obj = backend.query_class except AttributeError as e: raise CuminError('{message}: query_class module attribute not found'.format(message=message)) from e if not issubclass(class_obj, backends.BaseQuery): raise CuminError('{message}: query_class module attribute is not a subclass of cumin.backends.BaseQuery') return (keyword, Backend(name=name, keyword=keyword, cls=class_obj)) wikimedia-cumin-36f957f/cumin/query.py000066400000000000000000000154051476500461000200030ustar00rootroot00000000000000"""Query handling: factory and builder.""" from pyparsing import ParseException, ParseResults from cumin import grammar from cumin.backends import BaseQuery, BaseQueryAggregator, InvalidQueryError class Query(BaseQueryAggregator): """Cumin main query class. It has multi-query capability and allow to use a default backend, if set, without additional syntax. If a ``default_backend`` is set in the configuration, it will try to execute the query string first with the default backend and only if the query is not parsable with that backend it will try to execute it with the multi-query grammar. When a query is executed, a :py:class:`ClusterShell.NodeSet.NodeSet` with the FQDN of the matched hosts is returned. Examples: >>> import cumin >>> from cumin.query import Query >>> config = cumin.Config() >>> hosts = Query(config).execute(query_string) """ def __init__(self, config): """Query constructor, initialize the registered backends. :Parameters: according to parent :py:meth:`cumin.backends.BaseQueryAggregator.__init__`. """ super().__init__(config) external = self.config.get('plugins', {}).get('backends', []) self.registered_backends = grammar.get_registered_backends(external=external) self.grammar = grammar.grammar(self.registered_backends.keys()) def execute(self, query_string): """Override parent class execute method to implement the multi-query capability. :Parameters: according to parent :py:meth:`cumin.backends.BaseQueryAggregator.execute`. Returns: ClusterShell.NodeSet.NodeSet: with the FQDNs of the matching hosts. Raises: cumin.backends.InvalidQueryError: if unable to parse the query. """ if 'default_backend' not in self.config: try: # No default backend set, using directly the global grammar return super().execute(query_string) except ParseException as e: raise InvalidQueryError( ("Unable to parse the query '{query}' with the global grammar and no " "default backend is set:\n{error}").format(query=query_string, error=e)) from e try: # Default backend set, trying it first hosts = self._query_default_backend(query_string) except ParseException as e_default: try: # Trying global grammar as a fallback hosts = super().execute(query_string) except ParseException as e_global: raise InvalidQueryError( ("Unable to parse the query '{query}' neither with the default backend '{name}' nor with the " "global grammar:\n{name}: {e_def}\nglobal: {e_glob}").format( query=query_string, name=self.config['default_backend'], e_def=e_default, e_glob=e_global) ) from e_global return hosts def _query_default_backend(self, query_string): """Execute the query with the default backend, according to the configuration. Arguments: query_string (str): the query string to be parsed and executed with the default backend. Returns: ClusterShell.NodeSet.NodeSet: with the FQDNs of the matching hosts. Raises: cumin.backends.InvalidQueryError: if unable to get the default backend from the registered backends. """ for registered_backend in self.registered_backends.values(): if registered_backend.name == self.config['default_backend']: backend = registered_backend break else: raise InvalidQueryError("Default backend '{name}' is not registered: {backends}".format( name=self.config['default_backend'], backends=self.registered_backends)) query = backend.cls(self.config) return query.execute(query_string) def _parse_token(self, token): """Concrete implementation of parent abstract method. :Parameters: according to parent :py:meth:`cumin.backends.BaseQueryAggregator._parse_token`. Raises: cumin.backends.InvalidQueryError: on internal parsing error. """ if not isinstance(token, ParseResults): # pragma: no cover - this should never happen raise InvalidQueryError('Expecting ParseResults object, got {type}: {token}'.format( type=type(token), token=token)) token_dict = token.asDict() self.logger.trace('Token is: %s', token_dict) if self._replace_alias(token_dict): return # This token was an alias and got replaced if 'backend' in token_dict and 'query' in token_dict: element = self._get_stack_element() query = self.registered_backends[token_dict['backend']].cls(self.config) element['hosts'] = query.execute(token_dict['query']) if 'bool' in token_dict: element['bool'] = token_dict['bool'] self.stack_pointer['children'].append(element) elif 'open_subgroup' in token_dict and 'close_subgroup' in token_dict: self._open_subgroup() if 'bool' in token_dict: self.stack_pointer['bool'] = token_dict['bool'] for subtoken in token: if isinstance(subtoken, str): continue self._parse_token(subtoken) self._close_subgroup() else: # pragma: no cover - this should never happen raise InvalidQueryError('Got unexpected token: {token}'.format(token=token)) def _replace_alias(self, token_dict): """Replace any alias in the query in a recursive way, alias can reference other aliases. Arguments: token_dict (dict): the dictionary of the parsed token returned by the grammar parsing. Returns: bool: :py:data:`True` if a replacement was made, :py:data`False` otherwise. Raises: cumin.backends.InvalidQueryError: if unable to replace an alias. """ if 'alias' not in token_dict: return False alias_name = token_dict['alias'] if alias_name not in self.config.get('aliases', {}): raise InvalidQueryError("Unable to find alias replacement for '{alias}' in the configuration".format( alias=alias_name)) self._open_subgroup() if 'bool' in token_dict: self.stack_pointer['bool'] = token_dict['bool'] # Calling BaseQuery._build() directly and not the parent's one to avoid resetting the stack BaseQuery._build(self, self.config['aliases'][alias_name]) # pylint: disable=protected-access self._close_subgroup() return True wikimedia-cumin-36f957f/cumin/tests/000077500000000000000000000000001476500461000174215ustar00rootroot00000000000000wikimedia-cumin-36f957f/cumin/tests/__init__.py000066400000000000000000000015501476500461000215330ustar00rootroot00000000000000"""Tests utils.""" import logging import os logging.basicConfig(level=logging.DEBUG) _TESTS_BASE_PATH = os.path.realpath(os.path.dirname(__file__)) def get_fixture(path, as_string=False): """Return the content of a fixture file. Arguments: path: the relative path to the test's fixture directory to be opened. as_string: return the content as a multiline string instead of a list of lines [optional, default: False] """ with open(get_fixture_path(path), encoding='utf8') as f: if as_string: content = f.read() else: content = f.readlines() return content def get_fixture_path(path): """Return the absolute path of the given fixture. Arguments: path: the relative path to the test's fixture directory. """ return os.path.join(_TESTS_BASE_PATH, 'fixtures', path) wikimedia-cumin-36f957f/cumin/tests/fixtures/000077500000000000000000000000001476500461000212725ustar00rootroot00000000000000wikimedia-cumin-36f957f/cumin/tests/fixtures/backends/000077500000000000000000000000001476500461000230445ustar00rootroot00000000000000wikimedia-cumin-36f957f/cumin/tests/fixtures/backends/grammars/000077500000000000000000000000001476500461000246555ustar00rootroot00000000000000wikimedia-cumin-36f957f/cumin/tests/fixtures/backends/grammars/direct_invalid.txt000066400000000000000000000001561476500461000304000ustar00rootroot00000000000000# Invalid grammars some+host not host1 host1 or not host2 host1 and (not host2) Z:category Z:category = value wikimedia-cumin-36f957f/cumin/tests/fixtures/backends/grammars/direct_valid.txt000066400000000000000000000003701476500461000300470ustar00rootroot00000000000000# Valid grammars hostname host-name host_name.domain hostname and host_name.domain.tld host1 or host2 host1 and host2 host1 and not host2 host10[10-20,30-50] (hostname.domain.tld) (host1 or host2) and host1 ((host1[0-9] or host01) and host[01-10]) wikimedia-cumin-36f957f/cumin/tests/fixtures/backends/grammars/knownhosts_invalid.txt000066400000000000000000000001561476500461000313430ustar00rootroot00000000000000# Invalid grammars some+host not host1 host1 or not host2 host1 and (not host2) Z:category Z:category = value wikimedia-cumin-36f957f/cumin/tests/fixtures/backends/grammars/knownhosts_valid.txt000066400000000000000000000004421476500461000310120ustar00rootroot00000000000000# Valid grammars hostname host-name host_name.domain hostname and host_name.domain.tld host1 or host2 host1 and host2 host1 and not host2 host10[10-20,30-50] host10[10-20,30-50]* host?0[10-20,30-50]* (hostname.domain.tld) (host1 or host2) and host1 ((host1[0-9] or host01) and host[01-10]) wikimedia-cumin-36f957f/cumin/tests/fixtures/backends/grammars/openstack_invalid.txt000066400000000000000000000002031476500461000311060ustar00rootroot00000000000000# Invalid grammars * key:value key:value key key:value :value "key":value key%:value A:value a:value KEY:value Key:value kEy:value wikimedia-cumin-36f957f/cumin/tests/fixtures/backends/grammars/openstack_valid.txt000066400000000000000000000003351476500461000305650ustar00rootroot00000000000000# Valid grammars * name:host1 name:host10.* name:"^host10[1-9]\.domain$" name:'^host10[1-9]\.domain$' project:project_name project:"Project Name" project:project_name name:host1 project:"Project Name" name:host1 ip:value wikimedia-cumin-36f957f/cumin/tests/fixtures/backends/grammars/puppetdb_invalid.txt000066400000000000000000000007641476500461000307560ustar00rootroot00000000000000# Invalid grammars F:key1 = {(value} F:key1 != value some+host F:a.b_c-$3 >= 3.5-c F:a.b_c-3 = a"(complex string 3.5-c with symbols = > and parentheses)" F:key1 < 5 hostname F:key1 < 5 or and hostname and F:key2 = value not F:key1 = value not not F:key1 < 5 or not and hostname and F:key2 = value (F:key1 = value ()) (F:key1 < 5 and () or hostname) (F:key1 < 5 and (hostname not F:key2 = value)) F:key1 = value or (F:key2 < 5 and (hostname or F:key3 = (value)) and F:key4) = value Z:invalid_category wikimedia-cumin-36f957f/cumin/tests/fixtures/backends/grammars/puppetdb_valid.txt000066400000000000000000000024201476500461000304160ustar00rootroot00000000000000# Valid grammars * F:key1 = value not F:key1 = value hostname host-name host_name.domain not hostname hostname and host_name.domain.tld host1 or host2 host1 and host2 host1 and not host2 host1 or not host2 host01* host10[10-20,30-50] F:a.b_c-3 >= 3.5-c F:a.b_c-3 ~ "(complex string 3.5-c with symbols = > and parentheses)" F:key1 < 5 and hostname R:key1 R:key1%param = 'some value' R:key1@tag ~ 'some [r]egex' F:key1 < 5 or hostname and F:key2 = value not F:key1 < 5 or not hostname and F:key2 = value (F:key1 = value) (hostname.domain.tld) (F:key1 < 5 and hostname) (F:key1 < 5 and (hostname or F:key2 = value)) F:key1 = value or (F:key2 < 5 and (hostname or F:key3 = value)) (F:key2 < 5 and (hostname or F:key3 = value)) and F:key4 = value F:key1 = value or (F:key2 < 5 and (hostname or F:key3 = value)) and F:key4 = value not F:key1 = value or not ( not F:key2 < 5 and not (not hostname or not F:key3 = value)) and not F:key4 = value not F:key1 or not ( not F:key2 < 5 and not (not hostname or not F:key3)) and not F:key4 = value (F:key1 = value and F:key2 = "another value") or (F:key3 = Value and F:key4 > 8.1) O:Module::class O:Module::class%param = 'some value' O:Module::class@tag ~ 'some [r]egex' P:Module::class P:Module::class%param = 'some value' P:Module::class@tag ~ 'some [r]egex' wikimedia-cumin-36f957f/cumin/tests/fixtures/backends/knownhosts.txt000066400000000000000000000033041476500461000260220ustar00rootroot00000000000000# This is a comment and should be ignored, like empty lines # Hostname only host1.domain ecdsa-sha2-nistp256 AAAA...= # IPv4 only 127.0.1.2 ecdsa-sha2-nistp256 AAAA...= # IPv6 only fe80::3 ecdsa-sha2-nistp256 AAAA...= # Hostname and IPv4 host4.domain,127.0.1.7 ecdsa-sha2-nistp256 AAAA...= # Hostname and IPv6 host5.domain,fe80::9 ecdsa-sha2-nistp256 AAAA...= # IPv4 and IPv6 127.0.1.6,fe80::11 ecdsa-sha2-nistp256 AAAA...= # Hostname, IPv4 and IPv6 host7.domain,127.0.1.13,fe80::13 ecdsa-sha2-nistp256 AAAA...= # CA marker @cert-authority host8.domain ssh-rsa AAAA...= # Revoked marker @revoked host9.domain ssh-rsa AAAA...= # Hashed line |1|HaSh=|HaSh= ecdsa-sha2-nistp256 AAAA...= # Not enough fields host10.domain ssh-rsa # Not enough fields with marker @cert-authority host11.domain ssh-rsa # Unknown marker @marker host12.domain ssh-rsa AAAA...= # Patterns only *.domain ecdsa-sha2-nistp256 AAAA...= host?.domain ecdsa-sha2-nistp256 AAAA...= # Hostname and pattern host13.domain,*.otherdomain ecdsa-sha2-nistp256 AAAA...= *.otherdomain,host14.domain ecdsa-sha2-nistp256 AAAA...= # IPv4 and pattern 127.0.1.2,*.otherdomain ecdsa-sha2-nistp256 AAAA...= *.otherdomain,127.0.1.2 ecdsa-sha2-nistp256 AAAA...= # IPv6 and pattern fe80::3,*.otherdomain ecdsa-sha2-nistp256 AAAA...= *.otherdomain,fe80::3 ecdsa-sha2-nistp256 AAAA...= # Hostname, IPv4 and pattern host4.domain,*.otherdomain,127.0.1.7 ecdsa-sha2-nistp256 AAAA...= # Hostname, IPv6 and pattern host5.domain,*.otherdomain,fe80::9 ecdsa-sha2-nistp256 AAAA...= # IPv4, IPv6 and pattern 127.0.1.6,*.otherdomain,fe80::11 ecdsa-sha2-nistp256 AAAA...= # Hostname, IPv4, IPv6 and pattern host7.domain,127.0.1.13,*.otherdomain,fe80::13 ecdsa-sha2-nistp256 AAAA...= invalid line wikimedia-cumin-36f957f/cumin/tests/fixtures/backends/knownhosts_man.txt000066400000000000000000000006551476500461000266630ustar00rootroot00000000000000# Comments allowed at start of line closenet,192.0.2.53 1024 37 159...93 closenet.example.net cvs.example.net,192.0.2.10 ssh-rsa AAAA1234.....= # A hashed hostname |1|JfKTdBh7rNbXkVAQCRp4OQoPfmI=|USECr3SWf1JUPsms5AqfD5QfxkM= ssh-rsa AAAA1234.....= # A revoked key @revoked * ssh-rsa AAAAB5W... # A CA key, accepted for any host in *.mydomain.com or *.mydomain.org @cert-authority *.mydomain.org,*.mydomain.com ssh-rsa AAAAB5W... wikimedia-cumin-36f957f/cumin/tests/fixtures/config/000077500000000000000000000000001476500461000225375ustar00rootroot00000000000000wikimedia-cumin-36f957f/cumin/tests/fixtures/config/empty/000077500000000000000000000000001476500461000236755ustar00rootroot00000000000000wikimedia-cumin-36f957f/cumin/tests/fixtures/config/empty/config.yaml000066400000000000000000000000001476500461000260140ustar00rootroot00000000000000wikimedia-cumin-36f957f/cumin/tests/fixtures/config/invalid/000077500000000000000000000000001476500461000241655ustar00rootroot00000000000000wikimedia-cumin-36f957f/cumin/tests/fixtures/config/invalid/config.yaml000066400000000000000000000000511476500461000263120ustar00rootroot00000000000000key: other_key: value - invalid_item wikimedia-cumin-36f957f/cumin/tests/fixtures/config/valid/000077500000000000000000000000001476500461000236365ustar00rootroot00000000000000wikimedia-cumin-36f957f/cumin/tests/fixtures/config/valid/config.yaml000066400000000000000000000003561476500461000257730ustar00rootroot00000000000000transport: clustershell log_file: logs/cumin.log default_backend: puppetdb environment: ENV_VARIABLE: env_value puppetdb: host: puppetdb.local port: 443 clustershell: ssh_options: - 'some_option' fanout: 16 wikimedia-cumin-36f957f/cumin/tests/fixtures/config/valid_with_aliases/000077500000000000000000000000001476500461000263725ustar00rootroot00000000000000wikimedia-cumin-36f957f/cumin/tests/fixtures/config/valid_with_aliases/aliases.yaml000066400000000000000000000003211476500461000306730ustar00rootroot00000000000000group1: D{host10[10-22].example.org} group2: D{host20[10-22].example.org} group_all_direct: A:group1 OR A:group2 role1: P{R:Class = Role::Role1} dc1: P{*.dc1.example.org} group_all_puppetdb: A:role1 AND A:dc1 wikimedia-cumin-36f957f/cumin/tests/fixtures/config/valid_with_aliases/config.yaml000077700000000000000000000000001476500461000341612../valid/config.yamlustar00rootroot00000000000000wikimedia-cumin-36f957f/cumin/tests/fixtures/config/valid_with_empty_aliases/000077500000000000000000000000001476500461000276105ustar00rootroot00000000000000wikimedia-cumin-36f957f/cumin/tests/fixtures/config/valid_with_empty_aliases/aliases.yaml000066400000000000000000000000251476500461000321120ustar00rootroot00000000000000# Emtpy aliases file wikimedia-cumin-36f957f/cumin/tests/fixtures/config/valid_with_empty_aliases/config.yaml000077700000000000000000000000001476500461000353772../valid/config.yamlustar00rootroot00000000000000wikimedia-cumin-36f957f/cumin/tests/fixtures/config/valid_with_invalid_aliases/000077500000000000000000000000001476500461000301005ustar00rootroot00000000000000wikimedia-cumin-36f957f/cumin/tests/fixtures/config/valid_with_invalid_aliases/aliases.yaml000066400000000000000000000000331476500461000324010ustar00rootroot00000000000000name: query - invalid_name wikimedia-cumin-36f957f/cumin/tests/fixtures/config/valid_with_invalid_aliases/config.yaml000077700000000000000000000000001476500461000356672../valid/config.yamlustar00rootroot00000000000000wikimedia-cumin-36f957f/cumin/tests/fixtures/grammar/000077500000000000000000000000001476500461000227205ustar00rootroot00000000000000wikimedia-cumin-36f957f/cumin/tests/fixtures/grammar/invalid_grammars.txt000066400000000000000000000005361476500461000270040ustar00rootroot00000000000000# Invalid grammars A:alias/name P{puppetdb query with unquoted P{} P{puppetdb query with unquoted }} P{query1} or not D{query2} P{query} or not A:alias A:alias or not P{query} A:alias1 and (not P{query1} or D{query1}) and not A:alias2 A:alias1 and ((P{query1} or (D{query1} or) not A:alias2) A:alias1 and ((P{query1} or (D{query1} or) not A:alias2)) wikimedia-cumin-36f957f/cumin/tests/fixtures/grammar/valid_grammars.txt000066400000000000000000000011231476500461000264460ustar00rootroot00000000000000# Valid grammars A:alias_name A:alias-name A:alias.name A:alias+name P{puppetdb query (with subgroups) "and even P{}}, if quoted"} D{direct (query) with and or ((subgroups[10-20]))} P{query1} and D{query2} P{query1} and not D{query2} P{query1} or D{query1} P{query1} xor D{query1} P{query} and A:alias P{query} and not A:alias P{query} or A:alias P{query} xor A:alias A:alias and P{query} A:alias and not P{query} A:alias or P{query} A:alias xor P{query} A:alias1 and (P{query1} or D{query2}) and not A:alias2 xor D{query3} A:alias1 and ((P{query1} or D{query2}) and not A:alias2) xor D{query3} wikimedia-cumin-36f957f/cumin/tests/integration/000077500000000000000000000000001476500461000217445ustar00rootroot00000000000000wikimedia-cumin-36f957f/cumin/tests/integration/__init__.py000066400000000000000000000000311476500461000240470ustar00rootroot00000000000000"""Integration tests.""" wikimedia-cumin-36f957f/cumin/tests/integration/backends/000077500000000000000000000000001476500461000235165ustar00rootroot00000000000000wikimedia-cumin-36f957f/cumin/tests/integration/backends/__init__.py000066400000000000000000000000521476500461000256240ustar00rootroot00000000000000"""Backend specific integration tests.""" wikimedia-cumin-36f957f/cumin/tests/integration/conftest.py000066400000000000000000000007171476500461000241500ustar00rootroot00000000000000"""Pytest customization for integration tests.""" import pytest @pytest.hookimpl(hookwrapper=True) def pytest_runtest_makereport(item, call): # pylint: disable=unused-argument """If a custom variant_params marker is set, print a section with its content.""" outcome = yield marker = item.get_closest_marker('variant_params') if marker: rep = outcome.get_result() rep.sections.insert(0, ('test_variant parameters', marker.args)) wikimedia-cumin-36f957f/cumin/tests/integration/docker.sh000077500000000000000000000026311476500461000235540ustar00rootroot00000000000000#!/bin/bash set -e if ! which docker &> /dev/null; then echo "docker executable not found. Aborting!" exit 1 fi if [[ -z "${1}" ]]; then echo "Missing required positional argument ENV_NAME, the name of the environment to run the test on" exit 1 fi ENV_NAME="${1}" SKIP_DELETION=0 if [[ -n "${2}" && "${2}" -eq "1" ]]; then SKIP_DELETION=1 fi ENV_FILE="$(dirname "${0}")/${ENV_NAME}.sh" if [[ ! -f "${ENV_FILE}" ]]; then echo "Environment file '${ENV_FILE}' not found. Aborting!" exit 1 fi function _log() { echo "$(date +"%F %T") | ${*}" } # shellcheck source=/dev/null source "${ENV_FILE}" # The sourced ENV_FILE must register any docker instance in this variable for cleanup DOCKER_INSTANCES="" function exit_trap() { if [[ "${SKIP_DELETION}" -eq "1" ]]; then _log "Skip deletion set: docker instances and temporary directory were not removed" return fi _log "Removing docker instances" docker rm -f ${DOCKER_INSTANCES} > /dev/null if [[ -n "${CUMIN_TMPDIR}" ]]; then _log "Cleaning TMPDIR: ${CUMIN_TMPDIR}" rm -rf "${CUMIN_TMPDIR}" fi } export CUMIN_TMPDIR export CUMIN_IDENTIFIER CUMIN_TMPDIR="$(mktemp -d /tmp/cumin-XXXXXX)" _log "Temporary directory is: ${CUMIN_TMPDIR}" CUMIN_IDENTIFIER="$(basename "${CUMIN_TMPDIR}")" _log "Unique identifier is ${CUMIN_IDENTIFIER}" trap 'exit_trap' EXIT setup sleep 3 run_tests exit "${?}" wikimedia-cumin-36f957f/cumin/tests/integration/test_cli.py000066400000000000000000000523211476500461000241270ustar00rootroot00000000000000"""CLI integration tests.""" import copy import json import os import re import sys import pytest from cumin import __version__, cli # Dictionary with expected strings to match in the execution stderr: # {label: string_to_match} _EXPECTED_LINES = { 'all_targeted': '5 hosts will be targeted', 'failed': 'failed', 'global_timeout': 'global timeout', 'successfully': 'successfully', 'dry_run': 'DRY-RUN mode enabled, aborting', 'subfanout_targeted': '2 hosts will be targeted', 'ls_success': "100.0% (5/5) success ratio (>= 100.0% threshold) for command: 'ls -la /tmp'.", 'ls_success_threshold': "100.0% (5/5) success ratio (>= 50.0% threshold) for command: 'ls -la /tmp'.", 'ls_partial_success': "/5) of nodes failed to execute command 'ls -la /tmp/maybe'", 'ls_partial_success_ratio_re': r"[4-6]0\.0% \([2-3]/5\) success ratio \(< 100\.0% threshold\) for command: 'ls -la /tmp/maybe'\. Aborting.", 'ls_partial_success_threshold_ratio': "60.0% (3/5) success ratio (>= 50.0% threshold) for command: 'ls -la /tmp/maybe'.", 'ls_failure_batch': "40.0% (2/5) of nodes failed to execute command 'ls -la /tmp/non_existing'", 'ls_failure_batch_threshold': "80.0% (4/5) of nodes failed to execute command 'ls -la /tmp/non_existing'", 'ls_total_failure': "100.0% (5/5) of nodes failed to execute command 'ls -la /tmp/non_existing'", 'ls_total_failure_threshold_ratio': "0.0% (0/5) success ratio (< 50.0% threshold) for command: 'ls -la /tmp/non_existing'. Aborting.", 'date_success': "100.0% (5/5) success ratio (>= 100.0% threshold) for command: 'date'.", 'date_success_subfanout': "100.0% (2/2) success ratio (>= 100.0% threshold) for command: 'date'.", 'date_success_threshold': "100.0% (5/5) success ratio (>= 50.0% threshold) for command: 'date'.", 'date_success_threshold_partial': "60.0% (3/5) success ratio (>= 50.0% threshold) for command: 'date'.", 'all_success': '100.0% (5/5) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.', 'all_success_subfanout': '100.0% (2/2) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.', 'all_success_threshold': '100.0% (5/5) success ratio (>= 50.0% threshold) of nodes successfully executed all commands.', 'one_success': '100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.', 'all_failure': '0.0% (0/5) success ratio (< 100.0% threshold) of nodes successfully executed all commands. Aborting.', 'all_failure_threshold': '0.0% (0/5) success ratio (< 50.0% threshold) of nodes successfully executed all commands. Aborting.', 'global_timeout_executing_re': (r'([2-6]|)0\.0% \([0-3]/5\) of nodes were executing a command when the global ' r'timeout occurred'), 'global_timeout_executing_threshold_re': r'([2-6]|)0\.0% \([0-3]/5\) of nodes were executing a command when the global timeout occurred', 'global_timeout_pending_re': (r'([2-6]|)0\.0% \([0-3]/5\) of nodes were pending execution when the global timeout ' r'occurred'), 'global_timeout_pending_threshold_re': r'([2-6]|)0\.0% \([0-3]/5\) of nodes were pending execution when the global timeout occurred', 'sleep_total_failure': "0.0% (0/5) success ratio (< 100.0% threshold) for command: 'sleep 2'. Aborting.", 'sleep_success': "100.0% (5/5) success ratio (>= 100.0% threshold) for command: 'sleep 0.5'.", 'sleep_success_threshold': "100.0% (5/5) success ratio (>= 50.0% threshold) for command: 'sleep 0.5'.", 'sleep_timeout': "100.0% (5/5) of nodes timeout to execute command 'sleep 2'", 'sleep_timeout_threshold_re': r"[4-8]0\.0% \([2-4]/5\) of nodes timeout to execute command 'sleep 2'", 'sync': { 'ls_total_failure_ratio': "0.0% (0/5) success ratio (< 100.0% threshold) for command: 'ls -la /tmp/non_existing'. Aborting.", }, 'async': { 'ls_total_failure_ratio': "0.0% (0/5) success ratio (< 100.0% threshold). Aborting.", }, } # Tuple of dictionaries with commands to execute for each variant parameters, with the following fields: # rc: expected return code # commands: list of commands to execute # assert_true: list of labels of strings to match with assertTrue() against stderr. [optional] # assert_false: list of labels of strings to match with assertFalse() against stderr. [optional] # additional_params: list of additional parameters to pass to the CLI. [optional] _VARIANTS_COMMANDS = ( {'rc': 0, 'commands': ['ls -la /tmp', 'date'], 'assert_true': ['all_success'], 'assert_false': ['failed', 'global_timeout']}, {'rc': None, 'commands': ['ls -la /tmp/maybe', 'date']}, {'rc': 2, 'commands': ['ls -la /tmp/non_existing', 'date'], 'assert_true': ['all_failure'], 'assert_false': ['global_timeout']}, {'rc': 0, 'commands': ['date', 'date', 'date'], 'assert_true': ['all_success'], 'assert_false': ['failed', 'global_timeout']}, {'rc': 2, 'additional_params': ['--global-timeout', '1'], 'commands': ['date', 'sleep 2']}, {'rc': None, 'additional_params': ['--global-timeout', '1'], 'commands': ['sleep 0.99', 'date']}, {'rc': 2, 'additional_params': ['-t', '1'], 'commands': ['sleep 2', 'date'], 'assert_false': ['failed', 'global_timeout', 'date_success']}, {'rc': 0, 'additional_params': ['-t', '2'], 'commands': ['sleep 0.5', 'date'], 'assert_false': ['failed', 'global_timeout']}, ) # Tuple of lists of additional parameters to pass to the CLI. _VARIANTS_PARAMETERS = ( ['-m', 'sync'], ['-m', 'sync', '--batch-size', '2'], ['-m', 'sync', '--batch-size', '2', '--batch-sleep', '1.0'], ['-m', 'sync', '-p', '50'], ['-m', 'sync', '-p', '50', '--batch-size', '2'], ['-m', 'sync', '-p', '50', '--batch-size', '2', '--batch-sleep', '1.0'], ['-m', 'async'], ['-m', 'async', '--batch-size', '2'], ['-m', 'async', '--batch-size', '2', '--batch-sleep', '1.0'], ['-m', 'async', '-p', '50'], ['-m', 'async', '-p', '50', '--batch-size', '2'], ['-m', 'async', '-p', '50', '--batch-size', '2', '--batch-sleep', '1.0'], ) # Expected output for the -o/--out txt option for one node _TXT_EXPECTED_SINGLE_OUTPUT = """{prefix}{node_id}: First {prefix}{node_id}: Second {prefix}{node_id}: Third""" # Expected output for the -o/--out json option for one node _JSON_EXPECTED_SINGLE_OUTPUT = 'First\nSecond\nThird' # Expected output for the 'uname' command to stdout _UNAME_OUTPUT = "\x1b[34m----- OUTPUT of 'uname' -----\x1b[39m\nLinux\n\x1b[34m================\x1b[39m\n" def make_method(name, commands_set): """Method generator with a dynamic name and docstring.""" params = copy.deepcopy(commands_set) # Needed to have a different one for each method @pytest.mark.variant_params(params) def test_variant(self, capsys): """Test variant generated function.""" argv = self.default_params + params['params'] + [self.all_nodes] + params['commands'] rc = cli.main(argv=argv) out, err = capsys.readouterr() sys.stdout.write(out) sys.stderr.write(err) if params['rc'] is None: params['rc'] = get_rc(params) assert rc == params['rc'] assert _EXPECTED_LINES['all_targeted'] in err, _EXPECTED_LINES['all_targeted'] labels = params.get('assert_true', []) labels += get_global_timeout_expected_lines(params) if 'async' in params['params']: mode = 'async' else: mode = 'sync' labels += (get_ls_expected_lines(params) + get_date_expected_lines(params) + get_timeout_expected_lines(params)) for label in labels: if label in ('all_success', 'all_failure') and '-p' in params['params']: label = '{label}_threshold'.format(label=label) if label in _EXPECTED_LINES[mode]: string = _EXPECTED_LINES[mode][label] else: string = _EXPECTED_LINES[label] if label.endswith('_re'): assert re.search(string, err) is not None, string else: assert string in err, string for label in params.get('assert_false', []): assert _EXPECTED_LINES[label] not in err, _EXPECTED_LINES[label] # Dynamically set the name and docstring of the generated function to distinguish them test_variant.__name__ = 'test_variant_{name}'.format(name=name) test_variant.__doc__ = 'variant_function called with params: {params}'.format(params=params) return test_variant def add_variants_methods(indexes): """Decorator to add generated tests to a TestClass subclass.""" def func_wrapper(cls): """Dynamic test generator.""" for i in indexes: for j, commands_set in enumerate(_VARIANTS_COMMANDS): commands_set['params'] = _VARIANTS_PARAMETERS[i] + commands_set.get('additional_params', []) test_input = make_method('params{i:02d}_commands{j:02d}'.format(i=i, j=j), commands_set) if test_input.__doc__ is None: raise AssertionError("Missing __doc__ for test {name}".format(name=test_input.__name__)) setattr(cls, test_input.__name__, test_input) return cls return func_wrapper def get_rc(params): """Return the expected return code based on the parameters. Arguments: params: a dictionary with all the parameters passed to the variant_function """ return_value = 2 if '-p' in params['params'] and '--global-timeout' not in params['params']: return_value = 1 return return_value def get_global_timeout_expected_lines(params): """Return a list of expected lines labels for global timeout-based tests. Arguments: params: a dictionary with all the parameters passed to the variant_function """ expected = [] if '--global-timeout' not in params['params']: return expected if '-p' in params['params']: expected = ['global_timeout_executing_threshold_re', 'global_timeout_pending_threshold_re'] else: expected = ['global_timeout_executing_re', 'global_timeout_pending_re'] return expected def get_timeout_expected_lines(params): """Return a list of expected lines labels for timeout-based tests. Arguments: params: a dictionary with all the parameters passed to the variant_function """ expected = [] if '-t' not in params['params']: return expected if params['rc'] == 0: # Test successful cases if '-p' in params['params']: expected = ['sleep_success_threshold', 'date_success_threshold'] else: expected = ['date_success', 'sleep_success'] else: # Test timeout cases if '--batch-size' in params['params']: expected = ['sleep_timeout_threshold_re'] else: expected = ['sleep_timeout'] return expected def get_date_expected_lines(params): """Return a list of expected lines labels for the date command based on parameters. Arguments: params: a dictionary with all the parameters passed to the variant_function """ expected = [] if 'ls -la /tmp/non_existing' in params['commands']: return expected if '-p' in params['params']: if 'ls -la /tmp/maybe' in params['commands']: expected = ['date_success_threshold_partial'] elif 'ls -la /tmp' in params['commands']: expected = ['date_success_threshold'] elif 'ls -la /tmp' in params['commands']: expected = ['date_success'] return expected def get_ls_expected_lines(params): """Return a list of expected lines labels for the ls command based on the parameters. Arguments: params: a dictionary with all the parameters passed to the variant_function """ expected = [] if 'ls -la /tmp' in params['commands']: if '-p' in params['params']: expected = ['ls_success_threshold'] else: expected = ['ls_success'] elif 'ls -la /tmp/maybe' in params['commands']: if '-p' in params['params']: expected = ['ls_partial_success', 'ls_partial_success_threshold_ratio'] else: expected = ['ls_partial_success', 'ls_partial_success_ratio_re'] elif 'ls -la /tmp/non_existing' in params['commands']: if '--batch-size' in params['params']: if '-p' in params['params']: expected.append('ls_failure_batch_threshold') else: expected.append('ls_failure_batch') else: expected.append('ls_total_failure') if '-p' in params['params']: expected.append('ls_total_failure_threshold_ratio') else: expected.append('ls_total_failure_ratio') return expected @add_variants_methods(range(len(_VARIANTS_PARAMETERS))) class TestCLI: """CLI module tests.""" def setup_method(self, _): """Set default properties.""" # pylint: disable=attribute-defined-outside-init self.identifier = os.getenv('CUMIN_IDENTIFIER') assert self.identifier is not None, 'Unable to find CUMIN_IDENTIFIER environmental variable' self.config = os.path.join(os.getenv('CUMIN_TMPDIR', ''), 'config.yaml') self.default_params = ['--force', '-d', '-c', self.config] self.nodes_prefix = '{identifier}-'.format(identifier=self.identifier) self.all_nodes = '{prefix}[1-5]'.format(prefix=self.nodes_prefix) def _get_nodes(self, nodes): """Return the query for the nodes selection. Arguments: nodes: a string with the NodeSet nodes selection """ if nodes is None: return self.all_nodes return '{prefix}[{nodes}]'.format(prefix=self.nodes_prefix, nodes=nodes) def test_single_command_subfanout(self, capsys): """Executing one command on a subset of nodes smaller than the ClusterShell fanout.""" params = [self._get_nodes('1-2'), 'date'] rc = cli.main(argv=self.default_params + params) out, err = capsys.readouterr() sys.stdout.write(out) sys.stderr.write(err) assert _EXPECTED_LINES['subfanout_targeted'] in err, _EXPECTED_LINES['subfanout_targeted'] assert _EXPECTED_LINES['date_success_subfanout'] in err, _EXPECTED_LINES['date_success_subfanout'] assert _EXPECTED_LINES['all_success_subfanout'] in err, _EXPECTED_LINES['all_success_subfanout'] assert _EXPECTED_LINES['failed'] not in err, _EXPECTED_LINES['failed'] assert _EXPECTED_LINES['global_timeout'] not in err, _EXPECTED_LINES['global_timeout'] assert rc == 0 def test_single_command_supfanout(self, capsys): """Executing one command on a subset of nodes greater than the ClusterShell fanout.""" params = [self.all_nodes, 'date'] rc = cli.main(argv=self.default_params + params) out, err = capsys.readouterr() sys.stdout.write(out) sys.stderr.write(err) assert _EXPECTED_LINES['all_targeted'] in err, _EXPECTED_LINES['all_targeted'] assert _EXPECTED_LINES['date_success'] in err, _EXPECTED_LINES['date_success'] assert _EXPECTED_LINES['all_success'] in err, _EXPECTED_LINES['all_success'] assert _EXPECTED_LINES['failed'] not in err, _EXPECTED_LINES['failed'] assert _EXPECTED_LINES['global_timeout'] not in err, _EXPECTED_LINES['global_timeout'] assert rc == 0 def test_dry_run(self, capsys): """With --dry-run only the matching hosts are printed.""" params = ['--dry-run', self.all_nodes, 'date'] rc = cli.main(argv=self.default_params + params) out, err = capsys.readouterr() sys.stdout.write(out) sys.stderr.write(err) assert _EXPECTED_LINES['all_targeted'] in err, _EXPECTED_LINES['all_targeted'] assert _EXPECTED_LINES['dry_run'] in err, _EXPECTED_LINES['dry_run'] assert _EXPECTED_LINES['successfully'] not in err, _EXPECTED_LINES['successfully'] assert _EXPECTED_LINES['failed'] not in err, _EXPECTED_LINES['failed'] assert _EXPECTED_LINES['global_timeout'] not in err, _EXPECTED_LINES['global_timeout'] assert rc == 0 def test_timeout(self, capsys): """With a timeout shorter than a command it should fail.""" params = ['--global-timeout', '1', self.all_nodes, 'sleep 2'] rc = cli.main(argv=self.default_params + params) out, err = capsys.readouterr() sys.stdout.write(out) sys.stderr.write(err) assert _EXPECTED_LINES['all_targeted'] in err, _EXPECTED_LINES['all_targeted'] assert re.search(_EXPECTED_LINES['global_timeout_executing_re'], err) is not None, \ _EXPECTED_LINES['global_timeout_executing_re'] assert re.search(_EXPECTED_LINES['global_timeout_pending_re'], err) is not None, \ _EXPECTED_LINES['global_timeout_pending_re'] assert _EXPECTED_LINES['sleep_total_failure'] in err, _EXPECTED_LINES['sleep_total_failure'] assert _EXPECTED_LINES['all_failure'] in err, _EXPECTED_LINES['all_failure'] assert _EXPECTED_LINES['failed'] not in err, _EXPECTED_LINES['failed'] assert rc == 2 def test_version(self, capsys): """Calling --version should return the version and exit.""" with pytest.raises(SystemExit) as e: cli.main(argv=['--version']) out, err = capsys.readouterr() sys.stdout.write(out) sys.stderr.write(err) assert e.type == SystemExit assert e.value.code == 0 assert err == '' assert len(out.splitlines()) == 1 assert __version__ in out def test_out_txt(self, capsys): """The -o/--out txt option should print the output expanded for each host, prefixed by the hostname.""" params = ['-o', 'txt', self.all_nodes, 'cat /tmp/out'] rc = cli.main(argv=self.default_params + params) out, err = capsys.readouterr() sys.stdout.write(out) sys.stderr.write(err) assert _EXPECTED_LINES['all_targeted'] in err, _EXPECTED_LINES['all_targeted'] assert _EXPECTED_LINES['successfully'] in err, _EXPECTED_LINES['successfully'] assert _EXPECTED_LINES['failed'] not in err, _EXPECTED_LINES['failed'] assert rc == 0 expected_out = '\n'.join( _TXT_EXPECTED_SINGLE_OUTPUT.format(prefix=self.nodes_prefix, node_id=i) for i in range(1, 6)) assert out.split(cli.OUTPUT_SEPARATOR + '\n')[1] == expected_out + '\n' def test_out_json(self, capsys): """The -o/--out json option should print a JSON with hostnames as keys and output as values.""" params = ['-o', 'json', self.all_nodes, 'cat /tmp/out'] rc = cli.main(argv=self.default_params + params) out, err = capsys.readouterr() sys.stdout.write(out) sys.stderr.write(err) assert _EXPECTED_LINES['all_targeted'] in err, _EXPECTED_LINES['all_targeted'] assert _EXPECTED_LINES['successfully'] in err, _EXPECTED_LINES['successfully'] assert _EXPECTED_LINES['failed'] not in err, _EXPECTED_LINES['failed'] assert rc == 0 expected_out = {self.nodes_prefix + str(i): _JSON_EXPECTED_SINGLE_OUTPUT for i in range(1, 6)} assert json.loads(out.split(cli.OUTPUT_SEPARATOR + '\n')[1]) == expected_out def test_undeduplicated_output(self, capsys): """Executing a command without output deduplication (1 target host) should work as expected.""" params = [self._get_nodes('1'), 'uname'] rc = cli.main(argv=self.default_params + params) out, err = capsys.readouterr() sys.stdout.write(out) sys.stderr.write(err) assert _UNAME_OUTPUT == out assert _EXPECTED_LINES['one_success'] in err, _EXPECTED_LINES['one_success'] assert _EXPECTED_LINES['failed'] not in err, _EXPECTED_LINES['failed'] assert _EXPECTED_LINES['global_timeout'] not in err, _EXPECTED_LINES['global_timeout'] assert rc == 0 def test_cli_exception_no_trace(capsys): """When --trace is not set and an exception is raised it should not be printed to stderr.""" config = os.path.join(os.getenv('CUMIN_TMPDIR', ''), 'config.yaml') params = ['--force', '-c', config, 'Z{invalid query}'] rc = cli.main(argv=params) out, err = capsys.readouterr() assert not out assert 'Traceback' not in err assert 'Caught InvalidQueryError exception' in err assert rc == 99 def test_cli_exception_trace(capsys): """When --trace is set and an exception is raised it should be printed to stderr.""" config = os.path.join(os.getenv('CUMIN_TMPDIR', ''), 'config.yaml') params = ['--force', '--trace', '-c', config, 'Z{invalid query}'] rc = cli.main(argv=params) out, err = capsys.readouterr() assert not out assert 'Traceback' in err assert 'cumin.backends.InvalidQueryError' in err assert rc == 99 def test_cli_no_hosts_found(capsys): """If the query returns no hosts it should print that and exit with 0.""" config = os.path.join(os.getenv('CUMIN_TMPDIR', ''), 'config.yaml') params = ['--force', '-c', config, '(D{host1} and D{host2}) and D{host[1-5]}'] rc = cli.main(argv=params) out, err = capsys.readouterr() assert not out assert 'No hosts found that matches the query' in err assert rc == 0 wikimedia-cumin-36f957f/cumin/tests/integration/transports/000077500000000000000000000000001476500461000241635ustar00rootroot00000000000000wikimedia-cumin-36f957f/cumin/tests/integration/transports/__init__.py000066400000000000000000000000541476500461000262730ustar00rootroot00000000000000"""Transport specific integration tests.""" wikimedia-cumin-36f957f/cumin/tests/integration/transports/clustershell.sh000077500000000000000000000033161476500461000272360ustar00rootroot00000000000000#!/bin/bash set -e SSH_KEY_ALGO='ed25519' function setup() { ssh-keygen -t ${SSH_KEY_ALGO} -N "" -f "${CUMIN_TMPDIR}/id_${SSH_KEY_ALGO}" -C "cumin-integration-tests" > /dev/null cat < "${CUMIN_TMPDIR}/config.yaml" default_backend: direct transport: clustershell log_file: ${CUMIN_TMPDIR}/cumin.log clustershell: ssh_options: - '-F ${CUMIN_TMPDIR}/ssh_config' fanout: 3 EOF local SSH_ALIASES="" _log "Creating docker instances" for index in {1..5}; do HOST_NAME="${CUMIN_IDENTIFIER}-${index}" # TODO: use a custom-generated image docker run -d -p "222${index}:2222" -e PUBLIC_KEY="$(cat ${CUMIN_TMPDIR}/id_${SSH_KEY_ALGO}.pub)" \ -e USER_NAME=cumin -e SUDO_ACCESS=true --hostname "${HOST_NAME}" --name "${HOST_NAME}" \ "linuxserver/openssh-server:latest" > /dev/null DOCKER_INSTANCES="${DOCKER_INSTANCES} ${HOST_NAME}" SSH_ALIASES="${SSH_ALIASES} Host ${HOST_NAME} Port 222${index} " done cat < "${CUMIN_TMPDIR}/ssh_config" Host * User cumin Hostname localhost IdentityFile ${CUMIN_TMPDIR}/id_${SSH_KEY_ALGO} IdentitiesOnly yes LogLevel QUIET StrictHostKeyChecking no UserKnownHostsFile /dev/null ${SSH_ALIASES} EOF _log "Created docker instances: ${DOCKER_INSTANCES}" } function run_tests() { sleep 5 # Make sure all SSH servers are up and running cumin --force -c "${CUMIN_TMPDIR}/config.yaml" "${CUMIN_IDENTIFIER}-[1-2,5]" "touch /tmp/maybe" cumin --force -c "${CUMIN_TMPDIR}/config.yaml" "${CUMIN_IDENTIFIER}-[1-5]" 'echo -e "First\nSecond\nThird" > /tmp/out' py.test -n auto --strict-markers --cov-report term-missing --cov=cumin cumin/tests/integration } wikimedia-cumin-36f957f/cumin/tests/integration/transports/test_clustershell.py000066400000000000000000000047621476500461000303160ustar00rootroot00000000000000"""Clustershell module integration tests.""" import os import sys from pathlib import Path import cumin from cumin import query, transport, transports from cumin.transports.clustershell import NullReporter # Expected block output for a single hostname command _HOSTNAME_BLOCK_OUTPUT = """\x1b[34m===== NODE GROUP =====\x1b[39m \x1b[36m(1) {hostname}\x1b[39m \x1b[34m----- OUTPUT of 'hostname' -----\x1b[39m {hostname} """ class TestClustershellTransport: """Clustershell transport class tests.""" def setup_method(self, _): """Set default properties.""" # pylint: disable=attribute-defined-outside-init self.identifier = os.getenv('CUMIN_IDENTIFIER') assert self.identifier is not None, 'Unable to find CUMIN_IDENTIFIER environmental variable' self.nodes_prefix = '{identifier}-'.format(identifier=self.identifier) self.all_nodes = '{prefix}[1-5]'.format(prefix=self.nodes_prefix) self.config = cumin.Config(config=Path(os.getenv('CUMIN_TMPDIR', '')) / 'config.yaml') self.hosts = query.Query(self.config).execute('D{{{nodes}}}'.format(nodes=self.all_nodes)) self.target = transports.Target(self.hosts) self.worker = transport.Transport.new(self.config, self.target) self.worker.commands = ['hostname'] def test_execute_tqdm_reporter(self, capsys): """It should execute the command on the target hosts and print to stdout the stdout/err of the command.""" self.worker.handler = 'sync' exit_code = self.worker.execute() assert exit_code == 0 for nodes, output in self.worker.get_results(): assert str(nodes) == output.message().decode() out, err = capsys.readouterr() sys.stdout.write(out) sys.stderr.write(err) for host in self.hosts: assert _HOSTNAME_BLOCK_OUTPUT.format(hostname=host) in out def test_execute_null_reporter(self, capsys): """It should execute the command on the target hosts and not print to stdout the stdout/err of the command.""" self.worker.handler = 'sync' self.worker.reporter = NullReporter exit_code = self.worker.execute() assert exit_code == 0 for nodes, output in self.worker.get_results(): assert str(nodes) == output.message().decode() out, err = capsys.readouterr() sys.stdout.write(out) sys.stderr.write(err) for host in self.hosts: assert _HOSTNAME_BLOCK_OUTPUT.format(hostname=host) not in out wikimedia-cumin-36f957f/cumin/tests/unit/000077500000000000000000000000001476500461000204005ustar00rootroot00000000000000wikimedia-cumin-36f957f/cumin/tests/unit/__init__.py000066400000000000000000000000221476500461000225030ustar00rootroot00000000000000"""Unit tests.""" wikimedia-cumin-36f957f/cumin/tests/unit/backends/000077500000000000000000000000001476500461000221525ustar00rootroot00000000000000wikimedia-cumin-36f957f/cumin/tests/unit/backends/__init__.py000066400000000000000000000000361476500461000242620ustar00rootroot00000000000000"""Backend specific tests.""" wikimedia-cumin-36f957f/cumin/tests/unit/backends/external/000077500000000000000000000000001476500461000237745ustar00rootroot00000000000000wikimedia-cumin-36f957f/cumin/tests/unit/backends/external/__init__.py000066400000000000000000000035041476500461000261070ustar00rootroot00000000000000"""External backends package for testing.""" import pyparsing as pp from cumin import nodeset from cumin.backends import BaseQuery def grammar(): """Define the query grammar for the external backend used for testing.""" # Hosts selection: clustershell (,!&^[]) syntax is allowed: host10[10-42].domain hosts = pp.Word(pp.alphanums + '-_.,!&^[]')('hosts') # Final grammar, see the docstring for its BNF based on the tokens defined above # Groups are used to split the parsed results for an easy access full_grammar = pp.Forward() full_grammar << pp.Group(hosts) + pp.ZeroOrMore(pp.Group(hosts)) # pylint: disable=expression-not-assigned return full_grammar class ExternalBackendQuery(BaseQuery): """External backend test query class.""" grammar = grammar() """:py:class:`pyparsing.ParserElement`: load the grammar parser only once in a singleton-like way.""" def __init__(self, config): """Query constructor for the test external backend. :Parameters: according to parent :py:meth:`cumin.backends.BaseQuery.__init__`. """ super().__init__(config) self.hosts = nodeset() def _execute(self): """Concrete implementation of parent abstract method. :Parameters: according to parent :py:meth:`cumin.backends.BaseQuery._execute`. Returns: ClusterShell.NodeSet.NodeSet: with the FQDNs of the matching hosts. """ return self.hosts def _parse_token(self, token): """Concrete implementation of parent abstract method. :Parameters: according to parent :py:meth:`cumin.backends.BaseQuery._parse_token`. """ if isinstance(token, str): return token_dict = token.asDict() self.hosts |= nodeset(token_dict['hosts']) wikimedia-cumin-36f957f/cumin/tests/unit/backends/external/duplicate_prefix.py000066400000000000000000000010441476500461000276740ustar00rootroot00000000000000"""Test external backend module with a GRAMMAR_PREFIX that conflicts with an existing one.""" from cumin.tests.unit.backends.external import ExternalBackendQuery GRAMMAR_PREFIX = 'D' """:py:class:`str`: the prefix associate to this grammar, to register this backend into the general grammar. Required by the backend auto-loader in :py:meth:`cumin.grammar.get_registered_backends`.""" query_class = ExternalBackendQuery # pylint: disable=invalid-name """Required by the backend auto-loader in :py:meth:`cumin.grammar.get_registered_backends`.""" wikimedia-cumin-36f957f/cumin/tests/unit/backends/external/missing_grammar_prefix.py000066400000000000000000000004501476500461000311010ustar00rootroot00000000000000"""Test external backend module with missing GRAMMAR_PREFIX.""" from cumin.tests.unit.backends.external import ExternalBackendQuery query_class = ExternalBackendQuery # pylint: disable=invalid-name """Required by the backend auto-loader in :py:meth:`cumin.grammar.get_registered_backends`.""" wikimedia-cumin-36f957f/cumin/tests/unit/backends/external/missing_query_class.py000066400000000000000000000004361476500461000304340ustar00rootroot00000000000000"""Test external backend module with missing query class.""" GRAMMAR_PREFIX = '_Z' """:py:class:`str`: the prefix associate to this grammar, to register this backend into the general grammar. Required by the backend auto-loader in :py:meth:`cumin.grammar.get_registered_backends`.""" wikimedia-cumin-36f957f/cumin/tests/unit/backends/external/ok.py000066400000000000000000000007631476500461000247650ustar00rootroot00000000000000"""Test working external backend module.""" from cumin.tests.unit.backends.external import ExternalBackendQuery GRAMMAR_PREFIX = '_Z' """:py:class:`str`: the prefix associate to this grammar, to register this backend into the general grammar. Required by the backend auto-loader in :py:meth:`cumin.grammar.get_registered_backends`.""" query_class = ExternalBackendQuery # pylint: disable=invalid-name """Required by the backend auto-loader in :py:meth:`cumin.grammar.get_registered_backends`.""" wikimedia-cumin-36f957f/cumin/tests/unit/backends/external/wrong_inheritance.py000066400000000000000000000010321476500461000300470ustar00rootroot00000000000000"""Test external backend module with wrong inheritance of the query class.""" class WrongInheritance: """Test query class with wrong inheritance.""" GRAMMAR_PREFIX = '_Z' """:py:class:`str`: the prefix associate to this grammar, to register this backend into the general grammar. Required by the backend auto-loader in :py:meth:`cumin.grammar.get_registered_backends`.""" query_class = WrongInheritance # pylint: disable=invalid-name """Required by the backend auto-loader in :py:meth:`cumin.grammar.get_registered_backends`.""" wikimedia-cumin-36f957f/cumin/tests/unit/backends/test_direct.py000066400000000000000000000025131476500461000250360ustar00rootroot00000000000000"""Direct backend tests.""" from cumin import nodeset from cumin.backends import BaseQuery, direct def test_direct_query_class(): """An instance of query_class should be an instance of BaseQuery.""" query = direct.query_class({}) assert isinstance(query, BaseQuery) class TestDirectQuery: """Direct backend query test class.""" def setup_method(self, _): """Setup an instance of DirectQuery for each test.""" self.query = direct.DirectQuery({}) # pylint: disable=attribute-defined-outside-init def test_instantiation(self): """An instance of DirectQuery should be an instance of BaseQuery.""" assert isinstance(self.query, BaseQuery) assert self.query.config == {} def test_execute(self): """Calling execute() should return the list of hosts.""" assert self.query.execute('host1 or host2') == nodeset('host[1-2]') assert self.query.execute('host1 and host2') == nodeset() assert self.query.execute('host1 and not host2') == nodeset('host1') assert self.query.execute('host[1-5] xor host[3-7]') == nodeset('host[1-2,6-7]') assert self.query.execute('host1 or (host[10-20] and not host15)') == nodeset('host[1,10-14,16-20]') assert self.query.execute('(host1 or host[2-3]) and not (host[3-9] or host2)') == nodeset('host1') wikimedia-cumin-36f957f/cumin/tests/unit/backends/test_grammars.py000066400000000000000000000026461476500461000254040ustar00rootroot00000000000000"""Backends basic grammar tests.""" import importlib import os import pkgutil import pytest from cumin import backends from cumin.tests import get_fixture BACKENDS = [name for _, name, ispkg in pkgutil.iter_modules(backends.__path__) if not ispkg] BASE_PATH = os.path.join('backends', 'grammars') @pytest.mark.parametrize('backend_name', BACKENDS) def test_valid_grammars(backend_name): """Run quick pyparsing test over valid grammar strings for each backend that has the appropriate fixture.""" try: backend = importlib.import_module('cumin.backends.{backend}'.format(backend=backend_name)) except ImportError: return # Backend not available results = backend.grammar().runTests( get_fixture(os.path.join(BASE_PATH, '{backend}_valid.txt'.format(backend=backend_name)), as_string=True)) assert results[0] @pytest.mark.parametrize('backend_name', BACKENDS) def test_invalid_grammars(backend_name): """Run quick pyparsing test over invalid grammar strings for each backend that has the appropriate fixture.""" try: backend = importlib.import_module('cumin.backends.{backend}'.format(backend=backend_name)) except ImportError: return # Backend not available results = backend.grammar().runTests( get_fixture(os.path.join(BASE_PATH, '{backend}_invalid.txt'.format(backend=backend_name)), as_string=True), failureTests=True) assert results[0] wikimedia-cumin-36f957f/cumin/tests/unit/backends/test_knownhosts.py000066400000000000000000000217351476500461000260100ustar00rootroot00000000000000"""Known hosts backend tests.""" import os import pytest from ClusterShell.NodeSet import NodeSet, RESOLVER_NOGROUP from cumin.backends import BaseQuery from cumin.backends.knownhosts import KnownHostsLineError, KnownHostsQuery, KnownHostsSkippedLineError, query_class from cumin.tests import get_fixture_path def test_knownhosts_query_class(): """An instance of query_class should be an instance of BaseQuery.""" query = query_class({}) assert isinstance(query, BaseQuery) class TestKnownhostsQuery: """Knownhosts backend query test class.""" def setup_method(self, _): """Set up an instance of KnownHostsQuery for each test.""" # pylint: disable=attribute-defined-outside-init self.query = KnownHostsQuery({ 'knownhosts': {'files': [ get_fixture_path(os.path.join('backends', 'knownhosts.txt')), get_fixture_path(os.path.join('backends', 'knownhosts_man.txt')), ]}}) self.no_query = KnownHostsQuery({}) self.no_hosts = NodeSet(resolver=RESOLVER_NOGROUP) self.domain_hosts = NodeSet('host[1,4-5,7-8,13-14].domain', resolver=RESOLVER_NOGROUP) self.all_hosts = self.domain_hosts | NodeSet('closenet,cvs.example.net', resolver=RESOLVER_NOGROUP) def test_instantiation(self): """An instance of KnownHostsQuery should be an instance of BaseQuery.""" assert isinstance(self.query, BaseQuery) assert 'knownhosts' in self.query.config def test_execute(self): """Calling execute() with one host should return it.""" assert self.query.execute('host1.domain') == NodeSet('host1.domain', resolver=RESOLVER_NOGROUP) def test_execute_non_existent(self): """Calling execute() with one host that doens't exists should return no hosts.""" assert self.query.execute('nohost1.domain') == self.no_hosts def test_execute_or(self): """Calling execute() with two hosts in 'or' should return both hosts.""" expected = NodeSet('host[1,4].domain', resolver=RESOLVER_NOGROUP) assert self.query.execute('host1.domain or host4.domain') == expected def test_execute_and(self): """Calling execute() with two hosts in 'and' should return no hosts.""" assert self.query.execute('host1.domain and host2.domain') == self.no_hosts def test_execute_and_not(self): """Calling execute() with two hosts with 'and not' should return the first host.""" expected = NodeSet('host1.domain', resolver=RESOLVER_NOGROUP) assert self.query.execute('host1.domain and not host2.domain') == expected def test_execute_xor(self): """Calling execute() with two host groups with 'xor' should return the hosts that are not in both groups.""" expected = NodeSet('host[1,7-8].domain', resolver=RESOLVER_NOGROUP) assert self.query.execute('host[1-8].domain xor host[4-6].domain') == expected def test_execute_complex(self): """Calling execute() with a complex query should return the matching hosts.""" expected = NodeSet('host[1,5,8].domain', resolver=RESOLVER_NOGROUP) assert self.query.execute('host1.domain or (host[5-9].domain and not host7.domain)') == expected expected = NodeSet('host1.domain', resolver=RESOLVER_NOGROUP) assert self.query.execute( '(host1.domain or host[2-5].domain) and not (host[3-9].domain or host2.domain)') == expected def test_execute_all(self): """Calling execute() with broader matching should return all hosts.""" assert self.query.execute('*') == self.all_hosts assert self.query.execute('host[1-100].domain') == self.domain_hosts assert self.query.execute('host[1-100].domai?') == self.domain_hosts assert self.query.execute('host[1-100].*') == self.domain_hosts def test_execute_no_hosts(self): """Calling execute() without any known hosts to load should return no hosts.""" assert self.no_query.execute('host1.domain') == self.no_hosts assert self.no_query.execute('*') == self.no_hosts def test_parse_line_empty(): """Empty lines should raise KnownHostsSkippedLineError.""" with pytest.raises(KnownHostsSkippedLineError, match='empty line'): KnownHostsQuery.parse_known_hosts_line('') with pytest.raises(KnownHostsSkippedLineError, match='empty line'): KnownHostsQuery.parse_known_hosts_line('\n') def test_parse_line_comment(): """Comment lines should raise KnownHostsSkippedLineError.""" with pytest.raises(KnownHostsSkippedLineError, match='comment'): KnownHostsQuery.parse_known_hosts_line('# comment') def test_parse_line_hashed(): """Hashed lines should raise KnownHostsSkippedLineError.""" with pytest.raises(KnownHostsSkippedLineError, match='hashed'): KnownHostsQuery.parse_known_hosts_line('|1|HaSh=|HaSh= ecdsa-sha2-nistp256 AAAA...=') def test_parse_line_no_fields(): """Lines without enough fields should raise KnownHostsLineError.""" with pytest.raises(KnownHostsLineError, match='not enough fields'): KnownHostsQuery.parse_known_hosts_line('host1 ssh-rsa') def test_parse_line_no_fields_mark(): """Lines with a marker but without enough fields should raise KnownHostsLineError.""" with pytest.raises(KnownHostsLineError, match='not enough fields'): KnownHostsQuery.parse_known_hosts_line('@marker host1 ssh-rsa') def test_parse_line_revoked(): """Lines with a revoked marker should raise KnownHostsSkippedLineError.""" with pytest.raises(KnownHostsSkippedLineError, match='revoked'): KnownHostsQuery.parse_known_hosts_line('@revoked host1 ecdsa-sha2-nistp256 AAAA...=') def test_parse_line_unknown_marker(): """Lines with an unknown marker should raise KnownHostsLineError.""" with pytest.raises(KnownHostsLineError, match='unknown marker'): KnownHostsQuery.parse_known_hosts_line('@marker host1 ecdsa-sha2-nistp256 AAAA...=') def test_parse_line_ca(): """Lines with a cert-authority marker should parse the hostnames.""" expected = ({'host1'}, set()) assert KnownHostsQuery.parse_known_hosts_line('@cert-authority host1 ecdsa-sha2-nistp256 AAAA...=') == expected def test_parse_line(): """With a standard line should parse the hostnames.""" assert KnownHostsQuery.parse_known_hosts_line('host1 ecdsa-sha2-nistp256 AAAA...=') == ({'host1'}, set()) def test_parse_line_hosts_empty(): """Empty line hosts should be skipped.""" assert KnownHostsQuery.parse_line_hosts(',') == (set(), set()) assert KnownHostsQuery.parse_line_hosts('host1,,') == ({'host1'}, set()) def test_parse_line_hosts_negated(): """Negated line hosts should remove the negation.""" assert KnownHostsQuery.parse_line_hosts('!host1') == ({'host1'}, set()) expected = ({'host1', 'host2'}, set()) assert KnownHostsQuery.parse_line_hosts('!host1,host2') == expected assert KnownHostsQuery.parse_line_hosts('host1,!host2') == expected assert KnownHostsQuery.parse_line_hosts('!host1,!host2') == expected def test_parse_line_hosts_port(): """Line hosts with custom ports should remove the additional syntax.""" assert KnownHostsQuery.parse_line_hosts('[host1]:2222') == ({'host1'}, set()) expected = ({'host1', 'host2'}, set()) assert KnownHostsQuery.parse_line_hosts('[host1]:2222,host2') == expected assert KnownHostsQuery.parse_line_hosts('host1,[host2]:2222') == expected assert KnownHostsQuery.parse_line_hosts('[host1]:2222,[host2]:2222') == expected def test_parse_line_hosts_neg_port(): """Line hosts with custom ports and negated entries should remove the additional syntax.""" assert KnownHostsQuery.parse_line_hosts('![host1]:2222') == ({'host1'}, set()) expected = ({'host1', 'host2'}, set()) assert KnownHostsQuery.parse_line_hosts('![host1]:2222,!host2') == expected assert KnownHostsQuery.parse_line_hosts('!host1,![host2]:2222') == expected assert KnownHostsQuery.parse_line_hosts('![host1]:2222,![host2]:2222') == expected def test_parse_line_hosts_patterns(): """Line hosts with patterns should skip the patterns entries.""" assert KnownHostsQuery.parse_line_hosts('host?') == (set(), {'host?'}) assert KnownHostsQuery.parse_line_hosts('host*') == (set(), {'host*'}) assert KnownHostsQuery.parse_line_hosts('host?,host2') == ({'host2'}, {'host?'}) assert KnownHostsQuery.parse_line_hosts('host*,host2') == ({'host2'}, {'host*'}) assert KnownHostsQuery.parse_line_hosts('host*,host2,host?') == ({'host2'}, {'host?', 'host*'}) def test_parse_line_hosts_ips(): """Line hosts with IPs should skip the IP entries.""" assert KnownHostsQuery.parse_line_hosts('127.0.1.1') == (set(), {'127.0.1.1'}) assert KnownHostsQuery.parse_line_hosts('fe80::1') == (set(), {'fe80::1'}) assert KnownHostsQuery.parse_line_hosts('host1,127.0.1.1') == ({'host1'}, {'127.0.1.1'}) assert KnownHostsQuery.parse_line_hosts('host1,fe80::1') == ({'host1'}, {'fe80::1'}) assert KnownHostsQuery.parse_line_hosts('host1,127.0.1.1,fe80::1') == ({'host1'}, {'127.0.1.1', 'fe80::1'}) wikimedia-cumin-36f957f/cumin/tests/unit/backends/test_openstack.py000066400000000000000000000153721476500461000255620ustar00rootroot00000000000000"""OpenStack backend tests.""" from collections import namedtuple from unittest import mock from cumin import nodeset from cumin.backends import BaseQuery, openstack Project = namedtuple('Project', ['name']) Server = namedtuple('Server', ['name']) def test_openstack_query_class(): """An instance of query_class should be an instance of BaseQuery.""" query = openstack.query_class({}) assert isinstance(query, BaseQuery) def test_openstack_query_class_init(): """An instance of OpenStackQuery should be an instance of BaseQuery.""" config = {'key': 'value'} query = openstack.OpenStackQuery(config) assert isinstance(query, BaseQuery) assert query.config == config def test_all_selection(): """A selection for all hosts is properly parsed and interpreted.""" parsed = openstack.grammar().parseString('*', parseAll=True) assert parsed[0].asDict() == {'all': '*'} def test_key_value_token(): """A token is properly parsed and interpreted.""" parsed = openstack.grammar().parseString('project:project_name', parseAll=True) assert parsed[0].asDict() == {'key': 'project', 'value': 'project_name'} def test_key_value_tokens(): """Multiple tokens are properly parsed and interpreted.""" parsed = openstack.grammar().parseString('project:project_name name:hostname', parseAll=True) assert parsed[0].asDict() == {'key': 'project', 'value': 'project_name'} assert parsed[1].asDict() == {'key': 'name', 'value': 'hostname'} @mock.patch('cumin.backends.openstack.nova_client.Client') @mock.patch('cumin.backends.openstack.keystone_client.Client') @mock.patch('cumin.backends.openstack.keystone_session.Session') @mock.patch('cumin.backends.openstack.keystone_identity.Password') class TestOpenStackQuery: """OpenStack backend query test class.""" def setup_method(self, _): """Set an instance of OpenStackQuery for each test.""" self.config = {'openstack': {}} # pylint: disable=attribute-defined-outside-init self.query = openstack.OpenStackQuery(self.config) # pylint: disable=attribute-defined-outside-init def test_execute_all(self, keystone_identity, keystone_session, keystone_client, nova_client): """Calling execute() with a query that select all hosts should return the list of all hosts.""" keystone_client.return_value.projects.list.return_value = [Project('project1'), Project('project2')] nova_client.return_value.servers.list.side_effect = [ [Server('host1'), Server('host2')], [Server('host1'), Server('host2')]] hosts = self.query.execute('*') assert hosts == nodeset('host[1-2].project[1-2]') assert keystone_identity.call_count == 3 assert keystone_session.call_count == 3 keystone_client.assert_called_once_with(session=keystone_session(), timeout=10) assert nova_client.call_args_list == [ mock.call('2', endpoint_type='public', session=keystone_session(), timeout=10), mock.call('2', endpoint_type='public', session=keystone_session(), timeout=10)] assert nova_client().servers.list.call_args_list == [ mock.call(search_opts={'vm_state': 'ACTIVE', 'status': 'ACTIVE'})] * 2 def test_execute_project(self, keystone_identity, keystone_session, keystone_client, nova_client): """Calling execute() with a query that select all hosts in a project should return the list of hosts.""" nova_client.return_value.servers.list.return_value = [Server('host1'), Server('host2')] hosts = self.query.execute('project:project1') assert hosts == nodeset('host[1-2].project1') assert keystone_identity.call_count == 1 assert keystone_session.call_count == 1 keystone_client.assert_not_called() nova_client.assert_called_once_with('2', endpoint_type='public', session=keystone_session(), timeout=10) nova_client().servers.list.assert_called_once_with(search_opts={'vm_state': 'ACTIVE', 'status': 'ACTIVE'}) def test_execute_project_name(self, keystone_identity, keystone_session, keystone_client, nova_client): """Calling execute() with a query that select hosts matching a name in a project should return only those.""" nova_client.return_value.servers.list.return_value = [Server('host1'), Server('host2')] hosts = self.query.execute('project:project1 name:host') assert hosts == nodeset('host[1-2].project1') assert keystone_identity.call_count == 1 assert keystone_session.call_count == 1 keystone_client.assert_not_called() nova_client.assert_called_once_with('2', endpoint_type='public', session=keystone_session(), timeout=10) nova_client().servers.list.assert_called_once_with( search_opts={'vm_state': 'ACTIVE', 'status': 'ACTIVE', 'name': 'host'}) def test_execute_project_domain(self, keystone_identity, keystone_session, keystone_client, nova_client): """When the domain suffix is configured, it should append it to all hosts.""" nova_client.return_value.servers.list.return_value = [Server('host1'), Server('host2')] self.config['openstack']['domain_suffix'] = 'servers.local' query = openstack.OpenStackQuery(self.config) hosts = query.execute('project:project1') assert hosts == nodeset('host[1-2].project1.servers.local') assert keystone_identity.call_count == 1 assert keystone_session.call_count == 1 keystone_client.assert_not_called() def test_execute_project_dot_domain(self, keystone_identity, keystone_session, keystone_client, nova_client): """When the domain suffix is configured with a dot, it should append it to all hosts without the dot.""" nova_client.return_value.servers.list.return_value = [Server('host1'), Server('host2')] self.config['openstack']['domain_suffix'] = '.servers.local' query = openstack.OpenStackQuery(self.config) hosts = query.execute('project:project1') assert hosts == nodeset('host[1-2].project1.servers.local') assert keystone_identity.call_count == 1 assert keystone_session.call_count == 1 keystone_client.assert_not_called() def test_execute_query_params(self, keystone_identity, keystone_session, keystone_client, nova_client): """When the query_params are set, they must be loaded automatically.""" nova_client.return_value.servers.list.return_value = [Server('host1'), Server('host2')] self.config['openstack']['query_params'] = {'project': 'project1'} query = openstack.OpenStackQuery(self.config) hosts = query.execute('*') assert hosts == nodeset('host[1-2].project1') assert keystone_identity.call_count == 1 assert keystone_session.call_count == 1 keystone_client.assert_not_called() wikimedia-cumin-36f957f/cumin/tests/unit/backends/test_puppetdb.py000066400000000000000000000403311476500461000254070ustar00rootroot00000000000000"""PuppetDB backend tests.""" from unittest import mock import pytest from requests.exceptions import HTTPError from cumin import nodeset from cumin.backends import BaseQuery, InvalidQueryError, puppetdb def test_puppetdb_query_class(): """An instance of query_class should be an instance of BaseQuery.""" query = puppetdb.query_class({}) assert isinstance(query, BaseQuery) def _get_category_key_token(category='F', key='key1', operator='=', value='value1'): """Generate and return a category token string and it's expected dictionary of tokens when parsed.""" expected = {'category': category, 'key': key, 'operator': operator, 'quoted': value} token = '{category}:{key} {operator} {quoted}'.format(**expected) return token, expected def test_single_category_key_token(): """A valid single token with a category that has key is properly parsed and interpreted.""" token, expected = _get_category_key_token() parsed = puppetdb.grammar().parseString(token, parseAll=True) assert parsed[0].asDict() == expected def test_hosts_selection(): """A host selection is properly parsed and interpreted.""" hosts = 'host[10-20,30-40].domain' parsed = puppetdb.grammar().parseString(hosts, parseAll=True) # Backward compatibility with PyParsing<2.3.1, this check works both with a list or a string assert hosts in parsed[0].asDict()['hosts'] class TestPuppetDBQueryV4: """PuppetDB backend query test class for API version 4.""" def setup_method(self, _): """Set an instance of PuppetDBQuery for each test.""" self.query = puppetdb.PuppetDBQuery({}) # pylint: disable=attribute-defined-outside-init def test_instantiation(self): """An instance of PuppetDBQuery should be an instance of BaseQuery.""" assert isinstance(self.query, BaseQuery) assert self.query.url == 'https://localhost:443/pdb/query/v4/' def test_endpoint_getter(self): """Access to endpoint property should return nodes by default.""" assert self.query.endpoint == 'nodes' @pytest.mark.parametrize('endpoint', set(puppetdb.PuppetDBQuery.endpoints.values())) def test_endpoint_setter_valid(self, endpoint): """Setting the endpoint property should accept valid values.""" self.query.endpoint = endpoint assert self.query.endpoint == endpoint def test_endpoint_setter_invalid(self): """Setting the endpoint property should raise InvalidQueryError for an invalid value.""" with pytest.raises(InvalidQueryError, match="Invalid value 'invalid_value'"): self.query.endpoint = 'invalid_value' def test_endpoint_setter_mixed1(self): """Setting the endpoint property twice to different values should raise InvalidQueryError (combination 1).""" assert self.query.endpoint == 'nodes' self.query.endpoint = 'resources' assert self.query.endpoint == 'resources' with pytest.raises(InvalidQueryError, match='Mixed endpoints are not supported'): self.query.endpoint = 'nodes' def test_endpoint_setter_mixed2(self): """Setting the endpoint property twice to different values should raise InvalidQueryError (combination 2).""" assert self.query.endpoint == 'nodes' self.query.endpoint = 'nodes' assert self.query.endpoint == 'nodes' with pytest.raises(InvalidQueryError, match='Mixed endpoints are not supported'): self.query.endpoint = 'resources' @mock.patch.object(puppetdb.PuppetDBQuery, '_api_call') class TestPuppetDBQueryBuildV4: """PuppetDB backend API v4 query build test class.""" def setup_method(self, _): """Set an instace of PuppetDBQuery for each test.""" self.query = puppetdb.PuppetDBQuery({}) # pylint: disable=attribute-defined-outside-init @pytest.mark.parametrize('query, expected', ( ( # Base fact 'F:key=value', '["=", ["fact", "key"], "value"]'), ( # Negated 'not F:key = value', '["not", ["=", ["fact", "key"], "value"]]'), ( # Different operator 'F:key >= value', '[">=", ["fact", "key"], "value"]'), ( # Regex with backslash escaped r'F:key ~ value\\escaped', r'["~", ["fact", "key"], "value\\\\escaped"]'), ( # Regex with dot escaped r'F:key ~ value\.escaped', r'["~", ["fact", "key"], "value\\.escaped"]'), )) def test_add_category_fact(self, mocked_api_call, query, expected): """A fact query should add the proper query token to the current_group.""" expected = f'["extract", ["certname"], {expected}, ["group_by", "certname"]]' self.query.execute(query) mocked_api_call.assert_called_with(expected) @pytest.mark.parametrize('query, expected', ( ( # Base fact 'I:facts.key=value', '["=", "facts.key", "value"]'), ( # Negated 'not I:facts.key = value', '["not", ["=", "facts.key", "value"]]'), ( # Different operator 'I:facts.key >= value', '[">=", "facts.key", "value"]'), ( # Regex with backslash escaped r'I:facts.key ~ value\\escaped', r'["~", "facts.key", "value\\\\escaped"]'), ( # Regex with dot escaped r'I:facts.key ~ value\.escaped', r'["~", "facts.key", "value\\.escaped"]'), ( # Fact path with array 'I:facts.key[0].subkey = value', '["=", "facts.key[0].subkey", "value"]'), ( # Fact path with dot in the name 'I:facts.key."sub.key" = value', r'["=", "facts.key.\"sub.key\"", "value"]'), ( # Multiple query fact and trusted 'I:facts.key1 = 1 and I:trusted.key2 = 2', '["and", ["=", "facts.key1", 1], ["=", "trusted.key2", 2]]'), )) def test_add_category_inventory(self, mocked_api_call, query, expected): """An inventory query should add the proper query token to the current_group.""" expected = f'["extract", ["certname"], {expected}, ["group_by", "certname"]]' self.query.execute(query) mocked_api_call.assert_called_with(expected) @pytest.mark.parametrize('query, expected', ( ( # Base resource equality 'R:key = value', '["and", ["=", "type", "Key"], ["=", "title", "value"]]'), ( # Class title 'R:class = classtitle', '["and", ["=", "type", "Class"], ["=", "title", "Classtitle"]]'), ( # Class path 'R:class = resource::path::to::class', '["and", ["=", "type", "Class"], ["=", "title", "Resource::Path::To::Class"]]'), ( # Negated 'not R:key = value', '["not", ["and", ["=", "type", "Key"], ["=", "title", "value"]]]'), ( # Regex backslash escaped r'R:key ~ value\\escaped', r'["and", ["=", "type", "Key"], ["~", "title", "value\\\\escaped"]]'), ( # Regex dot escaped r'R:key ~ value\.escaped', r'["and", ["=", "type", "Key"], ["~", "title", "value\\.escaped"]]'), ( # Regex class r'R:Class ~ "Role::(One|Another)"', r'["and", ["=", "type", "Class"], ["~", "title", "Role::(One|Another)"]]'), ( # Resource parameter 'R:resource%param = value', '["and", ["=", "type", "Resource"], ["=", ["parameter", "param"], "value"]]'), ( # Resource parameter regex 'R:resource%param ~ value.*', '["and", ["=", "type", "Resource"], ["~", ["parameter", "param"], "value.*"]]'), ( # Resource field 'R:resource@field = value', '["and", ["=", "type", "Resource"], ["=", "field", "value"]]'), ( # Resource type 'R:Resource', '["and", ["=", "type", "Resource"]]'), ( # Class shortcut 'C:class_name', '["and", ["=", "type", "Class"], ["=", "title", "Class_name"]]'), ( # Class shortcut with path 'C:module::class::name', '["and", ["=", "type", "Class"], ["=", "title", "Module::Class::Name"]]'), ( # Class shortcut with parameter 'C:class_name%param = value', ('["and", ["and", ["=", "type", "Class"], ["=", "title", "Class_name"]], ' '["and", ["=", "type", "Class"], ["=", ["parameter", "param"], "value"]]]')), ( # Class shortcut with field 'C:class_name@field = value', ('["and", ["and", ["=", "type", "Class"], ["=", "title", "Class_name"]], ' '["and", ["=", "type", "Class"], ["=", "field", "value"]]]')), ( # Profile shortcut 'P:profile_name', '["and", ["=", "type", "Class"], ["=", "title", "Profile::Profile_name"]]'), ( # Profile shortcut path 'P:module::name', '["and", ["=", "type", "Class"], ["=", "title", "Profile::Module::Name"]]'), ( # Profile shortcut with parameter 'P:profile_name%param = value', ('["and", ["and", ["=", "type", "Class"], ["=", "title", "Profile::Profile_name"]], ' '["and", ["=", "type", "Class"], ["=", ["parameter", "param"], "value"]]]')), ( # Profile shortcut with field 'P:profile_name@field = value', ('["and", ["and", ["=", "type", "Class"], ["=", "title", "Profile::Profile_name"]], ' '["and", ["=", "type", "Class"], ["=", "field", "value"]]]')), ( # Role shortcut 'O:role_name', '["and", ["=", "type", "Class"], ["=", "title", "Role::Role_name"]]'), ( # Role shortcut path 'O:module::name', '["and", ["=", "type", "Class"], ["=", "title", "Role::Module::Name"]]'), ( # Role shortcut with parameter 'O:role_name%param = value', ('["and", ["and", ["=", "type", "Class"], ["=", "title", "Role::Role_name"]], ' '["and", ["=", "type", "Class"], ["=", ["parameter", "param"], "value"]]]')), ( # Role shortcut with field 'O:role_name@field = value', ('["and", ["and", ["=", "type", "Class"], ["=", "title", "Role::Role_name"]], ' '["and", ["=", "type", "Class"], ["=", "field", "value"]]]')), )) def test_add_category_resource(self, mocked_api_call, query, expected): """A resource query should add the proper query token to the current_group.""" expected = f'["extract", ["certname"], {expected}, ["group_by", "certname"]]' self.query.execute(query) mocked_api_call.assert_called_with(expected) @pytest.mark.parametrize('query, message', ( ( # Parameter and field 'R:resource%param@field', 'Resource key cannot contain both'), ( # Field and parameter 'R:resource@field%param', 'Resource key cannot contain both'), ( # Class shortcut with value 'C:class_name = value', 'The matching of a value is accepted only when using'), ( # Class shortcut with parameter and field 'C:class_name%param@field', 'Resource key cannot contain both'), ( # Class shortcut with field and parameter 'C:class_name@field%param', 'Resource key cannot contain both'), ( # Profile shortcut value 'P:profile_name = value', 'The matching of a value is accepted only when using'), ( # Profile shortcut with parameter and field 'P:profile_name%param@field', 'Resource key cannot contain both'), ( # Profile shortcut with field and parameter 'P:profile_name@field%param', 'Resource key cannot contain both'), ( # Role shortcut with value 'O:role_name = value', 'The matching of a value is accepted only when using'), ( # Role shortcut with parameter and field 'O:role_name%param@field', 'Resource key cannot contain both'), ( # Role shortcut with field and parameter 'O:role_name@field%param', 'Resource key cannot contain both'), )) def test_add_category_resource_raise(self, mocked_api_call, query, message): """A query with both a resource's parameter and field should raise InvalidQueryError.""" with pytest.raises(InvalidQueryError, match=message): self.query.execute(query) assert not mocked_api_call.called @pytest.mark.parametrize('query, expected', ( ( # No hosts 'host1!host1', ''), ( # Single host 'host', '["or", ["=", "certname", "host"]]'), ( # Multiple hosts 'host[1-2]', '["or", ["=", "certname", "host1"], ["=", "certname", "host2"]]'), ( # Negated query 'not host[1-2]', '["not", ["or", ["=", "certname", "host1"], ["=", "certname", "host2"]]]'), ( # Globbing hosts 'host1*.domain', r'["or", ["~", "certname", "^host1.*\\.domain$"]]'), )) def test_add_hosts(self, mocked_api_call, query, expected): """A host query should add the proper query token to the current_group.""" expected = f'["extract", ["certname"], {expected}, ["group_by", "certname"]]' self.query.execute(query) mocked_api_call.assert_called_with(expected) @pytest.mark.parametrize('query, operator, expected', ( ( # AND 'host1 and host2', 'and', '["and", ["or", ["=", "certname", "host1"]], ["or", ["=", "certname", "host2"]]]'), ( # OR 'host1 or host2', 'or', '["or", ["or", ["=", "certname", "host1"]], ["or", ["=", "certname", "host2"]]]'), ( # Multiple AND 'host1 and host2 and host3', 'and', ( '["and", ["or", ["=", "certname", "host1"]], ["or", ["=", "certname", "host2"]], ' '["or", ["=", "certname", "host3"]]]')), )) def test_operator(self, mocked_api_call, query, operator, expected): """A query with boolean operators should set the boolean property to the current group.""" expected = f'["extract", ["certname"], {expected}, ["group_by", "certname"]]' self.query.execute(query) assert self.query.current_group['bool'] == operator mocked_api_call.assert_called_with(expected) def test_and_or(self, mocked_api_call): """A query with 'and' and 'or' in the same group should raise InvalidQueryError.""" with pytest.raises(InvalidQueryError, match='boolean operator, current operator was'): self.query.execute('host1 and host2 or host3') assert not mocked_api_call.called @pytest.mark.parametrize('query, expected', ( ('nodes_host[1-2]', 'nodes_host[1-2]'), # Nodes ('R:Class = value', 'resources_host[1-2]'), # Resources ('I:facts.structured.property = value', 'inventory_host[1-2]'), # Inventory ('nodes_host1 or nodes_host2', 'nodes_host[1-2]'), # Nodes with AND ('(nodes_host1 or nodes_host2)', 'nodes_host[1-2]'), # Nodes with subgroup ('non_existent_host', None), # No match )) def test_endpoints(query_requests, query, expected): """Calling execute() with a query should go to the proper endpoint and return the list of hosts.""" hosts = query_requests[0].execute(query) assert hosts == nodeset(expected) assert query_requests[1].call_count == 1 def test_error(query_requests): """Calling execute() if the request fails it should raise the requests exception.""" with pytest.raises(HTTPError): query_requests[0].execute('invalid_query') assert query_requests[1].call_count == 1 def test_complex_query(query_requests): """Calling execute() with a complex query should return the expected structure.""" category = 'R' endpoint = query_requests[0].endpoints[category] query_requests[1].register_uri('POST', query_requests[0].url + endpoint + '?query=', status_code=200, json=[ {'certname': endpoint + '_host1', 'key': 'value1'}, {'certname': endpoint + '_host2', 'key': 'value2'}]) hosts = query_requests[0].execute('(resources_host1 or resources_host2) and R:Class = MyClass') assert hosts == nodeset('resources_host[1-2]') assert query_requests[1].call_count == 1 wikimedia-cumin-36f957f/cumin/tests/unit/conftest.py000066400000000000000000000032051476500461000225770ustar00rootroot00000000000000"""Pytest customization for unit tests.""" import pytest import requests_mock from cumin.backends import puppetdb def _requests_matcher_non_existent(request): return request.json() == { 'query': '["extract", ["certname"], ["or", ["=", "certname", "non_existent_host"]], ["group_by", "certname"]]' } def _requests_matcher_invalid(request): return request.json() == { 'query': '["extract", ["certname"], ["or", ["=", "certname", "invalid_query"]], ["group_by", "certname"]]' } @pytest.fixture() def mocked_requests(): """Set mocked requests fixture.""" with requests_mock.Mocker() as mocker: yield mocker @pytest.fixture() def query_requests(mocked_requests): # pylint: disable=redefined-outer-name """Set the requests library mock for each test.""" query = puppetdb.PuppetDBQuery({}) for endpoint in ('nodes', 'resources', 'inventory'): mocked_requests.register_uri( 'POST', query.url + endpoint, status_code=200, complete_qs=True, json=[ {'certname': endpoint + '_host1'}, {'certname': endpoint + '_host2'}, ]) # Register a requests response for a non matching query mocked_requests.register_uri( 'POST', query.url + query.endpoints['F'], status_code=200, json=[], complete_qs=True, additional_matcher=_requests_matcher_non_existent) # Register a requests response for an invalid query mocked_requests.register_uri( 'POST', query.url + query.endpoints['F'], status_code=400, complete_qs=True, additional_matcher=_requests_matcher_invalid) return query, mocked_requests wikimedia-cumin-36f957f/cumin/tests/unit/test_backends.py000066400000000000000000000004471476500461000235700ustar00rootroot00000000000000"""Abstract query tests.""" import pytest from cumin.backends import BaseQuery def test_base_query_instantiation(): """Class BaseQuery is not instantiable being an abstract class.""" with pytest.raises(TypeError): BaseQuery({}) # pylint: disable=abstract-class-instantiated wikimedia-cumin-36f957f/cumin/tests/unit/test_cli.py000066400000000000000000000251261476500461000225660ustar00rootroot00000000000000"""CLI tests.""" import argparse from logging import DEBUG, INFO from unittest import mock import pytest from cumin import cli, CuminError, LOGGING_TRACE_LEVEL_NUMBER, nodeset, transports from cumin.color import Colored # Command line arguments _ARGV = ['-c', 'doc/examples/config.yaml', '-d', '-m', 'sync', 'host', 'command1', 'command2'] def _validate_parsed_args(args, no_commands=False): """Validate that the parsed args have the proper values.""" assert args.debug assert args.config == 'doc/examples/config.yaml' assert args.hosts == 'host' if no_commands: assert args.dry_run else: assert args.commands == ['command1', 'command2'] def test_get_parser(): """Calling get_parser() should return a populated argparse.ArgumentParser object.""" parser = cli.get_parser() assert isinstance(parser, argparse.ArgumentParser) assert parser.prog == 'cumin' def test_parse_args_help(capsys): """Calling cumin with -h/--help should return its help message.""" with pytest.raises(SystemExit) as e: cli.parse_args(['-h']) out, _ = capsys.readouterr() assert e.value.code == 0 assert 'Cumin CLI - Automation and orchestration framework written in Python' in out def test_parse_args_ok(): """A standard set of command line parameters should be properly parsed into their respective variables.""" args = cli.parse_args(_ARGV) _validate_parsed_args(args) def test_parse_args_no_commands(): """If no commands are specified, dry-run mode should be implied.""" args = cli.parse_args(_ARGV[:-2]) _validate_parsed_args(args, no_commands=True) def test_parse_args_no_mode(): """If mode is not specified with multiple commands, parsing the args should raise a parser error.""" index = _ARGV.index('-m') with pytest.raises(SystemExit): cli.parse_args(_ARGV[:index] + _ARGV[index + 1:]) def test_parse_args_no_colors(): """If -n/--no-colors is specified, the colors should be globally disabled.""" assert not Colored.disabled cli.parse_args(_ARGV[:2] + ['-n'] + _ARGV[3:]) assert Colored.disabled Colored.disabled = False # Reset it def test_target_batch_size(): """Calling target_batch_size() should properly parse integer values.""" assert cli.target_batch_size('1') == {'value': 1, 'ratio': None} assert cli.target_batch_size('100') == {'value': 100, 'ratio': None} @pytest.mark.parametrize('percentage', (('0%', 0.0), ('50%', 0.5), ('100%', 1.0))) def test_target_batch_size_perc(percentage): """Calling target_batch_size() with a valid percentage should properly parse it.""" assert cli.target_batch_size(percentage[0]) == {'value': None, 'ratio': percentage[1]} @pytest.mark.parametrize('percentage', ('-1%', '101%')) def test_target_batch_size_perc_ko(percentage): """Calling target_batch_size() with invalid percentage should raise argparse.ArgumentTypeError.""" with pytest.raises(argparse.ArgumentTypeError, match='not a valid percentage'): cli.target_batch_size(percentage) @pytest.mark.parametrize('value', ('0', '-1')) def test_target_batch_size_val_ko(value): """Calling target_batch_size() with invalid value should raise argparse.ArgumentTypeError.""" with pytest.raises(argparse.ArgumentTypeError, match='is not a valid value'): cli.target_batch_size(value) def test_get_running_user(): """Unknown user should raise CuminError and a proper user should be detected.""" env = {'USER': 'root', 'SUDO_USER': None} with mock.patch('os.getenv', env.get): with pytest.raises(CuminError, match='Unable to determine real user'): cli.get_running_user() env = {'USER': 'user', 'SUDO_USER': None} with mock.patch('os.getenv', env.get): assert cli.get_running_user() == 'user' env = {'USER': 'root', 'SUDO_USER': 'user'} with mock.patch('os.getenv', env.get): assert cli.get_running_user() == 'user' @mock.patch('cumin.cli.os.path.exists') @mock.patch('cumin.cli.os.makedirs') @mock.patch('cumin.cli.RotatingFileHandler') @mock.patch('cumin.cli.logging.getLogger') def test_setup_logging(mocked_get_logger, mocked_file_handler, mocked_os_makedirs, mocked_os_path_exists): """Calling setup_logging() should properly setup the logger.""" mocked_os_path_exists.return_value = False cli.setup_logging('/path/to/filename.yaml') assert mock.call().setLevel(INFO) in mocked_get_logger.mock_calls assert mocked_file_handler.called assert mocked_os_makedirs.called assert mocked_os_path_exists.called mocked_file_handler.reset_mock() mocked_os_makedirs.reset_mock() mocked_os_path_exists.reset_mock() mocked_os_path_exists.side_effect = FileNotFoundError cli.setup_logging('filename.yaml') assert mock.call().setLevel(INFO) in mocked_get_logger.mock_calls assert mocked_file_handler.called assert not mocked_os_makedirs.called assert not mocked_os_path_exists.called mocked_os_path_exists.return_value = True cli.setup_logging('filename.yaml', debug=True) assert mock.call().setLevel(DEBUG) in mocked_get_logger.mock_calls mocked_os_path_exists.return_value = True cli.setup_logging('filename.yaml', trace=True) assert mock.call().setLevel(LOGGING_TRACE_LEVEL_NUMBER) in mocked_get_logger.mock_calls @mock.patch('cumin.cli.stderr') @mock.patch('builtins.input') @mock.patch('cumin.cli.sys.stdout.isatty') @mock.patch('cumin.cli.logger') def test_sigint_handler(logging, isatty, mocked_input, stderr): # pylint: disable=unused-argument """Calling the SIGINT handler should raise KeyboardInterrupt or not based on tty and answer.""" # Signal handler called without a tty isatty.return_value = False with pytest.raises(cli.KeyboardInterruptError): cli.sigint_handler(1, None) # Signal handler called with a tty isatty.return_value = True with pytest.raises(cli.KeyboardInterruptError): cli.sigint_handler(1, None) # # Signal handler called with a tty, answered 'y' # isatty.return_value = True # mocked_input.return_value = 'y' # with pytest.raises(cli.KeyboardInterruptError): # cli.sigint_handler(1, None) # # # Signal handler called with a tty, answered 'n' # isatty.return_value = True # mocked_input.return_value = 'n' # assert cli.sigint_handler(1, None) is None # # # Signal handler called with a tty, answered 'invalid_answer' # isatty.return_value = True # mocked_input.return_value = 'invalid_answer' # with pytest.raises(cli.KeyboardInterruptError): # cli.sigint_handler(1, None) # # # Signal handler called with a tty, empty answer # isatty.return_value = True # mocked_input.return_value = '' # with pytest.raises(cli.KeyboardInterruptError): # cli.sigint_handler(1, None) @mock.patch('cumin.cli.tqdm') def test_stderr(tqdm): """Calling stderr() should call tqdm.write().""" cli.stderr('message') assert tqdm.write.called @pytest.mark.parametrize('query, input_value', ( ('host1', '1'), ('host[1000-2000]', '1001'), )) @mock.patch('cumin.cli.stderr') @mock.patch('builtins.input') @mock.patch('cumin.cli.sys.stdout.isatty') def test_get_hosts_ok(isatty, mocked_input, stderr, query, input_value): """Calling get_hosts() should query the backend and return the list of hosts asking for confirmation in a TTY.""" args = cli.parse_args([query, 'command1']) config = {'backend': 'direct', 'default_backend': 'direct'} isatty.return_value = True mocked_input.return_value = input_value assert cli.get_hosts(args, config) == nodeset(query) assert stderr.called @pytest.mark.parametrize('query, input_value', ( ('host1', 'q'), ('host1', 'invalid_answer'), ('host1', ''), ('host1', '2'), )) @mock.patch('cumin.cli.stderr') @mock.patch('builtins.input') @mock.patch('cumin.cli.sys.stdout.isatty') def test_get_hosts_raise(isatty, mocked_input, stderr, query, input_value): """Calling get_hosts() should query the backend and raise KeyboardInterruptError without a confirmation.""" args = cli.parse_args([query, 'command1']) config = {'backend': 'direct', 'default_backend': 'direct'} isatty.return_value = True mocked_input.return_value = input_value with pytest.raises(cli.KeyboardInterruptError): cli.get_hosts(args, config) assert stderr.called @mock.patch('cumin.cli.stderr') @mock.patch('cumin.cli.sys.stdout.isatty') def test_get_hosts_no_tty_ko(isatty, stderr): """Calling get_hosts() without a TTY should raise CuminError if --dry-run or --force are not specified.""" args = cli.parse_args(['D{host1}', 'command1']) config = {'backend': 'direct'} isatty.return_value = False with pytest.raises(CuminError, match='Not in a TTY but neither DRY-RUN nor FORCE mode were specified'): cli.get_hosts(args, config) assert stderr.called @mock.patch('cumin.cli.stderr') @mock.patch('cumin.cli.sys.stdout.isatty') def test_get_hosts_no_tty_dry_run(isatty, stderr): """Calling get_hosts() with or without a TTY with --dry-run should return an empty list.""" args = cli.parse_args(['--dry-run', 'D{host1}', 'command1']) config = {'backend': 'direct'} assert cli.get_hosts(args, config) == [] isatty.return_value = True assert cli.get_hosts(args, config) == [] assert stderr.called @mock.patch('cumin.cli.stderr') @mock.patch('cumin.cli.sys.stdout.isatty') def test_get_hosts_no_tty_force(isatty, stderr): """Calling get_hosts() with or without a TTY with --force should return the list of hosts.""" args = cli.parse_args(['--force', 'D{host1}', 'command1']) config = {'backend': 'direct'} assert cli.get_hosts(args, config) == nodeset('host1') isatty.return_value = True assert cli.get_hosts(args, config) == nodeset('host1') assert stderr.called @mock.patch('cumin.cli.cumin.transport.Transport') @mock.patch('cumin.cli.stderr') def test_run(stderr, transport): """Calling run() should query the hosts and execute the commands on the transport.""" args = cli.parse_args(['--force', 'D{host1}', 'command1']) config = {'backend': 'direct', 'transport': 'clustershell'} cli.run(args, config) assert transport.new.call_args[0][0] is config assert isinstance(transport.new.call_args[0][1], transports.Target) assert stderr.called def test_validate_config_valid(): """A valid config should be validated without raising exception.""" cli.validate_config({'log_file': '/var/log/cumin/cumin.log'}) def test_validate_config_invalid(): """An invalid config should raise CuminError.""" with pytest.raises(CuminError, match='Missing required parameter'): cli.validate_config({}) wikimedia-cumin-36f957f/cumin/tests/unit/test_color.py000066400000000000000000000033571476500461000231370ustar00rootroot00000000000000"""Color tests.""" from unittest import mock import pytest from cumin.color import Colored def test_red(): """It should return the message enclosed in ASCII red color code.""" assert Colored.red('message') == '\x1b[31mmessage\x1b[39m' def test_green(): """It should return the message enclosed in ASCII green color code.""" assert Colored.green('message') == '\x1b[32mmessage\x1b[39m' def test_yellow(): """It should return the message enclosed in ASCII yellow color code.""" assert Colored.yellow('message') == '\x1b[33mmessage\x1b[39m' def test_blue(): """It should return the message enclosed in ASCII blue color code.""" assert Colored.blue('message') == '\x1b[34mmessage\x1b[39m' def test_cyan(): """It should return the message enclosed in ASCII cyan color code.""" assert Colored.cyan('message') == '\x1b[36mmessage\x1b[39m' def test_wrong_case(): """It should raise AttributeError if called with the wrong case.""" with pytest.raises(AttributeError, match="'Colored' object has no attribute 'Red'"): Colored.Red('') def test_non_existent(): """It should raise AttributeError if called with a non existent color.""" with pytest.raises(AttributeError, match="'Colored' object has no attribute 'missing'"): Colored.missing('') def test_emtpy(): """It should return an empty string if the object is empty.""" assert Colored.red('') == '' @mock.patch('cumin.color.Colored.disabled', new_callable=mock.PropertyMock) def test_disabled(mocked_colored_disabled): """It should return the message untouched if coloration is disabled.""" mocked_colored_disabled.return_value = True assert Colored.red('message') == 'message' assert mocked_colored_disabled.called wikimedia-cumin-36f957f/cumin/tests/unit/test_grammar.py000066400000000000000000000107501476500461000234420ustar00rootroot00000000000000"""Grammar tests.""" import os import sys from unittest import mock import pytest from cumin import CuminError, grammar from cumin.tests import get_fixture from cumin.tests.unit.backends.external import ExternalBackendQuery REGISTERED_BACKENDS = grammar.get_registered_backends() def test_valid_strings(): """Run quick pyparsing test over valid grammar strings.""" results = grammar.grammar(REGISTERED_BACKENDS.keys()).runTests( get_fixture(os.path.join('grammar', 'valid_grammars.txt'), as_string=True)) assert results[0] def test_invalid_strings(): """Run quick pyparsing test over invalid grammar strings.""" results = grammar.grammar(REGISTERED_BACKENDS.keys()).runTests( get_fixture(os.path.join('grammar', 'invalid_grammars.txt'), as_string=True), failureTests=True) assert results[0] # Built-in backends registration tests @mock.patch('cumin.grammar.pkgutil.iter_modules') def test_duplicate_backend(mocked_iter_modules): """Trying to register a backend with the same key of another should raise CuminError.""" backend = mock.MagicMock() backend.GRAMMAR_PREFIX = 'D' sys.modules['cumin.backends.test_backend'] = backend mocked_iter_modules.return_value = ((None, name, False) for name in ('direct', 'puppetdb', 'test_backend')) with pytest.raises(CuminError, match='Unable to register backend'): grammar.get_registered_backends() del sys.modules['cumin.backends.test_backend'] @mock.patch('cumin.grammar.importlib.import_module') def test_backend_import_error(mocked_import_modules): """If an internal backend raises ImportError because of missing dependencies, it should be skipped.""" mocked_import_modules.side_effect = ImportError backends = grammar.get_registered_backends() assert not backends assert isinstance(backends, dict) @mock.patch('cumin.grammar.importlib.import_module') def test_backend_import_error_ext(mocked_import_modules): """If an external backend raises ImportError because of missing dependencies, it should raise CuminError.""" mocked_import_modules.side_effect = ImportError with pytest.raises(CuminError, match='Unable to import backend'): grammar.get_registered_backends(external=['cumin.tests.unit.backends.external.ok']) @mock.patch('cumin.grammar.pkgutil.iter_modules') def test_import_error_backend(mocked_iter_modules): """Trying to register a backend that raises ImportError should silently skip it (missing optional dependencies).""" # Using a non-existent backend as it will raise ImportError like an existing backend with missing dependencies. mocked_iter_modules.return_value = ((None, name, False) for name in ('direct', 'puppetdb', 'non_existent')) backends = grammar.get_registered_backends() assert len(backends.keys()) == 2 assert sorted(backends.keys()) == ['D', 'P'] # External backends registration tests def test_register_ok(): """An external backend should be properly registered.""" backends = grammar.get_registered_backends(external=['cumin.tests.unit.backends.external.ok']) assert '_Z' in backends assert backends['_Z'].keyword == '_Z' assert backends['_Z'].name == 'ok' assert backends['_Z'].cls == ExternalBackendQuery def test_register_missing_prefix(): """Registering an external backend missing the GRAMMAR_PREFIX should raise CuminError.""" with pytest.raises(CuminError, match='GRAMMAR_PREFIX module attribute not found'): grammar.get_registered_backends(external=['cumin.tests.unit.backends.external.missing_grammar_prefix']) def test_register_duplicate_prefix(): """Registering an external backend with an already registered GRAMMAR_PREFIX should raise CuminError.""" with pytest.raises(CuminError, match='already registered'): grammar.get_registered_backends(external=['cumin.tests.unit.backends.external.duplicate_prefix']) def test_register_missing_class(): """Registering an external backend missing the query_class should raise CuminError.""" with pytest.raises(CuminError, match='query_class module attribute not found'): grammar.get_registered_backends(external=['cumin.tests.unit.backends.external.missing_query_class']) def test_register_inheritance(): """Registering an external backend with a query_class with the wrong inheritance should raise CuminError.""" with pytest.raises(CuminError, match='query_class module attribute is not a subclass'): grammar.get_registered_backends(external=['cumin.tests.unit.backends.external.wrong_inheritance']) wikimedia-cumin-36f957f/cumin/tests/unit/test_init.py000066400000000000000000000251321476500461000227570ustar00rootroot00000000000000"""Cumin package tests.""" import importlib import logging import os from subprocess import CalledProcessError, CompletedProcess from unittest import mock import pytest from ClusterShell.NodeSet import NodeSet import cumin from cumin.tests import get_fixture_path def test_config_class_valid(): """Should return the config. Multiple Config with the same path should return the same object.""" config_file = get_fixture_path(os.path.join('config', 'valid', 'config.yaml')) config1 = cumin.Config(config=config_file) assert 'log_file' in config1 config2 = cumin.Config(config=config_file) assert config1 is config2 def test_config_class_empty(): """An empty dictionary is returned if the configuration is empty.""" config = cumin.Config(config=get_fixture_path(os.path.join('config', 'empty', 'config.yaml'))) assert not config assert isinstance(config, dict) def test_config_class_invalid(): """A CuminError is raised if the configuration cannot be parsed.""" with pytest.raises(cumin.CuminError, match='Unable to parse configuration file'): cumin.Config(config=get_fixture_path(os.path.join('config', 'invalid', 'config.yaml'))) def test_config_class_valid_with_aliases(): """Should return the config including the aliases.""" config = cumin.Config(config=get_fixture_path(os.path.join('config', 'valid_with_aliases', 'config.yaml'))) assert 'log_file' in config assert 'aliases' in config assert 'role1' in config['aliases'] assert config['aliases']['role1'] == 'P{R:Class = Role::Role1}' assert 'group1' in config['aliases'] assert config['aliases']['group1'] == 'D{host10[10-22].example.org}' def test_config_class_empty_aliases(): """The configuration is loaded also if the aliases file is empty.""" config = cumin.Config(config=get_fixture_path(os.path.join('config', 'valid_with_empty_aliases', 'config.yaml'))) assert 'log_file' in config assert 'aliases' in config assert config['aliases'] == {} def test_config_class_invalid_aliases(): """A CuminError is raised if one of the backend aliases is invalid.""" with pytest.raises(cumin.CuminError, match='Unable to parse configuration file'): cumin.Config(config=get_fixture_path(os.path.join('config', 'valid_with_invalid_aliases', 'config.yaml'))) def test_parse_config_ok(): """The configuration file is properly parsed and accessible.""" config = cumin.parse_config(get_fixture_path(os.path.join('config', 'valid', 'config.yaml'))) assert 'log_file' in config def test_parse_config_non_existent(): """A CuminError is raised if the configuration file is not available.""" with pytest.raises(cumin.CuminError, match='Unable to read configuration file'): cumin.parse_config('not_existent_config.yaml') def test_parse_config_invalid(): """A CuminError is raised if the configuration cannot be parsed.""" with pytest.raises(cumin.CuminError, match='Unable to parse configuration file'): cumin.parse_config(get_fixture_path(os.path.join('config', 'invalid', 'config.yaml'))) def test_parse_config_empty(): """An empty dictionary is returned if the configuration is empty.""" config = cumin.parse_config(get_fixture_path(os.path.join('config', 'empty', 'config.yaml'))) assert config == {} def test_trace_logging_level_conflict(): """If the logging level for trace is already registered, should raise CuminError.""" importlib.reload(logging) # Avoid conflict given the singleton nature of this module logging.addLevelName(cumin.LOGGING_TRACE_LEVEL_NUMBER, 'CONFLICT') match = 'Unable to set custom logging for trace' try: # pytest.raises doesn't catch the reload exception importlib.reload(cumin) except cumin.CuminError as e: assert str(e).startswith(match) else: raise AssertionError("Failed: DID NOT RAISE {exc} matching '{match}'".format( exc=cumin.CuminError, match=match)) def test_trace_logging_level_existing_same(): """If the custom logging level is registered on the same level, it should use it and add a trace method.""" importlib.reload(logging) # Avoid conflict given the singleton nature of this module logging.addLevelName(cumin.LOGGING_TRACE_LEVEL_NUMBER, cumin.LOGGING_TRACE_LEVEL_NAME) assert not hasattr(logging.Logger, 'trace') importlib.reload(cumin) assert logging.getLevelName(cumin.LOGGING_TRACE_LEVEL_NUMBER) == cumin.LOGGING_TRACE_LEVEL_NAME assert logging.getLevelName(cumin.LOGGING_TRACE_LEVEL_NAME) == cumin.LOGGING_TRACE_LEVEL_NUMBER assert hasattr(logging.Logger, 'trace') def test_trace_logging_level_existing_different(): """If the custom logging level is registered on a different level, it should use it and add a trace method.""" importlib.reload(logging) # Avoid conflict given the singleton nature of this module logging.addLevelName(cumin.LOGGING_TRACE_LEVEL_NUMBER - 1, cumin.LOGGING_TRACE_LEVEL_NAME) assert not hasattr(logging.Logger, 'trace') importlib.reload(cumin) assert logging.getLevelName(cumin.LOGGING_TRACE_LEVEL_NAME) == cumin.LOGGING_TRACE_LEVEL_NUMBER - 1 assert logging.getLevelName(cumin.LOGGING_TRACE_LEVEL_NUMBER) != cumin.LOGGING_TRACE_LEVEL_NAME assert hasattr(logging.Logger, 'trace') def test_trace_logging_method_existing(): """If there is already a trace method registered, it should use it without problems adding the level.""" importlib.reload(logging) # Avoid conflict given the singleton nature of this module logging.Logger.trace = cumin.trace importlib.reload(cumin) assert logging.getLevelName(cumin.LOGGING_TRACE_LEVEL_NUMBER) == cumin.LOGGING_TRACE_LEVEL_NAME assert logging.getLevelName(cumin.LOGGING_TRACE_LEVEL_NAME) == cumin.LOGGING_TRACE_LEVEL_NUMBER assert hasattr(logging.Logger, 'trace') def test_nodeset(): """Calling nodeset() should return an instance of ClusterShell NodeSet with no resolver.""" nodeset = cumin.nodeset('node[1-2]') assert isinstance(nodeset, NodeSet) assert nodeset == NodeSet('node[1-2]') assert nodeset._resolver is None # pylint: disable=protected-access def test_nodeset_empty(): """Calling nodeset() without parameter should return an instance of ClusterShell NodeSet with no resolver.""" nodeset = cumin.nodeset() assert isinstance(nodeset, NodeSet) assert nodeset == NodeSet() assert nodeset._resolver is None # pylint: disable=protected-access def test_nodeset_fromlist(): """Calling nodeset_fromlist() should return an instance of ClusterShell NodeSet with no resolver.""" nodeset = cumin.nodeset_fromlist(['node1', 'node2']) assert isinstance(nodeset, NodeSet) assert nodeset == NodeSet('node[1-2]') assert nodeset._resolver is None # pylint: disable=protected-access def test_nodeset_fromlist_empty(): """Calling nodeset_fromlist() with empty list should return an instance of ClusterShell NodeSet with no resolver.""" nodeset = cumin.nodeset_fromlist([]) assert isinstance(nodeset, NodeSet) assert nodeset == NodeSet() assert nodeset._resolver is None # pylint: disable=protected-access @pytest.mark.parametrize('config', ( {}, {'kerberos': {}}, {'kerberos': {'ensure_ticket': False}}, )) def test_ensure_kerberos_ticket_config_disabled(config): """It should return without raising an exception if it's disabled from the config.""" cumin.ensure_kerberos_ticket(config) @pytest.mark.parametrize('config', ( {'kerberos': {'ensure_ticket': True}}, {'kerberos': {'ensure_ticket': True, 'ensure_ticket_root': False}}, )) @mock.patch('cumin.os.geteuid') def test_ensure_kerberos_ticket_root_excluded(mocked_geteuid, config): """It should not check the Kerberos ticket when running as root if ensure_ticket_root is not set.""" mocked_geteuid.return_value = 0 cumin.ensure_kerberos_ticket(config) mocked_geteuid.assert_called_once_with() @pytest.mark.parametrize('uid, ensure_ticket_root', ( (1000, False), (0, True), )) @mock.patch('cumin.os.access') @mock.patch('cumin.os.geteuid') def test_ensure_kerberos_ticket_no_klist(mocked_geteuid, mocked_os_access, uid, ensure_ticket_root): """It should raise CuminError if the klist executable is not found.""" mocked_geteuid.return_value = uid mocked_os_access.return_value = False with pytest.raises(cumin.CuminError, match='klist executable was not found'): cumin.ensure_kerberos_ticket({'kerberos': {'ensure_ticket': True, 'ensure_ticket_root': ensure_ticket_root}}) if ensure_ticket_root: assert not mocked_geteuid.called else: mocked_geteuid.assert_called_once_with() mocked_os_access.assert_called_once_with(cumin.KERBEROS_KLIST, os.X_OK) @pytest.mark.parametrize('uid, ensure_ticket_root', ( (1000, False), (0, True), )) @mock.patch('cumin.subprocess.run') @mock.patch('cumin.os.access') @mock.patch('cumin.os.geteuid') def test_ensure_kerberos_ticket_no_ticket(mocked_geteuid, mocked_os_access, mocked_run, uid, ensure_ticket_root): """It should raise CuminError if there is no valid Kerberos ticket.""" run_command = [cumin.KERBEROS_KLIST, '-s'] mocked_run.side_effect = CalledProcessError(1, run_command) mocked_geteuid.return_value = uid mocked_os_access.return_value = True with pytest.raises(cumin.CuminError, match='but no active Kerberos ticket was found'): cumin.ensure_kerberos_ticket({'kerberos': {'ensure_ticket': True, 'ensure_ticket_root': ensure_ticket_root}}) if ensure_ticket_root: assert not mocked_geteuid.called else: mocked_geteuid.assert_called_once_with() mocked_os_access.assert_called_once_with(cumin.KERBEROS_KLIST, os.X_OK) mocked_run.assert_called_once_with(run_command, check=True) @pytest.mark.parametrize('uid, ensure_ticket_root', ( (1000, False), (0, True), )) @mock.patch('cumin.subprocess.run') @mock.patch('cumin.os.access') @mock.patch('cumin.os.geteuid') def test_ensure_kerberos_ticket_valid(mocked_geteuid, mocked_os_access, mocked_run, uid, ensure_ticket_root): """It should return without raising any error if there is a valid Kerberos ticket.""" run_command = [cumin.KERBEROS_KLIST, '-s'] mocked_run.return_value = CompletedProcess(run_command, 0) mocked_geteuid.return_value = uid mocked_os_access.return_value = True cumin.ensure_kerberos_ticket({'kerberos': {'ensure_ticket': True, 'ensure_ticket_root': ensure_ticket_root}}) if ensure_ticket_root: assert not mocked_geteuid.called else: mocked_geteuid.assert_called_once_with() mocked_os_access.assert_called_once_with(cumin.KERBEROS_KLIST, os.X_OK) mocked_run.assert_called_once_with(run_command, check=True) wikimedia-cumin-36f957f/cumin/tests/unit/test_query.py000066400000000000000000000117131476500461000231610ustar00rootroot00000000000000"""Query handling tests.""" import pytest from cumin import backends, nodeset from cumin.query import Query def test_execute_valid_global(): """Executing a valid query should return the matching hosts.""" query = Query({}) hosts = query.execute('D{(host1 or host2) and host[1-5]}') assert hosts == nodeset('host[1-2]') def test_execute_global_or(): """Executing an 'or' between two queries should return the union of the hosts.""" query = Query({}) hosts = query.execute('D{host1} or D{host2}') assert hosts == nodeset('host[1-2]') def test_execute_global_and(): """Executing an 'and' between two queries should return the intersection of the hosts.""" query = Query({}) hosts = query.execute('D{host[1-5]} and D{host2}') assert hosts == nodeset('host2') def test_execute_global_and_not(): """Executing an 'and not' between two queries should return the difference of the hosts.""" query = Query({}) hosts = query.execute('D{host[1-5]} and not D{host2}') assert hosts == nodeset('host[1,3-5]') def test_execute_global_xor(): """Executing a 'xor' between two queries should return all the hosts that are in only in one of the queries.""" query = Query({}) hosts = query.execute('D{host[1-5]} xor D{host[3-7]}') assert hosts == nodeset('host[1-2,6-7]') def test_execute_valid_global_with_aliases(): """Executing a valid query with aliases should return the matching hosts.""" query = Query({'aliases': {'group1': 'D{host1 or host2}'}}) hosts = query.execute('A:group1') assert hosts == nodeset('host[1-2]') def test_execute_valid_global_with_nested_aliases(): """Executing a valid query with nested aliases should return the matching hosts.""" query = Query({ 'aliases': { 'group1': 'D{host1 or host2}', 'group2': 'D{host3 or host4}', 'all': 'A:group1 or A:group2', }}) hosts = query.execute('A:all') assert hosts == nodeset('host[1-4]') def test_execute_missing_alias(): """Executing a valid query with a missing alias should raise InvalidQueryError.""" query = Query({}) with pytest.raises(backends.InvalidQueryError, match='Unable to find alias replacement for'): query.execute('A:non_existent_group') query = Query({'aliases': {}}) with pytest.raises(backends.InvalidQueryError, match='Unable to find alias replacement for'): query.execute('A:non_existent_group') def test_execute_invalid_global(): """Executing a query with an invalid syntax should raise InvalidQueryError.""" query = Query({}) with pytest.raises(backends.InvalidQueryError, match='with the global grammar'): query.execute('invalid syntax') def test_execute_subgroup(): """Executing a query with a single subgroup should return the matching hosts.""" query = Query({}) hosts = query.execute('(D{host1})') assert hosts == nodeset('host1') def test_execute_subgroups(): """Executing a query with multiple subgroups should return the matching hosts.""" query = Query({}) hosts = query.execute('(D{host1} or D{host2}) and not (D{host1})') assert hosts == nodeset('host2') def test_execute_missing_default_backend(): """Executing a valid query with a missing default backend should raise InvalidQueryError.""" query = Query({'default_backend': 'non_existent_backend'}) with pytest.raises(backends.InvalidQueryError, match='is not registered'): query.execute('any_query') def test_execute_valid_default_backend(): """Executing a default backend valid query should return the matching hosts.""" query = Query({'default_backend': 'direct'}) hosts = query.execute('host1 or host2') assert hosts == nodeset('host[1-2]') def test_execute_invalid_default_valid_global(): """Executing a global grammar valid query in presence of a default backend should return the matching hosts.""" query = Query({'default_backend': 'direct'}) hosts = query.execute('D{host1 or host2}') assert hosts == nodeset('host[1-2]') def test_execute_invalid_default_invalid_global(): """Executing a query invalid for the default backend and the global grammar should raise InvalidQueryError.""" query = Query({'default_backend': 'direct'}) with pytest.raises(backends.InvalidQueryError, match='neither with the default backen'): query.execute('invalid syntax') def test_execute_complex_global(): """Executing a valid complex query should return the matching hosts.""" query = Query({}) hosts = query.execute( '(D{(host1 or host2) and host[1-5]}) or ((D{host[100-150]} and not D{host1[20-30]}) and D{host1[01,15,30]})') assert hosts == nodeset('host[1-2,101,115]') def test_execute_complex_empty_first_result(): """If the first block of a multi-part query returns no hosts it should not fail and return hosts accordingly.""" query = Query({}) hosts = query.execute('(D{host1} and D{host2}) and D{host[1-5]}') assert hosts == nodeset() wikimedia-cumin-36f957f/cumin/tests/unit/test_transport.py000066400000000000000000000027201476500461000240460ustar00rootroot00000000000000"""Transport class tests.""" import pkgutil from unittest import mock import pytest from cumin import CuminError, transports from cumin.transport import Transport def test_missing_transport(): """Not passing a transport should raise CuminError.""" with pytest.raises(CuminError, match=r"Missing required parameter 'transport'"): Transport.new({}, transports.Target(['host1'])) def test_invalid_transport(): """Passing an invalid transport should raise CuminError.""" with pytest.raises(CuminError, match=r"No module named 'cumin\.transports\.non_existent_transport'"): Transport.new({'transport': 'non_existent_transport'}, transports.Target(['host1'])) def test_missing_worker_class(): """Passing a transport without a defined worker_class should raise CuminError.""" module = mock.MagicMock() del module.worker_class with mock.patch('importlib.import_module', lambda _: module): with pytest.raises(CuminError, match=r'worker_class'): Transport.new({'transport': 'invalid_transport'}, transports.Target(['host1'])) @pytest.mark.parametrize('transport', [name for _, name, ispkg in pkgutil.iter_modules(transports.__path__) if not ispkg]) def test_valid_transport(transport): """Passing a valid transport should return an instance of BaseWorker.""" assert isinstance(Transport.new({'transport': transport}, transports.Target(['host1'])), transports.BaseWorker) wikimedia-cumin-36f957f/cumin/tests/unit/transports/000077500000000000000000000000001476500461000226175ustar00rootroot00000000000000wikimedia-cumin-36f957f/cumin/tests/unit/transports/__init__.py000066400000000000000000000000401476500461000247220ustar00rootroot00000000000000"""Transport specific tests.""" wikimedia-cumin-36f957f/cumin/tests/unit/transports/test_clustershell.py000066400000000000000000000537621476500461000267560ustar00rootroot00000000000000"""ClusterShell transport tests.""" # pylint: disable=no-member,protected-access,attribute-defined-outside-init from unittest import mock import pytest from cumin import CuminError, nodeset from cumin.transports import (BaseExecutionProgress, BaseWorker, clustershell, Command, State, Target, TqdmProgressBars, WorkerError) from cumin.transports.clustershell import Node, NullReporter, TqdmReporter def test_node_class_instantiation(): """Default values should be set when a Node instance is created.""" node = clustershell.Node('name', [Command('command1'), Command('command2')]) assert node.running_command_index == -1 assert isinstance(node.state, State) @mock.patch('cumin.transports.clustershell.Task.task_self') def test_worker_class(task_self): """An instance of worker_class should be an instance of BaseWorker.""" worker = clustershell.worker_class({}, Target(nodeset('node1'))) assert isinstance(worker, BaseWorker) task_self.assert_called_once_with() def test_null_reporter_does_nothing(capsys): """Tests that the NullReporter does nothing (and does not raise any error).""" reporter = NullReporter() # pylint: disable=abstract-class-instantiated message = "a message" command = "a command" commands = [Command("first command"), Command("second command")] node1 = Node("node1", commands) node2 = Node("node2", commands) nodes = [node1, node2] reporter.command_completed() reporter.command_output(iter(["node1", "node2"]), command) reporter.command_header(command) reporter.message_element(message) reporter.global_timeout_nodes(nodes, 2) reporter.failed_nodes(nodes, 2, commands) reporter.success_nodes(command, 2, 1.0, 2, 2, 1.0, nodes) out, err = capsys.readouterr() assert out == '' assert err == '' class TestClusterShellWorker: """ClusterShell backend worker test class.""" @mock.patch('cumin.transports.clustershell.Task.task_self') def setup_method(self, _, task_self): """Initialize default properties and instances.""" self.config = { 'clustershell': { 'ssh_options': ['-o StrictHostKeyChecking=no', '-o BatchMode=yes'], 'fanout': 3}} self.target = Target(nodeset('node[1-2]')) self.worker = clustershell.worker_class(self.config, self.target) self.commands = [Command('command1'), Command('command2', ok_codes=[0, 100], timeout=5)] self.task_self = task_self # Mock default handlers clustershell.DEFAULT_HANDLERS = { 'sync': mock.MagicMock(spec_set=clustershell.SyncEventHandler), 'async': mock.MagicMock(spec_set=clustershell.AsyncEventHandler)} # Initialize the worker self.worker.commands = self.commands @mock.patch('cumin.transports.clustershell.Task.task_self') def test_instantiation(self, task_self): """An instance of ClusterShellWorker should be an instance of BaseWorker and initialize ClusterShell.""" worker = clustershell.ClusterShellWorker(self.config, self.target) assert isinstance(worker, BaseWorker) task_self.assert_called_once_with() worker.task.set_info.assert_has_calls( [mock.call('fanout', 3), mock.call('ssh_options', '-o StrictHostKeyChecking=no -o BatchMode=yes')], any_order=True) def test_execute_default_sync_handler(self): """Calling execute() in sync mode without event handler should use the default sync event handler.""" self.worker.handler = 'sync' self.worker.execute() args, kwargs = self.worker.task.shell.call_args assert args == ('command1',) assert kwargs['nodes'] == self.target.first_batch assert kwargs['handler'] == self.worker._handler_instance assert clustershell.DEFAULT_HANDLERS['sync'].called def test_execute_different_reporter(self): """Calling execute() after changing the reporter should use the new reporter.""" self.worker.handler = 'sync' assert self.worker.reporter is TqdmReporter # Current default self.worker.reporter = NullReporter self.worker.execute() assert self.worker.reporter is NullReporter # TODO: improve assertions to check additional things def test_execute_default_async_handler(self): """Calling execute() in async mode without event handler should use the default async event handler.""" self.worker.handler = 'async' self.worker.execute() args, kwargs = self.worker.task.shell.call_args assert args == ('command1',) assert kwargs['nodes'] == self.target.first_batch assert kwargs['handler'] == self.worker._handler_instance assert clustershell.DEFAULT_HANDLERS['async'].called def test_execute_timeout(self): """Calling execute() and let the global timeout expire should call on_timeout.""" self.worker.task.run = mock.Mock(side_effect=clustershell.Task.TimeoutError) self.worker.handler = 'sync' self.worker.execute() self.worker._handler_instance.on_timeout.assert_called_once_with(self.worker.task) def test_execute_custom_handler(self): """Calling execute() using a custom handler should call ClusterShell task with the custom event handler.""" self.worker.handler = ConcreteBaseEventHandler self.worker.execute() assert isinstance(self.worker._handler_instance, ConcreteBaseEventHandler) args, kwargs = self.worker.task.shell.call_args assert args == ('command1',) assert kwargs['nodes'] == self.target.first_batch assert kwargs['handler'] == self.worker._handler_instance def test_execute_no_commands(self): """Calling execute() without commands should raise WorkerError.""" self.worker.handler = ConcreteBaseEventHandler self.worker.commands = None with pytest.raises(WorkerError, match=r'No commands provided\.'): self.worker.execute() assert not self.worker.task.shell.called def test_execute_one_command_no_mode(self): """Calling execute() with only one command without mode should raise WorkerError.""" self.worker.commands = [self.commands[0]] with pytest.raises(WorkerError, match=r'An EventHandler is mandatory\.'): self.worker.execute() assert not self.worker.task.shell.called def test_execute_wrong_mode(self): """Calling execute() without setting the mode with multiple commands should raise CuminError.""" with pytest.raises(CuminError, match=r'An EventHandler is mandatory\.'): self.worker.execute() def test_execute_batch_size(self): """Calling execute() with a batch_size specified should run in batches.""" self.worker.commands = [self.commands[0]] self.worker.handler = 'sync' self.worker.batch_size = 1 self.worker.execute() args, kwargs = self.worker.task.shell.call_args assert args == ('command1',) assert kwargs['nodes'] == self.target.first_batch assert kwargs['handler'] == self.worker._handler_instance def test_get_results(self): """Calling get_results() should call ClusterShell iter_buffers with the right parameters.""" self.worker.task.iter_buffers = TestClusterShellWorker.iter_buffers self.worker.handler = 'async' self.worker.execute() nodes = None output = None for nodes, output in self.worker.get_results(): pass assert str(nodes) == 'node[90-92]' assert output == 'output 9' def test_handler_getter(self): """Access to handler property should return the handler class or None.""" assert self.worker.handler is None self.worker.handler = 'sync' assert self.worker._handler == clustershell.DEFAULT_HANDLERS['sync'] def test_handler_setter_invalid(self): """Raise WorkerError if trying to set it to an invalid class or value.""" class InvalidClass: """Invalid class.""" with pytest.raises(WorkerError, match='handler must be one of'): self.worker.handler = 'invalid-handler' with pytest.raises(WorkerError, match='handler must be one of'): self.worker.handler = InvalidClass def test_handler_setter_default_sync(self): """Should set the handler to the default handler for the sync mode.""" self.worker.handler = 'sync' assert self.worker._handler == clustershell.DEFAULT_HANDLERS['sync'] def test_handler_setter_default_async(self): """Should set the handler to the default handler for the async mode.""" self.worker.handler = 'async' assert self.worker._handler == clustershell.DEFAULT_HANDLERS['async'] def test_handler_setter_custom(self): """Should set the handler to the given custom class that inherit from BaseEventHandler.""" self.worker.handler = ConcreteBaseEventHandler assert self.worker._handler == ConcreteBaseEventHandler def test_reporter_getter(self): """Access to the reporter getter should return the current value.""" assert self.worker.reporter is TqdmReporter # Current default self.worker.reporter = NullReporter assert self.worker.reporter is NullReporter def test_reporter_setter_invalid(self): """Raise WorkerError if trying to set the reporter to an invalid value.""" class InvalidReporter: """Reporter that does not inherit from BaseReporter.""" with pytest.raises(WorkerError, match='reporter must be a subclass of'): self.worker.reporter = InvalidReporter def test_progress_bars_getter(self): """Access to the progress_bars getter should return the current value.""" assert self.worker.progress_bars # Current default self.worker.progress_bars = False assert not self.worker.progress_bars def test_progress_bars_setter_invalid(self): """Raise WorkerError if trying to set the progress_bars to an invalid value.""" with pytest.raises(WorkerError, match='progress_bars must be a boolean'): self.worker.progress_bars = 'invalid' @staticmethod def iter_buffers(): """A generator to simulate the buffer iteration of ClusterShell objects.""" for i in range(10): yield 'output {}'.format(i), ['node{}0'.format(i), 'node{}1'.format(i), 'node{}2'.format(i)] class TestBaseEventHandler: """BaseEventHandler test class.""" def setup_method(self, *args): """Initialize default properties and instances.""" self.target = Target(nodeset('node[1-2]')) self.commands = [Command('command1', ok_codes=[0, 100]), Command('command2', timeout=5)] self.worker = mock.MagicMock() self.worker.current_node = 'node1' self.worker.command = 'command1' self.worker.nodes = self.target.hosts self.handler = None self.args = args self.progress_bars = mock.MagicMock(spec_set=BaseExecutionProgress) def test_close(self): """Calling close should raise NotImplementedError.""" self.handler = clustershell.BaseEventHandler(self.target, self.commands, TqdmReporter(), progress_bars=self.progress_bars) with pytest.raises(NotImplementedError): self.handler.close(self.worker) class ConcreteBaseEventHandler(clustershell.BaseEventHandler): """Concrete implementation of a BaseEventHandler.""" def __init__(self, nodes, commands, reporter, **kwargs): """Initialize progress bars.""" super().__init__(nodes, commands, reporter, **kwargs) self.progress = mock.Mock(spec_set=TqdmProgressBars) def close(self, task): """Required by the BaseEventHandler class.""" class TestConcreteBaseEventHandler(TestBaseEventHandler): """ConcreteBaseEventHandler test class.""" @mock.patch('cumin.transports.clustershell.tqdm') def setup_method(self, _, tqdm): # pylint: disable=arguments-differ """Initialize default properties and instances.""" super().setup_method() self.handler = ConcreteBaseEventHandler(self.target, self.commands, TqdmReporter(), progress_bars=self.progress_bars) self.worker.eh = self.handler assert not tqdm.write.called def test_instantiation(self): """An instance of ConcreteBaseEventHandler should be an instance of BaseEventHandler.""" assert sorted(self.handler.nodes.keys()) == list(self.target.hosts) @mock.patch('cumin.transports.clustershell.tqdm') def test_on_timeout(self, tqdm): """Calling on_timeout() should update the fail progress bar.""" for node in self.target.hosts: self.worker.current_node = node self.handler.ev_pickup(self.worker, node) self.handler.ev_close(self.worker, True) self.worker.task.num_timeout.return_value = 1 self.worker.task.iter_keys_timeout.return_value = [self.target.hosts[0]] assert not self.handler.global_timedout self.handler.on_timeout(self.worker.task) assert self.handler.progress.update_failed.called assert self.handler.global_timedout assert tqdm.write.called def test_ev_pickup(self): """Calling ev_pickup() should set the state of the current node to running.""" for node in self.target.hosts: self.handler.ev_pickup(self.worker, node) running_nodes = [node for node in self.worker.eh.nodes.values() if node.state.is_running] assert running_nodes == list(self.worker.eh.nodes.values()) @mock.patch('cumin.transports.clustershell.tqdm') def test_ev_read_many_hosts(self, tqdm): """Calling ev_read() should not print the worker message if matching multiple hosts.""" for node in self.target.hosts: self.handler.ev_read(self.worker, node, self.worker.SNAME_STDOUT, 'Node output') assert not tqdm.write.called @mock.patch('cumin.transports.clustershell.tqdm') def test_ev_read_single_host(self, tqdm): """Calling ev_read() should print the worker message if matching a single host.""" self.target = Target(nodeset('node1')) self.handler = ConcreteBaseEventHandler(self.target, self.commands, TqdmReporter(), progress_bars=self.progress_bars) output = b'node1 output' self.worker.nodes = self.target.hosts self.handler.ev_read(self.worker, self.target.hosts[0], self.worker.SNAME_STDOUT, output) assert tqdm.write.call_args[0][0] == output.decode() def test_ev_close(self): """Calling ev_close() should increase the counters for the timed out hosts.""" for node in self.target.hosts: self.handler.ev_pickup(self.worker, node) assert self.handler.counters['timeout'] == 0 self.worker.task.num_timeout.return_value = 2 self.handler.ev_close(self.worker, True) assert self.handler.counters['timeout'] == 2 class TestSyncEventHandler(TestBaseEventHandler): """SyncEventHandler test class.""" @mock.patch('cumin.transports.clustershell.logging') @mock.patch('cumin.transports.clustershell.tqdm') def setup_method(self, _, tqdm, logger): # pylint: disable=arguments-differ """Initialize default properties and instances.""" super().setup_method() self.handler = clustershell.SyncEventHandler(self.target, self.commands, TqdmReporter(), progress_bars=self.progress_bars, success_threshold=1) self.worker.eh = self.handler self.logger = logger assert not tqdm.write.called def test_instantiation(self): """An instance of SyncEventHandler should be an instance of BaseEventHandler.""" assert isinstance(self.handler, clustershell.BaseEventHandler) def test_start_command_no_schedule(self): """Calling start_command() should reset the success counter and initialize the progress bars.""" self.handler.start_command() assert self.handler.progress.init.called assert self.handler.counters['success'] == 0 @mock.patch('cumin.transports.clustershell.Task.task_self') def test_start_command_schedule(self, task_self): """Calling start_command() with schedule should also change the state of the first batch nodes.""" # Reset the state of nodes to pending for node in self.handler.nodes.values(): node.state.update(clustershell.State.running) node.state.update(clustershell.State.success) node.state.update(clustershell.State.pending) self.handler.start_command(schedule=True) assert self.handler.progress.init.called assert self.handler.counters['success'] == 0 scheduled_nodes = sorted(node.name for node in self.handler.nodes.values() if node.state.is_scheduled) assert scheduled_nodes == sorted(['node1', 'node2']) assert task_self.called @mock.patch('cumin.transports.clustershell.tqdm') def test_end_command(self, tqdm): """Calling end_command() should wrap up the command execution.""" assert not self.handler.end_command() self.handler.counters['success'] = 2 assert self.handler.end_command() self.handler.success_threshold = 0.5 self.handler.counters['success'] = 1 assert self.handler.end_command() self.handler.current_command_index = 1 assert not self.handler.end_command() assert tqdm.write.called @mock.patch('cumin.transports.clustershell.tqdm') def test_on_timeout(self, tqdm): """Calling on_timeout() should call end_command().""" self.worker.task.num_timeout.return_value = 0 self.worker.task.iter_keys_timeout.return_value = [] self.handler.on_timeout(self.worker.task) assert tqdm.write.called def test_ev_timer(self): """Calling ev_timer() should schedule the execution of the next node/command.""" # TODO: improve testing of ev_timer self.handler.ev_timer(mock.Mock()) @mock.patch('cumin.transports.clustershell.Task.Task.timer') def test_ev_hup_ok(self, timer): """Calling ev_hup with a worker that has exit status zero should update the success progress bar.""" self.handler.ev_pickup(self.worker, self.worker.current_node) self.handler.ev_hup(self.worker, self.worker.current_node, 100) assert self.handler.progress.update_success.called assert not timer.called assert self.handler.nodes[self.worker.current_node].state.is_success @mock.patch('cumin.transports.clustershell.Task.Task.timer') def test_ev_hup_ko(self, timer): """Calling ev_hup with a worker that has exit status non-zero should update the failed progress bar.""" self.handler.ev_pickup(self.worker, self.worker.current_node) self.handler.ev_hup(self.worker, self.worker.current_node, 1) assert self.handler.progress.update_failed.called assert not timer.called assert self.handler.nodes[self.worker.current_node].state.is_failed @mock.patch('cumin.transports.clustershell.tqdm') def test_close(self, tqdm): # pylint: disable=arguments-differ """Calling close should print the report when needed.""" self.handler.current_command_index = 2 self.handler.close(self.worker) assert tqdm.write.called class TestAsyncEventHandler(TestBaseEventHandler): """AsyncEventHandler test class.""" @mock.patch('cumin.transports.clustershell.logging') @mock.patch('cumin.transports.clustershell.tqdm') def setup_method(self, _, tqdm, logger): # pylint: disable=arguments-differ """Initialize default properties and instances.""" super().setup_method() self.handler = clustershell.AsyncEventHandler(self.target, self.commands, TqdmReporter(), progress_bars=self.progress_bars) self.worker.eh = self.handler self.logger = logger assert not tqdm.write.called def test_instantiation(self): """An instance of AsyncEventHandler should be an instance of BaseEventHandler and initialize progress bars.""" assert isinstance(self.handler, clustershell.BaseEventHandler) assert self.handler.progress.init.called def test_ev_hup_ok(self): """Calling ev_hup with a worker that has zero exit status should enqueue the next command.""" self.handler.ev_pickup(self.worker, self.worker.current_node) self.handler.ev_hup(self.worker, self.worker.current_node, 0) self.worker.task.shell.assert_called_once_with( 'command2', handler=self.handler, timeout=5, stdin=False, nodes=nodeset(self.worker.current_node)) # Calling it again self.worker.command = 'command2' self.handler.ev_pickup(self.worker, self.worker.current_node) self.handler.ev_hup(self.worker, self.worker.current_node, 0) assert self.handler.counters['success'] == 1 assert self.handler.progress.update_success.called def test_ev_hup_ko(self): """Calling ev_hup with a worker that has non-zero exit status should not enqueue the next command.""" self.handler.ev_pickup(self.worker, self.worker.current_node) self.handler.ev_hup(self.worker, self.worker.current_node, 1) assert self.handler.progress.update_failed.called def test_ev_timer(self): """Calling ev_timer() should schedule the execution of the next node/command.""" # TODO: improve testing of ev_timer self.handler.ev_timer(mock.Mock()) @mock.patch('cumin.transports.clustershell.tqdm') def test_close(self, tqdm): # pylint: disable=arguments-differ """Calling close with a worker should close progress bars.""" self.worker.task.iter_buffers = TestClusterShellWorker.iter_buffers self.worker.num_timeout.return_value = 0 self.handler.close(self.worker) assert self.handler.progress.close.called assert tqdm.write.called wikimedia-cumin-36f957f/cumin/tests/unit/transports/test_init.py000066400000000000000000000615411476500461000252020ustar00rootroot00000000000000"""Transport tests.""" # pylint: disable=protected-access from unittest import mock import pytest from ClusterShell.NodeSet import NodeSet import cumin # noqa: F401 (dynamically used in TestCommand) from cumin import transports from cumin.transports import TqdmProgressBars class ConcreteBaseWorker(transports.BaseWorker): """Concrete class for BaseWorker.""" def execute(self): """Required by BaseWorker.""" def get_results(self): """Required by BaseWorker.""" yield "node", "output" @property def handler(self): """Required by BaseWorker.""" return self._handler @handler.setter def handler(self, value): """Required by BaseWorker.""" self._handler = value class Commands: """Helper class to define a list of commands to test.""" command_with_options = r'command --with "options" -a -n -d params with\ spaces' command_with_options_equivalent = r"command --with 'options' -a -n -d params with\ spaces" command_with_nested_quotes = 'command --with \'nested "quotes"\' -a -n -d params with\\ spaces' different_command = r"command --with 'other options' -a -n -d other_params with\ spaces" def __init__(self): """Initialize test commands.""" self.commands = [ {'command': 'command1'}, {'command': 'command1', 'timeout': 5}, {'command': 'command1', 'ok_codes': [0, 255]}, {'command': 'command1', 'timeout': 5, 'ok_codes': [0, 255]}, {'command': self.command_with_options}, {'command': self.command_with_options, 'timeout': 5}, {'command': self.command_with_options, 'ok_codes': [0, 255]}, {'command': self.command_with_options, 'timeout': 5, 'ok_codes': [0, 255]}, {'command': self.command_with_nested_quotes}, {'command': self.command_with_nested_quotes, 'timeout': 5}, {'command': self.command_with_nested_quotes, 'ok_codes': [0, 255]}, {'command': self.command_with_nested_quotes, 'timeout': 5, 'ok_codes': [0, 255]}, ] for command in self.commands: command['obj'] = transports.Command( command['command'], timeout=command.get('timeout', None), ok_codes=command.get('ok_codes', None)) @pytest.mark.parametrize('command', Commands().commands) class TestCommandParametrized: """Command class tests executed for each parametrized command.""" def test_instantiation(self, command): """A new Command instance should set the command property to the given command.""" assert isinstance(command['obj'], transports.Command) assert command['obj'].command == command['command'] assert command['obj']._timeout == command.get('timeout', None) assert command['obj']._ok_codes == command.get('ok_codes', None) def test_repr(self, command): """A repr of a Command should allow to instantiate an instance with the same properties.""" # Bandit and pylint would require to use ast.literal_eval, but it will not work with objects command_repr = repr(command['obj']) if r'\ ' in command_repr: return # Skip tests with bash-escaped spaces are they will trigger DeprecationWarning command_instance = eval(command_repr) # nosec # pylint: disable=eval-used assert isinstance(command_instance, transports.Command) assert repr(command_instance) == repr(command['obj']) assert command_instance.command == command['obj'].command assert command_instance._timeout == command['obj']._timeout assert command_instance._ok_codes == command['obj']._ok_codes def test_str(self, command): """A cast to string of a Command should return its command.""" assert str(command['obj']) == command['command'] def test_eq(self, command): """A Command instance can be compared to another or to a string with the equality operator.""" assert command['obj'] == transports.Command( command['command'], timeout=command.get('timeout', None), ok_codes=command.get('ok_codes', None)) if command.get('timeout', None) is None and command.get('ok_codes', None) is None: assert command['obj'] == command['command'] with pytest.raises(ValueError, match='Unable to compare instance of'): command['obj'] == 1 # pylint: disable=pointless-statement def test_ne(self, command): """A Command instance can be compared to another or to a string with the inequality operator.""" # Different command with same or differnt properties assert command['obj'] != transports.Command( Commands.different_command, timeout=command.get('timeout', None), ok_codes=command.get('ok_codes', None)) assert command['obj'] != transports.Command( Commands.different_command, timeout=999, ok_codes=command.get('ok_codes', None)) assert command['obj'] != transports.Command( Commands.different_command, timeout=command.get('timeout', None), ok_codes=[99]) assert command['obj'] != transports.Command(Commands.different_command, timeout=999, ok_codes=[99]) assert command['obj'] != Commands.different_command # Same command, properties different assert command['obj'] != transports.Command( command['command'], timeout=999, ok_codes=command.get('ok_codes', None)) assert command['obj'] != transports.Command( command['command'], timeout=command.get('timeout', None), ok_codes=[99]) assert command['obj'] != transports.Command(command['command'], timeout=999, ok_codes=[99]) if command.get('timeout', None) is not None or command.get('ok_codes', None) is not None: assert command['obj'] != command['command'] with pytest.raises(ValueError, match='Unable to compare instance of'): command['obj'] == 1 # pylint: disable=pointless-statement def test_timeout_getter(self, command): """Should return the timeout set, None otherwise.""" if command['obj'].timeout is not None and command['obj']._timeout is not None: assert command['obj'].timeout == pytest.approx(command['obj']._timeout) def test_ok_codes_getter(self, command): """Should return the ok_codes set, [0] otherwise.""" assert command['obj'].ok_codes == command.get('ok_codes', [0]) class TestCommand: """Command class non parametrized tests.""" def test_eq_equivalent(self): """Two Commadn instances with equivalent comamnds just formatted differently should be considered equal.""" command1 = transports.Command(Commands.command_with_options) command2 = transports.Command(Commands.command_with_options_equivalent) assert command1 == command2 def test_timeout_setter(self): """Should set the timeout to its value, converted to float if integer. Unset it if None is passed.""" command = transports.Command('command1') command.timeout = 1.0 assert command._timeout == pytest.approx(1.0) command.timeout = None assert command._timeout is None command.timeout = 1 assert command._timeout == pytest.approx(1.0) with pytest.raises(transports.WorkerError, match='timeout must be a positive float'): command.timeout = -1.0 def test_ok_codes_getter_empty(self): """Should return the ok_codes set, [0] otherwise.""" # Test empty list command = transports.Command('command1') assert command.ok_codes == [0] command.ok_codes = [] assert command.ok_codes == [] command.ok_codes = [1, 255] assert command.ok_codes == [1, 255] def test_ok_codes_setter(self): """Should set the ok_codes to its value, unset it if None is passed.""" command = transports.Command('command1') assert command._ok_codes is None for i in range(256): codes = [i] command.ok_codes = codes assert command._ok_codes == codes codes.insert(0, 0) command.ok_codes = codes assert command._ok_codes == codes command.ok_codes = None assert command._ok_codes is None command.ok_codes = [] assert command._ok_codes == [] with pytest.raises(transports.WorkerError, match='ok_codes must be a list'): command.ok_codes = 'invalid_value' message = 'must be a list of integers in the range' for i in (-1, 0.0, 100.0, 256, 'invalid_value'): codes = [i] with pytest.raises(transports.WorkerError, match=message): command.ok_codes = codes codes.insert(0, 0) with pytest.raises(transports.WorkerError, match=message): command.ok_codes = codes class TestState: """State class tests.""" def test_instantiation_no_init(self): """A new State without an init value should start in the pending state.""" state = transports.State() assert state._state == transports.State.pending def test_instantiation_init_ok(self): """A new State with a valid init value should start in this state.""" state = transports.State(init=transports.State.running) assert state._state == transports.State.running def test_instantiation_init_ko(self): """A new State with an invalid init value should raise InvalidStateError.""" with pytest.raises(transports.InvalidStateError, match='is not a valid state'): transports.State(init='invalid_state') def test_getattr_current(self): """Accessing the 'current' property should return the current state.""" assert transports.State().current == transports.State.pending def test_getattr_is_valid_state(self): """Accessing a property named is_{a_valid_state_name} should return a boolean.""" state = transports.State(init=transports.State.failed) assert not state.is_pending assert not state.is_scheduled assert not state.is_running assert not state.is_timeout assert not state.is_success assert state.is_failed def test_getattr_invalid_property(self): """Accessing a property with an invalid name should raise AttributeError.""" state = transports.State(init=transports.State.failed) with pytest.raises(AttributeError, match='object has no attribute'): state.invalid_property # pylint: disable=pointless-statement def test_repr(self): """A State repr should return its representation that allows to recreate the same State instance.""" assert repr(transports.State()) == 'cumin.transports.State(init={state})'.format(state=transports.State.pending) state = transports.State.running assert repr(transports.State(init=state)) == 'cumin.transports.State(init={state})'.format(state=state) def test_str(self): """A State string should return its string representation.""" assert str(transports.State()) == 'pending' assert str(transports.State(init=transports.State.running)) == 'running' def test_cmp_state(self): """Two State instance can be compared between each other.""" state = transports.State() greater_state = transports.State(init=transports.State.failed) same_state = transports.State() assert greater_state > state assert greater_state >= state assert same_state >= state assert state < greater_state assert state <= greater_state assert state <= same_state assert state == same_state assert state != greater_state def test_cmp_int(self): """A State instance can be compared with integers.""" state = transports.State() greater_state = transports.State.running same_state = transports.State.pending assert greater_state > state assert greater_state >= state assert same_state >= state assert state < greater_state assert state <= greater_state assert state <= same_state assert state == same_state assert state != greater_state def test_cmp_invalid(self): """Trying to compare a State instance with an invalid object should raise ValueError.""" state = transports.State() invalid_state = 'invalid_state' with pytest.raises(ValueError, match='Unable to compare instance'): state == invalid_state # pylint: disable=pointless-statement def test_update_invalid_state(self): """Trying to update a State with an invalid value should raise ValueError.""" state = transports.State() with pytest.raises(ValueError, match='State must be one of'): state.update('invalid_state') def test_update_invalid_transition(self): """Trying to update a State with an invalid transition should raise StateTransitionError.""" state = transports.State() with pytest.raises(transports.StateTransitionError, match='the allowed states are'): state.update(transports.State.failed) def test_update_ok(self): """Properly updating a State should update it without errors.""" state = transports.State() state.update(transports.State.scheduled) assert state.current == transports.State.scheduled state.update(transports.State.running) assert state.current == transports.State.running state.update(transports.State.success) assert state.current == transports.State.success state.update(transports.State.pending) assert state.current == transports.State.pending class TestTarget: """Target class tests.""" def setup_method(self, _): """Initialize default properties and instances.""" # pylint: disable=attribute-defined-outside-init self.hosts_list = ['host' + str(i) for i in range(10)] self.hosts = cumin.nodeset_fromlist(self.hosts_list) def test_init_no_hosts(self): """Creating a Target instance with empty hosts should raise WorkerError.""" with pytest.raises(transports.WorkerError, match="must be a non-empty ClusterShell NodeSet or list"): transports.Target([]) def test_init_nodeset(self): """Creating a Target instance with a NodeSet and without optional parameter should return their defaults.""" target = transports.Target(self.hosts) assert target.hosts == self.hosts assert target.batch_size == len(self.hosts) assert target.batch_sleep == 0.0 def test_init_list(self): """Creating a Target instance with a list and without optional parameter should return their defaults.""" target = transports.Target(self.hosts_list) assert target.hosts == self.hosts assert target.batch_size == len(self.hosts) assert target.batch_sleep == 0.0 def test_init_invalid(self): """Creating a Target instance with invalid hosts should raise WorkerError.""" with pytest.raises(transports.WorkerError, match="must be a non-empty ClusterShell NodeSet or list"): transports.Target(set(self.hosts_list)) @mock.patch('cumin.transports.logging.Logger.debug') def test_init_batch_size(self, mocked_logger): """Creating a Target instance with a batch_size should set it to it's value, if valid.""" target = transports.Target(self.hosts, batch_size=5) assert target.batch_size == 5 target = transports.Target(self.hosts, batch_size=len(self.hosts) + 1) assert target.batch_size == len(self.hosts) assert mocked_logger.called target = transports.Target(self.hosts, batch_size=None) assert target.batch_size == len(self.hosts) with pytest.raises(transports.WorkerError, match='must be a positive integer'): transports.Target(self.hosts, batch_size=0) def test_init_batch_size_perc(self): """Creating a Target instance with a batch_size_ratio should set batch_size to the appropriate value.""" target = transports.Target(self.hosts, batch_size_ratio=0.5) assert target.batch_size == 5 target = transports.Target(self.hosts, batch_size_ratio=1.0) assert target.batch_size == len(self.hosts) target = transports.Target(self.hosts, batch_size_ratio=None) assert target.batch_size == len(self.hosts) with pytest.raises(transports.WorkerError, match='parameters are mutually exclusive'): transports.Target(self.hosts, batch_size=1, batch_size_ratio=0.5) with pytest.raises(transports.WorkerError, match='has generated a batch_size of 0 hosts'): transports.Target(self.hosts, batch_size_ratio=0.0) @pytest.mark.parametrize('ratio', (1, 2.0, -0.1)) def test_init_batch_size_perc_range(self, ratio): """Creating a Target instance with an invalid batch_size_ratio should raise WorkerError.""" with pytest.raises(transports.WorkerError, match='must be a float between 0.0 and 1.0'): transports.Target(self.hosts, batch_size_ratio=ratio) def test_init_batch_sleep(self): """Creating a Target instance with a batch_sleep should set it to it's value, if valid.""" target = transports.Target(self.hosts, batch_sleep=5.0) assert target.batch_sleep == pytest.approx(5.0) target = transports.Target(self.hosts, batch_sleep=None) assert target.batch_sleep == pytest.approx(0.0) with pytest.raises(transports.WorkerError): transports.Target(self.hosts, batch_sleep=0) with pytest.raises(transports.WorkerError): transports.Target(self.hosts, batch_sleep=-1.0) def test_first_batch(self): """The first_batch property should return the first_batch of hosts.""" size = 5 target = transports.Target(self.hosts, batch_size=size) assert len(target.first_batch) == size assert target.first_batch == cumin.nodeset_fromlist(self.hosts[:size]) assert isinstance(target.first_batch, NodeSet) class TestBaseWorker: """Concrete BaseWorker class for tests.""" def test_instantiation(self): """Raise if instantiated directly, should return an instance of BaseWorker if inherited.""" target = transports.Target(cumin.nodeset('node1')) with pytest.raises(TypeError): transports.BaseWorker({}, target) # pylint: disable=abstract-class-instantiated assert isinstance(ConcreteBaseWorker({}, transports.Target(cumin.nodeset('node[1-2]'))), transports.BaseWorker) @mock.patch.dict(transports.os.environ, {}, clear=True) def test_init(self): """Constructor should save config and set environment variables.""" env_dict = {'ENV_VARIABLE': 'env_value'} config = {'transport': 'test_transport', 'environment': env_dict} assert transports.os.environ == {} worker = ConcreteBaseWorker(config, transports.Target(cumin.nodeset('node[1-2]'))) assert transports.os.environ == env_dict assert worker.config == config class TestConcreteBaseWorker: """BaseWorker test class.""" def setup_method(self, _): """Initialize default properties and instances.""" # pylint: disable=attribute-defined-outside-init self.worker = ConcreteBaseWorker({}, transports.Target(cumin.nodeset('node[1-2]'))) self.commands = [transports.Command('command1'), transports.Command('command2')] def test_commands_getter(self): """Access to commands property should return an empty list if not set and the list of commands otherwise.""" assert self.worker.commands == [] self.worker._commands = self.commands assert self.worker.commands == self.commands self.worker._commands = None assert self.worker.commands == [] def test_commands_setter(self): """Raise WorkerError if trying to set it not to a list, set it otherwise.""" with pytest.raises(transports.WorkerError, match='commands must be a list'): self.worker.commands = 'invalid_value' with pytest.raises(transports.WorkerError, match='commands must be a list of Command objects'): self.worker.commands = [1, 'command2'] self.worker.commands = self.commands assert self.worker._commands == self.commands self.worker.commands = None assert self.worker._commands is None self.worker.commands = [] assert self.worker._commands == [] self.worker.commands = ['command1', 'command2'] assert self.worker._commands == self.commands def test_timeout_getter(self): """Return default value if not set, the value otherwise.""" assert self.worker.timeout == 0 self.worker._timeout = 10 assert self.worker.timeout == 10 def test_timeout_setter(self): """Raise WorkerError if not a positive integer, set it otherwise.""" message = r'timeout must be a positive integer' with pytest.raises(transports.WorkerError, match=message): self.worker.timeout = -1 with pytest.raises(transports.WorkerError, match=message): self.worker.timeout = 0 self.worker.timeout = 10 assert self.worker._timeout == 10 self.worker.timeout = None assert self.worker._timeout is None def test_success_threshold_getter(self): """Return default value if not set, the value otherwise.""" assert self.worker.success_threshold == pytest.approx(1.0) for success_threshold in (0.0, 0.0001, 0.5, 0.99): self.worker._success_threshold = success_threshold assert self.worker.success_threshold == pytest.approx(success_threshold) def test_success_threshold_setter(self): """Raise WorkerError if not float between 0 and 1, set it otherwise.""" message = r'success_threshold must be a float beween 0 and 1' with pytest.raises(transports.WorkerError, match=message): self.worker.success_threshold = 1 with pytest.raises(transports.WorkerError, match=message): self.worker.success_threshold = -0.1 self.worker.success_threshold = 0.3 assert self.worker._success_threshold == pytest.approx(0.3) class TestModuleFunctions: """Transports module functions test class.""" def test_validate_list(self): """Should raise a WorkerError if the argument is not a list.""" transports.validate_list('Test', ['value1']) transports.validate_list('Test', ['value1', 'value2']) transports.validate_list('Test', [], allow_empty=True) with pytest.raises(transports.WorkerError, match=r'Test must be a non-empty list'): transports.validate_list('Test', []) message = r'Test must be a list' for invalid_value in (0, None, 'invalid_value', {'invalid': 'value'}): with pytest.raises(transports.WorkerError, match=message): transports.validate_list('Test', invalid_value) def test_validate_positive_integer(self): """Should raise a WorkerError if the argument is not a positive integer or None.""" transports.validate_positive_integer('Test', None) transports.validate_positive_integer('Test', 1) transports.validate_positive_integer('Test', 100) message = r'Test must be a positive integer' for invalid_value in (0, -1, 'invalid_value', ['invalid_value']): with pytest.raises(transports.WorkerError, match=message): transports.validate_positive_integer('Test', invalid_value) def test_raise_error(self): """Should raise a WorkerError.""" with pytest.raises(transports.WorkerError, match='Test message'): transports.raise_error('Test', 'message', 'value') @mock.patch('cumin.transports.tqdm') class TestProgressBars: """A class that tests ProgressBars.""" def test_init_intialize_progress_bars_with_correct_size(self, tqdm): """Progress bars are initialized at the correct size.""" progress = TqdmProgressBars() progress.init(10) assert tqdm.call_count == 2 _, kwargs = tqdm.call_args assert kwargs['total'] == 10 def test_progress_bars_are_closed(self, tqdm): """Progress bars are closed.""" progress = TqdmProgressBars() progress.init(10) progress.close() assert tqdm.mock_calls[-2] == mock.call().close() assert tqdm.mock_calls[-1] == mock.call().close() def test_progress_bars_is_updated_on_success(self, tqdm): """Progress bar is updated on success.""" progress = TqdmProgressBars() progress.init(10) progress.update_success(5) assert mock.call().update(5) in tqdm.mock_calls def test_progress_bars_is_updated_on_failure(self, tqdm): """Progress bar is updated on failure.""" progress = TqdmProgressBars() progress.init(10) progress.update_failed(3) assert tqdm.mock_calls[-1] == mock.call().update(3) wikimedia-cumin-36f957f/cumin/tests/vulture_whitelist.py000066400000000000000000000020351476500461000235750ustar00rootroot00000000000000"""Vulture whitelist to avoid false positives.""" class Whitelist: """Helper class that allows mocking Python objects.""" def __getattr__(self, _): """Mocking magic method __getattr__.""" pass whitelist_logging = Whitelist() whitelist_logging.raiseExceptions whitelist_cli = Whitelist() whitelist_cli.run.h whitelist_color = Whitelist() whitelist_color.ColoredType.__getattr__ whitelist_Config = Whitelist() # noqa: N816 whitelist_Config.__new__ whitelist_transports_clustershell = Whitelist() whitelist_transports_clustershell.BaseEventHandler.kwargs whitelist_backends_openstack = Whitelist() whitelist_backends_openstack.TestOpenStackQuery.test_execute_all.nova_client.return_value.servers.list.side_effect whitelist_tests_integration_conftest = Whitelist() whitelist_tests_integration_conftest.pytest_cmdline_preparse whitelist_tests_integration_conftest.pytest_runtest_makereport whitelist_tests_integration_test_cli_TestCLI = Whitelist() # noqa: N816 whitelist_tests_integration_test_cli_TestCLI.setup_method wikimedia-cumin-36f957f/cumin/transport.py000066400000000000000000000040141476500461000206640ustar00rootroot00000000000000"""Transport factory.""" import importlib from cumin import CuminError class Transport: """Transport factory class. The transport layer is the one used to convey the commands to be executed into the selected hosts. The transport abstraction allow to specify a mode to choose the execution plan, an event handler class and a success threshold. Those can be used by the chosen transport to customize the behavior of the execution plan. All the transports share a common interface that is defined in the :py:class:`cumin.transports.BaseWorker` class and they are instantiated through the :py:class:`cumin.transport.Transport` factory class. Each transport module need to define a ``worker_class`` module variable that is a pointer to the transport class for dynamic instantiation. """ @staticmethod def new(config, target): """Create a transport worker class based on the configuration (`factory`). Arguments: config (dict): the configuration dictionary. target (cumin.transports.Target): a Target instance. Returns: BaseWorker: the created worker instance for the configured transport. Raises: cumin.CuminError: if the configuration is missing the required ``transport`` key. exceptions.ImportError: if unable to import the transport module. exceptions.AttributeError: if the transport module is missing the required ``worker_class`` attribute. """ if 'transport' not in config: raise CuminError("Missing required parameter 'transport' in the configuration dictionary") try: module = importlib.import_module('cumin.transports.{transport}'.format(transport=config['transport'])) return module.worker_class(config, target) except (AttributeError, ImportError) as e: raise CuminError("Unable to load worker class for transport '{transport}': {msg}".format( transport=config['transport'], msg=repr(e))) from e wikimedia-cumin-36f957f/cumin/transports/000077500000000000000000000000001476500461000204765ustar00rootroot00000000000000wikimedia-cumin-36f957f/cumin/transports/__init__.py000066400000000000000000000743551476500461000226250ustar00rootroot00000000000000"""Abstract transport and state machine for hosts state.""" import logging import os import shlex import sys from abc import ABCMeta, abstractmethod from typing import Callable, Optional from ClusterShell.NodeSet import NodeSet from tqdm import tqdm from cumin import CuminError, nodeset_fromlist from cumin.color import Colored class WorkerError(CuminError): """Custom exception class for worker errors.""" class StateTransitionError(CuminError): """Exception raised when an invalid transition for a node's State was attempted.""" class InvalidStateError(CuminError): """Exception raised when an invalid transition for a node's State was attempted.""" class Command: """Class to represent a command.""" def __init__(self, command, timeout=None, ok_codes=None): """Command constructor. Arguments: command (str): the command to execute. timeout (int, optional): the command's timeout in seconds. ok_codes (list, optional): a list of exit codes to be considered successful for the command. The exit code zero is considered successful by default, if this option is set it override it. If set to an empty list ``[]``, it means that any code is considered successful. """ self.command = command self._timeout = None self._ok_codes = None if timeout is not None: self.timeout = timeout if ok_codes is not None: self.ok_codes = ok_codes def __repr__(self): """Return the representation of the :py:class:`Command`. The representation allow to instantiate a new :py:class:`Command` instance with the same properties. Returns: str: the representation of the object. """ params = ["'{command}'".format(command=self.command.replace("'", r'\''))] for field in ('_timeout', '_ok_codes'): value = getattr(self, field) if value is not None: params.append('{key}={value}'.format(key=field[1:], value=value)) return 'cumin.transports.Command({params})'.format(params=', '.join(params)) def __str__(self): """Return the string representation of the command. Returns: str: the string representation of the object. """ return self.command def __eq__(self, other): """Equality operation. Allow to directly compare a :py:class:`Command` object to another or a string. :Parameters: according to Python's Data model :py:meth:`object.__eq__`. Returns: bool: :py:data:`True` if the `other` object is equal to this one, :py:data:`False` otherwise. Raises: exceptions.ValueError: if the comparing object is not an instance of :py:class:`Command` or a :py:class:`str`. """ if isinstance(other, str): other_command = other same_params = (self._timeout is None and self._ok_codes is None) elif isinstance(other, Command): other_command = other.command same_params = (self.timeout == other.timeout and self.ok_codes == other.ok_codes) else: raise ValueError("Unable to compare instance of '{other}' with Command instance".format(other=type(other))) return shlex.split(self.command) == shlex.split(other_command) and same_params def __ne__(self, other): """Inequality operation. Allow to directly compare a Command object to another or a string. :Parameters: according to Python's Data model :py:meth:`object.__ne__`. Returns: bool: :py:data:`True` if the `other` object is different to this one, :py:data:`False` otherwise. Raises: exceptions.ValueError: if the comparing object is not an instance of :py:class:`Command` or a :py:class:`str`. """ return not self == other @property def timeout(self): """Timeout of the :py:class:`Command`. :Getter: Returns the current `timeout` or :py:data:`None` if not set. :Setter: :py:class:`float`, :py:class:`int`, :py:data:`None`: the `timeout` in seconds for the execution of the `command` on each host. Both :py:class:`float` and :py:class:`int` are accepted and converted internally to :py:class:`float`. If :py:data:`None` the `timeout` is reset to its default value. Raises: cumin.transports.WorkerError: if trying to set it to an invalid value. """ return self._timeout @timeout.setter def timeout(self, value): """Setter for the timeout property. The relative documentation is in the getter.""" if isinstance(value, int): value = float(value) validate_positive_float('timeout', value) self._timeout = value @property def ok_codes(self): """List of exit codes to be considered successful for the execution of the :py:class:`Command`. :Getter: Returns the current `ok_codes` or a :py:class:`list` with the element ``0`` if not set. :Setter: :py:class:`list[int]`, :py:data:`None`: list of exit codes to be considered successful for the execution of the `command` on each host. Must be a :py:class:`list` of :py:class:`int` in the range ``0-255`` included, or :py:data:`None` to unset it. The exit code ``0`` is considered successful by default, but it can be overriden setting this property. Set it to an empty :py:class:`list` to consider any exit code successful. Raises: cumin.transports.WorkerError: if trying to set it to an invalid value. """ ok_codes = self._ok_codes if ok_codes is None: ok_codes = [0] return ok_codes @ok_codes.setter def ok_codes(self, value): """Setter for the ok_codes property. The relative documentation is in the getter.""" if value is None: self._ok_codes = value return validate_list('ok_codes', value, allow_empty=True) for code in value: if not isinstance(code, int) or code < 0 or code > 255: raise_error('ok_codes', 'must be a list of integers in the range 0-255 or None', value) self._ok_codes = value class State: """State machine for the state of a host. .. attribute:: current :py:class:`int`: the current state. .. attribute:: is_pending :py:class:`bool`: :py:data:`True` if the current state is `pending`, :py:data:`False` otherwise. .. attribute:: is_scheduled :py:class:`bool`: :py:data:`True` if the current state is `scheduled`, :py:data:`False` otherwise. .. attribute:: is_running :py:class:`bool`: :py:data:`True` if the current state is `running`, :py:data:`False` otherwise. .. attribute:: is_success :py:class:`bool`: :py:data:`True` if the current state is `success`, :py:data:`False` otherwise. .. attribute:: is_failed :py:class:`bool`: :py:data:`True` if the current state is `failed`, :py:data:`False` otherwise. .. attribute:: is_timeout :py:class:`bool`: :py:data:`True` if the current state is `timeout`, :py:data:`False` otherwise. """ valid_states = range(6) """:py:class:`list`: valid states integer indexes.""" pending, scheduled, running, success, failed, timeout = valid_states """Valid state property, one for each :py:data:`cumin.transports.State.valid_states`.""" states_representation = ('pending', 'scheduled', 'running', 'success', 'failed', 'timeout') """:py:func:`tuple`: Tuple with the string representations of the valid states.""" allowed_state_transitions = { pending: (scheduled, ), scheduled: (running, ), running: (running, success, failed, timeout), success: (pending, ), failed: (), timeout: (), } """:py:class:`dict`: Dictionary with ``{valid state: tuple of valid states}`` mapping of the allowed transitions between all the possile states. This is the diagram of the allowed transitions: .. image:: ../../examples/transports_state_transitions.png :alt: State class allowed transitions diagram | """ def __init__(self, init=None): """State constructor. The initial state is set to `pending` it not provided. Arguments: init (int, optional): the initial state from where to start. The `pending` state will be used if not set. Raises: cumin.transports.InvalidStateError: if `init` is an invalid state. """ if init is None: self._state = self.pending elif init in self.valid_states: self._state = init else: raise InvalidStateError("Initial state '{state}' is not a valid state. Expected one of {states}".format( state=init, states=self.valid_states)) def __getattr__(self, name): """Attribute accessor. :Accessible properties: * `current` (:py:class:`int`): retuns the current state. * `is_{valid_state_name}` (:py:class:`bool`): for each valid state name, returns :py:data:`True` if the current state matches the state in the variable name. :py:data:`False` otherwise. :Parameters: according to Python's Data model :py:meth:`object.__getattr__`. Raises: exceptions.AttributeError: if the attribute name is not available. """ if name == 'current': return self._state if name.startswith('is_') and name[3:] in self.states_representation: return getattr(self, name[3:]) == self._state raise AttributeError("'State' object has no attribute '{name}'".format(name=name)) def __repr__(self): """Return the representation of the :py:class:`State`. The representation allow to instantiate a new :py:class:`State` instance with the same properties. Returns: str: the representation of the object. """ return 'cumin.transports.State(init={state})'.format(state=self._state) def __str__(self): """Return the string representation of the state. Returns: str: the string representation of the object. """ return self.states_representation[self._state] def __eq__(self, other): """Equality operator for rich comparison. :Parameters: according to Python's Data model :py:meth:`object.__eq__`. Returns: bool: :py:data:`True` if `self` is equal to `other`, :py:data:`False` otherwise. Raises: exceptions.ValueError: if the comparing object is not an instance of :py:class:`State` or a :py:class:`int`. """ return self._cmp(other) == 0 def __lt__(self, other): """Less than operator for rich comparison. :Parameters: according to Python's Data model :py:meth:`object.__lt__`. Returns: bool: :py:data:`True` if `self` is lower than `other`, :py:data:`False` otherwise. Raises: exceptions.ValueError: if the comparing object is not an instance of :py:class:`State` or a :py:class:`int`. """ return self._cmp(other) < 0 def __le__(self, other): """Less than or equal operator for rich comparison. :Parameters: according to Python's Data model :py:meth:`object.__le__`. Returns: bool: :py:data:`True` if `self` is lower or equal than `other`, :py:data:`False` otherwise. Raises: exceptions.ValueError: if the comparing object is not an instance of :py:class:`State` or a :py:class:`int`. """ return self._cmp(other) <= 0 def __gt__(self, other): """Greater than operator for rich comparison. :Parameters: according to Python's Data model :py:meth:`object.__gt__`. Returns: bool: :py:data:`True` if `self` is greater than `other`, :py:data:`False` otherwise. Raises: exceptions.ValueError: if the comparing object is not an instance of :py:class:`State` or a :py:class:`int`. """ return self._cmp(other) > 0 def __ge__(self, other): """Greater than or equal operator for rich comparison. :Parameters: according to Python's Data model :py:meth:`object.__ge__`. Returns: bool: :py:data:`True` if `self` is greater or equal than `other`, :py:data:`False` otherwise. Raises: exceptions.ValueError: if the comparing object is not an instance of :py:class:`State` or a :py:class:`int`. """ return self._cmp(other) >= 0 def update(self, new): """Transition the state from the current state to the new one, if the transition is allowed. Arguments: new (int): the new state to set. Only specific state transitions are allowed. Raises: cumin.transports.StateTransitionError: if the transition is not allowed, see :py:attr:`allowed_state_transitions`. """ if new not in self.valid_states: raise ValueError("State must be one of {valid}, got '{new}'".format(valid=self.valid_states, new=new)) if new not in self.allowed_state_transitions[self._state]: raise StateTransitionError( "From the current state '{current}' the allowed states are '{allowed}', got '{new}'".format( current=self._state, allowed=self.allowed_state_transitions[self._state], new=new)) self._state = new def _cmp(self, other): """Comparison operation. Allow to directly compare a state object to another or to an integer. Arguments: other (mixed): the object to compare the current instance to. Raises: ValueError: if the comparing object is not an instance of State or an integer. """ if isinstance(other, int): return self._state - other if isinstance(other, State): return self._state - other._state # pylint: disable=protected-access raise ValueError("Unable to compare instance of '{other}' with State instance".format(other=type(other))) class Target: """Targets management class.""" def __init__(self, hosts, batch_size=None, batch_size_ratio=None, batch_sleep=None): """Constructor, inizialize the Target with the list of hosts and additional parameters. Arguments: hosts (ClusterShell.NodeSet.NodeSet, list): hosts that will be targeted, both :py:class:`ClusterShell.NodeSet.NodeSet` and :py:class:`list` are accepted and converted automatically to :py:class:`ClusterShell.NodeSet.NodeSet` internally. batch_size (int, optional): set the batch size so that no more that this number of hosts are targeted at any given time. It must be a positive integer. If greater than the number of hosts it will be auto-resized to the number of hosts. batch_size_ratio (float, optional): set the batch size with a ratio so that no more that this fraction of hosts are targeted at any given time. It must be a float between 0 and 1 and will raise exception if after rounding it there are 0 hosts selected. batch_sleep (float, optional): sleep time in seconds between the end of execution of one host in the batch and the start in the next host. It must be a positive float. Raises: cumin.transports.WorkerError: if the `hosts` parameter is empty or invalid, if both the `batch_size` and `batch_size_ratio` parameters are set or if the `batch_size_ratio` selects no hosts. """ self.logger = logging.getLogger('.'.join((self.__module__, self.__class__.__name__))) message = "must be a non-empty ClusterShell NodeSet or list" if not hosts: raise_error('hosts', message, hosts) elif isinstance(hosts, NodeSet): self.hosts = hosts elif isinstance(hosts, list): self.hosts = nodeset_fromlist(hosts) else: raise_error('hosts', message, hosts) if batch_size is not None and batch_size_ratio is not None: raise WorkerError(("The 'batch_size' and 'batch_size_ratio' parameters are mutually exclusive but they're " "both set.")) if batch_size_ratio is not None: if not isinstance(batch_size_ratio, float) or not 0.0 <= batch_size_ratio <= 1.0: raise_error('batch_size_ratio', 'must be a float between 0.0 and 1.0', batch_size_ratio) batch_size = round(len(self.hosts) * batch_size_ratio) if batch_size == 0: raise_error('batch_size_ratio', 'has generated a batch_size of 0 hosts', batch_size_ratio) self.batch_size = self._compute_batch_size(batch_size, self.hosts) self.batch_sleep = Target._compute_batch_sleep(batch_sleep) @property def first_batch(self): """First batch of the hosts to target. :Getter: Returns a :py:class:`ClusterShell.NodeSet.NodeSet` of the first batch of hosts, according to the `batch_size`. """ return self.hosts[:self.batch_size] def _compute_batch_size(self, batch_size, hosts): """Compute the batch_size based on the hosts size and return the value to be used. Arguments: batch_size (int, None): a positive integer to indicate the batch_size to apply when executing the worker or :py:data:`None` to get its default value of all the hosts. If greater than the number of hosts, the number of hosts will be used as value instead. hosts (ClusterShell.NodeSet.NodeSet): the list of hosts to use to calculate the batch size. Returns: int: the effective `batch_size` to use. """ validate_positive_integer('batch_size', batch_size) hosts_size = len(hosts) if batch_size is None: batch_size = hosts_size elif batch_size > hosts_size: self.logger.debug(("Provided batch_size '%d' is greater than the number of hosts '%d'" ", using '%d' as value"), batch_size, hosts_size, hosts_size) batch_size = hosts_size return batch_size @staticmethod def _compute_batch_sleep(batch_sleep): """Validate batch_sleep and return its value or a default value. Arguments: batch_sleep(float, None): a positive float indicating the sleep in seconds to apply between one batched host and the next, or :py:data:`None` to get its default value. Returns: float: the effective `batch_sleep` to use. """ validate_positive_float('batch_sleep', batch_sleep) return batch_sleep or 0.0 class BaseWorker(metaclass=ABCMeta): """Worker interface to be extended by concrete workers.""" def __init__(self, config, target): """Worker constructor. Setup environment variables and initialize properties. Arguments: config (dict): a dictionary with the parsed configuration file. target (Target): a Target instance. """ self.config = config self.target = target self.logger = logging.getLogger('.'.join((self.__module__, self.__class__.__name__))) self.logger.trace('Transport %s created with config: %s', type(self).__name__, config) # Initialize setters values self._commands = None self._handler = None self._timeout = None self._success_threshold = None for key, value in config.get('environment', {}).items(): os.environ[key] = value @abstractmethod def execute(self): """Execute the task as configured. Returns: int: ``0`` on success, a positive integer on failure. Raises: cumin.transports.WorkerError: if misconfigured. """ @abstractmethod def get_results(self): """Iterate over the results (`generator`). Yields: tuple: with ``(hosts, result)`` for each host(s) of the current execution. """ @property def commands(self): """Commands for the current execution. :Getter: Returns the current `command` :py:class:`list` or an empty :py:class:`list` if not set. :Setter: :py:class:`list[Command]`, :py:class:`list[str]`: a :py:class:`list` of :py:class:`Command` objects or :py:class:`str` to be executed in the hosts. The elements are converted to :py:class:`Command` automatically. Raises: cumin.transports.WorkerError: if trying to set it with invalid data. """ return self._commands or [] @commands.setter def commands(self, value): """Setter for the `commands` property. The relative documentation is in the getter.""" if value is None: self._commands = value return validate_list('commands', value, allow_empty=True) commands = [] for command in value: if isinstance(command, Command): commands.append(command) elif isinstance(command, str): commands.append(Command(command)) else: raise_error('commands', 'must be a list of Command objects or strings', value) self._commands = commands @property @abstractmethod def handler(self): """Get and set the `handler` for the current execution. :Getter: Returns the current `handler` or :py:data:`None` if not set. :Setter: :py:class:`str`, :py:class:`EventHandler`, :py:data:`None`: an event handler to be notified of the progress during execution. Its interface depends on the actual transport chosen. Accepted values are: * None => don't use an event handler (default) * str => a string label to choose one of the available default EventHandler classes in that transport, * an event handler class object (not instance) """ @handler.setter @abstractmethod def handler(self, value): """Setter for the `handler` property. The relative documentation is in the getter.""" @property def timeout(self): """Global timeout for the current execution. :Getter: int: returns the current `timeout` or ``0`` (no timeout) if not set. :Setter: :py:class:`int`, :py:data:`None`: timeout for the current execution in seconds. Must be a positive integer or :py:data:`None` to reset it. Raises: cumin.transports.WorkerError: if trying to set it to an invalid value. """ return self._timeout or 0 @timeout.setter def timeout(self, value): """Setter for the global `timeout` property. The relative documentation is in the getter.""" validate_positive_integer('timeout', value) self._timeout = value @property def success_threshold(self): """Success threshold for the current execution. :Getter: float: returns the current `success_threshold` or ``1.0`` (`100%`) if not set. :Setter: :py:class:`float`, :py:data:`None`: The success ratio threshold that must be reached to consider the run successful. A :py:class:`float` between ``0`` and ``1`` or :py:data:`None` to reset it. The specific meaning might change based on the chosen transport. Raises: cumin.transports.WorkerError: if trying to set it to an invalid value. """ success_threshold = self._success_threshold if success_threshold is None: success_threshold = 1.0 return success_threshold @success_threshold.setter def success_threshold(self, value): """Setter for the `success_threshold` property. The relative documentation is in the getter.""" if value is not None and (not isinstance(value, float) or not (0.0 <= value <= 1.0)): # pylint: disable=superfluous-parens raise WorkerError("success_threshold must be a float beween 0 and 1, got '{value_type}': {value}".format( value_type=type(value), value=value)) self._success_threshold = value def validate_list(property_name, value, allow_empty=False): """Validate a list. Arguments: property_name (str): the name of the property to validate. value (list): the value to validate. allow_empty (bool, optional): whether to consider an empty list valid. Raises: cumin.transports.WorkerError: if trying to set it to an invalid value. """ if not isinstance(value, list): raise_error(property_name, 'must be a list', value) if not allow_empty and not value: raise_error(property_name, 'must be a non-empty list', value) def validate_positive_integer(property_name, value): """Validate a positive integer or :py:data:`None`. Arguments: property_name (str): the name of the property to validate. value (int, None): the value to validate. Raises: cumin.transports.WorkerError: if trying to set it to an invalid value. """ if value is not None and (not isinstance(value, int) or value <= 0): raise_error(property_name, 'must be a positive integer or None', value) def validate_positive_float(property_name, value): """Validate a positive float or :py:data:`None`. Arguments: property_name (str): the name of the property to validate. value (float, None): the value to validate. Raises: cumin.transports.WorkerError: if trying to set it to an invalid value. """ if value is not None and (not isinstance(value, float) or value <= 0): raise_error(property_name, 'must be a positive float or None', value) def raise_error(property_name, message, value): """Raise a :py:class:`WorkerError` exception. Arguments: property_name (str): the name of the property that raised the exception. message (str): the message to use for the exception. value (mixed): the value that raised the exception. """ raise WorkerError("{property_name} {message}, got '{value_type}': {value}".format( property_name=property_name, message=message, value_type=type(value), value=value)) class BaseExecutionProgress(metaclass=ABCMeta): """Listener interface to consume notification of the status of successful / failed hosts. The listener needs to be notified of the total number of hosts when the operation starts, and then notified of successes and failures. """ @abstractmethod def init(self, num_hosts: int) -> None: """Initialize the progress bars. Arguments: num_hosts (int): the total number of hosts """ @abstractmethod def close(self) -> None: """Closes the progress bars.""" @abstractmethod def update_success(self, num_hosts: int = 1) -> None: """Updates the number of successful hosts. Arguments: num_hosts (int): increment to the number of hosts that have completed successfully """ @abstractmethod def update_failed(self, num_hosts: int = 1) -> None: """Updates the number of failed hosts. Arguments: num_hosts (int): increment to the number of hosts that have completed in error """ class TqdmProgressBars(BaseExecutionProgress): """Progress bars based on TQDM.""" def __init__(self) -> None: """Create the progress bars. Note: the progress bars themselves are not initalized at object creation. ``init()`` needs to be called before using the progress bars. """ self._pbar_success: Optional[tqdm] = None self._pbar_failed: Optional[tqdm] = None self._bar_format = ('{desc} |{bar}| {percentage:3.0f}% ({n_fmt}/{total_fmt}) ' '[{elapsed}<{remaining}, {rate_fmt}]') def init(self, num_hosts: int) -> None: """Initialize the progress bars. Arguments: num_hosts (int): the total number of hosts """ self._pbar_success = self._tqdm(num_hosts, 'PASS', Colored.green) self._pbar_failed = self._tqdm(num_hosts, 'FAIL', Colored.red) def _tqdm(self, num_hosts: int, desc: str, color: Callable[[str], str]) -> tqdm: pbar = tqdm(desc=desc, total=num_hosts, leave=True, unit='hosts', dynamic_ncols=True, bar_format=color(self._bar_format), file=sys.stderr) pbar.refresh() return pbar def close(self) -> None: """Closes the progress bars.""" self._success.close() self._failed.close() def update_success(self, num_hosts: int = 1) -> None: """Updates the number of successful hosts. Arguments: num_hosts (int): increment to the number of hosts that have completed successfully """ self._success.update(num_hosts) def update_failed(self, num_hosts: int = 1) -> None: """Updates the number of failed hosts. Arguments: num_hosts (int): increment to the number of hosts that have completed in error """ self._failed.update(num_hosts) @property def _success(self) -> tqdm: if self._pbar_success is None: raise ValueError('init() should be called before any other operation') return self._pbar_success @property def _failed(self) -> tqdm: if self._pbar_failed is None: raise ValueError('init() should be called before any other operation') return self._pbar_failed class NoProgress(BaseExecutionProgress): """Used as a null object to disable the display of execution progress.""" def init(self, num_hosts: int) -> None: """Does nothing.""" def close(self) -> None: """Does nothing.""" def update_success(self, num_hosts: int = 1) -> None: """Does nothing.""" def update_failed(self, num_hosts: int = 1) -> None: """Does nothing.""" wikimedia-cumin-36f957f/cumin/transports/clustershell.py000066400000000000000000001321111476500461000235600ustar00rootroot00000000000000"""Transport ClusterShell: worker and event handlers.""" import logging import sys import threading from abc import ABCMeta, abstractmethod from collections import Counter, defaultdict from typing import Any, Callable, Dict, List, Optional, Tuple, Type, Union from ClusterShell import Event, Task from ClusterShell.MsgTree import MsgTreeElem from tqdm import tqdm from cumin import nodeset, nodeset_fromlist from cumin.color import Colored from cumin.transports import (BaseExecutionProgress, BaseWorker, Command, NoProgress, raise_error, State, Target, TqdmProgressBars, WorkerError) class ClusterShellWorker(BaseWorker): """It provides a Cumin worker for SSH using the ClusterShell library. This transport uses the :py:mod:`ClusterShell` Python library to connect to the selected hosts and execute a list of commands. This transport accept the following customizations: * ``sync`` execution mode: given a list of commands, the first one will be executed on all the hosts, then, if the success ratio is reached, the second one will be executed on all hosts where the first one was successful, and so on. * ``async`` execution mode: given a list of commands, on each hosts the commands will be executed sequentially, interrupting the execution on any single host at the first command that fails. The execution on the hosts is independent between each other. * custom execution mode: can be achieved creating a custom event handler class that extends the ``BaseEventHandler`` class defined in ``cumin/transports/clustershell.py``, implementing its abstract methods and setting to this class object the handler to the transport. """ def __init__(self, config: dict, target: Target) -> None: """Worker ClusterShell constructor. :Parameters: according to parent :py:meth:`cumin.transports.BaseWorker.__init__`. """ super().__init__(config, target) self.task = Task.task_self() # Initialize a ClusterShell task self._handler_instance: Optional[Event.EventHandler] = None self._reporter: Type[BaseReporter] = TqdmReporter # TODO: change this to NullReporter when releasing v5.0.0 self._progress_bars: bool = True # TODO: change this to False when releasing v5.0.0 # Set any ClusterShell task options for key, value in config.get('clustershell', {}).items(): if isinstance(value, list): self.task.set_info(key, ' '.join(value)) else: self.task.set_info(key, value) def execute(self) -> int: """Execute the commands on all the targets using the handler. Concrete implementation of parent abstract method. :Parameters: according to parent :py:meth:`cumin.transports.BaseWorker.execute`. """ if not self.commands: raise WorkerError('No commands provided.') if self.handler is None: raise WorkerError('An EventHandler is mandatory.') # Instantiate handler # Schedule only the first command for the first batch, the following ones must be handled by the EventHandler reporter = self._reporter() # Instantiate a new Reporter at each execution progress_bars_instance = TqdmProgressBars() if self._progress_bars else NoProgress() self._handler_instance = self.handler( # pylint: disable=not-callable self.target, self.commands, reporter=reporter, success_threshold=self.success_threshold, progress_bars=progress_bars_instance) self.logger.info( "Executing commands %s on '%d' hosts: %s", self.commands, len(self.target.hosts), self.target.hosts) self.task.shell(self.commands[0].command, nodes=self.target.first_batch, handler=self._handler_instance, timeout=self.commands[0].timeout, stdin=False) # FIXME: return_value should not be optional return_value: Optional[int] = 0 try: self.task.run(timeout=self.timeout, stdin=False) self.task.join() except Task.TimeoutError: if self._handler_instance is not None: self._handler_instance.on_timeout(self.task) finally: if self._handler_instance is not None: self._handler_instance.close(self.task) return_value = self._handler_instance.return_value if return_value is None: return_value = 3 # The handler did not set a return value return return_value def get_results(self): """Get the results of the last task execution. Concrete implementation of parent abstract method. :Parameters: according to parent :py:meth:`cumin.transports.BaseWorker.get_results`. """ for output, nodelist in self.task.iter_buffers(): yield nodeset_fromlist(nodelist), output @property def handler(self) -> Optional[Type['BaseEventHandler']]: """Concrete implementation of parent abstract getter and setter. Accepted values for the setter: * an instance of a custom handler class derived from :py:class:`BaseEventHandler`. * a :py:class:`str` with one of the available default handler listed in :py:data:`DEFAULT_HANDLERS`. The event handler is mandatory for this transport. :Parameters: according to parent :py:attr:`cumin.transports.BaseWorker.handler`. """ return self._handler @handler.setter def handler(self, value: Union[Type['BaseEventHandler'], str]) -> None: """Setter for the `handler` property. The relative documentation is in the getter.""" if isinstance(value, type) and issubclass(value, BaseEventHandler): self._handler = value elif value in DEFAULT_HANDLERS: self._handler = DEFAULT_HANDLERS[value] else: raise_error( 'handler', 'must be one of ({default}, a class object derived from BaseEventHandler)'.format( default=', '.join(DEFAULT_HANDLERS.keys())), value) @property def reporter(self) -> Type['BaseReporter']: """Getter for the reporter property. It must be a subclass of :py:class:`cumin.transports.clustershell.BaseReporter`. """ return self._reporter @reporter.setter def reporter(self, value: Type['BaseReporter']) -> None: """Setter for the `reporter` property. The relative documentation is in the getter.""" if not issubclass(value, BaseReporter): raise_error('reporter', 'must be a subclass of cumin.transports.clustershell.BaseReporter', value) self._reporter = value @property def progress_bars(self) -> bool: """Getter for the boolean progress_bars property.""" return self._progress_bars @progress_bars.setter def progress_bars(self, value: bool) -> None: """Setter for the `progress_bars` property. The relative documentation is in the getter.""" if not isinstance(value, bool): raise_error('progress_bars', 'must be a boolean', value) self._progress_bars = value class Node: """Node class to represent each target node. Additional attributes available are: .. attribute:: state :py:class:`cumin.transports.State`: the state of the node. .. attribute:: running_command_index :py:class:`int`: the index of the current running command in the list of commands. """ def __init__(self, name: str, commands: List[Command]): """Node class constructor with default values. Arguments: name (str): the hostname of the node. commands (list): a list of :py:class:`cumin.transports.Command` objects to be executed on the node. """ self.name = name self.commands = commands self.state = State() # Initialize the state machine for this node. self.running_command_index = -1 # Pointer to the current running command in self.commands class BaseReporter(metaclass=ABCMeta): """Reporter base class that does not report anything.""" def __init__(self) -> None: """Initializes a Reporter.""" self.logger = logging.getLogger('.'.join((self.__module__, self.__class__.__name__))) @abstractmethod def global_timeout_nodes(self, nodes: Dict[str, Node], num_hosts: int) -> None: """Print the nodes that were caught by the global timeout in a colored and tqdm-friendly way.""" @abstractmethod def failed_nodes(self, nodes: Dict[str, Node], num_hosts: int, commands: List[Command], filter_command_index: int = -1) -> None: """Print the nodes that failed to execute commands in a colored and tqdm-friendly way. Arguments: nodes (list): the list of Nodes on which commands were executed num_hosts (int): the total number of nodes. commands (list): the list of Commands that were executed filter_command_index (int, optional): print only the nodes that failed to execute the command specified by this command index. """ # FIXME: refactor this to reduce number of arguments and pass a more structured execution context @abstractmethod def success_nodes(self, command: Optional[Command], # pylint: disable=too-many-arguments num_successfull_nodes: int, success_ratio: float, tot: int, num_hosts: int, success_threshold: float, nodes: Dict[str, Node]) -> None: """Print how many nodes successfully executed all commands in a colored and tqdm-friendly way. Arguments: command (Command): the command that was executed num_successfull_nodes (int): the number of nodes on which the execution was successful success_ratio (float): the ratio of successful nodes tot (int): total number of successful executions num_hosts (int): the total number of nodes. success_threshold (float): the threshold of successful nodes above which the command execution is deemed successful nodes (list): the nodes on which the command was executed """ @abstractmethod def command_completed(self) -> None: """To be called on completion of processing, when no command specific output is required.""" # FIXME: buffer_iterator should have a more specific type annotation @abstractmethod def command_output(self, buffer_iterator: Any, command: Optional[Command] = None) -> None: """Print the command output in a colored and tqdm-friendly way. Arguments: buffer_iterator (mixed): any `ClusterShell` object that implements ``iter_buffers()`` like :py:class:`ClusterShell.Task.Task` and all the `Worker` objects. command (Command, optional): the command the output is referring to. """ @abstractmethod def command_header(self, command: str) -> None: """Reports a single command execution. Arguments: command (str): the command the header belongs to. """ @abstractmethod def message_element(self, message: MsgTreeElem) -> None: """Report a single message as received from the execution of a command on a node. Arguments: message (ClusterShell.MsgTree.MsgTreeElem): the message to report. """ class NullReporter(BaseReporter): # pylint: disable=abstract-method are all generated dynamically """Reporter class that does not report anything.""" def _callable(self, *args, **kwargs): """Just a callable that does nothing.""" def __new__(cls, *args, **kwargs): """Override class instance creation, see Python's data model.""" for name in cls.__abstractmethods__: setattr(cls, name, cls._callable) cls.__abstractmethods__ = frozenset() return super().__new__(cls, *args, **kwargs) class TqdmQuietReporter(NullReporter): # pylint: disable=abstract-method some are generated dynamically """Reports the progress of command execution without the command output.""" short_command_length = 35 """:py:class:`int`: the length to which a command should be shortened in various outputs.""" def _report_line(self, message: str, # pylint: disable=no-self-use color_func: Callable[[str], str] = Colored.red, nodes_string: str = '') -> None: """Print a tqdm-friendly colored status line with success/failure ratio and optional list of nodes. Arguments: message (str): the message to print. color_func (function, optional): the coloring function, one of :py:class`cumin.color.Colored` methods. nodes_string (str, optional): the string representation of the affected nodes. """ tqdm.write(color_func(message) + Colored.cyan(nodes_string), file=sys.stderr) def _get_log_message(self, num: int, num_hosts: int, message: str, # pylint: disable=no-self-use nodes: Optional[List[str]] = None) -> Tuple[str, str]: """Get a pre-formatted message suitable for logging or printing. Arguments: num (int): the number of affected nodes. num_hosts (int): the total number of nodes. message (str): the message to print. nodes (list, optional): the list of nodes affected. Returns: tuple: a tuple of ``(logging message, NodeSet of the affected nodes)``. """ if nodes is None: nodes_string = '' message_end = '' else: nodes_string = str(nodeset_fromlist(nodes)) message_end = ': ' tot = num_hosts log_message = '{perc:.1%} ({num}/{tot}) {message}{message_end}'.format( perc=(num / tot), num=num, tot=tot, message=message, message_end=message_end) return log_message, nodes_string def _get_short_command(self, command: Union[str, Command]) -> str: """Return a shortened representation of a command omitting the central part, if it's too long. Arguments: command (str or Command optional): the command to be shortened. Returns: str: the short command. """ cmd = str(command) sublen = (self.short_command_length - 3) // 2 # The -3 is for the ellipsis return (cmd[:sublen] + '...' + cmd[-sublen:]) if len(cmd) > self.short_command_length else cmd def global_timeout_nodes(self, nodes: Dict[str, Node], num_hosts: int) -> None: """Print the nodes that were caught by the global timeout in a colored and tqdm-friendly way. :Parameters: according to parent :py:meth:`BaseReporter.global_timeout_nodes`. """ timeout = [node.name for node in nodes.values() if node.state.is_timeout] timeout_desc = 'of nodes were executing a command when the global timeout occurred' timeout_message, timeout_nodes = self._get_log_message(len(timeout), num_hosts=num_hosts, message=timeout_desc, nodes=timeout) self.logger.error('%s%s', timeout_message, timeout_nodes) self._report_line(timeout_message, nodes_string=timeout_nodes) not_run = [node.name for node in nodes.values() if node.state.is_pending or node.state.is_scheduled] not_run_desc = 'of nodes were pending execution when the global timeout occurred' not_run_message, not_run_nodes = self._get_log_message(len(not_run), num_hosts=num_hosts, message=not_run_desc, nodes=not_run) self.logger.error('%s%s', not_run_message, not_run_nodes) self._report_line(not_run_message, nodes_string=not_run_nodes) def failed_nodes(self, nodes: Dict[str, Node], num_hosts: int, commands: List[Command], filter_command_index: int = -1) -> None: # pylint: disable=no-self-use """Print the nodes that failed to execute commands in a colored and tqdm-friendly way. :Parameters: according to parent :py:meth:`BaseReporter.failed_nodes`. """ for state in (State.failed, State.timeout): failed_commands = defaultdict(list) for node in [node for node in nodes.values() if node.state == state]: failed_commands[node.running_command_index].append(node.name) for index, failed_nodes in failed_commands.items(): command = commands[index] if filter_command_index >= 0 and command is not None and index != filter_command_index: continue short_command = self._get_short_command(command) if command is not None else '' message = "of nodes {state} to execute command '{command}'".format( state=State.states_representation[state], command=short_command) log_message, nodes_string = self._get_log_message(len(failed_nodes), num_hosts=num_hosts, message=message, nodes=failed_nodes) self.logger.error('%s%s', log_message, nodes_string) self._report_line(log_message, nodes_string=nodes_string) def success_nodes(self, command: Optional[Command], # pylint: disable=too-many-arguments,too-many-locals num_successfull_nodes: int, success_ratio: float, tot: int, num_hosts: int, success_threshold: float, nodes: Dict[str, Node]) -> None: """Print how many nodes successfully executed all commands in a colored and tqdm-friendly way. :Parameters: according to parent :py:meth:`BaseReporter.success_nodes`. """ if success_ratio < success_threshold: comp = '<' post = '. Aborting.' else: comp = '>=' post = '.' message_string = ' of nodes successfully executed all commands' if command is not None: message_string = " for command: '{command}'".format(command=self._get_short_command(command)) nodes_to_log = None if num_successfull_nodes not in (0, tot): nodes_to_log = [node.name for node in nodes.values() if node.state.is_success] message = "success ratio ({comp} {perc:.1%} threshold){message_string}{post}".format( comp=comp, perc=success_threshold, message_string=message_string, post=post) log_message, nodes_string = self._get_log_message(num_successfull_nodes, num_hosts=num_hosts, message=message, nodes=nodes_to_log) if num_successfull_nodes == tot: color_func = Colored.green level = logging.INFO elif success_ratio >= success_threshold: color_func = Colored.yellow level = logging.WARNING else: color_func = Colored.red level = logging.CRITICAL self.logger.log(level, '%s%s', log_message, nodes_string) self._report_line(log_message, color_func=color_func, nodes_string=nodes_string) class TqdmReporter(TqdmQuietReporter): """Reports the progress of command execution with full command output.""" def command_completed(self) -> None: # pylint: disable=no-self-use """To be called on completion of processing, when no command specific output is required. :Parameters: according to parent :py:meth:`BaseReporter.command_completed`. """ tqdm.write(Colored.blue('================'), file=sys.stdout) def command_output(self, buffer_iterator: Any, command: Optional[Command] = None) -> None: """Print the command output in a colored and tqdm-friendly way. :Parameters: according to parent :py:meth:`BaseReporter.command_output`. """ nodelist = None if command is not None: output_message = "----- OUTPUT of '{command}' -----".format(command=self._get_short_command(command)) else: output_message = '----- OUTPUT -----' for output, nodelist in buffer_iterator.iter_buffers(): tqdm.write(Colored.blue('===== NODE GROUP ====='), file=sys.stdout) tqdm.write(Colored.cyan('({num}) {nodes}'.format(num=len(nodelist), nodes=nodeset_fromlist(nodelist))), file=sys.stdout) tqdm.write(Colored.blue(output_message), file=sys.stdout) tqdm.write(output.message().decode(), file=sys.stdout) if nodelist is None: message = '===== NO OUTPUT =====' else: message = '================' tqdm.write(Colored.blue(message), file=sys.stdout) def command_header(self, command: str) -> None: """Reports a single command execution. :Parameters: according to parent :py:meth:`BaseReporter.command_header`. """ output_message = "----- OUTPUT of '{command}' -----".format(command=self._get_short_command(command)) tqdm.write(Colored.blue(output_message), file=sys.stdout) def message_element(self, message: MsgTreeElem) -> None: # pylint: disable=no-self-use """Report a single message as received from the execution of a command on a node. :Parameters: according to parent :py:meth:`BaseReporter.message_element`. """ tqdm.write(message.decode(), file=sys.stdout) class BaseEventHandler(Event.EventHandler): """ClusterShell event handler base class. Inherit from :py:class:`ClusterShell.Event.EventHandler` class and define a base `EventHandler` class to be used in Cumin. It can be subclassed to generate custom `EventHandler` classes while taking advantage of some common functionalities. """ # FIXME: not sure what the type of **kwargs should be def __init__(self, target: Target, commands: List[Command], reporter: BaseReporter, progress_bars: BaseExecutionProgress, success_threshold: float = 1.0, **kwargs: Any) -> None: """Event handler ClusterShell extension constructor. Arguments: target (cumin.transports.Target): a Target instance. commands (list): the list of Command objects that has to be executed on the nodes. reporter (cumin.transports.clustershell.BaseReporter): reporter used to output progress. progress_bars (BaseExecutionProgress): the progress bars instance. success_threshold (float, optional): the success threshold, a :py:class:`float` between ``0`` and ``1``, to consider the execution successful. **kwargs (optional): additional keyword arguments that might be used by derived classes. """ super().__init__() self.success_threshold = success_threshold self.logger = logging.getLogger('.'.join((self.__module__, self.__class__.__name__))) self.target = target self.lock = threading.Lock() # Used to update instance variables coherently from within callbacks # Execution management variables self.return_value: Optional[int] = None self.commands = commands self.kwargs = kwargs # Allow to store custom parameters from subclasses without changing the signature self.counters: Dict[str, int] = Counter() self.counters['total'] = len(target.hosts) self.deduplicate_output = self.counters['total'] > 1 self.global_timedout = False # Instantiate all the node instances, slicing the commands list to get a copy self.nodes = {node: Node(node, commands[:]) for node in target.hosts} # Move already all the nodes in the first_batch to the scheduled state, it means that ClusterShell was # already instructed to execute a command on those nodes for node_name in target.first_batch: self.nodes[node_name].state.update(State.scheduled) self.progress = progress_bars self.reporter = reporter def close(self, task): """Additional method called at the end of the whole execution, useful for reporting and final actions. Arguments: task (ClusterShell.Task.Task): a ClusterShell Task instance. """ raise NotImplementedError def on_timeout(self, task): """Update the state of the nodes and the timeout counter. Callback called by the :py:class:`ClusterShellWorker` when a :py:exc:`ClusterShell.Task.TimeoutError` is raised. It means that the whole execution timed out. Arguments: task (ClusterShell.Task.Task): a ClusterShell Task instance. """ num_timeout = task.num_timeout() self.logger.error('Global timeout was triggered while %d nodes were executing a command', num_timeout) with self.lock: # Avoid modifications of the same data from other callbacks triggered by ClusterShell self.global_timedout = True # Considering timed out also the nodes that were pending the execution (for example when executing in # batches) and those that were already scheduled (for example when the # of nodes is greater than # ClusterShell fanout) pending_or_scheduled = sum( (node.state.is_pending or node.state.is_scheduled or (node.state.is_success and node.running_command_index < (len(node.commands) - 1)) ) for node in self.nodes.values()) if pending_or_scheduled > 0: self.progress.update_failed(num_timeout + pending_or_scheduled) if self.global_timedout: self.reporter.global_timeout_nodes(self.nodes, self.counters['total']) def ev_pickup(self, worker, node): """Command execution started on a node, remove the command from the node's queue. This callback is triggered by the `ClusterShell` library for each node when it starts executing a command. :Parameters: according to parent :py:meth:`ClusterShell.Event.EventHandler.ev_pickup`. """ self.logger.debug("node=%s, command='%s'", node, worker.command) with self.lock: # Avoid modifications of the same data from other callbacks triggered by ClusterShell curr_node = self.nodes[node] curr_node.state.update(State.running) # Update the node's state to running command = curr_node.commands[curr_node.running_command_index + 1].command # Security check, it should never be triggered if command != worker.command: raise WorkerError("ev_pickup: got unexpected command '{command}', expected '{expected}'".format( command=command, expected=worker.command)) curr_node.running_command_index += 1 # Move the pointer of the current command if not self.deduplicate_output: self.reporter.command_header(worker.command) def ev_read(self, worker, node, _, msg): """Worker has data to read from a specific node. Print it if running on a single host. This callback is triggered by ClusterShell for each node when output is available. :Parameters: according to parent :py:meth:`ClusterShell.Event.EventHandler.ev_read`. """ if self.deduplicate_output: return with self.lock: self.reporter.message_element(msg) def ev_close(self, worker, timedout): """Worker has finished or timed out. This callback is triggered by ClusterShell when the execution has completed or timed out. :Parameters: according to parent :py:meth:`ClusterShell.Event.EventHandler.ev_close`. """ if not timedout: return delta_timeout = worker.task.num_timeout() - self.counters['timeout'] self.logger.debug("command='%s', delta_timeout=%d", worker.command, delta_timeout) with self.lock: # Avoid modifications of the same data from other callbacks triggered by ClusterShell self.progress.update_failed(delta_timeout) self.counters['timeout'] = worker.task.num_timeout() for node in worker.task.iter_keys_timeout(): if not self.nodes[node].state.is_timeout: self.nodes[node].state.update(State.timeout) # Schedule a timer to run the current command on the next node or start the next command worker.task.timer(self.target.batch_sleep, worker.eh) def _success_nodes_report(self, command: Optional[Command] = None) -> None: """Print how many nodes successfully executed all commands in a colored and tqdm-friendly way. Arguments: command (Command, optional): the command the report is referring to. """ if self.global_timedout and command is None: num = sum(1 for node in self.nodes.values() if node.state.is_success and node.running_command_index == (len(self.commands) - 1)) else: num = self.counters['success'] tot = self.counters['total'] success_ratio = num / tot self.reporter.success_nodes(command, num, success_ratio, tot, self.counters['total'], self.success_threshold, self.nodes) class SyncEventHandler(BaseEventHandler): """Custom ClusterShell event handler class that execute commands synchronously. The implemented logic is: * execute command `#N` on all nodes where command #`N-1` was successful according to `batch_size`. * the success ratio is checked at each command completion on every node, and will abort if not met, however nodes already scheduled for execution with `ClusterShell` will execute the command anyway. The use of the `batch_size` allow to control this aspect. * if the execution of command `#N` is completed and the success ratio is greater than the success threshold, re-start from the top with `N=N+1`. The typical use case is to orchestrate some operation across a fleet, ensuring that each command is completed by enough nodes before proceeding with the next one. """ def __init__(self, target: Target, commands: List[Command], reporter: BaseReporter, progress_bars: BaseExecutionProgress, success_threshold: float = 1.0, **kwargs: Any) -> None: """Define a custom ClusterShell event handler to execute commands synchronously. :Parameters: according to parent :py:meth:`BaseEventHandler.__init__`. """ super().__init__(target, commands, reporter, success_threshold=success_threshold, progress_bars=progress_bars, **kwargs) self.current_command_index = 0 # Global pointer for the current command in execution across all nodes self.start_command() self.aborted = False def start_command(self, schedule: bool = False) -> None: """Initialize progress bars and variables for this command execution. Executed at the start of each command. Arguments: schedule (bool, optional): whether the next command should be sent to ClusterShell for execution or not. """ self.counters['success'] = 0 self.progress.init(self.counters['total']) # Schedule the next command, the first was already scheduled by ClusterShellWorker.execute() if schedule: with self.lock: # Avoid modifications of the same data from other callbacks triggered by ClusterShell # Available nodes for the next command execution were already update back to the pending state remaining_nodes = [node.name for node in self.nodes.values() if node.state.is_pending] first_batch = remaining_nodes[:self.target.batch_size] first_batch_set = nodeset_fromlist(first_batch) for node_name in first_batch: self.nodes[node_name].state.update(State.scheduled) command = self.commands[self.current_command_index] self.logger.debug( "command='%s', timeout=%s, first_batch=%s", command.command, command.timeout, first_batch_set) # Schedule the command for execution in ClusterShell Task.task_self().flush_buffers() Task.task_self().shell(command.command, nodes=first_batch_set, handler=self, timeout=command.timeout) def end_command(self) -> bool: """Command terminated, print the result and schedule the next command if criteria are met. Executed at the end of each command inside a lock. Returns: bool: :py:data:`True` if the next command should be scheduled, :py:data:`False` otherwise. """ if self.deduplicate_output: self.reporter.command_output(Task.task_self(), command=self.commands[self.current_command_index]) else: self.reporter.command_completed() self.progress.close() self.reporter.failed_nodes(nodes=self.nodes, num_hosts=self.counters['total'], commands=self.commands, filter_command_index=self.current_command_index) self._success_nodes_report(command=self.commands[self.current_command_index]) success_ratio = self.counters['success'] / self.counters['total'] # Abort on failure if success_ratio < self.success_threshold: self.return_value = 2 self.aborted = True # Tells other timers that might trigger after that the abort is already in progress return False if success_ratio == 1: self.return_value = 0 else: self.return_value = 1 if self.current_command_index == (len(self.commands) - 1): self.logger.debug('This was the last command') return False # This was the last command return True def on_timeout(self, task: Task) -> None: """Override parent class `on_timeout` method to run `end_command`. :Parameters: according to parent :py:meth:`BaseEventHandler.on_timeout`. """ super().on_timeout(task) self.end_command() def ev_hup(self, worker, node, rc): """Command execution completed on a node. This callback is triggered by ClusterShell for each node when it completes the execution of a command. Update the progress bars and keep track of nodes based on the success/failure of the command's execution. Schedule a timer for further decisions. :Parameters: according to parent :py:meth:`ClusterShell.Event.EventHandler.ev_hup`. """ self.logger.debug("node=%s, rc=%d, command='%s'", node, rc, worker.command) with self.lock: # Avoid modifications of the same data from other callbacks triggered by ClusterShell curr_node = self.nodes[node] ok_codes = curr_node.commands[curr_node.running_command_index].ok_codes if rc in ok_codes or not ok_codes: self.progress.update_success() self.counters['success'] += 1 new_state = State.success else: self.progress.update_failed() self.counters['failed'] += 1 new_state = State.failed curr_node.state.update(new_state) # Schedule a timer to run the current command on the next node or start the next command worker.task.timer(self.target.batch_sleep, worker.eh) def ev_timer(self, timer): # noqa, mccabe: MC0001 too complex (15) FIXME """Schedule the current command on the next node or the next command on the first batch of nodes. This callback is triggered by `ClusterShell` when a scheduled `Task.timer()` goes off. :Parameters: according to parent :py:meth:`ClusterShell.Event.EventHandler.ev_timer`. """ success_ratio = 1 - ((self.counters['failed'] + self.counters['timeout']) / self.counters['total']) node = None if success_ratio >= self.success_threshold: # Success ratio is still good, looking for the next node with self.lock: # Avoid modifications of the same data from other callbacks triggered by ClusterShell for new_node in self.nodes.values(): if new_node.state.is_pending: # Found the next node where to execute the command node = new_node node.state.update(State.scheduled) break if node is not None: # Schedule the execution with ClusterShell of the current command to the next node found above command = self.nodes[node.name].commands[self.nodes[node.name].running_command_index + 1] self.logger.debug("next_node=%s, timeout=%s, command='%s'", node.name, command.command, command.timeout) Task.task_self().shell(command.command, handler=timer.eh, timeout=command.timeout, nodes=nodeset(node.name)) return # No more nodes were left for the execution of the current command with self.lock: # Avoid modifications of the same data from other callbacks triggered by ClusterShell try: command: Optional[str] = self.commands[self.current_command_index].command except IndexError: command: Optional[str] = None # Last command reached # Get a list of the nodes still in pending state pending = [pending_node.name for pending_node in self.nodes.values() if pending_node.state.is_pending] # Nodes in running are still running the command and nodes in scheduled state will execute the command # anyway, they were already offloaded to ClusterShell accounted = len(pending) + self.counters['failed'] + self.counters['success'] + self.counters['timeout'] # Avoid race conditions if self.aborted or accounted != self.counters['total'] or command is None or self.global_timedout: self.logger.debug("Skipped timer") return if pending: # This usually happens when executing in batches self.logger.warning("Command '%s' was not executed on: %s", command, nodeset_fromlist(pending)) self.logger.info("Completed command '%s'", command) restart = self.end_command() self.current_command_index += 1 # Move the global pointer of the command in execution if restart: for node in self.nodes.values(): if node.state.is_success: # Only nodes in pending state will be scheduled for the next command node.state.update(State.pending) if restart: self.start_command(schedule=True) def close(self, task: Task) -> None: """Concrete implementation of parent abstract method to print the success nodes report. :Parameters: according to parent :py:meth:`cumin.transports.BaseEventHandler.close`. """ self._success_nodes_report() class AsyncEventHandler(BaseEventHandler): """Custom ClusterShell event handler class that execute commands asynchronously. The implemented logic is: * execute on all nodes independently every command in a sequence, aborting the execution on that node if any command fails. * The success ratio is checked at each node completion (either because it completed all commands or aborted earlier), however nodes already scheduled for execution with ClusterShell will execute the commands anyway. The use of the batch_size allows to control this aspect. * if the success ratio is met, schedule the execution of all commands to the next node. The typical use case is to execute read-only commands to gather the status of a fleet without any special need of orchestration between the nodes. """ def __init__(self, target: Target, commands: List[Command], reporter: BaseReporter, progress_bars: BaseExecutionProgress, success_threshold: float = 1.0, **kwargs: Any) -> None: """Define a custom ClusterShell event handler to execute commands asynchronously between nodes. :Parameters: according to parent :py:meth:`BaseEventHandler.__init__`. """ super().__init__(target, commands, reporter, success_threshold=success_threshold, progress_bars=progress_bars, **kwargs) self.progress.init(self.counters['total']) def ev_hup(self, worker, node, rc): """Command execution completed on a node. This callback is triggered by ClusterShell for each node when it completes the execution of a command. Enqueue the next command if the success criteria are met, track the failure otherwise. Update the progress bars accordingly. :Parameters: according to parent :py:meth:`ClusterShell.Event.EventHandler.ev_hup`. """ self.logger.debug("node=%s, rc=%d, command='%s'", node, rc, worker.command) schedule_next = False schedule_timer = False with self.lock: # Avoid modifications of the same data from other callbacks triggered by ClusterShell curr_node = self.nodes[node] ok_codes = curr_node.commands[curr_node.running_command_index].ok_codes if rc in ok_codes or not ok_codes: if curr_node.running_command_index == (len(curr_node.commands) - 1): self.progress.update_success() self.counters['success'] += 1 curr_node.state.update(State.success) schedule_timer = True # Continue the execution on other nodes if criteria are met else: schedule_next = True # Continue the execution in the current node with the next command else: self.progress.update_failed() self.counters['failed'] += 1 curr_node.state.update(State.failed) schedule_timer = True # Continue the execution on other nodes if criteria are met if schedule_next: # Schedule the execution of the next command on this node with ClusterShell command = curr_node.commands[curr_node.running_command_index + 1] worker.task.shell( command.command, nodes=nodeset(node), handler=worker.eh, timeout=command.timeout, stdin=False) elif schedule_timer: # Schedule a timer to allow to run all the commands in the next available node worker.task.timer(self.target.batch_sleep, worker.eh) def ev_timer(self, timer): """Schedule the current command on the next node or the next command on the first batch of nodes. This callback is triggered by `ClusterShell` when a scheduled `Task.timer()` goes off. :Parameters: according to parent :py:meth:`ClusterShell.Event.EventHandler.ev_timer`. """ success_ratio = 1 - ((self.counters['failed'] + self.counters['timeout']) / self.counters['total']) node = None if success_ratio >= self.success_threshold: # Success ratio is still good, looking for the next node with self.lock: # Avoid modifications of the same data from other callbacks triggered by ClusterShell for new_node in self.nodes.values(): if new_node.state.is_pending: # Found the next node where to execute all the commands node = new_node node.state.update(State.scheduled) break if node is not None: # Schedule the execution of the first command to the next node with ClusterShell command = node.commands[0] self.logger.debug("next_node=%s, timeout=%s, command='%s'", node.name, command.command, command.timeout) Task.task_self().shell( command.command, handler=timer.eh, timeout=command.timeout, nodes=nodeset(node.name)) else: self.logger.debug('No more nodes left') def close(self, task: Task) -> None: """Concrete implementation of parent abstract method to print the nodes reports and close progress bars. :Parameters: according to parent :py:meth:`cumin.transports.BaseEventHandler.close`. """ if self.deduplicate_output: self.reporter.command_output(task) else: self.reporter.command_completed() self.progress.close() self.reporter.failed_nodes(nodes=self.nodes, num_hosts=self.counters['total'], commands=self.commands) self._success_nodes_report() num = self.counters['success'] tot = self.counters['total'] success_ratio = num / tot if success_ratio == 1: self.return_value = 0 elif success_ratio < self.success_threshold: self.return_value = 2 else: self.return_value = 1 worker_class: Type[BaseWorker] = ClusterShellWorker # pylint: disable=invalid-name """Required by the transport auto-loader in :py:meth:`cumin.transport.Transport.new`.""" DEFAULT_HANDLERS: Dict[str, Type[Event.EventHandler]] = {'sync': SyncEventHandler, 'async': AsyncEventHandler} """dict: mapping of available default event handlers for :py:class:`ClusterShellWorker`.""" wikimedia-cumin-36f957f/doc/000077500000000000000000000000001476500461000157115ustar00rootroot00000000000000wikimedia-cumin-36f957f/doc/Makefile000066400000000000000000000011361476500461000173520ustar00rootroot00000000000000# Minimal makefile for Sphinx documentation # # You can set these variables from the command line. SPHINXOPTS = SPHINXBUILD = sphinx-build SPHINXPROJ = Cumin SOURCEDIR = source BUILDDIR = build # Put it first so that "make" without argument is like "make help". help: @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) .PHONY: help Makefile # Catch-all target: route all unknown targets to Sphinx using the new # "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). %: Makefile @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)wikimedia-cumin-36f957f/doc/examples/000077500000000000000000000000001476500461000175275ustar00rootroot00000000000000wikimedia-cumin-36f957f/doc/examples/aliases.yaml000066400000000000000000000011321476500461000220310ustar00rootroot00000000000000# Cumin aliases configuration # # Cumin looks for an aliases.yaml file in the same directory of the loaded main configuration file. # Aliases are resolved recursively at runtime, hence they can be nested. # Aliases must use the global grammar and defined in the form: # alias_name: query_string # alias_direct: D{host1 or host2} # Use the direct backend alias_puppetdb: P{R:Class = My::Class} # Use the PuppetDB backend alias_openstack: O{project:project_name} # Use the OpenStack backend alias_complex: A:alias_direct and (A:alias_puppetdb and not D{host3}) # Mix aliases and backend grammars wikimedia-cumin-36f957f/doc/examples/config.yaml000066400000000000000000000070331476500461000216630ustar00rootroot00000000000000# Cumin main configuration # # By default Cumin load the configuration from /etc/cumin/config.yaml, but it can be overriden by command line argument # transport: clustershell # Default transport to use, can be overriden by command line argument log_file: ~/.cumin/cumin.log # Absolute or relative path for the log file, expands ~ into the user's home directory # If set, use this backend to parse the query first and only if it fails, fallback to parse it with the general # multi-query grammar [optional] default_backend: direct # Environment variables that will be defined [optional] environment: ENV_VARIABLE: env_value # Backend-specific configurations [optional] puppetdb: host: puppetdb.local port: 443 # [optional] Allow to override the default HTTPS scheme with HTTP in case the connection to PuppetDB is secured in # other ways (e.g. SSH tunnel) scheme: https timeout: 30 # The timeout in seconds to pass to requests when calling the PuppetDB API [optional] # [optional] Whether to verify SSL CA certificate: true to verify with the default system CA # bundle, a path to a certificate to verify against that certificate, or false to disable # verification. This is passed to requests's 'verify' option, for more details see: # https://requests.readthedocs.io/en/latest/user/advanced/#ssl-cert-verification ssl_verify: true # [optional] Path to SSL client certificiate and key for communicating with PuppetDB. # If only ssl_client_cert is set, it is assumed that the single file contains both the private # key and the certificate. For more details, see: # https://requests.readthedocs.io/en/latest/user/advanced/#client-side-certificates ssl_client_cert: /path/to/cert.pem ssl_client_key: /path/to/key.pem urllib3_disable_warnings: # List of Python urllib3 exceptions to ignore - SomeWarning # See the urllib3.exceptions module for available warnings openstack: auth_url: http://keystone.local:5000 username: observer # Keystone API user's username password: observer_password # Keystone API user's password domain_suffix: openstack.local # OpenStack managed domain, to be added to all hostnames nova_api_version: 2.12 timeout: 2 # Used for both Keystone and Nova API calls # Additional parameters to set when instantiating the novaclient Client client_params: region_name: region1 # Default query parameters. The ones set by default anyway are: status: ACTIVE and vm_state: ACTIVE [optional] query_params: project: project_name # Parameter name: parameter value knownhosts: files: # List of SSH known hosts files to load - /path/to/known_hosts # Transport-specific configuration clustershell: ssh_options: # SSH options passed to ClusterShell [optional] - 'some_option' fanout: 16 # Max size of the sliding window of concurrent workers active at any given time [optional, default: 64] # Kerberos specific configuration [optional] kerberos: # Whether the SSH authentication to the hosts will be done via Kerberos. If true ensures that a valid Kerberos # ticket is present for the current user. [optional, default: false] ensure_ticket: false # Whether the check for a valid Kerberos ticket should be performed also when Cumin is run as root. # [optional, default: false] ensure_ticket_root: false # Plugins-specific configuration plugins: backends: # External backends. Each module must define GRAMMAR_PREFIX and query_class, and be in Python PATH - external.backend.module wikimedia-cumin-36f957f/doc/examples/transports_state_transitions.png000066400000000000000000000622411476500461000263160ustar00rootroot00000000000000PNG  IHDRP[/sRGB@IDATx/-m)n-@ŭxpC"-,@ŝ- nPR]EK_z=<>;3;s33rΜ9D!`@<:ig!`n!`!` 5,C01P{ ChtFmB{=;︷~[ן|g}k~7?q?O{,?4L㦝vZ]n:]:nfӥn'oe r!0X}n9׿^xsϹ~w?0 # N6d=%G?Pa}/*&駟:r9|_WnekU!,㌁6 Y?C?YGy=`|;2.s] gqF7Dh-M?+Ro<0s.D]vY2>¨ C @cTumݦ ?2JRK- j%?fcM?n5tk0"` .4U;`~-ܢ gne/3heqafyC{0)o <@3| ⋕zQGK=b6aEF`\ip4G +8upz?餓y68㢵;_|vXz&ZC "PJ㢋.fEs9|I&qGyBM9Dݎ@G+j<̣OQCVJ@G~{7bu/~ѭڱ"<P|Xcq]c1>#gF &@7 _rSL1E!Pha'跿/6jYd`dt o'3I8|lc;\s)|t !]='a_2}\vC-ipu22@e';QFL9$c9X+E:RMD"6F+ˏ+Àh;0PvHL(/$b_O=ԞdEB<̬v v!0+dgAm'etAtW\qNS }u裏49}dNPs} _G7aתQ}2̃Q2~]*+⚵]C +R izhI'UO>Dkc)ˇ0G|y_u h'<{hOuԬzzɬE=(Zׯ_݆@J IEvVeL^kFv BHB?tjjFۡ !@}K/afL.F@ZJ[om2d4|믿v|05h]\ZpPY3! o>Y-/%]Cmt-Cá> Mtm?h ѝ)*yG8k=jCgx($!` 4XU(\IM?Oe8ϵI7ʃi_|Ic1nmZ[@hndR4/^^J$°+yR.#rcK`pL.?=Vw16HI/FږqՍvĵ_~y]iōkȔ(^{x'նct])@) %)!bD1aܫ[z@t*:~9@RDprI1s|ӒwDa$&Ұ@>H}ݽb[캝f5"Eѥ|ͧR+o\|}|駛n#/'r-nVpiU.r':V'& 02 ցF M92I hhk,Zw0j]@(rP2$Fk0Ixe i2O@pdu}KlAs @)a$b'LJL0lH愯ҧEOJ@s1\|U{.D$쳏# gUX:@hyWפZj$M d%$J>#ws-ц@(-e<5eHPk{NJ7m1$ !nkոzG6w.gR3Ѐ/502 j~p*6 0]|Ūd$iȆ@0-j1뺔PF1j(GHfYD㽰B ~sюzugv#?(kHqٟʰs$91a!w'YI=đIakY(BX+nP?=֞!c1@J|%03[X`pSkb0AT M7V9`)Am&2,YdPy@Mքѭw}ZS FH~4Y?HA*}(0MT[ OeJCܦ]&e4arXCHiWN1cvmmCHK E33OȆD fDUfG¸I3H1(J:O#CbCс_~i& %ȍ;]kEt'|R%%)Ҷnw>}S;?!Wvxd&L~_vѣO;#C1x8evNGo &5,ײ߆@ ` # Zsb71D^{0PO8 wwkإ b ` μH@AϡCAJF!Pc| Zy~ '/B#2.`c<h'ݗ3ѺC]l:tyȑ#ofHC 3= B `((v1&;HEa?OM7n f41SO="LH[I5iFhFg Cu{{u38s?wqG8ֆ@)0ځa.**_}C/"5.{"UF 2!` ͣ__~(PQǞ|~aF3~6i3:16㏮fS)  I$U^}Ug)nGhG SOS6_?|7s1c(㄁V 32ʂ164E]WMv9 FFrdЯ8C$ !Ô!PyЊ6}' Hc~#$P3$CǶw#@8{ 'O~-Q~ I]16+&Z)| d|X>+be`d@`1(q4px`%nouw}mݦg?[r% +RK-w+vcࡇ+ʕ0Vk[L&tn+@O?a&k .w{TD:^mՔyax* >*07nM6QnCcMc{Gzv0zK.QiEi:jnEvAJGmFeTjc)F~؎5i}N"A `J`Fݏ1Ќ|($"oN?t7|i9?n5H2VLb*),˩#B*fSvX0`@?B"q1b#7'L=Qp a(+*z@@5&OҾG@",NFK#(g6zwvWv,H΢g]&hZ .@}81$9Q~ (Mܒʨd"3<6|W934裏V p @\;;Bhk\.=@3ĘzLZq;餓>a8U{kSPc!OR%jAߘgcՎivqG_*}k@'^ƮQL8iwq49JR4l7tSY':R׏{QFeYFǕ4{FFhM2}3f:NˊiTL4ND*-!$ڝr)TsǶzh}fO= jTlf8~I('xo3fMʈjD8y\Ww=Ρ}lj'r GKTfc x㍖%P*[RBcv˨ɚ eRlV$t;'V5W7s3̈рDV}@qϟQ`;[}+CHe H2hU[Ml8.;QVSc/rE$z~Knn Ht }ݧ%ic7v6wj, `=sޖr"FD$Ќƍ;̆ʔ /2{=2 Q #x#GG8)0hNGSćjÜZk-eGuZRpJJ'g_}tW٨(i 3ieI ENݛ~0ʡs밗ۋ'R݋DZ1c[8F^}^$1£LjB~9"V ssRv76>U8.W.ir\kS>~LᙎJD|7<$5?0zBFaA+L4luˀ-DR"oH2HH,n Ea;:}FRѽu1 F*n58LJ?600FgɴNPf&L E_<8B"jͥ3:BBoxcq.azl0s9a8묳pĎFLA s3fcTH5*0mTR ҋ~1瞾_~^te۪n@t^b-0HIrAf]vR%` 01=<{ $N7q,$XUW]EmE"zqO>w*=GXuq }}/1^^B\Z+uD#z}m,?`/^$[:X|RFF#5)x۪nF[czz~x0we*J+ @a3x+;L:YD*6\7b4}X%K,,Z[n wYgEME OUہ!LKXJ9/?!` 4 .$iD*MKe/uIaiN^mRtu^u݁uUܝp?E0S$W￯Hޖl)>)ɕ[9c'?"?'{%itvD< SI`&G6 pӟAkX]GJO{Zkq5jnT<3~&z=6+|=dZ܎~iqb_0D[@Ӕ>Cg&dw 7mF|HpbTt[oEsZ1Vs 4X簖6&Eʈ"b:T-5h'ehN=Tw:(DjevQkMr"xJnQJ[.ns:@N6n:1A&M™5$ 'Q>D+?YSq Jr"s55 7^8Lp6IXB}jKTTD3o=;y⎾3%;҈oGvF;0"0й瞻)$Z{챚*G}z(QoTM$x[ynȐ!ZCBKkf/8%YsOqI !;I5pAy!uM7]'VkJ^UeBAR6LZN }8#}g2c`jO-" -UC1p!Ԍ#N$f&zhgt`WBK6}8صGВƅ4L*oD8A18rQu%I꽼k~,? `sNG%3@J*58)I+Z.$RW6 wL+BeXf=PeruJfHmQ7D"7&L.QݠCa SI AYS7$7.=Q LbYK^S_f\&iL.} ` pEY OdLA"FC]I'i]upvH.&T?8<$<(,]g3}7֠`%W@-t'zA$6`L 0R<̺~,Hx,Ȣ -R"5 GnjdfTʒp||#v&RZ;餓67ڂ1`Ɗ/TLɮkǂXbʍ[!PO kX-ioWޘ1{٦)֫ HUEAh #j@+|C=M3n<H9)Iqڇ1ЄXKo 4!rI1ڋ1Єx3}teOnFZ EhB1 bq ͈CkM9$M8LHfOnh:8Z+Chf1R#~c ;dh#~c ;y0Y#`hr "` 4H6}O`թ&Vbc H q* Ԓ6c- Ʉ)uah8ͦ5@MFhC~kuMkb1G;b vZ5&V#bc-$߉7J@Zi@[ H-i&%@[c-i&%@[ZKZR q1%@[({v}bSб}yDh s1G g))|=dl{^0@I|-W@Bc;rq H-@cd c- s=gpktIyCh#(Ibnh \!` `m$;$Wmr8`DIbnhL ` ɡ0R5q8 6q69VƸI8)?v!Y&uDyذax{[o_}w*u8 ~zeW\z9 R0oo-MeB`gWfkgض1M ?'h"Bj4;;C:Chc@0`lɚ8!Cyq$ +o}kBZ;|CzOn2j쒔dfsݸ]L;C^2ĸaͽ‚ =aW:ofR~a] s#sϭjHHD0!{,H?4 7t/ C:CkB -R?@3<vkb}EB[n=CʴJE% i饗v?͐N_z0)xr]s5U-~ڵ 1FC{qJûf?L.sW\q{וl^j: /""~nsǏ? Wnfiv[!` 0NjLkOr_|0a#[l~_wݴÝw鮺*wM7iV(^fT <b 0* @ F5:0O;4wEDJsuftu]{WlѥGеM駟:TaY0tI‚!t6]tQ5M2$uzb 1FQ W^Qqس;|1znզC駟.% o Ƙgj*Vr,S{jB-v nV_}ub]˨pcb2u{gGo~v"_}^H=#>|I 2RUX\ |„.y䑖۶;Xc}.:-}yđX$+'{w1|8/Ro/Vg;DT~^.nwi5Wu/L_rߋ/4w庲mQxDb(YĦzH믿~>㩶oX3v" ȄU]$y183b4?n%\RoxHU i10PBr>.jo?jM1[m{'4t&GA"h3$I 5Sx㐟67c?>Fcuh(]s5]3QwqnvlJi_UwqF10ۥA2L^t]#:߸(\6pC/KR&S[OIhlLu>fHpm#}Q(E >si^\x5uz,H1b%͓" jsMs9, 6pOByI'չboƘBNB#Hb5gj}<ꨣ4V :>ws@n,`d!`Ej/xX$&j͂(Aq7ܤ q"492yI?2J^Ln6|s Z'4QehtFCa>eQ+ !iQ[)x@I%p>AZBZ-y1z_ɮnk %8AJ40B-n32^*Z_zjmI.%KKFu $ɂ+'o/a^nåJrf'|L~=X=FJ{aMM!'%LHT2D׫⴯I%$ W]w4(\>bO]k),S->`/g]FeGTʴ[l1/^to -@qﾛ8m @z`usgD8?3=(;/]^\~vU"X=Y>"Q-2xX6Zz HL!jNmX7 >;oT|w*H7ncdk8Swߟ8lOTo?,oZBO/vZDH5bb)1AIr׸dі|l.\qn^rᒠ#+cA-Le/ʹk -j) _򗪻AoV twW^y ˂0a$̨ŸbWJ nEЇW.ҏs=W]]ngtbʢyk!2 0b/SȖ/Y ))MUSN9G]y#~{x5Ww!~~'N#8‹!"V}q>kfXy:R;^]ƌj1efJ lL2 yn:{cG7JogH'!H-K{ԊdG9suV~Lq+i>:H+RYR8䤭qT5L)PDZh4;>k!0O0OCuD8.e8X$1:qw̠;VZI<@MŔz1Ƞh-DE8;)IbzEws(|d'%'>hNǰI~W;@tv2E&t!T$4ݡ0Ky=ӠzȋJiW %RȮn,3;^7Ƃ . ӾCS+,PӼ᳻첋ӋjM[[PΨ4tv;{cꇻ َFl{.!uUOqO[kUW,WT 迵h!ٯ_?mWwI<6F@hn'x I1?MÇU9~̱裫V2Dh\"<zݟsG)̍G&~CJ%ÔY^wsC=5>TȝJa:n;kZV!#`gB ;Z"x)7T.wPsTJ:( _V[MJ&(}fćT9wC9DKib!HZB>zLIzO`OEU*Z4T>1 0n"x}*MjJS'y8 Gaѿat"M^$#/}8uH(M%E #5_&x^@ "-x^aWOy=Cp&G7ia~ 7T%zMD$IXU{P ldBݦ^zI˟61JLgd2^mzI"2YfE?.d[p\If:K CnуHE-:LTG6!bόN/jz#@w Mbu$&[JGZk# ꚺiGU-$Ks5zHtPM h^r%N=j?Q}璉m֒*Dh8u5@^#Gś#tRWB+F:Ix % UCR#j@a-ޝh(,ڠ ^tA#|t9I?ǚ>,԰"? $$fcGb&ka8DjcFB8#Cg>nhkݠA4RwNʔ4S-5%aRQm@bB\fzF9KiauUd/ı<midB02K a>|5j;s}0;@C_>{3[F@5@K@J@2}СCڎx1Y.0v;7s/`\ˎ*d2*'@%%'p=Y d@cꩩeT>J@O,;1$!ʈZGE"0PpD+KRXQyn몗i(dV* OrJyb}Á4LS (x2M"1O<}H K+zQQ#Pj*K7&Dy )vE@."j\vGw.D ۦdo~nfE/*?<@:j%u/ed@1$>f̘&IrO:3.#۾Q!PZ$PHr-~?0<xWJք{W J@H͟3PQ)wީ5΋ˢKѻY92:=3vk/ *-QP71PƄc= [dXF0acMiN)ncOf Sc-ԘXgk#PZZ;W_}uԽܰxLr_#Ns#GTf ^TJZT kFm9(8ssy2 Q? )3 HW?Oi(X-FOHgW_}IύD(Lq(fd7cYCYF]v]{nI'whi~z.EGl_gJAx ~9/:=/:=EO=%ScTk\SƍR.U{DyaنTRiTY_?J/:2%^K~և]dv7 {2{1PqTˤO=ׯ#+}oc$Fv*cK3ǀmgg*:蠃u諯=e>䓪ܭQmPkTO'xbuboZyYg}9.sT10Tq4p~eͬϱ$/J;G"0/ u]׋o6"v$+㕙"$6JRMb%r.LߣCËG+;cGҼZ[K_n&l5gʇT#R0PP:`->twaY;ܘ1cw ĢfqFK馛86U#@ۮzs {{l!P e <`0ɾȮj_Ǖ}!;ＳdO2YSZ3hgD5Wt(OYg5CG}j5 ӄ =tP7~x ɫumAZi\x}]}xP{EҨQr&(Zo[o }glFT6$9 |X3q@"qa"$0,a܀釥Rb8_~\nx@Q`\z-="H/.dVeX3DXSJgX;ƌs L|CVc=_ .ە1CEIkw0A`,1vb)*axy?1wbi\n(`3RY+(o FJ  Z믿 tp?,ANqp -Pe׎kG2 d+ӚD),ٞx 2, C0W F0KwB]y!cMx_uM</[zU"c_LWw>vfX{(_<,nfe|I 3`3|qa%X"meu饗:V~ǦS14aەaD Il&0s91fB I(&nӅO7#.pH|16#xe /O "!f E\aH~VWIrZjZH " (`1rHK/bѯ#82f[ yѱg{᜴.3>/y*xP4RI$L/F s=~mV5$K+!C(%aD2mA#aȉ'E7D/}QI,ҍDl,?ob12e.ԭ4Kgu{2PRjø~@ PQ]1bK>0Ҵu~bLi3X#6Lw[ѣIb+5=ܳ*Y3k$Pt:w%P4@:Z5) v<*TF9[0IĕJlƐW'; 3d1|E]}Mٸp *I\yz# i *avp~/!>∝N_os811Eyn2 B߱~1ZŊʗ]6aT} ^WbR\!aqT#}3PI㈺jL:2 G4qر Lw7~r!$A&݋5Zq ,mx:×јg 6 eviF@ Ro[2pofz떈." Ǩ1xH0]${Y="څs:DPb<⋣gU-S %pli:JXó&\P<Y_گB@GNWmJI Umڿ# it*OSք7$Hz^3d/9(c55F x'U:æ֒M1p-_$2al=ztݩmJBSTY=4ZFPK-zwQ @נmOеeIh+&ITΜDX4I/'I+0=$/ҠOB蔹O\$di3K_h)_,1H";c@o @%986IR("L`J;dW~\|ssz-)ķF&hɢI^^tCIDJBpDVԩY׭"z,2ϬSkr(%97H-*!EⴑDz E]TcDDʣOyKV^^pCI` F&H=ipp'*%xLsdvbʖP(!z6DI DI/= QQCR=IȪybi k_FBM=O) ̮1w00OjKQH>Ztf 낹H=ZWJHƪkh8]Pj. F L4ʂ{-TوU_`EQk $VM4hFBݠb#:$s9Y0P, KkͻA,.WS1ۀROX~6c}3N,ɹE!4DG(l  "ϸf"m5NT) idi'1UT+t뭷z|Ȁjj~($ rT}«xUQ~ʭC@]4zy38C'Pe]fLH=8 ^TIs>:վGb]||d(r]k_ A:ShvE♢69:NAӗn4~| vV/cʉė%`RGDI$8n=5a{*ѭ>IV{m /T2 SR ! 82k 3&$<Aè!zx_v!:"+b6 Ѯdr2SWˆFZƷ,^0,w!% JGsFqԡpHpfΉ{,FQe,ԋbӊΰD3Rǎ) V`ia *׺o>^!HLͱCc9)K {9/z'F v(# ^t h0?|}Q)Δk4D%||Z!>(}_1`|S砆x7ub̃yaryM5٭X5'ZG%, 0QV~eAuBiTCos5F1F$ A -:OO/jC?O|wkhq$ơcdG44^%5W^紶 ʃLv܂>+2ң"iz2Yq}`ܠ র4Yi藯fR/ED+ac|RB>X'dI!ގ;ＳbT L|jJQZ}sUW釁u>eL2FAɌ/9_v3q]}u!_3>LӘ)pwyG?\ M3P@b$Q-T zH3Lni'B>iT S'xDe3l),Ufn]b:)f cK`T1Q}B\[d `0{D܀3P)a ^+g}VU%VظyKB`̹e8Vm>fey `!وA( }C' ))$>|:<(N$_,H8a1J$UgqjY^Sh=IlvRΕk<3"$C3d LՍ@a3U$'m7`hs(E(~y+J"h9\l:A? ]:K3C d-.gt)e\ d=`kU'S Vv†OBOd+I*u0L^֊)}%/0% E!Ȯt(7H"ɊY@ L J^ H!OESF¨Df[7:p% IHz}$ܓ$ ׼ ֻzID -eYz 4z(&,H0`m{{ WI2bm\/ q2{ uP1opP䐉[LjyF|`6 4 ImC "":ƴZH!}ҦT+A /qMR0 uM T~c̣8Z>͂ƇƹkPC+$T ?-2EEA"ug}kc' 0eNGKd EdŷJIIkȀ8'|eU[o=- KeѯI5II2LRctZH,'95$߀I2dWN$Lc2$$4뮻5Q9 %WbLI{HIeʛB}$h-k (S(eDAfoe@͚PJz:UoIxXQYMSk>|Xz[Q?dgczUiGM\괰e=`2qSL, Wikimedia Foundation, Inc." author = u'Riccardo Coccioli' # The version info for the project you're documenting, acts as replacement for # |version| and |release|, also used in various other places throughout the # built documents. # # The full version, including alpha/beta/rc tags. release = meta_version('cumin') # The short X.Y version. version = release # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. # # This is also used if you do content translation via gettext catalogs. # Usually you set "language" from the command line for these cases. language = 'en' # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. # This patterns also effect to html_static_path and html_extra_path exclude_patterns = [] # The name of the Pygments (syntax highlighting) style to use. pygments_style = 'sphinx' # If true, `todo` and `todoList` produce output, else they produce nothing. todo_include_todos = True # -- Options for HTML output ---------------------------------------------- html_theme = 'sphinx_rtd_theme' sphinx_version_parts = [int(i) for i in sphinx_version.split('.')] if sphinx_version_parts[0] == 1 and sphinx_version_parts[1] < 6: html_use_smartypants = False # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, # so a file named "default.css" will overwrite the builtin "default.css". html_static_path = ['_static'] # -- Options for HTMLHelp output ------------------------------------------ # Output file base name for HTML help builder. htmlhelp_basename = 'Cumindoc' # -- Options for manual page output --------------------------------------- # One entry per manual page. List of tuples # (source start file, name, description, authors, manual section). man_pages = [ ('cli', 'cumin', 'Automation and orchestration framework written in Python', [author], 1), ] # -- Options for intersphinx --------------------------------------- intersphinx_mapping = { 'python': ('https://docs.python.org/3/', None), 'requests': ('https://requests.readthedocs.io/en/master/', None), 'ClusterShell': ('https://clustershell.readthedocs.io/en/v1.8.1/', None), 'keystoneauth1': ('https://docs.openstack.org/keystoneauth/latest', None), 'novaclient': ('https://docs.openstack.org/python-novaclient/latest/', None), 'pyparsing': ('https://pythonhosted.org/pyparsing/', 'pyparsing.inv'), } # Napoleon settings napoleon_google_docstring = True napoleon_numpy_docstring = False napoleon_include_init_with_doc = False napoleon_include_private_with_doc = False napoleon_include_special_with_doc = False napoleon_use_admonition_for_examples = False napoleon_use_admonition_for_notes = False napoleon_use_admonition_for_references = False napoleon_use_ivar = False napoleon_use_param = True napoleon_use_rtype = True napoleon_use_keyword = True # Autodoc settings autodoc_default_options = { # Using None as value instead of True to support the version of Sphinx used in Buster 'members': None, 'member-order': 'bysource', 'private-members': None, 'show-inheritance': None, } autoclass_content = 'both' # -- Helper functions ----------------------------------------------------- def filter_namedtuple_docstrings(app, what, name, obj, options, lines): """Fix the automatically generated docstrings for namedtuples classes.""" if what == 'property' and len(lines) == 1 and lines[0].startswith('Alias for field number'): del lines[:] # Keep track of documented classes to avoid annotating both class and __init__. # Necessary when using autoclass_content 'both' and add_abstract_annotations(). _cumin_documented_classes = set() def add_abstract_annotations(app, what, name, obj, options, lines): """Workaround to add an abstract annotation for ABC abstract classes.""" if what == 'class' and len(getattr(obj, '__abstractmethods__', [])) > 0 and name not in _cumin_documented_classes: lines.insert(0, '``abstract``') _cumin_documented_classes.add(name) def setup(app): """Register the helper functions.""" app.connect('autodoc-process-docstring', filter_namedtuple_docstrings) app.connect('autodoc-process-docstring', add_abstract_annotations) app.add_css_file('theme_overrides.css') # override wide tables in RTD theme wikimedia-cumin-36f957f/doc/source/configuration.rst000066400000000000000000000023601476500461000226130ustar00rootroot00000000000000Configuration ============= .. _config.yaml: config.yaml ----------- The default configuration file for ``cumin`` is expected to be found at ``/etc/cumin/config.yaml``. Its path can be changed in the CLI via the command-line switch ``--config PATH``. A commented example configuration is available in the source code at ``doc/examples/config.yaml`` and included here below: .. literalinclude:: ../examples/config.yaml :language: yaml The example file is also shipped, depending on the installation method, to: * ``$VENV_PATH/share/doc/cumin/examples/config.yaml`` when installed in a Python ``virtualenv`` via ``pip``. * ``/usr/local/share/doc/cumin/examples/config.yaml`` when installed globally via ``pip``. * ``/usr/share/doc/cumin/examples/config.yaml`` when installed via the Debian package. aliases.yaml ------------ Cumin will also automatically load any aliases defined in a ``aliases.yaml`` file, if present in the same directory of the main configuration file. An aliases example file is available in the source code at ``doc/examples/aliases.yaml`` and included here below: .. literalinclude:: ../examples/aliases.yaml :language: yaml The file is also shipped in the same directory of the example configuration file, see `config.yaml`_. wikimedia-cumin-36f957f/doc/source/development.rst000066400000000000000000000045441476500461000222740ustar00rootroot00000000000000Development =========== Code Structure -------------- Query and global grammar ^^^^^^^^^^^^^^^^^^^^^^^^ The :py:class:`cumin.query.Query` class is the one taking care of replacing the aliases, building and executing the query parts with their respective backends and aggregating the results using the global grammar defined in :py:func:`cumin.grammar.grammar`. Once a query is executed, it returns a :py:class:`ClusterShell.NodeSet.NodeSet` with the FQDN of all the hosts that matches the selection. Backends ^^^^^^^^ All the backends share a minimal common interface that is defined in the :py:class:`cumin.backends.BaseQuery` class and they are instantiated by the :py:class:`Query` class when building and executing the query. Each backend module need to define a ``query_class`` module variable that is a pointer to the backend class for dynamic instantiation and a ``GRAMMAR_PREFIX`` constant string that is the identifier to be used in the main query syntax to identify the backend. ``A`` is a reserved ``GRAMMAR_PREFIX`` used in the main grammar for aliases. Some backends are optional, in the sense that their dependencies are not installed automatically, they are available as an ``extras_require`` when installing from ``pip`` or as ``Suggested`` in the Debian package. Given that the ``pyparsing`` library used to define the backend grammars uses a BNF-like style, for the details of the tokens not specified in each backend BNF, see directly the code in the ``grammar`` function in the backend module. Running tests ------------- The ``tox`` utility, a wrapper around virtualenv, is used to run the tests. To list the default environments that will be executed when running ``tox`` without parameters, run: .. code-block:: bash tox -lv To list all the available environments: .. code-block:: bash tox -av To run one specific environment only: .. code-block:: bash tox -e py311-flake8 It's possible to pass extra arguments to the underlying environment: .. code-block:: bash # Run only tests in a specific file: tox -e py311-unit -- -k test_puppetdb.py # Run only one specific test: tox -e py311-unit -- -k test_invalid_grammars Integration tests are also available, but are not run by default by tox. They depends on a running Docker instance. To run them: .. code-block:: bash tox -e py311-integration tox -e py39-integration-min wikimedia-cumin-36f957f/doc/source/docutils.conf000066400000000000000000000000541476500461000217050ustar00rootroot00000000000000[restructuredtext parser] smart_quotes = no wikimedia-cumin-36f957f/doc/source/index.rst000066400000000000000000000005641476500461000210570ustar00rootroot00000000000000Cumin |release| documentation ============================= An automation and orchestration framework written in Python. .. toctree:: :maxdepth: 3 introduction installation configuration cli api/index development .. toctree:: :maxdepth: 2 release Indices and tables ------------------ * :ref:`genindex` * :ref:`modindex` * :ref:`search` wikimedia-cumin-36f957f/doc/source/installation.rst000066400000000000000000000032551476500461000224510ustar00rootroot00000000000000Installation ============ PyPI ---- Cumin is available in the `Python Package Index`_ (PyPI) and can be installed via ``pip``: .. code-block:: none pip install cumin The dependencies of the optional backends are listed in dedicated ``extras_require`` keys in the ``setup.py``. To install Cumin with the support of an optional backend run for example: .. code-block:: none pip install cumin[with-openstack] Debian package -------------- Starting from Debian 12 (*"bookworm"*) Cumin is available directly from the official `Debian repositories`_. The Debian package for each release is also available for download on the `Release page`_ on GitHub, along with its GPG signature. To build the Debian package from the source code use ``gbp buildpackage`` in the ``debian`` branch. See the `Source code`_ section on how to get the source code. The dependencies of the optional backends are listed as ``Suggested`` packages. Source code ----------- A gzipped tar archive of the source code for each release is available for download on the `Release page`_ on GitHub, along with its GPG signature. The source code repository is available from `Wikimedia's Gerrit`_ website and mirrored on `GitHub`_. To install it, from the ``master`` branch run: .. code-block:: none python setup.py install .. _`Python Package Index`: https://pypi.org/project/cumin/ .. _`Wikimedia's Gerrit`: https://gerrit.wikimedia.org/r/#/admin/projects/operations/software/cumin .. _`GitHub`: https://github.com/wikimedia/cumin .. _`Release page`: https://github.com/wikimedia/cumin/releases .. _`Debian repositories`: https://packages.debian.org/search?keywords=cumin&searchon=names&exact=1&suite=all§ion=all wikimedia-cumin-36f957f/doc/source/introduction.rst000066400000000000000000000212351476500461000224670ustar00rootroot00000000000000Introduction ============ .. include:: ../../README.rst Main components --------------- Query language ^^^^^^^^^^^^^^ Cumin provides a user-friendly generic query language that allows to combine the results of subqueries from multiple backends. The details of the main grammar are: * Each query part can be composed with any other query part using boolean operators: ``and``, ``or``, ``and not``, ``xor``. * Multiple query parts can be grouped together with parentheses: ``(``, ``)``. * Each query part can be one of: * Specific backend query: ``I{backend-specific query syntax}`` (where ``I`` is an identifier for the specific backend). * Alias replacement, according to the aliases defined in the configuration: ``A:group1``. * If a ``default_backend`` is set in the configuration, Cumin will try to first execute the query directly with the default backend and only if the query is not parsable with that backend it will parse it with the main grammar. Backends ^^^^^^^^ The backends are the ones that allow to select the target hosts. Each backend is free to define its own grammar. Those are the available backends: * **PuppetDB**: allow to select hosts querying the PuppetDB API for Puppet facts or resources. See the :py:class:`cumin.backends.puppetdb.PuppetDBQuery` class documentation for the details. * **OpenStack**: allow to select hosts querying the OpenStack APIs to select based on project, instance name and so on. See the :py:class:`cumin.backends.openstack.OpenStackQuery` class documentation for the details. This is an optional backend. * **KnownHosts**: allow to select hosts listed in multiple SSH known hosts files that are not hashed. See the :py:class:`cumin.backends.knownhosts.KnownHostsQuery` class documentation for the details. * **Direct**: a fallback backend without extenal dependecies with :py:class:`ClusterShell.NodeSet.NodeSet` group expansion capabilities. See the :py:class:`cumin.backends.direct.DirectQuery` class documentation for the details. * **Custom**: is possible to plug-in custom backends developed externally from Cumin, as long as they: * are included in the Python ``PATH``. * define a ``GRAMMAR_PREFIX`` module constant that doesn't conflict with the other backend prefixes. * define a ``query_class`` module variable that points to a class that inherit from :py:class:`cumin.backends.BaseQuery`. * are listed in the configuration file in the ``plugins->backends`` section, see :ref:`config.yaml`. An example of external backend can be found in the source code as part of the tests in the ``cumin.tests.unit.backends.external.ok`` module. Transports ^^^^^^^^^^ The transport layer is the one used to convey the commands to be executed into the selected hosts. The transport abstraction allow to specify different execution strategies. Those are the available backends: * **ClusterShell**: SSH transport using the `ClusterShell `__ Python library. See the :py:class:`cumin.transports.clustershell.ClusterShellWorker` class documentation for the details. It's possible to set all SSH-related options in the configuration file, also passing directly an existing ssh_config file. Examples -------- CLI ^^^ Simple example without fine-tuning the options: * Execute the single command ``systemctl is-active nginx`` in parallel on all the hosts matching the query for the alias ``cp-esams``, as defined in the ``aliases.yaml`` configuration file. .. code-block:: none $ sudo cumin 'A:cp-esams' 'systemctl is-active nginx' 23 hosts will be targeted: cp[3007-3008,3010,3030-3049].esams.wmnet Confirm to continue [y/n]? y ===== NODE GROUP ===== (23) cp[3007-3008,3010,3030-3049].esams.wmnet ----- OUTPUT of 'systemctl is-active nginx' ----- active ================ PASS: |████████████████████████████████████████████████| 100% (23/23) [00:01<00:00, 12.61hosts/s] FAIL: | | 0% (0/23) [00:01= 100.0% threshold) for command: 'systemctl is-active nginx'. 100.0% (23/23) success ratio (>= 100.0% threshold) of nodes successfully executed all commands. More complex example fine-tuning many of the parameters using the long form of the options for clarity: * Execute two commands in each host in sequence in a moving window of 2 hosts at a time, moving to the next host 5 seconds after the previous one has finished. * Each command will be considered timed out if it takes more than 30 seconds to complete. * If the percentage of successful hosts goes below 95% at any point it will not schedule any more hosts for execution. .. code-block:: none $ sudo cumin --batch-size 2 --batch-sleep 5 --success-percentage 95 --timeout 30 --mode async \ '(P{R:class = role::puppetmaster::backend} or P{R:class = role::puppetmaster::frontend}) and not D{rhodium.eqiad.wmnet}' \ 'date' 'ls -la /tmp/foo' 4 hosts will be targeted: puppetmaster[2001-2002].codfw.wmnet,puppetmaster[1001-1002].eqiad.wmnet Confirm to continue [y/n]? y ===== NODE GROUP ===== (2) puppetmaster[2001-2002].codfw.wmnet ----- OUTPUT ----- Thu Nov 2 18:45:18 UTC 2017 ===== NODE GROUP ===== (1) puppetmaster2002.codfw.wmnet ----- OUTPUT ----- ls: cannot access /tmp/foo: No such file or directory ===== NODE GROUP ===== (1) puppetmaster2001.codfw.wmnet ----- OUTPUT ----- -rw-r--r-- 1 root root 0 Nov 2 18:44 /tmp/foo ================ PASS: |████████████▌ | 25% (1/4) [00:05<00:01, 2.10hosts/s] FAIL: |████████████▌ | 25% (1/4) [00:05<00:01, 2.45hosts/s] 25.0% (1/4) of nodes failed to execute command 'ls -la /tmp/foo': puppetmaster2002.codfw.wmnet 25.0% (1/4) success ratio (< 95.0% threshold) of nodes successfully executed all commands. Aborting.: puppetmaster2001.codfw.wmnet Library ^^^^^^^ Simple example without fine-tuning of optional parameters:: import cumin from cumin import query, transport, transports # Load configuration files /etc/cumin/config.yaml and /etc/cumin/aliases.yaml (if present). config = cumin.Config() # Assuming default_backend: direct is set in config.yaml, select with the direct backend 5 hosts. hosts = query.Query(config).execute('host[1-5]') target = transports.Target(hosts) worker = transport.Transport.new(config, target) worker.commands = ['systemctl is-active nginx'] worker.handler = 'sync' exit_code = worker.execute() # Execute the command on all hosts in parallel for nodes, output in worker.get_results(): # Cycle over the results print(nodes) print(output.message().decode()) print('-----') More complex example fine-tuning many of the parameters:: import cumin from cumin import query, transport, transports from cumin.transports.clustershell import NullReporter config = cumin.Config(config='/path/to/custom/cumin/config.yaml') hosts = query.Query(config).execute('A:nginx') # Match hosts defined by the query alias named 'nginx'. # Needed only if SSH is authenticated via Kerberos and the related configuration flags are set # (see also the example configuration). cumin.ensure_kerberos_ticket(config) # Moving window of 5 hosts a time with 30s sleep before adding a new host once the previous one has finished. target = transports.Target(hosts, batch_size=5, batch_sleep=30.0) worker = transport.Transport.new(config, target) worker.commands = [ transports.Command('systemctl is-active nginx'), # In each host, for this command apply a timeout of 30 seconds and consider successful an exit code of 0 or 42. transports.Command('depool_command', timeout=30, ok_codes=[0, 42]), transports.Command('systemctl restart nginx'), transports.Command('systemctl is-active nginx'), transports.Command('repool_command', ok_codes=[0, 42]), ] # On each host perform the above commands in a sequence, only if the previous command was successful. worker.handler = 'async' # Change the worker's default reporter from the current default that outputs to stdout all commands stdout/err # outputs to the empty reporter that does nothing. worker.reporter = NullReporter # Suppress the progress bars during execution worker.progress_bars = False exit_code = worker.execute() for nodes, output in worker.get_results(): print(nodes) print(output.message().decode()) print('-----') wikimedia-cumin-36f957f/doc/source/pyparsing.inv000066400000000000000000000003021476500461000217360ustar00rootroot00000000000000# Sphinx inventory version 2 # Project: pyparsing # Version: 2.1.10 # The remainder of this file is compressed using zlib. x+,H,*K ҩE9y% V9 @&V%`y].4EAť9%x@55wikimedia-cumin-36f957f/doc/source/release.rst000066400000000000000000000000761476500461000213660ustar00rootroot00000000000000Release Notes ============= .. include:: ../../CHANGELOG.rst wikimedia-cumin-36f957f/doc/source/transports_state_transitions.dot000066400000000000000000000003121476500461000257710ustar00rootroot00000000000000digraph G { rankdir=TB; pending -> scheduled; scheduled -> running; running -> running; running -> success; running -> failed; running -> timeout; success -> pending; } wikimedia-cumin-36f957f/prospector.yaml000066400000000000000000000024261476500461000202340ustar00rootroot00000000000000strictness: high inherits: - strictness_high doc-warnings: true member-warnings: true test-warnings: true autodetect: false output-format: grouped pep8: full: true options: max-line-length: 120 pep257: explain: true source: true disable: - D203 # 1 blank line required before class docstring, D211 (after) is enforce instead - D213 # Multi-line docstring summary should start at the second line, D212 (first line) is enforced instead - D406 # Section name should end with a newline, incompatible with Google Style Python Docstrings - D407 # Missing dashed underline after section, incompatible with Google Style Python Docstrings pylint: disable: - pointless-string-statement # used as documentation for class attributes - unsubscriptable-object # Breaks for latest pylint https://github.com/PyCQA/pylint/issues/3882 - consider-using-f-string # The code still use format(), disable for now options: ignore: vulture_whitelist.py max-line-length: 120 max-args: 6 max-positional-arguments: 7 max-attributes: 14 max-locals: 16 include-naming-hint: true variable-rgx: (([a-z][a-z0-9_]{0,30})|(_[a-z0-9_]*))$ variable-name-hint: (([a-z][a-z0-9_]{0,30})|(_[a-z0-9_]*))$ pyroma: run: true vulture: run: true wikimedia-cumin-36f957f/pytest.ini000066400000000000000000000001071476500461000171730ustar00rootroot00000000000000[pytest] markers = variant_params: test_cli.py variant parameters. wikimedia-cumin-36f957f/setup.cfg000066400000000000000000000011131476500461000167610ustar00rootroot00000000000000[aliases] test = pytest [build_sphinx] project = Cumin source-dir = doc/source build-dir = doc/build [mypy] disallow_incomplete_defs = True ignore_missing_imports = True no_implicit_optional = True warn_unused_ignores = True show_error_context = True warn_unused_configs = True # TODO: convert to True once the whole project has type hints disallow_untyped_defs = False check_untyped_defs = False disallow_untyped_decorators = False # TODO: remove to return to their default once the whole project has type hints allow_untyped_globals = True [mypy-cumin.tests.*] ignore_errors = True wikimedia-cumin-36f957f/setup.py000066400000000000000000000113241476500461000166570ustar00rootroot00000000000000#!/usr/bin/env python """Package configuration.""" import os from setuptools import find_packages, setup with open('README.rst', 'r') as readme: long_description = readme.read() # Required dependencies install_requires = [ 'clustershell>=1.8.3,<=1.9.99', 'pyparsing>=2.4.7,<=3.99.0', 'pyyaml>=5.3.1', 'requests>=2.25.1', 'tqdm>=4.57.0', ] # Extra dependencies extras_require = { # Optional dependencies for additional features 'with-openstack': [ 'keystoneauth1>=4.2.1', 'python-keystoneclient>=4.1.1', 'python-novaclient>=17.2.1', ], # Test dependencies 'tests': [ 'bandit>=1.6.1', 'flake8>=3.8.4', 'flake8-import-order>=0.18.2', 'mypy', 'pytest-cov>=2.10.1', 'pytest-xdist>=2.2.0', 'pytest>=6.0.2', 'requests-mock>=1.7.0', 'sphinx_rtd_theme>=1.0', 'sphinx-argparse>=0.2.5', # Temporary pinning due to https://github.com/sphinx-doc/sphinx/issues/11890 'sphinxcontrib-applehelp<=1.0.4', 'sphinxcontrib-devhelp<=1.0.2', 'sphinxcontrib-htmlhelp<=2.0.1', 'sphinxcontrib-serializinghtml<=1.1.6', 'sphinxcontrib-qthelp<=1.0.3', # End of temporary pinning 'Sphinx>=3.4.3', 'types-PyYAML', 'types-requests', ], 'prospector': [ 'prospector[with_everything]>=1.3.1', 'pytest>=6.0.2', 'requests-mock>=1.7.0', ], } # Copy tests requirements to test only base dependencies extras_require['tests-base'] = extras_require['tests'][:] # Copy tests requirements to test with the minimum version of the install_requires and Sphinx that is used to # generate the manpage during the Debian build process. extras_require['tests-min'] = [dep.split(',')[0].replace('>=', '==') if dep.lower().startswith('sphinx') else dep for dep in extras_require['tests']] # Add Jinja2 upper limit for min-tests, it breaks with more recent versions extras_require['tests-min'].append('jinja2<3.1.0') # Add Sphinx-related packates limit for min-tests, they are not pinned in Sphinx and break if too recent extras_require['tests-min'].append('sphinxcontrib-applehelp<1.0.6') extras_require['tests-min'].append('sphinxcontrib-devhelp<1.0.4') extras_require['tests-min'].append('sphinxcontrib-htmlhelp<2.0.3') extras_require['tests-min'].append('sphinxcontrib-serializinghtml<1.1.7') extras_require['tests-min'].append('sphinxcontrib-qthelp<1.0.4') # Add optional dependencies to the tests ones extras_require['tests'].extend(extras_require['with-openstack']) extras_require['tests-min'].extend(dep.replace('>=', '==') for dep in extras_require['with-openstack']) extras_require['prospector'].extend(extras_require['with-openstack']) if os.getenv('CUMIN_MIN_DEPS', False): install_requires = [dep.split(',')[0].replace('>=', '==') for dep in install_requires] setup_requires = [ 'pytest-runner>=2.11.1', 'setuptools_scm>=5.0.1', ] setup( author='Riccardo Coccioli', author_email='rcoccioli@wikimedia.org', classifiers=[ 'Development Status :: 5 - Production/Stable', 'Environment :: Console', 'Intended Audience :: System Administrators', 'License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)', 'Operating System :: MacOS :: MacOS X', 'Operating System :: POSIX :: BSD', 'Operating System :: POSIX :: Linux', 'Programming Language :: Python :: 3.9', 'Programming Language :: Python :: 3.10', 'Programming Language :: Python :: 3.11', 'Programming Language :: Python :: 3.12', 'Programming Language :: Python :: 3.13', 'Programming Language :: Python :: 3 :: Only', 'Topic :: Software Development :: Libraries :: Python Modules', 'Topic :: System :: Clustering', 'Topic :: System :: Distributed Computing', 'Topic :: System :: Systems Administration', ], data_files=[('share/doc/cumin/examples/', ['doc/examples/config.yaml', 'doc/examples/aliases.yaml'])], description='Automation and orchestration framework and CLI written in Python', entry_points={ 'console_scripts': [ 'cumin = cumin.cli:main', ], }, extras_require=extras_require, install_requires=install_requires, keywords=['cumin', 'automation', 'orchestration'], license='GPLv3+', long_description=long_description, long_description_content_type='text/x-rst', name='cumin', packages=find_packages(exclude=['*.tests', '*.tests.*']), platforms=['GNU/Linux', 'BSD', 'MacOSX'], python_requires='>=3.9', setup_requires=setup_requires, url='https://github.com/wikimedia/cumin', use_scm_version=True, zip_safe=False, ) wikimedia-cumin-36f957f/tox.ini000066400000000000000000000052651476500461000164670ustar00rootroot00000000000000[tox] minversion = 3.0.0 envlist = py{39,311,312,313}-{flake8,mypy,unit,unitbase,bandit,prospector,sphinx,man},py39-{unit-min,man-min} skip_missing_interpreters = True [testenv] usedevelop = True allowlist_externals = rm sed {toxinidir}/cumin/tests/integration/docker.sh description = flake8: Style consistency checker mypy: Static analyzer for type annotations unit: Run unit tests unitbase: Run unit tests with base dependencies only bandit: Security-oriented static analyzer prospector: Static analysis multi-tool sphinx: Build html documentation integration: Run integration tests man: Build the man page min: [minimum supported version of dependencies] py39: (Python 3.9) py311: (Python 3.11) py312: (Python 3.12) py313: (Python 3.13) commands = flake8: flake8 setup.py cumin doc mypy: mypy cumin/ unit: py.test -p no:logging --strict-markers --cov-report=term-missing --cov=cumin cumin/tests/unit {posargs} unitbase: py.test -p no:logging --strict-markers --cov-report=term-missing --cov=cumin --ignore=cumin/tests/unit/backends/test_openstack.py cumin/tests/unit {posargs} # Avoid bandit import_subprocess (B404) overall, the import itself it not unsafe bandit: bandit -l -i -r --skip B404 --exclude './cumin/tests' ./cumin/ # Avoid bandit assert_used (B101) in tests bandit: bandit -l -i -r --skip B101,B404 cumin/tests prospector: prospector --profile "{toxinidir}/prospector.yaml" cumin/ sphinx: sphinx-build -b html doc/source/ doc/build/html man: sphinx-build -b man doc/source/ doc/build/man # Fix missing space after bold blocks in man page: https://github.com/ribozz/sphinx-argparse/issues/80 # Use a syntax that works both on BSD/MacOS and Linux man: sed -i.orig -e 's/^\.B/.B /' '{toxinidir}/doc/build/man/cumin.1' man: rm -fv '{toxinidir}/doc/build/man/cumin.1.orig' integration: "{toxinidir}/cumin/tests/integration/docker.sh" "transports/clustershell" {posargs} deps = # Use install_requires and the additional extras_require[NAME] from setup.py unitbase: .[tests-base] min: .[tests-min] prospector: .[prospector] !min-!unitbase-!prospector: .[tests] setenv = min: CUMIN_MIN_DEPS=1 # Needed as long as tox 3 is supported [testenv:py39-integration] [testenv:py39-integration-min] [testenv:py311-integration] [testenv:py312-integration] [testenv:py313-integration] [flake8] max-line-length = 120 statistics = True ignore = W503 import-order-style = edited # Do not specify application-package-names to avoid to manually keep the list of Org-wide packages # application-package-names = # Mark cumin as local to separate its imports application-import-names = cumin