pax_global_header00006660000000000000000000000064147133231430014513gustar00rootroot0000000000000052 comment=f8e1e6d9b2fedd975ff1f217acab17cf0e0a56df optuna-4.1.0/000077500000000000000000000000001471332314300130235ustar00rootroot00000000000000optuna-4.1.0/.coveragerc000066400000000000000000000000741471332314300151450ustar00rootroot00000000000000[run] concurrency = multiprocessing,thread source = optuna/ optuna-4.1.0/.dockerignore000066400000000000000000000001131471332314300154720ustar00rootroot00000000000000# Ignore everything ** !pyproject.toml !README.md !optuna **/__pycache__ optuna-4.1.0/.github/000077500000000000000000000000001471332314300143635ustar00rootroot00000000000000optuna-4.1.0/.github/FUNDING.yml000066400000000000000000000000171471332314300161760ustar00rootroot00000000000000github: optuna optuna-4.1.0/.github/ISSUE_TEMPLATE/000077500000000000000000000000001471332314300165465ustar00rootroot00000000000000optuna-4.1.0/.github/ISSUE_TEMPLATE/bug-report.yml000066400000000000000000000041461471332314300213640ustar00rootroot00000000000000name: "\U0001F41BBug report" description: Create a report to help us improve Optuna. labels: ["bug"] body: - type: markdown attributes: value: | Thanks for taking the time to fill out this bug report! Please write a clear and concise description of what the bug is. - type: textarea id: expected-behavior attributes: label: Expected behavior description: Please write a clear and concise description of what you expected to happen. validations: required: true - type: textarea id: environment attributes: label: Environment description: | Please give us your environment information. You can get this information by typing the following. ``` python3 -c 'import optuna; print(f"- Optuna version:{optuna.__version__}")' python3 -c 'import platform; print(f"- Python version:{platform.python_version()}")' python3 -c 'import platform; print(f"- OS:{platform.platform()}")' ``` value: | - Optuna version: - Python version: - OS: - (Optional) Other libraries and their versions: validations: required: true - type: textarea id: logs attributes: label: Error messages, stack traces, or logs description: Please copy and paste any relevant error messages, stack traces, or log output. render: shell validations: required: true - type: textarea id: steps-to-reproduce attributes: label: Steps to reproduce description: Please provide how we reproduce your reported bugs. If possible, it is highly recommended to provide the reproducible example codes. Note that pickle files will not be accepted due to possible [security issues](https://docs.python.org/3/library/pickle.html). value: | 1. 2. 3. ```python # python code ``` validations: required: true - type: textarea id: additional-context attributes: label: Additional context (optional) description: Please provide additional contexts if you have. validations: required: false optuna-4.1.0/.github/ISSUE_TEMPLATE/code-fix.yml000066400000000000000000000020161471332314300207660ustar00rootroot00000000000000name: "\U0001F6A7Code fix" description: Suggest a code fix that does not change the behaviors of Optuna, such as code refactoring. labels: ["code-fix"] body: - type: markdown attributes: value: | Thanks for taking the time to raise a new issue! Please write a clear and concise description of the code fix. - type: textarea id: motivation attributes: label: Motivation description: | Please write the motivation for the proposal. If your code fix is related to a problem, please describe a clear and concise description of what the problem is. validations: required: true - type: textarea id: suggestion attributes: label: Suggestion description: Please explain your suggestion for the code change. validations: required: true - type: textarea id: additional-context attributes: label: Additional context (optional) description: Please provide additional contexts if you have. validations: required: false optuna-4.1.0/.github/ISSUE_TEMPLATE/config.yml000066400000000000000000000000341471332314300205330ustar00rootroot00000000000000blank_issues_enabled: false optuna-4.1.0/.github/ISSUE_TEMPLATE/documentation.yml000066400000000000000000000006511471332314300221440ustar00rootroot00000000000000name: "\U0001f4d6Documentation" description: Report an issue related to https://optuna.readthedocs.io/. labels: ["document"] body: - type: textarea id: explanation attributes: label: What is an issue? description: Thanks for taking time to raise an issue! Please write a clear and concise description of what content in https://optuna.readthedocs.io/ is an issue. validations: required: true optuna-4.1.0/.github/ISSUE_TEMPLATE/feature-request.yml000066400000000000000000000024141471332314300224130ustar00rootroot00000000000000name: "\U0001F4A1Feature request" description: Suggest an idea for Optuna. labels: ["feature"] body: - type: markdown attributes: value: | Thanks for taking the time for your feature requests! Please write a clear and concise description of the feature proposal. - type: textarea id: motivation attributes: label: Motivation description: | Please write the motivation for the proposal. If your feature request is related to a problem, please describe a clear and concise description of what the problem is. validations: required: true - type: textarea id: description attributes: label: Description description: Please write a detailed description of the new feature. validations: required: true - type: textarea id: alternatives attributes: label: Alternatives (optional) description: Please write a clear and concise description of any alternative solutions or features you've considered. validations: required: false - type: textarea id: additional-context attributes: label: Additional context (optional) description: Please add any other context or screenshots about the feature request here. validations: required: false optuna-4.1.0/.github/ISSUE_TEMPLATE/questions-help-support.yml000066400000000000000000000006751471332314300237730ustar00rootroot00000000000000name: "\U00002753Questions" description: Don't use GitHub Issues to ask support questions. body: - type: markdown attributes: value: | # PLEASE USE GITHUB DISCUSSIONS Don't use GitHub Issues to ask support questions. Use the [GitHub Discussions](https://github.com/optuna/optuna/discussions/categories/q-a) for that. - type: textarea attributes: label: "Don't use GitHub Issues to ask support questions." optuna-4.1.0/.github/PULL_REQUEST_TEMPLATE.md000066400000000000000000000012211471332314300201600ustar00rootroot00000000000000 ## Motivation ## Description of the changes optuna-4.1.0/.github/workflows/000077500000000000000000000000001471332314300164205ustar00rootroot00000000000000optuna-4.1.0/.github/workflows/checks.yml000066400000000000000000000021001471332314300203740ustar00rootroot00000000000000name: Checks on: push: branches: - master pull_request: {} schedule: - cron: '0 23 * * SUN-THU' concurrency: group: ${{ github.workflow }}-${{ github.ref == 'refs/heads/master' && github.run_number || github.ref }} cancel-in-progress: true jobs: checks: if: (github.event_name == 'schedule' && github.repository == 'optuna/optuna') || (github.event_name != 'schedule') runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-python@v5 with: python-version: 3.12 - name: Install run: | python -m pip install -U pip pip install --progress-bar off -U .[checking] - name: Output installed packages run: | pip freeze --all - name: Output dependency tree run: | pip install pipdeptree pipdeptree - name: black run: black . --check --diff - name: flake8 run: flake8 . - name: isort run: isort . --check --diff - name: mypy run: mypy . - name: blackdoc run: blackdoc . --check --diff optuna-4.1.0/.github/workflows/coverage.yml000066400000000000000000000044711471332314300207440ustar00rootroot00000000000000name: Coverage on: push: branches: - master pull_request: {} concurrency: group: ${{ github.workflow }}-${{ github.ref == 'refs/heads/master' && github.run_number || github.ref }} cancel-in-progress: true jobs: coverage: runs-on: ubuntu-latest # Not intended for forks. if: github.repository == 'optuna/optuna' steps: - name: Checkout uses: actions/checkout@v4 - name: Setup Python 3.12 uses: actions/setup-python@v5 with: python-version: 3.12 - name: Setup cache uses: actions/cache@v3 env: cache-name: coverage with: path: ~/.cache/pip key: ${{ runner.os }}-${{ env.cache-name }}-${{ hashFiles('**/pyproject.toml') }}-v1 restore-keys: | ${{ runner.os }}-${{ env.cache-name }}-${{ hashFiles('**/pyproject.toml') }} - name: Setup environment run: | sudo apt-get update sudo apt-get -y install openmpi-bin libopenmpi-dev libopenblas-dev - name: Install run: | python -m pip install --upgrade pip # Install minimal dependencies and confirm that `import optuna` is successful. pip install --progress-bar off . python -c 'import optuna' optuna --version pip install --progress-bar off .[test] --extra-index-url https://download.pytorch.org/whl/cpu pip install --progress-bar off .[optional] --extra-index-url https://download.pytorch.org/whl/cpu echo 'import coverage; coverage.process_startup()' > sitecustomize.py - name: Output installed packages run: | pip freeze --all - name: Output dependency tree run: | pip install pipdeptree pipdeptree - name: Tests env: OMP_NUM_THREADS: 1 PYTHONPATH: . # To invoke sitecutomize.py COVERAGE_PROCESS_START: .coveragerc # https://coverage.readthedocs.io/en/6.4.1/subprocess.html COVERAGE_COVERAGE: yes # https://github.com/nedbat/coveragepy/blob/65bf33fc03209ffb01bbbc0d900017614645ee7a/coverage/control.py#L255-L261 run: | coverage run --source=optuna -m pytest tests -m "not skip_coverage and not slow" coverage combine coverage xml - name: Upload coverage to Codecov uses: codecov/codecov-action@v3 with: file: ./coverage.xml optuna-4.1.0/.github/workflows/dockerimage.yml000066400000000000000000000045311471332314300214200ustar00rootroot00000000000000name: Build Docker Image on: push: branches: - master release: types: [published] pull_request: paths: - .github/workflows/dockerimage.yml - Dockerfile - .dockerignore - pyproject.toml - .github/workflows/dockerimage.yml concurrency: group: ${{ github.workflow }}-${{ github.ref == 'refs/heads/master' && github.run_number || github.ref }} cancel-in-progress: true env: DOCKER_HUB_BASE_NAME: optuna/optuna jobs: dockerimage: runs-on: ubuntu-latest strategy: matrix: python_version: ['3.8', '3.9', '3.10', '3.11'] build_type: ['', 'dev'] # "dev" installs all the dependencies including pytest. # This action cannot be executed in the forked repository. if: github.repository == 'optuna/optuna' steps: - uses: actions/checkout@v4 - name: Set ENV run: | export TAG_NAME="py${{ matrix.python_version }}" if [ "${{ github.event_name }}" = 'release' ]; then export TAG_NAME="${{ github.event.release.tag_name }}-${TAG_NAME}" fi if [ "${{matrix.build_type}}" = 'dev' ]; then export TAG_NAME="${TAG_NAME}-dev" fi echo "HUB_TAG=${DOCKER_HUB_BASE_NAME}:${TAG_NAME}" >> $GITHUB_ENV - name: Build the Docker image run: | if [ "${{ github.event_name }}" = 'release' ]; then # Cache is not available because the image tag includes the Optuna version. CACHE_FROM="" else CACHE_FROM="--cache-from=${HUB_TAG}" fi docker build ${CACHE_FROM} . --build-arg PYTHON_VERSION=${{ matrix.python_version }} --build-arg BUILD_TYPE=${{ matrix.build_type }} --file Dockerfile --tag "${HUB_TAG}" env: DOCKER_BUILDKIT: 1 - name: Output installed packages run: | docker run "${HUB_TAG}" sh -c "pip freeze --all" - name: Output dependency tree run: | docker run "${HUB_TAG}" sh -c "pip install pipdeptree && pipdeptree" - name: Verify the built image run: | docker run "${HUB_TAG}" optuna --version - name: Login & Push to Docker Hub if: ${{ github.event_name != 'pull_request' }} env: DOCKER_HUB_TOKEN: ${{ secrets.DOCKER_HUB_TOKEN }} run: | echo "${DOCKER_HUB_TOKEN}" | docker login -u optunabot --password-stdin docker push "${HUB_TAG}" optuna-4.1.0/.github/workflows/mac-tests.yml000066400000000000000000000042311471332314300210430ustar00rootroot00000000000000# Run tests on Mac, which are triggered by each master push. # Currently, Python3.12 is only used as an environment. # This is mainly for the sake of speed. name: Mac tests on: push: branches: - master pull_request: {} schedule: - cron: '0 23 * * SUN-THU' workflow_dispatch: concurrency: group: ${{ github.workflow }}-${{ github.ref == 'refs/heads/master' && github.run_number || github.ref }} cancel-in-progress: true jobs: tests-mac: runs-on: macos-latest # Scheduled Tests are disabled for forks. if: (github.event_name == 'schedule' && github.repository == 'optuna/optuna') || (github.event_name != 'schedule') steps: - name: Checkout uses: actions/checkout@v4 - name: Setup Python 3.12 uses: actions/setup-python@v5 with: python-version: 3.12 - name: Setup cache uses: actions/cache@v3 env: cache-name: test with: path: ~/Library/Caches/pip key: ${{ runner.os }}-3.12-${{ env.cache-name }}-${{ hashFiles('**/pyproject.toml') }}-v1 restore-keys: | ${{ runner.os }}-3.12-${{ env.cache-name }}-${{ hashFiles('**/pyproject.toml') }} - name: Install run: | python -m pip install --upgrade pip pip install --progress-bar off -U setuptools # Install minimal dependencies and confirm that `import optuna` is successful. pip install --progress-bar off . python -c 'import optuna' optuna --version pip install --progress-bar off .[test] --extra-index-url https://download.pytorch.org/whl/cpu pip install --progress-bar off .[optional] --extra-index-url https://download.pytorch.org/whl/cpu - name: Output installed packages run: | pip freeze --all - name: Output dependency tree run: | pip install pipdeptree pipdeptree - name: Scheduled tests if: ${{ github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' }} run: | pytest tests - name: Tests if: ${{ github.event_name != 'schedule' && github.event_name != 'workflow_dispatch' }} run: | pytest tests -m "not slow" optuna-4.1.0/.github/workflows/matplotlib-tests.yml000066400000000000000000000037221471332314300224560ustar00rootroot00000000000000name: matplotlib-tests on: pull_request: paths: - .github/workflows/matplotlib-tests.yml - optuna/visualization/**.py - tests/visualization_tests/** concurrency: group: ${{ github.workflow }}-${{ github.ref == 'refs/heads/master' && github.run_number || github.ref }} cancel-in-progress: true jobs: matplotlib-tests: runs-on: ubuntu-latest # Scheduled Tests are disabled for forks. if: (github.event_name == 'schedule' && github.repository == 'optuna/optuna') || (github.event_name != 'schedule') steps: - name: Checkout uses: actions/checkout@v4 - name: Setup Python 3.12 uses: actions/setup-python@v5 with: python-version: 3.12 - name: Setup cache uses: actions/cache@v3 env: cache-name: test-matplotlib with: path: ~/.cache/pip key: ${{ runner.os }}-3.12-${{ env.cache-name }}-${{ hashFiles('**/pyproject.toml') }}-v1 restore-keys: | ${{ runner.os }}-3.12-${{ env.cache-name }}-${{ hashFiles('**/pyproject.toml') }} - name: Install run: | python -m pip install --upgrade pip pip install --progress-bar off -U setuptools # Install minimal dependencies and confirm that `import optuna` is successful. pip install --progress-bar off . python -c 'import optuna' optuna --version pip install --progress-bar off .[test] --extra-index-url https://download.pytorch.org/whl/cpu pip install --progress-bar off .[optional] --extra-index-url https://download.pytorch.org/whl/cpu - name: Output installed packages run: | pip freeze --all - name: Output dependency tree run: | pip install pipdeptree pipdeptree - name: Tests without Plotly if: ${{ github.event_name != 'schedule' && github.event_name != 'workflow_dispatch' }} run: | pip uninstall -y plotly pytest tests/visualization_tests/matplotlib_tests optuna-4.1.0/.github/workflows/pypi-publish.yml000066400000000000000000000035671471332314300216030ustar00rootroot00000000000000name: Publish distributions to PyPI or TestPyPI # TestPyPI upload is scheduled in each weekday. # PyPI upload is only activated if the release is published. on: schedule: - cron: '0 15 * * *' release: types: - published jobs: build-n-publish: name: Build and publish Python distributions to PyPI or TestPyPI runs-on: ubuntu-latest permissions: id-token: write # Not intended for forks. if: github.repository == 'optuna/optuna' steps: - uses: actions/checkout@v4 - name: Set up Python 3.12 uses: actions/setup-python@v5 with: python-version: 3.12 - name: Install twine run: | python -m pip install -U pip python -m pip install -U twine wheel build - name: Output installed packages run: | pip freeze --all - name: Output dependency tree run: | pip install pipdeptree pipdeptree - name: Change the version file for scheduled TestPyPI upload if: github.event_name == 'schedule' run: | OPTUNA_VERSION=$(cut -d '"' -f 2 optuna/version.py) DATE=`date +"%Y%m%d"` echo "__version__ = \"${OPTUNA_VERSION}${DATE}\"" > optuna/version.py - name: Build a tar ball run: | python -m build --sdist --wheel - name: Verify the distributions run: twine check dist/* - name: Publish distribution to TestPyPI # The following upload action cannot be executed in the forked repository. if: (github.event_name == 'schedule') || (github.event_name == 'release') uses: pypa/gh-action-pypi-publish@release/v1 with: repository-url: https://test.pypi.org/legacy/ - name: Publish distribution to PyPI # The following upload action cannot be executed in the forked repository. if: github.event_name == 'release' uses: pypa/gh-action-pypi-publish@release/v1 optuna-4.1.0/.github/workflows/reviewdog.yml000066400000000000000000000015331471332314300211400ustar00rootroot00000000000000name: Reviewdog on: pull_request: types: [opened, review_requested] jobs: reviewdog: runs-on: ubuntu-latest if: github.event.action == 'opened' || github.event.requested_team.name == 'reviewdog' steps: - name: Checkout uses: actions/checkout@v4 - uses: actions/setup-python@v5 with: python-version: 3.12 - name: Install run: | python -m pip install -U pip pip install --progress-bar off -U .[checking] - name: Output installed packages run: | pip freeze --all - name: Output dependency tree run: | pip install pipdeptree pipdeptree - name: Apply formatters run: | black . blackdoc . isort . - name: Reviewdog uses: reviewdog/action-suggester@v1 with: tool_name: formatters optuna-4.1.0/.github/workflows/speed-benchmarks.yml000066400000000000000000000026101471332314300223550ustar00rootroot00000000000000name: Speed benchmarks on: schedule: - cron: '0 23 * * SUN-THU' jobs: speed-benchmarks: runs-on: ubuntu-latest # Not intended for forks. if: github.repository == 'optuna/optuna' steps: - name: Checkout uses: actions/checkout@v4 - name: Setup Python 3.12 uses: actions/setup-python@v5 with: python-version: 3.12 - name: Setup cache uses: actions/cache@v3 env: cache-name: speed-benchmarks with: path: ~/.cache/pip key: ${{ runner.os }}-3.12-${{ env.cache-name }}-${{ hashFiles('**/pyproject.toml') }}-v1 restore-keys: | ${{ runner.os }}-3.12-${{ env.cache-name }}-${{ hashFiles('**/pyproject.toml') }} - name: Install run: | python -m pip install --upgrade pip pip install --progress-bar off -U setuptools # Install minimal dependencies and confirm that `import optuna` is successful. pip install --progress-bar off . python -c 'import optuna' optuna --version pip install --progress-bar off .[benchmark] --extra-index-url https://download.pytorch.org/whl/cpu asv machine --yes - name: Output installed packages run: | pip freeze --all - name: Output dependency tree run: | pip install pipdeptree pipdeptree - name: Speed benchmarks run: | asv run optuna-4.1.0/.github/workflows/sphinx-build.yml000066400000000000000000000050561471332314300215570ustar00rootroot00000000000000name: Sphinx on: push: branches: - master pull_request: {} concurrency: group: ${{ github.workflow }}-${{ github.ref == 'refs/heads/master' && github.run_number || github.ref }} cancel-in-progress: true jobs: documentation: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 # TODO(c-bata): Use the newer Python version - uses: actions/setup-python@v5 with: python-version: 3.8 # note (crcrpar): We've not updated tutorial frequently enough so far thus # it'd be okay to discard cache by any small changes including typo fix under tutorial directory. - name: Sphinx Gallery Cache uses: actions/cache@v3 env: cache-name: sphx-glry-documentation with: path: | tutorial/MNIST docs/source/tutorial key: py3.8-${{ env.cache-name }}-${{ hashFiles('tutorial/**/*') }} - name: Install Dependencies run: | sudo apt-get install optipng python -m pip install -U pip pip install --progress-bar off -U .[document] --extra-index-url https://download.pytorch.org/whl/cpu - name: Output installed packages run: | pip freeze --all - name: Output dependency tree run: | pip install pipdeptree pipdeptree - name: Build Document run: | cd docs make html cd ../ - uses: actions/upload-artifact@v4 with: name: built-html path: | docs/build/html - uses: actions/upload-artifact@v4 with: name: tutorial path: | docs/source/tutorial doctest: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 # TODO(c-bata): Use the newer Python version - uses: actions/setup-python@v5 with: python-version: 3.8 - name: Sphinx Gallery Cache uses: actions/cache@v3 env: cache-name: sphx-glry-doctest with: path: | tutorial/MNIST docs/source/tutorial key: py3.8-${{ env.cache-name }}-${{ hashFiles('tutorial/**/*') }} - name: Install Dependencies run: | sudo apt-get install optipng python -m pip install -U pip pip install --progress-bar off -U .[document] --extra-index-url https://download.pytorch.org/whl/cpu - name: Output installed packages run: | pip freeze --all - name: Output dependency tree run: | pip install pipdeptree pipdeptree - name: Run Doctest run: | cd docs make doctest optuna-4.1.0/.github/workflows/stale.yml000066400000000000000000000020361471332314300202540ustar00rootroot00000000000000name: Stale on: schedule: - cron: '0 23 * * SUN-THU' jobs: stale: runs-on: ubuntu-latest if: github.repository == 'optuna/optuna' steps: - uses: actions/stale@v6 with: repo-token: ${{ secrets.GITHUB_TOKEN }} stale-issue-message: 'This issue has not seen any recent activity.' stale-pr-message: 'This pull request has not seen any recent activity.' close-issue-message: 'This issue was closed automatically because it had not seen any recent activity. If you want to discuss it, you can reopen it freely.' close-pr-message: 'This pull request was closed automatically because it had not seen any recent activity. If you want to discuss it, you can reopen it freely.' days-before-issue-stale: 300 days-before-issue-close: 0 days-before-pr-stale: 7 days-before-pr-close: 14 stale-issue-label: 'stale' stale-pr-label: 'stale' exempt-issue-labels: 'no-stale' exempt-pr-labels: 'no-stale' operations-per-run: 1000 optuna-4.1.0/.github/workflows/tests-storage.yml000066400000000000000000000103701471332314300217500ustar00rootroot00000000000000name: Tests (Storage with server) on: workflow_dispatch: push: branches: - master pull_request: {} schedule: - cron: '0 23 * * SUN-THU' concurrency: group: ${{ github.workflow }}-${{ github.ref == 'refs/heads/master' && github.run_number || github.ref }} cancel-in-progress: true jobs: # TODO(masap): Modify job name to "tests-storage-with-server" because this test is not only for # RDB. Since current name "tests-rdbstorage" is required in the Branch protection rules, you # need to modify the Branch protection rules as well. tests-rdbstorage: if: (github.event_name == 'schedule' && github.repository == 'optuna/optuna') || (github.event_name != 'schedule') runs-on: ubuntu-latest strategy: matrix: python-version: ['3.8', '3.9', '3.10', '3.11', '3.12', '3.13'] test-trigger-type: - ${{ (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') && 'Scheduled' || '' }} exclude: - test-trigger-type: "" python-version: "3.9" - test-trigger-type: "" python-version: "3.10" - test-trigger-type: "" python-version: "3.11" - test-trigger-type: "" python-version: "3.12" services: mysql: image: mysql:8 ports: - 3306:3306 env: MYSQL_ROOT_PASSWORD: mandatory_arguments MYSQL_DATABASE: optunatest MYSQL_USER: user MYSQL_PASSWORD: test options: >- --health-cmd "mysqladmin ping -h localhost" --health-interval 10s --health-timeout 5s --health-retries 5 postgres: image: postgres:latest ports: - 5432:5432 env: POSTGRES_DB: optunatest POSTGRES_USER: user POSTGRES_PASSWORD: test options: >- --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5 redis: image: redis ports: - 6379:6379 steps: - name: Checkout uses: actions/checkout@v4 - name: Setup Python${{ matrix.python-version }} uses: actions/setup-python@v5 with: python-version: ${{ matrix.python-version }} - name: Setup cache uses: actions/cache@v3 env: cache-name: test-storage-with-server with: path: ~/.cache/pip key: ${{ runner.os }}-${{ matrix.python-version }}-${{ env.cache-name }}-${{ hashFiles('**/pyproject.toml') }}-v1 restore-keys: | ${{ runner.os }}-${{ matrix.python-version }}-${{ env.cache-name }}-${{ hashFiles('**/pyproject.toml') }} - name: Setup environment run: | sudo apt-get update sudo apt-get -y install openmpi-bin libopenmpi-dev - name: Install run: | python -m pip install --upgrade pip pip install --progress-bar off -U setuptools # Install minimal dependencies and confirm that `import optuna` is successful. pip install --progress-bar off . python -c 'import optuna' optuna --version pip install --progress-bar off .[test] --extra-index-url https://download.pytorch.org/whl/cpu pip install --progress-bar off .[optional] --extra-index-url https://download.pytorch.org/whl/cpu - name: Install DB bindings run: | pip install --progress-bar off PyMySQL cryptography psycopg2-binary redis - name: Output installed packages run: | pip freeze --all - name: Output dependency tree run: | pip install pipdeptree pipdeptree - name: Tests MySQL run: | pytest tests/storages_tests/test_with_server.py env: SQLALCHEMY_WARN_20: 1 OMP_NUM_THREADS: 1 TEST_DB_URL: mysql+pymysql://user:test@127.0.0.1/optunatest - name: Tests PostgreSQL run: | pytest tests/storages_tests/test_with_server.py env: OMP_NUM_THREADS: 1 TEST_DB_URL: postgresql+psycopg2://user:test@127.0.0.1/optunatest - name: Tests Journal Redis run: | pytest tests/storages_tests/test_with_server.py env: OMP_NUM_THREADS: 1 TEST_DB_URL: redis://localhost:6379 TEST_DB_MODE: journal-redis optuna-4.1.0/.github/workflows/tests-with-minimum-versions.yml000066400000000000000000000062061471332314300246010ustar00rootroot00000000000000name: Tests with minimum versions on: push: branches: - master pull_request: {} schedule: - cron: '0 23 * * SUN-THU' workflow_dispatch: concurrency: group: ${{ github.workflow }}-${{ github.ref == 'refs/heads/master' && github.run_number || github.ref }} cancel-in-progress: true jobs: tests-with-minimum-versions: if: (github.event_name == 'schedule' && github.repository == 'optuna/optuna') || (github.event_name != 'schedule') runs-on: ubuntu-latest strategy: matrix: python-version: ['3.8', '3.9'] services: redis: image: redis options: >- --health-cmd "redis-cli ping" --health-interval 10s --health-timeout 5s --health-retries 5 steps: - name: Checkout uses: actions/checkout@v4 - name: Setup Python${{ matrix.python-version }} uses: actions/setup-python@v5 with: python-version: ${{ matrix.python-version }} - name: Setup cache uses: actions/cache@v3 env: cache-name: test-with-minimum-versions with: path: ~/.cache/pip key: ${{ runner.os }}-${{ matrix.python-version }}-${{ env.cache-name }}-${{ hashFiles('**/setup.py') }}-v1 restore-keys: | ${{ runner.os }}-${{ matrix.python-version }}-${{ env.cache-name }}-${{ hashFiles('**/setup.py') }} - name: Setup pip run: | python -m pip install --upgrade pip pip install --progress-bar off -U setuptools - name: Install run: | # Install minimal dependencies and confirm that `import optuna` is successful. pip install --progress-bar off . python -c 'import optuna' optuna --version pip install --progress-bar off .[test] --extra-index-url https://download.pytorch.org/whl/cpu pip install --progress-bar off .[optional] --extra-index-url https://download.pytorch.org/whl/cpu - name: Install dependencies with minimum versions run: | # Install dependencies with minimum versions. pip uninstall -y alembic cmaes packaging sqlalchemy plotly scikit-learn pillow pip install alembic==1.5.0 cmaes==0.10.0 packaging==20.0 sqlalchemy==1.4.2 tqdm==4.27.0 colorlog==0.3 PyYAML==5.1 'pillow<10.4.0' pip uninstall -y matplotlib pandas scipy if [ "${{ matrix.python-version }}" = "3.8" ]; then pip install matplotlib==3.7.5 pandas==2.0.3 scipy==1.10.1 numpy==1.20.3 elif [ "${{ matrix.python-version }}" = "3.9" ]; then pip install matplotlib==3.8.4 pandas==2.2.2 scipy==1.13.0 numpy==1.26.4 fi pip install plotly==5.0.0 scikit-learn==0.24.2 # optional extras - name: Output installed packages run: | pip freeze --all - name: Output dependency tree run: | pip install pipdeptree pipdeptree - name: Scheduled tests if: ${{ github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' }} run: | pytest tests - name: Tests if: ${{ github.event_name != 'schedule' && github.event_name != 'workflow_dispatch' }} run: | pytest tests -m "not slow" optuna-4.1.0/.github/workflows/tests.yml000066400000000000000000000057051471332314300203140ustar00rootroot00000000000000name: Tests on: push: branches: - master pull_request: {} schedule: - cron: '0 23 * * SUN-THU' workflow_dispatch: concurrency: group: ${{ github.workflow }}-${{ github.ref == 'refs/heads/master' && github.run_number || github.ref }} cancel-in-progress: true jobs: tests: if: (github.event_name == 'schedule' && github.repository == 'optuna/optuna') || (github.event_name != 'schedule') runs-on: ubuntu-latest strategy: matrix: python-version: ['3.8', '3.9', '3.10', '3.11', '3.12', '3.13'] test-trigger-type: - ${{ (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') && 'Scheduled' || '' }} exclude: - test-trigger-type: "" python-version: "3.9" - test-trigger-type: "" python-version: "3.10" - test-trigger-type: "" python-version: "3.11" services: redis: image: redis options: >- --health-cmd "redis-cli ping" --health-interval 10s --health-timeout 5s --health-retries 5 steps: - name: Checkout uses: actions/checkout@v4 - name: Setup Python${{ matrix.python-version }} uses: actions/setup-python@v5 with: python-version: ${{ matrix.python-version }} - name: Setup cache uses: actions/cache@v3 env: cache-name: test with: path: ~/.cache/pip key: ${{ runner.os }}-${{ matrix.python-version }}-${{ env.cache-name }}-${{ hashFiles('**/pyproject.toml') }}-v1 restore-keys: | ${{ runner.os }}-${{ matrix.python-version }}-${{ env.cache-name }}-${{ hashFiles('**/pyproject.toml') }} - name: Setup pip run: | python -m pip install --upgrade pip pip install --progress-bar off -U setuptools - name: Install run: | # Install minimal dependencies and confirm that `import optuna` is successful. pip install --progress-bar off . python -c 'import optuna' optuna --version pip install --progress-bar off .[test] --extra-index-url https://download.pytorch.org/whl/cpu pip install --progress-bar off .[optional] --extra-index-url https://download.pytorch.org/whl/cpu # TODO(gen740): Remove this after pytorch supports Python 3.13 if [ "${{ matrix.python-version }}" = "3.13" ] ; then pip install --pre torch --index-url https://download.pytorch.org/whl/nightly/cpu fi - name: Output installed packages run: | pip freeze --all - name: Output dependency tree run: | pip install pipdeptree pipdeptree - name: Tests run: | if [ "${{ github.event_name }}" = "schedule" ] || \ [ "${{ github.event_name }}" = "workflow_dispatch" ] ; then target="" else target="not slow" fi pytest tests -m "$target" env: SQLALCHEMY_WARN_20: 1 optuna-4.1.0/.github/workflows/windows-tests.yml000066400000000000000000000053431471332314300220020ustar00rootroot00000000000000# Run tests on Windows, which are triggered by each master push. # Currently, Python3.12 is only used as an environment. # This is mainly for the sake of speed. name: Windows tests on: push: branches: - master pull_request: {} schedule: - cron: '0 23 * * SUN-THU' workflow_dispatch: concurrency: group: ${{ github.workflow }}-${{ github.ref == 'refs/heads/master' && github.run_number || github.ref }} cancel-in-progress: true jobs: tests-windows: runs-on: windows-latest # Not intended for forks. if: (github.event_name == 'schedule' && github.repository == 'optuna/optuna') || (github.event_name != 'schedule') steps: - name: Checkout uses: actions/checkout@v4 - name: Setup Python 3.12 uses: actions/setup-python@v5 with: python-version: "3.12" - name: Setup cache uses: actions/cache@v3 env: cache-name: windows-test with: path: ~\\AppData\\Local\\pip\\Cache key: ${{ runner.os }}-3.12-${{ env.cache-name }}-${{ hashFiles('**/pyproject.toml') }}-v1 restore-keys: | ${{ runner.os }}-3.12-${{ env.cache-name }}-${{ hashFiles('**/pyproject.toml') }} - name: Install run: | python -m pip install --upgrade pip pip install --progress-bar off -U setuptools # Install minimal dependencies and confirm that `import optuna` is successful. pip install --progress-bar off . python -c 'import optuna' optuna --version pip install --progress-bar off .[test] --extra-index-url https://download.pytorch.org/whl/cpu pip install --progress-bar off .[optional] --extra-index-url https://download.pytorch.org/whl/cpu pip install PyQt6 # Install PyQT for using QtAgg as matplotlib backend. # TODO(HideakiImamura): Remove this after fixing https://github.com/plotly/Kaleido/issues/110 pip install "kaleido<=0.1.0post1" # TODO(nabe): Remove the version constraint once Torch supports NumPy v2.0.0 for Windows. pip uninstall numpy pip install --progress-bar off 'numpy<2.0.0' - name: Output installed packages run: | pip freeze --all - name: Output dependency tree run: | pip install pipdeptree pipdeptree - name: Scheduled tests if: ${{ github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' }} run: | pytest env: SQLALCHEMY_WARN_20: 1 MPLBACKEND: "QtAgg" # Use QtAgg as matplotlib backend. - name: Tests if: ${{ github.event_name != 'schedule' && github.event_name != 'workflow_dispatch' }} run: | pytest -m "not slow" env: MPLBACKEND: "QtAgg" # Use QtAgg as matplotlib backend. optuna-4.1.0/.gitignore000066400000000000000000000035631471332314300150220ustar00rootroot00000000000000# macOS metadata .DS_Store # Ignore files that examples create t10k-images-idx3-ubyte* t10k-labels-idx1-ubyte* train-images-idx3-ubyte* train-labels-idx1-ubyte* training.pt test.pt catboost_info/ # Byte-compiled / optimized / DLL files __pycache__/ *.py[cod] *$py.class # C extensions *.so # Distribution / packaging .Python env/ build/ develop-eggs/ dist/ downloads/ eggs/ .eggs/ lib/ lib64/ parts/ sdist/ var/ *.egg-info/ .installed.cfg *.egg # PyInstaller # Usually these files are written by a python script from a template # before PyInstaller builds the exe, so as to inject date/other infos into it. *.manifest *.spec # Installer logs pip-log.txt pip-delete-this-directory.txt # Unit test / coverage reports htmlcov/ .tox/ .coverage .coverage.* .cache nosetests.xml coverage.xml *,cover .hypothesis/ .pytest_cache/ # Translations *.mo *.pot # Django stuff: *.log !tests/storages_tests/journal_tests/assets/*.log local_settings.py # Flask stuff: instance/ .webassets-cache # Scrapy stuff: .scrapy # PyBuilder target/ # IPython Notebook .ipynb_checkpoints # pyenv .python-version # celery beat schedule file celerybeat-schedule # dotenv .env # virtualenv .venv/ venv/ ENV/ # Spyder project settings .spyderproject # Rope project settings .ropeproject # PyCharm .idea # VSCode .vscode .devcontainer # MyPy .mypy_cache # Sphinx tutorial/**/example.db tutorial/**/example-study.db tutorial/20_recipes/artifacts/ tutorial/20_recipes/best_atoms.png tutorial/20_recipes/tmp/ docs/_build/ docs/source/reference/generated/ docs/source/reference/multi_objective/generated/ docs/source/reference/visualization/generated/ docs/source/reference/visualization/matplotlib/generated/ docs/source/reference/samplers/generated docs/source/sg_execution_times.rst docs/source/tutorial/** !docs/source/tutorial/index.rst # asv .asv # Dask dask-worker-space/ # PyTorch Lightning lightning_logs/ optuna-4.1.0/.pre-commit-config.yaml000066400000000000000000000030251471332314300173040ustar00rootroot00000000000000# pre-commit package installation is necessary to use pre-commit. # $ pip install pre-commit # $ pre-commit install default_language_version: python: python3 repos: # Args are based on setup.cfg. - repo: https://github.com/psf/black rev: 24.8.0 hooks: - id: black - repo: https://github.com/pycqa/flake8 rev: 7.1.1 hooks: - id: flake8 exclude: tutorial|docs/visualization_examples|docs/visualization_matplotlib_examples args: [ "--max-line-length=99", "--ignore=E203,E704,W503", "--statistics", ] - repo: https://github.com/pycqa/isort rev: 5.13.2 hooks: - id: isort - repo: https://github.com/pre-commit/mirrors-mypy rev: v1.11.2 hooks: - id: mypy additional_dependencies: [ "alembic>=1.5.0", "colorlog", "numpy", "packaging>=20.0", "sqlalchemy>=1.3.0", "tqdm", "PyYAML", "mypy_boto3_s3", "types-PyYAML", "types-redis", "types-setuptools", "types-tqdm", "typing_extensions>=3.10.0.0", ] exclude: docs|tutorial|optuna/storages/_rdb/alembic args: [ --warn-unused-configs, --disallow-untyped-calls, --disallow-untyped-defs, --disallow-incomplete-defs, --check-untyped-defs, --no-implicit-optional, --warn-redundant-casts, --strict-equality, --extra-checks, --no-implicit-reexport, --ignore-missing-imports, ] optuna-4.1.0/.readthedocs.yml000066400000000000000000000014131471332314300161100ustar00rootroot00000000000000# Read the Docs configuration file # See https://docs.readthedocs.io/en/stable/config-file/v2.html for details # Required version: 2 # Set the version of Python and other tools you might need build: os: ubuntu-22.04 tools: python: "3.11" # Build documentation in the docs/ directory with Sphinx sphinx: configuration: docs/source/conf.py # Optionally build your docs in additional formats such as PDF and ePub formats: all # Optionally set the version of Python and requirements required to build your docs python: # `sphinx` requires either Python >= 3.8 or `typed-ast` to reflect type comments # in the documentation. See: https://github.com/sphinx-doc/sphinx/pull/6984 install: - method: pip path: . extra_requirements: - document optuna-4.1.0/CODE_OF_CONDUCT.md000066400000000000000000000004651471332314300156270ustar00rootroot00000000000000# Optuna Code of Conduct Optuna follows the [NumFOCUS Code of Conduct][homepage] available at https://numfocus.org/code-of-conduct. Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project team at optuna@preferred.jp. [homepage]: https://numfocus.org/ optuna-4.1.0/CONTRIBUTING.md000066400000000000000000000267611471332314300152700ustar00rootroot00000000000000# Contribution Guidelines It’s an honor to have you on board! We are proud of this project and have been working to make it great since day one. We believe you will love it, and we know there’s room for improvement. We want to - implement features that make what you want to do possible and/or easy. - write more tutorials and [examples](https://github.com/optuna/optuna-examples) that help you get familiar with Optuna. - make issues and pull requests on GitHub fruitful. - have more conversations and discussions on [GitHub Discussions](https://github.com/optuna/optuna/discussions). We need your help and everything about Optuna you have in your mind pushes this project forward. Join Us! If you feel like giving a hand, here are some ways: - Implement a feature - If you have some cool idea, please open an issue first to discuss design to make your idea in a better shape. - We also welcome PRs for [optunahub-registry](https://github.com/optuna/optunahub-registry). [OptunaHub](https://hub.optuna.org/) is a feature-sharing platform for Optuna. - Send a patch - Dirty your hands by tackling [issues with `contribution-welcome` label](https://github.com/optuna/optuna/issues?q=is%3Aissue+is%3Aopen+label%3Acontribution-welcome) - Report a bug - If you find a bug, please report it! Your reports are important. - Fix/Improve documentation - Documentation gets outdated easily and can always be better, so feel free to fix and improve - Let us and the Optuna community know your ideas and thoughts. - __Contribution to Optuna includes not only sending pull requests, but also writing down your comments on issues and pull requests by others, and joining conversations/discussions on [GitHub Discussions](https://github.com/optuna/optuna/discussions).__ - Also, sharing how you enjoy Optuna is a huge contribution! If you write a blog, let us know about it! ## Pull Request Guidelines If you make a pull request, please follow the guidelines below: - [Setup Optuna](#setup-optuna) - [Checking the Format, Coding Style, and Type Hints](#checking-the-format-coding-style-and-type-hints) - [Documentation](#documentation) - [Unit Tests](#unit-tests) - [Continuous Integration and Local Verification](#continuous-integration-and-local-verification) - [Creating a Pull Request](#creating-a-pull-request) Detailed conventions and policies to write, test, and maintain Optuna code are described in the [Optuna Wiki](https://github.com/optuna/optuna/wiki). - [Coding Style Conventions](https://github.com/optuna/optuna/wiki/Coding-Style-Conventions) - [Deprecation Policy](https://github.com/optuna/optuna/wiki/Deprecation-policy) - [Test Policy](https://github.com/optuna/optuna/wiki/Test-Policy) ### Setup Optuna First of all, fork Optuna on GitHub. You can learn about fork in the official [documentation](https://docs.github.com/en/github/getting-started-with-github/fork-a-repo). After forking, download and install Optuna on your computer. ```bash git clone git@github.com:YOUR_NAME/optuna.git cd optuna pip install -e . ``` ### Checking the Format, Coding Style, and Type Hints Code is formatted with [black](https://github.com/psf/black), and docstrings are formatted with [blackdoc](https://github.com/keewis/blackdoc). Coding style is checked with [flake8](http://flake8.pycqa.org) and [isort](https://pycqa.github.io/isort/), and additional conventions are described in the [Wiki](https://github.com/optuna/optuna/wiki/Coding-Style-Conventions). Type hints, [PEP484](https://www.python.org/dev/peps/pep-0484/), are checked with [mypy](http://mypy-lang.org/). You can check the format, coding style, and type hints at the same time just by executing a script `formats.sh`. If your environment is missing some dependencies such as black, blackdoc, flake8, isort or mypy, you will be asked to install them. The following commands automatically fix format errors by auto-formatters. ```bash # Install auto-formatters. $ pip install ".[checking]" $ ./formats.sh ``` You can use `pre-commit` to automatically check the format, coding style, and type hints before committing. The following commands automatically fix format errors by auto-formatters. ```bash # Install `pre-commit`. $ pip install pre-commit $ pre-commit install $ pre-commit run --all-files ``` ### Documentation When adding a new feature to the framework, you also need to document it in the reference. The documentation source is stored under the [docs](./docs) directory and written in [reStructuredText format](http://www.sphinx-doc.org/en/master/usage/restructuredtext/index.html). To build the documentation, you need to run: ```bash pip install -e ".[document]" ``` Note that the above command might try to install PyTorch without CUDA to your environment even if your environment has CUDA version already. Then you can build the documentation in HTML format locally: ```bash cd docs make html ``` HTML files are generated under `build/html` directory. Open `index.html` with the browser and see if it is rendered as expected. Optuna's tutorial is built with [Sphinx-Gallery](https://sphinx-gallery.github.io/stable/index.html) and some other requirements like [LightGBM](https://github.com/microsoft/LightGBM) and [PyTorch](https://pytorch.org) meaning that all .py files in `tutorial` directory are run during the documentation build if there's no build cache. Whether you edit any tutorial or not doesn't matter. To avoid having to run the tutorials, you may download executed tutorial artifacts named "tutorial" from our CI (see the capture below) and put them in `docs/build` before extracting the files in the zip to `docs/source/tutorial` directory. Note that the CI runs with Python 3.8 and the generated artifacts contain pickle files. The pickle files are serialized with [the protocol version 5](https://docs.python.org/3/library/pickle.html#data-stream-format) so you will see the error with Python 3.7 or older. Please use Python 3.8 or later if you build the documentation with artifacts. ![image](https://user-images.githubusercontent.com/16191443/107472296-0b211400-6bb2-11eb-9203-e2c42ce499ad.png) **Writing a Tutorial** Tutorials are part of Optuna’s documentation. Optuna depends on Sphinx to build the documentation HTML files from the corresponding reStructuredText (`.rst`) files in the docs/source directory, but as you may notice, [Tutorial directory](https://github.com/optuna/optuna/tree/master/tutorial) does not have any `.rst` files. Instead, it has a bunch of Python (`.py`) files. We have [Sphinx Gallery](https://sphinx-gallery.github.io/stable/index.html) that executes those `.py` files and generates `.rst` files with standard outputs from them and corresponding Jupyter Notebook (`.ipynb`) files. These generated `.rst` and `.ipynb` files are written to the docs/source/tutorial directory. The output directory (docs/source/tutorial) and source (tutorial) directory are configured in [`sphinx_gallery_conf` of docs/source/conf.py](https://github.com/optuna/optuna/blob/2e14273cab87f13edeb9d804a43bd63c44703cb5/docs/source/conf.py#L189-L199). These generated `.rst` files are handled by Sphinx like the other `.rst` files. The generated `.ipynb` files are hosted on Optuna’s documentation page and downloadable (check [Optuna tutorial](https://optuna.readthedocs.io/en/stable/tutorial/index.html)). The order of contents on [tutorial top page](https://optuna.readthedocs.io/en/stable/tutorial/index.html) is determined by two keys: one is the subdirectory name of tutorial and the other is the filename (note that there are some alternatives as documented in [Sphinx Gallery - sorting](https://sphinx-gallery.github.io/stable/gen_modules/sphinx_gallery.sorting.html?highlight=filenamesortkey), but we chose this key in https://github.com/optuna/optuna/blob/2e14273cab87f13edeb9d804a43bd63c44703cb5/docs/source/conf.py#L196). Optuna’s tutorial directory has two directories: (1) [10_key_features](https://github.com/optuna/optuna/tree/master/tutorial/10_key_features), which is meant to be aligned with and explain the key features listed on [README.md](https://github.com/optuna/optuna#key-features) and (2) [20_recipes](https://github.com/optuna/optuna/tree/master/tutorial/20_recipes), whose contents showcase how to use Optuna features conveniently. When adding new content to the Optuna tutorials, place it in `20_recipes` and its file name should conform to the other names, for example, `777_cool_feature.py`. In general, please number the prefix for your file consecutively with the last number. However, this is not mandatory and if you think your content deserves the smaller number (the order of recipes does not have a specific meaning, but in general, order could convey the priority order to readers), feel free to propose the renumbering in your PR. You may want to refer to the Sphinx Gallery for the syntax of `.py` files processed by Sphinx Gallery. Two specific conventions and limitations for Optuna tutorials: 1. 99 #s for block separation as in https://github.com/optuna/optuna/blob/2e14273cab87f13edeb9d804a43bd63c44703cb5/tutorial/10_key_features/001_first.py#L19 2. Execution time of the new content needs to be less than three minutes. This limitation derives from Read The Docs. If your content runs some hyperparameter optimization, set the `timeout` to 180 or less. You can check this limitation on [Read the Docs - Build Process](https://docs.readthedocs.io/en/stable/builds.html). ### Unit Tests When adding a new feature or fixing a bug, you also need to write sufficient test code. We use [pytest](https://pytest.org/) as the testing framework and unit tests are stored under the [tests directory](./tests). Please install some required packages at first. ```bash # Install required packages to test all modules. pip install ".[test,optional]" ``` You can run your tests as follows: ```bash # Run all the unit tests. pytest # Run all the unit tests defined in the specified test file. pytest tests/${TARGET_TEST_FILE_NAME} # Run the unit test function with the specified name defined in the specified test file. pytest tests/${TARGET_TEST_FILE_NAME} -k ${TARGET_TEST_FUNCTION_NAME} ``` See also the [Optuna Test Policy](https://github.com/optuna/optuna/wiki/Test-Policy), which describes the principles to write and maintain Optuna tests to meet certain quality requirements. ### Continuous Integration and Local Verification Optuna repository uses GitHub Actions. ### Creating a Pull Request When you are ready to create a pull request, please try to keep the following in mind. First, the **title** of your pull request should: - briefly describe and reflect the changes - wrap any code with backticks - not end with a period *The title will be directly visible in the release notes.* For example: - Introduces Tree-structured Parzen Estimator to `optuna.samplers` Second, the **description** of your pull request should: - describe the motivation - describe the changes - if still work-in-progress, describe remaining tasks ## Learning Optuna's Implementation With Optuna actively being developed and the amount of code growing, it has become difficult to get a hold of the overall flow from reading the code. So we created a tiny program called [Minituna](https://github.com/CyberAgentAILab/minituna). Once you get a good understanding of how Minituna is designed, it will not be too difficult to read the Optuna code. We encourage you to practice reading the Minituna code with the following article. [An Introduction to the Implementation of Optuna, a Hyperparameter Optimization Framework](https://medium.com/optuna/an-introduction-to-the-implementation-of-optuna-a-hyperparameter-optimization-framework-33995d9ec354) optuna-4.1.0/Dockerfile000066400000000000000000000014361471332314300150210ustar00rootroot00000000000000ARG PYTHON_VERSION=3.8 FROM python:${PYTHON_VERSION} ENV PIP_OPTIONS "--no-cache-dir --progress-bar off" RUN apt-get update \ && apt-get -y install openmpi-bin libopenmpi-dev libopenblas-dev \ && rm -rf /var/lib/apt/lists/* \ && pip install --no-cache-dir -U pip \ && pip install ${PIP_OPTIONS} -U setuptools WORKDIR /workspaces COPY . . ARG BUILD_TYPE='dev' RUN if [ "${BUILD_TYPE}" = "dev" ]; then \ pip install ${PIP_OPTIONS} -e '.[benchmark, checking, document, optional, test]' --extra-index-url https://download.pytorch.org/whl/cpu; \ else \ pip install ${PIP_OPTIONS} -e .; \ fi \ && pip install ${PIP_OPTIONS} jupyter notebook # Install RDB bindings. RUN pip install ${PIP_OPTIONS} PyMySQL cryptography psycopg2-binary ENV PIP_OPTIONS "" optuna-4.1.0/LICENSE000066400000000000000000000020711471332314300140300ustar00rootroot00000000000000MIT License Copyright (c) 2018 Preferred Networks, Inc. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. optuna-4.1.0/LICENSE_THIRD_PARTY000066400000000000000000000036111471332314300157420ustar00rootroot00000000000000Optuna contains code that is licensed by third-party developers. == SciPy The Optuna contains the codes from SciPy project. Copyright (c) 2001-2002 Enthought, Inc. 2003-2022, SciPy Developers. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. == fdlibm Copyright (C) 1993 by Sun Microsystems, Inc. All rights reserved. Developed at SunPro, a Sun Microsystems, Inc. business. Permission to use, copy, modify, and distribute this software is freely granted, provided that this notice is preserved. optuna-4.1.0/MANIFEST.in000066400000000000000000000000551471332314300145610ustar00rootroot00000000000000graft tests global-exclude *~ *.py[cod] *.so optuna-4.1.0/README.md000066400000000000000000000311421471332314300143030ustar00rootroot00000000000000
# Optuna: A hyperparameter optimization framework [![Python](https://img.shields.io/badge/python-3.8%20%7C%203.9%20%7C%203.10%20%7C%203.11%20%7C%203.12%20%7C%203.13-blue)](https://www.python.org) [![pypi](https://img.shields.io/pypi/v/optuna.svg)](https://pypi.python.org/pypi/optuna) [![conda](https://img.shields.io/conda/vn/conda-forge/optuna.svg)](https://anaconda.org/conda-forge/optuna) [![GitHub license](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/optuna/optuna) [![Read the Docs](https://readthedocs.org/projects/optuna/badge/?version=stable)](https://optuna.readthedocs.io/en/stable/) [![Codecov](https://codecov.io/gh/optuna/optuna/branch/master/graph/badge.svg)](https://codecov.io/gh/optuna/optuna) :link: [**Website**](https://optuna.org/) | :page_with_curl: [**Docs**](https://optuna.readthedocs.io/en/stable/) | :gear: [**Install Guide**](https://optuna.readthedocs.io/en/stable/installation.html) | :pencil: [**Tutorial**](https://optuna.readthedocs.io/en/stable/tutorial/index.html) | :bulb: [**Examples**](https://github.com/optuna/optuna-examples) | [**Twitter**](https://twitter.com/OptunaAutoML) | [**LinkedIn**](https://www.linkedin.com/showcase/optuna/) | [**Medium**](https://medium.com/optuna) *Optuna* is an automatic hyperparameter optimization software framework, particularly designed for machine learning. It features an imperative, *define-by-run* style user API. Thanks to our *define-by-run* API, the code written with Optuna enjoys high modularity, and the user of Optuna can dynamically construct the search spaces for the hyperparameters. ## :loudspeaker: News * **Oct 21, 2024**: We posted [an article](https://medium.com/optuna/an-introduction-to-moea-d-and-examples-of-multi-objective-optimization-comparisons-8630565a4e89) introducing [MOEA/D](https://hub.optuna.org/samplers/moead/) and an example comparison with other optimization methods. * **Oct 15, 2024**: We posted [an article](https://medium.com/optuna/introducing-a-new-terminator-early-termination-of-black-box-optimization-based-on-expected-9a660774fcdb) about `Terminator`, which is expanded in Optuna 4.0. * **Sep 18, 2024**: We posted [an article](https://medium.com/optuna/introducing-the-stabilized-journalstorage-in-optuna-4-0-from-mechanism-to-use-case-e320795ffb61) about `JournalStorage`, which is stabilized in Optuna 4.0. * **Sep 2, 2024**: Optuna 4.0 is available! You can install it by `pip install -U optuna`. Find the latest [here](https://github.com/optuna/optuna/releases) and check [our article](https://medium.com/optuna/optuna-4-0-whats-new-in-the-major-release-3325a8420d10). * **Aug 30, 2024**: We posted [an article](https://medium.com/optuna/optunahub-a-feature-sharing-platform-for-optuna-now-available-in-official-release-4b99efe9934d) about the official release of [OptunaHub](https://hub.optuna.org/). * **Aug 28, 2024**: We posted [an article](https://medium.com/optuna/a-natural-gradient-based-optimization-algorithm-registered-on-optunahub-0dbe17cb0f7d) about [implicit natural gradient optimization (`INGO`)](https://hub.optuna.org/samplers/implicit_natural_gradient/), a sampler newly supported in [OptunaHub](https://hub.optuna.org/). ## :fire: Key Features Optuna has modern functionalities as follows: - [Lightweight, versatile, and platform agnostic architecture](https://optuna.readthedocs.io/en/stable/tutorial/10_key_features/001_first.html) - Handle a wide variety of tasks with a simple installation that has few requirements. - [Pythonic search spaces](https://optuna.readthedocs.io/en/stable/tutorial/10_key_features/002_configurations.html) - Define search spaces using familiar Python syntax including conditionals and loops. - [Efficient optimization algorithms](https://optuna.readthedocs.io/en/stable/tutorial/10_key_features/003_efficient_optimization_algorithms.html) - Adopt state-of-the-art algorithms for sampling hyperparameters and efficiently pruning unpromising trials. - [Easy parallelization](https://optuna.readthedocs.io/en/stable/tutorial/10_key_features/004_distributed.html) - Scale studies to tens or hundreds of workers with little or no changes to the code. - [Quick visualization](https://optuna.readthedocs.io/en/stable/tutorial/10_key_features/005_visualization.html) - Inspect optimization histories from a variety of plotting functions. ## Basic Concepts We use the terms *study* and *trial* as follows: - Study: optimization based on an objective function - Trial: a single execution of the objective function Please refer to the sample code below. The goal of a *study* is to find out the optimal set of hyperparameter values (e.g., `regressor` and `svr_c`) through multiple *trials* (e.g., `n_trials=100`). Optuna is a framework designed for automation and acceleration of optimization *studies*.
Sample code with scikit-learn [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](http://colab.research.google.com/github/optuna/optuna-examples/blob/main/quickstart.ipynb) ```python import ... # Define an objective function to be minimized. def objective(trial): # Invoke suggest methods of a Trial object to generate hyperparameters. regressor_name = trial.suggest_categorical('regressor', ['SVR', 'RandomForest']) if regressor_name == 'SVR': svr_c = trial.suggest_float('svr_c', 1e-10, 1e10, log=True) regressor_obj = sklearn.svm.SVR(C=svr_c) else: rf_max_depth = trial.suggest_int('rf_max_depth', 2, 32) regressor_obj = sklearn.ensemble.RandomForestRegressor(max_depth=rf_max_depth) X, y = sklearn.datasets.fetch_california_housing(return_X_y=True) X_train, X_val, y_train, y_val = sklearn.model_selection.train_test_split(X, y, random_state=0) regressor_obj.fit(X_train, y_train) y_pred = regressor_obj.predict(X_val) error = sklearn.metrics.mean_squared_error(y_val, y_pred) return error # An objective value linked with the Trial object. study = optuna.create_study() # Create a new study. study.optimize(objective, n_trials=100) # Invoke optimization of the objective function. ```
> [!NOTE] > More examples can be found in [optuna/optuna-examples](https://github.com/optuna/optuna-examples). > > The examples cover diverse problem setups such as multi-objective optimization, constrained optimization, pruning, and distributed optimization. ## Installation Optuna is available at [the Python Package Index](https://pypi.org/project/optuna/) and on [Anaconda Cloud](https://anaconda.org/conda-forge/optuna). ```bash # PyPI $ pip install optuna ``` ```bash # Anaconda Cloud $ conda install -c conda-forge optuna ``` > [!IMPORTANT] > Optuna supports Python 3.8 or newer. > > Also, we provide Optuna docker images on [DockerHub](https://hub.docker.com/r/optuna/optuna). ## Integrations Optuna has integration features with various third-party libraries. Integrations can be found in [optuna/optuna-integration](https://github.com/optuna/optuna-integration) and the document is available [here](https://optuna-integration.readthedocs.io/en/stable/index.html).
Supported integration libraries * [Catboost](https://github.com/optuna/optuna-examples/tree/main/catboost/catboost_pruning.py) * [Dask](https://github.com/optuna/optuna-examples/tree/main/dask/dask_simple.py) * [fastai](https://github.com/optuna/optuna-examples/tree/main/fastai/fastai_simple.py) * [Keras](https://github.com/optuna/optuna-examples/tree/main/keras/keras_integration.py) * [LightGBM](https://github.com/optuna/optuna-examples/tree/main/lightgbm/lightgbm_integration.py) * [MLflow](https://github.com/optuna/optuna-examples/tree/main/mlflow/keras_mlflow.py) * [PyTorch](https://github.com/optuna/optuna-examples/tree/main/pytorch/pytorch_simple.py) * [PyTorch Ignite](https://github.com/optuna/optuna-examples/tree/main/pytorch/pytorch_ignite_simple.py) * [PyTorch Lightning](https://github.com/optuna/optuna-examples/tree/main/pytorch/pytorch_lightning_simple.py) * [TensorBoard](https://github.com/optuna/optuna-examples/tree/main/tensorboard/tensorboard_simple.py) * [TensorFlow](https://github.com/optuna/optuna-examples/tree/main/tensorflow/tensorflow_estimator_integration.py) * [tf.keras](https://github.com/optuna/optuna-examples/tree/main/tfkeras/tfkeras_integration.py) * [Weights & Biases](https://github.com/optuna/optuna-examples/tree/main/wandb/wandb_integration.py) * [XGBoost](https://github.com/optuna/optuna-examples/tree/main/xgboost/xgboost_integration.py)
## Web Dashboard [Optuna Dashboard](https://github.com/optuna/optuna-dashboard) is a real-time web dashboard for Optuna. You can check the optimization history, hyperparameter importance, etc. in graphs and tables. You don't need to create a Python script to call [Optuna's visualization](https://optuna.readthedocs.io/en/stable/reference/visualization/index.html) functions. Feature requests and bug reports are welcome! ![optuna-dashboard](https://user-images.githubusercontent.com/5564044/204975098-95c2cb8c-0fb5-4388-abc4-da32f56cb4e5.gif) `optuna-dashboard` can be installed via pip: ```shell $ pip install optuna-dashboard ``` > [!TIP] > Please check out the convenience of Optuna Dashboard using the sample code below.
Sample code to launch Optuna Dashboard Save the following code as `optimize_toy.py`. ```python import optuna def objective(trial): x1 = trial.suggest_float("x1", -100, 100) x2 = trial.suggest_float("x2", -100, 100) return x1 ** 2 + 0.01 * x2 ** 2 study = optuna.create_study(storage="sqlite:///db.sqlite3") # Create a new study with database. study.optimize(objective, n_trials=100) ``` Then try the commands below: ```shell # Run the study specified above $ python optimize_toy.py # Launch the dashboard based on the storage `sqlite:///db.sqlite3` $ optuna-dashboard sqlite:///db.sqlite3 ... Listening on http://localhost:8080/ Hit Ctrl-C to quit. ```
## OptunaHub [OptunaHub](https://hub.optuna.org/) is a feature-sharing platform for Optuna. You can use the registered features and publish your packages. ### Use registered features `optunahub` can be installed via pip: ```shell $ pip install optunahub ``` You can load registered module with `optunahub.load_module`. ```python import optuna import optunahub def objective(trial: optuna.Trial) -> float: x = trial.suggest_float("x", 0, 1) return x mod = optunahub.load_module("samplers/simulated_annealing") study = optuna.create_study(sampler=mod.SimulatedAnnealingSampler()) study.optimize(objective, n_trials=20) print(study.best_trial.value, study.best_trial.params) ``` For more details, please refer to [the optunahub documentation](https://optuna.github.io/optunahub/). ### Publish your packages You can publish your package via [optunahub-registry](https://github.com/optuna/optunahub-registry). See the [OptunaHub tutorial](https://optuna.github.io/optunahub-registry/index.html). ## Communication - [GitHub Discussions] for questions. - [GitHub Issues] for bug reports and feature requests. [GitHub Discussions]: https://github.com/optuna/optuna/discussions [GitHub issues]: https://github.com/optuna/optuna/issues ## Contribution Any contributions to Optuna are more than welcome! If you are new to Optuna, please check the [good first issues](https://github.com/optuna/optuna/labels/good%20first%20issue). They are relatively simple, well-defined, and often good starting points for you to get familiar with the contribution workflow and other developers. If you already have contributed to Optuna, we recommend the other [contribution-welcome issues](https://github.com/optuna/optuna/labels/contribution-welcome). For general guidelines on how to contribute to the project, take a look at [CONTRIBUTING.md](./CONTRIBUTING.md). ## Reference If you use Optuna in one of your research projects, please cite [our KDD paper](https://doi.org/10.1145/3292500.3330701) "Optuna: A Next-generation Hyperparameter Optimization Framework":
BibTeX ```bibtex @inproceedings{akiba2019optuna, title={{O}ptuna: A Next-Generation Hyperparameter Optimization Framework}, author={Akiba, Takuya and Sano, Shotaro and Yanase, Toshihiko and Ohta, Takeru and Koyama, Masanori}, booktitle={The 25th ACM SIGKDD International Conference on Knowledge Discovery \& Data Mining}, pages={2623--2631}, year={2019} } ```
## License MIT License (see [LICENSE](./LICENSE)). Optuna uses the codes from SciPy and fdlibm projects (see [LICENSE_THIRD_PARTY](./LICENSE_THIRD_PARTY)). optuna-4.1.0/asv.conf.json000066400000000000000000000153101471332314300154330ustar00rootroot00000000000000{ // The version of the config file format. Do not change, unless // you know what you are doing. "version": 1, // The name of the project being benchmarked "project": "Optuna", // The project's homepage "project_url": "https://optuna.org/", // The URL or local path of the source code repository for the // project being benchmarked "repo": ".", // The Python project's subdirectory in your repo. If missing or // the empty string, the project is assumed to be located at the root // of the repository. // "repo_subdir": "", // Customizable commands for building, installing, and // uninstalling the project. See asv.conf.json documentation. // // "install_command": ["in-dir={env_dir} python -mpip install {wheel_file}"], // "uninstall_command": ["return-code=any python -mpip uninstall -y {project}"], // "build_command": [ // "python setup.py build", // "PIP_NO_BUILD_ISOLATION=false python -mpip wheel --no-deps --no-index -w {build_cache_dir} {build_dir}" // ], "build_command": [ "python -m pip install build wheel", "python -m build --wheel -o {build_cache_dir} {build_dir}", "python -m pip install .[optional,test]" ], // List of branches to benchmark. If not provided, defaults to "master" // (for git) or "default" (for mercurial). // "branches": ["master"], // for git // "branches": ["default"], // for mercurial // The DVCS being used. If not set, it will be automatically // determined from "repo" by looking at the protocol in the URL // (if remote), or by looking for special directories, such as // ".git" (if local). // "dvcs": "git", // The tool to use to create environments. May be "conda", // "virtualenv" or other value depending on the plugins in use. // If missing or the empty string, the tool will be automatically // determined by looking for tools on the PATH environment // variable. "environment_type": "virtualenv", // timeout in seconds for installing any dependencies in environment // defaults to 10 min //"install_timeout": 600, // the base URL to show a commit for the project. "show_commit_url": "https://github.com/optuna/optuna/commit/", // The Pythons you'd like to test against. If not provided, defaults // to the current version of Python used to run `asv`. // "pythons": ["2.7", "3.6"], // The list of conda channel names to be searched for benchmark // dependency packages in the specified order // "conda_channels": ["conda-forge", "defaults"], // The matrix of dependencies to test. Each key is the name of a // package (in PyPI) and the values are version numbers. An empty // list or empty string indicates to just test against the default // (latest) version. null indicates that the package is to not be // installed. If the package to be tested is only available from // PyPi, and the 'environment_type' is conda, then you can preface // the package name by 'pip+', and the package will be installed via // pip (with all the conda available packages installed first, // followed by the pip installed packages). // // "matrix": { // }, // Combinations of libraries/python versions can be excluded/included // from the set to test. Each entry is a dictionary containing additional // key-value pairs to include/exclude. // // An exclude entry excludes entries where all values match. The // values are regexps that should match the whole string. // // An include entry adds an environment. Only the packages listed // are installed. The 'python' key is required. The exclude rules // do not apply to includes. // // In addition to package names, the following keys are available: // // - python // Python version, as in the *pythons* variable above. // - environment_type // Environment type, as above. // - sys_platform // Platform, as in sys.platform. Possible values for the common // cases: 'linux2', 'win32', 'cygwin', 'darwin'. // // "exclude": [ // {"python": "3.2", "sys_platform": "win32"}, // skip py3.2 on windows // {"environment_type": "conda", "six": null}, // don't run without six on conda // ], // // "include": [ // // additional env for python2.7 // {"python": "2.7", "numpy": "1.8"}, // // additional env if run on windows+conda // {"platform": "win32", "environment_type": "conda", "python": "2.7", "libpython": ""}, // ], // The directory (relative to the current directory) that benchmarks are // stored in. If not provided, defaults to "benchmarks" "benchmark_dir": "benchmarks/asv", // The directory (relative to the current directory) to cache the Python // environments in. If not provided, defaults to "env" "env_dir": ".asv/env", // The directory (relative to the current directory) that raw benchmark // results are stored in. If not provided, defaults to "results". "results_dir": ".asv/results", // The directory (relative to the current directory) that the html tree // should be written to. If not provided, defaults to "html". "html_dir": ".asv/html", // The number of characters to retain in the commit hashes. // "hash_length": 8, // `asv` will cache results of the recent builds in each // environment, making them faster to install next time. This is // the number of builds to keep, per environment. // "build_cache_size": 2, // The commits after which the regression search in `asv publish` // should start looking for regressions. Dictionary whose keys are // regexps matching to benchmark names, and values corresponding to // the commit (exclusive) after which to start looking for // regressions. The default is to start from the first commit // with results. If the commit is `null`, regression detection is // skipped for the matching benchmark. // // "regressions_first_commits": { // "some_benchmark": "352cdf", // Consider regressions only after this commit // "another_benchmark": null, // Skip regression detection altogether // }, // The thresholds for relative change in results, after which `asv // publish` starts reporting regressions. Dictionary of the same // form as in ``regressions_first_commits``, with values // indicating the thresholds. If multiple entries match, the // maximum is taken. If no entry matches, the default is 5%. // // "regressions_thresholds": { // "some_benchmark": 0.01, // Threshold of 1% // "another_benchmark": 0.5, // Threshold of 50% // }, } optuna-4.1.0/benchmarks/000077500000000000000000000000001471332314300151405ustar00rootroot00000000000000optuna-4.1.0/benchmarks/__init__.py000066400000000000000000000000001471332314300172370ustar00rootroot00000000000000optuna-4.1.0/benchmarks/asv/000077500000000000000000000000001471332314300157315ustar00rootroot00000000000000optuna-4.1.0/benchmarks/asv/__init__.py000066400000000000000000000000001471332314300200300ustar00rootroot00000000000000optuna-4.1.0/benchmarks/asv/optimize.py000066400000000000000000000071501471332314300201460ustar00rootroot00000000000000from __future__ import annotations from typing import cast import optuna from optuna.samplers import BaseSampler from optuna.samplers import CmaEsSampler from optuna.samplers import NSGAIISampler from optuna.samplers import RandomSampler from optuna.samplers import TPESampler from optuna.testing.storages import StorageSupplier def parse_args(args: str) -> list[int | str]: ret: list[int | str] = [] for arg in map(lambda s: s.strip(), args.split(",")): try: ret.append(int(arg)) except ValueError: ret.append(arg) return ret SAMPLER_MODES = [ "random", "tpe", "cmaes", ] def create_sampler(sampler_mode: str) -> BaseSampler: if sampler_mode == "random": return RandomSampler() elif sampler_mode == "tpe": return TPESampler() elif sampler_mode == "cmaes": return CmaEsSampler() elif sampler_mode == "nsgaii": return NSGAIISampler() else: assert False class OptimizeSuite: def single_objective(self, trial: optuna.Trial) -> float: x = trial.suggest_float("x", -100, 100) y = trial.suggest_int("y", -100, 100) return x**2 + y**2 def bi_objective(self, trial: optuna.Trial) -> tuple[float, float]: x = trial.suggest_float("x", -100, 100) y = trial.suggest_int("y", -100, 100) return x**2 + y**2, (x - 2) ** 2 + (y - 2) ** 2 def tri_objective(self, trial: optuna.Trial) -> tuple[float, float, float]: x = trial.suggest_float("x", -100, 100) y = trial.suggest_int("y", -100, 100) return x**2 + y**2, (x - 2) ** 2 + (y - 2) ** 2, (x + 2) ** 2 + (y + 2) ** 2 def optimize( self, storage_mode: str, sampler_mode: str, n_trials: int, n_objectives: int ) -> None: with StorageSupplier(storage_mode) as storage: sampler = create_sampler(sampler_mode) directions = ["minimize"] * n_objectives study = optuna.create_study(storage=storage, sampler=sampler, directions=directions) if n_objectives == 1: study.optimize(self.single_objective, n_trials=n_trials) elif n_objectives == 2: study.optimize(self.bi_objective, n_trials=n_trials) elif n_objectives == 3: study.optimize(self.tri_objective, n_trials=n_trials) else: assert False, "Should not be reached." def time_optimize(self, args: str) -> None: storage_mode, sampler_mode, n_trials, n_objectives = parse_args(args) storage_mode = cast(str, storage_mode) sampler_mode = cast(str, sampler_mode) n_trials = cast(int, n_trials) n_objectives = cast(int, n_objectives) self.optimize(storage_mode, sampler_mode, n_trials, n_objectives) params = ( "inmemory, random, 1000, 1", "inmemory, random, 10000, 1", "inmemory, tpe, 1000, 1", "inmemory, cmaes, 1000, 1", "sqlite, random, 1000, 1", "sqlite, tpe, 1000, 1", "sqlite, cmaes, 1000, 1", "journal, random, 1000, 1", "journal, tpe, 1000, 1", "journal, cmaes, 1000, 1", "inmemory, tpe, 1000, 2", "inmemory, nsgaii, 1000, 2", "sqlite, tpe, 1000, 2", "sqlite, nsgaii, 1000, 2", "journal, tpe, 1000, 2", "journal, nsgaii, 1000, 2", "inmemory, tpe, 1000, 3", "inmemory, nsgaii, 1000, 3", "sqlite, tpe, 1000, 3", "sqlite, nsgaii, 1000, 3", "journal, tpe, 1000, 3", "journal, nsgaii, 1000, 3", ) param_names = ["storage, sampler, n_trials, n_objectives"] timeout = 600 optuna-4.1.0/docs/000077500000000000000000000000001471332314300137535ustar00rootroot00000000000000optuna-4.1.0/docs/Makefile000066400000000000000000000023521471332314300154150ustar00rootroot00000000000000# Minimal makefile for Sphinx documentation # # You can set these variables from the command line. SPHINXOPTS = -W --keep-going SPHINXBUILD = sphinx-build SPHINXPROJ = Optuna SOURCEDIR = source BUILDDIR = build # Put it first so that "make" without argument is like "make help". help: @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) .PHONY: help Makefile # Catch-all target: route all unknown targets to Sphinx using the new # "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). %: Makefile @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) doctest: sphinx-autogen source/**/*.rst @$(SPHINXBUILD) -M doctest "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) # Copy from https://sphinx-gallery.github.io/stable/advanced.html#cleaning-the-gallery-files clean: rm -rf $(BUILDDIR)/* find source -type d -name generated -prune -exec rm -rf {} \; rm -f source/sg_execution_times.rst rm -rf source/tutorial/10_key_features rm -rf source/tutorial/20_recipes rm -f ../tutorial/**/*.db rm -f ../tutorial/**/*.log rm -rf ../tutorial/20_recipes/artifacts rm -f ../tutorial/20_recipes/journal.log rm -rf ../tutorial/20_recipes/tmp rm -f ../tutorial/20_recipes/best_atoms.png optuna-4.1.0/docs/image/000077500000000000000000000000001471332314300150355ustar00rootroot00000000000000optuna-4.1.0/docs/image/favicon.ico000066400000000000000000000420761471332314300171670ustar00rootroot00000000000000@@ (D(@   f33Uf@f3Uq{r&W&|%v3@UUUl-zY\mGe:a0Ƙ]'ؗ^+`-[!`1^*_,՚b2hlEc7ɗ\&[ZYWVTSTSUWXZ[Z]'f;uQq|'@UUʪuSod8ɖY!XVTSSTTUWWXWWUUTTTVX[^+nGcHf3f3yYb7ZTSTUTY^ab``g3e/g0a_aa_ZTVUTX\"jA`HUvTja3ZRRVTYa b$i4ۣm=rC}}V\\@W/,%z,W2~UEzP`sGm:g/ݞc$a ZVVSW]$nH|'@U}5iD›YRRVTZci5٤rCaB̦zaGrEi5ӡc ]VVRY_,{Yjf3@}an^1TPU UYai4ŭZ]ϯϯeSqDb&]UUT]pKȶ@f3pM[OSVT` i4ۮ\al!ZAQ_zFu}Gr {B|JyKc\Hf(տ pBqCc"[VSYi@θ$3f3 oMWNTUWb$xOɼf̳cP|Gt6Ƥo'm ppp pp ppnn't7˨yB`R̳aYh2ڢ_UUVe9㵏p03@ lHWOU TZf1gW$v4~Ls5ڥo$o ljhhjkihiihilo n#s7حOè&̳wMe$VVV^*篊p0f3UnKX OU S[k9չz,fu2{Go,njhlqttt$t'pv+s tvsnihgknq3Pζcjd-YUUe;弡&f3sSX NU S^sJȱոLo*mfjrrw/ح~=JqYM`5l(Mw+^6UNIs={4֧s u oihhn t:ܳeQqFf0YUVg>å@~b[[#MT TZrGgHt:۬ofhrt(K]Jĉ ժ `@Jz4wnihhn#Qx5k;XTYqO@!`4OTSXj:̭S~m%ihmt#Ji.+II+ff @ʵ\uz4ujjeo{Jı x5e/XR\vTpkGTQTUe0x1}Noehqx3ᵑ^O@UUѮOt$mjfmo+ĝv sGc,SR\'o.f3f3eGX!NTR`aaq2n eguEгUU߿p0\YGwJbBIm[Ir fdOx4pifju> `mc"RWkCkHRQT\rHzIkehrCf̙fĢq4R>β/'! '.鴊>_^ȶ`Ũ,w.ڪoifn{JuO]O]$}^\fhGY$OTUf5ťSndhrJ̙f¥v6M0"###)BԼa\UU+Šu#y7nhdm^rѹg6USe8տ UUjEURR_}WrcUm'bgo|=̻UR2&'&'*)&%''#'Oƪ@+y?u.ih dm.{4]]a!O]!yZck\*QSXf3åm1eg iu,v8ӹD &#*6:?뵐<᷑?嵏?:2'$)#Eֺ%3U[{ugfiwCm@WSc7sN_WPR^zUuzMkdgsTUfе&Dۺ &#(>N½_tj<ŭ{ªm̭z¦t.lZZF返7#'% GѶm}?ʫoick!e[~VG_ OZ!bI@@f>VQVb*洖q"x3k-bfjy8Ӽ`ӼBع%2OħvNνdB(%%DI^Qu#gghtBk>WUjCı ^/ӗQR[oEyIidfr RUJ#:a̙3IUUUժUIǯ@O߼-%#O|=phcj"`JyPc]Q\)๗f-Z(OR][\m3cfhr&m/m$Zi"~!3dժU@ժU%Sں$%!,¡q6q Uctgejt@sb.SX{ZUwTOXOT`(鱎j$`]hbfnz9Ͽ+۳ 'SU ȱz.˴Dʰ:нȱ.G$#HUħu,lgen2ȦlAXUg>kCkWPWh9uDhcfrIUWi} 9ǧt7ժUƬrk]ѾSTTVd˳J@nk8#/v)|=qgbj%fFvOnZS`1kFVQZn@m3ĦfefsNf= &TUĨl̪VĢIGC EGǥMäWĩkժIǻ'!Sl`Glrfdj{Pv^<\!S[(㵌klBVP\uLv{k.defrRKƪ-㳁1`rUǮy_ʨUDJJUYNGE̩Q§gIal4>f]?pheisAa.RX"|`@[VQ[sI[}M+fdehn }*=a7#7cM@]ƤLKI\ε>ƜƩn}RIEʩWɯycIȪy*:5q"s'jfgn2߿f6ʚSVrN\c2VO^~WOr8h+bfho@@EF~9ãjH@NţMHȦOhx¦dƤMJƣJ\ѹ Cո! Ȥmås'ܧkffp9i b+UUf@jDxXN_$dOU*f dfgtFnl4&!0VU@ʱRƦVEIYϺ;çjţLJHV™Hɹ ,ر ѹ t%ըlfeo7b!UVlEsQ_XO[qDVĝk/ݣdegsC̳*۴"%I ĨlͪQEIV`_QHIʧLbζ>ᶈ>ªs+jegm2b-UUd4wV>WN\tJno7fdhnx3̳ > !":if ĨkɨU IFƤK HEGɦKZϷ59 ¢lG5 Ffj)q%hehq:h;SU mIr\*PZpEwFh cgir!d=_V~ %&I!IUƬtlZսRģPƤPƥQR^̳<UW)~ YS `Mqfci t@{ƪ b/ΗRU jDa2SUj=bNgbfhtI9ɶ"'1V ̙3̙f ʮxHħjqīqĨmuƬwG@mĨwI:8ÖmKq fai[Lp _)PVmGskEW Pc0轜åp:٣dehmv)濤vnJ)~!%(3SĝU@@ɮ&FḊ" ~ [x{=êmfdl/ĝ[FZ NVtNXeGY%N]\V|Pi cgiu L@V#'*3K§pTm3ƫzCG๋!!9ѹ q9 j:q&hehr?rK}ZMX {W8۶b7җQXoFȤl1dgimw-t.N$(+-AT¦rS׼ɮ}/[=$"1p9USqebj!\Pi>UO[*ൕuoMXQc1켡&[vmchjsB S$(,)2BIڼV]~cgĦsdYjYOA0 $/jYU̙fu2v2hfgs?m1_,NSe=f3t,],O]sIvCǨhfijxXUV-%+,(.59TS^n@̹z,ı ~Kiu9ѩpojikquru)w&۬}<ԮL֩x1ըv+r$qrmghn n*uϿ@f30f=US[f/^j@ ^Q|Gs1npmlmnponpnmlkl k r4٫|In3}Za(RRQOa5깗,f3&g<ӟZSWcqEp9ħWR~Kw;s0٤o&n"jkjkn"o&s3էu:zBZDı @q4mA_QS PPZ#b93̿oI]RT]c)oh=̟]UTV_bi6ܦvK~WUt!տ j5zQule\2ya)/.=w79?ws?ztArļ'r9!CU$Ni7ugO#WUʩ_,\jQ_8T1'S=_mZ!_W~V/N`'v]BT(bb]r 9{~%F/0~0ċWXWCmN t̩ϞOvݐo::&s'kPDJyvǨ3#C *+(Xs~+.7zLOKNC#7ܼ uJzv7APcN)ߚ9(ٿ`:(*Vn֑/ܨCr^ݶ^;e(_4>tBS/B瘠Βr`A 6I<_0%1-,e#'F}q[dO}4 @(r᜸ȜtY; O i'[8*}ckMtɦC^)(~U+.&܎#?'.^!-Q7z{J+8zӠaQ'r B"W<vzAcܬ~C9J<6vL91q nExBn؉U/XaP,##vZJG:ŋ,a4P-ǔSGLxgcԙU?u>yM P)9G^yW 0̑î9ԌkLJ[O,#pJN!/\+_ESy;B-Ę?GrJEḚKfX+YTݦ$9)֜b%5r_5g Y[533fo{:Տ5}gB:A v\MsdAi2%r nl 8c:uUgQ+Q#6IVA ?~kL;¼sF9Y9ײ(@>+yNnqA]:8G9W 3<Ǐ`ֲ;pt_ qxמQl쵣Y<žj =2ܨ=+Ȋt%L=7̃N sc̷)dQä` y@zOl OjOLՂzPΙ9}Ue:sq~MjҨ Nȉ;ly$&|q9vIcDҝ;hͻȨK? Ñ$!' r Ztp7b Pϙ9ǘf7;k@h 5踲|QK϶{Ihλ QM@ g@*u =Y1GhEݭMotBXyThz_&h37`=,?FMKA=wyk?A;uY.NƜB-_eK*gOQvȍLsxŽwxobݫV4/%}7@\GyBu[>Z_t+[GR&.l~t׼v=v9'Qw:`< bܰݸoF7Mz7x :R{z'ezf^|-zz_2qs>SGӧcC ߾AÎQ?}'5>uUꝛtqFҝݫ.cZ9ݠQΠ?yI!3O(f?镵}.]qY]]QG_hgH t|=|G7YY: =>|MS."2.: /e*\9r)Yvb M[X9{|@W:Ѽ{5nzVt{AoYf|s (d t9P^3 ϱ9ǘaNI`:}FTnŅB1}Fzɤ;-g(?kxZ^aK/%6􋿾c؇eϗ~sNFo(u̓GFtiNJlw?̅M:3[^^{@7-_U)/Ї[>$I[ЩXyzrsbΑ5g#(v:թNlY>Y#DL#k?vl@%Pd ]qvM"}d:ܜc 6?W^yW^x|ʫ4?a%־f|5r-mw#tw&3)WRk5 81ɕ|x+C Jke596`sSns3AwOz|#E%M#t C?a7{=Fг u /6Dqlw sn_}i~sbԵH]8%5&ҥt݇tx'&]^#ۛnm{r?/z}Ma`bq'e`ݾu߉IL Y%b~ +[ҧ)ksj'YtAwI譻yn='y_A+8ϑ[B^=9s)>7}_0}ITxWj鉤>5ݫYMtA_*_#:Ƞ'J5:s~rd_:~0+oq-[li{4h^I~=@omZ޼Afa{{[wE^LԻerNYܠ]}|/] w߮s!!Qw{FR:I)/=HQӠVtv4J7^cd Io_Fm\LԓԩWYAl\G`wR!`bq.7sd9~)gIAM#apݬWm4ްxQo59(-oK ɟt-laj8Ď8NBR]i>[zN. :I}-\7y?et]쵝yj旚&7i^f+o5P Ǎtx!o,B !Df4%V=Ry;y͠;O!Α9eyb.$K[ᦙÇK+6׵8N,.a6 j\@w\!kxsY]rՏx9l_ :đ6s97oT+.Os9jIfUtiA!'ed׹+=齗~~I=_2~aҠuЧw_ tw"z#Vf q)~)DG1?9O[|K|,LYP❙& qtzȝ]=-ITf;7\rqo?8Dדʵ칟<~]-AX9lLP뵟H̙$o l_KM$#"aw |]^5ot/@oZ:<*nx I]x!g\-9{ /o⹳Oy\3Ok%t938;f/܉DmFPIVG@7xU}aB_f6}Ggskk8BHMV@IM|wE2Iu|tnǒRHm^Лtwv-/PͷhyV?2.VOթ~3m {c7w>t_]Hٸ ֶָ|@n-{o zUc)-? %u}qm.AP[a._ѯ !:*T2SL3-ןQm$aSɅrكd:Dz!WmUť6Zrq;kt ʼn'"/+8=#8 l[f&~[_.WFeKYzOoŔvű -r S9"9sߝ.VpN%̐u-R=T7,:gµІ&ebumWeFK[R~HRd{K /]pEًmvkPڝ)tȺ :n0_~j}lE7*ks7JtO2%q*(+e%77tgbE7vMP6Ct c9R@{ij}:o%K&>2Ax#fP)c?d=2oY }9;),@o[Ao@/%7m'qîVAm|-'H1.[?v3ߙ4޷nN;ff zKXj7gb;=FtCt3-i:KY& qZ[A' lff=~Umj%Fյ t*]\I?#?[]较AWA7::CEOOյtɢsq/l$5F ?A1(臲B+39]Ag3[("֠=/xpլ[3d\&}'W@7N8kK zc+bu$XkXN\wNkrq%15׻6*"r+1+LFVK :ehFU[(EZ[ 1dՅ) yyn+C%L~|06їTZo3LkL@oR/nAYfYwR^{9Xʺ嵃+cilY:Y4lSkHKW]MҮ}^Ze7׶X:L@UҢa[x =TEq-_;o03Hx0eoi*%=ukƵY\. ]ͮgk H2U3Dzg$uqY?s3Aw=_@dW]g1F͟tT@/51 \@-jổ{ݷBcb?o}^S̍ tE^:|EܔX] z.1Ɂ'5L]iv2YƆ=k\a~㍇ ?5̌{,|VC@SaӞ#t1ih>i/7VOJVu4xmZGBz,ݬ ף?)V2w@oKzgU.}5>a;p!]%e|WUld{' 3W6յBi%+`IDATJ1~pٹڟ{t@F&P0(?m8pH)!*_jlsխ,f2r.0/EBtݑf ҃CO:@A-L ;)tsokt &?k ];oy>)Iwz] nq@}I6aώṙ͔ TwpHeTs>ECX}|#tct96pHGt[@WJ,WUr d ;X\_A6d+fdwt-H7]nmf[i]ޒ,Ƕdv-lb~cMn - uz7 DҳX1RH/o@S.(;8 6{֩gf?Y^E!zmN^Iw1.bONΛ]{ M0\@oj̅5 L۠yb/^3n-g zl1CAփt}E:Yr|;}f-Xϻ{^-o<lu]ũ8aCC&XvM ځ@Gvw--*Yʰq}ޤBnf0u=#+_E+>;Kvϕ kwg=3}n7M4ia~etA:vhǮ]9.BN lŮBM:T.9_wն),BtSnvn*ˑPeh'( Ot9zkcW Cx^`sTh-T?) ]:a.,]vvKe٦ݐ*748ݺ3?erLP :}M&3*kAqsBTI?}G꼟]o󟾥YpayΝ\Ș;u{sLn7Gzdux5 {JӼb&b;M?{53r!INKk8rX# 圻sרnM5{84ߥ2nZձW&g@_{ D&rrbt%=3yqd=c-lsKk3qqĠܣAo;֤Yu,Jl5M35tZo F 8.I9ƨ/28L?{_ ؞#e4~z`wsG=>o7\Ʀ4@/}3+f35i}WYn@ ](=3ĤR\+L: ӑI S=:Es۵p[e|A'1Ͻ:Pqhh]"nX%4jRdjtsuSq[V sy&G/Ekk~U5ּE,QZ}}M6ľHarK韗6F.tͦ: $rJK GO#8+ݵA=݉{uWy;Wm`7Qfwn=AΝhYe zӊ:w ˆHTaSfDb-%54sب`7o_C##I>p\;At\wZdN=:A[uuY yg|nm*OM'PF9x#ΰd ȁkc޿EO|>sMziMHe0I!#C jSr8Z0@wm¶(DөIm3GY"tQNP;iG^N/ ͯb.?D7 *^f6mSt%SnV4dsŝ\IG Qd˓/wlտFy9:]AFd1,>&9('FaǝЙֽ~kN[iAw7yKGxF O^{íO2]Gܩ~ygP#ҹι?!>/_۔՝ްnsuگp` X@}㧼uR:k="іpY.VبIqHw͉/c~ ?M k[b9g!Oz;n5fox-[*:1ֺqjZx@J|#lXg}U\kX)~=c?j-r7BZybI@!I<G#V]j֦ĵ3کk#cF0y8?_&Y/tFȭuQI5Z4YռhxC[g}H|Ոl)\i1j~$2q͈kSn!v|!:R=YRZ#} \DȓB8WiA7v(㫆tx a?wqs |T:b|n/ ."Qm\)Ja/)tZ:;];uԵklB=>wS󣳤zBn>3@^'i{\~nJ: өM?2ۧKOJ|4<&2Wm3TlRkI=!GU}y/]Vqu,(8zT.m{e@^Ixg{|0JNFWZs1O5Ώ}R3sL| w6:C.Twvuy"Oh䓱A缴R%=M>$b%sF Է Mes^X95{>@|tVniX(#ȡ\lVGp,ߎsR&wEf+qIߪ֧GSQ=&Yc sn{+gCҁ@>5M')$%7m߇=qss}|\ʹb},\84{r73b-Rr%8't^+srŌvp@ Vǡ;Ӌ\Fs{߳rQ4z1*d- uU|=@mλfӻK 4Є+\4:PYP!oFzXt۶7BOC1gE5sKqRQa eqD%|WXOt+.6u}wŹ]b.S:e䏢H rnb]AQwk}:y7(ǘd;7珪Af5kz@)W\~&FaҹJmԬ]ܙ;1Ը^3aۜ_*̜.9wD V;ݤcrϚޑQs~UsgOȘy33&}w~%c5ʑ^ǜ _]U{T-tJ".cQ:ǐc 5E5}a$]wT P)אCӃ{;Fu݉vرˎ9Α9j8KhNgv8a@(?!͢Vs y8ME\%qs'Kbu4<*ODv ZrMO5. fqu)0~lc -R$ ; ٪R7,s-%֍mHNu/̔=7_œnA@e%ȨF]7u vL{ѢQR#FąkLIi+cC.PΜvW*,aEQөgUa[4oV_S"sٚ*/EnթY'wU!qіkd@E]u:3IJk] %vC.PNRp㡤.IPfVg3ʹώv P@([G<]=0:>="ݭo]#ۓK5 PETpTtӓ05$~ѱ2  m=:b%L[8TFHY/DIENDB`optuna-4.1.0/docs/image/sampling-sequence.png000066400000000000000000003063271471332314300211760ustar00rootroot00000000000000PNG  IHDRvf9zTXtmxGraphModelMVΫL}OYW &`29O3]dᦫtաug^uCZUA "iPF 1/e=O:GǟjEE}j?˫t8l!`< 6XzZ95BNɇ .WH*LA|r*xb+ ; Yx_ T=әwiP88  t`MyIJAUh1))waj[ MZSN7buIO1:p{Fu'H QBP(h f6jȿ\P7"5|Kl6Ħ6fۛOdm*Χ>Ck膏?G7zk& QC[f(L1P}Ekv,b#J9׷9^ R őGQ #Õ0&")NI/q"4 n6yیd,\US9 ;zb-prE|j.)_ov^~FNlo9M"6^nÃ[~zOL{QMM(B8Vݸ'bhMc ۶^i pHGqu 㸀썐P pBJR>Aq`VY[tGF=4)25Fw#N_&vǑ[l|ۮ6wu8hlS6$fC E>fe|Ӏ7knA5ŒQи|`B^B 2PڞO]pV|_k!X~VY)H \CQyȗ=ٶsyS2{yBpW)`@X+^:2+ݞȕH>$g_PÅի8U\{ѹJY5($A^10l}JEj%RMs)y}>[L;a2ٌxlvMAaN1 sN^halLv^TJ(tX9 Z0p_N酀6}1*'Mqkvϑ3.|{'{p=}(U^H(XFbՆς0 Fu#^Xyal PZ(,|lfho#~f+^rY ^7joϣ]r%1=p |RF>ŹpÛZ(QCHjuQKITU;<>,?* 4Aw(?-)%Nys\_v:ntG9B`c!F7kАpB{o= f.`TIvUx~&4y|3Boz(%{od&Vim%_c/ x?fo~ hVvq;7H2Жϴ.H-oVFOj'zrC?C=d76L IDATx^"P0آX"h4XА0Q^bXPQր+bC=7ͽwS|+sf~s>ȏ@ @ )=Ncǎ2c!@ @@^ :T eȐ!ye @ @ SVYeLE c!@ @v  @ @ v28̆ @ @;@ @ Q; fC @  @ @(!@ @c @ dNF @ @a1@ @2Ja'l@ @  @ @%a6 @ @@a @ @ @h0 @  0 @ @@F d4p @ @v @ @ v28̆ @ @;@ @ Q; fC @  @ @(!@ @c @ dNF @ @a1@ @2Ja'l@ @ Ja￑>xN._$-_,K?["} т@\6ZwsٸeVѺL& H*Y^%@bL*I.Ur4 ;}dv&Κo}YmcH?W~)_~|ظe %k'B|v: B|R!_*r|x`-*tQ;߬\!w/$/7__f?i:*@̑XGh}l71Ji @JCJhu.ʮxU#*ϪiWvn~js/v[!˓o'_}q4[{ȧ|=^'_UǍ%@JR"OHKJy2kd步jʵW~.~~zc000^xy[WL{k䫴GJ _k|oۼMCJ\dCd֔N;.8O,EAWKǖ|Hgm|e[D!_#ζ{I=?GGm2|/Oi,&0;D❵{oWMޓl/+cWG|ea*qag';Ix㧥[f\ z7]vCG,AU[#xȟ /SQgaUW]jtԡk& *q|U FJ̖^y민UY/COSgr_1A8br:"OԂM(LA伤nJl qE^$'}uf;oK.t649G7gNJ)Q-TO+pn;'6+~Nus*IFJvwQ ^;\ rě ϑ=ڣ h{qW]/40" 3;aҤ0 $>#9gKYed9]wa^m[q ;RF]s޿ rGJϣ{7_دB.W^xE^QcHw~GW>Gȶm[W_~Un0If͜U \]~#w?H::*;厩w;ws.|qcگK>dyaF7.@9Cp֗)(#!mJb<ޟxѹ29άEͥ쿏G6t22k].JvP%jWi񲔰c?d|wrg}\o)>ch4˦tE;ˑ!<(q#uf ϖcAβ|ǩN{oO }T{iS:KwC_zEQ4hPg~&N{y'_ݞ-9K8 _%ؑT2bwiΰ֏yLgIL.,䗛Hw/KMx軕~{W z 8NZʁwmg>R W~'7fYha]׶J2ғW;{CF~^ƫoʌfYsؓ[l.rάr];ӑ| ٽ΍W, -|i:d<3Bj;@2ҡ;?x~7i.wNKyxΔ^?aGǠn;k=g̀ҺEgtû9v4/C+W~_n zԍf?&t?VzO_qoyP166P~w0qrHvs3k?;Ͽ,NUڶm딾4\62%`۴Ȥk''-K{wjAio򋯐{N~1yApNѯAl3DEI2e+]4Ovf?:i׭g}ybΓrěh Ž42i}ˢK]*n©g6^c 2Iҷ_HYm-ҙ:3a=L6ڤy!gIﾽqǔ gDI>4agt?q}nV~`HKvlJ*_)M6X4۠S;+6YW 7;cA R{ֹd{CEϛd /}36E3v_|s ;A3n=#V?askwFO-%vDҽZ:IڴQVnشA9ޟ/63P qyvTKoSX6ˆmև!ݵ2!̇䒫F:3Tؙ<̗̓_|O?(k:v>_^4fm{nK a'`u!3!xꭇI] ] &/[ r7`-(2rpeWt׮A vm-G9&5/XĠ2&N+Tf5Vw ; vTN9T9 :c~/iB,0ԍS4.I,Swfzsıu%#u =kvr^@~ig蚍Мr諝4\3| YyDRŽ),V(ԦR6˽#V%KTعu#zE0rd'֛5i-J~ `slrvr{jk|Q N9aǯnMкx謺ZVYGgNx{9= Q︝s?j;pl ;> o.r m} җeAl lXrbwĪTn%~oTPFûRv=޲jeZUZ Kg˒ޖ%o-zy[gu̖)I%/=RYaGgKM*»]lg:{s]~Sa'-;Ҕtx<}ZY6lV0~%cTvJe~IZsvGr||s/=GםN֛.w,c0򪙱SL1cfGU_Esě5ZGRZkvBŖ[ ;mAc+LZ*{ 3揼 ,猈HqwhatYyg ƞ*8839h_USJ@ ʲ#oA@"=5 ageg;_uI.yG ,쨣)%twoGv47"D:TiiWyХ뺤RgJ:LvVضկ/#EZbF ;A߻\۞[k]%ޖ3?EƻӋPlYS_X*Yen6r\sNab6[u _i6MA- ˺̮X{.źu-2bO_3vIZد]fvw). ZGT-~YTl15E=3=#,3^02bQ-H8z3#Qk%EA6\e,2"ul b\)\jڻ䚱^=]U Kݾ3lGpպ ~}_Tk;:pΰaҾ7*vtyG \pMZLZg:GA+|aҡS_*y +w_Wi Pc3 MMeY{pqH*P_ rěԁ>SxS^qyQx>[|yVw]GW0ݥ-[zB a} F=@g謕7R%HG/T7ݤT6:{߽t<8GVkf`cDn䆫'96݅Alӵf[r.dݕ/:Evn%|]W1g-h,7*lGn|[fa٠^6Z{tFL]"snxPJsl ?|IqxyJ<1n;]6X=gV'}q7cm3U-zXE#,l/X,|S(Ԭۂ{Cs[v}hWwGqow^mD=[9M ;:shidћcz:9_V3 IvbTv喈[_>%_#߇@VaGg쌹hCovm-kMeه˜e;F$f5rk:~d=y7;sg˾[gc C:Ry6{펛[5ǜ5 mK\RaǦV;8$u#Wtֻdά Nnˁ\}9[wp׿eՓ__tb-3ZJ ggͶج^Q<41 {Cof<,mzgv}g=YG.|CvmԇN-}NLk֣XW_~Un0If͜%+[ ҽ^>Oo?u;@z嬇Ï_ՠ 8(ͰoLq<#p"Rnկ33{nGf5So\8:W R:tg*}@~:Yz:y{ؚvocڬ#Ɖ8hw j7kU&8;_zw6me/Ucre4hmiLt:w^ݝ'RP|U _VJZyw>^v)ΐ.|dfRwӝ*sa:A>дH>_UTwKmuN׿ڏuѵ<0u筿RaGjs܌H*_1c'F؇ޜNuaE0j6gR7r~О?$<Ĥ?A; 쌫*n{t{l-ӳ&vfg:tZۣ"/6n^g{ EfRR7rdpi}\z%gJx iWvӯ;vmwˈיUY}v\I#ż _Sر5|NGm|e{|8e*qaGkw^>Lo1? VP~ӨS=F{챇jʲU\EU~cUWY\vjgH HSJ!xͧe+7Ɋ>u\_6^+i&tMd%zϾZ&˾\*}\}{iѰu\sMi۶--%@4E(5B GN|9t@LrkhZU^x YSwvx6ڲ]vL}lıZ+WՒ$>W79 oUh:i䫜gWּOSJO瞓{?|7$"kH:pƾMkˮow$cK˵ _ƫWG.w=so @@vbGL;pW@2;l@ 9]ѣG'g=C~G3Wu.ᰏS)AF(U|+^A"fd1j @ (D9  0 @ P! q: (Vp$?v"GL @`mMǗ;v @ @ v>3vo;$и 0`|2f̘{ @ =v(,A؉j FNaP0 G`+Wٳ@ >2 40a6Qڃ Uڄ @Žu@&2Na'|œsl\ r;)PX1.,DI;Qҥm@ Llw&Mڂ Z Bk!@rIa'ai@$ʰ` @@ 9:x 1cw| @ ؉*MBjdhS0N*€ @ h D5-"%Ի<  SN9EV\)cƌ = @Aر{ ߂w9 4nB :u$~<#!@@z?=(M zv @Ne2{6NfC\g!iOt0Uv gqgz@&2N; C G_˜9s@ >2 40a6Qڃ Uڄ @Žu@&2Na'|œsl\ r;)PX1.,DI;Qҥm@ Llw&Mڂ Z Bk!@rIa'ai@$ʰ` @@ 9:x 1cw| @ ؉*MBjdhS0N*€ @ h D5-"%Ի< @ @Ii`B2 a'$io9>… 뮓CJFj2ΓKڵkWG}T?x:ul2PSLI&oӦM]=I޽7}C8:dM@ow|vo;7!P#c}]5jTMNZSvua1. @|Yvrh܄@v",5! Plg̙3C=T Pg),XPeɅPAfX?~f]l)Vs];Vvl.A#`#qAԨc~ I2mڴ@ΈB4z 1^l m{A{g츅`䭱7ErŽE%e #ţb ,瘾N;4GbNnK / NZ"@ !*h̘1Ù.6 H ҪU+B^!+:iۯ8r1aGms ;~6tӂUxr1-b-[)l73S̹Yfuae63K.$:Fa $*" ped̅ ![ K'Zϼb2* *4VJ5mkń# )]-I+7c، 0nf]t#aŌ[`j- ljWzr= xu[Lq^g뾘:^O Ҷjo}wm 3vǻW9eرuY/'cln>;1Pt@@Ž݃ac]N&PSjiB)RrcF1LKSsY=;51 y;aIv/ mN⋛v(t-,ZHB%\+!i]X-\θq ;d"M$vLMS(Vn5VR,q[c- CfBU;UaE(ً瓀Wh.->}z=aGIywb^dGn]jǩrm7jԨ^"w֍-x;*^]ܻi"dW,6badW,_v*g>G>^C TKaZra'c\(V4X5t\5;wvjh3K}cuח0SYkeRx[1m4k֬¶3X%QvLM!Sٯ_??~b3qn\z@ P<9Ad@@ر4^z@& R A=z|;zy"Iځ ~*q9 fϞ-{$ `" @H0#`ӧ޶YAO@ӠAYreMNѡ=@ Jnm#$, @vLW<؉!`tFLeC; FA M+!m#ANT&Nzb%L/C!8/tv 6bOؤiHE$\{v4 @ tAm~ ||5] 0( &T4z3w!vi @L"_uQa6K['NŋaKzU=KhG?{{,_Xv*F P! ete5r |w}eqb!Pj)[2d,o:U:}n-?q THaB`Y=wYvC _x-&' :.#D P jq  @@RNNˑ t @ l;a=@ Tp ;ZC,*fl3vRy@XD؉5=A $H@_ltL9Aǘ`5WCřvR  @goҤIvD@ vRCة`V.g]V"733+ !T@IE0NSNiaҤ-@ *￿,[L|ɨ]@ Fʼn@ ?<frD ;!o;?C mv ?;9;9 4nB v2@̇ev, (@ŽԻ7!qx1@ر,i  IDAT@R; @j#S? x Ù^ @2Fa'c\@(Vp=8v/A TIaJp\5WSP B 'A BaǖH c@ر;z@&2NଳΒ˗˕W^qO0Ž Q@@ =vo;n7!qp|SOėl cCP|{ _7!` ["Žq @@ر=?nj7!q; C2;w XJazb]N8jd< XP܁ `)K[ @@ةWC@<vL/ @#a. P+8;v @$S%8.RG I!섊 @l!cK$@ر{ ߂wLIq'p9Ȳed„ ! E|ŽaFIq'pҥKe޼y! E|;va''MBaǖH cGvl1c''MdNŽe@@ر4^z@&2N; C2;w XJa @@mvj x8  @@ d,` % Ž;@ * T jBA;1@ [ Iv;vǷSrh܄@ h>H{ vl">@Jaqcw| q#$и 8蠃wޑġl cCP|{ _7!` ["Žq @@ر=?nj7!q; C2;w XJazb]N8jd< XP܁ `)K[ @@ةWC@<vL/ @#a. P+8;v @$Uagʔ)ҳgOӧ5J5jTshңGڈ?رo̙2w\i׮]Eݫo 8йΓAɰa ?\X iӦԩSYfθlѢEUal6tx`  @@JQYpt],XPB5bI!EqvGa̽ӶmۂcRQ:g"d1jUԻ*q  ;s=W>Cc!x dUة%TY-[X [؉hK:A_ DG}Tڷo_0 N4NbEȑ`Q@ d|,YD~[9@Ȫ}ꫯ,U5kH֭[,{9W%ZnAhС2x`gOLnwe_=lR}v?77XB/&L30?G=S yg]1/*}~Μ?cYΝEoڴqrW:KLc\~9裝{~3eO&~Ko˱6muZDJL_Pa'0*N$ |Q yџ1cS}K. o^^L'dƏ_ogulܶy7+ofv%JW6ThӦMey^q?Q>:KJ CvJE kӰ6-U¢!%1@&2Na'|XFFa]b^P V  S׺_/A6EjœM{FpN)ǴM=6K}%u61-i6*1)\n[lւvBI3v3srlW)UzrvÜ]U錝RK- E9VA~ {Nvc @@I RG1SIh[c=üEjf/ n!-ty_=[-xo9a_-sy|kT#f-r;[T_ +\N5Pcd읋a1 @@ v~vJa1˓* Q1! H~ˆJ-:*曾R_b}VRg-+ ^㝝Nar6cG-3vZW1 64.OZ! WZ#S7sUy~˚j9}WE4Z~[E1Žcf4*\l3{lg&0 A1m d HHڢg1K.,"hiӦ9uLQo {{,_Xv*F vRh a]|iO~w AZ P} @XnAgȐ!Va+ I9;KVyHi@n_ D:MB@/J ?)Ne9y&' :Q a'#a'rpq2 - K % P SJ Si9rTTđ;ىUM"Ԅ! n @va$@ naGk%Wź"_E 5`b*5vjǵ@\iE)%΁ o:kc&_E촉Xa) @@xQ6]Au,_\4i *0*+ODر28@ P+^j%@\WqNg?; VA $L@@`<aʰ @@xQ Cq _E: 3.[EБ  ;#]w]$ T/Ά#@J}zFICbcL@9yW瞫- P+^j%@\WqNg?;KV!섎! D&! T5:.b&@xʺCIY@2a'* Ia'LJZ r= U\N:U ) BNPi/JUB@ f䫘;s @At+ W|Ž7@ T5:.b&@xʺCIY@0 @ xQJG(O|Ug ]| @/JUB@ f䫘;$*s(Yڅ$0tPYh Yڂ P^E@W @OQ;) Fyti*/ a5I;&R*f)a'e. &0i P+^j%@\WqNg?;KV!섎! D&! T5:.b&@xʺCIY@2;Q]@ L &mAEV\E|tθ` @@xQJ8t&@ v +NA JZ r= U\N:U @ E)= *0*+ODر28@ P+^j%@\WqNg?;KVQ<9t4D@@'[2qZI@E2^ $G|4􌰓(`۝. t]^xykn JZ r= U\N:U;#A@ ;@I@j(U ! bvRA؉,Ba@ &mAEV\E|tθn5vBGJ@T&R*f)a'e@ tE)q @<UyF6cst  @j(U ! bvŔ @H^ _gd;6G @&R*f)a'e'GEv!0 6L^ul TEq US%N)lw%]چ"paɳ>+/rXM Tݣ>*f͒V8 L"&MߦM2cŊ2x`9ꨣe˖K=#p7yU:␔;I_ Pq ^U馛FQ!㏗S:Ov1WCRV $E>~vbNw@UvE@DxQBq@;% %_U̲v, h1w@&2N; C2(Y7:d̙ο>}ȨQHꌜ &:w\i׮]aG֭Y,ͫɼudKő;O:w, p[nqflVҳg:cڵ?^,X wD1h[?3}k~WXզM޽viueyv*M 4 F~R>V2LK4O?tah-u+… C+zM)y17泐lWYA6"$I!@RK%m4Eȣ 2۷sR,?Ov„ێɓ';ˆ٢E(#t֭LLm9rd<~~q;c=SEE 0{6MVO^;i7.yƃgf͜^ne⨂WT4Bʼm 9ڎPj̧6Ql*a'? @@* d˷W1q]ca-ҧb^1}"R-lĀq93J ;* ٫سdY=^}3edr۪U+Bnwe_q/Ʈ=Ct[ .ƷZ2T.v8YNfl)ŸߘZd;df{JkVEJ @ V(\Y[|O*x {*_K̀q7cƌby5bY5f{F]?-'/f1̵ kbB"W̶=_Ś3N @E3&mą/oo5Wqh1>Z%#,:Zo-byxm3F{Y-m~N)YחKw'رcB'y2cgf+ygyrca'K@ة]xr…-Ç˫*7pCn8 yvmc^KJfT+xxG{βe|HUKpтz~3B~E^š9Kn ;|5U,NVlw^4.b'p;;j+MlQ*y;ޢ~˫J-ŪT[:Ǝp K,ѣ"33(ȝ讟uf3G9-*6 s=W>hib_ #(W/7&Nwc`VĴslWz#;9 4nB v2@̇elQ*&7=ǽە٢:ao$ob;]vd 3 WS5KXL)KZ̥ ~TVN,/֥bWl4Sٽ+VZ>iLu42OM;iF D! 섆 $tK@ZhQ>z9l}ɋ/H~!ϿdS1]DcN4\iNc Eymj@E |2c'[ a'[ZjT ! Pc'Ft%ŋ<;Nc'I@?|y饗dʔ) [BDv@V;;ًYUyUظG!O,#cY@qv, -7!q; C2;w XJazb]N8jd< XP܁ `)K[ @@ةWC@<vL/ @#a. P+8;v @$S%8.RG I!섊 @l!cK$@ر{ ߂wLIq'p / SLɸ'@adž( v;vǷ7rN8#=v/NN⋛Ž-Aaǎ8  cNN8!`;,%ci`n1.'Md5v2@̇ev, (@Ž-@  Ə!@ ;p@ @X0(Jvw @U@ةA#@ԅ$TvBIc @@ر% =vo;$и ?x! E|ŽaFIq'ЫW/;w! E|;va''MBaǖH cGvl1c''MdNŽe@@ر4^z@&2N; C2;w XJa @@mvj x8  @@ d,` % Ž;@ * T jBA;1@ [ Iv;vǷSrh܄@ \tE7ߜqO0Ž Q@@ =vo;n7!q_z+`> `Ca"$ [ Iv#^@Ž?f$и @x1@ر,@R;Srh܄@ Pc'|XFaDz K XX܂ @Nm!gz @ s!Xa@ر;x@ P%*q :\M]HB5a'T4@ ` ["Žcac]N8#FO?-7|s=|@;6D %c8@ر;︑sh܄@ oYfɢE2 C6@ر!0eٳGF%5 *[=Lfi+vJ1ΰu/9K|y(i6*NsoԩҲeˠ8&< @@1; ?QG{oݺ_D=ψ(N˂ m5o\:3O1ܢ$z/Gی3{v1?~|Arvts]:"hF @| 7,{7`1Z3vT`1׹#J4n)ܫ\02}pbD^\}ujw[~3Y3y9abꫯ 33bQlf'D{{bUv/A TIaJpU\杩RƊwVc?-xnW^Qe޼yu(?+Ye8x.cxSYPaGwUKv_^{ >ějU1ru 5W7Ž;@ * T ˊ-=+b@ŤJ%Kf E%[&VLq^Y%Ž{رcOtj [eFة2 'LӘz&Mڂ"œϟ`iѢE: ŠT@Iux0 @ );IOߨR;S=-W9*'GZgvf <'> a'ah@ DM8]##x-ZYl2g]۶meԨQbjkUzz3N dʕ5 ֞vbMg@ve@$vnZkml[IhV8q,^oN^w]ٳd{T㳷ŋabta'5h: p8  0ǟT0FVp@BR,3dg+0NȤt%t:R" @$ P50^vυ5?aO1W;r;ٍC DH %D[)%)9@\;يB Da'&t#vYrU *߃a'{@ "xQbh@IPaGg͔t}䫤"~v @H^Ŕ@,_\4icU`TVceX;E7!q^z<r뭷ḟl Q*q.%NNv9 4nB =X{wɸ'@%U>⌰8 NC #v2(̄@N𢔓@&, @ 5eR,E [!_;=C xQJcT #@@I@&2N; C2(YP܁W7k; q  @@𢔿1J|ȅc7N8i @2(YP܁W7k; q  @@𢔿1J|ȅc7N8i @2(YP܁W7k; p œm">@~]v,#eXL|eqp@ /J9C WY\8v#ÑV @,#eXL|eqp@ /J9C WY\8v#ÑV @,#eXL|eqp P<ن('œΝ+ӧOY<ROԇ! =vr;Iq'pqw!^=|@(E|@>b^"$;9 4nB v2@̇exQ, ,nv@Ž1Cd/JYBٻp:hhzt`dx0'rts~3ӑ$G bLFSx4AI,tO53JKrPt@3}{y{y*a{?O^^a }v v?{Rhx> pYA }qq3L`'  'Rx5g*+g;8   g\(yVP|_y\ S#ɀ!   p^͙1 }j7N9  J W7v2 p'PE怀˗/WַԦM,3DPD ~#UK`'@ 4p\믏B{0|A %C0l;ԟ`'B3M q PLFɇCv|"s@k pI %X[﫰OHc'B3M` L % z ݭ5j(&1H|_YFES4  8%Sg5k,qFʇqE 8^@  PJոV*C'ƸzNz5!@@D.Jє/^&NΝ{n5~(T0aB3ge˖A|PM6w-RzﶩSӧGi~K.1b3gNt֭[t'O_bE4G;3 "H7J3ֵ;Ԃ Rkc(Wa/@@p$4LmHLp`3NnÇgvVZtf:`}:g&CT$C'$Ʉ::4ҁUdnݺƘO=^3N3B+=V#4+Ýx )9ns[g2+^>U`jaKgdK 0@wީvܩ/# p$SdP`FfyEN`'-ϲɰ" p$Sд`ٝ7?lkފUfX+ƃ f{L4)'#iM;c챓ܿqFz?7{dcw2ѝ^rVU` UD@p^ %{[B6&80Tb_$4bkT=vLPkӋ챣d@O53+2Nr' +jV vԤ-@@o.֮].re4؄2^mBo~ƍU~ 3T'tqPfז?3*Ѩ4K͞=FNIv=A9/l5{+!ig|U6͝`Ǧj0@@kPҁ… ;'Ẍ́$PK6!`'*3G@@u](}t)@f\u]W;:~_+'Rhzd}>L> Pv>>@V;L>򑏨 6τ#U](:v T}e,M3@N f8.@x> }:wz5$??{j'J%Mf ;֕!  PFc~e˖hsxώMxww74ȌO$ܱ q^@@&*.'zH7"`LGɞK#O   T옩XZtQ,9rHOgڵꢋ.RGn:\;qk!UGq"(~V}P>h .PeCêC;W\LW_}un v21Ys5v Z_ZGrfϞz!uСr@@Oq* ;7t+dQ{jQHv*q(I`$HAR vއGv@ @Z b;Nmtv q*. > Dv+" H& ǎdx&@YA eZ  vq6 @=;8   c;"@S{q]_f  СNp u%)u@;r  /;Ty k`6fǭwi"w߭mۦկ:>>PEZ`u@w}H& tMP?3a Ud E`R_/;Ty ud ;We IDAT7㎝@ 4p\`2|< LT`&ŭwi"{8^@g;  xZX  PL`g##@S3  8&@X.4 ev   ti `{ZWRDS*'! "@K%vocvzH& ͓l٢9>>PEZ`u@w}H& ̙3GYF τ#;>T9 ~KH}&RI恀;~ԑY  ^̏;v)4Dq L`dz2@SO Rhx> xVPx*@ia  @1b~ PN=  cc T'8^;~ח! t(@!!uj]IJN4  /d @ 񻾍q] f8.rJy~@i ?%@SYGtRD >яիW_|i'@ϋ@^~oomNA@"($@cS5  @3@N f8.@x> xVPx*@iaֻ@ 4p\=v/ G3 t@Ov<-,B@(&@S̏@ ةǙ^@@ q`  ߋ`2;@@v:4N=W+I")@@| v^;~׷1;n L}Mm޼0|A`LJ*2;~ۘ@ 4@@4(iEp*`b`['ر@\ qR[`'Rh  FIC  TkSzgS5 ٳGm߾]͟?2e VSNUӧO3ˏnw֬YjƍjԨQYNwL~/^<-[ Q_ɓN85\K @.\\ $@#O @:83g>|XS vʬiW1.D;@v\cDTdůr e#Y:<Ǖ1YOOO(^fcAKrdX" /0 vJ~M<]y啩Ay›?hr `X`04]h Ub #BmZFq^9;J{+ Y-s'Q`!R&9dh_z\s&O] 빝]k#`a\ W_sxh v:Djh։SWr};uc[yivWSZՓ%iXwG$>:t@e;0 @;%balh{yP%yr;vv⏔MG`;v퍓(m=sr qnM`'s] fN 4{4?ϺǎyiҤIbI;W]uU1ݣX͖B(kccYݣ. rz2x< L L`dz2pWY 9kӂۜvewJ;9J{T=vo`{s+ڽe6w{sցe+@om Oլ`.;wT3f̨uD@@Ee9=])LmۢWGvc!킝xc_G:o?& 8qbs݌} J:_oJo7^s$ ;vօQ!@~o;.UƱꋱnMYFW:G@W @Q+ @cC!e(Y2й馛PGt5 kX @cA"@S X[r;ri 0RWȏ5ʽI0b(U`TNCAA&ةن.} tlc@@vl cBN69{jQHN4 |P?[:l@vʳ%@ve\}}6Ef`^//w߭; /@#_F ^`G[^}ѣG;vW\p >f-6\Pw??;Y\gٗ5vߟTōnV1NF@ E`'eZpZv4hKɓ_GA\k~Ԍ3876RswR; 4nz駢#G֭[:qĖor%* !@#.g<ѯ1׏cx㍼\.t  ל9s.[, l,^_;Y[u%a@  `@2n! v2Sq ,K9 "1 uD@N'c'Ds@ 2m`ڔ:,JNc Ё>9i&e˖@:li ةXIA&ੱ0t}nfu]wǏ#  P@_Kt'13~y|@'!@F%  @y;YZR'bp eeRx)*/_ ;;ԪH'#@MS4 @a+"jcǎۢ@\@@ v,;F ر, @@f@ )~ @@*` Pi*`hh`NJ20@@@ ةՖV vlDֻiJЛ'K_R۶m+=A@P`U@w}H& |^| g@@{E=b$;UZ&EaH O`E  @>|^Ml8A Un&lt< Wl>n Lc2|u" X^   `}5aD @;Vy# t,@1'"Op%Nt  {QGf@(w v/C@@\`@w}ֻ@ 4p\`jƍ_3a  =;ԢTja|-, CB~'Ԓ%KToo/:  % -m`”=,Ei ةB6@@| 񹺱Rh;#'TlX.@cyޕ%I; P{TK P;/S@\@@ v,;F ر, @@f@ )~ @@*` Pi*`hh`NJ20@@@ ةՖV vlDֻiJՆ ׿Rڣ@@v^;~׷1;>ȁi"ު>O^g@@{E=b$;UZ&EaH O`E  @>|^Ml8A Un&lt< Wl>n Lc2|u" X^   `}5aD @;Vy# t,@1'"Op%Nt  {QGf@(w v/C@@\`@w}ֻ@ 4p\ _Z~7L>  `=b$;UZ&d @ ŋUoo/:  % -m`”=,Ei ةB6@@| 񹺱Rh;#'TlX.@cyޕ%I; P{TK P;/S@\@@ v,;F ر, @@f@ )~ @@*` Pi*`hh`NJ20@@@ ةՖV vlDֻiJЛ'[Nm߾h@@@)Wm̎r f8.ݭ-Zz{{ G@ԢTja;!!@?  vy9{4c%@T,N yc W;^d.B3M` HםTlX.@cy  };Մ! Zy  бNt!!@? nM  P/ fv,-L")[@ *Ti@@gN f8.@x> yr@f ;q]YU Ne 25i @N=E@R`'Ȳ3i@J+ @@l ر: < @@ة&@2\֊ v(@@@@j\mi`ǖJT<n@x@}W;w,=A@P`U@w}H& ,ZHuww^g@@{E=b$;UZ&EaH O`E  @>|^Ml8A Un&lt< Wl>n Lc2|u" X^   `}5aD @;Vy# t,@1'"Op%Nt  {QGf@(w v/C@@\`@w}ֻ@ 4p\͓/ G@+v,Ki") ]F'/^,Xz{{!A@(I_ii3;a-J{ PN  Ս͍`'B3M q <9b3U@r T,IA*cJ]F2xy PD`" )@dٙ4 `eaP  6 \ƆI{M]_f  P{TJ P{VFkE;VA    PN5JcK%*U L Pu߯vYJ{4  ߫`6f9B3MUoo3a  =ߞZT1*T-l`¢0$'@â@@@ N>/g&qt v*7Ei6Ov| J`ǫr6 Rhx> T@v,/C@O`Ǿ0"@ TP+ϼ@@: 阎@@@'8k`FlB@C=v#@ \;~ח!  .@ 񻾍q] f8.Oܹ0|@@{jQHvPM>!!@?%KO|@@EI6CciaN٢UTJ  > \v)4Dq H͓*6SE, ر@e [ʒR=vԥm(SםI[ Evq.  @;AI#V XY  `al  ev5ٳG͚5KmܸQ1B͙3G >\͟?(֭S?ξ}Ԕ)SԊ+;+y22:?qDdjժݻw:Vc}wwwkF5J>|XM:UM>=O~@H k\`U;8>RNe0J`uRduΓ@@ ;~׷1;n;ugPh vʩayr  {E] 7/0aBEy3 e $ߊń===Wg_ym^7V,۫ovV]Id `?rO_yi$C0N”Vu־ɍX.J@@@dA-k;v,+HUֻ6Egăӂ@oxUJi O9MCܹs&Ly(+V*?v)yLCU[1wΘ1fG%-IK&~޼y1cZgǭުZicIey={ҩϻ>Csڹs`  U;VNjg|KQ}W8V˗/W{TWWWaOoVۢ@@@w}#797Cf䊀? vR3fP݅W8@@l ر`bO ~&0`zW <;,0pE͓]D .B3MLc.ǎ@:uq @;$ @2)# ɏ";n֍Q#> XU(,!ੱt NN P@'8 9p*E*cm  \?vvGM  @Q)* P{֩]_;ӣ֬Y#7"e-'iݹՌcԯvmW#'\PG'-   '񻾍q] fz+,\0zȁi" :fh;EK@Z_]~KH}&`USf3jƔ]@   N 玝@ 4;zUqnl4[͓?@f; n Li蘡.@t^w^:) "t(@!!8r:묳2sd@X`04@ 0 t@i ?!@S"m @];P+ϼO`Ǿ0"@@wc9N..F v*ĥiM͓k#@6;,n Lv(CD- $@S4  a% ;>T9 ~QGf8#@L( ay K  Z`uPN숗 Ɂš `'[[ чѱԭwQŪ8v?PWsf5W 2(ɑ#T|-u^zLǝz)[}B_ȼSO3Tw:&OJS4&@S% ! >]# XZUL`1eZ{m꿾}&w/zW2w9 v z]7Z!K[[ϞR?Fw;vwTFN%씀H .@#^91D/%L")`SY}E U݄^vo[y.}aש۫qW_>}>ṙ`Ϣ;6W!@V*q `Ԟ6̽1ԩM_ I7m_[]:xx AʩgTLUTJ  @szVN="uMNG[^ر-2Ǎ|;Nm6P6?Qv5r^{ut ?o{ԛA )C6@@;٭ISDs H}m vm˯yz_<7p(oRQo/7-Q5Y<[~imV`:2D3W;"vX  @Rw\7k̳g^wԸ1oW:Cݷ+^P_Z= "@òT@;G*UJ`G|y}V)!@#N P;/[@vܯ!3hX]wK( [z&W;^ H# pN9i<5u,-Xvڥȑ#+\wǎTowwZpaARxbs'xnC Qcǎu|& vX   @Rwqǎ| =SPGܬN ae  (@cU-E1ʁš8S*-<  aMT* u]|ao޼Y 8Pvi՗8 5f_<h@^wC PNf@|aoܸQwyjᖌas9jРAv!ЁNh XW 숗T! m"@9;8  ;z#(?Zbv¶!N #{xRH@ R{ۘ/Nh+J-욧Iw C`'" %씀 _ [5c*@ke *@SOe vꩯx/Rd-@@W@R7_2Ђ?Z\CׯW|T )~J=Uׄru##8+w;  :IZ,`G^Sz@@@ {.@S KSZ Hz'~GNP.FRi@V*q `;: B_i!uˣXuV96N=`gz!0v¬;F7v.tR ' v: DF`ǛR2zC ``Gjy/emLJ;SGUǎSGU{ߓfvخ~shڇrw*]tQv8(*pwGMp E|@r|VW_}u}TO$N?e#G|hR׿'s/p  6̙n }{N{{ՀaQ===;AT=IJzG;& ~׼(V dzKr{uo^.k׮%)wXۚY[*_;{eeR  `/HDC v;Xb;I~   `'ۼ>;NtB`щN'j tMѐ/_n [D?P;Y+q;nMz;('N9sÇ={Yf7QFߡ-۷OnS j;J4a5zT+ڽ{?~|׺)Sy橩S@Eu(bI~ v v$`&g@sRdjٲe}g~ӟI&EX;>M֯_9;} @;^eq?kynjQ[zyFK!~NAw#Z`GRݾ v]*Yع⋣'$]-f~0kӦMg=ݲeK#x?.w`nj1~N2Lw7+c{"ةߜ@s'8:q3EC v@ {Ws'?St:O}O~OG%cޮV-E}}7Y+{5/Pg1DlztT=JutXulh~h%@cFQ7{d+o|m7ع {eB}cwYd  IDAT[D&X1w~+mZMiwz+Vq9?WS9="@R{v>bgJ^`X8>wtkwǣM<>jcN 󯛬^>f;}m=UI>u: vY`YGLx;e߱ T)>f e vJ tE  E``':>;ѣտGDU_]~;V/`՗gO{:|>$رXR ~7&x;ߩMgy;c'*=iӦEo*{qs|v{Ǭj Ғ{{M0۹Vwb4 mRrQ,/=5ݟU_h6Uơj'>9ձM>S&رgi93JV"z:\?rd<7ɲQpWWۯ~RƫO/XzKC QcƌQ@+G&сާ&':m!|r78s;j(@4PbQ& 0wu5{_|R}k{~~~GLck?8Z"vոםzK9ƍSm 3 v*@ f$IKR5e]vEw闆:!T@DC v/0=+ 0~EfɗO><??D?R|mCG79Զܭ.Z j::`:6`Z&K' TNLi׏-[LW=bx0XL40΀+R(!aQ!KROpH^H~@YTe]U9V;KSx}6x:,*Ld;AI#`@`NJ0P``Eu H} vpVxRh3%;??' PUxagg\|NFEc PdE`'xP@`uաqxT5GE;֬YrpfߝZ;Fjv5re'#+t Q‹vlc,{kȊ;=}K> ? @Vb5"RA !)wEYҚԭw(heyk<3N R % F@Z蘎")\6@r ^(Vzq(V`uI9ʸP"Ρ ;2QK5Q# Pu(N=ED?P; #XT\|ނRvX %v:摫fJRe H=Q<'zJL犀`'2\\\@ x􅍾k]c:7'#ER{J^H~@*+`\(T($puYgenT t#GV^%_];vSRo0O@;Ky!{W#F"J %"+b N`` ؿ8x6B.|(A_-3C 4_K9^(T7`gS.vؑu?QOzԳ͸q5#Y T|_ @vxåci;)kwYсo,kJ@PLL<ΰ_]vEǎejRِܹ326O2D;6i#k\ʕJ1N@ סuJl'R>P;K|ꩧF`\(7C+;(@6\[+``'f] V|_;' Gy3ג ס;j ڼy8p:,ghgq3f=b$ p]I }mi H*mu(Tt>P7nTw>|xs9G 4ǩ1'KP  }Ֆ@6Ovd2VRԙ @J8ʎ:0 (.@CS|т`G P$ }@_~/u(oŒZi5KSOB[sY'Prl   L< YЃ22 lBɖJ0h'U;!S``ΕYpTRޅ*X&NG .(3D (#@BQySU-}!JVq#W՜#]nEh(k H %P@W,E@ )ЮC vVZ LB\(dQ *sIhRI-l~YNLB\(dQ (Q@zP_MZR@J8ʎ:0 @ T;yWZR`B)3m1dHzCR׃RH@ZRRV@!. q2(Ut HZZ4.u=(/J _-o eeB'#@|_ՈMW PN=/ӑ%ةcORI-l~8#A^. #C|_"@_r K,RȘ<Pa .x $ A~c'qzZ7WF^.7\8 X%ؑ\m5MCSr+Z pAWrR+ HK'@;-o ˉi"PH B|5 }U#6]!% H]JKS⡩R [_BG |_QF`' \R [\8@\(Zx|_9X4ROpHCzP_HR [@4($R!>NF@R=W+Tƥ%ؑZi+ LB\(dQ * ةe:Rס;~|i\;-կ=g$+ad W+V"@CZzRdEߠG J8 /@~_s7#u=(/w^"n@SOB`+R }չg"RT;ƾ vvj\nt@K.X  WTq"}~ vHz' d91M pT@FjĦ+@DA~ vJ\<4_@jaK@\(7C+;(@R׃R]!K@jaK T @ ϴpP+Ɛ@ U@ rH]JK#WjaKHY&P* PW5bT* jjѸT;R+-~Ti"PH B|5 }U#6]!@;LG:`ҏ=Kz'{P6  }Ŋ@_vv|YAC,H"cdB)#!W%` P/K~f厝KvIh\zPכ@s:L@@R@*``Gr7Nˍh) \ʕJ1N@T"/N [T,'@!. q2(Ut( u=(/N H-l~Y ^ Fv}eG W@zP_+s H-l~sp0 ph6 }`2 H=!UA~ vVZ J-l~)+DJ8jFlBJ\tR-`Gjү–72M pT@FjĦ+T`H]Tq[TT `Jֆ!@_X /k9yH}~^dL\(e0J JEI݌Tܱ{yN= Q#PJzt.UvH H,RH&!ةq-Pb +|_R)Ɖ X%  u–74($R!>NF@%)qT-/k pވ#@@ H]JKwp|.-o.F P.-4.u–מ3B02++| !e-=T"oЋ#Q P|_ @IR/i;vr/7O ة'usu0jB^ozC܎3@IE_Vc;;5.7BJ,pE+W*8@RT;|nZR&P* PW5b PT;%./ e @{.q!u` @^A~ v% ͅ*Rg8(Ec * T9%ؑZi+ LB\(۳gھ}?~![Nn.ÇԩSӣ,K7QFkJuϡCʊs JswԄT;-N* q,R8qB͙3G >`F;/,:LK`H]yl4RI-l~++ #J;u/gt}թ!m;;IӁY*`귃p  xdh֭[fΜ-[WߑjժFwޭƏ̝x3z'x,X}~dK?Zxb5qD5wo~9մi>`߾}jʔ)nSsO4~hѢ~w>,XИ%in' DZ~(H1tݻ?x`c.m!VC|yqm[U`ScߤIUrZH ?ɀ+x];xeuIӘ8p Z.1af{@1>%׼{5kVH+)KE X%ةsu EC#>>^(%s'IfN`'4Gdb ?OVXmv̈#!n<0 zt0?oӦM3N ~ΙxQme7lذ(j]CEu%Ӷ ֲgy&uY6ON\9^4deצW2@JE_*WEmKz' k|PJovQ6aHNf!C`'<4 |;8\PߙaLC`gɒ%[n$3n֮>/nÍ,ws]*d1>>OӪvW^yeXQ-i8`.e֟1kkA~ v[^HjaKU  x*i"{c]UOPZ0K('we;y!Nfs=jgW$iW`YZ$yӶog力z:j龜 8 zC NzP_HR [@4($RG]<;~.[=%iO~4)^(5 T>oM2dHߗRTFL % u%\ c" ~)?A~cGj/N= me;Bɾpm#rvޠeQ}eS5  ]@*``'pH0J %r8`f#rvf1ULo @YRT;eۑNjaKk2`xX!e` A HX( u=(/Ч!I-l~}sA*.]([﫲EiG@zP_zUH-l~-4L IDATG J98Ds(Q@ )A~ vr-+ (RUg)uc _@jUZH]JK#WjaKHY&P* PW5bT*@StC v*ӸԭwR [_{*HW %{k#=2󶤙3ge˖AIŋՂ ԢEskϞ=jݺu9U='N9s樞:t萚2e4iR皨}aQ Y*`ׯUlF j\}hQNڜ&LLгj*{n5~"AUPfx- ~)TA~cGj/N= me;Bɲ2"(3)2v6 qm'w27Y@*``N Ki: RE209nĉѣfÇMqeՌ}G?si/4 vZ%X6mZ;H}+ͣӹۨ/^7QFe 8.?G~ v^y.u–7OM8PPiѣU?-ǙӲw~ms9j۶m}>`NL 79vic0s۲eK$H vڙ\t v|ѩx`pL"K8XVkUv+Dl`ǦXT)!PJl0H1wa`1D߱s^<|0{~.&ٴiS#81!N|ɀŜ4muXQ<؉v'KLI,f,{?~qGR4V,)~~ѽrfR׃R+ZRVIx"RLީn]!i Bx voJ*OfJ%yᇣM~6 .3ѣtv+j`'[Z E̿2'?Ne+ iUQr4 ^``*>RT/u_(@g…GuwwG=V{ȘnjZSvfoxI _ӷb%iZ%Y<1~;'(/o*]U5@5_K@zP_ؑZi5KSOB[sY'P:NSOk(Iۯeر~7;]w1cFv-*]5>Yf=$I@iw´3:uJ*؉LiV5sύmFM@p 9QB HLAcW#D@b);b0N؊ (!PBWCI "ea7@jTaK-(!JCxf[Y@JEؙNxʓTaKB3M c@Gão@KџK.Mω'&yƮ ӿd괁 П|P.NR-eb H;^@3s)|3׃zCFEC@``R;8û;]Ҡ, [,x@)- <T/~>{㍊\8 RvvK*l)0Fa/ Pvf agj< Aa'e:RP!ޒZz'URv(\8 0  hGf' "K ,RvgOB;H!@/JeŎTF VA I;I !@@2);ɔf?GvvUOC`8;ñ'@@w91}N ,C"eaAjTaK- @$&:u*p ]jTR-ewTtL dH€@v /‡& 5u9TaKM?cD  vg@`8R;8[ORA); Hݎxh" v@v8c'& uT]J+ĮTaK-$ ^vza@ ЙNt;_4Zz'URvӬ@X;ayc   d@@EXfP* 쌎 @`_KAj>(e;R.N6pZ1$ $6 @X"$SEAWA<  ǒ 9,H ,Rvv*"Zz'URv3)€vFK@ \w4f @@j>(eag"R-eן -!P.rsOȉNN$@ 7RA);UpdH$eZp Ha#0CvpH-5#Uiؕ*l)0!ЋN/|< DB3v"In@^\rnFRA);#]A@{@N;#Z@ ! 섹LGj3q_RK [n%kN4@ !SHԋ,%Hͻ@#@ _RovRA)"a'Biq Q@؉*8@# %HEɮ! Rꄙ@+OnED@b);bְ;–6X@v^C p9@ ^RA);bI,FȀBa'f@|P.LeC [ʮ/Ad;%g!|rI$()RA);RV]–[HZ  @$8c'D E@U/Fh$53B ˗՞={ÇNܹSw}:qڿڻwrzGQVr㦛nR۷oOTt:xZbE>K9*+Mwe%6aV}qc{N97s;?ӿT"v_} k7Ca(_$yԶmٳgn9r$I6  ;a.];C?wRk7s^-j߾}J];/wʬNT.%{!uy%~ %@A [qXԋ+ ]|ŊkF[)}ۄI]+(tSJ5׾y똶vƤK' #w>כ+)Q-v mt{uk+V*O=ڴi5tqFe3ۺ/ѿ[/MVl;ׯWGU>sM'{UW]ܶo?:vt{[43ɴg>vC} 1?~vj5kˮy+WN$9n˶:{W{9b?ϮS7޾K}򓟜>ߴ ϻ{Y7|wk=§$(4 ]PĠ[-k׮UuL~F1nu T̓OT>ƸZͥʎѴjԤ< xX"xEvv֭[7ݢU4[qM6lEcǎbƞl;v\L]/\ۧc}z嗗Ξe>-VS~]"c=6K0/>uk׮ںi>qO_~g ;uE^' uvr5.ԍ1c|]ƒ:>easAmyu.qԧf|Dg7mgW>1vKv8<9DK@J`o-/LjTaڝuŎxeޞ&KoQp_&]*Ə/[7ɒ=ٰn&VH֭^Ҍ>)0f3\\?X]ۯ~nkܹsm\|5쪓_y8܎G?~~NWg'\YPm[lQޝj|Yԯ+O-|؜V~TyGWG} q>Ը +jda M@|C'eagL2Rk7agلqKKةLp A ?~|b;U~oV*ٓݺ׫k=YrB .V|bVS}$츞o;cMo[>yW_=) }k!mNU4 ;HUEsuBaWZ6{jYmto.E&q$Ku wN:ŷ?@|ph" ICRvKةPe5{̸՞("ִm%~Ӈk6m .VۿwZ͓of+Kƪ6\եKxjLlRkG{=iC ;UWwIةgYH>Ba9xpС+3_]۱|뫋 X3VHVW_aڅ;_}!4H#%HEؑBJYWt=]*yvmYJhcW`xgWgT+ޭ.Ž︊3/'A:suܨ{ퟔ]3IH]iaǶoOةM_w{KV,Wβbm9.au.Qۙ(pgXڶ~&p.[gn ngϷNvGQ+~G1Van+ϰߔc|6hUرۛCO%U5p|˗y1AWU|vl&؛hVkw\ѷ&VRZ   Oa}Ф}CEhI-*l_1 ;fbKi 5Mþ'qw9N>|{2<+$uL|5 4]Gnv3tMםv}nŚu+]}8Զ}MW|QwݹZw/txT3}coiq*V}y qHh{ԭ]VtSv mHaL/_է?iua&REmuv@dɓ]V'OmkB٧.]n_/3ش[-mכf{ 7@棩N\ï_R?u.{&m]k|nybPvYxT7՗WYݏKP̡bmc\5b4-ŽhM@M@J`„!tv4YT}LZϳyh)sO~S?WcLr#ʸZɹuD@J`S6"N}G5]SkCT<2 H+/2N}\]3K<3777qԩS!;]3IH\Wv"[O@@'I qaG)<= ;:Z);CgvkufPW7D0%v@ء, N qaaiD@j)E.ŽTbWV„@/;0  ؉$x:s˹I"PDt–K!v;h@$LgHƺ/y(Й?wR-e7"U:p  P("K ,Rv$"@; @B@]||P.+vB F\h D  @YX"dYWSHfvH.B8< X"숕ZXRK [nجb i@I3ox ,'0777S@@JE؉ssG?vƠJ@h;c?]ڠ  [ x@). , dV@vpHJEؑBJBJEa>"!;$7 /Rgz97B#]. URv= Na- @ syQs tR?35` ÿk^?cJp ;>_Yʿ:?CͫoRj͚0HCvdOH-*l)IǬN%!/ !@6|!ο=un٬zM7^7xo*wLھPb IDAT-U~'[[ /$~ w'3]>o^[.O[o0Kw Se4Hfo+w _7mVck Em,{Nv2f])nuI ,v/|]m/Zؚ!_sgo۟sOW׬@2u{*ƨR_[~jj?(]m8џ8 2 SYzIwyPLsP#T޲h1fB |<)| );)qaa' ;;4J$a/ÏQBɈ"ww__u_.BA骙$ў3vv4!Ek/Z;;Q{c}ekJo^qXu vqQRYo0L@gӛJb>֣9FrH.'P}Ta _H^Rh E ɹ]}-AڛzMO>3vSZ@ 4/a'S؃ #ag$tC 1\w.vIWvpX_䘔]J; &Ce@9:c'@ȂԙR);RV]–[HZ   @3ʙ = 5EO򭤖I NȘ @^ HR J 0:c ,# ~4HbGE N+ $$ӆ @ R]dJ;;*!0X GÓc@)E.N[EdRK [n&BΨxD  H"P$(ea' ,–~܁@vL NA tFs@@Rvv*R-e&z@酏!HpN$ @ԙ^΍Hj>(eag"7H]r@igD @ 0$0HCv|["KjTaKٍp @؉&8@@!vv )Ôz)yWA`;p@ K@ Nj>(e;CWP!Qh#M?nA *;Qg @@v);ٕ; BJ0 @ph%ɭh@@"eaGZz'URvfkHNyk@`9;" K@j>(ea'Z3–E#@0CA @3JEؙLxȗTaKB;La;!O.%!Vj>(eaG +URv I+aB^xgD܀HRvvF("|TaK%@;vF @C@ s<agȷ%⾤I݈K  hR# BaaR;LYJ`w!3 Gz @~RvY3tEN6"U:p dG@J`] BA) 3; $ !V܊H ,RvvJ-awR-e7lV4 줙7s*@Rvv,<*l)Y$ 02= N 0]ʄ| H]_.@vJ>C ;H PRlRvv*R-e&z@酏!HpN$ @ԙ^΍Hj>(eag"7H]r@igD @ 0$0HCv|["KjTaKٍp @؉&8@@!vv )Ôz)yWA`;p@ K@ Nj>(e;CWP!Qh#M?nA *;Qg @@v);ٕ; BJ0 @ph%ɭh@@"eaGZz'URvfkHNyk@`9;" K@j>(ea'Z3–E#@0CA @3JEؙLxȗTaKB;La;!O.%!Vj>(eaG +URv I+aB^jϞ=xڻwzꩧݻѣGڵk({9SZbEkӦM띬utq_k۶mSsھ}{mF3vfS 3WeUJj>(eaG +URv I+aB^a8p@^Žnř_~Ymݺu"#GyDرC9rd" D  v\#5E JZz'URveUEa'|uasʼn sm~I5Vov sX IϟgN¥= d^e'"K ,Rv˨&@?; ><1W۷o͛B,Numڟ1:JgNy03#8qbfګk WW|N~g9"?ق?zՃ>8Z$91Ѕ v~Q@b~/Ij>(e;R.N6pZ1$ ě6#zYf9-F":.g mE+W\_c1ۑlqF7zk~NFڹs$W_}uzfPgw+V?;7x@X"Wx3Ja)f 3@؂dnIx㍵IfŊ.v>+vnᆉSCBW?>]=cl?cM'{c|WT4ݯ;fUgya8<9 g@"eag*IwR-e7e'A_1BwXeco2m ;֭QceD}[w\cg>+"#J.||%6V5݊U.]ry/|< @`TRA);K]2@ig$Ѣ%n|sγ>;ҼϪ[ ^zj2j]vM-To2"vq\"S=栗w좶3vgv¼A;a8c,Rvvf& URv@؉3>N>V, #i[ni=cg;U'cZ@v\[\•v| NXXٟJEؙVx҃TaK@BOa';ƈz׻ԯگMno?sرgYS%ⳅJcǎmUCcni; Yjvڄ?vM6]q;X㌝0 C@aދ|P.N [n44@vMV,{\ #fRWaj\s`pӭXߦ[Y.Ug@d#hqf sx< ;0HCvbOjTaK(]t ĝ޺C !o]-,,{犭S6-{Mf>zUe&e >9}^o2*Wϵтw WiPhסʰoa'W DMaa'9?R/"e/@v荈}Nc_nߨOl#}NWk֬ gK" ~ХRvY#Ui"QhsHNijb\d0:h؜>}zlqqg&<@ ~R]krvv)$:vHWp]N@1Rvv)m@ [ʮ eB -;i o!:aga' O@jY]J+ĮTaK-$ ^v3f1 P!?U{qy 03W}>Rvvf& URvZ?C%t@+^xܝyA]';7kb a3:F 2y(/M]J-*l)1 = Ci+-,--MBbB m ]cԋ,%H JL@ yCL}ǔN%AF%Nbh ~HbGE N+ $!&J;I! Nv!ƭW H ,RvvƯ(, DQ8ڀC)A}rUB,y<)E.Ž|@jTaK L@ qLO C a^5&N6CEJE)^O HsHHK5\>7*B|P.NT嗟3R-e7 'DixCj ' #|[JEؑBJBJER/|< $x6 Q H:jP K"HUZ!v [n!i%L"D>` Sv\#5EswR-e7 %D) Baa'Z.:YJ`[t<< 0QE3@@x qŋjĉ̙3jƍ;{Nm۶mѣGڵk7TGPSW{#<ev[n\ٱcy%;Eav/5ˊJ la'B8@(%6@H!3F;uW'ϰ|>yVZ9- %HEɢlۃ@AiZ@ &Ja8cO?zC|3.v.a%\|YٳG>|U;Xj!%HEɪ|냑Zz'URv )'„@/Lza@ ƫfض`bZܹSC 1W E@‽b%U'ﶰSʞڵ agu*TMNV#%㓭CvU!駟nOsR'^M۪5DZ]%S忉spm ;MMjg,5LJ:Vӥ*Paz.QU[y5u.Ž}tK0sIz3vN0)Nt;ü'"NF_80Q x`4iT=Զ*6ɺnkn)j ;+Fʞ#<3Wl|V6hbaBn;o:-VڧSʪ Ž{aagJQR/"eW4@"(%(܄U}JUϱWjRؙa736W&_;6+aGe:z,+v8c}Eg㴐Je8u];aCKU}a];0\{LC ;.O@ꫯ^qNC݊Epv+ύWMl[׫\f77r+c"eaGA) 3LH.BWP=zrUpЫD֬Yl ҡC&g=cɓӳhlqi5G݁푿ޢϊ}>/z%ݻ'gjG W誘bGW=kI矿"vZUW}uU޾[ڥM@J`vz{/Nz'(OHU 㶣77J9VZ&UoN>OP> H"t@@v@CSKR'p$G*)fm=klc!7+ch3ͱҿ?}ZXXP<AK"t,w# URvѡ5$D̼5R$xw3v}wf֭['[6X.TsUW]b|P.ΐC_W*l) NR;#Z@q`#x aW:sU*wRA);RV]–[HZ 0Qꅏ! @U+ 섹LGjS"NRN ^(@@Wac vfxvvx2 "K ,Rv3(B(@ POşBk1=,5ˊoD_v(#! !&JWcCH@` CWG:uk}B O=z3ϫ >޹@ϫK/v: 顁/b!0D a'l&xw~1;cGl :KKKЇvWN@3vvxA  !x ĒM@/A f.a%vWN!–a p Nti!dI*˴ ` ;MΐN٤uTj>(e3vxuR즛)<@8L± ЏU?~< Nv:fU]WH"–끄&(_@Qvvz+%H KkH_<^CDW%f!&+4$X&0x'7+oT4 B@) IDAT0^! !sx2'3l@_<|(U Y@Žϛx;GIq?#|( 0^e`ƒ@Fvv2zBAA) 1|H$Q (+HN*ok?vvzB`P|'A#`.]C`BA3A؉2D|(1 4 0^7@vvJ|,u;@X1 #<"xUT I`BAIy)En;A/q/ vWhA aa'w1K/)Y& 00x 0^! 0^! R)h'vF xGh'x$߂Ó9<9"&l#T'x} `BAuv (sBL_<InB"R!xOã_<IgFKvv} aa'ī ţĬ3$xf%`BA)ʜ!D[@ |BAIuMQ۩즗!<@x|lfS@xW;;߻b,J ,RvI,Bxǣ@PWAqc A aa ģ) N/hA*< N aa=Ō)3b1E/E`!4ƫӇ(N<'sxreNG"M@Ó@ |BAIuQ;Gؙzxǰ< xlΰT! DY8U$xv@LNCHW;;;;9!&B/$ 7!bQ@2~:3{$%ay 0ƫ؆jqqQ}#Qk׮ a80'W;;y"u;H@_'Nz>sݻn/naҏ_UD=ڶm:{촯#GLlP.\Xа5NtelX۷\M.Mvv|wZJ`DRp!CQ hF,gj/z ; HOUVMf 0UZTٷoDԱE'#$iɈ:Z4҂m*b8+vfal V"gS |BAI0g t8q%W=8yd*d^Kf\Jam0}Lf+ 7캄W+ZaV!WlaUw֭['o%z,+v;{1?7gk aagw,RSI%@;x,go.ko#^AWɶ-H0sk\"C9lcCѷbעsMooj:cg[@צ9cG MUHؾzc3vzC aag@@J`K!v|@ia[ 츯x2+v{JzӇ jWw[YƧym 6P uUˆIѣj aGoz[ޢc_ĢEӬygkkwc?>ONoeN.WX1sU< +.UR]R=q䏙`T'oQJ ^o~顚挂˾'wuo|G4ruש;v,\i[͒|SR_җ&Eip*iG=^%5E؉*O&*\@09~'LVoݍQK9%.aG_Ek -{~՞&8k>zomLjobfeP]bDauk+]-|XW:66FfvT ;ZPjͻy9 e 5^ق^=\Oaǟ-5^2RA);jK'sxhbЫt ]궭ە+W^q@aPh4ݬb?㱯5|6f lZ_{c B R'+Hą*?rFgut/>[ ٭^z$ܴ_Szۇ+vXu0k߱'J:!Gj[ֶ I`jL[J`3FE'ם#DXT(x&uuS=oNdv|`>uOuOۡvWσ0ac&|3Q{{{kV~gD3-9`pf Afۘy;.NSl2tz̵ʵi+fbx.܈6,XN$lH`jDGZJ`3j99N<Ո'GcbI CiZy}Ц>pӞW'1f-~:Z=ƜZ3ӕq՟;銭IWƮӖ;LTg}'6Үq_MbzzgJ"؄@R*%HE^N!N GcV%ԥ;]V ;mV}N&f׶'wC ;]i[TSK/4"v…Vv}ܹ6f.qq;Zر쳛bƆ=5Sj2Dvf ٸ'A,A`t9~U79{N_дGi:Eoͩ;;رcm';V,J:qݹY0}^`}qN3vژyƎ>{ɮ g޽˶0fj9tu&n?|N>}aɴj=^uދbϮS{xo*wLEqlx*rR-e7t 0D)/u!׹5U>lcCb!v IZ23[bc4I}ŽMʾy [ND;sXϊ:^dD&v}UW(uK{;bo3?BmǾA+ArbOy嗕SbGhqK? QFuPO;?xURvي7*l)p 0 !&J;`a wU?D1[hvvڥ:c a COov'zň9׵>ciVn)j;<<+Y;sq|WŊj>RV.'7ε̇ΛcTo!35aVJɏn:zgN{1կ~Usvo| &0x-bgJ"̖/$ URv= EG|鯻;>OJ ܥ^y  JW}:g}bJES-Š^o/5Eؙ-_=\pƜ(e mRFF})Μ:&!+_3@r;;|v|(]ag6nGO@JaGnKNj}';; DipȘU%4dLaagrJ?pN~UMD`j`*/D \ HxJRV]–[HZ 0Qꅏ!  H" Xt/^T۷oW'NPgΜQ7nPGknR?(eagꉸ/ò [n%kT ݲ}*L{N޶*=9ېǬ0v3{nOnݪN]̤W{ꫯ.JuɊY~[lmv U7o^jդoHԎ;jm^|YٳG>|Ǝo\e}I[;wݻ'moŪ֋*hqm-[LKl "%@)JJEؑv[ PKRsqDxu"vp`[B6/su>~=w ;uv]$ڂLFةcd E.öojʉi-4EM~8 CM2;2*x"V۱ ?)E.ŽT+"Ki$D9m BÞ@ɺU9#?^m۶m"57+WNWg?>]b&{lҮjSGhVu9IرE{uݺuӕ=.>7nm UqL;:Oe3ˊV< : P]Z0F–+H@haG :Z9wZ\\{̟Uᶯf+SӍIC ;.V[\H!o(lcsn״=i+ck]vVfYcv7s++vb2 |P./¨ [0%؂^TAD|{e. wz[k.ag+cnv0]KlQmk)ٮKOnv*-NUCU)F䨊:&N]xU9^ oI۽{4:SZ"j׮][-[f-Liƫܸ iRA);ȾaYR-e7P:1 =QJYЉ%%v˺Fm, [ 4؆ |~Sǫ# =]J lWz;–8@ƚ(! Wc ;!uqP4vr8F*5^IƄm@LRA)RA);Rخԋ,URvsH^xazNj[bM Aypر6?s UVZʽzJe;hoptDG@j>(Bj>(eaGەz [nbIjJy衇&g $Y 8 5^@vO9B [RA)RA);RخԞJ–8@N~qDy[ޢM8<|N>}aɴj=ދbϮS{xu睑 0"!5v-5#Uiؕ*l)0!Ћ@_aGONj_~YYfbGuowDCmw~;yaרJE _cEY*l)E%`!0#!w1c~J̕31@ !+6[&S@Rvv.*l)'0DIYZZBI<Qc*@ )]L 9 [n,1ss sJCH|P.N:Sò [n$0 !bSH1&F&b9 0! 5/5#UiJ]o'URvsH@ȉO%\Po[RGWl3h@(RA)RA);Rخԋ,URvsH@Ha2>GWJџK.M؟_g/bmwW†qxu@cORA);>UAYfP* H;@ {ΝSSނ1NeB|P |P.ŽT+RN+ $$ӆ(Kع;Ֆ-[&|>;>hA@j>8F,>}J"Tmf& URvgŃ(NA&TdD%t a+1C|P.luS [ʮ'Ah;E!,dS@Rvv ,!KݐlT 줚9 @iJEI.R&(@@;ac  H@j>(ea":,KRN ^vza@ "]oŊu\ # 5,5#UiJ]o'URvsHNii@A@Z\\TKKK IDAT 5]J lWE*l)ӊ9$Ia'ɴ4 C @ J!JEؑv^d–8@vLNC; 02]J lWjOTaK VA I;I !W$@WWRA);]w" URv;1 %Sh  " 5J5#URvK3qC .h @ Е|P.N }'R-eCP;&!@@ RA); T3R-e<7@B @] H"tDK%URv-܆@P;Aqc ܜW #Zk@ H{>[RA);IrOI]o'URv+ 蘄F!_-..QS@C~RvvVH^d–hy6@ cvFK惃ѡ3]őrSY\#PvB 06 ? 0PwGj>(eak$^jOTaKMّ#Gۗc/wG?N81m~wXJ? 0%C1@ Ig>8});cd> Ǯsϩm۶g:* 2Ha'Ǭ @|p o"쌑MDة̙3jƍ|BKNJW@ )E.Nz5Rcn U׭U-Xm[J{Վoߦ(llJU)YM @ 8NIE#)uXTaؕvt+sJa'— z@鍐 HͩyG@M@j>(IE#)uTamv.sxSO=6mtEuk|bپb'— CQ@\2Iȟ|P|p ߤ"쌑z ǮKC߼cgT!_&\Ma7B:"!I"ph% 5lul>1LKE#)"K])aǔ_&\Ma7B:"!I"ph% 5lul>1LKE#)R} ;>[t*{ ͛'zVZչobEv( @ R߯rG@8W>1KE#9% U>vC ; #G۷/ѣGڵkvx$eZ  DCg>8Rvv&}f!4%4BΖ-[&bΉ'+^);|)@I!K@% %HEIV\}춭ZѢNݹ:-׍[ $Z$N%@ 0*HE#9% U>v% _RqÊ^ Ub @᭔]1aReI!=-լ؉-#"C]@Sjaa!Ȕ|P |p ߤ"쌑N}VWطQEF.^x=W{M--NI&DB I4aB RA)t>1|3F6#SE*l;*.Aa4$ SD YJᛔ]1aR/TaE؉Pq;E !P"LȂ|P |p ߤ"쌑S)URv#L=.A :;ѥ  H}] &Px%5S"t–)6 :3 @ nRA);qcIF@a'dL@ H"\!B*lm @ I`OP:}Zfz;9:/-ۀTas5\3'ʷPﾥl3ο.?>G @6]j="ReI ;vB*T @`<l-=Ca ͩy0 t$ 5(C濬tTz;F؉Bs$q@ R"Đ|ԋVAʈN 0suuw6D~Oѿ]6l@ zwN 3+}4g\Mta ! x'Nw|~>{w  A C@ +p H1+]C3`+ x(a 5ƫ0鑚rNkENHNIE@ ;S]Hvi %7 @"oV X $$\ @L@j9U<ЅTaw@X;ayc  @ /N[:,KŁ PKar!077B.! )`8kÒ"Ri%_w^H @I&U8 ߯ DM@j>5C*]A@E*lrKa' ЍN7^H",5eŎL[z ;8` BvQN =@`J@j>XZ ;TԞJ.$ $ $6_ @] 0^u%6[{/l)OR ;1 @ 0:/-ۀTaM!7w @"̖/$ U؞  # @"쌞ڲ Hvԉq@؉;?x@ 0/lJ)ò ;0 "SP Sjaa!H H|0un]"tT*Dӄ(Ni&HA΋H3AB R,uBj!I)7z ;\;r'{@)'D HS/NL%^E*Dӄ(Ni&HAa4$ 5^ ;rS=Rr@=rH}*0BC`ds?R_0-֊Ta ! @ Hv:*.>Ka',oA ! 5E bHv  @I I@ Йasx #m!v @@R_0"uXTa@B`nnNϫ\B"@ SRLqֆ%5E)Ҥ*BJHNii@A) @ RT ag Fޏԋ,Uؑ P4OȊNV$dM@j>5TGpR_B*ME*BJHNii@; 0`fr]j3S{HjOTa!<@9v5B wR߯rJ|gQj&Z*b8 @p @3/NT@RGBa 5@ 0;a[.8C ; $ !@:"tNt! U]|- NXX @C@j&V˒*lq8@ء8 \ͩyKHdJ@j>)ڰ;TvR]HZ I@I2m8 8Wjii >& 5IvFHf]JRc xBa'L' 5̟;Tԋ,U؅0!$$ӆeB@j> " S)Uؑ P4OȊ   UJj&Z*b8 @p @3/NT@RGBa 5@ 0;a[.8C ; $ !@:"tNt! {j=u9u5L%ʷPﾥl3K_PΟWWwM]fM~x @CZTg>N>wC?a@,- @9XN5_1Ag*DJvBP.F ۅT, M3aC lJ"M8 xsH4 |0 9eNJEYؒ 4@ءB \ I@bOyy!;TW /r.$ $ $6v( @ 1Sa5!;Ce-~ر@S'?qR]Ýag믿s?<@`wyڲe@1`'p^x{k@ ~|]m.g7lй @o}[6X3:b @MX @#3Kz ;h@ O;h@@#@ @@@)b8<Ԅ @ءD \ͩyKHdJ`AX;ywh„@vH.B^  0 #3"ܘE) 씝@Nvr&@ oSԛR-..#.4:^BONI%@`&;3a!@@AMb' lIS쩔m@&C=@U.$O*#_ RC @ @I8y@ E;)f !@b%kf @v2M,aA @Q@@)7D@ 0<F#eE@vL;AC Ksssj~~^-,,dAA`>O.] it\oWH  @I Ixs/L4" |0$Έpc9l &Sv9@) 0?NRoRJx M!BWd8 ?ƫsw~@tvK A $La':  줘5| @N/@@4@ BaG;F!Ka9 @vgeeZp E@)2 , ͩye|C`>tEw~q]!&L$@a'$" Eν00 ;#k^䘲/(N'zDa'l &|0:uJI)ZZZ;BE.4  D\f"360$+v–4ŞJI؆l; _I@1N%:@@؉.%8@ 0R$b @ V;f dJa' @ P.rsO @@i=rXVi)Iaȴ4$077B  NFv$0!;D#@ #Hˆ. 7yc@l;e!I,ț{)&kjii) y?Sɓj޽w>P=ڶmzƍ^xQm߾]vm|cG>vZ5I1O ? N(؁`>ؗ`ϳb' {*C9H{Qx;v%FoZڽ{:zDI郰3nvK@8| K@?W4N¿( 촧aQ-vJdu]vر߶.ڞyF?*"k׮]СCٳ6uS5Pt<6m22>-_7xd=ܳlW〉ק_}M}912qT~tfVX1髋I(#83T @(N'p31ml1f͚5 0L;׊=vb$-~9rd"t;wnPT͚';6d 't”\N][*vn喉pVEa3[\V{W;+tԷa7 @vƠJIhZqq_q6 pV\y;][H#PFEիW;WBnc\wݺu͊$g]vƎ*.FlU..o@&&>ά& $@ @h;òLkMȢ'^{N@zΨ{[@1["f˖-D-ʸ-~mM+vTz~mRf[]Uv?m[RvRC695??@ jNOovz#L&gO(aGo۲WؾZ'?>/Mر쩒V&ߪBXuJgyf Y5.?ꄒ~u?:l{ѬN7׳7h'uhA`y F//r7a>gtY$bzVP^v(_aǎA ?iqmm}V,bC=r 씓k"@fv4:^d?aIP3vy5[nފ5sN3v۷3}رhvC7iQGЬ?ugpSu+샏}VTVȰS&N@ȇN>$N`F;S/T'*boqo1׀7R]%*V,]oM>b2[7i[os:ڮU7٪tR5ƿjL?g~VwƎv˾ᬏ E0vJ6B o|;D0^+cA;WmSRO<ݫ;XZ=TWz֍쿳O#7ŽQ=,k.uСɵci\L8efŏO}Uvf MuMjo~o(7 -^Zs-!O;!ic  @ w;gxC;kfGb'P'7Ka'9 @@|vˉG:bE Y# Ƴ#]J  @@Ž XL#Ē~Ή K#SZƉ @`L;cҍoaY:%W PMȜܜ@8<9 a?$zCI"e8 l dZ@q0%uɦq/L7҉ _tICG~DmذAzcز(-Ϛ5k&?jŒ ]ƄWJy_U?4{Jz(Q IDATo@ #A,vf3;䪗:_I??c?L ѩS+͇61@m^ZjLonyr؆ ! ! 2ʎvLN=KKKJv]__Wq\e>aC@Žp0@ =xGa? @Maz@ P'dz @9@)Ă3D [t#}Ss/l>­X] @ ';9e!eUo\H&Bcc>O}CR7nra  "% tgB@ A9^o'\sDD\@Οɟ[nEٳGʯW;^h@GJ(f Nagpqvȋg^ @)i@I;x0;;yw/r!&L$Fa'. B`*Oٸ%{*KCH ꮻފU$( (Ҁ+H 7AI8y@ @@v?C @ 0 @ Ma= @ @ @I8y]\簬.h H0'ٳG}rͩy՞F0".NVN<8xw~'~B:tH}cx&pݹ{,C0+;elFygc@P;Aqc Ia'@`> !"gE@;x##@7J5Nj_T 8c'(nA= '@` ZŽvB @ v3@ @ BaG;F!@ @@; @ @!#=Q @ww>uw|; @ ͩy* t'|;@I)[=|zxFŋj_ڽ{w0 0 ;@JPga'kQKȢ1x@E3@ ;Q' = %a'uq -BRvc,x |Pz8;XZbO(~C8c (*4 ARMvNC @ P6O @ $La': @ @e@);D@ @@vN^9, -BROT/;>)7 @ܜW ^i@@A)a",nSW^yEZJ} _P=  9\w.@7J5Nj_^ NPz@ !`C-baG{xcNa;3 ȱ2 ЍnRkZf=31@ ( c@O| !`1# @ @?  @{jIu| 3=s7Cqd29NDAbڳCJ 1)! FB2^u:޽/UVժhBN{s] @N; @ `g}C# @ @:tž~Re77#͓oư{8 666fL& g5 nFoS H&ꫯ|7k":j!@ ׃]Sκә;79%;%F@/;h"(!zRƇv2n^ҽh9;]ɛ::j!@ ׃]SκәS) ()`P#@EA@ W%2>DqN @[@3[= @ `') @`gz @ @ cNͫRͲh9?K/4p +  @F ɤ"@@WO3`'sx]-P%^{s}8!Nλ73\VhNnY7rM8 T@d) Y$p=;9 . ةn tgof т:V^TքsI챓d)5N@2Wɨ;H I  @ @ v74 @ @N'&%@ @/ Y @ @N;fYH@u7x#\r%nW\qEAass3L&\V7 NNZV[ϩ$ӟN?pСdu<sB`줳t&oNMN@INI( N/ڠJ,!Wto*Z%@+NW%@`sB`줳t&TvorJ c$H"s]8O ~]]xE?"I&!@2׃~}v_#@)P;;#_^ tN @`'@@||7OOS  PQ@S @쌣VI;wP @4vJ%@q vFeіI sb{F.Ds} 1?B`줳t&xinY#@@;v\J@z׃S(I\⛚T$@ `')H"z0 sgv:O;7rZo PO@SY# iըp=<匂MMd) ɲm&@`a,; @Ԅs$$6 @@.\:N[@3[=#{\>O^//}{ ʰgϞl/K8x`Ygښzɩ6oL9 vr욚  0>znTL Sq5{o۰w裏n;9vrZ `+rХjYV @SLSQsu;uG@\##iֳy]o[ [n馭o׿w׆Cm4o?W}ȑۙ΍?sO;N~/ony5\veᗿEM;vہeu^G ?Iu.vʲqBbxsy睴y[=ܭ[ľopWQKp,SGQ@2[/?xmcBwq)Sށ,@NMN@P6P5& L6Ek׮ "(+J߾ivi^plox;,s^hyˢφ!޽{+8 Ef)Stg-}嗷;{oq¾Cez_ М`9K# Эn۞]ӶpOFI#1xe߸xN&^ W^ye7N8osΙpj1q+Lo[/~r<&:Ew'3 KERR߾ f_eǝ ͦϛ5)ԭwo4 $H@4.zuN'tʟnrT6Ӹ`3ܾgQʄ6ӡHϼ̓}se8W`g6<͆\|'41lw;1H zf{:=kո5Nk+SVW'/~ҭɧkNxU 0׃Cuv_#@ `gzo鲊 o1<'.ٍ=vkk%mX쌺Ol;ٴJ (ؙTqqOw綿uUt`g#g]W_ ,Ty{Jc3tzƋuήoH6E}؉F[΢'۹GN9gH>wGm PvFZYHm , vba{_y䑭ljǟ鰠X^1,!18xL-չ*p){wE 2i~8p |k_ {TyeWǼӲqg]wua2 }xرc~7d. ɼ'@`[_ ѣᡇ =\x'i.p嗇)PU췃rY: >wPܬ̏`cAsR;z}~wڮ}{_xCLlx≰s^Cq$0`gHkkZ ] SOm}+_G`<aZ3P+_ҕŀDx駅;| XRC C۽ɳ#&@`hEs)l_^C`a^3ۯ>O_ג @.P;:W<"x3^ vXۯV-W_UHw @ {`gU#ɾ@QvE,[ÇCfֿk @ey:ݷ}*3:JvJ tHl @v5g`g}J# @@qpx+M}e>9&@&0K/--fNnV/F& X/_*;W vJk9 ]w]T'@`dnŚt뮭Gcgd/%0`'![QŒc7|;60!@ W7|3ر#|; 7|sP7= t;=m,qɲ:AUh9rHxW`D @ `'!P`:YvFd /Ŧ?aL&$r9uC`:؉{\-7v* X- Ym?Vz術G>~{@{oYrNT? ϗwΪ@X`TS+;ub1y“O>L|'`UDt8 0wΝ;KESʁ С`C|S @W@ިx5 @# @;9tI @;MH5;5r v5 PG`|ӫ*xL@Byo}+z g5NxU 0׃Cuvy#іI`4 D@3FZ_aW3Z&! vMk!`';~5vչr$Lh " H#-zp/ΰku @5;5FI;IMF `'N쌻VO @@A9vr  @ vjvjxͲr욚 S ~^}_W^y8vz  5{~` j_8Ω{mop=؅z9;;S~ PA@Sˡ*?BǷM\x?=pՅ]X)Vdp,1|{WroJ\&@CN&@`ѣGC7677d2)%;D@ [@ѐ5RS٦ hR;Mj;_|ֿ2?2J!@ ׃mgLNz @ v*ة*x(# ) @Q vF~'@@;n @ @;^ @ @L;6j6˪*xWW_}uسgOW%>Ԡ"@@ [@ѐ5R<ޮM]c Ф@"@@ᮻj|l @I׃Mjo,NzJEȭ;-V;\ڣ!;=jFx#kl4i,촩klp=ؤf'TVX J@ iՐ"VX J@ >Z@ѐ5C) @ @**Z%@ @H@ӣf( @ PE@SE˱ @ @ vzԌ6KYV&@I'7i,a29  {=`i8kH+?pv'2:5N@2׃ɨ;H {Iӛzzn"@ `' 'z[.g vrԚuz# t vQ5;k:dQw2`2 '`zn"@ қz>r`'N @ 0# @ @d* ɴq&@ @v @ @ v2m\ղmUUt%?կKRass3L&R;] J>ͼ4Νv@Ww}w;K0t#qݸt`'nQ7xN%@ .IDAT `')XC@S H*z0)w;ɻw P]@St# ݬTp=X,3;9ukZSS H*`&#@` _T Jʝ|2Nrr @ @4h @ @@rNrr @ @4h @ @@rNrr @ @4h @ @@rNrr @ @4h @ @@rNrr @ @4h @ @@rNrr @ @4h @ @@rNrr @ @4h @ @@rNrr @ @4h @ @@rNrr @ @4h @ @@rNrr @ @4h @ @@rNrr @ @4h @ @@rNrr @ @4h @ @@rNrr @ @4h @ @@rNrr @ @4h @ @@rNrr @ @8!ifH @ @8|pÇjB @ @4#pg23yIENDB`optuna-4.1.0/docs/make.bat000066400000000000000000000014561471332314300153660ustar00rootroot00000000000000@ECHO OFF pushd %~dp0 REM Command file for Sphinx documentation if "%SPHINXBUILD%" == "" ( set SPHINXBUILD=sphinx-build ) set SOURCEDIR=source set BUILDDIR=build set SPHINXPROJ=Optuna if "%1" == "" goto help %SPHINXBUILD% >NUL 2>NUL if errorlevel 9009 ( echo. echo.The 'sphinx-build' command was not found. Make sure you have Sphinx echo.installed, then set the SPHINXBUILD environment variable to point echo.to the full path of the 'sphinx-build' executable. Alternatively you echo.may add the Sphinx directory to PATH. echo. echo.If you don't have Sphinx installed, grab it from echo.http://sphinx-doc.org/ exit /b 1 ) %SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% goto end :help %SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% :end popd optuna-4.1.0/docs/source/000077500000000000000000000000001471332314300152535ustar00rootroot00000000000000optuna-4.1.0/docs/source/_static/000077500000000000000000000000001471332314300167015ustar00rootroot00000000000000optuna-4.1.0/docs/source/_static/css/000077500000000000000000000000001471332314300174715ustar00rootroot00000000000000optuna-4.1.0/docs/source/_static/css/custom.css000066400000000000000000000105251471332314300215200ustar00rootroot00000000000000/* WARNING: This style sheet may not be compatible with `sphinx-rtd-theme >= 0.5.0` */ /* Code-block for console */ .highlight-console pre { background: #d3d5d4; } /* Parameter names and colons after them in a signature */ em.sig-param > span:nth-child(1), em.sig-param > span:nth-child(2) { color: #555555; } /* Type hints and default values in a signature */ em.sig-param > span:not(:nth-child(1)):not(:nth-child(2)) { color: #2980b9; } /* Internal links in a signature */ .rst-content dl.class > dt a.reference.internal, .rst-content dl.method > dt a.reference.internal, .rst-content dl.function > dt a.reference.internal { color: #2980b9; } /* External links in a signature */ .rst-content dl.class > dt a.reference.external, .rst-content dl.method > dt a.reference.external, .rst-content dl.function > dt a.reference.external { color: #2980b9; } /* Containers for a signature */ .rst-content dl.class > dt, .rst-content dl.function > dt { background-color: #f0f0f0; } /* Containers for methods, properties, parameters, and returns */ .rst-content dl:not(.docutils) dl dt { border-left: solid 3px #6ab0de; } /* Main content */ .wy-nav-content { max-width: 1200px; } /* Sidebar header (and topbar for mobile) */ .wy-side-nav-search, .wy-nav-top { background: #f1f3f4; } .wy-side-nav-search div.version { color: #404040; } .wy-nav-top a { color: #404040; } .wy-nav-top i { color: #404040; } /* Sidebar */ .wy-nav-side { background: #f1f3f4; } /* A tag */ .wy-menu-vertical a { color: #707070; } a { color: #2ba9cd; } .wy-menu-vertical a:active { background-color: #2ba9cd; cursor: pointer; color: #f1f3f4; } .highlight { background: #f1f3f4; } .navbar { background: #ffffff; } @media only screen and (max-width: 896px) { .navbar { height: 0px; } } .navbar-nav { background: #ffffff; list-style: none; display: flex; flex-direction: row; align-items: center; justify-content: flex-end; padding: 20px; max-width: 1200px; margin-left: 300px; } .ml-auto { margin-left: auto !important; } .header_link { margin: 15px 2px; font-size: 16px; font-weight: 600; font-family: "Helvetica"; cursor: pointer; padding: 0.5rem 0.8rem 0.5rem 0.5rem; color: #636a73; } .navbar-nav a:focus, a:hover { color: #2ba9cd; text-decoration: none; } .navbar-nav a:visited { color: #636a73; text-decoration: none; } .wy-alert.wy-alert-info .wy-alert-title, .rst-content .note .wy-alert-title, .rst-content .wy-alert-info.attention .wy-alert-title, .rst-content .wy-alert-info.caution .wy-alert-title, .rst-content .wy-alert-info.danger .wy-alert-title, .rst-content .wy-alert-info.error .wy-alert-title, .rst-content .wy-alert-info.hint .wy-alert-title, .rst-content .wy-alert-info.important .wy-alert-title, .rst-content .wy-alert-info.tip .wy-alert-title, .rst-content .wy-alert-info.warning .wy-alert-title, .rst-content .seealso .wy-alert-title, .rst-content .wy-alert-info.admonition-todo .wy-alert-title, .rst-content .wy-alert-info.admonition .wy-alert-title, .wy-alert.wy-alert-info .rst-content .admonition-title, .rst-content .wy-alert.wy-alert-info .admonition-title, .rst-content .note .admonition-title, .rst-content .wy-alert-info.attention .admonition-title, .rst-content .wy-alert-info.caution .admonition-title, .rst-content .wy-alert-info.danger .admonition-title, .rst-content .wy-alert-info.error .admonition-title, .rst-content .wy-alert-info.hint .admonition-title, .rst-content .wy-alert-info.important .admonition-title, .rst-content .wy-alert-info.tip .admonition-title, .rst-content .wy-alert-info.warning .admonition-title, .rst-content .seealso .admonition-title, .rst-content .wy-alert-info.admonition-todo .admonition-title, .rst-content .wy-alert-info.admonition .admonition-title { background: #2ba9cd; } .wy-alert, .rst-content .note, .rst-content .attention, .rst-content .caution, .rst-content .danger, .rst-content .error, .rst-content .hint, .rst-content .important, .rst-content .tip, .rst-content .warning, .rst-content .seealso, .rst-content .admonition-todo, .rst-content .admonition { background: #f1f3f4; } .sphx-glr-thumbnails { width: 100%; grid-template-columns: repeat(auto-fill, minmax(200px, 1fr)); } .sphx-glr-thumbcontainer { min-height: 200px !important; min-width: 200px !important; } .sphx-glr-thumbcontainer img { display: inline; max-height: 200px; max-width: 200px; } optuna-4.1.0/docs/source/_templates/000077500000000000000000000000001471332314300174105ustar00rootroot00000000000000optuna-4.1.0/docs/source/_templates/autosummary/000077500000000000000000000000001471332314300217765ustar00rootroot00000000000000optuna-4.1.0/docs/source/_templates/autosummary/class.rst000066400000000000000000000006411471332314300236360ustar00rootroot00000000000000{% extends "!autosummary/class.rst" %} {# An autosummary template to exclude the class constructor (__init__) which doesn't contain any docstring in Optuna. #} {% block methods %} {% set methods = methods | select("ne", "__init__") | list %} {% if methods %} .. rubric:: Methods .. autosummary:: {% for item in methods %} ~{{ name }}.{{ item }} {%- endfor %} {% endif %} {% endblock %} optuna-4.1.0/docs/source/_templates/breadcrumbs.html000066400000000000000000000004531471332314300225710ustar00rootroot00000000000000 {%- extends "sphinx_rtd_theme/breadcrumbs.html" %} {% block breadcrumbs_aside %} {% endblock %} optuna-4.1.0/docs/source/_templates/footer.html000066400000000000000000000003001471332314300215650ustar00rootroot00000000000000{% extends "!footer.html" %} {% block extrafooter %} {% trans path=pathto('privacy') %}Privacy Policy{{ privacy }}.{% endtrans %} {{ super() }} {% endblock %} optuna-4.1.0/docs/source/_templates/layout.html000066400000000000000000000024401471332314300216130ustar00rootroot00000000000000{% extends "!layout.html" %} {%- block extrabody %}
{{ super() }} {% endblock %} optuna-4.1.0/docs/source/conf.py000066400000000000000000000165371471332314300165660ustar00rootroot00000000000000# -*- coding: utf-8 -*- # # Configuration file for the Sphinx documentation builder. # # This file does only contain a selection of the most common options. For a # full list see the documentation: # http://www.sphinx-doc.org/en/master/config # -- Path setup -------------------------------------------------------------- # If extensions (or modules to document with autodoc) are in another directory, # add these directories to sys.path here. If the directory is relative to the # documentation root, use os.path.abspath to make it absolute, like shown here. # # import os # import sys # sys.path.insert(0, os.path.abspath('.')) import warnings import plotly.io as pio from sklearn.exceptions import ConvergenceWarning import optuna # -- Project information ----------------------------------------------------- project = "Optuna" copyright = "2018, Optuna Contributors" author = "Optuna Contributors." # The short X.Y version version = optuna.version.__version__ # The full version, including alpha/beta/rc tags release = optuna.version.__version__ # -- General configuration --------------------------------------------------- pio.renderers.default = "sphinx_gallery" # If your documentation needs a minimal Sphinx version, state it here. # # needs_sphinx = '1.0' # Add any Sphinx extension module names here, as strings. They can be # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom # ones. extensions = [ "sphinx.ext.autodoc", "sphinx.ext.autosummary", "sphinx.ext.doctest", "sphinx.ext.imgconverter", "sphinx.ext.intersphinx", "sphinx.ext.mathjax", "sphinx.ext.napoleon", "sphinx.ext.viewcode", "sphinx.ext.githubpages", "sphinx_copybutton", "sphinx_gallery.gen_gallery" ] # Add any paths that contain templates here, relative to this directory. templates_path = ["_templates"] # The suffix(es) of source filenames. # You can specify multiple suffix as a list of string: # # source_suffix = ['.rst', '.md'] source_suffix = ".rst" # The master toctree document. master_doc = "index" # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. # This pattern also affects html_static_path and html_extra_path . exclude_patterns = [ "reference/visualization/generated/index.rst", "reference/visualization/matplotlib/generated/index.rst", ] # The name of the Pygments (syntax highlighting) style to use. pygments_style = "sphinx" # -- Options for HTML output ------------------------------------------------- # The theme to use for HTML and HTML Help pages. See the documentation for # a list of builtin themes. # html_theme = "sphinx_rtd_theme" # Theme options are theme-specific and customize the look and feel of a theme # further. For a list of options available for each theme, see the # documentation. # html_theme_options = {"logo_only": True, "navigation_with_keys": True} html_favicon = "../image/favicon.ico" html_logo = "../image/optuna-logo.png" # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, # so a file named "default.css" will overwrite the builtin "default.css". html_static_path = ["_static"] html_css_files = ["css/custom.css"] # Custom sidebar templates, must be a dictionary that maps document names # to template names. # # The default sidebars (for documents that don't match any pattern) are # defined by theme itself. Builtin themes are using these templates by # default: ``['localtoc.html', 'relations.html', 'sourcelink.html', # 'searchbox.html']``. # # html_sidebars = {} # -- Options for HTMLHelp output --------------------------------------------- # Output file base name for HTML help builder. htmlhelp_basename = "Optunadoc" # -- Options for LaTeX output ------------------------------------------------ latex_elements = { # The paper size ('letterpaper' or 'a4paper'). # # 'papersize': 'letterpaper', # The font size ('10pt', '11pt' or '12pt'). # # 'pointsize': '10pt', # Additional stuff for the LaTeX preamble. # # 'preamble': '', # Latex figure (float) alignment # # 'figure_align': 'htbp', } # Grouping the document tree into LaTeX files. List of tuples # (source start file, target name, title, # author, documentclass [howto, manual, or own class]). latex_documents = [ (master_doc, "Optuna.tex", "Optuna Documentation", "Optuna Contributors.", "manual"), ] # -- Options for manual page output ------------------------------------------ # One entry per manual page. List of tuples # (source start file, name, description, authors, manual section). man_pages = [(master_doc, "optuna", "Optuna Documentation", [author], 1)] # -- Options for Texinfo output ---------------------------------------------- # Grouping the document tree into Texinfo files. List of tuples # (source start file, target name, title, author, # dir menu entry, description, category) texinfo_documents = [ ( master_doc, "Optuna", "Optuna Documentation", author, "Optuna", "One line description of project.", "Miscellaneous", ), ] intersphinx_mapping = { "python": ("https://docs.python.org/3", None), "distributed": ("https://distributed.dask.org/en/stable", None), "lightgbm": ("https://lightgbm.readthedocs.io/en/stable", None), "matplotlib": ("https://matplotlib.org/stable", None), "numpy": ("https://numpy.org/doc/stable", None), "scipy": ("https://docs.scipy.org/doc/scipy", None), "sklearn": ("https://scikit-learn.org/stable", None), "torch": ("https://pytorch.org/docs/stable", None), "pandas": ("https://pandas.pydata.org/docs", None), "plotly": ("https://plotly.com/python-api-reference", None), } # -- Extension configuration ------------------------------------------------- autosummary_generate = True autodoc_typehints = "description" autodoc_default_options = { "members": True, "inherited-members": True, "exclude-members": "with_traceback", } # sphinx_copybutton option to not copy prompt. copybutton_prompt_text = "$ " # Sphinx Gallery pio.renderers.default = "sphinx_gallery_png" sphinx_gallery_conf = { "doc_module": ("sphinx_gallery"), "examples_dirs": [ "../../tutorial/10_key_features", "../../tutorial/20_recipes", "../visualization_examples", "../visualization_matplotlib_examples", ], "gallery_dirs": [ "tutorial/10_key_features", "tutorial/20_recipes", "reference/visualization/generated", "reference/visualization/matplotlib/generated", ], "compress_images": ("images", "thumbnails"), "thumbnail_size": (400, 280), "within_subsection_order": "FileNameSortKey", "filename_pattern": r"/*\.py", "first_notebook_cell": None, "image_scrapers": ("matplotlib", "plotly.io._sg_scraper.plotly_sg_scraper"), } # matplotlib plot directive plot_include_source = True plot_formats = [("png", 90)] plot_html_show_formats = False plot_html_show_source_link = False # sphinx plotly directive plotly_include_source = True plotly_formats = ["html"] plotly_html_show_formats = False plotly_html_show_source_link = False # Not showing common warning messages as in # https://sphinx-gallery.github.io/stable/configuration.html#removing-warnings. warnings.filterwarnings("ignore", category=ConvergenceWarning, module="sklearn") optuna-4.1.0/docs/source/faq.rst000066400000000000000000001002041471332314300165510ustar00rootroot00000000000000FAQ === .. contents:: :local: Can I use Optuna with X? (where X is your favorite ML library) -------------------------------------------------------------- Optuna is compatible with most ML libraries, and it's easy to use Optuna with those. Please refer to `examples `__. .. _objective-func-additional-args: How to define objective functions that have own arguments? ---------------------------------------------------------- There are two ways to realize it. First, callable classes can be used for that purpose as follows: .. code-block:: python import optuna class Objective: def __init__(self, min_x, max_x): # Hold this implementation specific arguments as the fields of the class. self.min_x = min_x self.max_x = max_x def __call__(self, trial): # Calculate an objective value by using the extra arguments. x = trial.suggest_float("x", self.min_x, self.max_x) return (x - 2) ** 2 # Execute an optimization by using an `Objective` instance. study = optuna.create_study() study.optimize(Objective(-100, 100), n_trials=100) Second, you can use ``lambda`` or ``functools.partial`` for creating functions (closures) that hold extra arguments. Below is an example that uses ``lambda``: .. code-block:: python import optuna # Objective function that takes three arguments. def objective(trial, min_x, max_x): x = trial.suggest_float("x", min_x, max_x) return (x - 2) ** 2 # Extra arguments. min_x = -100 max_x = 100 # Execute an optimization by using the above objective function wrapped by `lambda`. study = optuna.create_study() study.optimize(lambda trial: objective(trial, min_x, max_x), n_trials=100) Please also refer to `sklearn_additional_args.py `__ example, which reuses the dataset instead of loading it in each trial execution. Can I use Optuna without remote RDB servers? -------------------------------------------- Yes, it's possible. In the simplest form, Optuna works with :class:`~optuna.storages.InMemoryStorage`: .. code-block:: python study = optuna.create_study() study.optimize(objective) If you want to save and resume studies, it's handy to use SQLite as the local storage: .. code-block:: python study = optuna.create_study(study_name="foo_study", storage="sqlite:///example.db") study.optimize(objective) # The state of `study` will be persisted to the local SQLite file. Please see :ref:`rdb` for more details. How can I save and resume studies? ---------------------------------------------------- There are two ways of persisting studies, which depend if you are using :class:`~optuna.storages.InMemoryStorage` (default) or remote databases (RDB). In-memory studies can be saved and loaded like usual Python objects using ``pickle`` or ``joblib``. For example, using ``joblib``: .. code-block:: python study = optuna.create_study() joblib.dump(study, "study.pkl") And to resume the study: .. code-block:: python study = joblib.load("study.pkl") print("Best trial until now:") print(" Value: ", study.best_trial.value) print(" Params: ") for key, value in study.best_trial.params.items(): print(f" {key}: {value}") Note that Optuna does not support saving/reloading across different Optuna versions with ``pickle``. To save/reload a study across different Optuna versions, please use RDBs and `upgrade storage schema `__ if necessary. If you are using RDBs, see :ref:`rdb` for more details. How to suppress log messages of Optuna? --------------------------------------- By default, Optuna shows log messages at the ``optuna.logging.INFO`` level. You can change logging levels by using :func:`optuna.logging.set_verbosity`. For instance, you can stop showing each trial result as follows: .. code-block:: python optuna.logging.set_verbosity(optuna.logging.WARNING) study = optuna.create_study() study.optimize(objective) # Logs like '[I 2020-07-21 13:41:45,627] Trial 0 finished with value:...' are disabled. Please refer to :class:`optuna.logging` for further details. How to save machine learning models trained in objective functions? ------------------------------------------------------------------- Optuna saves hyperparameter values with their corresponding objective values to storage, but it discards intermediate objects such as machine learning models and neural network weights. To save models or weights, we recommend utilizing Optuna's built-in ``ArtifactStore``. For example, you can use the :func:`~optuna.artifacts.upload_artifact` as follows: .. code-block:: python base_path = "./artifacts" os.makedirs(base_path, exist_ok=True) artifact_store = optuna.artifacts.FileSystemArtifactStore(base_path=base_path) def objective(trial): svc_c = trial.suggest_float("svc_c", 1e-10, 1e10, log=True) clf = sklearn.svm.SVC(C=svc_c) clf.fit(X_train, y_train) # Save the model using ArtifactStore with open("model.pickle", "wb") as fout: pickle.dump(clf, fout) artifact_id = optuna.artifacts.upload_artifact( artifact_store=artifact_store, file_path="model.pickle", study_or_trial=trial.study, ) trial.set_user_attr("artifact_id", artifact_id) return 1.0 - accuracy_score(y_valid, clf.predict(X_valid)) study = optuna.create_study() study.optimize(objective, n_trials=100) To retrieve models or weights, you can list and download them using :func:`~optuna.artifacts.get_all_artifact_meta` and :func:`~optuna.artifacts.download_artifact` as shown below: .. code-block:: python # List all models for artifact_meta in optuna.artifacts.get_all_artifact_meta(study_or_trial=study): print(artifact_meta) # Download the best model trial = study.best_trial best_artifact_id = trial.user_attrs["artifact_id"] optuna.artifacts.download_artifact( artifact_store=artifact_store, file_path='best_model.pickle', artifact_id=best_artifact_id, ) For a more comprehensive guide, refer to the `ArtifactStore tutorial `_. How can I obtain reproducible optimization results? --------------------------------------------------- To make the parameters suggested by Optuna reproducible, you can specify a fixed random seed via ``seed`` argument of an instance of :mod:`~optuna.samplers` as follows: .. code-block:: python sampler = TPESampler(seed=10) # Make the sampler behave in a deterministic way. study = optuna.create_study(sampler=sampler) study.optimize(objective) However, there are two caveats. First, when optimizing a study in distributed or parallel mode, there is inherent non-determinism. Thus it is very difficult to reproduce the same results in such condition. We recommend executing optimization of a study sequentially if you would like to reproduce the result. Second, if your objective function behaves in a non-deterministic way (i.e., it does not return the same value even if the same parameters were suggested), you cannot reproduce an optimization. To deal with this problem, please set an option (e.g., random seed) to make the behavior deterministic if your optimization target (e.g., an ML library) provides it. How are exceptions from trials handled? --------------------------------------- Trials that raise exceptions without catching them will be treated as failures, i.e. with the :obj:`~optuna.trial.TrialState.FAIL` status. By default, all exceptions except :class:`~optuna.exceptions.TrialPruned` raised in objective functions are propagated to the caller of :func:`~optuna.study.Study.optimize`. In other words, studies are aborted when such exceptions are raised. It might be desirable to continue a study with the remaining trials. To do so, you can specify in :func:`~optuna.study.Study.optimize` which exception types to catch using the ``catch`` argument. Exceptions of these types are caught inside the study and will not propagate further. You can find the failed trials in log messages. .. code-block:: sh [W 2018-12-07 16:38:36,889] Setting status of trial#0 as TrialState.FAIL because of \ the following error: ValueError('A sample error in objective.') You can also find the failed trials by checking the trial states as follows: .. code-block:: python study.trials_dataframe() .. csv-table:: number,state,value,...,params,system_attrs 0,TrialState.FAIL,,...,0,Setting status of trial#0 as TrialState.FAIL because of the following error: ValueError('A test error in objective.') 1,TrialState.COMPLETE,1269,...,1, .. seealso:: The ``catch`` argument in :func:`~optuna.study.Study.optimize`. How are NaNs returned by trials handled? ---------------------------------------- Trials that return NaN (``float('nan')``) are treated as failures, but they will not abort studies. Trials which return NaN are shown as follows: .. code-block:: sh [W 2018-12-07 16:41:59,000] Setting status of trial#2 as TrialState.FAIL because the \ objective function returned nan. What happens when I dynamically alter a search space? ----------------------------------------------------- Since parameters search spaces are specified in each call to the suggestion API, e.g. :func:`~optuna.trial.Trial.suggest_float` and :func:`~optuna.trial.Trial.suggest_int`, it is possible to, in a single study, alter the range by sampling parameters from different search spaces in different trials. The behavior when altered is defined by each sampler individually. .. note:: Discussion about the TPE sampler. https://github.com/optuna/optuna/issues/822 How can I use two GPUs for evaluating two trials simultaneously? ---------------------------------------------------------------- If your optimization target supports GPU (CUDA) acceleration and you want to specify which GPU is used in your script, ``main.py``, the easiest way is to set ``CUDA_VISIBLE_DEVICES`` environment variable: .. code-block:: bash # On a terminal. # # Specify to use the first GPU, and run an optimization. $ export CUDA_VISIBLE_DEVICES=0 $ python main.py # On another terminal. # # Specify to use the second GPU, and run another optimization. $ export CUDA_VISIBLE_DEVICES=1 $ python main.py Please refer to `CUDA C Programming Guide `__ for further details. How can I test my objective functions? -------------------------------------- When you test objective functions, you may prefer fixed parameter values to sampled ones. In that case, you can use :class:`~optuna.trial.FixedTrial`, which suggests fixed parameter values based on a given dictionary of parameters. For instance, you can input arbitrary values of :math:`x` and :math:`y` to the objective function :math:`x + y` as follows: .. code-block:: python def objective(trial): x = trial.suggest_float("x", -1.0, 1.0) y = trial.suggest_int("y", -5, 5) return x + y objective(FixedTrial({"x": 1.0, "y": -1})) # 0.0 objective(FixedTrial({"x": -1.0, "y": -4})) # -5.0 Using :class:`~optuna.trial.FixedTrial`, you can write unit tests as follows: .. code-block:: python # A test function of pytest def test_objective(): assert 1.0 == objective(FixedTrial({"x": 1.0, "y": 0})) assert -1.0 == objective(FixedTrial({"x": 0.0, "y": -1})) assert 0.0 == objective(FixedTrial({"x": -1.0, "y": 1})) .. _out-of-memory-gc-collect: How do I avoid running out of memory (OOM) when optimizing studies? ------------------------------------------------------------------- If the memory footprint increases as you run more trials, try to periodically run the garbage collector. Specify ``gc_after_trial`` to :obj:`True` when calling :func:`~optuna.study.Study.optimize` or call :func:`gc.collect` inside a callback. .. code-block:: python def objective(trial): x = trial.suggest_float("x", -1.0, 1.0) y = trial.suggest_int("y", -5, 5) return x + y study = optuna.create_study() study.optimize(objective, n_trials=10, gc_after_trial=True) # `gc_after_trial=True` is more or less identical to the following. study.optimize(objective, n_trials=10, callbacks=[lambda study, trial: gc.collect()]) There is a performance trade-off for running the garbage collector, which could be non-negligible depending on how fast your objective function otherwise is. Therefore, ``gc_after_trial`` is :obj:`False` by default. Note that the above examples are similar to running the garbage collector inside the objective function, except for the fact that :func:`gc.collect` is called even when errors, including :class:`~optuna.exceptions.TrialPruned` are raised. .. note:: :class:`~optuna.integration.ChainerMNStudy` does currently not provide ``gc_after_trial`` nor callbacks for :func:`~optuna.integration.ChainerMNStudy.optimize`. When using this class, you will have to call the garbage collector inside the objective function. How can I output a log only when the best value is updated? ----------------------------------------------------------- Here's how to replace the logging feature of optuna with your own logging callback function. The implemented callback can be passed to :func:`~optuna.study.Study.optimize`. Here's an example: .. code-block:: python import optuna # Turn off optuna log notes. optuna.logging.set_verbosity(optuna.logging.WARN) def objective(trial): x = trial.suggest_float("x", 0, 1) return x ** 2 def logging_callback(study, frozen_trial): previous_best_value = study.user_attrs.get("previous_best_value", None) if previous_best_value != study.best_value: study.set_user_attr("previous_best_value", study.best_value) print( "Trial {} finished with best value: {} and parameters: {}. ".format( frozen_trial.number, frozen_trial.value, frozen_trial.params, ) ) study = optuna.create_study() study.optimize(objective, n_trials=100, callbacks=[logging_callback]) Note that this callback may show incorrect values when you try to optimize an objective function with ``n_jobs!=1`` (or other forms of distributed optimization) due to its reads and writes to storage that are prone to race conditions. How do I suggest variables which represent the proportion, that is, are in accordance with Dirichlet distribution? ------------------------------------------------------------------------------------------------------------------ When you want to suggest :math:`n` variables which represent the proportion, that is, :math:`p[0], p[1], ..., p[n-1]` which satisfy :math:`0 \le p[k] \le 1` for any :math:`k` and :math:`p[0] + p[1] + ... + p[n-1] = 1`, try the below. For example, these variables can be used as weights when interpolating the loss functions. These variables are in accordance with the flat `Dirichlet distribution `__. .. code-block:: python import numpy as np import matplotlib.pyplot as plt import optuna def objective(trial): n = 5 x = [] for i in range(n): x.append(- np.log(trial.suggest_float(f"x_{i}", 0, 1))) p = [] for i in range(n): p.append(x[i] / sum(x)) for i in range(n): trial.set_user_attr(f"p_{i}", p[i]) return 0 study = optuna.create_study(sampler=optuna.samplers.RandomSampler()) study.optimize(objective, n_trials=1000) n = 5 p = [] for i in range(n): p.append([trial.user_attrs[f"p_{i}"] for trial in study.trials]) axes = plt.subplots(n, n, figsize=(20, 20))[1] for i in range(n): for j in range(n): axes[j][i].scatter(p[i], p[j], marker=".") axes[j][i].set_xlim(0, 1) axes[j][i].set_ylim(0, 1) axes[j][i].set_xlabel(f"p_{i}") axes[j][i].set_ylabel(f"p_{j}") plt.savefig("sampled_ps.png") This method is justified in the following way: First, if we apply the transformation :math:`x = - \log (u)` to the variable :math:`u` sampled from the uniform distribution :math:`Uni(0, 1)` in the interval :math:`[0, 1]`, the variable :math:`x` will follow the exponential distribution :math:`Exp(1)` with scale parameter :math:`1`. Furthermore, for :math:`n` variables :math:`x[0], ..., x[n-1]` that follow the exponential distribution of scale parameter :math:`1` independently, normalizing them with :math:`p[i] = x[i] / \sum_i x[i]`, the vector :math:`p` follows the Dirichlet distribution :math:`Dir(\alpha)` of scale parameter :math:`\alpha = (1, ..., 1)`. You can verify the transformation by calculating the elements of the Jacobian. How can I optimize a model with some constraints? ------------------------------------------------- When you want to optimize a model with constraints, you can use the following classes: :class:`~optuna.samplers.TPESampler`, :class:`~optuna.samplers.NSGAIISampler` or `BoTorchSampler `__. The following example is a benchmark of Binh and Korn function, a multi-objective optimization, with constraints using :class:`~optuna.samplers.NSGAIISampler`. This one has two constraints :math:`c_0 = (x-5)^2 + y^2 - 25 \le 0` and :math:`c_1 = -(x - 8)^2 - (y + 3)^2 + 7.7 \le 0` and finds the optimal solution satisfying these constraints. .. code-block:: python import optuna def objective(trial): # Binh and Korn function with constraints. x = trial.suggest_float("x", -15, 30) y = trial.suggest_float("y", -15, 30) # Constraints which are considered feasible if less than or equal to zero. # The feasible region is basically the intersection of a circle centered at (x=5, y=0) # and the complement to a circle centered at (x=8, y=-3). c0 = (x - 5) ** 2 + y ** 2 - 25 c1 = -((x - 8) ** 2) - (y + 3) ** 2 + 7.7 # Store the constraints as user attributes so that they can be restored after optimization. trial.set_user_attr("constraint", (c0, c1)) v0 = 4 * x ** 2 + 4 * y ** 2 v1 = (x - 5) ** 2 + (y - 5) ** 2 return v0, v1 def constraints(trial): return trial.user_attrs["constraint"] sampler = optuna.samplers.NSGAIISampler(constraints_func=constraints) study = optuna.create_study( directions=["minimize", "minimize"], sampler=sampler, ) study.optimize(objective, n_trials=32, timeout=600) print("Number of finished trials: ", len(study.trials)) print("Pareto front:") trials = sorted(study.best_trials, key=lambda t: t.values) for trial in trials: print(" Trial#{}".format(trial.number)) print( " Values: Values={}, Constraint={}".format( trial.values, trial.user_attrs["constraint"][0] ) ) print(" Params: {}".format(trial.params)) If you are interested in an example for `BoTorchSampler `__, please refer to `this sample code `__. There are two kinds of constrained optimizations, one with soft constraints and the other with hard constraints. Soft constraints do not have to be satisfied, but an objective function is penalized if they are unsatisfied. On the other hand, hard constraints must be satisfied. Optuna is adopting the soft one and **DOES NOT** support the hard one. In other words, Optuna **DOES NOT** have built-in samplers for the hard constraints. How can I parallelize optimization? ----------------------------------- The variations of parallelization are in the following three cases. 1. Multi-threading parallelization with single node 2. Multi-processing parallelization with single node 3. Multi-processing parallelization with multiple nodes 1. Multi-threading parallelization with a single node ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Parallelization can be achieved by setting the argument ``n_jobs`` in :func:`optuna.study.Study.optimize`. However, the python code will not be faster due to GIL because :func:`optuna.study.Study.optimize` with ``n_jobs!=1`` uses multi-threading. While optimizing, it will be faster in limited situations, such as waiting for other server requests or C/C++ processing with numpy, etc., but it will not be faster in other cases. For more information about 1., see APIReference_. .. _APIReference: https://optuna.readthedocs.io/en/stable/reference/index.html 2. Multi-processing parallelization with single node ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This can be achieved by using :class:`~optuna.storages.journal.JournalFileBackend` or client/server RDBs (such as PostgreSQL and MySQL). For more information about 2., see TutorialEasyParallelization_. .. _TutorialEasyParallelization: https://optuna.readthedocs.io/en/stable/tutorial/10_key_features/004_distributed.html 3. Multi-processing parallelization with multiple nodes ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This can be achieved by using client/server RDBs (such as PostgreSQL and MySQL). However, if you are in the environment where you can not install a client/server RDB, you can not run multi-processing parallelization with multiple nodes. For more information about 3., see TutorialEasyParallelization_. .. _sqlite_concurrency: How can I solve the error that occurs when performing parallel optimization with SQLite3? ----------------------------------------------------------------------------------------- We would never recommend SQLite3 for parallel optimization in the following reasons. - To concurrently evaluate trials enqueued by :func:`~optuna.study.Study.enqueue_trial`, :class:`~optuna.storages.RDBStorage` uses `SELECT ... FOR UPDATE` syntax, which is unsupported in `SQLite3 `__. - As described in `the SQLAlchemy's documentation `__, SQLite3 (and pysqlite driver) does not support a high level of concurrency. You may get a "database is locked" error, which occurs when one thread or process has an exclusive lock on a database connection (in reality a file handle) and another thread times out waiting for the lock to be released. You can increase the default `timeout `__ value like `optuna.storages.RDBStorage("sqlite:///example.db", engine_kwargs={"connect_args": {"timeout": 20.0}})` though. - For distributed optimization via NFS, SQLite3 does not work as described at `FAQ section of sqlite.org `__. If you want to use a file-based Optuna storage for these scenarios, please consider using :class:`~optuna.storages.journal.JournalFileBackend` instead. .. code-block:: python import optuna from optuna.storages import JournalStorage from optuna.storages.journal import JournalFileBackend storage = JournalStorage(JournalFileBackend("optuna_journal_storage.log")) study = optuna.create_study(storage=storage) ... See `the Medium blog post `__ for details. .. _heartbeat_monitoring: Can I monitor trials and make them failed automatically when they are killed unexpectedly? ------------------------------------------------------------------------------------------ .. note:: Heartbeat mechanism is experimental. API would change in the future. A process running a trial could be killed unexpectedly, typically by a job scheduler in a cluster environment. If trials are killed unexpectedly, they will be left on the storage with their states `RUNNING` until we remove them or update their state manually. For such a case, Optuna supports monitoring trials using `heartbeat `__ mechanism. Using heartbeat, if a process running a trial is killed unexpectedly, Optuna will automatically change the state of the trial that was running on that process to :obj:`~optuna.trial.TrialState.FAIL` from :obj:`~optuna.trial.TrialState.RUNNING`. .. code-block:: python import optuna def objective(trial): (Very time-consuming computation) # Recording heartbeats every 60 seconds. # Other processes' trials where more than 120 seconds have passed # since the last heartbeat was recorded will be automatically failed. storage = optuna.storages.RDBStorage(url="sqlite:///:memory:", heartbeat_interval=60, grace_period=120) study = optuna.create_study(storage=storage) study.optimize(objective, n_trials=100) .. note:: The heartbeat is supposed to be used with :meth:`~optuna.study.Study.optimize`. If you use :meth:`~optuna.study.Study.ask` and :meth:`~optuna.study.Study.tell`, please change the state of the killed trials by calling :meth:`~optuna.study.Study.tell` explicitly. You can also execute a callback function to process the failed trial. Optuna provides a callback to retry failed trials as :class:`~optuna.storages.RetryFailedTrialCallback`. Note that a callback is invoked at a beginning of each trial, which means :class:`~optuna.storages.RetryFailedTrialCallback` will retry failed trials when a new trial starts to evaluate. .. code-block:: python import optuna from optuna.storages import RetryFailedTrialCallback storage = optuna.storages.RDBStorage( url="sqlite:///:memory:", heartbeat_interval=60, grace_period=120, failed_trial_callback=RetryFailedTrialCallback(max_retry=3), ) study = optuna.create_study(storage=storage) How can I deal with permutation as a parameter? ----------------------------------------------- Although it is not straightforward to deal with combinatorial search spaces like permutations with existing API, there exists a convenient technique for handling them. It involves re-parametrization of permutation search space of :math:`n` items as an independent :math:`n`-dimensional integer search space. This technique is based on the concept of `Lehmer code `__. A Lehmer code of a sequence is the sequence of integers in the same size, whose :math:`i`-th entry denotes how many inversions the :math:`i`-th entry of the permutation has after itself. In other words, the :math:`i`-th entry of the Lehmer code represents the number of entries that are located after and are smaller than the :math:`i`-th entry of the original sequence. For instance, the Lehmer code of the permutation :math:`(3, 1, 4, 2, 0)` is :math:`(3, 1, 2, 1, 0)`. Not only does the Lehmer code provide a unique encoding of permutations into an integer space, but it also has some desirable properties. For example, the sum of Lehmer code entries is equal to the minimum number of adjacent transpositions necessary to transform the corresponding permutation into the identity permutation. Additionally, the lexicographical order of the encodings of two permutations is the same as that of the original sequence. Therefore, Lehmer code preserves "closeness" among permutations in some sense, which is important for the optimization algorithm. An Optuna implementation example to solve Euclid TSP is as follows: .. code-block:: python import numpy as np import optuna def decode(lehmer_code: list[int]) -> list[int]: """Decode Lehmer code to permutation. This function decodes Lehmer code represented as a list of integers to a permutation. """ all_indices = list(range(n)) output = [] for k in lehmer_code: value = all_indices[k] output.append(value) all_indices.remove(value) return output # Euclidean coordinates of cities for TSP. city_coordinates = np.array( [[0.0, 0.0], [1.0, 0.0], [0.0, 1.0], [1.0, 1.0], [2.0, 2.0], [-1.0, -1.0]] ) n = len(city_coordinates) def objective(trial: optuna.Trial) -> float: # Suggest a permutation in the Lehmer code representation. lehmer_code = [trial.suggest_int(f"x{i}", 0, n - i - 1) for i in range(n)] permutation = decode(lehmer_code) # Calculate the total distance of the suggested path. total_distance = 0.0 for i in range(n): total_distance += np.linalg.norm( city_coordinates[permutation[i]] - city_coordinates[np.roll(permutation, 1)[i]] ) return total_distance study = optuna.create_study() study.optimize(objective, n_trials=10) lehmer_code = study.best_params.values() print(decode(lehmer_code)) How can I ignore duplicated samples? ------------------------------------ Optuna may sometimes suggest parameters evaluated in the past and if you would like to avoid this problem, you can try out the following workaround: .. code-block:: python import optuna from optuna.trial import TrialState def objective(trial): # Sample parameters. x = trial.suggest_int("x", -5, 5) y = trial.suggest_int("y", -5, 5) # Fetch all the trials to consider. # In this example, we use only completed trials, but users can specify other states # such as TrialState.PRUNED and TrialState.FAIL. states_to_consider = (TrialState.COMPLETE,) trials_to_consider = trial.study.get_trials(deepcopy=False, states=states_to_consider) # Check whether we already evaluated the sampled `(x, y)`. for t in reversed(trials_to_consider): if trial.params == t.params: # Use the existing value as trial duplicated the parameters. return t.value # Compute the objective function if the parameters are not duplicated. # We use the 2D sphere function in this example. return x ** 2 + y ** 2 study = optuna.create_study() study.optimize(objective, n_trials=100) .. _remove_for_artifact_store: How can I delete all the artifacts uploaded to a study? ------------------------------------------------------- Optuna supports :mod:`~optuna.artifacts` for large data storage during an optimization. After you conduct enormous amount of experiments, you may want to remove the artifacts stored during optimizations. We strongly recommend to create a new directory or bucket for each study so that all the artifacts linked to a study can be entirely removed by deleting the directory or the bucket. However, if it is necessary to remove artifacts from a Python script, users can use the following code: .. warning:: :func:`~optuna.study.Study.add_trial` and :meth:`~optuna.study.copy_study` do not copy artifact files linked to :class:`~optuna.study.Study` or :class:`~optuna.trial.Trial`. Please make sure **NOT** to delete the artifacts from the source study or trial. Failing to do so may lead to unexpected behaviors as Optuna does not guarantee expected behaviors when users call :meth:`remove` externally. Due to the Optuna software design, it is hard to officially support the delete feature and we are not planning to support this feature in the future either. .. code-block:: python from optuna.artifacts import get_all_artifact_meta def remove_artifacts(study, artifact_store): # NOTE: ``artifact_store.remove`` is discouraged to use because it is an internal feature. storage = study._storage for trial in study.trials: for artifact_meta in get_all_artifact_meta(trial, storage=storage): # For each trial, remove the artifacts uploaded to ``base_path``. artifact_store.remove(artifact_meta.artifact_id) for artifact_meta in get_all_artifact_meta(study): # Remove the artifacts uploaded to ``base_path``. artifact_store.remove(artifact_meta.artifact_id) optuna-4.1.0/docs/source/index.rst000066400000000000000000000123721471332314300171210ustar00rootroot00000000000000|optunalogo| Optuna: A hyperparameter optimization framework =============================================== *Optuna* is an automatic hyperparameter optimization software framework, particularly designed for machine learning. It features an imperative, *define-by-run* style user API. Thanks to our *define-by-run* API, the code written with Optuna enjoys high modularity, and the user of Optuna can dynamically construct the search spaces for the hyperparameters. Key Features ------------ Optuna has modern functionalities as follows: - :doc:`Lightweight, versatile, and platform agnostic architecture ` - Handle a wide variety of tasks with a simple installation that has few requirements. - :doc:`Pythonic search spaces ` - Define search spaces using familiar Python syntax including conditionals and loops. - :doc:`Efficient optimization algorithms ` - Adopt state-of-the-art algorithms for sampling hyperparameters and efficiently pruning unpromising trials. - :doc:`Easy parallelization ` - Scale studies to tens or hundreds of workers with little or no changes to the code. - :doc:`Quick visualization ` - Inspect optimization histories from a variety of plotting functions. Basic Concepts -------------- We use the terms *study* and *trial* as follows: - Study: optimization based on an objective function - Trial: a single execution of the objective function Please refer to sample code below. The goal of a *study* is to find out the optimal set of hyperparameter values (e.g., ``classifier`` and ``svm_c``) through multiple *trials* (e.g., ``n_trials=100``). Optuna is a framework designed for the automation and the acceleration of the optimization *studies*. |Open in Colab| .. code:: python import ... # Define an objective function to be minimized. def objective(trial): # Invoke suggest methods of a Trial object to generate hyperparameters. regressor_name = trial.suggest_categorical('classifier', ['SVR', 'RandomForest']) if regressor_name == 'SVR': svr_c = trial.suggest_float('svr_c', 1e-10, 1e10, log=True) regressor_obj = sklearn.svm.SVR(C=svr_c) else: rf_max_depth = trial.suggest_int('rf_max_depth', 2, 32) regressor_obj = sklearn.ensemble.RandomForestRegressor(max_depth=rf_max_depth) X, y = sklearn.datasets.fetch_california_housing(return_X_y=True) X_train, X_val, y_train, y_val = sklearn.model_selection.train_test_split(X, y, random_state=0) regressor_obj.fit(X_train, y_train) y_pred = regressor_obj.predict(X_val) error = sklearn.metrics.mean_squared_error(y_val, y_pred) return error # An objective value linked with the Trial object. study = optuna.create_study() # Create a new study. study.optimize(objective, n_trials=100) # Invoke optimization of the objective function. Web Dashboard ------------- `Optuna Dashboard `__ is a real-time web dashboard for Optuna. You can check the optimization history, hyperparameter importance, etc. in graphs and tables. You don't need to create a Python script to call `Optuna's visualization `__ functions. Feature requests and bug reports are welcome! .. image:: https://user-images.githubusercontent.com/5564044/204975098-95c2cb8c-0fb5-4388-abc4-da32f56cb4e5.gif ``optuna-dashboard`` can be installed via pip: .. code-block:: console $ pip install optuna-dashboard .. TIP:: Please check out the `getting started `__ section of Optuna Dashboard's official documentation. Communication ------------- - `GitHub Discussions `__ for questions. - `GitHub Issues `__ for bug reports and feature requests. Contribution ------------ Any contributions to Optuna are welcome! When you send a pull request, please follow the `contribution guide `__. License ------- MIT License (see `LICENSE `__). Optuna uses the codes from SciPy and fdlibm projects (see :doc:`Third-party License `). Reference --------- Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A Next-generation Hyperparameter Optimization Framework. In KDD (`arXiv `__). .. toctree:: :maxdepth: 2 :caption: Contents: installation tutorial/index reference/index faq Indices and tables ================== * :ref:`genindex` * :ref:`modindex` * :ref:`search` .. |optunalogo| image:: https://raw.githubusercontent.com/optuna/optuna/master/docs/image/optuna-logo.png :width: 800 :alt: OPTUNA .. |Open in Colab| image:: https://colab.research.google.com/assets/colab-badge.svg :target: http://colab.research.google.com/github/optuna/optuna-examples/blob/main/quickstart.ipynb optuna-4.1.0/docs/source/installation.rst000066400000000000000000000006621471332314300205120ustar00rootroot00000000000000Installation ============ Optuna supports Python 3.8 or newer. We recommend to install Optuna via pip: .. code-block:: bash $ pip install optuna You can also install the development version of Optuna from master branch of Git repository: .. code-block:: bash $ pip install git+https://github.com/optuna/optuna.git You can also install Optuna via conda: .. code-block:: bash $ conda install -c conda-forge optuna optuna-4.1.0/docs/source/license_thirdparty.rst000066400000000000000000000036001471332314300217000ustar00rootroot00000000000000:orphan: Third-party License =================== SciPy ----- The Optuna contains the codes from SciPy project. Copyright (c) 2001-2002 Enthought, Inc. 2003-2022, SciPy Developers. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. fdlibm ------ Copyright (C) 1993 by Sun Microsystems, Inc. All rights reserved. Developed at SunPro, a Sun Microsystems, Inc. business. Permission to use, copy, modify, and distribute this software is freely granted, provided that this notice is preserved. optuna-4.1.0/docs/source/privacy.rst000066400000000000000000000011431471332314300174610ustar00rootroot00000000000000:orphan: Privacy Policy ============== Google Analytics ---------------- To collect information about how visitors use our website and to improve our services, we are using Google Analytics on this website. You can find out more about how Google Analytics works and about how information is collected on the Google Analytics terms of services and on Google's privacy policy. - Google Analytics Terms of Service: http://www.google.com/analytics/terms/us.html - Google Privacy Policy: https://policies.google.com/privacy?hl=en - Google Analytics Opt-out Add-on: https://tools.google.com/dlpage/gaoptout?hl=en optuna-4.1.0/docs/source/reference/000077500000000000000000000000001471332314300172115ustar00rootroot00000000000000optuna-4.1.0/docs/source/reference/artifacts.rst000066400000000000000000000034271471332314300217310ustar00rootroot00000000000000.. module:: optuna.artifacts optuna.artifacts ================ The :mod:`~optuna.artifacts` module provides the way to manage artifacts (output files) in Optuna. Please also check :ref:`artifact_tutorial` and `our article `__. The storages covered by :mod:`~optuna.artifacts` are the following: +-------------------------+----------------------------------------+ | Class Name | Supported Storage | +=========================+========================================+ | FileSystemArtifactStore | Local File System, Network File System | +-------------------------+----------------------------------------+ | Boto3ArtifactStore | Amazon S3 Compatible Object Storage | +-------------------------+----------------------------------------+ | GCSArtifactStore | Google Cloud Storage | +-------------------------+----------------------------------------+ .. note:: The methods defined in each ``ArtifactStore`` are not intended to be directly accessed by library users. .. note:: As ``ArtifactStore`` does not officially provide user API for artifact removal, please refer to :ref:`remove_for_artifact_store` for the hack. .. autoclass:: optuna.artifacts.FileSystemArtifactStore :no-members: .. autoclass:: optuna.artifacts.Boto3ArtifactStore :no-members: .. autoclass:: optuna.artifacts.GCSArtifactStore :no-members: .. autoclass:: optuna.artifacts.Backoff :no-members: .. autoclass:: optuna.artifacts.ArtifactMeta :no-members: .. autofunction:: optuna.artifacts.upload_artifact .. autofunction:: optuna.artifacts.get_all_artifact_meta .. autofunction:: optuna.artifacts.download_artifact optuna-4.1.0/docs/source/reference/cli.rst000066400000000000000000000004351471332314300205140ustar00rootroot00000000000000.. module:: optuna.cli optuna.cli ========== The :mod:`~optuna.cli` module implements Optuna's command-line functionality. For detail, please see the result of .. code-block:: console $ optuna --help .. seealso:: The :ref:`cli` tutorial provides use-cases with examples. optuna-4.1.0/docs/source/reference/distributions.rst000066400000000000000000000023631471332314300226510ustar00rootroot00000000000000.. module:: optuna.distributions optuna.distributions ==================== The :mod:`~optuna.distributions` module defines various classes representing probability distributions, mainly used to suggest initial hyperparameter values for an optimization trial. Distribution classes inherit from a library-internal :class:`~optuna.distributions.BaseDistribution`, and is initialized with specific parameters, such as the ``low`` and ``high`` endpoints for a :class:`~optuna.distributions.IntDistribution`. Optuna users should not use distribution classes directly, but instead use utility functions provided by :class:`~optuna.trial.Trial` such as :meth:`~optuna.trial.Trial.suggest_int`. .. autosummary:: :toctree: generated/ :nosignatures: optuna.distributions.FloatDistribution optuna.distributions.IntDistribution optuna.distributions.UniformDistribution optuna.distributions.LogUniformDistribution optuna.distributions.DiscreteUniformDistribution optuna.distributions.IntUniformDistribution optuna.distributions.IntLogUniformDistribution optuna.distributions.CategoricalDistribution optuna.distributions.distribution_to_json optuna.distributions.json_to_distribution optuna.distributions.check_distribution_compatibility optuna-4.1.0/docs/source/reference/exceptions.rst000066400000000000000000000012231471332314300221220ustar00rootroot00000000000000.. module:: optuna.exceptions optuna.exceptions ================= The :mod:`~optuna.exceptions` module defines Optuna-specific exceptions deriving from a base :class:`~optuna.exceptions.OptunaError` class. Of special importance for library users is the :class:`~optuna.exceptions.TrialPruned` exception to be raised if :func:`optuna.trial.Trial.should_prune` returns ``True`` for a trial that should be pruned. .. autosummary:: :toctree: generated/ :nosignatures: optuna.exceptions.OptunaError optuna.exceptions.TrialPruned optuna.exceptions.CLIUsageError optuna.exceptions.StorageInternalError optuna.exceptions.DuplicatedStudyError optuna-4.1.0/docs/source/reference/importance.rst000066400000000000000000000033421471332314300221060ustar00rootroot00000000000000.. module:: optuna.importance optuna.importance ================= The :mod:`~optuna.importance` module provides functionality for evaluating hyperparameter importances based on completed trials in a given study. The utility function :func:`~optuna.importance.get_param_importances` takes a :class:`~optuna.study.Study` and optional evaluator as two of its inputs. The evaluator must derive from :class:`~optuna.importance.BaseImportanceEvaluator`, and is initialized as a :class:`~optuna.importance.FanovaImportanceEvaluator` by default when not passed in. Users implementing custom evaluators should refer to either :class:`~optuna.importance.FanovaImportanceEvaluator`, :class:`~optuna.importance.MeanDecreaseImpurityImportanceEvaluator`, or :class:`~optuna.importance.PedAnovaImportanceEvaluator` as a guide, paying close attention to the format of the return value from the Evaluator's ``evaluate`` function. .. note:: :class:`~optuna.importance.FanovaImportanceEvaluator` takes over 1 minute when given a study that contains 1000+ trials. We published `optuna-fast-fanova `__ library, that is a Cython accelerated fANOVA implementation. By using it, you can get hyperparameter importances within a few seconds. If ``n_trials`` is more than 10000, the Cython implementation takes more than a minute, so you can use :class:`~optuna.importance.PedAnovaImportanceEvaluator` instead, enabling the evaluation to finish in a second. .. autosummary:: :toctree: generated/ :nosignatures: optuna.importance.get_param_importances optuna.importance.FanovaImportanceEvaluator optuna.importance.MeanDecreaseImpurityImportanceEvaluator optuna.importance.PedAnovaImportanceEvaluator optuna-4.1.0/docs/source/reference/index.rst000066400000000000000000000004411471332314300210510ustar00rootroot00000000000000API Reference ============= .. toctree:: :maxdepth: 1 optuna artifacts cli distributions exceptions importance integration logging pruners samplers/index search_space storages study terminator trial visualization/index optuna-4.1.0/docs/source/reference/integration.rst000066400000000000000000000307671471332314300223030ustar00rootroot00000000000000.. module:: optuna.integration optuna.integration ================== The :mod:`~optuna.integration` module contains classes used to integrate Optuna with external machine learning frameworks. .. note:: Optuna's integration modules for third-party libraries have started migrating from Optuna itself to a package called `optuna-integration`. Please check the `repository `__ and the `documentation `__. For most of the ML frameworks supported by Optuna, the corresponding Optuna integration class serves only to implement a callback object and functions, compliant with the framework's specific callback API, to be called with each intermediate step in the model training. The functionality implemented in these callbacks across the different ML frameworks includes: (1) Reporting intermediate model scores back to the Optuna trial using :func:`optuna.trial.Trial.report`, (2) According to the results of :func:`optuna.trial.Trial.should_prune`, pruning the current model by raising :func:`optuna.TrialPruned`, and (3) Reporting intermediate Optuna data such as the current trial number back to the framework, as done in :class:`~optuna.integration.MLflowCallback`. For scikit-learn, an integrated :class:`~optuna.integration.OptunaSearchCV` estimator is available that combines scikit-learn BaseEstimator functionality with access to a class-level ``Study`` object. Dependencies of each integration -------------------------------- We summarize the necessary dependencies for each integration. +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+ | Integration | Dependencies | +===================================================================================================================================================================================+====================================+ | `AllenNLP `__ | allennlp, torch, psutil, jsonnet | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+ | `BoTorch `__ | botorch, gpytorch, torch | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+ | `CatBoost `__ | catboost | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+ | `ChainerMN `__ | chainermn | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+ | `Chainer `__ | chainer | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+ | `pycma `__ | cma | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+ | `Dask `__ | distributed | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+ | `FastAI `__ | fastai | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+ | `Keras `__ | keras | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+ | `LightGBMTuner `__ | lightgbm, scikit-learn | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+ | `LightGBMPruningCallback `__ | lightgbm | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+ | `MLflow `__ | mlflow | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+ | `MXNet `__ | mxnet | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+ | PyTorch `Distributed `__ | torch | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+ | PyTorch (`Ignite `__) | pytorch-ignite | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+ | PyTorch (`Lightning `__) | pytorch-lightning | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+ | `SHAP `__ | scikit-learn, shap | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+ | `Scikit-learn `__ | pandas, scipy, scikit-learn | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+ | `SKorch `__ | skorch | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+ | `TensorBoard `__ | tensorboard, tensorflow | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+ | `TensorFlow `__ | tensorflow, tensorflow-estimator | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+ | `TensorFlow + Keras `__ | tensorflow | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+ | `Weights & Biases `__ | wandb | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+ | `XGBoost `__ | xgboost | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+ optuna-4.1.0/docs/source/reference/logging.rst000066400000000000000000000013651471332314300213760ustar00rootroot00000000000000.. module:: optuna.logging optuna.logging ============== The :mod:`~optuna.logging` module implements logging using the Python ``logging`` package. Library users may be especially interested in setting verbosity levels using :func:`~optuna.logging.set_verbosity` to one of ``optuna.logging.CRITICAL`` (aka ``optuna.logging.FATAL``), ``optuna.logging.ERROR``, ``optuna.logging.WARNING`` (aka ``optuna.logging.WARN``), ``optuna.logging.INFO``, or ``optuna.logging.DEBUG``. .. autosummary:: :toctree: generated/ :nosignatures: optuna.logging.get_verbosity optuna.logging.set_verbosity optuna.logging.disable_default_handler optuna.logging.enable_default_handler optuna.logging.disable_propagation optuna.logging.enable_propagation optuna-4.1.0/docs/source/reference/optuna.rst000066400000000000000000000011071471332314300212500ustar00rootroot00000000000000.. module:: optuna optuna ====== The :mod:`optuna` module is primarily used as an alias for basic Optuna functionality coded in other modules. Currently, two modules are aliased: (1) from :mod:`optuna.study`, functions regarding the Study lifecycle, and (2) from :mod:`optuna.exceptions`, the TrialPruned Exception raised when a trial is pruned. .. autosummary:: :toctree: generated/ :nosignatures: optuna.create_study optuna.load_study optuna.delete_study optuna.copy_study optuna.get_all_study_names optuna.get_all_study_summaries optuna.TrialPruned optuna-4.1.0/docs/source/reference/pruners.rst000066400000000000000000000025511471332314300214440ustar00rootroot00000000000000.. module:: optuna.pruners optuna.pruners ============== The :mod:`~optuna.pruners` module defines a :class:`~optuna.pruners.BasePruner` class characterized by an abstract :meth:`~optuna.pruners.BasePruner.prune` method, which, for a given trial and its associated study, returns a boolean value representing whether the trial should be pruned. This determination is made based on stored intermediate values of the objective function, as previously reported for the trial using :meth:`optuna.trial.Trial.report`. The remaining classes in this module represent child classes, inheriting from :class:`~optuna.pruners.BasePruner`, which implement different pruning strategies. .. warning:: Currently :mod:`~optuna.pruners` module is expected to be used only for single-objective optimization. .. seealso:: :ref:`pruning` tutorial explains the concept of the pruner classes and a minimal example. .. seealso:: :ref:`user_defined_pruner` tutorial could be helpful if you want to implement your own pruner classes. .. autosummary:: :toctree: generated/ :nosignatures: optuna.pruners.BasePruner optuna.pruners.MedianPruner optuna.pruners.NopPruner optuna.pruners.PatientPruner optuna.pruners.PercentilePruner optuna.pruners.SuccessiveHalvingPruner optuna.pruners.HyperbandPruner optuna.pruners.ThresholdPruner optuna.pruners.WilcoxonPruner optuna-4.1.0/docs/source/reference/samplers/000077500000000000000000000000001471332314300210375ustar00rootroot00000000000000optuna-4.1.0/docs/source/reference/samplers/index.rst000066400000000000000000000403121471332314300227000ustar00rootroot00000000000000.. module:: optuna.samplers optuna.samplers =============== The :mod:`~optuna.samplers` module defines a base class for parameter sampling as described extensively in :class:`~optuna.samplers.BaseSampler`. The remaining classes in this module represent child classes, deriving from :class:`~optuna.samplers.BaseSampler`, which implement different sampling strategies. .. seealso:: :ref:`pruning` tutorial explains the overview of the sampler classes. .. seealso:: :ref:`user_defined_sampler` tutorial could be helpful if you want to implement your own sampler classes. .. seealso:: If you are unsure about which sampler to use, please consider using `AutoSampler `__, which automatically selects a sampler during optimization. For more detail, see `the article on AutoSampler `__. +----------------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+-----------------------------------------------------------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------------------------------------------------------+ | | RandomSampler | GridSampler | TPESampler | CmaEsSampler | NSGAIISampler | QMCSampler | GPSampler | BoTorchSampler | BruteForceSampler | +==================================+===============================+===============================+===============================+===============================+=============================================================================+===============================+===============================+===============================+===============================================================================+ | Float parameters |:math:`\color{green}\checkmark`|:math:`\color{green}\checkmark`|:math:`\color{green}\checkmark`|:math:`\color{green}\checkmark`| :math:`\blacktriangle` |:math:`\color{green}\checkmark`|:math:`\color{green}\checkmark`|:math:`\color{green}\checkmark`|:math:`\color{green}\checkmark` (:math:`\color{red}\times` for infinite domain)| +----------------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+-----------------------------------------------------------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------------------------------------------------------+ | Integer parameters |:math:`\color{green}\checkmark`|:math:`\color{green}\checkmark`|:math:`\color{green}\checkmark`|:math:`\color{green}\checkmark`| :math:`\blacktriangle` |:math:`\color{green}\checkmark`|:math:`\color{green}\checkmark`| :math:`\blacktriangle` | :math:`\color{green}\checkmark` | +----------------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+-----------------------------------------------------------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------------------------------------------------------+ | Categorical parameters |:math:`\color{green}\checkmark`|:math:`\color{green}\checkmark`|:math:`\color{green}\checkmark`| :math:`\blacktriangle` | :math:`\color{green}\checkmark` | :math:`\blacktriangle` |:math:`\color{green}\checkmark`| :math:`\blacktriangle` | :math:`\color{green}\checkmark` | +----------------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+-----------------------------------------------------------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------------------------------------------------------+ | Pruning |:math:`\color{green}\checkmark`|:math:`\color{green}\checkmark`|:math:`\color{green}\checkmark`| :math:`\blacktriangle` | :math:`\color{red}\times` (:math:`\blacktriangle` for single-objective) |:math:`\color{green}\checkmark`| :math:`\blacktriangle` | :math:`\blacktriangle` | :math:`\color{green}\checkmark` | +----------------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+-----------------------------------------------------------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------------------------------------------------------+ | Multivariate optimization | :math:`\blacktriangle` | :math:`\blacktriangle` |:math:`\color{green}\checkmark`|:math:`\color{green}\checkmark`| :math:`\blacktriangle` | :math:`\blacktriangle` |:math:`\color{green}\checkmark`|:math:`\color{green}\checkmark`| :math:`\blacktriangle` | +----------------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+-----------------------------------------------------------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------------------------------------------------------+ | Conditional search space |:math:`\color{green}\checkmark`| :math:`\blacktriangle` |:math:`\color{green}\checkmark`| :math:`\blacktriangle` | :math:`\blacktriangle` | :math:`\blacktriangle` | :math:`\blacktriangle` | :math:`\blacktriangle` | :math:`\color{green}\checkmark` | +----------------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+-----------------------------------------------------------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------------------------------------------------------+ | Multi-objective optimization |:math:`\color{green}\checkmark`|:math:`\color{green}\checkmark`|:math:`\color{green}\checkmark`| :math:`\color{red}\times` |:math:`\color{green}\checkmark` (:math:`\blacktriangle` for single-objective)|:math:`\color{green}\checkmark`| :math:`\color{red}\times` |:math:`\color{green}\checkmark`| :math:`\color{green}\checkmark` | +----------------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+-----------------------------------------------------------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------------------------------------------------------+ | Batch optimization |:math:`\color{green}\checkmark`|:math:`\color{green}\checkmark`|:math:`\color{green}\checkmark`|:math:`\color{green}\checkmark`| :math:`\color{green}\checkmark` |:math:`\color{green}\checkmark`| :math:`\blacktriangle` |:math:`\color{green}\checkmark`| :math:`\color{green}\checkmark` | +----------------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+-----------------------------------------------------------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------------------------------------------------------+ | Distributed optimization |:math:`\color{green}\checkmark`|:math:`\color{green}\checkmark`|:math:`\color{green}\checkmark`|:math:`\color{green}\checkmark`| :math:`\color{green}\checkmark` |:math:`\color{green}\checkmark`| :math:`\blacktriangle` |:math:`\color{green}\checkmark`| :math:`\color{green}\checkmark` | +----------------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+-----------------------------------------------------------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------------------------------------------------------+ | Constrained optimization | :math:`\color{red}\times` | :math:`\color{red}\times` |:math:`\color{green}\checkmark`| :math:`\color{red}\times` | :math:`\color{green}\checkmark` | :math:`\color{red}\times` | :math:`\color{red}\times` |:math:`\color{green}\checkmark`| :math:`\color{red}\times` | +----------------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+-----------------------------------------------------------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------------------------------------------------------+ | Time complexity (per trial) (*) | :math:`O(d)` | :math:`O(dn)` | :math:`O(dn \log n)` | :math:`O(d^3)` | :math:`O(mp^2)` (\*\*\*) | :math:`O(dn)` | :math:`O(n^3)` | :math:`O(n^3)` | :math:`O(d)` | +----------------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+-----------------------------------------------------------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------------------------------------------------------+ | Recommended budgets (#trials) | as many as one likes | number of combinations | 100 – 1000 | 1000 – 10000 | 100 – 10000 | as many as one likes | – 500 | 10 – 100 | number of combinations | | (**) | | | | | | | | | | +----------------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+-----------------------------------------------------------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------------------------------------------------------+ .. note:: :math:`\color{green}\checkmark`: Supports this feature. :math:`\blacktriangle`: Works, but inefficiently. :math:`\color{red}\times`: Causes an error, or has no interface. (*): We assumes that :math:`d` is the dimension of the search space, :math:`n` is the number of finished trials, :math:`m` is the number of objectives, and :math:`p` is the population size (algorithm specific parameter). This table shows the time complexity of the sampling algorithms. We may omit other terms that depend on the implementation in Optuna, including :math:`O(d)` to call the sampling methods and :math:`O(n)` to collect the completed trials. This means that, for example, the actual time complexity of :class:`~optuna.samplers.RandomSampler` is :math:`O(d+n+d) = O(d+n)`. From another perspective, with the exception of :class:`~optuna.samplers.NSGAIISampler`, all time complexity is written for single-objective optimization. (**): (1) The budget depends on the number of parameters and the number of objectives. (2) This budget includes ``n_startup_trials`` if a sampler has ``n_startup_trials`` as one of its arguments. (\*\*\*): This time complexity assumes that the number of population size :math:`p` and the number of parallelization are regular. This means that the number of parallelization should not exceed the number of population size :math:`p`. .. note:: Samplers initialize their random number generators by specifying ``seed`` argument at initialization. However, samplers reseed them when ``n_jobs!=1`` of :func:`optuna.study.Study.optimize` to avoid sampling duplicated parameters by using the same generator. Thus we can hardly reproduce the optimization results with ``n_jobs!=1``. For the same reason, make sure that use either ``seed=None`` or different ``seed`` values among processes with distributed optimization explained in :ref:`distributed` tutorial. .. note:: For float, integer, or categorical parameters, see :ref:`configurations` tutorial. For pruning, see :ref:`pruning` tutorial. For multivariate optimization, see :class:`~optuna.samplers.BaseSampler`. The multivariate optimization is implemented as :func:`~optuna.samplers.BaseSampler.sample_relative` in Optuna. Please check the concrete documents of samplers for more details. For conditional search space, see :ref:`configurations` tutorial and :class:`~optuna.samplers.TPESampler`. The ``group`` option of :class:`~optuna.samplers.TPESampler` allows :class:`~optuna.samplers.TPESampler` to handle the conditional search space. For multi-objective optimization, see :ref:`multi_objective` tutorial. For batch optimization, see :ref:`Batch-Optimization` tutorial. Note that the ``constant_liar`` option of :class:`~optuna.samplers.TPESampler` allows :class:`~optuna.samplers.TPESampler` to handle the batch optimization. For distributed optimization, see :ref:`distributed` tutorial. Note that the ``constant_liar`` option of :class:`~optuna.samplers.TPESampler` allows :class:`~optuna.samplers.TPESampler` to handle the distributed optimization. For constrained optimization, see an `example `__. .. autosummary:: :toctree: generated/ :nosignatures: optuna.samplers.BaseSampler optuna.samplers.GridSampler optuna.samplers.RandomSampler optuna.samplers.TPESampler optuna.samplers.CmaEsSampler optuna.samplers.GPSampler optuna.samplers.PartialFixedSampler optuna.samplers.NSGAIISampler optuna.samplers.NSGAIIISampler optuna.samplers.QMCSampler optuna.samplers.BruteForceSampler .. note:: The following :mod:`optuna.samplers.nsgaii` module defines crossover operations used by :class:`~optuna.samplers.NSGAIISampler`. .. toctree:: :maxdepth: 1 nsgaii optuna-4.1.0/docs/source/reference/samplers/nsgaii.rst000066400000000000000000000010561471332314300230450ustar00rootroot00000000000000.. module:: optuna.samplers.nsgaii optuna.samplers.nsgaii ====================== The :mod:`~optuna.samplers.nsgaii` module defines crossover operations used by :class:`~optuna.samplers.NSGAIISampler`. .. autosummary:: :toctree: generated/ :nosignatures: optuna.samplers.nsgaii.BaseCrossover optuna.samplers.nsgaii.UniformCrossover optuna.samplers.nsgaii.BLXAlphaCrossover optuna.samplers.nsgaii.SPXCrossover optuna.samplers.nsgaii.SBXCrossover optuna.samplers.nsgaii.VSBXCrossover optuna.samplers.nsgaii.UNDXCrossover optuna-4.1.0/docs/source/reference/search_space.rst000066400000000000000000000005221471332314300223620ustar00rootroot00000000000000.. module:: optuna.search_space optuna.search_space =================== The :mod:`~optuna.search_space` module provides functionality for controlling search space of parameters. .. autosummary:: :toctree: generated/ :nosignatures: optuna.search_space.IntersectionSearchSpace optuna.search_space.intersection_search_space optuna-4.1.0/docs/source/reference/storages.rst000066400000000000000000000036331471332314300215770ustar00rootroot00000000000000.. module:: optuna.storages optuna.storages =============== The :mod:`~optuna.storages` module defines a :class:`~optuna.storages.BaseStorage` class which abstracts a backend database and provides library-internal interfaces to the read/write histories of the studies and trials. Library users who wish to use storage solutions other than the default :class:`~optuna.storages.InMemoryStorage` should use one of the child classes of :class:`~optuna.storages.BaseStorage` documented below. .. autosummary:: :toctree: generated/ :nosignatures: optuna.storages.RDBStorage optuna.storages.RetryFailedTrialCallback optuna.storages.fail_stale_trials optuna.storages.JournalStorage optuna.storages.InMemoryStorage optuna.storages.journal ----------------------- :class:`~optuna.storages.JournalStorage` requires its backend specification and here is the list of the supported backends: .. note:: If users would like to use any backends not supported by Optuna, it is possible to do so by creating a customized class by inheriting :class:`optuna.storages.journal.BaseJournalBackend`. .. autosummary:: :toctree: generated/ :nosignatures: optuna.storages.journal.JournalFileBackend optuna.storages.journal.JournalRedisBackend Users can flexibly choose a lock object for :class:`~optuna.storages.journal.JournalFileBackend` and here is the list of supported lock objects: .. autosummary:: :toctree: generated/ :nosignatures: optuna.storages.journal.JournalFileSymlinkLock optuna.storages.journal.JournalFileOpenLock Deprecated Modules ------------------ .. note:: The following modules are deprecated at v4.0.0 and will be removed in the future. Please use the modules defined in :mod:`optuna.storages.journal`. .. autosummary:: :toctree: generated/ :nosignatures: optuna.storages.BaseJournalLogStorage optuna.storages.JournalFileStorage optuna.storages.JournalRedisStorage optuna-4.1.0/docs/source/reference/study.rst000066400000000000000000000014721471332314300211170ustar00rootroot00000000000000.. module:: optuna.study optuna.study ============ The :mod:`~optuna.study` module implements the :class:`~optuna.study.Study` object and related functions. A public constructor is available for the :class:`~optuna.study.Study` class, but direct use of this constructor is not recommended. Instead, library users should create and load a :class:`~optuna.study.Study` using :func:`~optuna.study.create_study` and :func:`~optuna.study.load_study` respectively. .. autosummary:: :toctree: generated/ :nosignatures: optuna.study.Study optuna.study.create_study optuna.study.load_study optuna.study.delete_study optuna.study.copy_study optuna.study.get_all_study_names optuna.study.get_all_study_summaries optuna.study.MaxTrialsCallback optuna.study.StudyDirection optuna.study.StudySummary optuna-4.1.0/docs/source/reference/terminator.rst000066400000000000000000000022471471332314300221340ustar00rootroot00000000000000.. module:: optuna.terminator optuna.terminator ================= The :mod:`~optuna.terminator` module implements a mechanism for automatically terminating the optimization process, accompanied by a callback class for the termination and evaluators for the estimated room for improvement in the optimization and statistical error of the objective function. The terminator stops the optimization process when the estimated potential improvement is smaller than the statistical error. .. autosummary:: :toctree: generated/ :nosignatures: optuna.terminator.BaseTerminator optuna.terminator.Terminator optuna.terminator.BaseImprovementEvaluator optuna.terminator.RegretBoundEvaluator optuna.terminator.BestValueStagnationEvaluator optuna.terminator.EMMREvaluator optuna.terminator.BaseErrorEvaluator optuna.terminator.CrossValidationErrorEvaluator optuna.terminator.StaticErrorEvaluator optuna.terminator.MedianErrorEvaluator optuna.terminator.TerminatorCallback optuna.terminator.report_cross_validation_scores For an example of using this module, please refer to `this example `__. optuna-4.1.0/docs/source/reference/trial.rst000066400000000000000000000014161471332314300210600ustar00rootroot00000000000000.. module:: optuna.trial optuna.trial ============ The :mod:`~optuna.trial` module contains :class:`~optuna.trial.Trial` related classes and functions. A :class:`~optuna.trial.Trial` instance represents a process of evaluating an objective function. This instance is passed to an objective function and provides interfaces to get parameter suggestion, manage the trial's state, and set/get user-defined attributes of the trial, so that Optuna users can define a custom objective function through the interfaces. Basically, Optuna users only use it in their custom objective functions. .. autosummary:: :toctree: generated/ :nosignatures: optuna.trial.Trial optuna.trial.FixedTrial optuna.trial.FrozenTrial optuna.trial.TrialState optuna.trial.create_trial optuna-4.1.0/docs/source/reference/visualization/000077500000000000000000000000001471332314300221125ustar00rootroot00000000000000optuna-4.1.0/docs/source/reference/visualization/index.rst000066400000000000000000000004271471332314300237560ustar00rootroot00000000000000.. include:: ./generated/index.rst .. note:: The following :mod:`optuna.visualization.matplotlib` module uses Matplotlib as a backend. .. toctree:: :maxdepth: 1 matplotlib/index .. seealso:: The :ref:`visualization` tutorial provides use-cases with examples. optuna-4.1.0/docs/source/reference/visualization/matplotlib/000077500000000000000000000000001471332314300242615ustar00rootroot00000000000000optuna-4.1.0/docs/source/reference/visualization/matplotlib/index.rst000066400000000000000000000000431471332314300261170ustar00rootroot00000000000000.. include:: ./generated/index.rst optuna-4.1.0/docs/source/tutorial/000077500000000000000000000000001471332314300171165ustar00rootroot00000000000000optuna-4.1.0/docs/source/tutorial/index.rst000066400000000000000000000031301471332314300207540ustar00rootroot00000000000000:orphan: Tutorial ======== If you are new to Optuna or want a general introduction, we highly recommend the below video. .. raw:: html


Key Features ------------ Showcases Optuna's `Key Features `__. 1. :doc:`10_key_features/001_first` 2. :doc:`10_key_features/002_configurations` 3. :doc:`10_key_features/003_efficient_optimization_algorithms` 4. :doc:`10_key_features/004_distributed` 5. :doc:`10_key_features/005_visualization` Recipes ------- Showcases the recipes that might help you using Optuna with comfort. - :doc:`20_recipes/001_rdb` - :doc:`20_recipes/002_multi_objective` - :doc:`20_recipes/003_attributes` - :doc:`20_recipes/004_cli` - :doc:`20_recipes/005_user_defined_sampler` - :doc:`20_recipes/006_user_defined_pruner` - :doc:`20_recipes/007_optuna_callback` - :doc:`20_recipes/008_specify_params` - :doc:`20_recipes/009_ask_and_tell` - :doc:`20_recipes/010_reuse_best_trial` - :doc:`20_recipes/011_journal_storage` - `Human-in-the-loop Optimization with Optuna Dashboard `__ - :doc:`20_recipes/012_artifact_tutorial` - :doc:`20_recipes/013_wilcoxon_pruner` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `__ optuna-4.1.0/docs/visualization_examples/000077500000000000000000000000001471332314300205525ustar00rootroot00000000000000optuna-4.1.0/docs/visualization_examples/GALLERY_HEADER.rst000066400000000000000000000016661471332314300234240ustar00rootroot00000000000000.. _visualization-examples-index: .. _general_visualization_examples: optuna.visualization ==================== The :mod:`~optuna.visualization` module provides utility functions for plotting the optimization process using plotly and matplotlib. Plotting functions generally take a :class:`~optuna.study.Study` object and optional parameters are passed as a list to the ``params`` argument. .. note:: In the :mod:`optuna.visualization` module, the following functions use plotly to create figures, but `JupyterLab`_ cannot render them by default. Please follow this `installation guide`_ to show figures in `JupyterLab`_. .. note:: The :func:`~optuna.visualization.plot_param_importances` requires the Python package of `scikit-learn `__. .. _JupyterLab: https://github.com/jupyterlab/jupyterlab .. _installation guide: https://github.com/plotly/plotly.py#jupyterlab-support optuna-4.1.0/docs/visualization_examples/optuna.visualization.plot_contour.py000066400000000000000000000010711471332314300300570ustar00rootroot00000000000000""" plot_contour ============ .. autofunction:: optuna.visualization.plot_contour The following code snippet shows how to plot the parameter relationship as contour plot. """ import optuna from plotly.io import show def objective(trial): x = trial.suggest_float("x", -100, 100) y = trial.suggest_categorical("y", [-1, 0, 1]) return x**2 + y sampler = optuna.samplers.TPESampler(seed=10) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=30) fig = optuna.visualization.plot_contour(study, params=["x", "y"]) show(fig) optuna-4.1.0/docs/visualization_examples/optuna.visualization.plot_edf.py000066400000000000000000000022371471332314300271310ustar00rootroot00000000000000""" plot_edf ======== .. autofunction:: optuna.visualization.plot_edf The following code snippet shows how to plot EDF. """ import math import optuna from plotly.io import show def ackley(x, y): a = 20 * math.exp(-0.2 * math.sqrt(0.5 * (x**2 + y**2))) b = math.exp(0.5 * (math.cos(2 * math.pi * x) + math.cos(2 * math.pi * y))) return -a - b + math.e + 20 def objective(trial, low, high): x = trial.suggest_float("x", low, high) y = trial.suggest_float("y", low, high) return ackley(x, y) sampler = optuna.samplers.RandomSampler(seed=10) # Widest search space. study0 = optuna.create_study(study_name="x=[0,5), y=[0,5)", sampler=sampler) study0.optimize(lambda t: objective(t, 0, 5), n_trials=500) # Narrower search space. study1 = optuna.create_study(study_name="x=[0,4), y=[0,4)", sampler=sampler) study1.optimize(lambda t: objective(t, 0, 4), n_trials=500) # Narrowest search space but it doesn't include the global optimum point. study2 = optuna.create_study(study_name="x=[1,3), y=[1,3)", sampler=sampler) study2.optimize(lambda t: objective(t, 1, 3), n_trials=500) fig = optuna.visualization.plot_edf([study0, study1, study2]) show(fig) optuna-4.1.0/docs/visualization_examples/optuna.visualization.plot_hypervolume_history.py000066400000000000000000000012071471332314300325270ustar00rootroot00000000000000""" plot_hypervolume_history ======================== .. autofunction:: optuna.visualization.plot_hypervolume_history The following code snippet shows how to plot optimization history. """ import optuna from plotly.io import show def objective(trial): x = trial.suggest_float("x", 0, 5) y = trial.suggest_float("y", 0, 3) v0 = 4 * x**2 + 4 * y**2 v1 = (x - 5) ** 2 + (y - 5) ** 2 return v0, v1 study = optuna.create_study(directions=["minimize", "minimize"]) study.optimize(objective, n_trials=50) reference_point = [100.0, 50.0] fig = optuna.visualization.plot_hypervolume_history(study, reference_point) show(fig) optuna-4.1.0/docs/visualization_examples/optuna.visualization.plot_intermediate_values.py000066400000000000000000000014461471332314300324250ustar00rootroot00000000000000""" plot_intermediate_values ======================== .. autofunction:: optuna.visualization.plot_intermediate_values The following code snippet shows how to plot intermediate values. """ import optuna from plotly.io import show def f(x): return (x - 2) ** 2 def df(x): return 2 * x - 4 def objective(trial): lr = trial.suggest_float("lr", 1e-5, 1e-1, log=True) x = 3 for step in range(128): y = f(x) trial.report(y, step=step) if trial.should_prune(): raise optuna.TrialPruned() gy = df(x) x -= gy * lr return y sampler = optuna.samplers.TPESampler(seed=10) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=16) fig = optuna.visualization.plot_intermediate_values(study) show(fig) optuna-4.1.0/docs/visualization_examples/optuna.visualization.plot_optimization_history.py000066400000000000000000000011041471332314300326720ustar00rootroot00000000000000""" plot_optimization_history ========================= .. autofunction:: optuna.visualization.plot_optimization_history The following code snippet shows how to plot optimization history. """ import optuna from plotly.io import show def objective(trial): x = trial.suggest_float("x", -100, 100) y = trial.suggest_categorical("y", [-1, 0, 1]) return x**2 + y sampler = optuna.samplers.TPESampler(seed=10) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=10) fig = optuna.visualization.plot_optimization_history(study) show(fig) optuna-4.1.0/docs/visualization_examples/optuna.visualization.plot_parallel_coordinate.py000066400000000000000000000011531471332314300323720ustar00rootroot00000000000000""" plot_parallel_coordinate ======================== .. autofunction:: optuna.visualization.plot_parallel_coordinate The following code snippet shows how to plot the high-dimensional parameter relationships. """ import optuna from plotly.io import show def objective(trial): x = trial.suggest_float("x", -100, 100) y = trial.suggest_categorical("y", [-1, 0, 1]) return x**2 + y sampler = optuna.samplers.TPESampler(seed=10) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=10) fig = optuna.visualization.plot_parallel_coordinate(study, params=["x", "y"]) show(fig) optuna-4.1.0/docs/visualization_examples/optuna.visualization.plot_param_importances.py000066400000000000000000000011511471332314300320710ustar00rootroot00000000000000""" plot_param_importances ====================== .. autofunction:: optuna.visualization.plot_param_importances The following code snippet shows how to plot hyperparameter importances. """ import optuna from plotly.io import show def objective(trial): x = trial.suggest_int("x", 0, 2) y = trial.suggest_float("y", -1.0, 1.0) z = trial.suggest_float("z", 0.0, 1.5) return x**2 + y**3 - z**4 sampler = optuna.samplers.RandomSampler(seed=10) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=100) fig = optuna.visualization.plot_param_importances(study) show(fig) optuna-4.1.0/docs/visualization_examples/optuna.visualization.plot_pareto_front.py000066400000000000000000000025031471332314300310710ustar00rootroot00000000000000""" plot_pareto_front ================= .. autofunction:: optuna.visualization.plot_pareto_front The following code snippet shows how to plot the Pareto front of a study. """ import optuna from plotly.io import show def objective(trial): x = trial.suggest_float("x", 0, 5) y = trial.suggest_float("y", 0, 3) v0 = 4 * x**2 + 4 * y**2 v1 = (x - 5) ** 2 + (y - 5) ** 2 return v0, v1 study = optuna.create_study(directions=["minimize", "minimize"]) study.optimize(objective, n_trials=50) fig = optuna.visualization.plot_pareto_front(study) show(fig) # %% # The following code snippet shows how to plot a 2-dimensional Pareto front # of a 3-dimensional study. # This example is scalable, e.g., for plotting a 2- or 3-dimensional Pareto front # of a 4-dimensional study and so on. import optuna from plotly.io import show def objective(trial): x = trial.suggest_float("x", 0, 5) y = trial.suggest_float("y", 0, 3) v0 = 5 * x**2 + 3 * y**2 v1 = (x - 10) ** 2 + (y - 10) ** 2 v2 = x + y return v0, v1, v2 study = optuna.create_study(directions=["minimize", "minimize", "minimize"]) study.optimize(objective, n_trials=100) fig = optuna.visualization.plot_pareto_front( study, targets=lambda t: (t.values[0], t.values[1]), target_names=["Objective 0", "Objective 1"], ) show(fig) optuna-4.1.0/docs/visualization_examples/optuna.visualization.plot_rank.py000066400000000000000000000013301471332314300273170ustar00rootroot00000000000000""" plot_rank ========= .. autofunction:: optuna.visualization.plot_rank The following code snippet shows how to plot the parameter relationship as a rank plot. """ import optuna from plotly.io import show def objective(trial): x = trial.suggest_float("x", -100, 100) y = trial.suggest_categorical("y", [-1, 0, 1]) c0 = 400 - (x + y) ** 2 trial.set_user_attr("constraint", [c0]) return x**2 + y def constraints(trial): return trial.user_attrs["constraint"] sampler = optuna.samplers.TPESampler(seed=10, constraints_func=constraints) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=30) fig = optuna.visualization.plot_rank(study, params=["x", "y"]) show(fig) optuna-4.1.0/docs/visualization_examples/optuna.visualization.plot_slice.py000066400000000000000000000010571471332314300274710ustar00rootroot00000000000000""" plot_slice ========== .. autofunction:: optuna.visualization.plot_slice The following code snippet shows how to plot the parameter relationship as slice plot. """ import optuna from plotly.io import show def objective(trial): x = trial.suggest_float("x", -100, 100) y = trial.suggest_categorical("y", [-1, 0, 1]) return x**2 + y sampler = optuna.samplers.TPESampler(seed=10) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=10) fig = optuna.visualization.plot_slice(study, params=["x", "y"]) show(fig) optuna-4.1.0/docs/visualization_examples/optuna.visualization.plot_terminator_improvement.py000066400000000000000000000026511471332314300332040ustar00rootroot00000000000000""" plot_terminator_improvement =========================== .. autofunction:: optuna.visualization.plot_terminator_improvement The following code snippet shows how to plot improvement potentials, together with cross-validation errors. """ from lightgbm import LGBMClassifier from sklearn.datasets import load_wine from sklearn.model_selection import cross_val_score from sklearn.model_selection import KFold import optuna from plotly.io import show from optuna.terminator import report_cross_validation_scores from optuna.visualization import plot_terminator_improvement def objective(trial): X, y = load_wine(return_X_y=True) clf = LGBMClassifier( reg_alpha=trial.suggest_float("reg_alpha", 1e-8, 10.0, log=True), reg_lambda=trial.suggest_float("reg_lambda", 1e-8, 10.0, log=True), num_leaves=trial.suggest_int("num_leaves", 2, 256), colsample_bytree=trial.suggest_float("colsample_bytree", 0.4, 1.0), subsample=trial.suggest_float("subsample", 0.4, 1.0), subsample_freq=trial.suggest_int("subsample_freq", 1, 7), min_child_samples=trial.suggest_int("min_child_samples", 5, 100), ) scores = cross_val_score(clf, X, y, cv=KFold(n_splits=5, shuffle=True)) report_cross_validation_scores(trial, scores) return scores.mean() study = optuna.create_study() study.optimize(objective, n_trials=30) fig = plot_terminator_improvement(study, plot_error=True) show(fig) optuna-4.1.0/docs/visualization_examples/optuna.visualization.plot_timeline.py000066400000000000000000000013051471332314300301740ustar00rootroot00000000000000""" plot_timeline ============= .. autofunction:: optuna.visualization.plot_timeline The following code snippet shows how to plot the timeline of a study. Timeline plot can visualize trials with overlapping execution time (e.g., in distributed environments). """ import time import optuna from plotly.io import show def objective(trial): x = trial.suggest_float("x", 0, 1) time.sleep(x * 0.1) if x > 0.8: raise ValueError() if x > 0.4: raise optuna.TrialPruned() return x ** 2 study = optuna.create_study(direction="minimize") study.optimize( objective, n_trials=50, n_jobs=2, catch=(ValueError,) ) fig = optuna.visualization.plot_timeline(study) show(fig) optuna-4.1.0/docs/visualization_matplotlib_examples/000077500000000000000000000000001471332314300230015ustar00rootroot00000000000000optuna-4.1.0/docs/visualization_matplotlib_examples/GALLERY_HEADER.rst000066400000000000000000000003451471332314300256440ustar00rootroot00000000000000.. module:: optuna.visualization.matplotlib matplotlib ========== .. note:: The following functions use Matplotlib as a backend. .. _visualization-matplotlib-examples-index: .. _general_visualization_matplotlib_examples: optuna-4.1.0/docs/visualization_matplotlib_examples/optuna.visualization.matplotlib.contour.py000066400000000000000000000010441471332314300334160ustar00rootroot00000000000000""" plot_contour ============ .. autofunction:: optuna.visualization.matplotlib.plot_contour The following code snippet shows how to plot the parameter relationship as contour plot. """ import optuna def objective(trial): x = trial.suggest_float("x", -100, 100) y = trial.suggest_categorical("y", [-1, 0, 1]) return x**2 + y sampler = optuna.samplers.TPESampler(seed=10) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=30) optuna.visualization.matplotlib.plot_contour(study, params=["x", "y"]) optuna-4.1.0/docs/visualization_matplotlib_examples/optuna.visualization.matplotlib.edf.py000066400000000000000000000022121471332314300324610ustar00rootroot00000000000000""" plot_edf ======== .. autofunction:: optuna.visualization.matplotlib.plot_edf The following code snippet shows how to plot EDF. """ import math import optuna def ackley(x, y): a = 20 * math.exp(-0.2 * math.sqrt(0.5 * (x**2 + y**2))) b = math.exp(0.5 * (math.cos(2 * math.pi * x) + math.cos(2 * math.pi * y))) return -a - b + math.e + 20 def objective(trial, low, high): x = trial.suggest_float("x", low, high) y = trial.suggest_float("y", low, high) return ackley(x, y) sampler = optuna.samplers.RandomSampler(seed=10) # Widest search space. study0 = optuna.create_study(study_name="x=[0,5), y=[0,5)", sampler=sampler) study0.optimize(lambda t: objective(t, 0, 5), n_trials=500) # Narrower search space. study1 = optuna.create_study(study_name="x=[0,4), y=[0,4)", sampler=sampler) study1.optimize(lambda t: objective(t, 0, 4), n_trials=500) # Narrowest search space but it doesn't include the global optimum point. study2 = optuna.create_study(study_name="x=[1,3), y=[1,3)", sampler=sampler) study2.optimize(lambda t: objective(t, 1, 3), n_trials=500) optuna.visualization.matplotlib.plot_edf([study0, study1, study2]) optuna.visualization.matplotlib.hypervolume_history.py000066400000000000000000000012431471332314300360070ustar00rootroot00000000000000optuna-4.1.0/docs/visualization_matplotlib_examples""" plot_hypervolume_history ======================== .. autofunction:: optuna.visualization.matplotlib.plot_hypervolume_history The following code snippet shows how to plot optimization history. """ import optuna import matplotlib.pyplot as plt def objective(trial): x = trial.suggest_float("x", 0, 5) y = trial.suggest_float("y", 0, 3) v0 = 4 * x ** 2 + 4 * y ** 2 v1 = (x - 5) ** 2 + (y - 5) ** 2 return v0, v1 study = optuna.create_study(directions=["minimize", "minimize"]) study.optimize(objective, n_trials=50) reference_point=[100, 50] optuna.visualization.matplotlib.plot_hypervolume_history(study, reference_point) plt.tight_layout() optuna.visualization.matplotlib.intermediate_values.py000066400000000000000000000014211471332314300356760ustar00rootroot00000000000000optuna-4.1.0/docs/visualization_matplotlib_examples""" plot_intermediate_values ======================== .. autofunction:: optuna.visualization.matplotlib.plot_intermediate_values The following code snippet shows how to plot intermediate values. """ import optuna def f(x): return (x - 2) ** 2 def df(x): return 2 * x - 4 def objective(trial): lr = trial.suggest_float("lr", 1e-5, 1e-1, log=True) x = 3 for step in range(128): y = f(x) trial.report(y, step=step) if trial.should_prune(): raise optuna.TrialPruned() gy = df(x) x -= gy * lr return y sampler = optuna.samplers.TPESampler(seed=10) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=16) optuna.visualization.matplotlib.plot_intermediate_values(study) optuna.visualization.matplotlib.optimization_history.py000066400000000000000000000011421471332314300361540ustar00rootroot00000000000000optuna-4.1.0/docs/visualization_matplotlib_examples""" plot_optimization_history ========================= .. autofunction:: optuna.visualization.matplotlib.plot_optimization_history The following code snippet shows how to plot optimization history. """ import optuna import matplotlib.pyplot as plt def objective(trial): x = trial.suggest_float("x", -100, 100) y = trial.suggest_categorical("y", [-1, 0, 1]) return x**2 + y sampler = optuna.samplers.TPESampler(seed=10) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=10) optuna.visualization.matplotlib.plot_optimization_history(study) plt.tight_layout() optuna.visualization.matplotlib.parallel_coordinate.py000066400000000000000000000011261471332314300356520ustar00rootroot00000000000000optuna-4.1.0/docs/visualization_matplotlib_examples""" plot_parallel_coordinate ======================== .. autofunction:: optuna.visualization.matplotlib.plot_parallel_coordinate The following code snippet shows how to plot the high-dimensional parameter relationships. """ import optuna def objective(trial): x = trial.suggest_float("x", -100, 100) y = trial.suggest_categorical("y", [-1, 0, 1]) return x**2 + y sampler = optuna.samplers.TPESampler(seed=10) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=10) optuna.visualization.matplotlib.plot_parallel_coordinate(study, params=["x", "y"]) optuna.visualization.matplotlib.param_importances.py000066400000000000000000000011241471332314300353510ustar00rootroot00000000000000optuna-4.1.0/docs/visualization_matplotlib_examples""" plot_param_importances ====================== .. autofunction:: optuna.visualization.matplotlib.plot_param_importances The following code snippet shows how to plot hyperparameter importances. """ import optuna def objective(trial): x = trial.suggest_int("x", 0, 2) y = trial.suggest_float("y", -1.0, 1.0) z = trial.suggest_float("z", 0.0, 1.5) return x**2 + y**3 - z**4 sampler = optuna.samplers.RandomSampler(seed=10) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=100) optuna.visualization.matplotlib.plot_param_importances(study) optuna-4.1.0/docs/visualization_matplotlib_examples/optuna.visualization.matplotlib.pareto_front.py000066400000000000000000000010541471332314300344300ustar00rootroot00000000000000""" plot_pareto_front ================= .. autofunction:: optuna.visualization.matplotlib.plot_pareto_front The following code snippet shows how to plot the Pareto front of a study. """ import optuna def objective(trial): x = trial.suggest_float("x", 0, 5) y = trial.suggest_float("y", 0, 3) v0 = 4 * x**2 + 4 * y**2 v1 = (x - 5) ** 2 + (y - 5) ** 2 return v0, v1 study = optuna.create_study(directions=["minimize", "minimize"]) study.optimize(objective, n_trials=50) optuna.visualization.matplotlib.plot_pareto_front(study) optuna-4.1.0/docs/visualization_matplotlib_examples/optuna.visualization.matplotlib.rank.py000066400000000000000000000013031471332314300326560ustar00rootroot00000000000000""" plot_rank ========= .. autofunction:: optuna.visualization.matplotlib.plot_rank The following code snippet shows how to plot the parameter relationship as a rank plot. """ import optuna def objective(trial): x = trial.suggest_float("x", -100, 100) y = trial.suggest_categorical("y", [-1, 0, 1]) c0 = 400 - (x + y) ** 2 trial.set_user_attr("constraint", [c0]) return x**2 + y def constraints(trial): return trial.user_attrs["constraint"] sampler = optuna.samplers.TPESampler(seed=10, constraints_func=constraints) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=30) optuna.visualization.matplotlib.plot_rank(study, params=["x", "y"]) optuna-4.1.0/docs/visualization_matplotlib_examples/optuna.visualization.matplotlib.slice.py000066400000000000000000000010341471332314300330230ustar00rootroot00000000000000""" plot_slice ============ .. autofunction:: optuna.visualization.matplotlib.plot_slice The following code snippet shows how to plot the parameter relationship as slice plot. """ import optuna def objective(trial): x = trial.suggest_float("x", -100, 100) y = trial.suggest_categorical("y", [-1, 0, 1]) return x**2 + y sampler = optuna.samplers.TPESampler(seed=10) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=10) optuna.visualization.matplotlib.plot_slice(study, params=["x", "y"]) optuna.visualization.matplotlib.terminator_improvement.py000066400000000000000000000026241471332314300364640ustar00rootroot00000000000000optuna-4.1.0/docs/visualization_matplotlib_examples""" plot_terminator_improvement =========================== .. autofunction:: optuna.visualization.matplotlib.plot_terminator_improvement The following code snippet shows how to plot improvement potentials, together with cross-validation errors. """ from lightgbm import LGBMClassifier from sklearn.datasets import load_wine from sklearn.model_selection import cross_val_score from sklearn.model_selection import KFold import optuna from optuna.terminator import report_cross_validation_scores from optuna.visualization.matplotlib import plot_terminator_improvement def objective(trial): X, y = load_wine(return_X_y=True) clf = LGBMClassifier( reg_alpha=trial.suggest_float("reg_alpha", 1e-8, 10.0, log=True), reg_lambda=trial.suggest_float("reg_lambda", 1e-8, 10.0, log=True), num_leaves=trial.suggest_int("num_leaves", 2, 256), colsample_bytree=trial.suggest_float("colsample_bytree", 0.4, 1.0), subsample=trial.suggest_float("subsample", 0.4, 1.0), subsample_freq=trial.suggest_int("subsample_freq", 1, 7), min_child_samples=trial.suggest_int("min_child_samples", 5, 100), ) scores = cross_val_score(clf, X, y, cv=KFold(n_splits=5, shuffle=True)) report_cross_validation_scores(trial, scores) return scores.mean() study = optuna.create_study() study.optimize(objective, n_trials=30) plot_terminator_improvement(study, plot_error=True) optuna-4.1.0/docs/visualization_matplotlib_examples/optuna.visualization.matplotlib.timeline.py000066400000000000000000000010761471332314300335400ustar00rootroot00000000000000""" plot_timeline ============= .. autofunction:: optuna.visualization.matplotlib.plot_timeline The following code snippet shows how to plot the timeline of a study. """ import time import optuna def objective(trial): x = trial.suggest_float("x", 0, 1) time.sleep(x * 0.1) if x > 0.8: raise ValueError() if x > 0.4: raise optuna.TrialPruned() return x**2 study = optuna.create_study(direction="minimize") study.optimize(objective, n_trials=50, n_jobs=2, catch=(ValueError,)) optuna.visualization.matplotlib.plot_timeline(study) optuna-4.1.0/formats.sh000077500000000000000000000045471471332314300150470ustar00rootroot00000000000000#!/bin/bash # As described in `CONTRIBUTING.md`, this script checks and formats Optuna's source codes by # `black`, `blackdoc`, and `isort`. If you pass `-n` as an option, this script checks codes # without updating codebase. missing_dependencies=() command -v black &> /dev/null if [ $? -eq 1 ] ; then missing_dependencies+=(black) fi command -v blackdoc &> /dev/null if [ $? -eq 1 ] ; then missing_dependencies+=(blackdoc) fi command -v flake8 &> /dev/null if [ $? -eq 1 ] ; then missing_dependencies+=(flake8) fi command -v isort &> /dev/null if [ $? -eq 1 ] ; then missing_dependencies+=(isort) fi command -v mypy &> /dev/null if [ $? -eq 1 ] ; then missing_dependencies+=(mypy) fi if [ ! ${#missing_dependencies[@]} -eq 0 ]; then echo "The following dependencies are missing:" "${missing_dependencies[@]}" read -p "Would you like to install the missing dependencies? (y/N): " yn case "$yn" in [yY]*) ;; *) echo "abort." ; exit ;; esac pip install "${missing_dependencies[@]}" fi update=1 while getopts "n" OPT do case $OPT in n) update=0 ;; *) ;; esac done target="optuna tests benchmarks tutorial" mypy_target="optuna tests benchmarks" res_all=0 res_black=$(black $target --check --diff 2>&1) if [ $? -eq 1 ] ; then if [ $update -eq 1 ] ; then echo "black failed. The code will be formatted by black." black $target else echo "$res_black" echo "black failed." res_all=1 fi else echo "black succeeded." fi res_blackdoc=$(blackdoc $target --check --diff 2>&1) if [ $? -eq 1 ] ; then if [ $update -eq 1 ] ; then echo "blackdoc failed. The docstrings will be formatted by blackdoc." blackdoc $target else echo "$res_blackdoc" echo "blackdoc failed." res_all=1 fi else echo "blackdoc succeeded." fi res_flake8=$(flake8 $target) if [ $? -eq 1 ] ; then echo "$res_flake8" echo "flake8 failed." res_all=1 else echo "flake8 succeeded." fi res_isort=$(isort $target --check 2>&1) if [ $? -eq 1 ] ; then if [ $update -eq 1 ] ; then echo "isort failed. The code will be formatted by isort." isort $target else echo "$res_isort" echo "isort failed." res_all=1 fi else echo "isort succeeded." fi res_mypy=$(mypy $mypy_target) if [ $? -eq 1 ] ; then echo "$res_mypy" echo "mypy failed." res_all=1 else echo "mypy succeeded." fi if [ $res_all -eq 1 ] ; then exit 1 fi optuna-4.1.0/optuna/000077500000000000000000000000001471332314300143315ustar00rootroot00000000000000optuna-4.1.0/optuna/__init__.py000066400000000000000000000025571471332314300164530ustar00rootroot00000000000000from optuna import distributions from optuna import exceptions from optuna import integration from optuna import logging from optuna import pruners from optuna import samplers from optuna import search_space from optuna import storages from optuna import study from optuna import trial from optuna import version from optuna._imports import _LazyImport from optuna.exceptions import TrialPruned from optuna.study import copy_study from optuna.study import create_study from optuna.study import delete_study from optuna.study import get_all_study_names from optuna.study import get_all_study_summaries from optuna.study import load_study from optuna.study import Study from optuna.trial import create_trial from optuna.trial import Trial from optuna.version import __version__ __all__ = [ "Study", "Trial", "TrialPruned", "__version__", "artifacts", "copy_study", "create_study", "create_trial", "delete_study", "distributions", "exceptions", "get_all_study_names", "get_all_study_summaries", "importance", "integration", "load_study", "logging", "pruners", "samplers", "search_space", "storages", "study", "trial", "version", "visualization", ] artifacts = _LazyImport("optuna.artifacts") importance = _LazyImport("optuna.importance") visualization = _LazyImport("optuna.visualization") optuna-4.1.0/optuna/_callbacks.py000066400000000000000000000036251471332314300167670ustar00rootroot00000000000000from typing import Container from typing import Optional import optuna from optuna.trial import FrozenTrial from optuna.trial import TrialState class MaxTrialsCallback: """Set a maximum number of trials before ending the study. While the ``n_trials`` argument of :meth:`optuna.study.Study.optimize` sets the number of trials that will be run, you may want to continue running until you have a certain number of successfully completed trials or stop the study when you have a certain number of trials that fail. This ``MaxTrialsCallback`` class allows you to set a maximum number of trials for a particular :class:`~optuna.trial.TrialState` before stopping the study. Example: .. testcode:: import optuna from optuna.study import MaxTrialsCallback from optuna.trial import TrialState def objective(trial): x = trial.suggest_float("x", -1, 1) return x**2 study = optuna.create_study() study.optimize( objective, callbacks=[MaxTrialsCallback(10, states=(TrialState.COMPLETE,))], ) Args: n_trials: The max number of trials. Must be set to an integer. states: Tuple of the :class:`~optuna.trial.TrialState` to be counted towards the max trials limit. Default value is ``(TrialState.COMPLETE,)``. If :obj:`None`, count all states. """ def __init__( self, n_trials: int, states: Optional[Container[TrialState]] = (TrialState.COMPLETE,) ) -> None: self._n_trials = n_trials self._states = states def __call__(self, study: "optuna.study.Study", trial: FrozenTrial) -> None: trials = study.get_trials(deepcopy=False, states=self._states) n_complete = len(trials) if n_complete >= self._n_trials: study.stop() optuna-4.1.0/optuna/_convert_positional_args.py000066400000000000000000000063241471332314300220040ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Callable from collections.abc import Sequence from functools import wraps from inspect import Parameter from inspect import signature from typing import Any from typing import TYPE_CHECKING from typing import TypeVar import warnings if TYPE_CHECKING: from typing_extensions import ParamSpec _P = ParamSpec("_P") _T = TypeVar("_T") def _get_positional_arg_names(func: "Callable[_P, _T]") -> list[str]: params = signature(func).parameters positional_arg_names = [ name for name, p in params.items() if p.default == Parameter.empty and p.kind == p.POSITIONAL_OR_KEYWORD ] return positional_arg_names def _infer_kwargs(previous_positional_arg_names: Sequence[str], *args: Any) -> dict[str, Any]: inferred_kwargs = {arg_name: val for val, arg_name in zip(args, previous_positional_arg_names)} return inferred_kwargs def convert_positional_args( *, previous_positional_arg_names: Sequence[str], warning_stacklevel: int = 2, ) -> "Callable[[Callable[_P, _T]], Callable[_P, _T]]": """Convert positional arguments to keyword arguments. Args: previous_positional_arg_names: List of names previously given as positional arguments. warning_stacklevel: Level of the stack trace where decorated function locates. """ def converter_decorator(func: "Callable[_P, _T]") -> "Callable[_P, _T]": assert set(previous_positional_arg_names).issubset(set(signature(func).parameters)), ( f"{set(previous_positional_arg_names)} is not a subset of" f" {set(signature(func).parameters)}" ) @wraps(func) def converter_wrapper(*args: Any, **kwargs: Any) -> "_T": positional_arg_names = _get_positional_arg_names(func) inferred_kwargs = _infer_kwargs(previous_positional_arg_names, *args) if len(inferred_kwargs) > len(positional_arg_names): expected_kwds = set(inferred_kwargs) - set(positional_arg_names) warnings.warn( f"{func.__name__}() got {expected_kwds} as positional arguments " "but they were expected to be given as keyword arguments.", FutureWarning, stacklevel=warning_stacklevel, ) if len(args) > len(previous_positional_arg_names): raise TypeError( f"{func.__name__}() takes {len(previous_positional_arg_names)} positional" f" arguments but {len(args)} were given." ) duplicated_kwds = set(kwargs).intersection(inferred_kwargs) if len(duplicated_kwds): # When specifying positional arguments that are not located at the end of args as # keyword arguments, raise TypeError as follows by imitating the Python standard # behavior raise TypeError( f"{func.__name__}() got multiple values for arguments {duplicated_kwds}." ) kwargs.update(inferred_kwargs) return func(**kwargs) # type: ignore[call-arg] return converter_wrapper return converter_decorator optuna-4.1.0/optuna/_deprecated.py000066400000000000000000000154451471332314300171530ustar00rootroot00000000000000import functools import textwrap from typing import Any from typing import Callable from typing import Optional from typing import TYPE_CHECKING from typing import TypeVar import warnings from packaging import version from optuna._experimental import _get_docstring_indent from optuna._experimental import _validate_version if TYPE_CHECKING: from typing_extensions import ParamSpec FT = TypeVar("FT") FP = ParamSpec("FP") CT = TypeVar("CT") _DEPRECATION_NOTE_TEMPLATE = """ .. warning:: Deprecated in v{d_ver}. This feature will be removed in the future. The removal of this feature is currently scheduled for v{r_ver}, but this schedule is subject to change. See https://github.com/optuna/optuna/releases/tag/v{d_ver}. """ _DEPRECATION_WARNING_TEMPLATE = ( "{name} has been deprecated in v{d_ver}. " "This feature will be removed in v{r_ver}. " "See https://github.com/optuna/optuna/releases/tag/v{d_ver}." ) def _validate_two_version(old_version: str, new_version: str) -> None: if version.parse(old_version) > version.parse(new_version): raise ValueError( "Invalid version relationship. The deprecated version must be smaller than " "the removed version, but (deprecated version, removed version) = ({}, {}) are " "specified.".format(old_version, new_version) ) def _format_text(text: str) -> str: return "\n\n" + textwrap.indent(text.strip(), " ") + "\n" def deprecated_func( deprecated_version: str, removed_version: str, name: Optional[str] = None, text: Optional[str] = None, ) -> "Callable[[Callable[FP, FT]], Callable[FP, FT]]": """Decorate function as deprecated. Args: deprecated_version: The version in which the target feature is deprecated. removed_version: The version in which the target feature will be removed. name: The name of the feature. Defaults to the function name. Optional. text: The additional text for the deprecation note. The default note is build using specified ``deprecated_version`` and ``removed_version``. If you want to provide additional information, please specify this argument yourself. .. note:: The default deprecation note is as follows: "Deprecated in v{d_ver}. This feature will be removed in the future. The removal of this feature is currently scheduled for v{r_ver}, but this schedule is subject to change. See https://github.com/optuna/optuna/releases/tag/v{d_ver}." .. note:: The specified text is concatenated after the default deprecation note. """ _validate_version(deprecated_version) _validate_version(removed_version) _validate_two_version(deprecated_version, removed_version) def decorator(func: "Callable[FP, FT]") -> "Callable[FP, FT]": if func.__doc__ is None: func.__doc__ = "" note = _DEPRECATION_NOTE_TEMPLATE.format(d_ver=deprecated_version, r_ver=removed_version) if text is not None: note += _format_text(text) indent = _get_docstring_indent(func.__doc__) func.__doc__ = func.__doc__.strip() + textwrap.indent(note, indent) + indent @functools.wraps(func) def wrapper(*args: Any, **kwargs: Any) -> "FT": """Decorates a function as deprecated. This decorator is supposed to be applied to the deprecated function. """ message = _DEPRECATION_WARNING_TEMPLATE.format( name=(name if name is not None else func.__name__), d_ver=deprecated_version, r_ver=removed_version, ) if text is not None: message += " " + text warnings.warn(message, FutureWarning, stacklevel=2) return func(*args, **kwargs) return wrapper return decorator def deprecated_class( deprecated_version: str, removed_version: str, name: Optional[str] = None, text: Optional[str] = None, ) -> "Callable[[CT], CT]": """Decorate class as deprecated. Args: deprecated_version: The version in which the target feature is deprecated. removed_version: The version in which the target feature will be removed. name: The name of the feature. Defaults to the class name. Optional. text: The additional text for the deprecation note. The default note is build using specified ``deprecated_version`` and ``removed_version``. If you want to provide additional information, please specify this argument yourself. .. note:: The default deprecation note is as follows: "Deprecated in v{d_ver}. This feature will be removed in the future. The removal of this feature is currently scheduled for v{r_ver}, but this schedule is subject to change. See https://github.com/optuna/optuna/releases/tag/v{d_ver}." .. note:: The specified text is concatenated after the default deprecation note. """ _validate_version(deprecated_version) _validate_version(removed_version) _validate_two_version(deprecated_version, removed_version) def decorator(cls: "CT") -> "CT": def wrapper(cls: "CT") -> "CT": """Decorates a class as deprecated. This decorator is supposed to be applied to the deprecated class. """ _original_init = getattr(cls, "__init__") _original_name = getattr(cls, "__name__") @functools.wraps(_original_init) def wrapped_init(self: Any, *args: Any, **kwargs: Any) -> None: message = _DEPRECATION_WARNING_TEMPLATE.format( name=(name if name is not None else _original_name), d_ver=deprecated_version, r_ver=removed_version, ) if text is not None: message += " " + text warnings.warn( message, FutureWarning, stacklevel=2, ) _original_init(self, *args, **kwargs) setattr(cls, "__init__", wrapped_init) if cls.__doc__ is None: cls.__doc__ = "" note = _DEPRECATION_NOTE_TEMPLATE.format( d_ver=deprecated_version, r_ver=removed_version ) if text is not None: note += _format_text(text) indent = _get_docstring_indent(cls.__doc__) cls.__doc__ = cls.__doc__.strip() + textwrap.indent(note, indent) + indent return cls return wrapper(cls) return decorator optuna-4.1.0/optuna/_experimental.py000066400000000000000000000077171471332314300175530ustar00rootroot00000000000000from __future__ import annotations import functools import textwrap from typing import Any from typing import Callable from typing import TYPE_CHECKING from typing import TypeVar import warnings from optuna.exceptions import ExperimentalWarning if TYPE_CHECKING: from typing_extensions import ParamSpec FT = TypeVar("FT") FP = ParamSpec("FP") CT = TypeVar("CT") _EXPERIMENTAL_NOTE_TEMPLATE = """ .. note:: Added in v{ver} as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v{ver}. """ def warn_experimental_argument(option_name: str) -> None: warnings.warn( f"Argument ``{option_name}`` is an experimental feature." " The interface can change in the future.", ExperimentalWarning, ) def _validate_version(version: str) -> None: if not isinstance(version, str) or len(version.split(".")) != 3: raise ValueError( "Invalid version specification. Must follow `x.y.z` format but `{}` is given".format( version ) ) def _get_docstring_indent(docstring: str) -> str: return docstring.split("\n")[-1] if "\n" in docstring else "" def experimental_func( version: str, name: str | None = None, ) -> Callable[[Callable[FP, FT]], Callable[FP, FT]]: """Decorate function as experimental. Args: version: The first version that supports the target feature. name: The name of the feature. Defaults to the function name. Optional. """ _validate_version(version) def decorator(func: Callable[FP, FT]) -> Callable[FP, FT]: if func.__doc__ is None: func.__doc__ = "" note = _EXPERIMENTAL_NOTE_TEMPLATE.format(ver=version) indent = _get_docstring_indent(func.__doc__) func.__doc__ = func.__doc__.strip() + textwrap.indent(note, indent) + indent @functools.wraps(func) def wrapper(*args: Any, **kwargs: Any) -> FT: warnings.warn( "{} is experimental (supported from v{}). " "The interface can change in the future.".format( name if name is not None else func.__name__, version ), ExperimentalWarning, stacklevel=2, ) return func(*args, **kwargs) return wrapper return decorator def experimental_class( version: str, name: str | None = None, ) -> Callable[[CT], CT]: """Decorate class as experimental. Args: version: The first version that supports the target feature. name: The name of the feature. Defaults to the class name. Optional. """ _validate_version(version) def decorator(cls: CT) -> CT: def wrapper(cls: CT) -> CT: """Decorates a class as experimental. This decorator is supposed to be applied to the experimental class. """ _original_init = getattr(cls, "__init__") _original_name = getattr(cls, "__name__") @functools.wraps(_original_init) def wrapped_init(self: Any, *args: Any, **kwargs: Any) -> None: warnings.warn( "{} is experimental (supported from v{}). " "The interface can change in the future.".format( name if name is not None else _original_name, version ), ExperimentalWarning, stacklevel=2, ) _original_init(self, *args, **kwargs) setattr(cls, "__init__", wrapped_init) if cls.__doc__ is None: cls.__doc__ = "" note = _EXPERIMENTAL_NOTE_TEMPLATE.format(ver=version) indent = _get_docstring_indent(cls.__doc__) cls.__doc__ = cls.__doc__.strip() + textwrap.indent(note, indent) + indent return cls return wrapper(cls) return decorator optuna-4.1.0/optuna/_gp/000077500000000000000000000000001471332314300150765ustar00rootroot00000000000000optuna-4.1.0/optuna/_gp/__init__.py000066400000000000000000000000001471332314300171750ustar00rootroot00000000000000optuna-4.1.0/optuna/_gp/acqf.py000066400000000000000000000115471471332314300163720ustar00rootroot00000000000000from __future__ import annotations from dataclasses import dataclass from enum import IntEnum import math from typing import TYPE_CHECKING import numpy as np from optuna._gp.gp import kernel from optuna._gp.gp import KernelParamsTensor from optuna._gp.gp import posterior from optuna._gp.search_space import ScaleType from optuna._gp.search_space import SearchSpace if TYPE_CHECKING: import torch else: from optuna._imports import _LazyImport torch = _LazyImport("torch") def standard_logei(z: torch.Tensor) -> torch.Tensor: # Return E_{x ~ N(0, 1)}[max(0, x+z)] # We switch the implementation depending on the value of z to # avoid numerical instability. small = z < -25 vals = torch.empty_like(z) # Eq. (9) in ref: https://arxiv.org/pdf/2310.20708.pdf # NOTE: We do not use the third condition because ours is good enough. z_small = z[small] z_normal = z[~small] sqrt_2pi = math.sqrt(2 * math.pi) # First condition cdf = 0.5 * torch.special.erfc(-z_normal * math.sqrt(0.5)) pdf = torch.exp(-0.5 * z_normal**2) * (1 / sqrt_2pi) vals[~small] = torch.log(z_normal * cdf + pdf) # Second condition r = math.sqrt(0.5 * math.pi) * torch.special.erfcx(-z_small * math.sqrt(0.5)) vals[small] = -0.5 * z_small**2 + torch.log((z_small * r + 1) * (1 / sqrt_2pi)) return vals def logei(mean: torch.Tensor, var: torch.Tensor, f0: float) -> torch.Tensor: # Return E_{y ~ N(mean, var)}[max(0, y-f0)] sigma = torch.sqrt(var) st_val = standard_logei((mean - f0) / sigma) val = torch.log(sigma) + st_val return val def ucb(mean: torch.Tensor, var: torch.Tensor, beta: float) -> torch.Tensor: return mean + torch.sqrt(beta * var) def lcb(mean: torch.Tensor, var: torch.Tensor, beta: float) -> torch.Tensor: return mean - torch.sqrt(beta * var) # TODO(contramundum53): consider abstraction for acquisition functions. # NOTE: Acquisition function is not class on purpose to integrate numba in the future. class AcquisitionFunctionType(IntEnum): LOG_EI = 0 UCB = 1 LCB = 2 @dataclass(frozen=True) class AcquisitionFunctionParams: acqf_type: AcquisitionFunctionType kernel_params: KernelParamsTensor X: np.ndarray search_space: SearchSpace cov_Y_Y_inv: np.ndarray cov_Y_Y_inv_Y: np.ndarray max_Y: float beta: float | None acqf_stabilizing_noise: float def create_acqf_params( acqf_type: AcquisitionFunctionType, kernel_params: KernelParamsTensor, search_space: SearchSpace, X: np.ndarray, Y: np.ndarray, beta: float | None = None, acqf_stabilizing_noise: float = 1e-12, ) -> AcquisitionFunctionParams: X_tensor = torch.from_numpy(X) is_categorical = torch.from_numpy(search_space.scale_types == ScaleType.CATEGORICAL) with torch.no_grad(): cov_Y_Y = kernel(is_categorical, kernel_params, X_tensor, X_tensor).detach().numpy() cov_Y_Y[np.diag_indices(X.shape[0])] += kernel_params.noise_var.item() cov_Y_Y_inv = np.linalg.inv(cov_Y_Y) return AcquisitionFunctionParams( acqf_type=acqf_type, kernel_params=kernel_params, X=X, search_space=search_space, cov_Y_Y_inv=cov_Y_Y_inv, cov_Y_Y_inv_Y=cov_Y_Y_inv @ Y, max_Y=np.max(Y), beta=beta, acqf_stabilizing_noise=acqf_stabilizing_noise, ) def eval_acqf(acqf_params: AcquisitionFunctionParams, x: torch.Tensor) -> torch.Tensor: mean, var = posterior( acqf_params.kernel_params, torch.from_numpy(acqf_params.X), torch.from_numpy(acqf_params.search_space.scale_types == ScaleType.CATEGORICAL), torch.from_numpy(acqf_params.cov_Y_Y_inv), torch.from_numpy(acqf_params.cov_Y_Y_inv_Y), x, ) if acqf_params.acqf_type == AcquisitionFunctionType.LOG_EI: return logei(mean=mean, var=var + acqf_params.acqf_stabilizing_noise, f0=acqf_params.max_Y) elif acqf_params.acqf_type == AcquisitionFunctionType.UCB: assert acqf_params.beta is not None, "beta must be given to UCB." return ucb(mean=mean, var=var, beta=acqf_params.beta) elif acqf_params.acqf_type == AcquisitionFunctionType.LCB: assert acqf_params.beta is not None, "beta must be given to LCB." return lcb(mean=mean, var=var, beta=acqf_params.beta) else: assert False, "Unknown acquisition function type." def eval_acqf_no_grad(acqf_params: AcquisitionFunctionParams, x: np.ndarray) -> np.ndarray: with torch.no_grad(): return eval_acqf(acqf_params, torch.from_numpy(x)).detach().numpy() def eval_acqf_with_grad( acqf_params: AcquisitionFunctionParams, x: np.ndarray ) -> tuple[float, np.ndarray]: assert x.ndim == 1 x_tensor = torch.from_numpy(x) x_tensor.requires_grad_(True) val = eval_acqf(acqf_params, x_tensor) val.backward() # type: ignore return val.item(), x_tensor.grad.detach().numpy() # type: ignore optuna-4.1.0/optuna/_gp/gp.py000066400000000000000000000251071471332314300160630ustar00rootroot00000000000000from __future__ import annotations from dataclasses import dataclass import math import typing from typing import Callable from typing import TYPE_CHECKING import numpy as np from optuna.logging import get_logger if TYPE_CHECKING: import scipy.optimize as so import torch else: from optuna._imports import _LazyImport so = _LazyImport("scipy.optimize") torch = _LazyImport("torch") logger = get_logger(__name__) # This GP implementation uses the following notation: # X[len(trials), len(params)]: observed parameter values. # Y[len(trials)]: observed objective values. # x[(batch_len,) len(params)]: parameter value to evaluate. Possibly batched. # cov_fX_fX[len(trials), len(trials)]: kernel matrix of X = V[f(X)] # cov_fx_fX[(batch_len,) len(trials)]: kernel matrix of x and X = Cov[f(x), f(X)] # cov_fx_fx: kernel value (scalar) of x = V[f(x)]. # Since we use a Matern 5/2 kernel, we assume this value to be a constant. # cov_Y_Y_inv[len(trials), len(trials)]: inv of the covariance matrix of Y = (V[f(X) + noise])^-1 # cov_Y_Y_inv_Y[len(trials)]: cov_Y_Y_inv @ Y # max_Y: maximum of Y (Note that we transform the objective values such that it is maximized.) # d2: squared distance between two points class Matern52Kernel(torch.autograd.Function): @staticmethod def forward(ctx: typing.Any, squared_distance: torch.Tensor) -> torch.Tensor: # type: ignore sqrt5d = torch.sqrt(5 * squared_distance) exp_part = torch.exp(-sqrt5d) val = exp_part * ((5 / 3) * squared_distance + sqrt5d + 1) # Notice that the derivative is taken w.r.t. d^2, but not w.r.t. d. deriv = (-5 / 6) * (sqrt5d + 1) * exp_part ctx.save_for_backward(deriv) return val @staticmethod def backward(ctx: typing.Any, grad: torch.Tensor) -> torch.Tensor: # type: ignore # Let x be squared_distance, f(x) be forward(ctx, x), and g(f) be a provided function, # then deriv := df/dx, grad := dg/df, and deriv * grad = df/dx * dg/df = dg/dx. (deriv,) = ctx.saved_tensors return deriv * grad def matern52_kernel_from_squared_distance(squared_distance: torch.Tensor) -> torch.Tensor: # sqrt5d = sqrt(5 * squared_distance) # exp(sqrt5d) * (1/3 * sqrt5d ** 2 + sqrt5d + 1) # # We cannot let PyTorch differentiate the above expression because # the gradient runs into 0/0 at squared_distance=0. return Matern52Kernel.apply(squared_distance) # type: ignore @dataclass(frozen=True) class KernelParamsTensor: # Kernel parameters to fit. inverse_squared_lengthscales: torch.Tensor # [len(params)] kernel_scale: torch.Tensor # Scalar noise_var: torch.Tensor # Scalar def kernel( is_categorical: torch.Tensor, # [len(params)] kernel_params: KernelParamsTensor, X1: torch.Tensor, # [...batch_shape, n_A, len(params)] X2: torch.Tensor, # [...batch_shape, n_B, len(params)] ) -> torch.Tensor: # [...batch_shape, n_A, n_B] # kernel(x1, x2) = kernel_scale * matern52_kernel_from_squared_distance( # d2(x1, x2) * inverse_squared_lengthscales) # d2(x1, x2) = sum_i d2_i(x1_i, x2_i) # d2_i(x1_i, x2_i) = (x1_i - x2_i) ** 2 # if x_i is continuous # d2_i(x1_i, x2_i) = 1 if x1_i != x2_i else 0 # if x_i is categorical d2 = (X1[..., :, None, :] - X2[..., None, :, :]) ** 2 # Use the Hamming distance for categorical parameters. d2[..., is_categorical] = (d2[..., is_categorical] > 0.0).type(torch.float64) d2 = (d2 * kernel_params.inverse_squared_lengthscales).sum(dim=-1) return matern52_kernel_from_squared_distance(d2) * kernel_params.kernel_scale def kernel_at_zero_distance( kernel_params: KernelParamsTensor, ) -> torch.Tensor: # [...batch_shape, n_A, n_B] # kernel(x, x) = kernel_scale return kernel_params.kernel_scale def posterior( kernel_params: KernelParamsTensor, X: torch.Tensor, # [len(trials), len(params)] is_categorical: torch.Tensor, # bool[len(params)] cov_Y_Y_inv: torch.Tensor, # [len(trials), len(trials)] cov_Y_Y_inv_Y: torch.Tensor, # [len(trials)] x: torch.Tensor, # [(batch,) len(params)] ) -> tuple[torch.Tensor, torch.Tensor]: # (mean: [(batch,)], var: [(batch,)]) cov_fx_fX = kernel(is_categorical, kernel_params, x[..., None, :], X)[..., 0, :] cov_fx_fx = kernel_at_zero_distance(kernel_params) # mean = cov_fx_fX @ inv(cov_fX_fX + noise * I) @ Y # var = cov_fx_fx - cov_fx_fX @ inv(cov_fX_fX + noise * I) @ cov_fx_fX.T mean = cov_fx_fX @ cov_Y_Y_inv_Y # [batch] var = cov_fx_fx - (cov_fx_fX * (cov_fx_fX @ cov_Y_Y_inv)).sum(dim=-1) # [batch] # We need to clamp the variance to avoid negative values due to numerical errors. return (mean, torch.clamp(var, min=0.0)) def marginal_log_likelihood( X: torch.Tensor, # [len(trials), len(params)] Y: torch.Tensor, # [len(trials)] is_categorical: torch.Tensor, # [len(params)] kernel_params: KernelParamsTensor, ) -> torch.Tensor: # Scalar # -0.5 * log((2pi)^n |C|) - 0.5 * Y^T C^-1 Y, where C^-1 = cov_Y_Y_inv # We apply the cholesky decomposition to efficiently compute log(|C|) and C^-1. cov_fX_fX = kernel(is_categorical, kernel_params, X, X) cov_Y_Y_chol = torch.linalg.cholesky( cov_fX_fX + kernel_params.noise_var * torch.eye(X.shape[0], dtype=torch.float64) ) # log |L| = 0.5 * log|L^T L| = 0.5 * log|C| logdet = 2 * torch.log(torch.diag(cov_Y_Y_chol)).sum() # cov_Y_Y_chol @ cov_Y_Y_chol_inv_Y = Y --> cov_Y_Y_chol_inv_Y = inv(cov_Y_Y_chol) @ Y cov_Y_Y_chol_inv_Y = torch.linalg.solve_triangular(cov_Y_Y_chol, Y[:, None], upper=False)[:, 0] return -0.5 * ( logdet + X.shape[0] * math.log(2 * math.pi) # Y^T C^-1 Y = Y^T inv(L^T L) Y --> cov_Y_Y_chol_inv_Y @ cov_Y_Y_chol_inv_Y + (cov_Y_Y_chol_inv_Y @ cov_Y_Y_chol_inv_Y) ) def _fit_kernel_params( X: np.ndarray, # [len(trials), len(params)] Y: np.ndarray, # [len(trials)] is_categorical: np.ndarray, # [len(params)] log_prior: Callable[[KernelParamsTensor], torch.Tensor], minimum_noise: float, deterministic_objective: bool, initial_kernel_params: KernelParamsTensor, gtol: float, ) -> KernelParamsTensor: n_params = X.shape[1] # We apply log transform to enforce the positivity of the kernel parameters. # Note that we cannot just use the constraint because of the numerical unstability # of the marginal log likelihood. # We also enforce the noise parameter to be greater than `minimum_noise` to avoid # pathological behavior of maximum likelihood estimation. initial_raw_params = np.concatenate( [ np.log(initial_kernel_params.inverse_squared_lengthscales.detach().numpy()), [ np.log(initial_kernel_params.kernel_scale.item()), # We add 0.01 * minimum_noise to initial noise_var to avoid instability. np.log(initial_kernel_params.noise_var.item() - 0.99 * minimum_noise), ], ] ) def loss_func(raw_params: np.ndarray) -> tuple[float, np.ndarray]: raw_params_tensor = torch.from_numpy(raw_params) raw_params_tensor.requires_grad_(True) with torch.enable_grad(): params = KernelParamsTensor( inverse_squared_lengthscales=torch.exp(raw_params_tensor[:n_params]), kernel_scale=torch.exp(raw_params_tensor[n_params]), noise_var=( torch.tensor(minimum_noise, dtype=torch.float64) if deterministic_objective else torch.exp(raw_params_tensor[n_params + 1]) + minimum_noise ), ) loss = -marginal_log_likelihood( torch.from_numpy(X), torch.from_numpy(Y), torch.from_numpy(is_categorical), params ) - log_prior(params) loss.backward() # type: ignore # scipy.minimize requires all the gradients to be zero for termination. raw_noise_var_grad = raw_params_tensor.grad[n_params + 1] # type: ignore assert not deterministic_objective or raw_noise_var_grad == 0 return loss.item(), raw_params_tensor.grad.detach().numpy() # type: ignore # jac=True means loss_func returns the gradient for gradient descent. res = so.minimize( # Too small `gtol` causes instability in loss_func optimization. loss_func, initial_raw_params, jac=True, method="l-bfgs-b", options={"gtol": gtol}, ) if not res.success: raise RuntimeError(f"Optimization failed: {res.message}") raw_params_opt_tensor = torch.from_numpy(res.x) res = KernelParamsTensor( inverse_squared_lengthscales=torch.exp(raw_params_opt_tensor[:n_params]), kernel_scale=torch.exp(raw_params_opt_tensor[n_params]), noise_var=( torch.tensor(minimum_noise, dtype=torch.float64) if deterministic_objective else minimum_noise + torch.exp(raw_params_opt_tensor[n_params + 1]) ), ) return res def fit_kernel_params( X: np.ndarray, Y: np.ndarray, is_categorical: np.ndarray, log_prior: Callable[[KernelParamsTensor], torch.Tensor], minimum_noise: float, deterministic_objective: bool, initial_kernel_params: KernelParamsTensor | None = None, gtol: float = 1e-2, ) -> KernelParamsTensor: default_initial_kernel_params = KernelParamsTensor( inverse_squared_lengthscales=torch.ones(X.shape[1], dtype=torch.float64), kernel_scale=torch.tensor(1.0, dtype=torch.float64), noise_var=torch.tensor(1.0, dtype=torch.float64), ) if initial_kernel_params is None: initial_kernel_params = default_initial_kernel_params error = None # First try optimizing the kernel params with the provided initial_kernel_params, # but if it fails, rerun the optimization with the default initial_kernel_params. # This increases the robustness of the optimization. for init_kernel_params in [initial_kernel_params, default_initial_kernel_params]: try: return _fit_kernel_params( X=X, Y=Y, is_categorical=is_categorical, log_prior=log_prior, minimum_noise=minimum_noise, initial_kernel_params=init_kernel_params, deterministic_objective=deterministic_objective, gtol=gtol, ) except RuntimeError as e: error = e logger.warn( f"The optimization of kernel_params failed: \n{error}\n" "The default initial kernel params will be used instead." ) return default_initial_kernel_params optuna-4.1.0/optuna/_gp/optim_mixed.py000066400000000000000000000277541471332314300200050ustar00rootroot00000000000000from __future__ import annotations import math from typing import TYPE_CHECKING import numpy as np from optuna._gp.acqf import AcquisitionFunctionParams from optuna._gp.acqf import eval_acqf_no_grad from optuna._gp.acqf import eval_acqf_with_grad from optuna._gp.search_space import normalize_one_param from optuna._gp.search_space import sample_normalized_params from optuna._gp.search_space import ScaleType from optuna.logging import get_logger if TYPE_CHECKING: import scipy.optimize as so else: from optuna import _LazyImport so = _LazyImport("scipy.optimize") _logger = get_logger(__name__) def _gradient_ascent( acqf_params: AcquisitionFunctionParams, initial_params: np.ndarray, initial_fval: float, continuous_indices: np.ndarray, lengthscales: np.ndarray, tol: float, ) -> tuple[np.ndarray, float, bool]: """ This function optimizes the acquisition function using preconditioning. Preconditioning equalizes the variances caused by each parameter and speeds up the convergence. In Optuna, acquisition functions use Matern 5/2 kernel, which is a function of `x / l` where `x` is `normalized_params` and `l` is the corresponding lengthscales. Then acquisition functions are a function of `x / l`, i.e. `f(x / l)`. As `l` has different values for each param, it makes the function ill-conditioned. By transforming `x / l` to `zl / l = z`, the function becomes `f(z)` and has equal variances w.r.t. `z`. So optimization w.r.t. `z` instead of `x` is the preconditioning here and speeds up the convergence. As the domain of `x` is [0, 1], that of `z` becomes [0, 1/l]. """ if len(continuous_indices) == 0: return (initial_params, initial_fval, False) normalized_params = initial_params.copy() def negative_acqf_with_grad(scaled_x: np.ndarray) -> tuple[float, np.ndarray]: # Scale back to the original domain, i.e. [0, 1], from [0, 1/s]. normalized_params[continuous_indices] = scaled_x * lengthscales (fval, grad) = eval_acqf_with_grad(acqf_params, normalized_params) # Flip sign because scipy minimizes functions. # Let the scaled acqf be g(x) and the acqf be f(sx), then dg/dx = df/dx * s. return (-fval, -grad[continuous_indices] * lengthscales) scaled_cont_x_opt, neg_fval_opt, info = so.fmin_l_bfgs_b( func=negative_acqf_with_grad, x0=normalized_params[continuous_indices] / lengthscales, bounds=[(0, 1 / s) for s in lengthscales], pgtol=math.sqrt(tol), maxiter=200, ) if -neg_fval_opt > initial_fval and info["nit"] > 0: # Improved. # `nit` is the number of iterations. normalized_params[continuous_indices] = scaled_cont_x_opt * lengthscales return (normalized_params, -neg_fval_opt, True) return (initial_params, initial_fval, False) # No improvement. def _exhaustive_search( acqf_params: AcquisitionFunctionParams, initial_params: np.ndarray, initial_fval: float, param_idx: int, choices: np.ndarray, ) -> tuple[np.ndarray, float, bool]: choices_except_current = choices[choices != initial_params[param_idx]] all_params = np.repeat(initial_params[None, :], len(choices_except_current), axis=0) all_params[:, param_idx] = choices_except_current fvals = eval_acqf_no_grad(acqf_params, all_params) best_idx = np.argmax(fvals) if fvals[best_idx] > initial_fval: # Improved. return (all_params[best_idx, :], fvals[best_idx], True) return (initial_params, initial_fval, False) # No improvement. def _discrete_line_search( acqf_params: AcquisitionFunctionParams, initial_params: np.ndarray, initial_fval: float, param_idx: int, grids: np.ndarray, xtol: float, ) -> tuple[np.ndarray, float, bool]: if len(grids) == 1: # Do not optimize anything when there's only one choice. return (initial_params, initial_fval, False) def find_nearest_index(x: float) -> int: i = int(np.clip(np.searchsorted(grids, x), 1, len(grids) - 1)) return i - 1 if abs(x - grids[i - 1]) < abs(x - grids[i]) else i current_choice_i = find_nearest_index(initial_params[param_idx]) assert np.isclose(initial_params[param_idx], grids[current_choice_i]) negative_fval_cache = {current_choice_i: -initial_fval} normalized_params = initial_params.copy() def negative_acqf_with_cache(i: int) -> float: # Function value at choices[i]. cache_val = negative_fval_cache.get(i) if cache_val is not None: return cache_val normalized_params[param_idx] = grids[i] # Flip sign because scipy minimizes functions. negval = -float(eval_acqf_no_grad(acqf_params, normalized_params)) negative_fval_cache[i] = negval return negval def interpolated_negative_acqf(x: float) -> float: if x < grids[0] or x > grids[-1]: return np.inf right = int(np.clip(np.searchsorted(grids, x), 1, len(grids) - 1)) left = right - 1 neg_acqf_left, neg_acqf_right = negative_acqf_with_cache(left), negative_acqf_with_cache( right ) w_left = (grids[right] - x) / (grids[right] - grids[left]) w_right = 1.0 - w_left return w_left * neg_acqf_left + w_right * neg_acqf_right EPS = 1e-12 res = so.minimize_scalar( interpolated_negative_acqf, # The values of this bracket are (inf, -fval, inf). # This trivially satisfies the bracket condition if fval is finite. bracket=(grids[0] - EPS, grids[current_choice_i], grids[-1] + EPS), method="brent", tol=xtol, ) opt_idx = find_nearest_index(res.x) fval_opt = -negative_acqf_with_cache(opt_idx) # We check both conditions because of numerical errors. if opt_idx != current_choice_i and fval_opt > initial_fval: normalized_params[param_idx] = grids[opt_idx] return (normalized_params, fval_opt, True) return (initial_params, initial_fval, False) # No improvement. def _local_search_discrete( acqf_params: AcquisitionFunctionParams, initial_params: np.ndarray, initial_fval: float, param_idx: int, choices: np.ndarray, xtol: float, ) -> tuple[np.ndarray, float, bool]: # If the number of possible parameter values is small, we just perform an exhaustive search. # This is faster and better than the line search. MAX_INT_EXHAUSTIVE_SEARCH_PARAMS = 16 scale_type = acqf_params.search_space.scale_types[param_idx] if scale_type == ScaleType.CATEGORICAL or len(choices) <= MAX_INT_EXHAUSTIVE_SEARCH_PARAMS: return _exhaustive_search(acqf_params, initial_params, initial_fval, param_idx, choices) else: return _discrete_line_search( acqf_params, initial_params, initial_fval, param_idx, choices, xtol ) def local_search_mixed( acqf_params: AcquisitionFunctionParams, initial_normalized_params: np.ndarray, *, tol: float = 1e-4, max_iter: int = 100, ) -> tuple[np.ndarray, float]: scale_types = acqf_params.search_space.scale_types bounds = acqf_params.search_space.bounds steps = acqf_params.search_space.steps continuous_indices = np.where(steps == 0.0)[0] inverse_squared_lengthscales = ( acqf_params.kernel_params.inverse_squared_lengthscales.detach().numpy() ) # This is a technique for speeding up optimization. # We use an isotropic kernel, so scaling the gradient will make # the hessian better-conditioned. lengthscales = 1 / np.sqrt(inverse_squared_lengthscales[continuous_indices]) discrete_indices = np.where(steps > 0)[0] choices_of_discrete_params = [ ( np.arange(bounds[i, 1]) if scale_types[i] == ScaleType.CATEGORICAL else normalize_one_param( param_value=np.arange(bounds[i, 0], bounds[i, 1] + 0.5 * steps[i], steps[i]), scale_type=ScaleType(scale_types[i]), bounds=(bounds[i, 0], bounds[i, 1]), step=steps[i], ) ) for i in discrete_indices ] discrete_xtols = [ # Terminate discrete optimizations once the change in x becomes smaller than this. # Basically, if the change is smaller than min(dx) / 4, it is useless to see more details. np.min(np.diff(choices), initial=np.inf) / 4 for choices in choices_of_discrete_params ] best_normalized_params = initial_normalized_params.copy() best_fval = float(eval_acqf_no_grad(acqf_params, best_normalized_params)) CONTINUOUS = -1 last_changed_param: int | None = None for _ in range(max_iter): if last_changed_param == CONTINUOUS: # Parameters not changed since last time. return (best_normalized_params, best_fval) (best_normalized_params, best_fval, updated) = _gradient_ascent( acqf_params, best_normalized_params, best_fval, continuous_indices, lengthscales, tol, ) if updated: last_changed_param = CONTINUOUS for i, choices, xtol in zip(discrete_indices, choices_of_discrete_params, discrete_xtols): if last_changed_param == i: # Parameters not changed since last time. return (best_normalized_params, best_fval) (best_normalized_params, best_fval, updated) = _local_search_discrete( acqf_params, best_normalized_params, best_fval, i, choices, xtol ) if updated: last_changed_param = i if last_changed_param is None: # Parameters not changed from the beginning. return (best_normalized_params, best_fval) _logger.warning("local_search_mixed: Local search did not converge.") return (best_normalized_params, best_fval) def optimize_acqf_mixed( acqf_params: AcquisitionFunctionParams, *, warmstart_normalized_params_array: np.ndarray | None = None, n_preliminary_samples: int = 2048, n_local_search: int = 10, tol: float = 1e-4, rng: np.random.RandomState | None = None, ) -> tuple[np.ndarray, float]: rng = rng or np.random.RandomState() dim = acqf_params.search_space.scale_types.shape[0] if warmstart_normalized_params_array is None: warmstart_normalized_params_array = np.empty((0, dim)) assert ( len(warmstart_normalized_params_array) <= n_local_search - 1 ), "We must choose at least 1 best sampled point + given_initial_xs as start points." sampled_xs = sample_normalized_params(n_preliminary_samples, acqf_params.search_space, rng=rng) # Evaluate all values at initial samples f_vals = eval_acqf_no_grad(acqf_params, sampled_xs) assert isinstance(f_vals, np.ndarray) max_i = np.argmax(f_vals) # We use a modified roulette wheel selection to pick the initial param for each local search. probs = np.exp(f_vals - f_vals[max_i]) probs[max_i] = 0.0 # We already picked the best param, so remove it from roulette. probs /= probs.sum() n_non_zero_probs_improvement = np.count_nonzero(probs > 0.0) # n_additional_warmstart becomes smaller when study starts to converge. n_additional_warmstart = min( n_local_search - len(warmstart_normalized_params_array) - 1, n_non_zero_probs_improvement ) if n_additional_warmstart == n_non_zero_probs_improvement: _logger.warning("Study already converged, so the number of local search is reduced.") chosen_idxs = np.array([max_i]) if n_additional_warmstart > 0: additional_idxs = rng.choice( len(sampled_xs), size=n_additional_warmstart, replace=False, p=probs ) chosen_idxs = np.append(chosen_idxs, additional_idxs) best_x = sampled_xs[max_i, :] best_f = float(f_vals[max_i]) for x_warmstart in np.vstack([sampled_xs[chosen_idxs, :], warmstart_normalized_params_array]): x, f = local_search_mixed(acqf_params, x_warmstart, tol=tol) if f > best_f: best_x = x best_f = f return best_x, best_f optuna-4.1.0/optuna/_gp/optim_sample.py000066400000000000000000000011011471332314300201320ustar00rootroot00000000000000from __future__ import annotations import numpy as np from optuna._gp import acqf from optuna._gp.search_space import sample_normalized_params def optimize_acqf_sample( acqf_params: acqf.AcquisitionFunctionParams, *, n_samples: int = 2048, rng: np.random.RandomState | None = None, ) -> tuple[np.ndarray, float]: # Normalized parameter values are sampled. xs = sample_normalized_params(n_samples, acqf_params.search_space, rng=rng) res = acqf.eval_acqf_no_grad(acqf_params, xs) best_i = np.argmax(res) return xs[best_i, :], res[best_i] optuna-4.1.0/optuna/_gp/prior.py000066400000000000000000000021601471332314300166020ustar00rootroot00000000000000from __future__ import annotations from typing import TYPE_CHECKING from optuna._gp import gp if TYPE_CHECKING: import torch else: from optuna._imports import _LazyImport torch = _LazyImport("torch") DEFAULT_MINIMUM_NOISE_VAR = 1e-6 def default_log_prior(kernel_params: "gp.KernelParamsTensor") -> "torch.Tensor": # Log of prior distribution of kernel parameters. def gamma_log_prior(x: "torch.Tensor", concentration: float, rate: float) -> "torch.Tensor": # We omit the constant factor `rate ** concentration / Gamma(concentration)`. return (concentration - 1) * torch.log(x) - rate * x # NOTE(contramundum53): The priors below (params and function # shape for inverse_squared_lengthscales) were picked by heuristics. # TODO(contramundum53): Check whether these priors are appropriate. return ( -( 0.1 / kernel_params.inverse_squared_lengthscales + 0.1 * kernel_params.inverse_squared_lengthscales ).sum() + gamma_log_prior(kernel_params.kernel_scale, 2, 1) + gamma_log_prior(kernel_params.noise_var, 1.1, 30) ) optuna-4.1.0/optuna/_gp/search_space.py000066400000000000000000000144301471332314300200720ustar00rootroot00000000000000from __future__ import annotations from dataclasses import dataclass from enum import IntEnum import math import threading from typing import Any from typing import TYPE_CHECKING import numpy as np from optuna.distributions import BaseDistribution from optuna.distributions import CategoricalDistribution from optuna.distributions import FloatDistribution from optuna.distributions import IntDistribution from optuna.trial import FrozenTrial if TYPE_CHECKING: import scipy.stats.qmc as qmc else: from optuna._imports import _LazyImport qmc = _LazyImport("scipy.stats.qmc") _threading_lock = threading.Lock() class ScaleType(IntEnum): LINEAR = 0 LOG = 1 CATEGORICAL = 2 @dataclass(frozen=True) class SearchSpace: scale_types: np.ndarray bounds: np.ndarray steps: np.ndarray def unnormalize_one_param( param_value: np.ndarray, scale_type: ScaleType, bounds: tuple[float, float], step: float ) -> np.ndarray: # param_value can be batched, or not. if scale_type == ScaleType.CATEGORICAL: return param_value low, high = (bounds[0] - 0.5 * step, bounds[1] + 0.5 * step) if scale_type == ScaleType.LOG: low, high = (math.log(low), math.log(high)) param_value = param_value * (high - low) + low if scale_type == ScaleType.LOG: param_value = np.exp(param_value) return param_value def normalize_one_param( param_value: np.ndarray, scale_type: ScaleType, bounds: tuple[float, float], step: float ) -> np.ndarray: # param_value can be batched, or not. if scale_type == ScaleType.CATEGORICAL: return param_value low, high = (bounds[0] - 0.5 * step, bounds[1] + 0.5 * step) if scale_type == ScaleType.LOG: low, high = (math.log(low), math.log(high)) param_value = np.log(param_value) if high == low: return np.full_like(param_value, 0.5) param_value = (param_value - low) / (high - low) return param_value def round_one_normalized_param( param_value: np.ndarray, scale_type: ScaleType, bounds: tuple[float, float], step: float ) -> np.ndarray: assert scale_type != ScaleType.CATEGORICAL if step == 0.0: return param_value param_value = unnormalize_one_param(param_value, scale_type, bounds, step) param_value = np.clip( (param_value - bounds[0] + 0.5 * step) // step * step + bounds[0], bounds[0], bounds[1], ) param_value = normalize_one_param(param_value, scale_type, bounds, step) return param_value def sample_normalized_params( n: int, search_space: SearchSpace, rng: np.random.RandomState | None ) -> np.ndarray: rng = rng or np.random.RandomState() dim = search_space.scale_types.shape[0] scale_types = search_space.scale_types bounds = search_space.bounds steps = search_space.steps # Sobol engine likely shares its internal state among threads. # Without threading.Lock, ValueError exceptions are raised in Sobol engine as discussed in # https://github.com/optuna/optunahub-registry/pull/168#pullrequestreview-2404054969 with _threading_lock: qmc_engine = qmc.Sobol(dim, scramble=True, seed=rng.randint(np.iinfo(np.int32).max)) param_values = qmc_engine.random(n) for i in range(dim): if scale_types[i] == ScaleType.CATEGORICAL: param_values[:, i] = np.floor(param_values[:, i] * bounds[i, 1]) elif steps[i] != 0.0: param_values[:, i] = round_one_normalized_param( param_values[:, i], scale_types[i], (bounds[i, 0], bounds[i, 1]), steps[i] ) return param_values def get_search_space_and_normalized_params( trials: list[FrozenTrial], optuna_search_space: dict[str, BaseDistribution], ) -> tuple[SearchSpace, np.ndarray]: scale_types = np.zeros(len(optuna_search_space), dtype=np.int64) bounds = np.zeros((len(optuna_search_space), 2), dtype=np.float64) steps = np.zeros(len(optuna_search_space), dtype=np.float64) values = np.zeros((len(trials), len(optuna_search_space)), dtype=np.float64) for i, (param, distribution) in enumerate(optuna_search_space.items()): if isinstance(distribution, CategoricalDistribution): scale_types[i] = ScaleType.CATEGORICAL bounds[i, :] = (0.0, len(distribution.choices)) steps[i] = 1.0 values[:, i] = np.array( [distribution.to_internal_repr(trial.params[param]) for trial in trials] ) else: assert isinstance( distribution, ( FloatDistribution, IntDistribution, ), ) scale_types[i] = ScaleType.LOG if distribution.log else ScaleType.LINEAR steps[i] = 0.0 if distribution.step is None else distribution.step bounds[i, :] = (distribution.low, distribution.high) values[:, i] = normalize_one_param( np.array([trial.params[param] for trial in trials]), scale_types[i], (bounds[i, 0], bounds[i, 1]), steps[i], ) return SearchSpace(scale_types, bounds, steps), values def get_unnormalized_param( optuna_search_space: dict[str, BaseDistribution], normalized_param: np.ndarray, ) -> dict[str, Any]: ret = {} for i, (param, distribution) in enumerate(optuna_search_space.items()): if isinstance(distribution, CategoricalDistribution): ret[param] = distribution.to_external_repr(normalized_param[i]) else: assert isinstance( distribution, ( FloatDistribution, IntDistribution, ), ) scale_type = ScaleType.LOG if distribution.log else ScaleType.LINEAR step = 0.0 if distribution.step is None else distribution.step bounds = (distribution.low, distribution.high) param_value = float( np.clip( unnormalize_one_param(normalized_param[i], scale_type, bounds, step), distribution.low, distribution.high, ) ) if isinstance(distribution, IntDistribution): param_value = round(param_value) ret[param] = param_value return ret optuna-4.1.0/optuna/_hypervolume/000077500000000000000000000000001471332314300170475ustar00rootroot00000000000000optuna-4.1.0/optuna/_hypervolume/__init__.py000066400000000000000000000002341471332314300211570ustar00rootroot00000000000000from optuna._hypervolume.hssp import _solve_hssp from optuna._hypervolume.wfg import compute_hypervolume __all__ = ["_solve_hssp", "compute_hypervolume"] optuna-4.1.0/optuna/_hypervolume/hssp.py000066400000000000000000000142011471332314300203740ustar00rootroot00000000000000from __future__ import annotations import numpy as np import optuna def _solve_hssp_2d( rank_i_loss_vals: np.ndarray, rank_i_indices: np.ndarray, subset_size: int, reference_point: np.ndarray, ) -> np.ndarray: # This function can be used for non-unique rank_i_loss_vals as well. # The time complexity is O(subset_size * rank_i_loss_vals.shape[0]). assert rank_i_loss_vals.shape[-1] == 2 and subset_size <= rank_i_loss_vals.shape[0] n_trials = rank_i_loss_vals.shape[0] # rank_i_loss_vals is unique-lexsorted in solve_hssp. sorted_indices = np.arange(rank_i_loss_vals.shape[0]) sorted_loss_vals = rank_i_loss_vals.copy() # The diagonal points for each rectangular to calculate the hypervolume contributions. rect_diags = np.repeat(reference_point[np.newaxis, :], n_trials, axis=0) selected_indices = np.zeros(subset_size, dtype=int) for i in range(subset_size): contribs = np.prod(rect_diags - sorted_loss_vals, axis=-1) max_index = np.argmax(contribs) selected_indices[i] = rank_i_indices[sorted_indices[max_index]] loss_vals = sorted_loss_vals[max_index].copy() keep = np.ones(n_trials - i, dtype=bool) keep[max_index] = False # Remove the chosen point. sorted_indices = sorted_indices[keep] rect_diags = rect_diags[keep] sorted_loss_vals = sorted_loss_vals[keep] # Update the diagonal points for each hypervolume contribution calculation. rect_diags[:max_index, 0] = np.minimum(loss_vals[0], rect_diags[:max_index, 0]) rect_diags[max_index:, 1] = np.minimum(loss_vals[1], rect_diags[max_index:, 1]) return selected_indices def _lazy_contribs_update( contribs: np.ndarray, pareto_loss_values: np.ndarray, selected_vecs: np.ndarray, reference_point: np.ndarray, ) -> np.ndarray: """Lazy update the hypervolume contributions. S=selected_indices - {indices[max_index]}, T=selected_indices, and S' is a subset of S. As we would like to know argmax H(T v {i}) in the next iteration, we can skip HV calculations for j if H(T v {i}) - H(T) > H(S' v {j}) - H(S') >= H(T v {j}) - H(T). We used the submodularity for the inequality above. As the upper bound of contribs[i] is H(S' v {j}) - H(S'), we start to update from i with a higher upper bound so that we can skip more HV calculations. """ hv_selected = optuna._hypervolume.compute_hypervolume( selected_vecs[:-1], reference_point, assume_pareto=True ) max_contrib = 0.0 index_from_larger_upper_bound_contrib = np.argsort(-contribs) for i in index_from_larger_upper_bound_contrib: if contribs[i] < max_contrib: # Lazy evaluation to reduce HV calculations. # If contribs[i] will not be the maximum next, it is unnecessary to compute it. continue selected_vecs[-1] = pareto_loss_values[i].copy() hv_plus = optuna._hypervolume.compute_hypervolume( selected_vecs, reference_point, assume_pareto=True ) # inf - inf in the contribution calculation is always inf. contribs[i] = hv_plus - hv_selected if not np.isinf(hv_plus) else np.inf max_contrib = max(contribs[i], max_contrib) return contribs def _solve_hssp_on_unique_loss_vals( rank_i_loss_vals: np.ndarray, rank_i_indices: np.ndarray, subset_size: int, reference_point: np.ndarray, ) -> np.ndarray: if not np.isfinite(reference_point).all(): return rank_i_indices[:subset_size] if rank_i_indices.size == subset_size: return rank_i_indices if rank_i_loss_vals.shape[-1] == 2: return _solve_hssp_2d(rank_i_loss_vals, rank_i_indices, subset_size, reference_point) # The following logic can be used for non-unique rank_i_loss_vals as well. diff_of_loss_vals_and_ref_point = reference_point - rank_i_loss_vals assert subset_size <= rank_i_indices.size n_objectives = reference_point.size contribs = np.prod(diff_of_loss_vals_and_ref_point, axis=-1) selected_indices = np.zeros(subset_size, dtype=int) selected_vecs = np.empty((subset_size, n_objectives)) indices = np.arange(rank_i_loss_vals.shape[0], dtype=int) for k in range(subset_size): max_index = int(np.argmax(contribs)) selected_indices[k] = indices[max_index] selected_vecs[k] = rank_i_loss_vals[max_index].copy() keep = np.ones(contribs.size, dtype=bool) keep[max_index] = False contribs = contribs[keep] indices = indices[keep] rank_i_loss_vals = rank_i_loss_vals[keep] if k == subset_size - 1: # We do not need to update contribs at the last iteration. break contribs = _lazy_contribs_update( contribs, rank_i_loss_vals, selected_vecs[: k + 2], reference_point ) return rank_i_indices[selected_indices] def _solve_hssp( rank_i_loss_vals: np.ndarray, rank_i_indices: np.ndarray, subset_size: int, reference_point: np.ndarray, ) -> np.ndarray: """Solve a hypervolume subset selection problem (HSSP) via a greedy algorithm. This method is a 1-1/e approximation algorithm to solve HSSP. For further information about algorithms to solve HSSP, please refer to the following paper: - `Greedy Hypervolume Subset Selection in Low Dimensions `__ """ if subset_size == rank_i_indices.size: return rank_i_indices rank_i_unique_loss_vals, indices_of_unique_loss_vals = np.unique( rank_i_loss_vals, return_index=True, axis=0 ) n_unique = indices_of_unique_loss_vals.size if n_unique < subset_size: chosen = np.zeros(rank_i_indices.size, dtype=bool) chosen[indices_of_unique_loss_vals] = True duplicated_indices = np.arange(rank_i_indices.size)[~chosen] chosen[duplicated_indices[: subset_size - n_unique]] = True return rank_i_indices[chosen] selected_indices_of_unique_loss_vals = _solve_hssp_on_unique_loss_vals( rank_i_unique_loss_vals, indices_of_unique_loss_vals, subset_size, reference_point ) return rank_i_indices[selected_indices_of_unique_loss_vals] optuna-4.1.0/optuna/_hypervolume/wfg.py000066400000000000000000000142231471332314300202060ustar00rootroot00000000000000from __future__ import annotations import numpy as np from optuna.study._multi_objective import _is_pareto_front def _compute_2d(sorted_pareto_sols: np.ndarray, reference_point: np.ndarray) -> float: assert sorted_pareto_sols.shape[1] == 2 and reference_point.shape[0] == 2 rect_diag_y = np.append(reference_point[1], sorted_pareto_sols[:-1, 1]) edge_length_x = reference_point[0] - sorted_pareto_sols[:, 0] edge_length_y = rect_diag_y - sorted_pareto_sols[:, 1] return edge_length_x @ edge_length_y def _compute_hv(sorted_loss_vals: np.ndarray, reference_point: np.ndarray) -> float: inclusive_hvs = np.prod(reference_point - sorted_loss_vals, axis=-1) if inclusive_hvs.shape[0] == 1: return float(inclusive_hvs[0]) elif inclusive_hvs.shape[0] == 2: # S(A v B) = S(A) + S(B) - S(A ^ B). intersec = np.prod(reference_point - np.maximum(sorted_loss_vals[0], sorted_loss_vals[1])) return np.sum(inclusive_hvs) - intersec # c.f. Eqs. (6) and (7) of ``A Fast Way of Calculating Exact Hypervolumes``. limited_sols_array = np.maximum(sorted_loss_vals[:, np.newaxis], sorted_loss_vals) return sum( _compute_exclusive_hv(limited_sols_array[i, i + 1 :], inclusive_hv, reference_point) for i, inclusive_hv in enumerate(inclusive_hvs) ) def _compute_exclusive_hv( limited_sols: np.ndarray, inclusive_hv: float, reference_point: np.ndarray ) -> float: if limited_sols.shape[0] == 0: return inclusive_hv # NOTE(nabenabe): As the following line is a hack for speedup, I will describe several # important points to note. Even if we do not run _is_pareto_front below or use # assume_unique_lexsorted=False instead, the result of this function does not change, but this # function simply becomes slower. # # For simplicity, I call an array ``quasi-lexsorted`` if it is sorted by the first objective. # # Reason why it will be faster with _is_pareto_front # Hypervolume of a given solution set and a reference point does not change even when we # remove non Pareto solutions from the solution set. However, the calculation becomes slower # if the solution set contains many non Pareto solutions. By removing some obvious non Pareto # solutions, the calculation becomes faster. # # Reason why assume_unique_lexsorted must be True for _is_pareto_front # assume_unique_lexsorted=True actually checks weak dominance and solutions will be weakly # dominated if there are duplications, so we can remove duplicated solutions by this option. # In other words, assume_unique_lexsorted=False may significantly slow down when limited_sols # has many duplicated Pareto solutions because this function becomes an exponential algorithm # without duplication removal. # # NOTE(nabenabe): limited_sols can be non-unique and/or non-lexsorted, so I will describe why # it is fine. # # Reason why we can specify assume_unique_lexsorted=True even when limited_sols is not # All ``False`` in on_front will be correct (, but it may not be the case for ``True``) even # if limited_sols is not unique or not lexsorted as long as limited_sols is quasi-lexsorted, # which is guaranteed. As mentioned earlier, if all ``False`` in on_front is correct, the # result of this function does not change. on_front = _is_pareto_front(limited_sols, assume_unique_lexsorted=True) return inclusive_hv - _compute_hv(limited_sols[on_front], reference_point) def compute_hypervolume( loss_vals: np.ndarray, reference_point: np.ndarray, assume_pareto: bool = False ) -> float: """Hypervolume calculator for any dimension. This class exactly calculates the hypervolume for any dimension. For 3 dimensions or higher, the WFG algorithm will be used. Please refer to ``A Fast Way of Calculating Exact Hypervolumes`` for the WFG algorithm. .. note:: This class is used for computing the hypervolumes of points in multi-objective space. Each coordinate of each point represents a ``values`` of the multi-objective function. .. note:: We check that each objective is to be minimized. Transform objective values that are to be maximized before calling this class's ``compute`` method. Args: loss_vals: An array of loss value vectors to calculate the hypervolume. reference_point: The reference point used to calculate the hypervolume. assume_pareto: Whether to assume the Pareto optimality to ``loss_vals``. In other words, if ``True``, none of loss vectors are dominated by another. ``assume_pareto`` is used only for speedup and it does not change the result even if this argument is wrongly given. If there are many non-Pareto solutions in ``loss_vals``, ``assume_pareto=True`` will speed up the calculation. Returns: The hypervolume of the given arguments. """ if not np.all(loss_vals <= reference_point): raise ValueError( "All points must dominate or equal the reference point. " "That is, for all points in the loss_vals and the coordinate `i`, " "`loss_vals[i] <= reference_point[i]`." ) if not np.all(np.isfinite(reference_point)): # reference_point does not have nan, thanks to the verification above. return float("inf") if not assume_pareto: unique_lexsorted_loss_vals = np.unique(loss_vals, axis=0) on_front = _is_pareto_front(unique_lexsorted_loss_vals, assume_unique_lexsorted=True) sorted_pareto_sols = unique_lexsorted_loss_vals[on_front] else: # NOTE(nabenabe): The result of this function does not change both by # np.argsort(loss_vals[:, 0]) and np.unique(loss_vals, axis=0). # But many duplications in loss_vals significantly slows down the function. # TODO(nabenabe): Make an option to use np.unique. sorted_pareto_sols = loss_vals[loss_vals[:, 0].argsort()] if reference_point.shape[0] == 2: return _compute_2d(sorted_pareto_sols, reference_point) return _compute_hv(sorted_pareto_sols, reference_point) optuna-4.1.0/optuna/_imports.py000066400000000000000000000101411471332314300165340ustar00rootroot00000000000000from __future__ import annotations import importlib import types from types import TracebackType from typing import Any from typing import Type _INTEGRATION_IMPORT_ERROR_TEMPLATE = ( "\nCould not find `optuna-integration` for `{0}`.\n" "Please run `pip install optuna-integration[{0}]`." ) class _DeferredImportExceptionContextManager: """Context manager to defer exceptions from imports. Catches :exc:`ImportError` and :exc:`SyntaxError`. If any exception is caught, this class raises an :exc:`ImportError` when being checked. """ def __init__(self) -> None: self._deferred: tuple[Exception, str] | None = None def __enter__(self) -> "_DeferredImportExceptionContextManager": """Enter the context manager. Returns: Itself. """ return self def __exit__( self, exc_type: Type[Exception] | None, exc_value: Exception | None, traceback: TracebackType | None, ) -> bool | None: """Exit the context manager. Args: exc_type: Raised exception type. :obj:`None` if nothing is raised. exc_value: Raised exception object. :obj:`None` if nothing is raised. traceback: Associated traceback. :obj:`None` if nothing is raised. Returns: :obj:`None` if nothing is deferred, otherwise :obj:`True`. :obj:`True` will suppress any exceptions avoiding them from propagating. """ if isinstance(exc_value, (ImportError, SyntaxError)): if isinstance(exc_value, ImportError): message = ( "Tried to import '{}' but failed. Please make sure that the package is " "installed correctly to use this feature. Actual error: {}." ).format(exc_value.name, exc_value) elif isinstance(exc_value, SyntaxError): message = ( "Tried to import a package but failed due to a syntax error in {}. Please " "make sure that the Python version is correct to use this feature. Actual " "error: {}." ).format(exc_value.filename, exc_value) else: assert False self._deferred = (exc_value, message) return True return None def is_successful(self) -> bool: """Return whether the context manager has caught any exceptions. Returns: :obj:`True` if no exceptions are caught, :obj:`False` otherwise. """ return self._deferred is None def check(self) -> None: """Check whether the context manager has caught any exceptions. Raises: :exc:`ImportError`: If any exception was caught from the caught exception. """ if self._deferred is not None: exc_value, message = self._deferred raise ImportError(message) from exc_value def try_import() -> _DeferredImportExceptionContextManager: """Create a context manager that can wrap imports of optional packages to defer exceptions. Returns: Deferred import context manager. """ return _DeferredImportExceptionContextManager() class _LazyImport(types.ModuleType): """Module wrapper for lazy import. This class wraps the specified modules and lazily imports them only when accessed. Otherwise, `import optuna` is slowed down by importing all submodules and dependencies even if not required. Within this project's usage, importlib override this module's attribute on the first access and the imported submodule is directly accessed from the second access. Args: name: Name of module to apply lazy import. """ def __init__(self, name: str) -> None: super().__init__(name) self._name = name def _load(self) -> types.ModuleType: module = importlib.import_module(self._name) self.__dict__.update(module.__dict__) return module def __getattr__(self, item: str) -> Any: return getattr(self._load(), item) optuna-4.1.0/optuna/_transform.py000066400000000000000000000273331471332314300170650ustar00rootroot00000000000000import math from typing import Any from typing import Dict from typing import List from typing import Tuple from typing import Union import numpy as np from optuna.distributions import BaseDistribution from optuna.distributions import CategoricalDistribution from optuna.distributions import FloatDistribution from optuna.distributions import IntDistribution class _SearchSpaceTransform: """Transform a search space and parameter configurations to continuous space. The search space bounds and parameter configurations are represented as ``numpy.ndarray``s and transformed into continuous space. Bounds and parameters associated with categorical distributions are one-hot encoded. Parameter configurations in this space can additionally be untransformed, or mapped back to the original space. This type of transformation/untransformation is useful for e.g. implementing samplers without having to condition on distribution types before sampling parameter values. Args: search_space: The search space. If any transformations are to be applied, parameter configurations are assumed to hold parameter values for all of the distributions defined in this search space. Otherwise, assertion failures will be raised. transform_log: If :obj:`True`, apply log/exp operations to the bounds and parameters with corresponding distributions in log space during transformation/untransformation. Should always be :obj:`True` if any parameters are going to be sampled from the transformed space. transform_step: If :obj:`True`, offset the lower and higher bounds by a half step each, increasing the space by one step. This allows fair sampling for values close to the bounds. Should always be :obj:`True` if any parameters are going to be sampled from the transformed space. transform_0_1: If :obj:`True`, apply a linear transformation to the bounds and parameters so that they are in the unit cube. Attributes: bounds: Constructed bounds from the given search space. column_to_encoded_columns: Constructed mapping from original parameter column index to encoded column indices. encoded_column_to_column: Constructed mapping from encoded column index to original parameter column index. Note: Parameter values are not scaled to the unit cube. Note: ``transform_log`` and ``transform_step`` are useful for constructing bounds and parameters without any actual transformations by setting those arguments to :obj:`False`. This is needed for e.g. the hyperparameter importance assessments. """ def __init__( self, search_space: Dict[str, BaseDistribution], transform_log: bool = True, transform_step: bool = True, transform_0_1: bool = False, ) -> None: bounds, column_to_encoded_columns, encoded_column_to_column = _transform_search_space( search_space, transform_log, transform_step ) self._raw_bounds = bounds self._column_to_encoded_columns = column_to_encoded_columns self._encoded_column_to_column = encoded_column_to_column self._search_space = search_space self._transform_log = transform_log self._transform_0_1 = transform_0_1 @property def bounds(self) -> np.ndarray: if self._transform_0_1: return np.array([[0.0, 1.0]] * self._raw_bounds.shape[0]) else: return self._raw_bounds @property def column_to_encoded_columns(self) -> List[np.ndarray]: return self._column_to_encoded_columns @property def encoded_column_to_column(self) -> np.ndarray: return self._encoded_column_to_column def transform(self, params: Dict[str, Any]) -> np.ndarray: """Transform a parameter configuration from actual values to continuous space. Args: params: A parameter configuration to transform. Returns: A 1-dimensional ``numpy.ndarray`` holding the transformed parameters in the configuration. """ trans_params = np.zeros(self._raw_bounds.shape[0], dtype=np.float64) bound_idx = 0 for name, distribution in self._search_space.items(): assert name in params, "Parameter configuration must contain all distributions." param = params[name] if isinstance(distribution, CategoricalDistribution): choice_idx = distribution.to_internal_repr(param) trans_params[bound_idx + choice_idx] = 1 bound_idx += len(distribution.choices) else: trans_params[bound_idx] = _transform_numerical_param( param, distribution, self._transform_log ) bound_idx += 1 if self._transform_0_1: single_mask = self._raw_bounds[:, 0] == self._raw_bounds[:, 1] trans_params[single_mask] = 0.5 trans_params[~single_mask] = ( trans_params[~single_mask] - self._raw_bounds[~single_mask, 0] ) / (self._raw_bounds[~single_mask, 1] - self._raw_bounds[~single_mask, 0]) return trans_params def untransform(self, trans_params: np.ndarray) -> Dict[str, Any]: """Untransform a parameter configuration from continuous space to actual values. Args: trans_params: A 1-dimensional ``numpy.ndarray`` in the transformed space corresponding to a parameter configuration. Returns: A dictionary of an untransformed parameter configuration. Keys are parameter names. Values are untransformed parameter values. """ assert trans_params.shape == (self._raw_bounds.shape[0],) if self._transform_0_1: trans_params = self._raw_bounds[:, 0] + trans_params * ( self._raw_bounds[:, 1] - self._raw_bounds[:, 0] ) params = {} for (name, distribution), encoded_columns in zip( self._search_space.items(), self.column_to_encoded_columns ): trans_param = trans_params[encoded_columns] if isinstance(distribution, CategoricalDistribution): # Select the highest rated one-hot encoding. param = distribution.to_external_repr(trans_param.argmax()) else: param = _untransform_numerical_param( trans_param.item(), distribution, self._transform_log ) params[name] = param return params def _transform_search_space( search_space: Dict[str, BaseDistribution], transform_log: bool, transform_step: bool ) -> Tuple[np.ndarray, List[np.ndarray], np.ndarray]: assert len(search_space) > 0, "Cannot transform if no distributions are given." n_bounds = sum( len(d.choices) if isinstance(d, CategoricalDistribution) else 1 for d in search_space.values() ) bounds = np.empty((n_bounds, 2), dtype=np.float64) column_to_encoded_columns: List[np.ndarray] = [] encoded_column_to_column = np.empty(n_bounds, dtype=np.int64) bound_idx = 0 for distribution in search_space.values(): d = distribution if isinstance(d, CategoricalDistribution): n_choices = len(d.choices) bounds[bound_idx : bound_idx + n_choices] = (0, 1) # Broadcast across all choices. encoded_columns = np.arange(bound_idx, bound_idx + n_choices) encoded_column_to_column[encoded_columns] = len(column_to_encoded_columns) column_to_encoded_columns.append(encoded_columns) bound_idx += n_choices elif isinstance( d, ( FloatDistribution, IntDistribution, ), ): if isinstance(d, FloatDistribution): if d.step is not None: half_step = 0.5 * d.step if transform_step else 0.0 bds = ( _transform_numerical_param(d.low, d, transform_log) - half_step, _transform_numerical_param(d.high, d, transform_log) + half_step, ) else: bds = ( _transform_numerical_param(d.low, d, transform_log), _transform_numerical_param(d.high, d, transform_log), ) elif isinstance(d, IntDistribution): half_step = 0.5 * d.step if transform_step else 0.0 if d.log: bds = ( _transform_numerical_param(d.low - half_step, d, transform_log), _transform_numerical_param(d.high + half_step, d, transform_log), ) else: bds = ( _transform_numerical_param(d.low, d, transform_log) - half_step, _transform_numerical_param(d.high, d, transform_log) + half_step, ) else: assert False, "Should not reach. Unexpected distribution." bounds[bound_idx] = bds encoded_column = np.atleast_1d(bound_idx) encoded_column_to_column[encoded_column] = len(column_to_encoded_columns) column_to_encoded_columns.append(encoded_column) bound_idx += 1 else: assert False, "Should not reach. Unexpected distribution." assert bound_idx == n_bounds return bounds, column_to_encoded_columns, encoded_column_to_column def _transform_numerical_param( param: Union[int, float], distribution: BaseDistribution, transform_log: bool ) -> float: d = distribution if isinstance(d, CategoricalDistribution): assert False, "Should not reach. Should be one-hot encoded." elif isinstance(d, FloatDistribution): if d.log: trans_param = math.log(param) if transform_log else float(param) else: trans_param = float(param) elif isinstance(d, IntDistribution): if d.log: trans_param = math.log(param) if transform_log else float(param) else: trans_param = float(param) else: assert False, "Should not reach. Unexpected distribution." return trans_param def _untransform_numerical_param( trans_param: float, distribution: BaseDistribution, transform_log: bool ) -> Union[int, float]: d = distribution if isinstance(d, CategoricalDistribution): assert False, "Should not reach. Should be one-hot encoded." elif isinstance(d, FloatDistribution): if d.log: param = math.exp(trans_param) if transform_log else trans_param if d.single(): pass else: param = min(param, np.nextafter(d.high, d.high - 1)) elif d.step is not None: param = float( np.clip(np.round((trans_param - d.low) / d.step) * d.step + d.low, d.low, d.high) ) else: if d.single(): param = trans_param else: param = min(trans_param, np.nextafter(d.high, d.high - 1)) elif isinstance(d, IntDistribution): if d.log: if transform_log: param = int(np.clip(np.round(math.exp(trans_param)), d.low, d.high)) else: param = int(trans_param) else: param = int( np.clip(np.round((trans_param - d.low) / d.step) * d.step + d.low, d.low, d.high) ) else: assert False, "Should not reach. Unexpected distribution." return param optuna-4.1.0/optuna/_typing.py000066400000000000000000000004531471332314300163560ustar00rootroot00000000000000from __future__ import annotations from typing import Mapping from typing import Sequence from typing import Union JSONSerializable = Union[ Mapping[str, "JSONSerializable"], Sequence["JSONSerializable"], str, int, float, bool, None, ] __all__ = ["JSONSerializable"] optuna-4.1.0/optuna/artifacts/000077500000000000000000000000001471332314300163115ustar00rootroot00000000000000optuna-4.1.0/optuna/artifacts/__init__.py000066400000000000000000000012211471332314300204160ustar00rootroot00000000000000from optuna.artifacts._backoff import Backoff from optuna.artifacts._boto3 import Boto3ArtifactStore from optuna.artifacts._download import download_artifact from optuna.artifacts._filesystem import FileSystemArtifactStore from optuna.artifacts._gcs import GCSArtifactStore from optuna.artifacts._list_artifact_meta import get_all_artifact_meta from optuna.artifacts._upload import ArtifactMeta from optuna.artifacts._upload import upload_artifact __all__ = [ "ArtifactMeta", "FileSystemArtifactStore", "Boto3ArtifactStore", "GCSArtifactStore", "Backoff", "get_all_artifact_meta", "upload_artifact", "download_artifact", ] optuna-4.1.0/optuna/artifacts/_backoff.py000066400000000000000000000072041471332314300204200ustar00rootroot00000000000000from __future__ import annotations import logging import time from typing import TYPE_CHECKING from optuna.artifacts.exceptions import ArtifactNotFound _logger = logging.getLogger(__name__) if TYPE_CHECKING: from typing import BinaryIO from optuna.artifacts._protocol import ArtifactStore class Backoff: """An artifact store's middleware for exponential backoff. Example: .. code-block:: python import optuna from optuna.artifacts import upload_artifact from optuna.artifacts import Boto3ArtifactStore from optuna.artifacts import Backoff artifact_store = Backoff(Boto3ArtifactStore("my-bucket")) def objective(trial: optuna.Trial) -> float: ... = trial.suggest_float("x", -10, 10) file_path = generate_example(...) upload_artifact( artifact_store=artifact_store, file_path=file_path, study_or_trial=trial, ) return ... """ def __init__( self, backend: ArtifactStore, *, max_retries: int = 10, multiplier: float = 2, min_delay: float = 0.1, max_delay: float = 30, ) -> None: # Default sleep seconds: # 0.1, 0.2, 0.4, 0.8, 1.6, 3.2, 6.4, 12.8, 25.6, 30 self._backend = backend assert max_retries > 0 assert multiplier > 0 assert min_delay > 0 assert max_delay > min_delay self._max_retries = max_retries self._multiplier = multiplier self._min_delay = min_delay self._max_delay = max_delay def _get_sleep_secs(self, n_retry: int) -> float: return min(self._min_delay * self._multiplier**n_retry, self._max_delay) def open_reader(self, artifact_id: str) -> BinaryIO: for i in range(self._max_retries): try: return self._backend.open_reader(artifact_id) except ArtifactNotFound: raise except Exception as e: if i == self._max_retries - 1: raise else: _logger.error(f"Failed to open artifact={artifact_id} n_retry={i}", exc_info=e) time.sleep(self._get_sleep_secs(i)) assert False, "must not reach here" def write(self, artifact_id: str, content_body: BinaryIO) -> None: for i in range(self._max_retries): try: self._backend.write(artifact_id, content_body) break except ArtifactNotFound: raise except Exception as e: if i == self._max_retries - 1: raise else: _logger.error(f"Failed to open artifact={artifact_id} n_retry={i}", exc_info=e) content_body.seek(0) time.sleep(self._get_sleep_secs(i)) def remove(self, artifact_id: str) -> None: for i in range(self._max_retries): try: self._backend.remove(artifact_id) except ArtifactNotFound: raise except Exception as e: if i == self._max_retries - 1: raise else: _logger.error(f"Failed to delete artifact={artifact_id}", exc_info=e) time.sleep(self._get_sleep_secs(i)) if TYPE_CHECKING: # A mypy-runtime assertion to ensure that the Backoff middleware implements # all abstract methods in ArtifactStore. from optuna.artifacts import FileSystemArtifactStore _: ArtifactStore = Backoff(FileSystemArtifactStore(".")) optuna-4.1.0/optuna/artifacts/_boto3.py000066400000000000000000000067221471332314300200570ustar00rootroot00000000000000from __future__ import annotations import io import shutil from typing import TYPE_CHECKING from optuna._imports import try_import from optuna.artifacts.exceptions import ArtifactNotFound if TYPE_CHECKING: from typing import BinaryIO from mypy_boto3_s3 import S3Client with try_import() as _imports: import boto3 from botocore.exceptions import ClientError class Boto3ArtifactStore: """An artifact backend for Boto3. Args: bucket_name: The name of the bucket to store artifacts. client: A Boto3 client to use for storage operations. If not specified, a new client will be created. avoid_buf_copy: If True, skip procedure to copy the content of the source file object to a buffer before uploading it to S3 ins. This is default to False because using ``upload_fileobj()`` method of Boto3 client might close the source file object. Example: .. code-block:: python import optuna from optuna.artifacts import upload_artifact from optuna.artifacts import Boto3ArtifactStore artifact_store = Boto3ArtifactStore("my-bucket") def objective(trial: optuna.Trial) -> float: ... = trial.suggest_float("x", -10, 10) file_path = generate_example(...) upload_artifact( artifact_store=artifact_store, file_path=file_path, study_or_trial=trial, ) return ... """ def __init__( self, bucket_name: str, client: S3Client | None = None, *, avoid_buf_copy: bool = False ) -> None: _imports.check() self.bucket = bucket_name self.client = client or boto3.client("s3") # This flag is added to avoid that upload_fileobj() method of Boto3 client may close the # source file object. See https://github.com/boto/boto3/issues/929. self._avoid_buf_copy = avoid_buf_copy def open_reader(self, artifact_id: str) -> BinaryIO: try: obj = self.client.get_object(Bucket=self.bucket, Key=artifact_id) except ClientError as e: if _is_not_found_error(e): raise ArtifactNotFound( f"Artifact storage with bucket: {self.bucket}, artifact_id: {artifact_id} was" " not found" ) from e raise body = obj.get("Body") assert body is not None return body def write(self, artifact_id: str, content_body: BinaryIO) -> None: fsrc: BinaryIO = content_body if not self._avoid_buf_copy: buf = io.BytesIO() shutil.copyfileobj(content_body, buf) buf.seek(0) fsrc = buf self.client.upload_fileobj(fsrc, self.bucket, artifact_id) def remove(self, artifact_id: str) -> None: self.client.delete_object(Bucket=self.bucket, Key=artifact_id) def _is_not_found_error(e: ClientError) -> bool: error_code = e.response.get("Error", {}).get("Code") http_status_code = e.response.get("ResponseMetadata", {}).get("HTTPStatusCode") return error_code == "NoSuchKey" or http_status_code == 404 if TYPE_CHECKING: # A mypy-runtime assertion to ensure that Boto3ArtifactStore implements all abstract methods # in ArtifactStore. from optuna.artifacts._protocol import ArtifactStore _: ArtifactStore = Boto3ArtifactStore("") optuna-4.1.0/optuna/artifacts/_download.py000066400000000000000000000013301471332314300206260ustar00rootroot00000000000000from __future__ import annotations import os import shutil from optuna.artifacts._protocol import ArtifactStore def download_artifact(*, artifact_store: ArtifactStore, file_path: str, artifact_id: str) -> None: """Download an artifact from the artifact store. Args: artifact_store: An artifact store. file_path: A path to save the downloaded artifact. artifact_id: The identifier of the artifact to download. """ if os.path.exists(file_path): raise FileExistsError(f"File already exists: {file_path}") with artifact_store.open_reader(artifact_id) as reader, open(file_path, "wb") as writer: shutil.copyfileobj(reader, writer) optuna-4.1.0/optuna/artifacts/_filesystem.py000066400000000000000000000045601471332314300212130ustar00rootroot00000000000000from __future__ import annotations import os from pathlib import Path import shutil from typing import TYPE_CHECKING from optuna.artifacts.exceptions import ArtifactNotFound if TYPE_CHECKING: from typing import BinaryIO class FileSystemArtifactStore: """An artifact store for file systems. Args: base_path: The base path to a directory to store artifacts. Example: .. code-block:: python import os import optuna from optuna.artifacts import FileSystemArtifactStore from optuna.artifacts import upload_artifact base_path = "./artifacts" os.makedirs(base_path, exist_ok=True) artifact_store = FileSystemArtifactStore(base_path=base_path) def objective(trial: optuna.Trial) -> float: ... = trial.suggest_float("x", -10, 10) file_path = generate_example(...) upload_artifact( artifact_store=artifact_store, file_path=file_path, study_or_trial=trial, ) return ... """ def __init__(self, base_path: str | Path) -> None: if isinstance(base_path, str): base_path = Path(base_path) # TODO(Shinichi): Check if the base_path is valid directory. self._base_path = base_path def open_reader(self, artifact_id: str) -> BinaryIO: filepath = os.path.join(self._base_path, artifact_id) try: f = open(filepath, "rb") except FileNotFoundError as e: raise ArtifactNotFound("not found") from e return f def write(self, artifact_id: str, content_body: BinaryIO) -> None: filepath = os.path.join(self._base_path, artifact_id) with open(filepath, "wb") as f: shutil.copyfileobj(content_body, f) def remove(self, artifact_id: str) -> None: filepath = os.path.join(self._base_path, artifact_id) try: os.remove(filepath) except FileNotFoundError as e: raise ArtifactNotFound("not found") from e if TYPE_CHECKING: # A mypy-runtime assertion to ensure that LocalArtifactBackend # implements all abstract methods in ArtifactBackendProtocol. from optuna.artifacts._protocol import ArtifactStore _: ArtifactStore = FileSystemArtifactStore("") optuna-4.1.0/optuna/artifacts/_gcs.py000066400000000000000000000053741471332314300176070ustar00rootroot00000000000000from __future__ import annotations from io import BytesIO from typing import TYPE_CHECKING from optuna._experimental import experimental_class from optuna._imports import try_import from optuna.artifacts.exceptions import ArtifactNotFound if TYPE_CHECKING: from typing import BinaryIO with try_import() as _imports: import google.cloud.storage @experimental_class("3.4.0") class GCSArtifactStore: """An artifact backend for Google Cloud Storage (GCS). Args: bucket_name: The name of the bucket to store artifacts. client: A google-cloud-storage ``Client`` to use for storage operations. If not specified, a new client will be created with default settings. Example: .. code-block:: python import optuna from optuna.artifacts import GCSArtifactStore, upload_artifact artifact_backend = GCSArtifactStore("my-bucket") def objective(trial: optuna.Trial) -> float: ... = trial.suggest_float("x", -10, 10) file_path = generate_example(...) upload_artifact( artifact_store=artifact_store, file_path=file_path, study_or_trial=trial, ) return ... Before running this code, you will have to install ``gcloud`` and run .. code-block:: bash gcloud auth application-default login so that the Cloud Storage library can automatically find the credential. """ def __init__( self, bucket_name: str, client: google.cloud.storage.Client | None = None, ) -> None: _imports.check() self.bucket_name = bucket_name self.client = client or google.cloud.storage.Client() self.bucket_obj = self.client.bucket(bucket_name) def open_reader(self, artifact_id: str) -> "BinaryIO": blob = self.bucket_obj.get_blob(artifact_id) if blob is None: raise ArtifactNotFound( f"Artifact storage with bucket: {self.bucket_name}, artifact_id: {artifact_id} was" " not found" ) body = blob.download_as_bytes() return BytesIO(body) def write(self, artifact_id: str, content_body: "BinaryIO") -> None: blob = self.bucket_obj.blob(artifact_id) data = content_body.read() blob.upload_from_string(data) def remove(self, artifact_id: str) -> None: self.bucket_obj.delete_blob(artifact_id) if TYPE_CHECKING: # A mypy-runtime assertion to ensure that GCS3ArtifactStore implements all abstract methods # in ArtifactStore. from optuna.artifacts._protocol import ArtifactStore _: ArtifactStore = GCSArtifactStore("") optuna-4.1.0/optuna/artifacts/_list_artifact_meta.py000066400000000000000000000072611471332314300226660ustar00rootroot00000000000000from __future__ import annotations import json from optuna.artifacts._upload import ArtifactMeta from optuna.artifacts._upload import ARTIFACTS_ATTR_PREFIX from optuna.storages import BaseStorage from optuna.study import Study from optuna.trial import FrozenTrial from optuna.trial import Trial def get_all_artifact_meta( study_or_trial: Trial | FrozenTrial | Study, *, storage: BaseStorage | None = None ) -> list[ArtifactMeta]: """List the associated artifact information of the provided trial or study. Args: study_or_trial: A :class:`~optuna.trial.Trial` object, a :class:`~optuna.trial.FrozenTrial`, or a :class:`~optuna.study.Study` object. storage: A storage object. This argument is required only if ``study_or_trial`` is :class:`~optuna.trial.FrozenTrial`. Example: An example where this function is useful: .. code:: import os import optuna # Get the storage that contains the study of interest. storage = optuna.storages.get_storage(storage=...) # Instantiate the artifact store used for the study. # Optuna does not provide the API that stores the used artifact store information, so # please manage the information in the user side. artifact_store = ... # Load study that contains the artifacts of interest. study = optuna.load_study(study_name=..., storage=storage) # Fetch the best trial. best_trial = study.best_trial # Fetch all the artifact meta connected to the best trial. artifact_metas = optuna.artifacts.get_all_artifact_meta(best_trial, storage=storage) download_dir_path = "./best_trial_artifacts/" os.makedirs(download_dir_path, exist_ok=True) for artifact_meta in artifact_metas: download_file_path = os.path.join(download_dir_path, artifact_meta.filename) # Download the artifacts to ``download_file_path``. optuna.artifacts.download_artifact( artifact_store=artifact_store, artifact_id=artifact_meta.artifact_id, file_path=download_file_path, ) Returns: The list of artifact meta in the trial or study. Each artifact meta includes ``artifact_id``, ``filename``, ``mimetype``, and ``encoding``. Note that if :class:`~optuna.study.Study` is provided, we return the information of the artifacts uploaded to ``study``, but not to all the trials in the study. """ if isinstance(study_or_trial, Trial) and storage is None: storage = study_or_trial.storage elif isinstance(study_or_trial, Study) and storage is None: storage = study_or_trial._storage if storage is None: raise ValueError("storage is required for FrozenTrial.") if isinstance(study_or_trial, (Trial, FrozenTrial)): system_attrs = storage.get_trial_system_attrs(study_or_trial._trial_id) else: system_attrs = storage.get_study_system_attrs(study_or_trial._study_id) artifact_meta_list: list[ArtifactMeta] = [] for attr_key, attr_json_string in system_attrs.items(): if not attr_key.startswith(ARTIFACTS_ATTR_PREFIX): continue attr_content = json.loads(attr_json_string) artifact_meta = ArtifactMeta( artifact_id=attr_content["artifact_id"], filename=attr_content["filename"], mimetype=attr_content["mimetype"], encoding=attr_content["encoding"], ) artifact_meta_list.append(artifact_meta) return artifact_meta_list optuna-4.1.0/optuna/artifacts/_protocol.py000066400000000000000000000035451471332314300206720ustar00rootroot00000000000000from __future__ import annotations from typing import TYPE_CHECKING try: from typing import Protocol except ImportError: from typing_extensions import Protocol # type: ignore if TYPE_CHECKING: from typing import BinaryIO class ArtifactStore(Protocol): """A protocol defining the interface for an artifact backend. The methods defined in this protocol are not supposed to be directly called by library users. An artifact backend is responsible for managing the storage and retrieval of artifact data. The backend should provide methods for opening, writing and removing artifacts. """ def open_reader(self, artifact_id: str) -> BinaryIO: """Open the artifact identified by the artifact_id. This method should return a binary file-like object in read mode, similar to ``open(..., mode="rb")``. If the artifact does not exist, an :exc:`~optuna.artifacts.exceptions.ArtifactNotFound` exception should be raised. Args: artifact_id: The identifier of the artifact to open. Returns: BinaryIO: A binary file-like object that can be read from. """ ... def write(self, artifact_id: str, content_body: BinaryIO) -> None: """Save the content to the backend. Args: artifact_id: The identifier of the artifact to write to. content_body: The content to write to the artifact. """ ... def remove(self, artifact_id: str) -> None: """Remove the artifact identified by the artifact_id. This method should delete the artifact from the backend. If the artifact does not exist, an :exc:`~optuna.artifacts.exceptions.ArtifactNotFound` exception may be raised. Args: artifact_id: The identifier of the artifact to remove. """ ... optuna-4.1.0/optuna/artifacts/_upload.py000066400000000000000000000073171471332314300203160ustar00rootroot00000000000000from __future__ import annotations from dataclasses import asdict from dataclasses import dataclass import json import mimetypes import os import uuid from optuna._convert_positional_args import convert_positional_args from optuna.artifacts._protocol import ArtifactStore from optuna.storages import BaseStorage from optuna.study import Study from optuna.trial import FrozenTrial from optuna.trial import Trial ARTIFACTS_ATTR_PREFIX = "artifacts:" DEFAULT_MIME_TYPE = "application/octet-stream" @dataclass class ArtifactMeta: """Meta information for an artifact. .. note:: All the artifact meta linked to a study or trial can be listed by :func:`~optuna.artifacts.get_all_artifact_meta`. The artifact meta can be used for :func:`~optuna.artifacts.download_artifact`. Args: artifact_id: The identifier of the artifact. filename: The artifact file name used for the upload. mimetype: A MIME type of the artifact. If not specified, the MIME type is guessed from the file extension. encoding: An encoding of the artifact, which is suitable for use as a Content-Encoding header, e.g., gzip. If not specified, the encoding is guessed from the file extension. """ artifact_id: str filename: str mimetype: str encoding: str | None @convert_positional_args( previous_positional_arg_names=["study_or_trial", "file_path", "artifact_store"] ) def upload_artifact( *, artifact_store: ArtifactStore, file_path: str, study_or_trial: Trial | FrozenTrial | Study, storage: BaseStorage | None = None, mimetype: str | None = None, encoding: str | None = None, ) -> str: """Upload an artifact to the artifact store. Args: artifact_store: An artifact store. file_path: A path to the file to be uploaded. study_or_trial: A :class:`~optuna.trial.Trial` object, a :class:`~optuna.trial.FrozenTrial`, or a :class:`~optuna.study.Study` object. storage: A storage object. This argument is required only if ``study_or_trial`` is :class:`~optuna.trial.FrozenTrial`. mimetype: A MIME type of the artifact. If not specified, the MIME type is guessed from the file extension. encoding: An encoding of the artifact, which is suitable for use as a ``Content-Encoding`` header (e.g. gzip). If not specified, the encoding is guessed from the file extension. Returns: An artifact ID. """ filename = os.path.basename(file_path) if isinstance(study_or_trial, Trial) and storage is None: storage = study_or_trial.storage elif isinstance(study_or_trial, Study) and storage is None: storage = study_or_trial._storage if storage is None: raise ValueError("storage is required for FrozenTrial.") artifact_id = str(uuid.uuid4()) guess_mimetype, guess_encoding = mimetypes.guess_type(filename) artifact = ArtifactMeta( artifact_id=artifact_id, filename=filename, mimetype=mimetype or guess_mimetype or DEFAULT_MIME_TYPE, encoding=encoding or guess_encoding, ) attr_key = ARTIFACTS_ATTR_PREFIX + artifact_id if isinstance(study_or_trial, (Trial, FrozenTrial)): trial_id = study_or_trial._trial_id storage.set_trial_system_attr(trial_id, attr_key, json.dumps(asdict(artifact))) else: study_id = study_or_trial._study_id storage.set_study_system_attr(study_id, attr_key, json.dumps(asdict(artifact))) with open(file_path, "rb") as f: artifact_store.write(artifact_id, f) return artifact_id optuna-4.1.0/optuna/artifacts/exceptions.py000066400000000000000000000005161471332314300210460ustar00rootroot00000000000000from optuna.exceptions import OptunaError class ArtifactNotFound(OptunaError): """Exception raised when an artifact is not found. It is typically raised while calling :meth:`~optuna.artifacts._protocol.ArtifactStore.open_reader` or :meth:`~optuna.artifacts._protocol.ArtifactStore.remove` methods. """ ... optuna-4.1.0/optuna/cli.py000066400000000000000000001030451471332314300154550ustar00rootroot00000000000000"""Optuna CLI module. If you want to add a new command, you also need to update the constant `_COMMANDS` """ import argparse from argparse import ArgumentParser from argparse import Namespace import datetime from enum import Enum import inspect import json import logging import os import sys from typing import Any from typing import Dict from typing import List from typing import Optional from typing import Tuple from typing import Type from typing import Union import warnings import sqlalchemy.exc import yaml import optuna from optuna._imports import _LazyImport from optuna.exceptions import CLIUsageError from optuna.exceptions import ExperimentalWarning from optuna.storages import BaseStorage from optuna.storages import JournalFileStorage from optuna.storages import JournalRedisStorage from optuna.storages import JournalStorage from optuna.storages import RDBStorage from optuna.storages.journal import JournalFileBackend from optuna.storages.journal import JournalRedisBackend from optuna.trial import TrialState _dataframe = _LazyImport("optuna.study._dataframe") _DATETIME_FORMAT = "%Y-%m-%d %H:%M:%S" def _check_storage_url(storage_url: Optional[str]) -> str: if storage_url is not None: return storage_url env_storage = os.environ.get("OPTUNA_STORAGE") if env_storage is not None: warnings.warn( "Specifying the storage url via 'OPTUNA_STORAGE' environment variable" " is an experimental feature. The interface can change in the future.", ExperimentalWarning, ) return env_storage raise CLIUsageError("Storage URL is not specified.") def _get_storage(storage_url: Optional[str], storage_class: Optional[str]) -> BaseStorage: storage_url = _check_storage_url(storage_url) if storage_class: if storage_class == JournalRedisBackend.__name__: return JournalStorage(JournalRedisBackend(storage_url)) if storage_class == JournalRedisStorage.__name__: return JournalStorage(JournalRedisStorage(storage_url)) if storage_class == JournalFileBackend.__name__: return JournalStorage(JournalFileBackend(storage_url)) if storage_class == JournalFileStorage.__name__: return JournalStorage(JournalFileStorage(storage_url)) if storage_class == RDBStorage.__name__: return RDBStorage(storage_url) raise CLIUsageError("Unsupported storage class") if storage_url.startswith("redis"): return JournalStorage(JournalRedisBackend(storage_url)) if os.path.isfile(storage_url): return JournalStorage(JournalFileBackend(storage_url)) try: return RDBStorage(storage_url) except sqlalchemy.exc.ArgumentError: raise CLIUsageError("Failed to guess storage class from storage_url") def _format_value(value: Any) -> Any: # Format value that can be serialized to JSON or YAML. if value is None or isinstance(value, (int, float)): return value elif isinstance(value, datetime.datetime): return value.strftime(_DATETIME_FORMAT) elif isinstance(value, list): return list(_format_value(v) for v in value) elif isinstance(value, tuple): return tuple(_format_value(v) for v in value) elif isinstance(value, dict): return {_format_value(k): _format_value(v) for k, v in value.items()} else: return str(value) def _convert_to_dict( records: List[Dict[Tuple[str, str], Any]], columns: List[Tuple[str, str]], flatten: bool ) -> Tuple[List[Dict[str, Any]], List[str]]: header = [] ret = [] if flatten: for column in columns: if column[1] != "": header.append(f"{column[0]}_{column[1]}") elif any(isinstance(record.get(column), (list, tuple)) for record in records): max_length = 0 for record in records: if column in record: max_length = max(max_length, len(record[column])) for i in range(max_length): header.append(f"{column[0]}_{i}") else: header.append(column[0]) for record in records: row = {} for column in columns: if column not in record: continue value = _format_value(record[column]) if column[1] != "": row[f"{column[0]}_{column[1]}"] = value elif any(isinstance(record.get(column), (list, tuple)) for record in records): for i, v in enumerate(value): row[f"{column[0]}_{i}"] = v else: row[f"{column[0]}"] = value ret.append(row) else: for column in columns: if column[0] not in header: header.append(column[0]) for record in records: attrs: Dict[str, Any] = {column_name: {} for column_name in header} for column in columns: if column not in record: continue value = _format_value(record[column]) if isinstance(column[1], int): # Reconstruct list of values. `_dataframe._create_records_and_aggregate_column` # returns indices of list as the second key of column. if attrs[column[0]] == {}: attrs[column[0]] = [] attrs[column[0]] += [None] * max(column[1] + 1 - len(attrs[column[0]]), 0) attrs[column[0]][column[1]] = value elif column[1] != "": attrs[column[0]][column[1]] = value else: attrs[column[0]] = value ret.append(attrs) return ret, header class ValueType(Enum): NONE = 0 NUMERIC = 1 STRING = 2 class CellValue: def __init__(self, value: Any) -> None: self.value = value if value is None: self.value_type = ValueType.NONE elif isinstance(value, (int, float)): self.value_type = ValueType.NUMERIC else: self.value_type = ValueType.STRING def __str__(self) -> str: if isinstance(self.value, datetime.datetime): return self.value.strftime(_DATETIME_FORMAT) else: return str(self.value) def width(self) -> int: return len(str(self.value)) def get_string(self, value_type: ValueType, width: int) -> str: value = str(self.value) if self.value is None: return " " * width elif value_type == ValueType.NUMERIC: return f"{value:>{width}}" else: return f"{value:<{width}}" def _dump_value(records: List[Dict[str, Any]], header: List[str]) -> str: values = [] for record in records: row = [] for column_name in header: row.append(str(record.get(column_name, ""))) values.append(" ".join(row)) return "\n".join(values) def _dump_table(records: List[Dict[str, Any]], header: List[str]) -> str: rows = [] for record in records: row = [] for column_name in header: row.append(CellValue(record.get(column_name))) rows.append(row) separator = "+" header_string = "|" rows_string = ["|" for _ in rows] for column in range(len(header)): value_types = [row[column].value_type for row in rows] value_type = ValueType.NUMERIC for t in value_types: if t == ValueType.STRING: value_type = ValueType.STRING max_width = max(len(header[column]), max(row[column].width() for row in rows)) separator += "-" * (max_width + 2) + "+" if value_type == ValueType.NUMERIC: header_string += f" {header[column]:>{max_width}} |" else: header_string += f" {header[column]:<{max_width}} |" for i, row in enumerate(rows): rows_string[i] += " " + row[column].get_string(value_type, max_width) + " |" ret = "" ret += separator + "\n" ret += header_string + "\n" ret += separator + "\n" ret += "\n".join(rows_string) + "\n" ret += separator + "\n" return ret def _format_output( records: Union[List[Dict[Tuple[str, str], Any]], Dict[Tuple[str, str], Any]], columns: List[Tuple[str, str]], output_format: str, flatten: bool, ) -> str: if isinstance(records, list): values, header = _convert_to_dict(records, columns, flatten) else: values, header = _convert_to_dict([records], columns, flatten) if output_format == "value": if isinstance(records, list): return _dump_value(values, header).strip() else: return str(values[0]).strip() elif output_format == "table": return _dump_table(values, header).strip() elif output_format == "json": if isinstance(records, list): return json.dumps(values).strip() else: return json.dumps(values[0]).strip() elif output_format == "yaml": if isinstance(records, list): return yaml.safe_dump(values).strip() else: return yaml.safe_dump(values[0]).strip() else: raise CLIUsageError(f"Optuna CLI does not supported the {output_format} format.") class _BaseCommand: """Base class for commands. Note that command classes are not intended to be called by library users. They are exclusively used within this file to manage Optuna CLI commands. """ def __init__(self) -> None: self.logger = optuna.logging.get_logger(__name__) def add_arguments(self, parser: ArgumentParser) -> None: """Add arguments required for each command. Args: parser: `ArgumentParser` object to add arguments """ pass def take_action(self, parsed_args: Namespace) -> int: """Define action if the command is called. Args: parsed_args: `Namespace` object including arguments specified by user. Returns: Running status of the action. 0 if this method finishes normally, otherwise 1. """ raise NotImplementedError class _CreateStudy(_BaseCommand): """Create a new study.""" def add_arguments(self, parser: ArgumentParser) -> None: parser.add_argument( "--study-name", default=None, help="A human-readable name of a study to distinguish it from others.", ) parser.add_argument( "--direction", default=None, type=str, choices=("minimize", "maximize"), help="Set direction of optimization to a new study. Set 'minimize' " "for minimization and 'maximize' for maximization.", ) parser.add_argument( "--skip-if-exists", default=False, action="store_true", help="If specified, the creation of the study is skipped " "without any error when the study name is duplicated.", ) parser.add_argument( "--directions", type=str, default=None, choices=("minimize", "maximize"), help="Set directions of optimization to a new study." " Put whitespace between directions. Each direction should be" ' either "minimize" or "maximize".', nargs="+", ) def take_action(self, parsed_args: Namespace) -> int: storage = _get_storage(parsed_args.storage, parsed_args.storage_class) study_name = optuna.create_study( storage=storage, study_name=parsed_args.study_name, direction=parsed_args.direction, directions=parsed_args.directions, load_if_exists=parsed_args.skip_if_exists, ).study_name print(study_name) return 0 class _DeleteStudy(_BaseCommand): """Delete a specified study.""" def add_arguments(self, parser: ArgumentParser) -> None: parser.add_argument("--study-name", default=None, help="The name of the study to delete.") def take_action(self, parsed_args: Namespace) -> int: storage = _get_storage(parsed_args.storage, parsed_args.storage_class) study_id = storage.get_study_id_from_name(parsed_args.study_name) storage.delete_study(study_id) return 0 class _StudySetUserAttribute(_BaseCommand): """Set a user attribute to a study.""" def add_arguments(self, parser: ArgumentParser) -> None: parser.add_argument( "--study-name", required=True, help="The name of the study to set the user attribute to.", ) parser.add_argument("--key", "-k", required=True, help="Key of the user attribute.") parser.add_argument("--value", required=True, help="Value to be set.") def take_action(self, parsed_args: Namespace) -> int: storage = _get_storage(parsed_args.storage, parsed_args.storage_class) study = optuna.load_study(storage=storage, study_name=parsed_args.study_name) study.set_user_attr(parsed_args.key, parsed_args.value) self.logger.info("Attribute successfully written.") return 0 class _StudyNames(_BaseCommand): """Get all study names stored in a specified storage""" def add_arguments(self, parser: ArgumentParser) -> None: parser.add_argument( "-f", "--format", type=str, choices=("value", "json", "table", "yaml"), default="value", help="Output format.", ) def take_action(self, parsed_args: Namespace) -> int: storage = _get_storage(parsed_args.storage, parsed_args.storage_class) all_study_names = optuna.get_all_study_names(storage) records = [] record_key = ("name", "") for study_name in all_study_names: records.append({record_key: study_name}) print(_format_output(records, [record_key], parsed_args.format, flatten=False)) return 0 class _Studies(_BaseCommand): """Show a list of studies.""" _study_list_header = [ ("name", ""), ("direction", ""), ("n_trials", ""), ("datetime_start", ""), ] def add_arguments(self, parser: ArgumentParser) -> None: parser.add_argument( "-f", "--format", type=str, choices=("json", "table", "yaml"), default="table", help="Output format.", ) parser.add_argument( "--flatten", default=False, action="store_true", help="Flatten nested columns such as directions.", ) def take_action(self, parsed_args: Namespace) -> int: storage = _get_storage(parsed_args.storage, parsed_args.storage_class) summaries = optuna.get_all_study_summaries(storage, include_best_trial=False) records = [] for s in summaries: start = ( s.datetime_start.strftime(_DATETIME_FORMAT) if s.datetime_start is not None else None ) record: Dict[Tuple[str, str], Any] = {} record[("name", "")] = s.study_name record[("direction", "")] = tuple(d.name for d in s.directions) record[("n_trials", "")] = s.n_trials record[("datetime_start", "")] = start record[("user_attrs", "")] = s.user_attrs records.append(record) if any(r[("user_attrs", "")] != {} for r in records): self._study_list_header.append(("user_attrs", "")) print( _format_output( records, self._study_list_header, parsed_args.format, parsed_args.flatten ) ) return 0 class _Trials(_BaseCommand): """Show a list of trials.""" def add_arguments(self, parser: ArgumentParser) -> None: parser.add_argument( "--study-name", type=str, required=True, help="The name of the study which includes trials.", ) parser.add_argument( "-f", "--format", type=str, choices=("json", "table", "yaml"), default="table", help="Output format.", ) parser.add_argument( "--flatten", default=False, action="store_true", help="Flatten nested columns such as params and user_attrs.", ) def take_action(self, parsed_args: Namespace) -> int: warnings.warn( "'trials' is an experimental CLI command. The interface can change in the future.", ExperimentalWarning, ) storage = _get_storage(parsed_args.storage, parsed_args.storage_class) study = optuna.load_study(storage=storage, study_name=parsed_args.study_name) attrs = ( "number", "value" if not study._is_multi_objective() else "values", "datetime_start", "datetime_complete", "duration", "params", "user_attrs", "state", ) records, columns = _dataframe._create_records_and_aggregate_column(study, attrs) print(_format_output(records, columns, parsed_args.format, parsed_args.flatten)) return 0 class _BestTrial(_BaseCommand): """Show the best trial.""" def add_arguments(self, parser: ArgumentParser) -> None: parser.add_argument( "--study-name", type=str, required=True, help="The name of the study to get the best trial.", ) parser.add_argument( "-f", "--format", type=str, choices=("json", "table", "yaml"), default="table", help="Output format.", ) parser.add_argument( "--flatten", default=False, action="store_true", help="Flatten nested columns such as params and user_attrs.", ) def take_action(self, parsed_args: Namespace) -> int: warnings.warn( "'best-trial' is an experimental CLI command. The interface can change in the future.", ExperimentalWarning, ) storage = _get_storage(parsed_args.storage, parsed_args.storage_class) study = optuna.load_study(storage=storage, study_name=parsed_args.study_name) attrs = ( "number", "value" if not study._is_multi_objective() else "values", "datetime_start", "datetime_complete", "duration", "params", "user_attrs", "state", ) records, columns = _dataframe._create_records_and_aggregate_column(study, attrs) print( _format_output( records[study.best_trial.number], columns, parsed_args.format, parsed_args.flatten ) ) return 0 class _BestTrials(_BaseCommand): """Show a list of trials located at the Pareto front.""" def add_arguments(self, parser: ArgumentParser) -> None: parser.add_argument( "--study-name", type=str, required=True, help="The name of the study to get the best trials (trials at the Pareto front).", ) parser.add_argument( "-f", "--format", type=str, choices=("json", "table", "yaml"), default="table", help="Output format.", ) parser.add_argument( "--flatten", default=False, action="store_true", help="Flatten nested columns such as params and user_attrs.", ) def take_action(self, parsed_args: Namespace) -> int: warnings.warn( "'best-trials' is an experimental CLI command. The interface can change in the " "future.", ExperimentalWarning, ) storage = _get_storage(parsed_args.storage, parsed_args.storage_class) study = optuna.load_study(storage=storage, study_name=parsed_args.study_name) best_trials = [trial.number for trial in study.best_trials] attrs = ( "number", "value" if not study._is_multi_objective() else "values", "datetime_start", "datetime_complete", "duration", "params", "user_attrs", "state", ) records, columns = _dataframe._create_records_and_aggregate_column(study, attrs) best_records = list(filter(lambda record: record[("number", "")] in best_trials, records)) print(_format_output(best_records, columns, parsed_args.format, parsed_args.flatten)) return 0 class _StorageUpgrade(_BaseCommand): """Upgrade the schema of an RDB storage.""" def take_action(self, parsed_args: Namespace) -> int: storage_url = _check_storage_url(parsed_args.storage) try: storage = RDBStorage( storage_url, skip_compatibility_check=True, skip_table_creation=True ) except sqlalchemy.exc.ArgumentError: self.logger.error("Invalid RDBStorage URL.") return 1 current_version = storage.get_current_version() head_version = storage.get_head_version() known_versions = storage.get_all_versions() if current_version == head_version: self.logger.info("This storage is up-to-date.") elif current_version in known_versions: self.logger.info("Upgrading the storage schema to the latest version.") storage.upgrade() self.logger.info("Completed to upgrade the storage.") else: warnings.warn( "Your optuna version seems outdated against the storage version. " "Please try updating optuna to the latest version by " "`$ pip install -U optuna`." ) return 0 class _Ask(_BaseCommand): """Create a new trial and suggest parameters.""" def add_arguments(self, parser: ArgumentParser) -> None: parser.add_argument("--study-name", type=str, help="Name of study.") parser.add_argument("--sampler", type=str, help="Class name of sampler object to create.") parser.add_argument( "--sampler-kwargs", type=str, help="Sampler object initialization keyword arguments as JSON.", ) parser.add_argument( "--search-space", type=str, help=( "Search space as JSON. Keys are names and values are outputs from " ":func:`~optuna.distributions.distribution_to_json`." ), ) parser.add_argument( "-f", "--format", type=str, choices=("json", "table", "yaml"), default="json", help="Output format.", ) parser.add_argument( "--flatten", default=False, action="store_true", help="Flatten nested columns such as params.", ) def take_action(self, parsed_args: Namespace) -> int: warnings.warn( "'ask' is an experimental CLI command. The interface can change in the future.", ExperimentalWarning, ) storage = _get_storage(parsed_args.storage, parsed_args.storage_class) create_study_kwargs = { "storage": storage, "study_name": parsed_args.study_name, "load_if_exists": True, } if parsed_args.sampler is not None: if parsed_args.sampler_kwargs is not None: sampler_kwargs = json.loads(parsed_args.sampler_kwargs) else: sampler_kwargs = {} sampler_cls = getattr(optuna.samplers, parsed_args.sampler) sampler = sampler_cls(**sampler_kwargs) create_study_kwargs["sampler"] = sampler else: if parsed_args.sampler_kwargs is not None: raise ValueError( "`--sampler_kwargs` is set without `--sampler`. Please specify `--sampler` as" " well or omit `--sampler-kwargs`." ) if parsed_args.search_space is not None: # The search space is expected to be a JSON serialized string, e.g. # '{"x": {"name": "FloatDistribution", "attributes": {"low": 0.0, "high": 1.0}}, # "y": ...}'. search_space = { name: optuna.distributions.json_to_distribution(json.dumps(dist)) for name, dist in json.loads(parsed_args.search_space).items() } else: search_space = {} try: study = optuna.load_study( study_name=create_study_kwargs["study_name"], storage=create_study_kwargs["storage"], sampler=create_study_kwargs.get("sampler"), ) except KeyError: raise KeyError( "Implicit study creation within the 'ask' command was dropped in Optuna v4.0.0. " "Please use the 'create-study' command beforehand." ) trial = study.ask(fixed_distributions=search_space) self.logger.info(f"Asked trial {trial.number} with parameters {trial.params}.") record: Dict[Tuple[str, str], Any] = {("number", ""): trial.number} columns = [("number", "")] if len(trial.params) == 0 and not parsed_args.flatten: record[("params", "")] = {} columns.append(("params", "")) else: for param_name, param_value in trial.params.items(): record[("params", param_name)] = param_value columns.append(("params", param_name)) print(_format_output(record, columns, parsed_args.format, parsed_args.flatten)) return 0 class _Tell(_BaseCommand): """Finish a trial, which was created by the ask command.""" def add_arguments(self, parser: ArgumentParser) -> None: parser.add_argument("--study-name", type=str, help="Name of study.") parser.add_argument("--trial-number", type=int, help="Trial number.") parser.add_argument("--values", type=float, nargs="+", help="Objective values.") parser.add_argument( "--state", type=str, help="Trial state.", choices=("complete", "pruned", "fail"), ) parser.add_argument( "--skip-if-finished", default=False, action="store_true", help="If specified, tell is skipped without any error when the trial is already " "finished.", ) def take_action(self, parsed_args: Namespace) -> int: warnings.warn( "'tell' is an experimental CLI command. The interface can change in the future.", ExperimentalWarning, ) storage = _get_storage(parsed_args.storage, parsed_args.storage_class) study = optuna.load_study( storage=storage, study_name=parsed_args.study_name, ) if parsed_args.state is not None: state: Optional[TrialState] = TrialState[parsed_args.state.upper()] else: state = None trial_number = parsed_args.trial_number values = parsed_args.values study.tell( trial=trial_number, values=values, state=state, skip_if_finished=parsed_args.skip_if_finished, ) self.logger.info(f"Told trial {trial_number} with values {values} and state {state}.") return 0 _COMMANDS: Dict[str, Type[_BaseCommand]] = { "create-study": _CreateStudy, "delete-study": _DeleteStudy, "study set-user-attr": _StudySetUserAttribute, "study-names": _StudyNames, "studies": _Studies, "trials": _Trials, "best-trial": _BestTrial, "best-trials": _BestTrials, "storage upgrade": _StorageUpgrade, "ask": _Ask, "tell": _Tell, } def _parse_storage_class_without_suggesting_deprecated_choices(value: str) -> str: choices = [ RDBStorage.__name__, JournalFileBackend.__name__, JournalRedisBackend.__name__, ] deprecated_choices = [ JournalFileStorage.__name__, JournalRedisStorage.__name__, ] if value in choices + deprecated_choices: return value raise argparse.ArgumentTypeError( f"Invalid choice: {value} (choose from {str(choices)[1:-1]})" ) def _add_common_arguments(parser: ArgumentParser) -> ArgumentParser: parser.add_argument( "--storage", default=None, help=( "DB URL. (e.g. sqlite:///example.db) " "Also can be specified via OPTUNA_STORAGE environment variable." ), ) parser.add_argument( "--storage-class", help="Storage class hint (e.g. JournalFileBackend)", default=None, type=_parse_storage_class_without_suggesting_deprecated_choices, ) verbose_group = parser.add_mutually_exclusive_group() verbose_group.add_argument( "-v", "--verbose", action="count", dest="verbose_level", default=1, help="Increase verbosity of output. Can be repeated.", ) verbose_group.add_argument( "-q", "--quiet", action="store_const", dest="verbose_level", const=0, help="Suppress output except warnings and errors.", ) parser.add_argument( "--log-file", action="store", default=None, help="Specify a file to log output. Disabled by default.", ) parser.add_argument( "--debug", default=False, action="store_true", help="Show tracebacks on errors.", ) return parser def _add_commands( main_parser: ArgumentParser, parent_parser: ArgumentParser ) -> Dict[str, ArgumentParser]: subparsers = main_parser.add_subparsers() command_name_to_subparser = {} for command_name, command_type in _COMMANDS.items(): command = command_type() subparser = subparsers.add_parser( command_name, parents=[parent_parser], help=inspect.getdoc(command_type) ) command.add_arguments(subparser) subparser.set_defaults(handler=command.take_action) command_name_to_subparser[command_name] = subparser def _print_help(args: Namespace) -> None: main_parser.print_help() subparsers.add_parser("help", help="Show help message and exit.").set_defaults( handler=_print_help ) return command_name_to_subparser def _get_parser(description: str = "") -> Tuple[ArgumentParser, Dict[str, ArgumentParser]]: # Use `parent_parser` is necessary to avoid namespace conflict for -h/--help # between `main_parser` and `subparser`. parent_parser = ArgumentParser(add_help=False) parent_parser = _add_common_arguments(parent_parser) main_parser = ArgumentParser(description=description, parents=[parent_parser]) main_parser.add_argument( "--version", action="version", version="{0} {1}".format("optuna", optuna.__version__) ) command_name_to_subparser = _add_commands(main_parser, parent_parser) return main_parser, command_name_to_subparser def _preprocess_argv(argv: List[str]) -> List[str]: # Some preprocess is necessary for argv because some subcommand includes space # (e.g. optuna storage upgrade). argv = argv[1:] if len(argv) > 1 else ["help"] for i in range(len(argv)): for j in range(i, i + 2): # Commands consist of one or two words. command_candidate = " ".join(argv[i : j + 1]) if command_candidate in _COMMANDS: options = argv[:i] + argv[j + 1 :] return [command_candidate] + options # No subcommand is found. return argv def _set_verbosity(args: Namespace) -> None: root_logger = logging.getLogger() root_logger.setLevel(logging.DEBUG) stream_handler = logging.StreamHandler(sys.stderr) logging_level = { 0: logging.WARNING, 1: logging.INFO, 2: logging.DEBUG, }.get(args.verbose_level, logging.DEBUG) stream_handler.setLevel(logging_level) stream_handler.setFormatter(optuna.logging.create_default_formatter()) root_logger.addHandler(stream_handler) optuna.logging.set_verbosity(logging_level) def _set_log_file(args: Namespace) -> None: if args.log_file is None: return root_logger = logging.getLogger() root_logger.setLevel(logging.DEBUG) file_handler = logging.FileHandler( filename=args.log_file, ) file_handler.setFormatter(optuna.logging.create_default_formatter()) root_logger.addHandler(file_handler) def main() -> int: main_parser, command_name_to_subparser = _get_parser() argv = sys.argv preprocessed_argv = _preprocess_argv(argv) args = main_parser.parse_args(preprocessed_argv) _set_verbosity(args) _set_log_file(args) logger = logging.getLogger("optuna") try: return args.handler(args) except CLIUsageError as e: if args.debug: logger.exception(e) else: logger.error(e) # This code is required to show help for each subcommand. # NOTE: the first element of `preprocessed_argv` is command name. command_name_to_subparser[preprocessed_argv[0]].print_help() return 1 except AttributeError: # Exception for the case -v/--verbose/-q/--quiet/--log-file/--debug # without any subcommand. argv_str = " ".join(argv[1:]) logger.error(f"'{argv_str}' is not an optuna command. see 'optuna --help'") main_parser.print_help() return 1 optuna-4.1.0/optuna/distributions.py000066400000000000000000000702021471332314300176060ustar00rootroot00000000000000import abc import copy import decimal import json from numbers import Real from typing import Any from typing import cast from typing import Dict from typing import Sequence from typing import Union import warnings import numpy as np from optuna._deprecated import deprecated_class CategoricalChoiceType = Union[None, bool, int, float, str] _float_distribution_deprecated_msg = ( "Use :class:`~optuna.distributions.FloatDistribution` instead." ) _int_distribution_deprecated_msg = "Use :class:`~optuna.distributions.IntDistribution` instead." class BaseDistribution(abc.ABC): """Base class for distributions. Note that distribution classes are not supposed to be called by library users. They are used by :class:`~optuna.trial.Trial` and :class:`~optuna.samplers` internally. """ def to_external_repr(self, param_value_in_internal_repr: float) -> Any: """Convert internal representation of a parameter value into external representation. Args: param_value_in_internal_repr: Optuna's internal representation of a parameter value. Returns: Optuna's external representation of a parameter value. """ return param_value_in_internal_repr @abc.abstractmethod def to_internal_repr(self, param_value_in_external_repr: Any) -> float: """Convert external representation of a parameter value into internal representation. Args: param_value_in_external_repr: Optuna's external representation of a parameter value. Returns: Optuna's internal representation of a parameter value. """ raise NotImplementedError @abc.abstractmethod def single(self) -> bool: """Test whether the range of this distribution contains just a single value. Returns: :obj:`True` if the range of this distribution contains just a single value, otherwise :obj:`False`. """ raise NotImplementedError @abc.abstractmethod def _contains(self, param_value_in_internal_repr: float) -> bool: """Test if a parameter value is contained in the range of this distribution. Args: param_value_in_internal_repr: Optuna's internal representation of a parameter value. Returns: :obj:`True` if the parameter value is contained in the range of this distribution, otherwise :obj:`False`. """ raise NotImplementedError def _asdict(self) -> Dict: return self.__dict__ def __eq__(self, other: Any) -> bool: if not isinstance(other, BaseDistribution): return NotImplemented if type(self) is not type(other): return False return self.__dict__ == other.__dict__ def __hash__(self) -> int: return hash((self.__class__,) + tuple(sorted(self.__dict__.items()))) def __repr__(self) -> str: kwargs = ", ".join("{}={}".format(k, v) for k, v in sorted(self._asdict().items())) return "{}({})".format(self.__class__.__name__, kwargs) class FloatDistribution(BaseDistribution): """A distribution on floats. This object is instantiated by :func:`~optuna.trial.Trial.suggest_float`, and passed to :mod:`~optuna.samplers` in general. .. note:: When ``step`` is not :obj:`None`, if the range :math:`[\\mathsf{low}, \\mathsf{high}]` is not divisible by :math:`\\mathsf{step}`, :math:`\\mathsf{high}` will be replaced with the maximum of :math:`k \\times \\mathsf{step} + \\mathsf{low} < \\mathsf{high}`, where :math:`k` is an integer. Attributes: low: Lower endpoint of the range of the distribution. ``low`` is included in the range. ``low`` must be less than or equal to ``high``. If ``log`` is :obj:`True`, ``low`` must be larger than 0. high: Upper endpoint of the range of the distribution. ``high`` is included in the range. ``high`` must be greater than or equal to ``low``. log: If ``log`` is :obj:`True`, this distribution is in log-scaled domain. In this case, all parameters enqueued to the distribution must be positive values. This parameter must be :obj:`False` when the parameter ``step`` is not :obj:`None`. step: A discretization step. ``step`` must be larger than 0. This parameter must be :obj:`None` when the parameter ``log`` is :obj:`True`. """ def __init__( self, low: float, high: float, log: bool = False, step: Union[None, float] = None ) -> None: if log and step is not None: raise ValueError("The parameter `step` is not supported when `log` is true.") if low > high: raise ValueError( "The `low` value must be smaller than or equal to the `high` value " "(low={}, high={}).".format(low, high) ) if log and low <= 0.0: raise ValueError( "The `low` value must be larger than 0 for a log distribution " "(low={}, high={}).".format(low, high) ) if step is not None and step <= 0: raise ValueError( "The `step` value must be non-zero positive value, " "but step={}.".format(step) ) self.step = None if step is not None: high = _adjust_discrete_uniform_high(low, high, step) self.step = float(step) self.low = float(low) self.high = float(high) self.log = log def single(self) -> bool: if self.step is None: return self.low == self.high else: if self.low == self.high: return True high = decimal.Decimal(str(self.high)) low = decimal.Decimal(str(self.low)) step = decimal.Decimal(str(self.step)) return (high - low) < step def _contains(self, param_value_in_internal_repr: float) -> bool: value = param_value_in_internal_repr if self.step is None: return self.low <= value <= self.high else: k = (value - self.low) / self.step return self.low <= value <= self.high and abs(k - round(k)) < 1.0e-8 def to_internal_repr(self, param_value_in_external_repr: float) -> float: try: internal_repr = float(param_value_in_external_repr) except (ValueError, TypeError) as e: raise ValueError( f"'{param_value_in_external_repr}' is not a valid type. " "float-castable value is expected." ) from e if np.isnan(internal_repr): raise ValueError(f"`{param_value_in_external_repr}` is invalid value.") if self.log and internal_repr <= 0.0: raise ValueError( f"`{param_value_in_external_repr}` is invalid value for the case log=True." ) return internal_repr @deprecated_class("3.0.0", "6.0.0", text=_float_distribution_deprecated_msg) class UniformDistribution(FloatDistribution): """A uniform distribution in the linear domain. This object is instantiated by :func:`~optuna.trial.Trial.suggest_float`, and passed to :mod:`~optuna.samplers` in general. Attributes: low: Lower endpoint of the range of the distribution. ``low`` is included in the range. ``low`` must be less than or equal to ``high``. high: Upper endpoint of the range of the distribution. ``high`` is included in the range. ``high`` must be greater than or equal to ``low``. """ def __init__(self, low: float, high: float) -> None: super().__init__(low=low, high=high, log=False, step=None) def _asdict(self) -> Dict: d = copy.deepcopy(self.__dict__) d.pop("log") d.pop("step") return d @deprecated_class("3.0.0", "6.0.0", text=_float_distribution_deprecated_msg) class LogUniformDistribution(FloatDistribution): """A uniform distribution in the log domain. This object is instantiated by :func:`~optuna.trial.Trial.suggest_float` with ``log=True``, and passed to :mod:`~optuna.samplers` in general. Attributes: low: Lower endpoint of the range of the distribution. ``low`` is included in the range. ``low`` must be larger than 0. ``low`` must be less than or equal to ``high``. high: Upper endpoint of the range of the distribution. ``high`` is included in the range. ``high`` must be greater than or equal to ``low``. """ def __init__(self, low: float, high: float) -> None: super().__init__(low=low, high=high, log=True, step=None) def _asdict(self) -> Dict: d = copy.deepcopy(self.__dict__) d.pop("log") d.pop("step") return d @deprecated_class("3.0.0", "6.0.0", text=_float_distribution_deprecated_msg) class DiscreteUniformDistribution(FloatDistribution): """A discretized uniform distribution in the linear domain. This object is instantiated by :func:`~optuna.trial.Trial.suggest_float` with ``step`` argument, and passed to :mod:`~optuna.samplers` in general. .. note:: If the range :math:`[\\mathsf{low}, \\mathsf{high}]` is not divisible by :math:`q`, :math:`\\mathsf{high}` will be replaced with the maximum of :math:`k q + \\mathsf{low} < \\mathsf{high}`, where :math:`k` is an integer. Args: low: Lower endpoint of the range of the distribution. ``low`` is included in the range. ``low`` must be less than or equal to ``high``. high: Upper endpoint of the range of the distribution. ``high`` is included in the range. ``high`` must be greater than or equal to ``low``. q: A discretization step. ``q`` must be larger than 0. Attributes: low: Lower endpoint of the range of the distribution. ``low`` is included in the range. high: Upper endpoint of the range of the distribution. ``high`` is included in the range. """ def __init__(self, low: float, high: float, q: float) -> None: super().__init__(low=low, high=high, step=q) def _asdict(self) -> Dict: d = copy.deepcopy(self.__dict__) d.pop("log") step = d.pop("step") d["q"] = step return d @property def q(self) -> float: """Discretization step. :class:`~optuna.distributions.DiscreteUniformDistribution` is a subtype of :class:`~optuna.distributions.FloatDistribution`. This property is a proxy for its ``step`` attribute. """ return cast(float, self.step) @q.setter def q(self, v: float) -> None: self.step = v class IntDistribution(BaseDistribution): """A distribution on integers. This object is instantiated by :func:`~optuna.trial.Trial.suggest_int`, and passed to :mod:`~optuna.samplers` in general. .. note:: When ``step`` is not :obj:`None`, if the range :math:`[\\mathsf{low}, \\mathsf{high}]` is not divisible by :math:`\\mathsf{step}`, :math:`\\mathsf{high}` will be replaced with the maximum of :math:`k \\times \\mathsf{step} + \\mathsf{low} < \\mathsf{high}`, where :math:`k` is an integer. Attributes: low: Lower endpoint of the range of the distribution. ``low`` is included in the range. ``low`` must be less than or equal to ``high``. If ``log`` is :obj:`True`, ``low`` must be larger than or equal to 1. high: Upper endpoint of the range of the distribution. ``high`` is included in the range. ``high`` must be greater than or equal to ``low``. log: If ``log`` is :obj:`True`, this distribution is in log-scaled domain. In this case, all parameters enqueued to the distribution must be positive values. This parameter must be :obj:`False` when the parameter ``step`` is not 1. step: A discretization step. ``step`` must be a positive integer. This parameter must be 1 when the parameter ``log`` is :obj:`True`. """ def __init__(self, low: int, high: int, log: bool = False, step: int = 1) -> None: if log and step != 1: raise ValueError( "Samplers and other components in Optuna only accept step is 1 " "when `log` argument is True." ) if low > high: raise ValueError( "The `low` value must be smaller than or equal to the `high` value " "(low={}, high={}).".format(low, high) ) if log and low < 1: raise ValueError( "The `low` value must be equal to or greater than 1 for a log distribution " "(low={}, high={}).".format(low, high) ) if step <= 0: raise ValueError( "The `step` value must be non-zero positive value, but step={}.".format(step) ) self.log = log self.step = int(step) self.low = int(low) high = int(high) self.high = _adjust_int_uniform_high(self.low, high, self.step) def to_external_repr(self, param_value_in_internal_repr: float) -> int: return int(param_value_in_internal_repr) def to_internal_repr(self, param_value_in_external_repr: int) -> float: try: internal_repr = float(param_value_in_external_repr) except (ValueError, TypeError) as e: raise ValueError( f"'{param_value_in_external_repr}' is not a valid type. " "float-castable value is expected." ) from e if np.isnan(internal_repr): raise ValueError(f"`{param_value_in_external_repr}` is invalid value.") if self.log and internal_repr <= 0.0: raise ValueError( f"`{param_value_in_external_repr}` is invalid value for the case log=True." ) return internal_repr def single(self) -> bool: if self.log: return self.low == self.high if self.low == self.high: return True return (self.high - self.low) < self.step def _contains(self, param_value_in_internal_repr: float) -> bool: value = param_value_in_internal_repr return self.low <= value <= self.high and (value - self.low) % self.step == 0 @deprecated_class("3.0.0", "6.0.0", text=_int_distribution_deprecated_msg) class IntUniformDistribution(IntDistribution): """A uniform distribution on integers. This object is instantiated by :func:`~optuna.trial.Trial.suggest_int`, and passed to :mod:`~optuna.samplers` in general. .. note:: If the range :math:`[\\mathsf{low}, \\mathsf{high}]` is not divisible by :math:`\\mathsf{step}`, :math:`\\mathsf{high}` will be replaced with the maximum of :math:`k \\times \\mathsf{step} + \\mathsf{low} < \\mathsf{high}`, where :math:`k` is an integer. Attributes: low: Lower endpoint of the range of the distribution. ``low`` is included in the range. ``low`` must be less than or equal to ``high``. high: Upper endpoint of the range of the distribution. ``high`` is included in the range. ``high`` must be greater than or equal to ``low``. step: A discretization step. ``step`` must be a positive integer. """ def __init__(self, low: int, high: int, step: int = 1) -> None: super().__init__(low=low, high=high, log=False, step=step) def _asdict(self) -> Dict: d = copy.deepcopy(self.__dict__) d.pop("log") return d @deprecated_class("3.0.0", "6.0.0", text=_int_distribution_deprecated_msg) class IntLogUniformDistribution(IntDistribution): """A uniform distribution on integers in the log domain. This object is instantiated by :func:`~optuna.trial.Trial.suggest_int`, and passed to :mod:`~optuna.samplers` in general. Attributes: low: Lower endpoint of the range of the distribution. ``low`` is included in the range and must be larger than or equal to 1. ``low`` must be less than or equal to ``high``. high: Upper endpoint of the range of the distribution. ``high`` is included in the range. ``high`` must be greater than or equal to ``low``. step: A discretization step. ``step`` must be a positive integer. .. warning:: Deprecated in v2.0.0. ``step`` argument will be removed in the future. The removal of this feature is currently scheduled for v4.0.0, but this schedule is subject to change. Samplers and other components in Optuna relying on this distribution will ignore this value and assume that ``step`` is always 1. User-defined samplers may continue to use other values besides 1 during the deprecation. """ def __init__(self, low: int, high: int, step: int = 1) -> None: super().__init__(low=low, high=high, log=True, step=step) def _asdict(self) -> Dict: d = copy.deepcopy(self.__dict__) d.pop("log") return d def _categorical_choice_equal( value1: CategoricalChoiceType, value2: CategoricalChoiceType ) -> bool: """A function to check two choices equal considering NaN. This function can handle NaNs like np.float32("nan") other than float. """ value1_is_nan = isinstance(value1, Real) and np.isnan(float(value1)) value2_is_nan = isinstance(value2, Real) and np.isnan(float(value2)) return (value1 == value2) or (value1_is_nan and value2_is_nan) class CategoricalDistribution(BaseDistribution): """A categorical distribution. This object is instantiated by :func:`~optuna.trial.Trial.suggest_categorical`, and passed to :mod:`~optuna.samplers` in general. Args: choices: Parameter value candidates. ``choices`` must have one element at least. .. note:: Not all types are guaranteed to be compatible with all storages. It is recommended to restrict the types of the choices to :obj:`None`, :class:`bool`, :class:`int`, :class:`float` and :class:`str`. Attributes: choices: Parameter value candidates. """ def __init__(self, choices: Sequence[CategoricalChoiceType]) -> None: if len(choices) == 0: raise ValueError("The `choices` must contain one or more elements.") for choice in choices: if choice is not None and not isinstance(choice, (bool, int, float, str)): message = ( "Choices for a categorical distribution should be a tuple of None, bool, " "int, float and str for persistent storage but contains {} which is of type " "{}.".format(choice, type(choice).__name__) ) warnings.warn(message) self.choices = tuple(choices) def to_external_repr(self, param_value_in_internal_repr: float) -> CategoricalChoiceType: return self.choices[int(param_value_in_internal_repr)] def to_internal_repr(self, param_value_in_external_repr: CategoricalChoiceType) -> float: try: # NOTE(nabenabe): With this implementation, we cannot distinguish some values # such as True and 1, or 1.0 and 1. For example, if choices=[True, 1] and external_repr # is 1, this method wrongly returns 0 instead of 1. However, we decided to accept this # bug for such exceptional choices for less complexity and faster processing. return self.choices.index(param_value_in_external_repr) except ValueError: # ValueError: param_value_in_external_repr is not in choices. # ValueError also happens if external_repr is nan or includes precision error in float. for index, choice in enumerate(self.choices): if _categorical_choice_equal(param_value_in_external_repr, choice): return index raise ValueError(f"'{param_value_in_external_repr}' not in {self.choices}.") def single(self) -> bool: return len(self.choices) == 1 def _contains(self, param_value_in_internal_repr: float) -> bool: index = int(param_value_in_internal_repr) return 0 <= index < len(self.choices) def __eq__(self, other: Any) -> bool: if not isinstance(other, BaseDistribution): return NotImplemented if not isinstance(other, self.__class__): return False if self.__dict__.keys() != other.__dict__.keys(): return False for key, value in self.__dict__.items(): if key == "choices": if len(value) != len(getattr(other, key)): return False for choice, other_choice in zip(value, getattr(other, key)): if not _categorical_choice_equal(choice, other_choice): return False else: if value != getattr(other, key): return False return True __hash__ = BaseDistribution.__hash__ DISTRIBUTION_CLASSES = ( IntDistribution, IntLogUniformDistribution, IntUniformDistribution, FloatDistribution, UniformDistribution, LogUniformDistribution, DiscreteUniformDistribution, CategoricalDistribution, ) def json_to_distribution(json_str: str) -> BaseDistribution: """Deserialize a distribution in JSON format. Args: json_str: A JSON-serialized distribution. Returns: A deserialized distribution. """ json_dict = json.loads(json_str) if "name" in json_dict: if json_dict["name"] == CategoricalDistribution.__name__: json_dict["attributes"]["choices"] = tuple(json_dict["attributes"]["choices"]) for cls in DISTRIBUTION_CLASSES: if json_dict["name"] == cls.__name__: return cls(**json_dict["attributes"]) raise ValueError("Unknown distribution class: {}".format(json_dict["name"])) else: # Deserialize a distribution from an abbreviated format. if json_dict["type"] == "categorical": return CategoricalDistribution(json_dict["choices"]) elif json_dict["type"] in ("float", "int"): low = json_dict["low"] high = json_dict["high"] step = json_dict.get("step") log = json_dict.get("log", False) if json_dict["type"] == "float": return FloatDistribution(low, high, log=log, step=step) else: if step is None: step = 1 return IntDistribution(low=low, high=high, log=log, step=step) raise ValueError("Unknown distribution type: {}".format(json_dict["type"])) def distribution_to_json(dist: BaseDistribution) -> str: """Serialize a distribution to JSON format. Args: dist: A distribution to be serialized. Returns: A JSON string of a given distribution. """ return json.dumps({"name": dist.__class__.__name__, "attributes": dist._asdict()}) def check_distribution_compatibility( dist_old: BaseDistribution, dist_new: BaseDistribution ) -> None: """A function to check compatibility of two distributions. It checks whether ``dist_old`` and ``dist_new`` are the same kind of distributions. If ``dist_old`` is :class:`~optuna.distributions.CategoricalDistribution`, it further checks ``choices`` are the same between ``dist_old`` and ``dist_new``. Note that this method is not supposed to be called by library users. Args: dist_old: A distribution previously recorded in storage. dist_new: A distribution newly added to storage. """ if dist_old.__class__ != dist_new.__class__: raise ValueError("Cannot set different distribution kind to the same parameter name.") if isinstance(dist_old, (FloatDistribution, IntDistribution)): # For mypy. assert isinstance(dist_new, (FloatDistribution, IntDistribution)) if dist_old.log != dist_new.log: raise ValueError("Cannot set different log configuration to the same parameter name.") if not isinstance(dist_old, CategoricalDistribution): return if not isinstance(dist_new, CategoricalDistribution): return if dist_old != dist_new: raise ValueError( CategoricalDistribution.__name__ + " does not support dynamic value space." ) def _adjust_discrete_uniform_high(low: float, high: float, step: float) -> float: d_high = decimal.Decimal(str(high)) d_low = decimal.Decimal(str(low)) d_step = decimal.Decimal(str(step)) d_r = d_high - d_low if d_r % d_step != decimal.Decimal("0"): old_high = high high = float((d_r // d_step) * d_step + d_low) warnings.warn( "The distribution is specified by [{low}, {old_high}] and step={step}, but the range " "is not divisible by `step`. It will be replaced by [{low}, {high}].".format( low=low, old_high=old_high, high=high, step=step ) ) return high def _adjust_int_uniform_high(low: int, high: int, step: int) -> int: r = high - low if r % step != 0: old_high = high high = r // step * step + low warnings.warn( "The distribution is specified by [{low}, {old_high}] and step={step}, but the range " "is not divisible by `step`. It will be replaced by [{low}, {high}].".format( low=low, old_high=old_high, high=high, step=step ) ) return high def _get_single_value(distribution: BaseDistribution) -> Union[int, float, CategoricalChoiceType]: assert distribution.single() if isinstance( distribution, ( FloatDistribution, IntDistribution, ), ): return distribution.low elif isinstance(distribution, CategoricalDistribution): return distribution.choices[0] assert False # TODO(himkt): Remove this method with the deletion of deprecated distributions. # https://github.com/optuna/optuna/issues/2941 def _convert_old_distribution_to_new_distribution( distribution: BaseDistribution, suppress_warning: bool = False, ) -> BaseDistribution: new_distribution: BaseDistribution # Float distributions. if isinstance(distribution, UniformDistribution): new_distribution = FloatDistribution( low=distribution.low, high=distribution.high, log=False, step=None, ) elif isinstance(distribution, LogUniformDistribution): new_distribution = FloatDistribution( low=distribution.low, high=distribution.high, log=True, step=None, ) elif isinstance(distribution, DiscreteUniformDistribution): new_distribution = FloatDistribution( low=distribution.low, high=distribution.high, log=False, step=distribution.q, ) # Integer distributions. elif isinstance(distribution, IntUniformDistribution): new_distribution = IntDistribution( low=distribution.low, high=distribution.high, log=False, step=distribution.step, ) elif isinstance(distribution, IntLogUniformDistribution): new_distribution = IntDistribution( low=distribution.low, high=distribution.high, log=True, step=distribution.step, ) # Categorical distribution. else: new_distribution = distribution if new_distribution != distribution and not suppress_warning: message = ( f"{distribution} is deprecated and internally converted to" f" {new_distribution}. See https://github.com/optuna/optuna/issues/2941." ) warnings.warn(message, FutureWarning) return new_distribution def _is_distribution_log(distribution: BaseDistribution) -> bool: if isinstance(distribution, (FloatDistribution, IntDistribution)): return distribution.log return False optuna-4.1.0/optuna/exceptions.py000066400000000000000000000046121471332314300170670ustar00rootroot00000000000000class OptunaError(Exception): """Base class for Optuna specific errors.""" pass class TrialPruned(OptunaError): """Exception for pruned trials. This error tells a trainer that the current :class:`~optuna.trial.Trial` was pruned. It is supposed to be raised after :func:`optuna.trial.Trial.should_prune` as shown in the following example. See also: :class:`optuna.TrialPruned` is an alias of :class:`optuna.exceptions.TrialPruned`. Example: .. testcode:: import numpy as np from sklearn.datasets import load_iris from sklearn.linear_model import SGDClassifier from sklearn.model_selection import train_test_split import optuna X, y = load_iris(return_X_y=True) X_train, X_valid, y_train, y_valid = train_test_split(X, y) classes = np.unique(y) def objective(trial): alpha = trial.suggest_float("alpha", 0.0, 1.0) clf = SGDClassifier(alpha=alpha) n_train_iter = 100 for step in range(n_train_iter): clf.partial_fit(X_train, y_train, classes=classes) intermediate_value = clf.score(X_valid, y_valid) trial.report(intermediate_value, step) if trial.should_prune(): raise optuna.TrialPruned() return clf.score(X_valid, y_valid) study = optuna.create_study(direction="maximize") study.optimize(objective, n_trials=20) """ pass class CLIUsageError(OptunaError): """Exception for CLI. CLI raises this exception when it receives invalid configuration. """ pass class StorageInternalError(OptunaError): """Exception for storage operation. This error is raised when an operation failed in backend DB of storage. """ pass class DuplicatedStudyError(OptunaError): """Exception for a duplicated study name. This error is raised when a specified study name already exists in the storage. """ pass class ExperimentalWarning(Warning): """Experimental Warning class. This implementation exists here because the policy of `FutureWarning` has been changed since Python 3.7 was released. See the details in https://docs.python.org/3/library/warnings.html#warning-categories. """ pass optuna-4.1.0/optuna/importance/000077500000000000000000000000001471332314300164725ustar00rootroot00000000000000optuna-4.1.0/optuna/importance/__init__.py000066400000000000000000000120761471332314300206110ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Callable from optuna._experimental import warn_experimental_argument from optuna.importance._base import BaseImportanceEvaluator from optuna.importance._fanova import FanovaImportanceEvaluator from optuna.importance._mean_decrease_impurity import MeanDecreaseImpurityImportanceEvaluator from optuna.importance._ped_anova import PedAnovaImportanceEvaluator from optuna.study import Study from optuna.trial import FrozenTrial __all__ = [ "BaseImportanceEvaluator", "FanovaImportanceEvaluator", "MeanDecreaseImpurityImportanceEvaluator", "PedAnovaImportanceEvaluator", "get_param_importances", ] def get_param_importances( study: Study, *, evaluator: BaseImportanceEvaluator | None = None, params: list[str] | None = None, target: Callable[[FrozenTrial], float] | None = None, normalize: bool = True, ) -> dict[str, float]: """Evaluate parameter importances based on completed trials in the given study. The parameter importances are returned as a dictionary where the keys consist of parameter names and their values importances. The importances are represented by non-negative floating point numbers, where higher values mean that the parameters are more important. The returned dictionary is ordered by its values in a descending order. By default, the sum of the importance values are normalized to 1.0. If ``params`` is :obj:`None`, all parameter that are present in all of the completed trials are assessed. This implies that conditional parameters will be excluded from the evaluation. To assess the importances of conditional parameters, a :obj:`list` of parameter names can be specified via ``params``. If specified, only completed trials that contain all of the parameters will be considered. If no such trials are found, an error will be raised. If the given study does not contain completed trials, an error will be raised. .. note:: If ``params`` is specified as an empty list, an empty dictionary is returned. .. seealso:: See :func:`~optuna.visualization.plot_param_importances` to plot importances. Args: study: An optimized study. evaluator: An importance evaluator object that specifies which algorithm to base the importance assessment on. Defaults to :class:`~optuna.importance.FanovaImportanceEvaluator`. .. note:: :class:`~optuna.importance.FanovaImportanceEvaluator` takes over 1 minute when given a study that contains 1000+ trials. We published `optuna-fast-fanova `__ library, that is a Cython accelerated fANOVA implementation. By using it, you can get hyperparameter importances within a few seconds. If ``n_trials`` is more than 10000, the Cython implementation takes more than a minute, so you can use :class:`~optuna.importance.PedAnovaImportanceEvaluator` instead, enabling the evaluation to finish in a second. params: A list of names of parameters to assess. If :obj:`None`, all parameters that are present in all of the completed trials are assessed. target: A function to specify the value to evaluate importances. If it is :obj:`None` and ``study`` is being used for single-objective optimization, the objective values are used. ``target`` must be specified if ``study`` is being used for multi-objective optimization. .. note:: Specify this argument if ``study`` is being used for multi-objective optimization. For example, to get the hyperparameter importance of the first objective, use ``target=lambda t: t.values[0]`` for the target parameter. normalize: A boolean option to specify whether the sum of the importance values should be normalized to 1.0. Defaults to :obj:`True`. .. note:: Added in v3.0.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v3.0.0. Returns: A :obj:`dict` where the keys are parameter names and the values are assessed importances. """ if evaluator is None: evaluator = FanovaImportanceEvaluator() if not isinstance(evaluator, BaseImportanceEvaluator): raise TypeError("Evaluator must be a subclass of BaseImportanceEvaluator.") res = evaluator.evaluate(study, params=params, target=target) if normalize: s = sum(res.values()) if s == 0.0: n_params = len(res) return dict((param, 1.0 / n_params) for param in res.keys()) else: return dict((param, value / s) for (param, value) in res.items()) else: warn_experimental_argument("normalize") return res optuna-4.1.0/optuna/importance/_base.py000066400000000000000000000146631471332314300201270ustar00rootroot00000000000000from __future__ import annotations import abc from collections.abc import Callable from collections.abc import Collection from typing import cast import numpy as np from optuna._transform import _SearchSpaceTransform from optuna.distributions import BaseDistribution from optuna.search_space import intersection_search_space from optuna.study import Study from optuna.trial import FrozenTrial from optuna.trial import TrialState class BaseImportanceEvaluator(abc.ABC): """Abstract parameter importance evaluator.""" @abc.abstractmethod def evaluate( self, study: Study, params: list[str] | None = None, *, target: Callable[[FrozenTrial], float] | None = None, ) -> dict[str, float]: """Evaluate parameter importances based on completed trials in the given study. .. note:: This method is not meant to be called by library users. .. seealso:: Please refer to :func:`~optuna.importance.get_param_importances` for how a concrete evaluator should implement this method. Args: study: An optimized study. params: A list of names of parameters to assess. If :obj:`None`, all parameters that are present in all of the completed trials are assessed. target: A function to specify the value to evaluate importances. If it is :obj:`None` and ``study`` is being used for single-objective optimization, the objective values are used. Can also be used for other trial attributes, such as the duration, like ``target=lambda t: t.duration.total_seconds()``. .. note:: Specify this argument if ``study`` is being used for multi-objective optimization. For example, to get the hyperparameter importance of the first objective, use ``target=lambda t: t.values[0]`` for the target parameter. Returns: A :obj:`dict` where the keys are parameter names and the values are assessed importances. """ # TODO(hvy): Reconsider the interface as logic might violate DRY among multiple evaluators. raise NotImplementedError def _get_distributions(study: Study, params: list[str] | None) -> dict[str, BaseDistribution]: completed_trials = study.get_trials(deepcopy=False, states=(TrialState.COMPLETE,)) _check_evaluate_args(completed_trials, params) if params is None: return intersection_search_space(study.get_trials(deepcopy=False)) # New temporary required to pass mypy. Seems like a bug. params_not_none = params assert params_not_none is not None # Compute the search space based on the subset of trials containing all parameters. distributions = None for trial in completed_trials: trial_distributions = trial.distributions if not all(name in trial_distributions for name in params_not_none): continue if distributions is None: distributions = dict( filter( lambda name_and_distribution: name_and_distribution[0] in params_not_none, trial_distributions.items(), ) ) continue if any( trial_distributions[name] != distribution for name, distribution in distributions.items() ): raise ValueError( "Parameters importances cannot be assessed with dynamic search spaces if " "parameters are specified. Specified parameters: {}.".format(params) ) assert distributions is not None # Required to pass mypy. distributions = dict( sorted(distributions.items(), key=lambda name_and_distribution: name_and_distribution[0]) ) return distributions def _check_evaluate_args(completed_trials: list[FrozenTrial], params: list[str] | None) -> None: if len(completed_trials) == 0: raise ValueError("Cannot evaluate parameter importances without completed trials.") if len(completed_trials) == 1: raise ValueError("Cannot evaluate parameter importances with only a single trial.") if params is not None: if not isinstance(params, (list, tuple)): raise TypeError( "Parameters must be specified as a list. Actual parameters: {}.".format(params) ) if any(not isinstance(p, str) for p in params): raise TypeError( "Parameters must be specified by their names with strings. Actual parameters: " "{}.".format(params) ) if len(params) > 0: at_least_one_trial = False for trial in completed_trials: if all(p in trial.distributions for p in params): at_least_one_trial = True break if not at_least_one_trial: raise ValueError( "Study must contain completed trials with all specified parameters. " "Specified parameters: {}.".format(params) ) def _get_filtered_trials( study: Study, params: Collection[str], target: Callable[[FrozenTrial], float] | None ) -> list[FrozenTrial]: trials = study.get_trials(deepcopy=False, states=(TrialState.COMPLETE,)) return [ trial for trial in trials if set(params) <= set(trial.params) and np.isfinite(target(trial) if target is not None else cast(float, trial.value)) ] def _param_importances_to_dict( params: Collection[str], param_importances: np.ndarray | float ) -> dict[str, float]: return { name: value for name, value in zip(params, np.broadcast_to(param_importances, (len(params),))) } def _get_trans_params(trials: list[FrozenTrial], trans: _SearchSpaceTransform) -> np.ndarray: return np.array([trans.transform(trial.params) for trial in trials]) def _get_target_values( trials: list[FrozenTrial], target: Callable[[FrozenTrial], float] | None ) -> np.ndarray: return np.array([target(trial) if target is not None else trial.value for trial in trials]) def _sort_dict_by_importance(param_importances: dict[str, float]) -> dict[str, float]: return dict( reversed( sorted( param_importances.items(), key=lambda name_and_importance: name_and_importance[1] ) ) ) optuna-4.1.0/optuna/importance/_fanova/000077500000000000000000000000001471332314300201035ustar00rootroot00000000000000optuna-4.1.0/optuna/importance/_fanova/__init__.py000066400000000000000000000001651471332314300222160ustar00rootroot00000000000000from optuna.importance._fanova._evaluator import FanovaImportanceEvaluator __all__ = ["FanovaImportanceEvaluator"] optuna-4.1.0/optuna/importance/_fanova/_evaluator.py000066400000000000000000000117621471332314300226250ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Callable import numpy as np from optuna._transform import _SearchSpaceTransform from optuna.importance._base import _get_distributions from optuna.importance._base import _get_filtered_trials from optuna.importance._base import _get_target_values from optuna.importance._base import _get_trans_params from optuna.importance._base import _param_importances_to_dict from optuna.importance._base import _sort_dict_by_importance from optuna.importance._base import BaseImportanceEvaluator from optuna.importance._fanova._fanova import _Fanova from optuna.study import Study from optuna.trial import FrozenTrial class FanovaImportanceEvaluator(BaseImportanceEvaluator): """fANOVA importance evaluator. Implements the fANOVA hyperparameter importance evaluation algorithm in `An Efficient Approach for Assessing Hyperparameter Importance `__. fANOVA fits a random forest regression model that predicts the objective values of :class:`~optuna.trial.TrialState.COMPLETE` trials given their parameter configurations. The more accurate this model is, the more reliable the importances assessed by this class are. .. note:: This class takes over 1 minute when given a study that contains 1000+ trials. We published `optuna-fast-fanova `__ library, that is a Cython accelerated fANOVA implementation. By using it, you can get hyperparameter importances within a few seconds. .. note:: Requires the `sklearn `__ Python package. .. note:: The performance of fANOVA depends on the prediction performance of the underlying random forest model. In order to obtain high prediction performance, it is necessary to cover a wide range of the hyperparameter search space. It is recommended to use an exploration-oriented sampler such as :class:`~optuna.samplers.RandomSampler`. .. note:: For how to cite the original work, please refer to https://automl.github.io/fanova/cite.html. Args: n_trees: The number of trees in the forest. max_depth: The maximum depth of the trees in the forest. seed: Controls the randomness of the forest. For deterministic behavior, specify a value other than :obj:`None`. """ def __init__(self, *, n_trees: int = 64, max_depth: int = 64, seed: int | None = None) -> None: self._evaluator = _Fanova( n_trees=n_trees, max_depth=max_depth, min_samples_split=2, min_samples_leaf=1, seed=seed, ) def evaluate( self, study: Study, params: list[str] | None = None, *, target: Callable[[FrozenTrial], float] | None = None, ) -> dict[str, float]: if target is None and study._is_multi_objective(): raise ValueError( "If the `study` is being used for multi-objective optimization, " "please specify the `target`. For example, use " "`target=lambda t: t.values[0]` for the first objective value." ) distributions = _get_distributions(study, params=params) if params is None: params = list(distributions.keys()) assert params is not None # fANOVA does not support parameter distributions with a single value. # However, there is no reason to calculate parameter importance in such case anyway, # since it will always be 0 as the parameter is constant in the objective function. non_single_distributions = { name: dist for name, dist in distributions.items() if not dist.single() } single_distributions = { name: dist for name, dist in distributions.items() if dist.single() } if len(non_single_distributions) == 0: return {} trials: list[FrozenTrial] = _get_filtered_trials(study, params=params, target=target) trans = _SearchSpaceTransform( non_single_distributions, transform_log=False, transform_step=False ) trans_params: np.ndarray = _get_trans_params(trials, trans) target_values: np.ndarray = _get_target_values(trials, target) evaluator = self._evaluator evaluator.fit( X=trans_params, y=target_values, search_spaces=trans.bounds, column_to_encoded_columns=trans.column_to_encoded_columns, ) param_importances = np.array( [evaluator.get_importance(i)[0] for i in range(len(non_single_distributions))] ) return _sort_dict_by_importance( { **_param_importances_to_dict(non_single_distributions.keys(), param_importances), **_param_importances_to_dict(single_distributions.keys(), 0.0), } ) optuna-4.1.0/optuna/importance/_fanova/_fanova.py000066400000000000000000000077301471332314300220750ustar00rootroot00000000000000"""An implementation of `An Efficient Approach for Assessing Hyperparameter Importance`. See http://proceedings.mlr.press/v32/hutter14.pdf and https://automl.github.io/fanova/cite.html for how to cite the original work. This implementation is inspired by the efficient algorithm in `fanova` (https://github.com/automl/fanova) and `pyrfr` (https://github.com/automl/random_forest_run) by the original authors. Differences include relying on scikit-learn to fit random forests (`sklearn.ensemble.RandomForestRegressor`) and that it is otherwise written entirely in Python. This stands in contrast to the original implementation which is partially written in C++. Since Python runtime overhead may become noticeable, included are instead several optimizations, e.g. vectorized NumPy functions to compute the marginals, instead of keeping all running statistics. Known cases include assessing categorical features with a larger number of choices since each choice is given a unique one-hot encoded raw feature. """ from __future__ import annotations import numpy as np from optuna._imports import try_import from optuna.importance._fanova._tree import _FanovaTree with try_import() as _imports: from sklearn.ensemble import RandomForestRegressor class _Fanova: def __init__( self, n_trees: int, max_depth: int, min_samples_split: int | float, min_samples_leaf: int | float, seed: int | None, ) -> None: _imports.check() self._forest = RandomForestRegressor( n_estimators=n_trees, max_depth=max_depth, min_samples_split=min_samples_split, min_samples_leaf=min_samples_leaf, random_state=seed, ) self._trees: list[_FanovaTree] | None = None self._variances: dict[int, np.ndarray] | None = None self._column_to_encoded_columns: list[np.ndarray] | None = None def fit( self, X: np.ndarray, y: np.ndarray, search_spaces: np.ndarray, column_to_encoded_columns: list[np.ndarray], ) -> None: assert X.shape[0] == y.shape[0] assert X.shape[1] == search_spaces.shape[0] assert search_spaces.shape[1] == 2 self._forest.fit(X, y) self._trees = [_FanovaTree(e.tree_, search_spaces) for e in self._forest.estimators_] self._column_to_encoded_columns = column_to_encoded_columns self._variances = {} if all(tree.variance == 0 for tree in self._trees): # If all trees have 0 variance, we cannot assess any importances. # This could occur if for instance `X.shape[0] == 1`. raise RuntimeError("Encountered zero total variance in all trees.") def get_importance(self, feature: int) -> tuple[float, float]: # Assert that `fit` has been called. assert self._trees is not None assert self._variances is not None self._compute_variances(feature) fractions: list[float] | np.ndarray = [] for tree_index, tree in enumerate(self._trees): tree_variance = tree.variance if tree_variance > 0.0: fraction = self._variances[feature][tree_index] / tree_variance fractions = np.append(fractions, fraction) fractions = np.asarray(fractions) return float(fractions.mean()), float(fractions.std()) def _compute_variances(self, feature: int) -> None: assert self._trees is not None assert self._variances is not None assert self._column_to_encoded_columns is not None if feature in self._variances: return raw_features = self._column_to_encoded_columns[feature] variances = np.empty(len(self._trees), dtype=np.float64) for tree_index, tree in enumerate(self._trees): marginal_variance = tree.get_marginal_variance(raw_features) variances[tree_index] = np.clip(marginal_variance, 0.0, None) self._variances[feature] = variances optuna-4.1.0/optuna/importance/_fanova/_tree.py000066400000000000000000000306671471332314300215670ustar00rootroot00000000000000from __future__ import annotations from functools import lru_cache import itertools from typing import TYPE_CHECKING import numpy as np if TYPE_CHECKING: import sklearn.tree class _FanovaTree: def __init__(self, tree: "sklearn.tree._tree.Tree", search_spaces: np.ndarray) -> None: assert search_spaces.shape[0] == tree.n_features assert search_spaces.shape[1] == 2 self._tree = tree self._search_spaces = search_spaces statistics = self._precompute_statistics() split_midpoints, split_sizes = self._precompute_split_midpoints_and_sizes() subtree_active_features = self._precompute_subtree_active_features() self._statistics = statistics self._split_midpoints = split_midpoints self._split_sizes = split_sizes self._subtree_active_features = subtree_active_features self._variance = None # Computed lazily and requires `self._statistics`. @property def variance(self) -> float: if self._variance is None: leaf_node_indices = np.nonzero(np.array(self._tree.feature) < 0)[0] statistics = self._statistics[leaf_node_indices] values = statistics[:, 0] weights = statistics[:, 1] average_values = np.average(values, weights=weights) variance = np.average((values - average_values) ** 2, weights=weights) self._variance = variance assert self._variance is not None return self._variance def get_marginal_variance(self, features: np.ndarray) -> float: assert features.size > 0 # For each midpoint along the given dimensions, traverse this tree to compute the # marginal predictions. selected_midpoints = [self._split_midpoints[f] for f in features] selected_sizes = [self._split_sizes[f] for f in features] product_midpoints = itertools.product(*selected_midpoints) product_sizes = itertools.product(*selected_sizes) sample = np.full(self._n_features, fill_value=np.nan, dtype=np.float64) values: list[float] | np.ndarray = [] weights: list[float] | np.ndarray = [] for midpoints, sizes in zip(product_midpoints, product_sizes): sample[features] = np.array(midpoints) value, weight = self._get_marginalized_statistics(sample) weight *= float(np.prod(sizes)) values = np.append(values, value) weights = np.append(weights, weight) weights = np.asarray(weights) values = np.asarray(values) average_values = np.average(values, weights=weights) variance = np.average((values - average_values) ** 2, weights=weights) assert variance >= 0.0 return variance def _get_marginalized_statistics(self, feature_vector: np.ndarray) -> tuple[float, float]: assert feature_vector.size == self._n_features marginalized_features = np.isnan(feature_vector) active_features = ~marginalized_features # Reduce search space cardinalities to 1 for non-active features. search_spaces = self._search_spaces.copy() search_spaces[marginalized_features] = [0.0, 1.0] # Start from the root and traverse towards the leafs. active_nodes = [0] active_search_spaces = [search_spaces] node_indices = [] active_leaf_search_spaces = [] while len(active_nodes) > 0: node_index = active_nodes.pop() search_spaces = active_search_spaces.pop() feature = self._get_node_split_feature(node_index) if feature >= 0: # Not leaf. Avoid unnecessary call to `_is_node_leaf`. # If node splits on an active feature, push the child node that we end up in. response = feature_vector[feature] if not np.isnan(response): if response <= self._get_node_split_threshold(node_index): next_node_index = self._get_node_left_child(node_index) next_subspace = self._get_node_left_child_subspaces( node_index, search_spaces ) else: next_node_index = self._get_node_right_child(node_index) next_subspace = self._get_node_right_child_subspaces( node_index, search_spaces ) active_nodes.append(next_node_index) active_search_spaces.append(next_subspace) continue # If subtree starting from node splits on an active feature, push both child nodes. # Here, we use `any` for list because `ndarray.any` is slow. if any(self._subtree_active_features[node_index][active_features].tolist()): for child_node_index in self._get_node_children(node_index): active_nodes.append(child_node_index) active_search_spaces.append(search_spaces) continue # If node is a leaf or the subtree does not split on any of the active features. node_indices.append(node_index) active_leaf_search_spaces.append(search_spaces) statistics = self._statistics[node_indices] values = statistics[:, 0] weights = statistics[:, 1] active_features_cardinalities = _get_cardinality_batched(active_leaf_search_spaces) weights = weights / active_features_cardinalities value = np.average(values, weights=weights) weight = weights.sum() return value, weight def _precompute_statistics(self) -> np.ndarray: n_nodes = self._n_nodes # Holds for each node, its weighted average value and the sum of weights. statistics = np.empty((n_nodes, 2), dtype=np.float64) subspaces = np.array([None for _ in range(n_nodes)]) subspaces[0] = self._search_spaces # Compute marginals for leaf nodes. for node_index in range(n_nodes): subspace = subspaces[node_index] if self._is_node_leaf(node_index): value = self._get_node_value(node_index) weight = _get_cardinality(subspace) statistics[node_index] = [value, weight] else: for child_node_index, child_subspace in zip( self._get_node_children(node_index), self._get_node_children_subspaces(node_index, subspace), ): assert subspaces[child_node_index] is None subspaces[child_node_index] = child_subspace # Compute marginals for internal nodes. for node_index in reversed(range(n_nodes)): if not self._is_node_leaf(node_index): child_values = [] child_weights = [] for child_node_index in self._get_node_children(node_index): child_values.append(statistics[child_node_index, 0]) child_weights.append(statistics[child_node_index, 1]) value = np.average(child_values, weights=child_weights) weight = float(np.sum(child_weights)) statistics[node_index] = [value, weight] return statistics def _precompute_split_midpoints_and_sizes( self, ) -> tuple[list[np.ndarray], list[np.ndarray]]: midpoints = [] sizes = [] search_spaces = self._search_spaces for feature, feature_split_values in enumerate(self._compute_features_split_values()): feature_split_values = np.concatenate( ( np.atleast_1d(search_spaces[feature, 0]), feature_split_values, np.atleast_1d(search_spaces[feature, 1]), ) ) midpoint = 0.5 * (feature_split_values[1:] + feature_split_values[:-1]) size = feature_split_values[1:] - feature_split_values[:-1] midpoints.append(midpoint) sizes.append(size) return midpoints, sizes def _compute_features_split_values(self) -> list[np.ndarray]: all_split_values: list[set[float]] = [set() for _ in range(self._n_features)] for node_index in range(self._n_nodes): feature = self._get_node_split_feature(node_index) if feature >= 0: # Not leaf. Avoid unnecessary call to `_is_node_leaf`. threshold = self._get_node_split_threshold(node_index) all_split_values[feature].add(threshold) sorted_all_split_values: list[np.ndarray] = [] for split_values in all_split_values: split_values_array = np.array(list(split_values), dtype=np.float64) split_values_array.sort() sorted_all_split_values.append(split_values_array) return sorted_all_split_values def _precompute_subtree_active_features(self) -> np.ndarray: subtree_active_features = np.full((self._n_nodes, self._n_features), fill_value=False) for node_index in reversed(range(self._n_nodes)): feature = self._get_node_split_feature(node_index) if feature >= 0: # Not leaf. Avoid unnecessary call to `_is_node_leaf`. subtree_active_features[node_index, feature] = True for child_node_index in self._get_node_children(node_index): subtree_active_features[node_index] |= subtree_active_features[ child_node_index ] return subtree_active_features @property def _n_features(self) -> int: return len(self._search_spaces) @property def _n_nodes(self) -> int: return self._tree.node_count @lru_cache(maxsize=None) def _is_node_leaf(self, node_index: int) -> bool: return self._tree.feature[node_index] < 0 @lru_cache(maxsize=None) def _get_node_left_child(self, node_index: int) -> int: return self._tree.children_left[node_index] @lru_cache(maxsize=None) def _get_node_right_child(self, node_index: int) -> int: return self._tree.children_right[node_index] @lru_cache(maxsize=None) def _get_node_children(self, node_index: int) -> tuple[int, int]: return self._get_node_left_child(node_index), self._get_node_right_child(node_index) @lru_cache(maxsize=None) def _get_node_value(self, node_index: int) -> float: # self._tree.value: sklearn.tree._tree.Tree.value has # the shape (node_count, n_outputs, max_n_classes) return float(self._tree.value[node_index].reshape(-1)[0]) @lru_cache(maxsize=None) def _get_node_split_threshold(self, node_index: int) -> float: return self._tree.threshold[node_index] @lru_cache(maxsize=None) def _get_node_split_feature(self, node_index: int) -> int: return self._tree.feature[node_index] def _get_node_left_child_subspaces( self, node_index: int, search_spaces: np.ndarray ) -> np.ndarray: return _get_subspaces( search_spaces, search_spaces_column=1, feature=self._get_node_split_feature(node_index), threshold=self._get_node_split_threshold(node_index), ) def _get_node_right_child_subspaces( self, node_index: int, search_spaces: np.ndarray ) -> np.ndarray: return _get_subspaces( search_spaces, search_spaces_column=0, feature=self._get_node_split_feature(node_index), threshold=self._get_node_split_threshold(node_index), ) def _get_node_children_subspaces( self, node_index: int, search_spaces: np.ndarray ) -> tuple[np.ndarray, np.ndarray]: return ( self._get_node_left_child_subspaces(node_index, search_spaces), self._get_node_right_child_subspaces(node_index, search_spaces), ) def _get_cardinality(search_spaces: np.ndarray) -> float: return np.prod(search_spaces[:, 1] - search_spaces[:, 0]) def _get_cardinality_batched(search_spaces_list: list[np.ndarray]) -> float: search_spaces = np.asarray(search_spaces_list) return np.prod(search_spaces[:, :, 1] - search_spaces[:, :, 0], axis=1) def _get_subspaces( search_spaces: np.ndarray, *, search_spaces_column: int, feature: int, threshold: float ) -> np.ndarray: search_spaces_subspace = np.copy(search_spaces) search_spaces_subspace[feature, search_spaces_column] = threshold return search_spaces_subspace optuna-4.1.0/optuna/importance/_mean_decrease_impurity.py000066400000000000000000000073021471332314300237220ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Callable import numpy as np from optuna._imports import try_import from optuna._transform import _SearchSpaceTransform from optuna.importance._base import _get_distributions from optuna.importance._base import _get_filtered_trials from optuna.importance._base import _get_target_values from optuna.importance._base import _get_trans_params from optuna.importance._base import _param_importances_to_dict from optuna.importance._base import _sort_dict_by_importance from optuna.importance._base import BaseImportanceEvaluator from optuna.study import Study from optuna.trial import FrozenTrial with try_import() as _imports: from sklearn.ensemble import RandomForestRegressor class MeanDecreaseImpurityImportanceEvaluator(BaseImportanceEvaluator): """Mean Decrease Impurity (MDI) parameter importance evaluator. This evaluator fits fits a random forest regression model that predicts the objective values of :class:`~optuna.trial.TrialState.COMPLETE` trials given their parameter configurations. Feature importances are then computed using MDI. .. note:: This evaluator requires the `sklearn `__ Python package and is based on `sklearn.ensemble.RandomForestClassifier.feature_importances_ `__. Args: n_trees: Number of trees in the random forest. max_depth: The maximum depth of each tree in the random forest. seed: Seed for the random forest. """ def __init__(self, *, n_trees: int = 64, max_depth: int = 64, seed: int | None = None) -> None: _imports.check() self._forest = RandomForestRegressor( n_estimators=n_trees, max_depth=max_depth, min_samples_split=2, min_samples_leaf=1, random_state=seed, ) self._trans_params = np.empty(0) self._trans_values = np.empty(0) self._param_names: list[str] = list() def evaluate( self, study: Study, params: list[str] | None = None, *, target: Callable[[FrozenTrial], float] | None = None, ) -> dict[str, float]: if target is None and study._is_multi_objective(): raise ValueError( "If the `study` is being used for multi-objective optimization, " "please specify the `target`. For example, use " "`target=lambda t: t.values[0]` for the first objective value." ) distributions = _get_distributions(study, params=params) if params is None: params = list(distributions.keys()) assert params is not None if len(params) == 0: return {} trials: list[FrozenTrial] = _get_filtered_trials(study, params=params, target=target) trans = _SearchSpaceTransform(distributions, transform_log=False, transform_step=False) trans_params: np.ndarray = _get_trans_params(trials, trans) target_values: np.ndarray = _get_target_values(trials, target) forest = self._forest forest.fit(X=trans_params, y=target_values) feature_importances = forest.feature_importances_ # Untransform feature importances to param importances # by adding up relevant feature importances. param_importances = np.zeros(len(params)) np.add.at(param_importances, trans.encoded_column_to_column, feature_importances) return _sort_dict_by_importance(_param_importances_to_dict(params, param_importances)) optuna-4.1.0/optuna/importance/_ped_anova/000077500000000000000000000000001471332314300205655ustar00rootroot00000000000000optuna-4.1.0/optuna/importance/_ped_anova/__init__.py000066400000000000000000000001731471332314300226770ustar00rootroot00000000000000from optuna.importance._ped_anova.evaluator import PedAnovaImportanceEvaluator __all__ = ["PedAnovaImportanceEvaluator"] optuna-4.1.0/optuna/importance/_ped_anova/evaluator.py000066400000000000000000000224161471332314300231460ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Callable import warnings import numpy as np from optuna._experimental import experimental_class from optuna.distributions import BaseDistribution from optuna.importance._base import _get_distributions from optuna.importance._base import _get_filtered_trials from optuna.importance._base import _sort_dict_by_importance from optuna.importance._base import BaseImportanceEvaluator from optuna.importance._ped_anova.scott_parzen_estimator import _build_parzen_estimator from optuna.logging import get_logger from optuna.study import Study from optuna.study import StudyDirection from optuna.trial import FrozenTrial _logger = get_logger(__name__) class _QuantileFilter: def __init__( self, quantile: float, is_lower_better: bool, min_n_top_trials: int, target: Callable[[FrozenTrial], float] | None, ): assert 0 <= quantile <= 1, "quantile must be in [0, 1]." assert min_n_top_trials > 0, "min_n_top_trials must be positive." self._quantile = quantile self._is_lower_better = is_lower_better self._min_n_top_trials = min_n_top_trials self._target = target def filter(self, trials: list[FrozenTrial]) -> list[FrozenTrial]: target, min_n_top_trials = self._target, self._min_n_top_trials sign = 1.0 if self._is_lower_better else -1.0 loss_values = sign * np.asarray([t.value if target is None else target(t) for t in trials]) err_msg = "len(trials) must be larger than or equal to min_n_top_trials" assert min_n_top_trials <= loss_values.size, err_msg def _quantile(v: np.ndarray, q: float) -> float: cutoff_index = int(np.ceil(q * loss_values.size)) - 1 return float(np.partition(loss_values, cutoff_index)[cutoff_index]) cutoff_val = max( np.partition(loss_values, min_n_top_trials - 1)[min_n_top_trials - 1], # TODO(nabenabe0928): After dropping Python3.10, replace below with # np.quantile(loss_values, self._quantile, method="inverted_cdf"). _quantile(loss_values, self._quantile), ) should_keep_trials = loss_values <= cutoff_val return [t for t, should_keep in zip(trials, should_keep_trials) if should_keep] @experimental_class("3.6.0") class PedAnovaImportanceEvaluator(BaseImportanceEvaluator): """PED-ANOVA importance evaluator. Implements the PED-ANOVA hyperparameter importance evaluation algorithm. PED-ANOVA fits Parzen estimators of :class:`~optuna.trial.TrialState.COMPLETE` trials better than a user-specified baseline. Users can specify the baseline by a quantile. The importance can be interpreted as how important each hyperparameter is to get the performance better than baseline. For further information about PED-ANOVA algorithm, please refer to the following paper: - `PED-ANOVA: Efficiently Quantifying Hyperparameter Importance in Arbitrary Subspaces `__ .. note:: The performance of PED-ANOVA depends on how many trials to consider above baseline. To stabilize the analysis, it is preferable to include at least 5 trials above baseline. .. note:: Please refer to `the original work `__. Args: baseline_quantile: Compute the importance of achieving top-``baseline_quantile`` quantile objective value. For example, ``baseline_quantile=0.1`` means that the importances give the information of which parameters were important to achieve the top-10% performance during optimization. evaluate_on_local: Whether we measure the importance in the local or global space. If :obj:`True`, the importances imply how importance each parameter is during optimization. Meanwhile, ``evaluate_on_local=False`` gives the importances in the specified search_space. ``evaluate_on_local=True`` is especially useful when users modify search space during optimization. Example: An example of using PED-ANOVA is as follows: .. testcode:: import optuna from optuna.importance import PedAnovaImportanceEvaluator def objective(trial): x1 = trial.suggest_float("x1", -10, 10) x2 = trial.suggest_float("x2", -10, 10) return x1 + x2 / 1000 study = optuna.create_study() study.optimize(objective, n_trials=100) evaluator = PedAnovaImportanceEvaluator() importance = optuna.importance.get_param_importances(study, evaluator=evaluator) """ def __init__( self, *, baseline_quantile: float = 0.1, evaluate_on_local: bool = True, ): assert 0.0 <= baseline_quantile <= 1.0, "baseline_quantile must be in [0, 1]." self._baseline_quantile = baseline_quantile self._evaluate_on_local = evaluate_on_local # Advanced Setups. # Discretize a domain [low, high] as `np.linspace(low, high, n_steps)`. self._n_steps: int = 50 # Prior is used for regularization. self._consider_prior = True # Control the regularization effect. self._prior_weight = 1.0 # How many `trials` must be included in `top_trials`. self._min_n_top_trials = 2 def _get_top_trials( self, study: Study, trials: list[FrozenTrial], params: list[str], target: Callable[[FrozenTrial], float] | None, ) -> list[FrozenTrial]: is_lower_better = study.directions[0] == StudyDirection.MINIMIZE if target is not None: warnings.warn( f"{self.__class__.__name__} computes the importances of params to achieve " "low `target` values. If this is not what you want, " "please modify target, e.g., by multiplying the output by -1." ) is_lower_better = True top_trials = _QuantileFilter( self._baseline_quantile, is_lower_better, self._min_n_top_trials, target ).filter(trials) if len(trials) == len(top_trials): _logger.warning("All trials are in top trials, which gives equal importances.") return top_trials def _compute_pearson_divergence( self, param_name: str, dist: BaseDistribution, top_trials: list[FrozenTrial], all_trials: list[FrozenTrial], ) -> float: # When pdf_all == pdf_top, i.e. all_trials == top_trials, this method will give 0.0. consider_prior, prior_weight = self._consider_prior, self._prior_weight pe_top = _build_parzen_estimator( param_name, dist, top_trials, self._n_steps, consider_prior, prior_weight ) # NOTE: pe_top.n_steps could be different from self._n_steps. grids = np.arange(pe_top.n_steps) pdf_top = pe_top.pdf(grids) + 1e-12 if self._evaluate_on_local: # The importance of param during the study. pe_local = _build_parzen_estimator( param_name, dist, all_trials, self._n_steps, consider_prior, prior_weight ) pdf_local = pe_local.pdf(grids) + 1e-12 else: # The importance of param in the search space. pdf_local = np.full(pe_top.n_steps, 1.0 / pe_top.n_steps) return float(pdf_local @ ((pdf_top / pdf_local - 1) ** 2)) def evaluate( self, study: Study, params: list[str] | None = None, *, target: Callable[[FrozenTrial], float] | None = None, ) -> dict[str, float]: dists = _get_distributions(study, params=params) if params is None: params = list(dists.keys()) assert params is not None # PED-ANOVA does not support parameter distributions with a single value, # because the importance of such params become zero. non_single_dists = {name: dist for name, dist in dists.items() if not dist.single()} single_dists = {name: dist for name, dist in dists.items() if dist.single()} if len(non_single_dists) == 0: return {} trials = _get_filtered_trials(study, params=params, target=target) n_params = len(non_single_dists) # The following should be tested at _get_filtered_trials. assert target is not None or max([len(t.values) for t in trials], default=1) == 1 if len(trials) <= self._min_n_top_trials: param_importances = {k: 1.0 / n_params for k in non_single_dists} param_importances.update({k: 0.0 for k in single_dists}) return {k: 0.0 for k in param_importances} top_trials = self._get_top_trials(study, trials, params, target) quantile = len(top_trials) / len(trials) importance_sum = 0.0 param_importances = {} for param_name, dist in non_single_dists.items(): param_importances[param_name] = quantile * self._compute_pearson_divergence( param_name, dist, top_trials=top_trials, all_trials=trials ) importance_sum += param_importances[param_name] param_importances.update({k: 0.0 for k in single_dists}) return _sort_dict_by_importance(param_importances) optuna-4.1.0/optuna/importance/_ped_anova/scott_parzen_estimator.py000066400000000000000000000154331471332314300257470ustar00rootroot00000000000000from __future__ import annotations import numpy as np from optuna.distributions import BaseDistribution from optuna.distributions import CategoricalDistribution from optuna.distributions import FloatDistribution from optuna.distributions import IntDistribution from optuna.samplers._tpe.parzen_estimator import _ParzenEstimator from optuna.samplers._tpe.parzen_estimator import _ParzenEstimatorParameters from optuna.samplers._tpe.probability_distributions import _BatchedDiscreteTruncNormDistributions from optuna.samplers._tpe.probability_distributions import _BatchedDistributions from optuna.trial import FrozenTrial class _ScottParzenEstimator(_ParzenEstimator): """1D ParzenEstimator using the bandwidth selection by Scott's rule.""" def __init__( self, param_name: str, dist: IntDistribution | CategoricalDistribution, counts: np.ndarray, consider_prior: bool, prior_weight: float, ): assert isinstance(dist, (CategoricalDistribution, IntDistribution)) assert not isinstance(dist, IntDistribution) or dist.low == 0 n_choices = dist.high + 1 if isinstance(dist, IntDistribution) else len(dist.choices) assert len(counts) == n_choices, counts self._n_steps = len(counts) self._param_name = param_name self._counts = counts.copy() super().__init__( observations={param_name: np.arange(self._n_steps)[counts > 0.0]}, search_space={param_name: dist}, parameters=_ParzenEstimatorParameters( consider_prior=consider_prior, prior_weight=prior_weight, consider_magic_clip=False, consider_endpoints=False, weights=lambda x: np.empty(0), multivariate=True, categorical_distance_func={}, ), predetermined_weights=counts[counts > 0.0], ) def _calculate_numerical_distributions( self, observations: np.ndarray, low: float, # The type is actually int, but typing follows the original. high: float, # The type is actually int, but typing follows the original. step: float | None, parameters: _ParzenEstimatorParameters, ) -> _BatchedDistributions: # NOTE: The Optuna TPE bandwidth selection is too wide for this analysis. # So use the Scott's rule by Scott, D.W. (1992), # Multivariate Density Estimation: Theory, Practice, and Visualization. assert step is not None and np.isclose(step, 1.0), "MyPy redefinition." n_trials = np.sum(self._counts) counts_non_zero = self._counts[self._counts > 0] weights = counts_non_zero / n_trials mus = np.arange(self.n_steps)[self._counts > 0] mean_est = mus @ weights sigma_est = np.sqrt((mus - mean_est) ** 2 @ counts_non_zero / max(1, n_trials - 1)) count_cum = np.cumsum(counts_non_zero) idx_q25 = np.searchsorted(count_cum, n_trials // 4, side="left") idx_q75 = np.searchsorted(count_cum, n_trials * 3 // 4, side="right") interquantile_range = mus[min(mus.size - 1, idx_q75)] - mus[idx_q25] sigma_est = 1.059 * min(interquantile_range / 1.34, sigma_est) * n_trials ** (-0.2) # To avoid numerical errors. 0.5/1.64 means 1.64sigma (=90%) will fit in the target grid. sigma_min = 0.5 / 1.64 sigmas = np.full_like(mus, max(sigma_est, sigma_min), dtype=np.float64) if parameters.consider_prior: mus = np.append(mus, [0.5 * (low + high)]) sigmas = np.append(sigmas, [1.0 * (high - low + 1)]) return _BatchedDiscreteTruncNormDistributions( mu=mus, sigma=sigmas, low=0, high=self.n_steps - 1, step=1 ) @property def n_steps(self) -> int: return self._n_steps def pdf(self, samples: np.ndarray) -> np.ndarray: return np.exp(self.log_pdf({self._param_name: samples})) def _get_grids_and_grid_indices_of_trials( param_name: str, dist: IntDistribution | FloatDistribution, trials: list[FrozenTrial], n_steps: int, ) -> tuple[int, np.ndarray]: assert isinstance(dist, (FloatDistribution, IntDistribution)), "Unexpected distribution." if isinstance(dist, IntDistribution) and dist.log: log2_domain_size = int(np.ceil(np.log(dist.high - dist.low + 1) / np.log(2))) + 1 n_steps = min(log2_domain_size, n_steps) elif dist.step is not None: assert not dist.log, "log must be False when step is not None." n_steps = min(round((dist.high - dist.low) / dist.step) + 1, n_steps) scaler = np.log if dist.log else np.asarray grids = np.linspace(scaler(dist.low), scaler(dist.high), n_steps) # type: ignore[operator] params = scaler([t.params[param_name] for t in trials]) # type: ignore[operator] step_size = grids[1] - grids[0] # grids[indices[n] - 1] < param - step_size / 2 <= grids[indices[n]] indices = np.searchsorted(grids, params - step_size / 2) return grids.size, indices def _count_numerical_param_in_grid( param_name: str, dist: IntDistribution | FloatDistribution, trials: list[FrozenTrial], n_steps: int, ) -> np.ndarray: n_grids, grid_indices_of_trials = _get_grids_and_grid_indices_of_trials( param_name, dist, trials, n_steps ) unique_vals, counts_in_unique = np.unique(grid_indices_of_trials, return_counts=True) counts = np.zeros(n_grids, dtype=np.int32) counts[unique_vals] += counts_in_unique return counts def _count_categorical_param_in_grid( param_name: str, dist: CategoricalDistribution, trials: list[FrozenTrial] ) -> np.ndarray: cat_indices = [int(dist.to_internal_repr(t.params[param_name])) for t in trials] unique_vals, counts_in_unique = np.unique(cat_indices, return_counts=True) counts = np.zeros(len(dist.choices), dtype=np.int32) counts[unique_vals] += counts_in_unique return counts def _build_parzen_estimator( param_name: str, dist: BaseDistribution, trials: list[FrozenTrial], n_steps: int, consider_prior: bool, prior_weight: float, ) -> _ScottParzenEstimator: rounded_dist: IntDistribution | CategoricalDistribution if isinstance(dist, (IntDistribution, FloatDistribution)): counts = _count_numerical_param_in_grid(param_name, dist, trials, n_steps) rounded_dist = IntDistribution(low=0, high=counts.size - 1) elif isinstance(dist, CategoricalDistribution): counts = _count_categorical_param_in_grid(param_name, dist, trials) rounded_dist = dist else: assert False, f"Got an unknown dist with the type {type(dist)}." # counts.astype(float) is necessary for weight calculation in ParzenEstimator. return _ScottParzenEstimator( param_name, rounded_dist, counts.astype(np.float64), consider_prior, prior_weight ) optuna-4.1.0/optuna/integration/000077500000000000000000000000001471332314300166545ustar00rootroot00000000000000optuna-4.1.0/optuna/integration/__init__.py000066400000000000000000000124521471332314300207710ustar00rootroot00000000000000import os import sys from types import ModuleType from typing import Any from typing import TYPE_CHECKING from optuna._imports import _INTEGRATION_IMPORT_ERROR_TEMPLATE _import_structure = { "allennlp": ["AllenNLPExecutor", "AllenNLPPruningCallback"], "botorch": ["BoTorchSampler"], "catboost": ["CatBoostPruningCallback"], "chainer": ["ChainerPruningExtension"], "chainermn": ["ChainerMNStudy"], "cma": ["PyCmaSampler"], "dask": ["DaskStorage"], "mlflow": ["MLflowCallback"], "wandb": ["WeightsAndBiasesCallback"], "keras": ["KerasPruningCallback"], "lightgbm": ["LightGBMPruningCallback", "LightGBMTuner", "LightGBMTunerCV"], "pytorch_distributed": ["TorchDistributedTrial"], "pytorch_ignite": ["PyTorchIgnitePruningHandler"], "pytorch_lightning": ["PyTorchLightningPruningCallback"], "sklearn": ["OptunaSearchCV"], "shap": ["ShapleyImportanceEvaluator"], "skorch": ["SkorchPruningCallback"], "mxnet": ["MXNetPruningCallback"], "tensorboard": ["TensorBoardCallback"], "tensorflow": ["TensorFlowPruningHook"], "tfkeras": ["TFKerasPruningCallback"], "xgboost": ["XGBoostPruningCallback"], "fastaiv2": ["FastAIV2PruningCallback", "FastAIPruningCallback"], } __all__ = [ "AllenNLPExecutor", "AllenNLPPruningCallback", "BoTorchSampler", "CatBoostPruningCallback", "ChainerPruningExtension", "ChainerMNStudy", "PyCmaSampler", "DaskStorage", "MLflowCallback", "WeightsAndBiasesCallback", "KerasPruningCallback", "LightGBMPruningCallback", "LightGBMTuner", "LightGBMTunerCV", "TorchDistributedTrial", "PyTorchIgnitePruningHandler", "PyTorchLightningPruningCallback", "OptunaSearchCV", "ShapleyImportanceEvaluator", "SkorchPruningCallback", "MXNetPruningCallback", "TensorBoardCallback", "TensorFlowPruningHook", "TFKerasPruningCallback", "XGBoostPruningCallback", "FastAIV2PruningCallback", "FastAIPruningCallback", ] if TYPE_CHECKING: from optuna.integration.allennlp import AllenNLPExecutor from optuna.integration.allennlp import AllenNLPPruningCallback from optuna.integration.botorch import BoTorchSampler from optuna.integration.catboost import CatBoostPruningCallback from optuna.integration.chainer import ChainerPruningExtension from optuna.integration.chainermn import ChainerMNStudy from optuna.integration.cma import PyCmaSampler from optuna.integration.dask import DaskStorage from optuna.integration.fastaiv2 import FastAIPruningCallback from optuna.integration.fastaiv2 import FastAIV2PruningCallback from optuna.integration.keras import KerasPruningCallback from optuna.integration.lightgbm import LightGBMPruningCallback from optuna.integration.lightgbm import LightGBMTuner from optuna.integration.lightgbm import LightGBMTunerCV from optuna.integration.mlflow import MLflowCallback from optuna.integration.mxnet import MXNetPruningCallback from optuna.integration.pytorch_distributed import TorchDistributedTrial from optuna.integration.pytorch_ignite import PyTorchIgnitePruningHandler from optuna.integration.pytorch_lightning import PyTorchLightningPruningCallback from optuna.integration.shap import ShapleyImportanceEvaluator from optuna.integration.sklearn import OptunaSearchCV from optuna.integration.skorch import SkorchPruningCallback from optuna.integration.tensorboard import TensorBoardCallback from optuna.integration.tensorflow import TensorFlowPruningHook from optuna.integration.tfkeras import TFKerasPruningCallback from optuna.integration.wandb import WeightsAndBiasesCallback from optuna.integration.xgboost import XGBoostPruningCallback else: class _IntegrationModule(ModuleType): """Module class that implements `optuna.integration` package. This class applies lazy import under `optuna.integration`, where submodules are imported when they are actually accessed. Otherwise, `import optuna` becomes much slower because it imports all submodules and their dependencies (e.g., chainer, keras, lightgbm) all at once. """ __all__ = __all__ __file__ = globals()["__file__"] __path__ = [os.path.dirname(__file__)] _modules = set(_import_structure.keys()) _class_to_module = {} for key, values in _import_structure.items(): for value in values: _class_to_module[value] = key def __getattr__(self, name: str) -> Any: if name in self._modules: value = self._get_module(name) elif name in self._class_to_module.keys(): module = self._get_module(self._class_to_module[name]) value = getattr(module, name) else: raise AttributeError("module {} has no attribute {}".format(self.__name__, name)) setattr(self, name, value) return value def _get_module(self, module_name: str) -> ModuleType: import importlib try: return importlib.import_module("." + module_name, self.__name__) except ModuleNotFoundError: raise ModuleNotFoundError(_INTEGRATION_IMPORT_ERROR_TEMPLATE.format(module_name)) sys.modules[__name__] = _IntegrationModule(__name__) optuna-4.1.0/optuna/integration/allennlp/000077500000000000000000000000001471332314300204615ustar00rootroot00000000000000optuna-4.1.0/optuna/integration/allennlp/__init__.py000066400000000000000000000007511471332314300225750ustar00rootroot00000000000000from optuna._imports import _INTEGRATION_IMPORT_ERROR_TEMPLATE try: from optuna_integration.allennlp._dump_best_config import dump_best_config from optuna_integration.allennlp._executor import AllenNLPExecutor from optuna_integration.allennlp._pruner import AllenNLPPruningCallback except ModuleNotFoundError: raise ModuleNotFoundError(_INTEGRATION_IMPORT_ERROR_TEMPLATE.format("allennlp")) __all__ = ["dump_best_config", "AllenNLPExecutor", "AllenNLPPruningCallback"] optuna-4.1.0/optuna/integration/botorch.py000066400000000000000000000004071471332314300206670ustar00rootroot00000000000000from optuna._imports import _INTEGRATION_IMPORT_ERROR_TEMPLATE try: from optuna_integration import BoTorchSampler except ModuleNotFoundError: raise ModuleNotFoundError(_INTEGRATION_IMPORT_ERROR_TEMPLATE.format("botorch")) __all__ = ["BoTorchSampler"] optuna-4.1.0/optuna/integration/catboost.py000066400000000000000000000004431471332314300210450ustar00rootroot00000000000000from optuna._imports import _INTEGRATION_IMPORT_ERROR_TEMPLATE try: from optuna_integration.catboost import CatBoostPruningCallback except ModuleNotFoundError: raise ModuleNotFoundError(_INTEGRATION_IMPORT_ERROR_TEMPLATE.format("catboost")) __all__ = ["CatBoostPruningCallback"] optuna-4.1.0/optuna/integration/chainer.py000066400000000000000000000004411471332314300206360ustar00rootroot00000000000000from optuna._imports import _INTEGRATION_IMPORT_ERROR_TEMPLATE try: from optuna_integration.chainer import ChainerPruningExtension except ModuleNotFoundError: raise ModuleNotFoundError(_INTEGRATION_IMPORT_ERROR_TEMPLATE.format("chainer")) __all__ = ["ChainerPruningExtension"] optuna-4.1.0/optuna/integration/chainermn.py000066400000000000000000000004231471332314300211710ustar00rootroot00000000000000from optuna._imports import _INTEGRATION_IMPORT_ERROR_TEMPLATE try: from optuna_integration.chainermn import ChainerMNStudy except ModuleNotFoundError: raise ModuleNotFoundError(_INTEGRATION_IMPORT_ERROR_TEMPLATE.format("chainermn")) __all__ = ["ChainerMNStudy"] optuna-4.1.0/optuna/integration/cma.py000066400000000000000000000004031471332314300177630ustar00rootroot00000000000000from optuna._imports import _INTEGRATION_IMPORT_ERROR_TEMPLATE try: from optuna_integration.cma import PyCmaSampler except ModuleNotFoundError: raise ModuleNotFoundError(_INTEGRATION_IMPORT_ERROR_TEMPLATE.format("cma")) __all__ = ["PyCmaSampler"] optuna-4.1.0/optuna/integration/dask.py000066400000000000000000000004031471332314300201450ustar00rootroot00000000000000from optuna._imports import _INTEGRATION_IMPORT_ERROR_TEMPLATE try: from optuna_integration.dask import DaskStorage except ModuleNotFoundError: raise ModuleNotFoundError(_INTEGRATION_IMPORT_ERROR_TEMPLATE.format("dask")) __all__ = ["DaskStorage"] optuna-4.1.0/optuna/integration/fastaiv2.py000066400000000000000000000005761471332314300207550ustar00rootroot00000000000000from optuna._imports import _INTEGRATION_IMPORT_ERROR_TEMPLATE try: from optuna_integration.fastaiv2 import FastAIPruningCallback from optuna_integration.fastaiv2 import FastAIV2PruningCallback except ModuleNotFoundError: raise ModuleNotFoundError(_INTEGRATION_IMPORT_ERROR_TEMPLATE.format("fastaiv2")) __all__ = ["FastAIV2PruningCallback", "FastAIPruningCallback"] optuna-4.1.0/optuna/integration/keras.py000066400000000000000000000004271471332314300203360ustar00rootroot00000000000000from optuna._imports import _INTEGRATION_IMPORT_ERROR_TEMPLATE try: from optuna_integration.keras import KerasPruningCallback except ModuleNotFoundError: raise ModuleNotFoundError(_INTEGRATION_IMPORT_ERROR_TEMPLATE.format("keras")) __all__ = ["KerasPruningCallback"] optuna-4.1.0/optuna/integration/lightgbm.py000066400000000000000000000021371471332314300210260ustar00rootroot00000000000000import os import sys from types import ModuleType from typing import Any from typing import TYPE_CHECKING from optuna._imports import _INTEGRATION_IMPORT_ERROR_TEMPLATE try: import optuna_integration.lightgbm as lgb except ModuleNotFoundError: raise ModuleNotFoundError(_INTEGRATION_IMPORT_ERROR_TEMPLATE.format("lightgbm")) if TYPE_CHECKING: # These modules are from optuna-integration. from optuna.integration.lightgbm_tuner import LightGBMPruningCallback from optuna.integration.lightgbm_tuner import LightGBMTuner from optuna.integration.lightgbm_tuner import LightGBMTunerCV from optuna.integration.lightgbm_tuner import train __all__ = [ "LightGBMPruningCallback", "LightGBMTuner", "LightGBMTunerCV", "train", ] class _LightGBMModule(ModuleType): """Module class that implements `optuna.integration.lightgbm` package.""" __all__ = __all__ __file__ = globals()["__file__"] __path__ = [os.path.dirname(__file__)] def __getattr__(self, name: str) -> Any: return lgb.__dict__[name] sys.modules[__name__] = _LightGBMModule(__name__) optuna-4.1.0/optuna/integration/mlflow.py000066400000000000000000000004151471332314300205260ustar00rootroot00000000000000from optuna._imports import _INTEGRATION_IMPORT_ERROR_TEMPLATE try: from optuna_integration.mlflow import MLflowCallback except ModuleNotFoundError: raise ModuleNotFoundError(_INTEGRATION_IMPORT_ERROR_TEMPLATE.format("mlflow")) __all__ = ["MLflowCallback"] optuna-4.1.0/optuna/integration/mxnet.py000066400000000000000000000004271471332314300203640ustar00rootroot00000000000000from optuna._imports import _INTEGRATION_IMPORT_ERROR_TEMPLATE try: from optuna_integration.mxnet import MXNetPruningCallback except ModuleNotFoundError: raise ModuleNotFoundError(_INTEGRATION_IMPORT_ERROR_TEMPLATE.format("mxnet")) __all__ = ["MXNetPruningCallback"] optuna-4.1.0/optuna/integration/pytorch_distributed.py000066400000000000000000000004651471332314300233250ustar00rootroot00000000000000from optuna._imports import _INTEGRATION_IMPORT_ERROR_TEMPLATE try: from optuna_integration.pytorch_distributed import TorchDistributedTrial except ModuleNotFoundError: raise ModuleNotFoundError(_INTEGRATION_IMPORT_ERROR_TEMPLATE.format("pytorch_distributed")) __all__ = ["TorchDistributedTrial"] optuna-4.1.0/optuna/integration/pytorch_ignite.py000066400000000000000000000004671471332314300222640ustar00rootroot00000000000000from optuna._imports import _INTEGRATION_IMPORT_ERROR_TEMPLATE try: from optuna_integration.pytorch_ignite import PyTorchIgnitePruningHandler except ModuleNotFoundError: raise ModuleNotFoundError(_INTEGRATION_IMPORT_ERROR_TEMPLATE.format("pytorch_ignite")) __all__ = ["PyTorchIgnitePruningHandler"] optuna-4.1.0/optuna/integration/pytorch_lightning.py000066400000000000000000000005051471332314300227610ustar00rootroot00000000000000from optuna._imports import _INTEGRATION_IMPORT_ERROR_TEMPLATE try: from optuna_integration.pytorch_lightning import PyTorchLightningPruningCallback except ModuleNotFoundError: raise ModuleNotFoundError(_INTEGRATION_IMPORT_ERROR_TEMPLATE.format("pytorch_lightning")) __all__ = ["PyTorchLightningPruningCallback"] optuna-4.1.0/optuna/integration/shap.py000066400000000000000000000004411471332314300201600ustar00rootroot00000000000000from optuna._imports import _INTEGRATION_IMPORT_ERROR_TEMPLATE try: from optuna_integration.shap import ShapleyImportanceEvaluator except ModuleNotFoundError: raise ModuleNotFoundError(_INTEGRATION_IMPORT_ERROR_TEMPLATE.format("shap")) __all__ = ["ShapleyImportanceEvaluator"] optuna-4.1.0/optuna/integration/sklearn.py000066400000000000000000000004171471332314300206670ustar00rootroot00000000000000from optuna._imports import _INTEGRATION_IMPORT_ERROR_TEMPLATE try: from optuna_integration.sklearn import OptunaSearchCV except ModuleNotFoundError: raise ModuleNotFoundError(_INTEGRATION_IMPORT_ERROR_TEMPLATE.format("sklearn")) __all__ = ["OptunaSearchCV"] optuna-4.1.0/optuna/integration/skorch.py000066400000000000000000000004331471332314300205170ustar00rootroot00000000000000from optuna._imports import _INTEGRATION_IMPORT_ERROR_TEMPLATE try: from optuna_integration.skorch import SkorchPruningCallback except ModuleNotFoundError: raise ModuleNotFoundError(_INTEGRATION_IMPORT_ERROR_TEMPLATE.format("skorch")) __all__ = ["SkorchPruningCallback"] optuna-4.1.0/optuna/integration/tensorboard.py000066400000000000000000000004411471332314300215470ustar00rootroot00000000000000from optuna._imports import _INTEGRATION_IMPORT_ERROR_TEMPLATE try: from optuna_integration.tensorboard import TensorBoardCallback except ModuleNotFoundError: raise ModuleNotFoundError(_INTEGRATION_IMPORT_ERROR_TEMPLATE.format("tensorboard")) __all__ = ["TensorBoardCallback"] optuna-4.1.0/optuna/integration/tensorflow.py000066400000000000000000000004431471332314300214310ustar00rootroot00000000000000from optuna._imports import _INTEGRATION_IMPORT_ERROR_TEMPLATE try: from optuna_integration.tensorflow import TensorFlowPruningHook except ModuleNotFoundError: raise ModuleNotFoundError(_INTEGRATION_IMPORT_ERROR_TEMPLATE.format("tensorflow")) __all__ = ["TensorFlowPruningHook"] optuna-4.1.0/optuna/integration/tfkeras.py000066400000000000000000000004371471332314300206710ustar00rootroot00000000000000from optuna._imports import _INTEGRATION_IMPORT_ERROR_TEMPLATE try: from optuna_integration.tfkeras import TFKerasPruningCallback except ModuleNotFoundError: raise ModuleNotFoundError(_INTEGRATION_IMPORT_ERROR_TEMPLATE.format("tfkeras")) __all__ = ["TFKerasPruningCallback"] optuna-4.1.0/optuna/integration/wandb.py000066400000000000000000000004371471332314300203250ustar00rootroot00000000000000from optuna._imports import _INTEGRATION_IMPORT_ERROR_TEMPLATE try: from optuna_integration.wandb import WeightsAndBiasesCallback except ModuleNotFoundError: raise ModuleNotFoundError(_INTEGRATION_IMPORT_ERROR_TEMPLATE.format("wandb")) __all__ = ["WeightsAndBiasesCallback"] optuna-4.1.0/optuna/integration/xgboost.py000066400000000000000000000004371471332314300207170ustar00rootroot00000000000000from optuna._imports import _INTEGRATION_IMPORT_ERROR_TEMPLATE try: from optuna_integration.xgboost import XGBoostPruningCallback except ModuleNotFoundError: raise ModuleNotFoundError(_INTEGRATION_IMPORT_ERROR_TEMPLATE.format("xgboost")) __all__ = ["XGBoostPruningCallback"] optuna-4.1.0/optuna/logging.py000066400000000000000000000243311471332314300163340ustar00rootroot00000000000000from __future__ import annotations import logging from logging import CRITICAL from logging import DEBUG from logging import ERROR from logging import FATAL from logging import INFO from logging import WARN from logging import WARNING import os import sys import threading import colorlog __all__ = [ "CRITICAL", "DEBUG", "ERROR", "FATAL", "INFO", "WARN", "WARNING", ] _lock: threading.Lock = threading.Lock() _default_handler: logging.Handler | None = None def create_default_formatter() -> logging.Formatter: """Create a default formatter of log messages. This function is not supposed to be directly accessed by library users. """ header = "[%(levelname)1.1s %(asctime)s]" message = "%(message)s" if _color_supported(): return colorlog.ColoredFormatter( f"%(log_color)s{header}%(reset)s {message}", ) return logging.Formatter(f"{header} {message}") def _color_supported() -> bool: """Detection of color support.""" # NO_COLOR environment variable: if os.environ.get("NO_COLOR", None): return False if not hasattr(sys.stderr, "isatty") or not sys.stderr.isatty(): return False else: return True def _get_library_name() -> str: return __name__.split(".")[0] def _get_library_root_logger() -> logging.Logger: return logging.getLogger(_get_library_name()) def _configure_library_root_logger() -> None: global _default_handler with _lock: if _default_handler: # This library has already configured the library root logger. return _default_handler = logging.StreamHandler() # Set sys.stderr as stream. _default_handler.setFormatter(create_default_formatter()) # Apply our default configuration to the library root logger. library_root_logger: logging.Logger = _get_library_root_logger() library_root_logger.addHandler(_default_handler) library_root_logger.setLevel(logging.INFO) library_root_logger.propagate = False def _reset_library_root_logger() -> None: global _default_handler with _lock: if not _default_handler: return library_root_logger: logging.Logger = _get_library_root_logger() library_root_logger.removeHandler(_default_handler) library_root_logger.setLevel(logging.NOTSET) _default_handler = None def get_logger(name: str) -> logging.Logger: """Return a logger with the specified name. This function is not supposed to be directly accessed by library users. """ _configure_library_root_logger() return logging.getLogger(name) def get_verbosity() -> int: """Return the current level for the Optuna's root logger. Example: Get the default verbosity level. .. testsetup:: def objective(trial): x = trial.suggest_float("x", -100, 100) y = trial.suggest_categorical("y", [-1, 0, 1]) return x**2 + y .. testcode:: import optuna # The default verbosity level of Optuna is `optuna.logging.INFO`. print(optuna.logging.get_verbosity()) # 20 print(optuna.logging.INFO) # 20 # There are logs of the INFO level. study = optuna.create_study() study.optimize(objective, n_trials=5) # [I 2021-10-31 05:35:17,232] A new study created ... # [I 2021-10-31 05:35:17,238] Trial 0 finished with value: ... # [I 2021-10-31 05:35:17,245] Trial 1 finished with value: ... # ... .. testoutput:: :hide: 20 20 Returns: Logging level, e.g., ``optuna.logging.DEBUG`` and ``optuna.logging.INFO``. .. note:: Optuna has following logging levels: - ``optuna.logging.CRITICAL``, ``optuna.logging.FATAL`` - ``optuna.logging.ERROR`` - ``optuna.logging.WARNING``, ``optuna.logging.WARN`` - ``optuna.logging.INFO`` - ``optuna.logging.DEBUG`` """ _configure_library_root_logger() return _get_library_root_logger().getEffectiveLevel() def set_verbosity(verbosity: int) -> None: """Set the level for the Optuna's root logger. Example: Set the logging level ``optuna.logging.WARNING``. .. testsetup:: def objective(trial): x = trial.suggest_int("x", -10, 10) return x**2 .. testcode:: import optuna # There are INFO level logs. study = optuna.create_study() study.optimize(objective, n_trials=10) # [I 2021-10-31 02:59:35,088] Trial 0 finished with value: 16.0 ... # [I 2021-10-31 02:59:35,091] Trial 1 finished with value: 1.0 ... # [I 2021-10-31 02:59:35,096] Trial 2 finished with value: 1.0 ... # Setting the logging level WARNING, the INFO logs are suppressed. optuna.logging.set_verbosity(optuna.logging.WARNING) study.optimize(objective, n_trials=10) .. testcleanup:: optuna.logging.set_verbosity(optuna.logging.INFO) Args: verbosity: Logging level, e.g., ``optuna.logging.DEBUG`` and ``optuna.logging.INFO``. .. note:: Optuna has following logging levels: - ``optuna.logging.CRITICAL``, ``optuna.logging.FATAL`` - ``optuna.logging.ERROR`` - ``optuna.logging.WARNING``, ``optuna.logging.WARN`` - ``optuna.logging.INFO`` - ``optuna.logging.DEBUG`` """ _configure_library_root_logger() _get_library_root_logger().setLevel(verbosity) def disable_default_handler() -> None: """Disable the default handler of the Optuna's root logger. Example: Stop and then resume logging to :obj:`sys.stderr`. .. testsetup:: def objective(trial): x = trial.suggest_float("x", -100, 100) y = trial.suggest_categorical("y", [-1, 0, 1]) return x**2 + y .. testcode:: import optuna study = optuna.create_study() # There are no logs in sys.stderr. optuna.logging.disable_default_handler() study.optimize(objective, n_trials=10) # There are logs in sys.stderr. optuna.logging.enable_default_handler() study.optimize(objective, n_trials=10) # [I 2020-02-23 17:00:54,314] Trial 10 finished with value: ... # [I 2020-02-23 17:00:54,356] Trial 11 finished with value: ... # ... """ _configure_library_root_logger() assert _default_handler is not None _get_library_root_logger().removeHandler(_default_handler) def enable_default_handler() -> None: """Enable the default handler of the Optuna's root logger. Please refer to the example shown in :func:`~optuna.logging.disable_default_handler()`. """ _configure_library_root_logger() assert _default_handler is not None _get_library_root_logger().addHandler(_default_handler) def disable_propagation() -> None: """Disable propagation of the library log outputs. Note that log propagation is disabled by default. You only need to use this function to stop log propagation when you use :func:`~optuna.logging.enable_propagation()`. Example: Stop propagating logs to the root logger on the second optimize call. .. testsetup:: def objective(trial): x = trial.suggest_float("x", -100, 100) y = trial.suggest_categorical("y", [-1, 0, 1]) return x**2 + y .. testcode:: import optuna import logging optuna.logging.disable_default_handler() # Disable the default handler. logger = logging.getLogger() logger.setLevel(logging.INFO) # Setup the root logger. logger.addHandler(logging.FileHandler("foo.log", mode="w")) optuna.logging.enable_propagation() # Propagate logs to the root logger. study = optuna.create_study() logger.info("Logs from first optimize call") # The logs are saved in the logs file. study.optimize(objective, n_trials=10) optuna.logging.disable_propagation() # Stop propogating logs to the root logger. logger.info("Logs from second optimize call") # The new logs for second optimize call are not saved. study.optimize(objective, n_trials=10) with open("foo.log") as f: assert f.readline().startswith("A new study created") assert f.readline() == "Logs from first optimize call\\n" # Check for logs after second optimize call. assert f.read().split("Logs from second optimize call\\n")[-1] == "" """ _configure_library_root_logger() _get_library_root_logger().propagate = False def enable_propagation() -> None: """Enable propagation of the library log outputs. Please disable the Optuna's default handler to prevent double logging if the root logger has been configured. Example: Propagate all log output to the root logger in order to save them to the file. .. testsetup:: def objective(trial): x = trial.suggest_float("x", -100, 100) y = trial.suggest_categorical("y", [-1, 0, 1]) return x**2 + y .. testcode:: import optuna import logging logger = logging.getLogger() logger.setLevel(logging.INFO) # Setup the root logger. logger.addHandler(logging.FileHandler("foo.log", mode="w")) optuna.logging.enable_propagation() # Propagate logs to the root logger. optuna.logging.disable_default_handler() # Stop showing logs in sys.stderr. study = optuna.create_study() logger.info("Start optimization.") study.optimize(objective, n_trials=10) with open("foo.log") as f: assert f.readline().startswith("A new study created") assert f.readline() == "Start optimization.\\n" """ _configure_library_root_logger() _get_library_root_logger().propagate = True optuna-4.1.0/optuna/multi_objective/000077500000000000000000000000001471332314300175155ustar00rootroot00000000000000optuna-4.1.0/optuna/multi_objective/__init__.py000066400000000000000000000010071471332314300216240ustar00rootroot00000000000000# TODO(nabenabe0928): Come up with any ways to remove this file. # NOTE(nabenabe0928): Discuss when to remove this class. migration_url = "https://github.com/optuna/optuna/discussions/5573" raise ModuleNotFoundError( "\nThe features in `optuna.multi_objective` were integrated with the" "\nsingle objective optimization API and `optuna.multi_objective` were" "\ndeleted at v4.0.0. Please update your code based on the migration guide" f"\nat {migration_url}" "\nor downgrade your Optuna version." ) optuna-4.1.0/optuna/progress_bar.py000066400000000000000000000102141471332314300173710ustar00rootroot00000000000000from __future__ import annotations import logging from typing import Any from typing import TYPE_CHECKING import warnings from tqdm.auto import tqdm from optuna import logging as optuna_logging if TYPE_CHECKING: from optuna.study import Study _tqdm_handler: _TqdmLoggingHandler | None = None # Reference: https://gist.github.com/hvy/8b80c2cedf02b15c24f85d1fa17ebe02 class _TqdmLoggingHandler(logging.StreamHandler): def emit(self, record: Any) -> None: try: msg = self.format(record) tqdm.write(msg) self.flush() except (KeyboardInterrupt, SystemExit): raise except Exception: self.handleError(record) class _ProgressBar: """Progress Bar implementation for :func:`~optuna.study.Study.optimize` on the top of `tqdm`. Args: is_valid: Whether to show progress bars in :func:`~optuna.study.Study.optimize`. n_trials: The number of trials. timeout: Stop study after the given number of second(s). """ def __init__( self, is_valid: bool, n_trials: int | None = None, timeout: float | None = None, ) -> None: if is_valid and n_trials is None and timeout is None: warnings.warn("Progress bar won't be displayed because n_trials and timeout are None.") self._is_valid = is_valid and (n_trials or timeout) is not None self._n_trials = n_trials self._timeout = timeout self._last_elapsed_seconds = 0.0 if self._is_valid: if self._n_trials is not None: self._progress_bar = tqdm(total=self._n_trials) elif self._timeout is not None: total = tqdm.format_interval(self._timeout) fmt = "{desc} {percentage:3.0f}%|{bar}| {elapsed}/" + total self._progress_bar = tqdm(total=self._timeout, bar_format=fmt) else: assert False global _tqdm_handler _tqdm_handler = _TqdmLoggingHandler() _tqdm_handler.setLevel(logging.INFO) _tqdm_handler.setFormatter(optuna_logging.create_default_formatter()) optuna_logging.disable_default_handler() optuna_logging._get_library_root_logger().addHandler(_tqdm_handler) def update(self, elapsed_seconds: float, study: Study) -> None: """Update the progress bars if ``is_valid`` is :obj:`True`. Args: elapsed_seconds: The time past since :func:`~optuna.study.Study.optimize` started. study: The current study object. """ if self._is_valid: if not study._is_multi_objective(): # Not updating the progress bar when there are no complete trial. try: msg = ( f"Best trial: {study.best_trial.number}. " f"Best value: {study.best_value:.6g}" ) self._progress_bar.set_description(msg) except ValueError: pass if self._n_trials is not None: self._progress_bar.update(1) if self._timeout is not None: self._progress_bar.set_postfix_str( "{:.02f}/{} seconds".format(elapsed_seconds, self._timeout) ) elif self._timeout is not None: time_diff = elapsed_seconds - self._last_elapsed_seconds if elapsed_seconds > self._timeout: # Clip elapsed time to avoid tqdm warnings. time_diff -= elapsed_seconds - self._timeout self._progress_bar.update(time_diff) self._last_elapsed_seconds = elapsed_seconds else: assert False def close(self) -> None: """Close progress bars.""" if self._is_valid: self._progress_bar.close() assert _tqdm_handler is not None optuna_logging._get_library_root_logger().removeHandler(_tqdm_handler) optuna_logging.enable_default_handler() optuna-4.1.0/optuna/pruners/000077500000000000000000000000001471332314300160275ustar00rootroot00000000000000optuna-4.1.0/optuna/pruners/__init__.py000066400000000000000000000022351471332314300201420ustar00rootroot00000000000000from typing import TYPE_CHECKING from optuna.pruners._base import BasePruner from optuna.pruners._hyperband import HyperbandPruner from optuna.pruners._median import MedianPruner from optuna.pruners._nop import NopPruner from optuna.pruners._patient import PatientPruner from optuna.pruners._percentile import PercentilePruner from optuna.pruners._successive_halving import SuccessiveHalvingPruner from optuna.pruners._threshold import ThresholdPruner from optuna.pruners._wilcoxon import WilcoxonPruner if TYPE_CHECKING: from optuna.study import Study from optuna.trial import FrozenTrial __all__ = [ "BasePruner", "HyperbandPruner", "MedianPruner", "NopPruner", "PatientPruner", "PercentilePruner", "SuccessiveHalvingPruner", "ThresholdPruner", "WilcoxonPruner", ] def _filter_study(study: "Study", trial: "FrozenTrial") -> "Study": if isinstance(study.pruner, HyperbandPruner): # Create `_BracketStudy` to use trials that have the same bracket id. pruner: HyperbandPruner = study.pruner return pruner._create_bracket_study(study, pruner._get_bracket_id(study, trial)) else: return study optuna-4.1.0/optuna/pruners/_base.py000066400000000000000000000016161471332314300174560ustar00rootroot00000000000000import abc import optuna class BasePruner(abc.ABC): """Base class for pruners.""" @abc.abstractmethod def prune(self, study: "optuna.study.Study", trial: "optuna.trial.FrozenTrial") -> bool: """Judge whether the trial should be pruned based on the reported values. Note that this method is not supposed to be called by library users. Instead, :func:`optuna.trial.Trial.report` and :func:`optuna.trial.Trial.should_prune` provide user interfaces to implement pruning mechanism in an objective function. Args: study: Study object of the target study. trial: FrozenTrial object of the target trial. Take a copy before modifying this object. Returns: A boolean value representing whether the trial should be pruned. """ raise NotImplementedError optuna-4.1.0/optuna/pruners/_hyperband.py000066400000000000000000000332161471332314300205210ustar00rootroot00000000000000from __future__ import annotations import binascii from collections.abc import Container import math import optuna from optuna import logging from optuna.pruners._base import BasePruner from optuna.pruners._successive_halving import SuccessiveHalvingPruner from optuna.trial._state import TrialState _logger = logging.get_logger(__name__) class HyperbandPruner(BasePruner): """Pruner using Hyperband. As SuccessiveHalving (SHA) requires the number of configurations :math:`n` as its hyperparameter. For a given finite budget :math:`B`, all the configurations have the resources of :math:`B \\over n` on average. As you can see, there will be a trade-off of :math:`B` and :math:`B \\over n`. `Hyperband `__ attacks this trade-off by trying different :math:`n` values for a fixed budget. .. note:: * In the Hyperband paper, the counterpart of :class:`~optuna.samplers.RandomSampler` is used. * Optuna uses :class:`~optuna.samplers.TPESampler` by default. * `The benchmark result `__ shows that :class:`optuna.pruners.HyperbandPruner` supports both samplers. .. note:: If you use ``HyperbandPruner`` with :class:`~optuna.samplers.TPESampler`, it's recommended to consider setting larger ``n_trials`` or ``timeout`` to make full use of the characteristics of :class:`~optuna.samplers.TPESampler` because :class:`~optuna.samplers.TPESampler` uses some (by default, :math:`10`) :class:`~optuna.trial.Trial`\\ s for its startup. As Hyperband runs multiple :class:`~optuna.pruners.SuccessiveHalvingPruner` and collects trials based on the current :class:`~optuna.trial.Trial`\\ 's bracket ID, each bracket needs to observe more than :math:`10` :class:`~optuna.trial.Trial`\\ s for :class:`~optuna.samplers.TPESampler` to adapt its search space. Thus, for example, if ``HyperbandPruner`` has :math:`4` pruners in it, at least :math:`4 \\times 10` trials are consumed for startup. .. note:: Hyperband has several :class:`~optuna.pruners.SuccessiveHalvingPruner`\\ s. Each :class:`~optuna.pruners.SuccessiveHalvingPruner` is referred to as "bracket" in the original paper. The number of brackets is an important factor to control the early stopping behavior of Hyperband and is automatically determined by ``min_resource``, ``max_resource`` and ``reduction_factor`` as :math:`\\mathrm{The\\ number\\ of\\ brackets} = \\mathrm{floor}(\\log_{\\texttt{reduction}\\_\\texttt{factor}} (\\frac{\\texttt{max}\\_\\texttt{resource}}{\\texttt{min}\\_\\texttt{resource}})) + 1`. Please set ``reduction_factor`` so that the number of brackets is not too large (about 4 – 6 in most use cases). Please see Section 3.6 of the `original paper `__ for the detail. Example: We minimize an objective function with Hyperband pruning algorithm. .. testcode:: import numpy as np from sklearn.datasets import load_iris from sklearn.linear_model import SGDClassifier from sklearn.model_selection import train_test_split import optuna X, y = load_iris(return_X_y=True) X_train, X_valid, y_train, y_valid = train_test_split(X, y) classes = np.unique(y) n_train_iter = 100 def objective(trial): alpha = trial.suggest_float("alpha", 0.0, 1.0) clf = SGDClassifier(alpha=alpha) for step in range(n_train_iter): clf.partial_fit(X_train, y_train, classes=classes) intermediate_value = clf.score(X_valid, y_valid) trial.report(intermediate_value, step) if trial.should_prune(): raise optuna.TrialPruned() return clf.score(X_valid, y_valid) study = optuna.create_study( direction="maximize", pruner=optuna.pruners.HyperbandPruner( min_resource=1, max_resource=n_train_iter, reduction_factor=3 ), ) study.optimize(objective, n_trials=20) Args: min_resource: A parameter for specifying the minimum resource allocated to a trial noted as :math:`r` in the paper. A smaller :math:`r` will give a result faster, but a larger :math:`r` will give a better guarantee of successful judging between configurations. See the details for :class:`~optuna.pruners.SuccessiveHalvingPruner`. max_resource: A parameter for specifying the maximum resource allocated to a trial. :math:`R` in the paper corresponds to ``max_resource / min_resource``. This value represents and should match the maximum iteration steps (e.g., the number of epochs for neural networks). When this argument is "auto", the maximum resource is estimated according to the completed trials. The default value of this argument is "auto". .. note:: With "auto", the maximum resource will be the largest step reported by :meth:`~optuna.trial.Trial.report` in the first, or one of the first if trained in parallel, completed trial. No trials will be pruned until the maximum resource is determined. .. note:: If the step of the last intermediate value may change with each trial, please manually specify the maximum possible step to ``max_resource``. reduction_factor: A parameter for specifying reduction factor of promotable trials noted as :math:`\\eta` in the paper. See the details for :class:`~optuna.pruners.SuccessiveHalvingPruner`. bootstrap_count: Parameter specifying the number of trials required in a rung before any trial can be promoted. Incompatible with ``max_resource`` is ``"auto"``. See the details for :class:`~optuna.pruners.SuccessiveHalvingPruner`. """ def __init__( self, min_resource: int = 1, max_resource: str | int = "auto", reduction_factor: int = 3, bootstrap_count: int = 0, ) -> None: self._min_resource = min_resource self._max_resource = max_resource self._reduction_factor = reduction_factor self._pruners: list[SuccessiveHalvingPruner] = [] self._bootstrap_count = bootstrap_count self._total_trial_allocation_budget = 0 self._trial_allocation_budgets: list[int] = [] self._n_brackets: int | None = None if not isinstance(self._max_resource, int) and self._max_resource != "auto": raise ValueError( "The 'max_resource' should be integer or 'auto'. " "But max_resource = {}".format(self._max_resource) ) if self._bootstrap_count > 0 and self._max_resource == "auto": raise ValueError( "bootstrap_count > 0 and max_resource == 'auto' " "are mutually incompatible, bootstrap_count is {}".format(self._bootstrap_count) ) def prune(self, study: "optuna.study.Study", trial: "optuna.trial.FrozenTrial") -> bool: if len(self._pruners) == 0: self._try_initialization(study) if len(self._pruners) == 0: return False bracket_id = self._get_bracket_id(study, trial) _logger.debug("{}th bracket is selected".format(bracket_id)) bracket_study = self._create_bracket_study(study, bracket_id) return self._pruners[bracket_id].prune(bracket_study, trial) def _try_initialization(self, study: "optuna.study.Study") -> None: if self._max_resource == "auto": trials = study.get_trials(deepcopy=False, states=(TrialState.COMPLETE,)) n_steps = [t.last_step for t in trials if t.last_step is not None] if not n_steps: return self._max_resource = max(n_steps) + 1 assert isinstance(self._max_resource, int) if self._n_brackets is None: # In the original paper http://www.jmlr.org/papers/volume18/16-558/16-558.pdf, the # inputs of Hyperband are `R`: max resource and `\eta`: reduction factor. The # number of brackets (this is referred as `s_{max} + 1` in the paper) is calculated # by s_{max} + 1 = \floor{\log_{\eta} (R)} + 1 in Algorithm 1 of the original paper. # In this implementation, we combine this formula and that of ASHA paper # https://arxiv.org/abs/1502.07943 as # `n_brackets = floor(log_{reduction_factor}(max_resource / min_resource)) + 1` self._n_brackets = ( math.floor( math.log(self._max_resource / self._min_resource, self._reduction_factor) ) + 1 ) _logger.debug("Hyperband has {} brackets".format(self._n_brackets)) for bracket_id in range(self._n_brackets): trial_allocation_budget = self._calculate_trial_allocation_budget(bracket_id) self._total_trial_allocation_budget += trial_allocation_budget self._trial_allocation_budgets.append(trial_allocation_budget) pruner = SuccessiveHalvingPruner( min_resource=self._min_resource, reduction_factor=self._reduction_factor, min_early_stopping_rate=bracket_id, bootstrap_count=self._bootstrap_count, ) self._pruners.append(pruner) def _calculate_trial_allocation_budget(self, bracket_id: int) -> int: """Compute the trial allocated budget for a bracket of ``bracket_id``. In the `original paper `, the number of trials per one bracket is referred as ``n`` in Algorithm 1. Since we do not know the total number of trials in the leaning scheme of Optuna, we calculate the ratio of the number of trials here instead. """ assert self._n_brackets is not None s = self._n_brackets - 1 - bracket_id return math.ceil(self._n_brackets * (self._reduction_factor**s) / (s + 1)) def _get_bracket_id( self, study: "optuna.study.Study", trial: "optuna.trial.FrozenTrial" ) -> int: """Compute the index of bracket for a trial of ``trial_number``. The index of a bracket is noted as :math:`s` in `Hyperband paper `__. """ if len(self._pruners) == 0: return 0 assert self._n_brackets is not None n = ( binascii.crc32("{}_{}".format(study.study_name, trial.number).encode()) % self._total_trial_allocation_budget ) for bracket_id in range(self._n_brackets): n -= self._trial_allocation_budgets[bracket_id] if n < 0: return bracket_id assert False, "This line should be unreachable." def _create_bracket_study( self, study: "optuna.study.Study", bracket_id: int ) -> "optuna.study.Study": # This class is assumed to be passed to # `SuccessiveHalvingPruner.prune` in which `get_trials`, # `direction`, and `storage` are used. # But for safety, prohibit the other attributes explicitly. class _BracketStudy(optuna.study.Study): _VALID_ATTRS = ( "get_trials", "_get_trials", "directions", "direction", "_directions", "_storage", "_study_id", "pruner", "study_name", "_bracket_id", "sampler", "trials", "_is_multi_objective", "stop", "_study", "_thread_local", ) def __init__( self, study: "optuna.study.Study", pruner: HyperbandPruner, bracket_id: int ) -> None: super().__init__( study_name=study.study_name, storage=study._storage, sampler=study.sampler, pruner=pruner, ) self._study = study self._bracket_id = bracket_id def get_trials( self, deepcopy: bool = True, states: Container[TrialState] | None = None, ) -> list["optuna.trial.FrozenTrial"]: trials = super()._get_trials(deepcopy=deepcopy, states=states) pruner = self.pruner assert isinstance(pruner, HyperbandPruner) return [t for t in trials if pruner._get_bracket_id(self, t) == self._bracket_id] def stop(self) -> None: # `stop` should stop the original study's optimization loop instead of # `_BracketStudy`. self._study.stop() def __getattribute__(self, attr_name): # type: ignore if attr_name not in _BracketStudy._VALID_ATTRS: raise AttributeError( "_BracketStudy does not have attribute of '{}'".format(attr_name) ) else: return object.__getattribute__(self, attr_name) return _BracketStudy(study, self, bracket_id) optuna-4.1.0/optuna/pruners/_median.py000066400000000000000000000056151471332314300200040ustar00rootroot00000000000000from optuna.pruners._percentile import PercentilePruner class MedianPruner(PercentilePruner): """Pruner using the median stopping rule. Prune if the trial's best intermediate result is worse than median of intermediate results of previous trials at the same step. Example: We minimize an objective function with the median stopping rule. .. testcode:: import numpy as np from sklearn.datasets import load_iris from sklearn.linear_model import SGDClassifier from sklearn.model_selection import train_test_split import optuna X, y = load_iris(return_X_y=True) X_train, X_valid, y_train, y_valid = train_test_split(X, y) classes = np.unique(y) def objective(trial): alpha = trial.suggest_float("alpha", 0.0, 1.0) clf = SGDClassifier(alpha=alpha) n_train_iter = 100 for step in range(n_train_iter): clf.partial_fit(X_train, y_train, classes=classes) intermediate_value = clf.score(X_valid, y_valid) trial.report(intermediate_value, step) if trial.should_prune(): raise optuna.TrialPruned() return clf.score(X_valid, y_valid) study = optuna.create_study( direction="maximize", pruner=optuna.pruners.MedianPruner( n_startup_trials=5, n_warmup_steps=30, interval_steps=10 ), ) study.optimize(objective, n_trials=20) Args: n_startup_trials: Pruning is disabled until the given number of trials finish in the same study. n_warmup_steps: Pruning is disabled until the trial exceeds the given number of step. Note that this feature assumes that ``step`` starts at zero. interval_steps: Interval in number of steps between the pruning checks, offset by the warmup steps. If no value has been reported at the time of a pruning check, that particular check will be postponed until a value is reported. n_min_trials: Minimum number of reported trial results at a step to judge whether to prune. If the number of reported intermediate values from all trials at the current step is less than ``n_min_trials``, the trial will not be pruned. This can be used to ensure that a minimum number of trials are run to completion without being pruned. """ def __init__( self, n_startup_trials: int = 5, n_warmup_steps: int = 0, interval_steps: int = 1, *, n_min_trials: int = 1, ) -> None: super().__init__( 50.0, n_startup_trials, n_warmup_steps, interval_steps, n_min_trials=n_min_trials ) optuna-4.1.0/optuna/pruners/_nop.py000066400000000000000000000027401471332314300173370ustar00rootroot00000000000000import optuna from optuna.pruners import BasePruner class NopPruner(BasePruner): """Pruner which never prunes trials. Example: .. testcode:: import numpy as np from sklearn.datasets import load_iris from sklearn.linear_model import SGDClassifier from sklearn.model_selection import train_test_split import optuna X, y = load_iris(return_X_y=True) X_train, X_valid, y_train, y_valid = train_test_split(X, y) classes = np.unique(y) def objective(trial): alpha = trial.suggest_float("alpha", 0.0, 1.0) clf = SGDClassifier(alpha=alpha) n_train_iter = 100 for step in range(n_train_iter): clf.partial_fit(X_train, y_train, classes=classes) intermediate_value = clf.score(X_valid, y_valid) trial.report(intermediate_value, step) if trial.should_prune(): assert False, "should_prune() should always return False with this pruner." raise optuna.TrialPruned() return clf.score(X_valid, y_valid) study = optuna.create_study(direction="maximize", pruner=optuna.pruners.NopPruner()) study.optimize(objective, n_trials=20) """ def prune(self, study: "optuna.study.Study", trial: "optuna.trial.FrozenTrial") -> bool: return False optuna-4.1.0/optuna/pruners/_patient.py000066400000000000000000000101061471332314300202020ustar00rootroot00000000000000from typing import Optional import numpy as np import optuna from optuna._experimental import experimental_class from optuna.pruners import BasePruner from optuna.study._study_direction import StudyDirection @experimental_class("2.8.0") class PatientPruner(BasePruner): """Pruner which wraps another pruner with tolerance. Example: .. testcode:: import numpy as np from sklearn.datasets import load_iris from sklearn.linear_model import SGDClassifier from sklearn.model_selection import train_test_split import optuna X, y = load_iris(return_X_y=True) X_train, X_valid, y_train, y_valid = train_test_split(X, y) classes = np.unique(y) def objective(trial): alpha = trial.suggest_float("alpha", 0.0, 1.0) clf = SGDClassifier(alpha=alpha) n_train_iter = 100 for step in range(n_train_iter): clf.partial_fit(X_train, y_train, classes=classes) intermediate_value = clf.score(X_valid, y_valid) trial.report(intermediate_value, step) if trial.should_prune(): raise optuna.TrialPruned() return clf.score(X_valid, y_valid) study = optuna.create_study( direction="maximize", pruner=optuna.pruners.PatientPruner(optuna.pruners.MedianPruner(), patience=1), ) study.optimize(objective, n_trials=20) Args: wrapped_pruner: Wrapped pruner to perform pruning when :class:`~optuna.pruners.PatientPruner` allows a trial to be pruned. If it is :obj:`None`, this pruner is equivalent to early-stopping taken the intermediate values in the individual trial. patience: Pruning is disabled until the objective doesn't improve for ``patience`` consecutive steps. min_delta: Tolerance value to check whether or not the objective improves. This value should be non-negative. """ def __init__( self, wrapped_pruner: Optional[BasePruner], patience: int, min_delta: float = 0.0 ) -> None: if patience < 0: raise ValueError(f"patience cannot be negative but got {patience}.") if min_delta < 0: raise ValueError(f"min_delta cannot be negative but got {min_delta}.") self._wrapped_pruner = wrapped_pruner self._patience = patience self._min_delta = min_delta def prune(self, study: "optuna.study.Study", trial: "optuna.trial.FrozenTrial") -> bool: step = trial.last_step if step is None: return False intermediate_values = trial.intermediate_values steps = np.asarray(list(intermediate_values.keys())) # Do not prune if number of step to determine are insufficient. if steps.size <= self._patience + 1: return False steps.sort() # This is the score patience steps ago steps_before_patience = steps[: -self._patience - 1] scores_before_patience = np.asarray( list(intermediate_values[step] for step in steps_before_patience) ) # And these are the scores after that steps_after_patience = steps[-self._patience - 1 :] scores_after_patience = np.asarray( list(intermediate_values[step] for step in steps_after_patience) ) direction = study.direction if direction == StudyDirection.MINIMIZE: maybe_prune = np.nanmin(scores_before_patience) + self._min_delta < np.nanmin( scores_after_patience ) else: maybe_prune = np.nanmax(scores_before_patience) - self._min_delta > np.nanmax( scores_after_patience ) if maybe_prune: if self._wrapped_pruner is not None: return self._wrapped_pruner.prune(study, trial) else: return True else: return False optuna-4.1.0/optuna/pruners/_percentile.py000066400000000000000000000160431471332314300206760ustar00rootroot00000000000000from __future__ import annotations from collections.abc import KeysView import functools import math import numpy as np import optuna from optuna.pruners import BasePruner from optuna.study._study_direction import StudyDirection from optuna.trial._state import TrialState def _get_best_intermediate_result_over_steps( trial: "optuna.trial.FrozenTrial", direction: StudyDirection ) -> float: values = np.asarray(list(trial.intermediate_values.values()), dtype=float) if direction == StudyDirection.MAXIMIZE: return np.nanmax(values) return np.nanmin(values) def _get_percentile_intermediate_result_over_trials( completed_trials: list["optuna.trial.FrozenTrial"], direction: StudyDirection, step: int, percentile: float, n_min_trials: int, ) -> float: if len(completed_trials) == 0: raise ValueError("No trials have been completed.") intermediate_values = [ t.intermediate_values[step] for t in completed_trials if step in t.intermediate_values ] if len(intermediate_values) < n_min_trials: return math.nan if direction == StudyDirection.MAXIMIZE: percentile = 100 - percentile return float( np.nanpercentile( np.array(intermediate_values, dtype=float), percentile, ) ) def _is_first_in_interval_step( step: int, intermediate_steps: KeysView[int], n_warmup_steps: int, interval_steps: int ) -> bool: nearest_lower_pruning_step = ( step - n_warmup_steps ) // interval_steps * interval_steps + n_warmup_steps assert nearest_lower_pruning_step >= 0 # `intermediate_steps` may not be sorted so we must go through all elements. second_last_step = functools.reduce( lambda second_last_step, s: s if s > second_last_step and s != step else second_last_step, intermediate_steps, -1, ) return second_last_step < nearest_lower_pruning_step class PercentilePruner(BasePruner): """Pruner to keep the specified percentile of the trials. Prune if the best intermediate value is in the bottom percentile among trials at the same step. Example: .. testcode:: import numpy as np from sklearn.datasets import load_iris from sklearn.linear_model import SGDClassifier from sklearn.model_selection import train_test_split import optuna X, y = load_iris(return_X_y=True) X_train, X_valid, y_train, y_valid = train_test_split(X, y) classes = np.unique(y) def objective(trial): alpha = trial.suggest_float("alpha", 0.0, 1.0) clf = SGDClassifier(alpha=alpha) n_train_iter = 100 for step in range(n_train_iter): clf.partial_fit(X_train, y_train, classes=classes) intermediate_value = clf.score(X_valid, y_valid) trial.report(intermediate_value, step) if trial.should_prune(): raise optuna.TrialPruned() return clf.score(X_valid, y_valid) study = optuna.create_study( direction="maximize", pruner=optuna.pruners.PercentilePruner( 25.0, n_startup_trials=5, n_warmup_steps=30, interval_steps=10 ), ) study.optimize(objective, n_trials=20) Args: percentile: Percentile which must be between 0 and 100 inclusive (e.g., When given 25.0, top of 25th percentile trials are kept). n_startup_trials: Pruning is disabled until the given number of trials finish in the same study. n_warmup_steps: Pruning is disabled until the trial exceeds the given number of step. Note that this feature assumes that ``step`` starts at zero. interval_steps: Interval in number of steps between the pruning checks, offset by the warmup steps. If no value has been reported at the time of a pruning check, that particular check will be postponed until a value is reported. Value must be at least 1. n_min_trials: Minimum number of reported trial results at a step to judge whether to prune. If the number of reported intermediate values from all trials at the current step is less than ``n_min_trials``, the trial will not be pruned. This can be used to ensure that a minimum number of trials are run to completion without being pruned. """ def __init__( self, percentile: float, n_startup_trials: int = 5, n_warmup_steps: int = 0, interval_steps: int = 1, *, n_min_trials: int = 1, ) -> None: if not 0.0 <= percentile <= 100: raise ValueError( "Percentile must be between 0 and 100 inclusive but got {}.".format(percentile) ) if n_startup_trials < 0: raise ValueError( "Number of startup trials cannot be negative but got {}.".format(n_startup_trials) ) if n_warmup_steps < 0: raise ValueError( "Number of warmup steps cannot be negative but got {}.".format(n_warmup_steps) ) if interval_steps < 1: raise ValueError( "Pruning interval steps must be at least 1 but got {}.".format(interval_steps) ) if n_min_trials < 1: raise ValueError( "Number of trials for pruning must be at least 1 but got {}.".format(n_min_trials) ) self._percentile = percentile self._n_startup_trials = n_startup_trials self._n_warmup_steps = n_warmup_steps self._interval_steps = interval_steps self._n_min_trials = n_min_trials def prune(self, study: "optuna.study.Study", trial: "optuna.trial.FrozenTrial") -> bool: completed_trials = study.get_trials(deepcopy=False, states=(TrialState.COMPLETE,)) n_trials = len(completed_trials) if n_trials == 0: return False if n_trials < self._n_startup_trials: return False step = trial.last_step if step is None: return False n_warmup_steps = self._n_warmup_steps if step < n_warmup_steps: return False if not _is_first_in_interval_step( step, trial.intermediate_values.keys(), n_warmup_steps, self._interval_steps ): return False direction = study.direction best_intermediate_result = _get_best_intermediate_result_over_steps(trial, direction) if math.isnan(best_intermediate_result): return True p = _get_percentile_intermediate_result_over_trials( completed_trials, direction, step, self._percentile, self._n_min_trials ) if math.isnan(p): return False if direction == StudyDirection.MAXIMIZE: return best_intermediate_result < p return best_intermediate_result > p optuna-4.1.0/optuna/pruners/_successive_halving.py000066400000000000000000000246351471332314300224360ustar00rootroot00000000000000from __future__ import annotations import math import optuna from optuna.pruners._base import BasePruner from optuna.study._study_direction import StudyDirection from optuna.trial._state import TrialState class SuccessiveHalvingPruner(BasePruner): """Pruner using Asynchronous Successive Halving Algorithm. `Successive Halving `__ is a bandit-based algorithm to identify the best one among multiple configurations. This class implements an asynchronous version of Successive Halving. Please refer to the paper of `Asynchronous Successive Halving `__ for detailed descriptions. Note that, this class does not take care of the parameter for the maximum resource, referred to as :math:`R` in the paper. The maximum resource allocated to a trial is typically limited inside the objective function (e.g., ``step`` number in `simple_pruning.py `__, ``EPOCH`` number in `chainer_integration.py `__). .. seealso:: Please refer to :meth:`~optuna.trial.Trial.report`. Example: We minimize an objective function with ``SuccessiveHalvingPruner``. .. testcode:: import numpy as np from sklearn.datasets import load_iris from sklearn.linear_model import SGDClassifier from sklearn.model_selection import train_test_split import optuna X, y = load_iris(return_X_y=True) X_train, X_valid, y_train, y_valid = train_test_split(X, y) classes = np.unique(y) def objective(trial): alpha = trial.suggest_float("alpha", 0.0, 1.0) clf = SGDClassifier(alpha=alpha) n_train_iter = 100 for step in range(n_train_iter): clf.partial_fit(X_train, y_train, classes=classes) intermediate_value = clf.score(X_valid, y_valid) trial.report(intermediate_value, step) if trial.should_prune(): raise optuna.TrialPruned() return clf.score(X_valid, y_valid) study = optuna.create_study( direction="maximize", pruner=optuna.pruners.SuccessiveHalvingPruner() ) study.optimize(objective, n_trials=20) Args: min_resource: A parameter for specifying the minimum resource allocated to a trial (in the `paper `__ this parameter is referred to as :math:`r`). This parameter defaults to 'auto' where the value is determined based on a heuristic that looks at the number of required steps for the first trial to complete. A trial is never pruned until it executes :math:`\\mathsf{min}\\_\\mathsf{resource} \\times \\mathsf{reduction}\\_\\mathsf{factor}^{ \\mathsf{min}\\_\\mathsf{early}\\_\\mathsf{stopping}\\_\\mathsf{rate}}` steps (i.e., the completion point of the first rung). When the trial completes the first rung, it will be promoted to the next rung only if the value of the trial is placed in the top :math:`{1 \\over \\mathsf{reduction}\\_\\mathsf{factor}}` fraction of the all trials that already have reached the point (otherwise it will be pruned there). If the trial won the competition, it runs until the next completion point (i.e., :math:`\\mathsf{min}\\_\\mathsf{resource} \\times \\mathsf{reduction}\\_\\mathsf{factor}^{ (\\mathsf{min}\\_\\mathsf{early}\\_\\mathsf{stopping}\\_\\mathsf{rate} + \\mathsf{rung})}` steps) and repeats the same procedure. .. note:: If the step of the last intermediate value may change with each trial, please manually specify the minimum possible step to ``min_resource``. reduction_factor: A parameter for specifying reduction factor of promotable trials (in the `paper `__ this parameter is referred to as :math:`\\eta`). At the completion point of each rung, about :math:`{1 \\over \\mathsf{reduction}\\_\\mathsf{factor}}` trials will be promoted. min_early_stopping_rate: A parameter for specifying the minimum early-stopping rate (in the `paper `__ this parameter is referred to as :math:`s`). bootstrap_count: Minimum number of trials that need to complete a rung before any trial is considered for promotion into the next rung. """ def __init__( self, min_resource: str | int = "auto", reduction_factor: int = 4, min_early_stopping_rate: int = 0, bootstrap_count: int = 0, ) -> None: if isinstance(min_resource, str) and min_resource != "auto": raise ValueError( "The value of `min_resource` is {}, " "but must be either `min_resource` >= 1 or 'auto'".format(min_resource) ) if isinstance(min_resource, int) and min_resource < 1: raise ValueError( "The value of `min_resource` is {}, " "but must be either `min_resource >= 1` or 'auto'".format(min_resource) ) if reduction_factor < 2: raise ValueError( "The value of `reduction_factor` is {}, " "but must be `reduction_factor >= 2`".format(reduction_factor) ) if min_early_stopping_rate < 0: raise ValueError( "The value of `min_early_stopping_rate` is {}, " "but must be `min_early_stopping_rate >= 0`".format(min_early_stopping_rate) ) if bootstrap_count < 0: raise ValueError( "The value of `bootstrap_count` is {}, " "but must be `bootstrap_count >= 0`".format(bootstrap_count) ) if bootstrap_count > 0 and min_resource == "auto": raise ValueError( "bootstrap_count > 0 and min_resource == 'auto' " "are mutually incompatible, bootstrap_count is {}".format(bootstrap_count) ) self._min_resource: int | None = None if isinstance(min_resource, int): self._min_resource = min_resource self._reduction_factor = reduction_factor self._min_early_stopping_rate = min_early_stopping_rate self._bootstrap_count = bootstrap_count def prune(self, study: "optuna.study.Study", trial: "optuna.trial.FrozenTrial") -> bool: step = trial.last_step if step is None: return False rung = _get_current_rung(trial) value = trial.intermediate_values[step] trials: list["optuna.trial.FrozenTrial"] | None = None while True: if self._min_resource is None: if trials is None: trials = study.get_trials(deepcopy=False) self._min_resource = _estimate_min_resource(trials) if self._min_resource is None: return False assert self._min_resource is not None rung_promotion_step = self._min_resource * ( self._reduction_factor ** (self._min_early_stopping_rate + rung) ) if step < rung_promotion_step: return False if math.isnan(value): return True if trials is None: trials = study.get_trials(deepcopy=False) rung_key = _completed_rung_key(rung) study._storage.set_trial_system_attr(trial._trial_id, rung_key, value) competing = _get_competing_values(trials, value, rung_key) # 'competing' already includes the current trial # Therefore, we need to use the '<=' operator here if len(competing) <= self._bootstrap_count: return True if not _is_trial_promotable_to_next_rung( value, competing, self._reduction_factor, study.direction, ): return True rung += 1 def _estimate_min_resource(trials: list["optuna.trial.FrozenTrial"]) -> int | None: n_steps = [ t.last_step for t in trials if t.state == TrialState.COMPLETE and t.last_step is not None ] if not n_steps: return None # Get the maximum number of steps and divide it by 100. last_step = max(n_steps) return max(last_step // 100, 1) def _get_current_rung(trial: "optuna.trial.FrozenTrial") -> int: # The following loop takes `O(log step)` iterations. rung = 0 while _completed_rung_key(rung) in trial.system_attrs: rung += 1 return rung def _completed_rung_key(rung: int) -> str: return "completed_rung_{}".format(rung) def _get_competing_values( trials: list["optuna.trial.FrozenTrial"], value: float, rung_key: str ) -> list[float]: competing_values = [t.system_attrs[rung_key] for t in trials if rung_key in t.system_attrs] competing_values.append(value) return competing_values def _is_trial_promotable_to_next_rung( value: float, competing_values: list[float], reduction_factor: int, study_direction: StudyDirection, ) -> bool: promotable_idx = (len(competing_values) // reduction_factor) - 1 if promotable_idx == -1: # Optuna does not support suspending or resuming ongoing trials. Therefore, for the first # `eta - 1` trials, this implementation instead promotes the trial if its value is the # smallest one among the competing values. promotable_idx = 0 competing_values.sort() if study_direction == StudyDirection.MAXIMIZE: return value >= competing_values[-(promotable_idx + 1)] return value <= competing_values[promotable_idx] optuna-4.1.0/optuna/pruners/_threshold.py000066400000000000000000000106301471332314300205340ustar00rootroot00000000000000from __future__ import annotations import math from typing import Any import optuna from optuna.pruners import BasePruner from optuna.pruners._percentile import _is_first_in_interval_step def _check_value(value: Any) -> float: try: # For convenience, we allow users to report a value that can be cast to `float`. value = float(value) except (TypeError, ValueError): message = "The `value` argument is of type '{}' but supposed to be a float.".format( type(value).__name__ ) raise TypeError(message) from None return value class ThresholdPruner(BasePruner): """Pruner to detect outlying metrics of the trials. Prune if a metric exceeds upper threshold, falls behind lower threshold or reaches ``nan``. Example: .. testcode:: from optuna import create_study from optuna.pruners import ThresholdPruner from optuna import TrialPruned def objective_for_upper(trial): for step, y in enumerate(ys_for_upper): trial.report(y, step) if trial.should_prune(): raise TrialPruned() return ys_for_upper[-1] def objective_for_lower(trial): for step, y in enumerate(ys_for_lower): trial.report(y, step) if trial.should_prune(): raise TrialPruned() return ys_for_lower[-1] ys_for_upper = [0.0, 0.1, 0.2, 0.5, 1.2] ys_for_lower = [100.0, 90.0, 0.1, 0.0, -1] study = create_study(pruner=ThresholdPruner(upper=1.0)) study.optimize(objective_for_upper, n_trials=10) study = create_study(pruner=ThresholdPruner(lower=0.0)) study.optimize(objective_for_lower, n_trials=10) Args: lower: A minimum value which determines whether pruner prunes or not. If an intermediate value is smaller than lower, it prunes. upper: A maximum value which determines whether pruner prunes or not. If an intermediate value is larger than upper, it prunes. n_warmup_steps: Pruning is disabled if the step is less than the given number of warmup steps. interval_steps: Interval in number of steps between the pruning checks, offset by the warmup steps. If no value has been reported at the time of a pruning check, that particular check will be postponed until a value is reported. Value must be at least 1. """ def __init__( self, lower: float | None = None, upper: float | None = None, n_warmup_steps: int = 0, interval_steps: int = 1, ) -> None: if lower is None and upper is None: raise TypeError("Either lower or upper must be specified.") if lower is not None: lower = _check_value(lower) if upper is not None: upper = _check_value(upper) lower = lower if lower is not None else -float("inf") upper = upper if upper is not None else float("inf") if lower > upper: raise ValueError("lower should be smaller than upper.") if n_warmup_steps < 0: raise ValueError( "Number of warmup steps cannot be negative but got {}.".format(n_warmup_steps) ) if interval_steps < 1: raise ValueError( "Pruning interval steps must be at least 1 but got {}.".format(interval_steps) ) self._lower = lower self._upper = upper self._n_warmup_steps = n_warmup_steps self._interval_steps = interval_steps def prune(self, study: "optuna.study.Study", trial: "optuna.trial.FrozenTrial") -> bool: step = trial.last_step if step is None: return False n_warmup_steps = self._n_warmup_steps if step < n_warmup_steps: return False if not _is_first_in_interval_step( step, trial.intermediate_values.keys(), n_warmup_steps, self._interval_steps ): return False latest_value = trial.intermediate_values[step] if math.isnan(latest_value): return True if latest_value < self._lower: return True if latest_value > self._upper: return True return False optuna-4.1.0/optuna/pruners/_wilcoxon.py000066400000000000000000000221221471332314300204010ustar00rootroot00000000000000from __future__ import annotations from typing import TYPE_CHECKING import warnings import numpy as np import optuna from optuna._experimental import experimental_class from optuna.pruners import BasePruner from optuna.study._study_direction import StudyDirection from optuna.trial import FrozenTrial if TYPE_CHECKING: import scipy.stats as ss else: from optuna._imports import _LazyImport ss = _LazyImport("scipy.stats") @experimental_class("3.6.0") class WilcoxonPruner(BasePruner): """Pruner based on the `Wilcoxon signed-rank test `__. This pruner performs the Wilcoxon signed-rank test between the current trial and the current best trial, and stops whenever the pruner is sure up to a given p-value that the current trial is worse than the best one. This pruner is effective for optimizing the mean/median of some (costly-to-evaluate) performance scores over a set of problem instances. Example applications include the optimization of: * the mean performance of a heuristic method (simulated annealing, genetic algorithm, SAT solver, etc.) on a set of problem instances, * the k-fold cross-validation score of a machine learning model, and * the accuracy of outputs of a large language model (LLM) on a set of questions. There can be "easy" or "hard" instances (the pruner handles correspondence of the instances between different trials). In each trial, it is recommended to shuffle the evaluation order, so that the optimization doesn't overfit to the instances in the beginning. When you use this pruner, you must call ``Trial.report(value, step)`` method for each step (instance id) with the evaluated value. The instance id may not be in ascending order. This is different from other pruners in that the reported value need not converge to the real value. To use pruners such as :class:`~optuna.pruners.SuccessiveHalvingPruner` in the same setting, you must provide e.g., the historical average of the evaluated values. .. seealso:: Please refer to :meth:`~optuna.trial.Trial.report`. Example: .. testcode:: import optuna import numpy as np # We minimize the mean evaluation loss over all the problem instances. def evaluate(param, instance): # A toy loss function for demonstrative purpose. return (param - instance) ** 2 problem_instances = np.linspace(-1, 1, 100) def objective(trial): # Sample a parameter. param = trial.suggest_float("param", -1, 1) # Evaluate performance of the parameter. results = [] # For best results, shuffle the evaluation order in each trial. instance_ids = np.random.permutation(len(problem_instances)) for instance_id in instance_ids: loss = evaluate(param, problem_instances[instance_id]) results.append(loss) # Report loss together with the instance id. # CAVEAT: You need to pass the same id for the same instance, # otherwise WilcoxonPruner cannot correctly pair the losses across trials and # the pruning performance will degrade. trial.report(loss, instance_id) if trial.should_prune(): # Return the current predicted value instead of raising `TrialPruned`. # This is a workaround to tell the Optuna about the evaluation # results in pruned trials. (See the note below.) return sum(results) / len(results) return sum(results) / len(results) study = optuna.create_study(pruner=optuna.pruners.WilcoxonPruner(p_threshold=0.1)) study.optimize(objective, n_trials=100) .. note:: This pruner cannot handle ``infinity`` or ``nan`` values. Trials containing those values are never pruned. .. note:: If :func:`~optuna.trial.FrozenTrial.should_prune` returns :obj:`True`, you can return an estimation of the final value (e.g., the average of all evaluated values) instead of ``raise optuna.TrialPruned()``. This is a workaround for the problem that currently there is no way to tell Optuna the predicted objective value for trials raising :class:`optuna.TrialPruned`. Args: p_threshold: The p-value threshold for pruning. This value should be between 0 and 1. A trial will be pruned whenever the pruner is sure up to the given p-value that the current trial is worse than the best trial. The larger this value is, the more aggressive pruning will be performed. Defaults to 0.1. .. note:: This pruner repeatedly performs statistical tests between the current trial and the current best trial with increasing samples. The false-positive rate of such a sequential test is different from performing the test only once. To get the nominal false-positive rate, please specify the Pocock-corrected p-value. n_startup_steps: The number of steps before which no trials are pruned. Pruning starts only after you have ``n_startup_steps`` steps of available observations for comparison between the current trial and the best trial. Defaults to 0 (pruning kicks in from the very first step). """ # NOQA: E501 def __init__( self, *, p_threshold: float = 0.1, n_startup_steps: int = 0, ) -> None: if n_startup_steps < 0: raise ValueError(f"n_startup_steps must be nonnegative but got {n_startup_steps}.") if not 0.0 <= p_threshold <= 1.0: raise ValueError(f"p_threshold must be between 0 and 1 but got {p_threshold}.") self._n_startup_steps = n_startup_steps self._p_threshold = p_threshold def prune(self, study: "optuna.study.Study", trial: FrozenTrial) -> bool: if len(trial.intermediate_values) == 0: return False steps, step_values = np.array(list(trial.intermediate_values.items())).T if np.any(~np.isfinite(step_values)): warnings.warn( f"The intermediate values of the current trial (trial {trial.number}) " f"contain infinity/NaNs. WilcoxonPruner will not prune this trial." ) return False try: best_trial = study.best_trial except ValueError: return False if len(best_trial.intermediate_values) == 0: warnings.warn( "The best trial has no intermediate values so WilcoxonPruner cannot prune trials. " "If you have added the best trial with Study.add_trial, please consider setting " "intermediate_values argument." ) return False best_steps, best_step_values = np.array(list(best_trial.intermediate_values.items())).T if np.any(~np.isfinite(best_step_values)): warnings.warn( f"The intermediate values of the best trial (trial {best_trial.number}) " f"contain infinity/NaNs. WilcoxonPruner will not prune the current trial." ) return False _, idx1, idx2 = np.intersect1d(steps, best_steps, return_indices=True) if len(idx1) < len(step_values): # This if-statement is never satisfied if following "average_is_best" safety works, # because the safety ensures that the best trial always has the all steps. warnings.warn( "WilcoxonPruner finds steps existing in the current trial " "but does not exist in the best trial. " "Those values are ignored." ) diff_values = step_values[idx1] - best_step_values[idx2] if len(diff_values) < self._n_startup_steps: return False if study.direction == StudyDirection.MAXIMIZE: alt = "less" average_is_best = sum(best_step_values) / len(best_step_values) <= sum( step_values ) / len(step_values) else: alt = "greater" average_is_best = sum(best_step_values) / len(best_step_values) >= sum( step_values ) / len(step_values) # We use zsplit to avoid the problem when all values are zero. p = ss.wilcoxon(diff_values, alternative=alt, zero_method="zsplit").pvalue if p < self._p_threshold and average_is_best: # ss.wilcoxon found the current trial is probably worse than the best trial, # but the value of the best trial was not better than # the average of the current trial's intermediate values. # For safety, WilcoxonPruner concludes not to prune it for now. return False return p < self._p_threshold optuna-4.1.0/optuna/py.typed000066400000000000000000000000001471332314300160160ustar00rootroot00000000000000optuna-4.1.0/optuna/samplers/000077500000000000000000000000001471332314300161575ustar00rootroot00000000000000optuna-4.1.0/optuna/samplers/__init__.py000066400000000000000000000015451471332314300202750ustar00rootroot00000000000000from optuna.samplers import nsgaii from optuna.samplers._base import BaseSampler from optuna.samplers._brute_force import BruteForceSampler from optuna.samplers._cmaes import CmaEsSampler from optuna.samplers._gp.sampler import GPSampler from optuna.samplers._grid import GridSampler from optuna.samplers._nsgaiii._sampler import NSGAIIISampler from optuna.samplers._partial_fixed import PartialFixedSampler from optuna.samplers._qmc import QMCSampler from optuna.samplers._random import RandomSampler from optuna.samplers._tpe.sampler import TPESampler from optuna.samplers.nsgaii._sampler import NSGAIISampler __all__ = [ "BaseSampler", "BruteForceSampler", "CmaEsSampler", "GridSampler", "NSGAIISampler", "NSGAIIISampler", "PartialFixedSampler", "QMCSampler", "RandomSampler", "TPESampler", "GPSampler", "nsgaii", ] optuna-4.1.0/optuna/samplers/_base.py000066400000000000000000000217231471332314300176070ustar00rootroot00000000000000from __future__ import annotations import abc from collections.abc import Callable from collections.abc import Sequence from typing import Any from typing import TYPE_CHECKING import warnings import numpy as np from optuna.distributions import BaseDistribution from optuna.trial import FrozenTrial from optuna.trial import TrialState if TYPE_CHECKING: from optuna.study import Study class BaseSampler(abc.ABC): """Base class for samplers. Optuna combines two types of sampling strategies, which are called *relative sampling* and *independent sampling*. *The relative sampling* determines values of multiple parameters simultaneously so that sampling algorithms can use relationship between parameters (e.g., correlation). Target parameters of the relative sampling are described in a relative search space, which is determined by :func:`~optuna.samplers.BaseSampler.infer_relative_search_space`. *The independent sampling* determines a value of a single parameter without considering any relationship between parameters. Target parameters of the independent sampling are the parameters not described in the relative search space. More specifically, parameters are sampled by the following procedure. At the beginning of a trial, :meth:`~optuna.samplers.BaseSampler.infer_relative_search_space` is called to determine the relative search space for the trial. During the execution of the objective function, :meth:`~optuna.samplers.BaseSampler.sample_relative` is called only once when sampling the parameters belonging to the relative search space for the first time. :meth:`~optuna.samplers.BaseSampler.sample_independent` is used to sample parameters that don't belong to the relative search space. The following figure depicts the lifetime of a trial and how the above three methods are called in the trial. .. image:: ../../../../image/sampling-sequence.png | """ def __str__(self) -> str: return self.__class__.__name__ @abc.abstractmethod def infer_relative_search_space( self, study: Study, trial: FrozenTrial ) -> dict[str, BaseDistribution]: """Infer the search space that will be used by relative sampling in the target trial. This method is called right before :func:`~optuna.samplers.BaseSampler.sample_relative` method, and the search space returned by this method is passed to it. The parameters not contained in the search space will be sampled by using :func:`~optuna.samplers.BaseSampler.sample_independent` method. Args: study: Target study object. trial: Target trial object. Take a copy before modifying this object. Returns: A dictionary containing the parameter names and parameter's distributions. .. seealso:: Please refer to :func:`~optuna.search_space.intersection_search_space` as an implementation of :func:`~optuna.samplers.BaseSampler.infer_relative_search_space`. """ raise NotImplementedError @abc.abstractmethod def sample_relative( self, study: Study, trial: FrozenTrial, search_space: dict[str, BaseDistribution] ) -> dict[str, Any]: """Sample parameters in a given search space. This method is called once at the beginning of each trial, i.e., right before the evaluation of the objective function. This method is suitable for sampling algorithms that use relationship between parameters such as Gaussian Process and CMA-ES. .. note:: The failed trials are ignored by any build-in samplers when they sample new parameters. Thus, failed trials are regarded as deleted in the samplers' perspective. Args: study: Target study object. trial: Target trial object. Take a copy before modifying this object. search_space: The search space returned by :func:`~optuna.samplers.BaseSampler.infer_relative_search_space`. Returns: A dictionary containing the parameter names and the values. """ raise NotImplementedError @abc.abstractmethod def sample_independent( self, study: Study, trial: FrozenTrial, param_name: str, param_distribution: BaseDistribution, ) -> Any: """Sample a parameter for a given distribution. This method is called only for the parameters not contained in the search space returned by :func:`~optuna.samplers.BaseSampler.sample_relative` method. This method is suitable for sampling algorithms that do not use relationship between parameters such as random sampling and TPE. .. note:: The failed trials are ignored by any build-in samplers when they sample new parameters. Thus, failed trials are regarded as deleted in the samplers' perspective. Args: study: Target study object. trial: Target trial object. Take a copy before modifying this object. param_name: Name of the sampled parameter. param_distribution: Distribution object that specifies a prior and/or scale of the sampling algorithm. Returns: A parameter value. """ raise NotImplementedError def before_trial(self, study: Study, trial: FrozenTrial) -> None: """Trial pre-processing. This method is called before the objective function is called and right after the trial is instantiated. More precisely, this method is called during trial initialization, just before the :func:`~optuna.samplers.BaseSampler.infer_relative_search_space` call. In other words, it is responsible for pre-processing that should be done before inferring the search space. .. note:: Added in v3.3.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v3.3.0. Args: study: Target study object. trial: Target trial object. """ pass def after_trial( self, study: Study, trial: FrozenTrial, state: TrialState, values: Sequence[float] | None, ) -> None: """Trial post-processing. This method is called after the objective function returns and right before the trial is finished and its state is stored. .. note:: Added in v2.4.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v2.4.0. Args: study: Target study object. trial: Target trial object. Take a copy before modifying this object. state: Resulting trial state. values: Resulting trial values. Guaranteed to not be :obj:`None` if trial succeeded. """ pass def reseed_rng(self) -> None: """Reseed sampler's random number generator. This method is called by the :class:`~optuna.study.Study` instance if trials are executed in parallel with the option ``n_jobs>1``. In that case, the sampler instance will be replicated including the state of the random number generator, and they may suggest the same values. To prevent this issue, this method assigns a different seed to each random number generator. """ pass def _raise_error_if_multi_objective(self, study: Study) -> None: if study._is_multi_objective(): raise ValueError( "If the study is being used for multi-objective optimization, " f"{self.__class__.__name__} cannot be used." ) _CONSTRAINTS_KEY = "constraints" def _process_constraints_after_trial( constraints_func: Callable[[FrozenTrial], Sequence[float]], study: Study, trial: FrozenTrial, state: TrialState, ) -> None: if state not in [TrialState.COMPLETE, TrialState.PRUNED]: return constraints = None try: con = constraints_func(trial) if np.any(np.isnan(con)): raise ValueError("Constraint values cannot be NaN.") if not isinstance(con, (tuple, list)): warnings.warn( f"Constraints should be a sequence of floats but got {type(con).__name__}." ) constraints = tuple(con) finally: assert constraints is None or isinstance(constraints, tuple) study._storage.set_trial_system_attr( trial._trial_id, _CONSTRAINTS_KEY, constraints, ) optuna-4.1.0/optuna/samplers/_brute_force.py000066400000000000000000000233561471332314300212000ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Iterable from collections.abc import Sequence from dataclasses import dataclass import decimal from typing import Any from typing import TYPE_CHECKING import numpy as np from optuna._experimental import experimental_class from optuna.distributions import BaseDistribution from optuna.distributions import CategoricalDistribution from optuna.distributions import FloatDistribution from optuna.distributions import IntDistribution from optuna.samplers import BaseSampler from optuna.samplers._lazy_random_state import LazyRandomState from optuna.trial import create_trial from optuna.trial import FrozenTrial from optuna.trial import TrialState if TYPE_CHECKING: from optuna.study import Study @dataclass class _TreeNode: # This is a class to represent the tree of search space. # A tree node has three states: # 1. Unexpanded. This is represented by children=None. # 2. Leaf. This is represented by children={} and param_name=None. # 3. Normal node. It has a param_name and non-empty children. param_name: str | None = None children: dict[float, "_TreeNode"] | None = None def expand(self, param_name: str | None, search_space: Iterable[float]) -> None: # If the node is unexpanded, expand it. # Otherwise, check if the node is compatible with the given search space. if self.children is None: # Expand the node self.param_name = param_name self.children = {value: _TreeNode() for value in search_space} else: if self.param_name != param_name: raise ValueError(f"param_name mismatch: {self.param_name} != {param_name}") if self.children.keys() != set(search_space): raise ValueError( f"search_space mismatch: {set(self.children.keys())} != {set(search_space)}" ) def set_leaf(self) -> None: self.expand(None, []) def add_path( self, params_and_search_spaces: Iterable[tuple[str, Iterable[float], float]] ) -> "_TreeNode" | None: # Add a path (i.e. a list of suggested parameters in one trial) to the tree. current_node = self for param_name, search_space, value in params_and_search_spaces: current_node.expand(param_name, search_space) assert current_node.children is not None if value not in current_node.children: return None current_node = current_node.children[value] return current_node def count_unexpanded(self) -> int: # Count the number of unexpanded nodes in the subtree. return ( 1 if self.children is None else sum(child.count_unexpanded() for child in self.children.values()) ) def sample_child(self, rng: np.random.RandomState) -> float: assert self.children is not None # Sample an unexpanded node in the subtree uniformly, and return the first # parameter value in the path to the node. # Equivalently, we sample the child node with weights proportional to the number # of unexpanded nodes in the subtree. weights = np.array( [child.count_unexpanded() for child in self.children.values()], dtype=np.float64 ) weights /= weights.sum() return rng.choice(list(self.children.keys()), p=weights) @experimental_class("3.1.0") class BruteForceSampler(BaseSampler): """Sampler using brute force. This sampler performs exhaustive search on the defined search space. Example: .. testcode:: import optuna def objective(trial): c = trial.suggest_categorical("c", ["float", "int"]) if c == "float": return trial.suggest_float("x", 1, 3, step=0.5) elif c == "int": a = trial.suggest_int("a", 1, 3) b = trial.suggest_int("b", a, 3) return a + b study = optuna.create_study(sampler=optuna.samplers.BruteForceSampler()) study.optimize(objective) Note: The defined search space must be finite. Therefore, when using :class:`~optuna.distributions.FloatDistribution` or :func:`~optuna.trial.Trial.suggest_float`, ``step=None`` is not allowed. Note: The sampler may fail to try the entire search space in when the suggestion ranges or parameters are changed in the same :class:`~optuna.study.Study`. Args: seed: A seed to fix the order of trials as the search order randomly shuffled. Please note that it is not recommended using this option in distributed optimization settings since this option cannot ensure the order of trials and may increase the number of duplicate suggestions during distributed optimization. """ def __init__(self, seed: int | None = None) -> None: self._rng = LazyRandomState(seed) def infer_relative_search_space( self, study: Study, trial: FrozenTrial ) -> dict[str, BaseDistribution]: return {} def sample_relative( self, study: Study, trial: FrozenTrial, search_space: dict[str, BaseDistribution] ) -> dict[str, Any]: return {} @staticmethod def _populate_tree( tree: _TreeNode, trials: Iterable[FrozenTrial], params: dict[str, Any] ) -> None: # Populate tree under given params from the given trials. incomplete_leaves: list[_TreeNode] = [] for trial in trials: if not all(p in trial.params and trial.params[p] == v for p, v in params.items()): continue leaf = tree.add_path( ( ( param_name, _enumerate_candidates(param_distribution), param_distribution.to_internal_repr(trial.params[param_name]), ) for param_name, param_distribution in trial.distributions.items() if param_name not in params ) ) if leaf is not None: # The parameters are on the defined grid. if trial.state.is_finished(): leaf.set_leaf() else: incomplete_leaves.append(leaf) # Add all incomplete leaf nodes at the end because they may not have complete search space. for leaf in incomplete_leaves: if leaf.children is None: leaf.set_leaf() def sample_independent( self, study: Study, trial: FrozenTrial, param_name: str, param_distribution: BaseDistribution, ) -> Any: trials = study.get_trials( deepcopy=False, states=( TrialState.COMPLETE, TrialState.PRUNED, TrialState.RUNNING, TrialState.FAIL, ), ) tree = _TreeNode() candidates = _enumerate_candidates(param_distribution) tree.expand(param_name, candidates) # Populating must happen after the initialization above to prevent `tree` from # being initialized as an empty graph, which is created with n_jobs > 1 # where we get trials[i].params = {} for some i. self._populate_tree(tree, (t for t in trials if t.number != trial.number), trial.params) if tree.count_unexpanded() == 0: return param_distribution.to_external_repr(self._rng.rng.choice(candidates)) else: return param_distribution.to_external_repr(tree.sample_child(self._rng.rng)) def after_trial( self, study: Study, trial: FrozenTrial, state: TrialState, values: Sequence[float] | None, ) -> None: trials = study.get_trials( deepcopy=False, states=( TrialState.COMPLETE, TrialState.PRUNED, TrialState.RUNNING, TrialState.FAIL, ), ) tree = _TreeNode() self._populate_tree( tree, ( ( t if t.number != trial.number else create_trial( state=state, # Set current trial as complete. values=values, params=trial.params, distributions=trial.distributions, ) ) for t in trials ), {}, ) if tree.count_unexpanded() == 0: study.stop() def _enumerate_candidates(param_distribution: BaseDistribution) -> Sequence[float]: if isinstance(param_distribution, FloatDistribution): if param_distribution.step is None: raise ValueError( "FloatDistribution.step must be given for BruteForceSampler" " (otherwise, the search space will be infinite)." ) low = decimal.Decimal(str(param_distribution.low)) high = decimal.Decimal(str(param_distribution.high)) step = decimal.Decimal(str(param_distribution.step)) ret = [] value = low while value <= high: ret.append(float(value)) value += step return ret elif isinstance(param_distribution, IntDistribution): return list( range(param_distribution.low, param_distribution.high + 1, param_distribution.step) ) elif isinstance(param_distribution, CategoricalDistribution): return list(range(len(param_distribution.choices))) # Internal representations. else: raise ValueError(f"Unknown distribution {param_distribution}.") optuna-4.1.0/optuna/samplers/_cmaes.py000066400000000000000000000771021471332314300177670ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Callable from collections.abc import Sequence import copy import math import pickle from typing import Any from typing import cast from typing import NamedTuple from typing import TYPE_CHECKING from typing import Union import numpy as np import optuna from optuna import logging from optuna._experimental import warn_experimental_argument from optuna._imports import _LazyImport from optuna._transform import _SearchSpaceTransform from optuna.distributions import BaseDistribution from optuna.distributions import FloatDistribution from optuna.distributions import IntDistribution from optuna.samplers import BaseSampler from optuna.samplers._lazy_random_state import LazyRandomState from optuna.search_space import IntersectionSearchSpace from optuna.study._study_direction import StudyDirection from optuna.trial import FrozenTrial from optuna.trial import TrialState if TYPE_CHECKING: import cmaes CmaClass = Union[cmaes.CMA, cmaes.SepCMA, cmaes.CMAwM] else: cmaes = _LazyImport("cmaes") _logger = logging.get_logger(__name__) _EPS = 1e-10 # The value of system_attrs must be less than 2046 characters on RDBStorage. _SYSTEM_ATTR_MAX_LENGTH = 2045 class _CmaEsAttrKeys(NamedTuple): optimizer: Callable[[int], str] generation: Callable[[int], str] popsize: Callable[[], str] n_restarts: Callable[[], str] n_restarts_with_large: str poptype: str small_n_eval: str large_n_eval: str class CmaEsSampler(BaseSampler): """A sampler using `cmaes `__ as the backend. Example: Optimize a simple quadratic function by using :class:`~optuna.samplers.CmaEsSampler`. .. code-block:: console $ pip install cmaes .. testcode:: import optuna def objective(trial): x = trial.suggest_float("x", -1, 1) y = trial.suggest_int("y", -1, 1) return x**2 + y sampler = optuna.samplers.CmaEsSampler() study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=20) Please note that this sampler does not support CategoricalDistribution. However, :class:`~optuna.distributions.FloatDistribution` with ``step``, (:func:`~optuna.trial.Trial.suggest_float`) and :class:`~optuna.distributions.IntDistribution` (:func:`~optuna.trial.Trial.suggest_int`) are supported. If your search space contains categorical parameters, I recommend you to use :class:`~optuna.samplers.TPESampler` instead. Furthermore, there is room for performance improvements in parallel optimization settings. This sampler cannot use some trials for updating the parameters of multivariate normal distribution. For further information about CMA-ES algorithm, please refer to the following papers: - `N. Hansen, The CMA Evolution Strategy: A Tutorial. arXiv:1604.00772, 2016. `__ - `A. Auger and N. Hansen. A restart CMA evolution strategy with increasing population size. In Proceedings of the IEEE Congress on Evolutionary Computation (CEC 2005), pages 1769–1776. IEEE Press, 2005. `__ - `N. Hansen. Benchmarking a BI-Population CMA-ES on the BBOB-2009 Function Testbed. GECCO Workshop, 2009. `__ - `Raymond Ros, Nikolaus Hansen. A Simple Modification in CMA-ES Achieving Linear Time and Space Complexity. 10th International Conference on Parallel Problem Solving From Nature, Sep 2008, Dortmund, Germany. inria-00287367. `__ - `Masahiro Nomura, Shuhei Watanabe, Youhei Akimoto, Yoshihiko Ozaki, Masaki Onishi. Warm Starting CMA-ES for Hyperparameter Optimization, AAAI. 2021. `__ - `R. Hamano, S. Saito, M. Nomura, S. Shirakawa. CMA-ES with Margin: Lower-Bounding Marginal Probability for Mixed-Integer Black-Box Optimization, GECCO. 2022. `__ - `M. Nomura, Y. Akimoto, I. Ono. CMA-ES with Learning Rate Adaptation: Can CMA-ES with Default Population Size Solve Multimodal and Noisy Problems?, GECCO. 2023. `__ .. seealso:: You can also use `optuna_integration.PyCmaSampler `__ which is a sampler using cma library as the backend. Args: x0: A dictionary of an initial parameter values for CMA-ES. By default, the mean of ``low`` and ``high`` for each distribution is used. Note that ``x0`` is sampled uniformly within the search space domain for each restart if you specify ``restart_strategy`` argument. sigma0: Initial standard deviation of CMA-ES. By default, ``sigma0`` is set to ``min_range / 6``, where ``min_range`` denotes the minimum range of the distributions in the search space. seed: A random seed for CMA-ES. n_startup_trials: The independent sampling is used instead of the CMA-ES algorithm until the given number of trials finish in the same study. independent_sampler: A :class:`~optuna.samplers.BaseSampler` instance that is used for independent sampling. The parameters not contained in the relative search space are sampled by this sampler. The search space for :class:`~optuna.samplers.CmaEsSampler` is determined by :func:`~optuna.search_space.intersection_search_space()`. If :obj:`None` is specified, :class:`~optuna.samplers.RandomSampler` is used as the default. .. seealso:: :class:`optuna.samplers` module provides built-in independent samplers such as :class:`~optuna.samplers.RandomSampler` and :class:`~optuna.samplers.TPESampler`. warn_independent_sampling: If this is :obj:`True`, a warning message is emitted when the value of a parameter is sampled by using an independent sampler. Note that the parameters of the first trial in a study are always sampled via an independent sampler, so no warning messages are emitted in this case. restart_strategy: Strategy for restarting CMA-ES optimization when converges to a local minimum. If :obj:`None` is given, CMA-ES will not restart (default). If 'ipop' is given, CMA-ES will restart with increasing population size. if 'bipop' is given, CMA-ES will restart with the population size increased or decreased. Please see also ``inc_popsize`` parameter. .. note:: Added in v2.1.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v2.1.0. popsize: A population size of CMA-ES. When ``restart_strategy = 'ipop'`` or ``restart_strategy = 'bipop'`` is specified, this is used as the initial population size. inc_popsize: Multiplier for increasing population size before each restart. This argument will be used when ``restart_strategy = 'ipop'`` or ``restart_strategy = 'bipop'`` is specified. consider_pruned_trials: If this is :obj:`True`, the PRUNED trials are considered for sampling. .. note:: Added in v2.0.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v2.0.0. .. note:: It is suggested to set this flag :obj:`False` when the :class:`~optuna.pruners.MedianPruner` is used. On the other hand, it is suggested to set this flag :obj:`True` when the :class:`~optuna.pruners.HyperbandPruner` is used. Please see `the benchmark result `__ for the details. use_separable_cma: If this is :obj:`True`, the covariance matrix is constrained to be diagonal. Due to reduce the model complexity, the learning rate for the covariance matrix is increased. Consequently, this algorithm outperforms CMA-ES on separable functions. .. note:: Added in v2.6.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v2.6.0. with_margin: If this is :obj:`True`, CMA-ES with margin is used. This algorithm prevents samples in each discrete distribution (:class:`~optuna.distributions.FloatDistribution` with `step` and :class:`~optuna.distributions.IntDistribution`) from being fixed to a single point. Currently, this option cannot be used with ``use_separable_cma=True``. .. note:: Added in v3.1.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v3.1.0. lr_adapt: If this is :obj:`True`, CMA-ES with learning rate adaptation is used. This algorithm focuses on working well on multimodal and/or noisy problems with default settings. Currently, this option cannot be used with ``use_separable_cma=True`` or ``with_margin=True``. .. note:: Added in v3.3.0 or later, as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v3.3.0. source_trials: This option is for Warm Starting CMA-ES, a method to transfer prior knowledge on similar HPO tasks through the initialization of CMA-ES. This method estimates a promising distribution from ``source_trials`` and generates the parameter of multivariate gaussian distribution. Please note that it is prohibited to use ``x0``, ``sigma0``, or ``use_separable_cma`` argument together. .. note:: Added in v2.6.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v2.6.0. """ # NOQA: E501 def __init__( self, x0: dict[str, Any] | None = None, sigma0: float | None = None, n_startup_trials: int = 1, independent_sampler: BaseSampler | None = None, warn_independent_sampling: bool = True, seed: int | None = None, *, consider_pruned_trials: bool = False, restart_strategy: str | None = None, popsize: int | None = None, inc_popsize: int = 2, use_separable_cma: bool = False, with_margin: bool = False, lr_adapt: bool = False, source_trials: list[FrozenTrial] | None = None, ) -> None: self._x0 = x0 self._sigma0 = sigma0 self._independent_sampler = independent_sampler or optuna.samplers.RandomSampler(seed=seed) self._n_startup_trials = n_startup_trials self._warn_independent_sampling = warn_independent_sampling self._cma_rng = LazyRandomState(seed) self._search_space = IntersectionSearchSpace() self._consider_pruned_trials = consider_pruned_trials self._restart_strategy = restart_strategy self._initial_popsize = popsize self._inc_popsize = inc_popsize self._use_separable_cma = use_separable_cma self._with_margin = with_margin self._lr_adapt = lr_adapt self._source_trials = source_trials if self._restart_strategy: warn_experimental_argument("restart_strategy") if self._consider_pruned_trials: warn_experimental_argument("consider_pruned_trials") if self._use_separable_cma: warn_experimental_argument("use_separable_cma") if self._source_trials is not None: warn_experimental_argument("source_trials") if self._with_margin: warn_experimental_argument("with_margin") if self._lr_adapt: warn_experimental_argument("lr_adapt") if source_trials is not None and (x0 is not None or sigma0 is not None): raise ValueError( "It is prohibited to pass `source_trials` argument when " "x0 or sigma0 is specified." ) # TODO(c-bata): Support WS-sep-CMA-ES. if source_trials is not None and use_separable_cma: raise ValueError( "It is prohibited to pass `source_trials` argument when using separable CMA-ES." ) if lr_adapt and (use_separable_cma or with_margin): raise ValueError( "It is prohibited to pass `use_separable_cma` or `with_margin` argument when " "using `lr_adapt`." ) if restart_strategy not in ( "ipop", "bipop", None, ): raise ValueError( "restart_strategy={} is unsupported. " "Please specify: 'ipop', 'bipop', or None.".format(restart_strategy) ) # TODO(knshnb): Support sep-CMA-ES with margin. if self._use_separable_cma and self._with_margin: raise ValueError( "Currently, we do not support `use_separable_cma=True` and `with_margin=True`." ) def reseed_rng(self) -> None: # _cma_rng doesn't require reseeding because the relative sampling reseeds in each trial. self._independent_sampler.reseed_rng() def infer_relative_search_space( self, study: "optuna.Study", trial: "optuna.trial.FrozenTrial" ) -> dict[str, BaseDistribution]: search_space: dict[str, BaseDistribution] = {} for name, distribution in self._search_space.calculate(study).items(): if distribution.single(): # `cma` cannot handle distributions that contain just a single value, so we skip # them. Note that the parameter values for such distributions are sampled in # `Trial`. continue if not isinstance(distribution, (FloatDistribution, IntDistribution)): # Categorical distribution is unsupported. continue search_space[name] = distribution return search_space def sample_relative( self, study: "optuna.Study", trial: "optuna.trial.FrozenTrial", search_space: dict[str, BaseDistribution], ) -> dict[str, Any]: self._raise_error_if_multi_objective(study) if len(search_space) == 0: return {} completed_trials = self._get_trials(study) if len(completed_trials) < self._n_startup_trials: return {} if len(search_space) == 1: if self._warn_independent_sampling: _logger.warning( "`CmaEsSampler` only supports two or more dimensional continuous " "search space. `{}` is used instead of `CmaEsSampler`.".format( self._independent_sampler.__class__.__name__ ) ) self._warn_independent_sampling = False return {} # When `with_margin=True`, bounds in discrete dimensions are handled inside `CMAwM`. trans = _SearchSpaceTransform( search_space, transform_step=not self._with_margin, transform_0_1=True ) if self._initial_popsize is None: self._initial_popsize = 4 + math.floor(3 * math.log(len(trans.bounds))) popsize: int = self._initial_popsize n_restarts: int = 0 n_restarts_with_large: int = 0 poptype: str = "small" small_n_eval: int = 0 large_n_eval: int = 0 if len(completed_trials) != 0: latest_trial = completed_trials[-1] popsize_attr_key = self._attr_keys.popsize() if popsize_attr_key in latest_trial.system_attrs: popsize = latest_trial.system_attrs[popsize_attr_key] else: popsize = self._initial_popsize n_restarts_attr_key = self._attr_keys.n_restarts() n_restarts = latest_trial.system_attrs.get(n_restarts_attr_key, 0) n_restarts_with_large = latest_trial.system_attrs.get( self._attr_keys.n_restarts_with_large, 0 ) poptype = latest_trial.system_attrs.get(self._attr_keys.poptype, "small") small_n_eval = latest_trial.system_attrs.get(self._attr_keys.small_n_eval, 0) large_n_eval = latest_trial.system_attrs.get(self._attr_keys.large_n_eval, 0) optimizer = self._restore_optimizer(completed_trials, n_restarts) if optimizer is None: optimizer = self._init_optimizer( trans, study.direction, population_size=self._initial_popsize ) if optimizer.dim != len(trans.bounds): if self._warn_independent_sampling: _logger.warning( "`CmaEsSampler` does not support dynamic search space. " "`{}` is used instead of `CmaEsSampler`.".format( self._independent_sampler.__class__.__name__ ) ) self._warn_independent_sampling = False return {} # TODO(c-bata): Reduce the number of wasted trials during parallel optimization. # See https://github.com/optuna/optuna/pull/920#discussion_r385114002 for details. solution_trials = self._get_solution_trials( completed_trials, optimizer.generation, n_restarts ) if len(solution_trials) >= popsize: solutions: list[tuple[np.ndarray, float]] = [] for t in solution_trials[:popsize]: assert t.value is not None, "completed trials must have a value" if isinstance(optimizer, cmaes.CMAwM): x = np.array(t.system_attrs["x_for_tell"]) else: x = trans.transform(t.params) y = t.value if study.direction == StudyDirection.MINIMIZE else -t.value solutions.append((x, y)) optimizer.tell(solutions) if self._restart_strategy == "ipop" and optimizer.should_stop(): n_restarts += 1 popsize = popsize * self._inc_popsize optimizer = self._init_optimizer( trans, study.direction, population_size=popsize, randomize_start_point=True ) if self._restart_strategy == "bipop" and optimizer.should_stop(): n_restarts += 1 n_eval = popsize * optimizer.generation if poptype == "small": small_n_eval += n_eval else: # poptype == "large" large_n_eval += n_eval if small_n_eval < large_n_eval: poptype = "small" popsize_multiplier = self._inc_popsize**n_restarts_with_large popsize = math.floor( self._initial_popsize * popsize_multiplier ** (self._cma_rng.rng.uniform() ** 2) ) else: poptype = "large" n_restarts_with_large += 1 popsize = self._initial_popsize * (self._inc_popsize**n_restarts_with_large) optimizer = self._init_optimizer( trans, study.direction, population_size=popsize, randomize_start_point=True ) # Store optimizer. optimizer_str = pickle.dumps(optimizer).hex() optimizer_attrs = self._split_optimizer_str(optimizer_str, n_restarts) for key in optimizer_attrs: study._storage.set_trial_system_attr(trial._trial_id, key, optimizer_attrs[key]) # Caution: optimizer should update its seed value. seed = self._cma_rng.rng.randint(1, 2**16) + trial.number optimizer._rng.seed(seed) if isinstance(optimizer, cmaes.CMAwM): params, x_for_tell = optimizer.ask() study._storage.set_trial_system_attr( trial._trial_id, "x_for_tell", x_for_tell.tolist() ) else: params = optimizer.ask() generation_attr_key = self._attr_keys.generation(n_restarts) study._storage.set_trial_system_attr( trial._trial_id, generation_attr_key, optimizer.generation ) popsize_attr_key = self._attr_keys.popsize() study._storage.set_trial_system_attr(trial._trial_id, popsize_attr_key, popsize) n_restarts_attr_key = self._attr_keys.n_restarts() study._storage.set_trial_system_attr(trial._trial_id, n_restarts_attr_key, n_restarts) study._storage.set_trial_system_attr( trial._trial_id, self._attr_keys.n_restarts_with_large, n_restarts_with_large ) study._storage.set_trial_system_attr(trial._trial_id, self._attr_keys.poptype, poptype) study._storage.set_trial_system_attr( trial._trial_id, self._attr_keys.small_n_eval, small_n_eval ) study._storage.set_trial_system_attr( trial._trial_id, self._attr_keys.large_n_eval, large_n_eval ) external_values = trans.untransform(params) return external_values @property def _attr_keys(self) -> _CmaEsAttrKeys: if self._use_separable_cma: attr_prefix = "sepcma:" elif self._with_margin: attr_prefix = "cmawm:" else: attr_prefix = "cma:" def optimizer_key_template(restart: int) -> str: if self._restart_strategy is None: return attr_prefix + "optimizer" else: return attr_prefix + "{}:restart_{}:optimizer".format( self._restart_strategy, restart ) def generation_attr_key_template(restart: int) -> str: if self._restart_strategy is None: return attr_prefix + "generation" else: return attr_prefix + "{}:restart_{}:generation".format( self._restart_strategy, restart ) def popsize_attr_key_template() -> str: if self._restart_strategy is None: return attr_prefix + "popsize" else: return attr_prefix + "{}:popsize".format(self._restart_strategy) def n_restarts_attr_key_template() -> str: if self._restart_strategy is None: return attr_prefix + "n_restarts" else: return attr_prefix + "{}:n_restarts".format(self._restart_strategy) return _CmaEsAttrKeys( optimizer_key_template, generation_attr_key_template, popsize_attr_key_template, n_restarts_attr_key_template, attr_prefix + "n_restarts_with_large", attr_prefix + "poptype", attr_prefix + "small_n_eval", attr_prefix + "large_n_eval", ) def _concat_optimizer_attrs(self, optimizer_attrs: dict[str, str], n_restarts: int = 0) -> str: return "".join( optimizer_attrs["{}:{}".format(self._attr_keys.optimizer(n_restarts), i)] for i in range(len(optimizer_attrs)) ) def _split_optimizer_str(self, optimizer_str: str, n_restarts: int = 0) -> dict[str, str]: optimizer_len = len(optimizer_str) attrs = {} for i in range(math.ceil(optimizer_len / _SYSTEM_ATTR_MAX_LENGTH)): start = i * _SYSTEM_ATTR_MAX_LENGTH end = min((i + 1) * _SYSTEM_ATTR_MAX_LENGTH, optimizer_len) attrs["{}:{}".format(self._attr_keys.optimizer(n_restarts), i)] = optimizer_str[ start:end ] return attrs def _restore_optimizer( self, completed_trials: "list[optuna.trial.FrozenTrial]", n_restarts: int = 0, ) -> "CmaClass" | None: # Restore a previous CMA object. for trial in reversed(completed_trials): optimizer_attrs = { key: value for key, value in trial.system_attrs.items() if key.startswith(self._attr_keys.optimizer(n_restarts)) } if len(optimizer_attrs) == 0: continue optimizer_str = self._concat_optimizer_attrs(optimizer_attrs, n_restarts) return pickle.loads(bytes.fromhex(optimizer_str)) return None def _init_optimizer( self, trans: _SearchSpaceTransform, direction: StudyDirection, population_size: int | None = None, randomize_start_point: bool = False, ) -> "CmaClass": lower_bounds = trans.bounds[:, 0] upper_bounds = trans.bounds[:, 1] n_dimension = len(trans.bounds) if self._source_trials is None: if randomize_start_point: mean = lower_bounds + (upper_bounds - lower_bounds) * self._cma_rng.rng.rand( n_dimension ) elif self._x0 is None: mean = lower_bounds + (upper_bounds - lower_bounds) / 2 else: # `self._x0` is external representations. mean = trans.transform(self._x0) if self._sigma0 is None: sigma0 = np.min((upper_bounds - lower_bounds) / 6) else: sigma0 = self._sigma0 cov = None else: expected_states = [TrialState.COMPLETE] if self._consider_pruned_trials: expected_states.append(TrialState.PRUNED) # TODO(c-bata): Filter parameters by their values instead of checking search space. sign = 1 if direction == StudyDirection.MINIMIZE else -1 source_solutions = [ (trans.transform(t.params), sign * cast(float, t.value)) for t in self._source_trials if t.state in expected_states and _is_compatible_search_space(trans, t.distributions) ] if len(source_solutions) == 0: raise ValueError("No compatible source_trials") # TODO(c-bata): Add options to change prior parameters (alpha and gamma). mean, sigma0, cov = cmaes.get_warm_start_mgd(source_solutions) # Avoid ZeroDivisionError in cmaes. sigma0 = max(sigma0, _EPS) if self._use_separable_cma: return cmaes.SepCMA( mean=mean, sigma=sigma0, bounds=trans.bounds, seed=self._cma_rng.rng.randint(1, 2**31 - 2), n_max_resampling=10 * n_dimension, population_size=population_size, ) if self._with_margin: steps = np.empty(len(trans._search_space), dtype=float) for i, dist in enumerate(trans._search_space.values()): assert isinstance(dist, (IntDistribution, FloatDistribution)) # Set step 0.0 for continuous search space. if dist.step is None or dist.log: steps[i] = 0.0 elif dist.low == dist.high: steps[i] = 1.0 else: steps[i] = dist.step / (dist.high - dist.low) return cmaes.CMAwM( mean=mean, sigma=sigma0, bounds=trans.bounds, steps=steps, cov=cov, seed=self._cma_rng.rng.randint(1, 2**31 - 2), n_max_resampling=10 * n_dimension, population_size=population_size, ) return cmaes.CMA( mean=mean, sigma=sigma0, cov=cov, bounds=trans.bounds, seed=self._cma_rng.rng.randint(1, 2**31 - 2), n_max_resampling=10 * n_dimension, population_size=population_size, lr_adapt=self._lr_adapt, ) def sample_independent( self, study: "optuna.Study", trial: "optuna.trial.FrozenTrial", param_name: str, param_distribution: BaseDistribution, ) -> Any: self._raise_error_if_multi_objective(study) if self._warn_independent_sampling: complete_trials = self._get_trials(study) if len(complete_trials) >= self._n_startup_trials: self._log_independent_sampling(trial, param_name) return self._independent_sampler.sample_independent( study, trial, param_name, param_distribution ) def _log_independent_sampling(self, trial: FrozenTrial, param_name: str) -> None: _logger.warning( "The parameter '{}' in trial#{} is sampled independently " "by using `{}` instead of `CmaEsSampler` " "(optimization performance may be degraded). " "`CmaEsSampler` does not support dynamic search space or `CategoricalDistribution`. " "You can suppress this warning by setting `warn_independent_sampling` " "to `False` in the constructor of `CmaEsSampler`, " "if this independent sampling is intended behavior.".format( param_name, trial.number, self._independent_sampler.__class__.__name__ ) ) def _get_trials(self, study: "optuna.Study") -> list[FrozenTrial]: complete_trials = [] for t in study._get_trials(deepcopy=False, use_cache=True): if t.state == TrialState.COMPLETE: complete_trials.append(t) elif ( t.state == TrialState.PRUNED and len(t.intermediate_values) > 0 and self._consider_pruned_trials ): _, value = max(t.intermediate_values.items()) if value is None: continue # We rewrite the value of the trial `t` for sampling, so we need a deepcopy. copied_t = copy.deepcopy(t) copied_t.value = value complete_trials.append(copied_t) return complete_trials def _get_solution_trials( self, trials: list[FrozenTrial], generation: int, n_restarts: int ) -> list[FrozenTrial]: generation_attr_key = self._attr_keys.generation(n_restarts) return [t for t in trials if generation == t.system_attrs.get(generation_attr_key, -1)] def before_trial(self, study: optuna.Study, trial: FrozenTrial) -> None: self._independent_sampler.before_trial(study, trial) def after_trial( self, study: "optuna.Study", trial: "optuna.trial.FrozenTrial", state: TrialState, values: Sequence[float] | None, ) -> None: self._independent_sampler.after_trial(study, trial, state, values) def _is_compatible_search_space( trans: _SearchSpaceTransform, search_space: dict[str, BaseDistribution] ) -> bool: intersection_size = len(set(trans._search_space.keys()).intersection(search_space.keys())) return intersection_size == len(trans._search_space) == len(search_space) optuna-4.1.0/optuna/samplers/_gp/000077500000000000000000000000001471332314300167245ustar00rootroot00000000000000optuna-4.1.0/optuna/samplers/_gp/__init__.py000066400000000000000000000001141471332314300210310ustar00rootroot00000000000000from optuna.samplers._gp.sampler import GPSampler __all__ = ["GPSampler"] optuna-4.1.0/optuna/samplers/_gp/sampler.py000066400000000000000000000203541471332314300207450ustar00rootroot00000000000000from __future__ import annotations from typing import Any from typing import Callable from typing import cast from typing import Sequence from typing import TYPE_CHECKING import warnings import numpy as np import optuna from optuna._experimental import experimental_class from optuna.distributions import BaseDistribution from optuna.samplers._base import BaseSampler from optuna.samplers._lazy_random_state import LazyRandomState from optuna.study import StudyDirection from optuna.trial import FrozenTrial from optuna.trial import TrialState if TYPE_CHECKING: import torch import optuna._gp.acqf as acqf import optuna._gp.gp as gp import optuna._gp.optim_mixed as optim_mixed import optuna._gp.prior as prior import optuna._gp.search_space as gp_search_space from optuna.study import Study else: from optuna._imports import _LazyImport torch = _LazyImport("torch") gp_search_space = _LazyImport("optuna._gp.search_space") gp = _LazyImport("optuna._gp.gp") optim_mixed = _LazyImport("optuna._gp.optim_mixed") acqf = _LazyImport("optuna._gp.acqf") prior = _LazyImport("optuna._gp.prior") @experimental_class("3.6.0") class GPSampler(BaseSampler): """Sampler using Gaussian process-based Bayesian optimization. This sampler fits a Gaussian process (GP) to the objective function and optimizes the acquisition function to suggest the next parameters. The current implementation uses: - Matern kernel with nu=2.5 (twice differentiable), - Automatic relevance determination (ARD) for the length scale of each parameter, - Gamma prior for inverse squared lengthscales, kernel scale, and noise variance, - Log Expected Improvement (logEI) as the acquisition function, and - Quasi-Monte Carlo (QMC) sampling to optimize the acquisition function. .. note:: This sampler requires ``scipy`` and ``torch``. You can install these dependencies with ``pip install scipy torch``. Args: seed: Random seed to initialize internal random number generator. Defaults to :obj:`None` (a seed is picked randomly). independent_sampler: Sampler used for initial sampling (for the first ``n_startup_trials`` trials) and for conditional parameters. Defaults to :obj:`None` (a random sampler with the same ``seed`` is used). n_startup_trials: Number of initial trials. Defaults to 10. deterministic_objective: Whether the objective function is deterministic or not. If :obj:`True`, the sampler will fix the noise variance of the surrogate model to the minimum value (slightly above 0 to ensure numerical stability). Defaults to :obj:`False`. """ def __init__( self, *, seed: int | None = None, independent_sampler: BaseSampler | None = None, n_startup_trials: int = 10, deterministic_objective: bool = False, ) -> None: self._rng = LazyRandomState(seed) self._independent_sampler = independent_sampler or optuna.samplers.RandomSampler(seed=seed) self._intersection_search_space = optuna.search_space.IntersectionSearchSpace() self._n_startup_trials = n_startup_trials self._log_prior: "Callable[[gp.KernelParamsTensor], torch.Tensor]" = ( prior.default_log_prior ) self._minimum_noise: float = prior.DEFAULT_MINIMUM_NOISE_VAR # We cache the kernel parameters for initial values of fitting the next time. self._kernel_params_cache: "gp.KernelParamsTensor | None" = None self._optimize_n_samples: int = 2048 self._deterministic = deterministic_objective def reseed_rng(self) -> None: self._rng.rng.seed() self._independent_sampler.reseed_rng() def infer_relative_search_space( self, study: Study, trial: FrozenTrial ) -> dict[str, BaseDistribution]: search_space = {} for name, distribution in self._intersection_search_space.calculate(study).items(): if distribution.single(): continue search_space[name] = distribution return search_space def _optimize_acqf( self, acqf_params: "acqf.AcquisitionFunctionParams", best_params: np.ndarray, ) -> np.ndarray: # Advanced users can override this method to change the optimization algorithm. # However, we do not make any effort to keep backward compatibility between versions. # Particularly, we may remove this function in future refactoring. normalized_params, _acqf_val = optim_mixed.optimize_acqf_mixed( acqf_params, warmstart_normalized_params_array=best_params[None, :], n_preliminary_samples=2048, n_local_search=10, tol=1e-4, rng=self._rng.rng, ) return normalized_params def sample_relative( self, study: Study, trial: FrozenTrial, search_space: dict[str, BaseDistribution] ) -> dict[str, Any]: self._raise_error_if_multi_objective(study) if search_space == {}: return {} states = (TrialState.COMPLETE,) trials = study._get_trials(deepcopy=False, states=states, use_cache=True) if len(trials) < self._n_startup_trials: return {} ( internal_search_space, normalized_params, ) = gp_search_space.get_search_space_and_normalized_params(trials, search_space) _sign = -1.0 if study.direction == StudyDirection.MINIMIZE else 1.0 score_vals = np.array([_sign * cast(float, trial.value) for trial in trials]) if np.any(~np.isfinite(score_vals)): warnings.warn( "GPSampler cannot handle infinite values. " "We clamp those values to worst/best finite value." ) finite_score_vals = score_vals[np.isfinite(score_vals)] best_finite_score = np.max(finite_score_vals, initial=0.0) worst_finite_score = np.min(finite_score_vals, initial=0.0) score_vals = np.clip(score_vals, worst_finite_score, best_finite_score) standarized_score_vals = (score_vals - score_vals.mean()) / max(1e-10, score_vals.std()) if self._kernel_params_cache is not None and len( self._kernel_params_cache.inverse_squared_lengthscales ) != len(internal_search_space.scale_types): # Clear cache if the search space changes. self._kernel_params_cache = None kernel_params = gp.fit_kernel_params( X=normalized_params, Y=standarized_score_vals, is_categorical=( internal_search_space.scale_types == gp_search_space.ScaleType.CATEGORICAL ), log_prior=self._log_prior, minimum_noise=self._minimum_noise, initial_kernel_params=self._kernel_params_cache, deterministic_objective=self._deterministic, ) self._kernel_params_cache = kernel_params acqf_params = acqf.create_acqf_params( acqf_type=acqf.AcquisitionFunctionType.LOG_EI, kernel_params=kernel_params, search_space=internal_search_space, X=normalized_params, Y=standarized_score_vals, ) normalized_param = self._optimize_acqf( acqf_params, normalized_params[np.argmax(standarized_score_vals), :] ) return gp_search_space.get_unnormalized_param(search_space, normalized_param) def sample_independent( self, study: Study, trial: FrozenTrial, param_name: str, param_distribution: BaseDistribution, ) -> Any: self._raise_error_if_multi_objective(study) return self._independent_sampler.sample_independent( study, trial, param_name, param_distribution ) def before_trial(self, study: Study, trial: FrozenTrial) -> None: self._independent_sampler.before_trial(study, trial) def after_trial( self, study: Study, trial: FrozenTrial, state: TrialState, values: Sequence[float] | None, ) -> None: self._independent_sampler.after_trial(study, trial, state, values) optuna-4.1.0/optuna/samplers/_grid.py000066400000000000000000000266051471332314300176260ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Mapping from collections.abc import Sequence import itertools from numbers import Real from typing import Any from typing import TYPE_CHECKING from typing import Union import warnings import numpy as np from optuna.distributions import BaseDistribution from optuna.logging import get_logger from optuna.samplers import BaseSampler from optuna.samplers._lazy_random_state import LazyRandomState from optuna.trial import FrozenTrial from optuna.trial import TrialState if TYPE_CHECKING: from optuna.study import Study GridValueType = Union[str, float, int, bool, None] _logger = get_logger(__name__) class GridSampler(BaseSampler): """Sampler using grid search. With :class:`~optuna.samplers.GridSampler`, the trials suggest all combinations of parameters in the given search space during the study. Example: .. testcode:: import optuna def objective(trial): x = trial.suggest_float("x", -100, 100) y = trial.suggest_int("y", -100, 100) return x**2 + y**2 search_space = {"x": [-50, 0, 50], "y": [-99, 0, 99]} study = optuna.create_study(sampler=optuna.samplers.GridSampler(search_space)) study.optimize(objective) Note: This sampler with :ref:`ask_and_tell` raises :exc:`RuntimeError` just after evaluating the final grid. This is because :class:`~optuna.samplers.GridSampler` automatically stops the optimization if all combinations in the passed ``search_space`` have already been evaluated, internally invoking the :func:`~optuna.study.Study.stop` method. As a workaround, we need to handle the error manually as in https://github.com/optuna/optuna/issues/4121#issuecomment-1305289910. Note: :class:`~optuna.samplers.GridSampler` does not take care of a parameter's quantization specified by discrete suggest methods but just samples one of values specified in the search space. E.g., in the following code snippet, either of ``-0.5`` or ``0.5`` is sampled as ``x`` instead of an integer point. .. testcode:: import optuna def objective(trial): # The following suggest method specifies integer points between -5 and 5. x = trial.suggest_float("x", -5, 5, step=1) return x**2 # Non-int points are specified in the grid. search_space = {"x": [-0.5, 0.5]} study = optuna.create_study(sampler=optuna.samplers.GridSampler(search_space)) study.optimize(objective, n_trials=2) Note: A parameter configuration in the grid is not considered finished until its trial is finished. Therefore, during distributed optimization where trials run concurrently, different workers will occasionally suggest the same parameter configuration. The total number of actual trials may therefore exceed the size of the grid. Note: All parameters must be specified when using :class:`~optuna.samplers.GridSampler` with :meth:`~optuna.study.Study.enqueue_trial`. Args: search_space: A dictionary whose key and value are a parameter name and the corresponding candidates of values, respectively. seed: A seed to fix the order of trials as the grid is randomly shuffled. This shuffle is beneficial when the number of grids is larger than ``n_trials`` in :meth:`~optuna.Study.optimize` to suppress suggesting similar grids. Please note that fixing ``seed`` for each process is strongly recommended in distributed optimization to avoid duplicated suggestions. """ def __init__( self, search_space: Mapping[str, Sequence[GridValueType]], seed: int | None = None ) -> None: for param_name, param_values in search_space.items(): for value in param_values: self._check_value(param_name, value) self._search_space = {} for param_name, param_values in sorted(search_space.items()): self._search_space[param_name] = list(param_values) self._all_grids = list(itertools.product(*self._search_space.values())) self._param_names = sorted(search_space.keys()) self._n_min_trials = len(self._all_grids) self._rng = LazyRandomState(seed or 0) self._rng.rng.shuffle(self._all_grids) # type: ignore[arg-type] def reseed_rng(self) -> None: self._rng.rng.seed() def before_trial(self, study: Study, trial: FrozenTrial) -> None: # Instead of returning param values, GridSampler puts the target grid id as a system attr, # and the values are returned from `sample_independent`. This is because the distribution # object is hard to get at the beginning of trial, while we need the access to the object # to validate the sampled value. # When the trial is created by RetryFailedTrialCallback or enqueue_trial, we should not # assign a new grid_id. if "grid_id" in trial.system_attrs or "fixed_params" in trial.system_attrs: return if 0 <= trial.number and trial.number < self._n_min_trials: study._storage.set_trial_system_attr( trial._trial_id, "search_space", self._search_space ) study._storage.set_trial_system_attr(trial._trial_id, "grid_id", trial.number) return target_grids = self._get_unvisited_grid_ids(study) if len(target_grids) == 0: # This case may occur with distributed optimization or trial queue. If there is no # target grid, `GridSampler` evaluates a visited, duplicated point with the current # trial. After that, the optimization stops. _logger.warning( "`GridSampler` is re-evaluating a configuration because the grid has been " "exhausted. This may happen due to a timing issue during distributed optimization " "or when re-running optimizations on already finished studies." ) # One of all grids is randomly picked up in this case. target_grids = list(range(len(self._all_grids))) # In distributed optimization, multiple workers may simultaneously pick up the same grid. # To make the conflict less frequent, the grid is chosen randomly. grid_id = int(self._rng.rng.choice(target_grids)) study._storage.set_trial_system_attr(trial._trial_id, "search_space", self._search_space) study._storage.set_trial_system_attr(trial._trial_id, "grid_id", grid_id) def infer_relative_search_space( self, study: Study, trial: FrozenTrial ) -> dict[str, BaseDistribution]: return {} def sample_relative( self, study: Study, trial: FrozenTrial, search_space: dict[str, BaseDistribution] ) -> dict[str, Any]: return {} def sample_independent( self, study: Study, trial: FrozenTrial, param_name: str, param_distribution: BaseDistribution, ) -> Any: if "grid_id" not in trial.system_attrs: message = "All parameters must be specified when using GridSampler with enqueue_trial." raise ValueError(message) if param_name not in self._search_space: message = "The parameter name, {}, is not found in the given grid.".format(param_name) raise ValueError(message) grid_id = trial.system_attrs["grid_id"] param_value = self._all_grids[grid_id][self._param_names.index(param_name)] contains = param_distribution._contains(param_distribution.to_internal_repr(param_value)) if not contains: warnings.warn( f"The value `{param_value}` is out of range of the parameter `{param_name}`. " f"The value will be used but the actual distribution is: `{param_distribution}`." ) return param_value def after_trial( self, study: Study, trial: FrozenTrial, state: TrialState, values: Sequence[float] | None, ) -> None: target_grids = self._get_unvisited_grid_ids(study) if len(target_grids) == 0: study.stop() elif len(target_grids) == 1: grid_id = study._storage.get_trial_system_attrs(trial._trial_id)["grid_id"] if grid_id == target_grids[0]: study.stop() @staticmethod def _check_value(param_name: str, param_value: Any) -> None: if param_value is None or isinstance(param_value, (str, int, float, bool)): return message = ( "{} contains a value with the type of {}, which is not supported by " "`GridSampler`. Please make sure a value is `str`, `int`, `float`, `bool`" " or `None` for persistent storage.".format(param_name, type(param_value)) ) warnings.warn(message) def _get_unvisited_grid_ids(self, study: Study) -> list[int]: # List up unvisited grids based on already finished ones. visited_grids = [] running_grids = [] # We directly query the storage to get trials here instead of `study.get_trials`, # since some pruners such as `HyperbandPruner` use the study transformed # to filter trials. See https://github.com/optuna/optuna/issues/2327 for details. trials = study._storage.get_all_trials(study._study_id, deepcopy=False) for t in trials: if "grid_id" in t.system_attrs and self._same_search_space( t.system_attrs["search_space"] ): if t.state.is_finished(): visited_grids.append(t.system_attrs["grid_id"]) elif t.state == TrialState.RUNNING: running_grids.append(t.system_attrs["grid_id"]) unvisited_grids = set(range(self._n_min_trials)) - set(visited_grids) - set(running_grids) # If evaluations for all grids have been started, return grids that have not yet finished # because all grids should be evaluated before stopping the optimization. if len(unvisited_grids) == 0: unvisited_grids = set(range(self._n_min_trials)) - set(visited_grids) return list(unvisited_grids) @staticmethod def _grid_value_equal(value1: GridValueType, value2: GridValueType) -> bool: value1_is_nan = isinstance(value1, Real) and np.isnan(float(value1)) value2_is_nan = isinstance(value2, Real) and np.isnan(float(value2)) return (value1 == value2) or (value1_is_nan and value2_is_nan) def _same_search_space(self, search_space: Mapping[str, Sequence[GridValueType]]) -> bool: if set(search_space.keys()) != set(self._search_space.keys()): return False for param_name in search_space.keys(): if len(search_space[param_name]) != len(self._search_space[param_name]): return False for i, param_value in enumerate(search_space[param_name]): if not self._grid_value_equal(param_value, self._search_space[param_name][i]): return False return True def is_exhausted(self, study: Study) -> bool: """ Return True if all the possible params are evaluated, otherwise return False. """ return len(self._get_unvisited_grid_ids(study)) == 0 optuna-4.1.0/optuna/samplers/_lazy_random_state.py000066400000000000000000000013311471332314300224050ustar00rootroot00000000000000from __future__ import annotations import numpy as np class LazyRandomState: """Lazy Random State class. This is a class to initialize the random state just before use to prevent duplication of the same random state when deepcopy is applied to the instance of sampler. """ def __init__(self, seed: int | None = None) -> None: self._rng: np.random.RandomState | None = None if seed is not None: self.rng.seed(seed=seed) def _set_rng(self) -> None: self._rng = np.random.RandomState() @property def rng(self) -> np.random.RandomState: if self._rng is None: self._set_rng() assert self._rng is not None return self._rng optuna-4.1.0/optuna/samplers/_nsgaiii/000077500000000000000000000000001471332314300177415ustar00rootroot00000000000000optuna-4.1.0/optuna/samplers/_nsgaiii/__init__.py000066400000000000000000000000001471332314300220400ustar00rootroot00000000000000optuna-4.1.0/optuna/samplers/_nsgaiii/_elite_population_selection_strategy.py000066400000000000000000000341701471332314300300220ustar00rootroot00000000000000from __future__ import annotations from collections import defaultdict from collections.abc import Callable from collections.abc import Sequence import itertools import math from typing import TYPE_CHECKING import numpy as np from optuna.samplers._lazy_random_state import LazyRandomState from optuna.samplers.nsgaii._constraints_evaluation import _validate_constraints from optuna.samplers.nsgaii._elite_population_selection_strategy import _rank_population from optuna.trial import FrozenTrial if TYPE_CHECKING: from optuna.study import Study # Define a coefficient for scaling intervals, used in _filter_inf() to replace +-inf. _COEF = 3 class NSGAIIIElitePopulationSelectionStrategy: def __init__( self, *, population_size: int, constraints_func: Callable[[FrozenTrial], Sequence[float]] | None = None, reference_points: np.ndarray | None = None, dividing_parameter: int = 3, rng: LazyRandomState, ) -> None: if population_size < 2: raise ValueError("`population_size` must be greater than or equal to 2.") self._population_size = population_size self._constraints_func = constraints_func self._reference_points = reference_points self._dividing_parameter = dividing_parameter self._rng = rng def __call__(self, study: Study, population: list[FrozenTrial]) -> list[FrozenTrial]: """Select elite population from the given trials by NSGA-III algorithm. Args: study: Target study object. population: Trials in the study. Returns: A list of trials that are selected as elite population. """ _validate_constraints(population, is_constrained=self._constraints_func is not None) population_per_rank = _rank_population( population, study.directions, is_constrained=self._constraints_func is not None ) elite_population: list[FrozenTrial] = [] for population in population_per_rank: if len(elite_population) + len(population) < self._population_size: elite_population.extend(population) else: n_objectives = len(study.directions) # Construct reference points in the first run. if self._reference_points is None: self._reference_points = _generate_default_reference_point( n_objectives, self._dividing_parameter ) elif np.shape(self._reference_points)[1] != n_objectives: raise ValueError( "The dimension of reference points vectors must be the same as the number " "of objectives of the study." ) # Normalize objective values after filtering +-inf. objective_matrix = _normalize_objective_values( _filter_inf(elite_population + population) ) ( closest_reference_points, distance_reference_points, ) = _associate_individuals_with_reference_points( objective_matrix, self._reference_points ) elite_population_num = len(elite_population) target_population_size = self._population_size - elite_population_num additional_elite_population = _preserve_niche_individuals( target_population_size, elite_population_num, population, closest_reference_points, distance_reference_points, self._rng.rng, ) elite_population.extend(additional_elite_population) break return elite_population def _multi_choose(n: int, k: int) -> int: return math.comb(n + k - 1, k) def _generate_default_reference_point( n_objectives: int, dividing_parameter: int = 3 ) -> np.ndarray: """Generates default reference points which are `uniformly` spread on a hyperplane.""" reference_points = np.zeros( ( _multi_choose(n_objectives, dividing_parameter), n_objectives, ) ) for i, comb in enumerate( itertools.combinations_with_replacement(range(n_objectives), dividing_parameter) ): for j in comb: reference_points[i, j] += 1.0 return reference_points def _filter_inf(population: list[FrozenTrial]) -> np.ndarray: # Collect all objective values. n_objectives = len(population[0].values) objective_matrix = np.zeros((len(population), n_objectives)) for i, trial in enumerate(population): objective_matrix[i] = np.array(trial.values, dtype=float) mask_posinf = np.isposinf(objective_matrix) mask_neginf = np.isneginf(objective_matrix) # Replace +-inf with nan temporary to get max and min. objective_matrix[mask_posinf + mask_neginf] = np.nan nadir_point = np.nanmax(objective_matrix, axis=0) ideal_point = np.nanmin(objective_matrix, axis=0) interval = nadir_point - ideal_point # TODO(Shinichi) reconsider alternative value for inf. rows_posinf, cols_posinf = np.where(mask_posinf) objective_matrix[rows_posinf, cols_posinf] = ( nadir_point[cols_posinf] + _COEF * interval[cols_posinf] ) rows_neginf, cols_neginf = np.where(mask_neginf) objective_matrix[rows_neginf, cols_neginf] = ( ideal_point[cols_neginf] - _COEF * interval[cols_neginf] ) return objective_matrix def _normalize_objective_values(objective_matrix: np.ndarray) -> np.ndarray: """Normalizes objective values of population. An ideal point z* consists of minimums in each axis. Each objective value of population is then subtracted by the ideal point. An extreme point of each axis is (originally) defined as a minimum solution of achievement scalarizing function from the population. After that, intercepts are calculate as intercepts of hyperplane which has all the extreme points on it and used to rescale objective values. We adopt weights and achievement scalarizing function(ASF) used in pre-print of the NSGA-III paper (See https://www.egr.msu.edu/~kdeb/papers/k2012009.pdf). """ n_objectives = np.shape(objective_matrix)[1] # Subtract ideal point from objective values. objective_matrix -= np.min(objective_matrix, axis=0) # Initialize weights. weights = np.eye(n_objectives) weights[weights == 0] = 1e6 # Calculate extreme points to normalize objective values. # TODO(Shinichi) Reimplement to reduce time complexity. asf_value = np.max( np.einsum("nm,dm->dnm", objective_matrix, weights), axis=2, ) extreme_points = objective_matrix[np.argmin(asf_value, axis=1), :] # Normalize objective_matrix with extreme points. # Note that extreme_points can be degenerate, but no proper operation is remarked in the # paper. Therefore, the maximum value of population in each axis is used in such cases. if np.all(np.isfinite(extreme_points)) and np.linalg.matrix_rank(extreme_points) == len( extreme_points ): intercepts_inv = np.linalg.solve(extreme_points, np.ones(n_objectives)) else: intercepts = np.max(objective_matrix, axis=0) intercepts_inv = 1 / np.where(intercepts == 0, 1, intercepts) objective_matrix *= np.where(np.isfinite(intercepts_inv), intercepts_inv, 1) return objective_matrix def _associate_individuals_with_reference_points( objective_matrix: np.ndarray, reference_points: np.ndarray ) -> tuple[np.ndarray, np.ndarray]: """Associates each objective value to the closest reference point. Associate each normalized objective value to the closest reference point. The distance is calculated by Euclidean norm. Args: objective_matrix: A 2 dimension ``numpy.ndarray`` with columns of objective dimension and rows of generation size. Each row is the normalized objective value of the corresponding individual. Returns: closest_reference_points: A ``numpy.ndarray`` with rows of generation size. Each row is the index of the closest reference point to the corresponding individual. distance_reference_points: A ``numpy.ndarray`` with rows of generation size. Each row is the distance from the corresponding individual to the closest reference point. """ # TODO(Shinichi) Implement faster assignment for the default reference points because it does # not seem necessary to calculate distance from all reference points. # TODO(Shinichi) Normalize reference_points in constructor to remove reference_point_norms. # In addition, the minimum distance from each reference point can be replaced with maximum # inner product between the given individual and each normalized reference points. # distance_from_reference_lines is a ndarray of shape (n, p), where n is the size of the # population and p is the number of reference points. Its (i,j) entry keeps distance between # the i-th individual values and the j-th reference line. reference_point_norm_squared = np.linalg.norm(reference_points, axis=1) ** 2 perpendicular_vectors_to_reference_lines = np.einsum( "ni,pi,p,pm->npm", objective_matrix, reference_points, 1 / reference_point_norm_squared, reference_points, ) distance_from_reference_lines = np.linalg.norm( objective_matrix[:, np.newaxis, :] - perpendicular_vectors_to_reference_lines, axis=2, ) closest_reference_points: np.ndarray = np.argmin(distance_from_reference_lines, axis=1) distance_reference_points: np.ndarray = np.min(distance_from_reference_lines, axis=1) return closest_reference_points, distance_reference_points def _preserve_niche_individuals( target_population_size: int, elite_population_num: int, population: list[FrozenTrial], closest_reference_points: np.ndarray, distance_reference_points: np.ndarray, rng: np.random.RandomState, ) -> list[FrozenTrial]: """Determine who survives form the borderline front. Who survive form the borderline front is determined according to the sparsity of each closest reference point. The algorithm picks a reference point from those who have the least neighbors in elite population and adds one of borderline front member who has the same closest reference point. Args: target_population_size: The number of individuals to select. elite_population_num: The number of individuals which are already selected as the elite population. population: List of all the trials in the current surviving generation. distance_reference_points: A ``numpy.ndarray`` with rows of generation size. Each row is the distance from the corresponding individual to the closest reference point. closest_reference_points: A ``numpy.ndarray`` with rows of generation size. Each row is the index of the closest reference point to the corresponding individual. rng: Random number generator. Returns: A list of trials which are selected as the next generation. """ if len(population) < target_population_size: raise ValueError( "The population size must be greater than or equal to the target population size." ) # reference_point_to_borderline_population keeps pairs of a neighbor and the distance of # each reference point from borderline front population. reference_point_to_borderline_population = defaultdict(list) for i, reference_point_idx in enumerate(closest_reference_points[elite_population_num:]): population_idx = i + elite_population_num reference_point_to_borderline_population[reference_point_idx].append( (distance_reference_points[population_idx], i) ) # reference_points_to_elite_population_count keeps how many elite neighbors each reference # point has. reference_point_to_elite_population_count: dict[int, int] = defaultdict(int) for i, reference_point_idx in enumerate(closest_reference_points[:elite_population_num]): reference_point_to_elite_population_count[reference_point_idx] += 1 # nearest_points_count_to_reference_points classifies reference points which have at least one # closest borderline population member by the number of elite neighbors they have. Each key # corresponds to the number of elite neighbors and the value to the reference point indices. nearest_points_count_to_reference_points = defaultdict(list) for reference_point_idx in reference_point_to_borderline_population: elite_population_count = reference_point_to_elite_population_count[reference_point_idx] nearest_points_count_to_reference_points[elite_population_count].append( reference_point_idx ) count = -1 additional_elite_population: list[FrozenTrial] = [] is_shuffled: defaultdict[int, bool] = defaultdict(bool) while len(additional_elite_population) < target_population_size: if len(nearest_points_count_to_reference_points[count]) == 0: count += 1 rng.shuffle(nearest_points_count_to_reference_points[count]) continue reference_point_idx = nearest_points_count_to_reference_points[count].pop() if count > 0 and not is_shuffled[reference_point_idx]: rng.shuffle(reference_point_to_borderline_population[reference_point_idx]) is_shuffled[reference_point_idx] = True elif count == 0: reference_point_to_borderline_population[reference_point_idx].sort(reverse=True) _, selected_individual_id = reference_point_to_borderline_population[ reference_point_idx ].pop() additional_elite_population.append(population[selected_individual_id]) if reference_point_to_borderline_population[reference_point_idx]: nearest_points_count_to_reference_points[count + 1].append(reference_point_idx) return additional_elite_population optuna-4.1.0/optuna/samplers/_nsgaiii/_sampler.py000066400000000000000000000303521471332314300221200ustar00rootroot00000000000000from __future__ import annotations from collections import defaultdict from collections.abc import Callable from collections.abc import Sequence import hashlib from typing import Any from typing import TYPE_CHECKING import numpy as np import optuna from optuna._experimental import experimental_class from optuna.distributions import BaseDistribution from optuna.samplers._base import BaseSampler from optuna.samplers._lazy_random_state import LazyRandomState from optuna.samplers._nsgaiii._elite_population_selection_strategy import ( NSGAIIIElitePopulationSelectionStrategy, ) from optuna.samplers._random import RandomSampler from optuna.samplers.nsgaii._after_trial_strategy import NSGAIIAfterTrialStrategy from optuna.samplers.nsgaii._child_generation_strategy import NSGAIIChildGenerationStrategy from optuna.samplers.nsgaii._crossovers._base import BaseCrossover from optuna.samplers.nsgaii._crossovers._uniform import UniformCrossover from optuna.search_space import IntersectionSearchSpace from optuna.trial import FrozenTrial from optuna.trial import TrialState if TYPE_CHECKING: from optuna.study import Study # Define key names of `Trial.system_attrs`. _GENERATION_KEY = "nsga3:generation" _POPULATION_CACHE_KEY_PREFIX = "nsga3:population" @experimental_class("3.2.0") class NSGAIIISampler(BaseSampler): """Multi-objective sampler using the NSGA-III algorithm. NSGA-III stands for "Nondominated Sorting Genetic Algorithm III", which is a modified version of NSGA-II for many objective optimization problem. For further information about NSGA-III, please refer to the following papers: - `An Evolutionary Many-Objective Optimization Algorithm Using Reference-Point-Based Nondominated Sorting Approach, Part I: Solving Problems With Box Constraints `__ - `An Evolutionary Many-Objective Optimization Algorithm Using Reference-Point-Based Nondominated Sorting Approach, Part II: Handling Constraints and Extending to an Adaptive Approach `__ Args: reference_points: A 2 dimension ``numpy.ndarray`` with objective dimension columns. Represents a list of reference points which is used to determine who to survive. After non-dominated sort, who out of borderline front are going to survived is determined according to how sparse the closest reference point of each individual is. In the default setting the algorithm uses `uniformly` spread points to diversify the result. It is also possible to reflect your `preferences` by giving an arbitrary set of `target` points since the algorithm prioritizes individuals around reference points. dividing_parameter: A parameter to determine the density of default reference points. This parameter determines how many divisions are made between reference points on each axis. The smaller this value is, the less reference points you have. The default value is 3. Note that this parameter is not used when ``reference_points`` is not :obj:`None`. .. note:: Other parameters than ``reference_points`` and ``dividing_parameter`` are the same as :class:`~optuna.samplers.NSGAIISampler`. """ def __init__( self, *, population_size: int = 50, mutation_prob: float | None = None, crossover: BaseCrossover | None = None, crossover_prob: float = 0.9, swapping_prob: float = 0.5, seed: int | None = None, constraints_func: Callable[[FrozenTrial], Sequence[float]] | None = None, reference_points: np.ndarray | None = None, dividing_parameter: int = 3, elite_population_selection_strategy: ( Callable[[Study, list[FrozenTrial]], list[FrozenTrial]] | None ) = None, child_generation_strategy: ( Callable[[Study, dict[str, BaseDistribution], list[FrozenTrial]], dict[str, Any]] | None ) = None, after_trial_strategy: ( Callable[[Study, FrozenTrial, TrialState, Sequence[float] | None], None] | None ) = None, ) -> None: # TODO(ohta): Reconsider the default value of each parameter. if population_size < 2: raise ValueError("`population_size` must be greater than or equal to 2.") if crossover is None: crossover = UniformCrossover(swapping_prob) if not isinstance(crossover, BaseCrossover): raise ValueError( f"'{crossover}' is not a valid crossover." " For valid crossovers see" " https://optuna.readthedocs.io/en/stable/reference/samplers.html." ) if population_size < crossover.n_parents: raise ValueError( f"Using {crossover}," f" the population size should be greater than or equal to {crossover.n_parents}." f" The specified `population_size` is {population_size}." ) self._population_size = population_size self._random_sampler = RandomSampler(seed=seed) self._rng = LazyRandomState(seed) self._constraints_func = constraints_func self._search_space = IntersectionSearchSpace() self._elite_population_selection_strategy = ( elite_population_selection_strategy or NSGAIIIElitePopulationSelectionStrategy( population_size=population_size, constraints_func=constraints_func, reference_points=reference_points, dividing_parameter=dividing_parameter, rng=self._rng, ) ) self._child_generation_strategy = ( child_generation_strategy or NSGAIIChildGenerationStrategy( crossover_prob=crossover_prob, mutation_prob=mutation_prob, swapping_prob=swapping_prob, crossover=crossover, constraints_func=constraints_func, rng=self._rng, ) ) self._after_trial_strategy = after_trial_strategy or NSGAIIAfterTrialStrategy( constraints_func=constraints_func ) def reseed_rng(self) -> None: self._random_sampler.reseed_rng() self._rng.rng.seed() def infer_relative_search_space( self, study: Study, trial: FrozenTrial ) -> dict[str, BaseDistribution]: search_space: dict[str, BaseDistribution] = {} for name, distribution in self._search_space.calculate(study).items(): if distribution.single(): # The `untransform` method of `optuna._transform._SearchSpaceTransform` # does not assume a single value, # so single value objects are not sampled with the `sample_relative` method, # but with the `sample_independent` method. continue search_space[name] = distribution return search_space def sample_relative( self, study: Study, trial: FrozenTrial, search_space: dict[str, BaseDistribution], ) -> dict[str, Any]: parent_generation, parent_population = self._collect_parent_population(study) generation = parent_generation + 1 study._storage.set_trial_system_attr(trial._trial_id, _GENERATION_KEY, generation) if parent_generation < 0: return {} return self._child_generation_strategy(study, search_space, parent_population) def sample_independent( self, study: Study, trial: FrozenTrial, param_name: str, param_distribution: BaseDistribution, ) -> Any: # Following parameters are randomly sampled here. # 1. A parameter in the initial population/first generation. # 2. A parameter to mutate. # 3. A parameter excluded from the intersection search space. return self._random_sampler.sample_independent( study, trial, param_name, param_distribution ) def _collect_parent_population(self, study: Study) -> tuple[int, list[FrozenTrial]]: trials = study.get_trials(deepcopy=False) generation_to_runnings = defaultdict(list) generation_to_population = defaultdict(list) for trial in trials: if _GENERATION_KEY not in trial.system_attrs: continue generation = trial.system_attrs[_GENERATION_KEY] if trial.state != optuna.trial.TrialState.COMPLETE: if trial.state == optuna.trial.TrialState.RUNNING: generation_to_runnings[generation].append(trial) continue # Do not use trials whose states are not COMPLETE, or `constraint` will be unavailable. generation_to_population[generation].append(trial) hasher = hashlib.sha256() parent_population: list[FrozenTrial] = [] parent_generation = -1 while True: generation = parent_generation + 1 population = generation_to_population[generation] # Under multi-worker settings, the population size might become larger than # `self._population_size`. if len(population) < self._population_size: break # [NOTE] # It's generally safe to assume that once the above condition is satisfied, # there are no additional individuals added to the generation (i.e., the members of # the generation have been fixed). # If the number of parallel workers is huge, this assumption can be broken, but # this is a very rare case and doesn't significantly impact optimization performance. # So we can ignore the case. # The cache key is calculated based on the key of the previous generation and # the remaining running trials in the current population. # If there are no running trials, the new cache key becomes exactly the same as # the previous one, and the cached content will be overwritten. This allows us to # skip redundant cache key calculations when this method is called for the subsequent # trials. for trial in generation_to_runnings[generation]: hasher.update(bytes(str(trial.number), "utf-8")) cache_key = "{}:{}".format(_POPULATION_CACHE_KEY_PREFIX, hasher.hexdigest()) study_system_attrs = study._storage.get_study_system_attrs(study._study_id) cached_generation, cached_population_numbers = study_system_attrs.get( cache_key, (-1, []) ) if cached_generation >= generation: generation = cached_generation population = [trials[n] for n in cached_population_numbers] else: population.extend(parent_population) population = self._elite_population_selection_strategy(study, population) # To reduce the number of system attribute entries, # we cache the population information only if there are no running trials # (i.e., the information of the population has been fixed). # Usually, if there are no too delayed running trials, the single entry # will be used. if len(generation_to_runnings[generation]) == 0: population_numbers = [t.number for t in population] study._storage.set_study_system_attr( study._study_id, cache_key, (generation, population_numbers) ) parent_generation = generation parent_population = population return parent_generation, parent_population def before_trial(self, study: Study, trial: FrozenTrial) -> None: self._random_sampler.before_trial(study, trial) def after_trial( self, study: Study, trial: FrozenTrial, state: TrialState, values: Sequence[float] | None, ) -> None: assert state in [TrialState.COMPLETE, TrialState.FAIL, TrialState.PRUNED] self._after_trial_strategy(study, trial, state, values) self._random_sampler.after_trial(study, trial, state, values) optuna-4.1.0/optuna/samplers/_partial_fixed.py000066400000000000000000000073771471332314300215210ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Sequence from typing import Any from typing import TYPE_CHECKING import warnings from optuna._experimental import experimental_class from optuna.distributions import BaseDistribution from optuna.samplers import BaseSampler from optuna.trial import FrozenTrial from optuna.trial import TrialState if TYPE_CHECKING: from optuna.study import Study @experimental_class("2.4.0") class PartialFixedSampler(BaseSampler): """Sampler with partially fixed parameters. Example: After several steps of optimization, you can fix the value of ``y`` and re-optimize it. .. testcode:: import optuna def objective(trial): x = trial.suggest_float("x", -1, 1) y = trial.suggest_int("y", -1, 1) return x**2 + y study = optuna.create_study() study.optimize(objective, n_trials=10) best_params = study.best_params fixed_params = {"y": best_params["y"]} partial_sampler = optuna.samplers.PartialFixedSampler(fixed_params, study.sampler) study.sampler = partial_sampler study.optimize(objective, n_trials=10) Args: fixed_params: A dictionary of parameters to be fixed. base_sampler: A sampler which samples unfixed parameters. """ def __init__(self, fixed_params: dict[str, Any], base_sampler: BaseSampler) -> None: self._fixed_params = fixed_params self._base_sampler = base_sampler def reseed_rng(self) -> None: self._base_sampler.reseed_rng() def infer_relative_search_space( self, study: Study, trial: FrozenTrial ) -> dict[str, BaseDistribution]: search_space = self._base_sampler.infer_relative_search_space(study, trial) # Remove fixed params from relative search space to return fixed values. for param_name in self._fixed_params.keys(): if param_name in search_space: del search_space[param_name] return search_space def sample_relative( self, study: Study, trial: FrozenTrial, search_space: dict[str, BaseDistribution], ) -> dict[str, Any]: # Fixed params are never sampled here. return self._base_sampler.sample_relative(study, trial, search_space) def sample_independent( self, study: Study, trial: FrozenTrial, param_name: str, param_distribution: BaseDistribution, ) -> Any: if param_name not in self._fixed_params: # Unfixed params are sampled here. return self._base_sampler.sample_independent( study, trial, param_name, param_distribution ) else: # Fixed params are sampled here. # Check if a parameter value is contained in the range of this distribution. param_value = self._fixed_params[param_name] param_value_in_internal_repr = param_distribution.to_internal_repr(param_value) contained = param_distribution._contains(param_value_in_internal_repr) if not contained: warnings.warn( f"Fixed parameter '{param_name}' with value {param_value} is out of range " f"for distribution {param_distribution}." ) return param_value def before_trial(self, study: Study, trial: FrozenTrial) -> None: self._base_sampler.before_trial(study, trial) def after_trial( self, study: Study, trial: FrozenTrial, state: TrialState, values: Sequence[float] | None, ) -> None: self._base_sampler.after_trial(study, trial, state, values) optuna-4.1.0/optuna/samplers/_qmc.py000066400000000000000000000322011471332314300174460ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Sequence import threading from typing import Any from typing import TYPE_CHECKING import numpy as np import optuna from optuna import logging from optuna._experimental import experimental_class from optuna._imports import _LazyImport from optuna._transform import _SearchSpaceTransform from optuna.distributions import BaseDistribution from optuna.distributions import CategoricalDistribution from optuna.samplers import BaseSampler from optuna.trial import FrozenTrial from optuna.trial import TrialState if TYPE_CHECKING: from optuna.study import Study _logger = logging.get_logger(__name__) _SUGGESTED_STATES = (TrialState.COMPLETE, TrialState.PRUNED) _threading_lock = threading.Lock() @experimental_class("3.0.0") class QMCSampler(BaseSampler): """A Quasi Monte Carlo Sampler that generates low-discrepancy sequences. Quasi Monte Carlo (QMC) sequences are designed to have lower discrepancies than standard random sequences. They are known to perform better than the standard random sequences in hyperparameter optimization. For further information about the use of QMC sequences for hyperparameter optimization, please refer to the following paper: - `Bergstra, James, and Yoshua Bengio. Random search for hyper-parameter optimization. Journal of machine learning research 13.2, 2012. `__ We use the QMC implementations in Scipy. For the details of the QMC algorithm, see the Scipy API references on `scipy.stats.qmc `__. .. note: If your search space contains categorical parameters, it samples the categorical parameters by its `independent_sampler` without using QMC algorithm. .. note:: The search space of the sampler is determined by either previous trials in the study or the first trial that this sampler samples. If there are previous trials in the study, :class:`~optuna.samplers.QMCSampler` infers its search space using the trial which was created first in the study. Otherwise (if the study has no previous trials), :class:`~optuna.samplers.QMCSampler` samples the first trial using its `independent_sampler` and then infers the search space in the second trial. As mentioned above, the search space of the :class:`~optuna.samplers.QMCSampler` is determined by the first trial of the study. Once the search space is determined, it cannot be changed afterwards. Args: qmc_type: The type of QMC sequence to be sampled. This must be one of `"halton"` and `"sobol"`. Default is `"sobol"`. .. note:: Sobol' sequence is designed to have low-discrepancy property when the number of samples is :math:`n=2^m` for each positive integer :math:`m`. When it is possible to pre-specify the number of trials suggested by `QMCSampler`, it is recommended that the number of trials should be set as power of two. scramble: If this option is :obj:`True`, scrambling (randomization) is applied to the QMC sequences. seed: A seed for ``QMCSampler``. This argument is used only when ``scramble`` is :obj:`True`. If this is :obj:`None`, the seed is initialized randomly. Default is :obj:`None`. .. note:: When using multiple :class:`~optuna.samplers.QMCSampler`'s in parallel and/or distributed optimization, all the samplers must share the same seed when the `scrambling` is enabled. Otherwise, the low-discrepancy property of the samples will be degraded. independent_sampler: A :class:`~optuna.samplers.BaseSampler` instance that is used for independent sampling. The first trial of the study and the parameters not contained in the relative search space are sampled by this sampler. If :obj:`None` is specified, :class:`~optuna.samplers.RandomSampler` is used as the default. .. seealso:: :class:`~optuna.samplers` module provides built-in independent samplers such as :class:`~optuna.samplers.RandomSampler` and :class:`~optuna.samplers.TPESampler`. warn_independent_sampling: If this is :obj:`True`, a warning message is emitted when the value of a parameter is sampled by using an independent sampler. Note that the parameters of the first trial in a study are sampled via an independent sampler in most cases, so no warning messages are emitted in such cases. warn_asynchronous_seeding: If this is :obj:`True`, a warning message is emitted when the scrambling (randomization) is applied to the QMC sequence and the random seed of the sampler is not set manually. .. note:: When using parallel and/or distributed optimization without manually setting the seed, the seed is set randomly for each instances of :class:`~optuna.samplers.QMCSampler` for different workers, which ends up asynchronous seeding for multiple samplers used in the optimization. .. seealso:: See parameter ``seed`` in :class:`~optuna.samplers.QMCSampler`. Example: Optimize a simple quadratic function by using :class:`~optuna.samplers.QMCSampler`. .. testcode:: import optuna def objective(trial): x = trial.suggest_float("x", -1, 1) y = trial.suggest_int("y", -1, 1) return x**2 + y sampler = optuna.samplers.QMCSampler() study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=8) """ def __init__( self, *, qmc_type: str = "sobol", scramble: bool = False, # default is False for simplicity in distributed environment. seed: int | None = None, independent_sampler: BaseSampler | None = None, warn_asynchronous_seeding: bool = True, warn_independent_sampling: bool = True, ) -> None: self._scramble = scramble self._seed = np.random.PCG64().random_raw() if seed is None else seed self._independent_sampler = independent_sampler or optuna.samplers.RandomSampler(seed=seed) self._initial_search_space: dict[str, BaseDistribution] | None = None self._warn_independent_sampling = warn_independent_sampling if qmc_type in ("halton", "sobol"): self._qmc_type = qmc_type else: message = ( f'The `qmc_type`, "{qmc_type}", is not a valid. ' 'It must be one of "halton" and "sobol".' ) raise ValueError(message) if seed is None and scramble and warn_asynchronous_seeding: # Sobol/Halton sequences without scrambling do not use seed. self._log_asynchronous_seeding() def reseed_rng(self) -> None: # We must not reseed the `self._seed` like below. Otherwise, workers will have different # seed under parallel execution because `self.reseed_rng()` is called when starting each # parallel executor. # >>> self._seed = np.random.MT19937().random_raw() self._independent_sampler.reseed_rng() def infer_relative_search_space( self, study: Study, trial: FrozenTrial ) -> dict[str, BaseDistribution]: if self._initial_search_space is not None: return self._initial_search_space past_trials = study._get_trials(deepcopy=False, states=_SUGGESTED_STATES, use_cache=True) # The initial trial is sampled by the independent sampler. if len(past_trials) == 0: return {} # If an initial trial was already made, # construct search_space of this sampler from the initial trial. first_trial = min(past_trials, key=lambda t: t.number) self._initial_search_space = self._infer_initial_search_space(first_trial) return self._initial_search_space def _infer_initial_search_space(self, trial: FrozenTrial) -> dict[str, BaseDistribution]: search_space: dict[str, BaseDistribution] = {} for param_name, distribution in trial.distributions.items(): if isinstance(distribution, CategoricalDistribution): continue search_space[param_name] = distribution return search_space @staticmethod def _log_asynchronous_seeding() -> None: _logger.warning( "No seed is provided for `QMCSampler` and the seed is set randomly. " "If you are running multiple `QMCSampler`s in parallel and/or distributed " " environment, the same seed must be used in all samplers to ensure that resulting " "samples are taken from the same QMC sequence. " ) def _log_independent_sampling(self, trial: FrozenTrial, param_name: str) -> None: _logger.warning( f"The parameter '{param_name}' in trial#{trial.number} is sampled independently " f"by using `{self._independent_sampler.__class__.__name__}` instead of `QMCSampler` " "(optimization performance may be degraded). " "`QMCSampler` does not support dynamic search space or `CategoricalDistribution`. " "You can suppress this warning by setting `warn_independent_sampling` " "to `False` in the constructor of `QMCSampler`, " "if this independent sampling is intended behavior." ) def sample_independent( self, study: Study, trial: FrozenTrial, param_name: str, param_distribution: BaseDistribution, ) -> Any: if self._initial_search_space is not None: if self._warn_independent_sampling: self._log_independent_sampling(trial, param_name) return self._independent_sampler.sample_independent( study, trial, param_name, param_distribution ) def sample_relative( self, study: Study, trial: FrozenTrial, search_space: dict[str, BaseDistribution] ) -> dict[str, Any]: if search_space == {}: return {} sample = self._sample_qmc(study, search_space) trans = _SearchSpaceTransform(search_space) sample = trans.bounds[:, 0] + sample * (trans.bounds[:, 1] - trans.bounds[:, 0]) return trans.untransform(sample[0, :]) def before_trial(self, study: Study, trial: FrozenTrial) -> None: self._independent_sampler.before_trial(study, trial) def after_trial( self, study: Study, trial: "optuna.trial.FrozenTrial", state: TrialState, values: Sequence[float] | None, ) -> None: self._independent_sampler.after_trial(study, trial, state, values) def _sample_qmc(self, study: Study, search_space: dict[str, BaseDistribution]) -> np.ndarray: # Lazy import because the `scipy.stats.qmc` is slow to import. qmc_module = _LazyImport("scipy.stats.qmc") sample_id = self._find_sample_id(study) d = len(search_space) if self._qmc_type == "halton": qmc_engine = qmc_module.Halton(d, seed=self._seed, scramble=self._scramble) elif self._qmc_type == "sobol": # Sobol engine likely shares its internal state among threads. # Without threading.Lock, ValueError exceptions are raised in Sobol engine as discussed # in https://github.com/optuna/optunahub-registry/pull/168#pullrequestreview-2404054969 with _threading_lock: qmc_engine = qmc_module.Sobol(d, seed=self._seed, scramble=self._scramble) else: raise ValueError("Invalid `qmc_type`") forward_size = sample_id # `sample_id` starts from 0. # Skip fast_forward with forward_size==0 because Sobol doesn't support the case, # and fast_forward(0) doesn't affect sampling. if forward_size > 0: qmc_engine.fast_forward(forward_size) sample = qmc_engine.random(1) return sample def _find_sample_id(self, study: Study) -> int: qmc_id = "" qmc_id += self._qmc_type # Sobol/Halton sequences without scrambling do not use seed. if self._scramble: qmc_id += f" (scramble=True, seed={self._seed})" else: qmc_id += " (scramble=False)" key_qmc_id = qmc_id + "'s last sample id" # TODO(kstoneriv3): Here, we ideally assume that the following block is # an atomic transaction. Without such an assumption, the current implementation # only ensures that each `sample_id` is sampled at least once. system_attrs = study._storage.get_study_system_attrs(study._study_id) if key_qmc_id in system_attrs.keys(): sample_id = system_attrs[key_qmc_id] sample_id += 1 else: sample_id = 0 study._storage.set_study_system_attr(study._study_id, key_qmc_id, sample_id) return sample_id optuna-4.1.0/optuna/samplers/_random.py000066400000000000000000000037551471332314300201620ustar00rootroot00000000000000from __future__ import annotations from typing import Any from typing import Dict from typing import Optional from typing import TYPE_CHECKING from optuna import distributions from optuna._transform import _SearchSpaceTransform from optuna.distributions import BaseDistribution from optuna.samplers import BaseSampler from optuna.samplers._lazy_random_state import LazyRandomState from optuna.trial import FrozenTrial if TYPE_CHECKING: from optuna.study import Study class RandomSampler(BaseSampler): """Sampler using random sampling. This sampler is based on *independent sampling*. See also :class:`~optuna.samplers.BaseSampler` for more details of 'independent sampling'. Example: .. testcode:: import optuna from optuna.samplers import RandomSampler def objective(trial): x = trial.suggest_float("x", -5, 5) return x**2 study = optuna.create_study(sampler=RandomSampler()) study.optimize(objective, n_trials=10) Args: seed: Seed for random number generator. """ def __init__(self, seed: Optional[int] = None) -> None: self._rng = LazyRandomState(seed) def reseed_rng(self) -> None: self._rng.rng.seed() def infer_relative_search_space( self, study: Study, trial: FrozenTrial ) -> Dict[str, BaseDistribution]: return {} def sample_relative( self, study: Study, trial: FrozenTrial, search_space: Dict[str, BaseDistribution] ) -> Dict[str, Any]: return {} def sample_independent( self, study: Study, trial: FrozenTrial, param_name: str, param_distribution: distributions.BaseDistribution, ) -> Any: search_space = {param_name: param_distribution} trans = _SearchSpaceTransform(search_space) trans_params = self._rng.rng.uniform(trans.bounds[:, 0], trans.bounds[:, 1]) return trans.untransform(trans_params)[param_name] optuna-4.1.0/optuna/samplers/_tpe/000077500000000000000000000000001471332314300171065ustar00rootroot00000000000000optuna-4.1.0/optuna/samplers/_tpe/__init__.py000066400000000000000000000000001471332314300212050ustar00rootroot00000000000000optuna-4.1.0/optuna/samplers/_tpe/_erf.py000066400000000000000000000140571471332314300204020ustar00rootroot00000000000000# This code is the modified version of erf function in FreeBSD's standard C library. # origin: FreeBSD /usr/src/lib/msun/src/s_erf.c # https://github.com/freebsd/freebsd-src/blob/main/lib/msun/src/s_erf.c # /* @(#)s_erf.c 5.1 93/09/24 */ # /* # * ==================================================== # * Copyright (C) 1993 by Sun Microsystems, Inc. All rights reserved. # * # * Developed at SunPro, a Sun Microsystems, Inc. business. # * Permission to use, copy, modify, and distribute this # * software is freely granted, provided that this notice # * is preserved. # * ==================================================== # */ import numpy as np from numpy.polynomial import Polynomial half = 0.5 one = 1 two = 2 erx = 8.45062911510467529297e-01 # /* # * In the domain [0, 2**-28], only the first term in the power series # * expansion of erf(x) is used. The magnitude of the first neglected # * terms is less than 2**-84. # */ efx = 1.28379167095512586316e-01 efx8 = 1.02703333676410069053e00 # Coefficients for approximation to erf on [0,0.84375] pp0 = 1.28379167095512558561e-01 pp1 = -3.25042107247001499370e-01 pp2 = -2.84817495755985104766e-02 pp3 = -5.77027029648944159157e-03 pp4 = -2.37630166566501626084e-05 pp = Polynomial([pp0, pp1, pp2, pp3, pp4]) # type: ignore[no-untyped-call] qq1 = 3.97917223959155352819e-01 qq2 = 6.50222499887672944485e-02 qq3 = 5.08130628187576562776e-03 qq4 = 1.32494738004321644526e-04 qq5 = -3.96022827877536812320e-06 qq = Polynomial([one, qq1, qq2, qq3, qq4, qq5]) # type: ignore[no-untyped-call] # Coefficients for approximation to erf in [0.84375,1.25] pa0 = -2.36211856075265944077e-03 pa1 = 4.14856118683748331666e-01 pa2 = -3.72207876035701323847e-01 pa3 = 3.18346619901161753674e-01 pa4 = -1.10894694282396677476e-01 pa5 = 3.54783043256182359371e-02 pa6 = -2.16637559486879084300e-03 pa = Polynomial([pa0, pa1, pa2, pa3, pa4, pa5, pa6]) # type: ignore[no-untyped-call] qa1 = 1.06420880400844228286e-01 qa2 = 5.40397917702171048937e-01 qa3 = 7.18286544141962662868e-02 qa4 = 1.26171219808761642112e-01 qa5 = 1.36370839120290507362e-02 qa6 = 1.19844998467991074170e-02 qa = Polynomial([one, qa1, qa2, qa3, qa4, qa5, qa6]) # type: ignore[no-untyped-call] # Coefficients for approximation to erfc in [1.25,1/0.35] ra0 = -9.86494403484714822705e-03 ra1 = -6.93858572707181764372e-01 ra2 = -1.05586262253232909814e01 ra3 = -6.23753324503260060396e01 ra4 = -1.62396669462573470355e02 ra5 = -1.84605092906711035994e02 ra6 = -8.12874355063065934246e01 ra7 = -9.81432934416914548592e00 ra = Polynomial([ra0, ra1, ra2, ra3, ra4, ra5, ra6, ra7]) # type: ignore[no-untyped-call] sa1 = 1.96512716674392571292e01 sa2 = 1.37657754143519042600e02 sa3 = 4.34565877475229228821e02 sa4 = 6.45387271733267880336e02 sa5 = 4.29008140027567833386e02 sa6 = 1.08635005541779435134e02 sa7 = 6.57024977031928170135e00 sa8 = -6.04244152148580987438e-02 sa = Polynomial([one, sa1, sa2, sa3, sa4, sa5, sa6, sa7, sa8]) # type: ignore[no-untyped-call] # Coefficients for approximation to erfc in [1/.35,28] rb0 = -9.86494292470009928597e-03 rb1 = -7.99283237680523006574e-01 rb2 = -1.77579549177547519889e01 rb3 = -1.60636384855821916062e02 rb4 = -6.37566443368389627722e02 rb5 = -1.02509513161107724954e03 rb6 = -4.83519191608651397019e02 rb = Polynomial([rb0, rb1, rb2, rb3, rb4, rb5, rb6]) # type: ignore[no-untyped-call] sb1 = 3.03380607434824582924e01 sb2 = 3.25792512996573918826e02 sb3 = 1.53672958608443695994e03 sb4 = 3.19985821950859553908e03 sb5 = 2.55305040643316442583e03 sb6 = 4.74528541206955367215e02 sb7 = -2.24409524465858183362e01 sb = Polynomial([one, sb1, sb2, sb3, sb4, sb5, sb6, sb7]) # type: ignore[no-untyped-call] def erf(x: np.ndarray) -> np.ndarray: a = np.abs(x) case_nan = np.isnan(x) case_posinf = np.isposinf(x) case_neginf = np.isneginf(x) case_tiny = a < 2**-28 case_small1 = (2**-28 <= a) & (a < 0.84375) case_small2 = (0.84375 <= a) & (a < 1.25) case_med1 = (1.25 <= a) & (a < 1 / 0.35) case_med2 = (1 / 0.35 <= a) & (a < 6) case_big = a >= 6 def calc_case_tiny(x: np.ndarray) -> np.ndarray: return x + efx * x def calc_case_small1(x: np.ndarray) -> np.ndarray: z = x * x r = pp(z) s = qq(z) y = r / s return x + x * y def calc_case_small2(x: np.ndarray) -> np.ndarray: s = np.abs(x) - one P = pa(s) Q = qa(s) absout = erx + P / Q return absout * np.sign(x) def calc_case_med1(x: np.ndarray) -> np.ndarray: sign = np.sign(x) x = np.abs(x) s = one / (x * x) R = ra(s) S = sa(s) # the following 3 lines are omitted for the following reasons: # (1) there are no easy way to implement SET_LOW_WORD equivalent method in NumPy # (2) we don't need very high accuracy in our use case. # z = x # SET_LOW_WORD(z, 0) # r = np.exp(-z * z - 0.5625) * np.exp((z - x) * (z + x) + R / S) r = np.exp(-x * x - 0.5625) * np.exp(R / S) return (one - r / x) * sign def calc_case_med2(x: np.ndarray) -> np.ndarray: sign = np.sign(x) x = np.abs(x) s = one / (x * x) R = rb(s) S = sb(s) # z = x # SET_LOW_WORD(z, 0) # r = np.exp(-z * z - 0.5625) * np.exp((z - x) * (z + x) + R / S) r = np.exp(-x * x - 0.5625) * np.exp(R / S) return (one - r / x) * sign def calc_case_big(x: np.ndarray) -> np.ndarray: return np.sign(x) out = np.full_like(a, fill_value=np.nan, dtype=np.float64) out[case_nan] = np.nan out[case_posinf] = 1.0 out[case_neginf] = -1.0 if x[case_tiny].size: out[case_tiny] = calc_case_tiny(x[case_tiny]) if x[case_small1].size: out[case_small1] = calc_case_small1(x[case_small1]) if x[case_small2].size: out[case_small2] = calc_case_small2(x[case_small2]) if x[case_med1].size: out[case_med1] = calc_case_med1(x[case_med1]) if x[case_med2].size: out[case_med2] = calc_case_med2(x[case_med2]) if x[case_big].size: out[case_big] = calc_case_big(x[case_big]) return out optuna-4.1.0/optuna/samplers/_tpe/_truncnorm.py000066400000000000000000000167371471332314300216640ustar00rootroot00000000000000# This file contains the codes from SciPy project. # # Copyright (c) 2001-2002 Enthought, Inc. 2003-2022, SciPy Developers. # All rights reserved. # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions # are met: # 1. Redistributions of source code must retain the above copyright # notice, this list of conditions and the following disclaimer. # 2. Redistributions in binary form must reproduce the above # copyright notice, this list of conditions and the following # disclaimer in the documentation and/or other materials provided # with the distribution. # 3. Neither the name of the copyright holder nor the names of its # contributors may be used to endorse or promote products derived # from this software without specific prior written permission. # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS # "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT # OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, # SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT # LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, # DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY # THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. from __future__ import annotations from collections.abc import Callable import functools import math import sys import numpy as np from optuna.samplers._tpe._erf import erf _norm_pdf_C = math.sqrt(2 * math.pi) _norm_pdf_logC = math.log(_norm_pdf_C) def _log_sum(log_p: np.ndarray, log_q: np.ndarray) -> np.ndarray: return np.logaddexp(log_p, log_q) def _log_diff(log_p: np.ndarray, log_q: np.ndarray) -> np.ndarray: return log_p + np.log1p(-np.exp(log_q - log_p)) @functools.lru_cache(1000) def _ndtr_single(a: float) -> float: x = a / 2**0.5 if x < -1 / 2**0.5: y = 0.5 * math.erfc(-x) elif x < 1 / 2**0.5: y = 0.5 + 0.5 * math.erf(x) else: y = 1.0 - 0.5 * math.erfc(x) return y def _ndtr(a: np.ndarray) -> np.ndarray: # todo(amylase): implement erfc in _erf.py and use it for big |a| inputs. return 0.5 + 0.5 * erf(a / 2**0.5) @functools.lru_cache(1000) def _log_ndtr_single(a: float) -> float: if a > 6: return -_ndtr_single(-a) if a > -20: return math.log(_ndtr_single(a)) log_LHS = -0.5 * a**2 - math.log(-a) - 0.5 * math.log(2 * math.pi) last_total = 0.0 right_hand_side = 1.0 numerator = 1.0 denom_factor = 1.0 denom_cons = 1 / a**2 sign = 1 i = 0 while abs(last_total - right_hand_side) > sys.float_info.epsilon: i += 1 last_total = right_hand_side sign = -sign denom_factor *= denom_cons numerator *= 2 * i - 1 right_hand_side += sign * numerator * denom_factor return log_LHS + math.log(right_hand_side) def _log_ndtr(a: np.ndarray) -> np.ndarray: return np.frompyfunc(_log_ndtr_single, 1, 1)(a).astype(float) def _norm_logpdf(x: np.ndarray) -> np.ndarray: return -(x**2) / 2.0 - _norm_pdf_logC def _log_gauss_mass(a: np.ndarray, b: np.ndarray) -> np.ndarray: """Log of Gaussian probability mass within an interval""" # Calculations in right tail are inaccurate, so we'll exploit the # symmetry and work only in the left tail case_left = b <= 0 case_right = a > 0 case_central = ~(case_left | case_right) def mass_case_left(a: np.ndarray, b: np.ndarray) -> np.ndarray: return _log_diff(_log_ndtr(b), _log_ndtr(a)) def mass_case_right(a: np.ndarray, b: np.ndarray) -> np.ndarray: return mass_case_left(-b, -a) def mass_case_central(a: np.ndarray, b: np.ndarray) -> np.ndarray: # Previously, this was implemented as: # left_mass = mass_case_left(a, 0) # right_mass = mass_case_right(0, b) # return _log_sum(left_mass, right_mass) # Catastrophic cancellation occurs as np.exp(log_mass) approaches 1. # Correct for this with an alternative formulation. # We're not concerned with underflow here: if only one term # underflows, it was insignificant; if both terms underflow, # the result can't accurately be represented in logspace anyway # because sc.log1p(x) ~ x for small x. return np.log1p(-_ndtr(a) - _ndtr(-b)) # _lazyselect not working; don't care to debug it out = np.full_like(a, fill_value=np.nan, dtype=np.complex128) if a[case_left].size: out[case_left] = mass_case_left(a[case_left], b[case_left]) if a[case_right].size: out[case_right] = mass_case_right(a[case_right], b[case_right]) if a[case_central].size: out[case_central] = mass_case_central(a[case_central], b[case_central]) return np.real(out) # discard ~0j def _bisect(f: Callable[[float], float], a: float, b: float, c: float) -> float: if f(a) > c: a, b = b, a # In the algorithm, it is assumed that all of (a + b), (a * 2), and (b * 2) are finite. for _ in range(100): m = (a + b) / 2 if a == m or b == m: return m if f(m) < c: a = m else: b = m return (a + b) / 2 def _ndtri_exp_single(y: float) -> float: # TODO(amylase): Justify this constant return _bisect(_log_ndtr_single, -100, +100, y) def _ndtri_exp(y: np.ndarray) -> np.ndarray: return np.frompyfunc(_ndtri_exp_single, 1, 1)(y).astype(float) def ppf(q: np.ndarray, a: np.ndarray | float, b: np.ndarray | float) -> np.ndarray: q, a, b = np.atleast_1d(q, a, b) q, a, b = np.broadcast_arrays(q, a, b) case_left = a < 0 case_right = ~case_left def ppf_left(q: np.ndarray, a: np.ndarray, b: np.ndarray) -> np.ndarray: log_Phi_x = _log_sum(_log_ndtr(a), np.log(q) + _log_gauss_mass(a, b)) return _ndtri_exp(log_Phi_x) def ppf_right(q: np.ndarray, a: np.ndarray, b: np.ndarray) -> np.ndarray: log_Phi_x = _log_sum(_log_ndtr(-b), np.log1p(-q) + _log_gauss_mass(a, b)) return -_ndtri_exp(log_Phi_x) out = np.empty_like(q) q_left = q[case_left] q_right = q[case_right] if q_left.size: out[case_left] = ppf_left(q_left, a[case_left], b[case_left]) if q_right.size: out[case_right] = ppf_right(q_right, a[case_right], b[case_right]) out[q == 0] = a[q == 0] out[q == 1] = b[q == 1] out[a == b] = math.nan return out def rvs( a: np.ndarray, b: np.ndarray, loc: np.ndarray | float = 0, scale: np.ndarray | float = 1, random_state: np.random.RandomState | None = None, ) -> np.ndarray: random_state = random_state or np.random.RandomState() size = np.broadcast(a, b, loc, scale).shape percentiles = random_state.uniform(low=0, high=1, size=size) return ppf(percentiles, a, b) * scale + loc def logpdf( x: np.ndarray, a: np.ndarray | float, b: np.ndarray | float, loc: np.ndarray | float = 0, scale: np.ndarray | float = 1, ) -> np.ndarray: x = (x - loc) / scale x, a, b = np.atleast_1d(x, a, b) out = _norm_logpdf(x) - _log_gauss_mass(a, b) - np.log(scale) x, a, b = np.broadcast_arrays(x, a, b) out[(x < a) | (b < x)] = -np.inf out[a == b] = math.nan return out optuna-4.1.0/optuna/samplers/_tpe/parzen_estimator.py000066400000000000000000000270141471332314300230520ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Callable from typing import NamedTuple import numpy as np from optuna.distributions import BaseDistribution from optuna.distributions import CategoricalChoiceType from optuna.distributions import CategoricalDistribution from optuna.distributions import FloatDistribution from optuna.distributions import IntDistribution from optuna.samplers._tpe.probability_distributions import _BatchedCategoricalDistributions from optuna.samplers._tpe.probability_distributions import _BatchedDiscreteTruncNormDistributions from optuna.samplers._tpe.probability_distributions import _BatchedDistributions from optuna.samplers._tpe.probability_distributions import _BatchedTruncNormDistributions from optuna.samplers._tpe.probability_distributions import _MixtureOfProductDistribution EPS = 1e-12 class _ParzenEstimatorParameters(NamedTuple): consider_prior: bool prior_weight: float | None consider_magic_clip: bool consider_endpoints: bool weights: Callable[[int], np.ndarray] multivariate: bool categorical_distance_func: dict[ str, Callable[[CategoricalChoiceType, CategoricalChoiceType], float] ] class _ParzenEstimator: def __init__( self, observations: dict[str, np.ndarray], search_space: dict[str, BaseDistribution], parameters: _ParzenEstimatorParameters, predetermined_weights: np.ndarray | None = None, ) -> None: if parameters.consider_prior: if parameters.prior_weight is None: raise ValueError("Prior weight must be specified when consider_prior==True.") elif parameters.prior_weight <= 0: raise ValueError("Prior weight must be positive.") self._search_space = search_space transformed_observations = self._transform(observations) assert predetermined_weights is None or len(transformed_observations) == len( predetermined_weights ) weights = ( predetermined_weights if predetermined_weights is not None else self._call_weights_func(parameters.weights, len(transformed_observations)) ) if len(transformed_observations) == 0: weights = np.array([1.0]) elif parameters.consider_prior: assert parameters.prior_weight is not None weights = np.append(weights, [parameters.prior_weight]) weights /= weights.sum() self._mixture_distribution = _MixtureOfProductDistribution( weights=weights, distributions=[ self._calculate_distributions( transformed_observations[:, i], param, search_space[param], parameters ) for i, param in enumerate(search_space) ], ) def sample(self, rng: np.random.RandomState, size: int) -> dict[str, np.ndarray]: sampled = self._mixture_distribution.sample(rng, size) return self._untransform(sampled) def log_pdf(self, samples_dict: dict[str, np.ndarray]) -> np.ndarray: transformed_samples = self._transform(samples_dict) return self._mixture_distribution.log_pdf(transformed_samples) @staticmethod def _call_weights_func(weights_func: Callable[[int], np.ndarray], n: int) -> np.ndarray: w = np.array(weights_func(n))[:n] if np.any(w < 0): raise ValueError( f"The `weights` function is not allowed to return negative values {w}. " + f"The argument of the `weights` function is {n}." ) if len(w) > 0 and np.sum(w) <= 0: raise ValueError( f"The `weight` function is not allowed to return all-zero values {w}." + f" The argument of the `weights` function is {n}." ) if not np.all(np.isfinite(w)): raise ValueError( "The `weights`function is not allowed to return infinite or NaN values " + f"{w}. The argument of the `weights` function is {n}." ) # TODO(HideakiImamura) Raise `ValueError` if the weight function returns an ndarray of # unexpected size. return w @staticmethod def _is_log(dist: BaseDistribution) -> bool: return isinstance(dist, (FloatDistribution, IntDistribution)) and dist.log def _transform(self, samples_dict: dict[str, np.ndarray]) -> np.ndarray: return np.array( [ ( np.log(samples_dict[param]) if self._is_log(self._search_space[param]) else samples_dict[param] ) for param in self._search_space ] ).T def _untransform(self, samples_array: np.ndarray) -> dict[str, np.ndarray]: res = { param: ( np.exp(samples_array[:, i]) if self._is_log(self._search_space[param]) else samples_array[:, i] ) for i, param in enumerate(self._search_space) } # TODO(contramundum53): Remove this line after fixing log-Int hack. return { param: ( np.clip( dist.low + np.round((res[param] - dist.low) / dist.step) * dist.step, dist.low, dist.high, ) if isinstance(dist, IntDistribution) else res[param] ) for (param, dist) in self._search_space.items() } def _calculate_distributions( self, transformed_observations: np.ndarray, param_name: str, search_space: BaseDistribution, parameters: _ParzenEstimatorParameters, ) -> _BatchedDistributions: if isinstance(search_space, CategoricalDistribution): return self._calculate_categorical_distributions( transformed_observations, param_name, search_space, parameters ) else: assert isinstance(search_space, (FloatDistribution, IntDistribution)) if search_space.log: low = np.log(search_space.low) high = np.log(search_space.high) else: low = search_space.low high = search_space.high step = search_space.step # TODO(contramundum53): This is a hack and should be fixed. if step is not None and search_space.log: low = np.log(search_space.low - step / 2) high = np.log(search_space.high + step / 2) step = None return self._calculate_numerical_distributions( transformed_observations, low, high, step, parameters ) def _calculate_categorical_distributions( self, observations: np.ndarray, param_name: str, search_space: CategoricalDistribution, parameters: _ParzenEstimatorParameters, ) -> _BatchedDistributions: choices = search_space.choices n_choices = len(choices) if len(observations) == 0: return _BatchedCategoricalDistributions( weights=np.full((1, n_choices), fill_value=1.0 / n_choices) ) n_kernels = len(observations) + parameters.consider_prior assert parameters.prior_weight is not None weights = np.full( shape=(n_kernels, n_choices), fill_value=parameters.prior_weight / n_kernels, ) observed_indices = observations.astype(int) if param_name in parameters.categorical_distance_func: # TODO(nabenabe0928): Think about how to handle combinatorial explosion. # The time complexity is O(n_choices * used_indices.size), so n_choices cannot be huge. used_indices, rev_indices = np.unique(observed_indices, return_inverse=True) dist_func = parameters.categorical_distance_func[param_name] dists = np.array([[dist_func(choices[i], c) for c in choices] for i in used_indices]) coef = np.log(n_kernels / parameters.prior_weight) * np.log(n_choices) / np.log(6) cat_weights = np.exp(-((dists / np.max(dists, axis=1)[:, np.newaxis]) ** 2) * coef) weights[: len(observed_indices)] = cat_weights[rev_indices] else: weights[np.arange(len(observed_indices)), observed_indices] += 1 weights /= weights.sum(axis=1, keepdims=True) return _BatchedCategoricalDistributions(weights) def _calculate_numerical_distributions( self, observations: np.ndarray, low: float, high: float, step: float | None, parameters: _ParzenEstimatorParameters, ) -> _BatchedDistributions: step_or_0 = step or 0 mus = observations consider_prior = parameters.consider_prior or len(observations) == 0 def compute_sigmas() -> np.ndarray: if parameters.multivariate: SIGMA0_MAGNITUDE = 0.2 sigma = ( SIGMA0_MAGNITUDE * max(len(observations), 1) ** (-1.0 / (len(self._search_space) + 4)) * (high - low + step_or_0) ) sigmas = np.full(shape=(len(observations),), fill_value=sigma) else: # TODO(contramundum53): Remove dependency on prior_mu prior_mu = 0.5 * (low + high) mus_with_prior = np.append(mus, prior_mu) if consider_prior else mus sorted_indices = np.argsort(mus_with_prior) sorted_mus = mus_with_prior[sorted_indices] sorted_mus_with_endpoints = np.empty(len(mus_with_prior) + 2, dtype=float) sorted_mus_with_endpoints[0] = low - step_or_0 / 2 sorted_mus_with_endpoints[1:-1] = sorted_mus sorted_mus_with_endpoints[-1] = high + step_or_0 / 2 sorted_sigmas = np.maximum( sorted_mus_with_endpoints[1:-1] - sorted_mus_with_endpoints[0:-2], sorted_mus_with_endpoints[2:] - sorted_mus_with_endpoints[1:-1], ) if not parameters.consider_endpoints and sorted_mus_with_endpoints.shape[0] >= 4: sorted_sigmas[0] = sorted_mus_with_endpoints[2] - sorted_mus_with_endpoints[1] sorted_sigmas[-1] = ( sorted_mus_with_endpoints[-2] - sorted_mus_with_endpoints[-3] ) sigmas = sorted_sigmas[np.argsort(sorted_indices)][: len(observations)] # We adjust the range of the 'sigmas' according to the 'consider_magic_clip' flag. maxsigma = 1.0 * (high - low + step_or_0) if parameters.consider_magic_clip: # TODO(contramundum53): Remove dependency of minsigma on consider_prior. minsigma = ( 1.0 * (high - low + step_or_0) / min(100.0, (1.0 + len(observations) + consider_prior)) ) else: minsigma = EPS return np.asarray(np.clip(sigmas, minsigma, maxsigma)) sigmas = compute_sigmas() if consider_prior: prior_mu = 0.5 * (low + high) prior_sigma = 1.0 * (high - low + step_or_0) mus = np.append(mus, [prior_mu]) sigmas = np.append(sigmas, [prior_sigma]) if step is None: return _BatchedTruncNormDistributions(mus, sigmas, low, high) else: return _BatchedDiscreteTruncNormDistributions(mus, sigmas, low, high, step) optuna-4.1.0/optuna/samplers/_tpe/probability_distributions.py000066400000000000000000000116411471332314300247650ustar00rootroot00000000000000from __future__ import annotations from typing import NamedTuple from typing import Union import numpy as np from optuna.samplers._tpe import _truncnorm class _BatchedCategoricalDistributions(NamedTuple): weights: np.ndarray class _BatchedTruncNormDistributions(NamedTuple): mu: np.ndarray sigma: np.ndarray low: float # Currently, low and high do not change per trial. high: float class _BatchedDiscreteTruncNormDistributions(NamedTuple): mu: np.ndarray sigma: np.ndarray low: float # Currently, low, high and step do not change per trial. high: float step: float _BatchedDistributions = Union[ _BatchedCategoricalDistributions, _BatchedTruncNormDistributions, _BatchedDiscreteTruncNormDistributions, ] class _MixtureOfProductDistribution(NamedTuple): weights: np.ndarray distributions: list[_BatchedDistributions] def sample(self, rng: np.random.RandomState, batch_size: int) -> np.ndarray: active_indices = rng.choice(len(self.weights), p=self.weights, size=batch_size) ret = np.empty((batch_size, len(self.distributions)), dtype=np.float64) for i, d in enumerate(self.distributions): if isinstance(d, _BatchedCategoricalDistributions): active_weights = d.weights[active_indices, :] rnd_quantile = rng.rand(batch_size) cum_probs = np.cumsum(active_weights, axis=-1) assert np.isclose(cum_probs[:, -1], 1).all() cum_probs[:, -1] = 1 # Avoid numerical errors. ret[:, i] = np.sum(cum_probs < rnd_quantile[:, None], axis=-1) elif isinstance(d, _BatchedTruncNormDistributions): active_mus = d.mu[active_indices] active_sigmas = d.sigma[active_indices] ret[:, i] = _truncnorm.rvs( a=(d.low - active_mus) / active_sigmas, b=(d.high - active_mus) / active_sigmas, loc=active_mus, scale=active_sigmas, random_state=rng, ) elif isinstance(d, _BatchedDiscreteTruncNormDistributions): active_mus = d.mu[active_indices] active_sigmas = d.sigma[active_indices] samples = _truncnorm.rvs( a=(d.low - d.step / 2 - active_mus) / active_sigmas, b=(d.high + d.step / 2 - active_mus) / active_sigmas, loc=active_mus, scale=active_sigmas, random_state=rng, ) ret[:, i] = np.clip( d.low + np.round((samples - d.low) / d.step) * d.step, d.low, d.high ) else: assert False return ret def log_pdf(self, x: np.ndarray) -> np.ndarray: batch_size, n_vars = x.shape log_pdfs = np.empty((batch_size, len(self.weights), n_vars), dtype=np.float64) for i, d in enumerate(self.distributions): xi = x[:, i] if isinstance(d, _BatchedCategoricalDistributions): log_pdfs[:, :, i] = np.log( np.take_along_axis( d.weights[None, :, :], xi[:, None, None].astype(np.int64), axis=-1 ) )[:, :, 0] elif isinstance(d, _BatchedTruncNormDistributions): log_pdfs[:, :, i] = _truncnorm.logpdf( x=xi[:, None], a=(d.low - d.mu[None, :]) / d.sigma[None, :], b=(d.high - d.mu[None, :]) / d.sigma[None, :], loc=d.mu[None, :], scale=d.sigma[None, :], ) elif isinstance(d, _BatchedDiscreteTruncNormDistributions): lower_limit = d.low - d.step / 2 upper_limit = d.high + d.step / 2 x_lower = np.maximum(xi - d.step / 2, lower_limit) x_upper = np.minimum(xi + d.step / 2, upper_limit) log_gauss_mass = _truncnorm._log_gauss_mass( (x_lower[:, None] - d.mu[None, :]) / d.sigma[None, :], (x_upper[:, None] - d.mu[None, :]) / d.sigma[None, :], ) log_p_accept = _truncnorm._log_gauss_mass( (d.low - d.step / 2 - d.mu[None, :]) / d.sigma[None, :], (d.high + d.step / 2 - d.mu[None, :]) / d.sigma[None, :], ) log_pdfs[:, :, i] = log_gauss_mass - log_p_accept else: assert False weighted_log_pdf = np.sum(log_pdfs, axis=-1) + np.log(self.weights[None, :]) max_ = weighted_log_pdf.max(axis=1) # We need to avoid (-inf) - (-inf) when the probability is zero. max_[np.isneginf(max_)] = 0 with np.errstate(divide="ignore"): # Suppress warning in log(0). return np.log(np.exp(weighted_log_pdf - max_[:, None]).sum(axis=1)) + max_ optuna-4.1.0/optuna/samplers/_tpe/sampler.py000066400000000000000000001031111471332314300211200ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Callable from collections.abc import Sequence import math from typing import Any from typing import cast from typing import TYPE_CHECKING import warnings import numpy as np from optuna._experimental import warn_experimental_argument from optuna._hypervolume import compute_hypervolume from optuna._hypervolume.hssp import _solve_hssp from optuna.distributions import BaseDistribution from optuna.distributions import CategoricalChoiceType from optuna.logging import get_logger from optuna.samplers._base import _CONSTRAINTS_KEY from optuna.samplers._base import _process_constraints_after_trial from optuna.samplers._base import BaseSampler from optuna.samplers._lazy_random_state import LazyRandomState from optuna.samplers._random import RandomSampler from optuna.samplers._tpe.parzen_estimator import _ParzenEstimator from optuna.samplers._tpe.parzen_estimator import _ParzenEstimatorParameters from optuna.search_space import IntersectionSearchSpace from optuna.search_space.group_decomposed import _GroupDecomposedSearchSpace from optuna.search_space.group_decomposed import _SearchSpaceGroup from optuna.study._multi_objective import _fast_non_domination_rank from optuna.study._study_direction import StudyDirection from optuna.trial import FrozenTrial from optuna.trial import TrialState if TYPE_CHECKING: from optuna.study import Study EPS = 1e-12 _logger = get_logger(__name__) def default_gamma(x: int) -> int: return min(int(np.ceil(0.1 * x)), 25) def hyperopt_default_gamma(x: int) -> int: return min(int(np.ceil(0.25 * np.sqrt(x))), 25) def default_weights(x: int) -> np.ndarray: if x == 0: return np.asarray([]) elif x < 25: return np.ones(x) else: ramp = np.linspace(1.0 / x, 1.0, num=x - 25) flat = np.ones(25) return np.concatenate([ramp, flat], axis=0) class TPESampler(BaseSampler): """Sampler using TPE (Tree-structured Parzen Estimator) algorithm. On each trial, for each parameter, TPE fits one Gaussian Mixture Model (GMM) ``l(x)`` to the set of parameter values associated with the best objective values, and another GMM ``g(x)`` to the remaining parameter values. It chooses the parameter value ``x`` that maximizes the ratio ``l(x)/g(x)``. For further information about TPE algorithm, please refer to the following papers: - `Algorithms for Hyper-Parameter Optimization `__ - `Making a Science of Model Search: Hyperparameter Optimization in Hundreds of Dimensions for Vision Architectures `__ - `Tree-Structured Parzen Estimator: Understanding Its Algorithm Components and Their Roles for Better Empirical Performance `__ For multi-objective TPE (MOTPE), please refer to the following papers: - `Multiobjective Tree-Structured Parzen Estimator for Computationally Expensive Optimization Problems `__ - `Multiobjective Tree-Structured Parzen Estimator `__ Please also check our articles: - `Significant Speed Up of Multi-Objective TPESampler in Optuna v4.0.0 `__ - `Multivariate TPE Makes Optuna Even More Powerful `__ Example: An example of a single-objective optimization is as follows: .. testcode:: import optuna from optuna.samplers import TPESampler def objective(trial): x = trial.suggest_float("x", -10, 10) return x**2 study = optuna.create_study(sampler=TPESampler()) study.optimize(objective, n_trials=10) .. note:: :class:`~optuna.samplers.TPESampler`, which became much faster in v4.0.0, c.f. `our article `__, can handle multi-objective optimization with many trials as well. Please note that :class:`~optuna.samplers.NSGAIISampler` will be used by default for multi-objective optimization, so if users would like to use :class:`~optuna.samplers.TPESampler` for multi-objective optimization, ``sampler`` must be explicitly specified when study is created. Args: consider_prior: Enhance the stability of Parzen estimator by imposing a Gaussian prior when :obj:`True`. The prior is only effective if the sampling distribution is either :class:`~optuna.distributions.FloatDistribution`, or :class:`~optuna.distributions.IntDistribution`. prior_weight: The weight of the prior. This argument is used in :class:`~optuna.distributions.FloatDistribution`, :class:`~optuna.distributions.IntDistribution`, and :class:`~optuna.distributions.CategoricalDistribution`. consider_magic_clip: Enable a heuristic to limit the smallest variances of Gaussians used in the Parzen estimator. consider_endpoints: Take endpoints of domains into account when calculating variances of Gaussians in Parzen estimator. See the original paper for details on the heuristics to calculate the variances. n_startup_trials: The random sampling is used instead of the TPE algorithm until the given number of trials finish in the same study. n_ei_candidates: Number of candidate samples used to calculate the expected improvement. gamma: A function that takes the number of finished trials and returns the number of trials to form a density function for samples with low grains. See the original paper for more details. weights: A function that takes the number of finished trials and returns a weight for them. See `Making a Science of Model Search: Hyperparameter Optimization in Hundreds of Dimensions for Vision Architectures `__ for more details. .. note:: In the multi-objective case, this argument is only used to compute the weights of bad trials, i.e., trials to construct `g(x)` in the `paper `__ ). The weights of good trials, i.e., trials to construct `l(x)`, are computed by a rule based on the hypervolume contribution proposed in the `paper of MOTPE `__. seed: Seed for random number generator. multivariate: If this is :obj:`True`, the multivariate TPE is used when suggesting parameters. The multivariate TPE is reported to outperform the independent TPE. See `BOHB: Robust and Efficient Hyperparameter Optimization at Scale `__ and `our article `__ for more details. .. note:: Added in v2.2.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v2.2.0. group: If this and ``multivariate`` are :obj:`True`, the multivariate TPE with the group decomposed search space is used when suggesting parameters. The sampling algorithm decomposes the search space based on past trials and samples from the joint distribution in each decomposed subspace. The decomposed subspaces are a partition of the whole search space. Each subspace is a maximal subset of the whole search space, which satisfies the following: for a trial in completed trials, the intersection of the subspace and the search space of the trial becomes subspace itself or an empty set. Sampling from the joint distribution on the subspace is realized by multivariate TPE. If ``group`` is :obj:`True`, ``multivariate`` must be :obj:`True` as well. .. note:: Added in v2.8.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v2.8.0. Example: .. testcode:: import optuna def objective(trial): x = trial.suggest_categorical("x", ["A", "B"]) if x == "A": return trial.suggest_float("y", -10, 10) else: return trial.suggest_int("z", -10, 10) sampler = optuna.samplers.TPESampler(multivariate=True, group=True) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=10) warn_independent_sampling: If this is :obj:`True` and ``multivariate=True``, a warning message is emitted when the value of a parameter is sampled by using an independent sampler. If ``multivariate=False``, this flag has no effect. constant_liar: If :obj:`True`, penalize running trials to avoid suggesting parameter configurations nearby. .. note:: Abnormally terminated trials often leave behind a record with a state of ``RUNNING`` in the storage. Such "zombie" trial parameters will be avoided by the constant liar algorithm during subsequent sampling. When using an :class:`~optuna.storages.RDBStorage`, it is possible to enable the ``heartbeat_interval`` to change the records for abnormally terminated trials to ``FAIL``. .. note:: It is recommended to set this value to :obj:`True` during distributed optimization to avoid having multiple workers evaluating similar parameter configurations. In particular, if each objective function evaluation is costly and the durations of the running states are significant, and/or the number of workers is high. .. note:: Added in v2.8.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v2.8.0. constraints_func: An optional function that computes the objective constraints. It must take a :class:`~optuna.trial.FrozenTrial` and return the constraints. The return value must be a sequence of :obj:`float` s. A value strictly larger than 0 means that a constraints is violated. A value equal to or smaller than 0 is considered feasible. If ``constraints_func`` returns more than one value for a trial, that trial is considered feasible if and only if all values are equal to 0 or smaller. The ``constraints_func`` will be evaluated after each successful trial. The function won't be called when trials fail or they are pruned, but this behavior is subject to change in the future releases. .. note:: Added in v3.0.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v3.0.0. categorical_distance_func: A dictionary of distance functions for categorical parameters. The key is the name of the categorical parameter and the value is a distance function that takes two :class:`~optuna.distributions.CategoricalChoiceType` s and returns a :obj:`float` value. The distance function must return a non-negative value. While categorical choices are handled equally by default, this option allows users to specify prior knowledge on the structure of categorical parameters. When specified, categorical choices closer to current best choices are more likely to be sampled. .. note:: Added in v3.4.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v3.4.0. """ def __init__( self, consider_prior: bool = True, prior_weight: float = 1.0, consider_magic_clip: bool = True, consider_endpoints: bool = False, n_startup_trials: int = 10, n_ei_candidates: int = 24, gamma: Callable[[int], int] = default_gamma, weights: Callable[[int], np.ndarray] = default_weights, seed: int | None = None, *, multivariate: bool = False, group: bool = False, warn_independent_sampling: bool = True, constant_liar: bool = False, constraints_func: Callable[[FrozenTrial], Sequence[float]] | None = None, categorical_distance_func: ( dict[str, Callable[[CategoricalChoiceType, CategoricalChoiceType], float]] | None ) = None, ) -> None: self._parzen_estimator_parameters = _ParzenEstimatorParameters( consider_prior, prior_weight, consider_magic_clip, consider_endpoints, weights, multivariate, categorical_distance_func or {}, ) self._n_startup_trials = n_startup_trials self._n_ei_candidates = n_ei_candidates self._gamma = gamma self._warn_independent_sampling = warn_independent_sampling self._rng = LazyRandomState(seed) self._random_sampler = RandomSampler(seed=seed) self._multivariate = multivariate self._group = group self._group_decomposed_search_space: _GroupDecomposedSearchSpace | None = None self._search_space_group: _SearchSpaceGroup | None = None self._search_space = IntersectionSearchSpace(include_pruned=True) self._constant_liar = constant_liar self._constraints_func = constraints_func # NOTE(nabenabe0928): Users can overwrite _ParzenEstimator to customize the TPE behavior. self._parzen_estimator_cls = _ParzenEstimator if multivariate: warn_experimental_argument("multivariate") if group: if not multivariate: raise ValueError( "``group`` option can only be enabled when ``multivariate`` is enabled." ) warn_experimental_argument("group") self._group_decomposed_search_space = _GroupDecomposedSearchSpace(True) if constant_liar: warn_experimental_argument("constant_liar") if constraints_func is not None: warn_experimental_argument("constraints_func") if categorical_distance_func is not None: warn_experimental_argument("categorical_distance_func") def reseed_rng(self) -> None: self._rng.rng.seed() self._random_sampler.reseed_rng() def infer_relative_search_space( self, study: Study, trial: FrozenTrial ) -> dict[str, BaseDistribution]: if not self._multivariate: return {} search_space: dict[str, BaseDistribution] = {} if self._group: assert self._group_decomposed_search_space is not None self._search_space_group = self._group_decomposed_search_space.calculate(study) for sub_space in self._search_space_group.search_spaces: # Sort keys because Python's string hashing is nondeterministic. for name, distribution in sorted(sub_space.items()): if distribution.single(): continue search_space[name] = distribution return search_space for name, distribution in self._search_space.calculate(study).items(): if distribution.single(): continue search_space[name] = distribution return search_space def sample_relative( self, study: Study, trial: FrozenTrial, search_space: dict[str, BaseDistribution] ) -> dict[str, Any]: if self._group: assert self._search_space_group is not None params = {} for sub_space in self._search_space_group.search_spaces: search_space = {} # Sort keys because Python's string hashing is nondeterministic. for name, distribution in sorted(sub_space.items()): if not distribution.single(): search_space[name] = distribution params.update(self._sample_relative(study, trial, search_space)) return params else: return self._sample_relative(study, trial, search_space) def _sample_relative( self, study: Study, trial: FrozenTrial, search_space: dict[str, BaseDistribution] ) -> dict[str, Any]: if search_space == {}: return {} states = (TrialState.COMPLETE, TrialState.PRUNED) trials = study._get_trials(deepcopy=False, states=states, use_cache=True) # If the number of samples is insufficient, we run random trial. if len(trials) < self._n_startup_trials: return {} return self._sample(study, trial, search_space) def sample_independent( self, study: Study, trial: FrozenTrial, param_name: str, param_distribution: BaseDistribution, ) -> Any: states = (TrialState.COMPLETE, TrialState.PRUNED) trials = study._get_trials(deepcopy=False, states=states, use_cache=True) # If the number of samples is insufficient, we run random trial. if len(trials) < self._n_startup_trials: return self._random_sampler.sample_independent( study, trial, param_name, param_distribution ) if self._warn_independent_sampling and self._multivariate: # Avoid independent warning at the first sampling of `param_name`. if any(param_name in trial.params for trial in trials): _logger.warning( f"The parameter '{param_name}' in trial#{trial.number} is sampled " "independently instead of being sampled by multivariate TPE sampler. " "(optimization performance may be degraded). " "You can suppress this warning by setting `warn_independent_sampling` " "to `False` in the constructor of `TPESampler`, " "if this independent sampling is intended behavior." ) return self._sample(study, trial, {param_name: param_distribution})[param_name] def _get_internal_repr( self, trials: list[FrozenTrial], search_space: dict[str, BaseDistribution] ) -> dict[str, np.ndarray]: values: dict[str, list[float]] = {param_name: [] for param_name in search_space} for trial in trials: if all((param_name in trial.params) for param_name in search_space): for param_name in search_space: param = trial.params[param_name] distribution = trial.distributions[param_name] values[param_name].append(distribution.to_internal_repr(param)) return {k: np.asarray(v) for k, v in values.items()} def _sample( self, study: Study, trial: FrozenTrial, search_space: dict[str, BaseDistribution] ) -> dict[str, Any]: if self._constant_liar: states = [TrialState.COMPLETE, TrialState.PRUNED, TrialState.RUNNING] else: states = [TrialState.COMPLETE, TrialState.PRUNED] use_cache = not self._constant_liar trials = study._get_trials(deepcopy=False, states=states, use_cache=use_cache) # We divide data into below and above. n = sum(trial.state != TrialState.RUNNING for trial in trials) # Ignore running trials. below_trials, above_trials = _split_trials( study, trials, self._gamma(n), self._constraints_func is not None, ) mpe_below = self._build_parzen_estimator( study, search_space, below_trials, handle_below=True ) mpe_above = self._build_parzen_estimator( study, search_space, above_trials, handle_below=False ) samples_below = mpe_below.sample(self._rng.rng, self._n_ei_candidates) acq_func_vals = self._compute_acquisition_func(samples_below, mpe_below, mpe_above) ret = TPESampler._compare(samples_below, acq_func_vals) for param_name, dist in search_space.items(): ret[param_name] = dist.to_external_repr(ret[param_name]) return ret def _build_parzen_estimator( self, study: Study, search_space: dict[str, BaseDistribution], trials: list[FrozenTrial], handle_below: bool, ) -> _ParzenEstimator: observations = self._get_internal_repr(trials, search_space) if handle_below and study._is_multi_objective(): param_mask_below = [] for trial in trials: param_mask_below.append( all((param_name in trial.params) for param_name in search_space) ) weights_below = _calculate_weights_below_for_multi_objective( study, trials, self._constraints_func )[param_mask_below] assert np.isfinite(weights_below).all() mpe = self._parzen_estimator_cls( observations, search_space, self._parzen_estimator_parameters, weights_below ) else: mpe = self._parzen_estimator_cls( observations, search_space, self._parzen_estimator_parameters ) if not isinstance(mpe, _ParzenEstimator): raise RuntimeError("_parzen_estimator_cls must override _ParzenEstimator.") return mpe def _compute_acquisition_func( self, samples: dict[str, np.ndarray], mpe_below: _ParzenEstimator, mpe_above: _ParzenEstimator, ) -> np.ndarray: log_likelihoods_below = mpe_below.log_pdf(samples) log_likelihoods_above = mpe_above.log_pdf(samples) acq_func_vals = log_likelihoods_below - log_likelihoods_above return acq_func_vals @classmethod def _compare( cls, samples: dict[str, np.ndarray], acquisition_func_vals: np.ndarray ) -> dict[str, int | float]: sample_size = next(iter(samples.values())).size if sample_size == 0: raise ValueError(f"The size of `samples` must be positive, but got {sample_size}.") if sample_size != acquisition_func_vals.size: raise ValueError( "The sizes of `samples` and `acquisition_func_vals` must be same, but got " "(samples.size, acquisition_func_vals.size) = " f"({sample_size}, {acquisition_func_vals.size})." ) best_idx = np.argmax(acquisition_func_vals) return {k: v[best_idx].item() for k, v in samples.items()} @staticmethod def hyperopt_parameters() -> dict[str, Any]: """Return the the default parameters of hyperopt (v0.1.2). :class:`~optuna.samplers.TPESampler` can be instantiated with the parameters returned by this method. Example: Create a :class:`~optuna.samplers.TPESampler` instance with the default parameters of `hyperopt `__. .. testcode:: import optuna from optuna.samplers import TPESampler def objective(trial): x = trial.suggest_float("x", -10, 10) return x**2 sampler = TPESampler(**TPESampler.hyperopt_parameters()) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=10) Returns: A dictionary containing the default parameters of hyperopt. """ return { "consider_prior": True, "prior_weight": 1.0, "consider_magic_clip": True, "consider_endpoints": False, "n_startup_trials": 20, "n_ei_candidates": 24, "gamma": hyperopt_default_gamma, "weights": default_weights, } def before_trial(self, study: Study, trial: FrozenTrial) -> None: self._random_sampler.before_trial(study, trial) def after_trial( self, study: Study, trial: FrozenTrial, state: TrialState, values: Sequence[float] | None, ) -> None: assert state in [TrialState.COMPLETE, TrialState.FAIL, TrialState.PRUNED] if self._constraints_func is not None: _process_constraints_after_trial(self._constraints_func, study, trial, state) self._random_sampler.after_trial(study, trial, state, values) def _split_trials( study: Study, trials: list[FrozenTrial], n_below: int, constraints_enabled: bool ) -> tuple[list[FrozenTrial], list[FrozenTrial]]: complete_trials = [] pruned_trials = [] running_trials = [] infeasible_trials = [] for trial in trials: if trial.state == TrialState.RUNNING: # We should check if the trial is RUNNING before the feasibility check # because its constraint values have not yet been set. running_trials.append(trial) elif constraints_enabled and _get_infeasible_trial_score(trial) > 0: infeasible_trials.append(trial) elif trial.state == TrialState.COMPLETE: complete_trials.append(trial) elif trial.state == TrialState.PRUNED: pruned_trials.append(trial) else: assert False # We divide data into below and above. below_complete, above_complete = _split_complete_trials(complete_trials, study, n_below) # This ensures `n_below` is non-negative to prevent unexpected trial splits. n_below = max(0, n_below - len(below_complete)) below_pruned, above_pruned = _split_pruned_trials(pruned_trials, study, n_below) # This ensures `n_below` is non-negative to prevent unexpected trial splits. n_below = max(0, n_below - len(below_pruned)) below_infeasible, above_infeasible = _split_infeasible_trials(infeasible_trials, n_below) below_trials = below_complete + below_pruned + below_infeasible above_trials = above_complete + above_pruned + above_infeasible + running_trials below_trials.sort(key=lambda trial: trial.number) above_trials.sort(key=lambda trial: trial.number) return below_trials, above_trials def _split_complete_trials( trials: Sequence[FrozenTrial], study: Study, n_below: int ) -> tuple[list[FrozenTrial], list[FrozenTrial]]: n_below = min(n_below, len(trials)) if len(study.directions) <= 1: return _split_complete_trials_single_objective(trials, study, n_below) else: return _split_complete_trials_multi_objective(trials, study, n_below) def _split_complete_trials_single_objective( trials: Sequence[FrozenTrial], study: Study, n_below: int ) -> tuple[list[FrozenTrial], list[FrozenTrial]]: if study.direction == StudyDirection.MINIMIZE: sorted_trials = sorted(trials, key=lambda trial: cast(float, trial.value)) else: sorted_trials = sorted(trials, key=lambda trial: cast(float, trial.value), reverse=True) return sorted_trials[:n_below], sorted_trials[n_below:] def _split_complete_trials_multi_objective( trials: Sequence[FrozenTrial], study: Study, n_below: int ) -> tuple[list[FrozenTrial], list[FrozenTrial]]: if n_below == 0: # The type of trials must be `list`, but not `Sequence`. return [], list(trials) lvals = np.array([trial.values for trial in trials]) lvals *= np.array([-1.0 if d == StudyDirection.MAXIMIZE else 1.0 for d in study.directions]) # Solving HSSP for variables number of times is a waste of time. nondomination_ranks = _fast_non_domination_rank(lvals, n_below=n_below) assert 0 <= n_below <= len(lvals) indices = np.array(range(len(lvals))) indices_below = np.empty(n_below, dtype=int) # Nondomination rank-based selection i = 0 last_idx = 0 while last_idx < n_below and last_idx + sum(nondomination_ranks == i) <= n_below: length = indices[nondomination_ranks == i].shape[0] indices_below[last_idx : last_idx + length] = indices[nondomination_ranks == i] last_idx += length i += 1 # Hypervolume subset selection problem (HSSP)-based selection subset_size = n_below - last_idx if subset_size > 0: rank_i_lvals = lvals[nondomination_ranks == i] rank_i_indices = indices[nondomination_ranks == i] worst_point = np.max(rank_i_lvals, axis=0) reference_point = np.maximum(1.1 * worst_point, 0.9 * worst_point) reference_point[reference_point == 0] = EPS selected_indices = _solve_hssp(rank_i_lvals, rank_i_indices, subset_size, reference_point) indices_below[last_idx:] = selected_indices below_trials = [] above_trials = [] for index in range(len(trials)): if index in indices_below: below_trials.append(trials[index]) else: above_trials.append(trials[index]) return below_trials, above_trials def _get_pruned_trial_score(trial: FrozenTrial, study: Study) -> tuple[float, float]: if len(trial.intermediate_values) > 0: step, intermediate_value = max(trial.intermediate_values.items()) if math.isnan(intermediate_value): return -step, float("inf") elif study.direction == StudyDirection.MINIMIZE: return -step, intermediate_value else: return -step, -intermediate_value else: return 1, 0.0 def _split_pruned_trials( trials: Sequence[FrozenTrial], study: Study, n_below: int ) -> tuple[list[FrozenTrial], list[FrozenTrial]]: n_below = min(n_below, len(trials)) sorted_trials = sorted(trials, key=lambda trial: _get_pruned_trial_score(trial, study)) return sorted_trials[:n_below], sorted_trials[n_below:] def _get_infeasible_trial_score(trial: FrozenTrial) -> float: constraint = trial.system_attrs.get(_CONSTRAINTS_KEY) if constraint is None: warnings.warn( f"Trial {trial.number} does not have constraint values." " It will be treated as a lower priority than other trials." ) return float("inf") else: # Violation values of infeasible dimensions are summed up. return sum(v for v in constraint if v > 0) def _split_infeasible_trials( trials: Sequence[FrozenTrial], n_below: int ) -> tuple[list[FrozenTrial], list[FrozenTrial]]: n_below = min(n_below, len(trials)) sorted_trials = sorted(trials, key=_get_infeasible_trial_score) return sorted_trials[:n_below], sorted_trials[n_below:] def _calculate_weights_below_for_multi_objective( study: Study, below_trials: list[FrozenTrial], constraints_func: Callable[[FrozenTrial], Sequence[float]] | None, ) -> np.ndarray: loss_vals = [] feasible_mask = np.ones(len(below_trials), dtype=bool) for i, trial in enumerate(below_trials): # Hypervolume contributions are calculated only using feasible trials. if constraints_func is not None: if any(constraint > 0 for constraint in constraints_func(trial)): feasible_mask[i] = False continue values = [] for value, direction in zip(trial.values, study.directions): if direction == StudyDirection.MINIMIZE: values.append(value) else: values.append(-value) loss_vals.append(values) lvals = np.asarray(loss_vals, dtype=float) # Calculate weights based on hypervolume contributions. n_below = len(lvals) weights_below: np.ndarray if n_below == 0: weights_below = np.asarray([]) elif n_below == 1: weights_below = np.asarray([1.0]) else: worst_point = np.max(lvals, axis=0) reference_point = np.maximum(1.1 * worst_point, 0.9 * worst_point) reference_point[reference_point == 0] = EPS hv = compute_hypervolume(lvals, reference_point) indices_mat = ~np.eye(n_below).astype(bool) contributions = np.asarray( [ hv - compute_hypervolume(lvals[indices_mat[i]], reference_point) for i in range(n_below) ] ) contributions[np.isnan(contributions)] = np.inf max_contribution = np.maximum(np.max(contributions), EPS) if not np.isfinite(max_contribution): weights_below = np.ones_like(contributions, dtype=float) # TODO(nabenabe0928): Make the weights for non Pareto solutions to zero. weights_below[np.isfinite(contributions)] = EPS else: weights_below = np.clip(contributions / max_contribution, EPS, 1) # For now, EPS weight is assigned to infeasible trials. weights_below_all = np.full(len(below_trials), EPS) weights_below_all[feasible_mask] = weights_below return weights_below_all optuna-4.1.0/optuna/samplers/nsgaii/000077500000000000000000000000001471332314300174315ustar00rootroot00000000000000optuna-4.1.0/optuna/samplers/nsgaii/__init__.py000066400000000000000000000012071471332314300215420ustar00rootroot00000000000000from optuna.samplers.nsgaii._crossovers._base import BaseCrossover from optuna.samplers.nsgaii._crossovers._blxalpha import BLXAlphaCrossover from optuna.samplers.nsgaii._crossovers._sbx import SBXCrossover from optuna.samplers.nsgaii._crossovers._spx import SPXCrossover from optuna.samplers.nsgaii._crossovers._undx import UNDXCrossover from optuna.samplers.nsgaii._crossovers._uniform import UniformCrossover from optuna.samplers.nsgaii._crossovers._vsbx import VSBXCrossover __all__ = [ "BaseCrossover", "BLXAlphaCrossover", "SBXCrossover", "SPXCrossover", "UNDXCrossover", "UniformCrossover", "VSBXCrossover", ] optuna-4.1.0/optuna/samplers/nsgaii/_after_trial_strategy.py000066400000000000000000000021421471332314300243570ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Callable from collections.abc import Sequence from typing import TYPE_CHECKING from optuna.samplers._base import _process_constraints_after_trial from optuna.trial import FrozenTrial from optuna.trial import TrialState if TYPE_CHECKING: from optuna.study import Study class NSGAIIAfterTrialStrategy: def __init__( self, *, constraints_func: Callable[[FrozenTrial], Sequence[float]] | None = None ) -> None: self._constraints_func = constraints_func def __call__( self, study: Study, trial: FrozenTrial, state: TrialState, values: Sequence[float] | None = None, ) -> None: """Carry out the after trial process of default NSGA-II. This method is called after each trial of the study, examines whether the trial result is valid in terms of constraints, and store the results in system_attrs of the study. """ if self._constraints_func is not None: _process_constraints_after_trial(self._constraints_func, study, trial, state) optuna-4.1.0/optuna/samplers/nsgaii/_child_generation_strategy.py000066400000000000000000000074011471332314300253640ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Callable from collections.abc import Sequence from typing import Any from typing import TYPE_CHECKING from optuna.distributions import BaseDistribution from optuna.samplers._lazy_random_state import LazyRandomState from optuna.samplers.nsgaii._constraints_evaluation import _constrained_dominates from optuna.samplers.nsgaii._crossover import perform_crossover from optuna.samplers.nsgaii._crossovers._base import BaseCrossover from optuna.study._multi_objective import _dominates from optuna.trial import FrozenTrial if TYPE_CHECKING: from optuna.study import Study class NSGAIIChildGenerationStrategy: def __init__( self, *, mutation_prob: float | None = None, crossover: BaseCrossover, crossover_prob: float, swapping_prob: float, constraints_func: Callable[[FrozenTrial], Sequence[float]] | None = None, rng: LazyRandomState, ) -> None: if not (mutation_prob is None or 0.0 <= mutation_prob <= 1.0): raise ValueError( "`mutation_prob` must be None or a float value within the range [0.0, 1.0]." ) if not (0.0 <= crossover_prob <= 1.0): raise ValueError("`crossover_prob` must be a float value within the range [0.0, 1.0].") if not (0.0 <= swapping_prob <= 1.0): raise ValueError("`swapping_prob` must be a float value within the range [0.0, 1.0].") if not isinstance(crossover, BaseCrossover): raise ValueError( f"'{crossover}' is not a valid crossover." " For valid crossovers see" " https://optuna.readthedocs.io/en/stable/reference/samplers.html." ) self._crossover_prob = crossover_prob self._mutation_prob = mutation_prob self._swapping_prob = swapping_prob self._crossover = crossover self._constraints_func = constraints_func self._rng = rng def __call__( self, study: Study, search_space: dict[str, BaseDistribution], parent_population: list[FrozenTrial], ) -> dict[str, Any]: """Generate a child parameter from the given parent population by NSGA-II algorithm. Args: study: Target study object. search_space: A dictionary containing the parameter names and parameter's distributions. parent_population: A list of trials that are selected as parent population. Returns: A dictionary containing the parameter names and parameter's values. """ dominates = _dominates if self._constraints_func is None else _constrained_dominates # We choose a child based on the specified crossover method. if self._rng.rng.rand() < self._crossover_prob: child_params = perform_crossover( self._crossover, study, parent_population, search_space, self._rng.rng, self._swapping_prob, dominates, ) else: parent_population_size = len(parent_population) parent_params = parent_population[self._rng.rng.choice(parent_population_size)].params child_params = {name: parent_params[name] for name in search_space.keys()} n_params = len(child_params) if self._mutation_prob is None: mutation_prob = 1.0 / max(1.0, n_params) else: mutation_prob = self._mutation_prob params = {} for param_name in child_params.keys(): if self._rng.rng.rand() >= mutation_prob: params[param_name] = child_params[param_name] return params optuna-4.1.0/optuna/samplers/nsgaii/_constraints_evaluation.py000066400000000000000000000103751471332314300247460ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Sequence import warnings import numpy as np from optuna.samplers._base import _CONSTRAINTS_KEY from optuna.study import StudyDirection from optuna.study._multi_objective import _dominates from optuna.trial import FrozenTrial from optuna.trial import TrialState def _constrained_dominates( trial0: FrozenTrial, trial1: FrozenTrial, directions: Sequence[StudyDirection] ) -> bool: """Checks constrained-domination. A trial x is said to constrained-dominate a trial y, if any of the following conditions is true: 1) Trial x is feasible and trial y is not. 2) Trial x and y are both infeasible, but solution x has a smaller overall constraint violation. 3) Trial x and y are feasible and trial x dominates trial y. """ constraints0 = trial0.system_attrs.get(_CONSTRAINTS_KEY) constraints1 = trial1.system_attrs.get(_CONSTRAINTS_KEY) if constraints0 is None: warnings.warn( f"Trial {trial0.number} does not have constraint values." " It will be dominated by the other trials." ) if constraints1 is None: warnings.warn( f"Trial {trial1.number} does not have constraint values." " It will be dominated by the other trials." ) if constraints0 is None and constraints1 is None: # Neither Trial x nor y has constraints values return _dominates(trial0, trial1, directions) if constraints0 is not None and constraints1 is None: # Trial x has constraint values, but y doesn't. return True if constraints0 is None and constraints1 is not None: # If Trial y has constraint values, but x doesn't. return False assert isinstance(constraints0, (list, tuple)) assert isinstance(constraints1, (list, tuple)) if len(constraints0) != len(constraints1): raise ValueError("Trials with different numbers of constraints cannot be compared.") if trial0.state != TrialState.COMPLETE: return False if trial1.state != TrialState.COMPLETE: return True satisfy_constraints0 = all(v <= 0 for v in constraints0) satisfy_constraints1 = all(v <= 0 for v in constraints1) if satisfy_constraints0 and satisfy_constraints1: # Both trials satisfy the constraints. return _dominates(trial0, trial1, directions) if satisfy_constraints0: # trial0 satisfies the constraints, but trial1 violates them. return True if satisfy_constraints1: # trial1 satisfies the constraints, but trial0 violates them. return False # Both trials violate the constraints. violation0 = sum(v for v in constraints0 if v > 0) violation1 = sum(v for v in constraints1 if v > 0) return violation0 < violation1 def _evaluate_penalty(population: Sequence[FrozenTrial]) -> np.ndarray: """Evaluate feasibility of trials in population. Returns: A list of feasibility status T/F/None of trials in population, where T/F means feasible/infeasible and None means that the trial does not have constraint values. """ penalty: list[float] = [] for trial in population: constraints = trial.system_attrs.get(_CONSTRAINTS_KEY) if constraints is None: penalty.append(np.nan) else: penalty.append(sum(v for v in constraints if v > 0)) return np.array(penalty) def _validate_constraints( population: list[FrozenTrial], *, is_constrained: bool = False, ) -> None: if not is_constrained: return num_constraints = max( [len(t.system_attrs.get(_CONSTRAINTS_KEY, [])) for t in population], default=0 ) for _trial in population: _constraints = _trial.system_attrs.get(_CONSTRAINTS_KEY) if _constraints is None: warnings.warn( f"Trial {_trial.number} does not have constraint values." " It will be dominated by the other trials." ) continue if np.any(np.isnan(np.array(_constraints))): raise ValueError("NaN is not acceptable as constraint value.") elif len(_constraints) != num_constraints: raise ValueError("Trials with different numbers of constraints cannot be compared.") optuna-4.1.0/optuna/samplers/nsgaii/_crossover.py000066400000000000000000000135541471332314300221770ustar00rootroot00000000000000from __future__ import annotations from typing import Any from typing import Callable from typing import Dict from typing import List from typing import Optional from typing import Sequence from typing import TYPE_CHECKING import numpy as np from optuna._transform import _SearchSpaceTransform from optuna.distributions import BaseDistribution from optuna.distributions import FloatDistribution from optuna.distributions import IntDistribution from optuna.samplers.nsgaii._crossovers._base import BaseCrossover from optuna.study import StudyDirection from optuna.trial import FrozenTrial if TYPE_CHECKING: from optuna.study import Study _NUMERICAL_DISTRIBUTIONS = ( FloatDistribution, IntDistribution, ) def _try_crossover( parents: List[FrozenTrial], crossover: BaseCrossover, study: Study, rng: np.random.RandomState, swapping_prob: float, categorical_search_space: Dict[str, BaseDistribution], numerical_search_space: Dict[str, BaseDistribution], numerical_transform: Optional[_SearchSpaceTransform], ) -> Dict[str, Any]: child_params: Dict[str, Any] = {} if len(categorical_search_space) > 0: parents_categorical_params = np.array( [ [parent.params[p] for p in categorical_search_space] for parent in [parents[0], parents[-1]] ], dtype=object, ) child_categorical_array = _inlined_categorical_uniform_crossover( parents_categorical_params, rng, swapping_prob, categorical_search_space ) child_categorical_params = { param: value for param, value in zip(categorical_search_space, child_categorical_array) } child_params.update(child_categorical_params) if numerical_transform is None: return child_params # The following is applied only for numerical parameters. parents_numerical_params = np.stack( [ numerical_transform.transform( { param_key: parent.params[param_key] for param_key in numerical_search_space.keys() } ) for parent in parents ] ) # Parent individual with NUMERICAL_DISTRIBUTIONS parameter. child_numerical_array = crossover.crossover( parents_numerical_params, rng, study, numerical_transform.bounds ) child_numerical_params = numerical_transform.untransform(child_numerical_array) child_params.update(child_numerical_params) return child_params def perform_crossover( crossover: BaseCrossover, study: Study, parent_population: Sequence[FrozenTrial], search_space: Dict[str, BaseDistribution], rng: np.random.RandomState, swapping_prob: float, dominates: Callable[[FrozenTrial, FrozenTrial, Sequence[StudyDirection]], bool], ) -> Dict[str, Any]: numerical_search_space: Dict[str, BaseDistribution] = {} categorical_search_space: Dict[str, BaseDistribution] = {} for key, value in search_space.items(): if isinstance(value, _NUMERICAL_DISTRIBUTIONS): numerical_search_space[key] = value else: categorical_search_space[key] = value numerical_transform: Optional[_SearchSpaceTransform] = None if len(numerical_search_space) != 0: numerical_transform = _SearchSpaceTransform(numerical_search_space) while True: # Repeat while parameters lie outside search space boundaries. parents = _select_parents(crossover, study, parent_population, rng, dominates) child_params = _try_crossover( parents, crossover, study, rng, swapping_prob, categorical_search_space, numerical_search_space, numerical_transform, ) if _is_contained(child_params, search_space): break return child_params def _select_parents( crossover: BaseCrossover, study: Study, parent_population: Sequence[FrozenTrial], rng: np.random.RandomState, dominates: Callable[[FrozenTrial, FrozenTrial, Sequence[StudyDirection]], bool], ) -> List[FrozenTrial]: parents: List[FrozenTrial] = [] for _ in range(crossover.n_parents): parent = _select_parent( study, [t for t in parent_population if t not in parents], rng, dominates ) parents.append(parent) return parents def _select_parent( study: Study, parent_population: Sequence[FrozenTrial], rng: np.random.RandomState, dominates: Callable[[FrozenTrial, FrozenTrial, Sequence[StudyDirection]], bool], ) -> FrozenTrial: population_size = len(parent_population) candidate0 = parent_population[rng.choice(population_size)] candidate1 = parent_population[rng.choice(population_size)] # TODO(ohta): Consider crowding distance. if dominates(candidate0, candidate1, study.directions): return candidate0 else: return candidate1 def _is_contained(params: Dict[str, Any], search_space: Dict[str, BaseDistribution]) -> bool: for param_name in params.keys(): param, param_distribution = params[param_name], search_space[param_name] if not param_distribution._contains(param_distribution.to_internal_repr(param)): return False return True def _inlined_categorical_uniform_crossover( parent_params: np.ndarray, rng: np.random.RandomState, swapping_prob: float, search_space: Dict[str, BaseDistribution], ) -> np.ndarray: # We can't use uniform crossover implementation of `BaseCrossover` for # parameters from `CategoricalDistribution`, since categorical params are # passed to crossover untransformed, which is not what `BaseCrossover` # implementations expect. n_categorical_params = len(search_space) masks = (rng.rand(n_categorical_params) >= swapping_prob).astype(int) return parent_params[masks, range(n_categorical_params)] optuna-4.1.0/optuna/samplers/nsgaii/_crossovers/000077500000000000000000000000001471332314300220005ustar00rootroot00000000000000optuna-4.1.0/optuna/samplers/nsgaii/_crossovers/__init__.py000066400000000000000000000000001471332314300240770ustar00rootroot00000000000000optuna-4.1.0/optuna/samplers/nsgaii/_crossovers/_base.py000066400000000000000000000040171471332314300234250ustar00rootroot00000000000000from __future__ import annotations import abc from typing import TYPE_CHECKING import numpy as np if TYPE_CHECKING: from optuna.study import Study class BaseCrossover(abc.ABC): """Base class for crossovers. A crossover operation is used by :class:`~optuna.samplers.NSGAIISampler` to create new parameter combination from parameters of ``n`` parent individuals. .. note:: Concrete implementations of this class are expected to only accept parameters from numerical distributions. At the moment, only crossover operation for categorical parameters (uniform crossover) is built-in into :class:`~optuna.samplers.NSGAIISampler`. """ def __str__(self) -> str: return self.__class__.__name__ @property @abc.abstractmethod def n_parents(self) -> int: """Number of parent individuals required to perform crossover.""" raise NotImplementedError @abc.abstractmethod def crossover( self, parents_params: np.ndarray, rng: np.random.RandomState, study: Study, search_space_bounds: np.ndarray, ) -> np.ndarray: """Perform crossover of selected parent individuals. This method is called in :func:`~optuna.samplers.NSGAIISampler.sample_relative`. Args: parents_params: A ``numpy.ndarray`` with dimensions ``num_parents x num_parameters``. Represents a parameter space for each parent individual. This space is continuous for numerical parameters. rng: An instance of ``numpy.random.RandomState``. study: Target study object. search_space_bounds: A ``numpy.ndarray`` with dimensions ``len_search_space x 2`` representing numerical distribution bounds constructed from transformed search space. Returns: A 1-dimensional ``numpy.ndarray`` containing new parameter combination. """ raise NotImplementedError optuna-4.1.0/optuna/samplers/nsgaii/_crossovers/_blxalpha.py000066400000000000000000000032131471332314300243030ustar00rootroot00000000000000from __future__ import annotations from typing import TYPE_CHECKING import numpy as np from optuna._experimental import experimental_class from optuna.samplers.nsgaii._crossovers._base import BaseCrossover if TYPE_CHECKING: from optuna.study import Study @experimental_class("3.0.0") class BLXAlphaCrossover(BaseCrossover): """Blend Crossover operation used by :class:`~optuna.samplers.NSGAIISampler`. Uniformly samples child individuals from the hyper-rectangles created by the two parent individuals. For further information about BLX-alpha crossover, please refer to the following paper: - `Eshelman, L. and J. D. Schaffer. Real-Coded Genetic Algorithms and Interval-Schemata. FOGA (1992). `__ Args: alpha: Parametrizes blend operation. """ n_parents = 2 def __init__(self, alpha: float = 0.5) -> None: self._alpha = alpha def crossover( self, parents_params: np.ndarray, rng: np.random.RandomState, study: Study, search_space_bounds: np.ndarray, ) -> np.ndarray: # https://doi.org/10.1109/CEC.2001.934452 # Section 2 Crossover Operators for RCGA 2.1 Blend Crossover parents_min = parents_params.min(axis=0) parents_max = parents_params.max(axis=0) diff = self._alpha * (parents_max - parents_min) # Equation (1). low = parents_min - diff # Equation (1). high = parents_max + diff # Equation (1). r = rng.rand(len(search_space_bounds)) child_params = (high - low) * r + low return child_params optuna-4.1.0/optuna/samplers/nsgaii/_crossovers/_sbx.py000066400000000000000000000076311471332314300233140ustar00rootroot00000000000000from __future__ import annotations from typing import Optional from typing import TYPE_CHECKING import numpy as np from optuna._experimental import experimental_class from optuna.samplers.nsgaii._crossovers._base import BaseCrossover if TYPE_CHECKING: from optuna.study import Study @experimental_class("3.0.0") class SBXCrossover(BaseCrossover): """Simulated Binary Crossover operation used by :class:`~optuna.samplers.NSGAIISampler`. Generates a child from two parent individuals according to the polynomial probability distribution. - `Deb, K. and R. Agrawal. “Simulated Binary Crossover for Continuous Search Space.” Complex Syst. 9 (1995): n. pag. `__ Args: eta: Distribution index. A small value of ``eta`` allows distant solutions to be selected as children solutions. If not specified, takes default value of ``2`` for single objective functions and ``20`` for multi objective. """ n_parents = 2 def __init__(self, eta: Optional[float] = None) -> None: self._eta = eta def crossover( self, parents_params: np.ndarray, rng: np.random.RandomState, study: Study, search_space_bounds: np.ndarray, ) -> np.ndarray: # https://www.researchgate.net/profile/M-M-Raghuwanshi/publication/267198495_Simulated_Binary_Crossover_with_Lognormal_Distribution/links/5576c78408ae7536375205d7/Simulated-Binary-Crossover-with-Lognormal-Distribution.pdf # Section 2 Simulated Binary Crossover (SBX) # To avoid generating solutions that violate the box constraints, # alpha1, alpha2, xls and xus are introduced, unlike the reference. xls = search_space_bounds[..., 0] xus = search_space_bounds[..., 1] xs_min = np.min(parents_params, axis=0) xs_max = np.max(parents_params, axis=0) if self._eta is None: eta = 20.0 if study._is_multi_objective() else 2.0 else: eta = self._eta xs_diff = np.clip(xs_max - xs_min, 1e-10, None) beta1 = 1 + 2 * (xs_min - xls) / xs_diff beta2 = 1 + 2 * (xus - xs_max) / xs_diff alpha1 = 2 - np.power(beta1, -(eta + 1)) alpha2 = 2 - np.power(beta2, -(eta + 1)) us = rng.rand(len(search_space_bounds)) mask1 = us > 1 / alpha1 # Equation (3). betaq1 = np.power(us * alpha1, 1 / (eta + 1)) # Equation (3). betaq1[mask1] = np.power((1 / (2 - us * alpha1)), 1 / (eta + 1))[mask1] # Equation (3). mask2 = us > 1 / alpha2 # Equation (3). betaq2 = np.power(us * alpha2, 1 / (eta + 1)) # Equation (3) betaq2[mask2] = np.power((1 / (2 - us * alpha2)), 1 / (eta + 1))[mask2] # Equation (3). c1 = 0.5 * ((xs_min + xs_max) - betaq1 * xs_diff) # Equation (4). c2 = 0.5 * ((xs_min + xs_max) + betaq2 * xs_diff) # Equation (5). # SBX applies crossover with establishment 0.5, and with probability 0.5, # the gene of the parent individual is the gene of the child individual. # The original SBX creates two child individuals, # but optuna's implementation creates only one child individual. # Therefore, when there is no crossover, # the gene is selected with equal probability from the parent individuals x1 and x2. child_params_list = [] for c1_i, c2_i, x1_i, x2_i in zip(c1, c2, parents_params[0], parents_params[1]): if rng.rand() < 0.5: if rng.rand() < 0.5: child_params_list.append(c1_i) else: child_params_list.append(c2_i) else: if rng.rand() < 0.5: child_params_list.append(x1_i) else: child_params_list.append(x2_i) child_params = np.array(child_params_list) return child_params optuna-4.1.0/optuna/samplers/nsgaii/_crossovers/_spx.py000066400000000000000000000042561471332314300233320ustar00rootroot00000000000000from __future__ import annotations from typing import Optional from typing import TYPE_CHECKING import numpy as np from optuna._experimental import experimental_class from optuna.samplers.nsgaii._crossovers._base import BaseCrossover if TYPE_CHECKING: from optuna.study import Study @experimental_class("3.0.0") class SPXCrossover(BaseCrossover): """Simplex Crossover operation used by :class:`~optuna.samplers.NSGAIISampler`. Uniformly samples child individuals from within a single simplex that is similar to the simplex produced by the parent individual. For further information about SPX crossover, please refer to the following paper: - `Shigeyoshi Tsutsui and Shigeyoshi Tsutsui and David E. Goldberg and David E. Goldberg and Kumara Sastry and Kumara Sastry Progress Toward Linkage Learning in Real-Coded GAs with Simplex Crossover. IlliGAL Report. 2000. `__ Args: epsilon: Expansion rate. If not specified, defaults to ``sqrt(len(search_space) + 2)``. """ n_parents = 3 def __init__(self, epsilon: Optional[float] = None) -> None: self._epsilon = epsilon def crossover( self, parents_params: np.ndarray, rng: np.random.RandomState, study: Study, search_space_bounds: np.ndarray, ) -> np.ndarray: # https://www.researchgate.net/publication/2388486_Progress_Toward_Linkage_Learning_in_Real-Coded_GAs_with_Simplex_Crossover # Section 2 A Brief Review of SPX n = self.n_parents - 1 G = np.mean(parents_params, axis=0) # Equation (1). rs = np.power(rng.rand(n), 1 / (np.arange(n) + 1)) # Equation (2). epsilon = np.sqrt(len(search_space_bounds) + 2) if self._epsilon is None else self._epsilon xks = [G + epsilon * (pk - G) for pk in parents_params] # Equation (3). ck = 0 # Equation (4). for k in range(1, self.n_parents): ck = rs[k - 1] * (xks[k - 1] - xks[k] + ck) child_params = xks[-1] + ck # Equation (5). return child_params optuna-4.1.0/optuna/samplers/nsgaii/_crossovers/_undx.py000066400000000000000000000101111471332314300234610ustar00rootroot00000000000000from __future__ import annotations from typing import Optional from typing import TYPE_CHECKING import numpy as np from optuna._experimental import experimental_class from optuna.samplers.nsgaii._crossovers._base import BaseCrossover if TYPE_CHECKING: from optuna.study import Study @experimental_class("3.0.0") class UNDXCrossover(BaseCrossover): """Unimodal Normal Distribution Crossover used by :class:`~optuna.samplers.NSGAIISampler`. Generates child individuals from the three parents using a multivariate normal distribution. - `H. Kita, I. Ono and S. Kobayashi, Multi-parental extension of the unimodal normal distribution crossover for real-coded genetic algorithms, Proceedings of the 1999 Congress on Evolutionary Computation-CEC99 (Cat. No. 99TH8406), 1999, pp. 1581-1588 Vol. 2 `__ Args: sigma_xi: Parametrizes normal distribution from which ``xi`` is drawn. sigma_eta: Parametrizes normal distribution from which ``etas`` are drawn. If not specified, defaults to ``0.35 / sqrt(len(search_space))``. """ n_parents = 3 def __init__(self, sigma_xi: float = 0.5, sigma_eta: Optional[float] = None) -> None: self._sigma_xi = sigma_xi self._sigma_eta = sigma_eta def _distance_from_x_to_psl(self, parents_params: np.ndarray) -> np.floating: # The line connecting x1 to x2 is called psl (primary search line). # Compute the 2-norm of the vector orthogonal to psl from x3. e_12 = UNDXCrossover._normalized_x1_to_x2( parents_params ) # Normalized vector from x1 to x2. v_13 = parents_params[2] - parents_params[0] # Vector from x1 to x3. v_12_3 = v_13 - np.dot(v_13, e_12) * e_12 # Vector orthogonal to v_12 through x3. m_12_3 = np.linalg.norm(v_12_3, ord=2) # 2-norm of v_12_3. return m_12_3 def _orthonormal_basis_vector_to_psl(self, parents_params: np.ndarray, n: int) -> np.ndarray: # Compute orthogonal basis vectors for the subspace orthogonal to psl. e_12 = UNDXCrossover._normalized_x1_to_x2( parents_params ) # Normalized vector from x1 to x2. basis_matrix = np.identity(n) if np.count_nonzero(e_12) != 0: basis_matrix[0] = e_12 basis_matrix_t = basis_matrix.T Q, _ = np.linalg.qr(basis_matrix_t) return Q.T[1:] def crossover( self, parents_params: np.ndarray, rng: np.random.RandomState, study: Study, search_space_bounds: np.ndarray, ) -> np.ndarray: # https://doi.org/10.1109/CEC.1999.782672 # Section 2 Unimodal Normal Distribution Crossover n = len(search_space_bounds) xp = (parents_params[0] + parents_params[1]) / 2 # Section 2 (2). d = parents_params[0] - parents_params[1] # Section 2 (3). if self._sigma_eta is None: sigma_eta = 0.35 / np.sqrt(n) else: sigma_eta = self._sigma_eta etas = rng.normal(0, sigma_eta**2, size=n) xi = rng.normal(0, self._sigma_xi**2) es = self._orthonormal_basis_vector_to_psl( parents_params, n ) # Orthonormal basis vectors of the subspace orthogonal to the psl. one = xp # Section 2 (5). two = xi * d # Section 2 (5). if n > 1: # When n=1, there is no subsearch component. three = np.zeros(n) # Section 2 (5). D = self._distance_from_x_to_psl(parents_params) # Section 2 (4). for i in range(n - 1): three += etas[i] * es[i] three *= D child_params = one + two + three else: child_params = one + two return child_params @staticmethod def _normalized_x1_to_x2(parents_params: np.ndarray) -> np.ndarray: # Compute the normalized vector from x1 to x2. v_12 = parents_params[1] - parents_params[0] m_12 = np.linalg.norm(v_12, ord=2) e_12 = v_12 / np.clip(m_12, 1e-10, None) return e_12 optuna-4.1.0/optuna/samplers/nsgaii/_crossovers/_uniform.py000066400000000000000000000033401471332314300241700ustar00rootroot00000000000000from __future__ import annotations from typing import TYPE_CHECKING import numpy as np from optuna.samplers.nsgaii._crossovers._base import BaseCrossover if TYPE_CHECKING: from optuna.study import Study class UniformCrossover(BaseCrossover): """Uniform Crossover operation used by :class:`~optuna.samplers.NSGAIISampler`. Select each parameter with equal probability from the two parent individuals. For further information about uniform crossover, please refer to the following paper: - `Gilbert Syswerda. 1989. Uniform Crossover in Genetic Algorithms. In Proceedings of the 3rd International Conference on Genetic Algorithms. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 2-9. `__ Args: swapping_prob: Probability of swapping each parameter of the parents during crossover. """ n_parents = 2 def __init__(self, swapping_prob: float = 0.5) -> None: if not (0.0 <= swapping_prob <= 1.0): raise ValueError("`swapping_prob` must be a float value within the range [0.0, 1.0].") self._swapping_prob = swapping_prob def crossover( self, parents_params: np.ndarray, rng: np.random.RandomState, study: Study, search_space_bounds: np.ndarray, ) -> np.ndarray: # https://www.researchgate.net/publication/201976488_Uniform_Crossover_in_Genetic_Algorithms # Section 1 Introduction n_params = len(search_space_bounds) masks = (rng.rand(n_params) >= self._swapping_prob).astype(int) child_params = parents_params[masks, range(n_params)] return child_params optuna-4.1.0/optuna/samplers/nsgaii/_crossovers/_vsbx.py000066400000000000000000000063251471332314300235010ustar00rootroot00000000000000from __future__ import annotations from typing import Optional from typing import TYPE_CHECKING import numpy as np from optuna._experimental import experimental_class from optuna.samplers.nsgaii._crossovers._base import BaseCrossover if TYPE_CHECKING: from optuna.study import Study @experimental_class("3.0.0") class VSBXCrossover(BaseCrossover): """Modified Simulated Binary Crossover operation used by :class:`~optuna.samplers.NSGAIISampler`. vSBX generates child individuals without excluding any region of the parameter space, while maintaining the excellent properties of SBX. - `Pedro J. Ballester, Jonathan N. Carter. Real-Parameter Genetic Algorithms for Finding Multiple Optimal Solutions in Multi-modal Optimization. GECCO 2003: 706-717 `__ Args: eta: Distribution index. A small value of ``eta`` allows distant solutions to be selected as children solutions. If not specified, takes default value of ``2`` for single objective functions and ``20`` for multi objective. """ n_parents = 2 def __init__(self, eta: Optional[float] = None) -> None: self._eta = eta def crossover( self, parents_params: np.ndarray, rng: np.random.RandomState, study: Study, search_space_bounds: np.ndarray, ) -> np.ndarray: # https://doi.org/10.1007/3-540-45105-6_86 # Section 3.2 Crossover Schemes (vSBX) if self._eta is None: eta = 20.0 if study._is_multi_objective() else 2.0 else: eta = self._eta us = rng.rand(len(search_space_bounds)) beta_1 = np.power(1 / 2 * us, 1 / (eta + 1)) beta_2 = np.power(1 / 2 * (1 - us), 1 / (eta + 1)) mask = us > 0.5 c1 = 0.5 * ((1 + beta_1) * parents_params[0] + (1 - beta_1) * parents_params[1]) c1[mask] = ( 0.5 * ((1 - beta_1) * parents_params[0] + (1 + beta_1) * parents_params[1])[mask] ) c2 = 0.5 * ((3 - beta_2) * parents_params[0] - (1 - beta_2) * parents_params[1]) c2[mask] = ( 0.5 * (-(1 - beta_2) * parents_params[0] + (3 - beta_2) * parents_params[1])[mask] ) # vSBX applies crossover with establishment 0.5, and with probability 0.5, # the gene of the parent individual is the gene of the child individual. # The original SBX creates two child individuals, # but optuna's implementation creates only one child individual. # Therefore, when there is no crossover, # the gene is selected with equal probability from the parent individuals x1 and x2. child_params_list = [] for c1_i, c2_i, x1_i, x2_i in zip(c1, c2, parents_params[0], parents_params[1]): if rng.rand() < 0.5: if rng.rand() < 0.5: child_params_list.append(c1_i) else: child_params_list.append(c2_i) else: if rng.rand() < 0.5: child_params_list.append(x1_i) else: child_params_list.append(x2_i) child_params = np.array(child_params_list) return child_params optuna-4.1.0/optuna/samplers/nsgaii/_elite_population_selection_strategy.py000066400000000000000000000122441471332314300275100ustar00rootroot00000000000000from __future__ import annotations from collections import defaultdict from collections.abc import Callable from collections.abc import Sequence from typing import TYPE_CHECKING import numpy as np from optuna.samplers.nsgaii._constraints_evaluation import _evaluate_penalty from optuna.samplers.nsgaii._constraints_evaluation import _validate_constraints from optuna.study import StudyDirection from optuna.study._multi_objective import _fast_non_domination_rank from optuna.trial import FrozenTrial if TYPE_CHECKING: from optuna.study import Study class NSGAIIElitePopulationSelectionStrategy: def __init__( self, *, population_size: int, constraints_func: Callable[[FrozenTrial], Sequence[float]] | None = None, ) -> None: if population_size < 2: raise ValueError("`population_size` must be greater than or equal to 2.") self._population_size = population_size self._constraints_func = constraints_func def __call__(self, study: Study, population: list[FrozenTrial]) -> list[FrozenTrial]: """Select elite population from the given trials by NSGA-II algorithm. Args: study: Target study object. population: Trials in the study. Returns: A list of trials that are selected as elite population. """ _validate_constraints(population, is_constrained=self._constraints_func is not None) population_per_rank = _rank_population( population, study.directions, is_constrained=self._constraints_func is not None ) elite_population: list[FrozenTrial] = [] for individuals in population_per_rank: if len(elite_population) + len(individuals) < self._population_size: elite_population.extend(individuals) else: n = self._population_size - len(elite_population) _crowding_distance_sort(individuals) elite_population.extend(individuals[:n]) break return elite_population def _calc_crowding_distance(population: list[FrozenTrial]) -> defaultdict[int, float]: """Calculates the crowding distance of population. We define the crowding distance as the summation of the crowding distance of each dimension of value calculated as follows: * If all values in that dimension are the same, i.e., [1, 1, 1] or [inf, inf], the crowding distances of all trials in that dimension are zero. * Otherwise, the crowding distances of that dimension is the difference between two nearest values besides that value, one above and one below, divided by the difference between the maximal and minimal finite value of that dimension. Please note that: * the nearest value below the minimum is considered to be -inf and the nearest value above the maximum is considered to be inf, and * inf - inf and (-inf) - (-inf) is considered to be zero. """ manhattan_distances: defaultdict[int, float] = defaultdict(float) if len(population) == 0: return manhattan_distances for i in range(len(population[0].values)): population.sort(key=lambda x: x.values[i]) # If all trials in population have the same value in the i-th dimension, ignore the # objective dimension since it does not make difference. if population[0].values[i] == population[-1].values[i]: continue vs = [-float("inf")] + [trial.values[i] for trial in population] + [float("inf")] # Smallest finite value. v_min = next(x for x in vs if x != -float("inf")) # Largest finite value. v_max = next(x for x in reversed(vs) if x != float("inf")) width = v_max - v_min if width <= 0: # width == 0 or width == -inf width = 1.0 for j in range(len(population)): # inf - inf and (-inf) - (-inf) is considered to be zero. gap = 0.0 if vs[j] == vs[j + 2] else vs[j + 2] - vs[j] manhattan_distances[population[j].number] += gap / width return manhattan_distances def _crowding_distance_sort(population: list[FrozenTrial]) -> None: manhattan_distances = _calc_crowding_distance(population) population.sort(key=lambda x: manhattan_distances[x.number]) population.reverse() def _rank_population( population: list[FrozenTrial], directions: Sequence[StudyDirection], *, is_constrained: bool = False, ) -> list[list[FrozenTrial]]: if len(population) == 0: return [] objective_values = np.array([trial.values for trial in population], dtype=np.float64) objective_values *= np.array( [-1.0 if d == StudyDirection.MAXIMIZE else 1.0 for d in directions] ) penalty = _evaluate_penalty(population) if is_constrained else None domination_ranks = _fast_non_domination_rank(objective_values, penalty=penalty) population_per_rank: list[list[FrozenTrial]] = [[] for _ in range(max(domination_ranks) + 1)] for trial, rank in zip(population, domination_ranks): if rank == -1: continue population_per_rank[rank].append(trial) return population_per_rank optuna-4.1.0/optuna/samplers/nsgaii/_sampler.py000066400000000000000000000404511471332314300216110ustar00rootroot00000000000000from __future__ import annotations from collections import defaultdict from collections.abc import Callable from collections.abc import Sequence import hashlib from typing import Any from typing import TYPE_CHECKING import optuna from optuna._experimental import warn_experimental_argument from optuna.distributions import BaseDistribution from optuna.samplers._base import BaseSampler from optuna.samplers._lazy_random_state import LazyRandomState from optuna.samplers._random import RandomSampler from optuna.samplers.nsgaii._after_trial_strategy import NSGAIIAfterTrialStrategy from optuna.samplers.nsgaii._child_generation_strategy import NSGAIIChildGenerationStrategy from optuna.samplers.nsgaii._crossovers._base import BaseCrossover from optuna.samplers.nsgaii._crossovers._uniform import UniformCrossover from optuna.samplers.nsgaii._elite_population_selection_strategy import ( NSGAIIElitePopulationSelectionStrategy, ) from optuna.search_space import IntersectionSearchSpace from optuna.trial import FrozenTrial from optuna.trial import TrialState if TYPE_CHECKING: from optuna.study import Study # Define key names of `Trial.system_attrs`. _GENERATION_KEY = "nsga2:generation" _POPULATION_CACHE_KEY_PREFIX = "nsga2:population" class NSGAIISampler(BaseSampler): """Multi-objective sampler using the NSGA-II algorithm. NSGA-II stands for "Nondominated Sorting Genetic Algorithm II", which is a well known, fast and elitist multi-objective genetic algorithm. For further information about NSGA-II, please refer to the following paper: - `A fast and elitist multiobjective genetic algorithm: NSGA-II `__ .. note:: :class:`~optuna.samplers.TPESampler` became much faster in v4.0.0 and supports several features not supported by ``NSGAIISampler`` such as handling of dynamic search space and categorical distance. To use :class:`~optuna.samplers.TPESampler`, you need to explicitly specify the sampler as follows: .. testcode:: import optuna def objective(trial): x = trial.suggest_float("x", -100, 100) y = trial.suggest_categorical("y", [-1, 0, 1]) f1 = x**2 + y f2 = -((x - 2) ** 2 + y) return f1, f2 # We minimize the first objective and maximize the second objective. sampler = optuna.samplers.TPESampler() study = optuna.create_study(directions=["minimize", "maximize"], sampler=sampler) study.optimize(objective, n_trials=100) Please also check `our article `__ for more details of the speedup in v4.0.0. Args: population_size: Number of individuals (trials) in a generation. ``population_size`` must be greater than or equal to ``crossover.n_parents``. For :class:`~optuna.samplers.nsgaii.UNDXCrossover` and :class:`~optuna.samplers.nsgaii.SPXCrossover`, ``n_parents=3``, and for the other algorithms, ``n_parents=2``. mutation_prob: Probability of mutating each parameter when creating a new individual. If :obj:`None` is specified, the value ``1.0 / len(parent_trial.params)`` is used where ``parent_trial`` is the parent trial of the target individual. crossover: Crossover to be applied when creating child individuals. The available crossovers are listed here: https://optuna.readthedocs.io/en/stable/reference/samplers/nsgaii.html. :class:`~optuna.samplers.nsgaii.UniformCrossover` is always applied to parameters sampled from :class:`~optuna.distributions.CategoricalDistribution`, and by default for parameters sampled from other distributions unless this argument is specified. For more information on each of the crossover method, please refer to specific crossover documentation. crossover_prob: Probability that a crossover (parameters swapping between parents) will occur when creating a new individual. swapping_prob: Probability of swapping each parameter of the parents during crossover. seed: Seed for random number generator. constraints_func: An optional function that computes the objective constraints. It must take a :class:`~optuna.trial.FrozenTrial` and return the constraints. The return value must be a sequence of :obj:`float` s. A value strictly larger than 0 means that a constraints is violated. A value equal to or smaller than 0 is considered feasible. If ``constraints_func`` returns more than one value for a trial, that trial is considered feasible if and only if all values are equal to 0 or smaller. The ``constraints_func`` will be evaluated after each successful trial. The function won't be called when trials fail or they are pruned, but this behavior is subject to change in the future releases. The constraints are handled by the constrained domination. A trial x is said to constrained-dominate a trial y, if any of the following conditions is true: 1. Trial x is feasible and trial y is not. 2. Trial x and y are both infeasible, but trial x has a smaller overall violation. 3. Trial x and y are feasible and trial x dominates trial y. .. note:: Added in v2.5.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v2.5.0. elite_population_selection_strategy: The selection strategy for determining the individuals to survive from the current population pool. Default to :obj:`None`. .. note:: The arguments ``elite_population_selection_strategy`` was added in v3.3.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v3.3.0. child_generation_strategy: The strategy for generating child parameters from parent trials. Defaults to :obj:`None`. .. note:: The arguments ``child_generation_strategy`` was added in v3.3.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v3.3.0. after_trial_strategy: A set of procedure to be conducted after each trial. Defaults to :obj:`None`. .. note:: The arguments ``after_trial_strategy`` was added in v3.3.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v3.3.0. """ def __init__( self, *, population_size: int = 50, mutation_prob: float | None = None, crossover: BaseCrossover | None = None, crossover_prob: float = 0.9, swapping_prob: float = 0.5, seed: int | None = None, constraints_func: Callable[[FrozenTrial], Sequence[float]] | None = None, elite_population_selection_strategy: ( Callable[[Study, list[FrozenTrial]], list[FrozenTrial]] | None ) = None, child_generation_strategy: ( Callable[[Study, dict[str, BaseDistribution], list[FrozenTrial]], dict[str, Any]] | None ) = None, after_trial_strategy: ( Callable[[Study, FrozenTrial, TrialState, Sequence[float] | None], None] | None ) = None, ) -> None: # TODO(ohta): Reconsider the default value of each parameter. if population_size < 2: raise ValueError("`population_size` must be greater than or equal to 2.") if constraints_func is not None: warn_experimental_argument("constraints_func") if after_trial_strategy is not None: warn_experimental_argument("after_trial_strategy") if child_generation_strategy is not None: warn_experimental_argument("child_generation_strategy") if elite_population_selection_strategy is not None: warn_experimental_argument("elite_population_selection_strategy") if crossover is None: crossover = UniformCrossover(swapping_prob) if not isinstance(crossover, BaseCrossover): raise ValueError( f"'{crossover}' is not a valid crossover." " For valid crossovers see" " https://optuna.readthedocs.io/en/stable/reference/samplers.html." ) if population_size < crossover.n_parents: raise ValueError( f"Using {crossover}," f" the population size should be greater than or equal to {crossover.n_parents}." f" The specified `population_size` is {population_size}." ) self._population_size = population_size self._random_sampler = RandomSampler(seed=seed) self._rng = LazyRandomState(seed) self._constraints_func = constraints_func self._search_space = IntersectionSearchSpace() self._elite_population_selection_strategy = ( elite_population_selection_strategy or NSGAIIElitePopulationSelectionStrategy( population_size=population_size, constraints_func=constraints_func ) ) self._child_generation_strategy = ( child_generation_strategy or NSGAIIChildGenerationStrategy( crossover_prob=crossover_prob, mutation_prob=mutation_prob, swapping_prob=swapping_prob, crossover=crossover, constraints_func=constraints_func, rng=self._rng, ) ) self._after_trial_strategy = after_trial_strategy or NSGAIIAfterTrialStrategy( constraints_func=constraints_func ) def reseed_rng(self) -> None: self._random_sampler.reseed_rng() self._rng.rng.seed() def infer_relative_search_space( self, study: Study, trial: FrozenTrial ) -> dict[str, BaseDistribution]: search_space: dict[str, BaseDistribution] = {} for name, distribution in self._search_space.calculate(study).items(): if distribution.single(): # The `untransform` method of `optuna._transform._SearchSpaceTransform` # does not assume a single value, # so single value objects are not sampled with the `sample_relative` method, # but with the `sample_independent` method. continue search_space[name] = distribution return search_space def sample_relative( self, study: Study, trial: FrozenTrial, search_space: dict[str, BaseDistribution], ) -> dict[str, Any]: parent_generation, parent_population = self._collect_parent_population(study) generation = parent_generation + 1 study._storage.set_trial_system_attr(trial._trial_id, _GENERATION_KEY, generation) if parent_generation < 0: return {} return self._child_generation_strategy(study, search_space, parent_population) def sample_independent( self, study: Study, trial: FrozenTrial, param_name: str, param_distribution: BaseDistribution, ) -> Any: # Following parameters are randomly sampled here. # 1. A parameter in the initial population/first generation. # 2. A parameter to mutate. # 3. A parameter excluded from the intersection search space. return self._random_sampler.sample_independent( study, trial, param_name, param_distribution ) def _collect_parent_population(self, study: Study) -> tuple[int, list[FrozenTrial]]: trials = study._get_trials(deepcopy=False, use_cache=True) generation_to_runnings = defaultdict(list) generation_to_population = defaultdict(list) for trial in trials: if _GENERATION_KEY not in trial.system_attrs: continue generation = trial.system_attrs[_GENERATION_KEY] if trial.state != optuna.trial.TrialState.COMPLETE: if trial.state == optuna.trial.TrialState.RUNNING: generation_to_runnings[generation].append(trial) continue # Do not use trials whose states are not COMPLETE, or `constraint` will be unavailable. generation_to_population[generation].append(trial) hasher = hashlib.sha256() parent_population: list[FrozenTrial] = [] parent_generation = -1 while True: generation = parent_generation + 1 population = generation_to_population[generation] # Under multi-worker settings, the population size might become larger than # `self._population_size`. if len(population) < self._population_size: break # [NOTE] # It's generally safe to assume that once the above condition is satisfied, # there are no additional individuals added to the generation (i.e., the members of # the generation have been fixed). # If the number of parallel workers is huge, this assumption can be broken, but # this is a very rare case and doesn't significantly impact optimization performance. # So we can ignore the case. # The cache key is calculated based on the key of the previous generation and # the remaining running trials in the current population. # If there are no running trials, the new cache key becomes exactly the same as # the previous one, and the cached content will be overwritten. This allows us to # skip redundant cache key calculations when this method is called for the subsequent # trials. for trial in generation_to_runnings[generation]: hasher.update(bytes(str(trial.number), "utf-8")) cache_key = "{}:{}".format(_POPULATION_CACHE_KEY_PREFIX, hasher.hexdigest()) study_system_attrs = study._storage.get_study_system_attrs(study._study_id) cached_generation, cached_population_numbers = study_system_attrs.get( cache_key, (-1, []) ) if cached_generation >= generation: generation = cached_generation population = [trials[n] for n in cached_population_numbers] else: population.extend(parent_population) population = self._elite_population_selection_strategy(study, population) # To reduce the number of system attribute entries, # we cache the population information only if there are no running trials # (i.e., the information of the population has been fixed). # Usually, if there are no too delayed running trials, the single entry # will be used. if len(generation_to_runnings[generation]) == 0: population_numbers = [t.number for t in population] study._storage.set_study_system_attr( study._study_id, cache_key, (generation, population_numbers) ) parent_generation = generation parent_population = population return parent_generation, parent_population def before_trial(self, study: Study, trial: FrozenTrial) -> None: self._random_sampler.before_trial(study, trial) def after_trial( self, study: Study, trial: FrozenTrial, state: TrialState, values: Sequence[float] | None, ) -> None: assert state in [TrialState.COMPLETE, TrialState.FAIL, TrialState.PRUNED] self._after_trial_strategy(study, trial, state, values) self._random_sampler.after_trial(study, trial, state, values) optuna-4.1.0/optuna/search_space/000077500000000000000000000000001471332314300167515ustar00rootroot00000000000000optuna-4.1.0/optuna/search_space/__init__.py000066400000000000000000000006501471332314300210630ustar00rootroot00000000000000from optuna.search_space.group_decomposed import _GroupDecomposedSearchSpace from optuna.search_space.group_decomposed import _SearchSpaceGroup from optuna.search_space.intersection import intersection_search_space from optuna.search_space.intersection import IntersectionSearchSpace __all__ = [ "_GroupDecomposedSearchSpace", "_SearchSpaceGroup", "IntersectionSearchSpace", "intersection_search_space", ] optuna-4.1.0/optuna/search_space/group_decomposed.py000066400000000000000000000047121471332314300226650ustar00rootroot00000000000000from __future__ import annotations import copy from typing import Dict from typing import List from typing import Optional from typing import Tuple from typing import TYPE_CHECKING from optuna.distributions import BaseDistribution from optuna.trial import TrialState if TYPE_CHECKING: from optuna.study import Study class _SearchSpaceGroup: def __init__(self) -> None: self._search_spaces: List[Dict[str, BaseDistribution]] = [] @property def search_spaces(self) -> List[Dict[str, BaseDistribution]]: return self._search_spaces def add_distributions(self, distributions: Dict[str, BaseDistribution]) -> None: dist_keys = set(distributions.keys()) next_search_spaces = [] for search_space in self._search_spaces: keys = set(search_space.keys()) next_search_spaces.append({name: search_space[name] for name in keys & dist_keys}) next_search_spaces.append({name: search_space[name] for name in keys - dist_keys}) dist_keys -= keys next_search_spaces.append({name: distributions[name] for name in dist_keys}) self._search_spaces = list( filter(lambda search_space: len(search_space) > 0, next_search_spaces) ) class _GroupDecomposedSearchSpace: def __init__(self, include_pruned: bool = False) -> None: self._search_space = _SearchSpaceGroup() self._study_id: Optional[int] = None self._include_pruned = include_pruned def calculate(self, study: Study) -> _SearchSpaceGroup: if self._study_id is None: self._study_id = study._study_id else: # Note that the check below is meaningless when # :class:`~optuna.storages.InMemoryStorage` is used because # :func:`~optuna.storages.InMemoryStorage.create_new_study` # always returns the same study ID. if self._study_id != study._study_id: raise ValueError("`_GroupDecomposedSearchSpace` cannot handle multiple studies.") states_of_interest: Tuple[TrialState, ...] if self._include_pruned: states_of_interest = (TrialState.COMPLETE, TrialState.PRUNED) else: states_of_interest = (TrialState.COMPLETE,) for trial in study._get_trials(deepcopy=False, states=states_of_interest, use_cache=False): self._search_space.add_distributions(trial.distributions) return copy.deepcopy(self._search_space) optuna-4.1.0/optuna/search_space/intersection.py000066400000000000000000000123061471332314300220330ustar00rootroot00000000000000from __future__ import annotations import copy from typing import Dict from typing import Tuple from typing import TYPE_CHECKING import optuna from optuna.distributions import BaseDistribution if TYPE_CHECKING: from optuna.study import Study def _calculate( trials: list[optuna.trial.FrozenTrial], include_pruned: bool = False, search_space: Dict[str, BaseDistribution] | None = None, cached_trial_number: int = -1, ) -> Tuple[Dict[str, BaseDistribution] | None, int]: states_of_interest = [ optuna.trial.TrialState.COMPLETE, optuna.trial.TrialState.WAITING, optuna.trial.TrialState.RUNNING, ] if include_pruned: states_of_interest.append(optuna.trial.TrialState.PRUNED) trials_of_interest = [trial for trial in trials if trial.state in states_of_interest] next_cached_trial_number = ( trials_of_interest[-1].number + 1 if len(trials_of_interest) > 0 else -1 ) for trial in reversed(trials_of_interest): if cached_trial_number > trial.number: break if not trial.state.is_finished(): next_cached_trial_number = trial.number continue if search_space is None: search_space = copy.copy(trial.distributions) continue search_space = { name: distribution for name, distribution in search_space.items() if trial.distributions.get(name) == distribution } return search_space, next_cached_trial_number class IntersectionSearchSpace: """A class to calculate the intersection search space of a :class:`~optuna.study.Study`. Intersection search space contains the intersection of parameter distributions that have been suggested in the completed trials of the study so far. If there are multiple parameters that have the same name but different distributions, neither is included in the resulting search space (i.e., the parameters with dynamic value ranges are excluded). Note that an instance of this class is supposed to be used for only one study. If different studies are passed to :func:`~optuna.search_space.IntersectionSearchSpace.calculate`, a :obj:`ValueError` is raised. Args: include_pruned: Whether pruned trials should be included in the search space. """ def __init__(self, include_pruned: bool = False) -> None: self._cached_trial_number: int = -1 self._search_space: Dict[str, BaseDistribution] | None = None self._study_id: int | None = None self._include_pruned = include_pruned def calculate(self, study: Study) -> Dict[str, BaseDistribution]: """Returns the intersection search space of the :class:`~optuna.study.Study`. Args: study: A study with completed trials. The same study must be passed for one instance of this class through its lifetime. Returns: A dictionary containing the parameter names and parameter's distributions sorted by parameter names. """ if self._study_id is None: self._study_id = study._study_id else: # Note that the check below is meaningless when # :class:`~optuna.storages.InMemoryStorage` is used because # :func:`~optuna.storages.InMemoryStorage.create_new_study` # always returns the same study ID. if self._study_id != study._study_id: raise ValueError("`IntersectionSearchSpace` cannot handle multiple studies.") self._search_space, self._cached_trial_number = _calculate( study.get_trials(deepcopy=False), self._include_pruned, self._search_space, self._cached_trial_number, ) search_space = self._search_space or {} search_space = dict(sorted(search_space.items(), key=lambda x: x[0])) return copy.deepcopy(search_space) def intersection_search_space( trials: list[optuna.trial.FrozenTrial], include_pruned: bool = False, ) -> Dict[str, BaseDistribution]: """Return the intersection search space of the given trials. Intersection search space contains the intersection of parameter distributions that have been suggested in the completed trials of the study so far. If there are multiple parameters that have the same name but different distributions, neither is included in the resulting search space (i.e., the parameters with dynamic value ranges are excluded). .. note:: :class:`~optuna.search_space.IntersectionSearchSpace` provides the same functionality with a much faster way. Please consider using it if you want to reduce execution time as much as possible. Args: trials: A list of trials. include_pruned: Whether pruned trials should be included in the search space. Returns: A dictionary containing the parameter names and parameter's distributions sorted by parameter names. """ search_space, _ = _calculate(trials, include_pruned) search_space = search_space or {} search_space = dict(sorted(search_space.items(), key=lambda x: x[0])) return search_space optuna-4.1.0/optuna/storages/000077500000000000000000000000001471332314300161605ustar00rootroot00000000000000optuna-4.1.0/optuna/storages/__init__.py000066400000000000000000000032701471332314300202730ustar00rootroot00000000000000from __future__ import annotations from optuna.storages._base import BaseStorage from optuna.storages._cached_storage import _CachedStorage from optuna.storages._callbacks import RetryFailedTrialCallback from optuna.storages._heartbeat import fail_stale_trials from optuna.storages._in_memory import InMemoryStorage from optuna.storages._rdb.storage import RDBStorage from optuna.storages.journal._base import BaseJournalLogStorage from optuna.storages.journal._file import ( DeprecatedJournalFileSymlinkLock as JournalFileSymlinkLock, ) from optuna.storages.journal._file import DeprecatedJournalFileOpenLock as JournalFileOpenLock from optuna.storages.journal._file import JournalFileStorage from optuna.storages.journal._redis import JournalRedisStorage from optuna.storages.journal._storage import JournalStorage __all__ = [ "BaseStorage", "BaseJournalLogStorage", "InMemoryStorage", "RDBStorage", "JournalStorage", "JournalFileStorage", "JournalRedisStorage", "JournalFileSymlinkLock", "JournalFileOpenLock", "RetryFailedTrialCallback", "_CachedStorage", "fail_stale_trials", ] def get_storage(storage: None | str | BaseStorage) -> BaseStorage: """Only for internal usage. It might be deprecated in the future.""" if storage is None: return InMemoryStorage() if isinstance(storage, str): if storage.startswith("redis"): raise ValueError( "RedisStorage is removed at Optuna v3.1.0. Please use JournalRedisBackend instead." ) return _CachedStorage(RDBStorage(storage)) elif isinstance(storage, RDBStorage): return _CachedStorage(storage) else: return storage optuna-4.1.0/optuna/storages/_base.py000066400000000000000000000452401471332314300176100ustar00rootroot00000000000000from __future__ import annotations import abc from collections.abc import Container from collections.abc import Sequence from typing import Any from typing import cast from optuna._typing import JSONSerializable from optuna.distributions import BaseDistribution from optuna.study._frozen import FrozenStudy from optuna.study._study_direction import StudyDirection from optuna.trial import FrozenTrial from optuna.trial import TrialState DEFAULT_STUDY_NAME_PREFIX = "no-name-" class BaseStorage(abc.ABC): """Base class for storages. This class is not supposed to be directly accessed by library users. This class abstracts a backend database and provides internal interfaces to read/write histories of studies and trials. A storage class implementing this class must meet the following requirements. **Thread safety** A storage class instance can be shared among multiple threads, and must therefore be thread-safe. It must guarantee that a data instance read from the storage must not be modified by subsequent writes. For example, `FrozenTrial` instance returned by `get_trial` should not be updated by the subsequent `set_trial_xxx`. This is usually achieved by replacing the old data with a copy on `set_trial_xxx`. A storage class can also assume that a data instance returned are never modified by its user. When a user modifies a return value from a storage class, the internal state of the storage may become inconsistent. Consequences are undefined. **Ownership of RUNNING trials** Trials in finished states are not allowed to be modified. Trials in the WAITING state are not allowed to be modified except for the `state` field. """ # Basic study manipulation @abc.abstractmethod def create_new_study( self, directions: Sequence[StudyDirection], study_name: str | None = None ) -> int: """Create a new study from a name. If no name is specified, the storage class generates a name. The returned study ID is unique among all current and deleted studies. Args: directions: A sequence of direction whose element is either :obj:`~optuna.study.StudyDirection.MAXIMIZE` or :obj:`~optuna.study.StudyDirection.MINIMIZE`. study_name: Name of the new study to create. Returns: ID of the created study. Raises: :exc:`optuna.exceptions.DuplicatedStudyError`: If a study with the same ``study_name`` already exists. """ raise NotImplementedError @abc.abstractmethod def delete_study(self, study_id: int) -> None: """Delete a study. Args: study_id: ID of the study. Raises: :exc:`KeyError`: If no study with the matching ``study_id`` exists. """ raise NotImplementedError @abc.abstractmethod def set_study_user_attr(self, study_id: int, key: str, value: Any) -> None: """Register a user-defined attribute to a study. This method overwrites any existing attribute. Args: study_id: ID of the study. key: Attribute key. value: Attribute value. It should be JSON serializable. Raises: :exc:`KeyError`: If no study with the matching ``study_id`` exists. """ raise NotImplementedError @abc.abstractmethod def set_study_system_attr(self, study_id: int, key: str, value: JSONSerializable) -> None: """Register an optuna-internal attribute to a study. This method overwrites any existing attribute. Args: study_id: ID of the study. key: Attribute key. value: Attribute value. It should be JSON serializable. Raises: :exc:`KeyError`: If no study with the matching ``study_id`` exists. """ raise NotImplementedError # Basic study access @abc.abstractmethod def get_study_id_from_name(self, study_name: str) -> int: """Read the ID of a study. Args: study_name: Name of the study. Returns: ID of the study. Raises: :exc:`KeyError`: If no study with the matching ``study_name`` exists. """ raise NotImplementedError @abc.abstractmethod def get_study_name_from_id(self, study_id: int) -> str: """Read the study name of a study. Args: study_id: ID of the study. Returns: Name of the study. Raises: :exc:`KeyError`: If no study with the matching ``study_id`` exists. """ raise NotImplementedError @abc.abstractmethod def get_study_directions(self, study_id: int) -> list[StudyDirection]: """Read whether a study maximizes or minimizes an objective. Args: study_id: ID of a study. Returns: Optimization directions list of the study. Raises: :exc:`KeyError`: If no study with the matching ``study_id`` exists. """ raise NotImplementedError @abc.abstractmethod def get_study_user_attrs(self, study_id: int) -> dict[str, Any]: """Read the user-defined attributes of a study. Args: study_id: ID of the study. Returns: Dictionary with the user attributes of the study. Raises: :exc:`KeyError`: If no study with the matching ``study_id`` exists. """ raise NotImplementedError @abc.abstractmethod def get_study_system_attrs(self, study_id: int) -> dict[str, Any]: """Read the optuna-internal attributes of a study. Args: study_id: ID of the study. Returns: Dictionary with the optuna-internal attributes of the study. Raises: :exc:`KeyError`: If no study with the matching ``study_id`` exists. """ raise NotImplementedError @abc.abstractmethod def get_all_studies(self) -> list[FrozenStudy]: """Read a list of :class:`~optuna.study.FrozenStudy` objects. Returns: A list of :class:`~optuna.study.FrozenStudy` objects, sorted by ``study_id``. """ raise NotImplementedError # Basic trial manipulation @abc.abstractmethod def create_new_trial(self, study_id: int, template_trial: FrozenTrial | None = None) -> int: """Create and add a new trial to a study. The returned trial ID is unique among all current and deleted trials. Args: study_id: ID of the study. template_trial: Template :class:`~optuna.trial.FrozenTrial` with default user-attributes, system-attributes, intermediate-values, and a state. Returns: ID of the created trial. Raises: :exc:`KeyError`: If no study with the matching ``study_id`` exists. """ raise NotImplementedError @abc.abstractmethod def set_trial_param( self, trial_id: int, param_name: str, param_value_internal: float, distribution: BaseDistribution, ) -> None: """Set a parameter to a trial. Args: trial_id: ID of the trial. param_name: Name of the parameter. param_value_internal: Internal representation of the parameter value. distribution: Sampled distribution of the parameter. Raises: :exc:`KeyError`: If no trial with the matching ``trial_id`` exists. :exc:`RuntimeError`: If the trial is already finished. """ raise NotImplementedError def get_trial_id_from_study_id_trial_number(self, study_id: int, trial_number: int) -> int: """Read the trial ID of a trial. Args: study_id: ID of the study. trial_number: Number of the trial. Returns: ID of the trial. Raises: :exc:`KeyError`: If no trial with the matching ``study_id`` and ``trial_number`` exists. """ trials = self.get_all_trials(study_id, deepcopy=False) if len(trials) <= trial_number: raise KeyError( "No trial with trial number {} exists in study with study_id {}.".format( trial_number, study_id ) ) return trials[trial_number]._trial_id def get_trial_number_from_id(self, trial_id: int) -> int: """Read the trial number of a trial. .. note:: The trial number is only unique within a study, and is sequential. Args: trial_id: ID of the trial. Returns: Number of the trial. Raises: :exc:`KeyError`: If no trial with the matching ``trial_id`` exists. """ return self.get_trial(trial_id).number def get_trial_param(self, trial_id: int, param_name: str) -> float: """Read the parameter of a trial. Args: trial_id: ID of the trial. param_name: Name of the parameter. Returns: Internal representation of the parameter. Raises: :exc:`KeyError`: If no trial with the matching ``trial_id`` exists. If no such parameter exists. """ trial = self.get_trial(trial_id) return trial.distributions[param_name].to_internal_repr(trial.params[param_name]) @abc.abstractmethod def set_trial_state_values( self, trial_id: int, state: TrialState, values: Sequence[float] | None = None ) -> bool: """Update the state and values of a trial. Set return values of an objective function to values argument. If values argument is not :obj:`None`, this method overwrites any existing trial values. Args: trial_id: ID of the trial. state: New state of the trial. values: Values of the objective function. Returns: :obj:`True` if the state is successfully updated. :obj:`False` if the state is kept the same. The latter happens when this method tries to update the state of :obj:`~optuna.trial.TrialState.RUNNING` trial to :obj:`~optuna.trial.TrialState.RUNNING`. Raises: :exc:`KeyError`: If no trial with the matching ``trial_id`` exists. :exc:`RuntimeError`: If the trial is already finished. """ raise NotImplementedError @abc.abstractmethod def set_trial_intermediate_value( self, trial_id: int, step: int, intermediate_value: float ) -> None: """Report an intermediate value of an objective function. This method overwrites any existing intermediate value associated with the given step. Args: trial_id: ID of the trial. step: Step of the trial (e.g., the epoch when training a neural network). intermediate_value: Intermediate value corresponding to the step. Raises: :exc:`KeyError`: If no trial with the matching ``trial_id`` exists. :exc:`RuntimeError`: If the trial is already finished. """ raise NotImplementedError @abc.abstractmethod def set_trial_user_attr(self, trial_id: int, key: str, value: Any) -> None: """Set a user-defined attribute to a trial. This method overwrites any existing attribute. Args: trial_id: ID of the trial. key: Attribute key. value: Attribute value. It should be JSON serializable. Raises: :exc:`KeyError`: If no trial with the matching ``trial_id`` exists. :exc:`RuntimeError`: If the trial is already finished. """ raise NotImplementedError @abc.abstractmethod def set_trial_system_attr(self, trial_id: int, key: str, value: JSONSerializable) -> None: """Set an optuna-internal attribute to a trial. This method overwrites any existing attribute. Args: trial_id: ID of the trial. key: Attribute key. value: Attribute value. It should be JSON serializable. Raises: :exc:`KeyError`: If no trial with the matching ``trial_id`` exists. :exc:`RuntimeError`: If the trial is already finished. """ raise NotImplementedError # Basic trial access @abc.abstractmethod def get_trial(self, trial_id: int) -> FrozenTrial: """Read a trial. Args: trial_id: ID of the trial. Returns: Trial with a matching trial ID. Raises: :exc:`KeyError`: If no trial with the matching ``trial_id`` exists. """ raise NotImplementedError @abc.abstractmethod def get_all_trials( self, study_id: int, deepcopy: bool = True, states: Container[TrialState] | None = None, ) -> list[FrozenTrial]: """Read all trials in a study. Args: study_id: ID of the study. deepcopy: Whether to copy the list of trials before returning. Set to :obj:`True` if you intend to update the list or elements of the list. states: Trial states to filter on. If :obj:`None`, include all states. Returns: List of trials in the study, sorted by ``trial_id``. Raises: :exc:`KeyError`: If no study with the matching ``study_id`` exists. """ raise NotImplementedError def get_n_trials( self, study_id: int, state: tuple[TrialState, ...] | TrialState | None = None ) -> int: """Count the number of trials in a study. Args: study_id: ID of the study. state: Trial states to filter on. If :obj:`None`, include all states. Returns: Number of trials in the study. Raises: :exc:`KeyError`: If no study with the matching ``study_id`` exists. """ # TODO(hvy): Align the name and the behavior or the `state` parameter with # `get_all_trials`'s `states`. if isinstance(state, TrialState): state = (state,) return len(self.get_all_trials(study_id, deepcopy=False, states=state)) def get_best_trial(self, study_id: int) -> FrozenTrial: """Return the trial with the best value in a study. This method is valid only during single-objective optimization. Args: study_id: ID of the study. Returns: The trial with the best objective value among all finished trials in the study. Raises: :exc:`KeyError`: If no study with the matching ``study_id`` exists. :exc:`RuntimeError`: If the study has more than one direction. :exc:`ValueError`: If no trials have been completed. """ all_trials = self.get_all_trials(study_id, deepcopy=False, states=[TrialState.COMPLETE]) if len(all_trials) == 0: raise ValueError("No trials are completed yet.") directions = self.get_study_directions(study_id) if len(directions) > 1: raise RuntimeError( "Best trial can be obtained only for single-objective optimization." ) direction = directions[0] if direction == StudyDirection.MAXIMIZE: best_trial = max(all_trials, key=lambda t: cast(float, t.value)) else: best_trial = min(all_trials, key=lambda t: cast(float, t.value)) return best_trial def get_trial_params(self, trial_id: int) -> dict[str, Any]: """Read the parameter dictionary of a trial. Args: trial_id: ID of the trial. Returns: Dictionary of a parameters. Keys are parameter names and values are internal representations of the parameter values. Raises: :exc:`KeyError`: If no trial with the matching ``trial_id`` exists. """ return self.get_trial(trial_id).params def get_trial_user_attrs(self, trial_id: int) -> dict[str, Any]: """Read the user-defined attributes of a trial. Args: trial_id: ID of the trial. Returns: Dictionary with the user-defined attributes of the trial. Raises: :exc:`KeyError`: If no trial with the matching ``trial_id`` exists. """ return self.get_trial(trial_id).user_attrs def get_trial_system_attrs(self, trial_id: int) -> dict[str, Any]: """Read the optuna-internal attributes of a trial. Args: trial_id: ID of the trial. Returns: Dictionary with the optuna-internal attributes of the trial. Raises: :exc:`KeyError`: If no trial with the matching ``trial_id`` exists. """ return self.get_trial(trial_id).system_attrs def remove_session(self) -> None: """Clean up all connections to a database.""" pass def check_trial_is_updatable(self, trial_id: int, trial_state: TrialState) -> None: """Check whether a trial state is updatable. Args: trial_id: ID of the trial. Only used for an error message. trial_state: Trial state to check. Raises: :exc:`RuntimeError`: If the trial is already finished. """ if trial_state.is_finished(): trial = self.get_trial(trial_id) raise RuntimeError( "Trial#{} has already finished and can not be updated.".format(trial.number) ) optuna-4.1.0/optuna/storages/_cached_storage.py000066400000000000000000000267731471332314300216430ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Callable from collections.abc import Container from collections.abc import Sequence import copy import threading from typing import Any import optuna from optuna import distributions from optuna._typing import JSONSerializable from optuna.storages import BaseStorage from optuna.storages._heartbeat import BaseHeartbeat from optuna.storages._rdb.storage import RDBStorage from optuna.study._frozen import FrozenStudy from optuna.study._study_direction import StudyDirection from optuna.trial import FrozenTrial from optuna.trial import TrialState class _StudyInfo: def __init__(self) -> None: # Trial number to corresponding FrozenTrial. self.trials: dict[int, FrozenTrial] = {} # A list of trials and the last trial number which require storage access to read latest # attributes. self.unfinished_trial_ids: set[int] = set() self.last_finished_trial_id: int = -1 # Cache distributions to avoid storage access on distribution consistency check. self.param_distribution: dict[str, distributions.BaseDistribution] = {} self.directions: list[StudyDirection] | None = None self.name: str | None = None class _CachedStorage(BaseStorage, BaseHeartbeat): """A wrapper class of storage backends. This class is used in :func:`~optuna.get_storage` function and automatically wraps :class:`~optuna.storages.RDBStorage` class. :class:`~optuna.storages._CachedStorage` meets the following **Data persistence** requirements. **Data persistence** :class:`~optuna.storages._CachedStorage` does not guarantee that write operations are logged into a persistent storage, even when write methods succeed. Thus, when process failure occurs, some writes might be lost. As exceptions, when a persistent storage is available, any writes on any attributes of `Study` and writes on `state` of `Trial` are guaranteed to be persistent. Additionally, any preceding writes on any attributes of `Trial` are guaranteed to be written into a persistent storage before writes on `state` of `Trial` succeed. The same applies for `param`, `user_attrs', 'system_attrs' and 'intermediate_values` attributes. Args: backend: :class:`~optuna.storages.RDBStorage` class instance to wrap. """ def __init__(self, backend: RDBStorage) -> None: self._backend = backend self._studies: dict[int, _StudyInfo] = {} self._trial_id_to_study_id_and_number: dict[int, tuple[int, int]] = {} self._study_id_and_number_to_trial_id: dict[tuple[int, int], int] = {} self._lock = threading.Lock() def __getstate__(self) -> dict[Any, Any]: state = self.__dict__.copy() del state["_lock"] return state def __setstate__(self, state: dict[Any, Any]) -> None: self.__dict__.update(state) self._lock = threading.Lock() def create_new_study( self, directions: Sequence[StudyDirection], study_name: str | None = None ) -> int: study_id = self._backend.create_new_study(directions=directions, study_name=study_name) with self._lock: study = _StudyInfo() study.name = study_name study.directions = list(directions) self._studies[study_id] = study return study_id def delete_study(self, study_id: int) -> None: with self._lock: if study_id in self._studies: for trial_number in self._studies[study_id].trials: trial_id = self._study_id_and_number_to_trial_id.get((study_id, trial_number)) if trial_id in self._trial_id_to_study_id_and_number: del self._trial_id_to_study_id_and_number[trial_id] if (study_id, trial_number) in self._study_id_and_number_to_trial_id: del self._study_id_and_number_to_trial_id[(study_id, trial_number)] del self._studies[study_id] self._backend.delete_study(study_id) def set_study_user_attr(self, study_id: int, key: str, value: Any) -> None: self._backend.set_study_user_attr(study_id, key, value) def set_study_system_attr(self, study_id: int, key: str, value: JSONSerializable) -> None: self._backend.set_study_system_attr(study_id, key, value) def get_study_id_from_name(self, study_name: str) -> int: return self._backend.get_study_id_from_name(study_name) def get_study_name_from_id(self, study_id: int) -> str: with self._lock: if study_id in self._studies: name = self._studies[study_id].name if name is not None: return name name = self._backend.get_study_name_from_id(study_id) with self._lock: if study_id not in self._studies: self._studies[study_id] = _StudyInfo() self._studies[study_id].name = name return name def get_study_directions(self, study_id: int) -> list[StudyDirection]: with self._lock: if study_id in self._studies: directions = self._studies[study_id].directions if directions is not None: return directions directions = self._backend.get_study_directions(study_id) with self._lock: if study_id not in self._studies: self._studies[study_id] = _StudyInfo() self._studies[study_id].directions = directions return directions def get_study_user_attrs(self, study_id: int) -> dict[str, Any]: return self._backend.get_study_user_attrs(study_id) def get_study_system_attrs(self, study_id: int) -> dict[str, Any]: return self._backend.get_study_system_attrs(study_id) def get_all_studies(self) -> list[FrozenStudy]: return self._backend.get_all_studies() def create_new_trial(self, study_id: int, template_trial: FrozenTrial | None = None) -> int: frozen_trial = self._backend._create_new_trial(study_id, template_trial) trial_id = frozen_trial._trial_id with self._lock: if study_id not in self._studies: self._studies[study_id] = _StudyInfo() study = self._studies[study_id] self._add_trials_to_cache(study_id, [frozen_trial]) # Since finished trials will not be modified by any worker, we do not # need storage access for them. if frozen_trial.state.is_finished(): study.last_finished_trial_id = max(study.last_finished_trial_id, trial_id) else: study.unfinished_trial_ids.add(trial_id) return trial_id def set_trial_param( self, trial_id: int, param_name: str, param_value_internal: float, distribution: distributions.BaseDistribution, ) -> None: self._backend.set_trial_param(trial_id, param_name, param_value_internal, distribution) def get_trial_id_from_study_id_trial_number(self, study_id: int, trial_number: int) -> int: key = (study_id, trial_number) with self._lock: if key in self._study_id_and_number_to_trial_id: return self._study_id_and_number_to_trial_id[key] return self._backend.get_trial_id_from_study_id_trial_number(study_id, trial_number) def get_best_trial(self, study_id: int) -> FrozenTrial: return self._backend.get_best_trial(study_id) def set_trial_state_values( self, trial_id: int, state: TrialState, values: Sequence[float] | None = None ) -> bool: return self._backend.set_trial_state_values(trial_id, state=state, values=values) def set_trial_intermediate_value( self, trial_id: int, step: int, intermediate_value: float ) -> None: self._backend.set_trial_intermediate_value(trial_id, step, intermediate_value) def set_trial_user_attr(self, trial_id: int, key: str, value: Any) -> None: self._backend.set_trial_user_attr(trial_id, key=key, value=value) def set_trial_system_attr(self, trial_id: int, key: str, value: JSONSerializable) -> None: self._backend.set_trial_system_attr(trial_id, key=key, value=value) def _get_cached_trial(self, trial_id: int) -> FrozenTrial | None: if trial_id not in self._trial_id_to_study_id_and_number: return None study_id, number = self._trial_id_to_study_id_and_number[trial_id] study = self._studies[study_id] return study.trials[number] if trial_id not in study.unfinished_trial_ids else None def get_trial(self, trial_id: int) -> FrozenTrial: with self._lock: trial = self._get_cached_trial(trial_id) if trial is not None: return trial return self._backend.get_trial(trial_id) def get_all_trials( self, study_id: int, deepcopy: bool = True, states: Container[TrialState] | None = None, ) -> list[FrozenTrial]: self._read_trials_from_remote_storage(study_id) with self._lock: study = self._studies[study_id] # We need to sort trials by their number because some samplers assume this behavior. # The following two lines are latency-sensitive. trials: dict[int, FrozenTrial] | list[FrozenTrial] if states is not None: trials = {number: t for number, t in study.trials.items() if t.state in states} else: trials = study.trials trials = list(sorted(trials.values(), key=lambda t: t.number)) return copy.deepcopy(trials) if deepcopy else trials def _read_trials_from_remote_storage(self, study_id: int) -> None: with self._lock: if study_id not in self._studies: self._studies[study_id] = _StudyInfo() study = self._studies[study_id] trials = self._backend._get_trials( study_id, states=None, included_trial_ids=study.unfinished_trial_ids, trial_id_greater_than=study.last_finished_trial_id, ) if not trials: return self._add_trials_to_cache(study_id, trials) for trial in trials: if not trial.state.is_finished(): study.unfinished_trial_ids.add(trial._trial_id) continue study.last_finished_trial_id = max(study.last_finished_trial_id, trial._trial_id) if trial._trial_id in study.unfinished_trial_ids: study.unfinished_trial_ids.remove(trial._trial_id) def _add_trials_to_cache(self, study_id: int, trials: list[FrozenTrial]) -> None: study = self._studies[study_id] for trial in trials: self._trial_id_to_study_id_and_number[trial._trial_id] = ( study_id, trial.number, ) self._study_id_and_number_to_trial_id[(study_id, trial.number)] = trial._trial_id study.trials[trial.number] = trial def record_heartbeat(self, trial_id: int) -> None: self._backend.record_heartbeat(trial_id) def _get_stale_trial_ids(self, study_id: int) -> list[int]: return self._backend._get_stale_trial_ids(study_id) def get_heartbeat_interval(self) -> int | None: return self._backend.get_heartbeat_interval() def get_failed_trial_callback(self) -> Callable[["optuna.Study", FrozenTrial], None] | None: return self._backend.get_failed_trial_callback() optuna-4.1.0/optuna/storages/_callbacks.py000066400000000000000000000106511471332314300206130ustar00rootroot00000000000000from __future__ import annotations from typing import Any import optuna from optuna._experimental import experimental_class from optuna._experimental import experimental_func from optuna.trial import FrozenTrial @experimental_class("2.8.0") class RetryFailedTrialCallback: """Retry a failed trial up to a maximum number of times. When a trial fails, this callback can be used with a class in :mod:`optuna.storages` to recreate the trial in ``TrialState.WAITING`` to queue up the trial to be run again. The failed trial can be identified by the :func:`~optuna.storages.RetryFailedTrialCallback.retried_trial_number` function. Even if repetitive failure occurs (a retried trial fails again), this method returns the number of the original trial. To get a full list including the numbers of the retried trials as well as their original trial, call the :func:`~optuna.storages.RetryFailedTrialCallback.retry_history` function. This callback is helpful in environments where trials may fail due to external conditions, such as being preempted by other processes. Usage: .. testcode:: import optuna from optuna.storages import RetryFailedTrialCallback storage = optuna.storages.RDBStorage( url="sqlite:///:memory:", heartbeat_interval=60, grace_period=120, failed_trial_callback=RetryFailedTrialCallback(max_retry=3), ) study = optuna.create_study( storage=storage, ) .. seealso:: See :class:`~optuna.storages.RDBStorage`. Args: max_retry: The max number of times a trial can be retried. Must be set to :obj:`None` or an integer. If set to the default value of :obj:`None` will retry indefinitely. If set to an integer, will only retry that many times. inherit_intermediate_values: Option to inherit `trial.intermediate_values` reported by :func:`optuna.trial.Trial.report` from the failed trial. Default is :obj:`False`. """ def __init__( self, max_retry: int | None = None, inherit_intermediate_values: bool = False ) -> None: self._max_retry = max_retry self._inherit_intermediate_values = inherit_intermediate_values def __call__(self, study: "optuna.study.Study", trial: FrozenTrial) -> None: system_attrs: dict[str, Any] = { "failed_trial": trial.number, "retry_history": [], **trial.system_attrs, } system_attrs["retry_history"].append(trial.number) if self._max_retry is not None: if self._max_retry < len(system_attrs["retry_history"]): return study.add_trial( optuna.create_trial( state=optuna.trial.TrialState.WAITING, params=trial.params, distributions=trial.distributions, user_attrs=trial.user_attrs, system_attrs=system_attrs, intermediate_values=( trial.intermediate_values if self._inherit_intermediate_values else None ), ) ) @staticmethod @experimental_func("2.8.0") def retried_trial_number(trial: FrozenTrial) -> int | None: """Return the number of the original trial being retried. Args: trial: The trial object. Returns: The number of the first failed trial. If not retry of a previous trial, returns :obj:`None`. """ return trial.system_attrs.get("failed_trial", None) @staticmethod @experimental_func("3.0.0") def retry_history(trial: FrozenTrial) -> list[int]: """Return the list of retried trial numbers with respect to the specified trial. Args: trial: The trial object. Returns: A list of trial numbers in ascending order of the series of retried trials. The first item of the list indicates the original trial which is identical to the :func:`~optuna.storages.RetryFailedTrialCallback.retried_trial_number`, and the last item is the one right before the specified trial in the retry series. If the specified trial is not a retry of any trial, returns an empty list. """ return trial.system_attrs.get("retry_history", []) optuna-4.1.0/optuna/storages/_heartbeat.py000066400000000000000000000135271471332314300206400ustar00rootroot00000000000000from __future__ import annotations import abc from collections.abc import Callable import copy from threading import Event from threading import Thread from types import TracebackType from typing import Type import optuna from optuna._experimental import experimental_func from optuna.storages import BaseStorage from optuna.trial import FrozenTrial from optuna.trial import TrialState class BaseHeartbeat(metaclass=abc.ABCMeta): """Base class for heartbeat. This class is not supposed to be directly accessed by library users. The heartbeat mechanism periodically checks whether each trial process is alive during an optimization loop. To support this mechanism, the methods of :class:`~optuna.storages._heartbeat.BaseHeartbeat` is implemented for the target database backend, typically with multiple inheritance of :class:`~optuna.storages._base.BaseStorage` and :class:`~optuna.storages._heartbeat.BaseHeartbeat`. .. seealso:: See :class:`~optuna.storages.RDBStorage`, where the backend supports heartbeat. """ @abc.abstractmethod def record_heartbeat(self, trial_id: int) -> None: """Record the heartbeat of the trial. Args: trial_id: ID of the trial. """ raise NotImplementedError() @abc.abstractmethod def _get_stale_trial_ids(self, study_id: int) -> list[int]: """Get the stale trial ids of the study. Args: study_id: ID of the study. Returns: List of IDs of trials whose heartbeat has not been updated for a long time. """ raise NotImplementedError() @abc.abstractmethod def get_heartbeat_interval(self) -> int | None: """Get the heartbeat interval if it is set. Returns: The heartbeat interval if it is set, otherwise :obj:`None`. """ raise NotImplementedError() @abc.abstractmethod def get_failed_trial_callback(self) -> Callable[["optuna.Study", FrozenTrial], None] | None: """Get the failed trial callback function. Returns: The failed trial callback function if it is set, otherwise :obj:`None`. """ raise NotImplementedError() class BaseHeartbeatThread(metaclass=abc.ABCMeta): def __enter__(self) -> None: self.start() def __exit__( self, exc_type: Type[Exception] | None, exc_value: Exception | None, traceback: TracebackType | None, ) -> None: self.join() @abc.abstractmethod def start(self) -> None: raise NotImplementedError() @abc.abstractmethod def join(self) -> None: raise NotImplementedError() class NullHeartbeatThread(BaseHeartbeatThread): def __init__(self) -> None: pass def start(self) -> None: pass def join(self) -> None: pass class HeartbeatThread(BaseHeartbeatThread): def __init__(self, trial_id: int, heartbeat: BaseHeartbeat) -> None: self._trial_id = trial_id self._heartbeat = heartbeat self._thread: Thread | None = None self._stop_event: Event | None = None def start(self) -> None: self._stop_event = Event() self._thread = Thread( target=self._record_heartbeat, args=(self._trial_id, self._heartbeat, self._stop_event) ) self._thread.start() def join(self) -> None: assert self._stop_event is not None assert self._thread is not None self._stop_event.set() self._thread.join() @staticmethod def _record_heartbeat(trial_id: int, heartbeat: BaseHeartbeat, stop_event: Event) -> None: heartbeat_interval = heartbeat.get_heartbeat_interval() assert heartbeat_interval is not None while True: heartbeat.record_heartbeat(trial_id) if stop_event.wait(timeout=heartbeat_interval): return def get_heartbeat_thread(trial_id: int, storage: BaseStorage) -> BaseHeartbeatThread: if is_heartbeat_enabled(storage): assert isinstance(storage, BaseHeartbeat) return HeartbeatThread(trial_id, storage) else: return NullHeartbeatThread() @experimental_func("2.9.0") def fail_stale_trials(study: "optuna.Study") -> None: """Fail stale trials and run their failure callbacks. The running trials whose heartbeat has not been updated for a long time will be failed, that is, those states will be changed to :obj:`~optuna.trial.TrialState.FAIL`. .. seealso:: See :class:`~optuna.storages.RDBStorage`. Args: study: Study holding the trials to check. """ storage = study._storage if not isinstance(storage, BaseHeartbeat): return if not is_heartbeat_enabled(storage): return failed_trial_ids = [] for trial_id in storage._get_stale_trial_ids(study._study_id): try: if storage.set_trial_state_values(trial_id, state=TrialState.FAIL): failed_trial_ids.append(trial_id) except RuntimeError: # If another process fails the trial, the storage raises RuntimeError. pass failed_trial_callback = storage.get_failed_trial_callback() if failed_trial_callback is not None: for trial_id in failed_trial_ids: failed_trial = copy.deepcopy(storage.get_trial(trial_id)) failed_trial_callback(study, failed_trial) def is_heartbeat_enabled(storage: BaseStorage) -> bool: """Check whether the storage enables the heartbeat. Returns: :obj:`True` if the storage also inherits :class:`~optuna.storages._heartbeat.BaseHeartbeat` and the return value of :meth:`~optuna.storages.BaseStorage.get_heartbeat_interval` is an integer, otherwise :obj:`False`. """ return isinstance(storage, BaseHeartbeat) and storage.get_heartbeat_interval() is not None optuna-4.1.0/optuna/storages/_in_memory.py000066400000000000000000000345071471332314300207000ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Container from collections.abc import Sequence import copy from datetime import datetime import threading from typing import Any import uuid import optuna from optuna import distributions # NOQA from optuna._typing import JSONSerializable from optuna.exceptions import DuplicatedStudyError from optuna.storages import BaseStorage from optuna.storages._base import DEFAULT_STUDY_NAME_PREFIX from optuna.study._frozen import FrozenStudy from optuna.study._study_direction import StudyDirection from optuna.trial import FrozenTrial from optuna.trial import TrialState _logger = optuna.logging.get_logger(__name__) class InMemoryStorage(BaseStorage): """Storage class that stores data in memory of the Python process. Example: Create an :class:`~optuna.storages.InMemoryStorage` instance. .. testcode:: import optuna def objective(trial): x = trial.suggest_float("x", -100, 100) return x**2 storage = optuna.storages.InMemoryStorage() study = optuna.create_study(storage=storage) study.optimize(objective, n_trials=10) """ def __init__(self) -> None: self._trial_id_to_study_id_and_number: dict[int, tuple[int, int]] = {} self._study_name_to_id: dict[str, int] = {} self._studies: dict[int, _StudyInfo] = {} self._max_study_id = -1 self._max_trial_id = -1 self._lock = threading.RLock() def __getstate__(self) -> dict[Any, Any]: state = self.__dict__.copy() del state["_lock"] return state def __setstate__(self, state: dict[Any, Any]) -> None: self.__dict__.update(state) self._lock = threading.RLock() def create_new_study( self, directions: Sequence[StudyDirection], study_name: str | None = None ) -> int: with self._lock: study_id = self._max_study_id + 1 self._max_study_id += 1 if study_name is not None: if study_name in self._study_name_to_id: raise DuplicatedStudyError else: study_uuid = str(uuid.uuid4()) study_name = DEFAULT_STUDY_NAME_PREFIX + study_uuid self._studies[study_id] = _StudyInfo(study_name, list(directions)) self._study_name_to_id[study_name] = study_id _logger.info("A new study created in memory with name: {}".format(study_name)) return study_id def delete_study(self, study_id: int) -> None: with self._lock: self._check_study_id(study_id) for trial in self._studies[study_id].trials: del self._trial_id_to_study_id_and_number[trial._trial_id] study_name = self._studies[study_id].name del self._study_name_to_id[study_name] del self._studies[study_id] def set_study_user_attr(self, study_id: int, key: str, value: Any) -> None: with self._lock: self._check_study_id(study_id) self._studies[study_id].user_attrs[key] = value def set_study_system_attr(self, study_id: int, key: str, value: JSONSerializable) -> None: with self._lock: self._check_study_id(study_id) self._studies[study_id].system_attrs[key] = value def get_study_id_from_name(self, study_name: str) -> int: with self._lock: if study_name not in self._study_name_to_id: raise KeyError("No such study {}.".format(study_name)) return self._study_name_to_id[study_name] def get_study_name_from_id(self, study_id: int) -> str: with self._lock: self._check_study_id(study_id) return self._studies[study_id].name def get_study_directions(self, study_id: int) -> list[StudyDirection]: with self._lock: self._check_study_id(study_id) return self._studies[study_id].directions def get_study_user_attrs(self, study_id: int) -> dict[str, Any]: with self._lock: self._check_study_id(study_id) return self._studies[study_id].user_attrs def get_study_system_attrs(self, study_id: int) -> dict[str, Any]: with self._lock: self._check_study_id(study_id) return self._studies[study_id].system_attrs def get_all_studies(self) -> list[FrozenStudy]: with self._lock: return [self._build_frozen_study(study_id) for study_id in self._studies] def _build_frozen_study(self, study_id: int) -> FrozenStudy: study = self._studies[study_id] return FrozenStudy( study_name=study.name, direction=None, directions=study.directions, user_attrs=copy.deepcopy(study.user_attrs), system_attrs=copy.deepcopy(study.system_attrs), study_id=study_id, ) def create_new_trial(self, study_id: int, template_trial: FrozenTrial | None = None) -> int: with self._lock: self._check_study_id(study_id) if template_trial is None: trial = self._create_running_trial() else: trial = copy.deepcopy(template_trial) trial_id = self._max_trial_id + 1 self._max_trial_id += 1 trial.number = len(self._studies[study_id].trials) trial._trial_id = trial_id self._trial_id_to_study_id_and_number[trial_id] = (study_id, trial.number) self._studies[study_id].trials.append(trial) self._update_cache(trial_id, study_id) return trial_id @staticmethod def _create_running_trial() -> FrozenTrial: return FrozenTrial( trial_id=-1, # dummy value. number=-1, # dummy value. state=TrialState.RUNNING, params={}, distributions={}, user_attrs={}, system_attrs={}, value=None, intermediate_values={}, datetime_start=datetime.now(), datetime_complete=None, ) def set_trial_param( self, trial_id: int, param_name: str, param_value_internal: float, distribution: distributions.BaseDistribution, ) -> None: with self._lock: trial = self._get_trial(trial_id) self.check_trial_is_updatable(trial_id, trial.state) study_id = self._trial_id_to_study_id_and_number[trial_id][0] # Check param distribution compatibility with previous trial(s). if param_name in self._studies[study_id].param_distribution: distributions.check_distribution_compatibility( self._studies[study_id].param_distribution[param_name], distribution ) # Set param distribution. self._studies[study_id].param_distribution[param_name] = distribution # Set param. trial = copy.copy(trial) trial.params = copy.copy(trial.params) trial.params[param_name] = distribution.to_external_repr(param_value_internal) trial.distributions = copy.copy(trial.distributions) trial.distributions[param_name] = distribution self._set_trial(trial_id, trial) def get_trial_id_from_study_id_trial_number(self, study_id: int, trial_number: int) -> int: with self._lock: study = self._studies.get(study_id) if study is None: raise KeyError("No study with study_id {} exists.".format(study_id)) trials = study.trials if len(trials) <= trial_number: raise KeyError( "No trial with trial number {} exists in study with study_id {}.".format( trial_number, study_id ) ) trial = trials[trial_number] assert trial.number == trial_number return trial._trial_id def get_trial_number_from_id(self, trial_id: int) -> int: with self._lock: self._check_trial_id(trial_id) return self._trial_id_to_study_id_and_number[trial_id][1] def get_best_trial(self, study_id: int) -> FrozenTrial: with self._lock: self._check_study_id(study_id) best_trial_id = self._studies[study_id].best_trial_id if best_trial_id is None: raise ValueError("No trials are completed yet.") elif len(self._studies[study_id].directions) > 1: raise RuntimeError( "Best trial can be obtained only for single-objective optimization." ) return self.get_trial(best_trial_id) def get_trial_param(self, trial_id: int, param_name: str) -> float: with self._lock: trial = self._get_trial(trial_id) distribution = trial.distributions[param_name] return distribution.to_internal_repr(trial.params[param_name]) def set_trial_state_values( self, trial_id: int, state: TrialState, values: Sequence[float] | None = None ) -> bool: with self._lock: trial = copy.copy(self._get_trial(trial_id)) self.check_trial_is_updatable(trial_id, trial.state) if state == TrialState.RUNNING and trial.state != TrialState.WAITING: return False trial.state = state if values is not None: trial.values = values if state == TrialState.RUNNING: trial.datetime_start = datetime.now() if state.is_finished(): trial.datetime_complete = datetime.now() self._set_trial(trial_id, trial) study_id = self._trial_id_to_study_id_and_number[trial_id][0] self._update_cache(trial_id, study_id) else: self._set_trial(trial_id, trial) return True def _update_cache(self, trial_id: int, study_id: int) -> None: trial = self._get_trial(trial_id) if trial.state != TrialState.COMPLETE: return best_trial_id = self._studies[study_id].best_trial_id if best_trial_id is None: self._studies[study_id].best_trial_id = trial_id return _directions = self.get_study_directions(study_id) if len(_directions) > 1: return direction = _directions[0] best_trial = self._get_trial(best_trial_id) assert best_trial is not None if best_trial.value is None: self._studies[study_id].best_trial_id = trial_id return # Complete trials do not have `None` values. assert trial.value is not None best_value = best_trial.value new_value = trial.value if direction == StudyDirection.MAXIMIZE: if best_value < new_value: self._studies[study_id].best_trial_id = trial_id else: if best_value > new_value: self._studies[study_id].best_trial_id = trial_id def set_trial_intermediate_value( self, trial_id: int, step: int, intermediate_value: float ) -> None: with self._lock: trial = self._get_trial(trial_id) self.check_trial_is_updatable(trial_id, trial.state) trial = copy.copy(trial) trial.intermediate_values = copy.copy(trial.intermediate_values) trial.intermediate_values[step] = intermediate_value self._set_trial(trial_id, trial) def set_trial_user_attr(self, trial_id: int, key: str, value: Any) -> None: with self._lock: self._check_trial_id(trial_id) trial = self._get_trial(trial_id) self.check_trial_is_updatable(trial_id, trial.state) trial = copy.copy(trial) trial.user_attrs = copy.copy(trial.user_attrs) trial.user_attrs[key] = value self._set_trial(trial_id, trial) def set_trial_system_attr(self, trial_id: int, key: str, value: JSONSerializable) -> None: with self._lock: trial = self._get_trial(trial_id) self.check_trial_is_updatable(trial_id, trial.state) trial = copy.copy(trial) trial.system_attrs = copy.copy(trial.system_attrs) trial.system_attrs[key] = value self._set_trial(trial_id, trial) def get_trial(self, trial_id: int) -> FrozenTrial: with self._lock: return self._get_trial(trial_id) def _get_trial(self, trial_id: int) -> FrozenTrial: self._check_trial_id(trial_id) study_id, trial_number = self._trial_id_to_study_id_and_number[trial_id] return self._studies[study_id].trials[trial_number] def _set_trial(self, trial_id: int, trial: FrozenTrial) -> None: study_id, trial_number = self._trial_id_to_study_id_and_number[trial_id] self._studies[study_id].trials[trial_number] = trial def get_all_trials( self, study_id: int, deepcopy: bool = True, states: Container[TrialState] | None = None, ) -> list[FrozenTrial]: with self._lock: self._check_study_id(study_id) trials = self._studies[study_id].trials if states is not None: trials = [t for t in trials if t.state in states] if deepcopy: trials = copy.deepcopy(trials) else: # This copy is required for the replacing trick in `set_trial_xxx`. trials = copy.copy(trials) return trials def _check_study_id(self, study_id: int) -> None: if study_id not in self._studies: raise KeyError("No study with study_id {} exists.".format(study_id)) def _check_trial_id(self, trial_id: int) -> None: if trial_id not in self._trial_id_to_study_id_and_number: raise KeyError("No trial with trial_id {} exists.".format(trial_id)) class _StudyInfo: def __init__(self, name: str, directions: list[StudyDirection]) -> None: self.trials: list[FrozenTrial] = [] self.param_distribution: dict[str, distributions.BaseDistribution] = {} self.user_attrs: dict[str, Any] = {} self.system_attrs: dict[str, Any] = {} self.name: str = name self.directions: list[StudyDirection] = directions self.best_trial_id: int | None = None optuna-4.1.0/optuna/storages/_rdb/000077500000000000000000000000001471332314300170665ustar00rootroot00000000000000optuna-4.1.0/optuna/storages/_rdb/__init__.py000066400000000000000000000000001471332314300211650ustar00rootroot00000000000000optuna-4.1.0/optuna/storages/_rdb/alembic.ini000066400000000000000000000032411471332314300211630ustar00rootroot00000000000000# A generic, single database configuration. [alembic] # path to migration scripts script_location = alembic # template used to generate migration files # file_template = %%(rev)s_%%(slug)s # timezone to use when rendering the date # within the migration file as well as the filename. # string value is passed to dateutil.tz.gettz() # leave blank for localtime # timezone = # max length of characters to apply to the # "slug" field #truncate_slug_length = 40 # set to 'true' to run the environment during # the 'revision' command, regardless of autogenerate # revision_environment = false # set to 'true' to allow .pyc and .pyo files without # a source .py file to be detected as revisions in the # versions/ directory # sourceless = false # version location specification; this defaults # to alembic/versions. When using multiple version # directories, initial revisions must be specified with --version-path # version_locations = %(here)s/bar %(here)s/bat alembic/versions # the output encoding used when revision files # are written from script.py.mako # output_encoding = utf-8 # NOTE: This URL is only used when generating migration scripts. sqlalchemy.url = sqlite:///alembic.db # Logging configuration [loggers] keys = root,sqlalchemy,alembic [handlers] keys = console [formatters] keys = generic [logger_root] level = WARN handlers = console qualname = [logger_sqlalchemy] level = WARN handlers = qualname = sqlalchemy.engine [logger_alembic] level = INFO handlers = qualname = alembic [handler_console] class = StreamHandler args = (sys.stderr,) level = NOTSET formatter = generic [formatter_generic] format = %(levelname)-5.5s [%(name)s] %(message)s datefmt = %H:%M:%S optuna-4.1.0/optuna/storages/_rdb/alembic/000077500000000000000000000000001471332314300204625ustar00rootroot00000000000000optuna-4.1.0/optuna/storages/_rdb/alembic/env.py000066400000000000000000000041611471332314300216260ustar00rootroot00000000000000import logging from logging.config import fileConfig from alembic import context from sqlalchemy import engine_from_config from sqlalchemy import pool import optuna.storages._rdb.models # this is the Alembic Config object, which provides # access to the values within the .ini file in use. config = context.config # Interpret the config file for Python logging. # This line sets up loggers basically. if len(logging.getLogger().handlers) == 0: fileConfig(config.config_file_name) # add your model's MetaData object here # for 'autogenerate' support # from myapp import mymodel # target_metadata = mymodel.Base.metadata target_metadata = optuna.storages._rdb.models.BaseModel.metadata # other values from the config, defined by the needs of env.py, # can be acquired: # my_important_option = config.get_main_option("my_important_option") # ... etc. def run_migrations_offline(): """Run migrations in 'offline' mode. This configures the context with just a URL and not an Engine, though an Engine is acceptable here as well. By skipping the Engine creation we don't even need a DBAPI to be available. Calls to context.execute() here emit the given string to the script output. """ url = config.get_main_option("sqlalchemy.url") context.configure( url=url, target_metadata=target_metadata, literal_binds=True, render_as_batch=True ) with context.begin_transaction(): context.run_migrations() def run_migrations_online(): """Run migrations in 'online' mode. In this scenario we need to create an Engine and associate a connection with the context. """ connectable = engine_from_config( config.get_section(config.config_ini_section), prefix="sqlalchemy.", poolclass=pool.NullPool, ) with connectable.connect() as connection: context.configure( connection=connection, target_metadata=target_metadata, render_as_batch=True ) with context.begin_transaction(): context.run_migrations() if context.is_offline_mode(): run_migrations_offline() else: run_migrations_online() optuna-4.1.0/optuna/storages/_rdb/alembic/script.py.mako000066400000000000000000000007561471332314300232760ustar00rootroot00000000000000"""${message} Revision ID: ${up_revision} Revises: ${down_revision | comma,n} Create Date: ${create_date} """ from alembic import op import sqlalchemy as sa ${imports if imports else ""} # revision identifiers, used by Alembic. revision = ${repr(up_revision)} down_revision = ${repr(down_revision)} branch_labels = ${repr(branch_labels)} depends_on = ${repr(depends_on)} def upgrade(): ${upgrades if upgrades else "pass"} def downgrade(): ${downgrades if downgrades else "pass"} optuna-4.1.0/optuna/storages/_rdb/alembic/versions/000077500000000000000000000000001471332314300223325ustar00rootroot00000000000000optuna-4.1.0/optuna/storages/_rdb/alembic/versions/v0.9.0.a.py000066400000000000000000000125201471332314300237550ustar00rootroot00000000000000"""empty message Revision ID: v0.9.0.a Revises: Create Date: 2019-03-12 12:30:31.178819 """ from alembic import op import sqlalchemy as sa # revision identifiers, used by Alembic. revision = "v0.9.0.a" down_revision = None branch_labels = None depends_on = None def upgrade(): # ### commands auto generated by Alembic - please adjust! ### op.create_table( "studies", sa.Column("study_id", sa.Integer(), nullable=False), sa.Column("study_name", sa.String(length=512), nullable=False), sa.Column( "direction", sa.Enum("NOT_SET", "MINIMIZE", "MAXIMIZE", name="studydirection"), nullable=False, ), sa.PrimaryKeyConstraint("study_id"), ) op.create_index(op.f("ix_studies_study_name"), "studies", ["study_name"], unique=True) op.create_table( "version_info", sa.Column("version_info_id", sa.Integer(), autoincrement=False, nullable=False), sa.Column("schema_version", sa.Integer(), nullable=True), sa.Column("library_version", sa.String(length=256), nullable=True), sa.CheckConstraint("version_info_id=1"), sa.PrimaryKeyConstraint("version_info_id"), ) op.create_table( "study_system_attributes", sa.Column("study_system_attribute_id", sa.Integer(), nullable=False), sa.Column("study_id", sa.Integer(), nullable=True), sa.Column("key", sa.String(length=512), nullable=True), sa.Column("value_json", sa.String(length=2048), nullable=True), sa.ForeignKeyConstraint(["study_id"], ["studies.study_id"]), sa.PrimaryKeyConstraint("study_system_attribute_id"), sa.UniqueConstraint("study_id", "key"), ) op.create_table( "study_user_attributes", sa.Column("study_user_attribute_id", sa.Integer(), nullable=False), sa.Column("study_id", sa.Integer(), nullable=True), sa.Column("key", sa.String(length=512), nullable=True), sa.Column("value_json", sa.String(length=2048), nullable=True), sa.ForeignKeyConstraint(["study_id"], ["studies.study_id"]), sa.PrimaryKeyConstraint("study_user_attribute_id"), sa.UniqueConstraint("study_id", "key"), ) op.create_table( "trials", sa.Column("trial_id", sa.Integer(), nullable=False), sa.Column("study_id", sa.Integer(), nullable=True), sa.Column( "state", sa.Enum("RUNNING", "COMPLETE", "PRUNED", "FAIL", name="trialstate"), nullable=False, ), sa.Column("value", sa.Float(), nullable=True), sa.Column("datetime_start", sa.DateTime(), nullable=True), sa.Column("datetime_complete", sa.DateTime(), nullable=True), sa.ForeignKeyConstraint(["study_id"], ["studies.study_id"]), sa.PrimaryKeyConstraint("trial_id"), ) op.create_table( "trial_params", sa.Column("param_id", sa.Integer(), nullable=False), sa.Column("trial_id", sa.Integer(), nullable=True), sa.Column("param_name", sa.String(length=512), nullable=True), sa.Column("param_value", sa.Float(), nullable=True), sa.Column("distribution_json", sa.String(length=2048), nullable=True), sa.ForeignKeyConstraint(["trial_id"], ["trials.trial_id"]), sa.PrimaryKeyConstraint("param_id"), sa.UniqueConstraint("trial_id", "param_name"), ) op.create_table( "trial_system_attributes", sa.Column("trial_system_attribute_id", sa.Integer(), nullable=False), sa.Column("trial_id", sa.Integer(), nullable=True), sa.Column("key", sa.String(length=512), nullable=True), sa.Column("value_json", sa.String(length=2048), nullable=True), sa.ForeignKeyConstraint(["trial_id"], ["trials.trial_id"]), sa.PrimaryKeyConstraint("trial_system_attribute_id"), sa.UniqueConstraint("trial_id", "key"), ) op.create_table( "trial_user_attributes", sa.Column("trial_user_attribute_id", sa.Integer(), nullable=False), sa.Column("trial_id", sa.Integer(), nullable=True), sa.Column("key", sa.String(length=512), nullable=True), sa.Column("value_json", sa.String(length=2048), nullable=True), sa.ForeignKeyConstraint(["trial_id"], ["trials.trial_id"]), sa.PrimaryKeyConstraint("trial_user_attribute_id"), sa.UniqueConstraint("trial_id", "key"), ) op.create_table( "trial_values", sa.Column("trial_value_id", sa.Integer(), nullable=False), sa.Column("trial_id", sa.Integer(), nullable=True), sa.Column("step", sa.Integer(), nullable=True), sa.Column("value", sa.Float(), nullable=True), sa.ForeignKeyConstraint(["trial_id"], ["trials.trial_id"]), sa.PrimaryKeyConstraint("trial_value_id"), sa.UniqueConstraint("trial_id", "step"), ) # ### end Alembic commands ### def downgrade(): # ### commands auto generated by Alembic - please adjust! ### op.drop_table("trial_values") op.drop_table("trial_user_attributes") op.drop_table("trial_system_attributes") op.drop_table("trial_params") op.drop_table("trials") op.drop_table("study_user_attributes") op.drop_table("study_system_attributes") op.drop_table("version_info") op.drop_index(op.f("ix_studies_study_name"), table_name="studies") op.drop_table("studies") # ### end Alembic commands ### optuna-4.1.0/optuna/storages/_rdb/alembic/versions/v1.2.0.a.py000066400000000000000000000017041471332314300237510ustar00rootroot00000000000000"""empty message Revision ID: v1.2.0.a Revises: v0.9.0.a Create Date: 2020-02-05 15:17:41.458947 """ from alembic import op import sqlalchemy as sa # revision identifiers, used by Alembic. revision = "v1.2.0.a" down_revision = "v0.9.0.a" branch_labels = None depends_on = None def upgrade(): with op.batch_alter_table("trials") as batch_op: batch_op.alter_column( "state", type_=sa.Enum("RUNNING", "COMPLETE", "PRUNED", "FAIL", "WAITING", name="trialstate"), existing_type=sa.Enum("RUNNING", "COMPLETE", "PRUNED", "FAIL", name="trialstate"), ) def downgrade(): with op.batch_alter_table("trials") as batch_op: batch_op.alter_column( "state", type_=sa.Enum("RUNNING", "COMPLETE", "PRUNED", "FAIL", name="trialstate"), existing_type=sa.Enum( "RUNNING", "COMPLETE", "PRUNED", "FAIL", "WAITING", name="trialstate" ), ) optuna-4.1.0/optuna/storages/_rdb/alembic/versions/v1.3.0.a.py000066400000000000000000000054151471332314300237550ustar00rootroot00000000000000"""empty message Revision ID: v1.3.0.a Revises: v1.2.0.a Create Date: 2020-02-14 16:23:04.800808 """ import json from alembic import op import sqlalchemy as sa from sqlalchemy.exc import SQLAlchemyError from sqlalchemy import orm try: from sqlalchemy.orm import declarative_base except ImportError: # TODO(c-bata): Remove this after dropping support for SQLAlchemy v1.3 or prior. from sqlalchemy.ext.declarative import declarative_base # revision identifiers, used by Alembic. revision = "v1.3.0.a" down_revision = "v1.2.0.a" branch_labels = None depends_on = None # Model definition MAX_INDEXED_STRING_LENGTH = 512 MAX_STRING_LENGTH = 2048 BaseModel = declarative_base() class TrialModel(BaseModel): __tablename__ = "trials" trial_id = sa.Column(sa.Integer, primary_key=True) number = sa.Column(sa.Integer) class TrialSystemAttributeModel(BaseModel): __tablename__ = "trial_system_attributes" trial_system_attribute_id = sa.Column(sa.Integer, primary_key=True) trial_id = sa.Column(sa.Integer, sa.ForeignKey("trials.trial_id")) key = sa.Column(sa.String(MAX_INDEXED_STRING_LENGTH)) value_json = sa.Column(sa.String(MAX_STRING_LENGTH)) def upgrade(): bind = op.get_bind() session = orm.Session(bind=bind) with op.batch_alter_table("trials") as batch_op: batch_op.add_column(sa.Column("number", sa.Integer(), nullable=True, default=None)) try: number_records = ( session.query(TrialSystemAttributeModel) .filter(TrialSystemAttributeModel.key == "_number") .all() ) mapping = [ {"trial_id": r.trial_id, "number": json.loads(r.value_json)} for r in number_records ] session.bulk_update_mappings(TrialModel, mapping) stmt = ( sa.delete(TrialSystemAttributeModel) .where(TrialSystemAttributeModel.key == "_number") .execution_options(synchronize_session=False) ) session.execute(stmt) session.commit() except SQLAlchemyError as e: session.rollback() raise e finally: session.close() def downgrade(): bind = op.get_bind() session = orm.Session(bind=bind) try: number_attrs = [] trials = session.query(TrialModel).all() for trial in trials: number_attrs.append( TrialSystemAttributeModel( trial_id=trial.trial_id, key="_number", value_json=json.dumps(trial.number) ) ) session.bulk_save_objects(number_attrs) session.commit() except SQLAlchemyError as e: session.rollback() raise e finally: session.close() with op.batch_alter_table("trials") as batch_op: batch_op.drop_column("number") optuna-4.1.0/optuna/storages/_rdb/alembic/versions/v2.4.0.a.py000066400000000000000000000144071471332314300237600ustar00rootroot00000000000000"""empty message Revision ID: v2.4.0.a Revises: v1.3.0.a Create Date: 2020-11-17 02:16:16.536171 """ from alembic import op import sqlalchemy as sa from typing import Any from sqlalchemy import Column from sqlalchemy import Enum from sqlalchemy import Float from sqlalchemy import ForeignKey from sqlalchemy import Integer from sqlalchemy import UniqueConstraint from sqlalchemy.exc import SQLAlchemyError from sqlalchemy import orm from optuna.study import StudyDirection try: from sqlalchemy.orm import declarative_base except ImportError: # TODO(c-bata): Remove this after dropping support for SQLAlchemy v1.3 or prior. from sqlalchemy.ext.declarative import declarative_base # revision identifiers, used by Alembic. revision = "v2.4.0.a" down_revision = "v1.3.0.a" branch_labels = None depends_on = None # Model definition BaseModel = declarative_base() class StudyModel(BaseModel): __tablename__ = "studies" study_id = Column(Integer, primary_key=True) direction = sa.Column(sa.Enum(StudyDirection)) class StudyDirectionModel(BaseModel): __tablename__ = "study_directions" __table_args__: Any = (UniqueConstraint("study_id", "objective"),) study_direction_id = Column(Integer, primary_key=True) direction = Column(Enum(StudyDirection), nullable=False) study_id = Column(Integer, ForeignKey("studies.study_id"), nullable=False) objective = Column(Integer, nullable=False) class TrialModel(BaseModel): __tablename__ = "trials" trial_id = Column(Integer, primary_key=True) number = Column(Integer) study_id = Column(Integer, ForeignKey("studies.study_id")) value = sa.Column(sa.Float) class TrialValueModel(BaseModel): __tablename__ = "trial_values" __table_args__: Any = (UniqueConstraint("trial_id", "objective"),) trial_value_id = Column(Integer, primary_key=True) trial_id = Column(Integer, ForeignKey("trials.trial_id"), nullable=False) objective = Column(Integer, nullable=False) value = Column(Float, nullable=False) step = sa.Column(sa.Integer) class TrialIntermediateValueModel(BaseModel): __tablename__ = "trial_intermediate_values" __table_args__: Any = (UniqueConstraint("trial_id", "step"),) trial_intermediate_value_id = Column(Integer, primary_key=True) trial_id = Column(Integer, ForeignKey("trials.trial_id"), nullable=False) step = Column(Integer, nullable=False) intermediate_value = Column(Float, nullable=False) def upgrade(): bind = op.get_bind() inspector = sa.inspect(bind) tables = inspector.get_table_names() if "study_directions" not in tables: op.create_table( "study_directions", sa.Column("study_direction_id", sa.Integer(), nullable=False), sa.Column( "direction", sa.Enum("NOT_SET", "MINIMIZE", "MAXIMIZE", name="studydirection"), nullable=False, ), sa.Column("study_id", sa.Integer(), nullable=False), sa.Column("objective", sa.Integer(), nullable=False), sa.ForeignKeyConstraint( ["study_id"], ["studies.study_id"], ), sa.PrimaryKeyConstraint("study_direction_id"), sa.UniqueConstraint("study_id", "objective"), ) if "trial_intermediate_values" not in tables: op.create_table( "trial_intermediate_values", sa.Column("trial_intermediate_value_id", sa.Integer(), nullable=False), sa.Column("trial_id", sa.Integer(), nullable=False), sa.Column("step", sa.Integer(), nullable=False), sa.Column("intermediate_value", sa.Float(), nullable=False), sa.ForeignKeyConstraint( ["trial_id"], ["trials.trial_id"], ), sa.PrimaryKeyConstraint("trial_intermediate_value_id"), sa.UniqueConstraint("trial_id", "step"), ) session = orm.Session(bind=bind) try: studies_records = session.query(StudyModel).all() objects = [ StudyDirectionModel(study_id=r.study_id, direction=r.direction, objective=0) for r in studies_records ] session.bulk_save_objects(objects) intermediate_values_records = session.query( TrialValueModel.trial_id, TrialValueModel.value, TrialValueModel.step ).all() objects = [ TrialIntermediateValueModel( trial_id=r.trial_id, intermediate_value=r.value, step=r.step ) for r in intermediate_values_records ] session.bulk_save_objects(objects) session.query(TrialValueModel).delete() session.commit() with op.batch_alter_table("trial_values", schema=None) as batch_op: batch_op.add_column(sa.Column("objective", sa.Integer(), nullable=False)) # The name of this constraint is manually determined. # In the future, the naming convention may be determined based on # https://alembic.sqlalchemy.org/en/latest/naming.html batch_op.create_unique_constraint( "uq_trial_values_trial_id_objective", ["trial_id", "objective"] ) trials_records = session.query(TrialModel).all() objects = [ TrialValueModel(trial_id=r.trial_id, value=r.value, objective=0) for r in trials_records ] session.bulk_save_objects(objects) session.commit() except SQLAlchemyError as e: session.rollback() raise e finally: session.close() with op.batch_alter_table("studies", schema=None) as batch_op: batch_op.drop_column("direction") with op.batch_alter_table("trial_values", schema=None) as batch_op: batch_op.drop_column("step") with op.batch_alter_table("trials", schema=None) as batch_op: batch_op.drop_column("value") for c in inspector.get_unique_constraints("trial_values"): # MySQL changes the uniq constraint of (trial_id, step) to that of trial_id. if c["column_names"] == ["trial_id"]: with op.batch_alter_table("trial_values", schema=None) as batch_op: batch_op.drop_constraint(c["name"], type_="unique") break # TODO(imamura): Implement downgrade def downgrade(): pass optuna-4.1.0/optuna/storages/_rdb/alembic/versions/v2.6.0.a_.py000066400000000000000000000032731471332314300241200ustar00rootroot00000000000000"""empty message Revision ID: v2.6.0.a Revises: v2.4.0.a Create Date: 2021-03-01 11:30:32.214196 """ from alembic import op import sqlalchemy as sa # revision identifiers, used by Alembic. revision = "v2.6.0.a" down_revision = "v2.4.0.a" branch_labels = None depends_on = None MAX_STRING_LENGTH = 2048 def upgrade(): with op.batch_alter_table("study_user_attributes") as batch_op: batch_op.alter_column("value_json", type_=sa.TEXT) with op.batch_alter_table("study_system_attributes") as batch_op: batch_op.alter_column("value_json", type_=sa.TEXT) with op.batch_alter_table("trial_user_attributes") as batch_op: batch_op.alter_column("value_json", type_=sa.TEXT) with op.batch_alter_table("trial_system_attributes") as batch_op: batch_op.alter_column("value_json", type_=sa.TEXT) with op.batch_alter_table("trial_params") as batch_op: batch_op.alter_column("distribution_json", type_=sa.TEXT) def downgrade(): with op.batch_alter_table("study_user_attributes") as batch_op: batch_op.alter_column("value_json", type_=sa.String(MAX_STRING_LENGTH)) with op.batch_alter_table("study_system_attributes") as batch_op: batch_op.alter_column("value_json", type_=sa.String(MAX_STRING_LENGTH)) with op.batch_alter_table("trial_user_attributes") as batch_op: batch_op.alter_column("value_json", type_=sa.String(MAX_STRING_LENGTH)) with op.batch_alter_table("trial_system_attributes") as batch_op: batch_op.alter_column("value_json", type_=sa.String(MAX_STRING_LENGTH)) with op.batch_alter_table("trial_params") as batch_op: batch_op.alter_column("distribution_json", type_=sa.String(MAX_STRING_LENGTH)) optuna-4.1.0/optuna/storages/_rdb/alembic/versions/v3.0.0.a.py000066400000000000000000000140771471332314300237600ustar00rootroot00000000000000"""unify existing distributions to {int,float} distribution Revision ID: v3.0.0.a Revises: v2.6.0.a Create Date: 2021-11-21 23:48:42.424430 """ from typing import Any from typing import List import sqlalchemy as sa from alembic import op from sqlalchemy import Column from sqlalchemy import DateTime from sqlalchemy import Enum from sqlalchemy import Float from sqlalchemy import ForeignKey from sqlalchemy import Integer from sqlalchemy import orm from sqlalchemy import String from sqlalchemy import Text from sqlalchemy import UniqueConstraint from sqlalchemy.exc import SQLAlchemyError from optuna.distributions import _convert_old_distribution_to_new_distribution from optuna.distributions import BaseDistribution from optuna.distributions import DiscreteUniformDistribution from optuna.distributions import distribution_to_json from optuna.distributions import FloatDistribution from optuna.distributions import IntDistribution from optuna.distributions import IntLogUniformDistribution from optuna.distributions import IntUniformDistribution from optuna.distributions import json_to_distribution from optuna.distributions import LogUniformDistribution from optuna.distributions import UniformDistribution from optuna.trial import TrialState try: from sqlalchemy.orm import declarative_base except ImportError: # TODO(c-bata): Remove this after dropping support for SQLAlchemy v1.3 or prior. from sqlalchemy.ext.declarative import declarative_base # revision identifiers, used by Alembic. revision = "v3.0.0.a" down_revision = "v2.6.0.a" branch_labels = None depends_on = None MAX_INDEXED_STRING_LENGTH = 512 BATCH_SIZE = 5000 BaseModel = declarative_base() class StudyModel(BaseModel): __tablename__ = "studies" study_id = Column(Integer, primary_key=True) study_name = Column(String(MAX_INDEXED_STRING_LENGTH), index=True, unique=True, nullable=False) class TrialModel(BaseModel): __tablename__ = "trials" trial_id = Column(Integer, primary_key=True) number = Column(Integer) study_id = Column(Integer, ForeignKey("studies.study_id")) state = Column(Enum(TrialState), nullable=False) datetime_start = Column(DateTime) datetime_complete = Column(DateTime) class TrialParamModel(BaseModel): __tablename__ = "trial_params" __table_args__: Any = (UniqueConstraint("trial_id", "param_name"),) param_id = Column(Integer, primary_key=True) trial_id = Column(Integer, ForeignKey("trials.trial_id")) param_name = Column(String(MAX_INDEXED_STRING_LENGTH)) param_value = Column(Float) distribution_json = Column(Text()) def migrate_new_distribution(distribution_json: str) -> str: distribution = json_to_distribution(distribution_json) new_distribution = _convert_old_distribution_to_new_distribution( distribution, suppress_warning=True, ) return distribution_to_json(new_distribution) def restore_old_distribution(distribution_json: str) -> str: distribution = json_to_distribution(distribution_json) old_distribution: BaseDistribution # Float distributions. if isinstance(distribution, FloatDistribution): if distribution.log: old_distribution = LogUniformDistribution( low=distribution.low, high=distribution.high, ) else: if distribution.step is not None: old_distribution = DiscreteUniformDistribution( low=distribution.low, high=distribution.high, q=distribution.step, ) else: old_distribution = UniformDistribution( low=distribution.low, high=distribution.high, ) # Integer distributions. elif isinstance(distribution, IntDistribution): if distribution.log: old_distribution = IntLogUniformDistribution( low=distribution.low, high=distribution.high, step=distribution.step, ) else: old_distribution = IntUniformDistribution( low=distribution.low, high=distribution.high, step=distribution.step, ) # Categorical distribution. else: old_distribution = distribution return distribution_to_json(old_distribution) def persist(session: orm.Session, distributions: List[BaseDistribution]) -> None: if len(distributions) == 0: return session.bulk_save_objects(distributions) session.commit() def upgrade() -> None: bind = op.get_bind() inspector = sa.inspect(bind) tables = inspector.get_table_names() assert "trial_params" in tables session = orm.Session(bind=bind) try: distributions: List[BaseDistribution] = [] for distribution in session.query(TrialParamModel).yield_per(BATCH_SIZE): distribution.distribution_json = migrate_new_distribution( distribution.distribution_json, ) distributions.append(distribution) if len(distributions) == BATCH_SIZE: persist(session, distributions) distributions = [] persist(session, distributions) except SQLAlchemyError as e: session.rollback() raise e finally: session.close() def downgrade() -> None: bind = op.get_bind() inspector = sa.inspect(bind) tables = inspector.get_table_names() assert "trial_params" in tables session = orm.Session(bind=bind) try: distributions = [] for distribution in session.query(TrialParamModel).yield_per(BATCH_SIZE): distribution.distribution_json = restore_old_distribution( distribution.distribution_json, ) distributions.append(distribution) if len(distributions) == BATCH_SIZE: persist(session, distributions) distributions = [] persist(session, distributions) except SQLAlchemyError as e: session.rollback() raise e finally: session.close() optuna-4.1.0/optuna/storages/_rdb/alembic/versions/v3.0.0.b.py000066400000000000000000000056141471332314300237560ustar00rootroot00000000000000"""Change floating point precision and make intermediate_value nullable. Revision ID: v3.0.0.b Revises: v3.0.0.a Create Date: 2022-04-27 16:31:42.012666 """ import enum from alembic import op from sqlalchemy import and_ from sqlalchemy import Column from sqlalchemy import Enum from sqlalchemy import Float from sqlalchemy import ForeignKey from sqlalchemy import Integer from sqlalchemy.orm import Session try: from sqlalchemy.orm import declarative_base except ImportError: # TODO(c-bata): Remove this after dropping support for SQLAlchemy v1.3 or prior. from sqlalchemy.ext.declarative import declarative_base # revision identifiers, used by Alembic. revision = "v3.0.0.b" down_revision = "v3.0.0.a" branch_labels = None depends_on = None BaseModel = declarative_base() FLOAT_PRECISION = 53 class TrialState(enum.Enum): RUNNING = 0 COMPLETE = 1 PRUNED = 2 FAIL = 3 WAITING = 4 class TrialModel(BaseModel): __tablename__ = "trials" trial_id = Column(Integer, primary_key=True) number = Column(Integer) state = Column(Enum(TrialState), nullable=False) class TrialValueModel(BaseModel): __tablename__ = "trial_values" trial_value_id = Column(Integer, primary_key=True) trial_id = Column(Integer, ForeignKey("trials.trial_id"), nullable=False) value = Column(Float, nullable=False) def upgrade(): bind = op.get_bind() session = Session(bind=bind) if ( session.query(TrialValueModel) .join(TrialModel, TrialValueModel.trial_id == TrialModel.trial_id) .filter(and_(TrialModel.state == TrialState.COMPLETE, TrialValueModel.value.is_(None))) .count() ) != 0: raise ValueError("Found invalid trial_values records (value=None and state='COMPLETE')") session.query(TrialValueModel).filter(TrialValueModel.value.is_(None)).delete() with op.batch_alter_table("trial_intermediate_values") as batch_op: batch_op.alter_column( "intermediate_value", type_=Float(precision=FLOAT_PRECISION), nullable=True, ) with op.batch_alter_table("trial_params") as batch_op: batch_op.alter_column( "param_value", type_=Float(precision=FLOAT_PRECISION), existing_nullable=True, ) with op.batch_alter_table("trial_values") as batch_op: batch_op.alter_column( "value", type_=Float(precision=FLOAT_PRECISION), nullable=False, ) def downgrade(): with op.batch_alter_table("trial_intermediate_values") as batch_op: batch_op.alter_column("intermediate_value", type_=Float, nullable=False) with op.batch_alter_table("trial_params") as batch_op: batch_op.alter_column("param_value", type_=Float, existing_nullable=True) with op.batch_alter_table("trial_values") as batch_op: batch_op.alter_column("value", type_=Float, existing_nullable=False) optuna-4.1.0/optuna/storages/_rdb/alembic/versions/v3.0.0.c.py000066400000000000000000000144301471332314300237530ustar00rootroot00000000000000"""Add intermediate_value_type column to represent +inf and -inf Revision ID: v3.0.0.c Revises: v3.0.0.b Create Date: 2022-05-16 17:17:28.810792 """ import enum import numpy as np from alembic import op import sqlalchemy as sa from sqlalchemy.exc import SQLAlchemyError from sqlalchemy import orm from typing import Optional from typing import Tuple try: from sqlalchemy.orm import declarative_base except ImportError: # TODO(c-bata): Remove this after dropping support for SQLAlchemy v1.3 or prior. from sqlalchemy.ext.declarative import declarative_base # revision identifiers, used by Alembic. revision = "v3.0.0.c" down_revision = "v3.0.0.b" branch_labels = None depends_on = None BaseModel = declarative_base() RDB_MAX_FLOAT = np.finfo(np.float32).max RDB_MIN_FLOAT = np.finfo(np.float32).min FLOAT_PRECISION = 53 class IntermediateValueModel(BaseModel): class TrialIntermediateValueType(enum.Enum): FINITE = 1 INF_POS = 2 INF_NEG = 3 NAN = 4 __tablename__ = "trial_intermediate_values" trial_intermediate_value_id = sa.Column(sa.Integer, primary_key=True) intermediate_value = sa.Column(sa.Float(precision=FLOAT_PRECISION), nullable=True) intermediate_value_type = sa.Column(sa.Enum(TrialIntermediateValueType), nullable=False) @classmethod def intermediate_value_to_stored_repr( cls, value: float, ) -> Tuple[Optional[float], TrialIntermediateValueType]: if np.isnan(value): return (None, cls.TrialIntermediateValueType.NAN) elif value == float("inf"): return (None, cls.TrialIntermediateValueType.INF_POS) elif value == float("-inf"): return (None, cls.TrialIntermediateValueType.INF_NEG) else: return (value, cls.TrialIntermediateValueType.FINITE) def upgrade(): bind = op.get_bind() inspector = sa.inspect(bind) column_names = [c["name"] for c in inspector.get_columns("trial_intermediate_values")] sa.Enum(IntermediateValueModel.TrialIntermediateValueType).create(bind, checkfirst=True) # MySQL and PostgreSQL supports DEFAULT clause like 'ALTER TABLE # ADD COLUMN ... DEFAULT "FINITE_OR_NAN"', but seemingly Alembic # does not support such a SQL statement. So first add a column with schema-level # default value setting, then remove it by `batch_op.alter_column()`. if "intermediate_value_type" not in column_names: with op.batch_alter_table("trial_intermediate_values") as batch_op: batch_op.add_column( sa.Column( "intermediate_value_type", sa.Enum( "FINITE", "INF_POS", "INF_NEG", "NAN", name="trialintermediatevaluetype" ), nullable=False, server_default="FINITE", ), ) with op.batch_alter_table("trial_intermediate_values") as batch_op: batch_op.alter_column( "intermediate_value_type", existing_type=sa.Enum( "FINITE", "INF_POS", "INF_NEG", "NAN", name="trialintermediatevaluetype" ), existing_nullable=False, server_default=None, ) session = orm.Session(bind=bind) try: records = ( session.query(IntermediateValueModel) .filter( sa.or_( IntermediateValueModel.intermediate_value > 1e16, IntermediateValueModel.intermediate_value < -1e16, IntermediateValueModel.intermediate_value.is_(None), ) ) .all() ) mapping = [] for r in records: value: float if r.intermediate_value is None or np.isnan(r.intermediate_value): value = float("nan") elif np.isclose(r.intermediate_value, RDB_MAX_FLOAT) or np.isposinf( r.intermediate_value ): value = float("inf") elif np.isclose(r.intermediate_value, RDB_MIN_FLOAT) or np.isneginf( r.intermediate_value ): value = float("-inf") else: value = r.intermediate_value ( stored_value, float_type, ) = IntermediateValueModel.intermediate_value_to_stored_repr(value) mapping.append( { "trial_intermediate_value_id": r.trial_intermediate_value_id, "intermediate_value_type": float_type, "intermediate_value": stored_value, } ) session.bulk_update_mappings(IntermediateValueModel, mapping) session.commit() except SQLAlchemyError as e: session.rollback() raise e finally: session.close() def downgrade(): bind = op.get_bind() session = orm.Session(bind=bind) try: records = session.query(IntermediateValueModel).all() mapping = [] for r in records: if ( r.intermediate_value_type == IntermediateValueModel.TrialIntermediateValueType.FINITE or r.intermediate_value_type == IntermediateValueModel.TrialIntermediateValueType.NAN ): continue _intermediate_value = r.intermediate_value if ( r.intermediate_value_type == IntermediateValueModel.TrialIntermediateValueType.INF_POS ): _intermediate_value = RDB_MAX_FLOAT else: _intermediate_value = RDB_MIN_FLOAT mapping.append( { "trial_intermediate_value_id": r.trial_intermediate_value_id, "intermediate_value": _intermediate_value, } ) session.bulk_update_mappings(IntermediateValueModel, mapping) session.commit() except SQLAlchemyError as e: session.rollback() raise e finally: session.close() with op.batch_alter_table("trial_intermediate_values", schema=None) as batch_op: batch_op.drop_column("intermediate_value_type") sa.Enum(IntermediateValueModel.FloatTypeEnum).drop(bind, checkfirst=True) optuna-4.1.0/optuna/storages/_rdb/alembic/versions/v3.0.0.d.py000066400000000000000000000133401471332314300237530ustar00rootroot00000000000000"""Handle inf/-inf for trial_values table. Revision ID: v3.0.0.d Revises: v3.0.0.c Create Date: 2022-06-02 09:57:22.818798 """ import enum import numpy as np from alembic import op import sqlalchemy as sa from sqlalchemy.exc import SQLAlchemyError from sqlalchemy import orm from typing import Optional from typing import Tuple try: from sqlalchemy.orm import declarative_base except ImportError: # TODO(c-bata): Remove this after dropping support for SQLAlchemy v1.3 or prior. from sqlalchemy.ext.declarative import declarative_base # revision identifiers, used by Alembic. revision = "v3.0.0.d" down_revision = "v3.0.0.c" branch_labels = None depends_on = None BaseModel = declarative_base() RDB_MAX_FLOAT = np.finfo(np.float32).max RDB_MIN_FLOAT = np.finfo(np.float32).min FLOAT_PRECISION = 53 class TrialValueModel(BaseModel): class TrialValueType(enum.Enum): FINITE = 1 INF_POS = 2 INF_NEG = 3 __tablename__ = "trial_values" trial_value_id = sa.Column(sa.Integer, primary_key=True) value = sa.Column(sa.Float(precision=FLOAT_PRECISION), nullable=True) value_type = sa.Column(sa.Enum(TrialValueType), nullable=False) @classmethod def value_to_stored_repr( cls, value: float, ) -> Tuple[Optional[float], TrialValueType]: if value == float("inf"): return (None, cls.TrialValueType.INF_POS) elif value == float("-inf"): return (None, cls.TrialValueType.INF_NEG) else: return (value, cls.TrialValueType.FINITE) @classmethod def stored_repr_to_value(cls, value: Optional[float], float_type: TrialValueType) -> float: if float_type == cls.TrialValueType.INF_POS: assert value is None return float("inf") elif float_type == cls.TrialValueType.INF_NEG: assert value is None return float("-inf") else: assert float_type == cls.TrialValueType.FINITE assert value is not None return value def upgrade(): bind = op.get_bind() inspector = sa.inspect(bind) column_names = [c["name"] for c in inspector.get_columns("trial_values")] sa.Enum(TrialValueModel.TrialValueType).create(bind, checkfirst=True) # MySQL and PostgreSQL supports DEFAULT clause like 'ALTER TABLE # ADD COLUMN ... DEFAULT "FINITE"', but seemingly Alembic # does not support such a SQL statement. So first add a column with schema-level # default value setting, then remove it by `batch_op.alter_column()`. if "value_type" not in column_names: with op.batch_alter_table("trial_values") as batch_op: batch_op.add_column( sa.Column( "value_type", sa.Enum("FINITE", "INF_POS", "INF_NEG", name="trialvaluetype"), nullable=False, server_default="FINITE", ), ) with op.batch_alter_table("trial_values") as batch_op: batch_op.alter_column( "value_type", existing_type=sa.Enum("FINITE", "INF_POS", "INF_NEG", name="trialvaluetype"), existing_nullable=False, server_default=None, ) batch_op.alter_column( "value", existing_type=sa.Float(precision=FLOAT_PRECISION), nullable=True, ) session = orm.Session(bind=bind) try: records = ( session.query(TrialValueModel) .filter( sa.or_( TrialValueModel.value > 1e16, TrialValueModel.value < -1e16, ) ) .all() ) mapping = [] for r in records: value: float if np.isclose(r.value, RDB_MAX_FLOAT) or np.isposinf(r.value): value = float("inf") elif np.isclose(r.value, RDB_MIN_FLOAT) or np.isneginf(r.value): value = float("-inf") else: value = r.value ( stored_value, float_type, ) = TrialValueModel.value_to_stored_repr(value) mapping.append( { "trial_value_id": r.trial_value_id, "value_type": float_type, "value": stored_value, } ) session.bulk_update_mappings(TrialValueModel, mapping) session.commit() except SQLAlchemyError as e: session.rollback() raise e finally: session.close() def downgrade(): bind = op.get_bind() session = orm.Session(bind=bind) try: records = session.query(TrialValueModel).all() mapping = [] for r in records: if r.value_type == TrialValueModel.TrialValueType.FINITE: continue _value = r.value if r.value_type == TrialValueModel.TrialValueType.INF_POS: _value = RDB_MAX_FLOAT else: _value = RDB_MIN_FLOAT mapping.append( { "trial_value_id": r.trial_value_id, "value": _value, } ) session.bulk_update_mappings(TrialValueModel, mapping) session.commit() except SQLAlchemyError as e: session.rollback() raise e finally: session.close() with op.batch_alter_table("trial_values", schema=None) as batch_op: batch_op.drop_column("value_type") batch_op.alter_column( "value", existing_type=sa.Float(precision=FLOAT_PRECISION), nullable=False, ) sa.Enum(TrialValueModel.TrialValueType).drop(bind, checkfirst=True) optuna-4.1.0/optuna/storages/_rdb/alembic/versions/v3.2.0.a_.py000066400000000000000000000013121471332314300241050ustar00rootroot00000000000000"""Add index to study_id column in trials table Revision ID: v3.2.0.a Revises: v3.0.0.d Create Date: 2023-02-25 13:21:00.730272 """ from alembic import op # revision identifiers, used by Alembic. revision = "v3.2.0.a" down_revision = "v3.0.0.d" branch_labels = None depends_on = None def upgrade(): op.create_index(op.f("trials_study_id_key"), "trials", ["study_id"], unique=False) def downgrade(): # The following operation doesn't work on MySQL due to a foreign key constraint. # # mysql> DROP INDEX ix_trials_study_id ON trials; # ERROR: Cannot drop index 'ix_trials_study_id': needed in a foreign key constraint. op.drop_index(op.f("trials_study_id_key"), table_name="trials") optuna-4.1.0/optuna/storages/_rdb/models.py000066400000000000000000000464571471332314300207430ustar00rootroot00000000000000from __future__ import annotations import enum import math from typing import Any from sqlalchemy import asc from sqlalchemy import case from sqlalchemy import CheckConstraint from sqlalchemy import DateTime from sqlalchemy import desc from sqlalchemy import Enum from sqlalchemy import Float from sqlalchemy import ForeignKey from sqlalchemy import func from sqlalchemy import Integer from sqlalchemy import orm from sqlalchemy import String from sqlalchemy import Text from sqlalchemy import UniqueConstraint from optuna import distributions from optuna.study._study_direction import StudyDirection from optuna.trial import TrialState try: from sqlalchemy.orm import declarative_base except ImportError: # TODO(c-bata): Remove this after dropping support for SQLAlchemy v1.3 or prior. from sqlalchemy.ext.declarative import declarative_base try: from sqlalchemy.orm import mapped_column _Column = mapped_column except ImportError: # TODO(Shinichi): Remove this after dropping support for SQLAlchemy<2.0. from sqlalchemy import Column as _Column # type: ignore[assignment, no-redef] # Don't modify this version number anymore. # The schema management functionality has been moved to alembic. SCHEMA_VERSION = 12 MAX_INDEXED_STRING_LENGTH = 512 MAX_VERSION_LENGTH = 256 NOT_FOUND_MSG = "Record does not exist." FLOAT_PRECISION = 53 BaseModel: Any = declarative_base() class StudyModel(BaseModel): __tablename__ = "studies" study_id = _Column(Integer, primary_key=True) study_name = _Column( String(MAX_INDEXED_STRING_LENGTH), index=True, unique=True, nullable=False ) @classmethod def find_or_raise_by_id( cls, study_id: int, session: orm.Session, for_update: bool = False ) -> "StudyModel": query = session.query(cls).filter(cls.study_id == study_id) if for_update: query = query.with_for_update() study = query.one_or_none() if study is None: raise KeyError(NOT_FOUND_MSG) return study @classmethod def find_by_name(cls, study_name: str, session: orm.Session) -> "StudyModel" | None: study = session.query(cls).filter(cls.study_name == study_name).one_or_none() return study @classmethod def find_or_raise_by_name(cls, study_name: str, session: orm.Session) -> "StudyModel": study = cls.find_by_name(study_name, session) if study is None: raise KeyError(NOT_FOUND_MSG) return study class StudyDirectionModel(BaseModel): __tablename__ = "study_directions" __table_args__: Any = (UniqueConstraint("study_id", "objective"),) study_direction_id = _Column(Integer, primary_key=True) direction = _Column(Enum(StudyDirection), nullable=False) study_id = _Column(Integer, ForeignKey("studies.study_id"), nullable=False) objective = _Column(Integer, nullable=False) study = orm.relationship( StudyModel, backref=orm.backref("directions", cascade="all, delete-orphan") ) @classmethod def where_study_id(cls, study_id: int, session: orm.Session) -> list["StudyDirectionModel"]: return session.query(cls).filter(cls.study_id == study_id).all() class StudyUserAttributeModel(BaseModel): __tablename__ = "study_user_attributes" __table_args__: Any = (UniqueConstraint("study_id", "key"),) study_user_attribute_id = _Column(Integer, primary_key=True) study_id = _Column(Integer, ForeignKey("studies.study_id")) key = _Column(String(MAX_INDEXED_STRING_LENGTH)) value_json = _Column(Text()) study = orm.relationship( StudyModel, backref=orm.backref("user_attributes", cascade="all, delete-orphan") ) @classmethod def find_by_study_and_key( cls, study: StudyModel, key: str, session: orm.Session ) -> "StudyUserAttributeModel" | None: attribute = ( session.query(cls) .filter(cls.study_id == study.study_id) .filter(cls.key == key) .one_or_none() ) return attribute @classmethod def where_study_id( cls, study_id: int, session: orm.Session ) -> list["StudyUserAttributeModel"]: return session.query(cls).filter(cls.study_id == study_id).all() class StudySystemAttributeModel(BaseModel): __tablename__ = "study_system_attributes" __table_args__: Any = (UniqueConstraint("study_id", "key"),) study_system_attribute_id = _Column(Integer, primary_key=True) study_id = _Column(Integer, ForeignKey("studies.study_id")) key = _Column(String(MAX_INDEXED_STRING_LENGTH)) value_json = _Column(Text()) study = orm.relationship( StudyModel, backref=orm.backref("system_attributes", cascade="all, delete-orphan") ) @classmethod def find_by_study_and_key( cls, study: StudyModel, key: str, session: orm.Session ) -> "StudySystemAttributeModel" | None: attribute = ( session.query(cls) .filter(cls.study_id == study.study_id) .filter(cls.key == key) .one_or_none() ) return attribute @classmethod def where_study_id( cls, study_id: int, session: orm.Session ) -> list["StudySystemAttributeModel"]: return session.query(cls).filter(cls.study_id == study_id).all() class TrialModel(BaseModel): __tablename__ = "trials" trial_id = _Column(Integer, primary_key=True) # No `UniqueConstraint` is put on the `number` columns although it in practice is constrained # to be unique. This is to reduce code complexity as table-level locking would be required # otherwise. See https://github.com/optuna/optuna/pull/939#discussion_r387447632. number = _Column(Integer) study_id = _Column(Integer, ForeignKey("studies.study_id"), index=True) state = _Column(Enum(TrialState), nullable=False) datetime_start = _Column(DateTime) datetime_complete = _Column(DateTime) study = orm.relationship( StudyModel, backref=orm.backref("trials", cascade="all, delete-orphan") ) @classmethod def find_max_value_trial_id(cls, study_id: int, objective: int, session: orm.Session) -> int: trial = ( session.query(cls) .with_entities(cls.trial_id) .filter(cls.study_id == study_id) .filter(cls.state == TrialState.COMPLETE) .join(TrialValueModel) .filter(TrialValueModel.objective == objective) .order_by( desc( case( {"INF_NEG": -1, "FINITE": 0, "INF_POS": 1}, value=TrialValueModel.value_type, ) ), desc(TrialValueModel.value), ) .limit(1) .one_or_none() ) if trial is None: raise ValueError(NOT_FOUND_MSG) return trial[0] @classmethod def find_min_value_trial_id(cls, study_id: int, objective: int, session: orm.Session) -> int: trial = ( session.query(cls) .with_entities(cls.trial_id) .filter(cls.study_id == study_id) .filter(cls.state == TrialState.COMPLETE) .join(TrialValueModel) .filter(TrialValueModel.objective == objective) .order_by( asc( case( {"INF_NEG": -1, "FINITE": 0, "INF_POS": 1}, value=TrialValueModel.value_type, ) ), asc(TrialValueModel.value), ) .limit(1) .one_or_none() ) if trial is None: raise ValueError(NOT_FOUND_MSG) return trial[0] @classmethod def find_or_raise_by_id( cls, trial_id: int, session: orm.Session, for_update: bool = False ) -> "TrialModel": query = session.query(cls).filter(cls.trial_id == trial_id) # "FOR UPDATE" clause is used for row-level locking. # Please note that SQLite3 doesn't support this clause. if for_update: query = query.with_for_update() trial = query.one_or_none() if trial is None: raise KeyError(NOT_FOUND_MSG) return trial @classmethod def count( cls, session: orm.Session, study: StudyModel | None = None, state: TrialState | None = None ) -> int: trial_count = session.query(func.count(cls.trial_id)) if study is not None: trial_count = trial_count.filter(cls.study_id == study.study_id) if state is not None: trial_count = trial_count.filter(cls.state == state) return trial_count.scalar() def count_past_trials(self, session: orm.Session) -> int: trial_count = session.query(func.count(TrialModel.trial_id)).filter( TrialModel.study_id == self.study_id, TrialModel.trial_id < self.trial_id ) return trial_count.scalar() class TrialUserAttributeModel(BaseModel): __tablename__ = "trial_user_attributes" __table_args__: Any = (UniqueConstraint("trial_id", "key"),) trial_user_attribute_id = _Column(Integer, primary_key=True) trial_id = _Column(Integer, ForeignKey("trials.trial_id")) key = _Column(String(MAX_INDEXED_STRING_LENGTH)) value_json = _Column(Text()) trial = orm.relationship( TrialModel, backref=orm.backref("user_attributes", cascade="all, delete-orphan") ) @classmethod def find_by_trial_and_key( cls, trial: TrialModel, key: str, session: orm.Session ) -> "TrialUserAttributeModel" | None: attribute = ( session.query(cls) .filter(cls.trial_id == trial.trial_id) .filter(cls.key == key) .one_or_none() ) return attribute @classmethod def where_trial_id( cls, trial_id: int, session: orm.Session ) -> list["TrialUserAttributeModel"]: return session.query(cls).filter(cls.trial_id == trial_id).all() class TrialSystemAttributeModel(BaseModel): __tablename__ = "trial_system_attributes" __table_args__: Any = (UniqueConstraint("trial_id", "key"),) trial_system_attribute_id = _Column(Integer, primary_key=True) trial_id = _Column(Integer, ForeignKey("trials.trial_id")) key = _Column(String(MAX_INDEXED_STRING_LENGTH)) value_json = _Column(Text()) trial = orm.relationship( TrialModel, backref=orm.backref("system_attributes", cascade="all, delete-orphan") ) @classmethod def find_by_trial_and_key( cls, trial: TrialModel, key: str, session: orm.Session ) -> "TrialSystemAttributeModel" | None: attribute = ( session.query(cls) .filter(cls.trial_id == trial.trial_id) .filter(cls.key == key) .one_or_none() ) return attribute @classmethod def where_trial_id( cls, trial_id: int, session: orm.Session ) -> list["TrialSystemAttributeModel"]: return session.query(cls).filter(cls.trial_id == trial_id).all() class TrialParamModel(BaseModel): __tablename__ = "trial_params" __table_args__: Any = (UniqueConstraint("trial_id", "param_name"),) param_id = _Column(Integer, primary_key=True) trial_id = _Column(Integer, ForeignKey("trials.trial_id")) param_name = _Column(String(MAX_INDEXED_STRING_LENGTH)) param_value = _Column(Float(precision=FLOAT_PRECISION)) distribution_json = _Column(Text()) trial = orm.relationship( TrialModel, backref=orm.backref("params", cascade="all, delete-orphan") ) def check_and_add(self, session: orm.Session, study_id: int) -> None: self._check_compatibility_with_previous_trial_param_distributions(session, study_id) session.add(self) def _check_compatibility_with_previous_trial_param_distributions( self, session: orm.Session, study_id: int ) -> None: previous_record = ( session.query(TrialParamModel) .join(TrialModel) .filter(TrialModel.study_id == study_id) .filter(TrialParamModel.param_name == self.param_name) .first() ) if previous_record is not None: distributions.check_distribution_compatibility( distributions.json_to_distribution(previous_record.distribution_json), distributions.json_to_distribution(self.distribution_json), ) @classmethod def find_by_trial_and_param_name( cls, trial: TrialModel, param_name: str, session: orm.Session ) -> "TrialParamModel" | None: param_distribution = ( session.query(cls) .filter(cls.trial_id == trial.trial_id) .filter(cls.param_name == param_name) .one_or_none() ) return param_distribution @classmethod def find_or_raise_by_trial_and_param_name( cls, trial: TrialModel, param_name: str, session: orm.Session ) -> "TrialParamModel": param_distribution = cls.find_by_trial_and_param_name(trial, param_name, session) if param_distribution is None: raise KeyError(NOT_FOUND_MSG) return param_distribution @classmethod def where_trial_id(cls, trial_id: int, session: orm.Session) -> list["TrialParamModel"]: trial_params = session.query(cls).filter(cls.trial_id == trial_id).all() return trial_params class TrialValueModel(BaseModel): class TrialValueType(enum.Enum): FINITE = 1 INF_POS = 2 INF_NEG = 3 __tablename__ = "trial_values" __table_args__: Any = (UniqueConstraint("trial_id", "objective"),) trial_value_id = _Column(Integer, primary_key=True) trial_id = _Column(Integer, ForeignKey("trials.trial_id"), nullable=False) objective = _Column(Integer, nullable=False) value = _Column(Float(precision=FLOAT_PRECISION), nullable=True) value_type = _Column(Enum(TrialValueType), nullable=False) trial = orm.relationship( TrialModel, backref=orm.backref("values", cascade="all, delete-orphan") ) @classmethod def value_to_stored_repr(cls, value: float) -> tuple[float | None, TrialValueType]: if value == float("inf"): return (None, cls.TrialValueType.INF_POS) elif value == float("-inf"): return (None, cls.TrialValueType.INF_NEG) else: return (value, cls.TrialValueType.FINITE) @classmethod def stored_repr_to_value(cls, value: float | None, float_type: TrialValueType) -> float: if float_type == cls.TrialValueType.INF_POS: assert value is None return float("inf") elif float_type == cls.TrialValueType.INF_NEG: assert value is None return float("-inf") else: assert float_type == cls.TrialValueType.FINITE assert value is not None return value @classmethod def find_by_trial_and_objective( cls, trial: TrialModel, objective: int, session: orm.Session ) -> "TrialValueModel" | None: trial_value = ( session.query(cls) .filter(cls.trial_id == trial.trial_id) .filter(cls.objective == objective) .one_or_none() ) return trial_value @classmethod def where_trial_id(cls, trial_id: int, session: orm.Session) -> list["TrialValueModel"]: trial_values = ( session.query(cls).filter(cls.trial_id == trial_id).order_by(asc(cls.objective)).all() ) return trial_values class TrialIntermediateValueModel(BaseModel): class TrialIntermediateValueType(enum.Enum): FINITE = 1 INF_POS = 2 INF_NEG = 3 NAN = 4 __tablename__ = "trial_intermediate_values" __table_args__: Any = (UniqueConstraint("trial_id", "step"),) trial_intermediate_value_id = _Column(Integer, primary_key=True) trial_id = _Column(Integer, ForeignKey("trials.trial_id"), nullable=False) step = _Column(Integer, nullable=False) intermediate_value = _Column(Float(precision=FLOAT_PRECISION), nullable=True) intermediate_value_type = _Column(Enum(TrialIntermediateValueType), nullable=False) trial = orm.relationship( TrialModel, backref=orm.backref("intermediate_values", cascade="all, delete-orphan") ) @classmethod def intermediate_value_to_stored_repr( cls, value: float ) -> tuple[float | None, TrialIntermediateValueType]: if math.isnan(value): return (None, cls.TrialIntermediateValueType.NAN) elif value == float("inf"): return (None, cls.TrialIntermediateValueType.INF_POS) elif value == float("-inf"): return (None, cls.TrialIntermediateValueType.INF_NEG) else: return (value, cls.TrialIntermediateValueType.FINITE) @classmethod def stored_repr_to_intermediate_value( cls, value: float | None, float_type: TrialIntermediateValueType ) -> float: if float_type == cls.TrialIntermediateValueType.NAN: assert value is None return float("nan") elif float_type == cls.TrialIntermediateValueType.INF_POS: assert value is None return float("inf") elif float_type == cls.TrialIntermediateValueType.INF_NEG: assert value is None return float("-inf") else: assert float_type == cls.TrialIntermediateValueType.FINITE assert value is not None return value @classmethod def find_by_trial_and_step( cls, trial: TrialModel, step: int, session: orm.Session ) -> "TrialIntermediateValueModel" | None: trial_intermediate_value = ( session.query(cls) .filter(cls.trial_id == trial.trial_id) .filter(cls.step == step) .one_or_none() ) return trial_intermediate_value @classmethod def where_trial_id( cls, trial_id: int, session: orm.Session ) -> list["TrialIntermediateValueModel"]: trial_intermediate_values = session.query(cls).filter(cls.trial_id == trial_id).all() return trial_intermediate_values class TrialHeartbeatModel(BaseModel): __tablename__ = "trial_heartbeats" __table_args__: Any = (UniqueConstraint("trial_id"),) trial_heartbeat_id = _Column(Integer, primary_key=True) trial_id = _Column(Integer, ForeignKey("trials.trial_id"), nullable=False) heartbeat = _Column(DateTime, nullable=False, default=func.current_timestamp()) trial = orm.relationship( TrialModel, backref=orm.backref("heartbeats", cascade="all, delete-orphan") ) @classmethod def where_trial_id( cls, trial_id: int, session: orm.Session, for_update: bool = False ) -> "TrialHeartbeatModel" | None: query = session.query(cls).filter(cls.trial_id == trial_id) if for_update: query = query.with_for_update() return query.one_or_none() class VersionInfoModel(BaseModel): __tablename__ = "version_info" # setting check constraint to ensure the number of rows is at most 1 __table_args__: Any = (CheckConstraint("version_info_id=1"),) version_info_id = _Column(Integer, primary_key=True, autoincrement=False, default=1) schema_version = _Column(Integer) library_version = _Column(String(MAX_VERSION_LENGTH)) @classmethod def find(cls, session: orm.Session) -> "VersionInfoModel" | None: version_info = session.query(cls).one_or_none() return version_info optuna-4.1.0/optuna/storages/_rdb/storage.py000066400000000000000000001410631471332314300211110ustar00rootroot00000000000000from __future__ import annotations from collections import defaultdict from contextlib import contextmanager import copy from datetime import datetime from datetime import timedelta import json import logging import os import random import sqlite3 import time from typing import Any from typing import Callable from typing import Container from typing import Dict from typing import Generator from typing import Iterable from typing import List from typing import Optional from typing import Sequence from typing import Set from typing import TYPE_CHECKING import uuid import optuna from optuna import distributions from optuna import version from optuna._imports import _LazyImport from optuna._typing import JSONSerializable from optuna.storages._base import BaseStorage from optuna.storages._base import DEFAULT_STUDY_NAME_PREFIX from optuna.storages._heartbeat import BaseHeartbeat from optuna.study._frozen import FrozenStudy from optuna.study._study_direction import StudyDirection from optuna.trial import FrozenTrial from optuna.trial import TrialState if TYPE_CHECKING: import alembic.command as alembic_command import alembic.config as alembic_config import alembic.migration as alembic_migration import alembic.script as alembic_script import sqlalchemy import sqlalchemy.dialects.mysql as sqlalchemy_dialects_mysql import sqlalchemy.dialects.sqlite as sqlalchemy_dialects_sqlite import sqlalchemy.exc as sqlalchemy_exc import sqlalchemy.orm as sqlalchemy_orm import sqlalchemy.sql.functions as sqlalchemy_sql_functions from optuna.storages._rdb import models else: alembic_command = _LazyImport("alembic.command") alembic_config = _LazyImport("alembic.config") alembic_migration = _LazyImport("alembic.migration") alembic_script = _LazyImport("alembic.script") sqlalchemy = _LazyImport("sqlalchemy") sqlalchemy_dialects_mysql = _LazyImport("sqlalchemy.dialects.mysql") sqlalchemy_dialects_sqlite = _LazyImport("sqlalchemy.dialects.sqlite") sqlalchemy_exc = _LazyImport("sqlalchemy.exc") sqlalchemy_orm = _LazyImport("sqlalchemy.orm") sqlalchemy_sql_functions = _LazyImport("sqlalchemy.sql.functions") models = _LazyImport("optuna.storages._rdb.models") _logger = optuna.logging.get_logger(__name__) @contextmanager def _create_scoped_session( scoped_session: "sqlalchemy_orm.scoped_session", ignore_integrity_error: bool = False, ) -> Generator["sqlalchemy_orm.Session", None, None]: session = scoped_session() try: yield session session.commit() except sqlalchemy_exc.IntegrityError as e: session.rollback() if ignore_integrity_error: _logger.debug( "Ignoring {}. This happens due to a timing issue among threads/processes/nodes. " "Another one might have committed a record with the same key(s).".format(repr(e)) ) else: raise except sqlalchemy_exc.SQLAlchemyError as e: session.rollback() message = ( "An exception is raised during the commit. " "This typically happens due to invalid data in the commit, " "e.g. exceeding max length. " ) raise optuna.exceptions.StorageInternalError(message) from e except Exception: session.rollback() raise finally: session.close() class RDBStorage(BaseStorage, BaseHeartbeat): """Storage class for RDB backend. Note that library users can instantiate this class, but the attributes provided by this class are not supposed to be directly accessed by them. Example: Create an :class:`~optuna.storages.RDBStorage` instance with customized ``pool_size`` and ``timeout`` settings. .. testcode:: import optuna def objective(trial): x = trial.suggest_float("x", -100, 100) return x**2 storage = optuna.storages.RDBStorage( url="sqlite:///:memory:", engine_kwargs={"pool_size": 20, "connect_args": {"timeout": 10}}, ) study = optuna.create_study(storage=storage) study.optimize(objective, n_trials=10) Args: url: URL of the storage. engine_kwargs: A dictionary of keyword arguments that is passed to `sqlalchemy.engine.create_engine`_ function. skip_compatibility_check: Flag to skip schema compatibility check if set to :obj:`True`. heartbeat_interval: Interval to record the heartbeat. It is recorded every ``interval`` seconds. ``heartbeat_interval`` must be :obj:`None` or a positive integer. .. note:: The heartbeat is supposed to be used with :meth:`~optuna.study.Study.optimize`. If you use :meth:`~optuna.study.Study.ask` and :meth:`~optuna.study.Study.tell` instead, it will not work. grace_period: Grace period before a running trial is failed from the last heartbeat. ``grace_period`` must be :obj:`None` or a positive integer. If it is :obj:`None`, the grace period will be `2 * heartbeat_interval`. failed_trial_callback: A callback function that is invoked after failing each stale trial. The function must accept two parameters with the following types in this order: :class:`~optuna.study.Study` and :class:`~optuna.trial.FrozenTrial`. .. note:: The procedure to fail existing stale trials is called just before asking the study for a new trial. skip_table_creation: Flag to skip table creation if set to :obj:`True`. .. _sqlalchemy.engine.create_engine: https://docs.sqlalchemy.org/en/latest/core/engines.html#sqlalchemy.create_engine .. note:: If you use MySQL, `pool_pre_ping`_ will be set to :obj:`True` by default to prevent connection timeout. You can turn it off with ``engine_kwargs['pool_pre_ping']=False``, but it is recommended to keep the setting if execution time of your objective function is longer than the `wait_timeout` of your MySQL configuration. .. _pool_pre_ping: https://docs.sqlalchemy.org/en/13/core/engines.html#sqlalchemy.create_engine.params. pool_pre_ping .. note:: We would never recommend SQLite3 for parallel optimization. Please see the FAQ :ref:`sqlite_concurrency` for details. .. note:: Mainly in a cluster environment, running trials are often killed unexpectedly. If you want to detect a failure of trials, please use the heartbeat mechanism. Set ``heartbeat_interval``, ``grace_period``, and ``failed_trial_callback`` appropriately according to your use case. For more details, please refer to the :ref:`tutorial ` and `Example page `__. .. seealso:: You can use :class:`~optuna.storages.RetryFailedTrialCallback` to automatically retry failed trials detected by heartbeat. """ def __init__( self, url: str, engine_kwargs: Optional[Dict[str, Any]] = None, skip_compatibility_check: bool = False, *, heartbeat_interval: Optional[int] = None, grace_period: Optional[int] = None, failed_trial_callback: Optional[ Callable[["optuna.study.Study", FrozenTrial], None] ] = None, skip_table_creation: bool = False, ) -> None: self.engine_kwargs = engine_kwargs or {} self.url = self._fill_storage_url_template(url) self.skip_compatibility_check = skip_compatibility_check if heartbeat_interval is not None and heartbeat_interval <= 0: raise ValueError("The value of `heartbeat_interval` should be a positive integer.") if grace_period is not None and grace_period <= 0: raise ValueError("The value of `grace_period` should be a positive integer.") self.heartbeat_interval = heartbeat_interval self.grace_period = grace_period self.failed_trial_callback = failed_trial_callback self._set_default_engine_kwargs_for_mysql(url, self.engine_kwargs) try: self.engine = sqlalchemy.engine.create_engine(self.url, **self.engine_kwargs) except ImportError as e: raise ImportError( "Failed to import DB access module for the specified storage URL. " "Please install appropriate one." ) from e self.scoped_session = sqlalchemy_orm.scoped_session( sqlalchemy_orm.sessionmaker(bind=self.engine) ) if not skip_table_creation: models.BaseModel.metadata.create_all(self.engine) self._version_manager = _VersionManager(self.url, self.engine, self.scoped_session) if not skip_compatibility_check: self._version_manager.check_table_schema_compatibility() def __getstate__(self) -> Dict[Any, Any]: state = self.__dict__.copy() del state["scoped_session"] del state["engine"] del state["_version_manager"] return state def __setstate__(self, state: Dict[Any, Any]) -> None: self.__dict__.update(state) try: self.engine = sqlalchemy.engine.create_engine(self.url, **self.engine_kwargs) except ImportError as e: raise ImportError( "Failed to import DB access module for the specified storage URL. " "Please install appropriate one." ) from e self.scoped_session = sqlalchemy_orm.scoped_session( sqlalchemy_orm.sessionmaker(bind=self.engine) ) models.BaseModel.metadata.create_all(self.engine) self._version_manager = _VersionManager(self.url, self.engine, self.scoped_session) if not self.skip_compatibility_check: self._version_manager.check_table_schema_compatibility() def create_new_study( self, directions: Sequence[StudyDirection], study_name: Optional[str] = None ) -> int: try: with _create_scoped_session(self.scoped_session) as session: if study_name is None: study_name = self._create_unique_study_name(session) direction_models = [ models.StudyDirectionModel(objective=objective, direction=d) for objective, d in enumerate(list(directions)) ] session.add(models.StudyModel(study_name=study_name, directions=direction_models)) except sqlalchemy_exc.IntegrityError: raise optuna.exceptions.DuplicatedStudyError( "Another study with name '{}' already exists. " "Please specify a different name, or reuse the existing one " "by setting `load_if_exists` (for Python API) or " "`--skip-if-exists` flag (for CLI).".format(study_name) ) _logger.info("A new study created in RDB with name: {}".format(study_name)) return self.get_study_id_from_name(study_name) def delete_study(self, study_id: int) -> None: with _create_scoped_session(self.scoped_session, True) as session: study = models.StudyModel.find_or_raise_by_id(study_id, session) session.delete(study) @staticmethod def _create_unique_study_name(session: "sqlalchemy_orm.Session") -> str: while True: study_uuid = str(uuid.uuid4()) study_name = DEFAULT_STUDY_NAME_PREFIX + study_uuid study = models.StudyModel.find_by_name(study_name, session) if study is None: break return study_name def set_study_user_attr(self, study_id: int, key: str, value: Any) -> None: with _create_scoped_session(self.scoped_session, True) as session: study = models.StudyModel.find_or_raise_by_id(study_id, session) attribute = models.StudyUserAttributeModel.find_by_study_and_key(study, key, session) if attribute is None: attribute = models.StudyUserAttributeModel( study_id=study_id, key=key, value_json=json.dumps(value) ) session.add(attribute) else: attribute.value_json = json.dumps(value) def set_study_system_attr(self, study_id: int, key: str, value: JSONSerializable) -> None: with _create_scoped_session(self.scoped_session, True) as session: study = models.StudyModel.find_or_raise_by_id(study_id, session) attribute = models.StudySystemAttributeModel.find_by_study_and_key(study, key, session) if attribute is None: attribute = models.StudySystemAttributeModel( study_id=study_id, key=key, value_json=json.dumps(value) ) session.add(attribute) else: attribute.value_json = json.dumps(value) def get_study_id_from_name(self, study_name: str) -> int: with _create_scoped_session(self.scoped_session) as session: study = models.StudyModel.find_or_raise_by_name(study_name, session) study_id = study.study_id return study_id def get_study_name_from_id(self, study_id: int) -> str: with _create_scoped_session(self.scoped_session) as session: study = models.StudyModel.find_or_raise_by_id(study_id, session) study_name = study.study_name return study_name def get_study_directions(self, study_id: int) -> List[StudyDirection]: with _create_scoped_session(self.scoped_session) as session: study = models.StudyModel.find_or_raise_by_id(study_id, session) directions = [d.direction for d in study.directions] return directions def get_study_user_attrs(self, study_id: int) -> Dict[str, Any]: with _create_scoped_session(self.scoped_session) as session: # Ensure that that study exists. models.StudyModel.find_or_raise_by_id(study_id, session) attributes = models.StudyUserAttributeModel.where_study_id(study_id, session) user_attrs = {attr.key: json.loads(attr.value_json) for attr in attributes} return user_attrs def get_study_system_attrs(self, study_id: int) -> Dict[str, Any]: with _create_scoped_session(self.scoped_session) as session: # Ensure that that study exists. models.StudyModel.find_or_raise_by_id(study_id, session) attributes = models.StudySystemAttributeModel.where_study_id(study_id, session) system_attrs = {attr.key: json.loads(attr.value_json) for attr in attributes} return system_attrs def get_trial_user_attrs(self, trial_id: int) -> Dict[str, Any]: with _create_scoped_session(self.scoped_session) as session: # Ensure trial exists. models.TrialModel.find_or_raise_by_id(trial_id, session) attributes = models.TrialUserAttributeModel.where_trial_id(trial_id, session) user_attrs = {attr.key: json.loads(attr.value_json) for attr in attributes} return user_attrs def get_trial_system_attrs(self, trial_id: int) -> Dict[str, Any]: with _create_scoped_session(self.scoped_session) as session: # Ensure trial exists. models.TrialModel.find_or_raise_by_id(trial_id, session) attributes = models.TrialSystemAttributeModel.where_trial_id(trial_id, session) system_attrs = {attr.key: json.loads(attr.value_json) for attr in attributes} return system_attrs def get_all_studies(self) -> List[FrozenStudy]: with _create_scoped_session(self.scoped_session) as session: studies = ( session.query( models.StudyModel.study_id, models.StudyModel.study_name, ) .order_by(models.StudyModel.study_id) .all() ) _directions = defaultdict(list) for direction_model in session.query(models.StudyDirectionModel).all(): _directions[direction_model.study_id].append(direction_model.direction) _user_attrs = defaultdict(list) for attribute_model in session.query(models.StudyUserAttributeModel).all(): _user_attrs[attribute_model.study_id].append(attribute_model) _system_attrs = defaultdict(list) for attribute_model in session.query(models.StudySystemAttributeModel).all(): _system_attrs[attribute_model.study_id].append(attribute_model) frozen_studies = [] for study in studies: directions = _directions[study.study_id] user_attrs = _user_attrs.get(study.study_id, []) system_attrs = _system_attrs.get(study.study_id, []) frozen_studies.append( FrozenStudy( study_name=study.study_name, direction=None, directions=directions, user_attrs={i.key: json.loads(i.value_json) for i in user_attrs}, system_attrs={i.key: json.loads(i.value_json) for i in system_attrs}, study_id=study.study_id, ) ) return frozen_studies def create_new_trial(self, study_id: int, template_trial: Optional[FrozenTrial] = None) -> int: return self._create_new_trial(study_id, template_trial)._trial_id def _create_new_trial( self, study_id: int, template_trial: Optional[FrozenTrial] = None ) -> FrozenTrial: """Create a new trial and returns a :class:`~optuna.trial.FrozenTrial`. Args: study_id: Study id. template_trial: A :class:`~optuna.trial.FrozenTrial` with default values for trial attributes. Returns: A :class:`~optuna.trial.FrozenTrial` instance. """ def _create_frozen_trial( trial: "models.TrialModel", template_trial: FrozenTrial | None ) -> FrozenTrial: if template_trial: frozen = copy.deepcopy(template_trial) frozen.number = trial.number frozen.datetime_start = trial.datetime_start frozen._trial_id = trial.trial_id return frozen return FrozenTrial( number=trial.number, state=trial.state, value=None, values=None, datetime_start=trial.datetime_start, datetime_complete=None, params={}, distributions={}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=trial.trial_id, ) # Retry maximum five times. Deadlocks may occur in distributed environments. MAX_RETRIES = 5 for n_retries in range(1, MAX_RETRIES + 1): try: with _create_scoped_session(self.scoped_session) as session: # This lock is necessary because the trial creation is not an atomic operation # and the calculation of trial.number is prone to race conditions. models.StudyModel.find_or_raise_by_id(study_id, session, for_update=True) trial = self._get_prepared_new_trial(study_id, template_trial, session) return _create_frozen_trial(trial, template_trial) # sqlalchemy_exc.OperationalError is converted to ``StorageInternalError``. except optuna.exceptions.StorageInternalError as e: # ``OperationalError`` happens either by (1) invalid inputs, e.g., too long string, # or (2) timeout error, which relates to deadlock. Although Error (1) is not # intended to be caught here, it must be fixed to use RDBStorage anyways. if n_retries == MAX_RETRIES: raise e # Optuna defers to the DB administrator to reduce DB server congestion, hence # Optuna simply uses non-exponential backoff here for retries caused by deadlock. time.sleep(random.random() * 2.0) assert False, "Should not be reached." def _get_prepared_new_trial( self, study_id: int, template_trial: Optional[FrozenTrial], session: "sqlalchemy_orm.Session", ) -> "models.TrialModel": if template_trial is None: trial = models.TrialModel( study_id=study_id, number=None, state=TrialState.RUNNING, datetime_start=datetime.now(), ) else: # Because only `RUNNING` trials can be updated, # we temporarily set the state of the new trial to `RUNNING`. # After all fields of the trial have been updated, # the state is set to `template_trial.state`. temp_state = TrialState.RUNNING trial = models.TrialModel( study_id=study_id, number=None, state=temp_state, datetime_start=template_trial.datetime_start, datetime_complete=template_trial.datetime_complete, ) session.add(trial) # Flush the session cache to reflect the above addition operation to # the current RDB transaction. # # Without flushing, the following operations (e.g, `_set_trial_param_without_commit`) # will fail because the target trial doesn't exist in the storage yet. session.flush() if template_trial is not None: if template_trial.values is not None and len(template_trial.values) > 1: for objective, value in enumerate(template_trial.values): self._set_trial_value_without_commit(session, trial.trial_id, objective, value) elif template_trial.value is not None: self._set_trial_value_without_commit( session, trial.trial_id, 0, template_trial.value ) for param_name, param_value in template_trial.params.items(): distribution = template_trial.distributions[param_name] param_value_in_internal_repr = distribution.to_internal_repr(param_value) self._set_trial_param_without_commit( session, trial.trial_id, param_name, param_value_in_internal_repr, distribution ) for key, value in template_trial.user_attrs.items(): self._set_trial_attr_without_commit( session, models.TrialUserAttributeModel, trial.trial_id, key, value ) for key, value in template_trial.system_attrs.items(): self._set_trial_attr_without_commit( session, models.TrialSystemAttributeModel, trial.trial_id, key, value ) for step, intermediate_value in template_trial.intermediate_values.items(): self._set_trial_intermediate_value_without_commit( session, trial.trial_id, step, intermediate_value ) trial.state = template_trial.state trial.number = trial.count_past_trials(session) session.add(trial) return trial def set_trial_param( self, trial_id: int, param_name: str, param_value_internal: float, distribution: distributions.BaseDistribution, ) -> None: with _create_scoped_session(self.scoped_session, True) as session: self._set_trial_param_without_commit( session, trial_id, param_name, param_value_internal, distribution ) def _set_trial_param_without_commit( self, session: "sqlalchemy_orm.Session", trial_id: int, param_name: str, param_value_internal: float, distribution: distributions.BaseDistribution, ) -> None: trial = models.TrialModel.find_or_raise_by_id(trial_id, session) self.check_trial_is_updatable(trial_id, trial.state) trial_param = models.TrialParamModel( trial_id=trial_id, param_name=param_name, param_value=param_value_internal, distribution_json=distributions.distribution_to_json(distribution), ) trial_param.check_and_add(session, trial.study_id) def _check_and_set_param_distribution( self, study_id: int, trial_id: int, param_name: str, param_value_internal: float, distribution: distributions.BaseDistribution, ) -> None: with _create_scoped_session(self.scoped_session) as session: # Acquire lock. # # Assume that study exists. models.StudyModel.find_or_raise_by_id(study_id, session, for_update=True) models.TrialParamModel( trial_id=trial_id, param_name=param_name, param_value=param_value_internal, distribution_json=distributions.distribution_to_json(distribution), ).check_and_add(session, study_id) def get_trial_param(self, trial_id: int, param_name: str) -> float: with _create_scoped_session(self.scoped_session) as session: trial = models.TrialModel.find_or_raise_by_id(trial_id, session) trial_param = models.TrialParamModel.find_or_raise_by_trial_and_param_name( trial, param_name, session ) param_value = trial_param.param_value return param_value def set_trial_state_values( self, trial_id: int, state: TrialState, values: Optional[Sequence[float]] = None ) -> bool: try: with _create_scoped_session(self.scoped_session) as session: trial = models.TrialModel.find_or_raise_by_id(trial_id, session, for_update=True) self.check_trial_is_updatable(trial_id, trial.state) if values is not None: for objective, v in enumerate(values): self._set_trial_value_without_commit(session, trial_id, objective, v) if state == TrialState.RUNNING and trial.state != TrialState.WAITING: return False trial.state = state if state == TrialState.RUNNING: trial.datetime_start = datetime.now() if state.is_finished(): trial.datetime_complete = datetime.now() except sqlalchemy_exc.IntegrityError: return False return True def _set_trial_value_without_commit( self, session: "sqlalchemy_orm.Session", trial_id: int, objective: int, value: float ) -> None: trial = models.TrialModel.find_or_raise_by_id(trial_id, session) self.check_trial_is_updatable(trial_id, trial.state) stored_value, value_type = models.TrialValueModel.value_to_stored_repr(value) trial_value = models.TrialValueModel.find_by_trial_and_objective(trial, objective, session) if trial_value is None: trial_value = models.TrialValueModel( trial_id=trial_id, objective=objective, value=stored_value, value_type=value_type ) session.add(trial_value) else: trial_value.value = stored_value trial_value.value_type = value_type def set_trial_intermediate_value( self, trial_id: int, step: int, intermediate_value: float ) -> None: with _create_scoped_session(self.scoped_session, True) as session: self._set_trial_intermediate_value_without_commit( session, trial_id, step, intermediate_value ) def _set_trial_intermediate_value_without_commit( self, session: "sqlalchemy_orm.Session", trial_id: int, step: int, intermediate_value: float, ) -> None: trial = models.TrialModel.find_or_raise_by_id(trial_id, session) self.check_trial_is_updatable(trial_id, trial.state) ( stored_value, value_type, ) = models.TrialIntermediateValueModel.intermediate_value_to_stored_repr( intermediate_value ) trial_intermediate_value = models.TrialIntermediateValueModel.find_by_trial_and_step( trial, step, session ) if trial_intermediate_value is None: trial_intermediate_value = models.TrialIntermediateValueModel( trial_id=trial_id, step=step, intermediate_value=stored_value, intermediate_value_type=value_type, ) session.add(trial_intermediate_value) else: trial_intermediate_value.intermediate_value = stored_value trial_intermediate_value.intermediate_value_type = value_type def set_trial_user_attr(self, trial_id: int, key: str, value: Any) -> None: with _create_scoped_session(self.scoped_session, True) as session: self._set_trial_attr_without_commit( session, models.TrialUserAttributeModel, trial_id, key, value, ) def set_trial_system_attr(self, trial_id: int, key: str, value: JSONSerializable) -> None: with _create_scoped_session(self.scoped_session, True) as session: self._set_trial_attr_without_commit( session, models.TrialSystemAttributeModel, trial_id, key, value, ) def _set_trial_attr_without_commit( self, session: "sqlalchemy_orm.Session", model_cls: type[models.TrialUserAttributeModel | models.TrialSystemAttributeModel], trial_id: int, key: str, value: Any, ) -> None: trial = models.TrialModel.find_or_raise_by_id(trial_id, session) self.check_trial_is_updatable(trial_id, trial.state) if self.engine.name == "mysql": mysql_insert_stmt = sqlalchemy_dialects_mysql.insert(model_cls).values( trial_id=trial_id, key=key, value_json=json.dumps(value) ) mysql_upsert_stmt = mysql_insert_stmt.on_duplicate_key_update( value_json=mysql_insert_stmt.inserted.value_json ) session.execute(mysql_upsert_stmt) elif self.engine.name == "sqlite" and sqlite3.sqlite_version_info >= (3, 24, 0): sqlite_insert_stmt = sqlalchemy_dialects_sqlite.insert(model_cls).values( trial_id=trial_id, key=key, value_json=json.dumps(value) ) sqlite_upsert_stmt = sqlite_insert_stmt.on_conflict_do_update( index_elements=[model_cls.trial_id, model_cls.key], set_=dict(value_json=sqlite_insert_stmt.excluded.value_json), ) session.execute(sqlite_upsert_stmt) else: # TODO(porink0424): Add support for other databases, e.g., PostgreSQL. attribute = model_cls.find_by_trial_and_key(trial, key, session) if attribute is None: attribute = model_cls(trial_id=trial_id, key=key, value_json=json.dumps(value)) session.add(attribute) else: attribute.value_json = json.dumps(value) def get_trial_id_from_study_id_trial_number(self, study_id: int, trial_number: int) -> int: with _create_scoped_session(self.scoped_session) as session: trial_id = ( session.query(models.TrialModel.trial_id) .filter( models.TrialModel.number == trial_number, models.TrialModel.study_id == study_id, ) .one_or_none() ) if trial_id is None: raise KeyError( "No trial with trial number {} exists in study with study_id {}.".format( trial_number, study_id ) ) return trial_id[0] def get_trial(self, trial_id: int) -> FrozenTrial: with _create_scoped_session(self.scoped_session) as session: trial_model = models.TrialModel.find_or_raise_by_id(trial_id, session) frozen_trial = self._build_frozen_trial_from_trial_model(trial_model) return frozen_trial def get_all_trials( self, study_id: int, deepcopy: bool = True, states: Optional[Container[TrialState]] = None, ) -> List[FrozenTrial]: trials = self._get_trials(study_id, states, set(), -1) return copy.deepcopy(trials) if deepcopy else trials def _get_trials( self, study_id: int, states: Optional[Container[TrialState]], included_trial_ids: Set[int], trial_id_greater_than: int, ) -> List[FrozenTrial]: included_trial_ids = set( trial_id for trial_id in included_trial_ids if trial_id <= trial_id_greater_than ) with _create_scoped_session(self.scoped_session) as session: # Ensure that the study exists. models.StudyModel.find_or_raise_by_id(study_id, session) query = ( session.query(models.TrialModel) .options(sqlalchemy_orm.selectinload(models.TrialModel.params)) .options(sqlalchemy_orm.selectinload(models.TrialModel.values)) .options(sqlalchemy_orm.selectinload(models.TrialModel.user_attributes)) .options(sqlalchemy_orm.selectinload(models.TrialModel.system_attributes)) .options(sqlalchemy_orm.selectinload(models.TrialModel.intermediate_values)) .filter( models.TrialModel.study_id == study_id, ) ) if states is not None: # This assertion is for type checkers, since `states` is required to be Container # in the base class while `models.TrialModel.state.in_` requires Iterable. assert isinstance(states, Iterable) query = query.filter(models.TrialModel.state.in_(states)) try: if len(included_trial_ids) > 0 and trial_id_greater_than > -1: _query = query.filter( sqlalchemy.or_( models.TrialModel.trial_id.in_(included_trial_ids), models.TrialModel.trial_id > trial_id_greater_than, ) ) elif trial_id_greater_than > -1: _query = query.filter(models.TrialModel.trial_id > trial_id_greater_than) else: _query = query trial_models = _query.order_by(models.TrialModel.trial_id).all() except sqlalchemy_exc.OperationalError as e: # Likely exceeding the number of maximum allowed variables using IN. # This number differ between database dialects. For SQLite for instance, see # https://www.sqlite.org/limits.html and the section describing # SQLITE_MAX_VARIABLE_NUMBER. _logger.warning( "Caught an error from sqlalchemy: {}. Falling back to a slower alternative. " "".format(str(e)) ) trial_models = query.order_by(models.TrialModel.trial_id).all() trial_models = [ t for t in trial_models if t.trial_id in included_trial_ids or t.trial_id > trial_id_greater_than ] trials = [self._build_frozen_trial_from_trial_model(trial) for trial in trial_models] return trials def _build_frozen_trial_from_trial_model(self, trial: "models.TrialModel") -> FrozenTrial: values: Optional[List[float]] if trial.values: values = [0 for _ in trial.values] for value_model in trial.values: values[value_model.objective] = models.TrialValueModel.stored_repr_to_value( value_model.value, value_model.value_type ) else: values = None params = sorted(trial.params, key=lambda p: p.param_id) return FrozenTrial( number=trial.number, state=trial.state, value=None, values=values, datetime_start=trial.datetime_start, datetime_complete=trial.datetime_complete, params={ p.param_name: distributions.json_to_distribution( p.distribution_json ).to_external_repr(p.param_value) for p in params }, distributions={ p.param_name: distributions.json_to_distribution(p.distribution_json) for p in params }, user_attrs={attr.key: json.loads(attr.value_json) for attr in trial.user_attributes}, system_attrs={ attr.key: json.loads(attr.value_json) for attr in trial.system_attributes }, intermediate_values={ v.step: models.TrialIntermediateValueModel.stored_repr_to_intermediate_value( v.intermediate_value, v.intermediate_value_type ) for v in trial.intermediate_values }, trial_id=trial.trial_id, ) def get_best_trial(self, study_id: int) -> FrozenTrial: with _create_scoped_session(self.scoped_session) as session: _directions = self.get_study_directions(study_id) if len(_directions) > 1: raise RuntimeError( "Best trial can be obtained only for single-objective optimization." ) direction = _directions[0] if direction == StudyDirection.MAXIMIZE: trial_id = models.TrialModel.find_max_value_trial_id(study_id, 0, session) else: trial_id = models.TrialModel.find_min_value_trial_id(study_id, 0, session) return self.get_trial(trial_id) @staticmethod def _set_default_engine_kwargs_for_mysql(url: str, engine_kwargs: Dict[str, Any]) -> None: # Skip if RDB is not MySQL. if not url.startswith("mysql"): return # Do not overwrite value. if "pool_pre_ping" in engine_kwargs: return # If True, the connection pool checks liveness of connections at every checkout. # Without this option, trials that take longer than `wait_timeout` may cause connection # errors. For further details, please refer to the following document: # https://docs.sqlalchemy.org/en/13/core/pooling.html#pool-disconnects-pessimistic engine_kwargs["pool_pre_ping"] = True _logger.debug("pool_pre_ping=True was set to engine_kwargs to prevent connection timeout.") @staticmethod def _fill_storage_url_template(template: str) -> str: return template.format(SCHEMA_VERSION=models.SCHEMA_VERSION) def remove_session(self) -> None: """Removes the current session. A session is stored in SQLAlchemy's ThreadLocalRegistry for each thread. This method closes and removes the session which is associated to the current thread. Particularly, under multi-thread use cases, it is important to call this method *from each thread*. Otherwise, all sessions and their associated DB connections are destructed by a thread that occasionally invoked the garbage collector. By default, it is not allowed to touch a SQLite connection from threads other than the thread that created the connection. Therefore, we need to explicitly close the connection from each thread. """ self.scoped_session.remove() def upgrade(self) -> None: """Upgrade the storage schema.""" self._version_manager.upgrade() def get_current_version(self) -> str: """Return the schema version currently used by this storage.""" return self._version_manager.get_current_version() def get_head_version(self) -> str: """Return the latest schema version.""" return self._version_manager.get_head_version() def get_all_versions(self) -> List[str]: """Return the schema version list.""" return self._version_manager.get_all_versions() def record_heartbeat(self, trial_id: int) -> None: with _create_scoped_session(self.scoped_session, True) as session: # Fetch heartbeat with read-only. heartbeat = models.TrialHeartbeatModel.where_trial_id(trial_id, session) if heartbeat is None: # heartbeat record does not exist. heartbeat = models.TrialHeartbeatModel(trial_id=trial_id) session.add(heartbeat) else: # Re-fetch the existing heartbeat with the write authorization. heartbeat = models.TrialHeartbeatModel.where_trial_id(trial_id, session, True) assert heartbeat is not None heartbeat.heartbeat = session.execute(sqlalchemy.func.now()).scalar() def _get_stale_trial_ids(self, study_id: int) -> List[int]: assert self.heartbeat_interval is not None if self.grace_period is None: grace_period = 2 * self.heartbeat_interval else: grace_period = self.grace_period stale_trial_ids = [] with _create_scoped_session(self.scoped_session, True) as session: current_heartbeat = session.execute(sqlalchemy.func.now()).scalar() assert current_heartbeat is not None # Added the following line to prevent mixing of timezone-aware and timezone-naive # `datetime` in PostgreSQL. See # https://github.com/optuna/optuna/pull/2190#issuecomment-766605088 for details current_heartbeat = current_heartbeat.replace(tzinfo=None) running_trials = ( session.query(models.TrialModel) .options(sqlalchemy_orm.selectinload(models.TrialModel.heartbeats)) .filter(models.TrialModel.state == TrialState.RUNNING) .filter(models.TrialModel.study_id == study_id) .all() ) for trial in running_trials: if len(trial.heartbeats) == 0: continue assert len(trial.heartbeats) == 1 heartbeat = trial.heartbeats[0].heartbeat if current_heartbeat - heartbeat > timedelta(seconds=grace_period): stale_trial_ids.append(trial.trial_id) return stale_trial_ids def get_heartbeat_interval(self) -> Optional[int]: return self.heartbeat_interval def get_failed_trial_callback( self, ) -> Optional[Callable[["optuna.study.Study", FrozenTrial], None]]: return self.failed_trial_callback class _VersionManager: def __init__( self, url: str, engine: "sqlalchemy.engine.Engine", scoped_session: "sqlalchemy_orm.scoped_session", ) -> None: self.url = url self.engine = engine self.scoped_session = scoped_session self._init_version_info_model() self._init_alembic() def _init_version_info_model(self) -> None: with _create_scoped_session(self.scoped_session, True) as session: version_info = models.VersionInfoModel.find(session) if version_info is not None: return version_info = models.VersionInfoModel( schema_version=models.SCHEMA_VERSION, library_version=version.__version__, ) session.add(version_info) def _init_alembic(self) -> None: logging.getLogger("alembic").setLevel(logging.WARN) with self.engine.connect() as connection: context = alembic_migration.MigrationContext.configure(connection) is_initialized = context.get_current_revision() is not None if is_initialized: # The `alembic_version` table already exists and is not empty. return if self._is_alembic_supported(): revision = self.get_head_version() else: # The storage has been created before alembic is introduced. revision = self._get_base_version() self._set_alembic_revision(revision) def _set_alembic_revision(self, revision: str) -> None: with self.engine.connect() as connection: context = alembic_migration.MigrationContext.configure(connection) with connection.begin(): script = self._create_alembic_script() context.stamp(script, revision) def check_table_schema_compatibility(self) -> None: with _create_scoped_session(self.scoped_session) as session: # NOTE: After invocation of `_init_version_info_model` method, # it is ensured that a `VersionInfoModel` entry exists. version_info = models.VersionInfoModel.find(session) assert version_info is not None current_version = self.get_current_version() head_version = self.get_head_version() if current_version == head_version: return message = ( "The runtime optuna version {} is no longer compatible with the table schema " "(set up by optuna {}). ".format(version.__version__, version_info.library_version) ) known_versions = self.get_all_versions() if current_version in known_versions: message += ( "Please execute `$ optuna storage upgrade --storage $STORAGE_URL` " "for upgrading the storage." ) else: message += ( "Please try updating optuna to the latest version by `$ pip install -U optuna`." ) raise RuntimeError(message) def get_current_version(self) -> str: with self.engine.connect() as connection: context = alembic_migration.MigrationContext.configure(connection) version = context.get_current_revision() assert version is not None return version def get_head_version(self) -> str: script = self._create_alembic_script() current_head = script.get_current_head() assert current_head is not None return current_head def _get_base_version(self) -> str: script = self._create_alembic_script() base = script.get_base() assert base is not None, "There should be exactly one base, i.e. v0.9.0.a." return base def get_all_versions(self) -> List[str]: script = self._create_alembic_script() return [r.revision for r in script.walk_revisions()] def upgrade(self) -> None: config = self._create_alembic_config() alembic_command.upgrade(config, "head") with _create_scoped_session(self.scoped_session, True) as session: version_info = models.VersionInfoModel.find(session) assert version_info is not None version_info.schema_version = models.SCHEMA_VERSION version_info.library_version = version.__version__ def _is_alembic_supported(self) -> bool: with _create_scoped_session(self.scoped_session) as session: version_info = models.VersionInfoModel.find(session) if version_info is None: # `None` means this storage was created just now. return True return version_info.schema_version == models.SCHEMA_VERSION def _create_alembic_script(self) -> "alembic_script.ScriptDirectory": config = self._create_alembic_config() script = alembic_script.ScriptDirectory.from_config(config) return script def _create_alembic_config(self) -> "alembic_config.Config": alembic_dir = os.path.join(os.path.dirname(__file__), "alembic") config = alembic_config.Config(os.path.join(os.path.dirname(__file__), "alembic.ini")) config.set_main_option("script_location", escape_alembic_config_value(alembic_dir)) config.set_main_option("sqlalchemy.url", escape_alembic_config_value(self.url)) return config def escape_alembic_config_value(value: str) -> str: # We must escape '%' in a value string because the character # is regarded as the trigger of variable expansion. # Please see the documentation of `configparser.BasicInterpolation` for more details. return value.replace("%", "%%") optuna-4.1.0/optuna/storages/journal/000077500000000000000000000000001471332314300176325ustar00rootroot00000000000000optuna-4.1.0/optuna/storages/journal/__init__.py000066400000000000000000000014021471332314300217400ustar00rootroot00000000000000from optuna.storages.journal._base import BaseJournalBackend from optuna.storages.journal._file import JournalFileBackend from optuna.storages.journal._file import JournalFileOpenLock from optuna.storages.journal._file import JournalFileSymlinkLock from optuna.storages.journal._redis import JournalRedisBackend from optuna.storages.journal._storage import JournalStorage # NOTE(nabenabe0928): Do not add objects deprecated at v4.0.0 here, e.g., JournalFileStorage # because ``optuna/storages/journal`` was added at v4.0.0 and it will be confusing to keep them in # the non-deprecated directory. __all__ = [ "JournalFileBackend", "BaseJournalBackend", "JournalFileOpenLock", "JournalFileSymlinkLock", "JournalRedisBackend", "JournalStorage", ] optuna-4.1.0/optuna/storages/journal/_base.py000066400000000000000000000054461471332314300212660ustar00rootroot00000000000000import abc from typing import Any from typing import Dict from typing import List from typing import Optional from optuna._deprecated import deprecated_class class BaseJournalBackend(abc.ABC): """Base class for Journal storages. Storage classes implementing this base class must guarantee process safety. This means, multiple processes might concurrently call ``read_logs`` and ``append_logs``. If the backend storage does not internally support mutual exclusion mechanisms, such as locks, you might want to use :class:`~optuna.storages.journal.JournalFileSymlinkLock` or :class:`~optuna.storages.journal.JournalFileOpenLock` for creating a critical section. """ @abc.abstractmethod def read_logs(self, log_number_from: int) -> List[Dict[str, Any]]: """Read logs with a log number greater than or equal to ``log_number_from``. If ``log_number_from`` is 0, read all the logs. Args: log_number_from: A non-negative integer value indicating which logs to read. Returns: Logs with log number greater than or equal to ``log_number_from``. """ raise NotImplementedError @abc.abstractmethod def append_logs(self, logs: List[Dict[str, Any]]) -> None: """Append logs to the backend. Args: logs: A list that contains json-serializable logs. """ raise NotImplementedError class BaseJournalSnapshot(abc.ABC): """Optional base class for Journal storages. Storage classes implementing this base class may work faster when constructing the internal state from the large amount of logs. """ @abc.abstractmethod def save_snapshot(self, snapshot: bytes) -> None: """Save snapshot to the backend. Args: snapshot: A serialized snapshot (bytes) """ raise NotImplementedError @abc.abstractmethod def load_snapshot(self) -> Optional[bytes]: """Load snapshot from the backend. Returns: A serialized snapshot (bytes) if found, otherwise :obj:`None`. """ raise NotImplementedError @deprecated_class( "4.0.0", "6.0.0", text="Use :class:`~optuna.storages.journal.BaseJournalBackend` instead." ) class BaseJournalLogStorage(BaseJournalBackend): """Base class for Journal storages. Storage classes implementing this base class must guarantee process safety. This means, multiple processes might concurrently call ``read_logs`` and ``append_logs``. If the backend storage does not internally support mutual exclusion mechanisms, such as locks, you might want to use :class:`~optuna.storages.journal.JournalFileSymlinkLock` or :class:`~optuna.storages.journal.JournalFileOpenLock` for creating a critical section. """ optuna-4.1.0/optuna/storages/journal/_file.py000066400000000000000000000226311471332314300212660ustar00rootroot00000000000000from __future__ import annotations import abc from contextlib import contextmanager import errno import json import os import time from typing import Any from typing import Iterator import uuid from optuna._deprecated import deprecated_class from optuna.storages.journal._base import BaseJournalBackend LOCK_FILE_SUFFIX = ".lock" RENAME_FILE_SUFFIX = ".rename" class JournalFileBackend(BaseJournalBackend): """File storage class for Journal log backend. Compared to SQLite3, the benefit of this backend is that it is more suitable for environments where the file system does not support ``fcntl()`` file locking. For example, as written in the `SQLite3 FAQ `__, SQLite3 might not work on NFS (Network File System) since ``fcntl()`` file locking is broken on many NFS implementations. In such scenarios, this backend provides several workarounds for locking files. For more details, refer to the `Medium blog post`_. .. _Medium blog post: https://medium.com/optuna/distributed-optimization-via-nfs\ -using-optunas-new-operation-based-logging-storage-9815f9c3f932 It's important to note that, similar to SQLite3, this class doesn't support a high level of write concurrency, as outlined in the `SQLAlchemy documentation`_. However, in typical situations where the objective function is computationally expensive, Optuna users don't need to be concerned about this limitation. The reason being, the write operations are not the bottleneck as long as the objective function doesn't invoke :meth:`~optuna.trial.Trial.report` and :meth:`~optuna.trial.Trial.set_user_attr` excessively. .. _SQLAlchemy documentation: https://docs.sqlalchemy.org/en/20/dialects/sqlite.html\ #database-locking-behavior-concurrency Args: file_path: Path of file to persist the log to. lock_obj: Lock object for process exclusivity. An instance of :class:`~optuna.storages.journal.JournalFileSymlinkLock` and :class:`~optuna.storages.journal.JournalFileOpenLock` can be passed. """ def __init__(self, file_path: str, lock_obj: BaseJournalFileLock | None = None) -> None: self._file_path: str = file_path self._lock = lock_obj or JournalFileSymlinkLock(self._file_path) if not os.path.exists(self._file_path): open(self._file_path, "ab").close() # Create a file if it does not exist. self._log_number_offset: dict[int, int] = {0: 0} def read_logs(self, log_number_from: int) -> list[dict[str, Any]]: logs = [] with open(self._file_path, "rb") as f: # Maintain remaining_log_size to allow writing by another process # while reading the log. remaining_log_size = os.stat(self._file_path).st_size log_number_start = 0 if log_number_from in self._log_number_offset: f.seek(self._log_number_offset[log_number_from]) log_number_start = log_number_from remaining_log_size -= self._log_number_offset[log_number_from] last_decode_error = None for log_number, line in enumerate(f, start=log_number_start): byte_len = len(line) remaining_log_size -= byte_len if remaining_log_size < 0: break if last_decode_error is not None: raise last_decode_error if log_number + 1 not in self._log_number_offset: self._log_number_offset[log_number + 1] = ( self._log_number_offset[log_number] + byte_len ) if log_number < log_number_from: continue # Ensure that each line ends with line separators (\n, \r\n). if not line.endswith(b"\n"): last_decode_error = ValueError("Invalid log format.") del self._log_number_offset[log_number + 1] continue try: logs.append(json.loads(line)) except json.JSONDecodeError as err: last_decode_error = err del self._log_number_offset[log_number + 1] return logs def append_logs(self, logs: list[dict[str, Any]]) -> None: with get_lock_file(self._lock): what_to_write = ( "\n".join([json.dumps(log, separators=(",", ":")) for log in logs]) + "\n" ) with open(self._file_path, "ab") as f: f.write(what_to_write.encode("utf-8")) f.flush() os.fsync(f.fileno()) class BaseJournalFileLock(abc.ABC): @abc.abstractmethod def acquire(self) -> bool: raise NotImplementedError @abc.abstractmethod def release(self) -> None: raise NotImplementedError class JournalFileSymlinkLock(BaseJournalFileLock): """Lock class for synchronizing processes for NFSv2 or later. On acquiring the lock, link system call is called to create an exclusive file. The file is deleted when the lock is released. In NFS environments prior to NFSv3, use this instead of :class:`~optuna.storages.journal.JournalFileOpenLock`. Args: filepath: The path of the file whose race condition must be protected. """ def __init__(self, filepath: str) -> None: self._lock_target_file = filepath self._lock_file = filepath + LOCK_FILE_SUFFIX def acquire(self) -> bool: """Acquire a lock in a blocking way by creating a symbolic link of a file. Returns: :obj:`True` if it succeeded in creating a symbolic link of ``self._lock_target_file``. """ sleep_secs = 0.001 while True: try: os.symlink(self._lock_target_file, self._lock_file) return True except OSError as err: if err.errno == errno.EEXIST: time.sleep(sleep_secs) sleep_secs = min(sleep_secs * 2, 1) continue raise err except BaseException: self.release() raise def release(self) -> None: """Release a lock by removing the symbolic link.""" lock_rename_file = self._lock_file + str(uuid.uuid4()) + RENAME_FILE_SUFFIX try: os.rename(self._lock_file, lock_rename_file) os.unlink(lock_rename_file) except OSError: raise RuntimeError("Error: did not possess lock") except BaseException: os.unlink(lock_rename_file) raise class JournalFileOpenLock(BaseJournalFileLock): """Lock class for synchronizing processes for NFSv3 or later. On acquiring the lock, open system call is called with the O_EXCL option to create an exclusive file. The file is deleted when the lock is released. This class is only supported when using NFSv3 or later on kernel 2.6 or later. In prior NFS environments, use :class:`~optuna.storages.journal.JournalFileSymlinkLock`. Args: filepath: The path of the file whose race condition must be protected. """ def __init__(self, filepath: str) -> None: self._lock_file = filepath + LOCK_FILE_SUFFIX def acquire(self) -> bool: """Acquire a lock in a blocking way by creating a lock file. Returns: :obj:`True` if it succeeded in creating a ``self._lock_file``. """ sleep_secs = 0.001 while True: try: open_flags = os.O_CREAT | os.O_EXCL | os.O_WRONLY os.close(os.open(self._lock_file, open_flags)) return True except OSError as err: if err.errno == errno.EEXIST: time.sleep(sleep_secs) sleep_secs = min(sleep_secs * 2, 1) continue raise err except BaseException: self.release() raise def release(self) -> None: """Release a lock by removing the created file.""" lock_rename_file = self._lock_file + str(uuid.uuid4()) + RENAME_FILE_SUFFIX try: os.rename(self._lock_file, lock_rename_file) os.unlink(lock_rename_file) except OSError: raise RuntimeError("Error: did not possess lock") except BaseException: os.unlink(lock_rename_file) raise @contextmanager def get_lock_file(lock_obj: BaseJournalFileLock) -> Iterator[None]: lock_obj.acquire() try: yield finally: lock_obj.release() @deprecated_class( "4.0.0", "6.0.0", text="Use :class:`~optuna.storages.journal.JournalFileBackend` instead." ) class JournalFileStorage(JournalFileBackend): pass @deprecated_class( deprecated_version="4.0.0", removed_version="6.0.0", name="The import path :class:`~optuna.storages.JournalFileOpenLock`", text="Use :class:`~optuna.storages.journal.JournalFileOpenLock` instead.", ) class DeprecatedJournalFileOpenLock(JournalFileOpenLock): pass @deprecated_class( deprecated_version="4.0.0", removed_version="6.0.0", name="The import path :class:`~optuna.storages.JournalFileSymlinkLock`", text="Use :class:`~optuna.storages.journal.JournalFileSymlinkLock` instead.", ) class DeprecatedJournalFileSymlinkLock(JournalFileSymlinkLock): pass optuna-4.1.0/optuna/storages/journal/_redis.py000066400000000000000000000076661471332314300214700ustar00rootroot00000000000000import json import time from typing import Any from typing import Dict from typing import List from typing import Optional from optuna._deprecated import deprecated_class from optuna._experimental import experimental_class from optuna._imports import try_import from optuna.storages.journal._base import BaseJournalBackend from optuna.storages.journal._base import BaseJournalSnapshot with try_import() as _imports: import redis @experimental_class("3.1.0") class JournalRedisBackend(BaseJournalBackend, BaseJournalSnapshot): """Redis storage class for Journal log backend. Args: url: URL of the redis storage, password and db are optional. (ie: ``redis://localhost:6379``) use_cluster: Flag whether you use the Redis cluster. If this is :obj:`False`, it is assumed that you use the standalone Redis server and ensured that a write operation is atomic. This provides the consistency of the preserved logs. If this is :obj:`True`, it is assumed that you use the Redis cluster and not ensured that a write operation is atomic. This means the preserved logs can be inconsistent due to network errors, and may cause errors. prefix: Prefix of the preserved key of logs. This is useful when multiple users work on one Redis server. """ def __init__(self, url: str, use_cluster: bool = False, prefix: str = "") -> None: _imports.check() self._url = url self._redis = redis.Redis.from_url(url) self._use_cluster = use_cluster self._prefix = prefix def __getstate__(self) -> Dict[Any, Any]: state = self.__dict__.copy() del state["_redis"] return state def __setstate__(self, state: Dict[Any, Any]) -> None: self.__dict__.update(state) self._redis = redis.Redis.from_url(self._url) def read_logs(self, log_number_from: int) -> List[Dict[str, Any]]: max_log_number_bytes = self._redis.get(f"{self._prefix}:log_number") if max_log_number_bytes is None: return [] max_log_number = int(max_log_number_bytes) logs = [] for log_number in range(log_number_from, max_log_number + 1): sleep_secs = 0.1 while True: log = self._redis.get(self._key_log_id(log_number)) if log is not None: break time.sleep(sleep_secs) sleep_secs = min(sleep_secs * 2, 10) try: logs.append(json.loads(log)) except json.JSONDecodeError as err: if log_number != max_log_number: raise err return logs def append_logs(self, logs: List[Dict[str, Any]]) -> None: self._redis.setnx(f"{self._prefix}:log_number", -1) for log in logs: if not self._use_cluster: self._redis.eval( # type: ignore "local i = redis.call('incr', string.format('%s:log_number', ARGV[1])) " "redis.call('set', string.format('%s:log:%d', ARGV[1], i), ARGV[2])", 0, self._prefix, json.dumps(log), ) else: log_number = self._redis.incr(f"{self._prefix}:log_number", 1) self._redis.set(self._key_log_id(log_number), json.dumps(log)) def save_snapshot(self, snapshot: bytes) -> None: self._redis.set(f"{self._prefix}:snapshot", snapshot) def load_snapshot(self) -> Optional[bytes]: snapshot_bytes = self._redis.get(f"{self._prefix}:snapshot") return snapshot_bytes def _key_log_id(self, log_number: int) -> str: return f"{self._prefix}:log:{log_number}" @deprecated_class( "4.0.0", "6.0.0", text="Use :class:`~optuna.storages.journal.JournalRedisBackend` instead." ) class JournalRedisStorage(JournalRedisBackend): pass optuna-4.1.0/optuna/storages/journal/_storage.py000066400000000000000000000623501471332314300220150ustar00rootroot00000000000000import copy import datetime import enum import pickle import threading from typing import Any from typing import Container from typing import Dict from typing import List from typing import Optional from typing import Sequence import uuid import optuna from optuna._typing import JSONSerializable from optuna.distributions import BaseDistribution from optuna.distributions import check_distribution_compatibility from optuna.distributions import distribution_to_json from optuna.distributions import json_to_distribution from optuna.exceptions import DuplicatedStudyError from optuna.storages import BaseStorage from optuna.storages._base import DEFAULT_STUDY_NAME_PREFIX from optuna.storages.journal._base import BaseJournalBackend from optuna.storages.journal._base import BaseJournalSnapshot from optuna.study._frozen import FrozenStudy from optuna.study._study_direction import StudyDirection from optuna.trial import FrozenTrial from optuna.trial import TrialState _logger = optuna.logging.get_logger(__name__) NOT_FOUND_MSG = "Record does not exist." # A heuristic interval number to dump snapshots SNAPSHOT_INTERVAL = 100 class JournalOperation(enum.IntEnum): CREATE_STUDY = 0 DELETE_STUDY = 1 SET_STUDY_USER_ATTR = 2 SET_STUDY_SYSTEM_ATTR = 3 CREATE_TRIAL = 4 SET_TRIAL_PARAM = 5 SET_TRIAL_STATE_VALUES = 6 SET_TRIAL_INTERMEDIATE_VALUE = 7 SET_TRIAL_USER_ATTR = 8 SET_TRIAL_SYSTEM_ATTR = 9 class JournalStorage(BaseStorage): """Storage class for Journal storage backend. Note that library users can instantiate this class, but the attributes provided by this class are not supposed to be directly accessed by them. Journal storage writes a record of every operation to the database as it is executed and at the same time, keeps a latest snapshot of the database in-memory. If the database crashes for any reason, the storage can re-establish the contents in memory by replaying the operations stored from the beginning. Journal storage has several benefits over the conventional value logging storages. 1. The number of IOs can be reduced because of larger granularity of logs. 2. Journal storage has simpler backend API than value logging storage. 3. Journal storage keeps a snapshot in-memory so no need to add more cache. Example: .. code:: import optuna def objective(trial): ... storage = optuna.storages.JournalStorage( optuna.storages.journal.JournalFileBackend("./optuna_journal_storage.log") ) study = optuna.create_study(storage=storage) study.optimize(objective) In a Windows environment, an error message "A required privilege is not held by the client" may appear. In this case, you can solve the problem with creating storage by specifying :class:`~optuna.storages.journal.JournalFileOpenLock` as follows. .. code:: file_path = "./optuna_journal_storage.log" lock_obj = optuna.storages.journal.JournalFileOpenLock(file_path) storage = optuna.storages.JournalStorage( optuna.storages.journal.JournalFileBackend(file_path, lock_obj=lock_obj), ) """ def __init__(self, log_storage: BaseJournalBackend) -> None: self._worker_id_prefix = str(uuid.uuid4()) + "-" self._backend = log_storage self._thread_lock = threading.Lock() self._replay_result = JournalStorageReplayResult(self._worker_id_prefix) with self._thread_lock: if isinstance(self._backend, BaseJournalSnapshot): snapshot = self._backend.load_snapshot() if snapshot is not None: self.restore_replay_result(snapshot) self._sync_with_backend() def __getstate__(self) -> Dict[Any, Any]: state = self.__dict__.copy() del state["_worker_id_prefix"] del state["_replay_result"] del state["_thread_lock"] return state def __setstate__(self, state: Dict[Any, Any]) -> None: self.__dict__.update(state) self._worker_id_prefix = str(uuid.uuid4()) + "-" self._replay_result = JournalStorageReplayResult(self._worker_id_prefix) self._thread_lock = threading.Lock() def restore_replay_result(self, snapshot: bytes) -> None: try: r: Optional[JournalStorageReplayResult] = pickle.loads(snapshot) except (pickle.UnpicklingError, KeyError): _logger.warning("Failed to restore `JournalStorageReplayResult`.") return if r is None: return if not isinstance(r, JournalStorageReplayResult): _logger.warning("The restored object is not `JournalStorageReplayResult`.") return r._worker_id_prefix = self._worker_id_prefix r._worker_id_to_owned_trial_id = {} r._last_created_trial_id_by_this_process = -1 self._replay_result = r def _write_log(self, op_code: int, extra_fields: Dict[str, Any]) -> None: worker_id = self._replay_result.worker_id self._backend.append_logs([{"op_code": op_code, "worker_id": worker_id, **extra_fields}]) def _sync_with_backend(self) -> None: logs = self._backend.read_logs(self._replay_result.log_number_read) self._replay_result.apply_logs(logs) def create_new_study( self, directions: Sequence[StudyDirection], study_name: Optional[str] = None ) -> int: study_name = study_name or DEFAULT_STUDY_NAME_PREFIX + str(uuid.uuid4()) with self._thread_lock: self._write_log( JournalOperation.CREATE_STUDY, {"study_name": study_name, "directions": directions} ) self._sync_with_backend() for frozen_study in self._replay_result.get_all_studies(): if frozen_study.study_name != study_name: continue _logger.info("A new study created in Journal with name: {}".format(study_name)) study_id = frozen_study._study_id # Dump snapshot here. if ( isinstance(self._backend, BaseJournalSnapshot) and study_id != 0 and study_id % SNAPSHOT_INTERVAL == 0 ): self._backend.save_snapshot(pickle.dumps(self._replay_result)) return study_id assert False, "Should not reach." def delete_study(self, study_id: int) -> None: with self._thread_lock: self._write_log(JournalOperation.DELETE_STUDY, {"study_id": study_id}) self._sync_with_backend() def set_study_user_attr(self, study_id: int, key: str, value: Any) -> None: log: Dict[str, Any] = {"study_id": study_id, "user_attr": {key: value}} with self._thread_lock: self._write_log(JournalOperation.SET_STUDY_USER_ATTR, log) self._sync_with_backend() def set_study_system_attr(self, study_id: int, key: str, value: JSONSerializable) -> None: log: Dict[str, Any] = {"study_id": study_id, "system_attr": {key: value}} with self._thread_lock: self._write_log(JournalOperation.SET_STUDY_SYSTEM_ATTR, log) self._sync_with_backend() def get_study_id_from_name(self, study_name: str) -> int: with self._thread_lock: self._sync_with_backend() for study in self._replay_result.get_all_studies(): if study.study_name == study_name: return study._study_id raise KeyError(NOT_FOUND_MSG) def get_study_name_from_id(self, study_id: int) -> str: with self._thread_lock: self._sync_with_backend() return self._replay_result.get_study(study_id).study_name def get_study_directions(self, study_id: int) -> List[StudyDirection]: with self._thread_lock: self._sync_with_backend() return self._replay_result.get_study(study_id).directions def get_study_user_attrs(self, study_id: int) -> Dict[str, Any]: with self._thread_lock: self._sync_with_backend() return self._replay_result.get_study(study_id).user_attrs def get_study_system_attrs(self, study_id: int) -> Dict[str, Any]: with self._thread_lock: self._sync_with_backend() return self._replay_result.get_study(study_id).system_attrs def get_all_studies(self) -> List[FrozenStudy]: with self._thread_lock: self._sync_with_backend() return copy.deepcopy(self._replay_result.get_all_studies()) # Basic trial manipulation def create_new_trial(self, study_id: int, template_trial: Optional[FrozenTrial] = None) -> int: log: Dict[str, Any] = { "study_id": study_id, "datetime_start": datetime.datetime.now().isoformat(timespec="microseconds"), } if template_trial: log["state"] = template_trial.state if template_trial.values is not None and len(template_trial.values) > 1: log["value"] = None log["values"] = template_trial.values else: log["value"] = template_trial.value log["values"] = None if template_trial.datetime_start: log["datetime_start"] = template_trial.datetime_start.isoformat( timespec="microseconds" ) else: log["datetime_start"] = None if template_trial.datetime_complete: log["datetime_complete"] = template_trial.datetime_complete.isoformat( timespec="microseconds" ) log["distributions"] = { k: distribution_to_json(dist) for k, dist in template_trial.distributions.items() } log["params"] = { k: template_trial.distributions[k].to_internal_repr(param) for k, param in template_trial.params.items() } log["user_attrs"] = template_trial.user_attrs log["system_attrs"] = template_trial.system_attrs log["intermediate_values"] = template_trial.intermediate_values with self._thread_lock: self._write_log(JournalOperation.CREATE_TRIAL, log) self._sync_with_backend() trial_id = self._replay_result._last_created_trial_id_by_this_process # Dump snapshot here. if ( isinstance(self._backend, BaseJournalSnapshot) and trial_id != 0 and trial_id % SNAPSHOT_INTERVAL == 0 ): self._backend.save_snapshot(pickle.dumps(self._replay_result)) return trial_id def set_trial_param( self, trial_id: int, param_name: str, param_value_internal: float, distribution: BaseDistribution, ) -> None: log: Dict[str, Any] = { "trial_id": trial_id, "param_name": param_name, "param_value_internal": param_value_internal, "distribution": distribution_to_json(distribution), } with self._thread_lock: self._write_log(JournalOperation.SET_TRIAL_PARAM, log) self._sync_with_backend() def get_trial_id_from_study_id_trial_number(self, study_id: int, trial_number: int) -> int: with self._thread_lock: self._sync_with_backend() if len(self._replay_result._study_id_to_trial_ids[study_id]) <= trial_number: raise KeyError( "No trial with trial number {} exists in study with study_id {}.".format( trial_number, study_id ) ) return self._replay_result._study_id_to_trial_ids[study_id][trial_number] def set_trial_state_values( self, trial_id: int, state: TrialState, values: Optional[Sequence[float]] = None ) -> bool: log: Dict[str, Any] = { "trial_id": trial_id, "state": state, "values": values, } if state == TrialState.RUNNING: log["datetime_start"] = datetime.datetime.now().isoformat(timespec="microseconds") elif state.is_finished(): log["datetime_complete"] = datetime.datetime.now().isoformat(timespec="microseconds") with self._thread_lock: self._write_log(JournalOperation.SET_TRIAL_STATE_VALUES, log) self._sync_with_backend() if state == TrialState.RUNNING and trial_id != self._replay_result.owned_trial_id: return False else: return True def set_trial_intermediate_value( self, trial_id: int, step: int, intermediate_value: float ) -> None: log: Dict[str, Any] = { "trial_id": trial_id, "step": step, "intermediate_value": intermediate_value, } with self._thread_lock: self._write_log(JournalOperation.SET_TRIAL_INTERMEDIATE_VALUE, log) self._sync_with_backend() def set_trial_user_attr(self, trial_id: int, key: str, value: Any) -> None: log: Dict[str, Any] = { "trial_id": trial_id, "user_attr": {key: value}, } with self._thread_lock: self._write_log(JournalOperation.SET_TRIAL_USER_ATTR, log) self._sync_with_backend() def set_trial_system_attr(self, trial_id: int, key: str, value: JSONSerializable) -> None: log: Dict[str, Any] = { "trial_id": trial_id, "system_attr": {key: value}, } with self._thread_lock: self._write_log(JournalOperation.SET_TRIAL_SYSTEM_ATTR, log) self._sync_with_backend() def get_trial(self, trial_id: int) -> FrozenTrial: with self._thread_lock: self._sync_with_backend() return self._replay_result.get_trial(trial_id) def get_all_trials( self, study_id: int, deepcopy: bool = True, states: Optional[Container[TrialState]] = None, ) -> List[FrozenTrial]: with self._thread_lock: self._sync_with_backend() frozen_trials = self._replay_result.get_all_trials(study_id, states) if deepcopy: return copy.deepcopy(frozen_trials) return frozen_trials class JournalStorageReplayResult: def __init__(self, worker_id_prefix: str) -> None: self.log_number_read = 0 self._worker_id_prefix = worker_id_prefix self._studies: Dict[int, FrozenStudy] = {} self._trials: Dict[int, FrozenTrial] = {} self._study_id_to_trial_ids: Dict[int, List[int]] = {} self._trial_id_to_study_id: Dict[int, int] = {} self._next_study_id: int = 0 self._worker_id_to_owned_trial_id: Dict[str, int] = {} def apply_logs(self, logs: List[Dict[str, Any]]) -> None: for log in logs: self.log_number_read += 1 op = log["op_code"] if op == JournalOperation.CREATE_STUDY: self._apply_create_study(log) elif op == JournalOperation.DELETE_STUDY: self._apply_delete_study(log) elif op == JournalOperation.SET_STUDY_USER_ATTR: self._apply_set_study_user_attr(log) elif op == JournalOperation.SET_STUDY_SYSTEM_ATTR: self._apply_set_study_system_attr(log) elif op == JournalOperation.CREATE_TRIAL: self._apply_create_trial(log) elif op == JournalOperation.SET_TRIAL_PARAM: self._apply_set_trial_param(log) elif op == JournalOperation.SET_TRIAL_STATE_VALUES: self._apply_set_trial_state_values(log) elif op == JournalOperation.SET_TRIAL_INTERMEDIATE_VALUE: self._apply_set_trial_intermediate_value(log) elif op == JournalOperation.SET_TRIAL_USER_ATTR: self._apply_set_trial_user_attr(log) elif op == JournalOperation.SET_TRIAL_SYSTEM_ATTR: self._apply_set_trial_system_attr(log) else: assert False, "Should not reach." def get_study(self, study_id: int) -> FrozenStudy: if study_id not in self._studies: raise KeyError(NOT_FOUND_MSG) return self._studies[study_id] def get_all_studies(self) -> List[FrozenStudy]: return list(self._studies.values()) def get_trial(self, trial_id: int) -> FrozenTrial: if trial_id not in self._trials: raise KeyError(NOT_FOUND_MSG) return self._trials[trial_id] def get_all_trials( self, study_id: int, states: Optional[Container[TrialState]] ) -> List[FrozenTrial]: if study_id not in self._studies: raise KeyError(NOT_FOUND_MSG) frozen_trials: List[FrozenTrial] = [] for trial_id in self._study_id_to_trial_ids[study_id]: trial = self._trials[trial_id] if states is None or trial.state in states: frozen_trials.append(trial) return frozen_trials @property def worker_id(self) -> str: return self._worker_id_prefix + str(threading.get_ident()) @property def owned_trial_id(self) -> Optional[int]: return self._worker_id_to_owned_trial_id.get(self.worker_id) def _is_issued_by_this_worker(self, log: Dict[str, Any]) -> bool: return log["worker_id"] == self.worker_id def _study_exists(self, study_id: int, log: Dict[str, Any]) -> bool: if study_id in self._studies: return True if self._is_issued_by_this_worker(log): raise KeyError(NOT_FOUND_MSG) return False def _apply_create_study(self, log: Dict[str, Any]) -> None: study_name = log["study_name"] directions = [StudyDirection(d) for d in log["directions"]] if study_name in [s.study_name for s in self._studies.values()]: if self._is_issued_by_this_worker(log): raise DuplicatedStudyError( "Another study with name '{}' already exists. " "Please specify a different name, or reuse the existing one " "by setting `load_if_exists` (for Python API) or " "`--skip-if-exists` flag (for CLI).".format(study_name) ) return study_id = self._next_study_id self._next_study_id += 1 self._studies[study_id] = FrozenStudy( study_name=study_name, direction=None, user_attrs={}, system_attrs={}, study_id=study_id, directions=directions, ) self._study_id_to_trial_ids[study_id] = [] def _apply_delete_study(self, log: Dict[str, Any]) -> None: study_id = log["study_id"] if self._study_exists(study_id, log): fs = self._studies.pop(study_id) assert fs._study_id == study_id def _apply_set_study_user_attr(self, log: Dict[str, Any]) -> None: study_id = log["study_id"] if self._study_exists(study_id, log): assert len(log["user_attr"]) == 1 self._studies[study_id].user_attrs.update(log["user_attr"]) def _apply_set_study_system_attr(self, log: Dict[str, Any]) -> None: study_id = log["study_id"] if self._study_exists(study_id, log): assert len(log["system_attr"]) == 1 self._studies[study_id].system_attrs.update(log["system_attr"]) def _apply_create_trial(self, log: Dict[str, Any]) -> None: study_id = log["study_id"] if not self._study_exists(study_id, log): return trial_id = len(self._trials) distributions = {} if "distributions" in log: distributions = {k: json_to_distribution(v) for k, v in log["distributions"].items()} params = {} if "params" in log: params = {k: distributions[k].to_external_repr(p) for k, p in log["params"].items()} if log["datetime_start"] is not None: datetime_start = datetime.datetime.fromisoformat(log["datetime_start"]) else: datetime_start = None if "datetime_complete" in log: datetime_complete = datetime.datetime.fromisoformat(log["datetime_complete"]) else: datetime_complete = None self._trials[trial_id] = FrozenTrial( trial_id=trial_id, number=len(self._study_id_to_trial_ids[study_id]), state=TrialState(log.get("state", TrialState.RUNNING.value)), params=params, distributions=distributions, user_attrs=log.get("user_attrs", {}), system_attrs=log.get("system_attrs", {}), value=log.get("value", None), intermediate_values={int(k): v for k, v in log.get("intermediate_values", {}).items()}, datetime_start=datetime_start, datetime_complete=datetime_complete, values=log.get("values", None), ) self._study_id_to_trial_ids[study_id].append(trial_id) self._trial_id_to_study_id[trial_id] = study_id if self._is_issued_by_this_worker(log): self._last_created_trial_id_by_this_process = trial_id if self._trials[trial_id].state == TrialState.RUNNING: self._worker_id_to_owned_trial_id[self.worker_id] = trial_id def _apply_set_trial_param(self, log: Dict[str, Any]) -> None: trial_id = log["trial_id"] if not self._trial_exists_and_updatable(trial_id, log): return param_name = log["param_name"] param_value_internal = log["param_value_internal"] distribution = json_to_distribution(log["distribution"]) study_id = self._trial_id_to_study_id[trial_id] for prev_trial_id in self._study_id_to_trial_ids[study_id]: prev_trial = self._trials[prev_trial_id] if param_name in prev_trial.params.keys(): try: check_distribution_compatibility( prev_trial.distributions[param_name], distribution ) except Exception: if self._is_issued_by_this_worker(log): raise return break trial = copy.copy(self._trials[trial_id]) trial.params = { **copy.copy(trial.params), param_name: distribution.to_external_repr(param_value_internal), } trial.distributions = {**copy.copy(trial.distributions), param_name: distribution} self._trials[trial_id] = trial def _apply_set_trial_state_values(self, log: Dict[str, Any]) -> None: trial_id = log["trial_id"] if not self._trial_exists_and_updatable(trial_id, log): return state = TrialState(log["state"]) if state == self._trials[trial_id].state and state == TrialState.RUNNING: return trial = copy.copy(self._trials[trial_id]) if state == TrialState.RUNNING: trial.datetime_start = datetime.datetime.fromisoformat(log["datetime_start"]) if self._is_issued_by_this_worker(log): self._worker_id_to_owned_trial_id[self.worker_id] = trial_id if state.is_finished(): trial.datetime_complete = datetime.datetime.fromisoformat(log["datetime_complete"]) trial.state = state if log["values"] is not None: trial.values = log["values"] self._trials[trial_id] = trial def _apply_set_trial_intermediate_value(self, log: Dict[str, Any]) -> None: trial_id = log["trial_id"] if self._trial_exists_and_updatable(trial_id, log): trial = copy.copy(self._trials[trial_id]) trial.intermediate_values = { **copy.copy(trial.intermediate_values), log["step"]: log["intermediate_value"], } self._trials[trial_id] = trial def _apply_set_trial_user_attr(self, log: Dict[str, Any]) -> None: trial_id = log["trial_id"] if self._trial_exists_and_updatable(trial_id, log): assert len(log["user_attr"]) == 1 trial = copy.copy(self._trials[trial_id]) trial.user_attrs = {**copy.copy(trial.user_attrs), **log["user_attr"]} self._trials[trial_id] = trial def _apply_set_trial_system_attr(self, log: Dict[str, Any]) -> None: trial_id = log["trial_id"] if self._trial_exists_and_updatable(trial_id, log): assert len(log["system_attr"]) == 1 trial = copy.copy(self._trials[trial_id]) trial.system_attrs = { **copy.copy(trial.system_attrs), **log["system_attr"], } self._trials[trial_id] = trial def _trial_exists_and_updatable(self, trial_id: int, log: Dict[str, Any]) -> bool: if trial_id not in self._trials: if self._is_issued_by_this_worker(log): raise KeyError(NOT_FOUND_MSG) return False elif self._trials[trial_id].state.is_finished(): if self._is_issued_by_this_worker(log): raise RuntimeError( "Trial#{} has already finished and can not be updated.".format( self._trials[trial_id].number ) ) return False else: return True optuna-4.1.0/optuna/study/000077500000000000000000000000001471332314300155015ustar00rootroot00000000000000optuna-4.1.0/optuna/study/__init__.py000066400000000000000000000012771471332314300176210ustar00rootroot00000000000000from optuna._callbacks import MaxTrialsCallback from optuna.study._study_direction import StudyDirection from optuna.study._study_summary import StudySummary from optuna.study.study import copy_study from optuna.study.study import create_study from optuna.study.study import delete_study from optuna.study.study import get_all_study_names from optuna.study.study import get_all_study_summaries from optuna.study.study import load_study from optuna.study.study import Study __all__ = [ "MaxTrialsCallback", "StudyDirection", "StudySummary", "copy_study", "create_study", "delete_study", "get_all_study_names", "get_all_study_summaries", "load_study", "Study", ] optuna-4.1.0/optuna/study/_constrained_optimization.py000066400000000000000000000013751471332314300233370ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Sequence from optuna.trial import FrozenTrial _CONSTRAINTS_KEY = "constraints" def _get_feasible_trials(trials: Sequence[FrozenTrial]) -> list[FrozenTrial]: """Return feasible trials from given trials. This function assumes that the trials were created in constrained optimization. Therefore, if there is no violation value in the trial, it is considered infeasible. Returns: A list of feasible trials. """ feasible_trials = [] for trial in trials: constraints = trial.system_attrs.get(_CONSTRAINTS_KEY) if constraints is not None and all(x <= 0.0 for x in constraints): feasible_trials.append(trial) return feasible_trials optuna-4.1.0/optuna/study/_dataframe.py000066400000000000000000000102451471332314300201400ustar00rootroot00000000000000from __future__ import annotations import collections from typing import Any from typing import DefaultDict from typing import Set import optuna from optuna._imports import try_import from optuna.trial._state import TrialState with try_import() as _imports: # `Study.trials_dataframe` is disabled if pandas is not available. import pandas as pd # Required for type annotation in `Study.trials_dataframe`. if not _imports.is_successful(): pd = object # NOQA __all__ = ["pd"] def _create_records_and_aggregate_column( study: "optuna.Study", attrs: tuple[str, ...] ) -> tuple[list[dict[tuple[str, str], Any]], list[tuple[str, str]]]: attrs_to_df_columns: dict[str, str] = {} for attr in attrs: if attr.startswith("_"): # Python conventional underscores are omitted in the dataframe. df_column = attr[1:] else: df_column = attr attrs_to_df_columns[attr] = df_column # column_agg is an aggregator of column names. # Keys of column agg are attributes of `FrozenTrial` such as 'trial_id' and 'params'. # Values are dataframe columns such as ('trial_id', '') and ('params', 'n_layers'). column_agg: DefaultDict[str, Set] = collections.defaultdict(set) non_nested_attr = "" metric_names = study.metric_names records = [] for trial in study.get_trials(deepcopy=False): record = {} for attr, df_column in attrs_to_df_columns.items(): value = getattr(trial, attr) if isinstance(value, TrialState): value = value.name if isinstance(value, dict): for nested_attr, nested_value in value.items(): record[(df_column, nested_attr)] = nested_value column_agg[attr].add((df_column, nested_attr)) elif attr == "values": # Expand trial.values. # trial.values should be None when the trial's state is FAIL or PRUNED. trial_values = [None] * len(study.directions) if value is None else value iterator = ( enumerate(trial_values) if metric_names is None else zip(metric_names, trial_values) ) for nested_attr, nested_value in iterator: record[(df_column, nested_attr)] = nested_value column_agg[attr].add((df_column, nested_attr)) elif isinstance(value, list): for nested_attr, nested_value in enumerate(value): record[(df_column, nested_attr)] = nested_value column_agg[attr].add((df_column, nested_attr)) elif attr == "value": nested_attr = non_nested_attr if metric_names is None else metric_names[0] record[(df_column, nested_attr)] = value column_agg[attr].add((df_column, nested_attr)) else: record[(df_column, non_nested_attr)] = value column_agg[attr].add((df_column, non_nested_attr)) records.append(record) columns: list[tuple[str, str]] = sum( (sorted(column_agg[k]) for k in attrs if k in column_agg), [] ) return records, columns def _flatten_columns(columns: list[tuple[str, str]]) -> list[str]: # Flatten the `MultiIndex` columns where names are concatenated with underscores. # Filtering is required to omit non-nested columns avoiding unwanted trailing underscores. return ["_".join(filter(lambda c: c, map(lambda c: str(c), col))) for col in columns] def _trials_dataframe( study: "optuna.Study", attrs: tuple[str, ...], multi_index: bool ) -> "pd.DataFrame": _imports.check() # If no trials, return an empty dataframe. if len(study.get_trials(deepcopy=False)) == 0: return pd.DataFrame() if "value" in attrs and study._is_multi_objective(): attrs = tuple("values" if attr == "value" else attr for attr in attrs) records, columns = _create_records_and_aggregate_column(study, attrs) df = pd.DataFrame(records, columns=pd.MultiIndex.from_tuples(columns)) if not multi_index: df.columns = _flatten_columns(columns) return df optuna-4.1.0/optuna/study/_frozen.py000066400000000000000000000053401471332314300175170ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Sequence from typing import Any from optuna import logging from optuna.study._study_direction import StudyDirection _logger = logging.get_logger(__name__) class FrozenStudy: """Basic attributes of a :class:`~optuna.study.Study`. This class is private and not referenced by Optuna users. Attributes: study_name: Name of the :class:`~optuna.study.Study`. direction: :class:`~optuna.study.StudyDirection` of the :class:`~optuna.study.Study`. .. note:: This attribute is only available during single-objective optimization. directions: A list of :class:`~optuna.study.StudyDirection` objects. user_attrs: Dictionary that contains the attributes of the :class:`~optuna.study.Study` set with :func:`optuna.study.Study.set_user_attr`. system_attrs: Dictionary that contains the attributes of the :class:`~optuna.study.Study` internally set by Optuna. """ def __init__( self, study_name: str, direction: StudyDirection | None, user_attrs: dict[str, Any], system_attrs: dict[str, Any], study_id: int, *, directions: Sequence[StudyDirection] | None = None, ): self.study_name = study_name if direction is None and directions is None: raise ValueError("Specify one of `direction` and `directions`.") elif directions is not None: self._directions = list(directions) elif direction is not None: self._directions = [direction] else: raise ValueError("Specify only one of `direction` and `directions`.") self.user_attrs = user_attrs self.system_attrs = system_attrs self._study_id = study_id def __eq__(self, other: Any) -> bool: if not isinstance(other, FrozenStudy): return NotImplemented return other.__dict__ == self.__dict__ def __lt__(self, other: Any) -> bool: if not isinstance(other, FrozenStudy): return NotImplemented return self._study_id < other._study_id def __le__(self, other: Any) -> bool: if not isinstance(other, FrozenStudy): return NotImplemented return self._study_id <= other._study_id @property def direction(self) -> StudyDirection: if len(self._directions) > 1: raise RuntimeError( "This attribute is not available during multi-objective optimization." ) return self._directions[0] @property def directions(self) -> list[StudyDirection]: return self._directions optuna-4.1.0/optuna/study/_multi_objective.py000066400000000000000000000254011471332314300214000ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Sequence import numpy as np import optuna from optuna.study._constrained_optimization import _get_feasible_trials from optuna.study._study_direction import StudyDirection from optuna.trial import FrozenTrial from optuna.trial import TrialState def _get_pareto_front_trials_by_trials( trials: Sequence[FrozenTrial], directions: Sequence[StudyDirection], consider_constraint: bool = False, ) -> list[FrozenTrial]: # NOTE(nabenabe0928): Vectorization relies on all the trials being complete. trials = [t for t in trials if t.state == TrialState.COMPLETE] if consider_constraint: trials = _get_feasible_trials(trials) if len(trials) == 0: return [] if any(len(t.values) != len(directions) for t in trials): raise ValueError( "The number of the values and the number of the objectives must be identical." ) loss_values = np.asarray( [[_normalize_value(v, d) for v, d in zip(t.values, directions)] for t in trials] ) on_front = _is_pareto_front(loss_values, assume_unique_lexsorted=False) return [t for t, is_pareto in zip(trials, on_front) if is_pareto] def _get_pareto_front_trials( study: "optuna.study.Study", consider_constraint: bool = False ) -> list[FrozenTrial]: return _get_pareto_front_trials_by_trials(study.trials, study.directions, consider_constraint) def _fast_non_domination_rank( loss_values: np.ndarray, *, penalty: np.ndarray | None = None, n_below: int | None = None ) -> np.ndarray: """Calculate non-domination rank based on the fast non-dominated sort algorithm. The fast non-dominated sort algorithm assigns a rank to each trial based on the dominance relationship of the trials, determined by the objective values and the penalty values. The algorithm is based on `the constrained NSGA-II algorithm `__, but the handling of the case when penalty values are None is different. The algorithm assigns the rank according to the following rules: 1. Feasible trials: First, the algorithm assigns the rank to feasible trials, whose penalty values are less than or equal to 0, according to unconstrained version of fast non- dominated sort. 2. Infeasible trials: Next, the algorithm assigns the rank from the minimum penalty value of to the maximum penalty value. 3. Trials with no penalty information (constraints value is None): Finally, The algorithm assigns the rank to trials with no penalty information according to unconstrained version of fast non-dominated sort. Note that only this step is different from the original constrained NSGA-II algorithm. Plus, the algorithm terminates whenever the number of sorted trials reaches n_below. Args: loss_values: Objective values, which is better when it is lower, of each trials. penalty: Constraints values of each trials. Defaults to None. n_below: The minimum number of top trials required to be sorted. The algorithm will terminate when the number of sorted trials reaches n_below. Defaults to None. Returns: An ndarray in the shape of (n_trials,), where each element is the non-domination rank of each trial. The rank is 0-indexed. This function guarantees the correctness of the ranks only up to the top-``n_below`` solutions. If a solution's rank is worse than the top-``n_below`` solution, its rank will be guaranteed to be greater than the rank of the top-``n_below`` solution. """ if len(loss_values) == 0: return np.array([], dtype=int) n_below = n_below or len(loss_values) assert n_below > 0, "n_below must be a positive integer." if penalty is None: return _calculate_nondomination_rank(loss_values, n_below=n_below) if len(penalty) != len(loss_values): raise ValueError( "The length of penalty and loss_values must be same, but got " f"len(penalty)={len(penalty)} and len(loss_values)={len(loss_values)}." ) ranks = np.full(len(loss_values), -1, dtype=int) is_penalty_nan = np.isnan(penalty) is_feasible = np.logical_and(~is_penalty_nan, penalty <= 0) is_infeasible = np.logical_and(~is_penalty_nan, penalty > 0) # First, we calculate the domination rank for feasible trials. ranks[is_feasible] = _calculate_nondomination_rank(loss_values[is_feasible], n_below=n_below) n_below -= np.count_nonzero(is_feasible) # Second, we calculate the domination rank for infeasible trials. top_rank_infeasible = np.max(ranks[is_feasible], initial=-1) + 1 ranks[is_infeasible] = top_rank_infeasible + _calculate_nondomination_rank( penalty[is_infeasible][:, np.newaxis], n_below=n_below ) n_below -= np.count_nonzero(is_infeasible) # Third, we calculate the domination rank for trials with no penalty information. top_rank_penalty_nan = np.max(ranks[~is_penalty_nan], initial=-1) + 1 ranks[is_penalty_nan] = top_rank_penalty_nan + _calculate_nondomination_rank( loss_values[is_penalty_nan], n_below=n_below ) assert np.all(ranks != -1), "All the rank must be updated." return ranks def _is_pareto_front_nd(unique_lexsorted_loss_values: np.ndarray) -> np.ndarray: # NOTE(nabenabe0928): I tried the Kung's algorithm below, but it was not really quick. # https://github.com/optuna/optuna/pull/5302#issuecomment-1988665532 # As unique_lexsorted_loss_values[:, 0] is sorted, we do not need it to judge dominance. loss_values = unique_lexsorted_loss_values[:, 1:] n_trials = loss_values.shape[0] on_front = np.zeros(n_trials, dtype=bool) nondominated_indices = np.arange(n_trials) while len(loss_values): # The following judges `np.any(loss_values[i] < loss_values[0])` for each `i`. nondominated_and_not_top = np.any(loss_values < loss_values[0], axis=1) # NOTE: trials[j] cannot dominate trials[i] for i < j because of lexsort. # Therefore, nondominated_indices[0] is always non-dominated. on_front[nondominated_indices[0]] = True loss_values = loss_values[nondominated_and_not_top] nondominated_indices = nondominated_indices[nondominated_and_not_top] return on_front def _is_pareto_front_2d(unique_lexsorted_loss_values: np.ndarray) -> np.ndarray: n_trials = unique_lexsorted_loss_values.shape[0] cummin_value1 = np.minimum.accumulate(unique_lexsorted_loss_values[:, 1]) on_front = np.ones(n_trials, dtype=bool) on_front[1:] = cummin_value1[1:] < cummin_value1[:-1] # True if cummin value1 is new minimum. return on_front def _is_pareto_front_for_unique_sorted(unique_lexsorted_loss_values: np.ndarray) -> np.ndarray: (n_trials, n_objectives) = unique_lexsorted_loss_values.shape if n_objectives == 1: on_front = np.zeros(len(unique_lexsorted_loss_values), dtype=bool) on_front[0] = True # Only the first element is Pareto optimal. return on_front elif n_objectives == 2: return _is_pareto_front_2d(unique_lexsorted_loss_values) else: return _is_pareto_front_nd(unique_lexsorted_loss_values) def _is_pareto_front(loss_values: np.ndarray, assume_unique_lexsorted: bool) -> np.ndarray: # NOTE(nabenabe): If assume_unique_lexsorted=True, but loss_values is not a unique array, # Duplicated Pareto solutions will be filtered out except for the earliest occurrences. # If assume_unique_lexsorted=True and loss_values[:, 0] is not sorted, then the result will be # incorrect. if assume_unique_lexsorted: return _is_pareto_front_for_unique_sorted(loss_values) unique_lexsorted_loss_values, order_inv = np.unique(loss_values, axis=0, return_inverse=True) on_front = _is_pareto_front_for_unique_sorted(unique_lexsorted_loss_values) # NOTE(nabenabe): We can remove `.reshape(-1)` if ``numpy==2.0.0`` is not used. # https://github.com/numpy/numpy/issues/26738 # TODO: Remove `.reshape(-1)` once `numpy==2.0.0` is obsolete. return on_front[order_inv.reshape(-1)] def _calculate_nondomination_rank( loss_values: np.ndarray, *, n_below: int | None = None ) -> np.ndarray: if len(loss_values) == 0 or (n_below is not None and n_below <= 0): return np.zeros(len(loss_values), dtype=int) (n_trials, n_objectives) = loss_values.shape if n_objectives == 1: _, ranks = np.unique(loss_values[:, 0], return_inverse=True) return ranks # It ensures that trials[j] will not dominate trials[i] for i < j. # np.unique does lexsort. unique_lexsorted_loss_values, order_inv = np.unique(loss_values, return_inverse=True, axis=0) n_unique = unique_lexsorted_loss_values.shape[0] # Clip n_below. n_below = min(n_below or len(unique_lexsorted_loss_values), len(unique_lexsorted_loss_values)) ranks = np.zeros(n_unique, dtype=int) rank = 0 indices = np.arange(n_unique) while n_unique - indices.size < n_below: on_front = _is_pareto_front(unique_lexsorted_loss_values, assume_unique_lexsorted=True) ranks[indices[on_front]] = rank # Remove the recent Pareto solutions. indices = indices[~on_front] unique_lexsorted_loss_values = unique_lexsorted_loss_values[~on_front] rank += 1 ranks[indices] = rank # Rank worse than the top n_below is defined as the worst rank. # NOTE(nabenabe): We can remove `.reshape(-1)` if ``numpy==2.0.0`` is not used. # https://github.com/numpy/numpy/issues/26738 # TODO: Remove `.reshape(-1)` once `numpy==2.0.0` is obsolete. return ranks[order_inv.reshape(-1)] def _dominates( trial0: FrozenTrial, trial1: FrozenTrial, directions: Sequence[StudyDirection] ) -> bool: values0 = trial0.values values1 = trial1.values if trial0.state != TrialState.COMPLETE: return False if trial1.state != TrialState.COMPLETE: return True assert values0 is not None assert values1 is not None if len(values0) != len(values1): raise ValueError("Trials with different numbers of objectives cannot be compared.") if len(values0) != len(directions): raise ValueError( "The number of the values and the number of the objectives are mismatched." ) normalized_values0 = [_normalize_value(v, d) for v, d in zip(values0, directions)] normalized_values1 = [_normalize_value(v, d) for v, d in zip(values1, directions)] if normalized_values0 == normalized_values1: return False return all(v0 <= v1 for v0, v1 in zip(normalized_values0, normalized_values1)) def _normalize_value(value: float | None, direction: StudyDirection) -> float: if value is None: return float("inf") if direction is StudyDirection.MAXIMIZE: value = -value return value optuna-4.1.0/optuna/study/_optimize.py000066400000000000000000000212771471332314300200630ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Iterable from collections.abc import Sequence from concurrent.futures import FIRST_COMPLETED from concurrent.futures import Future from concurrent.futures import ThreadPoolExecutor from concurrent.futures import wait import datetime import gc import itertools import os import sys from typing import Any from typing import Callable import warnings import optuna from optuna import exceptions from optuna import logging from optuna import progress_bar as pbar_module from optuna import trial as trial_module from optuna.storages._heartbeat import get_heartbeat_thread from optuna.storages._heartbeat import is_heartbeat_enabled from optuna.study._tell import _tell_with_warning from optuna.study._tell import STUDY_TELL_WARNING_KEY from optuna.trial import FrozenTrial from optuna.trial import TrialState _logger = logging.get_logger(__name__) def _optimize( study: "optuna.Study", func: "optuna.study.study.ObjectiveFuncType", n_trials: int | None = None, timeout: float | None = None, n_jobs: int = 1, catch: tuple[type[Exception], ...] = (), callbacks: Iterable[Callable[["optuna.Study", FrozenTrial], None]] | None = None, gc_after_trial: bool = False, show_progress_bar: bool = False, ) -> None: if not isinstance(catch, tuple): raise TypeError( "The catch argument is of type '{}' but must be a tuple.".format(type(catch).__name__) ) if study._thread_local.in_optimize_loop: raise RuntimeError("Nested invocation of `Study.optimize` method isn't allowed.") if show_progress_bar and n_trials is None and timeout is not None and n_jobs != 1: warnings.warn("The timeout-based progress bar is not supported with n_jobs != 1.") show_progress_bar = False progress_bar = pbar_module._ProgressBar(show_progress_bar, n_trials, timeout) study._stop_flag = False try: if n_jobs == 1: _optimize_sequential( study, func, n_trials, timeout, catch, callbacks, gc_after_trial, reseed_sampler_rng=False, time_start=None, progress_bar=progress_bar, ) else: if n_jobs == -1: n_jobs = os.cpu_count() or 1 time_start = datetime.datetime.now() futures: set[Future] = set() with ThreadPoolExecutor(max_workers=n_jobs) as executor: for n_submitted_trials in itertools.count(): if study._stop_flag: break if ( timeout is not None and (datetime.datetime.now() - time_start).total_seconds() > timeout ): break if n_trials is not None and n_submitted_trials >= n_trials: break if len(futures) >= n_jobs: completed, futures = wait(futures, return_when=FIRST_COMPLETED) # Raise if exception occurred in executing the completed futures. for f in completed: f.result() futures.add( executor.submit( _optimize_sequential, study, func, 1, timeout, catch, callbacks, gc_after_trial, True, time_start, progress_bar, ) ) finally: study._thread_local.in_optimize_loop = False progress_bar.close() def _optimize_sequential( study: "optuna.Study", func: "optuna.study.study.ObjectiveFuncType", n_trials: int | None, timeout: float | None, catch: tuple[type[Exception], ...], callbacks: Iterable[Callable[["optuna.Study", FrozenTrial], None]] | None, gc_after_trial: bool, reseed_sampler_rng: bool, time_start: datetime.datetime | None, progress_bar: pbar_module._ProgressBar | None, ) -> None: # Here we set `in_optimize_loop = True`, not at the beginning of the `_optimize()` function. # Because it is a thread-local object and `n_jobs` option spawns new threads. study._thread_local.in_optimize_loop = True if reseed_sampler_rng: study.sampler.reseed_rng() i_trial = 0 if time_start is None: time_start = datetime.datetime.now() while True: if study._stop_flag: break if n_trials is not None: if i_trial >= n_trials: break i_trial += 1 if timeout is not None: elapsed_seconds = (datetime.datetime.now() - time_start).total_seconds() if elapsed_seconds >= timeout: break try: frozen_trial = _run_trial(study, func, catch) finally: # The following line mitigates memory problems that can be occurred in some # environments (e.g., services that use computing containers such as GitHub Actions). # Please refer to the following PR for further details: # https://github.com/optuna/optuna/pull/325. if gc_after_trial: gc.collect() if callbacks is not None: for callback in callbacks: callback(study, frozen_trial) if progress_bar is not None: elapsed_seconds = (datetime.datetime.now() - time_start).total_seconds() progress_bar.update(elapsed_seconds, study) study._storage.remove_session() def _run_trial( study: "optuna.Study", func: "optuna.study.study.ObjectiveFuncType", catch: tuple[type[Exception], ...], ) -> trial_module.FrozenTrial: if is_heartbeat_enabled(study._storage): optuna.storages.fail_stale_trials(study) trial = study.ask() state: TrialState | None = None value_or_values: float | Sequence[float] | None = None func_err: Exception | KeyboardInterrupt | None = None func_err_fail_exc_info: Any | None = None with get_heartbeat_thread(trial._trial_id, study._storage): try: value_or_values = func(trial) except exceptions.TrialPruned as e: # TODO(mamu): Handle multi-objective cases. state = TrialState.PRUNED func_err = e except (Exception, KeyboardInterrupt) as e: state = TrialState.FAIL func_err = e func_err_fail_exc_info = sys.exc_info() # `_tell_with_warning` may raise during trial post-processing. try: frozen_trial = _tell_with_warning( study=study, trial=trial, value_or_values=value_or_values, state=state, suppress_warning=True, ) except Exception: frozen_trial = study._storage.get_trial(trial._trial_id) raise finally: if frozen_trial.state == TrialState.COMPLETE: study._log_completed_trial(frozen_trial) elif frozen_trial.state == TrialState.PRUNED: _logger.info("Trial {} pruned. {}".format(frozen_trial.number, str(func_err))) elif frozen_trial.state == TrialState.FAIL: if func_err is not None: _log_failed_trial( frozen_trial, repr(func_err), exc_info=func_err_fail_exc_info, value_or_values=value_or_values, ) elif STUDY_TELL_WARNING_KEY in frozen_trial.system_attrs: _log_failed_trial( frozen_trial, frozen_trial.system_attrs[STUDY_TELL_WARNING_KEY], value_or_values=value_or_values, ) else: assert False, "Should not reach." else: assert False, "Should not reach." if ( frozen_trial.state == TrialState.FAIL and func_err is not None and not isinstance(func_err, catch) ): raise func_err return frozen_trial def _log_failed_trial( trial: FrozenTrial, message: str | Warning, exc_info: Any = None, value_or_values: Any = None, ) -> None: _logger.warning( "Trial {} failed with parameters: {} because of the following error: {}.".format( trial.number, trial.params, message ), exc_info=exc_info, ) _logger.warning("Trial {} failed with value {}.".format(trial.number, repr(value_or_values))) optuna-4.1.0/optuna/study/_study_direction.py000066400000000000000000000006451471332314300214270ustar00rootroot00000000000000import enum class StudyDirection(enum.IntEnum): """Direction of a :class:`~optuna.study.Study`. Attributes: NOT_SET: Direction has not been set. MINIMIZE: :class:`~optuna.study.Study` minimizes the objective function. MAXIMIZE: :class:`~optuna.study.Study` maximizes the objective function. """ NOT_SET = 0 MINIMIZE = 1 MAXIMIZE = 2 optuna-4.1.0/optuna/study/_study_summary.py000066400000000000000000000101331471332314300211350ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Sequence import datetime from typing import Any import warnings from optuna import logging from optuna import trial from optuna.study._study_direction import StudyDirection _logger = logging.get_logger(__name__) class StudySummary: """Basic attributes and aggregated results of a :class:`~optuna.study.Study`. See also :func:`optuna.study.get_all_study_summaries`. Attributes: study_name: Name of the :class:`~optuna.study.Study`. direction: :class:`~optuna.study.StudyDirection` of the :class:`~optuna.study.Study`. .. note:: This attribute is only available during single-objective optimization. directions: A sequence of :class:`~optuna.study.StudyDirection` objects. best_trial: :class:`optuna.trial.FrozenTrial` with best objective value in the :class:`~optuna.study.Study`. user_attrs: Dictionary that contains the attributes of the :class:`~optuna.study.Study` set with :func:`optuna.study.Study.set_user_attr`. system_attrs: Dictionary that contains the attributes of the :class:`~optuna.study.Study` internally set by Optuna. .. warning:: Deprecated in v3.1.0. ``system_attrs`` argument will be removed in the future. The removal of this feature is currently scheduled for v5.0.0, but this schedule is subject to change. See https://github.com/optuna/optuna/releases/tag/v3.1.0. n_trials: The number of trials ran in the :class:`~optuna.study.Study`. datetime_start: Datetime where the :class:`~optuna.study.Study` started. """ def __init__( self, study_name: str, direction: StudyDirection | None, best_trial: trial.FrozenTrial | None, user_attrs: dict[str, Any], system_attrs: dict[str, Any], n_trials: int, datetime_start: datetime.datetime | None, study_id: int, *, directions: Sequence[StudyDirection] | None = None, ): self.study_name = study_name if direction is None and directions is None: raise ValueError("Specify one of `direction` and `directions`.") elif directions is not None: self._directions = list(directions) elif direction is not None: self._directions = [direction] else: raise ValueError("Specify only one of `direction` and `directions`.") self.best_trial = best_trial self.user_attrs = user_attrs self._system_attrs = system_attrs self.n_trials = n_trials self.datetime_start = datetime_start self._study_id = study_id def __eq__(self, other: Any) -> bool: if not isinstance(other, StudySummary): return NotImplemented return other.__dict__ == self.__dict__ def __lt__(self, other: Any) -> bool: if not isinstance(other, StudySummary): return NotImplemented return self._study_id < other._study_id def __le__(self, other: Any) -> bool: if not isinstance(other, StudySummary): return NotImplemented return self._study_id <= other._study_id @property def direction(self) -> StudyDirection: if len(self._directions) > 1: raise RuntimeError( "This attribute is not available during multi-objective optimization." ) return self._directions[0] @property def directions(self) -> Sequence[StudyDirection]: return self._directions @property def system_attrs(self) -> dict[str, Any]: warnings.warn( "`system_attrs` has been deprecated in v3.1.0. " "The removal of this feature is currently scheduled for v5.0.0, " "but this schedule is subject to change. " "See https://github.com/optuna/optuna/releases/tag/v3.1.0.", FutureWarning, ) return self._system_attrs optuna-4.1.0/optuna/study/_tell.py000066400000000000000000000152031471332314300171530ustar00rootroot00000000000000from __future__ import annotations import copy import math from typing import Optional from typing import Sequence from typing import TYPE_CHECKING from typing import Union import warnings import optuna from optuna import logging from optuna import pruners from optuna.trial import FrozenTrial from optuna.trial import TrialState if TYPE_CHECKING: from optuna import Study from optuna import Trial # This is used for propagating warning message to Study.optimize. STUDY_TELL_WARNING_KEY = "STUDY_TELL_WARNING" _logger = logging.get_logger(__name__) def _get_frozen_trial(study: Study, trial: Union[Trial, int]) -> FrozenTrial: if isinstance(trial, optuna.Trial): trial_id = trial._trial_id elif isinstance(trial, int): trial_number = trial try: trial_id = study._storage.get_trial_id_from_study_id_trial_number( study._study_id, trial_number ) except KeyError as e: raise ValueError( f"Cannot tell for trial with number {trial_number} since it has not been " "created." ) from e else: raise TypeError("Trial must be a trial object or trial number.") return study._storage.get_trial(trial_id) def _check_state_and_values( state: Optional[TrialState], values: Optional[Union[float, Sequence[float]]] ) -> None: if state == TrialState.COMPLETE: if values is None: raise ValueError( "No values were told. Values are required when state is TrialState.COMPLETE." ) elif state in (TrialState.PRUNED, TrialState.FAIL): if values is not None: raise ValueError( "Values were told. Values cannot be specified when state is " "TrialState.PRUNED or TrialState.FAIL." ) elif state is not None: raise ValueError(f"Cannot tell with state {state}.") def _check_values_are_feasible(study: Study, values: Sequence[float]) -> Optional[str]: for v in values: # TODO(Imamura): Construct error message taking into account all values and do not early # return `value` is assumed to be ignored on failure so we can set it to any value. try: float(v) except (ValueError, TypeError): return f"The value {repr(v)} could not be cast to float" if math.isnan(v): return f"The value {v} is not acceptable" if len(study.directions) != len(values): return ( f"The number of the values {len(values)} did not match the number of the objectives " f"{len(study.directions)}" ) return None def _tell_with_warning( study: Study, trial: Union[Trial, int], value_or_values: Optional[Union[float, Sequence[float]]] = None, state: Optional[TrialState] = None, skip_if_finished: bool = False, suppress_warning: bool = False, ) -> FrozenTrial: """Internal method of :func:`~optuna.study.Study.tell`. Refer to the document for :func:`~optuna.study.Study.tell` for the reference. This method has one additional parameter ``suppress_warning``. Args: suppress_warning: If :obj:`True`, tell will not show warnings when tell receives an invalid values. This flag is expected to be :obj:`True` only when it is invoked by Study.optimize. """ # We must invalidate all trials cache here as it is only valid within a trial. study._thread_local.cached_all_trials = None # Validate the trial argument. frozen_trial = _get_frozen_trial(study, trial) if frozen_trial.state.is_finished() and skip_if_finished: _logger.info( f"Skipped telling trial {frozen_trial.number} with values " f"{value_or_values} and state {state} since trial was already finished. " f"Finished trial has values {frozen_trial.values} and state {frozen_trial.state}." ) return copy.deepcopy(frozen_trial) elif frozen_trial.state != TrialState.RUNNING: raise ValueError(f"Cannot tell a {frozen_trial.state.name} trial.") # Validate the state and values arguments. values: Optional[Sequence[float]] if value_or_values is None: values = None elif isinstance(value_or_values, Sequence): values = value_or_values else: values = [value_or_values] _check_state_and_values(state, values) warning_message = None if state == TrialState.COMPLETE: assert values is not None values_conversion_failure_message = _check_values_are_feasible(study, values) if values_conversion_failure_message is not None: raise ValueError(values_conversion_failure_message) elif state == TrialState.PRUNED: # Register the last intermediate value if present as the value of the trial. # TODO(hvy): Whether a pruned trials should have an actual value can be discussed. assert values is None last_step = frozen_trial.last_step if last_step is not None: last_intermediate_value = frozen_trial.intermediate_values[last_step] # intermediate_values can be unacceptable value, i.e., NaN. if _check_values_are_feasible(study, [last_intermediate_value]) is None: values = [last_intermediate_value] elif state is None: if values is None: values_conversion_failure_message = "The value None could not be cast to float." else: values_conversion_failure_message = _check_values_are_feasible(study, values) if values_conversion_failure_message is None: state = TrialState.COMPLETE else: state = TrialState.FAIL values = None if not suppress_warning: warnings.warn(values_conversion_failure_message) else: warning_message = values_conversion_failure_message assert state is not None # Cast values to list of floats. if values is not None: # values have been checked to be castable to floats in _check_values_are_feasible. values = [float(value) for value in values] # Post-processing and storing the trial. try: # Sampler defined trial post-processing. study = pruners._filter_study(study, frozen_trial) study.sampler.after_trial(study, frozen_trial, state, values) finally: study._storage.set_trial_state_values(frozen_trial._trial_id, state, values) frozen_trial = copy.deepcopy(study._storage.get_trial(frozen_trial._trial_id)) if warning_message is not None: frozen_trial._system_attrs[STUDY_TELL_WARNING_KEY] = warning_message return frozen_trial optuna-4.1.0/optuna/study/study.py000066400000000000000000001607471471332314300172420ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Container from collections.abc import Iterable from collections.abc import Mapping import copy from numbers import Real import threading from typing import Any from typing import Callable from typing import cast from typing import Sequence from typing import TYPE_CHECKING from typing import Union import warnings import numpy as np import optuna from optuna import exceptions from optuna import logging from optuna import pruners from optuna import samplers from optuna import storages from optuna._convert_positional_args import convert_positional_args from optuna._deprecated import deprecated_func from optuna._experimental import experimental_func from optuna._imports import _LazyImport from optuna._typing import JSONSerializable from optuna.distributions import _convert_old_distribution_to_new_distribution from optuna.distributions import BaseDistribution from optuna.storages._heartbeat import is_heartbeat_enabled from optuna.study._constrained_optimization import _CONSTRAINTS_KEY from optuna.study._constrained_optimization import _get_feasible_trials from optuna.study._multi_objective import _get_pareto_front_trials from optuna.study._optimize import _optimize from optuna.study._study_direction import StudyDirection from optuna.study._study_summary import StudySummary # NOQA from optuna.study._tell import _tell_with_warning from optuna.trial import create_trial from optuna.trial import TrialState _dataframe = _LazyImport("optuna.study._dataframe") if TYPE_CHECKING: from optuna.study._dataframe import pd from optuna.trial import FrozenTrial from optuna.trial import Trial ObjectiveFuncType = Callable[["Trial"], Union[float, Sequence[float]]] _SYSTEM_ATTR_METRIC_NAMES = "study:metric_names" _logger = logging.get_logger(__name__) class _ThreadLocalStudyAttribute(threading.local): in_optimize_loop: bool = False cached_all_trials: list[FrozenTrial] | None = None class Study: """A study corresponds to an optimization task, i.e., a set of trials. This object provides interfaces to run a new :class:`~optuna.trial.Trial`, access trials' history, set/get user-defined attributes of the study itself. Note that the direct use of this constructor is not recommended. To create and load a study, please refer to the documentation of :func:`~optuna.study.create_study` and :func:`~optuna.study.load_study` respectively. """ def __init__( self, study_name: str, storage: str | storages.BaseStorage, sampler: "samplers.BaseSampler" | None = None, pruner: pruners.BasePruner | None = None, ) -> None: self.study_name = study_name storage = storages.get_storage(storage) study_id = storage.get_study_id_from_name(study_name) self._study_id = study_id self._storage = storage self._directions = storage.get_study_directions(study_id) self.sampler = sampler or samplers.TPESampler() self.pruner = pruner or pruners.MedianPruner() self._thread_local = _ThreadLocalStudyAttribute() self._stop_flag = False def __getstate__(self) -> dict[Any, Any]: state = self.__dict__.copy() del state["_thread_local"] return state def __setstate__(self, state: dict[Any, Any]) -> None: self.__dict__.update(state) self._thread_local = _ThreadLocalStudyAttribute() @property def best_params(self) -> dict[str, Any]: """Return parameters of the best trial in the study. .. note:: This feature can only be used for single-objective optimization. Returns: A dictionary containing parameters of the best trial. """ return self.best_trial.params @property def best_value(self) -> float: """Return the best objective value in the study. .. note:: This feature can only be used for single-objective optimization. Returns: A float representing the best objective value. """ best_value = self.best_trial.value assert best_value is not None return best_value @property def best_trial(self) -> FrozenTrial: """Return the best trial in the study. .. note:: This feature can only be used for single-objective optimization. If your study is multi-objective, use :attr:`~optuna.study.Study.best_trials` instead. Returns: A :class:`~optuna.trial.FrozenTrial` object of the best trial. .. seealso:: The :ref:`reuse_best_trial` tutorial provides a detailed example of how to use this method. """ if self._is_multi_objective(): raise RuntimeError( "A single best trial cannot be retrieved from a multi-objective study. Consider " "using Study.best_trials to retrieve a list containing the best trials." ) best_trial = self._storage.get_best_trial(self._study_id) # If the trial with the best value is infeasible, select the best trial from all feasible # trials. Note that the behavior is undefined when constrained optimization without the # violation value in the best-valued trial. constraints = best_trial.system_attrs.get(_CONSTRAINTS_KEY) if constraints is not None and any([x > 0.0 for x in constraints]): complete_trials = self.get_trials(deepcopy=False, states=[TrialState.COMPLETE]) feasible_trials = _get_feasible_trials(complete_trials) if len(feasible_trials) == 0: raise ValueError("No feasible trials are completed yet.") if self.direction == StudyDirection.MAXIMIZE: best_trial = max(feasible_trials, key=lambda t: cast(float, t.value)) else: best_trial = min(feasible_trials, key=lambda t: cast(float, t.value)) return copy.deepcopy(best_trial) @property def best_trials(self) -> list[FrozenTrial]: """Return trials located at the Pareto front in the study. A trial is located at the Pareto front if there are no trials that dominate the trial. It's called that a trial ``t0`` dominates another trial ``t1`` if ``all(v0 <= v1) for v0, v1 in zip(t0.values, t1.values)`` and ``any(v0 < v1) for v0, v1 in zip(t0.values, t1.values)`` are held. Returns: A list of :class:`~optuna.trial.FrozenTrial` objects. """ # Check whether the study is constrained optimization. trials = self.get_trials(deepcopy=False) is_constrained = any((_CONSTRAINTS_KEY in trial.system_attrs) for trial in trials) return _get_pareto_front_trials(self, consider_constraint=is_constrained) @property def direction(self) -> StudyDirection: """Return the direction of the study. .. note:: This feature can only be used for single-objective optimization. If your study is multi-objective, use :attr:`~optuna.study.Study.directions` instead. Returns: A :class:`~optuna.study.StudyDirection` object. """ if self._is_multi_objective(): raise RuntimeError( "A single direction cannot be retrieved from a multi-objective study. Consider " "using Study.directions to retrieve a list containing all directions." ) return self.directions[0] @property def directions(self) -> list[StudyDirection]: """Return the directions of the study. Returns: A list of :class:`~optuna.study.StudyDirection` objects. """ return self._directions @property def trials(self) -> list[FrozenTrial]: """Return all trials in the study. The returned trials are ordered by trial number. This is a short form of ``self.get_trials(deepcopy=True, states=None)``. Returns: A list of :class:`~optuna.trial.FrozenTrial` objects. .. seealso:: See :func:`~optuna.study.Study.get_trials` for related method. """ return self.get_trials(deepcopy=True, states=None) def get_trials( self, deepcopy: bool = True, states: Container[TrialState] | None = None, ) -> list[FrozenTrial]: """Return all trials in the study. The returned trials are ordered by trial number. .. seealso:: See :attr:`~optuna.study.Study.trials` for related property. Example: .. testcode:: import optuna def objective(trial): x = trial.suggest_float("x", -1, 1) return x**2 study = optuna.create_study() study.optimize(objective, n_trials=3) trials = study.get_trials() assert len(trials) == 3 Args: deepcopy: Flag to control whether to apply ``copy.deepcopy()`` to the trials. Note that if you set the flag to :obj:`False`, you shouldn't mutate any fields of the returned trial. Otherwise the internal state of the study may corrupt and unexpected behavior may happen. states: Trial states to filter on. If :obj:`None`, include all states. Returns: A list of :class:`~optuna.trial.FrozenTrial` objects. """ return self._get_trials(deepcopy, states, use_cache=False) def _get_trials( self, deepcopy: bool = True, states: Container[TrialState] | None = None, use_cache: bool = False, ) -> list[FrozenTrial]: if use_cache: if self._thread_local.cached_all_trials is None: self._thread_local.cached_all_trials = self._storage.get_all_trials( self._study_id, deepcopy=False ) trials = self._thread_local.cached_all_trials if states is not None: filtered_trials = [t for t in trials if t.state in states] else: filtered_trials = trials return copy.deepcopy(filtered_trials) if deepcopy else filtered_trials return self._storage.get_all_trials(self._study_id, deepcopy=deepcopy, states=states) @property def user_attrs(self) -> dict[str, Any]: """Return user attributes. .. seealso:: See :func:`~optuna.study.Study.set_user_attr` for related method. Example: .. testcode:: import optuna def objective(trial): x = trial.suggest_float("x", 0, 1) y = trial.suggest_float("y", 0, 1) return x**2 + y**2 study = optuna.create_study() study.set_user_attr("objective function", "quadratic function") study.set_user_attr("dimensions", 2) study.set_user_attr("contributors", ["Akiba", "Sano"]) assert study.user_attrs == { "objective function": "quadratic function", "dimensions": 2, "contributors": ["Akiba", "Sano"], } Returns: A dictionary containing all user attributes. """ return copy.deepcopy(self._storage.get_study_user_attrs(self._study_id)) @property @deprecated_func("3.1.0", "5.0.0") def system_attrs(self) -> dict[str, Any]: """Return system attributes. Returns: A dictionary containing all system attributes. """ return copy.deepcopy(self._storage.get_study_system_attrs(self._study_id)) @property def metric_names(self) -> list[str] | None: """Return metric names. .. note:: Use :meth:`~optuna.study.Study.set_metric_names` to set the metric names first. Returns: A list with names for each dimension of the returned values of the objective function. """ return self._storage.get_study_system_attrs(self._study_id).get(_SYSTEM_ATTR_METRIC_NAMES) def optimize( self, func: ObjectiveFuncType, n_trials: int | None = None, timeout: float | None = None, n_jobs: int = 1, catch: Iterable[type[Exception]] | type[Exception] = (), callbacks: Iterable[Callable[[Study, FrozenTrial], None]] | None = None, gc_after_trial: bool = False, show_progress_bar: bool = False, ) -> None: """Optimize an objective function. Optimization is done by choosing a suitable set of hyperparameter values from a given range. Uses a sampler which implements the task of value suggestion based on a specified distribution. The sampler is specified in :func:`~optuna.study.create_study` and the default choice for the sampler is TPE. See also :class:`~optuna.samplers.TPESampler` for more details on 'TPE'. Optimization will be stopped when receiving a termination signal such as SIGINT and SIGTERM. Unlike other signals, a trial is automatically and cleanly failed when receiving SIGINT (Ctrl+C). If ``n_jobs`` is greater than one or if another signal than SIGINT is used, the interrupted trial state won't be properly updated. Example: .. testcode:: import optuna def objective(trial): x = trial.suggest_float("x", -1, 1) return x**2 study = optuna.create_study() study.optimize(objective, n_trials=3) Args: func: A callable that implements objective function. n_trials: The number of trials for each process. :obj:`None` represents no limit in terms of the number of trials. The study continues to create trials until the number of trials reaches ``n_trials``, ``timeout`` period elapses, :func:`~optuna.study.Study.stop` is called, or a termination signal such as SIGTERM or Ctrl+C is received. .. seealso:: :class:`optuna.study.MaxTrialsCallback` can ensure how many times trials will be performed across all processes. timeout: Stop study after the given number of second(s). :obj:`None` represents no limit in terms of elapsed time. The study continues to create trials until the number of trials reaches ``n_trials``, ``timeout`` period elapses, :func:`~optuna.study.Study.stop` is called or, a termination signal such as SIGTERM or Ctrl+C is received. n_jobs: The number of parallel jobs. If this argument is set to ``-1``, the number is set to CPU count. .. note:: ``n_jobs`` allows parallelization using :obj:`threading` and may suffer from `Python's GIL `__. It is recommended to use :ref:`process-based parallelization` if ``func`` is CPU bound. catch: A study continues to run even when a trial raises one of the exceptions specified in this argument. Default is an empty tuple, i.e. the study will stop for any exception except for :class:`~optuna.exceptions.TrialPruned`. callbacks: List of callback functions that are invoked at the end of each trial. Each function must accept two parameters with the following types in this order: :class:`~optuna.study.Study` and :class:`~optuna.trial.FrozenTrial`. .. seealso:: See the tutorial of :ref:`optuna_callback` for how to use and implement callback functions. gc_after_trial: Flag to determine whether to automatically run garbage collection after each trial. Set to :obj:`True` to run the garbage collection, :obj:`False` otherwise. When it runs, it runs a full collection by internally calling :func:`gc.collect`. If you see an increase in memory consumption over several trials, try setting this flag to :obj:`True`. .. seealso:: :ref:`out-of-memory-gc-collect` show_progress_bar: Flag to show progress bars or not. To show progress bar, set this :obj:`True`. Note that it is disabled when ``n_trials`` is :obj:`None`, ``timeout`` is not :obj:`None`, and ``n_jobs`` :math:`\\ne 1`. Raises: RuntimeError: If nested invocation of this method occurs. """ _optimize( study=self, func=func, n_trials=n_trials, timeout=timeout, n_jobs=n_jobs, catch=tuple(catch) if isinstance(catch, Iterable) else (catch,), callbacks=callbacks, gc_after_trial=gc_after_trial, show_progress_bar=show_progress_bar, ) def ask(self, fixed_distributions: dict[str, BaseDistribution] | None = None) -> Trial: """Create a new trial from which hyperparameters can be suggested. This method is part of an alternative to :func:`~optuna.study.Study.optimize` that allows controlling the lifetime of a trial outside the scope of ``func``. Each call to this method should be followed by a call to :func:`~optuna.study.Study.tell` to finish the created trial. .. seealso:: The :ref:`ask_and_tell` tutorial provides use-cases with examples. Example: Getting the trial object with the :func:`~optuna.study.Study.ask` method. .. testcode:: import optuna study = optuna.create_study() trial = study.ask() x = trial.suggest_float("x", -1, 1) study.tell(trial, x**2) Example: Passing previously defined distributions to the :func:`~optuna.study.Study.ask` method. .. testcode:: import optuna study = optuna.create_study() distributions = { "optimizer": optuna.distributions.CategoricalDistribution(["adam", "sgd"]), "lr": optuna.distributions.FloatDistribution(0.0001, 0.1, log=True), } # You can pass the distributions previously defined. trial = study.ask(fixed_distributions=distributions) # `optimizer` and `lr` are already suggested and accessible with `trial.params`. assert "optimizer" in trial.params assert "lr" in trial.params Args: fixed_distributions: A dictionary containing the parameter names and parameter's distributions. Each parameter in this dictionary is automatically suggested for the returned trial, even when the suggest method is not explicitly invoked by the user. If this argument is set to :obj:`None`, no parameter is automatically suggested. Returns: A :class:`~optuna.trial.Trial`. """ if not self._thread_local.in_optimize_loop and is_heartbeat_enabled(self._storage): warnings.warn("Heartbeat of storage is supposed to be used with Study.optimize.") fixed_distributions = fixed_distributions or {} fixed_distributions = { key: _convert_old_distribution_to_new_distribution(dist) for key, dist in fixed_distributions.items() } # Sync storage once every trial. self._thread_local.cached_all_trials = None trial_id = self._pop_waiting_trial_id() if trial_id is None: trial_id = self._storage.create_new_trial(self._study_id) trial = optuna.Trial(self, trial_id) for name, param in fixed_distributions.items(): trial._suggest(name, param) return trial def tell( self, trial: Trial | int, values: float | Sequence[float] | None = None, state: TrialState | None = None, skip_if_finished: bool = False, ) -> FrozenTrial: """Finish a trial created with :func:`~optuna.study.Study.ask`. .. seealso:: The :ref:`ask_and_tell` tutorial provides use-cases with examples. Example: .. testcode:: import optuna from optuna.trial import TrialState def f(x): return (x - 2) ** 2 def df(x): return 2 * x - 4 study = optuna.create_study() n_trials = 30 for _ in range(n_trials): trial = study.ask() lr = trial.suggest_float("lr", 1e-5, 1e-1, log=True) # Iterative gradient descent objective function. x = 3 # Initial value. for step in range(128): y = f(x) trial.report(y, step=step) if trial.should_prune(): # Finish the trial with the pruned state. study.tell(trial, state=TrialState.PRUNED) break gy = df(x) x -= gy * lr else: # Finish the trial with the final value after all iterations. study.tell(trial, y) Args: trial: A :class:`~optuna.trial.Trial` object or a trial number. values: Optional objective value or a sequence of such values in case the study is used for multi-objective optimization. Argument must be provided if ``state`` is :class:`~optuna.trial.TrialState.COMPLETE` and should be :obj:`None` if ``state`` is :class:`~optuna.trial.TrialState.FAIL` or :class:`~optuna.trial.TrialState.PRUNED`. state: State to be reported. Must be :obj:`None`, :class:`~optuna.trial.TrialState.COMPLETE`, :class:`~optuna.trial.TrialState.FAIL` or :class:`~optuna.trial.TrialState.PRUNED`. If ``state`` is :obj:`None`, it will be updated to :class:`~optuna.trial.TrialState.COMPLETE` or :class:`~optuna.trial.TrialState.FAIL` depending on whether validation for ``values`` reported succeed or not. skip_if_finished: Flag to control whether exception should be raised when values for already finished trial are told. If :obj:`True`, tell is skipped without any error when the trial is already finished. Returns: A :class:`~optuna.trial.FrozenTrial` representing the resulting trial. A returned trial is deep copied thus user can modify it as needed. """ return _tell_with_warning( study=self, trial=trial, value_or_values=values, state=state, skip_if_finished=skip_if_finished, ) def set_user_attr(self, key: str, value: Any) -> None: """Set a user attribute to the study. .. seealso:: See :attr:`~optuna.study.Study.user_attrs` for related attribute. .. seealso:: See the recipe on :ref:`attributes`. Example: .. testcode:: import optuna def objective(trial): x = trial.suggest_float("x", 0, 1) y = trial.suggest_float("y", 0, 1) return x**2 + y**2 study = optuna.create_study() study.set_user_attr("objective function", "quadratic function") study.set_user_attr("dimensions", 2) study.set_user_attr("contributors", ["Akiba", "Sano"]) assert study.user_attrs == { "objective function": "quadratic function", "dimensions": 2, "contributors": ["Akiba", "Sano"], } Args: key: A key string of the attribute. value: A value of the attribute. The value should be JSON serializable. """ self._storage.set_study_user_attr(self._study_id, key, value) @deprecated_func("3.1.0", "5.0.0") def set_system_attr(self, key: str, value: Any) -> None: """Set a system attribute to the study. Note that Optuna internally uses this method to save system messages. Please use :func:`~optuna.study.Study.set_user_attr` to set users' attributes. Args: key: A key string of the attribute. value: A value of the attribute. The value should be JSON serializable. """ self._storage.set_study_system_attr(self._study_id, key, value) def trials_dataframe( self, attrs: tuple[str, ...] = ( "number", "value", "datetime_start", "datetime_complete", "duration", "params", "user_attrs", "system_attrs", "state", ), multi_index: bool = False, ) -> "pd.DataFrame": """Export trials as a pandas DataFrame_. The DataFrame_ provides various features to analyze studies. It is also useful to draw a histogram of objective values and to export trials as a CSV file. If there are no trials, an empty DataFrame_ is returned. Example: .. testcode:: import optuna import pandas def objective(trial): x = trial.suggest_float("x", -1, 1) return x**2 study = optuna.create_study() study.optimize(objective, n_trials=3) # Create a dataframe from the study. df = study.trials_dataframe() assert isinstance(df, pandas.DataFrame) assert df.shape[0] == 3 # n_trials. Args: attrs: Specifies field names of :class:`~optuna.trial.FrozenTrial` to include them to a DataFrame of trials. multi_index: Specifies whether the returned DataFrame_ employs MultiIndex_ or not. Columns that are hierarchical by nature such as ``(params, x)`` will be flattened to ``params_x`` when set to :obj:`False`. Returns: A pandas DataFrame_ of trials in the :class:`~optuna.study.Study`. .. _DataFrame: http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html .. _MultiIndex: https://pandas.pydata.org/pandas-docs/stable/advanced.html Note: If ``value`` is in ``attrs`` during multi-objective optimization, it is implicitly replaced with ``values``. Note: If :meth:`~optuna.study.Study.set_metric_names` is called, the ``value`` or ``values`` is implicitly replaced with the dictionary with the objective name as key and the objective value as value. """ return _dataframe._trials_dataframe(self, attrs, multi_index) def stop(self) -> None: """Exit from the current optimization loop after the running trials finish. This method lets the running :meth:`~optuna.study.Study.optimize` method return immediately after all trials which the :meth:`~optuna.study.Study.optimize` method spawned finishes. This method does not affect any behaviors of parallel or successive study processes. This method only works when it is called inside an objective function or callback. Example: .. testcode:: import optuna def objective(trial): if trial.number == 4: trial.study.stop() x = trial.suggest_float("x", 0, 10) return x**2 study = optuna.create_study() study.optimize(objective, n_trials=10) assert len(study.trials) == 5 """ if not self._thread_local.in_optimize_loop: raise RuntimeError( "`Study.stop` is supposed to be invoked inside an objective function or a " "callback." ) self._stop_flag = True def enqueue_trial( self, params: dict[str, Any], user_attrs: dict[str, Any] | None = None, skip_if_exists: bool = False, ) -> None: """Enqueue a trial with given parameter values. You can fix the next sampling parameters which will be evaluated in your objective function. Example: .. testcode:: import optuna def objective(trial): x = trial.suggest_float("x", 0, 10) return x**2 study = optuna.create_study() study.enqueue_trial({"x": 5}) study.enqueue_trial({"x": 0}, user_attrs={"memo": "optimal"}) study.optimize(objective, n_trials=2) assert study.trials[0].params == {"x": 5} assert study.trials[1].params == {"x": 0} assert study.trials[1].user_attrs == {"memo": "optimal"} Args: params: Parameter values to pass your objective function. user_attrs: A dictionary of user-specific attributes other than ``params``. skip_if_exists: When :obj:`True`, prevents duplicate trials from being enqueued again. .. note:: This method might produce duplicated trials if called simultaneously by multiple processes at the same time with same ``params`` dict. .. seealso:: Please refer to :ref:`enqueue_trial_tutorial` for the tutorial of specifying hyperparameters manually. """ if not isinstance(params, dict): raise TypeError("params must be a dictionary.") if skip_if_exists and self._should_skip_enqueue(params): _logger.info(f"Trial with params {params} already exists. Skipping enqueue.") return self.add_trial( create_trial( state=TrialState.WAITING, system_attrs={"fixed_params": params}, user_attrs=user_attrs, ) ) def add_trial(self, trial: FrozenTrial) -> None: """Add trial to study. The trial is validated before being added. Example: .. testcode:: import optuna from optuna.distributions import FloatDistribution def objective(trial): x = trial.suggest_float("x", 0, 10) return x**2 study = optuna.create_study() assert len(study.trials) == 0 trial = optuna.trial.create_trial( params={"x": 2.0}, distributions={"x": FloatDistribution(0, 10)}, value=4.0, ) study.add_trial(trial) assert len(study.trials) == 1 study.optimize(objective, n_trials=3) assert len(study.trials) == 4 other_study = optuna.create_study() for trial in study.trials: other_study.add_trial(trial) assert len(other_study.trials) == len(study.trials) other_study.optimize(objective, n_trials=2) assert len(other_study.trials) == len(study.trials) + 2 .. seealso:: This method should in general be used to add already evaluated trials (``trial.state.is_finished() == True``). To queue trials for evaluation, please refer to :func:`~optuna.study.Study.enqueue_trial`. .. seealso:: See :func:`~optuna.trial.create_trial` for how to create trials. .. seealso:: Please refer to :ref:`add_trial_tutorial` for the tutorial of specifying hyperparameters with the evaluated value manually. Args: trial: Trial to add. """ trial._validate() if trial.values is not None and len(self.directions) != len(trial.values): raise ValueError( f"The added trial has {len(trial.values)} values, which is different from the " f"number of objectives {len(self.directions)} in the study (determined by " "Study.directions)." ) self._storage.create_new_trial(self._study_id, template_trial=trial) def add_trials(self, trials: Iterable[FrozenTrial]) -> None: """Add trials to study. The trials are validated before being added. Example: .. testcode:: import optuna def objective(trial): x = trial.suggest_float("x", 0, 10) return x**2 study = optuna.create_study() study.optimize(objective, n_trials=3) assert len(study.trials) == 3 other_study = optuna.create_study() other_study.add_trials(study.trials) assert len(other_study.trials) == len(study.trials) other_study.optimize(objective, n_trials=2) assert len(other_study.trials) == len(study.trials) + 2 .. seealso:: See :func:`~optuna.study.Study.add_trial` for addition of each trial. Args: trials: Trials to add. """ for trial in trials: self.add_trial(trial) @experimental_func("3.2.0") def set_metric_names(self, metric_names: list[str]) -> None: """Set metric names. This method names each dimension of the returned values of the objective function. It is particularly useful in multi-objective optimization. The metric names are mainly referenced by the visualization functions. Example: .. testcode:: import optuna import pandas def objective(trial): x = trial.suggest_float("x", 0, 10) return x**2, x + 1 study = optuna.create_study(directions=["minimize", "minimize"]) study.set_metric_names(["x**2", "x+1"]) study.optimize(objective, n_trials=3) df = study.trials_dataframe(multi_index=True) assert isinstance(df, pandas.DataFrame) assert list(df.get("values").keys()) == ["x**2", "x+1"] .. seealso:: The names set by this method are used in :meth:`~optuna.study.Study.trials_dataframe` and :func:`~optuna.visualization.plot_pareto_front`. Args: metric_names: A list of metric names for the objective function. """ if len(self.directions) != len(metric_names): raise ValueError("The number of objectives must match the length of the metric names.") self._storage.set_study_system_attr( self._study_id, _SYSTEM_ATTR_METRIC_NAMES, metric_names ) def _is_multi_objective(self) -> bool: """Return :obj:`True` if the study has multiple objectives. Returns: A boolean value indicates if `self.directions` has more than 1 element or not. """ return len(self.directions) > 1 def _pop_waiting_trial_id(self) -> int | None: for trial in self._storage.get_all_trials( self._study_id, deepcopy=False, states=(TrialState.WAITING,) ): if not self._storage.set_trial_state_values(trial._trial_id, state=TrialState.RUNNING): continue _logger.debug("Trial {} popped from the trial queue.".format(trial.number)) return trial._trial_id return None def _should_skip_enqueue(self, params: Mapping[str, JSONSerializable]) -> bool: for trial in self.get_trials(deepcopy=False): trial_params = trial.system_attrs.get("fixed_params", trial.params) if trial_params.keys() != params.keys(): # Can't have repeated trials if different params are suggested. continue repeated_params: list[bool] = [] for param_name, param_value in params.items(): existing_param = trial_params[param_name] if not isinstance(param_value, type(existing_param)): # Enqueued param has distribution that does not match existing param # (e.g. trying to enqueue categorical to float param). # We are not doing anything about it here, since sanitization should # be handled regardless if `skip_if_exists` is `True`. repeated_params.append(False) continue is_repeated = ( np.isnan(float(param_value)) or np.isclose(float(param_value), float(existing_param), atol=0.0) if isinstance(param_value, Real) else param_value == existing_param ) repeated_params.append(bool(is_repeated)) if all(repeated_params): return True return False def _log_completed_trial(self, trial: FrozenTrial) -> None: if not _logger.isEnabledFor(logging.INFO): return metric_names = self.metric_names if len(trial.values) > 1: trial_values: list[float] | dict[str, float] if metric_names is None: trial_values = trial.values else: trial_values = {name: value for name, value in zip(metric_names, trial.values)} _logger.info( "Trial {} finished with values: {} and parameters: {}.".format( trial.number, trial_values, trial.params ) ) elif len(trial.values) == 1: trial_value: float | dict[str, float] if metric_names is None: trial_value = trial.values[0] else: trial_value = {metric_names[0]: trial.values[0]} message = ( f"Trial {trial.number} finished with value: {trial_value} and parameters: " f"{trial.params}." ) try: best_trial = self.best_trial message += f" Best is trial {best_trial.number} with value: {best_trial.value}." except ValueError: # If no feasible trials are completed yet, study.best_trial raises ValueError. pass _logger.info(message) else: assert False, "Should not reach." @convert_positional_args( previous_positional_arg_names=[ "storage", "sampler", "pruner", "study_name", "direction", "load_if_exists", ], ) def create_study( *, storage: str | storages.BaseStorage | None = None, sampler: "samplers.BaseSampler" | None = None, pruner: pruners.BasePruner | None = None, study_name: str | None = None, direction: str | StudyDirection | None = None, load_if_exists: bool = False, directions: Sequence[str | StudyDirection] | None = None, ) -> Study: """Create a new :class:`~optuna.study.Study`. Example: .. testcode:: import optuna def objective(trial): x = trial.suggest_float("x", 0, 10) return x**2 study = optuna.create_study() study.optimize(objective, n_trials=3) Args: storage: Database URL. If this argument is set to None, :class:`~optuna.storages.InMemoryStorage` is used, and the :class:`~optuna.study.Study` will not be persistent. .. note:: When a database URL is passed, Optuna internally uses `SQLAlchemy`_ to handle the database. Please refer to `SQLAlchemy's document`_ for further details. If you want to specify non-default options to `SQLAlchemy Engine`_, you can instantiate :class:`~optuna.storages.RDBStorage` with your desired options and pass it to the ``storage`` argument instead of a URL. .. _SQLAlchemy: https://www.sqlalchemy.org/ .. _SQLAlchemy's document: https://docs.sqlalchemy.org/en/latest/core/engines.html#database-urls .. _SQLAlchemy Engine: https://docs.sqlalchemy.org/en/latest/core/engines.html sampler: A sampler object that implements background algorithm for value suggestion. If :obj:`None` is specified, :class:`~optuna.samplers.TPESampler` is used during single-objective optimization and :class:`~optuna.samplers.NSGAIISampler` during multi-objective optimization. See also :class:`~optuna.samplers`. pruner: A pruner object that decides early stopping of unpromising trials. If :obj:`None` is specified, :class:`~optuna.pruners.MedianPruner` is used as the default. See also :class:`~optuna.pruners`. study_name: Study's name. If this argument is set to None, a unique name is generated automatically. direction: Direction of optimization. Set ``minimize`` for minimization and ``maximize`` for maximization. You can also pass the corresponding :class:`~optuna.study.StudyDirection` object. ``direction`` and ``directions`` must not be specified at the same time. .. note:: If none of `direction` and `directions` are specified, the direction of the study is set to "minimize". load_if_exists: Flag to control the behavior to handle a conflict of study names. In the case where a study named ``study_name`` already exists in the ``storage``, a :class:`~optuna.exceptions.DuplicatedStudyError` is raised if ``load_if_exists`` is set to :obj:`False`. Otherwise, the creation of the study is skipped, and the existing one is returned. directions: A sequence of directions during multi-objective optimization. ``direction`` and ``directions`` must not be specified at the same time. Returns: A :class:`~optuna.study.Study` object. See also: :func:`optuna.create_study` is an alias of :func:`optuna.study.create_study`. See also: The :ref:`rdb` tutorial provides concrete examples to save and resume optimization using RDB. """ if direction is None and directions is None: directions = ["minimize"] elif direction is not None and directions is not None: raise ValueError("Specify only one of `direction` and `directions`.") elif direction is not None: directions = [direction] elif directions is not None: directions = list(directions) else: assert False if len(directions) < 1: raise ValueError("The number of objectives must be greater than 0.") elif any( d not in ["minimize", "maximize", StudyDirection.MINIMIZE, StudyDirection.MAXIMIZE] for d in directions ): raise ValueError( "Please set either 'minimize' or 'maximize' to direction. You can also set the " "corresponding `StudyDirection` member." ) direction_objects = [ d if isinstance(d, StudyDirection) else StudyDirection[d.upper()] for d in directions ] storage = storages.get_storage(storage) try: study_id = storage.create_new_study(direction_objects, study_name) except exceptions.DuplicatedStudyError: if load_if_exists: assert study_name is not None _logger.info( "Using an existing study with name '{}' instead of " "creating a new one.".format(study_name) ) study_id = storage.get_study_id_from_name(study_name) else: raise if sampler is None and len(direction_objects) > 1: sampler = samplers.NSGAIISampler() study_name = storage.get_study_name_from_id(study_id) study = Study(study_name=study_name, storage=storage, sampler=sampler, pruner=pruner) return study @convert_positional_args( previous_positional_arg_names=[ "study_name", "storage", "sampler", "pruner", ], ) def load_study( *, study_name: str | None, storage: str | storages.BaseStorage, sampler: "samplers.BaseSampler" | None = None, pruner: pruners.BasePruner | None = None, ) -> Study: """Load the existing :class:`~optuna.study.Study` that has the specified name. Example: .. testsetup:: import os if os.path.exists("example.db"): raise RuntimeError("'example.db' already exists. Please remove it.") .. testcode:: import optuna def objective(trial): x = trial.suggest_float("x", 0, 10) return x**2 study = optuna.create_study(storage="sqlite:///example.db", study_name="my_study") study.optimize(objective, n_trials=3) loaded_study = optuna.load_study(study_name="my_study", storage="sqlite:///example.db") assert len(loaded_study.trials) == len(study.trials) .. testcleanup:: os.remove("example.db") Args: study_name: Study's name. Each study has a unique name as an identifier. If :obj:`None`, checks whether the storage contains a single study, and if so loads that study. ``study_name`` is required if there are multiple studies in the storage. storage: Database URL such as ``sqlite:///example.db``. Please see also the documentation of :func:`~optuna.study.create_study` for further details. sampler: A sampler object that implements background algorithm for value suggestion. If :obj:`None` is specified, :class:`~optuna.samplers.TPESampler` is used as the default. See also :class:`~optuna.samplers`. pruner: A pruner object that decides early stopping of unpromising trials. If :obj:`None` is specified, :class:`~optuna.pruners.MedianPruner` is used as the default. See also :class:`~optuna.pruners`. Returns: A :class:`~optuna.study.Study` object. See also: :func:`optuna.load_study` is an alias of :func:`optuna.study.load_study`. """ if study_name is None: study_names = get_all_study_names(storage) if len(study_names) != 1: raise ValueError( f"Could not determine the study name since the storage {storage} does not " "contain exactly 1 study. Specify `study_name`." ) study_name = study_names[0] _logger.info( f"Study name was omitted but trying to load '{study_name}' because that was the only " "study found in the storage." ) return Study(study_name=study_name, storage=storage, sampler=sampler, pruner=pruner) @convert_positional_args( previous_positional_arg_names=[ "study_name", "storage", ], ) def delete_study( *, study_name: str, storage: str | storages.BaseStorage, ) -> None: """Delete a :class:`~optuna.study.Study` object. Example: .. testsetup:: import os if os.path.exists("example.db"): raise RuntimeError("'example.db' already exists. Please remove it.") .. testcode:: import optuna def objective(trial): x = trial.suggest_float("x", -10, 10) return (x - 2) ** 2 study = optuna.create_study(study_name="example-study", storage="sqlite:///example.db") study.optimize(objective, n_trials=3) optuna.delete_study(study_name="example-study", storage="sqlite:///example.db") .. testcleanup:: os.remove("example.db") Args: study_name: Study's name. storage: Database URL such as ``sqlite:///example.db``. Please see also the documentation of :func:`~optuna.study.create_study` for further details. See also: :func:`optuna.delete_study` is an alias of :func:`optuna.study.delete_study`. """ storage = storages.get_storage(storage) study_id = storage.get_study_id_from_name(study_name) storage.delete_study(study_id) @convert_positional_args( previous_positional_arg_names=[ "from_study_name", "from_storage", "to_storage", "to_study_name", ], warning_stacklevel=3, ) def copy_study( *, from_study_name: str, from_storage: str | storages.BaseStorage, to_storage: str | storages.BaseStorage, to_study_name: str | None = None, ) -> None: """Copy study from one storage to another. The direction(s) of the objective(s) in the study, trials, user attributes and system attributes are copied. .. note:: :func:`~optuna.copy_study` copies a study even if the optimization is working on. It means users will get a copied study that contains a trial that is not finished. Example: .. testsetup:: import os if os.path.exists("example.db"): raise RuntimeError("'example.db' already exists. Please remove it.") if os.path.exists("example_copy.db"): raise RuntimeError("'example_copy.db' already exists. Please remove it.") .. testcode:: import optuna def objective(trial): x = trial.suggest_float("x", -10, 10) return (x - 2) ** 2 study = optuna.create_study( study_name="example-study", storage="sqlite:///example.db", ) study.optimize(objective, n_trials=3) optuna.copy_study( from_study_name="example-study", from_storage="sqlite:///example.db", to_storage="sqlite:///example_copy.db", ) study = optuna.load_study( study_name=None, storage="sqlite:///example_copy.db", ) .. testcleanup:: os.remove("example.db") os.remove("example_copy.db") Args: from_study_name: Name of study. from_storage: Source database URL such as ``sqlite:///example.db``. Please see also the documentation of :func:`~optuna.study.create_study` for further details. to_storage: Destination database URL. to_study_name: Name of the created study. If omitted, ``from_study_name`` is used. Raises: :class:`~optuna.exceptions.DuplicatedStudyError`: If a study with a conflicting name already exists in the destination storage. """ from_study = load_study(study_name=from_study_name, storage=from_storage) to_study = create_study( study_name=to_study_name or from_study_name, storage=to_storage, directions=from_study.directions, load_if_exists=False, ) for key, value in from_study._storage.get_study_system_attrs(from_study._study_id).items(): to_study._storage.set_study_system_attr(to_study._study_id, key, value) for key, value in from_study.user_attrs.items(): to_study.set_user_attr(key, value) # Trials are deep copied on `add_trials`. to_study.add_trials(from_study.get_trials(deepcopy=False)) def get_all_study_summaries( storage: str | storages.BaseStorage, include_best_trial: bool = True ) -> list[StudySummary]: """Get all history of studies stored in a specified storage. Example: .. testsetup:: import os if os.path.exists("example.db"): raise RuntimeError("'example.db' already exists. Please remove it.") .. testcode:: import optuna def objective(trial): x = trial.suggest_float("x", -10, 10) return (x - 2) ** 2 study = optuna.create_study(study_name="example-study", storage="sqlite:///example.db") study.optimize(objective, n_trials=3) study_summaries = optuna.study.get_all_study_summaries(storage="sqlite:///example.db") assert len(study_summaries) == 1 study_summary = study_summaries[0] assert study_summary.study_name == "example-study" .. testcleanup:: os.remove("example.db") Args: storage: Database URL such as ``sqlite:///example.db``. Please see also the documentation of :func:`~optuna.study.create_study` for further details. include_best_trial: Include the best trials if exist. It potentially increases the number of queries and may take longer to fetch summaries depending on the storage. Returns: List of study history summarized as :class:`~optuna.study.StudySummary` objects. See also: :func:`optuna.get_all_study_summaries` is an alias of :func:`optuna.study.get_all_study_summaries`. """ storage = storages.get_storage(storage) frozen_studies = storage.get_all_studies() study_summaries = [] for s in frozen_studies: all_trials = storage.get_all_trials(s._study_id) completed_trials = [t for t in all_trials if t.state == TrialState.COMPLETE] n_trials = len(all_trials) if len(s.directions) == 1: direction = s.direction directions = None if include_best_trial and len(completed_trials) != 0: if direction == StudyDirection.MAXIMIZE: best_trial = max(completed_trials, key=lambda t: cast(float, t.value)) else: best_trial = min(completed_trials, key=lambda t: cast(float, t.value)) else: best_trial = None else: direction = None directions = s.directions best_trial = None datetime_start = min( [t.datetime_start for t in all_trials if t.datetime_start is not None], default=None ) study_summaries.append( StudySummary( study_name=s.study_name, direction=direction, best_trial=best_trial, user_attrs=s.user_attrs, system_attrs=s.system_attrs, n_trials=n_trials, datetime_start=datetime_start, study_id=s._study_id, directions=directions, ) ) return study_summaries def get_all_study_names(storage: str | storages.BaseStorage) -> list[str]: """Get all study names stored in a specified storage. Example: .. testsetup:: import os if os.path.exists("example.db"): raise RuntimeError("'example.db' already exists. Please remove it.") .. testcode:: import optuna def objective(trial): x = trial.suggest_float("x", -10, 10) return (x - 2) ** 2 study = optuna.create_study(study_name="example-study", storage="sqlite:///example.db") study.optimize(objective, n_trials=3) study_names = optuna.study.get_all_study_names(storage="sqlite:///example.db") assert len(study_names) == 1 assert study_names[0] == "example-study" .. testcleanup:: os.remove("example.db") Args: storage: Database URL such as ``sqlite:///example.db``. Please see also the documentation of :func:`~optuna.study.create_study` for further details. Returns: List of all study names in the storage. See also: :func:`optuna.get_all_study_names` is an alias of :func:`optuna.study.get_all_study_names`. """ storage = storages.get_storage(storage) study_names = [study.study_name for study in storage.get_all_studies()] return study_names optuna-4.1.0/optuna/terminator/000077500000000000000000000000001471332314300165155ustar00rootroot00000000000000optuna-4.1.0/optuna/terminator/__init__.py000066400000000000000000000021671471332314300206340ustar00rootroot00000000000000from optuna.terminator.callback import TerminatorCallback from optuna.terminator.erroreval import BaseErrorEvaluator from optuna.terminator.erroreval import CrossValidationErrorEvaluator from optuna.terminator.erroreval import report_cross_validation_scores from optuna.terminator.erroreval import StaticErrorEvaluator from optuna.terminator.improvement.emmr import EMMREvaluator from optuna.terminator.improvement.evaluator import BaseImprovementEvaluator from optuna.terminator.improvement.evaluator import BestValueStagnationEvaluator from optuna.terminator.improvement.evaluator import RegretBoundEvaluator from optuna.terminator.median_erroreval import MedianErrorEvaluator from optuna.terminator.terminator import BaseTerminator from optuna.terminator.terminator import Terminator __all__ = [ "TerminatorCallback", "BaseErrorEvaluator", "CrossValidationErrorEvaluator", "report_cross_validation_scores", "StaticErrorEvaluator", "MedianErrorEvaluator", "BaseImprovementEvaluator", "BestValueStagnationEvaluator", "RegretBoundEvaluator", "EMMREvaluator", "BaseTerminator", "Terminator", ] optuna-4.1.0/optuna/terminator/callback.py000066400000000000000000000052401471332314300206240ustar00rootroot00000000000000from __future__ import annotations from optuna._experimental import experimental_class from optuna.logging import get_logger from optuna.study.study import Study from optuna.terminator.terminator import BaseTerminator from optuna.terminator.terminator import Terminator from optuna.trial import FrozenTrial _logger = get_logger(__name__) @experimental_class("3.2.0") class TerminatorCallback: """A callback that terminates the optimization using Terminator. This class implements a callback which wraps :class:`~optuna.terminator.Terminator` so that it can be used with the :func:`~optuna.study.Study.optimize` method. Args: terminator: A terminator object which determines whether to terminate the optimization by assessing the room for optimization and statistical error. Defaults to a :class:`~optuna.terminator.Terminator` object with default ``improvement_evaluator`` and ``error_evaluator``. Example: .. testcode:: from sklearn.datasets import load_wine from sklearn.ensemble import RandomForestClassifier from sklearn.model_selection import cross_val_score from sklearn.model_selection import KFold import optuna from optuna.terminator import TerminatorCallback from optuna.terminator import report_cross_validation_scores def objective(trial): X, y = load_wine(return_X_y=True) clf = RandomForestClassifier( max_depth=trial.suggest_int("max_depth", 2, 32), min_samples_split=trial.suggest_float("min_samples_split", 0, 1), criterion=trial.suggest_categorical("criterion", ("gini", "entropy")), ) scores = cross_val_score(clf, X, y, cv=KFold(n_splits=5, shuffle=True)) report_cross_validation_scores(trial, scores) return scores.mean() study = optuna.create_study(direction="maximize") terminator = TerminatorCallback() study.optimize(objective, n_trials=50, callbacks=[terminator]) .. seealso:: Please refer to :class:`~optuna.terminator.Terminator` for the details of the terminator mechanism. """ def __init__(self, terminator: BaseTerminator | None = None) -> None: self._terminator = terminator or Terminator() def __call__(self, study: Study, trial: FrozenTrial) -> None: should_terminate = self._terminator.should_terminate(study=study) if should_terminate: _logger.info("The study has been stopped by the terminator.") study.stop() optuna-4.1.0/optuna/terminator/erroreval.py000066400000000000000000000100611471332314300210660ustar00rootroot00000000000000from __future__ import annotations import abc from typing import cast import numpy as np from optuna._experimental import experimental_class from optuna.study import StudyDirection from optuna.trial import FrozenTrial from optuna.trial import Trial from optuna.trial._state import TrialState _CROSS_VALIDATION_SCORES_KEY = "terminator:cv_scores" class BaseErrorEvaluator(metaclass=abc.ABCMeta): """Base class for error evaluators.""" @abc.abstractmethod def evaluate( self, trials: list[FrozenTrial], study_direction: StudyDirection, ) -> float: pass @experimental_class("3.2.0") class CrossValidationErrorEvaluator(BaseErrorEvaluator): """An error evaluator for objective functions based on cross-validation. This evaluator evaluates the objective function's statistical error, which comes from the randomness of dataset. This evaluator assumes that the objective function is the average of the cross-validation and uses the scaled variance of the cross-validation scores in the best trial at the moment as the statistical error. """ def evaluate( self, trials: list[FrozenTrial], study_direction: StudyDirection, ) -> float: """Evaluate the statistical error of the objective function based on cross-validation. Args: trials: A list of trials to consider. The best trial in ``trials`` is used to compute the statistical error. study_direction: The direction of the study. Returns: A float representing the statistical error of the objective function. """ trials = [trial for trial in trials if trial.state == TrialState.COMPLETE] assert len(trials) > 0 if study_direction == StudyDirection.MAXIMIZE: best_trial = max(trials, key=lambda t: cast(float, t.value)) else: best_trial = min(trials, key=lambda t: cast(float, t.value)) best_trial_attrs = best_trial.system_attrs if _CROSS_VALIDATION_SCORES_KEY in best_trial_attrs: cv_scores = best_trial_attrs[_CROSS_VALIDATION_SCORES_KEY] else: raise ValueError( "Cross-validation scores have not been reported. Please call " "`report_cross_validation_scores(trial, scores)` during a trial and pass the " "list of scores as `scores`." ) k = len(cv_scores) assert k > 1, "Should be guaranteed by `report_cross_validation_scores`." scale = 1 / k + 1 / (k - 1) var = scale * np.var(cv_scores) std = np.sqrt(var) return float(std) @experimental_class("3.2.0") def report_cross_validation_scores(trial: Trial, scores: list[float]) -> None: """A function to report cross-validation scores of a trial. This function should be called within the objective function to report the cross-validation scores. The reported scores are used to evaluate the statistical error for termination judgement. Args: trial: A :class:`~optuna.trial.Trial` object to report the cross-validation scores. scores: The cross-validation scores of the trial. """ if len(scores) <= 1: raise ValueError("The length of `scores` is expected to be greater than one.") trial.storage.set_trial_system_attr(trial._trial_id, _CROSS_VALIDATION_SCORES_KEY, scores) @experimental_class("3.2.0") class StaticErrorEvaluator(BaseErrorEvaluator): """An error evaluator that always returns a constant value. This evaluator can be used to terminate the optimization when the evaluated improvement potential is below the fixed threshold. Args: constant: A user-specified constant value to always return as an error estimate. """ def __init__(self, constant: float) -> None: self._constant = constant def evaluate( self, trials: list[FrozenTrial], study_direction: StudyDirection, ) -> float: return self._constant optuna-4.1.0/optuna/terminator/improvement/000077500000000000000000000000001471332314300210625ustar00rootroot00000000000000optuna-4.1.0/optuna/terminator/improvement/__init__.py000066400000000000000000000000001471332314300231610ustar00rootroot00000000000000optuna-4.1.0/optuna/terminator/improvement/emmr.py000066400000000000000000000353041471332314300224010ustar00rootroot00000000000000from __future__ import annotations import math import sys from typing import cast from typing import TYPE_CHECKING import warnings import numpy as np from optuna._experimental import experimental_class from optuna.samplers._lazy_random_state import LazyRandomState from optuna.search_space import intersection_search_space from optuna.study import StudyDirection from optuna.terminator.improvement.evaluator import _compute_standardized_regret_bound from optuna.terminator.improvement.evaluator import BaseImprovementEvaluator from optuna.trial import FrozenTrial from optuna.trial import TrialState if TYPE_CHECKING: import scipy.stats as scipy_stats import torch from optuna._gp import acqf from optuna._gp import gp from optuna._gp import prior from optuna._gp import search_space as gp_search_space else: from optuna._imports import _LazyImport torch = _LazyImport("torch") gp = _LazyImport("optuna._gp.gp") acqf = _LazyImport("optuna._gp.acqf") prior = _LazyImport("optuna._gp.prior") gp_search_space = _LazyImport("optuna._gp.search_space") scipy_stats = _LazyImport("scipy.stats") MARGIN_FOR_NUMARICAL_STABILITY = 0.1 @experimental_class("4.0.0") class EMMREvaluator(BaseImprovementEvaluator): """Evaluates a kind of regrets, called the Expected Minimum Model Regret(EMMR). EMMR is an upper bound of "expected minimum simple regret" in the optimization process. Expected minimum simple regret is a quantity that converges to zero only if the optimization process has found the global optima. For further information about expected minimum simple regret and the algorithm, please refer to the following paper: - `A stopping criterion for Bayesian optimization by the gap of expected minimum simple regrets `__ Also, there is our blog post explaining this evaluator: - `Introducing A New Terminator: Early Termination of Black-box Optimization Based on Expected Minimum Model Regret `__ Args: deterministic_objective: A boolean value which indicates whether the objective function is deterministic. Default is :obj:`False`. delta: A float number related to the criterion for termination. Default to 0.1. For further information about this parameter, please see the aforementioned paper. min_n_trials: A minimum number of complete trials to compute the criterion. Default to 2. seed: A random seed for EMMREvaluator. Example: .. testcode:: import optuna from optuna.terminator import EMMREvaluator from optuna.terminator import MedianErrorEvaluator from optuna.terminator import Terminator sampler = optuna.samplers.TPESampler(seed=0) study = optuna.create_study(sampler=sampler, direction="minimize") emmr_improvement_evaluator = EMMREvaluator() median_error_evaluator = MedianErrorEvaluator(emmr_improvement_evaluator) terminator = Terminator( improvement_evaluator=emmr_improvement_evaluator, error_evaluator=median_error_evaluator, ) for i in range(1000): trial = study.ask() ys = [trial.suggest_float(f"x{i}", -10.0, 10.0) for i in range(5)] value = sum(ys[i] ** 2 for i in range(5)) study.tell(trial, value) if terminator.should_terminate(study): # Terminated by Optuna Terminator! break """ def __init__( self, deterministic_objective: bool = False, delta: float = 0.1, min_n_trials: int = 2, seed: int | None = None, ) -> None: if min_n_trials <= 1 or not np.isfinite(min_n_trials): raise ValueError("`min_n_trials` is expected to be a finite integer more than one.") self._deterministic = deterministic_objective self._delta = delta self.min_n_trials = min_n_trials self._rng = LazyRandomState(seed) def evaluate(self, trials: list[FrozenTrial], study_direction: StudyDirection) -> float: optuna_search_space = intersection_search_space(trials) complete_trials = [t for t in trials if t.state == TrialState.COMPLETE] if len(complete_trials) < self.min_n_trials: return sys.float_info.max * MARGIN_FOR_NUMARICAL_STABILITY # Do not terminate. search_space, normalized_params = gp_search_space.get_search_space_and_normalized_params( complete_trials, optuna_search_space ) if len(search_space.scale_types) == 0: warnings.warn( f"{self.__class__.__name__} cannot consider any search space." "Termination will never occur in this study." ) return sys.float_info.max * MARGIN_FOR_NUMARICAL_STABILITY # Do not terminate. len_trials = len(complete_trials) len_params = len(search_space.scale_types) assert normalized_params.shape == (len_trials, len_params) # _gp module assumes that optimization direction is maximization sign = -1 if study_direction == StudyDirection.MINIMIZE else 1 score_vals = np.array([cast(float, t.value) for t in complete_trials]) * sign if np.any(~np.isfinite(score_vals)): warnings.warn( f"{self.__class__.__name__} cannot handle infinite values." "Those values are clamped to worst/best finite value." ) finite_score_vals = score_vals[np.isfinite(score_vals)] best_finite_score = np.max(finite_score_vals, initial=0.0) worst_finite_score = np.min(finite_score_vals, initial=0.0) score_vals = np.clip(score_vals, worst_finite_score, best_finite_score) standarized_score_vals = (score_vals - score_vals.mean()) / max( sys.float_info.min, score_vals.std() ) assert len(standarized_score_vals) == len(normalized_params) kernel_params_t1 = gp.fit_kernel_params( # Fit kernel with up to (t-1)-th observation X=normalized_params[..., :-1, :], Y=standarized_score_vals[:-1], is_categorical=(search_space.scale_types == gp_search_space.ScaleType.CATEGORICAL), log_prior=prior.default_log_prior, minimum_noise=prior.DEFAULT_MINIMUM_NOISE_VAR, initial_kernel_params=None, deterministic_objective=self._deterministic, ) kernel_params_t = gp.fit_kernel_params( # Fit kernel with up to t-th observation X=normalized_params, Y=standarized_score_vals, is_categorical=(search_space.scale_types == gp_search_space.ScaleType.CATEGORICAL), log_prior=prior.default_log_prior, minimum_noise=prior.DEFAULT_MINIMUM_NOISE_VAR, initial_kernel_params=kernel_params_t1, deterministic_objective=self._deterministic, ) theta_t_star_index = int(np.argmax(standarized_score_vals)) theta_t1_star_index = int(np.argmax(standarized_score_vals[:-1])) theta_t_star = normalized_params[theta_t_star_index, :] theta_t1_star = normalized_params[theta_t1_star_index, :] cov_t_between_theta_t_star_and_theta_t1_star = _compute_gp_posterior_cov_two_thetas( search_space, normalized_params, standarized_score_vals, kernel_params_t, theta_t_star_index, theta_t1_star_index, ) mu_t1_theta_t_with_nu_t, variance_t1_theta_t_with_nu_t = _compute_gp_posterior( search_space, normalized_params[:-1, :], standarized_score_vals[:-1], normalized_params[-1, :], kernel_params_t, # Use kernel_params_t instead of kernel_params_t1. # Use "t" under the assumption that "t" and "t1" are approximately the same. # This is because kernel should same when computing KLD. # For detailed information, please see section 4.4 of the paper: # https://proceedings.mlr.press/v206/ishibashi23a/ishibashi23a.pdf ) _, variance_t_theta_t1_star = _compute_gp_posterior( search_space, normalized_params, standarized_score_vals, theta_t1_star, kernel_params_t, ) mu_t_theta_t_star, variance_t_theta_t_star = _compute_gp_posterior( search_space, normalized_params, standarized_score_vals, theta_t_star, kernel_params_t, ) mu_t1_theta_t1_star, _ = _compute_gp_posterior( search_space, normalized_params[:-1, :], standarized_score_vals[:-1], theta_t1_star, kernel_params_t1, ) y_t = standarized_score_vals[-1] kappa_t1 = _compute_standardized_regret_bound( kernel_params_t1, search_space, normalized_params[:-1, :], standarized_score_vals[:-1], self._delta, rng=self._rng.rng, ) theorem1_delta_mu_t_star = mu_t1_theta_t1_star - mu_t_theta_t_star alg1_delta_r_tilde_t_term1 = theorem1_delta_mu_t_star theorem1_v = math.sqrt( max( 1e-10, variance_t_theta_t_star - 2.0 * cov_t_between_theta_t_star_and_theta_t1_star + variance_t_theta_t1_star, ) ) theorem1_g = (mu_t_theta_t_star - mu_t1_theta_t1_star) / theorem1_v alg1_delta_r_tilde_t_term2 = theorem1_v * scipy_stats.norm.pdf(theorem1_g) alg1_delta_r_tilde_t_term3 = theorem1_v * theorem1_g * scipy_stats.norm.cdf(theorem1_g) _lambda = prior.DEFAULT_MINIMUM_NOISE_VAR**-1 eq4_rhs_term1 = 0.5 * math.log(1.0 + _lambda * variance_t1_theta_t_with_nu_t) eq4_rhs_term2 = ( -0.5 * variance_t1_theta_t_with_nu_t / (variance_t1_theta_t_with_nu_t + _lambda**-1) ) eq4_rhs_term3 = ( 0.5 * variance_t1_theta_t_with_nu_t * (y_t - mu_t1_theta_t_with_nu_t) ** 2 / (variance_t1_theta_t_with_nu_t + _lambda**-1) ** 2 ) alg1_delta_r_tilde_t_term4 = kappa_t1 * math.sqrt( 0.5 * (eq4_rhs_term1 + eq4_rhs_term2 + eq4_rhs_term3) ) return min( sys.float_info.max * 0.5, alg1_delta_r_tilde_t_term1 + alg1_delta_r_tilde_t_term2 + alg1_delta_r_tilde_t_term3 + alg1_delta_r_tilde_t_term4, ) def _compute_gp_posterior( search_space: gp_search_space.SearchSpace, X: np.ndarray, Y: np.ndarray, x_params: np.ndarray, kernel_params: gp.KernelParamsTensor, ) -> tuple[float, float]: # mean, var acqf_params = acqf.create_acqf_params( acqf_type=acqf.AcquisitionFunctionType.LOG_EI, kernel_params=kernel_params, search_space=search_space, X=X, # normalized_params[..., :-1, :], Y=Y, # standarized_score_vals[:-1], ) mean, var = gp.posterior( acqf_params.kernel_params, torch.from_numpy(acqf_params.X), torch.from_numpy( acqf_params.search_space.scale_types == gp_search_space.ScaleType.CATEGORICAL ), torch.from_numpy(acqf_params.cov_Y_Y_inv), torch.from_numpy(acqf_params.cov_Y_Y_inv_Y), torch.from_numpy(x_params), # best_params or normalized_params[..., -1, :]), ) mean = mean.detach().numpy().flatten() var = var.detach().numpy().flatten() assert len(mean) == 1 and len(var) == 1 return float(mean[0]), float(var[0]) def _posterior_of_batched_theta( kernel_params: gp.KernelParamsTensor, X: torch.Tensor, # [len(trials), len(params)] is_categorical: torch.Tensor, # bool[len(params)] cov_Y_Y_inv: torch.Tensor, # [len(trials), len(trials)] cov_Y_Y_inv_Y: torch.Tensor, # [len(trials)] theta: torch.Tensor, # [batch, len(params)] ) -> tuple[torch.Tensor, torch.Tensor]: # (mean: [(batch,)], var: [(batch,batch)]) assert len(X.shape) == 2 len_trials, len_params = X.shape assert len(theta.shape) == 2 len_batch = theta.shape[0] assert theta.shape == (len_batch, len_params) assert is_categorical.shape == (len_params,) assert cov_Y_Y_inv.shape == (len_trials, len_trials) assert cov_Y_Y_inv_Y.shape == (len_trials,) cov_ftheta_fX = gp.kernel(is_categorical, kernel_params, theta[..., None, :], X)[..., 0, :] assert cov_ftheta_fX.shape == (len_batch, len_trials) cov_ftheta_ftheta = gp.kernel(is_categorical, kernel_params, theta[..., None, :], theta)[ ..., 0, : ] assert cov_ftheta_ftheta.shape == (len_batch, len_batch) assert torch.allclose(cov_ftheta_ftheta.diag(), gp.kernel_at_zero_distance(kernel_params)) assert torch.allclose(cov_ftheta_ftheta, cov_ftheta_ftheta.T) mean = cov_ftheta_fX @ cov_Y_Y_inv_Y assert mean.shape == (len_batch,) var = cov_ftheta_ftheta - cov_ftheta_fX @ cov_Y_Y_inv @ cov_ftheta_fX.T assert var.shape == (len_batch, len_batch) # We need to clamp the variance to avoid negative values due to numerical errors. return mean, torch.clamp(var, min=0.0) def _compute_gp_posterior_cov_two_thetas( search_space: gp_search_space.SearchSpace, normalized_params: np.ndarray, standarized_score_vals: np.ndarray, kernel_params: gp.KernelParamsTensor, theta1_index: int, theta2_index: int, ) -> float: # cov if theta1_index == theta2_index: return _compute_gp_posterior( search_space, normalized_params, standarized_score_vals, normalized_params[theta1_index], kernel_params, )[1] assert normalized_params.shape[0] == standarized_score_vals.shape[0] acqf_params = acqf.create_acqf_params( acqf_type=acqf.AcquisitionFunctionType.LOG_EI, kernel_params=kernel_params, search_space=search_space, X=normalized_params, Y=standarized_score_vals, ) _, var = _posterior_of_batched_theta( acqf_params.kernel_params, torch.from_numpy(acqf_params.X), torch.from_numpy( acqf_params.search_space.scale_types == gp_search_space.ScaleType.CATEGORICAL ), torch.from_numpy(acqf_params.cov_Y_Y_inv), torch.from_numpy(acqf_params.cov_Y_Y_inv_Y), torch.from_numpy(normalized_params[[theta1_index, theta2_index]]), ) assert var.shape == (2, 2) var = var.detach().numpy()[0, 1] return float(var) optuna-4.1.0/optuna/terminator/improvement/evaluator.py000066400000000000000000000244731471332314300234500ustar00rootroot00000000000000from __future__ import annotations import abc from typing import TYPE_CHECKING import numpy as np from optuna._experimental import experimental_class from optuna.distributions import BaseDistribution from optuna.samplers._lazy_random_state import LazyRandomState from optuna.search_space import intersection_search_space from optuna.study import StudyDirection from optuna.trial import FrozenTrial from optuna.trial import TrialState if TYPE_CHECKING: from optuna._gp import acqf from optuna._gp import gp from optuna._gp import optim_sample from optuna._gp import prior from optuna._gp import search_space as gp_search_space else: from optuna._imports import _LazyImport gp = _LazyImport("optuna._gp.gp") optim_sample = _LazyImport("optuna._gp.optim_sample") acqf = _LazyImport("optuna._gp.acqf") prior = _LazyImport("optuna._gp.prior") gp_search_space = _LazyImport("optuna._gp.search_space") DEFAULT_TOP_TRIALS_RATIO = 0.5 DEFAULT_MIN_N_TRIALS = 20 def _get_beta(n_params: int, n_trials: int, delta: float = 0.1) -> float: # TODO(nabenabe0928): Check the original implementation to verify. # Especially, |D| seems to be the domain size, but not the dimension based on Theorem 1. beta = 2 * np.log(n_params * n_trials**2 * np.pi**2 / 6 / delta) # The following div is according to the original paper: "We then further scale it down # by a factor of 5 as defined in the experiments in # `Srinivas et al. (2010) `__" beta /= 5 return beta def _compute_standardized_regret_bound( kernel_params: gp.KernelParamsTensor, search_space: gp_search_space.SearchSpace, normalized_top_n_params: np.ndarray, standarized_top_n_values: np.ndarray, delta: float = 0.1, optimize_n_samples: int = 2048, rng: np.random.RandomState | None = None, ) -> float: """ # In the original paper, f(x) was intended to be minimized, but here we would like to # maximize f(x). Hence, the following changes happen: # 1. min(ucb) over top trials becomes max(lcb) over top trials, and # 2. min(lcb) over the search space becomes max(ucb) over the search space, and # 3. Regret bound becomes max(ucb) over the search space minus max(lcb) over top trials. """ n_trials, n_params = normalized_top_n_params.shape # calculate max_ucb beta = _get_beta(n_params, n_trials, delta) ucb_acqf_params = acqf.create_acqf_params( acqf_type=acqf.AcquisitionFunctionType.UCB, kernel_params=kernel_params, search_space=search_space, X=normalized_top_n_params, Y=standarized_top_n_values, beta=beta, ) # UCB over the search space. (Original: LCB over the search space. See Change 1 above.) standardized_ucb_value = max( acqf.eval_acqf_no_grad(ucb_acqf_params, normalized_top_n_params).max(), optim_sample.optimize_acqf_sample(ucb_acqf_params, n_samples=optimize_n_samples, rng=rng)[ 1 ], ) # calculate min_lcb lcb_acqf_params = acqf.create_acqf_params( acqf_type=acqf.AcquisitionFunctionType.LCB, kernel_params=kernel_params, search_space=search_space, X=normalized_top_n_params, Y=standarized_top_n_values, beta=beta, ) # LCB over the top trials. (Original: UCB over the top trials. See Change 2 above.) standardized_lcb_value = np.max( acqf.eval_acqf_no_grad(lcb_acqf_params, normalized_top_n_params) ) # max(UCB) - max(LCB). (Original: min(UCB) - min(LCB). See Change 3 above.) return standardized_ucb_value - standardized_lcb_value # standardized regret bound @experimental_class("3.2.0") class BaseImprovementEvaluator(metaclass=abc.ABCMeta): """Base class for improvement evaluators.""" @abc.abstractmethod def evaluate(self, trials: list[FrozenTrial], study_direction: StudyDirection) -> float: pass @experimental_class("3.2.0") class RegretBoundEvaluator(BaseImprovementEvaluator): """An error evaluator for upper bound on the regret with high-probability confidence. This evaluator evaluates the regret of current best solution, which defined as the difference between the objective value of the best solution and of the global optimum. To be specific, this evaluator calculates the upper bound on the regret based on the fact that empirical estimator of the objective function is bounded by lower and upper confidence bounds with high probability under the Gaussian process model assumption. Args: top_trials_ratio: A ratio of top trials to be considered when estimating the regret. Default to 0.5. min_n_trials: A minimum number of complete trials to estimate the regret. Default to 20. seed: Seed for random number generator. For further information about this evaluator, please refer to the following paper: - `Automatic Termination for Hyperparameter Optimization `__ """ # NOQA: E501 def __init__( self, top_trials_ratio: float = DEFAULT_TOP_TRIALS_RATIO, min_n_trials: int = DEFAULT_MIN_N_TRIALS, seed: int | None = None, ) -> None: self._top_trials_ratio = top_trials_ratio self._min_n_trials = min_n_trials self._log_prior = prior.default_log_prior self._minimum_noise = prior.DEFAULT_MINIMUM_NOISE_VAR self._optimize_n_samples = 2048 self._rng = LazyRandomState(seed) def _get_top_n( self, normalized_params: np.ndarray, values: np.ndarray ) -> tuple[np.ndarray, np.ndarray]: assert len(normalized_params) == len(values) n_trials = len(normalized_params) top_n = np.clip(int(n_trials * self._top_trials_ratio), self._min_n_trials, n_trials) top_n_val = np.partition(values, n_trials - top_n)[n_trials - top_n] top_n_mask = values >= top_n_val return normalized_params[top_n_mask], values[top_n_mask] def evaluate(self, trials: list[FrozenTrial], study_direction: StudyDirection) -> float: optuna_search_space = intersection_search_space(trials) self._validate_input(trials, optuna_search_space) complete_trials = [t for t in trials if t.state == TrialState.COMPLETE] # _gp module assumes that optimization direction is maximization sign = -1 if study_direction == StudyDirection.MINIMIZE else 1 values = np.array([t.value for t in complete_trials]) * sign search_space, normalized_params = gp_search_space.get_search_space_and_normalized_params( complete_trials, optuna_search_space ) normalized_top_n_params, top_n_values = self._get_top_n(normalized_params, values) top_n_values_mean = top_n_values.mean() top_n_values_std = max(1e-10, top_n_values.std()) standarized_top_n_values = (top_n_values - top_n_values_mean) / top_n_values_std kernel_params = gp.fit_kernel_params( X=normalized_top_n_params, Y=standarized_top_n_values, is_categorical=(search_space.scale_types == gp_search_space.ScaleType.CATEGORICAL), log_prior=self._log_prior, minimum_noise=self._minimum_noise, # TODO(contramundum53): Add option to specify this. deterministic_objective=False, # TODO(y0z): Add `kernel_params_cache` to speedup. initial_kernel_params=None, ) standardized_regret_bound = _compute_standardized_regret_bound( kernel_params, search_space, normalized_top_n_params, standarized_top_n_values, rng=self._rng.rng, ) return standardized_regret_bound * top_n_values_std # regret bound @classmethod def _validate_input( cls, trials: list[FrozenTrial], search_space: dict[str, BaseDistribution] ) -> None: if len([t for t in trials if t.state == TrialState.COMPLETE]) == 0: raise ValueError( "Because no trial has been completed yet, the regret bound cannot be evaluated." ) if len(search_space) == 0: raise ValueError( "The intersection search space is empty. This condition is not supported by " f"{cls.__name__}." ) @experimental_class("3.4.0") class BestValueStagnationEvaluator(BaseImprovementEvaluator): """Evaluates the stagnation period of the best value in an optimization process. This class is initialized with a maximum stagnation period (`max_stagnation_trials`) and is designed to evaluate the remaining trials before reaching this maximum period of allowed stagnation. If this remaining trials reach zero, the trial terminates. Therefore, the default error evaluator is instantiated by StaticErrorEvaluator(const=0). Args: max_stagnation_trials: The maximum number of trials allowed for stagnation. """ def __init__(self, max_stagnation_trials: int = 30) -> None: if max_stagnation_trials < 0: raise ValueError("The maximum number of stagnant trials must not be negative.") self._max_stagnation_trials = max_stagnation_trials def evaluate(self, trials: list[FrozenTrial], study_direction: StudyDirection) -> float: self._validate_input(trials) is_maximize_direction = True if (study_direction == StudyDirection.MAXIMIZE) else False trials = [t for t in trials if t.state == TrialState.COMPLETE] current_step = len(trials) - 1 best_step = 0 for i, trial in enumerate(trials): best_value = trials[best_step].value current_value = trial.value assert best_value is not None assert current_value is not None if is_maximize_direction and (best_value < current_value): best_step = i elif (not is_maximize_direction) and (best_value > current_value): best_step = i return self._max_stagnation_trials - (current_step - best_step) @classmethod def _validate_input(cls, trials: list[FrozenTrial]) -> None: if len([t for t in trials if t.state == TrialState.COMPLETE]) == 0: raise ValueError( "Because no trial has been completed yet, the improvement cannot be evaluated." ) optuna-4.1.0/optuna/terminator/median_erroreval.py000066400000000000000000000067151471332314300224160ustar00rootroot00000000000000from __future__ import annotations import sys import numpy as np from optuna._experimental import experimental_class from optuna.study import StudyDirection from optuna.terminator.erroreval import BaseErrorEvaluator from optuna.terminator.improvement.evaluator import BaseImprovementEvaluator from optuna.trial import FrozenTrial from optuna.trial._state import TrialState @experimental_class("4.0.0") class MedianErrorEvaluator(BaseErrorEvaluator): """An error evaluator that returns the ratio to initial median. This error evaluator is introduced as a heuristics in the following paper: - `A stopping criterion for Bayesian optimization by the gap of expected minimum simple regrets `__ Args: paired_improvement_evaluator: The ``improvement_evaluator`` instance which is set with this ``error_evaluator``. warm_up_trials: A parameter specifies the number of initial trials to be discarded before the calculation of median. Default to 10. In optuna, the first 10 trials are often random sampling. The ``warm_up_trials`` can exclude them from the calculation. n_initial_trials: A parameter specifies the number of initial trials considered in the calculation of median after `warm_up_trials`. Default to 20. threshold_ratio: A parameter specifies the ratio between the threshold and initial median. Default to 0.01. """ def __init__( self, paired_improvement_evaluator: BaseImprovementEvaluator, warm_up_trials: int = 10, n_initial_trials: int = 20, threshold_ratio: float = 0.01, ) -> None: if warm_up_trials < 0: raise ValueError("`warm_up_trials` is expected to be a non-negative integer.") if n_initial_trials <= 0: raise ValueError("`n_initial_trials` is expected to be a positive integer.") if threshold_ratio <= 0.0 or not np.isfinite(threshold_ratio): raise ValueError("`threshold_ratio_to_initial_median` is expected to be a positive.") self._paired_improvement_evaluator = paired_improvement_evaluator self._warm_up_trials = warm_up_trials self._n_initial_trials = n_initial_trials self._threshold_ratio = threshold_ratio self._threshold: float | None = None def evaluate( self, trials: list[FrozenTrial], study_direction: StudyDirection, ) -> float: if self._threshold is not None: return self._threshold trials = [trial for trial in trials if trial.state == TrialState.COMPLETE] if len(trials) < (self._warm_up_trials + self._n_initial_trials): return ( -sys.float_info.min ) # Do not terminate. It assumes that improvement must non-negative. trials.sort(key=lambda trial: trial.number) criteria = [] for i in range(1, self._n_initial_trials + 1): criteria.append( self._paired_improvement_evaluator.evaluate( trials[self._warm_up_trials : self._warm_up_trials + i], study_direction ) ) criteria.sort() self._threshold = criteria[len(criteria) // 2] assert self._threshold is not None self._threshold = min(sys.float_info.max, self._threshold * self._threshold_ratio) return self._threshold optuna-4.1.0/optuna/terminator/terminator.py000066400000000000000000000122111471332314300212500ustar00rootroot00000000000000from __future__ import annotations import abc from optuna._experimental import experimental_class from optuna.study.study import Study from optuna.terminator.erroreval import BaseErrorEvaluator from optuna.terminator.erroreval import CrossValidationErrorEvaluator from optuna.terminator.erroreval import StaticErrorEvaluator from optuna.terminator.improvement.evaluator import BaseImprovementEvaluator from optuna.terminator.improvement.evaluator import BestValueStagnationEvaluator from optuna.terminator.improvement.evaluator import DEFAULT_MIN_N_TRIALS from optuna.terminator.improvement.evaluator import RegretBoundEvaluator from optuna.trial import TrialState class BaseTerminator(metaclass=abc.ABCMeta): """Base class for terminators.""" @abc.abstractmethod def should_terminate(self, study: Study) -> bool: pass @experimental_class("3.2.0") class Terminator(BaseTerminator): """Automatic stopping mechanism for Optuna studies. This class implements an automatic stopping mechanism for Optuna studies, aiming to prevent unnecessary computation. The study is terminated when the statistical error, e.g. cross-validation error, exceeds the room left for optimization. For further information about the algorithm, please refer to the following paper: - `A. Makarova et al. Automatic termination for hyperparameter optimization. `__ Args: improvement_evaluator: An evaluator object for assessing the room left for optimization. Defaults to a :class:`~optuna.terminator.improvement.evaluator.RegretBoundEvaluator` object. error_evaluator: An evaluator for calculating the statistical error, e.g. cross-validation error. Defaults to a :class:`~optuna.terminator.CrossValidationErrorEvaluator` object. min_n_trials: The minimum number of trials before termination is considered. Defaults to ``20``. Raises: ValueError: If ``min_n_trials`` is not a positive integer. Example: .. testcode:: import logging import sys from sklearn.datasets import load_wine from sklearn.ensemble import RandomForestClassifier from sklearn.model_selection import cross_val_score from sklearn.model_selection import KFold import optuna from optuna.terminator import Terminator from optuna.terminator import report_cross_validation_scores study = optuna.create_study(direction="maximize") terminator = Terminator() min_n_trials = 20 while True: trial = study.ask() X, y = load_wine(return_X_y=True) clf = RandomForestClassifier( max_depth=trial.suggest_int("max_depth", 2, 32), min_samples_split=trial.suggest_float("min_samples_split", 0, 1), criterion=trial.suggest_categorical("criterion", ("gini", "entropy")), ) scores = cross_val_score(clf, X, y, cv=KFold(n_splits=5, shuffle=True)) report_cross_validation_scores(trial, scores) value = scores.mean() logging.info(f"Trial #{trial.number} finished with value {value}.") study.tell(trial, value) if trial.number > min_n_trials and terminator.should_terminate(study): logging.info("Terminated by Optuna Terminator!") break .. seealso:: Please refer to :class:`~optuna.terminator.TerminatorCallback` for how to use the terminator mechanism with the :func:`~optuna.study.Study.optimize` method. """ def __init__( self, improvement_evaluator: BaseImprovementEvaluator | None = None, error_evaluator: BaseErrorEvaluator | None = None, min_n_trials: int = DEFAULT_MIN_N_TRIALS, ) -> None: if min_n_trials <= 0: raise ValueError("`min_n_trials` is expected to be a positive integer.") self._improvement_evaluator = improvement_evaluator or RegretBoundEvaluator() self._error_evaluator = error_evaluator or self._initialize_error_evaluator() self._min_n_trials = min_n_trials def _initialize_error_evaluator(self) -> BaseErrorEvaluator: if isinstance(self._improvement_evaluator, BestValueStagnationEvaluator): return StaticErrorEvaluator(constant=0) return CrossValidationErrorEvaluator() def should_terminate(self, study: Study) -> bool: """Judge whether the study should be terminated based on the reported values.""" trials = study.get_trials(states=[TrialState.COMPLETE]) if len(trials) < self._min_n_trials: return False improvement = self._improvement_evaluator.evaluate( trials=study.trials, study_direction=study.direction, ) error = self._error_evaluator.evaluate( trials=study.trials, study_direction=study.direction ) should_terminate = improvement < error return should_terminate optuna-4.1.0/optuna/testing/000077500000000000000000000000001471332314300160065ustar00rootroot00000000000000optuna-4.1.0/optuna/testing/__init__.py000066400000000000000000000000001471332314300201050ustar00rootroot00000000000000optuna-4.1.0/optuna/testing/distributions.py000066400000000000000000000007411471332314300212640ustar00rootroot00000000000000from __future__ import annotations from typing import Any from optuna.distributions import BaseDistribution class UnsupportedDistribution(BaseDistribution): def single(self) -> bool: return False def _contains(self, param_value_in_internal_repr: float) -> bool: return True def _asdict(self) -> dict: return {} def to_internal_repr(self, param_value_in_external_repr: Any) -> float: return float(param_value_in_external_repr) optuna-4.1.0/optuna/testing/objectives.py000066400000000000000000000003051471332314300205130ustar00rootroot00000000000000from optuna import TrialPruned from optuna.trial import Trial def fail_objective(_: Trial) -> float: raise ValueError() def pruned_objective(trial: Trial) -> float: raise TrialPruned() optuna-4.1.0/optuna/testing/pruners.py000066400000000000000000000004761471332314300200650ustar00rootroot00000000000000from __future__ import annotations import optuna class DeterministicPruner(optuna.pruners.BasePruner): def __init__(self, is_pruning: bool) -> None: self.is_pruning = is_pruning def prune(self, study: "optuna.study.Study", trial: "optuna.trial.FrozenTrial") -> bool: return self.is_pruning optuna-4.1.0/optuna/testing/samplers.py000066400000000000000000000037261471332314300202160ustar00rootroot00000000000000from __future__ import annotations from typing import Any import optuna from optuna.distributions import BaseDistribution class DeterministicSampler(optuna.samplers.BaseSampler): def __init__(self, params: dict[str, Any]) -> None: self.params = params def infer_relative_search_space( self, study: "optuna.study.Study", trial: "optuna.trial.FrozenTrial" ) -> dict[str, BaseDistribution]: return {} def sample_relative( self, study: "optuna.study.Study", trial: "optuna.trial.FrozenTrial", search_space: dict[str, BaseDistribution], ) -> dict[str, Any]: return {} def sample_independent( self, study: "optuna.study.Study", trial: "optuna.trial.FrozenTrial", param_name: str, param_distribution: BaseDistribution, ) -> Any: param_value = self.params[param_name] assert param_distribution._contains(param_distribution.to_internal_repr(param_value)) return param_value class FirstTrialOnlyRandomSampler(optuna.samplers.RandomSampler): def sample_relative( self, study: "optuna.study.Study", trial: "optuna.trial.FrozenTrial", search_space: dict[str, BaseDistribution], ) -> dict[str, float]: if len(study.trials) > 1: raise RuntimeError("`FirstTrialOnlyRandomSampler` only works on the first trial.") return super(FirstTrialOnlyRandomSampler, self).sample_relative(study, trial, search_space) def sample_independent( self, study: "optuna.study.Study", trial: "optuna.trial.FrozenTrial", param_name: str, param_distribution: BaseDistribution, ) -> float: if len(study.trials) > 1: raise RuntimeError("`FirstTrialOnlyRandomSampler` only works on the first trial.") return super(FirstTrialOnlyRandomSampler, self).sample_independent( study, trial, param_name, param_distribution ) optuna-4.1.0/optuna/testing/storages.py000066400000000000000000000050321471332314300202070ustar00rootroot00000000000000from __future__ import annotations from types import TracebackType from typing import Any from typing import IO import fakeredis import optuna from optuna.storages.journal import JournalFileBackend from optuna.testing.tempfile_pool import NamedTemporaryFilePool STORAGE_MODES: list[Any] = [ "inmemory", "sqlite", "cached_sqlite", "journal", "journal_redis", ] STORAGE_MODES_HEARTBEAT = [ "sqlite", "cached_sqlite", ] SQLITE3_TIMEOUT = 300 class StorageSupplier: def __init__(self, storage_specifier: str, **kwargs: Any) -> None: self.storage_specifier = storage_specifier self.extra_args = kwargs self.tempfile: IO[Any] | None = None def __enter__( self, ) -> ( optuna.storages.InMemoryStorage | optuna.storages._CachedStorage | optuna.storages.RDBStorage | optuna.storages.JournalStorage ): if self.storage_specifier == "inmemory": if len(self.extra_args) > 0: raise ValueError("InMemoryStorage does not accept any arguments!") return optuna.storages.InMemoryStorage() elif "sqlite" in self.storage_specifier: self.tempfile = NamedTemporaryFilePool().tempfile() url = "sqlite:///{}".format(self.tempfile.name) rdb_storage = optuna.storages.RDBStorage( url, engine_kwargs={"connect_args": {"timeout": SQLITE3_TIMEOUT}}, **self.extra_args, ) return ( optuna.storages._CachedStorage(rdb_storage) if "cached" in self.storage_specifier else rdb_storage ) elif self.storage_specifier == "journal_redis": journal_redis_storage = optuna.storages.journal.JournalRedisBackend( "redis://localhost" ) journal_redis_storage._redis = self.extra_args.get( "redis", fakeredis.FakeStrictRedis() # type: ignore[no-untyped-call] ) return optuna.storages.JournalStorage(journal_redis_storage) elif "journal" in self.storage_specifier: self.tempfile = NamedTemporaryFilePool().tempfile() file_storage = JournalFileBackend(self.tempfile.name) return optuna.storages.JournalStorage(file_storage) else: assert False def __exit__( self, exc_type: type[BaseException], exc_val: BaseException, exc_tb: TracebackType ) -> None: if self.tempfile: self.tempfile.close() optuna-4.1.0/optuna/testing/tempfile_pool.py000066400000000000000000000023721471332314300212220ustar00rootroot00000000000000# On Windows, temporary file shold delete "after" storage was deleted # NamedTemporaryFilePool ensures tempfile delete after tests. from __future__ import annotations import atexit import gc import os import tempfile from types import TracebackType from typing import Any from typing import IO class NamedTemporaryFilePool: tempfile_pool: list[IO[Any]] = [] def __new__(cls, **kwargs: Any) -> "NamedTemporaryFilePool": if not hasattr(cls, "_instance"): cls._instance = super(NamedTemporaryFilePool, cls).__new__(cls) atexit.register(cls._instance.cleanup) return cls._instance def __init__(self, **kwargs: Any) -> None: self.kwargs = kwargs def tempfile(self) -> IO[Any]: self._tempfile = tempfile.NamedTemporaryFile(delete=False, **self.kwargs) self.tempfile_pool.append(self._tempfile) return self._tempfile def cleanup(self) -> None: gc.collect() for i in self.tempfile_pool: os.unlink(i.name) def __enter__(self) -> IO[Any]: return self.tempfile() def __exit__( self, exc_type: type[BaseException], exc_val: BaseException, exc_tb: TracebackType, ) -> None: self._tempfile.close() optuna-4.1.0/optuna/testing/threading.py000066400000000000000000000011561471332314300203300ustar00rootroot00000000000000from __future__ import annotations import threading from typing import Any from typing import Callable class _TestableThread(threading.Thread): def __init__(self, target: Callable[..., Any], args: tuple): threading.Thread.__init__(self, target=target, args=args) self.exc: BaseException | None = None def run(self) -> None: try: threading.Thread.run(self) except BaseException as e: self.exc = e def join(self, timeout: float | None = None) -> None: super(_TestableThread, self).join(timeout) if self.exc: raise self.exc optuna-4.1.0/optuna/testing/trials.py000066400000000000000000000020511471332314300176540ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Sequence from typing import Any import optuna from optuna.distributions import BaseDistribution from optuna.samplers._base import _CONSTRAINTS_KEY from optuna.trial import FrozenTrial from optuna.trial import TrialState def _create_frozen_trial( number: int = 0, values: Sequence[float] | None = None, constraints: Sequence[float] | None = None, params: dict[str, Any] | None = None, param_distributions: dict[str, BaseDistribution] | None = None, state: TrialState = TrialState.COMPLETE, ) -> optuna.trial.FrozenTrial: return FrozenTrial( number=number, value=1.0 if values is None else None, values=values, state=state, user_attrs={}, system_attrs={} if constraints is None else {_CONSTRAINTS_KEY: list(constraints)}, params=params or {}, distributions=param_distributions or {}, intermediate_values={}, datetime_start=None, datetime_complete=None, trial_id=number, ) optuna-4.1.0/optuna/testing/visualization.py000066400000000000000000000046511471332314300212670ustar00rootroot00000000000000from optuna import Study from optuna.distributions import FloatDistribution from optuna.study import create_study from optuna.trial import create_trial def prepare_study_with_trials( n_objectives: int = 1, direction: str = "minimize", value_for_first_trial: float = 0.0, ) -> Study: """Return a dummy study object for tests. This function is added to reduce the code to set up dummy study object in each test case. However, you can only use this function for unit tests that are loosely coupled with the dummy study object. Unit tests that are tightly coupled with the study become difficult to read because of `Mystery Guest `__ and/or `Eager Test `__ anti-patterns. Args: n_objectives: Number of objective values. direction: Study's optimization direction. value_for_first_trial: Objective value in first trial. This value will be broadcasted to all objectives in multi-objective optimization. Returns: :class:`~optuna.study.Study` """ study = create_study(directions=[direction] * n_objectives) study.add_trial( create_trial( values=[value_for_first_trial] * n_objectives, params={"param_a": 1.0, "param_b": 2.0, "param_c": 3.0, "param_d": 4.0}, distributions={ "param_a": FloatDistribution(0.0, 3.0), "param_b": FloatDistribution(0.0, 3.0), "param_c": FloatDistribution(2.0, 5.0), "param_d": FloatDistribution(2.0, 5.0), }, ) ) study.add_trial( create_trial( values=[2.0] * n_objectives, params={"param_b": 0.0, "param_d": 4.0}, distributions={ "param_b": FloatDistribution(0.0, 3.0), "param_d": FloatDistribution(2.0, 5.0), }, ) ) study.add_trial( create_trial( values=[1.0] * n_objectives, params={"param_a": 2.5, "param_b": 1.0, "param_c": 4.5, "param_d": 2.0}, distributions={ "param_a": FloatDistribution(0.0, 3.0), "param_b": FloatDistribution(0.0, 3.0), "param_c": FloatDistribution(2.0, 5.0), "param_d": FloatDistribution(2.0, 5.0), }, ) ) return study optuna-4.1.0/optuna/trial/000077500000000000000000000000001471332314300154445ustar00rootroot00000000000000optuna-4.1.0/optuna/trial/__init__.py000066400000000000000000000005711471332314300175600ustar00rootroot00000000000000from optuna.trial._base import BaseTrial from optuna.trial._fixed import FixedTrial from optuna.trial._frozen import create_trial from optuna.trial._frozen import FrozenTrial from optuna.trial._state import TrialState from optuna.trial._trial import Trial __all__ = [ "BaseTrial", "FixedTrial", "FrozenTrial", "Trial", "TrialState", "create_trial", ] optuna-4.1.0/optuna/trial/_base.py000066400000000000000000000070701471332314300170730ustar00rootroot00000000000000from __future__ import annotations import abc from collections.abc import Sequence import datetime from typing import Any from typing import overload from optuna._deprecated import deprecated_func from optuna.distributions import BaseDistribution from optuna.distributions import CategoricalChoiceType _SUGGEST_INT_POSITIONAL_ARGS = ["self", "name", "low", "high", "step", "log"] class BaseTrial(abc.ABC): """Base class for trials. Note that this class is not supposed to be directly accessed by library users. """ @abc.abstractmethod def suggest_float( self, name: str, low: float, high: float, *, step: float | None = None, log: bool = False, ) -> float: raise NotImplementedError @deprecated_func("3.0.0", "6.0.0") @abc.abstractmethod def suggest_uniform(self, name: str, low: float, high: float) -> float: raise NotImplementedError @deprecated_func("3.0.0", "6.0.0") @abc.abstractmethod def suggest_loguniform(self, name: str, low: float, high: float) -> float: raise NotImplementedError @deprecated_func("3.0.0", "6.0.0") @abc.abstractmethod def suggest_discrete_uniform(self, name: str, low: float, high: float, q: float) -> float: raise NotImplementedError @abc.abstractmethod def suggest_int( self, name: str, low: int, high: int, *, step: int = 1, log: bool = False ) -> int: raise NotImplementedError @overload @abc.abstractmethod def suggest_categorical(self, name: str, choices: Sequence[None]) -> None: ... @overload @abc.abstractmethod def suggest_categorical(self, name: str, choices: Sequence[bool]) -> bool: ... @overload @abc.abstractmethod def suggest_categorical(self, name: str, choices: Sequence[int]) -> int: ... @overload @abc.abstractmethod def suggest_categorical(self, name: str, choices: Sequence[float]) -> float: ... @overload @abc.abstractmethod def suggest_categorical(self, name: str, choices: Sequence[str]) -> str: ... @overload @abc.abstractmethod def suggest_categorical( self, name: str, choices: Sequence[CategoricalChoiceType] ) -> CategoricalChoiceType: ... @abc.abstractmethod def suggest_categorical( self, name: str, choices: Sequence[CategoricalChoiceType] ) -> CategoricalChoiceType: raise NotImplementedError @abc.abstractmethod def report(self, value: float, step: int) -> None: raise NotImplementedError @abc.abstractmethod def should_prune(self) -> bool: raise NotImplementedError @abc.abstractmethod def set_user_attr(self, key: str, value: Any) -> None: raise NotImplementedError @abc.abstractmethod @deprecated_func("3.1.0", "5.0.0") def set_system_attr(self, key: str, value: Any) -> None: raise NotImplementedError @property @abc.abstractmethod def params(self) -> dict[str, Any]: raise NotImplementedError @property @abc.abstractmethod def distributions(self) -> dict[str, BaseDistribution]: raise NotImplementedError @property @abc.abstractmethod def user_attrs(self) -> dict[str, Any]: raise NotImplementedError @property @abc.abstractmethod def system_attrs(self) -> dict[str, Any]: raise NotImplementedError @property @abc.abstractmethod def datetime_start(self) -> datetime.datetime | None: raise NotImplementedError @property def number(self) -> int: raise NotImplementedError optuna-4.1.0/optuna/trial/_fixed.py000066400000000000000000000142501471332314300172560ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Sequence import datetime from typing import Any from typing import overload import warnings from optuna import distributions from optuna._convert_positional_args import convert_positional_args from optuna._deprecated import deprecated_func from optuna.distributions import BaseDistribution from optuna.distributions import CategoricalChoiceType from optuna.distributions import CategoricalDistribution from optuna.distributions import FloatDistribution from optuna.distributions import IntDistribution from optuna.trial._base import _SUGGEST_INT_POSITIONAL_ARGS from optuna.trial._base import BaseTrial _suggest_deprecated_msg = "Use suggest_float{args} instead." class FixedTrial(BaseTrial): """A trial class which suggests a fixed value for each parameter. This object has the same methods as :class:`~optuna.trial.Trial`, and it suggests pre-defined parameter values. The parameter values can be determined at the construction of the :class:`~optuna.trial.FixedTrial` object. In contrast to :class:`~optuna.trial.Trial`, :class:`~optuna.trial.FixedTrial` does not depend on :class:`~optuna.study.Study`, and it is useful for deploying optimization results. Example: Evaluate an objective function with parameter values given by a user. .. testcode:: import optuna def objective(trial): x = trial.suggest_float("x", -100, 100) y = trial.suggest_categorical("y", [-1, 0, 1]) return x**2 + y assert objective(optuna.trial.FixedTrial({"x": 1, "y": 0})) == 1 .. note:: Please refer to :class:`~optuna.trial.Trial` for details of methods and properties. Args: params: A dictionary containing all parameters. number: A trial number. Defaults to ``0``. """ def __init__(self, params: dict[str, Any], number: int = 0) -> None: self._params = params self._suggested_params: dict[str, Any] = {} self._distributions: dict[str, BaseDistribution] = {} self._user_attrs: dict[str, Any] = {} self._system_attrs: dict[str, Any] = {} self._datetime_start = datetime.datetime.now() self._number = number def suggest_float( self, name: str, low: float, high: float, *, step: float | None = None, log: bool = False, ) -> float: return self._suggest(name, FloatDistribution(low, high, log=log, step=step)) @deprecated_func("3.0.0", "6.0.0", text=_suggest_deprecated_msg.format(args="")) def suggest_uniform(self, name: str, low: float, high: float) -> float: return self.suggest_float(name, low, high) @deprecated_func("3.0.0", "6.0.0", text=_suggest_deprecated_msg.format(args="(..., log=True)")) def suggest_loguniform(self, name: str, low: float, high: float) -> float: return self.suggest_float(name, low, high, log=True) @deprecated_func("3.0.0", "6.0.0", text=_suggest_deprecated_msg.format(args="(..., step=...)")) def suggest_discrete_uniform(self, name: str, low: float, high: float, q: float) -> float: return self.suggest_float(name, low, high, step=q) @convert_positional_args(previous_positional_arg_names=_SUGGEST_INT_POSITIONAL_ARGS) def suggest_int( self, name: str, low: int, high: int, *, step: int = 1, log: bool = False ) -> int: return int(self._suggest(name, IntDistribution(low, high, log=log, step=step))) @overload def suggest_categorical(self, name: str, choices: Sequence[None]) -> None: ... @overload def suggest_categorical(self, name: str, choices: Sequence[bool]) -> bool: ... @overload def suggest_categorical(self, name: str, choices: Sequence[int]) -> int: ... @overload def suggest_categorical(self, name: str, choices: Sequence[float]) -> float: ... @overload def suggest_categorical(self, name: str, choices: Sequence[str]) -> str: ... @overload def suggest_categorical( self, name: str, choices: Sequence[CategoricalChoiceType] ) -> CategoricalChoiceType: ... def suggest_categorical( self, name: str, choices: Sequence[CategoricalChoiceType] ) -> CategoricalChoiceType: return self._suggest(name, CategoricalDistribution(choices=choices)) def report(self, value: float, step: int) -> None: pass def should_prune(self) -> bool: return False def set_user_attr(self, key: str, value: Any) -> None: self._user_attrs[key] = value @deprecated_func("3.1.0", "5.0.0") def set_system_attr(self, key: str, value: Any) -> None: self._system_attrs[key] = value def _suggest(self, name: str, distribution: BaseDistribution) -> Any: if name not in self._params: raise ValueError( "The value of the parameter '{}' is not found. Please set it at " "the construction of the FixedTrial object.".format(name) ) value = self._params[name] param_value_in_internal_repr = distribution.to_internal_repr(value) if not distribution._contains(param_value_in_internal_repr): warnings.warn( "The value {} of the parameter '{}' is out of " "the range of the distribution {}.".format(value, name, distribution) ) if name in self._distributions: distributions.check_distribution_compatibility(self._distributions[name], distribution) self._suggested_params[name] = value self._distributions[name] = distribution return value @property def params(self) -> dict[str, Any]: return self._suggested_params @property def distributions(self) -> dict[str, BaseDistribution]: return self._distributions @property def user_attrs(self) -> dict[str, Any]: return self._user_attrs @property def system_attrs(self) -> dict[str, Any]: return self._system_attrs @property def datetime_start(self) -> datetime.datetime | None: return self._datetime_start @property def number(self) -> int: return self._number optuna-4.1.0/optuna/trial/_frozen.py000066400000000000000000000476421471332314300174750ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Mapping from collections.abc import Sequence import datetime import math from typing import Any from typing import cast from typing import Dict from typing import overload import warnings from optuna import distributions from optuna import logging from optuna._convert_positional_args import convert_positional_args from optuna._deprecated import deprecated_func from optuna._typing import JSONSerializable from optuna.distributions import _convert_old_distribution_to_new_distribution from optuna.distributions import BaseDistribution from optuna.distributions import CategoricalChoiceType from optuna.distributions import CategoricalDistribution from optuna.distributions import FloatDistribution from optuna.distributions import IntDistribution from optuna.trial._base import _SUGGEST_INT_POSITIONAL_ARGS from optuna.trial._base import BaseTrial from optuna.trial._state import TrialState _logger = logging.get_logger(__name__) _suggest_deprecated_msg = "Use suggest_float{args} instead." class FrozenTrial(BaseTrial): """Status and results of a :class:`~optuna.trial.Trial`. An object of this class has the same methods as :class:`~optuna.trial.Trial`, but is not associated with, nor has any references to a :class:`~optuna.study.Study`. It is therefore not possible to make persistent changes to a storage from this object by itself, for instance by using :func:`~optuna.trial.FrozenTrial.set_user_attr`. It will suggest the parameter values stored in :attr:`params` and will not sample values from any distributions. It can be passed to objective functions (see :func:`~optuna.study.Study.optimize`) and is useful for deploying optimization results. Example: Re-evaluate an objective function with parameter values optimized study. .. testcode:: import optuna def objective(trial): x = trial.suggest_float("x", -1, 1) return x**2 study = optuna.create_study() study.optimize(objective, n_trials=3) assert objective(study.best_trial) == study.best_value .. note:: Instances are mutable, despite the name. For instance, :func:`~optuna.trial.FrozenTrial.set_user_attr` will update user attributes of objects in-place. Example: Overwritten attributes. .. testcode:: import copy import datetime import optuna def objective(trial): x = trial.suggest_float("x", -1, 1) # this user attribute always differs trial.set_user_attr("evaluation time", datetime.datetime.now()) return x**2 study = optuna.create_study() study.optimize(objective, n_trials=3) best_trial = study.best_trial best_trial_copy = copy.deepcopy(best_trial) # re-evaluate objective(best_trial) # the user attribute is overwritten by re-evaluation assert best_trial.user_attrs != best_trial_copy.user_attrs .. note:: Please refer to :class:`~optuna.trial.Trial` for details of methods and properties. Attributes: number: Unique and consecutive number of :class:`~optuna.trial.Trial` for each :class:`~optuna.study.Study`. Note that this field uses zero-based numbering. state: :class:`TrialState` of the :class:`~optuna.trial.Trial`. value: Objective value of the :class:`~optuna.trial.Trial`. ``value`` and ``values`` must not be specified at the same time. values: Sequence of objective values of the :class:`~optuna.trial.Trial`. The length is greater than 1 if the problem is multi-objective optimization. ``value`` and ``values`` must not be specified at the same time. datetime_start: Datetime where the :class:`~optuna.trial.Trial` started. datetime_complete: Datetime where the :class:`~optuna.trial.Trial` finished. params: Dictionary that contains suggested parameters. distributions: Dictionary that contains the distributions of :attr:`params`. user_attrs: Dictionary that contains the attributes of the :class:`~optuna.trial.Trial` set with :func:`optuna.trial.Trial.set_user_attr`. system_attrs: Dictionary that contains the attributes of the :class:`~optuna.trial.Trial` set with :func:`optuna.trial.Trial.set_system_attr`. intermediate_values: Intermediate objective values set with :func:`optuna.trial.Trial.report`. """ def __init__( self, number: int, state: TrialState, value: float | None, datetime_start: datetime.datetime | None, datetime_complete: datetime.datetime | None, params: dict[str, Any], distributions: dict[str, BaseDistribution], user_attrs: dict[str, Any], system_attrs: dict[str, Any], intermediate_values: dict[int, float], trial_id: int, *, values: Sequence[float] | None = None, ) -> None: self._number = number self.state = state self._values: list[float] | None = None if value is not None and values is not None: raise ValueError("Specify only one of `value` and `values`.") elif value is not None: self._values = [value] elif values is not None: self._values = list(values) self._datetime_start = datetime_start self.datetime_complete = datetime_complete self._params = params self._user_attrs = user_attrs self._system_attrs = system_attrs self.intermediate_values = intermediate_values self._distributions = distributions self._trial_id = trial_id def __eq__(self, other: Any) -> bool: if not isinstance(other, FrozenTrial): return NotImplemented return other.__dict__ == self.__dict__ def __lt__(self, other: Any) -> bool: if not isinstance(other, FrozenTrial): return NotImplemented return self.number < other.number def __le__(self, other: Any) -> bool: if not isinstance(other, FrozenTrial): return NotImplemented return self.number <= other.number def __hash__(self) -> int: return hash(tuple(getattr(self, field) for field in self.__dict__)) def __repr__(self) -> str: return "{cls}({kwargs})".format( cls=self.__class__.__name__, kwargs=", ".join( "{field}={value}".format( field=field if not field.startswith("_") else field[1:], value=repr(getattr(self, field)), ) for field in self.__dict__ ) + ", value=None", ) def suggest_float( self, name: str, low: float, high: float, *, step: float | None = None, log: bool = False, ) -> float: return self._suggest(name, FloatDistribution(low, high, log=log, step=step)) @deprecated_func("3.0.0", "6.0.0", text=_suggest_deprecated_msg.format(args="")) def suggest_uniform(self, name: str, low: float, high: float) -> float: return self.suggest_float(name, low, high) @deprecated_func("3.0.0", "6.0.0", text=_suggest_deprecated_msg.format(args="(..., log=True)")) def suggest_loguniform(self, name: str, low: float, high: float) -> float: return self.suggest_float(name, low, high, log=True) @deprecated_func("3.0.0", "6.0.0", text=_suggest_deprecated_msg.format(args="(..., step=...)")) def suggest_discrete_uniform(self, name: str, low: float, high: float, q: float) -> float: return self.suggest_float(name, low, high, step=q) @convert_positional_args(previous_positional_arg_names=_SUGGEST_INT_POSITIONAL_ARGS) def suggest_int( self, name: str, low: int, high: int, *, step: int = 1, log: bool = False ) -> int: return int(self._suggest(name, IntDistribution(low, high, log=log, step=step))) @overload def suggest_categorical(self, name: str, choices: Sequence[None]) -> None: ... @overload def suggest_categorical(self, name: str, choices: Sequence[bool]) -> bool: ... @overload def suggest_categorical(self, name: str, choices: Sequence[int]) -> int: ... @overload def suggest_categorical(self, name: str, choices: Sequence[float]) -> float: ... @overload def suggest_categorical(self, name: str, choices: Sequence[str]) -> str: ... @overload def suggest_categorical( self, name: str, choices: Sequence[CategoricalChoiceType] ) -> CategoricalChoiceType: ... def suggest_categorical( self, name: str, choices: Sequence[CategoricalChoiceType] ) -> CategoricalChoiceType: return self._suggest(name, CategoricalDistribution(choices=choices)) def report(self, value: float, step: int) -> None: """Interface of report function. Since :class:`~optuna.trial.FrozenTrial` is not pruned, this report function does nothing. .. seealso:: Please refer to :func:`~optuna.trial.FrozenTrial.should_prune`. Args: value: A value returned from the objective function. step: Step of the trial (e.g., Epoch of neural network training). Note that pruners assume that ``step`` starts at zero. For example, :class:`~optuna.pruners.MedianPruner` simply checks if ``step`` is less than ``n_warmup_steps`` as the warmup mechanism. """ pass def should_prune(self) -> bool: """Suggest whether the trial should be pruned or not. The suggestion is always :obj:`False` regardless of a pruning algorithm. .. note:: :class:`~optuna.trial.FrozenTrial` only samples one combination of parameters. Returns: :obj:`False`. """ return False def set_user_attr(self, key: str, value: Any) -> None: self._user_attrs[key] = value @deprecated_func("3.1.0", "5.0.0") def set_system_attr(self, key: str, value: Any) -> None: self._system_attrs[key] = value def _validate(self) -> None: if self.state != TrialState.WAITING and self.datetime_start is None: raise ValueError( "`datetime_start` is supposed to be set when the trial state is not waiting." ) if self.state.is_finished(): if self.datetime_complete is None: raise ValueError("`datetime_complete` is supposed to be set for a finished trial.") else: if self.datetime_complete is not None: raise ValueError( "`datetime_complete` is supposed to be None for an unfinished trial." ) if self.state == TrialState.FAIL and self._values is not None: raise ValueError(f"values should be None for a failed trial, but got {self._values}.") if self.state == TrialState.COMPLETE: if self._values is None: raise ValueError("values should be set for a complete trial.") elif any(math.isnan(x) for x in self._values): raise ValueError("values should not contain NaN.") if set(self.params.keys()) != set(self.distributions.keys()): raise ValueError( "Inconsistent parameters {} and distributions {}.".format( set(self.params.keys()), set(self.distributions.keys()) ) ) for param_name, param_value in self.params.items(): distribution = self.distributions[param_name] param_value_in_internal_repr = distribution.to_internal_repr(param_value) if not distribution._contains(param_value_in_internal_repr): raise ValueError( "The value {} of parameter '{}' isn't contained in the distribution " "{}.".format(param_value, param_name, distribution) ) def _suggest(self, name: str, distribution: BaseDistribution) -> Any: if name not in self._params: raise ValueError( "The value of the parameter '{}' is not found. Please set it at " "the construction of the FrozenTrial object.".format(name) ) value = self._params[name] param_value_in_internal_repr = distribution.to_internal_repr(value) if not distribution._contains(param_value_in_internal_repr): warnings.warn( "The value {} of the parameter '{}' is out of " "the range of the distribution {}.".format(value, name, distribution) ) if name in self._distributions: distributions.check_distribution_compatibility(self._distributions[name], distribution) self._distributions[name] = distribution return value @property def number(self) -> int: return self._number @number.setter def number(self, value: int) -> None: self._number = value @property def value(self) -> float | None: if self._values is not None: if len(self._values) > 1: raise RuntimeError( "This attribute is not available during multi-objective optimization." ) return self._values[0] return None @value.setter def value(self, v: float | None) -> None: if self._values is not None: if len(self._values) > 1: raise RuntimeError( "This attribute is not available during multi-objective optimization." ) if v is not None: self._values = [v] else: self._values = None # These `_get_values`, `_set_values`, and `values = property(_get_values, _set_values)` are # defined to pass the mypy. # See https://github.com/python/mypy/issues/3004#issuecomment-726022329. def _get_values(self) -> list[float] | None: return self._values def _set_values(self, v: Sequence[float] | None) -> None: if v is not None: self._values = list(v) else: self._values = None values = property(_get_values, _set_values) @property def datetime_start(self) -> datetime.datetime | None: return self._datetime_start @datetime_start.setter def datetime_start(self, value: datetime.datetime | None) -> None: self._datetime_start = value @property def params(self) -> dict[str, Any]: return self._params @params.setter def params(self, params: dict[str, Any]) -> None: self._params = params @property def distributions(self) -> dict[str, BaseDistribution]: return self._distributions @distributions.setter def distributions(self, value: dict[str, BaseDistribution]) -> None: self._distributions = value @property def user_attrs(self) -> dict[str, Any]: return self._user_attrs @user_attrs.setter def user_attrs(self, value: dict[str, Any]) -> None: self._user_attrs = value @property def system_attrs(self) -> dict[str, Any]: return self._system_attrs @system_attrs.setter def system_attrs(self, value: Mapping[str, JSONSerializable]) -> None: self._system_attrs = cast(Dict[str, Any], value) @property def last_step(self) -> int | None: """Return the maximum step of :attr:`intermediate_values` in the trial. Returns: The maximum step of intermediates. """ if len(self.intermediate_values) == 0: return None else: return max(self.intermediate_values.keys()) @property def duration(self) -> datetime.timedelta | None: """Return the elapsed time taken to complete the trial. Returns: The duration. """ if self.datetime_start and self.datetime_complete: return self.datetime_complete - self.datetime_start else: return None def create_trial( *, state: TrialState = TrialState.COMPLETE, value: float | None = None, values: Sequence[float] | None = None, params: dict[str, Any] | None = None, distributions: dict[str, BaseDistribution] | None = None, user_attrs: dict[str, Any] | None = None, system_attrs: dict[str, Any] | None = None, intermediate_values: dict[int, float] | None = None, ) -> FrozenTrial: """Create a new :class:`~optuna.trial.FrozenTrial`. Example: .. testcode:: import optuna from optuna.distributions import CategoricalDistribution from optuna.distributions import FloatDistribution trial = optuna.trial.create_trial( params={"x": 1.0, "y": 0}, distributions={ "x": FloatDistribution(0, 10), "y": CategoricalDistribution([-1, 0, 1]), }, value=5.0, ) assert isinstance(trial, optuna.trial.FrozenTrial) assert trial.value == 5.0 assert trial.params == {"x": 1.0, "y": 0} .. seealso:: See :func:`~optuna.study.Study.add_trial` for how this function can be used to create a study from existing trials. .. note:: Please note that this is a low-level API. In general, trials that are passed to objective functions are created inside :func:`~optuna.study.Study.optimize`. .. note:: When ``state`` is :class:`TrialState.COMPLETE`, the following parameters are required: * ``params`` * ``distributions`` * ``value`` or ``values`` Args: state: Trial state. value: Trial objective value. Must be specified if ``state`` is :class:`TrialState.COMPLETE`. ``value`` and ``values`` must not be specified at the same time. values: Sequence of the trial objective values. The length is greater than 1 if the problem is multi-objective optimization. Must be specified if ``state`` is :class:`TrialState.COMPLETE`. ``value`` and ``values`` must not be specified at the same time. params: Dictionary with suggested parameters of the trial. distributions: Dictionary with parameter distributions of the trial. user_attrs: Dictionary with user attributes. system_attrs: Dictionary with system attributes. Should not have to be used for most users. intermediate_values: Dictionary with intermediate objective values of the trial. Returns: Created trial. """ params = params or {} distributions = distributions or {} distributions = { key: _convert_old_distribution_to_new_distribution(dist) for key, dist in distributions.items() } user_attrs = user_attrs or {} system_attrs = system_attrs or {} intermediate_values = intermediate_values or {} if state == TrialState.WAITING: datetime_start = None else: datetime_start = datetime.datetime.now() if state.is_finished(): datetime_complete: datetime.datetime | None = datetime_start else: datetime_complete = None trial = FrozenTrial( number=-1, trial_id=-1, state=state, value=value, values=values, datetime_start=datetime_start, datetime_complete=datetime_complete, params=params, distributions=distributions, user_attrs=user_attrs, system_attrs=system_attrs, intermediate_values=intermediate_values, ) trial._validate() return trial optuna-4.1.0/optuna/trial/_state.py000066400000000000000000000020041471332314300172710ustar00rootroot00000000000000import enum class TrialState(enum.IntEnum): """State of a :class:`~optuna.trial.Trial`. Attributes: RUNNING: The :class:`~optuna.trial.Trial` is running. WAITING: The :class:`~optuna.trial.Trial` is waiting and unfinished. COMPLETE: The :class:`~optuna.trial.Trial` has been finished without any error. PRUNED: The :class:`~optuna.trial.Trial` has been pruned with :class:`~optuna.exceptions.TrialPruned`. FAIL: The :class:`~optuna.trial.Trial` has failed due to an uncaught error. """ RUNNING = 0 COMPLETE = 1 PRUNED = 2 FAIL = 3 WAITING = 4 def __repr__(self) -> str: return str(self) def is_finished(self) -> bool: """Return a bool value to represent whether the trial state is unfinished or not. The unfinished state is either ``RUNNING`` or ``WAITING``. """ return self != TrialState.RUNNING and self != TrialState.WAITING optuna-4.1.0/optuna/trial/_trial.py000066400000000000000000000722261471332314300173010ustar00rootroot00000000000000from __future__ import annotations from collections import UserDict from collections.abc import Sequence import copy import datetime from typing import Any from typing import overload import warnings import optuna from optuna import distributions from optuna import logging from optuna import pruners from optuna._convert_positional_args import convert_positional_args from optuna._deprecated import deprecated_func from optuna.distributions import BaseDistribution from optuna.distributions import CategoricalChoiceType from optuna.distributions import CategoricalDistribution from optuna.distributions import FloatDistribution from optuna.distributions import IntDistribution from optuna.trial import FrozenTrial from optuna.trial._base import _SUGGEST_INT_POSITIONAL_ARGS from optuna.trial._base import BaseTrial _logger = logging.get_logger(__name__) _suggest_deprecated_msg = "Use suggest_float{args} instead." class Trial(BaseTrial): """A trial is a process of evaluating an objective function. This object is passed to an objective function and provides interfaces to get parameter suggestion, manage the trial's state, and set/get user-defined attributes of the trial. Note that the direct use of this constructor is not recommended. This object is seamlessly instantiated and passed to the objective function behind the :func:`optuna.study.Study.optimize()` method; hence library users do not care about instantiation of this object. Args: study: A :class:`~optuna.study.Study` object. trial_id: A trial ID that is automatically generated. """ def __init__(self, study: "optuna.study.Study", trial_id: int) -> None: self.study = study self._trial_id = trial_id self.storage = self.study._storage self._cached_frozen_trial = self.storage.get_trial(self._trial_id) study = pruners._filter_study(self.study, self._cached_frozen_trial) self.study.sampler.before_trial(study, self._cached_frozen_trial) self.relative_search_space = self.study.sampler.infer_relative_search_space( study, self._cached_frozen_trial ) self._relative_params: dict[str, Any] | None = None self._fixed_params = self._cached_frozen_trial.system_attrs.get("fixed_params", {}) @property def relative_params(self) -> dict[str, Any]: if self._relative_params is None: study = pruners._filter_study(self.study, self._cached_frozen_trial) self._relative_params = self.study.sampler.sample_relative( study, self._cached_frozen_trial, self.relative_search_space ) return self._relative_params def suggest_float( self, name: str, low: float, high: float, *, step: float | None = None, log: bool = False, ) -> float: """Suggest a value for the floating point parameter. Example: Suggest a momentum, learning rate and scaling factor of learning rate for neural network training. .. testcode:: import numpy as np from sklearn.datasets import load_iris from sklearn.model_selection import train_test_split from sklearn.neural_network import MLPClassifier import optuna X, y = load_iris(return_X_y=True) X_train, X_valid, y_train, y_valid = train_test_split(X, y, random_state=0) def objective(trial): momentum = trial.suggest_float("momentum", 0.0, 1.0) learning_rate_init = trial.suggest_float( "learning_rate_init", 1e-5, 1e-3, log=True ) power_t = trial.suggest_float("power_t", 0.2, 0.8, step=0.1) clf = MLPClassifier( hidden_layer_sizes=(100, 50), momentum=momentum, learning_rate_init=learning_rate_init, solver="sgd", random_state=0, power_t=power_t, ) clf.fit(X_train, y_train) return clf.score(X_valid, y_valid) study = optuna.create_study(direction="maximize") study.optimize(objective, n_trials=3) Args: name: A parameter name. low: Lower endpoint of the range of suggested values. ``low`` is included in the range. ``low`` must be less than or equal to ``high``. If ``log`` is :obj:`True`, ``low`` must be larger than 0. high: Upper endpoint of the range of suggested values. ``high`` is included in the range. ``high`` must be greater than or equal to ``low``. step: A step of discretization. .. note:: The ``step`` and ``log`` arguments cannot be used at the same time. To set the ``step`` argument to a float number, set the ``log`` argument to :obj:`False`. log: A flag to sample the value from the log domain or not. If ``log`` is true, the value is sampled from the range in the log domain. Otherwise, the value is sampled from the range in the linear domain. .. note:: The ``step`` and ``log`` arguments cannot be used at the same time. To set the ``log`` argument to :obj:`True`, set the ``step`` argument to :obj:`None`. Returns: A suggested float value. .. seealso:: :ref:`configurations` tutorial describes more details and flexible usages. """ distribution = FloatDistribution(low, high, log=log, step=step) suggested_value = self._suggest(name, distribution) self._check_distribution(name, distribution) return suggested_value @deprecated_func("3.0.0", "6.0.0", text=_suggest_deprecated_msg.format(args="")) def suggest_uniform(self, name: str, low: float, high: float) -> float: """Suggest a value for the continuous parameter. The value is sampled from the range :math:`[\\mathsf{low}, \\mathsf{high})` in the linear domain. When :math:`\\mathsf{low} = \\mathsf{high}`, the value of :math:`\\mathsf{low}` will be returned. Args: name: A parameter name. low: Lower endpoint of the range of suggested values. ``low`` is included in the range. high: Upper endpoint of the range of suggested values. ``high`` is included in the range. Returns: A suggested float value. """ return self.suggest_float(name, low, high) @deprecated_func("3.0.0", "6.0.0", text=_suggest_deprecated_msg.format(args="(..., log=True)")) def suggest_loguniform(self, name: str, low: float, high: float) -> float: """Suggest a value for the continuous parameter. The value is sampled from the range :math:`[\\mathsf{low}, \\mathsf{high})` in the log domain. When :math:`\\mathsf{low} = \\mathsf{high}`, the value of :math:`\\mathsf{low}` will be returned. Args: name: A parameter name. low: Lower endpoint of the range of suggested values. ``low`` is included in the range. high: Upper endpoint of the range of suggested values. ``high`` is included in the range. Returns: A suggested float value. """ return self.suggest_float(name, low, high, log=True) @deprecated_func("3.0.0", "6.0.0", text=_suggest_deprecated_msg.format(args="(..., step=...)")) def suggest_discrete_uniform(self, name: str, low: float, high: float, q: float) -> float: """Suggest a value for the discrete parameter. The value is sampled from the range :math:`[\\mathsf{low}, \\mathsf{high}]`, and the step of discretization is :math:`q`. More specifically, this method returns one of the values in the sequence :math:`\\mathsf{low}, \\mathsf{low} + q, \\mathsf{low} + 2 q, \\dots, \\mathsf{low} + k q \\le \\mathsf{high}`, where :math:`k` denotes an integer. Note that :math:`high` may be changed due to round-off errors if :math:`q` is not an integer. Please check warning messages to find the changed values. Args: name: A parameter name. low: Lower endpoint of the range of suggested values. ``low`` is included in the range. high: Upper endpoint of the range of suggested values. ``high`` is included in the range. q: A step of discretization. Returns: A suggested float value. """ return self.suggest_float(name, low, high, step=q) @convert_positional_args(previous_positional_arg_names=_SUGGEST_INT_POSITIONAL_ARGS) def suggest_int( self, name: str, low: int, high: int, *, step: int = 1, log: bool = False ) -> int: """Suggest a value for the integer parameter. The value is sampled from the integers in :math:`[\\mathsf{low}, \\mathsf{high}]`. Example: Suggest the number of trees in `RandomForestClassifier `__. .. testcode:: import numpy as np from sklearn.datasets import load_iris from sklearn.ensemble import RandomForestClassifier from sklearn.model_selection import train_test_split import optuna X, y = load_iris(return_X_y=True) X_train, X_valid, y_train, y_valid = train_test_split(X, y) def objective(trial): n_estimators = trial.suggest_int("n_estimators", 50, 400) clf = RandomForestClassifier(n_estimators=n_estimators, random_state=0) clf.fit(X_train, y_train) return clf.score(X_valid, y_valid) study = optuna.create_study(direction="maximize") study.optimize(objective, n_trials=3) Args: name: A parameter name. low: Lower endpoint of the range of suggested values. ``low`` is included in the range. ``low`` must be less than or equal to ``high``. If ``log`` is :obj:`True`, ``low`` must be larger than 0. high: Upper endpoint of the range of suggested values. ``high`` is included in the range. ``high`` must be greater than or equal to ``low``. step: A step of discretization. .. note:: Note that :math:`\\mathsf{high}` is modified if the range is not divisible by :math:`\\mathsf{step}`. Please check the warning messages to find the changed values. .. note:: The method returns one of the values in the sequence :math:`\\mathsf{low}, \\mathsf{low} + \\mathsf{step}, \\mathsf{low} + 2 * \\mathsf{step}, \\dots, \\mathsf{low} + k * \\mathsf{step} \\le \\mathsf{high}`, where :math:`k` denotes an integer. .. note:: The ``step != 1`` and ``log`` arguments cannot be used at the same time. To set the ``step`` argument :math:`\\mathsf{step} \\ge 2`, set the ``log`` argument to :obj:`False`. log: A flag to sample the value from the log domain or not. .. note:: If ``log`` is true, at first, the range of suggested values is divided into grid points of width 1. The range of suggested values is then converted to a log domain, from which a value is sampled. The uniformly sampled value is re-converted to the original domain and rounded to the nearest grid point that we just split, and the suggested value is determined. For example, if `low = 2` and `high = 8`, then the range of suggested values is `[2, 3, 4, 5, 6, 7, 8]` and lower values tend to be more sampled than higher values. .. note:: The ``step != 1`` and ``log`` arguments cannot be used at the same time. To set the ``log`` argument to :obj:`True`, set the ``step`` argument to 1. .. seealso:: :ref:`configurations` tutorial describes more details and flexible usages. """ distribution = IntDistribution(low=low, high=high, log=log, step=step) suggested_value = int(self._suggest(name, distribution)) self._check_distribution(name, distribution) return suggested_value @overload def suggest_categorical(self, name: str, choices: Sequence[None]) -> None: ... @overload def suggest_categorical(self, name: str, choices: Sequence[bool]) -> bool: ... @overload def suggest_categorical(self, name: str, choices: Sequence[int]) -> int: ... @overload def suggest_categorical(self, name: str, choices: Sequence[float]) -> float: ... @overload def suggest_categorical(self, name: str, choices: Sequence[str]) -> str: ... @overload def suggest_categorical( self, name: str, choices: Sequence[CategoricalChoiceType] ) -> CategoricalChoiceType: ... def suggest_categorical( self, name: str, choices: Sequence[CategoricalChoiceType] ) -> CategoricalChoiceType: """Suggest a value for the categorical parameter. The value is sampled from ``choices``. Example: Suggest a kernel function of `SVC `__. .. testcode:: import numpy as np from sklearn.datasets import load_iris from sklearn.model_selection import train_test_split from sklearn.svm import SVC import optuna X, y = load_iris(return_X_y=True) X_train, X_valid, y_train, y_valid = train_test_split(X, y) def objective(trial): kernel = trial.suggest_categorical("kernel", ["linear", "poly", "rbf"]) clf = SVC(kernel=kernel, gamma="scale", random_state=0) clf.fit(X_train, y_train) return clf.score(X_valid, y_valid) study = optuna.create_study(direction="maximize") study.optimize(objective, n_trials=3) Args: name: A parameter name. choices: Parameter value candidates. .. seealso:: :class:`~optuna.distributions.CategoricalDistribution`. Returns: A suggested value. .. seealso:: :ref:`configurations` tutorial describes more details and flexible usages. """ # There is no need to call self._check_distribution because # CategoricalDistribution does not support dynamic value space. return self._suggest(name, CategoricalDistribution(choices=choices)) def report(self, value: float, step: int) -> None: """Report an objective function value for a given step. The reported values are used by the pruners to determine whether this trial should be pruned. .. seealso:: Please refer to :class:`~optuna.pruners.BasePruner`. .. note:: The reported value is converted to ``float`` type by applying ``float()`` function internally. Thus, it accepts all float-like types (e.g., ``numpy.float32``). If the conversion fails, a ``TypeError`` is raised. .. note:: If this method is called multiple times at the same ``step`` in a trial, the reported ``value`` only the first time is stored and the reported values from the second time are ignored. .. note:: :func:`~optuna.trial.Trial.report` does not support multi-objective optimization. Example: Report intermediate scores of `SGDClassifier `__ training. .. testcode:: import numpy as np from sklearn.datasets import load_iris from sklearn.linear_model import SGDClassifier from sklearn.model_selection import train_test_split import optuna X, y = load_iris(return_X_y=True) X_train, X_valid, y_train, y_valid = train_test_split(X, y) def objective(trial): clf = SGDClassifier(random_state=0) for step in range(100): clf.partial_fit(X_train, y_train, np.unique(y)) intermediate_value = clf.score(X_valid, y_valid) trial.report(intermediate_value, step=step) if trial.should_prune(): raise optuna.TrialPruned() return clf.score(X_valid, y_valid) study = optuna.create_study(direction="maximize") study.optimize(objective, n_trials=3) Args: value: A value returned from the objective function. step: Step of the trial (e.g., Epoch of neural network training). Note that pruners assume that ``step`` starts at zero. For example, :class:`~optuna.pruners.MedianPruner` simply checks if ``step`` is less than ``n_warmup_steps`` as the warmup mechanism. ``step`` must be a positive integer. """ if len(self.study.directions) > 1: raise NotImplementedError( "Trial.report is not supported for multi-objective optimization." ) try: # For convenience, we allow users to report a value that can be cast to `float`. value = float(value) except (TypeError, ValueError): message = ( f"The `value` argument is of type '{type(value)}' but supposed to be a float." ) raise TypeError(message) from None try: step = int(step) except (TypeError, ValueError): message = f"The `step` argument is of type '{type(step)}' but supposed to be an int." raise TypeError(message) from None if step < 0: raise ValueError(f"The `step` argument is {step} but cannot be negative.") if step in self._cached_frozen_trial.intermediate_values: # Do nothing if already reported. warnings.warn( f"The reported value is ignored because this `step` {step} is already reported." ) return self.storage.set_trial_intermediate_value(self._trial_id, step, value) self._cached_frozen_trial.intermediate_values[step] = value def should_prune(self) -> bool: """Suggest whether the trial should be pruned or not. The suggestion is made by a pruning algorithm associated with the trial and is based on previously reported values. The algorithm can be specified when constructing a :class:`~optuna.study.Study`. .. note:: If no values have been reported, the algorithm cannot make meaningful suggestions. Similarly, if this method is called multiple times with the exact same set of reported values, the suggestions will be the same. .. seealso:: Please refer to the example code in :func:`optuna.trial.Trial.report`. .. note:: :func:`~optuna.trial.Trial.should_prune` does not support multi-objective optimization. Returns: A boolean value. If :obj:`True`, the trial should be pruned according to the configured pruning algorithm. Otherwise, the trial should continue. """ if len(self.study.directions) > 1: raise NotImplementedError( "Trial.should_prune is not supported for multi-objective optimization." ) trial = self._get_latest_trial() return self.study.pruner.prune(self.study, trial) def set_user_attr(self, key: str, value: Any) -> None: """Set user attributes to the trial. The user attributes in the trial can be access via :func:`optuna.trial.Trial.user_attrs`. .. seealso:: See the recipe on :ref:`attributes`. Example: Save fixed hyperparameters of neural network training. .. testcode:: import numpy as np from sklearn.datasets import load_iris from sklearn.model_selection import train_test_split from sklearn.neural_network import MLPClassifier import optuna X, y = load_iris(return_X_y=True) X_train, X_valid, y_train, y_valid = train_test_split(X, y, random_state=0) def objective(trial): trial.set_user_attr("BATCHSIZE", 128) momentum = trial.suggest_float("momentum", 0, 1.0) clf = MLPClassifier( hidden_layer_sizes=(100, 50), batch_size=trial.user_attrs["BATCHSIZE"], momentum=momentum, solver="sgd", random_state=0, ) clf.fit(X_train, y_train) return clf.score(X_valid, y_valid) study = optuna.create_study(direction="maximize") study.optimize(objective, n_trials=3) assert "BATCHSIZE" in study.best_trial.user_attrs.keys() assert study.best_trial.user_attrs["BATCHSIZE"] == 128 Args: key: A key string of the attribute. value: A value of the attribute. The value should be JSON serializable. """ self.storage.set_trial_user_attr(self._trial_id, key, value) self._cached_frozen_trial.user_attrs[key] = value @deprecated_func("3.1.0", "5.0.0") def set_system_attr(self, key: str, value: Any) -> None: """Set system attributes to the trial. Note that Optuna internally uses this method to save system messages such as failure reason of trials. Please use :func:`~optuna.trial.Trial.set_user_attr` to set users' attributes. Args: key: A key string of the attribute. value: A value of the attribute. The value should be JSON serializable. """ self.storage.set_trial_system_attr(self._trial_id, key, value) self._cached_frozen_trial.system_attrs[key] = value def _suggest(self, name: str, distribution: BaseDistribution) -> Any: storage = self.storage trial_id = self._trial_id trial = self._get_latest_trial() if name in trial.distributions: # No need to sample if already suggested. distributions.check_distribution_compatibility(trial.distributions[name], distribution) param_value = trial.params[name] else: if self._is_fixed_param(name, distribution): param_value = self._fixed_params[name] elif distribution.single(): param_value = distributions._get_single_value(distribution) elif self._is_relative_param(name, distribution): param_value = self.relative_params[name] else: study = pruners._filter_study(self.study, trial) param_value = self.study.sampler.sample_independent( study, trial, name, distribution ) # `param_value` is validated here (invalid value like `np.nan` raises ValueError). param_value_in_internal_repr = distribution.to_internal_repr(param_value) storage.set_trial_param(trial_id, name, param_value_in_internal_repr, distribution) self._cached_frozen_trial.distributions[name] = distribution self._cached_frozen_trial.params[name] = param_value return param_value def _is_fixed_param(self, name: str, distribution: BaseDistribution) -> bool: if name not in self._fixed_params: return False param_value = self._fixed_params[name] param_value_in_internal_repr = distribution.to_internal_repr(param_value) contained = distribution._contains(param_value_in_internal_repr) if not contained: warnings.warn( "Fixed parameter '{}' with value {} is out of range " "for distribution {}.".format(name, param_value, distribution) ) return True def _is_relative_param(self, name: str, distribution: BaseDistribution) -> bool: if name not in self.relative_params: return False if name not in self.relative_search_space: raise ValueError( "The parameter '{}' was sampled by `sample_relative` method " "but it is not contained in the relative search space.".format(name) ) relative_distribution = self.relative_search_space[name] distributions.check_distribution_compatibility(relative_distribution, distribution) param_value = self.relative_params[name] param_value_in_internal_repr = distribution.to_internal_repr(param_value) return distribution._contains(param_value_in_internal_repr) def _check_distribution(self, name: str, distribution: BaseDistribution) -> None: old_distribution = self._cached_frozen_trial.distributions.get(name, distribution) if old_distribution != distribution: warnings.warn( 'Inconsistent parameter values for distribution with name "{}"! ' "This might be a configuration mistake. " "Optuna allows to call the same distribution with the same " "name more than once in a trial. " "When the parameter values are inconsistent optuna only " "uses the values of the first call and ignores all following. " "Using these values: {}".format(name, old_distribution._asdict()), RuntimeWarning, ) def _get_latest_trial(self) -> FrozenTrial: # TODO(eukaryo): Remove this method after `system_attrs` property is removed. latest_trial = copy.copy(self._cached_frozen_trial) latest_trial.system_attrs = _LazyTrialSystemAttrs( # type: ignore[assignment] self._trial_id, self.storage ) return latest_trial @property def params(self) -> dict[str, Any]: """Return parameters to be optimized. Returns: A dictionary containing all parameters. """ return copy.deepcopy(self._cached_frozen_trial.params) @property def distributions(self) -> dict[str, BaseDistribution]: """Return distributions of parameters to be optimized. Returns: A dictionary containing all distributions. """ return copy.deepcopy(self._cached_frozen_trial.distributions) @property def user_attrs(self) -> dict[str, Any]: """Return user attributes. Returns: A dictionary containing all user attributes. """ return copy.deepcopy(self._cached_frozen_trial.user_attrs) @property @deprecated_func("3.1.0", "5.0.0") def system_attrs(self) -> dict[str, Any]: """Return system attributes. Returns: A dictionary containing all system attributes. """ return copy.deepcopy(self.storage.get_trial_system_attrs(self._trial_id)) @property def datetime_start(self) -> datetime.datetime | None: """Return start datetime. Returns: Datetime where the :class:`~optuna.trial.Trial` started. """ return self._cached_frozen_trial.datetime_start @property def number(self) -> int: """Return trial's number which is consecutive and unique in a study. Returns: A trial number. """ return self._cached_frozen_trial.number class _LazyTrialSystemAttrs(UserDict): def __init__(self, trial_id: int, storage: optuna.storages.BaseStorage) -> None: super().__init__() self._trial_id = trial_id self._storage = storage self._initialized = False def __getattribute__(self, key: str) -> Any: if key == "data": if not self._initialized: self._initialized = True super().update(self._storage.get_trial_system_attrs(self._trial_id)) return super().__getattribute__(key) optuna-4.1.0/optuna/version.py000066400000000000000000000000261471332314300163660ustar00rootroot00000000000000__version__ = "4.1.0" optuna-4.1.0/optuna/visualization/000077500000000000000000000000001471332314300172325ustar00rootroot00000000000000optuna-4.1.0/optuna/visualization/__init__.py000066400000000000000000000023601471332314300213440ustar00rootroot00000000000000from optuna.visualization import matplotlib from optuna.visualization._contour import plot_contour from optuna.visualization._edf import plot_edf from optuna.visualization._hypervolume_history import plot_hypervolume_history from optuna.visualization._intermediate_values import plot_intermediate_values from optuna.visualization._optimization_history import plot_optimization_history from optuna.visualization._parallel_coordinate import plot_parallel_coordinate from optuna.visualization._param_importances import plot_param_importances from optuna.visualization._pareto_front import plot_pareto_front from optuna.visualization._rank import plot_rank from optuna.visualization._slice import plot_slice from optuna.visualization._terminator_improvement import plot_terminator_improvement from optuna.visualization._timeline import plot_timeline from optuna.visualization._utils import is_available __all__ = [ "is_available", "matplotlib", "plot_contour", "plot_edf", "plot_hypervolume_history", "plot_intermediate_values", "plot_optimization_history", "plot_parallel_coordinate", "plot_param_importances", "plot_pareto_front", "plot_slice", "plot_rank", "plot_terminator_improvement", "plot_timeline", ] optuna-4.1.0/optuna/visualization/_contour.py000066400000000000000000000347761471332314300214550ustar00rootroot00000000000000from __future__ import annotations import math from typing import Any from typing import Callable from typing import NamedTuple import warnings import numpy as np from optuna.logging import get_logger from optuna.samplers._base import _CONSTRAINTS_KEY from optuna.study import Study from optuna.study import StudyDirection from optuna.trial import FrozenTrial from optuna.trial import TrialState from optuna.visualization._plotly_imports import _imports from optuna.visualization._utils import _check_plot_args from optuna.visualization._utils import _filter_nonfinite from optuna.visualization._utils import _is_log_scale from optuna.visualization._utils import _is_numerical from optuna.visualization._utils import _is_reverse_scale if _imports.is_successful(): from optuna.visualization._plotly_imports import Contour from optuna.visualization._plotly_imports import go from optuna.visualization._plotly_imports import make_subplots from optuna.visualization._plotly_imports import Scatter from optuna.visualization._utils import COLOR_SCALE _logger = get_logger(__name__) PADDING_RATIO = 0.05 class _AxisInfo(NamedTuple): name: str range: tuple[float, float] is_log: bool is_cat: bool indices: list[str | int | float] values: list[str | float | None] class _SubContourInfo(NamedTuple): xaxis: _AxisInfo yaxis: _AxisInfo z_values: dict[tuple[int, int], float] constraints: list[bool] = [] class _ContourInfo(NamedTuple): sorted_params: list[str] sub_plot_infos: list[list[_SubContourInfo]] reverse_scale: bool target_name: str class _PlotValues(NamedTuple): x: list[Any] y: list[Any] def plot_contour( study: Study, params: list[str] | None = None, *, target: Callable[[FrozenTrial], float] | None = None, target_name: str = "Objective Value", ) -> "go.Figure": """Plot the parameter relationship as contour plot in a study. Note that, if a parameter contains missing values, a trial with missing values is not plotted. Args: study: A :class:`~optuna.study.Study` object whose trials are plotted for their target values. params: Parameter list to visualize. The default is all parameters. target: A function to specify the value to display. If it is :obj:`None` and ``study`` is being used for single-objective optimization, the objective values are plotted. .. note:: Specify this argument if ``study`` is being used for multi-objective optimization. target_name: Target's name to display on the color bar. Returns: A :class:`plotly.graph_objects.Figure` object. .. note:: The colormap is reversed when the ``target`` argument isn't :obj:`None` or ``direction`` of :class:`~optuna.study.Study` is ``minimize``. """ _imports.check() info = _get_contour_info(study, params, target, target_name) return _get_contour_plot(info) def _get_contour_plot(info: _ContourInfo) -> "go.Figure": layout = go.Layout(title="Contour Plot") sorted_params = info.sorted_params sub_plot_infos = info.sub_plot_infos reverse_scale = info.reverse_scale target_name = info.target_name if len(sorted_params) <= 1: return go.Figure(data=[], layout=layout) if len(sorted_params) == 2: x_param = sorted_params[0] y_param = sorted_params[1] sub_plot_info = sub_plot_infos[0][0] sub_plots = _get_contour_subplot(sub_plot_info, reverse_scale, target_name) figure = go.Figure(data=sub_plots, layout=layout) figure.update_xaxes(title_text=x_param, range=sub_plot_info.xaxis.range) figure.update_yaxes(title_text=y_param, range=sub_plot_info.yaxis.range) if sub_plot_info.xaxis.is_cat: figure.update_xaxes(type="category") if sub_plot_info.yaxis.is_cat: figure.update_yaxes(type="category") if sub_plot_info.xaxis.is_log: log_range = [math.log10(p) for p in sub_plot_info.xaxis.range] figure.update_xaxes(range=log_range, type="log") if sub_plot_info.yaxis.is_log: log_range = [math.log10(p) for p in sub_plot_info.yaxis.range] figure.update_yaxes(range=log_range, type="log") else: figure = make_subplots( rows=len(sorted_params), cols=len(sorted_params), shared_xaxes=True, shared_yaxes=True ) figure.update_layout(layout) showscale = True # showscale option only needs to be specified once. for x_i, x_param in enumerate(sorted_params): for y_i, y_param in enumerate(sorted_params): if x_param == y_param: figure.add_trace(go.Scatter(), row=y_i + 1, col=x_i + 1) else: sub_plots = _get_contour_subplot( sub_plot_infos[y_i][x_i], reverse_scale, target_name ) contour = sub_plots[0] scatter = sub_plots[1] contour.update(showscale=showscale) # showscale's default is True. if showscale: showscale = False figure.add_trace(contour, row=y_i + 1, col=x_i + 1) figure.add_trace(scatter, row=y_i + 1, col=x_i + 1) xaxis = sub_plot_infos[y_i][x_i].xaxis yaxis = sub_plot_infos[y_i][x_i].yaxis figure.update_xaxes(range=xaxis.range, row=y_i + 1, col=x_i + 1) figure.update_yaxes(range=yaxis.range, row=y_i + 1, col=x_i + 1) if xaxis.is_cat: figure.update_xaxes(type="category", row=y_i + 1, col=x_i + 1) if yaxis.is_cat: figure.update_yaxes(type="category", row=y_i + 1, col=x_i + 1) if xaxis.is_log: log_range = [math.log10(p) for p in xaxis.range] figure.update_xaxes(range=log_range, type="log", row=y_i + 1, col=x_i + 1) if yaxis.is_log: log_range = [math.log10(p) for p in yaxis.range] figure.update_yaxes(range=log_range, type="log", row=y_i + 1, col=x_i + 1) if x_i == 0: figure.update_yaxes(title_text=y_param, row=y_i + 1, col=x_i + 1) if y_i == len(sorted_params) - 1: figure.update_xaxes(title_text=x_param, row=y_i + 1, col=x_i + 1) return figure def _get_contour_subplot( info: _SubContourInfo, reverse_scale: bool, target_name: str = "Objective Value", ) -> tuple["Contour", "Scatter", "Scatter"]: x_indices = info.xaxis.indices y_indices = info.yaxis.indices if len(x_indices) < 2 or len(y_indices) < 2: return go.Contour(), go.Scatter(), go.Scatter() if len(info.z_values) == 0: warnings.warn( f"Contour plot will not be displayed because `{info.xaxis.name}` and " f"`{info.yaxis.name}` cannot co-exist in `trial.params`." ) return go.Contour(), go.Scatter(), go.Scatter() feasible = _PlotValues([], []) infeasible = _PlotValues([], []) for x_value, y_value, c in zip(info.xaxis.values, info.yaxis.values, info.constraints): if x_value is not None and y_value is not None: if c: feasible.x.append(x_value) feasible.y.append(y_value) else: infeasible.x.append(x_value) infeasible.y.append(y_value) z_values = np.full((len(y_indices), len(x_indices)), np.nan) xys = np.array(list(info.z_values.keys())) zs = np.array(list(info.z_values.values())) z_values[xys[:, 1], xys[:, 0]] = zs contour = go.Contour( x=x_indices, y=y_indices, z=z_values, colorbar={"title": target_name}, colorscale=COLOR_SCALE, connectgaps=True, contours_coloring="heatmap", hoverinfo="none", line_smoothing=1.3, reversescale=reverse_scale, ) return ( contour, _create_scatter(feasible.x, feasible.y, is_feasible=True), _create_scatter(infeasible.x, infeasible.y, is_feasible=False), ) def _create_scatter(x: list[Any], y: list[Any], is_feasible: bool) -> Scatter: edge_color = "Gray" marker_color = "black" if is_feasible else "#cccccc" name = "Feasible Trial" if is_feasible else "Infeasible Trial" return go.Scatter( x=x, y=y, marker={ "line": {"width": 2.0, "color": edge_color}, "color": marker_color, }, mode="markers", name=name, showlegend=False, ) def _get_contour_info( study: Study, params: list[str] | None = None, target: Callable[[FrozenTrial], float] | None = None, target_name: str = "Objective Value", ) -> _ContourInfo: _check_plot_args(study, target, target_name) trials = _filter_nonfinite( study.get_trials(deepcopy=False, states=(TrialState.COMPLETE,)), target=target ) all_params = {p_name for t in trials for p_name in t.params.keys()} if len(trials) == 0: _logger.warning("Your study does not have any completed trials.") sorted_params = [] elif params is None: sorted_params = sorted(all_params) else: if len(params) <= 1: _logger.warning("The length of params must be greater than 1.") for input_p_name in params: if input_p_name not in all_params: raise ValueError("Parameter {} does not exist in your study.".format(input_p_name)) sorted_params = sorted(set(params)) sub_plot_infos: list[list[_SubContourInfo]] if len(sorted_params) == 2: x_param = sorted_params[0] y_param = sorted_params[1] sub_plot_info = _get_contour_subplot_info(study, trials, x_param, y_param, target) sub_plot_infos = [[sub_plot_info]] else: sub_plot_infos = [] for i, y_param in enumerate(sorted_params): sub_plot_infos.append([]) for x_param in sorted_params: sub_plot_info = _get_contour_subplot_info(study, trials, x_param, y_param, target) sub_plot_infos[i].append(sub_plot_info) reverse_scale = _is_reverse_scale(study, target) return _ContourInfo( sorted_params=sorted_params, sub_plot_infos=sub_plot_infos, reverse_scale=reverse_scale, target_name=target_name, ) def _get_contour_subplot_info( study: Study, trials: list[FrozenTrial], x_param: str, y_param: str, target: Callable[[FrozenTrial], float] | None, ) -> _SubContourInfo: xaxis = _get_axis_info(trials, x_param) yaxis = _get_axis_info(trials, y_param) if x_param == y_param: return _SubContourInfo(xaxis=xaxis, yaxis=yaxis, z_values={}) if len(xaxis.indices) < 2: _logger.warning("Param {} unique value length is less than 2.".format(x_param)) return _SubContourInfo(xaxis=xaxis, yaxis=yaxis, z_values={}) if len(yaxis.indices) < 2: _logger.warning("Param {} unique value length is less than 2.".format(y_param)) return _SubContourInfo(xaxis=xaxis, yaxis=yaxis, z_values={}) z_values: dict[tuple[int, int], float] = {} for i, trial in enumerate(trials): if x_param not in trial.params or y_param not in trial.params: continue x_value = xaxis.values[i] y_value = yaxis.values[i] assert x_value is not None assert y_value is not None x_i = xaxis.indices.index(x_value) y_i = yaxis.indices.index(y_value) if target is None: value = trial.value else: value = target(trial) assert value is not None existing = z_values.get((x_i, y_i)) if existing is None or target is not None: # When target function is present, we can't be sure what the z-value # represents and therefore we don't know how to select the best one. z_values[(x_i, y_i)] = value else: z_values[(x_i, y_i)] = ( min(existing, value) if study.direction is StudyDirection.MINIMIZE else max(existing, value) ) return _SubContourInfo( xaxis=xaxis, yaxis=yaxis, z_values=z_values, constraints=[_satisfy_constraints(t) for t in trials], ) def _satisfy_constraints(trial: FrozenTrial) -> bool: constraints = trial.system_attrs.get(_CONSTRAINTS_KEY) return constraints is None or all([x <= 0.0 for x in constraints]) def _get_axis_info(trials: list[FrozenTrial], param_name: str) -> _AxisInfo: values: list[str | float | None] if _is_numerical(trials, param_name): values = [t.params.get(param_name) for t in trials] else: values = [ str(t.params.get(param_name)) if param_name in t.params else None for t in trials ] min_value = min([v for v in values if v is not None]) max_value = max([v for v in values if v is not None]) if _is_log_scale(trials, param_name): min_value = float(min_value) max_value = float(max_value) padding = (math.log10(max_value) - math.log10(min_value)) * PADDING_RATIO min_value = math.pow(10, math.log10(min_value) - padding) max_value = math.pow(10, math.log10(max_value) + padding) is_log = True is_cat = False elif _is_numerical(trials, param_name): min_value = float(min_value) max_value = float(max_value) padding = (max_value - min_value) * PADDING_RATIO min_value = min_value - padding max_value = max_value + padding is_log = False is_cat = False else: unique_values = set(values) span = len(unique_values) - 1 if None in unique_values: span -= 1 padding = span * PADDING_RATIO min_value = -padding max_value = span + padding is_log = False is_cat = True indices = sorted(set([v for v in values if v is not None])) if len(indices) < 2: return _AxisInfo( name=param_name, range=(min_value, max_value), is_log=is_log, is_cat=is_cat, indices=indices, values=values, ) if _is_numerical(trials, param_name): indices.insert(0, min_value) indices.append(max_value) return _AxisInfo( name=param_name, range=(min_value, max_value), is_log=is_log, is_cat=is_cat, indices=indices, values=values, ) optuna-4.1.0/optuna/visualization/_edf.py000066400000000000000000000106021471332314300205000ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Callable from collections.abc import Sequence from typing import cast from typing import NamedTuple import numpy as np from optuna.logging import get_logger from optuna.study import Study from optuna.trial import FrozenTrial from optuna.trial import TrialState from optuna.visualization._plotly_imports import _imports from optuna.visualization._utils import _check_plot_args from optuna.visualization._utils import _filter_nonfinite if _imports.is_successful(): from optuna.visualization._plotly_imports import go _logger = get_logger(__name__) NUM_SAMPLES_X_AXIS = 100 class _EDFLineInfo(NamedTuple): study_name: str y_values: np.ndarray class _EDFInfo(NamedTuple): lines: list[_EDFLineInfo] x_values: np.ndarray def plot_edf( study: Study | Sequence[Study], *, target: Callable[[FrozenTrial], float] | None = None, target_name: str = "Objective Value", ) -> "go.Figure": """Plot the objective value EDF (empirical distribution function) of a study. Note that only the complete trials are considered when plotting the EDF. .. note:: EDF is useful to analyze and improve search spaces. For instance, you can see a practical use case of EDF in the paper `Designing Network Design Spaces `__. .. note:: The plotted EDF assumes that the value of the objective function is in accordance with the uniform distribution over the objective space. Args: study: A target :class:`~optuna.study.Study` object. You can pass multiple studies if you want to compare those EDFs. target: A function to specify the value to display. If it is :obj:`None` and ``study`` is being used for single-objective optimization, the objective values are plotted. .. note:: Specify this argument if ``study`` is being used for multi-objective optimization. target_name: Target's name to display on the axis label. Returns: A :class:`plotly.graph_objects.Figure` object. """ _imports.check() layout = go.Layout( title="Empirical Distribution Function Plot", xaxis={"title": target_name}, yaxis={"title": "Cumulative Probability"}, ) info = _get_edf_info(study, target, target_name) edf_lines = info.lines if len(edf_lines) == 0: return go.Figure(data=[], layout=layout) traces = [] for study_name, y_values in edf_lines: traces.append(go.Scatter(x=info.x_values, y=y_values, name=study_name, mode="lines")) figure = go.Figure(data=traces, layout=layout) figure.update_yaxes(range=[0, 1]) return figure def _get_edf_info( study: Study | Sequence[Study], target: Callable[[FrozenTrial], float] | None = None, target_name: str = "Objective Value", ) -> _EDFInfo: if isinstance(study, Study): studies = [study] else: studies = list(study) _check_plot_args(studies, target, target_name) if len(studies) == 0: _logger.warning("There are no studies.") return _EDFInfo(lines=[], x_values=np.array([])) if target is None: def _target(t: FrozenTrial) -> float: return cast(float, t.value) target = _target study_names = [] all_values: list[np.ndarray] = [] for study in studies: trials = _filter_nonfinite( study.get_trials(deepcopy=False, states=(TrialState.COMPLETE,)), target=target ) values = np.array([target(trial) for trial in trials]) all_values.append(values) study_names.append(study.study_name) if all(len(values) == 0 for values in all_values): _logger.warning("There are no complete trials.") return _EDFInfo(lines=[], x_values=np.array([])) min_x_value = np.min(np.concatenate(all_values)) max_x_value = np.max(np.concatenate(all_values)) x_values = np.linspace(min_x_value, max_x_value, NUM_SAMPLES_X_AXIS) edf_line_info_list = [] for study_name, values in zip(study_names, all_values): y_values = np.sum(values[:, np.newaxis] <= x_values, axis=0) / values.size edf_line_info_list.append(_EDFLineInfo(study_name=study_name, y_values=y_values)) return _EDFInfo(lines=edf_line_info_list, x_values=x_values) optuna-4.1.0/optuna/visualization/_hypervolume_history.py000066400000000000000000000106601471332314300241060ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Sequence from typing import NamedTuple import numpy as np from optuna._experimental import experimental_func from optuna._hypervolume import compute_hypervolume from optuna.logging import get_logger from optuna.samplers._base import _CONSTRAINTS_KEY from optuna.study import Study from optuna.study._multi_objective import _dominates from optuna.study._study_direction import StudyDirection from optuna.trial import FrozenTrial from optuna.trial import TrialState from optuna.visualization._plotly_imports import _imports if _imports.is_successful(): from optuna.visualization._plotly_imports import go _logger = get_logger(__name__) class _HypervolumeHistoryInfo(NamedTuple): trial_numbers: list[int] values: list[float] @experimental_func("3.3.0") def plot_hypervolume_history( study: Study, reference_point: Sequence[float], ) -> "go.Figure": """Plot hypervolume history of all trials in a study. Args: study: A :class:`~optuna.study.Study` object whose trials are plotted for their hypervolumes. The number of objectives must be 2 or more. reference_point: A reference point to use for hypervolume computation. The dimension of the reference point must be the same as the number of objectives. Returns: A :class:`plotly.graph_objects.Figure` object. """ _imports.check() if not study._is_multi_objective(): raise ValueError( "Study must be multi-objective. For single-objective optimization, " "please use plot_optimization_history instead." ) if len(reference_point) != len(study.directions): raise ValueError( "The dimension of the reference point must be the same as the number of objectives." ) info = _get_hypervolume_history_info(study, np.asarray(reference_point, dtype=np.float64)) return _get_hypervolume_history_plot(info) def _get_hypervolume_history_plot( info: _HypervolumeHistoryInfo, ) -> "go.Figure": layout = go.Layout( title="Hypervolume History Plot", xaxis={"title": "Trial"}, yaxis={"title": "Hypervolume"}, ) data = go.Scatter( x=info.trial_numbers, y=info.values, mode="lines+markers", ) return go.Figure(data=data, layout=layout) def _get_hypervolume_history_info( study: Study, reference_point: np.ndarray, ) -> _HypervolumeHistoryInfo: completed_trials = study.get_trials(deepcopy=False, states=(TrialState.COMPLETE,)) if len(completed_trials) == 0: _logger.warning("Your study does not have any completed trials.") # Our hypervolume computation module assumes that all objectives are minimized. # Here we transform the objective values and the reference point. signs = np.asarray([1 if d == StudyDirection.MINIMIZE else -1 for d in study.directions]) minimization_reference_point = signs * reference_point # Only feasible trials are considered in hypervolume computation. trial_numbers = [] values = [] best_trials: list[FrozenTrial] = [] hypervolume = 0.0 for trial in completed_trials: trial_numbers.append(trial.number) has_constraints = _CONSTRAINTS_KEY in trial.system_attrs if has_constraints: constraints_values = trial.system_attrs[_CONSTRAINTS_KEY] if any(map(lambda x: x > 0.0, constraints_values)): # The trial is infeasible. values.append(hypervolume) continue if any(map(lambda t: _dominates(t, trial, study.directions), best_trials)): # The trial is not on the Pareto front. values.append(hypervolume) continue best_trials = list( filter(lambda t: not _dominates(trial, t, study.directions), best_trials) ) + [trial] loss_vals = np.asarray( list( filter( lambda v: (v <= minimization_reference_point).all(), [signs * trial.values for trial in best_trials], ) ) ) if loss_vals.size > 0: hypervolume = compute_hypervolume(loss_vals, minimization_reference_point) values.append(hypervolume) if len(best_trials) == 0: _logger.warning("Your study does not have any feasible trials.") return _HypervolumeHistoryInfo(trial_numbers, values) optuna-4.1.0/optuna/visualization/_intermediate_values.py000066400000000000000000000056341471332314300240040ustar00rootroot00000000000000from __future__ import annotations from typing import NamedTuple from optuna.logging import get_logger from optuna.samplers._base import _CONSTRAINTS_KEY from optuna.study import Study from optuna.trial import FrozenTrial from optuna.trial import TrialState from optuna.visualization._plotly_imports import _imports if _imports.is_successful(): from optuna.visualization._plotly_imports import go _logger = get_logger(__name__) class _TrialInfo(NamedTuple): trial_number: int sorted_intermediate_values: list[tuple[int, float]] feasible: bool class _IntermediatePlotInfo(NamedTuple): trial_infos: list[_TrialInfo] def _get_intermediate_plot_info(study: Study) -> _IntermediatePlotInfo: trials = study.get_trials( deepcopy=False, states=(TrialState.PRUNED, TrialState.COMPLETE, TrialState.RUNNING) ) def _satisfies_constraints(trial: FrozenTrial) -> bool: constraints = trial.system_attrs.get(_CONSTRAINTS_KEY) return constraints is None or all([x <= 0.0 for x in constraints]) trial_infos = [ _TrialInfo( trial.number, sorted(trial.intermediate_values.items()), _satisfies_constraints(trial) ) for trial in trials if len(trial.intermediate_values) > 0 ] if len(trials) == 0: _logger.warning("Study instance does not contain trials.") elif len(trial_infos) == 0: _logger.warning( "You need to set up the pruning feature to utilize `plot_intermediate_values()`" ) return _IntermediatePlotInfo(trial_infos) def plot_intermediate_values(study: Study) -> "go.Figure": """Plot intermediate values of all trials in a study. Args: study: A :class:`~optuna.study.Study` object whose trials are plotted for their intermediate values. Returns: A :class:`plotly.graph_objects.Figure` object. """ _imports.check() return _get_intermediate_plot(_get_intermediate_plot_info(study)) def _get_intermediate_plot(info: _IntermediatePlotInfo) -> "go.Figure": layout = go.Layout( title="Intermediate Values Plot", xaxis={"title": "Step"}, yaxis={"title": "Intermediate Value"}, showlegend=False, ) trial_infos = info.trial_infos if len(trial_infos) == 0: return go.Figure(data=[], layout=layout) default_marker = {"maxdisplayed": 10} traces = [ go.Scatter( x=tuple((x for x, _ in tinfo.sorted_intermediate_values)), y=tuple((y for _, y in tinfo.sorted_intermediate_values)), mode="lines+markers", marker=( default_marker if tinfo.feasible else {**default_marker, "color": "#CCCCCC"} # type: ignore[dict-item] ), name="Trial{}".format(tinfo.trial_number), ) for tinfo in trial_infos ] return go.Figure(data=traces, layout=layout) optuna-4.1.0/optuna/visualization/_optimization_history.py000066400000000000000000000260041471332314300242540ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Callable from collections.abc import Sequence from enum import Enum import math from typing import cast from typing import NamedTuple import numpy as np from optuna.logging import get_logger from optuna.samplers._base import _CONSTRAINTS_KEY from optuna.study import Study from optuna.study._study_direction import StudyDirection from optuna.trial import FrozenTrial from optuna.trial import TrialState from optuna.visualization._plotly_imports import _imports from optuna.visualization._utils import _check_plot_args if _imports.is_successful(): from optuna.visualization._plotly_imports import go _logger = get_logger(__name__) class _ValueState(Enum): Feasible = 0 Infeasible = 1 Incomplete = 2 class _ValuesInfo(NamedTuple): values: list[float] stds: list[float] | None label_name: str states: list[_ValueState] class _OptimizationHistoryInfo(NamedTuple): trial_numbers: list[int] values_info: _ValuesInfo best_values_info: _ValuesInfo | None def _get_optimization_history_info_list( study: Study | Sequence[Study], target: Callable[[FrozenTrial], float] | None, target_name: str, error_bar: bool, ) -> list[_OptimizationHistoryInfo]: _check_plot_args(study, target, target_name) if isinstance(study, Study): studies = [study] else: studies = list(study) info_list: list[_OptimizationHistoryInfo] = [] for study in studies: trials = study.get_trials() label_name = target_name if len(studies) == 1 else f"{target_name} of {study.study_name}" values = [] value_states = [] for trial in trials: if trial.state != TrialState.COMPLETE: values.append(float("nan")) value_states.append(_ValueState.Incomplete) continue constraints = trial.system_attrs.get(_CONSTRAINTS_KEY) if constraints is None or all([x <= 0.0 for x in constraints]): value_states.append(_ValueState.Feasible) else: value_states.append(_ValueState.Infeasible) if target is not None: values.append(target(trial)) else: values.append(cast(float, trial.value)) if target is not None: # We don't calculate best for user-defined target function since we cannot tell # which direction is better. best_values_info: _ValuesInfo | None = None else: feasible_best_values = [] if study.direction == StudyDirection.MINIMIZE: feasible_best_values = [ v if s == _ValueState.Feasible else float("inf") for v, s in zip(values, value_states) ] best_values = list(np.minimum.accumulate(feasible_best_values)) else: feasible_best_values = [ v if s == _ValueState.Feasible else -float("inf") for v, s in zip(values, value_states) ] best_values = list(np.maximum.accumulate(feasible_best_values)) best_label_name = ( "Best Value" if len(studies) == 1 else f"Best Value of {study.study_name}" ) best_values_info = _ValuesInfo(best_values, None, best_label_name, value_states) info_list.append( _OptimizationHistoryInfo( trial_numbers=[t.number for t in trials], values_info=_ValuesInfo(values, None, label_name, value_states), best_values_info=best_values_info, ) ) if len(info_list) == 0: _logger.warning("There are no studies.") feasible_trial_count = sum( info.values_info.states.count(_ValueState.Feasible) for info in info_list ) infeasible_trial_count = sum( info.values_info.states.count(_ValueState.Infeasible) for info in info_list ) if feasible_trial_count + infeasible_trial_count == 0: _logger.warning("There are no complete trials.") info_list.clear() if not error_bar: return info_list # When error_bar=True, a list of 0 or 1 element is returned. if len(info_list) == 0: return [] if feasible_trial_count == 0: _logger.warning("There are no feasible trials.") return [] all_trial_numbers = [number for info in info_list for number in info.trial_numbers] max_num_trial = max(all_trial_numbers) + 1 def _aggregate(label_name: str, use_best_value: bool) -> tuple[list[int], _ValuesInfo]: # Calculate mean and std of values for each trial number. values: list[list[float]] = [[] for _ in range(max_num_trial)] states: list[list[_ValueState]] = [[] for _ in range(max_num_trial)] assert info_list is not None for trial_numbers, values_info, best_values_info in info_list: if use_best_value: assert best_values_info is not None values_info = best_values_info for n, v, s in zip(trial_numbers, values_info.values, values_info.states): if not math.isinf(v): if not use_best_value and s == _ValueState.Feasible: values[n].append(v) elif use_best_value: values[n].append(v) states[n].append(s) trial_numbers_union: list[int] = [] value_states: list[_ValueState] = [] value_means: list[float] = [] value_stds: list[float] = [] for i in range(max_num_trial): if len(states[i]) > 0 and _ValueState.Feasible in states[i]: value_states.append(_ValueState.Feasible) trial_numbers_union.append(i) value_means.append(np.mean(values[i]).item()) value_stds.append(np.std(values[i]).item()) else: value_states.append(_ValueState.Infeasible) return trial_numbers_union, _ValuesInfo(value_means, value_stds, label_name, value_states) eb_trial_numbers, eb_values_info = _aggregate(target_name, False) eb_best_values_info: _ValuesInfo | None = None if target is None: _, eb_best_values_info = _aggregate("Best Value", True) return [_OptimizationHistoryInfo(eb_trial_numbers, eb_values_info, eb_best_values_info)] def plot_optimization_history( study: Study | Sequence[Study], *, target: Callable[[FrozenTrial], float] | None = None, target_name: str = "Objective Value", error_bar: bool = False, ) -> "go.Figure": """Plot optimization history of all trials in a study. Args: study: A :class:`~optuna.study.Study` object whose trials are plotted for their target values. You can pass multiple studies if you want to compare those optimization histories. target: A function to specify the value to display. If it is :obj:`None` and ``study`` is being used for single-objective optimization, the objective values are plotted. .. note:: Specify this argument if ``study`` is being used for multi-objective optimization. target_name: Target's name to display on the axis label and the legend. error_bar: A flag to show the error bar. Returns: A :class:`plotly.graph_objects.Figure` object. """ _imports.check() info_list = _get_optimization_history_info_list(study, target, target_name, error_bar) return _get_optimization_history_plot(info_list, target_name) def _get_optimization_history_plot( info_list: list[_OptimizationHistoryInfo], target_name: str, ) -> "go.Figure": layout = go.Layout( title="Optimization History Plot", xaxis={"title": "Trial"}, yaxis={"title": target_name}, ) traces = [] for trial_numbers, values_info, best_values_info in info_list: infeasible_trial_numbers = [ n for n, s in zip(trial_numbers, values_info.states) if s == _ValueState.Infeasible ] if values_info.stds is None: error_y = None feasible_trial_numbers = [ num for num, s in zip(trial_numbers, values_info.states) if s == _ValueState.Feasible ] feasible_trial_values = [] for num in feasible_trial_numbers: feasible_trial_values.append(values_info.values[num]) infeasible_trial_values = [] for num in infeasible_trial_numbers: infeasible_trial_values.append(values_info.values[num]) else: if ( _ValueState.Infeasible in values_info.states or _ValueState.Incomplete in values_info.states ): _logger.warning( "Your study contains infeasible trials. " "In optimization history plot, " "error bars are calculated for only feasible trial values." ) error_y = {"type": "data", "array": values_info.stds, "visible": True} feasible_trial_numbers = trial_numbers feasible_trial_values = values_info.values infeasible_trial_values = [] traces.append( go.Scatter( x=feasible_trial_numbers, y=feasible_trial_values, error_y=error_y, mode="markers", name=values_info.label_name, ) ) if best_values_info is not None: traces.append( go.Scatter( x=trial_numbers, y=best_values_info.values, name=best_values_info.label_name, mode="lines", ) ) if best_values_info.stds is not None: upper = np.array(best_values_info.values) + np.array(best_values_info.stds) traces.append( go.Scatter( x=trial_numbers, y=upper, mode="lines", line=dict(width=0.01), showlegend=False, ) ) lower = np.array(best_values_info.values) - np.array(best_values_info.stds) traces.append( go.Scatter( x=trial_numbers, y=lower, mode="none", showlegend=False, fill="tonexty", fillcolor="rgba(255,0,0,0.2)", ) ) traces.append( go.Scatter( x=infeasible_trial_numbers, y=infeasible_trial_values, error_y=error_y, mode="markers", name="Infeasible Trial", marker={"color": "#cccccc"}, showlegend=False, ) ) return go.Figure(data=traces, layout=layout) optuna-4.1.0/optuna/visualization/_parallel_coordinate.py000066400000000000000000000240541471332314300237530ustar00rootroot00000000000000from __future__ import annotations from collections import defaultdict import math from typing import Any from typing import Callable from typing import cast from typing import NamedTuple import numpy as np from optuna.distributions import CategoricalDistribution from optuna.logging import get_logger from optuna.study import Study from optuna.trial import FrozenTrial from optuna.trial import TrialState from optuna.visualization._plotly_imports import _imports from optuna.visualization._utils import _check_plot_args from optuna.visualization._utils import _filter_nonfinite from optuna.visualization._utils import _get_skipped_trial_numbers from optuna.visualization._utils import _is_log_scale from optuna.visualization._utils import _is_numerical from optuna.visualization._utils import _is_reverse_scale if _imports.is_successful(): from optuna.visualization._plotly_imports import go from optuna.visualization._utils import COLOR_SCALE _logger = get_logger(__name__) class _DimensionInfo(NamedTuple): label: str values: tuple[float, ...] range: tuple[float, float] is_log: bool is_cat: bool tickvals: list[int | float] ticktext: list[str] class _ParallelCoordinateInfo(NamedTuple): dim_objective: _DimensionInfo dims_params: list[_DimensionInfo] reverse_scale: bool target_name: str def plot_parallel_coordinate( study: Study, params: list[str] | None = None, *, target: Callable[[FrozenTrial], float] | None = None, target_name: str = "Objective Value", ) -> "go.Figure": """Plot the high-dimensional parameter relationships in a study. Note that, if a parameter contains missing values, a trial with missing values is not plotted. Args: study: A :class:`~optuna.study.Study` object whose trials are plotted for their target values. params: Parameter list to visualize. The default is all parameters. target: A function to specify the value to display. If it is :obj:`None` and ``study`` is being used for single-objective optimization, the objective values are plotted. .. note:: Specify this argument if ``study`` is being used for multi-objective optimization. target_name: Target's name to display on the axis label and the legend. Returns: A :class:`plotly.graph_objects.Figure` object. .. note:: The colormap is reversed when the ``target`` argument isn't :obj:`None` or ``direction`` of :class:`~optuna.study.Study` is ``minimize``. """ _imports.check() info = _get_parallel_coordinate_info(study, params, target, target_name) return _get_parallel_coordinate_plot(info) def _get_parallel_coordinate_plot(info: _ParallelCoordinateInfo) -> "go.Figure": layout = go.Layout(title="Parallel Coordinate Plot") if len(info.dims_params) == 0 or len(info.dim_objective.values) == 0: return go.Figure(data=[], layout=layout) dims = _get_dims_from_info(info) reverse_scale = info.reverse_scale target_name = info.target_name traces = [ go.Parcoords( dimensions=dims, labelangle=30, labelside="bottom", line={ "color": dims[0]["values"], "colorscale": COLOR_SCALE, "colorbar": {"title": target_name}, "showscale": True, "reversescale": reverse_scale, }, ) ] figure = go.Figure(data=traces, layout=layout) return figure def _get_parallel_coordinate_info( study: Study, params: list[str] | None = None, target: Callable[[FrozenTrial], float] | None = None, target_name: str = "Objective Value", ) -> _ParallelCoordinateInfo: _check_plot_args(study, target, target_name) reverse_scale = _is_reverse_scale(study, target) trials = _filter_nonfinite( study.get_trials(deepcopy=False, states=(TrialState.COMPLETE,)), target=target ) all_params = {p_name for t in trials for p_name in t.params.keys()} if params is not None: for input_p_name in params: if input_p_name not in all_params: raise ValueError("Parameter {} does not exist in your study.".format(input_p_name)) all_params = set(params) sorted_params = sorted(all_params) if target is None: def _target(t: FrozenTrial) -> float: return cast(float, t.value) target = _target skipped_trial_numbers = _get_skipped_trial_numbers(trials, sorted_params) objectives = tuple([target(t) for t in trials if t.number not in skipped_trial_numbers]) # The value of (0, 0) is a dummy range. It is ignored when we plot. objective_range = (min(objectives), max(objectives)) if len(objectives) > 0 else (0, 0) dim_objective = _DimensionInfo( label=target_name, values=objectives, range=objective_range, is_log=False, is_cat=False, tickvals=[], ticktext=[], ) if len(trials) == 0: _logger.warning("Your study does not have any completed trials.") return _ParallelCoordinateInfo( dim_objective=dim_objective, dims_params=[], reverse_scale=reverse_scale, target_name=target_name, ) if len(objectives) == 0: _logger.warning("Your study has only completed trials with missing parameters.") return _ParallelCoordinateInfo( dim_objective=dim_objective, dims_params=[], reverse_scale=reverse_scale, target_name=target_name, ) numeric_cat_params_indices: list[int] = [] dims = [] for dim_index, p_name in enumerate(sorted_params, start=1): values = [] is_categorical = False for t in trials: if t.number in skipped_trial_numbers: continue if p_name in t.params: values.append(t.params[p_name]) is_categorical |= isinstance(t.distributions[p_name], CategoricalDistribution) if _is_log_scale(trials, p_name): values = [math.log10(v) for v in values] min_value = min(values) max_value = max(values) tickvals: list[int | float] = list( range(math.ceil(min_value), math.floor(max_value) + 1) ) if min_value not in tickvals: tickvals = [min_value] + tickvals if max_value not in tickvals: tickvals = tickvals + [max_value] dim = _DimensionInfo( label=_truncate_label(p_name), values=tuple(values), range=(min_value, max_value), is_log=True, is_cat=False, tickvals=tickvals, ticktext=["{:.3g}".format(math.pow(10, x)) for x in tickvals], ) elif is_categorical: vocab: defaultdict[int | str, int] = defaultdict(lambda: len(vocab)) ticktext: list[str] if _is_numerical(trials, p_name): _ = [vocab[v] for v in sorted(values)] values = [vocab[v] for v in values] ticktext = [str(v) for v in list(sorted(vocab.keys()))] numeric_cat_params_indices.append(dim_index) else: values = [vocab[v] for v in values] ticktext = [str(v) for v in list(sorted(vocab.keys(), key=lambda x: vocab[x]))] dim = _DimensionInfo( label=_truncate_label(p_name), values=tuple(values), range=(min(values), max(values)), is_log=False, is_cat=True, tickvals=list(range(len(vocab))), ticktext=ticktext, ) else: dim = _DimensionInfo( label=_truncate_label(p_name), values=tuple(values), range=(min(values), max(values)), is_log=False, is_cat=False, tickvals=[], ticktext=[], ) dims.append(dim) if numeric_cat_params_indices: dims.insert(0, dim_objective) # np.lexsort consumes the sort keys the order from back to front. # So the values of parameters have to be reversed the order. idx = np.lexsort([dims[index].values for index in numeric_cat_params_indices][::-1]) updated_dims = [] for dim in dims: # Since the values are mapped to other categories by the index, # the index will be swapped according to the sorted index of numeric params. updated_dims.append( _DimensionInfo( label=dim.label, values=tuple(np.array(dim.values)[idx]), range=dim.range, is_log=dim.is_log, is_cat=dim.is_cat, tickvals=dim.tickvals, ticktext=dim.ticktext, ) ) dim_objective = updated_dims[0] dims = updated_dims[1:] return _ParallelCoordinateInfo( dim_objective=dim_objective, dims_params=dims, reverse_scale=reverse_scale, target_name=target_name, ) def _get_dims_from_info(info: _ParallelCoordinateInfo) -> list[dict[str, Any]]: dims = [ { "label": info.dim_objective.label, "values": info.dim_objective.values, "range": info.dim_objective.range, } ] for dim in info.dims_params: if dim.is_log or dim.is_cat: dims.append( { "label": dim.label, "values": dim.values, "range": dim.range, "tickvals": dim.tickvals, "ticktext": dim.ticktext, } ) else: dims.append({"label": dim.label, "values": dim.values, "range": dim.range}) return dims def _truncate_label(label: str) -> str: return label if len(label) < 20 else "{}...".format(label[:17]) optuna-4.1.0/optuna/visualization/_param_importances.py000066400000000000000000000163771471332314300234650ustar00rootroot00000000000000from __future__ import annotations from typing import Callable from typing import NamedTuple import optuna from optuna.distributions import BaseDistribution from optuna.importance._base import BaseImportanceEvaluator from optuna.logging import get_logger from optuna.study import Study from optuna.trial import FrozenTrial from optuna.trial import TrialState from optuna.visualization._plotly_imports import _imports from optuna.visualization._utils import _check_plot_args from optuna.visualization._utils import _filter_nonfinite if _imports.is_successful(): from optuna.visualization._plotly_imports import go logger = get_logger(__name__) class _ImportancesInfo(NamedTuple): importance_values: list[float] param_names: list[str] importance_labels: list[str] target_name: str def _get_importances_info( study: Study, evaluator: BaseImportanceEvaluator | None, params: list[str] | None, target: Callable[[FrozenTrial], float] | None, target_name: str, ) -> _ImportancesInfo: _check_plot_args(study, target, target_name) trials = _filter_nonfinite( study.get_trials(deepcopy=False, states=(TrialState.COMPLETE,)), target=target ) if len(trials) == 0: logger.warning("Study instance does not contain completed trials.") return _ImportancesInfo( importance_values=[], param_names=[], importance_labels=[], target_name=target_name, ) importances = optuna.importance.get_param_importances( study, evaluator=evaluator, params=params, target=target ) importances = dict(reversed(list(importances.items()))) importance_values = list(importances.values()) param_names = list(importances.keys()) importance_labels = [f"{val:.2f}" if val >= 0.01 else "<0.01" for val in importance_values] return _ImportancesInfo( importance_values=importance_values, param_names=param_names, importance_labels=importance_labels, target_name=target_name, ) def _get_importances_infos( study: Study, evaluator: BaseImportanceEvaluator | None, params: list[str] | None, target: Callable[[FrozenTrial], float] | None, target_name: str, ) -> tuple[_ImportancesInfo, ...]: metric_names = study.metric_names if target or not study._is_multi_objective(): target_name = metric_names[0] if metric_names is not None and not target else target_name importances_infos: tuple[_ImportancesInfo, ...] = ( _get_importances_info( study, evaluator, params, target=target, target_name=target_name, ), ) else: n_objectives = len(study.directions) target_names = ( metric_names if metric_names is not None else (f"{target_name} {objective_id}" for objective_id in range(n_objectives)) ) importances_infos = tuple( _get_importances_info( study, evaluator, params, target=lambda t: t.values[objective_id], target_name=target_name, ) for objective_id, target_name in enumerate(target_names) ) return importances_infos def plot_param_importances( study: Study, evaluator: BaseImportanceEvaluator | None = None, params: list[str] | None = None, *, target: Callable[[FrozenTrial], float] | None = None, target_name: str = "Objective Value", ) -> "go.Figure": """Plot hyperparameter importances. .. seealso:: This function visualizes the results of :func:`optuna.importance.get_param_importances`. Args: study: An optimized study. evaluator: An importance evaluator object that specifies which algorithm to base the importance assessment on. Defaults to :class:`~optuna.importance.FanovaImportanceEvaluator`. .. note:: :class:`~optuna.importance.FanovaImportanceEvaluator` takes over 1 minute when given a study that contains 1000+ trials. We published `optuna-fast-fanova `__ library, that is a Cython accelerated fANOVA implementation. By using it, you can get hyperparameter importances within a few seconds. params: A list of names of parameters to assess. If :obj:`None`, all parameters that are present in all of the completed trials are assessed. target: A function to specify the value to display. If it is :obj:`None` and ``study`` is being used for single-objective optimization, the objective values are plotted. For multi-objective optimization, all objectives will be plotted if ``target`` is :obj:`None`. .. note:: This argument can be used to specify which objective to plot if ``study`` is being used for multi-objective optimization. For example, to get only the hyperparameter importance of the first objective, use ``target=lambda t: t.values[0]`` for the target parameter. target_name: Target's name to display on the legend. Names set via :meth:`~optuna.study.Study.set_metric_names` will be used if ``target`` is :obj:`None`, overriding this argument. Returns: A :class:`plotly.graph_objects.Figure` object. """ _imports.check() importances_infos = _get_importances_infos(study, evaluator, params, target, target_name) return _get_importances_plot(importances_infos, study) def _get_importances_plot(infos: tuple[_ImportancesInfo, ...], study: Study) -> "go.Figure": layout = go.Layout( title="Hyperparameter Importances", xaxis={"title": "Hyperparameter Importance"}, yaxis={"title": "Hyperparameter"}, ) data: list[go.Bar] = [] for info in infos: if not info.importance_values: continue data.append( go.Bar( x=info.importance_values, y=info.param_names, name=info.target_name, text=info.importance_labels, textposition="outside", cliponaxis=False, # Ensure text is not clipped. hovertemplate=_get_hover_template(info, study), orientation="h", ) ) return go.Figure(data, layout) def _get_distribution(param_name: str, study: Study) -> BaseDistribution: for trial in study.trials: if param_name in trial.distributions: return trial.distributions[param_name] assert False def _make_hovertext(param_name: str, importance: float, study: Study) -> str: return "{} ({}): {}".format( param_name, _get_distribution(param_name, study).__class__.__name__, importance ) def _get_hover_template(importances_info: _ImportancesInfo, study: Study) -> list[str]: return [ _make_hovertext(param_name, importance, study) for param_name, importance in zip( importances_info.param_names, importances_info.importance_values ) ] optuna-4.1.0/optuna/visualization/_pareto_front.py000066400000000000000000000373741471332314300224630ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Callable from collections.abc import Sequence from typing import Any from typing import NamedTuple import warnings import optuna from optuna import _deprecated from optuna.samplers._base import _CONSTRAINTS_KEY from optuna.study import Study from optuna.study._multi_objective import _get_pareto_front_trials_by_trials from optuna.trial import FrozenTrial from optuna.trial import TrialState from optuna.visualization._plotly_imports import _imports from optuna.visualization._utils import _make_hovertext if _imports.is_successful(): from optuna.visualization._plotly_imports import go _logger = optuna.logging.get_logger(__name__) class _ParetoFrontInfo(NamedTuple): n_targets: int target_names: list[str] best_trials_with_values: list[tuple[FrozenTrial, list[float]]] non_best_trials_with_values: list[tuple[FrozenTrial, list[float]]] infeasible_trials_with_values: list[tuple[FrozenTrial, list[float]]] axis_order: list[int] include_dominated_trials: bool has_constraints: bool def plot_pareto_front( study: Study, *, target_names: list[str] | None = None, include_dominated_trials: bool = True, axis_order: list[int] | None = None, constraints_func: Callable[[FrozenTrial], Sequence[float]] | None = None, targets: Callable[[FrozenTrial], Sequence[float]] | None = None, ) -> "go.Figure": """Plot the Pareto front of a study. .. seealso:: Please refer to :ref:`multi_objective` for the tutorial of the Pareto front visualization. Args: study: A :class:`~optuna.study.Study` object whose trials are plotted for their objective values. The number of objectives must be either 2 or 3 when ``targets`` is :obj:`None`. target_names: Objective name list used as the axis titles. If :obj:`None` is specified, "Objective {objective_index}" is used instead. If ``targets`` is specified for a study that does not contain any completed trial, ``target_name`` must be specified. include_dominated_trials: A flag to include all dominated trial's objective values. axis_order: A list of indices indicating the axis order. If :obj:`None` is specified, default order is used. ``axis_order`` and ``targets`` cannot be used at the same time. .. warning:: Deprecated in v3.0.0. This feature will be removed in the future. The removal of this feature is currently scheduled for v5.0.0, but this schedule is subject to change. See https://github.com/optuna/optuna/releases/tag/v3.0.0. constraints_func: An optional function that computes the objective constraints. It must take a :class:`~optuna.trial.FrozenTrial` and return the constraints. The return value must be a sequence of :obj:`float` s. A value strictly larger than 0 means that a constraint is violated. A value equal to or smaller than 0 is considered feasible. This specification is the same as in, for example, :class:`~optuna.samplers.NSGAIISampler`. If given, trials are classified into three categories: feasible and best, feasible but non-best, and infeasible. Categories are shown in different colors. Here, whether a trial is best (on Pareto front) or not is determined ignoring all infeasible trials. .. warning:: Deprecated in v4.0.0. This feature will be removed in the future. The removal of this feature is currently scheduled for v6.0.0, but this schedule is subject to change. See https://github.com/optuna/optuna/releases/tag/v4.0.0. targets: A function that returns targets values to display. The argument to this function is :class:`~optuna.trial.FrozenTrial`. ``axis_order`` and ``targets`` cannot be used at the same time. If ``study.n_objectives`` is neither 2 nor 3, ``targets`` must be specified. .. note:: Added in v3.0.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v3.0.0. Returns: A :class:`plotly.graph_objects.Figure` object. """ _imports.check() info = _get_pareto_front_info( study, target_names, include_dominated_trials, axis_order, constraints_func, targets ) return _get_pareto_front_plot(info) def _get_pareto_front_plot(info: _ParetoFrontInfo) -> "go.Figure": include_dominated_trials = info.include_dominated_trials has_constraints = info.has_constraints if not has_constraints: data = [ _make_scatter_object( info.n_targets, info.axis_order, include_dominated_trials, info.non_best_trials_with_values, hovertemplate="%{text}Trial", dominated_trials=True, ), _make_scatter_object( info.n_targets, info.axis_order, include_dominated_trials, info.best_trials_with_values, hovertemplate="%{text}Best Trial", dominated_trials=False, ), ] else: data = [ _make_scatter_object( info.n_targets, info.axis_order, include_dominated_trials, info.infeasible_trials_with_values, hovertemplate="%{text}Infeasible Trial", infeasible=True, ), _make_scatter_object( info.n_targets, info.axis_order, include_dominated_trials, info.non_best_trials_with_values, hovertemplate="%{text}Feasible Trial", dominated_trials=True, ), _make_scatter_object( info.n_targets, info.axis_order, include_dominated_trials, info.best_trials_with_values, hovertemplate="%{text}Best Trial", dominated_trials=False, ), ] if info.n_targets == 2: layout = go.Layout( title="Pareto-front Plot", xaxis_title=info.target_names[info.axis_order[0]], yaxis_title=info.target_names[info.axis_order[1]], ) else: layout = go.Layout( title="Pareto-front Plot", scene={ "xaxis_title": info.target_names[info.axis_order[0]], "yaxis_title": info.target_names[info.axis_order[1]], "zaxis_title": info.target_names[info.axis_order[2]], }, ) return go.Figure(data=data, layout=layout) def _get_pareto_front_info( study: Study, target_names: list[str] | None = None, include_dominated_trials: bool = True, axis_order: list[int] | None = None, constraints_func: Callable[[FrozenTrial], Sequence[float]] | None = None, targets: Callable[[FrozenTrial], Sequence[float]] | None = None, ) -> _ParetoFrontInfo: if axis_order is not None: msg = _deprecated._DEPRECATION_WARNING_TEMPLATE.format( name="`axis_order`", d_ver="3.0.0", r_ver="5.0.0" ) warnings.warn(msg, FutureWarning) if constraints_func is not None: msg = _deprecated._DEPRECATION_WARNING_TEMPLATE.format( name="`constraints_func`", d_ver="4.0.0", r_ver="6.0.0" ) warnings.warn(msg, FutureWarning) if targets is not None and axis_order is not None: raise ValueError( "Using both `targets` and `axis_order` is not supported. " "Use either `targets` or `axis_order`." ) feasible_trials = [] infeasible_trials = [] has_constraints = False for trial in study.get_trials(deepcopy=False, states=(TrialState.COMPLETE,)): if constraints_func is not None: # NOTE(nabenabe0928): This part is deprecated. has_constraints = True if all(map(lambda x: x <= 0.0, constraints_func(trial))): feasible_trials.append(trial) else: infeasible_trials.append(trial) continue constraints = trial.system_attrs.get(_CONSTRAINTS_KEY) has_constraints |= constraints is not None if constraints is None or all(x <= 0.0 for x in constraints): feasible_trials.append(trial) else: infeasible_trials.append(trial) best_trials = _get_pareto_front_trials_by_trials(feasible_trials, study.directions) if include_dominated_trials: non_best_trials = _get_non_pareto_front_trials(feasible_trials, best_trials) else: non_best_trials = [] if len(best_trials) == 0: what_trial = "completed" if has_constraints else "completed and feasible" _logger.warning(f"Your study does not have any {what_trial} trials. ") _targets = targets if _targets is None: if len(study.directions) in (2, 3): _targets = _targets_default else: raise ValueError( "`plot_pareto_front` function only supports 2 or 3 objective" " studies when using `targets` is `None`. Please use `targets`" " if your objective studies have more than 3 objectives." ) def _make_trials_with_values( trials: list[FrozenTrial], targets: Callable[[FrozenTrial], Sequence[float]], ) -> list[tuple[FrozenTrial, list[float]]]: target_values = [targets(trial) for trial in trials] for v in target_values: if not isinstance(v, Sequence): raise ValueError( "`targets` should return a sequence of target values." " your `targets` returns {}".format(type(v)) ) return [(trial, list(v)) for trial, v in zip(trials, target_values)] best_trials_with_values = _make_trials_with_values(best_trials, _targets) non_best_trials_with_values = _make_trials_with_values(non_best_trials, _targets) infeasible_trials_with_values = _make_trials_with_values(infeasible_trials, _targets) def _infer_n_targets( trials_with_values: Sequence[tuple[FrozenTrial, Sequence[float]]] ) -> int | None: if len(trials_with_values) > 0: return len(trials_with_values[0][1]) return None # Check for `non_best_trials_with_values` can be skipped, because if `best_trials_with_values` # is empty, then `non_best_trials_with_values` will also be empty. n_targets = _infer_n_targets(best_trials_with_values) or _infer_n_targets( infeasible_trials_with_values ) if n_targets is None: if target_names is not None: n_targets = len(target_names) elif targets is None: n_targets = len(study.directions) else: raise ValueError( "If `targets` is specified for empty studies, `target_names` must be specified." ) if n_targets not in (2, 3): raise ValueError( "`plot_pareto_front` function only supports 2 or 3 targets." " you used {} targets now.".format(n_targets) ) if target_names is None: metric_names = study.metric_names if metric_names is None: target_names = [f"Objective {i}" for i in range(n_targets)] else: target_names = metric_names elif len(target_names) != n_targets: raise ValueError(f"The length of `target_names` is supposed to be {n_targets}.") if axis_order is None: axis_order = list(range(n_targets)) else: if len(axis_order) != n_targets: raise ValueError( f"Size of `axis_order` {axis_order}. Expect: {n_targets}, " f"Actual: {len(axis_order)}." ) if len(set(axis_order)) != n_targets: raise ValueError(f"Elements of given `axis_order` {axis_order} are not unique!.") if max(axis_order) > n_targets - 1: raise ValueError( f"Given `axis_order` {axis_order} contains invalid index {max(axis_order)} " f"higher than {n_targets - 1}." ) if min(axis_order) < 0: raise ValueError( f"Given `axis_order` {axis_order} contains invalid index {min(axis_order)} " "lower than 0." ) return _ParetoFrontInfo( n_targets=n_targets, target_names=target_names, best_trials_with_values=best_trials_with_values, non_best_trials_with_values=non_best_trials_with_values, infeasible_trials_with_values=infeasible_trials_with_values, axis_order=axis_order, include_dominated_trials=include_dominated_trials, has_constraints=has_constraints, ) def _targets_default(trial: FrozenTrial) -> Sequence[float]: return trial.values def _get_non_pareto_front_trials( trials: list[FrozenTrial], pareto_trials: list[FrozenTrial] ) -> list[FrozenTrial]: non_pareto_trials = [] for trial in trials: if trial not in pareto_trials: non_pareto_trials.append(trial) return non_pareto_trials def _make_scatter_object( n_targets: int, axis_order: Sequence[int], include_dominated_trials: bool, trials_with_values: Sequence[tuple[FrozenTrial, Sequence[float]]], hovertemplate: str, infeasible: bool = False, dominated_trials: bool = False, ) -> "go.Scatter" | "go.Scatter3d": trials_with_values = trials_with_values or [] marker = _make_marker( [trial for trial, _ in trials_with_values], include_dominated_trials, dominated_trials=dominated_trials, infeasible=infeasible, ) if n_targets == 2: return go.Scatter( x=[values[axis_order[0]] for _, values in trials_with_values], y=[values[axis_order[1]] for _, values in trials_with_values], text=[_make_hovertext(trial) for trial, _ in trials_with_values], mode="markers", hovertemplate=hovertemplate, marker=marker, showlegend=False, ) elif n_targets == 3: return go.Scatter3d( x=[values[axis_order[0]] for _, values in trials_with_values], y=[values[axis_order[1]] for _, values in trials_with_values], z=[values[axis_order[2]] for _, values in trials_with_values], text=[_make_hovertext(trial) for trial, _ in trials_with_values], mode="markers", hovertemplate=hovertemplate, marker=marker, showlegend=False, ) else: assert False, "Must not reach here" def _make_marker( trials: Sequence[FrozenTrial], include_dominated_trials: bool, dominated_trials: bool = False, infeasible: bool = False, ) -> dict[str, Any]: if dominated_trials and not include_dominated_trials: assert len(trials) == 0 if infeasible: return { "color": "#cccccc", } elif dominated_trials: return { "line": {"width": 0.5, "color": "Grey"}, "color": [t.number for t in trials], "colorscale": "Blues", "colorbar": { "title": "Trial", }, } else: return { "line": {"width": 0.5, "color": "Grey"}, "color": [t.number for t in trials], "colorscale": "Reds", "colorbar": { "title": "Best Trial", "x": 1.1 if include_dominated_trials else 1, "xpad": 40, }, } optuna-4.1.0/optuna/visualization/_plotly_imports.py000066400000000000000000000015341471332314300230460ustar00rootroot00000000000000from packaging import version from optuna._imports import try_import with try_import() as _imports: import plotly from plotly import __version__ as plotly_version import plotly.graph_objects as go from plotly.graph_objects import Contour from plotly.graph_objects import Scatter from plotly.subplots import make_subplots if version.parse(plotly_version) < version.parse("4.0.0"): raise ImportError( "Your version of Plotly is " + plotly_version + " . " "Please install plotly version 4.0.0 or higher. " "Plotly can be installed by executing `$ pip install -U plotly>=4.0.0`. " "For further information, please refer to the installation guide of plotly. ", name="plotly", ) __all__ = ["_imports", "plotly", "go", "Contour", "Scatter", "make_subplots"] optuna-4.1.0/optuna/visualization/_rank.py000066400000000000000000000327731471332314300207120ustar00rootroot00000000000000from __future__ import annotations import math import typing from typing import Any from typing import Callable from typing import NamedTuple import numpy as np from optuna.logging import get_logger from optuna.samplers._base import _CONSTRAINTS_KEY from optuna.study import Study from optuna.trial import FrozenTrial from optuna.trial import TrialState from optuna.visualization._plotly_imports import _imports from optuna.visualization._utils import _check_plot_args from optuna.visualization._utils import _is_log_scale from optuna.visualization._utils import _is_numerical from optuna.visualization.matplotlib._matplotlib_imports import _imports as matplotlib_imports plotly_is_available = _imports.is_successful() if plotly_is_available: from optuna.visualization._plotly_imports import go from optuna.visualization._plotly_imports import make_subplots from optuna.visualization._plotly_imports import plotly from optuna.visualization._plotly_imports import Scatter if matplotlib_imports.is_successful(): # TODO(c-bata): Refactor to remove matplotlib and plotly dependencies in `_get_rank_info()`. # See https://github.com/optuna/optuna/pull/5133#discussion_r1414761672 for the discussion. from optuna.visualization.matplotlib._matplotlib_imports import plt as matplotlib_plt _logger = get_logger(__name__) PADDING_RATIO = 0.05 class _AxisInfo(NamedTuple): name: str range: tuple[float, float] is_log: bool is_cat: bool class _RankSubplotInfo(NamedTuple): xaxis: _AxisInfo yaxis: _AxisInfo xs: list[Any] ys: list[Any] trials: list[FrozenTrial] zs: np.ndarray colors: np.ndarray class _RankPlotInfo(NamedTuple): params: list[str] sub_plot_infos: list[list[_RankSubplotInfo]] target_name: str zs: np.ndarray colors: np.ndarray has_custom_target: bool def plot_rank( study: Study, params: list[str] | None = None, *, target: Callable[[FrozenTrial], float] | None = None, target_name: str = "Objective Value", ) -> "go.Figure": """Plot parameter relations as scatter plots with colors indicating ranks of target value. Note that trials missing the specified parameters will not be plotted. Args: study: A :class:`~optuna.study.Study` object whose trials are plotted for their target values. params: Parameter list to visualize. The default is all parameters. target: A function to specify the value to display. If it is :obj:`None` and ``study`` is being used for single-objective optimization, the objective values are plotted. .. note:: Specify this argument if ``study`` is being used for multi-objective optimization. target_name: Target's name to display on the color bar. Returns: A :class:`plotly.graph_objects.Figure` object. .. note:: This function requires plotly >= 5.0.0. """ _imports.check() info = _get_rank_info(study, params, target, target_name) return _get_rank_plot(info) def _get_order_with_same_order_averaging(data: np.ndarray) -> np.ndarray: order = np.zeros_like(data, dtype=float) data_sorted = np.sort(data) for i, d in enumerate(data): indices = np.where(data_sorted == d)[0] order[i] = sum(indices) / len(indices) return order def _get_rank_info( study: Study, params: list[str] | None, target: Callable[[FrozenTrial], float] | None, target_name: str, ) -> _RankPlotInfo: _check_plot_args(study, target, target_name) trials = study.get_trials(deepcopy=False, states=(TrialState.COMPLETE,)) all_params = {p_name for t in trials for p_name in t.params.keys()} if len(trials) == 0: _logger.warning("Your study does not have any completed trials.") params = [] elif params is None: params = sorted(all_params) else: for input_p_name in params: if input_p_name not in all_params: raise ValueError("Parameter {} does not exist in your study.".format(input_p_name)) if len(params) == 0: _logger.warning("params is an empty list.") has_custom_target = True if target is None: def target(trial: FrozenTrial) -> float: return typing.cast(float, trial.value) has_custom_target = False target_values = np.array([target(trial) for trial in trials]) raw_ranks = _get_order_with_same_order_averaging(target_values) color_idxs = raw_ranks / (len(trials) - 1) if len(trials) >= 2 else np.array([0.5]) colors = _convert_color_idxs_to_scaled_rgb_colors(color_idxs) sub_plot_infos: list[list[_RankSubplotInfo]] if len(params) == 2: x_param = params[0] y_param = params[1] sub_plot_info = _get_rank_subplot_info(trials, target_values, colors, x_param, y_param) sub_plot_infos = [[sub_plot_info]] else: sub_plot_infos = [ [ _get_rank_subplot_info(trials, target_values, colors, x_param, y_param) for x_param in params ] for y_param in params ] return _RankPlotInfo( params=params, sub_plot_infos=sub_plot_infos, target_name=target_name, zs=target_values, colors=colors, has_custom_target=has_custom_target, ) def _get_rank_subplot_info( trials: list[FrozenTrial], target_values: np.ndarray, colors: np.ndarray, x_param: str, y_param: str, ) -> _RankSubplotInfo: xaxis = _get_axis_info(trials, x_param) yaxis = _get_axis_info(trials, y_param) infeasible_trial_ids = [] filtered_ids = [] for idx, trial in enumerate(trials): constraints = trial.system_attrs.get(_CONSTRAINTS_KEY) if constraints is not None and any([x > 0.0 for x in constraints]): infeasible_trial_ids.append(idx) if x_param in trial.params and y_param in trial.params: filtered_ids.append(idx) filtered_trials = [trials[i] for i in filtered_ids] xs = [trial.params[x_param] for trial in filtered_trials] ys = [trial.params[y_param] for trial in filtered_trials] zs = target_values[filtered_ids] colors[infeasible_trial_ids] = (204, 204, 204) colors = colors[filtered_ids] return _RankSubplotInfo( xaxis=xaxis, yaxis=yaxis, xs=xs, ys=ys, trials=filtered_trials, zs=np.array(zs), colors=colors, ) def _get_axis_info(trials: list[FrozenTrial], param_name: str) -> _AxisInfo: values: list[str | float | None] is_numerical = _is_numerical(trials, param_name) if is_numerical: values = [t.params.get(param_name) for t in trials] else: values = [ str(t.params.get(param_name)) if param_name in t.params else None for t in trials ] min_value = min([v for v in values if v is not None]) max_value = max([v for v in values if v is not None]) if _is_log_scale(trials, param_name): min_value = float(min_value) max_value = float(max_value) padding = (math.log10(max_value) - math.log10(min_value)) * PADDING_RATIO min_value = math.pow(10, math.log10(min_value) - padding) max_value = math.pow(10, math.log10(max_value) + padding) is_log = True is_cat = False elif is_numerical: min_value = float(min_value) max_value = float(max_value) padding = (max_value - min_value) * PADDING_RATIO min_value = min_value - padding max_value = max_value + padding is_log = False is_cat = False else: unique_values = set(values) span = len(unique_values) - 1 if None in unique_values: span -= 1 padding = span * PADDING_RATIO min_value = -padding max_value = span + padding is_log = False is_cat = True return _AxisInfo( name=param_name, range=(min_value, max_value), is_log=is_log, is_cat=is_cat, ) def _get_rank_subplot( info: _RankSubplotInfo, target_name: str, print_raw_objectives: bool ) -> "Scatter": def get_hover_text(trial: FrozenTrial, target_value: float) -> str: lines = [f"Trial #{trial.number}"] lines += [f"{k}: {v}" for k, v in trial.params.items()] lines += [f"{target_name}: {target_value}"] if print_raw_objectives: lines += [f"Objective #{i}: {v}" for i, v in enumerate(trial.values)] return "
".join(lines) scatter = go.Scatter( x=[str(x) for x in info.xs] if info.xaxis.is_cat else info.xs, y=[str(y) for y in info.ys] if info.yaxis.is_cat else info.ys, marker={ "color": list(map(plotly.colors.label_rgb, info.colors)), "line": {"width": 0.5, "color": "Grey"}, }, mode="markers", showlegend=False, hovertemplate="%{hovertext}", hovertext=[ get_hover_text(trial, target_value) for trial, target_value in zip(info.trials, info.zs) ], ) return scatter class _TickInfo(NamedTuple): coloridxs: list[float] text: list[str] def _get_tick_info(target_values: np.ndarray) -> _TickInfo: sorted_target_values = np.sort(target_values) coloridxs = [0, 0.25, 0.5, 0.75, 1] values = np.quantile(sorted_target_values, coloridxs) rank_text = ["min.", "25%", "50%", "75%", "max."] text = [f"{rank_text[i]} ({values[i]:3g})" for i in range(len(values))] return _TickInfo(coloridxs=coloridxs, text=text) def _get_rank_plot( info: _RankPlotInfo, ) -> "go.Figure": params = info.params sub_plot_infos = info.sub_plot_infos layout = go.Layout(title=f"Rank ({info.target_name})") if len(params) == 0: return go.Figure(data=[], layout=layout) if len(params) == 2: x_param = params[0] y_param = params[1] sub_plot_info = sub_plot_infos[0][0] sub_plots = _get_rank_subplot(sub_plot_info, info.target_name, info.has_custom_target) figure = go.Figure(data=sub_plots, layout=layout) figure.update_xaxes(title_text=x_param, range=sub_plot_info.xaxis.range) figure.update_yaxes(title_text=y_param, range=sub_plot_info.yaxis.range) if sub_plot_info.xaxis.is_cat: figure.update_xaxes(type="category") if sub_plot_info.yaxis.is_cat: figure.update_yaxes(type="category") if sub_plot_info.xaxis.is_log: log_range = [math.log10(p) for p in sub_plot_info.xaxis.range] figure.update_xaxes(range=log_range, type="log") if sub_plot_info.yaxis.is_log: log_range = [math.log10(p) for p in sub_plot_info.yaxis.range] figure.update_yaxes(range=log_range, type="log") else: figure = make_subplots( rows=len(params), cols=len(params), shared_xaxes=True, shared_yaxes=True, horizontal_spacing=0.08 / len(params), vertical_spacing=0.08 / len(params), ) figure.update_layout(layout) for x_i, x_param in enumerate(params): for y_i, y_param in enumerate(params): scatter = _get_rank_subplot( sub_plot_infos[y_i][x_i], info.target_name, info.has_custom_target ) figure.add_trace(scatter, row=y_i + 1, col=x_i + 1) xaxis = sub_plot_infos[y_i][x_i].xaxis yaxis = sub_plot_infos[y_i][x_i].yaxis figure.update_xaxes(range=xaxis.range, row=y_i + 1, col=x_i + 1) figure.update_yaxes(range=yaxis.range, row=y_i + 1, col=x_i + 1) if xaxis.is_cat: figure.update_xaxes(type="category", row=y_i + 1, col=x_i + 1) if yaxis.is_cat: figure.update_yaxes(type="category", row=y_i + 1, col=x_i + 1) if xaxis.is_log: log_range = [math.log10(p) for p in xaxis.range] figure.update_xaxes(range=log_range, type="log", row=y_i + 1, col=x_i + 1) if yaxis.is_log: log_range = [math.log10(p) for p in yaxis.range] figure.update_yaxes(range=log_range, type="log", row=y_i + 1, col=x_i + 1) if x_i == 0: figure.update_yaxes(title_text=y_param, row=y_i + 1, col=x_i + 1) if y_i == len(params) - 1: figure.update_xaxes(title_text=x_param, row=y_i + 1, col=x_i + 1) tick_info = _get_tick_info(info.zs) colormap = "RdYlBu_r" colorbar_trace = go.Scatter( x=[None], y=[None], mode="markers", marker=dict( colorscale=colormap, showscale=True, cmin=0, cmax=1, colorbar=dict(thickness=10, tickvals=tick_info.coloridxs, ticktext=tick_info.text), ), hoverinfo="none", showlegend=False, ) figure.add_trace(colorbar_trace) return figure def _convert_color_idxs_to_scaled_rgb_colors(color_idxs: np.ndarray) -> np.ndarray: colormap = "RdYlBu_r" if plotly_is_available: # sample_colorscale requires plotly >= 5.0.0. labeled_colors = plotly.colors.sample_colorscale(colormap, color_idxs) scaled_rgb_colors = np.array([plotly.colors.unlabel_rgb(cl) for cl in labeled_colors]) return scaled_rgb_colors else: cmap = matplotlib_plt.get_cmap(colormap) colors = cmap(color_idxs)[:, :3] # Drop alpha values. rgb_colors = np.asarray(colors * 255, dtype=int) return rgb_colors optuna-4.1.0/optuna/visualization/_slice.py000066400000000000000000000215271471332314300210510ustar00rootroot00000000000000from __future__ import annotations from typing import Any from typing import Callable from typing import cast from typing import NamedTuple from optuna.distributions import CategoricalChoiceType from optuna.distributions import CategoricalDistribution from optuna.logging import get_logger from optuna.samplers._base import _CONSTRAINTS_KEY from optuna.study import Study from optuna.trial import FrozenTrial from optuna.trial import TrialState from optuna.visualization._plotly_imports import _imports from optuna.visualization._utils import _check_plot_args from optuna.visualization._utils import _filter_nonfinite from optuna.visualization._utils import _is_log_scale if _imports.is_successful(): from optuna.visualization._plotly_imports import go from optuna.visualization._plotly_imports import make_subplots from optuna.visualization._plotly_imports import Scatter from optuna.visualization._utils import COLOR_SCALE _logger = get_logger(__name__) class _SliceSubplotInfo(NamedTuple): param_name: str x: list[Any] y: list[float] trial_numbers: list[int] is_log: bool is_numerical: bool constraints: list[bool] x_labels: tuple[CategoricalChoiceType, ...] | None class _SlicePlotInfo(NamedTuple): target_name: str subplots: list[_SliceSubplotInfo] class _PlotValues(NamedTuple): x: list[Any] y: list[float] trial_numbers: list[int] def _get_slice_subplot_info( trials: list[FrozenTrial], param: str, target: Callable[[FrozenTrial], float] | None, log_scale: bool, numerical: bool, x_labels: tuple[CategoricalChoiceType, ...] | None, ) -> _SliceSubplotInfo: if target is None: def _target(t: FrozenTrial) -> float: return cast(float, t.value) target = _target plot_info = _SliceSubplotInfo( param_name=param, x=[], y=[], trial_numbers=[], is_log=log_scale, is_numerical=numerical, x_labels=x_labels, constraints=[], ) for t in trials: if param not in t.params: continue plot_info.x.append(t.params[param]) plot_info.y.append(target(t)) plot_info.trial_numbers.append(t.number) constraints = t.system_attrs.get(_CONSTRAINTS_KEY) plot_info.constraints.append(constraints is None or all([x <= 0.0 for x in constraints])) return plot_info def _get_slice_plot_info( study: Study, params: list[str] | None, target: Callable[[FrozenTrial], float] | None, target_name: str, ) -> _SlicePlotInfo: _check_plot_args(study, target, target_name) trials = _filter_nonfinite( study.get_trials(deepcopy=False, states=(TrialState.COMPLETE,)), target=target ) if len(trials) == 0: _logger.warning("Your study does not have any completed trials.") return _SlicePlotInfo(target_name, []) all_params = {p_name for t in trials for p_name in t.params.keys()} distributions = {} for trial in trials: for param_name, distribution in trial.distributions.items(): if param_name not in distributions: distributions[param_name] = distribution x_labels = {} for param_name, distribution in distributions.items(): if isinstance(distribution, CategoricalDistribution): x_labels[param_name] = distribution.choices if params is None: sorted_params = sorted(all_params) else: for input_p_name in params: if input_p_name not in all_params: raise ValueError(f"Parameter {input_p_name} does not exist in your study.") sorted_params = sorted(set(params)) return _SlicePlotInfo( target_name=target_name, subplots=[ _get_slice_subplot_info( trials=trials, param=param, target=target, log_scale=_is_log_scale(trials, param), numerical=not isinstance(distributions[param], CategoricalDistribution), x_labels=x_labels.get(param), ) for param in sorted_params ], ) def plot_slice( study: Study, params: list[str] | None = None, *, target: Callable[[FrozenTrial], float] | None = None, target_name: str = "Objective Value", ) -> "go.Figure": """Plot the parameter relationship as slice plot in a study. Note that, if a parameter contains missing values, a trial with missing values is not plotted. Args: study: A :class:`~optuna.study.Study` object whose trials are plotted for their target values. params: Parameter list to visualize. The default is all parameters. target: A function to specify the value to display. If it is :obj:`None` and ``study`` is being used for single-objective optimization, the objective values are plotted. .. note:: Specify this argument if ``study`` is being used for multi-objective optimization. target_name: Target's name to display on the axis label. Returns: A :class:`plotly.graph_objects.Figure` object. """ _imports.check() return _get_slice_plot(_get_slice_plot_info(study, params, target, target_name)) def _get_slice_plot(info: _SlicePlotInfo) -> "go.Figure": layout = go.Layout(title="Slice Plot") if len(info.subplots) == 0: return go.Figure(data=[], layout=layout) elif len(info.subplots) == 1: figure = go.Figure(data=_generate_slice_subplot(info.subplots[0]), layout=layout) figure.update_xaxes(title_text=info.subplots[0].param_name) figure.update_yaxes(title_text=info.target_name) if not info.subplots[0].is_numerical: figure.update_xaxes( type="category", categoryorder="array", categoryarray=info.subplots[0].x_labels ) elif info.subplots[0].is_log: figure.update_xaxes(type="log") else: figure = make_subplots(rows=1, cols=len(info.subplots), shared_yaxes=True) figure.update_layout(layout) showscale = True # showscale option only needs to be specified once. for column_index, subplot_info in enumerate(info.subplots, start=1): trace = _generate_slice_subplot(subplot_info) trace[0].update(marker={"showscale": showscale}) # showscale's default is True. if showscale: showscale = False for t in trace: figure.add_trace(t, row=1, col=column_index) figure.update_xaxes(title_text=subplot_info.param_name, row=1, col=column_index) if column_index == 1: figure.update_yaxes(title_text=info.target_name, row=1, col=column_index) if not subplot_info.is_numerical: figure.update_xaxes( type="category", categoryorder="array", categoryarray=subplot_info.x_labels, row=1, col=column_index, ) elif subplot_info.is_log: figure.update_xaxes(type="log", row=1, col=column_index) if len(info.subplots) > 3: # Ensure that each subplot has a minimum width without relying on autusizing. figure.update_layout(width=300 * len(info.subplots)) return figure def _generate_slice_subplot(subplot_info: _SliceSubplotInfo) -> list[Scatter]: trace = [] feasible = _PlotValues([], [], []) infeasible = _PlotValues([], [], []) for x, y, num, c in zip( subplot_info.x, subplot_info.y, subplot_info.trial_numbers, subplot_info.constraints ): if x is not None or x != "None" or y is not None or y != "None": if c: feasible.x.append(x) feasible.y.append(y) feasible.trial_numbers.append(num) else: infeasible.x.append(x) infeasible.y.append(y) trace.append( go.Scatter( x=feasible.x, y=feasible.y, mode="markers", name="Feasible Trial", marker={ "line": {"width": 0.5, "color": "Grey"}, "color": feasible.trial_numbers, "colorscale": COLOR_SCALE, "colorbar": { "title": "Trial", "x": 1.0, # Offset the colorbar position with a fixed width `xpad`. "xpad": 40, }, }, showlegend=False, ) ) if len(infeasible.x) > 0: trace.append( go.Scatter( x=infeasible.x, y=infeasible.y, mode="markers", name="Infeasible Trial", marker={ "color": "#cccccc", }, showlegend=False, ) ) return trace optuna-4.1.0/optuna/visualization/_terminator_improvement.py000066400000000000000000000164441471332314300245650ustar00rootroot00000000000000from __future__ import annotations from typing import NamedTuple import tqdm import optuna from optuna._experimental import experimental_func from optuna.logging import get_logger from optuna.study.study import Study from optuna.terminator import BaseErrorEvaluator from optuna.terminator import BaseImprovementEvaluator from optuna.terminator import CrossValidationErrorEvaluator from optuna.terminator import RegretBoundEvaluator from optuna.terminator.erroreval import StaticErrorEvaluator from optuna.terminator.improvement.evaluator import BestValueStagnationEvaluator from optuna.terminator.improvement.evaluator import DEFAULT_MIN_N_TRIALS from optuna.visualization._plotly_imports import _imports if _imports.is_successful(): from optuna.visualization._plotly_imports import go _logger = get_logger(__name__) PADDING_RATIO_Y = 0.05 OPACITY = 0.25 class _ImprovementInfo(NamedTuple): trial_numbers: list[int] improvements: list[float] errors: list[float] | None @experimental_func("3.2.0") def plot_terminator_improvement( study: Study, plot_error: bool = False, improvement_evaluator: BaseImprovementEvaluator | None = None, error_evaluator: BaseErrorEvaluator | None = None, min_n_trials: int = DEFAULT_MIN_N_TRIALS, ) -> "go.Figure": """Plot the potentials for future objective improvement. This function visualizes the objective improvement potentials, evaluated with ``improvement_evaluator``. It helps to determine whether we should continue the optimization or not. You can also plot the error evaluated with ``error_evaluator`` if the ``plot_error`` argument is set to :obj:`True`. Note that this function may take some time to compute the improvement potentials. Args: study: A :class:`~optuna.study.Study` object whose trials are plotted for their improvement. plot_error: A flag to show the error. If it is set to :obj:`True`, errors evaluated by ``error_evaluator`` are also plotted as line graph. Defaults to :obj:`False`. improvement_evaluator: An object that evaluates the improvement of the objective function. Defaults to :class:`~optuna.terminator.RegretBoundEvaluator`. error_evaluator: An object that evaluates the error inherent in the objective function. Defaults to :class:`~optuna.terminator.CrossValidationErrorEvaluator`. min_n_trials: The minimum number of trials before termination is considered. Terminator improvements for trials below this value are shown in a lighter color. Defaults to ``20``. Returns: A :class:`plotly.graph_objects.Figure` object. """ _imports.check() info = _get_improvement_info(study, plot_error, improvement_evaluator, error_evaluator) return _get_improvement_plot(info, min_n_trials) def _get_improvement_info( study: Study, get_error: bool = False, improvement_evaluator: BaseImprovementEvaluator | None = None, error_evaluator: BaseErrorEvaluator | None = None, ) -> _ImprovementInfo: if study._is_multi_objective(): raise ValueError("This function does not support multi-objective optimization study.") if improvement_evaluator is None: improvement_evaluator = RegretBoundEvaluator() if error_evaluator is None: if isinstance(improvement_evaluator, BestValueStagnationEvaluator): error_evaluator = StaticErrorEvaluator(constant=0) else: error_evaluator = CrossValidationErrorEvaluator() trial_numbers = [] completed_trials = [] improvements = [] errors = [] for trial in tqdm.tqdm(study.trials): if trial.state == optuna.trial.TrialState.COMPLETE: completed_trials.append(trial) if len(completed_trials) == 0: continue trial_numbers.append(trial.number) improvement = improvement_evaluator.evaluate( trials=completed_trials, study_direction=study.direction ) improvements.append(improvement) if get_error: error = error_evaluator.evaluate( trials=completed_trials, study_direction=study.direction ) errors.append(error) if len(errors) == 0: return _ImprovementInfo( trial_numbers=trial_numbers, improvements=improvements, errors=None ) else: return _ImprovementInfo( trial_numbers=trial_numbers, improvements=improvements, errors=errors ) def _get_improvement_scatter( trial_numbers: list[int], improvements: list[float], opacity: float = 1.0, showlegend: bool = True, ) -> "go.Scatter": plotly_blue_with_opacity = f"rgba(99, 110, 250, {opacity})" return go.Scatter( x=trial_numbers, y=improvements, mode="markers+lines", marker=dict(color=plotly_blue_with_opacity), line=dict(color=plotly_blue_with_opacity), name="Terminator Improvement", showlegend=showlegend, legendgroup="improvement", ) def _get_error_scatter( trial_numbers: list[int], errors: list[float] | None, ) -> "go.Scatter": if errors is None: return go.Scatter() plotly_red = "rgb(239, 85, 59)" return go.Scatter( x=trial_numbers, y=errors, mode="markers+lines", name="Error", marker=dict(color=plotly_red), line=dict(color=plotly_red), ) def _get_y_range(info: _ImprovementInfo, min_n_trials: int) -> tuple[float, float]: min_value = min(info.improvements) if info.errors is not None: min_value = min(min_value, min(info.errors)) # Determine the display range based on trials after min_n_trials. if len(info.trial_numbers) > min_n_trials: max_value = max(info.improvements[min_n_trials:]) # If there are no trials after min_trials, determine the display range based on all trials. else: max_value = max(info.improvements) if info.errors is not None: max_value = max(max_value, max(info.errors)) padding = (max_value - min_value) * PADDING_RATIO_Y return (min_value - padding, max_value + padding) def _get_improvement_plot(info: _ImprovementInfo, min_n_trials: int) -> "go.Figure": n_trials = len(info.trial_numbers) fig = go.Figure( layout=go.Layout( title="Terminator Improvement Plot", xaxis=dict(title="Trial"), yaxis=dict(title="Terminator Improvement"), ) ) if n_trials == 0: _logger.warning("There are no complete trials.") return fig fig.add_trace( _get_improvement_scatter( info.trial_numbers[: min_n_trials + 1], info.improvements[: min_n_trials + 1], # Plot line with a lighter color until the number of trials reaches min_n_trials. OPACITY, n_trials <= min_n_trials, # Avoid showing legend twice. ) ) if n_trials > min_n_trials: fig.add_trace( _get_improvement_scatter( info.trial_numbers[min_n_trials:], info.improvements[min_n_trials:], ) ) fig.add_trace(_get_error_scatter(info.trial_numbers, info.errors)) fig.update_yaxes(range=_get_y_range(info, min_n_trials)) return fig optuna-4.1.0/optuna/visualization/_timeline.py000066400000000000000000000126101471332314300215510ustar00rootroot00000000000000from __future__ import annotations import datetime from typing import NamedTuple from optuna.logging import get_logger from optuna.samplers._base import _CONSTRAINTS_KEY from optuna.study import Study from optuna.trial import TrialState from optuna.visualization._plotly_imports import _imports from optuna.visualization._utils import _make_hovertext if _imports.is_successful(): from optuna.visualization._plotly_imports import go _logger = get_logger(__name__) class _TimelineBarInfo(NamedTuple): number: int start: datetime.datetime complete: datetime.datetime state: TrialState hovertext: str infeasible: bool class _TimelineInfo(NamedTuple): bars: list[_TimelineBarInfo] def plot_timeline(study: Study) -> "go.Figure": """Plot the timeline of a study. Args: study: A :class:`~optuna.study.Study` object whose trials are plotted with their lifetime. Returns: A :class:`plotly.graph_objects.Figure` object. """ _imports.check() info = _get_timeline_info(study) return _get_timeline_plot(info) def _get_max_datetime_complete(study: Study) -> datetime.datetime: max_run_duration = max( [ t.datetime_complete - t.datetime_start for t in study.trials if t.datetime_complete is not None and t.datetime_start is not None ], default=None, ) if _is_running_trials_in_study(study, max_run_duration): return datetime.datetime.now() return max( [t.datetime_complete for t in study.trials if t.datetime_complete is not None], default=datetime.datetime.now(), ) def _is_running_trials_in_study(study: Study, max_run_duration: datetime.timedelta | None) -> bool: running_trials = study.get_trials(states=(TrialState.RUNNING,), deepcopy=False) if max_run_duration is None: return len(running_trials) > 0 now = datetime.datetime.now() # This heuristic is to check whether we have trials that were somehow killed, # still remain as `RUNNING` in `study`. return any( now - t.datetime_start < 5 * max_run_duration for t in running_trials # MyPy redefinition: Running trial should have datetime_start. if t.datetime_start is not None ) def _get_timeline_info(study: Study) -> _TimelineInfo: bars = [] max_datetime = _get_max_datetime_complete(study) timedelta_for_small_bar = datetime.timedelta(seconds=1) for trial in study.get_trials(deepcopy=False): datetime_start = trial.datetime_start or max_datetime datetime_complete = ( max_datetime + timedelta_for_small_bar if trial.state == TrialState.RUNNING else trial.datetime_complete or datetime_start + timedelta_for_small_bar ) infeasible = ( False if _CONSTRAINTS_KEY not in trial.system_attrs else any([x > 0 for x in trial.system_attrs[_CONSTRAINTS_KEY]]) ) if datetime_complete < datetime_start: _logger.warning( ( f"The start and end times for Trial {trial.number} seem to be reversed. " f"The start time is {datetime_start} and the end time is {datetime_complete}." ) ) bars.append( _TimelineBarInfo( number=trial.number, start=datetime_start, complete=datetime_complete, state=trial.state, hovertext=_make_hovertext(trial), infeasible=infeasible, ) ) if len(bars) == 0: _logger.warning("Your study does not have any trials.") return _TimelineInfo(bars) def _get_timeline_plot(info: _TimelineInfo) -> "go.Figure": _cm = { "COMPLETE": "blue", "FAIL": "red", "PRUNED": "orange", "RUNNING": "green", "WAITING": "gray", } fig = go.Figure() for state in sorted(TrialState, key=lambda x: x.name): if state.name == "COMPLETE": infeasible_bars = [b for b in info.bars if b.state == state and b.infeasible] feasible_bars = [b for b in info.bars if b.state == state and not b.infeasible] _plot_bars(infeasible_bars, "#cccccc", "INFEASIBLE", fig) _plot_bars(feasible_bars, _cm[state.name], state.name, fig) else: bars = [b for b in info.bars if b.state == state] _plot_bars(bars, _cm[state.name], state.name, fig) fig.update_xaxes(type="date") fig.update_layout( go.Layout( title="Timeline Plot", xaxis={"title": "Datetime"}, yaxis={"title": "Trial"}, ) ) fig.update_layout(showlegend=True) # Draw a legend even if all TrialStates are the same. return fig def _plot_bars(bars: list[_TimelineBarInfo], color: str, name: str, fig: go.Figure) -> None: if len(bars) == 0: return fig.add_trace( go.Bar( name=name, x=[(b.complete - b.start).total_seconds() * 1000 for b in bars], y=[b.number for b in bars], base=[b.start.isoformat() for b in bars], text=[b.hovertext for b in bars], hovertemplate="%{text}" + name + "", orientation="h", marker=dict(color=color), textposition="none", # Avoid drawing hovertext in a bar. ) ) optuna-4.1.0/optuna/visualization/_utils.py000066400000000000000000000141351471332314300211070ustar00rootroot00000000000000from __future__ import annotations import json from typing import Any from typing import Callable from typing import cast from typing import Sequence import warnings import numpy as np import optuna from optuna.distributions import CategoricalDistribution from optuna.distributions import FloatDistribution from optuna.distributions import IntDistribution from optuna.study import Study from optuna.study._study_direction import StudyDirection from optuna.trial import FrozenTrial from optuna.visualization import _plotly_imports __all__ = ["is_available"] _logger = optuna.logging.get_logger(__name__) def is_available() -> bool: """Returns whether visualization with plotly is available or not. .. note:: :mod:`~optuna.visualization` module depends on plotly version 4.0.0 or higher. If a supported version of plotly isn't installed in your environment, this function will return :obj:`False`. In such case, please execute ``$ pip install -U plotly>=4.0.0`` to install plotly. Returns: :obj:`True` if visualization with plotly is available, :obj:`False` otherwise. """ return _plotly_imports._imports.is_successful() if is_available(): import plotly.colors COLOR_SCALE = plotly.colors.sequential.Blues def _check_plot_args( study: Study | Sequence[Study], target: Callable[[FrozenTrial], float] | None, target_name: str, ) -> None: studies: Sequence[Study] if isinstance(study, Study): studies = [study] else: studies = study if target is None and any(study._is_multi_objective() for study in studies): raise ValueError( "If the `study` is being used for multi-objective optimization, " "please specify the `target`." ) if target is not None and target_name == "Objective Value": warnings.warn( "`target` is specified, but `target_name` is the default value, 'Objective Value'." ) def _is_log_scale(trials: list[FrozenTrial], param: str) -> bool: for trial in trials: if param not in trial.params: continue dist = trial.distributions[param] return isinstance(dist, (FloatDistribution, IntDistribution)) and dist.log return False def _is_numerical(trials: list[FrozenTrial], param: str) -> bool: for trial in trials: if param not in trial.params: continue dist = trial.distributions[param] if isinstance(dist, (IntDistribution, FloatDistribution)): return True elif isinstance(dist, CategoricalDistribution): # NOTE: Although it is a bit odd to do so, we keep it as is only for visualization. return all( isinstance(v, (int, float)) and not isinstance(v, bool) for v in dist.choices ) else: assert False, "Should not reach." return True def _get_param_values(trials: list[FrozenTrial], p_name: str) -> list[Any]: values = [t.params[p_name] for t in trials if p_name in t.params] if _is_numerical(trials, p_name): return values return list(map(str, values)) def _get_skipped_trial_numbers( trials: list[FrozenTrial], used_param_names: Sequence[str] ) -> set[int]: """Utility function for ``plot_parallel_coordinate``. If trial's parameters do not contain a parameter in ``used_param_names``, ``plot_parallel_coordinate`` methods do not use such trials. Args: trials: List of ``FrozenTrial``s. used_param_names: The parameter names used in ``plot_parallel_coordinate``. Returns: A set of invalid trial numbers. """ skipped_trial_numbers = set() for trial in trials: for used_param in used_param_names: if used_param not in trial.params.keys(): skipped_trial_numbers.add(trial.number) break return skipped_trial_numbers def _filter_nonfinite( trials: list[FrozenTrial], target: Callable[[FrozenTrial], float] | None = None, with_message: bool = True, ) -> list[FrozenTrial]: # For multi-objective optimization target must be specified to select # one of objective values to filter trials by (and plot by later on). # This function is not raising when target is missing, since we're # assuming plot args have been sanitized before. if target is None: def _target(t: FrozenTrial) -> float: return cast(float, t.value) target = _target filtered_trials: list[FrozenTrial] = [] for trial in trials: value = target(trial) try: value = float(value) except ( ValueError, TypeError, ): warnings.warn( f"Trial{trial.number}'s target value {repr(value)} could not be cast to float." ) raise # Not a Number, positive infinity and negative infinity are considered to be non-finite. if not np.isfinite(value): if with_message: _logger.warning( f"Trial {trial.number} is omitted in visualization " "because its objective value is inf or nan." ) else: filtered_trials.append(trial) return filtered_trials def _is_reverse_scale(study: Study, target: Callable[[FrozenTrial], float] | None) -> bool: return target is not None or study.direction == StudyDirection.MINIMIZE def _make_json_compatible(value: Any) -> Any: try: json.dumps(value) return value except TypeError: # The value can't be converted to JSON directly, so return a string representation. return str(value) def _make_hovertext(trial: FrozenTrial) -> str: user_attrs = {key: _make_json_compatible(value) for key, value in trial.user_attrs.items()} user_attrs_dict = {"user_attrs": user_attrs} if user_attrs else {} text = json.dumps( { "number": trial.number, "values": trial.values, "params": trial.params, **user_attrs_dict, }, indent=2, ) return text.replace("\n", "
") optuna-4.1.0/optuna/visualization/matplotlib/000077500000000000000000000000001471332314300214015ustar00rootroot00000000000000optuna-4.1.0/optuna/visualization/matplotlib/__init__.py000066400000000000000000000025011471332314300235100ustar00rootroot00000000000000from optuna.visualization.matplotlib._contour import plot_contour from optuna.visualization.matplotlib._edf import plot_edf from optuna.visualization.matplotlib._hypervolume_history import plot_hypervolume_history from optuna.visualization.matplotlib._intermediate_values import plot_intermediate_values from optuna.visualization.matplotlib._optimization_history import plot_optimization_history from optuna.visualization.matplotlib._parallel_coordinate import plot_parallel_coordinate from optuna.visualization.matplotlib._param_importances import plot_param_importances from optuna.visualization.matplotlib._pareto_front import plot_pareto_front from optuna.visualization.matplotlib._rank import plot_rank from optuna.visualization.matplotlib._slice import plot_slice from optuna.visualization.matplotlib._terminator_improvement import plot_terminator_improvement from optuna.visualization.matplotlib._timeline import plot_timeline from optuna.visualization.matplotlib._utils import is_available __all__ = [ "is_available", "plot_contour", "plot_edf", "plot_intermediate_values", "plot_hypervolume_history", "plot_optimization_history", "plot_parallel_coordinate", "plot_param_importances", "plot_pareto_front", "plot_rank", "plot_slice", "plot_terminator_improvement", "plot_timeline", ] optuna-4.1.0/optuna/visualization/matplotlib/_contour.py000066400000000000000000000320711471332314300236060ustar00rootroot00000000000000from __future__ import annotations from typing import Callable from typing import Sequence import numpy as np from optuna._experimental import experimental_func from optuna._imports import try_import from optuna.logging import get_logger from optuna.study import Study from optuna.trial import FrozenTrial from optuna.visualization._contour import _AxisInfo from optuna.visualization._contour import _ContourInfo from optuna.visualization._contour import _get_contour_info from optuna.visualization._contour import _PlotValues from optuna.visualization._contour import _SubContourInfo from optuna.visualization.matplotlib._matplotlib_imports import _imports with try_import() as _optuna_imports: import scipy if _imports.is_successful(): from optuna.visualization.matplotlib._matplotlib_imports import Axes from optuna.visualization.matplotlib._matplotlib_imports import Colormap from optuna.visualization.matplotlib._matplotlib_imports import ContourSet from optuna.visualization.matplotlib._matplotlib_imports import plt _logger = get_logger(__name__) CONTOUR_POINT_NUM = 100 @experimental_func("2.2.0") def plot_contour( study: Study, params: list[str] | None = None, *, target: Callable[[FrozenTrial], float] | None = None, target_name: str = "Objective Value", ) -> "Axes": """Plot the parameter relationship as contour plot in a study with Matplotlib. Note that, if a parameter contains missing values, a trial with missing values is not plotted. .. seealso:: Please refer to :func:`optuna.visualization.plot_contour` for an example. Warnings: Output figures of this Matplotlib-based :func:`~optuna.visualization.matplotlib.plot_contour` function would be different from those of the Plotly-based :func:`~optuna.visualization.plot_contour`. Args: study: A :class:`~optuna.study.Study` object whose trials are plotted for their target values. params: Parameter list to visualize. The default is all parameters. target: A function to specify the value to display. If it is :obj:`None` and ``study`` is being used for single-objective optimization, the objective values are plotted. .. note:: Specify this argument if ``study`` is being used for multi-objective optimization. target_name: Target's name to display on the color bar. Returns: A :class:`matplotlib.axes.Axes` object. .. note:: The colormap is reversed when the ``target`` argument isn't :obj:`None` or ``direction`` of :class:`~optuna.study.Study` is ``minimize``. """ _imports.check() _logger.warning( "Output figures of this Matplotlib-based `plot_contour` function would be different from " "those of the Plotly-based `plot_contour`." ) info = _get_contour_info(study, params, target, target_name) return _get_contour_plot(info) def _get_contour_plot(info: _ContourInfo) -> "Axes": sorted_params = info.sorted_params sub_plot_infos = info.sub_plot_infos reverse_scale = info.reverse_scale target_name = info.target_name if len(sorted_params) <= 1: _, ax = plt.subplots() return ax n_params = len(sorted_params) plt.style.use("ggplot") # Use ggplot style sheet for similar outputs to plotly. if n_params == 2: # Set up the graph style. fig, axs = plt.subplots() axs.set_title("Contour Plot") cmap = _set_cmap(reverse_scale) cs = _generate_contour_subplot(sub_plot_infos[0][0], axs, cmap) if isinstance(cs, ContourSet): axcb = fig.colorbar(cs) axcb.set_label(target_name) else: # Set up the graph style. fig, axs = plt.subplots(n_params, n_params) fig.suptitle("Contour Plot") cmap = _set_cmap(reverse_scale) # Prepare data and draw contour plots. cs_list = [] for x_i in range(len(sorted_params)): for y_i in range(len(sorted_params)): ax = axs[y_i, x_i] cs = _generate_contour_subplot(sub_plot_infos[y_i][x_i], ax, cmap) if isinstance(cs, ContourSet): cs_list.append(cs) if cs_list: axcb = fig.colorbar(cs_list[0], ax=axs) axcb.set_label(target_name) return axs def _set_cmap(reverse_scale: bool) -> "Colormap": cmap = "Blues_r" if not reverse_scale else "Blues" return plt.get_cmap(cmap) class _LabelEncoder: def __init__(self) -> None: self.labels: list[str] = [] def fit(self, labels: list[str]) -> "_LabelEncoder": self.labels = sorted(set(labels)) return self def transform(self, labels: list[str]) -> list[int]: return [self.labels.index(label) for label in labels] def fit_transform(self, labels: list[str]) -> list[int]: return self.fit(labels).transform(labels) def get_labels(self) -> list[str]: return self.labels def get_indices(self) -> list[int]: return list(range(len(self.labels))) def _calculate_griddata( info: _SubContourInfo, ) -> tuple[ np.ndarray, np.ndarray, np.ndarray, list[int], list[str], list[int], list[str], _PlotValues, _PlotValues, ]: xaxis = info.xaxis yaxis = info.yaxis z_values_dict = info.z_values x_values = [] y_values = [] z_values = [] for x_value, y_value in zip(xaxis.values, yaxis.values): if x_value is not None and y_value is not None: x_values.append(x_value) y_values.append(y_value) x_i = xaxis.indices.index(x_value) y_i = yaxis.indices.index(y_value) z_values.append(z_values_dict[(x_i, y_i)]) # Return empty values when x or y has no value. if len(x_values) == 0 or len(y_values) == 0: return ( np.array([]), np.array([]), np.array([]), [], [], [], [], _PlotValues([], []), _PlotValues([], []), ) def _calculate_axis_data( axis: _AxisInfo, values: Sequence[str | float], ) -> tuple[np.ndarray, list[str], list[int], list[int | float]]: # Convert categorical values to int. cat_param_labels: list[str] = [] cat_param_pos: list[int] = [] returned_values: Sequence[int | float] if axis.is_cat: enc = _LabelEncoder() returned_values = enc.fit_transform(list(map(str, values))) cat_param_labels = enc.get_labels() cat_param_pos = enc.get_indices() else: returned_values = list(map(lambda x: float(x), values)) # For x and y, create 1-D array of evenly spaced coordinates on linear or log scale. if axis.is_log: ci = np.logspace(np.log10(axis.range[0]), np.log10(axis.range[1]), CONTOUR_POINT_NUM) else: ci = np.linspace(axis.range[0], axis.range[1], CONTOUR_POINT_NUM) return ci, cat_param_labels, cat_param_pos, list(returned_values) xi, cat_param_labels_x, cat_param_pos_x, transformed_x_values = _calculate_axis_data( xaxis, x_values, ) yi, cat_param_labels_y, cat_param_pos_y, transformed_y_values = _calculate_axis_data( yaxis, y_values, ) # Calculate grid data points. zi: np.ndarray = np.array([]) # Create irregularly spaced map of trial values # and interpolate it with Plotly's interpolation formulation. if xaxis.name != yaxis.name: zmap = _create_zmap(transformed_x_values, transformed_y_values, z_values, xi, yi) zi = _interpolate_zmap(zmap, CONTOUR_POINT_NUM) # categorize by constraints feasible = _PlotValues([], []) infeasible = _PlotValues([], []) for x_value, y_value, c in zip(transformed_x_values, transformed_y_values, info.constraints): if c: feasible.x.append(x_value) feasible.y.append(y_value) else: infeasible.x.append(x_value) infeasible.y.append(y_value) return ( xi, yi, zi, cat_param_pos_x, cat_param_labels_x, cat_param_pos_y, cat_param_labels_y, feasible, infeasible, ) def _generate_contour_subplot( info: _SubContourInfo, ax: "Axes", cmap: "Colormap" ) -> "ContourSet" | None: if len(info.xaxis.indices) < 2 or len(info.yaxis.indices) < 2: ax.label_outer() return None ax.set(xlabel=info.xaxis.name, ylabel=info.yaxis.name) ax.set_xlim(info.xaxis.range[0], info.xaxis.range[1]) ax.set_ylim(info.yaxis.range[0], info.yaxis.range[1]) if info.xaxis.name == info.yaxis.name: ax.label_outer() return None ( xi, yi, zi, x_cat_param_pos, x_cat_param_label, y_cat_param_pos, y_cat_param_label, feasible_plot_values, infeasible_plot_values, ) = _calculate_griddata(info) cs = None if len(zi) > 0: if info.xaxis.is_log: ax.set_xscale("log") if info.yaxis.is_log: ax.set_yscale("log") if info.xaxis.name != info.yaxis.name: # Contour the gridded data. ax.contour(xi, yi, zi, 15, linewidths=0.5, colors="k") cs = ax.contourf(xi, yi, zi, 15, cmap=cmap.reversed()) assert isinstance(cs, ContourSet) # Plot data points. ax.scatter( feasible_plot_values.x, feasible_plot_values.y, marker="o", c="black", s=20, edgecolors="grey", linewidth=2.0, ) ax.scatter( infeasible_plot_values.x, infeasible_plot_values.y, marker="o", c="#cccccc", s=20, edgecolors="grey", linewidth=2.0, ) if info.xaxis.is_cat: ax.set_xticks(x_cat_param_pos) ax.set_xticklabels(x_cat_param_label) if info.yaxis.is_cat: ax.set_yticks(y_cat_param_pos) ax.set_yticklabels(y_cat_param_label) ax.label_outer() return cs def _create_zmap( x_values: list[int | float], y_values: list[int | float], z_values: list[float], xi: np.ndarray, yi: np.ndarray, ) -> dict[tuple[int, int], float]: # Creates z-map from trial values and params. # z-map is represented by hashmap of coordinate and trial value pairs. # # Coordinates are represented by tuple of integers, where the first item # indicates x-axis index and the second item indicates y-axis index # and refer to a position of trial value on irregular param grid. # # Since params were resampled either with linspace or logspace # original params might not be on the x and y axes anymore # so we are going with close approximations of trial value positions. zmap = dict() for x, y, z in zip(x_values, y_values, z_values): xindex = int(np.argmin(np.abs(xi - x))) yindex = int(np.argmin(np.abs(yi - y))) zmap[(xindex, yindex)] = z return zmap def _interpolate_zmap(zmap: dict[tuple[int, int], float], contour_plot_num: int) -> np.ndarray: # Implements interpolation formulation used in Plotly # to interpolate heatmaps and contour plots # https://github.com/plotly/plotly.js/blob/95b3bd1bb19d8dc226627442f8f66bce9576def8/src/traces/heatmap/interp2d.js#L15-L20 # citing their doc: # # > Fill in missing data from a 2D array using an iterative # > poisson equation solver with zero-derivative BC at edges. # > Amazingly, this just amounts to repeatedly averaging all the existing # > nearest neighbors # # Plotly's algorithm is equivalent to solve the following linear simultaneous equation. # It is discretization form of the Poisson equation. # # z[x, y] = zmap[(x, y)] (if zmap[(x, y)] is given) # 4 * z[x, y] = z[x-1, y] + z[x+1, y] + z[x, y-1] + z[x, y+1] (if zmap[(x, y)] is not given) a_data = [] a_row = [] a_col = [] b = np.zeros(contour_plot_num**2) for x in range(contour_plot_num): for y in range(contour_plot_num): grid_index = y * contour_plot_num + x if (x, y) in zmap: a_data.append(1) a_row.append(grid_index) a_col.append(grid_index) b[grid_index] = zmap[(x, y)] else: for dx, dy in ((-1, 0), (1, 0), (0, -1), (0, 1)): if 0 <= x + dx < contour_plot_num and 0 <= y + dy < contour_plot_num: a_data.append(1) a_row.append(grid_index) a_col.append(grid_index) a_data.append(-1) a_row.append(grid_index) a_col.append(grid_index + dy * contour_plot_num + dx) z = scipy.sparse.linalg.spsolve(scipy.sparse.csc_matrix((a_data, (a_row, a_col))), b) return z.reshape((contour_plot_num, contour_plot_num)) optuna-4.1.0/optuna/visualization/matplotlib/_edf.py000066400000000000000000000052111471332314300226470ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Callable from collections.abc import Sequence from optuna._experimental import experimental_func from optuna.logging import get_logger from optuna.study import Study from optuna.trial import FrozenTrial from optuna.visualization._edf import _get_edf_info from optuna.visualization.matplotlib._matplotlib_imports import _imports if _imports.is_successful(): from optuna.visualization.matplotlib._matplotlib_imports import Axes from optuna.visualization.matplotlib._matplotlib_imports import plt _logger = get_logger(__name__) @experimental_func("2.2.0") def plot_edf( study: Study | Sequence[Study], *, target: Callable[[FrozenTrial], float] | None = None, target_name: str = "Objective Value", ) -> "Axes": """Plot the objective value EDF (empirical distribution function) of a study with Matplotlib. Note that only the complete trials are considered when plotting the EDF. .. seealso:: Please refer to :func:`optuna.visualization.plot_edf` for an example, where this function can be replaced with it. .. note:: Please refer to `matplotlib.pyplot.legend `_ to adjust the style of the generated legend. Args: study: A target :class:`~optuna.study.Study` object. You can pass multiple studies if you want to compare those EDFs. target: A function to specify the value to display. If it is :obj:`None` and ``study`` is being used for single-objective optimization, the objective values are plotted. .. note:: Specify this argument if ``study`` is being used for multi-objective optimization. target_name: Target's name to display on the axis label. Returns: A :class:`matplotlib.axes.Axes` object. """ _imports.check() # Set up the graph style. plt.style.use("ggplot") # Use ggplot style sheet for similar outputs to plotly. _, ax = plt.subplots() ax.set_title("Empirical Distribution Function Plot") ax.set_xlabel(target_name) ax.set_ylabel("Cumulative Probability") ax.set_ylim(0, 1) cmap = plt.get_cmap("tab20") # Use tab20 colormap for multiple line plots. info = _get_edf_info(study, target, target_name) edf_lines = info.lines if len(edf_lines) == 0: return ax for i, (study_name, y_values) in enumerate(edf_lines): ax.plot(info.x_values, y_values, color=cmap(i), alpha=0.7, label=study_name) if len(edf_lines) >= 2: ax.legend() return ax optuna-4.1.0/optuna/visualization/matplotlib/_hypervolume_history.py000066400000000000000000000047371471332314300262650ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Sequence import numpy as np from optuna._experimental import experimental_func from optuna.study import Study from optuna.visualization._hypervolume_history import _get_hypervolume_history_info from optuna.visualization._hypervolume_history import _HypervolumeHistoryInfo from optuna.visualization.matplotlib._matplotlib_imports import _imports if _imports.is_successful(): from optuna.visualization.matplotlib._matplotlib_imports import Axes from optuna.visualization.matplotlib._matplotlib_imports import plt @experimental_func("3.3.0") def plot_hypervolume_history( study: Study, reference_point: Sequence[float], ) -> "Axes": """Plot hypervolume history of all trials in a study with Matplotlib. .. note:: You need to adjust the size of the plot by yourself using ``plt.tight_layout()`` or ``plt.savefig(IMAGE_NAME, bbox_inches='tight')``. Args: study: A :class:`~optuna.study.Study` object whose trials are plotted for their hypervolumes. The number of objectives must be 2 or more. reference_point: A reference point to use for hypervolume computation. The dimension of the reference point must be the same as the number of objectives. Returns: A :class:`matplotlib.axes.Axes` object. """ _imports.check() if not study._is_multi_objective(): raise ValueError( "Study must be multi-objective. For single-objective optimization, " "please use plot_optimization_history instead." ) if len(reference_point) != len(study.directions): raise ValueError( "The dimension of the reference point must be the same as the number of objectives." ) info = _get_hypervolume_history_info(study, np.asarray(reference_point, dtype=np.float64)) return _get_hypervolume_history_plot(info) def _get_hypervolume_history_plot( info: _HypervolumeHistoryInfo, ) -> "Axes": # Set up the graph style. plt.style.use("ggplot") # Use ggplot style sheet for similar outputs to plotly. _, ax = plt.subplots() ax.set_title("Hypervolume History Plot") ax.set_xlabel("Trial") ax.set_ylabel("Hypervolume") cmap = plt.get_cmap("tab10") # Use tab10 colormap for similar outputs to plotly. ax.plot( info.trial_numbers, info.values, marker="o", color=cmap(0), alpha=0.5, ) return ax optuna-4.1.0/optuna/visualization/matplotlib/_intermediate_values.py000066400000000000000000000043651471332314300261530ustar00rootroot00000000000000from optuna._experimental import experimental_func from optuna.logging import get_logger from optuna.study import Study from optuna.visualization._intermediate_values import _get_intermediate_plot_info from optuna.visualization._intermediate_values import _IntermediatePlotInfo from optuna.visualization.matplotlib._matplotlib_imports import _imports if _imports.is_successful(): from optuna.visualization.matplotlib._matplotlib_imports import Axes from optuna.visualization.matplotlib._matplotlib_imports import plt _logger = get_logger(__name__) @experimental_func("2.2.0") def plot_intermediate_values(study: Study) -> "Axes": """Plot intermediate values of all trials in a study with Matplotlib. .. seealso:: Please refer to :func:`optuna.visualization.plot_intermediate_values` for an example. .. note:: Please refer to `matplotlib.pyplot.legend `__ to adjust the style of the generated legend. Args: study: A :class:`~optuna.study.Study` object whose trials are plotted for their intermediate values. Returns: A :class:`matplotlib.axes.Axes` object. """ _imports.check() return _get_intermediate_plot(_get_intermediate_plot_info(study)) def _get_intermediate_plot(info: _IntermediatePlotInfo) -> "Axes": # Set up the graph style. plt.style.use("ggplot") # Use ggplot style sheet for similar outputs to plotly. _, ax = plt.subplots(tight_layout=True) ax.set_title("Intermediate Values Plot") ax.set_xlabel("Step") ax.set_ylabel("Intermediate Value") cmap = plt.get_cmap("tab20") # Use tab20 colormap for multiple line plots. trial_infos = info.trial_infos for i, tinfo in enumerate(trial_infos): ax.plot( tuple((x for x, _ in tinfo.sorted_intermediate_values)), tuple((y for _, y in tinfo.sorted_intermediate_values)), color=cmap(i) if tinfo.feasible else "#CCCCCC", marker=".", alpha=0.7, label="Trial{}".format(tinfo.trial_number), ) if len(trial_infos) >= 2: ax.legend(bbox_to_anchor=(1.05, 1), loc="upper left", borderaxespad=0.0) return ax optuna-4.1.0/optuna/visualization/matplotlib/_matplotlib_imports.py000066400000000000000000000023361471332314300260420ustar00rootroot00000000000000from packaging import version from optuna._imports import try_import with try_import() as _imports: # TODO(ytknzw): Add specific imports. import matplotlib from matplotlib import __version__ as matplotlib_version from matplotlib import pyplot as plt from matplotlib.axes._axes import Axes from matplotlib.collections import LineCollection from matplotlib.collections import PathCollection from matplotlib.colors import Colormap from matplotlib.contour import ContourSet from matplotlib.figure import Figure # TODO(ytknzw): Set precise version. if version.parse(matplotlib_version) < version.parse("3.0.0"): raise ImportError( "Your version of Matplotlib is " + matplotlib_version + " . " "Please install Matplotlib version 3.0.0 or higher. " "Matplotlib can be installed by executing `$ pip install -U matplotlib>=3.0.0`. " "For further information, please refer to the installation guide of Matplotlib. ", name="matplotlib", ) __all__ = [ "_imports", "matplotlib", "matplotlib_version", "plt", "Axes", "LineCollection", "PathCollection", "Colormap", "ContourSet", "Figure", ] optuna-4.1.0/optuna/visualization/matplotlib/_optimization_history.py000066400000000000000000000130331471332314300264210ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Callable from collections.abc import Sequence import numpy as np from optuna._experimental import experimental_func from optuna.logging import get_logger from optuna.study import Study from optuna.trial import FrozenTrial from optuna.visualization._optimization_history import _get_optimization_history_info_list from optuna.visualization._optimization_history import _OptimizationHistoryInfo from optuna.visualization._optimization_history import _ValueState from optuna.visualization.matplotlib._matplotlib_imports import _imports if _imports.is_successful(): from optuna.visualization.matplotlib._matplotlib_imports import Axes from optuna.visualization.matplotlib._matplotlib_imports import plt _logger = get_logger(__name__) @experimental_func("2.2.0") def plot_optimization_history( study: Study | Sequence[Study], *, target: Callable[[FrozenTrial], float] | None = None, target_name: str = "Objective Value", error_bar: bool = False, ) -> "Axes": """Plot optimization history of all trials in a study with Matplotlib. .. seealso:: Please refer to :func:`optuna.visualization.plot_optimization_history` for an example. .. note:: You need to adjust the size of the plot by yourself using ``plt.tight_layout()`` or ``plt.savefig(IMAGE_NAME, bbox_inches='tight')``. Args: study: A :class:`~optuna.study.Study` object whose trials are plotted for their target values. You can pass multiple studies if you want to compare those optimization histories. target: A function to specify the value to display. If it is :obj:`None` and ``study`` is being used for single-objective optimization, the objective values are plotted. .. note:: Specify this argument if ``study`` is being used for multi-objective optimization. target_name: Target's name to display on the axis label and the legend. error_bar: A flag to show the error bar. Returns: A :class:`matplotlib.axes.Axes` object. """ _imports.check() info_list = _get_optimization_history_info_list(study, target, target_name, error_bar) return _get_optimization_history_plot(info_list, target_name) def _get_optimization_history_plot( info_list: list[_OptimizationHistoryInfo], target_name: str, ) -> "Axes": # Set up the graph style. plt.style.use("ggplot") # Use ggplot style sheet for similar outputs to plotly. _, ax = plt.subplots() ax.set_title("Optimization History Plot") ax.set_xlabel("Trial") ax.set_ylabel(target_name) cmap = plt.get_cmap("tab10") # Use tab10 colormap for similar outputs to plotly. for i, (trial_numbers, values_info, best_values_info) in enumerate(info_list): if values_info.stds is not None: if ( _ValueState.Infeasible in values_info.states or _ValueState.Incomplete in values_info.states ): _logger.warning( "Your study contains infeasible trials. " "In optimization history plot, " "error bars are calculated for only feasible trial values." ) feasible_trial_numbers = trial_numbers feasible_trial_values = values_info.values plt.errorbar( x=feasible_trial_numbers, y=feasible_trial_values, yerr=values_info.stds, capsize=5, fmt="o", color="tab:blue", ) infeasible_trial_numbers: list[int] = [] infeasible_trial_values: list[float] = [] else: feasible_trial_numbers = [ n for n, s in zip(trial_numbers, values_info.states) if s == _ValueState.Feasible ] infeasible_trial_numbers = [ n for n, s in zip(trial_numbers, values_info.states) if s == _ValueState.Infeasible ] feasible_trial_values = [] for num in feasible_trial_numbers: feasible_trial_values.append(values_info.values[num]) infeasible_trial_values = [] for num in infeasible_trial_numbers: infeasible_trial_values.append(values_info.values[num]) ax.scatter( x=feasible_trial_numbers, y=feasible_trial_values, color=cmap(0) if len(info_list) == 1 else cmap(2 * i), alpha=1, label=values_info.label_name, ) if best_values_info is not None: ax.plot( trial_numbers, best_values_info.values, color=cmap(3) if len(info_list) == 1 else cmap(2 * i + 1), alpha=0.5, label=best_values_info.label_name, ) if best_values_info.stds is not None: lower = np.array(best_values_info.values) - np.array(best_values_info.stds) upper = np.array(best_values_info.values) + np.array(best_values_info.stds) ax.fill_between( x=trial_numbers, y1=lower, y2=upper, color="tab:red", alpha=0.4, ) ax.legend() ax.scatter( x=infeasible_trial_numbers, y=infeasible_trial_values, color="#cccccc", ) plt.legend(bbox_to_anchor=(1.05, 1.0), loc="upper left") return ax optuna-4.1.0/optuna/visualization/matplotlib/_parallel_coordinate.py000066400000000000000000000111231471332314300261130ustar00rootroot00000000000000from __future__ import annotations from typing import Callable import numpy as np from optuna._experimental import experimental_func from optuna.study import Study from optuna.trial import FrozenTrial from optuna.visualization._parallel_coordinate import _get_parallel_coordinate_info from optuna.visualization._parallel_coordinate import _ParallelCoordinateInfo from optuna.visualization.matplotlib._matplotlib_imports import _imports if _imports.is_successful(): from optuna.visualization.matplotlib._matplotlib_imports import Axes from optuna.visualization.matplotlib._matplotlib_imports import LineCollection from optuna.visualization.matplotlib._matplotlib_imports import plt @experimental_func("2.2.0") def plot_parallel_coordinate( study: Study, params: list[str] | None = None, *, target: Callable[[FrozenTrial], float] | None = None, target_name: str = "Objective Value", ) -> "Axes": """Plot the high-dimensional parameter relationships in a study with Matplotlib. Note that, if a parameter contains missing values, a trial with missing values is not plotted. .. seealso:: Please refer to :func:`optuna.visualization.plot_parallel_coordinate` for an example. Args: study: A :class:`~optuna.study.Study` object whose trials are plotted for their target values. params: Parameter list to visualize. The default is all parameters. target: A function to specify the value to display. If it is :obj:`None` and ``study`` is being used for single-objective optimization, the objective values are plotted. .. note:: Specify this argument if ``study`` is being used for multi-objective optimization. target_name: Target's name to display on the axis label and the legend. Returns: A :class:`matplotlib.axes.Axes` object. .. note:: The colormap is reversed when the ``target`` argument isn't :obj:`None` or ``direction`` of :class:`~optuna.study.Study` is ``minimize``. """ _imports.check() info = _get_parallel_coordinate_info(study, params, target, target_name) return _get_parallel_coordinate_plot(info) def _get_parallel_coordinate_plot(info: _ParallelCoordinateInfo) -> "Axes": reversescale = info.reverse_scale target_name = info.target_name # Set up the graph style. fig, ax = plt.subplots() cmap = plt.get_cmap("Blues_r" if reversescale else "Blues") ax.set_title("Parallel Coordinate Plot") ax.spines["top"].set_visible(False) ax.spines["bottom"].set_visible(False) # Prepare data for plotting. if len(info.dims_params) == 0 or len(info.dim_objective.values) == 0: return ax obj_min = info.dim_objective.range[0] obj_max = info.dim_objective.range[1] obj_w = obj_max - obj_min dims_obj_base = [[o] for o in info.dim_objective.values] for dim in info.dims_params: p_min = dim.range[0] p_max = dim.range[1] p_w = p_max - p_min if p_w == 0.0: center = obj_w / 2 + obj_min for i in range(len(dim.values)): dims_obj_base[i].append(center) else: for i, v in enumerate(dim.values): dims_obj_base[i].append((v - p_min) / p_w * obj_w + obj_min) # Draw multiple line plots and axes. # Ref: https://stackoverflow.com/a/50029441 n_params = len(info.dims_params) ax.set_xlim(0, n_params) ax.set_ylim(info.dim_objective.range[0], info.dim_objective.range[1]) xs = [range(n_params + 1) for _ in range(len(dims_obj_base))] segments = [np.column_stack([x, y]) for x, y in zip(xs, dims_obj_base)] lc = LineCollection(segments, cmap=cmap) lc.set_array(np.asarray(info.dim_objective.values)) axcb = fig.colorbar(lc, pad=0.1, ax=ax) axcb.set_label(target_name) var_names = [info.dim_objective.label] + [dim.label for dim in info.dims_params] plt.xticks(range(n_params + 1), var_names, rotation=330) for i, dim in enumerate(info.dims_params): ax2 = ax.twinx() if dim.is_log: ax2.set_ylim(np.power(10, dim.range[0]), np.power(10, dim.range[1])) ax2.set_yscale("log") else: ax2.set_ylim(dim.range[0], dim.range[1]) ax2.spines["top"].set_visible(False) ax2.spines["bottom"].set_visible(False) ax2.xaxis.set_visible(False) ax2.spines["right"].set_position(("axes", (i + 1) / n_params)) if dim.is_cat: ax2.set_yticks(dim.tickvals) ax2.set_yticklabels(dim.ticktext) ax.add_collection(lc) return ax optuna-4.1.0/optuna/visualization/matplotlib/_param_importances.py000066400000000000000000000111151471332314300256150ustar00rootroot00000000000000from __future__ import annotations from typing import Callable import numpy as np from optuna._experimental import experimental_func from optuna.importance._base import BaseImportanceEvaluator from optuna.logging import get_logger from optuna.study import Study from optuna.trial import FrozenTrial from optuna.visualization._param_importances import _get_importances_infos from optuna.visualization._param_importances import _ImportancesInfo from optuna.visualization.matplotlib._matplotlib_imports import _imports if _imports.is_successful(): from optuna.visualization.matplotlib._matplotlib_imports import Axes from optuna.visualization.matplotlib._matplotlib_imports import Figure from optuna.visualization.matplotlib._matplotlib_imports import plt _logger = get_logger(__name__) AXES_PADDING_RATIO = 1.05 @experimental_func("2.2.0") def plot_param_importances( study: Study, evaluator: BaseImportanceEvaluator | None = None, params: list[str] | None = None, *, target: Callable[[FrozenTrial], float] | None = None, target_name: str = "Objective Value", ) -> "Axes": """Plot hyperparameter importances with Matplotlib. .. seealso:: Please refer to :func:`optuna.visualization.plot_param_importances` for an example. Args: study: An optimized study. evaluator: An importance evaluator object that specifies which algorithm to base the importance assessment on. Defaults to :class:`~optuna.importance.FanovaImportanceEvaluator`. params: A list of names of parameters to assess. If :obj:`None`, all parameters that are present in all of the completed trials are assessed. target: A function to specify the value to display. If it is :obj:`None` and ``study`` is being used for single-objective optimization, the objective values are plotted. For multi-objective optimization, all objectives will be plotted if ``target`` is :obj:`None`. .. note:: This argument can be used to specify which objective to plot if ``study`` is being used for multi-objective optimization. For example, to get only the hyperparameter importance of the first objective, use ``target=lambda t: t.values[0]`` for the target parameter. target_name: Target's name to display on the axis label. Names set via :meth:`~optuna.study.Study.set_metric_names` will be used if ``target`` is :obj:`None`, overriding this argument. Returns: A :class:`matplotlib.axes.Axes` object. """ _imports.check() importances_infos = _get_importances_infos(study, evaluator, params, target, target_name) return _get_importances_plot(importances_infos) def _get_importances_plot(infos: tuple[_ImportancesInfo, ...]) -> "Axes": # Set up the graph style. plt.style.use("ggplot") # Use ggplot style sheet for similar outputs to plotly. fig, ax = plt.subplots() ax.set_title("Hyperparameter Importances", loc="left") ax.set_xlabel("Hyperparameter Importance") ax.set_ylabel("Hyperparameter") height = 0.8 / len(infos) # Default height split between objectives. for objective_id, info in enumerate(infos): param_names = info.param_names pos = np.arange(len(param_names)) offset = height * objective_id importance_values = info.importance_values if not importance_values: continue # Draw horizontal bars. ax.barh( pos + offset, importance_values, height=height, align="center", label=info.target_name, color=plt.get_cmap("tab20c")(objective_id), ) _set_bar_labels(info, fig, ax, offset) ax.set_yticks(pos + offset / 2, param_names) ax.legend(loc="best") return ax def _set_bar_labels(info: _ImportancesInfo, fig: "Figure", ax: "Axes", offset: float) -> None: renderer = fig.canvas.get_renderer() for idx, (val, label) in enumerate(zip(info.importance_values, info.importance_labels)): text = ax.text(val, idx + offset, label, va="center") # Sometimes horizontal axis needs to be re-scaled # to avoid text going over plot area. bbox = text.get_window_extent(renderer) bbox = bbox.transformed(ax.transData.inverted()) _, plot_xmax = ax.get_xlim() bbox_xmax = bbox.xmax if bbox_xmax > plot_xmax: ax.set_xlim(xmax=AXES_PADDING_RATIO * bbox_xmax) optuna-4.1.0/optuna/visualization/matplotlib/_pareto_front.py000066400000000000000000000201271471332314300246160ustar00rootroot00000000000000from __future__ import annotations from typing import Callable from typing import Sequence from optuna._experimental import experimental_func from optuna.study import Study from optuna.trial import FrozenTrial from optuna.visualization._pareto_front import _get_pareto_front_info from optuna.visualization._pareto_front import _ParetoFrontInfo from optuna.visualization.matplotlib._matplotlib_imports import _imports if _imports.is_successful(): from optuna.visualization.matplotlib._matplotlib_imports import Axes from optuna.visualization.matplotlib._matplotlib_imports import plt @experimental_func("2.8.0") def plot_pareto_front( study: Study, *, target_names: list[str] | None = None, include_dominated_trials: bool = True, axis_order: list[int] | None = None, constraints_func: Callable[[FrozenTrial], Sequence[float]] | None = None, targets: Callable[[FrozenTrial], Sequence[float]] | None = None, ) -> "Axes": """Plot the Pareto front of a study. .. seealso:: Please refer to :func:`optuna.visualization.plot_pareto_front` for an example. Args: study: A :class:`~optuna.study.Study` object whose trials are plotted for their objective values. ``study.n_objectives`` must be either 2 or 3 when ``targets`` is :obj:`None`. target_names: Objective name list used as the axis titles. If :obj:`None` is specified, "Objective {objective_index}" is used instead. If ``targets`` is specified for a study that does not contain any completed trial, ``target_name`` must be specified. include_dominated_trials: A flag to include all dominated trial's objective values. axis_order: A list of indices indicating the axis order. If :obj:`None` is specified, default order is used. ``axis_order`` and ``targets`` cannot be used at the same time. .. warning:: Deprecated in v3.0.0. This feature will be removed in the future. The removal of this feature is currently scheduled for v5.0.0, but this schedule is subject to change. See https://github.com/optuna/optuna/releases/tag/v3.0.0. constraints_func: An optional function that computes the objective constraints. It must take a :class:`~optuna.trial.FrozenTrial` and return the constraints. The return value must be a sequence of :obj:`float` s. A value strictly larger than 0 means that a constraint is violated. A value equal to or smaller than 0 is considered feasible. This specification is the same as in, for example, :class:`~optuna.samplers.NSGAIISampler`. If given, trials are classified into three categories: feasible and best, feasible but non-best, and infeasible. Categories are shown in different colors. Here, whether a trial is best (on Pareto front) or not is determined ignoring all infeasible trials. .. warning:: Deprecated in v4.0.0. This feature will be removed in the future. The removal of this feature is currently scheduled for v6.0.0, but this schedule is subject to change. See https://github.com/optuna/optuna/releases/tag/v4.0.0. targets: A function that returns a tuple of target values to display. The argument to this function is :class:`~optuna.trial.FrozenTrial`. ``targets`` must be :obj:`None` or return 2 or 3 values. ``axis_order`` and ``targets`` cannot be used at the same time. If the number of objectives is neither 2 nor 3, ``targets`` must be specified. .. note:: Added in v3.0.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v3.0.0. Returns: A :class:`matplotlib.axes.Axes` object. """ _imports.check() info = _get_pareto_front_info( study, target_names, include_dominated_trials, axis_order, constraints_func, targets ) return _get_pareto_front_plot(info) def _get_pareto_front_plot(info: _ParetoFrontInfo) -> "Axes": if info.n_targets == 2: return _get_pareto_front_2d(info) elif info.n_targets == 3: return _get_pareto_front_3d(info) else: assert False, "Must not reach here" def _get_pareto_front_2d(info: _ParetoFrontInfo) -> "Axes": # Set up the graph style. plt.style.use("ggplot") # Use ggplot style sheet for similar outputs to plotly. _, ax = plt.subplots() ax.set_title("Pareto-front Plot") cmap = plt.get_cmap("tab10") # Use tab10 colormap for similar outputs to plotly. ax.set_xlabel(info.target_names[info.axis_order[0]]) ax.set_ylabel(info.target_names[info.axis_order[1]]) trial_label: str = "Trial" if len(info.infeasible_trials_with_values) > 0: ax.scatter( x=[values[info.axis_order[0]] for _, values in info.infeasible_trials_with_values], y=[values[info.axis_order[1]] for _, values in info.infeasible_trials_with_values], color="#cccccc", label="Infeasible Trial", ) trial_label = "Feasible Trial" if len(info.non_best_trials_with_values) > 0: ax.scatter( x=[values[info.axis_order[0]] for _, values in info.non_best_trials_with_values], y=[values[info.axis_order[1]] for _, values in info.non_best_trials_with_values], color=cmap(0), label=trial_label, ) if len(info.best_trials_with_values) > 0: ax.scatter( x=[values[info.axis_order[0]] for _, values in info.best_trials_with_values], y=[values[info.axis_order[1]] for _, values in info.best_trials_with_values], color=cmap(3), label="Best Trial", ) if info.non_best_trials_with_values is not None and ax.has_data(): ax.legend() return ax def _get_pareto_front_3d(info: _ParetoFrontInfo) -> "Axes": # Set up the graph style. plt.style.use("ggplot") # Use ggplot style sheet for similar outputs to plotly. fig = plt.figure() ax = fig.add_subplot(projection="3d") ax.set_title("Pareto-front Plot") cmap = plt.get_cmap("tab10") # Use tab10 colormap for similar outputs to plotly. ax.set_xlabel(info.target_names[info.axis_order[0]]) ax.set_ylabel(info.target_names[info.axis_order[1]]) ax.set_zlabel(info.target_names[info.axis_order[2]]) trial_label: str = "Trial" if ( info.infeasible_trials_with_values is not None and len(info.infeasible_trials_with_values) > 0 ): ax.scatter( xs=[values[info.axis_order[0]] for _, values in info.infeasible_trials_with_values], ys=[values[info.axis_order[1]] for _, values in info.infeasible_trials_with_values], zs=[values[info.axis_order[2]] for _, values in info.infeasible_trials_with_values], color="#cccccc", label="Infeasible Trial", ) trial_label = "Feasible Trial" if info.non_best_trials_with_values is not None and len(info.non_best_trials_with_values) > 0: ax.scatter( xs=[values[info.axis_order[0]] for _, values in info.non_best_trials_with_values], ys=[values[info.axis_order[1]] for _, values in info.non_best_trials_with_values], zs=[values[info.axis_order[2]] for _, values in info.non_best_trials_with_values], color=cmap(0), label=trial_label, ) if info.best_trials_with_values is not None and len(info.best_trials_with_values): ax.scatter( xs=[values[info.axis_order[0]] for _, values in info.best_trials_with_values], ys=[values[info.axis_order[1]] for _, values in info.best_trials_with_values], zs=[values[info.axis_order[2]] for _, values in info.best_trials_with_values], color=cmap(3), label="Best Trial", ) if info.non_best_trials_with_values is not None and ax.has_data(): ax.legend() return ax optuna-4.1.0/optuna/visualization/matplotlib/_rank.py000066400000000000000000000107521471332314300230520ustar00rootroot00000000000000from __future__ import annotations from typing import Callable from optuna._experimental import experimental_func from optuna.logging import get_logger from optuna.study import Study from optuna.trial import FrozenTrial from optuna.visualization._rank import _get_rank_info from optuna.visualization._rank import _get_tick_info from optuna.visualization._rank import _RankPlotInfo from optuna.visualization._rank import _RankSubplotInfo from optuna.visualization.matplotlib._matplotlib_imports import _imports if _imports.is_successful(): from optuna.visualization.matplotlib._matplotlib_imports import Axes from optuna.visualization.matplotlib._matplotlib_imports import PathCollection from optuna.visualization.matplotlib._matplotlib_imports import plt _logger = get_logger(__name__) @experimental_func("3.2.0") def plot_rank( study: Study, params: list[str] | None = None, *, target: Callable[[FrozenTrial], float] | None = None, target_name: str = "Objective Value", ) -> "Axes": """Plot parameter relations as scatter plots with colors indicating ranks of target value. Note that trials missing the specified parameters will not be plotted. .. seealso:: Please refer to :func:`optuna.visualization.plot_rank` for an example. Warnings: Output figures of this Matplotlib-based :func:`~optuna.visualization.matplotlib.plot_rank` function would be different from those of the Plotly-based :func:`~optuna.visualization.plot_rank`. Args: study: A :class:`~optuna.study.Study` object whose trials are plotted for their target values. params: Parameter list to visualize. The default is all parameters. target: A function to specify the value to display. If it is :obj:`None` and ``study`` is being used for single-objective optimization, the objective values are plotted. .. note:: Specify this argument if ``study`` is being used for multi-objective optimization. target_name: Target's name to display on the color bar. Returns: A :class:`matplotlib.axes.Axes` object. """ _imports.check() _logger.warning( "Output figures of this Matplotlib-based `plot_rank` function would be different from " "those of the Plotly-based `plot_rank`." ) info = _get_rank_info(study, params, target, target_name) return _get_rank_plot(info) def _get_rank_plot( info: _RankPlotInfo, ) -> "Axes": params = info.params sub_plot_infos = info.sub_plot_infos plt.style.use("ggplot") # Use ggplot style sheet for similar outputs to plotly. title = f"Rank ({info.target_name})" n_params = len(params) if n_params == 0: _, ax = plt.subplots() ax.set_title(title) return ax if n_params == 1 or n_params == 2: fig, axs = plt.subplots() axs.set_title(title) pc = _add_rank_subplot(axs, sub_plot_infos[0][0]) else: fig, axs = plt.subplots(n_params, n_params) fig.suptitle(title) for x_i in range(n_params): for y_i in range(n_params): ax = axs[x_i, y_i] # Set the x or y label only if the subplot is in the edge of the overall figure. pc = _add_rank_subplot( ax, sub_plot_infos[x_i][y_i], set_x_label=x_i == (n_params - 1), set_y_label=y_i == 0, ) tick_info = _get_tick_info(info.zs) pc.set_cmap(plt.get_cmap("RdYlBu_r")) cbar = fig.colorbar(pc, ax=axs, ticks=tick_info.coloridxs) cbar.ax.set_yticklabels(tick_info.text) cbar.outline.set_edgecolor("gray") return axs def _add_rank_subplot( ax: "Axes", info: _RankSubplotInfo, set_x_label: bool = True, set_y_label: bool = True ) -> "PathCollection": if set_x_label: ax.set_xlabel(info.xaxis.name) if set_y_label: ax.set_ylabel(info.yaxis.name) if not info.xaxis.is_cat: ax.set_xlim(info.xaxis.range[0], info.xaxis.range[1]) if not info.yaxis.is_cat: ax.set_ylim(info.yaxis.range[0], info.yaxis.range[1]) if info.xaxis.is_log: ax.set_xscale("log") if info.yaxis.is_log: ax.set_yscale("log") return ax.scatter( x=[str(x) for x in info.xs] if info.xaxis.is_cat else info.xs, y=[str(y) for y in info.ys] if info.yaxis.is_cat else info.ys, c=info.colors / 255, edgecolors="grey", ) optuna-4.1.0/optuna/visualization/matplotlib/_slice.py000066400000000000000000000145401471332314300232150ustar00rootroot00000000000000from __future__ import annotations from collections import defaultdict import math from typing import Any from typing import Callable from optuna._experimental import experimental_func from optuna.study import Study from optuna.trial import FrozenTrial from optuna.visualization._slice import _get_slice_plot_info from optuna.visualization._slice import _PlotValues from optuna.visualization._slice import _SlicePlotInfo from optuna.visualization._slice import _SliceSubplotInfo from optuna.visualization.matplotlib._matplotlib_imports import _imports if _imports.is_successful(): from optuna.visualization.matplotlib._matplotlib_imports import Axes from optuna.visualization.matplotlib._matplotlib_imports import Colormap from optuna.visualization.matplotlib._matplotlib_imports import matplotlib from optuna.visualization.matplotlib._matplotlib_imports import PathCollection from optuna.visualization.matplotlib._matplotlib_imports import plt @experimental_func("2.2.0") def plot_slice( study: Study, params: list[str] | None = None, *, target: Callable[[FrozenTrial], float] | None = None, target_name: str = "Objective Value", ) -> "Axes": """Plot the parameter relationship as slice plot in a study with Matplotlib. .. seealso:: Please refer to :func:`optuna.visualization.plot_slice` for an example. Args: study: A :class:`~optuna.study.Study` object whose trials are plotted for their target values. params: Parameter list to visualize. The default is all parameters. target: A function to specify the value to display. If it is :obj:`None` and ``study`` is being used for single-objective optimization, the objective values are plotted. .. note:: Specify this argument if ``study`` is being used for multi-objective optimization. target_name: Target's name to display on the axis label. Returns: A :class:`matplotlib.axes.Axes` object. """ _imports.check() return _get_slice_plot(_get_slice_plot_info(study, params, target, target_name)) def _get_slice_plot(info: _SlicePlotInfo) -> "Axes": if len(info.subplots) == 0: _, ax = plt.subplots() return ax # Set up the graph style. cmap = plt.get_cmap("Blues") padding_ratio = 0.05 plt.style.use("ggplot") # Use ggplot style sheet for similar outputs to plotly. if len(info.subplots) == 1: # Set up the graph style. fig, axs = plt.subplots() axs.set_title("Slice Plot") # Draw a scatter plot. sc = _generate_slice_subplot(info.subplots[0], axs, cmap, padding_ratio, info.target_name) else: # Set up the graph style. min_figwidth = matplotlib.rcParams["figure.figsize"][0] / 2 fighight = matplotlib.rcParams["figure.figsize"][1] # Ensure that each subplot has a minimum width without relying on auto-sizing. fig, axs = plt.subplots( 1, len(info.subplots), sharey=True, figsize=(min_figwidth * len(info.subplots), fighight), ) fig.suptitle("Slice Plot") # Draw scatter plots. for i, subplot in enumerate(info.subplots): ax = axs[i] sc = _generate_slice_subplot(subplot, ax, cmap, padding_ratio, info.target_name) axcb = fig.colorbar(sc, ax=axs) axcb.set_label("Trial") return axs def _generate_slice_subplot( subplot_info: _SliceSubplotInfo, ax: "Axes", cmap: "Colormap", padding_ratio: float, target_name: str, ) -> "PathCollection": ax.set(xlabel=subplot_info.param_name, ylabel=target_name) scale = None feasible = _PlotValues([], [], []) infeasible = _PlotValues([], [], []) for x, y, num, c in zip( subplot_info.x, subplot_info.y, subplot_info.trial_numbers, subplot_info.constraints ): if x is not None or x != "None" or y is not None or y != "None": if c: feasible.x.append(x) feasible.y.append(y) feasible.trial_numbers.append(num) else: infeasible.x.append(x) infeasible.y.append(y) infeasible.trial_numbers.append(num) if subplot_info.is_log: ax.set_xscale("log") scale = "log" if subplot_info.is_numerical: feasible_x = feasible.x feasible_y = feasible.y feasible_c = feasible.trial_numbers infeasible_x = infeasible.x infeasible_y = infeasible.y else: feasible_x, feasible_y, feasible_c = _get_categorical_plot_values(subplot_info, feasible) infeasible_x, infeasible_y, _ = _get_categorical_plot_values(subplot_info, infeasible) scale = "categorical" xlim = _calc_lim_with_padding(feasible_x + infeasible_x, padding_ratio, scale) ax.set_xlim(xlim[0], xlim[1]) sc = ax.scatter(feasible_x, feasible_y, c=feasible_c, cmap=cmap, edgecolors="grey") ax.scatter(infeasible_x, infeasible_y, c="#cccccc", label="Infeasible Trial") ax.label_outer() return sc def _get_categorical_plot_values( subplot_info: _SliceSubplotInfo, values: _PlotValues ) -> tuple[list[Any], list[float], list[int]]: assert subplot_info.x_labels is not None value_x = [] value_y = [] value_c = [] points_dict = defaultdict(list) for x, y, number in zip(values.x, values.y, values.trial_numbers): points_dict[x].append((y, number)) for x_label in subplot_info.x_labels: for y, number in points_dict[x_label]: value_x.append(str(x_label)) value_y.append(y) value_c.append(number) return value_x, value_y, value_c def _calc_lim_with_padding( values: list[Any], padding_ratio: float, scale: str | None ) -> tuple[float, float]: value_max = max(values) value_min = min(values) if scale == "log": padding = (math.log10(value_max) - math.log10(value_min)) * padding_ratio return ( math.pow(10, math.log10(value_min) - padding), math.pow(10, math.log10(value_max) + padding), ) elif scale == "categorical": width = len(set(values)) - 1 padding = width * padding_ratio return -padding, width + padding else: padding = (value_max - value_min) * padding_ratio return value_min - padding, value_max + padding optuna-4.1.0/optuna/visualization/matplotlib/_terminator_improvement.py000066400000000000000000000102371471332314300267260ustar00rootroot00000000000000from __future__ import annotations from optuna._experimental import experimental_func from optuna.logging import get_logger from optuna.study.study import Study from optuna.terminator import BaseErrorEvaluator from optuna.terminator import BaseImprovementEvaluator from optuna.terminator.improvement.evaluator import DEFAULT_MIN_N_TRIALS from optuna.visualization._terminator_improvement import _get_improvement_info from optuna.visualization._terminator_improvement import _get_y_range from optuna.visualization._terminator_improvement import _ImprovementInfo from optuna.visualization.matplotlib._matplotlib_imports import _imports if _imports.is_successful(): from optuna.visualization.matplotlib._matplotlib_imports import Axes from optuna.visualization.matplotlib._matplotlib_imports import plt _logger = get_logger(__name__) PADDING_RATIO_Y = 0.05 ALPHA = 0.25 @experimental_func("3.2.0") def plot_terminator_improvement( study: Study, plot_error: bool = False, improvement_evaluator: BaseImprovementEvaluator | None = None, error_evaluator: BaseErrorEvaluator | None = None, min_n_trials: int = DEFAULT_MIN_N_TRIALS, ) -> "Axes": """Plot the potentials for future objective improvement. This function visualizes the objective improvement potentials, evaluated with ``improvement_evaluator``. It helps to determine whether we should continue the optimization or not. You can also plot the error evaluated with ``error_evaluator`` if the ``plot_error`` argument is set to :obj:`True`. Note that this function may take some time to compute the improvement potentials. .. seealso:: Please refer to :func:`optuna.visualization.plot_terminator_improvement`. Args: study: A :class:`~optuna.study.Study` object whose trials are plotted for their improvement. plot_error: A flag to show the error. If it is set to :obj:`True`, errors evaluated by ``error_evaluator`` are also plotted as line graph. Defaults to :obj:`False`. improvement_evaluator: An object that evaluates the improvement of the objective function. Default to :class:`~optuna.terminator.RegretBoundEvaluator`. error_evaluator: An object that evaluates the error inherent in the objective function. Default to :class:`~optuna.terminator.CrossValidationErrorEvaluator`. min_n_trials: The minimum number of trials before termination is considered. Terminator improvements for trials below this value are shown in a lighter color. Defaults to ``20``. Returns: A :class:`matplotlib.axes.Axes` object. """ _imports.check() info = _get_improvement_info(study, plot_error, improvement_evaluator, error_evaluator) return _get_improvement_plot(info, min_n_trials) def _get_improvement_plot(info: _ImprovementInfo, min_n_trials: int) -> "Axes": n_trials = len(info.trial_numbers) # Set up the graph style. plt.style.use("ggplot") # Use ggplot style sheet for similar outputs to plotly. _, ax = plt.subplots() ax.set_title("Terminator Improvement Plot") ax.set_xlabel("Trial") ax.set_ylabel("Terminator Improvement") cmap = plt.get_cmap("tab10") # Use tab10 colormap for similar outputs to plotly. if n_trials == 0: _logger.warning("There are no complete trials.") return ax ax.plot( info.trial_numbers[: min_n_trials + 1], info.improvements[: min_n_trials + 1], marker="o", color=cmap(0), alpha=ALPHA, label="Terminator Improvement" if n_trials <= min_n_trials else None, ) if n_trials > min_n_trials: ax.plot( info.trial_numbers[min_n_trials:], info.improvements[min_n_trials:], marker="o", color=cmap(0), label="Terminator Improvement", ) if info.errors is not None: ax.plot( info.trial_numbers, info.errors, marker="o", color=cmap(3), label="Error", ) ax.legend() ax.set_ylim(_get_y_range(info, min_n_trials)) return ax optuna-4.1.0/optuna/visualization/matplotlib/_timeline.py000066400000000000000000000061211471332314300237200ustar00rootroot00000000000000from optuna._experimental import experimental_func from optuna.study import Study from optuna.trial import TrialState from optuna.visualization._timeline import _get_timeline_info from optuna.visualization._timeline import _TimelineBarInfo from optuna.visualization._timeline import _TimelineInfo from optuna.visualization.matplotlib._matplotlib_imports import _imports if _imports.is_successful(): from optuna.visualization.matplotlib._matplotlib_imports import Axes from optuna.visualization.matplotlib._matplotlib_imports import matplotlib from optuna.visualization.matplotlib._matplotlib_imports import plt _INFEASIBLE_KEY = "INFEASIBLE" @experimental_func("3.2.0") def plot_timeline(study: Study) -> "Axes": """Plot the timeline of a study. .. seealso:: Please refer to :func:`optuna.visualization.plot_timeline` for an example. Args: study: A :class:`~optuna.study.Study` object whose trials are plotted with their lifetime. Returns: A :class:`matplotlib.axes.Axes` object. """ _imports.check() info = _get_timeline_info(study) return _get_timeline_plot(info) def _get_state_name(bar_info: _TimelineBarInfo) -> str: if bar_info.state == TrialState.COMPLETE and bar_info.infeasible: return _INFEASIBLE_KEY else: return bar_info.state.name def _get_timeline_plot(info: _TimelineInfo) -> "Axes": _cm = { TrialState.COMPLETE.name: "tab:blue", TrialState.FAIL.name: "tab:red", TrialState.PRUNED.name: "tab:orange", _INFEASIBLE_KEY: "#CCCCCC", TrialState.RUNNING.name: "tab:green", TrialState.WAITING.name: "tab:gray", } # Set up the graph style. plt.style.use("ggplot") # Use ggplot style sheet for similar outputs to plotly. fig, ax = plt.subplots() ax.set_title("Timeline Plot") ax.set_xlabel("Datetime") ax.set_ylabel("Trial") if len(info.bars) == 0: return ax ax.barh( y=[b.number for b in info.bars], width=[b.complete - b.start for b in info.bars], left=[b.start for b in info.bars], color=[_cm[_get_state_name(b)] for b in info.bars], ) # There are 5 types of TrialState in total. # However, the legend depicts only types present in the arguments. legend_handles = [] for state_name, color in _cm.items(): if any(_get_state_name(b) == state_name for b in info.bars): legend_handles.append(matplotlib.patches.Patch(color=color, label=state_name)) ax.legend(handles=legend_handles, loc="upper left", bbox_to_anchor=(1.05, 1.0)) fig.tight_layout() assert len(info.bars) > 0 first_start_time = min([b.start for b in info.bars]) last_complete_time = max([b.complete for b in info.bars]) margin = (last_complete_time - first_start_time) * 0.05 ax.set_xlim(right=last_complete_time + margin, left=first_start_time - margin) ax.yaxis.set_major_locator(matplotlib.ticker.MaxNLocator(integer=True)) ax.xaxis.set_major_formatter(matplotlib.dates.DateFormatter("%H:%M:%S")) plt.gcf().autofmt_xdate() return ax optuna-4.1.0/optuna/visualization/matplotlib/_utils.py000066400000000000000000000034541471332314300232600ustar00rootroot00000000000000from __future__ import annotations from optuna._experimental import experimental_func from optuna.distributions import CategoricalDistribution from optuna.distributions import FloatDistribution from optuna.distributions import IntDistribution from optuna.trial import FrozenTrial from optuna.visualization.matplotlib import _matplotlib_imports __all__ = ["is_available"] @experimental_func("2.2.0") def is_available() -> bool: """Returns whether visualization with Matplotlib is available or not. .. note:: :mod:`~optuna.visualization.matplotlib` module depends on Matplotlib version 3.0.0 or higher. If a supported version of Matplotlib isn't installed in your environment, this function will return :obj:`False`. In such a case, please execute ``$ pip install -U matplotlib>=3.0.0`` to install Matplotlib. Returns: :obj:`True` if visualization with Matplotlib is available, :obj:`False` otherwise. """ return _matplotlib_imports._imports.is_successful() def _is_log_scale(trials: list[FrozenTrial], param: str) -> bool: for trial in trials: if param in trial.params: dist = trial.distributions[param] if isinstance(dist, (FloatDistribution, IntDistribution)): if dist.log: return True return False def _is_categorical(trials: list[FrozenTrial], param: str) -> bool: return any( isinstance(t.distributions[param], CategoricalDistribution) for t in trials if param in t.params ) def _is_numerical(trials: list[FrozenTrial], param: str) -> bool: return all( (isinstance(t.params[param], int) or isinstance(t.params[param], float)) and not isinstance(t.params[param], bool) for t in trials if param in t.params ) optuna-4.1.0/pyproject.toml000066400000000000000000000100451471332314300157370ustar00rootroot00000000000000[build-system] requires = ["setuptools >= 61.1.0", "wheel"] build-backend = "setuptools.build_meta" [project] name = "optuna" description = "A hyperparameter optimization framework" readme = "README.md" authors = [ {name = "Takuya Akiba"} ] classifiers = [ "Development Status :: 5 - Production/Stable", "Intended Audience :: Science/Research", "Intended Audience :: Developers", "License :: OSI Approved :: MIT License", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.8", "Programming Language :: Python :: 3.9", "Programming Language :: Python :: 3.10", "Programming Language :: Python :: 3.11", "Programming Language :: Python :: 3.12", "Programming Language :: Python :: 3.13", "Programming Language :: Python :: 3 :: Only", "Topic :: Scientific/Engineering", "Topic :: Scientific/Engineering :: Mathematics", "Topic :: Scientific/Engineering :: Artificial Intelligence", "Topic :: Software Development", "Topic :: Software Development :: Libraries", "Topic :: Software Development :: Libraries :: Python Modules", ] requires-python = ">=3.8" dependencies = [ "alembic>=1.5.0", "colorlog", "numpy", "packaging>=20.0", "sqlalchemy>=1.4.2", "tqdm", "PyYAML", # Only used in `optuna/cli.py`. ] dynamic = ["version"] [project.optional-dependencies] benchmark = [ "asv>=0.5.0", "cma", "virtualenv" ] checking = [ "black", "blackdoc", "flake8", "isort", "mypy", "mypy_boto3_s3", "types-PyYAML", "types-redis", "types-setuptools", "types-tqdm", "typing_extensions>=3.10.0.0", ] document = [ "ase", "cmaes>=0.10.0", # optuna/samplers/_cmaes.py. "fvcore", "kaleido", "lightgbm", "matplotlib!=3.6.0", "pandas", "pillow", "plotly>=4.9.0", # optuna/visualization. "scikit-learn", "sphinx", "sphinx-copybutton", "sphinx-gallery", "sphinx_rtd_theme>=1.2.0", "torch", "torchvision", ] optional = [ "boto3", # optuna/artifacts/_boto3.py. "cmaes>=0.10.0", # optuna/samplers/_cmaes.py. "google-cloud-storage", # optuna/artifacts/_gcs.py. "matplotlib!=3.6.0", # optuna/visualization/matplotlib. "pandas", # optuna/study.py. "plotly>=4.9.0", # optuna/visualization. "redis", # optuna/storages/redis.py. "scikit-learn>=0.24.2", # optuna/visualization/param_importances.py. "scipy", # optuna/samplers/_gp "torch; python_version<='3.12'", # TODO(gen740): Remove this line when 'torch', a dependency of 'optuna/_gp', supports Python 3.13 ] test = [ "coverage", "fakeredis[lua]", "kaleido", "moto", "pytest", "scipy>=1.9.2", "torch; python_version<='3.12'", # TODO(gen740): Remove this line when 'torch', a dependency of 'optuna/_gp', supports Python 3.13 ] [project.urls] homepage = "https://optuna.org/" repository = "https://github.com/optuna/optuna" documentation = "https://optuna.readthedocs.io" bugtracker = "https://github.com/optuna/optuna/issues" [project.scripts] optuna = "optuna.cli:main" [tool.setuptools.packages.find] include = ["optuna*"] [tool.setuptools.dynamic] version = {attr = "optuna.version.__version__"} [tool.setuptools.package-data] "optuna" = [ "storages/_rdb/alembic.ini", "storages/_rdb/alembic/*.*", "storages/_rdb/alembic/versions/*.*", "py.typed", ] [tool.black] line-length = 99 target-version = ['py38'] force-exclude = ''' /( \.eggs | \.git | \.hg | \.mypy_cache | \.venv | venv | _build | buck-out | build | dist | docs )/ ''' [tool.isort] profile = 'black' src_paths = ['optuna', 'tests', 'docs', 'benchmarks'] skip_glob = [ 'docs/source/conf.py', '**/alembic/versions/*.py', 'tutorial/**/*.py', 'docs/visualization_examples/*.py', 'docs/visualization_matplotlib_examples/*.py', ] line_length = 99 lines_after_imports = 2 force_single_line = 'True' force_sort_within_sections = 'True' order_by_type = 'False' [tool.pytest.ini_options] addopts = "--color=yes" filterwarnings = 'ignore::optuna.exceptions.ExperimentalWarning' markers = [ "skip_coverage: marks tests are skipped when calculating the coverage", "slow: marks tests as slow (deselect with '-m \"not slow\"')", ] optuna-4.1.0/setup.cfg000066400000000000000000000012401471332314300146410ustar00rootroot00000000000000# This section is for flake8. [flake8] ignore = E203, E704, W503 max-line-length = 99 statistics = True exclude = .venv,venv,build,tutorial,.asv,docs/visualization_examples,docs/visualization_matplotlib_examples # This section is for mypy. [mypy] # Options configure mypy's strict mode. warn_unused_configs = True disallow_untyped_calls = True disallow_untyped_defs = True disallow_incomplete_defs = True check_untyped_defs = True no_implicit_optional = True warn_redundant_casts = True strict_equality = True extra_checks = True no_implicit_reexport = True ignore_missing_imports = True exclude = .venv|venv|build|docs|tutorial|optuna/storages/_rdb/alembic optuna-4.1.0/tests/000077500000000000000000000000001471332314300141655ustar00rootroot00000000000000optuna-4.1.0/tests/__init__.py000066400000000000000000000000001471332314300162640ustar00rootroot00000000000000optuna-4.1.0/tests/artifacts_tests/000077500000000000000000000000001471332314300173675ustar00rootroot00000000000000optuna-4.1.0/tests/artifacts_tests/__init__.py000066400000000000000000000000001471332314300214660ustar00rootroot00000000000000optuna-4.1.0/tests/artifacts_tests/stubs.py000066400000000000000000000031111471332314300210750ustar00rootroot00000000000000from __future__ import annotations import copy import io import shutil import threading from typing import TYPE_CHECKING from optuna.artifacts.exceptions import ArtifactNotFound if TYPE_CHECKING: from typing import BinaryIO class FailArtifactStore: def open_reader(self, artifact_id: str) -> BinaryIO: raise Exception("something error raised") def write(self, artifact_id: str, content_body: BinaryIO) -> None: raise Exception("something error raised") def remove(self, artifact_id: str) -> None: raise Exception("something error raised") class InMemoryArtifactStore: def __init__(self) -> None: self._data: dict[str, io.BytesIO] = {} self._lock = threading.Lock() def open_reader(self, artifact_id: str) -> BinaryIO: with self._lock: data = self._data.get(artifact_id) if data is None: raise ArtifactNotFound("not found") return copy.deepcopy(data) def write(self, artifact_id: str, content_body: BinaryIO) -> None: buf = io.BytesIO() shutil.copyfileobj(content_body, buf) buf.seek(0) with self._lock: self._data[artifact_id] = buf def remove(self, artifact_id: str) -> None: with self._lock: if artifact_id not in self._data: raise ArtifactNotFound("not found") del self._data[artifact_id] if TYPE_CHECKING: from optuna.artifacts._protocol import ArtifactStore _fail: ArtifactStore = FailArtifactStore() _inmemory: ArtifactStore = InMemoryArtifactStore() optuna-4.1.0/tests/artifacts_tests/test_backoff.py000066400000000000000000000015331471332314300223750ustar00rootroot00000000000000import io import uuid from optuna.artifacts import Backoff from .stubs import FailArtifactStore from .stubs import InMemoryArtifactStore def test_backoff_time() -> None: backend = Backoff( backend=FailArtifactStore(), min_delay=0.1, multiplier=10, max_delay=10, ) assert backend._get_sleep_secs(0) == 0.1 assert backend._get_sleep_secs(1) == 1 assert backend._get_sleep_secs(2) == 10 def test_read_and_write() -> None: artifact_id = f"test-{uuid.uuid4()}" dummy_content = b"Hello World" backend = Backoff( backend=InMemoryArtifactStore(), min_delay=0.1, multiplier=10, max_delay=10, ) backend.write(artifact_id, io.BytesIO(dummy_content)) with backend.open_reader(artifact_id) as f: actual = f.read() assert actual == dummy_content optuna-4.1.0/tests/artifacts_tests/test_boto3.py000066400000000000000000000054411471332314300220320ustar00rootroot00000000000000from __future__ import annotations import io from typing import TYPE_CHECKING import boto3 from moto import mock_aws import pytest from optuna.artifacts import Boto3ArtifactStore from optuna.artifacts.exceptions import ArtifactNotFound if TYPE_CHECKING: from collections.abc import Iterator from mypy_boto3_s3 import S3Client from typing_extensions import Annotated # TODO(Shinichi) import Annotated from typing after python 3.8 support is dropped. @pytest.fixture() def init_mock_client() -> Iterator[tuple[str, S3Client]]: with mock_aws(): # Runs before each test bucket_name = "moto-bucket" s3_client = boto3.client("s3") s3_client.create_bucket(Bucket=bucket_name) yield bucket_name, s3_client # Runs after each test objects = s3_client.list_objects(Bucket=bucket_name).get("Contents", []) if objects: s3_client.delete_objects( Bucket=bucket_name, Delete={"Objects": [{"Key": obj["Key"] for obj in objects}], "Quiet": True}, ) s3_client.delete_bucket(Bucket=bucket_name) @pytest.mark.parametrize("avoid_buf_copy", [True, False]) def test_upload_download( init_mock_client: Annotated[tuple[str, S3Client], pytest.fixture], avoid_buf_copy: bool, ) -> None: bucket_name, s3_client = init_mock_client backend = Boto3ArtifactStore(bucket_name, avoid_buf_copy=avoid_buf_copy) artifact_id = "dummy-uuid" dummy_content = b"Hello World" buf = io.BytesIO(dummy_content) backend.write(artifact_id, buf) assert len(s3_client.list_objects(Bucket=bucket_name)["Contents"]) == 1 obj = s3_client.get_object(Bucket=bucket_name, Key=artifact_id) assert obj["Body"].read() == dummy_content with backend.open_reader(artifact_id) as f: actual = f.read() assert actual == dummy_content if avoid_buf_copy is False: assert buf.closed is False def test_remove(init_mock_client: Annotated[tuple[str, S3Client], pytest.fixture]) -> None: bucket_name, s3_client = init_mock_client backend = Boto3ArtifactStore(bucket_name) artifact_id = "dummy-uuid" backend.write(artifact_id, io.BytesIO(b"Hello")) objects = s3_client.list_objects(Bucket=bucket_name)["Contents"] assert len([obj for obj in objects if obj["Key"] == artifact_id]) == 1 backend.remove(artifact_id) objects = s3_client.list_objects(Bucket=bucket_name).get("Contents", []) assert len([obj for obj in objects if obj["Key"] == artifact_id]) == 0 def test_file_not_found_exception( init_mock_client: Annotated[tuple[str, S3Client], pytest.fixture] ) -> None: bucket_name, _ = init_mock_client backend = Boto3ArtifactStore(bucket_name) with pytest.raises(ArtifactNotFound): backend.open_reader("not-found-id") optuna-4.1.0/tests/artifacts_tests/test_download_artifact.py000066400000000000000000000031751471332314300244720ustar00rootroot00000000000000from __future__ import annotations import pathlib import pytest import optuna from optuna.artifacts import download_artifact from optuna.artifacts import FileSystemArtifactStore from optuna.artifacts import upload_artifact from optuna.artifacts._protocol import ArtifactStore @pytest.fixture(params=["FileSystem"]) def artifact_store(tmp_path: pathlib.PurePath, request: pytest.FixtureRequest) -> ArtifactStore: if request.param == "FileSystem": return FileSystemArtifactStore(str(tmp_path)) assert False, f"Unknown artifact store: {request.param}" def test_download_artifact(tmp_path: pathlib.PurePath, artifact_store: ArtifactStore) -> None: study = optuna.create_study() artifact_ids: list[str] = [] def objective(trial: optuna.Trial) -> float: x = trial.suggest_int("x", 0, 100) y = trial.suggest_int("y", 0, 100) dummy_file = str(tmp_path / f"dummy_{trial.number}.txt") with open(dummy_file, "w") as f: f.write(f"{x} {y}") artifact_ids.append( upload_artifact( study_or_trial=trial, file_path=dummy_file, artifact_store=artifact_store ) ) return x**2 + y**2 study.optimize(objective, n_trials=5) for i, artifact_id in enumerate(artifact_ids): dummy_downloaded_file = str(tmp_path / f"dummy_downloaded_{i}.txt") download_artifact( file_path=dummy_downloaded_file, artifact_store=artifact_store, artifact_id=artifact_id ) with open(dummy_downloaded_file, "r") as f: assert f.read() == f"{study.trials[i].params['x']} {study.trials[i].params['y']}" optuna-4.1.0/tests/artifacts_tests/test_filesystem.py000066400000000000000000000023221471332314300231630ustar00rootroot00000000000000import io from pathlib import Path import pytest from optuna.artifacts import FileSystemArtifactStore from optuna.artifacts.exceptions import ArtifactNotFound def test_upload_download(tmp_path: Path) -> None: artifact_id = "dummy-uuid" dummy_content = b"Hello World" backend = FileSystemArtifactStore(tmp_path) backend.write(artifact_id, io.BytesIO(dummy_content)) with backend.open_reader(artifact_id) as f: actual = f.read() assert actual == dummy_content def test_remove(tmp_path: Path) -> None: artifact_id = "dummy-uuid" dummy_content = b"Hello World" backend = FileSystemArtifactStore(tmp_path) backend.write(artifact_id, io.BytesIO(dummy_content)) objects = list(tmp_path.glob("*")) assert len([obj for obj in objects if obj.name == artifact_id]) == 1 backend.remove(artifact_id) objects = list(tmp_path.glob("*")) assert len([obj for obj in objects if obj.name == artifact_id]) == 0 def test_file_not_found(tmp_path: str) -> None: backend = FileSystemArtifactStore(tmp_path) with pytest.raises(ArtifactNotFound): backend.open_reader("not-found-id") with pytest.raises(ArtifactNotFound): backend.remove("not-found-id") optuna-4.1.0/tests/artifacts_tests/test_gcs.py000066400000000000000000000071701471332314300215610ustar00rootroot00000000000000from __future__ import annotations import contextlib import io import os from typing import Dict from typing import Optional from typing import TYPE_CHECKING from unittest.mock import patch import google.cloud.storage import pytest from optuna.artifacts import GCSArtifactStore from optuna.artifacts.exceptions import ArtifactNotFound if TYPE_CHECKING: from collections.abc import Iterator _MOCK_BUCKET_CONTENT: Dict[str, bytes] = dict() class MockBucket: def get_blob(self, blob_name: str) -> Optional["MockBlob"]: if blob_name in _MOCK_BUCKET_CONTENT: return MockBlob(blob_name) else: return None def blob(self, blob_name: str) -> "MockBlob": return MockBlob(blob_name) def delete_blob(self, blob_name: str) -> None: global _MOCK_BUCKET_CONTENT del _MOCK_BUCKET_CONTENT[blob_name] def list_blobs(self) -> Iterator["MockBlob"]: for blob_name in _MOCK_BUCKET_CONTENT.keys(): yield MockBlob(blob_name) class MockBlob: def __init__(self, blob_name: str) -> None: self.blob_name = blob_name def download_as_bytes(self) -> bytes: return _MOCK_BUCKET_CONTENT[self.blob_name] def upload_from_string(self, data: bytes) -> None: global _MOCK_BUCKET_CONTENT _MOCK_BUCKET_CONTENT[self.blob_name] = data @contextlib.contextmanager def init_mock_client() -> Iterator[None]: # In case we fail to patch `google.cloud.storage.Client`, we deliberately set an invalid # credential path so that we do not accidentally access GCS. # Note that this is not a perfect measure; it can become ineffective in future when the # mechanism for finding the default credential is changed in the Cloud Storage API. os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "/dev/null" with patch("google.cloud.storage.Client") as MockClient: instance = MockClient.return_value def bucket(name: str) -> MockBucket: assert name == "mock-bucket" return MockBucket() instance.bucket.side_effect = bucket yield @pytest.mark.parametrize("explicit_client", [False, True]) def test_upload_download(explicit_client: bool) -> None: with init_mock_client(): bucket_name = "mock-bucket" if explicit_client: backend = GCSArtifactStore(bucket_name, google.cloud.storage.Client()) else: backend = GCSArtifactStore(bucket_name) artifact_id = "dummy-uuid" dummy_content = b"Hello World" buf = io.BytesIO(dummy_content) backend.write(artifact_id, buf) client = google.cloud.storage.Client() assert len(list(client.bucket(bucket_name).list_blobs())) == 1 blob = client.bucket(bucket_name).blob(artifact_id) assert blob.download_as_bytes() == dummy_content with backend.open_reader(artifact_id) as f: actual = f.read() assert actual == dummy_content def test_remove() -> None: with init_mock_client(): bucket_name = "mock-bucket" backend = GCSArtifactStore(bucket_name) client = google.cloud.storage.Client() artifact_id = "dummy-uuid" backend.write(artifact_id, io.BytesIO(b"Hello")) assert len(list(client.bucket(bucket_name).list_blobs())) == 1 backend.remove(artifact_id) assert len(list(client.bucket(bucket_name).list_blobs())) == 0 def test_file_not_found_exception() -> None: with init_mock_client(): bucket_name = "mock-bucket" backend = GCSArtifactStore(bucket_name) with pytest.raises(ArtifactNotFound): backend.open_reader("not-found-id") optuna-4.1.0/tests/artifacts_tests/test_list_artifact_meta.py000066400000000000000000000100111471332314300246270ustar00rootroot00000000000000from __future__ import annotations import pathlib import pytest import optuna from optuna.artifacts import FileSystemArtifactStore from optuna.artifacts import get_all_artifact_meta from optuna.artifacts import upload_artifact from optuna.artifacts._protocol import ArtifactStore from optuna.storages import BaseStorage from optuna.study import Study from optuna.trial import FrozenTrial from optuna.trial import Trial parametrize_artifact_meta = pytest.mark.parametrize( "filename,mimetype,expected_mimetype,encoding", [("dummy.txt", None, "text/plain", None), ("dummy.obj", "model/obj", "model/obj", "utf-8")], ) @pytest.fixture(params=["FileSystem"]) def artifact_store(tmp_path: pathlib.PurePath, request: pytest.FixtureRequest) -> ArtifactStore: if request.param == "FileSystem": return FileSystemArtifactStore(str(tmp_path)) assert False, f"Unknown artifact store: {request.param}" def _check_uploaded_artifact_meta( study_or_trial: Study | Trial | FrozenTrial, storage: BaseStorage, artifact_store: ArtifactStore, filename: str, file_path: str, mimetype: str | None, expected_mimetype: str, encoding: str | None, ) -> None: artifact_id = upload_artifact( study_or_trial=study_or_trial, file_path=file_path, artifact_store=artifact_store, storage=storage, mimetype=mimetype, encoding=encoding, ) artifact_meta_list = get_all_artifact_meta(study_or_trial, storage=storage) assert len(artifact_meta_list) == 1 assert artifact_meta_list[0].artifact_id == artifact_id assert artifact_meta_list[0].filename == filename assert artifact_meta_list[0].mimetype == expected_mimetype assert artifact_meta_list[0].encoding == encoding @parametrize_artifact_meta def test_get_all_artifact_meta_in_trial( tmp_path: pathlib.PurePath, artifact_store: ArtifactStore, filename: str, mimetype: str | None, expected_mimetype: str, encoding: str | None, ) -> None: file_path = str(tmp_path / filename) with open(file_path, "w") as f: f.write("foo") storage = optuna.storages.InMemoryStorage() study = optuna.create_study(storage=storage) trial = study.ask() _check_uploaded_artifact_meta( study_or_trial=trial, storage=storage, artifact_store=artifact_store, filename=filename, file_path=file_path, mimetype=mimetype, expected_mimetype=expected_mimetype, encoding=encoding, ) @parametrize_artifact_meta def test_get_all_artifact_meta_in_frozen_trial( tmp_path: pathlib.PurePath, artifact_store: ArtifactStore, filename: str, mimetype: str | None, expected_mimetype: str, encoding: str | None, ) -> None: file_path = str(tmp_path / filename) with open(file_path, "w") as f: f.write("foo") storage = optuna.storages.InMemoryStorage() study = optuna.create_study(storage=storage) trial = study.ask() frozen_trial = study._storage.get_trial(trial._trial_id) _check_uploaded_artifact_meta( study_or_trial=frozen_trial, storage=storage, artifact_store=artifact_store, filename=filename, file_path=file_path, mimetype=mimetype, expected_mimetype=expected_mimetype, encoding=encoding, ) @parametrize_artifact_meta def test_get_all_artifact_meta_in_study( tmp_path: pathlib.PurePath, artifact_store: ArtifactStore, filename: str, mimetype: str | None, expected_mimetype: str, encoding: str | None, ) -> None: file_path = str(tmp_path / filename) with open(file_path, "w") as f: f.write("foo") storage = optuna.storages.InMemoryStorage() study = optuna.create_study(storage=storage) _check_uploaded_artifact_meta( study_or_trial=study, storage=storage, artifact_store=artifact_store, filename=filename, file_path=file_path, mimetype=mimetype, expected_mimetype=expected_mimetype, encoding=encoding, ) optuna-4.1.0/tests/artifacts_tests/test_upload_artifact.py000066400000000000000000000114661471332314300241510ustar00rootroot00000000000000from __future__ import annotations import pathlib import pytest import optuna from optuna.artifacts import FileSystemArtifactStore from optuna.artifacts import get_all_artifact_meta from optuna.artifacts import upload_artifact from optuna.artifacts._protocol import ArtifactStore @pytest.fixture(params=["FileSystem"]) def artifact_store(tmp_path: pathlib.PurePath, request: pytest.FixtureRequest) -> ArtifactStore: if request.param == "FileSystem": return FileSystemArtifactStore(str(tmp_path)) assert False, f"Unknown artifact store: {request.param}" def test_upload_trial_artifact(tmp_path: pathlib.PurePath, artifact_store: ArtifactStore) -> None: file_path = str(tmp_path / "dummy.txt") with open(file_path, "w") as f: f.write("foo") storage = optuna.storages.InMemoryStorage() study = optuna.create_study(storage=storage) trial = study.ask() upload_artifact(study_or_trial=trial, file_path=file_path, artifact_store=artifact_store) frozen_trial = study._storage.get_trial(trial._trial_id) with pytest.raises(ValueError): upload_artifact( study_or_trial=frozen_trial, file_path=file_path, artifact_store=artifact_store ) upload_artifact( study_or_trial=frozen_trial, file_path=file_path, artifact_store=artifact_store, storage=trial.study._storage, ) artifact_items = get_all_artifact_meta(frozen_trial, storage=storage) assert len(artifact_items) == 2 assert artifact_items[0].artifact_id != artifact_items[1].artifact_id assert artifact_items[0].filename == "dummy.txt" assert artifact_items[0].mimetype == "text/plain" assert artifact_items[0].encoding is None def test_upload_study_artifact(tmp_path: pathlib.PurePath, artifact_store: ArtifactStore) -> None: file_path = str(tmp_path / "dummy.txt") with open(file_path, "w") as f: f.write("foo") storage = optuna.storages.InMemoryStorage() study = optuna.create_study(storage=storage) artifact_id = upload_artifact( study_or_trial=study, file_path=file_path, artifact_store=artifact_store ) artifact_items = get_all_artifact_meta(study) assert len(artifact_items) == 1 assert artifact_items[0].artifact_id == artifact_id assert artifact_items[0].filename == "dummy.txt" assert artifact_items[0].mimetype == "text/plain" assert artifact_items[0].encoding is None def test_upload_artifact_with_mimetype( tmp_path: pathlib.PurePath, artifact_store: ArtifactStore ) -> None: file_path = str(tmp_path / "dummy.obj") with open(file_path, "w") as f: f.write("foo") study = optuna.create_study() trial = study.ask() upload_artifact( study_or_trial=trial, file_path=file_path, artifact_store=artifact_store, mimetype="model/obj", encoding="utf-8", ) frozen_trial = study._storage.get_trial(trial._trial_id) with pytest.raises(ValueError): upload_artifact( study_or_trial=frozen_trial, file_path=file_path, artifact_store=artifact_store ) upload_artifact( study_or_trial=frozen_trial, file_path=file_path, artifact_store=artifact_store, storage=trial.study._storage, ) artifact_items = get_all_artifact_meta(frozen_trial, storage=study._storage) assert len(artifact_items) == 2 assert artifact_items[0].artifact_id != artifact_items[1].artifact_id assert artifact_items[0].filename == "dummy.obj" assert artifact_items[0].mimetype == "model/obj" assert artifact_items[0].encoding == "utf-8" def test_upload_artifact_with_positional_args( tmp_path: pathlib.PurePath, artifact_store: ArtifactStore ) -> None: storage = optuna.storages.InMemoryStorage() study = optuna.create_study(storage=storage) trial = study.ask() def _validate(artifact_id: str) -> None: artifact_items = get_all_artifact_meta(trial, storage=storage) assert artifact_items[-1].artifact_id == artifact_id assert artifact_items[-1].filename == "dummy.txt" assert artifact_items[-1].mimetype == "text/plain" assert artifact_items[-1].encoding is None file_path = str(tmp_path / "dummy.txt") with open(file_path, "w") as f: f.write("foo") with pytest.warns(FutureWarning): artifact_id = upload_artifact(trial, file_path, artifact_store) # type: ignore _validate(artifact_id=artifact_id) with pytest.warns(FutureWarning): artifact_id = upload_artifact( trial, file_path, artifact_store=artifact_store # type: ignore ) _validate(artifact_id=artifact_id) with pytest.warns(FutureWarning): artifact_id = upload_artifact( trial, file_path=file_path, artifact_store=artifact_store # type: ignore ) _validate(artifact_id=artifact_id) optuna-4.1.0/tests/gp_tests/000077500000000000000000000000001471332314300160155ustar00rootroot00000000000000optuna-4.1.0/tests/gp_tests/test_acqf.py000066400000000000000000000036071471332314300203460ustar00rootroot00000000000000from __future__ import annotations import numpy as np import pytest import torch from optuna._gp.acqf import AcquisitionFunctionType from optuna._gp.acqf import create_acqf_params from optuna._gp.acqf import eval_acqf from optuna._gp.gp import KernelParamsTensor from optuna._gp.search_space import ScaleType from optuna._gp.search_space import SearchSpace @pytest.mark.parametrize( "acqf_type, beta", [ (AcquisitionFunctionType.LOG_EI, None), (AcquisitionFunctionType.UCB, 2.0), (AcquisitionFunctionType.LCB, 2.0), ], ) @pytest.mark.parametrize( "x", [np.array([0.15, 0.12]), np.array([[0.15, 0.12], [0.0, 1.0]])] # unbatched # batched ) def test_eval_acqf( acqf_type: AcquisitionFunctionType, beta: float | None, x: np.ndarray, ) -> None: n_dims = 2 X = np.array([[0.1, 0.2], [0.2, 0.3], [0.3, 0.1]]) Y = np.array([1.0, 2.0, 3.0]) kernel_params = KernelParamsTensor( inverse_squared_lengthscales=torch.tensor([2.0, 3.0], dtype=torch.float64), kernel_scale=torch.tensor(4.0, dtype=torch.float64), noise_var=torch.tensor(0.1, dtype=torch.float64), ) search_space = SearchSpace( scale_types=np.full(n_dims, ScaleType.LINEAR), bounds=np.array([[0.0, 1.0] * n_dims]), steps=np.zeros(n_dims), ) acqf_params = create_acqf_params( acqf_type=acqf_type, kernel_params=kernel_params, search_space=search_space, X=X, Y=Y, beta=beta, acqf_stabilizing_noise=0.0, ) x_tensor = torch.from_numpy(x) x_tensor.requires_grad_(True) acqf_value = eval_acqf(acqf_params, x_tensor) acqf_value.sum().backward() # type: ignore acqf_grad = x_tensor.grad assert acqf_grad is not None assert acqf_value.shape == x.shape[:-1] assert torch.all(torch.isfinite(acqf_value)) assert torch.all(torch.isfinite(acqf_grad)) optuna-4.1.0/tests/gp_tests/test_gp.py000066400000000000000000000043611471332314300200400ustar00rootroot00000000000000from __future__ import annotations import numpy as np import pytest import torch from optuna._gp.gp import _fit_kernel_params from optuna._gp.gp import KernelParamsTensor import optuna._gp.prior as prior @pytest.mark.parametrize( "X, Y, is_categorical", [ ( np.array([[0.1, 0.2], [0.2, 0.3], [0.3, 0.1]]), np.array([1.0, 2.0, 3.0]), np.array([False, False]), ), ( np.array([[0.1, 0.2, 0.0], [0.2, 0.3, 1.0]]), np.array([1.0, 2.0]), np.array([False, False, True]), ), (np.array([[1.0, 0.0], [0.0, 1.0]]), np.array([1.0, 2.0]), np.array([True, True])), (np.array([[0.0]]), np.array([0.0]), np.array([True])), (np.array([[0.0]]), np.array([0.0]), np.array([False])), ], ) @pytest.mark.parametrize("deterministic_objective", [True, False]) @pytest.mark.parametrize("torch_set_grad_enabled", [True, False]) def test_fit_kernel_params( X: np.ndarray, Y: np.ndarray, is_categorical: np.ndarray, deterministic_objective: bool, torch_set_grad_enabled: bool, ) -> None: with torch.set_grad_enabled(torch_set_grad_enabled): log_prior = prior.default_log_prior minimum_noise = prior.DEFAULT_MINIMUM_NOISE_VAR initial_kernel_params = KernelParamsTensor( inverse_squared_lengthscales=torch.ones(X.shape[1], dtype=torch.float64), kernel_scale=torch.tensor(1.0, dtype=torch.float64), noise_var=torch.tensor(1.0, dtype=torch.float64), ) gtol: float = 1e-2 kernel_params = _fit_kernel_params( X=X, Y=Y, is_categorical=is_categorical, log_prior=log_prior, minimum_noise=minimum_noise, initial_kernel_params=initial_kernel_params, deterministic_objective=deterministic_objective, gtol=gtol, ) assert ( ( kernel_params.inverse_squared_lengthscales != initial_kernel_params.inverse_squared_lengthscales ).sum() + (kernel_params.kernel_scale != initial_kernel_params.kernel_scale).sum() + (kernel_params.noise_var != initial_kernel_params.noise_var).sum() ) optuna-4.1.0/tests/gp_tests/test_search_space.py000066400000000000000000000144421471332314300220530ustar00rootroot00000000000000from __future__ import annotations import numpy as np import pytest import optuna from optuna._gp.search_space import get_search_space_and_normalized_params from optuna._gp.search_space import get_unnormalized_param from optuna._gp.search_space import normalize_one_param from optuna._gp.search_space import round_one_normalized_param from optuna._gp.search_space import sample_normalized_params from optuna._gp.search_space import ScaleType from optuna._gp.search_space import SearchSpace from optuna._gp.search_space import unnormalize_one_param from optuna._transform import _SearchSpaceTransform @pytest.mark.parametrize( "scale_type,bounds,step,unnormalized,normalized", [ (ScaleType.LINEAR, (0.0, 10.0), 0.0, 2.0, 0.2), (ScaleType.LINEAR, (0.0, 9.0), 1.0, 2.0, 0.25), (ScaleType.LINEAR, (0.5, 8.5), 2.0, 2.5, 0.3), (ScaleType.LINEAR, (0.0, 0.0), 0.0, 0.0, 0.5), (ScaleType.LOG, (10**0.0, 10**10.0), 0.0, 10**2.0, 0.2), ( ScaleType.LOG, (1.0, 10.0), 1.0, 2.0, (np.log(2.0) - np.log(0.5)) / (np.log(10.5) - np.log(0.5)), ), (ScaleType.CATEGORICAL, (0.0, 10.0), 0.0, 3.0, 3.0), ], ) def test_normalize_unnormalize_one_param( scale_type: ScaleType, bounds: tuple[float, float], step: float, unnormalized: float, normalized: float, ) -> None: assert np.isclose( normalize_one_param( np.array(unnormalized), scale_type, bounds, step, ), normalized, ) assert np.isclose( unnormalize_one_param( np.array(normalized), scale_type, bounds, step, ), unnormalized, ) @pytest.mark.parametrize( "scale_type,bounds,step,value,expected", [ (ScaleType.LINEAR, (0.0, 9.0), 1.0, 0.21, 0.25), ( ScaleType.LOG, (1.0, 10.0), 1.0, (np.log(1.8) - np.log(0.5)) / (np.log(10.5) - np.log(0.5)), (np.log(2.0) - np.log(0.5)) / (np.log(10.5) - np.log(0.5)), ), (ScaleType.LINEAR, (-1, 1), 0.5, 0.0, 0.1), (ScaleType.LINEAR, (-1, 1), 0.5, 1.0, 0.9), (ScaleType.LINEAR, (-0.1, 0.7), 0.4, -0.1, 1 / 6), (ScaleType.LINEAR, (-0.1, 0.7), 0.4, 0.7, 5 / 6), ], ) def test_round_one_normalized_param( scale_type: ScaleType, bounds: tuple[float, float], step: float, value: float, expected: float ) -> None: res = round_one_normalized_param( np.array(value), scale_type, bounds, step, ) assert np.isclose(res, expected) assert 0.0 <= res <= 1.0 def test_sample_normalized_params() -> None: search_space = SearchSpace( scale_types=np.array( [ ScaleType.LINEAR, ScaleType.LINEAR, ScaleType.LOG, ScaleType.LOG, ScaleType.CATEGORICAL, ] ), bounds=np.array([(0.0, 10.0), (1.0, 10.0), (10.0, 100.0), (10.0, 100.0), (0.0, 5.0)]), steps=np.array([0.0, 1.0, 0.0, 1.0, 1.0]), ) samples = sample_normalized_params( n=128, search_space=search_space, rng=np.random.RandomState(0) ) assert samples.shape == (128, 5) assert np.all((samples[:, :4] >= 0.0) & (samples[:, :4] <= 1.0)) integer_params = [1, 3, 4] for i in integer_params: params = unnormalize_one_param( samples[:, i], search_space.scale_types[i], search_space.bounds[i], search_space.steps[i], ) # assert params are close to integers assert np.allclose((params + 0.5) % 1.0, 0.5) def test_get_search_space_and_normalized_params_no_categorical() -> None: optuna_search_space = { "a": optuna.distributions.FloatDistribution(0.0, 10.0), "b": optuna.distributions.IntDistribution(0, 10), "c": optuna.distributions.FloatDistribution(1.0, 10.0, log=True), "d": optuna.distributions.IntDistribution(1, 10, log=True), "e": optuna.distributions.CategoricalDistribution(["x", "y", "z"]), } trials = [ optuna.create_trial( params={"a": 2.0, "b": 2, "c": 2.0, "d": 2, "e": "x"}, distributions=optuna_search_space, value=0.0, ) ] search_space, normalized_params = get_search_space_and_normalized_params( trials, optuna_search_space ) assert np.all( search_space.scale_types == np.array( [ ScaleType.LINEAR, ScaleType.LINEAR, ScaleType.LOG, ScaleType.LOG, ScaleType.CATEGORICAL, ] ) ) assert np.all( search_space.bounds == np.array([(0.0, 10.0), (0.0, 10.0), (1.0, 10.0), (1.0, 10.0), (0.0, 3.0)]) ) assert np.all(search_space.steps == np.array([0.0, 1.0, 0.0, 1.0, 1.0])) non_categorical_search_space = { param: dist for param, dist in optuna_search_space.items() if not isinstance(dist, optuna.distributions.CategoricalDistribution) } search_space_transform = _SearchSpaceTransform( search_space=non_categorical_search_space, transform_log=True, transform_step=True, transform_0_1=True, ) expected = search_space_transform.transform(trials[0].params) assert np.allclose(normalized_params[:, :4], expected) assert normalized_params[0, 4] == 0.0 def test_get_untransform_search_space() -> None: optuna_search_space = { "a": optuna.distributions.FloatDistribution(0.0, 10.0), "b": optuna.distributions.IntDistribution(0, 9), "c": optuna.distributions.FloatDistribution(2.0**0, 2.0**10, log=True), "d": optuna.distributions.IntDistribution(1, 10, log=True), "e": optuna.distributions.CategoricalDistribution(["x", "y", "z"]), } normalized_values = np.array( [ 0.25, 0.25, 0.5, (np.log(2.0) - np.log(0.5)) / (np.log(10.5) - np.log(0.5)), 0.0, ] ) params = get_unnormalized_param(optuna_search_space, normalized_values) expected = { "a": 2.5, "b": 2, "c": 2.0**5, "d": 2, "e": "x", } assert params == expected optuna-4.1.0/tests/hypervolume_tests/000077500000000000000000000000001471332314300177665ustar00rootroot00000000000000optuna-4.1.0/tests/hypervolume_tests/__init__.py000066400000000000000000000000001471332314300220650ustar00rootroot00000000000000optuna-4.1.0/tests/hypervolume_tests/test_hssp.py000066400000000000000000000037511471332314300223620ustar00rootroot00000000000000import itertools import math from typing import Tuple import numpy as np import pytest import optuna def _compute_hssp_truth_and_approx(test_case: np.ndarray, subset_size: int) -> Tuple[float, float]: r = 1.1 * np.max(test_case, axis=0) truth = 0.0 for subset in itertools.permutations(test_case, subset_size): hv = optuna._hypervolume.compute_hypervolume(np.asarray(subset), r) assert not math.isnan(hv) truth = max(truth, hv) indices = optuna._hypervolume.hssp._solve_hssp( test_case, np.arange(len(test_case)), subset_size, r ) approx = optuna._hypervolume.compute_hypervolume(test_case[indices], r) assert not math.isnan(approx) return truth, approx @pytest.mark.parametrize("dim", [2, 3]) def test_solve_hssp(dim: int) -> None: rng = np.random.RandomState(128) for i in range(1, 9): subset_size = np.random.randint(1, i + 1) test_case = rng.rand(8, dim) truth, approx = _compute_hssp_truth_and_approx(test_case, subset_size) assert approx / truth > 0.6321 # 1 - 1/e def test_solve_hssp_infinite_loss() -> None: rng = np.random.RandomState(128) subset_size = 4 for dim in range(2, 4): test_case = rng.rand(9, dim) test_case[-1].fill(float("inf")) truth, approx = _compute_hssp_truth_and_approx(test_case, subset_size) assert np.isinf(truth) assert np.isinf(approx) test_case = rng.rand(9, dim) test_case[-1].fill(-float("inf")) truth, approx = _compute_hssp_truth_and_approx(test_case, subset_size) assert np.isinf(truth) assert np.isinf(approx) def test_solve_hssp_duplicated_infinite_loss() -> None: test_case = np.array([[np.inf, 0, 0], [np.inf, 0, 0], [0, np.inf, 0], [0, 0, np.inf]]) r = np.full(3, np.inf) res = optuna._hypervolume._solve_hssp( rank_i_loss_vals=test_case, rank_i_indices=np.arange(4), subset_size=2, reference_point=r ) assert (0 not in res) or (1 not in res) optuna-4.1.0/tests/hypervolume_tests/test_wfg.py000066400000000000000000000051731471332314300221700ustar00rootroot00000000000000import numpy as np import pytest import optuna def _shuffle_and_filter_sols( sols: np.ndarray, assume_pareto: bool, rng: np.random.RandomState ) -> np.ndarray: rng.shuffle(sols) if assume_pareto: on_front = optuna.study._multi_objective._is_pareto_front(sols, False) sols = sols[on_front] return sols @pytest.mark.parametrize("assume_pareto", (True, False)) @pytest.mark.parametrize("n_sols", list(range(2, 30))) def test_wfg_2d(assume_pareto: bool, n_sols: int) -> None: n = n_sols rng = np.random.RandomState(42) r = n * np.ones(2) s = np.empty((2 * n + 1, 2), dtype=int) s[:n] = np.stack([np.arange(n), np.arange(n)[::-1]], axis=-1) s[n:] = np.stack([np.arange(n + 1), np.arange(n + 1)[::-1]], axis=-1) s = _shuffle_and_filter_sols(s, assume_pareto, rng) assert optuna._hypervolume.compute_hypervolume(s, r, assume_pareto) == n * n - n * (n - 1) // 2 @pytest.mark.parametrize("n_objs", list(range(2, 10))) @pytest.mark.parametrize("assume_pareto", (True, False)) def test_wfg_nd(n_objs: int, assume_pareto: bool) -> None: rng = np.random.RandomState(42) r = 10 * np.ones(n_objs) s = np.vstack([np.identity(n_objs), rng.randint(1, 10, size=(10, n_objs))]) s = _shuffle_and_filter_sols(s, assume_pareto, rng) assert optuna._hypervolume.compute_hypervolume(s, r, assume_pareto) == 10**n_objs - 1 @pytest.mark.parametrize("n_objs", list(range(2, 10))) def test_wfg_with_inf(n_objs: int) -> None: s = np.ones((1, n_objs), dtype=float) s[0, n_objs // 2] = np.inf r = 1.1 * s assert optuna._hypervolume.compute_hypervolume(s, r) == np.inf @pytest.mark.parametrize("n_objs", list(range(2, 10))) def test_wfg_with_nan(n_objs: int) -> None: s = np.ones((1, n_objs), dtype=float) s[0, n_objs // 2] = np.inf r = 1.1 * s r[-1] = np.nan with pytest.raises(ValueError): optuna._hypervolume.compute_hypervolume(s, r) @pytest.mark.parametrize("assume_pareto", (True, False)) def test_wfg_duplicate_points(assume_pareto: bool) -> None: rng = np.random.RandomState(42) n = 3 r = 10 * np.ones(n) s = np.vstack([np.identity(n), rng.randint(1, 10, size=(10, n))]) ground_truth = optuna._hypervolume.compute_hypervolume(s, r, assume_pareto=False) s = np.vstack([s, s[-1]]) # Add an already existing point. s = _shuffle_and_filter_sols(s, assume_pareto, rng) assert optuna._hypervolume.compute_hypervolume(s, r, assume_pareto) == ground_truth def test_invalid_input() -> None: r = np.ones(3) s = np.atleast_2d(2 * np.ones(3)) with pytest.raises(ValueError): _ = optuna._hypervolume.compute_hypervolume(s, r) optuna-4.1.0/tests/importance_tests/000077500000000000000000000000001471332314300175505ustar00rootroot00000000000000optuna-4.1.0/tests/importance_tests/__init__.py000066400000000000000000000000001471332314300216470ustar00rootroot00000000000000optuna-4.1.0/tests/importance_tests/fanova_tests/000077500000000000000000000000001471332314300222445ustar00rootroot00000000000000optuna-4.1.0/tests/importance_tests/fanova_tests/__init__.py000066400000000000000000000000001471332314300243430ustar00rootroot00000000000000optuna-4.1.0/tests/importance_tests/fanova_tests/test_tree.py000066400000000000000000000255371471332314300246300ustar00rootroot00000000000000from __future__ import annotations import math from unittest.mock import Mock import numpy as np import pytest from optuna.importance._fanova._tree import _FanovaTree @pytest.fixture def tree() -> _FanovaTree: sklearn_tree = Mock() sklearn_tree.n_features = 3 sklearn_tree.node_count = 5 sklearn_tree.feature = [1, 2, -1, -1, -1] sklearn_tree.children_left = [1, 2, -1, -1, -1] sklearn_tree.children_right = [4, 3, -1, -1, -1] # value has the shape (node_count, n_output, max_n_classes) sklearn_tree.value = np.array([[[-1.0]], [[-1.0]], [[0.1]], [[0.2]], [[0.5]]]) sklearn_tree.threshold = [0.5, 1.5, -1.0, -1.0, -1.0] search_spaces = np.array([[0.0, 1.0], [0.0, 1.0], [0.0, 2.0]]) return _FanovaTree(tree=sklearn_tree, search_spaces=search_spaces) @pytest.fixture def expected_tree_statistics() -> list[dict[str, list]]: # Statistics the each node in the tree. return [ {"values": [0.1, 0.2, 0.5], "weights": [0.75, 0.25, 1.0]}, {"values": [0.1, 0.2], "weights": [0.75, 0.25]}, {"values": [0.1], "weights": [0.75]}, {"values": [0.2], "weights": [0.25]}, {"values": [0.5], "weights": [1.0]}, ] def test_tree_variance(tree: _FanovaTree, expected_tree_statistics: list[dict[str, list]]) -> None: # The root node at node index `0` holds the values and weights for all nodes in the tree. expected_statistics = expected_tree_statistics[0] expected_values = expected_statistics["values"] expected_weights = expected_statistics["weights"] expected_average_value = np.average(expected_values, weights=expected_weights) expected_variance = np.average( (expected_values - expected_average_value) ** 2, weights=expected_weights ) assert math.isclose(tree.variance, expected_variance) Size = float NodeIndex = int Cardinality = float @pytest.mark.parametrize( "features,expected", [ ([0], [([1.0], [(0, 1.0)])]), ([1], [([0.5], [(1, 0.5)]), ([0.5], [(4, 0.5)])]), ([2], [([1.5], [(2, 1.5), (4, 2.0)]), ([0.5], [(3, 0.5), (4, 2.0)])]), ([0, 1], [([1.0, 0.5], [(1, 0.5)]), ([1.0, 0.5], [(4, 0.5)])]), ([0, 2], [([1.0, 1.5], [(2, 1.5), (4, 2.0)]), ([1.0, 0.5], [(3, 0.5), (4, 2.0)])]), ( [1, 2], [ ([0.5, 1.5], [(2, 0.5 * 1.5)]), ([0.5, 1.5], [(4, 0.5 * 2.0)]), ([0.5, 0.5], [(3, 0.5 * 0.5)]), ([0.5, 0.5], [(4, 0.5 * 2.0)]), ], ), ( [0, 1, 2], [ ([1.0, 0.5, 1.5], [(2, 1.0 * 0.5 * 1.5)]), ([1.0, 0.5, 1.5], [(4, 1.0 * 0.5 * 2.0)]), ([1.0, 0.5, 0.5], [(3, 1.0 * 0.5 * 0.5)]), ([1.0, 0.5, 0.5], [(4, 1.0 * 0.5 * 2.0)]), ], ), ], ) def test_tree_get_marginal_variance( tree: _FanovaTree, features: list[int], expected: list[tuple[list[Size], list[tuple[NodeIndex, Cardinality]]]], expected_tree_statistics: list[dict[str, list]], ) -> None: variance = tree.get_marginal_variance(np.array(features)) expected_values = [] expected_weights = [] for sizes, node_indices_and_corrections in expected: expected_split_values = [] expected_split_weights = [] for node_index, cardinality in node_indices_and_corrections: expected_statistics = expected_tree_statistics[node_index] expected_split_values.append(expected_statistics["values"]) expected_split_weights.append( [w / cardinality for w in expected_statistics["weights"]] ) expected_value = np.average(expected_split_values, weights=expected_split_weights) expected_weight = np.prod(np.array(sizes) * np.sum(expected_split_weights)) expected_values.append(expected_value) expected_weights.append(expected_weight) expected_average_value = np.average(expected_values, weights=expected_weights) expected_variance = np.average( (expected_values - expected_average_value) ** 2, weights=expected_weights ) assert math.isclose(variance, expected_variance) @pytest.mark.parametrize( "feature_vector,expected", [ ([0.5, float("nan"), float("nan")], [(0, 1.0)]), ([float("nan"), 0.25, float("nan")], [(1, 0.5)]), ([float("nan"), 0.75, float("nan")], [(4, 0.5)]), ([float("nan"), float("nan"), 0.75], [(2, 1.5), (4, 2.0)]), ([float("nan"), float("nan"), 1.75], [(3, 0.5), (4, 2.0)]), ([0.5, 0.25, float("nan")], [(1, 1.0 * 0.5)]), ([0.5, 0.75, float("nan")], [(4, 1.0 * 0.5)]), ([0.5, float("nan"), 0.75], [(2, 1.0 * 1.5), (4, 1.0 * 2.0)]), ([0.5, float("nan"), 1.75], [(3, 1.0 * 0.5), (4, 1.0 * 2.0)]), ([float("nan"), 0.25, 0.75], [(2, 0.5 * 1.5)]), ([float("nan"), 0.25, 1.75], [(3, 0.5 * 0.5)]), ([float("nan"), 0.75, 0.75], [(4, 0.5 * 2.0)]), ([float("nan"), 0.75, 1.75], [(4, 0.5 * 2.0)]), ([0.5, 0.25, 0.75], [(2, 1.0 * 0.5 * 1.5)]), ([0.5, 0.25, 1.75], [(3, 1.0 * 0.5 * 0.5)]), ([0.5, 0.75, 0.75], [(4, 1.0 * 0.5 * 2.0)]), ([0.5, 0.75, 1.75], [(4, 1.0 * 0.5 * 2.0)]), ], ) def test_tree_get_marginalized_statistics( tree: _FanovaTree, feature_vector: list[float], expected: list[tuple[NodeIndex, Cardinality]], expected_tree_statistics: list[dict[str, list]], ) -> None: value, weight = tree._get_marginalized_statistics(np.array(feature_vector)) expected_values = [] expected_weights = [] for node_index, cardinality in expected: expected_statistics = expected_tree_statistics[node_index] expected_values.append(expected_statistics["values"]) expected_weights.append([w / cardinality for w in expected_statistics["weights"]]) expected_value = np.average(expected_values, weights=expected_weights) expected_weight = np.sum(expected_weights) assert math.isclose(value, expected_value) assert math.isclose(weight, expected_weight) def test_tree_statistics( tree: _FanovaTree, expected_tree_statistics: list[dict[str, list]] ) -> None: statistics = tree._statistics for statistic, expected_statistic in zip(statistics, expected_tree_statistics): value, weight = statistic expected_values = expected_statistic["values"] expected_weights = expected_statistic["weights"] expected_value = np.average(expected_values, weights=expected_weights) assert math.isclose(value, expected_value) assert math.isclose(weight, sum(expected_weights)) @pytest.mark.parametrize("node_index,expected", [(0, [0.5]), (1, [0.25, 0.75]), (2, [0.75, 1.75])]) def test_tree_split_midpoints( tree: _FanovaTree, node_index: NodeIndex, expected: list[float] ) -> None: np.testing.assert_equal(tree._split_midpoints[node_index], expected) @pytest.mark.parametrize("node_index,expected", [(0, [1.0]), (1, [0.5, 0.5]), (2, [1.5, 0.5])]) def test_tree_split_sizes(tree: _FanovaTree, node_index: NodeIndex, expected: list[float]) -> None: np.testing.assert_equal(tree._split_sizes[node_index], expected) @pytest.mark.parametrize( "node_index,expected", [ (0, [False, True, True]), (1, [False, False, True]), (2, [False, False, False]), (3, [False, False, False]), (4, [False, False, False]), ], ) def test_tree_subtree_active_features( tree: _FanovaTree, node_index: NodeIndex, expected: list[bool] ) -> None: active_features: np.ndarray = tree._subtree_active_features[node_index] == expected assert active_features.all() def test_tree_attrs(tree: _FanovaTree) -> None: assert tree._n_features == 3 assert tree._n_nodes == 5 assert not tree._is_node_leaf(0) assert not tree._is_node_leaf(1) assert tree._is_node_leaf(2) assert tree._is_node_leaf(3) assert tree._is_node_leaf(4) assert tree._get_node_left_child(0) == 1 assert tree._get_node_left_child(1) == 2 assert tree._get_node_left_child(2) == -1 assert tree._get_node_left_child(3) == -1 assert tree._get_node_left_child(4) == -1 assert tree._get_node_right_child(0) == 4 assert tree._get_node_right_child(1) == 3 assert tree._get_node_right_child(2) == -1 assert tree._get_node_right_child(3) == -1 assert tree._get_node_right_child(4) == -1 assert tree._get_node_children(0) == (1, 4) assert tree._get_node_children(1) == (2, 3) assert tree._get_node_children(2) == (-1, -1) assert tree._get_node_children(3) == (-1, -1) assert tree._get_node_children(4) == (-1, -1) assert tree._get_node_value(0) == -1.0 assert tree._get_node_value(1) == -1.0 assert tree._get_node_value(2) == 0.1 assert tree._get_node_value(3) == 0.2 assert tree._get_node_value(4) == 0.5 assert tree._get_node_split_threshold(0) == 0.5 assert tree._get_node_split_threshold(1) == 1.5 assert tree._get_node_split_threshold(2) == -1.0 assert tree._get_node_split_threshold(3) == -1.0 assert tree._get_node_split_threshold(4) == -1.0 assert tree._get_node_split_feature(0) == 1 assert tree._get_node_split_feature(1) == 2 def test_tree_get_node_subspaces(tree: _FanovaTree) -> None: search_spaces = np.array([[0.0, 1.0], [0.0, 1.0], [0.0, 2.0]]) search_spaces_copy = search_spaces.copy() # Test splitting on second feature, first node. expected_left_child_subspace = np.array([[0.0, 1.0], [0.0, 0.5], [0.0, 2.0]]) expected_right_child_subspace = np.array([[0.0, 1.0], [0.5, 1.0], [0.0, 2.0]]) np.testing.assert_array_equal( tree._get_node_left_child_subspaces(0, search_spaces), expected_left_child_subspace ) np.testing.assert_array_equal( tree._get_node_right_child_subspaces(0, search_spaces), expected_right_child_subspace ) np.testing.assert_array_equal( tree._get_node_children_subspaces(0, search_spaces)[0], expected_left_child_subspace ) np.testing.assert_array_equal( tree._get_node_children_subspaces(0, search_spaces)[1], expected_right_child_subspace ) np.testing.assert_array_equal(search_spaces, search_spaces_copy) # Test splitting on third feature, second node. expected_left_child_subspace = np.array([[0.0, 1.0], [0.0, 1.0], [0.0, 1.5]]) expected_right_child_subspace = np.array([[0.0, 1.0], [0.0, 1.0], [1.5, 2.0]]) np.testing.assert_array_equal( tree._get_node_left_child_subspaces(1, search_spaces), expected_left_child_subspace ) np.testing.assert_array_equal( tree._get_node_right_child_subspaces(1, search_spaces), expected_right_child_subspace ) np.testing.assert_array_equal( tree._get_node_children_subspaces(1, search_spaces)[0], expected_left_child_subspace ) np.testing.assert_array_equal( tree._get_node_children_subspaces(1, search_spaces)[1], expected_right_child_subspace ) np.testing.assert_array_equal(search_spaces, search_spaces_copy) optuna-4.1.0/tests/importance_tests/pedanova_tests/000077500000000000000000000000001471332314300225675ustar00rootroot00000000000000optuna-4.1.0/tests/importance_tests/pedanova_tests/__init__.py000066400000000000000000000000001471332314300246660ustar00rootroot00000000000000optuna-4.1.0/tests/importance_tests/pedanova_tests/test_evaluator.py000066400000000000000000000106001471332314300261770ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Callable from copy import deepcopy import numpy as np import pytest import optuna from optuna.importance import PedAnovaImportanceEvaluator from optuna.importance._ped_anova.evaluator import _QuantileFilter from optuna.trial import FrozenTrial from tests.importance_tests.test_importance_evaluators import get_study _VALUES = list([[float(i)] for i in range(10)])[::-1] _MULTI_VALUES = [[float(i), float(j)] for i, j in zip(range(10), reversed(range(10)))] @pytest.mark.parametrize( "quantile,is_lower_better,values,target,filtered_indices", [ (0.1, True, [[1.0], [2.0]], None, [0, 1]), # Check min_n_trials = 2 (0.49, True, deepcopy(_VALUES), None, list(range(10))[-5:]), (0.5, True, deepcopy(_VALUES), None, list(range(10))[-5:]), (0.51, True, deepcopy(_VALUES), None, list(range(10))[-6:]), (1.0, True, [[1.0], [2.0]], None, [0, 1]), (0.49, False, deepcopy(_VALUES), None, list(range(10))[:5]), (0.5, False, deepcopy(_VALUES), None, list(range(10))[:5]), (0.51, False, deepcopy(_VALUES), None, list(range(10))[:6]), # No tests for target!=None and is_lower_better=False because it is not used. (0.49, True, deepcopy(_MULTI_VALUES), lambda t: t.values[0], list(range(10))[:5]), (0.5, True, deepcopy(_MULTI_VALUES), lambda t: t.values[0], list(range(10))[:5]), (0.51, True, deepcopy(_MULTI_VALUES), lambda t: t.values[0], list(range(10))[:6]), (0.49, True, deepcopy(_MULTI_VALUES), lambda t: t.values[1], list(range(10))[-5:]), (0.5, True, deepcopy(_MULTI_VALUES), lambda t: t.values[1], list(range(10))[-5:]), (0.51, True, deepcopy(_MULTI_VALUES), lambda t: t.values[1], list(range(10))[-6:]), ], ) def test_filter( quantile: float, is_lower_better: bool, values: list[list[float]], target: Callable[[FrozenTrial], float] | None, filtered_indices: list[int], ) -> None: _filter = _QuantileFilter(quantile, is_lower_better, min_n_top_trials=2, target=target) trials = [optuna.create_trial(values=vs) for vs in values] for i, t in enumerate(trials): t.set_user_attr("index", i) indices = [t.user_attrs["index"] for t in _filter.filter(trials)] assert len(indices) == len(filtered_indices) assert all(i == j for i, j in zip(indices, filtered_indices)) def test_error_in_ped_anova() -> None: with pytest.raises(RuntimeError): evaluator = PedAnovaImportanceEvaluator() study = get_study(seed=0, n_trials=5, is_multi_obj=True) evaluator.evaluate(study) def test_n_trials_equal_to_min_n_top_trials() -> None: evaluator = PedAnovaImportanceEvaluator() study = get_study(seed=0, n_trials=evaluator._min_n_top_trials, is_multi_obj=False) param_importance = list(evaluator.evaluate(study).values()) n_params = len(param_importance) assert np.allclose(param_importance, np.zeros(n_params)) def test_baseline_quantile_is_1() -> None: study = get_study(seed=0, n_trials=100, is_multi_obj=False) # baseline_quantile=1.0 enforces top_trials == all_trials identical. evaluator = PedAnovaImportanceEvaluator(baseline_quantile=1.0) param_importance = list(evaluator.evaluate(study).values()) n_params = len(param_importance) # When top_trials == all_trials, all the importances become identical. assert np.allclose(param_importance, np.zeros(n_params)) def test_direction() -> None: study_minimize = get_study(seed=0, n_trials=20, is_multi_obj=False) study_maximize = optuna.create_study(direction="maximize") study_maximize.add_trials(study_minimize.trials) evaluator = PedAnovaImportanceEvaluator() assert evaluator.evaluate(study_minimize) != evaluator.evaluate(study_maximize) def test_baseline_quantile() -> None: study = get_study(seed=0, n_trials=20, is_multi_obj=False) default_evaluator = PedAnovaImportanceEvaluator(baseline_quantile=0.1) evaluator = PedAnovaImportanceEvaluator(baseline_quantile=0.3) assert evaluator.evaluate(study) != default_evaluator.evaluate(study) def test_evaluate_on_local() -> None: study = get_study(seed=0, n_trials=20, is_multi_obj=False) default_evaluator = PedAnovaImportanceEvaluator(evaluate_on_local=True) global_evaluator = PedAnovaImportanceEvaluator(evaluate_on_local=False) assert global_evaluator.evaluate(study) != default_evaluator.evaluate(study) optuna-4.1.0/tests/importance_tests/pedanova_tests/test_scott_parzen_estimator.py000066400000000000000000000167571471332314300310220ustar00rootroot00000000000000from __future__ import annotations from typing import Any import numpy as np import pytest from optuna import create_trial from optuna.distributions import BaseDistribution from optuna.distributions import CategoricalDistribution from optuna.distributions import FloatDistribution from optuna.distributions import IntDistribution from optuna.importance._ped_anova.scott_parzen_estimator import _build_parzen_estimator from optuna.importance._ped_anova.scott_parzen_estimator import _count_categorical_param_in_grid from optuna.importance._ped_anova.scott_parzen_estimator import _count_numerical_param_in_grid from optuna.importance._ped_anova.scott_parzen_estimator import _ScottParzenEstimator from optuna.samplers._tpe.probability_distributions import _BatchedCategoricalDistributions from optuna.samplers._tpe.probability_distributions import _BatchedDiscreteTruncNormDistributions from optuna.samplers._tpe.probability_distributions import _MixtureOfProductDistribution from tests.samplers_tests.tpe_tests.test_parzen_estimator import assert_distribution_almost_equal DIST_TYPES = ["int", "cat"] @pytest.mark.parametrize("dist_type", DIST_TYPES) def test_init_scott_parzen_estimator(dist_type: str) -> None: counts = np.array([1, 1, 1, 1]).astype(float) is_cat = dist_type == "cat" pe = _ScottParzenEstimator( param_name="a", dist=( IntDistribution(low=0, high=counts.size - 1) if not is_cat else CategoricalDistribution(choices=["a" * i for i in range(counts.size)]) ), counts=counts, consider_prior=False, prior_weight=0.0, ) assert len(pe._mixture_distribution.distributions) == 1 assert pe.n_steps == counts.size target_pe = pe._mixture_distribution.distributions[0] if is_cat: assert isinstance(target_pe, _BatchedCategoricalDistributions) else: assert isinstance(target_pe, _BatchedDiscreteTruncNormDistributions) @pytest.mark.parametrize( "counts,mu,sigma,weights", [ # NOTE: sigma could change depending on sigma_min picked by heuristic. (np.array([0, 0, 0, 1]), np.array([3]), np.array([0.304878]), np.array([1.0])), (np.array([0, 0, 100, 0]), np.array([2]), np.array([0.304878]), np.array([1.0])), (np.array([1, 2, 3, 4]), np.arange(4), np.array([0.7043276] * 4), (np.arange(4) + 1) / 10), ( np.array([90, 0, 0, 90]), np.array([0, 3]), np.array([0.5638226] * 2), np.array([0.5] * 2), ), (np.array([1, 0, 0, 1]), np.array([0, 3]), np.array([1.9556729] * 2), np.array([0.5] * 2)), ], ) def test_build_int_scott_parzen_estimator( counts: np.ndarray, mu: np.ndarray, sigma: np.ndarray, weights: np.ndarray ) -> None: _counts = counts.astype(float) pe = _ScottParzenEstimator( param_name="a", dist=IntDistribution(low=0, high=_counts.size - 1), counts=_counts, consider_prior=False, prior_weight=0.0, ) dist = _BatchedDiscreteTruncNormDistributions( mu=mu, sigma=sigma, low=0, high=_counts.size - 1, step=1 ) expected_dist = _MixtureOfProductDistribution(weights=weights, distributions=[dist]) assert_distribution_almost_equal(pe._mixture_distribution, expected_dist) @pytest.mark.parametrize( "counts,weights", [ (np.array([0, 0, 0, 1]), np.array([1.0])), (np.array([0, 0, 100, 0]), np.array([1.0])), (np.array([1, 2, 3, 4]), (np.arange(4) + 1) / 10), (np.array([90, 0, 0, 90]), np.array([0.5] * 2)), (np.array([1, 0, 0, 1]), np.array([0.5] * 2)), ], ) def test_build_cat_scott_parzen_estimator(counts: np.ndarray, weights: np.ndarray) -> None: _counts = counts.astype(float) pe = _ScottParzenEstimator( param_name="a", dist=CategoricalDistribution(choices=["a" * i for i in range(counts.size)]), counts=_counts, consider_prior=False, prior_weight=0.0, ) dist = _BatchedCategoricalDistributions(weights=np.identity(counts.size)[counts > 0.0]) expected_dist = _MixtureOfProductDistribution(weights=weights, distributions=[dist]) assert_distribution_almost_equal(pe._mixture_distribution, expected_dist) @pytest.mark.parametrize( "dist,params,expected_outcome", [ (IntDistribution(low=-5, high=5), [-5, -5, 1, 5, 5], [2, 0, 1, 0, 2]), (IntDistribution(low=1, high=8, log=True), list(range(1, 9)), [1, 1, 3, 3]), (FloatDistribution(low=-5.0, high=5.0), np.linspace(-5, 5, 100), [13, 25, 24, 25, 13]), ( FloatDistribution(low=1, high=8, log=True), [float(i) for i in range(1, 9)], [1, 1, 1, 3, 2], ), ], ) def test_count_numerical_param_in_grid( dist: IntDistribution | FloatDistribution, params: list[int] | list[float], expected_outcome: list[int], ) -> None: trials = [create_trial(value=0.0, params={"a": p}, distributions={"a": dist}) for p in params] res = _count_numerical_param_in_grid(param_name="a", dist=dist, trials=trials, n_steps=5) assert np.all(np.asarray(expected_outcome) == res), res def test_count_categorical_param_in_grid() -> None: params = ["a", "b", "a", "d", "a", "a", "d"] dist = CategoricalDistribution(choices=["a", "b", "c", "d"]) expected_outcome = [4, 1, 0, 2] trials = [create_trial(value=0.0, params={"a": p}, distributions={"a": dist}) for p in params] res = _count_categorical_param_in_grid(param_name="a", dist=dist, trials=trials) assert np.all(np.asarray(expected_outcome) == res) @pytest.mark.parametrize( "dist,params", [ (IntDistribution(low=-5, high=5), [1, 2, 3]), (IntDistribution(low=1, high=8, log=True), [1, 2, 4, 8]), (IntDistribution(low=-5, high=5, step=2), [1, 3, 5]), (FloatDistribution(low=-5.0, high=5.0), [1.0, 2.0, 3.0]), (FloatDistribution(low=1.0, high=8.0, log=True), [1.0, 2.0, 8.0]), (FloatDistribution(low=-5.0, high=5.0, step=0.5), [1.0, 2.0, 3.0]), (CategoricalDistribution(choices=["a", "b", "c"]), ["a", "b", "b"]), ], ) def test_build_parzen_estimator( dist: BaseDistribution, params: list[int] | list[float] | list[str], ) -> None: trials = [create_trial(value=0.0, params={"a": p}, distributions={"a": dist}) for p in params] pe = _build_parzen_estimator( param_name="a", dist=dist, trials=trials, n_steps=50, consider_prior=True, prior_weight=1.0, ) if isinstance(dist, (IntDistribution, FloatDistribution)): assert isinstance( pe._mixture_distribution.distributions[0], _BatchedDiscreteTruncNormDistributions ) elif isinstance(dist, CategoricalDistribution): assert isinstance( pe._mixture_distribution.distributions[0], _BatchedCategoricalDistributions ) else: assert False, "Should not be reached." def test_assert_in_build_parzen_estimator() -> None: class UnknownDistribution(BaseDistribution): def to_internal_repr(self, param_value_in_external_repr: Any) -> float: raise NotImplementedError def single(self) -> bool: raise NotImplementedError def _contains(self, param_value_in_internal_repr: float) -> bool: raise NotImplementedError with pytest.raises(AssertionError): _build_parzen_estimator( param_name="a", dist=UnknownDistribution(), trials=[], n_steps=50, consider_prior=True, prior_weight=1.0, ) optuna-4.1.0/tests/importance_tests/test_importance_evaluators.py000066400000000000000000000121631471332314300255720ustar00rootroot00000000000000from __future__ import annotations import pytest from optuna import create_study from optuna import Study from optuna import Trial from optuna.distributions import FloatDistribution from optuna.importance import BaseImportanceEvaluator from optuna.importance import FanovaImportanceEvaluator from optuna.importance import MeanDecreaseImpurityImportanceEvaluator from optuna.importance import PedAnovaImportanceEvaluator from optuna.samplers import RandomSampler from optuna.trial import create_trial parametrize_tree_based_evaluator_class = pytest.mark.parametrize( "evaluator_cls", (FanovaImportanceEvaluator, MeanDecreaseImpurityImportanceEvaluator) ) parametrize_evaluator = pytest.mark.parametrize( "evaluator", ( FanovaImportanceEvaluator(seed=0), MeanDecreaseImpurityImportanceEvaluator(seed=0), PedAnovaImportanceEvaluator(), ), ) def objective(trial: Trial) -> float: x1 = trial.suggest_float("x1", 0.1, 3) x2 = trial.suggest_float("x2", 0.1, 3, log=True) x3 = trial.suggest_float("x3", 2, 4, log=True) return x1 + x2 * x3 def multi_objective_function(trial: Trial) -> tuple[float, float]: x1 = trial.suggest_float("x1", 0.1, 3) x2 = trial.suggest_float("x2", 0.1, 3, log=True) x3 = trial.suggest_float("x3", 2, 4, log=True) return x1, x2 * x3 def get_study(seed: int, n_trials: int, is_multi_obj: bool) -> Study: # Assumes that `seed` can be fixed to reproduce identical results. directions = ["minimize", "minimize"] if is_multi_obj else ["minimize"] study = create_study(sampler=RandomSampler(seed=seed), directions=directions) if is_multi_obj: study.optimize(multi_objective_function, n_trials=n_trials) else: study.optimize(objective, n_trials=n_trials) return study @parametrize_tree_based_evaluator_class def test_n_trees_of_tree_based_evaluator( evaluator_cls: type[FanovaImportanceEvaluator | MeanDecreaseImpurityImportanceEvaluator], ) -> None: study = get_study(seed=0, n_trials=3, is_multi_obj=False) evaluator = evaluator_cls(n_trees=10, seed=0) param_importance = evaluator.evaluate(study) evaluator = evaluator_cls(n_trees=20, seed=0) param_importance_different_n_trees = evaluator.evaluate(study) assert param_importance != param_importance_different_n_trees @parametrize_tree_based_evaluator_class def test_max_depth_of_tree_based_evaluator( evaluator_cls: type[FanovaImportanceEvaluator | MeanDecreaseImpurityImportanceEvaluator], ) -> None: study = get_study(seed=0, n_trials=3, is_multi_obj=False) evaluator = evaluator_cls(max_depth=1, seed=0) param_importance = evaluator.evaluate(study) evaluator = evaluator_cls(max_depth=2, seed=0) param_importance_different_max_depth = evaluator.evaluate(study) assert param_importance != param_importance_different_max_depth @pytest.mark.filterwarnings("ignore::UserWarning") @parametrize_evaluator @pytest.mark.parametrize("inf_value", [float("inf"), -float("inf")]) @pytest.mark.parametrize("target_idx", [0, 1, None]) def test_evaluator_with_infinite( evaluator: BaseImportanceEvaluator, inf_value: float, target_idx: int | None ) -> None: # The test ensures that trials with infinite values are ignored to calculate importance scores. is_multi_obj = target_idx is not None study = get_study(seed=13, n_trials=10, is_multi_obj=is_multi_obj) target = (lambda t: t.values[target_idx]) if is_multi_obj else None # noqa: E731 # Importance scores are calculated without a trial with an inf value. param_importance_without_inf = evaluator.evaluate(study, target=target) # A trial with an inf value is added into the study manually. study.add_trial( create_trial( values=[inf_value] if not is_multi_obj else [inf_value, inf_value], params={"x1": 1.0, "x2": 1.0, "x3": 3.0}, distributions={ "x1": FloatDistribution(low=0.1, high=3), "x2": FloatDistribution(low=0.1, high=3, log=True), "x3": FloatDistribution(low=2, high=4, log=True), }, ) ) # Importance scores are calculated with a trial with an inf value. param_importance_with_inf = evaluator.evaluate(study, target=target) # Obtained importance scores should be the same between with inf and without inf, # because the last trial whose objective value is an inf is ignored. # PED-ANOVA can handle inf, so anyways the length should be identical. assert param_importance_with_inf == param_importance_without_inf @parametrize_evaluator def test_evaluator_with_only_single_dists(evaluator: BaseImportanceEvaluator) -> None: if isinstance(evaluator, MeanDecreaseImpurityImportanceEvaluator): # MeanDecreaseImpurityImportanceEvaluator does not handle as intended. # TODO(nabenabe0928): Fix MeanDecreaseImpurityImportanceEvaluator so that it behaves # identically to the other evaluators. return study = create_study(sampler=RandomSampler(seed=0)) study.optimize(lambda trial: trial.suggest_float("a", 0.0, 0.0), n_trials=3) param_importance = evaluator.evaluate(study) assert param_importance == {} optuna-4.1.0/tests/importance_tests/test_init.py000066400000000000000000000304601471332314300221270ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Callable from typing import Any from typing import Type import numpy as np import pytest import optuna from optuna import samplers from optuna.exceptions import ExperimentalWarning from optuna.importance import BaseImportanceEvaluator from optuna.importance import FanovaImportanceEvaluator from optuna.importance import get_param_importances from optuna.importance import MeanDecreaseImpurityImportanceEvaluator from optuna.samplers import RandomSampler from optuna.study import create_study from optuna.testing.objectives import pruned_objective from optuna.testing.storages import STORAGE_MODES from optuna.testing.storages import StorageSupplier from optuna.trial import Trial evaluators: list[Type[BaseImportanceEvaluator]] = [ MeanDecreaseImpurityImportanceEvaluator, FanovaImportanceEvaluator, ] parametrize_evaluator = pytest.mark.parametrize("evaluator_init_func", evaluators) @parametrize_evaluator @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_get_param_importance_target_is_none_and_study_is_multi_obj( storage_mode: str, evaluator_init_func: Callable[[], BaseImportanceEvaluator], ) -> None: def objective(trial: Trial) -> tuple[float, float]: x1 = trial.suggest_float("x1", 0.1, 3) x2 = trial.suggest_float("x2", 0.1, 3, log=True) x3 = trial.suggest_float("x3", 0, 3, step=1) x4 = trial.suggest_int("x4", -3, 3) x5 = trial.suggest_int("x5", 1, 5, log=True) x6 = trial.suggest_categorical("x6", [1.0, 1.1, 1.2]) if trial.number % 2 == 0: # Conditional parameters are ignored unless `params` is specified and is not `None`. x7 = trial.suggest_float("x7", 0.1, 3) value = x1**4 + x2 + x3 - x4**2 - x5 + x6 if trial.number % 2 == 0: value += x7 return value, 0.0 with StorageSupplier(storage_mode) as storage: study = create_study(directions=["minimize", "minimize"], storage=storage) study.optimize(objective, n_trials=3) with pytest.raises(ValueError): get_param_importances(study, evaluator=evaluator_init_func()) @parametrize_evaluator @pytest.mark.parametrize("storage_mode", STORAGE_MODES) @pytest.mark.parametrize("normalize", [True, False]) def test_get_param_importances( storage_mode: str, evaluator_init_func: Callable[[], BaseImportanceEvaluator], normalize: bool ) -> None: def objective(trial: Trial) -> float: x1 = trial.suggest_float("x1", 0.1, 3) x2 = trial.suggest_float("x2", 0.1, 3, log=True) x3 = trial.suggest_float("x3", 0, 3, step=1) x4 = trial.suggest_int("x4", -3, 3) x5 = trial.suggest_int("x5", 1, 5, log=True) x6 = trial.suggest_categorical("x6", [1.0, 1.1, 1.2]) if trial.number % 2 == 0: # Conditional parameters are ignored unless `params` is specified and is not `None`. x7 = trial.suggest_float("x7", 0.1, 3) value = x1**4 + x2 + x3 - x4**2 - x5 + x6 if trial.number % 2 == 0: value += x7 return value with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage, sampler=samplers.RandomSampler()) study.optimize(objective, n_trials=3) param_importance = get_param_importances( study, evaluator=evaluator_init_func(), normalize=normalize ) assert isinstance(param_importance, dict) assert len(param_importance) == 6 assert all( param_name in param_importance for param_name in ["x1", "x2", "x3", "x4", "x5", "x6"] ) prev_importance = float("inf") for param_name, importance in param_importance.items(): assert isinstance(param_name, str) assert isinstance(importance, float) assert importance <= prev_importance prev_importance = importance # Sanity check for param importances assert all(0 <= x < float("inf") for x in param_importance.values()) if normalize: assert np.isclose(sum(param_importance.values()), 1.0) @parametrize_evaluator @pytest.mark.parametrize("storage_mode", STORAGE_MODES) @pytest.mark.parametrize("params", [[], ["x1"], ["x1", "x3"], ["x1", "x4"]]) @pytest.mark.parametrize("normalize", [True, False]) def test_get_param_importances_with_params( storage_mode: str, params: list[str], evaluator_init_func: Callable[[], BaseImportanceEvaluator], normalize: bool, ) -> None: def objective(trial: Trial) -> float: x1 = trial.suggest_float("x1", 0.1, 3) x2 = trial.suggest_float("x2", 0.1, 3, log=True) x3 = trial.suggest_float("x3", 0, 3, step=1) if trial.number % 2 == 0: x4 = trial.suggest_float("x4", 0.1, 3) value = x1**4 + x2 + x3 if trial.number % 2 == 0: value += x4 return value with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) study.optimize(objective, n_trials=10) param_importance = get_param_importances( study, evaluator=evaluator_init_func(), params=params, normalize=normalize ) assert isinstance(param_importance, dict) assert len(param_importance) == len(params) assert all(param in param_importance for param in params) for param_name, importance in param_importance.items(): assert isinstance(param_name, str) assert isinstance(importance, float) # Sanity check for param importances assert all(0 <= x < float("inf") for x in param_importance.values()) if normalize: assert len(param_importance) == 0 or np.isclose(sum(param_importance.values()), 1.0) def test_get_param_importances_unnormalized_experimental() -> None: def objective(trial: Trial) -> float: x1 = trial.suggest_float("x1", 0.1, 3) return x1**2 study = create_study() study.optimize(objective, n_trials=4) with pytest.warns(ExperimentalWarning): get_param_importances(study, normalize=False) @parametrize_evaluator @pytest.mark.parametrize("storage_mode", STORAGE_MODES) @pytest.mark.parametrize("normalize", [True, False]) def test_get_param_importances_with_target( storage_mode: str, evaluator_init_func: Callable[[], BaseImportanceEvaluator], normalize: bool ) -> None: def objective(trial: Trial) -> float: x1 = trial.suggest_float("x1", 0.1, 3) x2 = trial.suggest_float("x2", 0.1, 3, log=True) x3 = trial.suggest_float("x3", 0, 3, step=1) if trial.number % 2 == 0: x4 = trial.suggest_float("x4", 0.1, 3) value = x1**4 + x2 + x3 if trial.number % 2 == 0: value += x4 return value with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) study.optimize(objective, n_trials=3) param_importance = get_param_importances( study, evaluator=evaluator_init_func(), target=lambda t: t.params["x1"] + t.params["x2"], normalize=normalize, ) assert isinstance(param_importance, dict) assert len(param_importance) == 3 assert all(param_name in param_importance for param_name in ["x1", "x2", "x3"]) prev_importance = float("inf") for param_name, importance in param_importance.items(): assert isinstance(param_name, str) assert isinstance(importance, float) assert importance <= prev_importance prev_importance = importance # Sanity check for param importances assert all(0 <= x < float("inf") for x in param_importance.values()) if normalize: assert np.isclose(sum(param_importance.values()), 1.0) @parametrize_evaluator def test_get_param_importances_invalid_empty_study( evaluator_init_func: Callable[[], BaseImportanceEvaluator] ) -> None: study = create_study() with pytest.raises(ValueError): get_param_importances(study, evaluator=evaluator_init_func()) study.optimize(pruned_objective, n_trials=3) with pytest.raises(ValueError): get_param_importances(study, evaluator=evaluator_init_func()) @parametrize_evaluator def test_get_param_importances_invalid_single_trial( evaluator_init_func: Callable[[], BaseImportanceEvaluator] ) -> None: def objective(trial: Trial) -> float: x1 = trial.suggest_float("x1", 0.1, 3) return x1**2 study = create_study() study.optimize(objective, n_trials=1) with pytest.raises(ValueError): get_param_importances(study, evaluator=evaluator_init_func()) @parametrize_evaluator def test_get_param_importances_invalid_no_completed_trials_params( evaluator_init_func: Callable[[], BaseImportanceEvaluator] ) -> None: def objective(trial: Trial) -> float: x1 = trial.suggest_float("x1", 0.1, 3) if trial.number % 2 == 0: _ = trial.suggest_float("x2", 0.1, 3, log=True) raise optuna.TrialPruned return x1**2 study = create_study() study.optimize(objective, n_trials=3) # None of the trials with `x2` are completed. with pytest.raises(ValueError): get_param_importances(study, evaluator=evaluator_init_func(), params=["x2"]) # None of the trials with `x2` are completed. Adding "x1" should not matter. with pytest.raises(ValueError): get_param_importances(study, evaluator=evaluator_init_func(), params=["x1", "x2"]) # None of the trials contain `x3`. with pytest.raises(ValueError): get_param_importances(study, evaluator=evaluator_init_func(), params=["x3"]) @parametrize_evaluator def test_get_param_importances_invalid_dynamic_search_space_params( evaluator_init_func: Callable[[], BaseImportanceEvaluator] ) -> None: def objective(trial: Trial) -> float: x1 = trial.suggest_float("x1", 0.1, trial.number + 0.1) return x1**2 study = create_study() study.optimize(objective, n_trials=3) with pytest.raises(ValueError): get_param_importances(study, evaluator=evaluator_init_func(), params=["x1"]) @parametrize_evaluator def test_get_param_importances_empty_search_space( evaluator_init_func: Callable[[], BaseImportanceEvaluator] ) -> None: def objective(trial: Trial) -> float: x = trial.suggest_float("x", 0, 5) y = trial.suggest_float("y", 1, 1) return 4 * x**2 + 4 * y**2 study = create_study() study.optimize(objective, n_trials=3) param_importance = get_param_importances(study, evaluator=evaluator_init_func()) assert len(param_importance) == 2 assert all([param in param_importance for param in ["x", "y"]]) assert param_importance["x"] > 0.0 assert param_importance["y"] == 0.0 @parametrize_evaluator def test_importance_evaluator_seed(evaluator_init_func: Any) -> None: def objective(trial: Trial) -> float: x1 = trial.suggest_float("x1", 0.1, 3) x2 = trial.suggest_float("x2", 0.1, 3, log=True) x3 = trial.suggest_float("x3", 2, 4, log=True) return x1 + x2 * x3 study = create_study(sampler=RandomSampler(seed=0)) study.optimize(objective, n_trials=3) evaluator = evaluator_init_func(seed=2) param_importance = evaluator.evaluate(study) evaluator = evaluator_init_func(seed=2) param_importance_same_seed = evaluator.evaluate(study) assert param_importance == param_importance_same_seed evaluator = evaluator_init_func(seed=3) param_importance_different_seed = evaluator.evaluate(study) assert param_importance != param_importance_different_seed @parametrize_evaluator def test_importance_evaluator_with_target(evaluator_init_func: Any) -> None: def objective(trial: Trial) -> float: x1 = trial.suggest_float("x1", 0.1, 3) x2 = trial.suggest_float("x2", 0.1, 3, log=True) x3 = trial.suggest_float("x3", 2, 4, log=True) return x1 + x2 * x3 # Assumes that `seed` can be fixed to reproduce identical results. study = create_study(sampler=RandomSampler(seed=0)) study.optimize(objective, n_trials=3) evaluator = evaluator_init_func(seed=0) param_importance = evaluator.evaluate(study) param_importance_with_target = evaluator.evaluate( study, target=lambda t: t.params["x1"] + t.params["x2"], ) assert param_importance != param_importance_with_target optuna-4.1.0/tests/pruners_tests/000077500000000000000000000000001471332314300171055ustar00rootroot00000000000000optuna-4.1.0/tests/pruners_tests/__init__.py000066400000000000000000000000001471332314300212040ustar00rootroot00000000000000optuna-4.1.0/tests/pruners_tests/test_hyperband.py000066400000000000000000000176571471332314300225120ustar00rootroot00000000000000from typing import Callable from unittest import mock import numpy as np import pytest import optuna MIN_RESOURCE = 1 MAX_RESOURCE = 16 REDUCTION_FACTOR = 2 N_BRACKETS = 4 EARLY_STOPPING_RATE_LOW = 0 EARLY_STOPPING_RATE_HIGH = 3 N_REPORTS = 10 EXPECTED_N_TRIALS_PER_BRACKET = 10 def test_hyperband_pruner_intermediate_values() -> None: pruner = optuna.pruners.HyperbandPruner( min_resource=MIN_RESOURCE, max_resource=MAX_RESOURCE, reduction_factor=REDUCTION_FACTOR ) study = optuna.study.create_study(sampler=optuna.samplers.RandomSampler(), pruner=pruner) def objective(trial: optuna.trial.Trial) -> float: for i in range(N_REPORTS): trial.report(i, step=i) return 1.0 study.optimize(objective, n_trials=N_BRACKETS * EXPECTED_N_TRIALS_PER_BRACKET) trials = study.trials assert len(trials) == N_BRACKETS * EXPECTED_N_TRIALS_PER_BRACKET def test_bracket_study() -> None: pruner = optuna.pruners.HyperbandPruner( min_resource=MIN_RESOURCE, max_resource=MAX_RESOURCE, reduction_factor=REDUCTION_FACTOR ) study = optuna.study.create_study(sampler=optuna.samplers.RandomSampler(), pruner=pruner) bracket_study = pruner._create_bracket_study(study, 0) with pytest.raises(AttributeError): bracket_study.optimize(lambda *args: 1.0) with pytest.raises(AttributeError): bracket_study.set_user_attr("abc", 100) for attr in ("user_attrs", "system_attrs"): with pytest.raises(AttributeError): getattr(bracket_study, attr) with pytest.raises(AttributeError): bracket_study.trials_dataframe() bracket_study.get_trials() bracket_study.direction bracket_study._storage bracket_study._study_id bracket_study.pruner bracket_study.study_name # As `_BracketStudy` is defined inside `HyperbandPruner`, # we cannot do `assert isinstance(bracket_study, _BracketStudy)`. # This is why the below line is ignored by mypy checks. bracket_study._bracket_id # type: ignore def test_hyperband_max_resource_is_auto() -> None: pruner = optuna.pruners.HyperbandPruner( min_resource=MIN_RESOURCE, reduction_factor=REDUCTION_FACTOR ) study = optuna.study.create_study(sampler=optuna.samplers.RandomSampler(), pruner=pruner) def objective(trial: optuna.trial.Trial) -> float: for i in range(N_REPORTS): trial.report(1.0, i) if trial.should_prune(): raise optuna.TrialPruned() return 1.0 study.optimize(objective, n_trials=N_BRACKETS * EXPECTED_N_TRIALS_PER_BRACKET) assert N_REPORTS == pruner._max_resource def test_hyperband_max_resource_value_error() -> None: with pytest.raises(ValueError): _ = optuna.pruners.HyperbandPruner(max_resource="not_appropriate") @pytest.mark.parametrize( "sampler_init_func", [ lambda: optuna.samplers.RandomSampler(), (lambda: optuna.samplers.TPESampler(n_startup_trials=1)), ( lambda: optuna.samplers.GridSampler( search_space={"value": np.linspace(0.0, 1.0, 8, endpoint=False).tolist()} ) ), (lambda: optuna.samplers.CmaEsSampler(n_startup_trials=1)), ], ) def test_hyperband_filter_study( sampler_init_func: Callable[[], optuna.samplers.BaseSampler] ) -> None: def objective(trial: optuna.trial.Trial) -> float: return trial.suggest_float("value", 0.0, 1.0) n_trials = 8 n_brackets = 4 expected_n_trials_per_bracket = n_trials // n_brackets with mock.patch( "optuna.pruners.HyperbandPruner._get_bracket_id", new=mock.Mock(side_effect=lambda study, trial: trial.number % n_brackets), ): for method_name in [ "infer_relative_search_space", "sample_relative", "sample_independent", ]: sampler = sampler_init_func() pruner = optuna.pruners.HyperbandPruner( min_resource=MIN_RESOURCE, max_resource=MAX_RESOURCE, reduction_factor=REDUCTION_FACTOR, ) with mock.patch( "optuna.samplers.{}.{}".format(sampler.__class__.__name__, method_name), wraps=getattr(sampler, method_name), ) as method_mock: study = optuna.study.create_study(sampler=sampler, pruner=pruner) study.optimize(objective, n_trials=n_trials) args = method_mock.call_args[0] study = args[0] trials = study.get_trials() assert len(trials) == expected_n_trials_per_bracket @pytest.mark.parametrize( "pruner_init_func", [ lambda: optuna.pruners.NopPruner(), lambda: optuna.pruners.MedianPruner(), lambda: optuna.pruners.ThresholdPruner(lower=0.5), lambda: optuna.pruners.SuccessiveHalvingPruner(), ], ) def test_hyperband_no_filter_study( pruner_init_func: Callable[[], optuna.pruners.BasePruner] ) -> None: def objective(trial: optuna.trial.Trial) -> float: return trial.suggest_float("value", 0.0, 1.0) n_trials = 10 for method_name in [ "infer_relative_search_space", "sample_relative", "sample_independent", ]: sampler = optuna.samplers.RandomSampler() pruner = pruner_init_func() with mock.patch( "optuna.samplers.{}.{}".format(sampler.__class__.__name__, method_name), wraps=getattr(sampler, method_name), ) as method_mock: study = optuna.study.create_study(sampler=sampler, pruner=pruner) study.optimize(objective, n_trials=n_trials) args = method_mock.call_args[0] study = args[0] trials = study.get_trials() assert len(trials) == n_trials @pytest.mark.parametrize( "sampler_init_func", [ lambda: optuna.samplers.RandomSampler(), (lambda: optuna.samplers.TPESampler(n_startup_trials=1)), ( lambda: optuna.samplers.GridSampler( search_space={"value": np.linspace(0.0, 1.0, 10, endpoint=False).tolist()} ) ), (lambda: optuna.samplers.CmaEsSampler(n_startup_trials=1)), ], ) def test_hyperband_no_call_of_filter_study_in_should_prune( sampler_init_func: Callable[[], optuna.samplers.BaseSampler] ) -> None: def objective(trial: optuna.trial.Trial) -> float: with mock.patch("optuna.pruners._filter_study") as method_mock: for i in range(N_REPORTS): trial.report(i, step=i) if trial.should_prune(): method_mock.assert_not_called() raise optuna.TrialPruned() else: method_mock.assert_not_called() return 1.0 sampler = sampler_init_func() pruner = optuna.pruners.HyperbandPruner( min_resource=MIN_RESOURCE, max_resource=MAX_RESOURCE, reduction_factor=REDUCTION_FACTOR ) study = optuna.study.create_study(sampler=sampler, pruner=pruner) study.optimize(objective, n_trials=10) def test_incompatibility_between_bootstrap_count_and_auto_max_resource() -> None: with pytest.raises(ValueError): optuna.pruners.HyperbandPruner(max_resource="auto", bootstrap_count=1) def test_hyperband_pruner_and_grid_sampler() -> None: pruner = optuna.pruners.HyperbandPruner( min_resource=MIN_RESOURCE, max_resource=MAX_RESOURCE, reduction_factor=REDUCTION_FACTOR ) search_space = {"x": [-50, 0, 50], "y": [-99, 0, 99]} sampler = optuna.samplers.GridSampler(search_space) study = optuna.study.create_study(sampler=sampler, pruner=pruner) def objective(trial: optuna.trial.Trial) -> float: for i in range(N_REPORTS): trial.report(i, step=i) x = trial.suggest_float("x", -100, 100) y = trial.suggest_int("y", -100, 100) return x**2 + y**2 study.optimize(objective, n_trials=10) trials = study.trials assert len(trials) == 9 optuna-4.1.0/tests/pruners_tests/test_median.py000066400000000000000000000112221471332314300217510ustar00rootroot00000000000000from typing import List from typing import Tuple import pytest import optuna def test_median_pruner_with_one_trial() -> None: pruner = optuna.pruners.MedianPruner(0, 0) study = optuna.study.create_study(pruner=pruner) trial = study.ask() trial.report(1, 1) # A pruner is not activated at a first trial. assert not trial.should_prune() @pytest.mark.parametrize("direction_value", [("minimize", 2), ("maximize", 0.5)]) def test_median_pruner_intermediate_values(direction_value: Tuple[str, float]) -> None: direction, intermediate_value = direction_value pruner = optuna.pruners.MedianPruner(0, 0) study = optuna.study.create_study(direction=direction, pruner=pruner) trial = study.ask() trial.report(1, 1) study.tell(trial, 1) trial = study.ask() # A pruner is not activated if a trial has no intermediate values. assert not trial.should_prune() trial.report(intermediate_value, 1) # A pruner is activated if a trial has an intermediate value. assert trial.should_prune() @pytest.mark.filterwarnings("ignore::RuntimeWarning") def test_median_pruner_intermediate_values_nan() -> None: pruner = optuna.pruners.MedianPruner(0, 0) study = optuna.study.create_study(pruner=pruner) trial = study.ask() trial.report(float("nan"), 1) # A pruner is not activated if the study does not have any previous trials. assert not trial.should_prune() study.tell(trial, -1) # -1 is used because we can not tell with nan. trial = study.ask() trial.report(float("nan"), 1) # A pruner is activated if the best intermediate value of this trial is NaN. assert trial.should_prune() study.tell(trial, -1) # -1 is used because we can not tell with nan. trial = study.ask() trial.report(1, 1) # A pruner is not activated if the median intermediate value is NaN. assert not trial.should_prune() def test_median_pruner_n_startup_trials() -> None: pruner = optuna.pruners.MedianPruner(2, 0) study = optuna.study.create_study(pruner=pruner) trial = study.ask() trial.report(1, 1) study.tell(trial, 1) trial = study.ask() trial.report(2, 1) # A pruner is not activated during startup trials. assert not trial.should_prune() study.tell(trial, 2) trial = study.ask() trial.report(3, 1) # A pruner is activated after startup trials. assert trial.should_prune() def test_median_pruner_n_warmup_steps() -> None: pruner = optuna.pruners.MedianPruner(0, 1) study = optuna.study.create_study(pruner=pruner) trial = study.ask() trial.report(1, 0) trial.report(1, 1) study.tell(trial, 1) trial = study.ask() trial.report(2, 0) # A pruner is not activated during warm-up steps. assert not trial.should_prune() trial.report(2, 1) # A pruner is activated after warm-up steps. assert trial.should_prune() @pytest.mark.parametrize( "n_warmup_steps,interval_steps,report_steps,expected_prune_steps", [ (0, 1, 1, [0, 1, 2, 3, 4, 5]), (1, 1, 1, [1, 2, 3, 4, 5]), (1, 2, 1, [1, 3, 5]), (0, 3, 10, list(range(29))), (2, 3, 10, list(range(10, 29))), (0, 10, 3, [0, 1, 2, 12, 13, 14, 21, 22, 23]), (2, 10, 3, [3, 4, 5, 12, 13, 14, 24, 25, 26]), ], ) def test_median_pruner_interval_steps( n_warmup_steps: int, interval_steps: int, report_steps: int, expected_prune_steps: List[int] ) -> None: pruner = optuna.pruners.MedianPruner(0, n_warmup_steps, interval_steps) study = optuna.study.create_study(pruner=pruner) trial = study.ask() last_step = max(expected_prune_steps) + 1 for i in range(last_step): trial.report(0, i) study.tell(trial, 0) trial = study.ask() pruned = [] for i in range(last_step): if i % report_steps == 0: trial.report(2, i) if trial.should_prune(): pruned.append(i) assert pruned == expected_prune_steps def test_median_pruner_n_min_trials() -> None: pruner = optuna.pruners.MedianPruner(2, 0, 1, n_min_trials=2) study = optuna.study.create_study(pruner=pruner) trial = study.ask() trial.report(4, 1) trial.report(2, 2) study.tell(trial, 2) trial = study.ask() trial.report(3, 1) study.tell(trial, 3) trial = study.ask() trial.report(4, 1) trial.report(3, 2) # A pruner is not activated before the values at step 2 observed n_min_trials times. assert not trial.should_prune() study.tell(trial, 3) trial = study.ask() trial.report(4, 1) trial.report(3, 2) # A pruner is activated after the values at step 2 observed n_min_trials times. assert trial.should_prune() optuna-4.1.0/tests/pruners_tests/test_nop.py000066400000000000000000000004221471332314300213100ustar00rootroot00000000000000import optuna def test_nop_pruner() -> None: pruner = optuna.pruners.NopPruner() study = optuna.study.create_study(pruner=pruner) trial = study.ask() trial.report(1, 1) # A NopPruner instance is always deactivated. assert not trial.should_prune() optuna-4.1.0/tests/pruners_tests/test_patient.py000066400000000000000000000053411471332314300221650ustar00rootroot00000000000000from typing import List import pytest import optuna def test_patient_pruner_experimental_warning() -> None: with pytest.warns(optuna.exceptions.ExperimentalWarning): optuna.pruners.PatientPruner(None, 0) def test_patient_pruner_patience() -> None: optuna.pruners.PatientPruner(None, 0) optuna.pruners.PatientPruner(None, 1) with pytest.raises(ValueError): optuna.pruners.PatientPruner(None, -1) def test_patient_pruner_min_delta() -> None: optuna.pruners.PatientPruner(None, 0, 0.0) optuna.pruners.PatientPruner(None, 0, 1.0) with pytest.raises(ValueError): optuna.pruners.PatientPruner(None, 0, -1) def test_patient_pruner_with_one_trial() -> None: pruner = optuna.pruners.PatientPruner(None, 0) study = optuna.study.create_study(pruner=pruner) trial = study.ask() trial.report(1, 0) # The pruner is not activated at a first trial. assert not trial.should_prune() @pytest.mark.filterwarnings("ignore::RuntimeWarning") def test_patient_pruner_intermediate_values_nan() -> None: pruner = optuna.pruners.PatientPruner(None, 0, 0) study = optuna.study.create_study(pruner=pruner) trial = study.ask() # A pruner is not activated if a trial does not have any intermediate values. assert not trial.should_prune() trial.report(float("nan"), 0) # A pruner is not activated if a trial has only one intermediate value. assert not trial.should_prune() trial.report(1.0, 1) # A pruner is not activated if a trial has only nan in intermediate values. assert not trial.should_prune() trial.report(float("nan"), 2) # A pruner is not activated if a trial has only nan in intermediate values. assert not trial.should_prune() @pytest.mark.parametrize( "patience,min_delta,direction,intermediates,expected_prune_steps", [ (0, 0, "maximize", [1, 0], [1]), (1, 0, "maximize", [2, 1, 0], [2]), (0, 0, "minimize", [0, 1], [1]), (1, 0, "minimize", [0, 1, 2], [2]), (0, 1.0, "maximize", [1, 0], []), (1, 1.0, "maximize", [3, 2, 1, 0], [3]), (0, 1.0, "minimize", [0, 1], []), (1, 1.0, "minimize", [0, 1, 2, 3], [3]), ], ) def test_patient_pruner_intermediate_values( patience: int, min_delta: float, direction: str, intermediates: List[int], expected_prune_steps: List[int], ) -> None: pruner = optuna.pruners.PatientPruner(None, patience, min_delta) study = optuna.study.create_study(pruner=pruner, direction=direction) trial = study.ask() pruned = [] for step, value in enumerate(intermediates): trial.report(value, step) if trial.should_prune(): pruned.append(step) assert pruned == expected_prune_steps optuna-4.1.0/tests/pruners_tests/test_percentile.py000066400000000000000000000205711471332314300226550ustar00rootroot00000000000000import math from typing import List from typing import Tuple import warnings import pytest import optuna from optuna.pruners import _percentile from optuna.study import Study from optuna.study import StudyDirection from optuna.trial import TrialState def test_percentile_pruner_percentile() -> None: optuna.pruners.PercentilePruner(0.0) optuna.pruners.PercentilePruner(25.0) optuna.pruners.PercentilePruner(100.0) with pytest.raises(ValueError): optuna.pruners.PercentilePruner(-0.1) with pytest.raises(ValueError): optuna.pruners.PercentilePruner(100.1) def test_percentile_pruner_n_startup_trials() -> None: optuna.pruners.PercentilePruner(25.0, n_startup_trials=0) optuna.pruners.PercentilePruner(25.0, n_startup_trials=5) with pytest.raises(ValueError): optuna.pruners.PercentilePruner(25.0, n_startup_trials=-1) def test_percentile_pruner_n_warmup_steps() -> None: optuna.pruners.PercentilePruner(25.0, n_warmup_steps=0) optuna.pruners.PercentilePruner(25.0, n_warmup_steps=5) with pytest.raises(ValueError): optuna.pruners.PercentilePruner(25.0, n_warmup_steps=-1) def test_percentile_pruner_interval_steps() -> None: optuna.pruners.PercentilePruner(25.0, interval_steps=1) optuna.pruners.PercentilePruner(25.0, interval_steps=5) with pytest.raises(ValueError): optuna.pruners.PercentilePruner(25.0, interval_steps=-1) with pytest.raises(ValueError): optuna.pruners.PercentilePruner(25.0, interval_steps=0) def test_percentile_pruner_with_one_trial() -> None: pruner = optuna.pruners.PercentilePruner(25.0, 0, 0) study = optuna.study.create_study(pruner=pruner) trial = study.ask() trial.report(1, 1) # A pruner is not activated at a first trial. assert not trial.should_prune() @pytest.mark.parametrize( "direction_value", [("minimize", [1, 2, 3, 4, 5], 2.1), ("maximize", [1, 2, 3, 4, 5], 3.9)] ) def test_25_percentile_pruner_intermediate_values( direction_value: Tuple[str, List[float], float] ) -> None: direction, intermediate_values, latest_value = direction_value pruner = optuna.pruners.PercentilePruner(25.0, 0, 0) study = optuna.study.create_study(direction=direction, pruner=pruner) for v in intermediate_values: trial = study.ask() trial.report(v, 1) study.tell(trial, v) trial = study.ask() # A pruner is not activated if a trial has no intermediate values. assert not trial.should_prune() trial.report(latest_value, 1) # A pruner is activated if a trial has an intermediate value. assert trial.should_prune() @pytest.mark.filterwarnings("ignore::RuntimeWarning") def test_25_percentile_pruner_intermediate_values_nan() -> None: pruner = optuna.pruners.PercentilePruner(25.0, 0, 0) study = optuna.study.create_study(pruner=pruner) trial = study.ask() trial.report(float("nan"), 1) # A pruner is not activated if the study does not have any previous trials. assert not trial.should_prune() study.tell(trial, -1) trial = study.ask() trial.report(float("nan"), 1) # A pruner is activated if the best intermediate value of this trial is NaN. assert trial.should_prune() study.tell(trial, -1) trial = study.ask() trial.report(1, 1) # A pruner is not activated if the 25 percentile intermediate value is NaN. assert not trial.should_prune() @pytest.mark.parametrize( "direction_expected", [(StudyDirection.MINIMIZE, 0.1), (StudyDirection.MAXIMIZE, 0.2)] ) def test_get_best_intermediate_result_over_steps( direction_expected: Tuple[StudyDirection, float] ) -> None: direction, expected = direction_expected if direction == StudyDirection.MINIMIZE: study = optuna.study.create_study(direction="minimize") else: study = optuna.study.create_study(direction="maximize") # FrozenTrial.intermediate_values has no elements. trial_id_empty = study._storage.create_new_trial(study._study_id) trial_empty = study._storage.get_trial(trial_id_empty) with pytest.raises(ValueError): _percentile._get_best_intermediate_result_over_steps(trial_empty, direction) # Input value has no NaNs but float values. trial_id_float = study._storage.create_new_trial(study._study_id) trial_float = optuna.trial.Trial(study, trial_id_float) trial_float.report(0.1, step=0) trial_float.report(0.2, step=1) frozen_trial_float = study._storage.get_trial(trial_id_float) assert expected == _percentile._get_best_intermediate_result_over_steps( frozen_trial_float, direction ) # Input value has a float value and a NaN. trial_id_float_nan = study._storage.create_new_trial(study._study_id) trial_float_nan = optuna.trial.Trial(study, trial_id_float_nan) trial_float_nan.report(0.3, step=0) trial_float_nan.report(float("nan"), step=1) frozen_trial_float_nan = study._storage.get_trial(trial_id_float_nan) assert 0.3 == _percentile._get_best_intermediate_result_over_steps( frozen_trial_float_nan, direction ) # Input value has a NaN only. trial_id_nan = study._storage.create_new_trial(study._study_id) trial_nan = optuna.trial.Trial(study, trial_id_nan) trial_nan.report(float("nan"), step=0) frozen_trial_nan = study._storage.get_trial(trial_id_nan) with warnings.catch_warnings(): warnings.simplefilter("ignore", category=RuntimeWarning) assert math.isnan( _percentile._get_best_intermediate_result_over_steps(frozen_trial_nan, direction) ) def test_get_percentile_intermediate_result_over_trials() -> None: def setup_study(trial_num: int, _intermediate_values: List[List[float]]) -> Study: _study = optuna.study.create_study(direction="minimize") trial_ids = [_study._storage.create_new_trial(_study._study_id) for _ in range(trial_num)] for step, values in enumerate(_intermediate_values): # Study does not have any complete trials. with pytest.raises(ValueError): completed_trials = _study.get_trials(deepcopy=False, states=(TrialState.COMPLETE,)) _direction = _study.direction _percentile._get_percentile_intermediate_result_over_trials( completed_trials, _direction, step, 25, 1 ) for i in range(trial_num): trial_id = trial_ids[i] value = values[i] _study._storage.set_trial_intermediate_value(trial_id, step, value) # Set trial states complete because this method ignores incomplete trials. for trial_id in trial_ids: _study._storage.set_trial_state_values(trial_id, state=TrialState.COMPLETE) return _study # Input value has no NaNs but float values (step=0). intermediate_values = [[0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]] study = setup_study(9, intermediate_values) all_trials = study.get_trials() direction = study.direction assert 0.3 == _percentile._get_percentile_intermediate_result_over_trials( all_trials, direction, 0, 25.0, 1 ) # Input value has a float value and NaNs (step=1). intermediate_values.append( [0.1, 0.2, 0.3, 0.4, 0.5, float("nan"), float("nan"), float("nan"), float("nan")] ) study = setup_study(9, intermediate_values) all_trials = study.get_trials() direction = study.direction assert 0.2 == _percentile._get_percentile_intermediate_result_over_trials( all_trials, direction, 1, 25.0, 1 ) # Input value has NaNs only (step=2). intermediate_values.append( [ float("nan"), float("nan"), float("nan"), float("nan"), float("nan"), float("nan"), float("nan"), float("nan"), float("nan"), ] ) study = setup_study(9, intermediate_values) all_trials = study.get_trials() direction = study.direction with warnings.catch_warnings(): warnings.simplefilter("ignore", category=RuntimeWarning) assert math.isnan( _percentile._get_percentile_intermediate_result_over_trials( all_trials, direction, 2, 75, 1 ) ) # n_min_trials = 2. assert math.isnan( _percentile._get_percentile_intermediate_result_over_trials( all_trials, direction, 2, 75, 2 ) ) optuna-4.1.0/tests/pruners_tests/test_successive_halving.py000066400000000000000000000242421471332314300244060ustar00rootroot00000000000000from typing import Tuple import pytest import optuna @pytest.mark.parametrize("direction_value", [("minimize", 2), ("maximize", 0.5)]) def test_successive_halving_pruner_intermediate_values(direction_value: Tuple[str, float]) -> None: direction, intermediate_value = direction_value pruner = optuna.pruners.SuccessiveHalvingPruner( min_resource=1, reduction_factor=2, min_early_stopping_rate=0 ) study = optuna.study.create_study(direction=direction, pruner=pruner) trial = study.ask() trial.report(1, 1) # A pruner is not activated at a first trial. assert not trial.should_prune() trial = study.ask() # A pruner is not activated if a trial has no intermediate values. assert not trial.should_prune() trial.report(intermediate_value, 1) # A pruner is activated if a trial has an intermediate value. assert trial.should_prune() def test_successive_halving_pruner_rung_check() -> None: pruner = optuna.pruners.SuccessiveHalvingPruner( min_resource=1, reduction_factor=2, min_early_stopping_rate=0 ) study = optuna.study.create_study(pruner=pruner) # Report 7 trials in advance. for i in range(7): trial = study.ask() trial.report(0.1 * (i + 1), step=7) pruner.prune(study=study, trial=study._storage.get_trial(trial._trial_id)) # Report a trial that has the 7-th value from bottom. trial = study.ask() trial.report(0.75, step=7) pruner.prune(study=study, trial=study._storage.get_trial(trial._trial_id)) trial_system_attrs = trial.storage.get_trial_system_attrs(trial._trial_id) assert "completed_rung_0" in trial_system_attrs assert "completed_rung_1" not in trial_system_attrs # Report a trial that has the third value from bottom. trial = study.ask() trial.report(0.25, step=7) trial.should_prune() trial_system_attrs = trial.storage.get_trial_system_attrs(trial._trial_id) assert "completed_rung_1" in trial_system_attrs assert "completed_rung_2" not in trial_system_attrs # Report a trial that has the lowest value. trial = study.ask() trial.report(0.05, step=7) trial.should_prune() trial_system_attrs = trial.storage.get_trial_system_attrs(trial._trial_id) assert "completed_rung_2" in trial_system_attrs assert "completed_rung_3" not in trial_system_attrs def test_successive_halving_pruner_first_trial_is_not_pruned() -> None: pruner = optuna.pruners.SuccessiveHalvingPruner( min_resource=1, reduction_factor=2, min_early_stopping_rate=0 ) study = optuna.study.create_study(pruner=pruner) trial = study.ask() for i in range(10): trial.report(1, step=i) # The first trial is not pruned. assert not trial.should_prune() # The trial completed until rung 3. trial_system_attrs = trial.storage.get_trial_system_attrs(trial._trial_id) assert "completed_rung_0" in trial_system_attrs assert "completed_rung_1" in trial_system_attrs assert "completed_rung_2" in trial_system_attrs assert "completed_rung_3" in trial_system_attrs assert "completed_rung_4" not in trial_system_attrs def test_successive_halving_pruner_with_nan() -> None: pruner = optuna.pruners.SuccessiveHalvingPruner( min_resource=2, reduction_factor=2, min_early_stopping_rate=0 ) study = optuna.study.create_study(pruner=pruner) trial = optuna.trial.Trial(study, study._storage.create_new_trial(study._study_id)) # A pruner is not activated if the step is not a rung completion point. trial.report(float("nan"), step=1) assert not trial.should_prune() # A pruner is activated if the step is a rung completion point and # the intermediate value is NaN. trial.report(float("nan"), step=2) assert trial.should_prune() @pytest.mark.parametrize("n_reports", range(3)) @pytest.mark.parametrize("n_trials", [1, 2]) def test_successive_halving_pruner_with_auto_min_resource(n_reports: int, n_trials: int) -> None: pruner = optuna.pruners.SuccessiveHalvingPruner(min_resource="auto") study = optuna.study.create_study(sampler=optuna.samplers.RandomSampler(), pruner=pruner) assert pruner._min_resource is None def objective(trial: optuna.trial.Trial) -> float: for i in range(n_reports): trial.report(1.0 / (i + 1), i) if trial.should_prune(): raise optuna.TrialPruned() return 1.0 study.optimize(objective, n_trials=n_trials) if n_reports > 0 and n_trials > 1: assert pruner._min_resource is not None and pruner._min_resource > 0 else: assert pruner._min_resource is None def test_successive_halving_pruner_with_invalid_str_to_min_resource() -> None: with pytest.raises(ValueError): optuna.pruners.SuccessiveHalvingPruner(min_resource="fixed") def test_successive_halving_pruner_min_resource_parameter() -> None: # min_resource=0: Error (must be `min_resource >= 1`). with pytest.raises(ValueError): optuna.pruners.SuccessiveHalvingPruner( min_resource=0, reduction_factor=2, min_early_stopping_rate=0 ) # min_resource=1: The rung 0 ends at step 1. pruner = optuna.pruners.SuccessiveHalvingPruner( min_resource=1, reduction_factor=2, min_early_stopping_rate=0 ) study = optuna.study.create_study(pruner=pruner) trial = study.ask() trial.report(1, step=1) assert not trial.should_prune() trial_system_attrs = trial.storage.get_trial_system_attrs(trial._trial_id) assert "completed_rung_0" in trial_system_attrs assert "completed_rung_1" not in trial_system_attrs # min_resource=2: The rung 0 ends at step 2. pruner = optuna.pruners.SuccessiveHalvingPruner( min_resource=2, reduction_factor=2, min_early_stopping_rate=0 ) study = optuna.study.create_study(pruner=pruner) trial = study.ask() trial.report(1, step=1) assert not trial.should_prune() trial_system_attrs = trial.storage.get_trial_system_attrs(trial._trial_id) assert "completed_rung_0" not in trial_system_attrs trial.report(1, step=2) assert not trial.should_prune() trial_system_attrs = trial.storage.get_trial_system_attrs(trial._trial_id) assert "completed_rung_0" in trial_system_attrs assert "completed_rung_1" not in trial_system_attrs def test_successive_halving_pruner_reduction_factor_parameter() -> None: # reduction_factor=1: Error (must be `reduction_factor >= 2`). with pytest.raises(ValueError): optuna.pruners.SuccessiveHalvingPruner( min_resource=1, reduction_factor=1, min_early_stopping_rate=0 ) # reduction_factor=2: The rung 0 ends at step 1. pruner = optuna.pruners.SuccessiveHalvingPruner( min_resource=1, reduction_factor=2, min_early_stopping_rate=0 ) study = optuna.study.create_study(pruner=pruner) trial = study.ask() trial.report(1, step=1) assert not trial.should_prune() trial_system_attrs = trial.storage.get_trial_system_attrs(trial._trial_id) assert "completed_rung_0" in trial_system_attrs assert "completed_rung_1" not in trial_system_attrs # reduction_factor=3: The rung 1 ends at step 3. pruner = optuna.pruners.SuccessiveHalvingPruner( min_resource=1, reduction_factor=3, min_early_stopping_rate=0 ) study = optuna.study.create_study(pruner=pruner) trial = study.ask() trial.report(1, step=1) assert not trial.should_prune() trial_system_attrs = trial.storage.get_trial_system_attrs(trial._trial_id) assert "completed_rung_0" in trial_system_attrs assert "completed_rung_1" not in trial_system_attrs trial.report(1, step=2) assert not trial.should_prune() trial_system_attrs = trial.storage.get_trial_system_attrs(trial._trial_id) assert "completed_rung_1" not in trial_system_attrs trial.report(1, step=3) assert not trial.should_prune() trial_system_attrs = trial.storage.get_trial_system_attrs(trial._trial_id) assert "completed_rung_1" in trial_system_attrs assert "completed_rung_2" not in trial_system_attrs def test_successive_halving_pruner_min_early_stopping_rate_parameter() -> None: # min_early_stopping_rate=-1: Error (must be `min_early_stopping_rate >= 0`). with pytest.raises(ValueError): optuna.pruners.SuccessiveHalvingPruner( min_resource=1, reduction_factor=2, min_early_stopping_rate=-1 ) # min_early_stopping_rate=0: The rung 0 ends at step 1. pruner = optuna.pruners.SuccessiveHalvingPruner( min_resource=1, reduction_factor=2, min_early_stopping_rate=0 ) study = optuna.study.create_study(pruner=pruner) trial = study.ask() trial.report(1, step=1) assert not trial.should_prune() trial_system_attrs = trial.storage.get_trial_system_attrs(trial._trial_id) assert "completed_rung_0" in trial_system_attrs # min_early_stopping_rate=1: The rung 0 ends at step 2. pruner = optuna.pruners.SuccessiveHalvingPruner( min_resource=1, reduction_factor=2, min_early_stopping_rate=1 ) study = optuna.study.create_study(pruner=pruner) trial = study.ask() trial.report(1, step=1) assert not trial.should_prune() trial_system_attrs = trial.storage.get_trial_system_attrs(trial._trial_id) assert "completed_rung_0" not in trial_system_attrs assert "completed_rung_1" not in trial_system_attrs trial.report(1, step=2) assert not trial.should_prune() trial_system_attrs = trial.storage.get_trial_system_attrs(trial._trial_id) assert "completed_rung_0" in trial_system_attrs assert "completed_rung_1" not in trial_system_attrs def test_successive_halving_pruner_bootstrap_parameter() -> None: with pytest.raises(ValueError): optuna.pruners.SuccessiveHalvingPruner(bootstrap_count=-1) with pytest.raises(ValueError): optuna.pruners.SuccessiveHalvingPruner(bootstrap_count=1, min_resource="auto") pruner = optuna.pruners.SuccessiveHalvingPruner( min_resource=1, reduction_factor=2, bootstrap_count=1 ) study = optuna.study.create_study(pruner=pruner) trial1 = study.ask() trial2 = study.ask() trial1.report(1, step=1) assert trial1.should_prune() trial2.report(1, step=1) assert not trial2.should_prune() optuna-4.1.0/tests/pruners_tests/test_threshold.py000066400000000000000000000056621471332314300225230ustar00rootroot00000000000000import pytest import optuna def test_threshold_pruner_with_ub() -> None: pruner = optuna.pruners.ThresholdPruner(upper=2.0, n_warmup_steps=0, interval_steps=1) study = optuna.study.create_study(pruner=pruner) trial = study.ask() trial.report(1.0, 1) assert not trial.should_prune() trial.report(3.0, 2) assert trial.should_prune() def test_threshold_pruner_with_lt() -> None: pruner = optuna.pruners.ThresholdPruner(lower=2.0, n_warmup_steps=0, interval_steps=1) study = optuna.study.create_study(pruner=pruner) trial = study.ask() trial.report(3.0, 1) assert not trial.should_prune() trial.report(1.0, 2) assert trial.should_prune() def test_threshold_pruner_with_two_side() -> None: pruner = optuna.pruners.ThresholdPruner( lower=0.0, upper=1.0, n_warmup_steps=0, interval_steps=1 ) study = optuna.study.create_study(pruner=pruner) trial = study.ask() trial.report(-0.1, 1) assert trial.should_prune() trial.report(0.0, 2) assert not trial.should_prune() trial.report(0.4, 3) assert not trial.should_prune() trial.report(1.0, 4) assert not trial.should_prune() trial.report(1.1, 5) assert trial.should_prune() def test_threshold_pruner_with_invalid_inputs() -> None: with pytest.raises(TypeError): optuna.pruners.ThresholdPruner(lower="val", upper=1.0) # type: ignore with pytest.raises(TypeError): optuna.pruners.ThresholdPruner(lower=0.0, upper="val") # type: ignore with pytest.raises(TypeError): optuna.pruners.ThresholdPruner(lower=None, upper=None) def test_threshold_pruner_with_nan() -> None: pruner = optuna.pruners.ThresholdPruner( lower=0.0, upper=1.0, n_warmup_steps=0, interval_steps=1 ) study = optuna.study.create_study(pruner=pruner) trial = study.ask() trial.report(float("nan"), 1) assert trial.should_prune() def test_threshold_pruner_n_warmup_steps() -> None: pruner = optuna.pruners.ThresholdPruner(lower=0.0, upper=1.0, n_warmup_steps=2) study = optuna.study.create_study(pruner=pruner) trial = study.ask() trial.report(-10.0, 0) assert not trial.should_prune() trial.report(100.0, 1) assert not trial.should_prune() trial.report(-100.0, 3) assert trial.should_prune() trial.report(1.0, 4) assert not trial.should_prune() trial.report(1000.0, 5) assert trial.should_prune() def test_threshold_pruner_interval_steps() -> None: pruner = optuna.pruners.ThresholdPruner(lower=0.0, upper=1.0, interval_steps=2) study = optuna.study.create_study(pruner=pruner) trial = study.ask() trial.report(-10.0, 0) assert trial.should_prune() trial.report(100.0, 1) assert not trial.should_prune() trial.report(-100.0, 2) assert trial.should_prune() trial.report(10.0, 3) assert not trial.should_prune() trial.report(1000.0, 4) assert trial.should_prune() optuna-4.1.0/tests/pruners_tests/test_wilcoxon.py000066400000000000000000000106151471332314300223630ustar00rootroot00000000000000from __future__ import annotations import pytest import optuna def test_wilcoxon_pruner_constructor() -> None: optuna.pruners.WilcoxonPruner() optuna.pruners.WilcoxonPruner(p_threshold=0) optuna.pruners.WilcoxonPruner(p_threshold=1) optuna.pruners.WilcoxonPruner(p_threshold=0.05) optuna.pruners.WilcoxonPruner(n_startup_steps=5) with pytest.raises(ValueError): optuna.pruners.WilcoxonPruner(p_threshold=-0.1) with pytest.raises(ValueError): optuna.pruners.WilcoxonPruner(n_startup_steps=-5) def test_wilcoxon_pruner_first_trial() -> None: # A pruner is not activated at a first trial. pruner = optuna.pruners.WilcoxonPruner() study = optuna.study.create_study(pruner=pruner) trial = study.ask() assert not trial.should_prune() trial.report(1, 1) assert not trial.should_prune() trial.report(2, 2) assert not trial.should_prune() def test_wilcoxon_pruner_when_best_trial_has_no_intermediate_value() -> None: # A pruner is not activated at a first trial. pruner = optuna.pruners.WilcoxonPruner() study = optuna.study.create_study(pruner=pruner) trial = study.ask() study.tell(trial, 10) trial = study.ask() assert not trial.should_prune() trial.report(1, 1) assert not trial.should_prune() trial.report(2, 2) assert not trial.should_prune() @pytest.mark.parametrize( "p_threshold,step_values,expected_should_prune", [ (0.2, [-1, 1, 2, 3, 4, 5, -2, -3], [False, False, False, True, True, True, True, True]), (0.15, [-1, 1, 2, 3, 4, 5, -2, -3], [False, False, False, False, True, True, True, False]), ], ) def test_wilcoxon_pruner_normal( p_threshold: float, step_values: list[float], expected_should_prune: list[bool], ) -> None: pruner = optuna.pruners.WilcoxonPruner(n_startup_steps=0, p_threshold=p_threshold) study = optuna.study.create_study(pruner=pruner) # Insert the best trial study.add_trial( optuna.trial.create_trial( value=0, params={}, distributions={}, intermediate_values={step: 0 for step in range(10)}, ) ) trial = study.ask() should_prune = [False] * len(step_values) for step_i in range(len(step_values)): trial.report(step_values[step_i], step_i) should_prune[step_i] = trial.should_prune() assert should_prune == expected_should_prune @pytest.mark.parametrize( "best_intermediate_values,intermediate_values", [ ({1: 1}, {1: 1, 2: 2}), # Current trial has more steps than the best trial ({1: 1}, {1: float("nan")}), # NaN value ({1: float("nan")}, {1: 1}), # NaN value ({1: 1}, {1: float("inf")}), # infinite value ({1: float("inf")}, {1: 1}), # infinite value ], ) @pytest.mark.parametrize( "direction", ("minimize", "maximize"), ) def test_wilcoxon_pruner_warn_bad_best_trial( best_intermediate_values: dict[int, float], intermediate_values: dict[int, float], direction: str, ) -> None: pruner = optuna.pruners.WilcoxonPruner() study = optuna.study.create_study(direction=direction, pruner=pruner) # Insert best trial study.add_trial( optuna.trial.create_trial( value=0, params={}, distributions={}, intermediate_values=best_intermediate_values ) ) trial = study.ask() with pytest.warns(UserWarning): for step, value in intermediate_values.items(): trial.report(value, step) trial.should_prune() def test_wilcoxon_pruner_if_average_is_best_then_not_prune() -> None: pruner = optuna.pruners.WilcoxonPruner(p_threshold=0.5) study = optuna.study.create_study(direction="minimize", pruner=pruner) best_intermediate_values_value = [0.0 for _ in range(10)] + [8.0 for _ in range(10)] best_intermediate_values = dict(zip(list(range(20)), best_intermediate_values_value)) # Insert best trial study.add_trial( optuna.trial.create_trial( value=4.0, params={}, distributions={}, intermediate_values=best_intermediate_values ) ) trial = study.ask() intermediate_values = [1.0 for _ in range(10)] + [9.0 for _ in range(10)] for step, value in enumerate(intermediate_values): trial.report(value, step) average = sum(intermediate_values[: step + 1]) / (step + 1) if average <= 4.0: assert not trial.should_prune() optuna-4.1.0/tests/samplers_tests/000077500000000000000000000000001471332314300172355ustar00rootroot00000000000000optuna-4.1.0/tests/samplers_tests/__init__.py000066400000000000000000000000001471332314300213340ustar00rootroot00000000000000optuna-4.1.0/tests/samplers_tests/test_brute_force.py000066400000000000000000000224241471332314300231510ustar00rootroot00000000000000import numpy as np import pytest import optuna from optuna import samplers from optuna.samplers._brute_force import _TreeNode from optuna.trial import Trial def test_tree_node_add_paths() -> None: tree = _TreeNode() leafs = [ tree.add_path([("a", [0, 1, 2], 0), ("b", [0.0, 1.0], 0.0)]), tree.add_path([("a", [0, 1, 2], 0), ("b", [0.0, 1.0], 1.0)]), tree.add_path([("a", [0, 1, 2], 0), ("b", [0.0, 1.0], 1.0)]), tree.add_path([("a", [0, 1, 2], 1), ("b", [0.0, 1.0], 0.0), ("c", [0, 1], 0)]), tree.add_path([("a", [0, 1, 2], 1), ("b", [0.0, 1.0], 0.0)]), ] for leaf in leafs: assert leaf is not None if leaf.children is None: leaf.set_leaf() assert tree == _TreeNode( param_name="a", children={ 0: _TreeNode( param_name="b", children={ 0.0: _TreeNode(param_name=None, children={}), 1.0: _TreeNode(param_name=None, children={}), }, ), 1: _TreeNode( param_name="b", children={ 0.0: _TreeNode( param_name="c", children={ 0: _TreeNode(param_name=None, children={}), 1: _TreeNode(), }, ), 1.0: _TreeNode(), }, ), 2: _TreeNode(), }, ) def test_tree_node_add_paths_error() -> None: with pytest.raises(ValueError): tree = _TreeNode() tree.add_path([("a", [0, 1, 2], 0)]) tree.add_path([("a", [0, 1], 0)]) with pytest.raises(ValueError): tree = _TreeNode() tree.add_path([("a", [0, 1, 2], 0)]) tree.add_path([("b", [0, 1, 2], 0)]) def test_tree_node_count_unexpanded() -> None: tree = _TreeNode( param_name="a", children={ 0: _TreeNode( param_name="b", children={ 0.0: _TreeNode(param_name=None, children={}), 1.0: _TreeNode(param_name=None, children={}), }, ), 1: _TreeNode( param_name="b", children={ 0.0: _TreeNode( param_name="c", children={ 0: _TreeNode(param_name=None, children={}), 1: _TreeNode(), }, ), 1.0: _TreeNode(), }, ), 2: _TreeNode(), }, ) assert tree.count_unexpanded() == 3 def test_study_optimize_with_single_search_space() -> None: def objective(trial: Trial) -> float: a = trial.suggest_int("a", 0, 2) if a == 0: b = trial.suggest_float("b", -1.0, 1.0, step=0.5) return a + b elif a == 1: c = trial.suggest_categorical("c", ["x", "y", None]) if c == "x": return a + 1 else: return a - 1 else: return a * 2 study = optuna.create_study(sampler=samplers.BruteForceSampler()) study.optimize(objective) expected_suggested_values = [ {"a": 0, "b": -1.0}, {"a": 0, "b": -0.5}, {"a": 0, "b": 0.0}, {"a": 0, "b": 0.5}, {"a": 0, "b": 1.0}, {"a": 1, "c": "x"}, {"a": 1, "c": "y"}, {"a": 1, "c": None}, {"a": 2}, ] all_suggested_values = [t.params for t in study.trials] assert len(all_suggested_values) == len(expected_suggested_values) for a in all_suggested_values: assert a in expected_suggested_values def test_study_optimize_with_pruned_trials() -> None: def objective(trial: Trial) -> float: a = trial.suggest_int("a", 0, 2) if a == 0: trial.suggest_float("b", -1.0, 1.0, step=0.5) raise optuna.TrialPruned elif a == 1: c = trial.suggest_categorical("c", ["x", "y", None]) if c == "x": return a + 1 else: return a - 1 else: return a * 2 study = optuna.create_study(sampler=samplers.BruteForceSampler()) study.optimize(objective) expected_suggested_values = [ {"a": 0, "b": -1.0}, {"a": 0, "b": -0.5}, {"a": 0, "b": 0.0}, {"a": 0, "b": 0.5}, {"a": 0, "b": 1.0}, {"a": 1, "c": "x"}, {"a": 1, "c": "y"}, {"a": 1, "c": None}, {"a": 2}, ] all_suggested_values = [t.params for t in study.trials] assert len(all_suggested_values) == len(expected_suggested_values) for a in all_suggested_values: assert a in expected_suggested_values def test_study_optimize_with_infinite_search_space() -> None: def objective(trial: Trial) -> float: return trial.suggest_float("a", 0, 2) study = optuna.create_study(sampler=samplers.BruteForceSampler()) with pytest.raises(ValueError): study.optimize(objective) def test_study_optimize_with_nan() -> None: def objective(trial: Trial) -> float: trial.suggest_categorical("a", [0.0, float("nan")]) return 1.0 study = optuna.create_study(sampler=samplers.BruteForceSampler()) study.optimize(objective) all_suggested_values = [t.params["a"] for t in study.trials] assert len(all_suggested_values) == 2 assert 0.0 in all_suggested_values assert np.isnan(all_suggested_values[0]) or np.isnan(all_suggested_values[1]) def test_study_optimize_with_single_search_space_user_added() -> None: def objective(trial: Trial) -> float: a = trial.suggest_int("a", 0, 2) if a == 0: b = trial.suggest_float("b", -1.0, 1.0, step=0.5) return a + b elif a == 1: c = trial.suggest_categorical("c", ["x", "y", None]) if c == "x": return a + 1 else: return a - 1 else: return a * 2 study = optuna.create_study(sampler=samplers.BruteForceSampler()) # Manually add a trial. This should not be tried again. study.add_trial( optuna.create_trial( params={"a": 0, "b": -1.0}, value=0.0, distributions={ "a": optuna.distributions.IntDistribution(0, 2), "b": optuna.distributions.FloatDistribution(-1.0, 1.0, step=0.5), }, ) ) study.optimize(objective) expected_suggested_values = [ {"a": 0, "b": -1.0}, {"a": 0, "b": -0.5}, {"a": 0, "b": 0.0}, {"a": 0, "b": 0.5}, {"a": 0, "b": 1.0}, {"a": 1, "c": "x"}, {"a": 1, "c": "y"}, {"a": 1, "c": None}, {"a": 2}, ] all_suggested_values = [t.params for t in study.trials] assert len(all_suggested_values) == len(expected_suggested_values) for a in all_suggested_values: assert a in expected_suggested_values def test_study_optimize_with_nonconstant_search_space() -> None: def objective_nonconstant_range(trial: Trial) -> float: x = trial.suggest_int("x", -1, trial.number) return x study = optuna.create_study(sampler=samplers.BruteForceSampler()) with pytest.raises(ValueError): study.optimize(objective_nonconstant_range, n_trials=10) def objective_increasing_variable(trial: Trial) -> float: return sum(trial.suggest_int(f"x{i}", 0, 0) for i in range(2)) study = optuna.create_study(sampler=samplers.BruteForceSampler()) study.add_trial( optuna.create_trial( params={"x0": 0}, value=0.0, distributions={"x0": optuna.distributions.IntDistribution(0, 0)}, ) ) with pytest.raises(ValueError): study.optimize(objective_increasing_variable, n_trials=10) def objective_decreasing_variable(trial: Trial) -> float: return trial.suggest_int("x0", 0, 0) study = optuna.create_study(sampler=samplers.BruteForceSampler()) study.add_trial( optuna.create_trial( params={"x0": 0, "x1": 0}, value=0.0, distributions={ "x0": optuna.distributions.IntDistribution(0, 0), "x1": optuna.distributions.IntDistribution(0, 0), }, ) ) with pytest.raises(ValueError): study.optimize(objective_decreasing_variable, n_trials=10) def test_study_optimize_with_failed_trials() -> None: def objective(trial: Trial) -> float: trial.suggest_int("x", 0, 99) return np.nan study = optuna.create_study(sampler=samplers.BruteForceSampler()) study.optimize(objective, n_trials=100) expected_suggested_values = [{"x": i} for i in range(100)] all_suggested_values = [t.params for t in study.trials] assert len(all_suggested_values) == len(expected_suggested_values) for a in expected_suggested_values: assert a in all_suggested_values def test_parallel_optimize() -> None: study = optuna.create_study(sampler=samplers.BruteForceSampler()) trial1 = study.ask() trial2 = study.ask() x1 = trial1.suggest_categorical("x", ["a", "b"]) x2 = trial2.suggest_categorical("x", ["a", "b"]) assert {x1, x2} == {"a", "b"} optuna-4.1.0/tests/samplers_tests/test_cmaes.py000066400000000000000000000700611471332314300217420ustar00rootroot00000000000000from __future__ import annotations import math from typing import Any from unittest.mock import MagicMock from unittest.mock import Mock from unittest.mock import patch import warnings import _pytest.capture from cmaes import CMA from cmaes import CMAwM from cmaes import SepCMA import numpy as np import pytest import optuna from optuna import create_trial from optuna._transform import _SearchSpaceTransform from optuna.testing.storages import StorageSupplier from optuna.trial import FrozenTrial from optuna.trial import TrialState def test_consider_pruned_trials_experimental_warning() -> None: with pytest.warns(optuna.exceptions.ExperimentalWarning): optuna.samplers.CmaEsSampler(consider_pruned_trials=True) def test_with_margin_experimental_warning() -> None: with pytest.warns(optuna.exceptions.ExperimentalWarning): optuna.samplers.CmaEsSampler(with_margin=True) def test_lr_adapt_experimental_warning() -> None: with pytest.warns(optuna.exceptions.ExperimentalWarning): optuna.samplers.CmaEsSampler(lr_adapt=True) @pytest.mark.filterwarnings("ignore::optuna.exceptions.ExperimentalWarning") @pytest.mark.parametrize( "use_separable_cma, cma_class_str", [(False, "optuna.samplers._cmaes.cmaes.CMA"), (True, "optuna.samplers._cmaes.cmaes.SepCMA")], ) @pytest.mark.parametrize("popsize", [None, 8]) def test_init_cmaes_opts(use_separable_cma: bool, cma_class_str: str, popsize: int | None) -> None: sampler = optuna.samplers.CmaEsSampler( x0={"x": 0, "y": 0}, sigma0=0.1, seed=1, n_startup_trials=1, use_separable_cma=use_separable_cma, popsize=popsize, ) study = optuna.create_study(sampler=sampler) with patch(cma_class_str) as cma_class: cma_obj = MagicMock() cma_obj.ask.return_value = np.array((-1, -1)) cma_obj.generation = 0 cma_class.return_value = cma_obj study.optimize( lambda t: t.suggest_float("x", -1, 1) + t.suggest_float("y", -1, 1), n_trials=2 ) assert cma_class.call_count == 1 _, actual_kwargs = cma_class.call_args assert np.array_equal(actual_kwargs["mean"], np.array([0.5, 0.5])) assert actual_kwargs["sigma"] == 0.1 assert np.allclose(actual_kwargs["bounds"], np.array([(0, 1), (0, 1)])) assert actual_kwargs["seed"] == np.random.RandomState(1).randint(1, np.iinfo(np.int32).max) assert actual_kwargs["n_max_resampling"] == 10 * 2 expected_popsize = 4 + math.floor(3 * math.log(2)) if popsize is None else popsize assert actual_kwargs["population_size"] == expected_popsize @pytest.mark.filterwarnings("ignore::optuna.exceptions.ExperimentalWarning") @pytest.mark.parametrize("popsize", [None, 8]) def test_init_cmaes_opts_with_margin(popsize: int | None) -> None: sampler = optuna.samplers.CmaEsSampler( x0={"x": 0, "y": 0}, sigma0=0.1, seed=1, n_startup_trials=1, popsize=popsize, with_margin=True, ) study = optuna.create_study(sampler=sampler) with patch("optuna.samplers._cmaes.cmaes.CMAwM") as cma_class: cma_obj = MagicMock() cma_obj.ask.return_value = np.array((-1, -1)) cma_obj.generation = 0 cma_class.return_value = cma_obj study.optimize( lambda t: t.suggest_float("x", -1, 1) + t.suggest_int("y", -1, 1), n_trials=2 ) assert cma_class.call_count == 1 _, actual_kwargs = cma_class.call_args assert np.array_equal(actual_kwargs["mean"], np.array([0.5, 0.5])) assert actual_kwargs["sigma"] == 0.1 assert np.allclose(actual_kwargs["bounds"], np.array([(0, 1), (0, 1)])) assert np.allclose(actual_kwargs["steps"], np.array([0.0, 0.5])) assert actual_kwargs["seed"] == np.random.RandomState(1).randint(1, np.iinfo(np.int32).max) assert actual_kwargs["n_max_resampling"] == 10 * 2 expected_popsize = 4 + math.floor(3 * math.log(2)) if popsize is None else popsize assert actual_kwargs["population_size"] == expected_popsize @pytest.mark.filterwarnings("ignore::optuna.exceptions.ExperimentalWarning") @pytest.mark.parametrize("popsize", [None, 8]) def test_init_cmaes_opts_lr_adapt(popsize: int | None) -> None: sampler = optuna.samplers.CmaEsSampler( x0={"x": 0, "y": 0}, sigma0=0.1, seed=1, n_startup_trials=1, popsize=popsize, lr_adapt=True, ) study = optuna.create_study(sampler=sampler) with patch("optuna.samplers._cmaes.cmaes.CMA") as cma_class: cma_obj = MagicMock() cma_obj.ask.return_value = np.array((-1, -1)) cma_obj.generation = 0 cma_class.return_value = cma_obj study.optimize( lambda t: t.suggest_float("x", -1, 1) + t.suggest_float("y", -1, 1), n_trials=2 ) assert cma_class.call_count == 1 _, actual_kwargs = cma_class.call_args assert actual_kwargs["lr_adapt"] is True @pytest.mark.filterwarnings("ignore::optuna.exceptions.ExperimentalWarning") @pytest.mark.parametrize("with_margin", [False, True]) def test_warm_starting_cmaes(with_margin: bool) -> None: def objective(trial: optuna.Trial) -> float: x = trial.suggest_float("x", -10, 10) y = trial.suggest_int("y", -10, 10) return x**2 + y source_study = optuna.create_study() source_study.optimize(objective, 20) source_trials = source_study.get_trials(deepcopy=False) with patch("optuna.samplers._cmaes.cmaes.get_warm_start_mgd") as mock_func_ws: mock_func_ws.return_value = (np.zeros(2), 0.0, np.zeros((2, 2))) sampler = optuna.samplers.CmaEsSampler( seed=1, n_startup_trials=1, with_margin=with_margin, source_trials=source_trials ) study = optuna.create_study(sampler=sampler) study.optimize(objective, 2) assert mock_func_ws.call_count == 1 @pytest.mark.filterwarnings("ignore::optuna.exceptions.ExperimentalWarning") @pytest.mark.parametrize("with_margin", [False, True]) def test_warm_starting_cmaes_maximize(with_margin: bool) -> None: def objective(trial: optuna.Trial) -> float: x = trial.suggest_float("x", -10, 10) y = trial.suggest_int("y", -10, 10) # Objective values are negative. return -(x**2) - (y - 5) ** 2 source_study = optuna.create_study(direction="maximize") source_study.optimize(objective, 20) source_trials = source_study.get_trials(deepcopy=False) with patch("optuna.samplers._cmaes.cmaes.get_warm_start_mgd") as mock_func_ws: mock_func_ws.return_value = (np.zeros(2), 0.0, np.zeros((2, 2))) sampler = optuna.samplers.CmaEsSampler( seed=1, n_startup_trials=1, with_margin=with_margin, source_trials=source_trials ) study = optuna.create_study(sampler=sampler, direction="maximize") study.optimize(objective, 2) assert mock_func_ws.call_count == 1 solutions_arg = mock_func_ws.call_args[0][0] is_positive = [x[1] >= 0 for x in solutions_arg] assert all(is_positive) @pytest.mark.filterwarnings("ignore::optuna.exceptions.ExperimentalWarning") def test_should_raise_exception() -> None: dummy_source_trials = [create_trial(value=i, state=TrialState.COMPLETE) for i in range(10)] with pytest.raises(ValueError): optuna.samplers.CmaEsSampler( x0={"x": 0.1, "y": 0.1}, source_trials=dummy_source_trials, ) with pytest.raises(ValueError): optuna.samplers.CmaEsSampler( sigma0=0.1, source_trials=dummy_source_trials, ) with pytest.raises(ValueError): optuna.samplers.CmaEsSampler( use_separable_cma=True, source_trials=dummy_source_trials, ) with pytest.raises(ValueError): optuna.samplers.CmaEsSampler( restart_strategy="invalid-restart-strategy", ) with pytest.raises(ValueError): optuna.samplers.CmaEsSampler(use_separable_cma=True, with_margin=True) @pytest.mark.filterwarnings("ignore::optuna.exceptions.ExperimentalWarning") @pytest.mark.parametrize("with_margin", [False, True]) def test_incompatible_search_space(with_margin: bool) -> None: def objective1(trial: optuna.Trial) -> float: x0 = trial.suggest_float("x0", 2, 3) x1 = trial.suggest_float("x1", 1e-2, 1e2, log=True) return x0 + x1 source_study = optuna.create_study() source_study.optimize(objective1, 20) # Should not raise an exception. sampler = optuna.samplers.CmaEsSampler( with_margin=with_margin, source_trials=source_study.trials ) target_study1 = optuna.create_study(sampler=sampler) target_study1.optimize(objective1, 20) def objective2(trial: optuna.Trial) -> float: x0 = trial.suggest_float("x0", 2, 3) x1 = trial.suggest_float("x1", 1e-2, 1e2, log=True) x2 = trial.suggest_float("x2", 1e-2, 1e2, log=True) return x0 + x1 + x2 # Should raise an exception. sampler = optuna.samplers.CmaEsSampler( with_margin=with_margin, source_trials=source_study.trials ) target_study2 = optuna.create_study(sampler=sampler) with pytest.raises(ValueError): target_study2.optimize(objective2, 20) def test_infer_relative_search_space_1d() -> None: sampler = optuna.samplers.CmaEsSampler() study = optuna.create_study(sampler=sampler) # The distribution has only one candidate. study.optimize(lambda t: t.suggest_int("x", 1, 1), n_trials=1) assert sampler.infer_relative_search_space(study, study.best_trial) == {} def test_sample_relative_1d() -> None: independent_sampler = optuna.samplers.RandomSampler() sampler = optuna.samplers.CmaEsSampler(independent_sampler=independent_sampler) study = optuna.create_study(sampler=sampler) # If search space is one dimensional, the independent sampler is always used. with patch.object( independent_sampler, "sample_independent", wraps=independent_sampler.sample_independent ) as mock_object: study.optimize(lambda t: t.suggest_int("x", -1, 1), n_trials=2) assert mock_object.call_count == 2 def test_sample_relative_n_startup_trials() -> None: independent_sampler = optuna.samplers.RandomSampler() sampler = optuna.samplers.CmaEsSampler( n_startup_trials=2, independent_sampler=independent_sampler ) study = optuna.create_study(sampler=sampler) def objective(t: optuna.Trial) -> float: value = t.suggest_int("x", -1, 1) + t.suggest_int("y", -1, 1) if t.number == 0: raise Exception("first trial is failed") return float(value) # The independent sampler is used for Trial#0 (FAILED), Trial#1 (COMPLETE) # and Trial#2 (COMPLETE). The CMA-ES is used for Trial#3 (COMPLETE). with patch.object( independent_sampler, "sample_independent", wraps=independent_sampler.sample_independent ) as mock_independent, patch.object( sampler, "sample_relative", wraps=sampler.sample_relative ) as mock_relative: study.optimize(objective, n_trials=4, catch=(Exception,)) assert mock_independent.call_count == 6 # The objective function has two parameters. assert mock_relative.call_count == 4 def test_get_trials() -> None: with patch( "optuna.Study._get_trials", new=Mock(side_effect=lambda deepcopy, use_cache: _create_trials()), ): sampler = optuna.samplers.CmaEsSampler(consider_pruned_trials=False) study = optuna.create_study(sampler=sampler) trials = sampler._get_trials(study) assert len(trials) == 1 sampler = optuna.samplers.CmaEsSampler(consider_pruned_trials=True) study = optuna.create_study(sampler=sampler) trials = sampler._get_trials(study) assert len(trials) == 2 assert trials[0].value == 1.0 assert trials[1].value == 2.0 def _create_trials() -> list[FrozenTrial]: trials = [] trials.append( FrozenTrial( number=0, value=1.0, state=optuna.trial.TrialState.COMPLETE, user_attrs={}, system_attrs={}, params={}, distributions={}, intermediate_values={}, datetime_start=None, datetime_complete=None, trial_id=0, ) ) trials.append( FrozenTrial( number=1, value=None, state=optuna.trial.TrialState.PRUNED, user_attrs={}, system_attrs={}, params={}, distributions={}, intermediate_values={0: 2.0}, datetime_start=None, datetime_complete=None, trial_id=0, ) ) return trials @pytest.mark.parametrize( "options, key", [ ({"with_margin": False, "use_separable_cma": False}, "cma:"), ({"with_margin": True, "use_separable_cma": False}, "cmawm:"), ({"with_margin": False, "use_separable_cma": True}, "sepcma:"), ], ) def test_sampler_attr_key(options: dict[str, bool], key: str) -> None: # Test sampler attr_key property. sampler = optuna.samplers.CmaEsSampler( with_margin=options["with_margin"], use_separable_cma=options["use_separable_cma"] ) assert sampler._attr_keys.optimizer(0).startswith(key) assert sampler._attr_keys.popsize().startswith(key) assert sampler._attr_keys.n_restarts().startswith(key) assert sampler._attr_keys.n_restarts_with_large.startswith(key) assert sampler._attr_keys.poptype.startswith(key) assert sampler._attr_keys.small_n_eval.startswith(key) assert sampler._attr_keys.large_n_eval.startswith(key) assert sampler._attr_keys.generation(0).startswith(key) for restart_strategy in ["ipop", "bipop"]: sampler._restart_strategy = restart_strategy for i in range(3): assert sampler._attr_keys.generation(i).startswith( (key + "{}:restart_{}:".format(restart_strategy, i) + "generation") ) @pytest.mark.parametrize("popsize", [None, 16]) def test_population_size_is_multiplied_when_enable_ipop(popsize: int | None) -> None: inc_popsize = 2 sampler = optuna.samplers.CmaEsSampler( x0={"x": 0, "y": 0}, sigma0=0.1, seed=1, n_startup_trials=1, restart_strategy="ipop", popsize=popsize, inc_popsize=inc_popsize, ) study = optuna.create_study(sampler=sampler) def objective(trial: optuna.Trial) -> float: _ = trial.suggest_float("x", -1, 1) _ = trial.suggest_float("y", -1, 1) return 1.0 with patch("optuna.samplers._cmaes.cmaes.CMA") as cma_class_mock, patch( "optuna.samplers._cmaes.pickle" ) as pickle_mock: pickle_mock.dump.return_value = b"serialized object" should_stop_mock = MagicMock() should_stop_mock.return_value = True cma_obj = CMA( mean=np.array([-1, -1], dtype=float), sigma=1.3, bounds=np.array([[-1, 1], [-1, 1]], dtype=float), population_size=popsize, # Already tested by test_init_cmaes_opts(). ) cma_obj.should_stop = should_stop_mock cma_class_mock.return_value = cma_obj initial_popsize = cma_obj.population_size study.optimize(objective, n_trials=2 + initial_popsize) assert cma_obj.should_stop.call_count == 1 _, actual_kwargs = cma_class_mock.call_args assert actual_kwargs["population_size"] == inc_popsize * initial_popsize @pytest.mark.parametrize("sampler_opts", [{}, {"use_separable_cma": True}, {"with_margin": True}]) def test_restore_optimizer_from_substrings(sampler_opts: dict[str, Any]) -> None: popsize = 8 sampler = optuna.samplers.CmaEsSampler(popsize=popsize, **sampler_opts) optimizer = sampler._restore_optimizer([]) assert optimizer is None def objective(trial: optuna.Trial) -> float: x1 = trial.suggest_float("x1", -10, 10, step=1) x2 = trial.suggest_float("x2", -10, 10) return x1**2 + x2**2 study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=popsize + 2) optimizer = sampler._restore_optimizer(study.trials) assert optimizer is not None assert optimizer.generation == 1 if sampler._with_margin: assert isinstance(optimizer, CMAwM) elif sampler._use_separable_cma: assert isinstance(optimizer, SepCMA) else: assert isinstance(optimizer, CMA) @pytest.mark.parametrize( "sampler_opts", [ {"restart_strategy": "ipop"}, {"restart_strategy": "bipop"}, {"restart_strategy": "ipop", "use_separable_cma": True}, {"restart_strategy": "bipop", "use_separable_cma": True}, {"restart_strategy": "ipop", "with_margin": True}, {"restart_strategy": "bipop", "with_margin": True}, ], ) def test_restore_optimizer_after_restart(sampler_opts: dict[str, Any]) -> None: def objective(trial: optuna.Trial) -> float: x1 = trial.suggest_float("x1", -10, 10, step=1) x2 = trial.suggest_float("x2", -10, 10) return x1**2 + x2**2 if sampler_opts.get("with_margin"): cma_class = CMAwM elif sampler_opts.get("use_separable_cma"): cma_class = SepCMA else: cma_class = CMA with patch.object(cma_class, "should_stop") as mock_method: mock_method.return_value = True sampler = optuna.samplers.CmaEsSampler(popsize=5, **sampler_opts) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=5 + 2) optimizer = sampler._restore_optimizer(study.trials, 1) assert optimizer is not None assert optimizer.generation == 0 @pytest.mark.parametrize( "sampler_opts, restart_strategy", [ ({"use_separable_cma": True}, "ipop"), ({"use_separable_cma": True}, "bipop"), ({"with_margin": True}, "ipop"), ({"with_margin": True}, "bipop"), ], ) def test_restore_optimizer_with_other_option( sampler_opts: dict[str, Any], restart_strategy: str ) -> None: def objective(trial: optuna.Trial) -> float: x1 = trial.suggest_float("x1", -10, 10, step=1) x2 = trial.suggest_float("x2", -10, 10) return x1**2 + x2**2 with patch.object(CMA, "should_stop") as mock_method: mock_method.return_value = True sampler = optuna.samplers.CmaEsSampler(popsize=5, restart_strategy=restart_strategy) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=5 + 2) # Restore optimizer via SepCMA or CMAwM samplers. sampler = optuna.samplers.CmaEsSampler(**sampler_opts) optimizer = sampler._restore_optimizer(study.trials) assert optimizer is None @pytest.mark.parametrize( "sampler_opts", [ {"restart_strategy": "ipop"}, {"restart_strategy": "bipop"}, {"restart_strategy": "ipop", "use_separable_cma": True}, {"restart_strategy": "bipop", "use_separable_cma": True}, {"restart_strategy": "ipop", "with_margin": True}, {"restart_strategy": "bipop", "with_margin": True}, ], ) def test_get_solution_trials(sampler_opts: dict[str, Any]) -> None: def objective(trial: optuna.Trial) -> float: x1 = trial.suggest_float("x1", -10, 10, step=1) x2 = trial.suggest_float("x2", -10, 10) return x1**2 + x2**2 popsize = 5 sampler = optuna.samplers.CmaEsSampler(popsize=popsize, **sampler_opts) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=popsize + 2) # The number of solutions for generation 0 equals population size. assert len(sampler._get_solution_trials(study.trials, 0, 0)) == popsize # The number of solutions for generation 1 is 1. assert len(sampler._get_solution_trials(study.trials, 1, 0)) == 1 @pytest.mark.parametrize( "sampler_opts, restart_strategy", [ ({"use_separable_cma": True}, "ipop"), ({"use_separable_cma": True}, "bipop"), ({"with_margin": True}, "ipop"), ({"with_margin": True}, "bipop"), ], ) def test_get_solution_trials_with_other_options( sampler_opts: dict[str, Any], restart_strategy: str ) -> None: def objective(trial: optuna.Trial) -> float: x1 = trial.suggest_float("x1", -10, 10, step=1) x2 = trial.suggest_float("x2", -10, 10) return x1**2 + x2**2 sampler = optuna.samplers.CmaEsSampler(popsize=5, restart_strategy=restart_strategy) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=5 + 2) # The number of solutions is 0 after changed samplers sampler = optuna.samplers.CmaEsSampler(**sampler_opts) assert len(sampler._get_solution_trials(study.trials, 0, 0)) == 0 @pytest.mark.parametrize( "sampler_opts", [ {"restart_strategy": "ipop"}, {"restart_strategy": "bipop"}, {"restart_strategy": "ipop", "use_separable_cma": True}, {"restart_strategy": "bipop", "use_separable_cma": True}, {"restart_strategy": "ipop", "with_margin": True}, {"restart_strategy": "bipop", "with_margin": True}, ], ) def test_get_solution_trials_after_restart(sampler_opts: dict[str, Any]) -> None: def objective(trial: optuna.Trial) -> float: x1 = trial.suggest_float("x1", -10, 10, step=1) x2 = trial.suggest_float("x2", -10, 10) return x1**2 + x2**2 if sampler_opts.get("with_margin"): cma_class = CMAwM elif sampler_opts.get("use_separable_cma"): cma_class = SepCMA else: cma_class = CMA popsize = 5 with patch.object(cma_class, "should_stop") as mock_method: mock_method.return_value = True sampler = optuna.samplers.CmaEsSampler(popsize=popsize, **sampler_opts) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=popsize + 2) # The number of solutions for generation=0 and n_restarts=0 equals population size. assert len(sampler._get_solution_trials(study.trials, 0, 0)) == popsize # The number of solutions for generation=1 and n_restarts=0 is 0. assert len(sampler._get_solution_trials(study.trials, 1, 0)) == 0 # The number of solutions for generation=0 and n_restarts=1 is 1 since it was restarted. assert len(sampler._get_solution_trials(study.trials, 0, 1)) == 1 @pytest.mark.parametrize( "dummy_optimizer_str,attr_len", [ ("012", 1), ("01234", 1), ("012345", 2), ], ) def test_split_and_concat_optimizer_string(dummy_optimizer_str: str, attr_len: int) -> None: sampler = optuna.samplers.CmaEsSampler() with patch("optuna.samplers._cmaes._SYSTEM_ATTR_MAX_LENGTH", 5): attrs = sampler._split_optimizer_str(dummy_optimizer_str) assert len(attrs) == attr_len actual = sampler._concat_optimizer_attrs(attrs) assert dummy_optimizer_str == actual def test_call_after_trial_of_base_sampler() -> None: independent_sampler = optuna.samplers.RandomSampler() with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = optuna.samplers.CmaEsSampler(independent_sampler=independent_sampler) study = optuna.create_study(sampler=sampler) with patch.object( independent_sampler, "after_trial", wraps=independent_sampler.after_trial ) as mock_object: study.optimize(lambda _: 1.0, n_trials=1) assert mock_object.call_count == 1 def test_is_compatible_search_space() -> None: transform = _SearchSpaceTransform( { "x0": optuna.distributions.FloatDistribution(2, 3), "x1": optuna.distributions.CategoricalDistribution(["foo", "bar", "baz", "qux"]), } ) assert optuna.samplers._cmaes._is_compatible_search_space( transform, { "x1": optuna.distributions.CategoricalDistribution(["foo", "bar", "baz", "qux"]), "x0": optuna.distributions.FloatDistribution(2, 3), }, ) # Same search space size, but different param names. assert not optuna.samplers._cmaes._is_compatible_search_space( transform, { "x0": optuna.distributions.FloatDistribution(2, 3), "foo": optuna.distributions.CategoricalDistribution(["foo", "bar", "baz", "qux"]), }, ) # x2 is added. assert not optuna.samplers._cmaes._is_compatible_search_space( transform, { "x0": optuna.distributions.FloatDistribution(2, 3), "x1": optuna.distributions.CategoricalDistribution(["foo", "bar", "baz", "qux"]), "x2": optuna.distributions.FloatDistribution(2, 3, step=0.1), }, ) # x0 is not found. assert not optuna.samplers._cmaes._is_compatible_search_space( transform, { "x1": optuna.distributions.CategoricalDistribution(["foo", "bar", "baz", "qux"]), }, ) def test_internal_optimizer_with_margin() -> None: def objective_discrete(trial: optuna.Trial) -> float: x = trial.suggest_int("x", -10, 10) y = trial.suggest_int("y", -10, 10) return x**2 + y def objective_mixed(trial: optuna.Trial) -> float: x = trial.suggest_float("x", -10, 10) y = trial.suggest_int("y", -10, 10) return x**2 + y def objective_continuous(trial: optuna.Trial) -> float: x = trial.suggest_float("x", -10, 10) y = trial.suggest_float("y", -10, 10) return x**2 + y objectives = [objective_discrete, objective_mixed, objective_continuous] for objective in objectives: with patch("optuna.samplers._cmaes.cmaes.CMAwM") as cmawm_class_mock: sampler = optuna.samplers.CmaEsSampler(with_margin=True) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=2) assert cmawm_class_mock.call_count == 1 @pytest.mark.parametrize("warn_independent_sampling", [True, False]) def test_warn_independent_sampling( capsys: _pytest.capture.CaptureFixture, warn_independent_sampling: bool ) -> None: def objective_single(trial: optuna.trial.Trial) -> float: return trial.suggest_float("x", 0, 1) def objective_shrink(trial: optuna.trial.Trial) -> float: if trial.number != 5: x = trial.suggest_float("x", 0, 1) y = trial.suggest_float("y", 0, 1) z = trial.suggest_float("z", 0, 1) return x + y + z else: x = trial.suggest_float("x", 0, 1) y = trial.suggest_float("y", 0, 1) return x + y def objective_expand(trial: optuna.trial.Trial) -> float: if trial.number != 5: x = trial.suggest_float("x", 0, 1) y = trial.suggest_float("y", 0, 1) return x + y else: x = trial.suggest_float("x", 0, 1) y = trial.suggest_float("y", 0, 1) z = trial.suggest_float("z", 0, 1) return x + y + z for objective in [objective_single, objective_shrink, objective_expand]: # We need to reconstruct our default handler to properly capture stderr. optuna.logging._reset_library_root_logger() optuna.logging.enable_default_handler() optuna.logging.set_verbosity(optuna.logging.WARNING) sampler = optuna.samplers.CmaEsSampler(warn_independent_sampling=warn_independent_sampling) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=10) _, err = capsys.readouterr() assert (err != "") == warn_independent_sampling @pytest.mark.filterwarnings("ignore::optuna.exceptions.ExperimentalWarning") @pytest.mark.parametrize("with_margin", [False, True]) @pytest.mark.parametrize("storage_name", ["sqlite", "journal"]) def test_rdb_storage(with_margin: bool, storage_name: str) -> None: # Confirm `study._storage.set_trial_system_attr` does not fail in several storages. def objective(trial: optuna.Trial) -> float: x = trial.suggest_float("x", -10, 10) y = trial.suggest_int("y", -10, 10) return x**2 + y with StorageSupplier(storage_name) as storage: study = optuna.create_study( sampler=optuna.samplers.CmaEsSampler(with_margin=with_margin), storage=storage, ) study.optimize(objective, n_trials=3) optuna-4.1.0/tests/samplers_tests/test_gp.py000066400000000000000000000033041471332314300212540ustar00rootroot00000000000000from __future__ import annotations from _pytest.logging import LogCaptureFixture import numpy as np import optuna import optuna._gp.acqf as acqf import optuna._gp.optim_mixed as optim_mixed import optuna._gp.prior as prior import optuna._gp.search_space as gp_search_space def test_after_convergence(caplog: LogCaptureFixture) -> None: # A large `optimal_trials` causes the instability in the kernel inversion, leading to # instability in the variance calculation. X_uniform = [(i + 1) / 10 for i in range(10)] X_uniform_near_optimal = [(i + 1) / 1e5 for i in range(20)] X_optimal = [0.0] * 10 X = np.array(X_uniform + X_uniform_near_optimal + X_optimal) score_vals = -(X - np.mean(X)) / np.std(X) search_space = gp_search_space.SearchSpace( scale_types=np.array([gp_search_space.ScaleType.LINEAR]), bounds=np.array([[0.0, 1.0]]), steps=np.zeros(1, dtype=float), ) kernel_params = optuna._gp.gp.fit_kernel_params( X=X[:, np.newaxis], Y=score_vals, is_categorical=np.array([False]), log_prior=prior.default_log_prior, minimum_noise=prior.DEFAULT_MINIMUM_NOISE_VAR, deterministic_objective=False, ) acqf_params = acqf.create_acqf_params( acqf_type=acqf.AcquisitionFunctionType.LOG_EI, kernel_params=kernel_params, search_space=search_space, X=X[:, np.newaxis], Y=score_vals, ) caplog.clear() optuna.logging.enable_propagation() optim_mixed.optimize_acqf_mixed(acqf_params, rng=np.random.RandomState(42)) # len(caplog.text) > 0 means the optimization has already converged. assert len(caplog.text) > 0, "Did you change the kernel implementation?" optuna-4.1.0/tests/samplers_tests/test_grid.py000066400000000000000000000226231471332314300216000ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Mapping from collections.abc import Sequence import itertools from typing import ValuesView import numpy as np import pytest import optuna from optuna import samplers from optuna.samplers._grid import GridValueType from optuna.storages import RetryFailedTrialCallback from optuna.testing.objectives import fail_objective from optuna.testing.objectives import pruned_objective from optuna.testing.storages import StorageSupplier from optuna.trial import Trial def test_study_optimize_with_single_search_space() -> None: def objective(trial: Trial) -> float: a = trial.suggest_int("a", 0, 100) b = trial.suggest_float("b", -0.1, 0.1) c = trial.suggest_categorical("c", ("x", "y", None, 1, 2.0)) d = trial.suggest_float("d", -5, 5, step=1) e = trial.suggest_float("e", 0.0001, 1, log=True) if c == "x": return a * d else: return b * e # Test that all combinations of the grid is sampled. search_space = { "b": np.arange(-0.1, 0.1, 0.05), "c": ("x", "y", None, 1, 2.0), "d": [-5.0, 5.0], "e": [0.1], "a": list(range(0, 100, 20)), } study = optuna.create_study(sampler=samplers.GridSampler(search_space)) # type: ignore study.optimize(objective) def sorted_values( d: Mapping[str, Sequence[GridValueType]] ) -> ValuesView[Sequence[GridValueType]]: return dict(sorted(d.items())).values() all_grids = itertools.product(*sorted_values(search_space)) # type: ignore all_suggested_values = [tuple([p for p in sorted_values(t.params)]) for t in study.trials] assert set(all_grids) == set(all_suggested_values) # Test a non-existing parameter name in the grid. search_space = {"a": list(range(0, 100, 20))} study = optuna.create_study(sampler=samplers.GridSampler(search_space)) # type: ignore with pytest.raises(ValueError): study.optimize(objective) # Test a value with out of range. search_space = { "a": [110], # 110 is out of range specified by the suggest method. "b": [0], "c": ["x"], "d": [0], "e": [0.1], } study = optuna.create_study(sampler=samplers.GridSampler(search_space)) # type: ignore with pytest.warns(UserWarning): study.optimize(objective) def test_study_optimize_with_exceeding_number_of_trials() -> None: def objective(trial: Trial) -> float: return trial.suggest_int("a", 0, 100) # When `n_trials` is `None`, the optimization stops just after all grids are evaluated. search_space: dict[str, list[GridValueType]] = {"a": [0, 50]} study = optuna.create_study(sampler=samplers.GridSampler(search_space)) study.optimize(objective, n_trials=None) assert len(study.trials) == 2 # If the optimization is triggered after all grids are evaluated, an additional trial runs. study.optimize(objective, n_trials=None) assert len(study.trials) == 3 def test_study_optimize_with_pruning() -> None: # Pruned trials should count towards grid consumption. search_space: dict[str, list[GridValueType]] = {"a": [0, 50]} study = optuna.create_study(sampler=samplers.GridSampler(search_space)) study.optimize(pruned_objective, n_trials=None) assert len(study.trials) == 2 def test_study_optimize_with_fail() -> None: def objective(trial: Trial) -> float: return trial.suggest_int("a", 0, 100) # Failed trials should count towards grid consumption. search_space: dict[str, list[GridValueType]] = {"a": [0, 50]} study = optuna.create_study(sampler=samplers.GridSampler(search_space)) study.optimize(fail_objective, n_trials=1, catch=ValueError) study.optimize(objective, n_trials=None) assert len(study.trials) == 2 def test_study_optimize_with_numpy_related_search_space() -> None: def objective(trial: Trial) -> float: a = trial.suggest_float("a", 0, 10) b = trial.suggest_float("b", -0.1, 0.1) return a + b # Test that all combinations of the grid is sampled. search_space = { "a": np.linspace(0, 10, 11), "b": np.arange(-0.1, 0.1, 0.05), } with StorageSupplier("sqlite") as storage: study = optuna.create_study( sampler=samplers.GridSampler(search_space), # type: ignore storage=storage, ) study.optimize(objective, n_trials=None) def test_study_optimize_with_multiple_search_spaces() -> None: def objective(trial: Trial) -> float: a = trial.suggest_int("a", 0, 100) b = trial.suggest_float("b", -100, 100) return a * b # Run 3 trials with a search space. search_space_0 = {"a": [0, 50], "b": [-50, 0, 50]} sampler_0 = samplers.GridSampler(search_space_0) study = optuna.create_study(sampler=sampler_0) study.optimize(objective, n_trials=3) assert len(study.trials) == 3 for t in study.trials: assert sampler_0._same_search_space(t.system_attrs["search_space"]) # Run 2 trials with another space. search_space_1 = {"a": [0, 25], "b": [-50]} sampler_1 = samplers.GridSampler(search_space_1) study.sampler = sampler_1 study.optimize(objective, n_trials=2) assert not sampler_0._same_search_space(sampler_1._search_space) assert len(study.trials) == 5 for t in study.trials[:3]: assert sampler_0._same_search_space(t.system_attrs["search_space"]) for t in study.trials[3:5]: assert sampler_1._same_search_space(t.system_attrs["search_space"]) # Run 3 trials with the first search space again. study.sampler = sampler_0 study.optimize(objective, n_trials=3) assert len(study.trials) == 8 for t in study.trials[:3]: assert sampler_0._same_search_space(t.system_attrs["search_space"]) for t in study.trials[3:5]: assert sampler_1._same_search_space(t.system_attrs["search_space"]) for t in study.trials[5:]: assert sampler_0._same_search_space(t.system_attrs["search_space"]) def test_cast_value() -> None: samplers.GridSampler._check_value("x", None) samplers.GridSampler._check_value("x", True) samplers.GridSampler._check_value("x", False) samplers.GridSampler._check_value("x", -1) samplers.GridSampler._check_value("x", -1.5) samplers.GridSampler._check_value("x", float("nan")) samplers.GridSampler._check_value("x", "foo") samplers.GridSampler._check_value("x", "") with pytest.warns(UserWarning): samplers.GridSampler._check_value("x", [1]) def test_has_same_search_space() -> None: search_space: dict[str, list[int | str]] = {"x": [3, 2, 1], "y": ["a", "b", "c"]} sampler = samplers.GridSampler(search_space) assert sampler._same_search_space(search_space) assert sampler._same_search_space({"x": [3, 2, 1], "y": ["a", "b", "c"]}) assert not sampler._same_search_space({"y": ["c", "a", "b"], "x": [1, 2, 3]}) assert not sampler._same_search_space({"x": [3, 2, 1, 0], "y": ["a", "b", "c"]}) assert not sampler._same_search_space({"x": [3, 2], "y": ["a", "b", "c"]}) def test_retried_trial() -> None: sampler = samplers.GridSampler({"a": [0, 50]}) study = optuna.create_study(sampler=sampler) trial = study.ask() trial.suggest_int("a", 0, 100) callback = RetryFailedTrialCallback() callback(study, study.trials[0]) study.optimize(lambda trial: trial.suggest_int("a", 0, 100)) assert len(study.trials) == 3 assert study.trials[0].params["a"] == study.trials[1].params["a"] assert study.trials[0].system_attrs["grid_id"] == study.trials[1].system_attrs["grid_id"] def test_enqueued_trial() -> None: sampler = samplers.GridSampler({"a": [0, 50]}) study = optuna.create_study(sampler=sampler) study.enqueue_trial({"a": 100}) study.optimize(lambda trial: trial.suggest_int("a", 0, 100)) assert len(study.trials) == 3 assert study.trials[0].params["a"] == 100 assert sorted([study.trials[1].params["a"], study.trials[2].params["a"]]) == [0, 50] def test_same_seed_trials() -> None: grid_values = [0, 20, 40, 60, 80, 100] seed = 0 sampler1 = samplers.GridSampler({"a": grid_values}, seed) study1 = optuna.create_study(sampler=sampler1) study1.optimize(lambda trial: trial.suggest_int("a", 0, 100)) sampler2 = samplers.GridSampler({"a": grid_values}, seed) study2 = optuna.create_study(sampler=sampler2) study2.optimize(lambda trial: trial.suggest_int("a", 0, 100)) for i in range(len(grid_values)): assert study1.trials[i].params["a"] == study2.trials[i].params["a"] def test_enqueued_insufficient_trial() -> None: sampler = samplers.GridSampler({"a": [0, 50]}) study = optuna.create_study(sampler=sampler) study.enqueue_trial({}) with pytest.raises(ValueError): study.optimize(lambda trial: trial.suggest_int("a", 0, 100)) def test_nan() -> None: sampler = optuna.samplers.GridSampler({"x": [0, float("nan")]}) study = optuna.create_study(sampler=sampler) study.optimize( lambda trial: 1 if np.isnan(trial.suggest_categorical("x", [0, float("nan")])) else 0 ) assert len(study.get_trials()) == 2 def test_is_exhausted() -> None: search_space = {"a": [0, 50]} sampler = samplers.GridSampler(search_space) study = optuna.create_study(sampler=sampler) assert not sampler.is_exhausted(study) study.optimize(lambda trial: trial.suggest_categorical("a", [0, 50])) assert sampler.is_exhausted(study) optuna-4.1.0/tests/samplers_tests/test_lazy_random_state.py000066400000000000000000000003241471332314300243640ustar00rootroot00000000000000from optuna.samplers._lazy_random_state import LazyRandomState def test_lazy_state() -> None: state = LazyRandomState() assert state._rng is None state.rng.seed(1) assert state._rng is not None optuna-4.1.0/tests/samplers_tests/test_nsgaii.py000066400000000000000000001133031471332314300221210ustar00rootroot00000000000000from __future__ import annotations from collections import Counter from collections.abc import Callable from collections.abc import Sequence import itertools from typing import Any from unittest.mock import MagicMock from unittest.mock import Mock from unittest.mock import patch import warnings import numpy as np import pytest import optuna from optuna._transform import _SearchSpaceTransform from optuna.distributions import BaseDistribution from optuna.distributions import CategoricalDistribution from optuna.distributions import FloatDistribution from optuna.distributions import IntDistribution from optuna.samplers import BaseSampler from optuna.samplers import NSGAIISampler from optuna.samplers._base import _CONSTRAINTS_KEY from optuna.samplers._lazy_random_state import LazyRandomState from optuna.samplers.nsgaii import BaseCrossover from optuna.samplers.nsgaii import BLXAlphaCrossover from optuna.samplers.nsgaii import SBXCrossover from optuna.samplers.nsgaii import SPXCrossover from optuna.samplers.nsgaii import UNDXCrossover from optuna.samplers.nsgaii import UniformCrossover from optuna.samplers.nsgaii import VSBXCrossover from optuna.samplers.nsgaii._after_trial_strategy import NSGAIIAfterTrialStrategy from optuna.samplers.nsgaii._child_generation_strategy import NSGAIIChildGenerationStrategy from optuna.samplers.nsgaii._constraints_evaluation import _constrained_dominates from optuna.samplers.nsgaii._constraints_evaluation import _validate_constraints from optuna.samplers.nsgaii._crossover import _inlined_categorical_uniform_crossover from optuna.samplers.nsgaii._elite_population_selection_strategy import ( NSGAIIElitePopulationSelectionStrategy, ) from optuna.samplers.nsgaii._elite_population_selection_strategy import _calc_crowding_distance from optuna.samplers.nsgaii._elite_population_selection_strategy import _crowding_distance_sort from optuna.samplers.nsgaii._elite_population_selection_strategy import _rank_population from optuna.samplers.nsgaii._sampler import _GENERATION_KEY from optuna.study._multi_objective import _dominates from optuna.study._study_direction import StudyDirection from optuna.testing.trials import _create_frozen_trial from optuna.trial import FrozenTrial def _nan_equal(a: Any, b: Any) -> bool: if isinstance(a, float) and isinstance(b, float) and np.isnan(a) and np.isnan(b): return True return a == b def test_population_size() -> None: # Set `population_size` to 10. sampler = NSGAIISampler(population_size=10) study = optuna.create_study(directions=["minimize"], sampler=sampler) study.optimize(lambda t: [t.suggest_float("x", 0, 9)], n_trials=40) generations = Counter( [t.system_attrs[optuna.samplers.nsgaii._sampler._GENERATION_KEY] for t in study.trials] ) assert generations == {0: 10, 1: 10, 2: 10, 3: 10} # Set `population_size` to 2. sampler = NSGAIISampler(population_size=2) study = optuna.create_study(directions=["minimize"], sampler=sampler) study.optimize(lambda t: [t.suggest_float("x", 0, 9)], n_trials=40) generations = Counter( [t.system_attrs[optuna.samplers.nsgaii._sampler._GENERATION_KEY] for t in study.trials] ) assert generations == {i: 2 for i in range(20)} # Invalid population size. with pytest.raises(ValueError): # Less than 2. NSGAIISampler(population_size=1) with pytest.raises(ValueError): mock_crossover = MagicMock(spec=BaseCrossover) mock_crossover.configure_mock(n_parents=3) NSGAIISampler(population_size=2, crossover=mock_crossover) def test_mutation_prob() -> None: NSGAIISampler(mutation_prob=None) NSGAIISampler(mutation_prob=0.0) NSGAIISampler(mutation_prob=0.5) NSGAIISampler(mutation_prob=1.0) with pytest.raises(ValueError): NSGAIISampler(mutation_prob=-0.5) with pytest.raises(ValueError): NSGAIISampler(mutation_prob=1.1) def test_crossover_prob() -> None: NSGAIISampler(crossover_prob=0.0) NSGAIISampler(crossover_prob=0.5) NSGAIISampler(crossover_prob=1.0) with pytest.raises(ValueError): NSGAIISampler(crossover_prob=-0.5) with pytest.raises(ValueError): NSGAIISampler(crossover_prob=1.1) def test_swapping_prob() -> None: NSGAIISampler(swapping_prob=0.0) NSGAIISampler(swapping_prob=0.5) NSGAIISampler(swapping_prob=1.0) with pytest.raises(ValueError): NSGAIISampler(swapping_prob=-0.5) with pytest.raises(ValueError): NSGAIISampler(swapping_prob=1.1) with pytest.raises(ValueError): UniformCrossover(swapping_prob=-0.5) with pytest.raises(ValueError): UniformCrossover(swapping_prob=1.1) @pytest.mark.parametrize("choices", [[-1, 0, 1], [True, False]]) def test_crossover_casting(choices: list[Any]) -> None: str_choices = list(map(str, choices)) def objective(trial: optuna.Trial) -> Sequence[float]: cat_1 = trial.suggest_categorical("cat_1", choices) cat_2 = trial.suggest_categorical("cat_2", str_choices) assert isinstance(cat_1, type(choices[0])) assert isinstance(cat_2, type(str_choices[0])) return 1.0, 2.0 population_size = 10 sampler = NSGAIISampler(population_size=population_size) study = optuna.create_study(directions=["minimize"] * 2, sampler=sampler) study.optimize(objective, n_trials=population_size * 2) def test_constraints_func_none() -> None: n_trials = 4 n_objectives = 2 sampler = NSGAIISampler(population_size=2) study = optuna.create_study(directions=["minimize"] * n_objectives, sampler=sampler) study.optimize( lambda t: [t.suggest_float(f"x{i}", 0, 1) for i in range(n_objectives)], n_trials=n_trials ) assert len(study.trials) == n_trials for trial in study.trials: assert _CONSTRAINTS_KEY not in trial.system_attrs @pytest.mark.parametrize("constraint_value", [-1.0, 0.0, 1.0, -float("inf"), float("inf")]) def test_constraints_func(constraint_value: float) -> None: n_trials = 4 n_objectives = 2 constraints_func_call_count = 0 def constraints_func(trial: FrozenTrial) -> Sequence[float]: nonlocal constraints_func_call_count constraints_func_call_count += 1 return (constraint_value + trial.number,) with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = NSGAIISampler(population_size=2, constraints_func=constraints_func) study = optuna.create_study(directions=["minimize"] * n_objectives, sampler=sampler) study.optimize( lambda t: [t.suggest_float(f"x{i}", 0, 1) for i in range(n_objectives)], n_trials=n_trials ) assert len(study.trials) == n_trials assert constraints_func_call_count == n_trials for trial in study.trials: for x, y in zip(trial.system_attrs[_CONSTRAINTS_KEY], (constraint_value + trial.number,)): assert x == y def test_constraints_func_nan() -> None: n_trials = 4 n_objectives = 2 def constraints_func(_: FrozenTrial) -> Sequence[float]: return (float("nan"),) with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = NSGAIISampler(population_size=2, constraints_func=constraints_func) study = optuna.create_study(directions=["minimize"] * n_objectives, sampler=sampler) with pytest.raises(ValueError): study.optimize( lambda t: [t.suggest_float(f"x{i}", 0, 1) for i in range(n_objectives)], n_trials=n_trials, ) trials = study.get_trials() assert len(trials) == 1 # The error stops optimization, but completed trials are recorded. assert all(0 <= x <= 1 for x in trials[0].params.values()) # The params are normal. assert trials[0].values == list(trials[0].params.values()) # The values are normal. assert trials[0].system_attrs[_CONSTRAINTS_KEY] is None # None is set for constraints. @pytest.mark.parametrize("direction1", [StudyDirection.MINIMIZE, StudyDirection.MAXIMIZE]) @pytest.mark.parametrize("direction2", [StudyDirection.MINIMIZE, StudyDirection.MAXIMIZE]) @pytest.mark.parametrize( "constraints_list", [ [[]], # empty constraint [[-float("inf")], [-1], [0]], # single constraint [ [c1, c2] for c1 in [-float("inf"), -1, 0] for c2 in [-float("inf"), -1, 0] ], # multiple constraints ], ) def test_constrained_dominates_feasible_vs_feasible( direction1: StudyDirection, direction2: StudyDirection, constraints_list: list[list[float]] ) -> None: directions = [direction1, direction2] # Check all pairs of trials consisting of these values, i.e., # [-inf, -inf], [-inf, -1], [-inf, 1], [-inf, inf], [-1, -inf], ... values_list = [ [x, y] for x in [-float("inf"), -1, 1, float("inf")] for y in [-float("inf"), -1, 1, float("inf")] ] values_constraints_list = [(vs, cs) for vs in values_list for cs in constraints_list] # The results of _constrained_dominates match _dominates in all feasible cases. for values1, constraints1 in values_constraints_list: for values2, constraints2 in values_constraints_list: t1 = _create_frozen_trial(number=0, values=values1, constraints=constraints1) t2 = _create_frozen_trial(number=1, values=values2, constraints=constraints2) assert _constrained_dominates(t1, t2, directions) == _dominates(t1, t2, directions) @pytest.mark.parametrize("direction", [StudyDirection.MINIMIZE, StudyDirection.MAXIMIZE]) def test_constrained_dominates_feasible_vs_infeasible( direction: StudyDirection, ) -> None: # Check all pairs of trials consisting of these constraint values. constraints_1d_feasible = [-float("inf"), -1, 0] constraints_1d_infeasible = [2, float("inf")] directions = [direction] # Feasible constraints. constraints_list1 = [ [c1, c2] for c1 in constraints_1d_feasible for c2 in constraints_1d_feasible ] # Infeasible constraints. constraints_list2 = [ [c1, c2] for c1 in constraints_1d_feasible + constraints_1d_infeasible for c2 in constraints_1d_infeasible ] # In the following code, we test that the feasible trials always dominate # the infeasible trials. for constraints1 in constraints_list1: for constraints2 in constraints_list2: t1 = _create_frozen_trial(number=0, values=[0], constraints=constraints1) t2 = _create_frozen_trial(number=1, values=[1], constraints=constraints2) assert _constrained_dominates(t1, t2, directions) assert not _constrained_dominates(t2, t1, directions) t1 = _create_frozen_trial(number=0, values=[1], constraints=constraints1) t2 = _create_frozen_trial(number=1, values=[0], constraints=constraints2) assert _constrained_dominates(t1, t2, directions) assert not _constrained_dominates(t2, t1, directions) @pytest.mark.parametrize("direction", [StudyDirection.MINIMIZE, StudyDirection.MAXIMIZE]) def test_constrained_dominates_infeasible_vs_infeasible(direction: StudyDirection) -> None: inf = float("inf") directions = [direction] # The following table illustrates the violations of some constraint values. # When both trials are infeasible, the trial with smaller violation dominates # the one with larger violation. # # c2 # ╔═════╤═════╤═════╤═════╤═════╤═════╗ # ║ │ -1 │ 0 │ 1 │ 2 │ ∞ ║ # ╟─────┼─────┼─────┼─────┼─────┼─────╢ # ║ -1 │ │ 1 │ 2 │ ∞ ║ # ╟─────┼─feasible ─┼─────┼─────┼─────╢ # ║ 0 │ │ 1 │ 2 │ ∞ ║ # c1 ╟─────┼─────┼─────┼─────┼─────┼─────╢ # ║ 1 │ 1 │ 1 │ 2 │ 3 │ ∞ ║ # ╟─────┼─────┼─────┼─────┼─────┼─────╢ # ║ 2 │ 2 │ 2 │ 3 │ 4 │ ∞ ║ # ╟─────┼─────┼─────┼─────┼─────┼─────╢ # ║ ∞ │ ∞ │ ∞ │ ∞ │ ∞ │ ∞ ║ # ╚═════╧═════╧═════╧═════╧═════╧═════╝ # # Check all pairs of these constraints. constraints_infeasible_sorted: list[list[list[float]]] constraints_infeasible_sorted = [ # These constraints have violation 1. [[1, -inf], [1, -1], [1, 0], [0, 1], [-1, 1], [-inf, 1]], # These constraints have violation 2. [[2, -inf], [2, -1], [2, 0], [1, 1], [0, 2], [-1, 2], [-inf, 2]], # These constraints have violation 3. [[3, -inf], [3, -1], [3, 0], [2, 1], [1, 2], [0, 3], [-1, 3], [-inf, 3]], # These constraints have violation inf. [ [-inf, inf], [-1, inf], [0, inf], [1, inf], [inf, inf], [inf, 1], [inf, 0], [inf, -1], [inf, -inf], ], ] # Check that constraints with smaller violations dominate constraints with larger violation. for i in range(len(constraints_infeasible_sorted)): for j in range(i + 1, len(constraints_infeasible_sorted)): # Every constraint in constraints_infeasible_sorted[i] dominates # every constraint in constraints_infeasible_sorted[j]. for constraints1 in constraints_infeasible_sorted[i]: for constraints2 in constraints_infeasible_sorted[j]: t1 = _create_frozen_trial(number=0, values=[0], constraints=constraints1) t2 = _create_frozen_trial(number=1, values=[1], constraints=constraints2) assert _constrained_dominates(t1, t2, directions) assert not _constrained_dominates(t2, t1, directions) t1 = _create_frozen_trial(number=0, values=[1], constraints=constraints1) t2 = _create_frozen_trial(number=1, values=[0], constraints=constraints2) assert _constrained_dominates(t1, t2, directions) assert not _constrained_dominates(t2, t1, directions) # Check that constraints with same violations are incomparable. for constraints_with_same_violations in constraints_infeasible_sorted: for constraints1 in constraints_with_same_violations: for constraints2 in constraints_with_same_violations: t1 = _create_frozen_trial(number=0, values=[0], constraints=constraints1) t2 = _create_frozen_trial(number=1, values=[1], constraints=constraints2) assert not _constrained_dominates(t1, t2, directions) assert not _constrained_dominates(t2, t1, directions) def _assert_population_per_rank( trials: list[FrozenTrial], direction: list[StudyDirection], population_per_rank: list[list[FrozenTrial]], ) -> None: # Check that the number of trials do not change. flattened = [trial for rank in population_per_rank for trial in rank] assert len(flattened) == len(trials) with warnings.catch_warnings(): warnings.simplefilter("ignore", UserWarning) # Check that the trials in the same rank do not dominate each other. for i in range(len(population_per_rank)): for trial1 in population_per_rank[i]: for trial2 in population_per_rank[i]: assert not _constrained_dominates(trial1, trial2, direction) # Check that each trial is dominated by some trial in the rank above. for i in range(len(population_per_rank) - 1): for trial2 in population_per_rank[i + 1]: assert any( _constrained_dominates(trial1, trial2, direction) for trial1 in population_per_rank[i] ) @pytest.mark.parametrize("direction1", [StudyDirection.MINIMIZE, StudyDirection.MAXIMIZE]) @pytest.mark.parametrize("direction2", [StudyDirection.MINIMIZE, StudyDirection.MAXIMIZE]) def test_rank_population_no_constraints( direction1: StudyDirection, direction2: StudyDirection ) -> None: directions = [direction1, direction2] value_list = [10, 20, 20, 30, float("inf"), float("inf"), -float("inf")] values = [[v1, v2] for v1 in value_list for v2 in value_list] trials = [_create_frozen_trial(number=i, values=v) for i, v in enumerate(values)] population_per_rank = _rank_population(trials, directions) _assert_population_per_rank(trials, directions, population_per_rank) def test_rank_population_with_constraints() -> None: value_list = [10, 20, 20, 30, float("inf"), float("inf"), -float("inf")] values = [[v1, v2] for v1 in value_list for v2 in value_list] constraint_list = [-float("inf"), -2, 0, 1, 2, 3, float("inf")] constraints = [[c1, c2] for c1 in constraint_list for c2 in constraint_list] trials = [ _create_frozen_trial(number=i, values=v, constraints=c) for i, (v, c) in enumerate(itertools.product(values, constraints)) ] directions = [StudyDirection.MINIMIZE, StudyDirection.MAXIMIZE] population_per_rank = _rank_population(trials, directions, is_constrained=True) _assert_population_per_rank(trials, directions, population_per_rank) def test_validate_constraints() -> None: # Nan is not allowed in constraints. with pytest.raises(ValueError): _validate_constraints( [_create_frozen_trial(number=0, values=[1], constraints=[0, float("nan")])], is_constrained=True, ) # Different numbers of constraints are not allowed. with pytest.raises(ValueError): _validate_constraints( [ _create_frozen_trial(number=0, values=[1], constraints=[0]), _create_frozen_trial(number=1, values=[1], constraints=[0, 1]), ], is_constrained=True, ) @pytest.mark.parametrize( "values_and_constraints", [ [([10], None), ([20], None), ([20], [0]), ([20], [1]), ([30], [-1])], [ ([50, 30], None), ([30, 50], None), ([20, 20], [3, 3]), ([30, 10], [0, -1]), ([15, 15], [4, 4]), ], ], ) def test_rank_population_missing_constraint_values( values_and_constraints: list[tuple[list[float], list[float]]] ) -> None: values_dim = len(values_and_constraints[0][0]) for directions in itertools.product( [StudyDirection.MINIMIZE, StudyDirection.MAXIMIZE], repeat=values_dim ): trials = [ _create_frozen_trial(number=i, values=v, constraints=c) for i, (v, c) in enumerate(values_and_constraints) ] with pytest.warns(UserWarning): _validate_constraints(trials, is_constrained=True) population_per_rank = _rank_population(trials, list(directions), is_constrained=True) _assert_population_per_rank(trials, list(directions), population_per_rank) @pytest.mark.parametrize("n_dims", [1, 2, 3]) def test_rank_population_empty(n_dims: int) -> None: for directions in itertools.product( [StudyDirection.MINIMIZE, StudyDirection.MAXIMIZE], repeat=n_dims ): trials: list[FrozenTrial] = [] population_per_rank = _rank_population(trials, list(directions)) assert population_per_rank == [] @pytest.mark.parametrize( "values, expected_dist", [ ([[5], [6], [9], [0]], [6 / 9, 4 / 9, float("inf"), float("inf")]), ([[5, 0], [6, 0], [9, 0], [0, 0]], [6 / 9, 4 / 9, float("inf"), float("inf")]), ( [[5, -1], [6, 0], [9, 1], [0, 2]], [float("inf"), 4 / 9 + 2 / 3, float("inf"), float("inf")], ), ([[5]], [0]), ([[5], [5]], [0, 0]), ( [[1], [2], [float("inf")]], [float("inf"), float("inf"), float("inf")], ), ( [[float("-inf")], [1], [2]], [float("inf"), float("inf"), float("inf")], ), ([[float("inf")], [float("inf")], [float("inf")]], [0, 0, 0]), ([[-float("inf")], [-float("inf")], [-float("inf")]], [0, 0, 0]), ([[-float("inf")], [float("inf")]], [float("inf"), float("inf")]), ( [[-float("inf")], [-float("inf")], [-float("inf")], [0], [1], [2], [float("inf")]], [0, 0, float("inf"), float("inf"), 1, float("inf"), float("inf")], ), ], ) def test_calc_crowding_distance(values: list[list[float]], expected_dist: list[float]) -> None: trials = [_create_frozen_trial(number=i, values=value) for i, value in enumerate(values)] crowding_dist = _calc_crowding_distance(trials) for i in range(len(trials)): assert _nan_equal(crowding_dist[i], expected_dist[i]), i @pytest.mark.parametrize( "values", [ [[5], [6], [9], [0]], [[5, 0], [6, 0], [9, 0], [0, 0]], [[5, -1], [6, 0], [9, 1], [0, 2]], [[1], [2], [float("inf")]], [[float("-inf")], [1], [2]], ], ) def test_crowding_distance_sort(values: list[list[float]]) -> None: """Checks that trials are sorted by the values of `_calc_crowding_distance`.""" trials = [_create_frozen_trial(number=i, values=value) for i, value in enumerate(values)] crowding_dist = _calc_crowding_distance(trials) _crowding_distance_sort(trials) sorted_dist = [crowding_dist[t.number] for t in trials] assert sorted_dist == sorted(sorted_dist, reverse=True) def test_study_system_attr_for_population_cache() -> None: sampler = NSGAIISampler(population_size=10) study = optuna.create_study(directions=["minimize"], sampler=sampler) def get_cached_entries( study: optuna.study.Study, ) -> list[tuple[int, list[int]]]: study_system_attrs = study._storage.get_study_system_attrs(study._study_id) return [ v for k, v in study_system_attrs.items() if k.startswith(optuna.samplers.nsgaii._sampler._POPULATION_CACHE_KEY_PREFIX) ] study.optimize(lambda t: [t.suggest_float("x", 0, 9)], n_trials=10) cached_entries = get_cached_entries(study) assert len(cached_entries) == 0 study.optimize(lambda t: [t.suggest_float("x", 0, 9)], n_trials=1) cached_entries = get_cached_entries(study) assert len(cached_entries) == 1 assert cached_entries[0][0] == 0 # Cached generation. assert len(cached_entries[0][1]) == 10 # Population size. study.optimize(lambda t: [t.suggest_float("x", 0, 9)], n_trials=10) cached_entries = get_cached_entries(study) assert len(cached_entries) == 1 assert cached_entries[0][0] == 1 # Cached generation. assert len(cached_entries[0][1]) == 10 # Population size. def test_constraints_func_experimental_warning() -> None: with pytest.warns(optuna.exceptions.ExperimentalWarning): NSGAIISampler(constraints_func=lambda _: [0]) def test_elite_population_selection_strategy_experimental_warning() -> None: with pytest.warns(optuna.exceptions.ExperimentalWarning): NSGAIISampler(elite_population_selection_strategy=lambda study, population: []) def test_child_generation_strategy_experimental_warning() -> None: with pytest.warns(optuna.exceptions.ExperimentalWarning): NSGAIISampler(child_generation_strategy=lambda study, search_space, parent_population: {}) def test_after_trial_strategy_experimental_warning() -> None: with pytest.warns(optuna.exceptions.ExperimentalWarning): NSGAIISampler(after_trial_strategy=lambda study, trial, state, value: None) def test_elite_population_selection_strategy_invalid_value() -> None: with pytest.raises(ValueError): NSGAIIElitePopulationSelectionStrategy(population_size=1) @pytest.mark.parametrize( "objectives, expected_elite_population", [ ( [[1.0, 4.0], [2.0, 3.0], [3.0, 2.0], [4.0, 1.0]], [[1.0, 4.0], [2.0, 3.0], [3.0, 2.0], [4.0, 1.0]], ), ( [[1.0, 2.0], [2.0, 1.0], [3.0, 3.0], [4.0, 4.0]], [[1.0, 2.0], [2.0, 1.0], [3.0, 3.0], [4.0, 4.0]], ), ( [[1.0, 2.0], [2.0, 1.0], [5.0, 3.0], [3.0, 5.0], [4.0, 4.0]], [[1.0, 2.0], [2.0, 1.0], [5.0, 3.0], [3.0, 5.0]], ), ], ) def test_elite_population_selection_strategy_result( objectives: list[list[float]], expected_elite_population: list[list[float]], ) -> None: population_size = 4 elite_population_selection_strategy = NSGAIIElitePopulationSelectionStrategy( population_size=population_size ) study = optuna.create_study(directions=["minimize", "minimize"]) study.add_trials([optuna.create_trial(values=values) for values in objectives]) elite_population_values = [ trial.values for trial in elite_population_selection_strategy(study, study.get_trials()) ] assert len(elite_population_values) == population_size for values in elite_population_values: assert values in expected_elite_population @pytest.mark.parametrize( "mutation_prob,crossover,crossover_prob,swapping_prob", [ (1.2, UniformCrossover(), 0.9, 0.5), (-0.2, UniformCrossover(), 0.9, 0.5), (None, UniformCrossover(), 1.2, 0.5), (None, UniformCrossover(), -0.2, 0.5), (None, UniformCrossover(), 0.9, 1.2), (None, UniformCrossover(), 0.9, -0.2), (None, 3, 0.9, 0.5), ], ) def test_child_generation_strategy_invalid_value( mutation_prob: float, crossover: BaseCrossover | int, crossover_prob: float, swapping_prob: float, ) -> None: with pytest.raises(ValueError): NSGAIIChildGenerationStrategy( mutation_prob=mutation_prob, crossover=crossover, # type: ignore[arg-type] crossover_prob=crossover_prob, swapping_prob=swapping_prob, rng=LazyRandomState(), ) @pytest.mark.parametrize( "mutation_prob,child_params", [(0.0, {"x": 1.0, "y": 0.0}), (1.0, {})], ) def test_child_generation_strategy_mutation_prob( mutation_prob: int, child_params: dict[str, float] ) -> None: child_generation_strategy = NSGAIIChildGenerationStrategy( crossover_prob=0.0, crossover=UniformCrossover(), mutation_prob=mutation_prob, swapping_prob=0.5, rng=LazyRandomState(seed=1), ) study = MagicMock(spec=optuna.study.Study) search_space = MagicMock(spec=dict) search_space.keys.return_value = ["x", "y"] parent_population = [ optuna.trial.create_trial( params={"x": 1.0, "y": 0}, distributions={ "x": FloatDistribution(0, 10), "y": CategoricalDistribution([-1, 0, 1]), }, value=5.0, ) ] assert child_generation_strategy(study, search_space, parent_population) == child_params def test_child_generation_strategy_generation_key() -> None: n_params = 2 def objective(trial: optuna.Trial) -> list[float]: xs = [trial.suggest_float(f"x{dim}", -10, 10) for dim in range(n_params)] return xs mock_func = MagicMock(spec=Callable, return_value={"x0": 0.0, "x1": 1.1}) with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) study = optuna.create_study( sampler=NSGAIISampler(population_size=2, child_generation_strategy=mock_func), directions=["minimize", "minimize"], ) study.optimize(objective, n_trials=3) assert mock_func.call_count == 1 for i, trial in enumerate(study.get_trials()): if i < 2: assert trial.system_attrs[_GENERATION_KEY] == 0 elif i == 2: assert trial.system_attrs[_GENERATION_KEY] == 1 @patch( "optuna.samplers.nsgaii._child_generation_strategy.perform_crossover", return_value={"x": 3.0, "y": 2.0}, ) def test_child_generation_strategy_crossover_prob(mock_func: MagicMock) -> None: study = MagicMock(spec=optuna.study.Study) search_space = MagicMock(spec=dict) search_space.keys.return_value = ["x", "y"] parent_population = [ optuna.trial.create_trial( params={"x": 1.0, "y": 0}, distributions={ "x": FloatDistribution(0, 10), "y": CategoricalDistribution([-1, 0, 1]), }, value=5.0, ) ] child_generation_strategy_always_not_crossover = NSGAIIChildGenerationStrategy( crossover_prob=0.0, crossover=UniformCrossover(), mutation_prob=None, swapping_prob=0.5, rng=LazyRandomState(seed=1), ) assert child_generation_strategy_always_not_crossover( study, search_space, parent_population ) == {"x": 1.0} assert mock_func.call_count == 0 child_generation_strategy_always_crossover = NSGAIIChildGenerationStrategy( crossover_prob=1.0, crossover=UniformCrossover(), mutation_prob=0.0, swapping_prob=0.5, rng=LazyRandomState(), ) assert child_generation_strategy_always_crossover(study, search_space, parent_population) == { "x": 3.0, "y": 2.0, } assert mock_func.call_count == 1 def test_call_after_trial_of_random_sampler() -> None: sampler = NSGAIISampler() study = optuna.create_study(sampler=sampler) with patch.object( sampler._random_sampler, "after_trial", wraps=sampler._random_sampler.after_trial ) as mock_object: study.optimize(lambda _: 1.0, n_trials=1) assert mock_object.call_count == 1 def test_call_after_trial_of_after_trial_strategy() -> None: sampler = NSGAIISampler() study = optuna.create_study(sampler=sampler) with patch.object(sampler, "_after_trial_strategy") as mock_object: study.optimize(lambda _: 1.0, n_trials=1) assert mock_object.call_count == 1 @patch("optuna.samplers.nsgaii._after_trial_strategy._process_constraints_after_trial") def test_nsgaii_after_trial_strategy(mock_func: MagicMock) -> None: def constraints_func(_: FrozenTrial) -> Sequence[float]: return (float("nan"),) state = optuna.trial.TrialState.FAIL study = optuna.create_study() trial = optuna.trial.create_trial(state=state) after_trial_strategy_without_constrains = NSGAIIAfterTrialStrategy() after_trial_strategy_without_constrains(study, trial, state) assert mock_func.call_count == 0 after_trial_strategy_with_constrains = NSGAIIAfterTrialStrategy( constraints_func=constraints_func ) after_trial_strategy_with_constrains(study, trial, state) assert mock_func.call_count == 1 parametrize_nsga2_sampler = pytest.mark.parametrize( "sampler_class", [ lambda: NSGAIISampler(population_size=2, crossover=UniformCrossover()), lambda: NSGAIISampler(population_size=2, crossover=BLXAlphaCrossover()), lambda: NSGAIISampler(population_size=2, crossover=SBXCrossover()), lambda: NSGAIISampler(population_size=2, crossover=VSBXCrossover()), lambda: NSGAIISampler(population_size=3, crossover=UNDXCrossover()), lambda: NSGAIISampler(population_size=3, crossover=UNDXCrossover()), ], ) @parametrize_nsga2_sampler @pytest.mark.parametrize("n_objectives", [1, 2, 3]) def test_crossover_objectives(n_objectives: int, sampler_class: Callable[[], BaseSampler]) -> None: n_trials = 8 study = optuna.create_study(directions=["minimize"] * n_objectives, sampler=sampler_class()) study.optimize( lambda t: [t.suggest_float(f"x{i}", 0, 1) for i in range(n_objectives)], n_trials=n_trials ) assert len(study.trials) == n_trials @parametrize_nsga2_sampler @pytest.mark.parametrize("n_params", [1, 2, 3]) def test_crossover_dims(n_params: int, sampler_class: Callable[[], BaseSampler]) -> None: def objective(trial: optuna.Trial) -> float: xs = [trial.suggest_float(f"x{dim}", -10, 10) for dim in range(n_params)] return sum(xs) n_objectives = 1 n_trials = 8 study = optuna.create_study(directions=["minimize"] * n_objectives, sampler=sampler_class()) study.optimize(objective, n_trials=n_trials) assert len(study.trials) == n_trials @pytest.mark.parametrize( "crossover,population_size", [ (UniformCrossover(), 1), (BLXAlphaCrossover(), 1), (SBXCrossover(), 1), (VSBXCrossover(), 1), (UNDXCrossover(), 2), (SPXCrossover(), 2), ], ) def test_crossover_invalid_population(crossover: BaseCrossover, population_size: int) -> None: with pytest.raises(ValueError): NSGAIISampler(population_size=population_size, crossover=crossover) @pytest.mark.parametrize( "crossover", [ UniformCrossover(), BLXAlphaCrossover(), SPXCrossover(), SBXCrossover(), VSBXCrossover(), UNDXCrossover(), ], ) def test_crossover_numerical_distribution(crossover: BaseCrossover) -> None: study = optuna.study.create_study() rng = np.random.RandomState() search_space = {"x": FloatDistribution(1, 10), "y": IntDistribution(1, 10)} numerical_transform = _SearchSpaceTransform(search_space) parent_params = np.array([[1.0, 2], [3.0, 4]]) if crossover.n_parents == 3: parent_params = np.append(parent_params, [[5.0, 6]], axis=0) child_params = crossover.crossover(parent_params, rng, study, numerical_transform.bounds) assert child_params.ndim == 1 assert len(child_params) == len(search_space) assert not any(np.isnan(child_params)) assert not any(np.isinf(child_params)) def test_crossover_inlined_categorical_distribution() -> None: search_space: dict[str, BaseDistribution] = { "x": CategoricalDistribution(choices=["a", "c"]), "y": CategoricalDistribution(choices=["b", "d"]), } parent_params = np.array([["a", "b"], ["c", "d"]]) rng = np.random.RandomState() child_params = _inlined_categorical_uniform_crossover(parent_params, rng, 0.5, search_space) assert child_params.ndim == 1 assert len(child_params) == len(search_space) assert all([isinstance(param, str) for param in child_params]) # Is child param from correct distribution? search_space["x"].to_internal_repr(child_params[0]) search_space["y"].to_internal_repr(child_params[1]) @pytest.mark.parametrize( "crossover", [ UniformCrossover(), BLXAlphaCrossover(), SPXCrossover(), SBXCrossover(), VSBXCrossover(), UNDXCrossover(), ], ) def test_crossover_duplicated_param_values(crossover: BaseCrossover) -> None: param_values = [1.0, 2.0] study = optuna.study.create_study() rng = np.random.RandomState() search_space = {"x": FloatDistribution(1, 10), "y": IntDistribution(1, 10)} numerical_transform = _SearchSpaceTransform(search_space) parent_params = np.array([param_values, param_values]) if crossover.n_parents == 3: parent_params = np.append(parent_params, [param_values], axis=0) child_params = crossover.crossover(parent_params, rng, study, numerical_transform.bounds) assert child_params.ndim == 1 np.testing.assert_almost_equal(child_params, param_values) @pytest.mark.parametrize( "crossover,rand_value,expected_params", [ (UniformCrossover(), 0.0, np.array([1.0, 2.0])), # p1. (UniformCrossover(), 0.5, np.array([3.0, 4.0])), # p2. (UniformCrossover(), 1.0, np.array([3.0, 4.0])), # p2. (BLXAlphaCrossover(), 0.0, np.array([0.0, 1.0])), # p1 - [1, 1]. (BLXAlphaCrossover(), 0.5, np.array([2.0, 3.0])), # (p1 + p2) / 2. (BLXAlphaCrossover(), 1.0, np.array([4.0, 5.0])), # p2 + [1, 1]. # G = [3, 4], xks=[[-1, 0], [3, 4]. [7, 8]]. (SPXCrossover(), 0.0, np.array([7, 8])), # rs = [0, 0], xks[-1]. (SPXCrossover(), 0.5, np.array([2.75735931, 3.75735931])), # rs = [0.5, 0.25]. (SPXCrossover(), 1.0, np.array([-1.0, 0.0])), # rs = [1, 1], xks[0]. (SBXCrossover(), 0.0, np.array([2.0, 3.0])), # c1 = (p1 + p2) / 2. (SBXCrossover(), 0.5, np.array([3.0, 4.0])), # p2. (SBXCrossover(), 1.0, np.array([3.0, 4.0])), # p2. (VSBXCrossover(), 0.0, np.array([2.0, 3.0])), # c1 = (p1 + p2) / 2. (VSBXCrossover(), 0.5, np.array([3.0, 4.0])), # p2. (VSBXCrossover(), 1.0, np.array([3.0, 4.0])), # p2. # p1, p2 and p3 are on x + 1, and distance from child to PSL is 0. (UNDXCrossover(), -0.5, np.array([3.0, 4.0])), # [2, 3] + [-1, -1] + [0, 0]. (UNDXCrossover(), 0.0, np.array([2.0, 3.0])), # [2, 3] + [0, 0] + [0, 0]. (UNDXCrossover(), 0.5, np.array([1.0, 2.0])), # [2, 3] + [-1, -1] + [0, 0]. ], ) def test_crossover_deterministic( crossover: BaseCrossover, rand_value: float, expected_params: np.ndarray ) -> None: study = optuna.study.create_study() search_space: dict[str, BaseDistribution] = { "x": FloatDistribution(1, 10), "y": FloatDistribution(1, 10), } numerical_transform = _SearchSpaceTransform(search_space) parent_params = np.array([[1.0, 2.0], [3.0, 4.0]]) if crossover.n_parents == 3: parent_params = np.append(parent_params, [[5.0, 6.0]], axis=0) def _rand(*args: Any, **kwargs: Any) -> Any: if len(args) == 0: return rand_value return np.full(args[0], rand_value) def _normal(*args: Any, **kwargs: Any) -> Any: if kwargs.get("size") is None: return rand_value return np.full(kwargs.get("size"), rand_value) # type: ignore[arg-type] rng = Mock() rng.rand = Mock(side_effect=_rand) rng.normal = Mock(side_effect=_normal) child_params = crossover.crossover(parent_params, rng, study, numerical_transform.bounds) np.testing.assert_almost_equal(child_params, expected_params) optuna-4.1.0/tests/samplers_tests/test_nsgaiii.py000066400000000000000000000476361471332314300223110ustar00rootroot00000000000000from __future__ import annotations from collections import Counter from collections.abc import Sequence from unittest.mock import MagicMock from unittest.mock import patch import warnings import numpy as np import pytest import optuna from optuna.samplers import BaseSampler from optuna.samplers._base import _CONSTRAINTS_KEY from optuna.samplers._nsgaiii._elite_population_selection_strategy import ( _associate_individuals_with_reference_points, ) from optuna.samplers._nsgaiii._elite_population_selection_strategy import ( _generate_default_reference_point, ) from optuna.samplers._nsgaiii._elite_population_selection_strategy import ( _normalize_objective_values, ) from optuna.samplers._nsgaiii._elite_population_selection_strategy import ( _preserve_niche_individuals, ) from optuna.samplers._nsgaiii._elite_population_selection_strategy import _COEF from optuna.samplers._nsgaiii._elite_population_selection_strategy import _filter_inf from optuna.samplers._nsgaiii._sampler import _POPULATION_CACHE_KEY_PREFIX from optuna.samplers._nsgaiii._sampler import NSGAIIISampler from optuna.samplers.nsgaii import BaseCrossover from optuna.samplers.nsgaii import BLXAlphaCrossover from optuna.samplers.nsgaii import SBXCrossover from optuna.samplers.nsgaii import SPXCrossover from optuna.samplers.nsgaii import UNDXCrossover from optuna.samplers.nsgaii import UniformCrossover from optuna.samplers.nsgaii import VSBXCrossover from optuna.samplers.nsgaii._after_trial_strategy import NSGAIIAfterTrialStrategy from optuna.trial import create_trial from optuna.trial import FrozenTrial def test_population_size() -> None: # Set `population_size` to 10. sampler = NSGAIIISampler(population_size=10) study = optuna.create_study(directions=["minimize"], sampler=sampler) study.optimize(lambda t: [t.suggest_float("x", 0, 9)], n_trials=40) generations = Counter( [t.system_attrs[optuna.samplers._nsgaiii._sampler._GENERATION_KEY] for t in study.trials] ) assert generations == {0: 10, 1: 10, 2: 10, 3: 10} # Set `population_size` to 2. sampler = NSGAIIISampler(population_size=2) study = optuna.create_study(directions=["minimize"], sampler=sampler) study.optimize(lambda t: [t.suggest_float("x", 0, 9)], n_trials=40) generations = Counter( [t.system_attrs[optuna.samplers._nsgaiii._sampler._GENERATION_KEY] for t in study.trials] ) assert generations == {i: 2 for i in range(20)} # Invalid population size. with pytest.raises(ValueError): # Less than 2. NSGAIIISampler(population_size=1) with pytest.raises(ValueError): mock_crossover = MagicMock(spec=BaseCrossover) mock_crossover.configure_mock(n_parents=3) NSGAIIISampler(population_size=2, crossover=mock_crossover) def test_mutation_prob() -> None: NSGAIIISampler(mutation_prob=None) NSGAIIISampler(mutation_prob=0.0) NSGAIIISampler(mutation_prob=0.5) NSGAIIISampler(mutation_prob=1.0) with pytest.raises(ValueError): NSGAIIISampler(mutation_prob=-0.5) with pytest.raises(ValueError): NSGAIIISampler(mutation_prob=1.1) def test_crossover_prob() -> None: NSGAIIISampler(crossover_prob=0.0) NSGAIIISampler(crossover_prob=0.5) NSGAIIISampler(crossover_prob=1.0) with pytest.raises(ValueError): NSGAIIISampler(crossover_prob=-0.5) with pytest.raises(ValueError): NSGAIIISampler(crossover_prob=1.1) def test_swapping_prob() -> None: NSGAIIISampler(swapping_prob=0.0) NSGAIIISampler(swapping_prob=0.5) NSGAIIISampler(swapping_prob=1.0) with pytest.raises(ValueError): NSGAIIISampler(swapping_prob=-0.5) with pytest.raises(ValueError): NSGAIIISampler(swapping_prob=1.1) def test_constraints_func_none() -> None: n_trials = 4 n_objectives = 2 sampler = NSGAIIISampler(population_size=2) study = optuna.create_study(directions=["minimize"] * n_objectives, sampler=sampler) study.optimize( lambda t: [t.suggest_float(f"x{i}", 0, 1) for i in range(n_objectives)], n_trials=n_trials, ) assert len(study.trials) == n_trials for trial in study.trials: assert _CONSTRAINTS_KEY not in trial.system_attrs @pytest.mark.parametrize("constraint_value", [-1.0, 0.0, 1.0, -float("inf"), float("inf")]) def test_constraints_func(constraint_value: float) -> None: n_trials = 4 n_objectives = 2 constraints_func_call_count = 0 def constraints_func(trial: FrozenTrial) -> Sequence[float]: nonlocal constraints_func_call_count constraints_func_call_count += 1 return (constraint_value + trial.number,) with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = NSGAIIISampler(population_size=2, constraints_func=constraints_func) study = optuna.create_study(directions=["minimize"] * n_objectives, sampler=sampler) study.optimize( lambda t: [t.suggest_float(f"x{i}", 0, 1) for i in range(n_objectives)], n_trials=n_trials, ) assert len(study.trials) == n_trials assert constraints_func_call_count == n_trials for trial in study.trials: for x, y in zip(trial.system_attrs[_CONSTRAINTS_KEY], (constraint_value + trial.number,)): assert x == y def test_constraints_func_nan() -> None: n_trials = 4 n_objectives = 2 def constraints_func(_: FrozenTrial) -> Sequence[float]: return (float("nan"),) with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = NSGAIIISampler(population_size=2, constraints_func=constraints_func) study = optuna.create_study(directions=["minimize"] * n_objectives, sampler=sampler) with pytest.raises(ValueError): study.optimize( lambda t: [t.suggest_float(f"x{i}", 0, 1) for i in range(n_objectives)], n_trials=n_trials, ) trials = study.get_trials() assert len(trials) == 1 # The error stops optimization, but completed trials are recorded. assert all(0 <= x <= 1 for x in trials[0].params.values()) # The params are normal. assert trials[0].values == list(trials[0].params.values()) # The values are normal. assert trials[0].system_attrs[_CONSTRAINTS_KEY] is None # None is set for constraints. def test_study_system_attr_for_population_cache() -> None: sampler = NSGAIIISampler(population_size=10) study = optuna.create_study(directions=["minimize"], sampler=sampler) def get_cached_entries( study: optuna.study.Study, ) -> list[tuple[int, list[int]]]: study_system_attrs = study._storage.get_study_system_attrs(study._study_id) return [ v for k, v in study_system_attrs.items() if k.startswith(_POPULATION_CACHE_KEY_PREFIX) ] study.optimize(lambda t: [t.suggest_float("x", 0, 9)], n_trials=10) cached_entries = get_cached_entries(study) assert len(cached_entries) == 0 study.optimize(lambda t: [t.suggest_float("x", 0, 9)], n_trials=1) cached_entries = get_cached_entries(study) assert len(cached_entries) == 1 assert cached_entries[0][0] == 0 # Cached generation. assert len(cached_entries[0][1]) == 10 # Population size. study.optimize(lambda t: [t.suggest_float("x", 0, 9)], n_trials=10) cached_entries = get_cached_entries(study) assert len(cached_entries) == 1 assert cached_entries[0][0] == 1 # Cached generation. assert len(cached_entries[0][1]) == 10 # Population size. def test_constraints_func_experimental_warning() -> None: with pytest.warns(optuna.exceptions.ExperimentalWarning): NSGAIIISampler(constraints_func=lambda _: [0]) def test_child_generation_strategy_experimental_warning() -> None: with pytest.warns(optuna.exceptions.ExperimentalWarning): NSGAIIISampler(child_generation_strategy=lambda study, search_space, parent_population: {}) def test_after_trial_strategy_experimental_warning() -> None: with pytest.warns(optuna.exceptions.ExperimentalWarning): NSGAIIISampler(after_trial_strategy=lambda study, trial, state, value: None) def test_call_after_trial_of_random_sampler() -> None: sampler = NSGAIIISampler() study = optuna.create_study(sampler=sampler) with patch.object( sampler._random_sampler, "after_trial", wraps=sampler._random_sampler.after_trial ) as mock_object: study.optimize(lambda _: 1.0, n_trials=1) assert mock_object.call_count == 1 def test_call_after_trial_of_after_trial_strategy() -> None: sampler = NSGAIIISampler() study = optuna.create_study(sampler=sampler) with patch.object(sampler, "_after_trial_strategy") as mock_object: study.optimize(lambda _: 1.0, n_trials=1) assert mock_object.call_count == 1 @patch("optuna.samplers.nsgaii._after_trial_strategy._process_constraints_after_trial") def test_nsgaii_after_trial_strategy(mock_func: MagicMock) -> None: def constraints_func(_: FrozenTrial) -> Sequence[float]: return (float("nan"),) state = optuna.trial.TrialState.FAIL study = optuna.create_study() trial = optuna.trial.create_trial(state=state) after_trial_strategy_without_constrains = NSGAIIAfterTrialStrategy() after_trial_strategy_without_constrains(study, trial, state) assert mock_func.call_count == 0 after_trial_strategy_with_constrains = NSGAIIAfterTrialStrategy( constraints_func=constraints_func ) after_trial_strategy_with_constrains(study, trial, state) assert mock_func.call_count == 1 parametrize_crossover_population = pytest.mark.parametrize( "crossover,population_size", [ (UniformCrossover(), 2), (BLXAlphaCrossover(), 2), (SBXCrossover(), 2), (VSBXCrossover(), 2), (UNDXCrossover(), 2), (UNDXCrossover(), 3), ], ) @parametrize_crossover_population @pytest.mark.parametrize("n_objectives", [1, 2, 3]) def test_crossover_objectives( n_objectives: int, crossover: BaseSampler, population_size: int ) -> None: n_trials = 8 sampler = NSGAIIISampler(population_size=population_size) study = optuna.create_study(directions=["minimize"] * n_objectives, sampler=sampler) study.optimize( lambda t: [t.suggest_float(f"x{i}", 0, 1) for i in range(n_objectives)], n_trials=n_trials, ) assert len(study.trials) == n_trials @parametrize_crossover_population @pytest.mark.parametrize("n_params", [1, 2, 3]) def test_crossover_dims(n_params: int, crossover: BaseSampler, population_size: int) -> None: def objective(trial: optuna.Trial) -> float: xs = [trial.suggest_float(f"x{dim}", -10, 10) for dim in range(n_params)] return sum(xs) n_trials = 8 sampler = NSGAIIISampler(population_size=population_size) study = optuna.create_study(directions=["minimize"], sampler=sampler) study.optimize(objective, n_trials=n_trials) assert len(study.trials) == n_trials @pytest.mark.parametrize( "crossover,population_size", [ (UniformCrossover(), 1), (BLXAlphaCrossover(), 1), (SBXCrossover(), 1), (VSBXCrossover(), 1), (UNDXCrossover(), 2), (SPXCrossover(), 2), ], ) def test_crossover_invalid_population(crossover: BaseCrossover, population_size: int) -> None: with pytest.raises(ValueError): NSGAIIISampler(population_size=population_size, crossover=crossover) @pytest.mark.parametrize( "n_objectives,dividing_parameter,expected_reference_points", [ (1, 3, [[3.0]]), (2, 2, [[0.0, 2.0], [1.0, 1.0], [2.0, 0.0]]), (2, 3, [[0.0, 3.0], [1.0, 2.0], [2.0, 1.0], [3.0, 0.0]]), ( 3, 2, [ [0.0, 0.0, 2.0], [0.0, 1.0, 1.0], [0.0, 2.0, 0.0], [1.0, 0.0, 1.0], [1.0, 1.0, 0.0], [2.0, 0.0, 0.0], ], ), ], ) def test_generate_reference_point( n_objectives: int, dividing_parameter: int, expected_reference_points: Sequence[Sequence[int]] ) -> None: actual_reference_points = sorted( _generate_default_reference_point(n_objectives, dividing_parameter).tolist() ) assert actual_reference_points == expected_reference_points @pytest.mark.parametrize( "objective_values, expected_normalized_value", [ ( [ [1.0, 2.0], [float("inf"), 0.5], ], [ [1.0, 2.0], [1.0, 0.5], ], ), ( [ [1.0, float("inf")], [float("inf"), 0.5], ], [ [1.0, 0.5], [1.0, 0.5], ], ), ( [ [1.0, float("inf")], [3.0, 1.0], [2.0, 3.0], [float("inf"), 0.5], ], [ [1.0, 3.0 + _COEF * 2.5], [3.0, 1.0], [2.0, 3.0], [3.0 + _COEF * 2.0, 0.5], ], ), ( [ [2.0, 3.0], [-float("inf"), 3.5], ], [ [2.0, 3.0], [2.0, 3.5], ], ), ( [ [2.0, -float("inf")], [-float("inf"), 3.5], ], [ [2.0, 3.5], [2.0, 3.5], ], ), ( [ [4.0, -float("inf")], [3.0, 1.0], [2.0, 3.0], [-float("inf"), 3.5], ], [ [4.0, 1.0 - _COEF * 2.5], [3.0, 1.0], [2.0, 3.0], [2.0 - _COEF * 2.0, 3.5], ], ), ( [ [1.0, float("inf")], [3.0, -float("inf")], [float("inf"), 2.0], [-float("inf"), 3.5], ], [ [1.0, 3.5 + _COEF * 1.5], [3.0, 2.0 - _COEF * 1.5], [3.0 + _COEF * 2.0, 2.0], [1.0 - _COEF * 2.0, 3.5], ], ), ], ) def test_filter_inf( objective_values: Sequence[Sequence[int]], expected_normalized_value: Sequence[Sequence[int]] ) -> None: population = [create_trial(values=values) for values in objective_values] np.testing.assert_almost_equal(_filter_inf(population), np.array(expected_normalized_value)) @pytest.mark.parametrize( "objective_values, expected_normalized_value", [ ( [ [2.71], [1.41], [3.14], ], [ [(2.71 - 1.41) / (3.14 - 1.41)], [0], [1.0], ], ), ( [ [1.0, 2.0, 3.0], [3.0, 1.0, 2.0], [2.0, 3.0, 1.0], [2.0, 2.0, 2.0], [4.0, 5.0, 6.0], [6.0, 4.0, 5.0], [5.0, 6.0, 4.0], [4.0, 4.0, 4.0], ], [ [0.0, 1.0 / 3.0, 2.0 / 3.0], [2.0 / 3.0, 0.0, 1.0 / 3.0], [1.0 / 3.0, 2.0 / 3.0, 0.0], [1.0 / 3.0, 1.0 / 3.0, 1.0 / 3.0], [1.0, 4.0 / 3.0, 5.0 / 3.0], [5.0 / 3.0, 1.0, 4.0 / 3.0], [4.0 / 3.0, 5.0 / 3.0, 1.0], [1.0, 1.0, 1.0], ], ), ( [ [1.0, 2.0, 3.0], [3.0, 1.0, 2.0], ], [ [0.0, 1.0, 1.0], [1.0, 0.0, 0.0], ], ), ], ) def test_normalize( objective_values: Sequence[Sequence[int]], expected_normalized_value: Sequence[Sequence[int]] ) -> None: np.testing.assert_almost_equal( _normalize_objective_values(np.array(objective_values)), np.array(expected_normalized_value), ) @pytest.mark.parametrize( "objective_values, expected_indices, expected_distances", [ ([[1.0], [2.0], [0.0], [3.0]], [0, 0, 0, 0], [0.0, 0.0, 0.0, 0.0]), ( [ [1.0, 2.0, 3.0], [3.0, 1.0, 2.0], [2.0, 3.0, 1.0], [2.0, 2.0, 2.0], [4.0, 5.0, 6.0], [6.0, 4.0, 5.0], [5.0, 6.0, 4.0], [4.0, 4.0, 4.0], [0.0, 1.0, 10.0], [10.0, 0.0, 1.0], [1.0, 10.0, 0.0], ], [4, 2, 1, 1, 4, 2, 1, 1, 5, 0, 3], [ 1.22474487, 1.22474487, 1.22474487, 2.0, 4.0620192, 4.0620192, 4.0620192, 4.0, 1.0, 1.0, 1.0, ], ), ], ) def test_associate( objective_values: Sequence[Sequence[float]], expected_indices: Sequence[int], expected_distances: Sequence[float], ) -> None: population = np.array(objective_values) n_objectives = population.shape[1] reference_points = _generate_default_reference_point( n_objectives=n_objectives, dividing_parameter=2 ) ( closest_reference_points, distance_reference_points, ) = _associate_individuals_with_reference_points(population, reference_points) assert np.all(closest_reference_points == expected_indices) np.testing.assert_almost_equal(distance_reference_points, expected_distances) @pytest.mark.parametrize( "population_value,closest_reference_points, distance_reference_points, " "expected_population_indices", [ ( [[1.0], [2.0], [0.0], [3.0], [3.5], [5.5], [1.2], [3.3], [4.8]], [0, 0, 0, 0, 0, 0, 0, 0, 0.0], [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [3, 0, 2, 4, 1], ), ( [ [4.0, 5.0, 6.0], [6.0, 4.0, 5.0], [5.0, 6.0, 4.0], [4.0, 4.0, 4.0], [0.0, 1.0, 10.0], [10.0, 0.0, 1.0], [1.0, 10.0, 0.0], ], [4, 2, 1, 1, 4, 2, 1, 1, 5, 0, 3], [ 1.22474487, 1.22474487, 1.22474487, 2.0, 4.0620192, 4.0620192, 4.0620192, 4.0, 1.0, 1.0, 1.0, ], [6, 5, 4, 0, 1], ), ], ) def test_niching( population_value: Sequence[Sequence[float]], closest_reference_points: Sequence[int], distance_reference_points: Sequence[float], expected_population_indices: Sequence[int], ) -> None: sampler = NSGAIIISampler(seed=42) target_population_size = 5 elite_population_num = 4 population = [create_trial(values=value) for value in population_value] actual_additional_elite_population = [ trial.values for trial in _preserve_niche_individuals( target_population_size, elite_population_num, population, np.array(closest_reference_points), np.array(distance_reference_points), sampler._rng.rng, ) ] expected_additional_elite_population = [ population[idx].values for idx in expected_population_indices ] assert np.all(actual_additional_elite_population == expected_additional_elite_population) def test_niching_unexpected_target_population_size() -> None: sampler = NSGAIIISampler(seed=42) target_population_size = 2 elite_population_num = 1 population = [create_trial(values=[1.0])] with pytest.raises(ValueError): _preserve_niche_individuals( target_population_size, elite_population_num, population, np.array([0]), np.array([0.0]), sampler._rng.rng, ) optuna-4.1.0/tests/samplers_tests/test_partial_fixed.py000066400000000000000000000113201471332314300234560ustar00rootroot00000000000000from unittest.mock import patch import warnings import pytest import optuna from optuna.samplers import PartialFixedSampler from optuna.samplers import RandomSampler from optuna.trial import Trial def test_fixed_sampling() -> None: def objective(trial: Trial) -> float: x = trial.suggest_float("x", -10, 10) y = trial.suggest_float("y", -10, 10) return x**2 + y**2 study0 = optuna.create_study() study0.sampler = RandomSampler(seed=42) study0.optimize(objective, n_trials=1) x_sampled0 = study0.trials[0].params["x"] # Fix parameter ``y`` as 0. study1 = optuna.create_study() with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) study1.sampler = PartialFixedSampler( fixed_params={"y": 0}, base_sampler=RandomSampler(seed=42) ) study1.optimize(objective, n_trials=1) x_sampled1 = study1.trials[0].params["x"] y_sampled1 = study1.trials[0].params["y"] assert x_sampled1 == x_sampled0 assert y_sampled1 == 0 def test_float_to_int() -> None: def objective(trial: Trial) -> float: x = trial.suggest_int("x", -10, 10) y = trial.suggest_int("y", -10, 10) return x**2 + y**2 fixed_y = 0.5 # Parameters of Int-type-distribution are rounded to int-type, # even if they are defined as float-type. study = optuna.create_study() with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) study.sampler = PartialFixedSampler( fixed_params={"y": fixed_y}, base_sampler=study.sampler ) # Since `fixed_y` is out-of-the-range value in the corresponding suggest_int, # `UserWarning` will occur. with pytest.warns(UserWarning): study.optimize(objective, n_trials=1) assert study.trials[0].params["y"] == int(fixed_y) @pytest.mark.parametrize("fixed_y", [-2, 2]) def test_out_of_the_range_numerical(fixed_y: int) -> None: def objective(trial: Trial) -> float: x = trial.suggest_int("x", -1, 1) y = trial.suggest_int("y", -1, 1) return x**2 + y**2 # It is possible to fix numerical parameters as out-of-the-range value. # `UserWarning` will occur. study = optuna.create_study() with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) study.sampler = PartialFixedSampler( fixed_params={"y": fixed_y}, base_sampler=study.sampler ) with pytest.warns(UserWarning): study.optimize(objective, n_trials=1) assert study.trials[0].params["y"] == fixed_y def test_out_of_the_range_categorical() -> None: def objective(trial: Trial) -> float: x = trial.suggest_int("x", -1, 1) y = trial.suggest_categorical("y", [-1, 0, 1]) return x**2 + y**2 fixed_y = 2 # It isn't possible to fix categorical parameters as out-of-the-range value. # `ValueError` will occur. study = optuna.create_study() with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) study.sampler = PartialFixedSampler( fixed_params={"y": fixed_y}, base_sampler=study.sampler ) with pytest.raises(ValueError): study.optimize(objective, n_trials=1) def test_partial_fixed_experimental_warning() -> None: study = optuna.create_study() with pytest.warns(optuna.exceptions.ExperimentalWarning): optuna.samplers.PartialFixedSampler(fixed_params={"x": 0}, base_sampler=study.sampler) def test_call_after_trial_of_base_sampler() -> None: base_sampler = RandomSampler() with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = PartialFixedSampler(fixed_params={}, base_sampler=base_sampler) study = optuna.create_study(sampler=sampler) with patch.object(base_sampler, "after_trial", wraps=base_sampler.after_trial) as mock_object: study.optimize(lambda _: 1.0, n_trials=1) assert mock_object.call_count == 1 def test_fixed_none_value_sampling() -> None: def objective(trial: Trial) -> float: trial.suggest_categorical("x", (None, 0)) return 0.0 tpe = optuna.samplers.TPESampler() with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) # In this following case , "x" should sample only `None` sampler = optuna.samplers.PartialFixedSampler(fixed_params={"x": None}, base_sampler=tpe) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=10) for trial in study.trials: assert trial.params["x"] is None optuna-4.1.0/tests/samplers_tests/test_qmc.py000066400000000000000000000321521471332314300214310ustar00rootroot00000000000000from typing import Any from typing import Callable from typing import Dict from unittest.mock import Mock from unittest.mock import patch import warnings import numpy as np import pytest import optuna from optuna.distributions import BaseDistribution from optuna.trial import Trial from optuna.trial import TrialState _SEARCH_SPACE = { "x1": optuna.distributions.IntDistribution(0, 10), "x2": optuna.distributions.IntDistribution(1, 10, log=True), "x3": optuna.distributions.FloatDistribution(0, 10), "x4": optuna.distributions.FloatDistribution(1, 10, log=True), "x5": optuna.distributions.FloatDistribution(1, 10, step=3), "x6": optuna.distributions.CategoricalDistribution([1, 4, 7, 10]), } # TODO(kstoneriv3): `QMCSampler` can be initialized without this wrapper # Remove this after the experimental warning is removed. def _init_QMCSampler_without_exp_warning(**kwargs: Any) -> optuna.samplers.QMCSampler: with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = optuna.samplers.QMCSampler(**kwargs) return sampler def test_experimental_warning() -> None: with pytest.warns(optuna.exceptions.ExperimentalWarning): optuna.samplers.QMCSampler() @pytest.mark.parametrize("qmc_type", ["sobol", "halton", "non-qmc"]) def test_invalid_qmc_type(qmc_type: str) -> None: if qmc_type == "non-qmc": with pytest.raises(ValueError): _init_QMCSampler_without_exp_warning(qmc_type=qmc_type) else: _init_QMCSampler_without_exp_warning(qmc_type=qmc_type) def test_initial_seeding() -> None: with patch.object(optuna.samplers.QMCSampler, "_log_asynchronous_seeding") as mock_log_async: sampler = _init_QMCSampler_without_exp_warning(scramble=True) mock_log_async.assert_called_once() assert isinstance(sampler._seed, int) def test_infer_relative_search_space() -> None: def objective(trial: Trial) -> float: ret: float = trial.suggest_int("x1", 0, 10) ret += trial.suggest_int("x2", 1, 10, log=True) ret += trial.suggest_float("x3", 0, 10) ret += trial.suggest_float("x4", 1, 10, log=True) ret += trial.suggest_float("x5", 1, 10, step=3) _ = trial.suggest_categorical("x6", [1, 4, 7, 10]) return ret sampler = _init_QMCSampler_without_exp_warning() study = optuna.create_study(sampler=sampler) trial = Mock() # In case no past trials. assert sampler.infer_relative_search_space(study, trial) == {} # In case there is a past trial. study.optimize(objective, n_trials=1) relative_search_space = sampler.infer_relative_search_space(study, trial) assert len(relative_search_space.keys()) == 5 assert set(relative_search_space.keys()) == {"x1", "x2", "x3", "x4", "x5"} # In case self._initial_trial already exists. new_search_space: Dict[str, BaseDistribution] = {"x": Mock()} sampler._initial_search_space = new_search_space assert sampler.infer_relative_search_space(study, trial) == new_search_space def test_infer_initial_search_space() -> None: trial = Mock() sampler = _init_QMCSampler_without_exp_warning() # Can it handle empty search space? trial.distributions = {} initial_search_space = sampler._infer_initial_search_space(trial) assert initial_search_space == {} # Does it exclude only categorical distribution? search_space = _SEARCH_SPACE.copy() trial.distributions = search_space initial_search_space = sampler._infer_initial_search_space(trial) search_space.pop("x6") assert initial_search_space == search_space def test_sample_independent() -> None: objective: Callable[[Trial], float] = lambda t: t.suggest_categorical("x", [1.0, 2.0]) independent_sampler = optuna.samplers.RandomSampler() with patch.object( independent_sampler, "sample_independent", wraps=independent_sampler.sample_independent ) as mock_sample_indep: sampler = _init_QMCSampler_without_exp_warning(independent_sampler=independent_sampler) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=1) assert mock_sample_indep.call_count == 1 # Relative sampling of `QMCSampler` does not support categorical distribution. # Thus, `independent_sampler.sample_independent` is called twice. study.optimize(objective, n_trials=1) assert mock_sample_indep.call_count == 2 # Unseen parameter is sampled by independent sampler. new_objective: Callable[[Trial], int] = lambda t: t.suggest_int("y", 0, 10) study.optimize(new_objective, n_trials=1) assert mock_sample_indep.call_count == 3 def test_warn_asynchronous_seeding() -> None: # Relative sampling of `QMCSampler` does not support categorical distribution. # Thus, `independent_sampler.sample_independent` is called twice. # '_log_independent_sampling is not called in the first trial so called once in total. objective: Callable[[Trial], float] = lambda t: t.suggest_categorical("x", [1.0, 2.0]) with patch.object(optuna.samplers.QMCSampler, "_log_asynchronous_seeding") as mock_log_async: sampler = _init_QMCSampler_without_exp_warning( scramble=True, warn_asynchronous_seeding=False ) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=2) assert mock_log_async.call_count == 0 sampler = _init_QMCSampler_without_exp_warning(scramble=True) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=2) assert mock_log_async.call_count == 1 def test_warn_independent_sampling() -> None: # Relative sampling of `QMCSampler` does not support categorical distribution. # Thus, `independent_sampler.sample_independent` is called twice. # '_log_independent_sampling is not called in the first trial so called once in total. objective: Callable[[Trial], float] = lambda t: t.suggest_categorical("x", [1.0, 2.0]) with patch.object(optuna.samplers.QMCSampler, "_log_independent_sampling") as mock_log_indep: sampler = _init_QMCSampler_without_exp_warning(warn_independent_sampling=False) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=2) assert mock_log_indep.call_count == 0 sampler = _init_QMCSampler_without_exp_warning() study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=2) assert mock_log_indep.call_count == 1 def test_sample_relative() -> None: search_space = _SEARCH_SPACE.copy() search_space.pop("x6") sampler = _init_QMCSampler_without_exp_warning() study = optuna.create_study(sampler=sampler) trial = Mock() # Make sure that sample type, shape is OK. for _ in range(3): sample = sampler.sample_relative(study, trial, search_space) assert 0 <= sample["x1"] <= 10 assert 1 <= sample["x2"] <= 10 assert 0 <= sample["x3"] <= 10 assert 1 <= sample["x4"] <= 10 assert 1 <= sample["x5"] <= 10 assert isinstance(sample["x1"], int) assert isinstance(sample["x2"], int) assert sample["x5"] in (1, 4, 7, 10) # If empty search_space, return {}. assert sampler.sample_relative(study, trial, {}) == {} def test_sample_relative_halton() -> None: n, d = 8, 5 search_space: Dict[str, BaseDistribution] = { f"x{i}": optuna.distributions.FloatDistribution(0, 1) for i in range(d) } sampler = _init_QMCSampler_without_exp_warning(scramble=False, qmc_type="halton") study = optuna.create_study(sampler=sampler) trial = Mock() # Make sure that sample type, shape is OK. samples = np.zeros((n, d)) for i in range(n): sample = sampler.sample_relative(study, trial, search_space) for j in range(d): samples[i, j] = sample[f"x{j}"] ref_samples = np.array( [ [0.0, 0.0, 0.0, 0.0, 0.0], [0.5, 0.33333333, 0.2, 0.14285714, 0.09090909], [0.25, 0.66666667, 0.4, 0.28571429, 0.18181818], [0.75, 0.11111111, 0.6, 0.42857143, 0.27272727], [0.125, 0.44444444, 0.8, 0.57142857, 0.36363636], [0.625, 0.77777778, 0.04, 0.71428571, 0.45454545], [0.375, 0.22222222, 0.24, 0.85714286, 0.54545455], [0.875, 0.55555556, 0.44, 0.02040816, 0.63636364], ] ) # If empty search_space, return {}. np.testing.assert_allclose(samples, ref_samples, rtol=1e-6) def test_sample_relative_sobol() -> None: n, d = 8, 5 search_space: Dict[str, BaseDistribution] = { f"x{i}": optuna.distributions.FloatDistribution(0, 1) for i in range(d) } sampler = _init_QMCSampler_without_exp_warning(scramble=False, qmc_type="sobol") study = optuna.create_study(sampler=sampler) trial = Mock() # Make sure that sample type, shape is OK. samples = np.zeros((n, d)) for i in range(n): sample = sampler.sample_relative(study, trial, search_space) for j in range(d): samples[i, j] = sample[f"x{j}"] ref_samples = np.array( [ [0.0, 0.0, 0.0, 0.0, 0.0], [0.5, 0.5, 0.5, 0.5, 0.5], [0.75, 0.25, 0.25, 0.25, 0.75], [0.25, 0.75, 0.75, 0.75, 0.25], [0.375, 0.375, 0.625, 0.875, 0.375], [0.875, 0.875, 0.125, 0.375, 0.875], [0.625, 0.125, 0.875, 0.625, 0.625], [0.125, 0.625, 0.375, 0.125, 0.125], ] ) # If empty search_space, return {}. np.testing.assert_allclose(samples, ref_samples, rtol=1e-6) @pytest.mark.parametrize("scramble", [True, False]) @pytest.mark.parametrize("qmc_type", ["sobol", "halton"]) @pytest.mark.parametrize("seed", [0, 12345]) def test_sample_relative_seeding(scramble: bool, qmc_type: str, seed: int) -> None: objective: Callable[[Trial], float] = lambda t: t.suggest_float("x", 0, 1) # Base case. sampler = _init_QMCSampler_without_exp_warning(scramble=scramble, qmc_type=qmc_type, seed=seed) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=10, n_jobs=1) past_trials = study._storage.get_all_trials(study._study_id, states=(TrialState.COMPLETE,)) past_trials = [t for t in past_trials if t.number > 0] values = [t.params["x"] for t in past_trials] # Sequential case. sampler = _init_QMCSampler_without_exp_warning(scramble=scramble, qmc_type=qmc_type, seed=seed) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=10, n_jobs=1) past_trials_sequential = study._storage.get_all_trials( study._study_id, states=(TrialState.COMPLETE,) ) past_trials_sequential = [t for t in past_trials_sequential if t.number > 0] values_sequential = [t.params["x"] for t in past_trials_sequential] np.testing.assert_allclose(values, values_sequential, rtol=1e-6) # Parallel case (n_jobs=3): # Same parameters might be evaluated multiple times. sampler = _init_QMCSampler_without_exp_warning(scramble=scramble, qmc_type=qmc_type, seed=seed) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=30, n_jobs=3) past_trials_parallel = study._storage.get_all_trials( study._study_id, states=(TrialState.COMPLETE,) ) past_trials_parallel = [t for t in past_trials_parallel if t.number > 0] values_parallel = [t.params["x"] for t in past_trials_parallel] for v in values: assert np.any( np.isclose(v, values_parallel, rtol=1e-6) ), f"v: {v} of values: {values} is not included in values_parallel: {values_parallel}." def test_call_after_trial() -> None: sampler = _init_QMCSampler_without_exp_warning() study = optuna.create_study(sampler=sampler) with patch.object( sampler._independent_sampler, "after_trial", wraps=sampler._independent_sampler.after_trial ) as mock_object: study.optimize(lambda _: 1.0, n_trials=1) assert mock_object.call_count == 1 @pytest.mark.parametrize("qmc_type", ["sobol", "halton"]) def test_sample_qmc(qmc_type: str) -> None: sampler = _init_QMCSampler_without_exp_warning(qmc_type=qmc_type) study = Mock() search_space = _SEARCH_SPACE.copy() search_space.pop("x6") with patch.object(sampler, "_find_sample_id", side_effect=[0, 1, 2, 4, 9]) as _: # Make sure that the shape of sample is correct. sample = sampler._sample_qmc(study, search_space) assert sample.shape == (1, 5) def test_find_sample_id() -> None: sampler = _init_QMCSampler_without_exp_warning(qmc_type="halton", seed=0) study = optuna.create_study() for i in range(5): assert sampler._find_sample_id(study) == i # Change seed but without scramble. The hash should remain the same. with patch.object(sampler, "_seed", 1) as _: assert sampler._find_sample_id(study) == 5 # Seed is considered only when scrambling is enabled. with patch.object(sampler, "_scramble", True) as _: assert sampler._find_sample_id(study) == 0 # Change qmc_type. with patch.object(sampler, "_qmc_type", "sobol") as _: assert sampler._find_sample_id(study) == 0 optuna-4.1.0/tests/samplers_tests/test_samplers.py000066400000000000000000001131731471332314300225020ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Callable from collections.abc import Sequence import multiprocessing from multiprocessing.managers import DictProxy import os import pickle from typing import Any from unittest.mock import patch import warnings from _pytest.fixtures import SubRequest from _pytest.mark.structures import MarkDecorator import numpy as np import pytest import optuna from optuna.distributions import BaseDistribution from optuna.distributions import CategoricalChoiceType from optuna.distributions import CategoricalDistribution from optuna.distributions import FloatDistribution from optuna.distributions import IntDistribution from optuna.samplers import BaseSampler from optuna.samplers._lazy_random_state import LazyRandomState from optuna.study import Study from optuna.testing.objectives import fail_objective from optuna.testing.objectives import pruned_objective from optuna.trial import FrozenTrial from optuna.trial import Trial from optuna.trial import TrialState def get_gp_sampler( *, n_startup_trials: int = 0, deterministic_objective: bool = False, seed: int | None = None ) -> optuna.samplers.GPSampler: return optuna.samplers.GPSampler( n_startup_trials=n_startup_trials, seed=seed, deterministic_objective=deterministic_objective, ) parametrize_sampler = pytest.mark.parametrize( "sampler_class", [ optuna.samplers.RandomSampler, lambda: optuna.samplers.TPESampler(n_startup_trials=0), lambda: optuna.samplers.TPESampler(n_startup_trials=0, multivariate=True), lambda: optuna.samplers.CmaEsSampler(n_startup_trials=0), lambda: optuna.samplers.CmaEsSampler(n_startup_trials=0, use_separable_cma=True), optuna.samplers.NSGAIISampler, optuna.samplers.NSGAIIISampler, optuna.samplers.QMCSampler, lambda: get_gp_sampler(n_startup_trials=0), lambda: get_gp_sampler(n_startup_trials=0, deterministic_objective=True), ], ) parametrize_relative_sampler = pytest.mark.parametrize( "relative_sampler_class", [ lambda: optuna.samplers.TPESampler(n_startup_trials=0, multivariate=True), lambda: optuna.samplers.CmaEsSampler(n_startup_trials=0), lambda: optuna.samplers.CmaEsSampler(n_startup_trials=0, use_separable_cma=True), lambda: get_gp_sampler(n_startup_trials=0), lambda: get_gp_sampler(n_startup_trials=0, deterministic_objective=True), ], ) parametrize_multi_objective_sampler = pytest.mark.parametrize( "multi_objective_sampler_class", [ optuna.samplers.NSGAIISampler, optuna.samplers.NSGAIIISampler, lambda: optuna.samplers.TPESampler(n_startup_trials=0), ], ) sampler_class_with_seed: dict[str, Callable[[int], BaseSampler]] = { "RandomSampler": lambda seed: optuna.samplers.RandomSampler(seed=seed), "TPESampler": lambda seed: optuna.samplers.TPESampler(seed=seed), "multivariate TPESampler": lambda seed: optuna.samplers.TPESampler( multivariate=True, seed=seed ), "CmaEsSampler": lambda seed: optuna.samplers.CmaEsSampler(seed=seed), "separable CmaEsSampler": lambda seed: optuna.samplers.CmaEsSampler( seed=seed, use_separable_cma=True ), "NSGAIISampler": lambda seed: optuna.samplers.NSGAIISampler(seed=seed), "NSGAIIISampler": lambda seed: optuna.samplers.NSGAIIISampler(seed=seed), "QMCSampler": lambda seed: optuna.samplers.QMCSampler(seed=seed), "GPSampler": lambda seed: get_gp_sampler(seed=seed, n_startup_trials=0), } param_sampler_with_seed = [] param_sampler_name_with_seed = [] for sampler_name, sampler_class in sampler_class_with_seed.items(): param_sampler_with_seed.append(pytest.param(sampler_class, id=sampler_name)) param_sampler_name_with_seed.append(pytest.param(sampler_name)) parametrize_sampler_with_seed = pytest.mark.parametrize("sampler_class", param_sampler_with_seed) parametrize_sampler_name_with_seed = pytest.mark.parametrize( "sampler_name", param_sampler_name_with_seed ) @pytest.mark.parametrize( "sampler_class,expected_has_rng,expected_has_another_sampler", [ (optuna.samplers.RandomSampler, True, False), (lambda: optuna.samplers.TPESampler(n_startup_trials=0), True, True), (lambda: optuna.samplers.TPESampler(n_startup_trials=0, multivariate=True), True, True), (lambda: optuna.samplers.CmaEsSampler(n_startup_trials=0), True, True), (optuna.samplers.NSGAIISampler, True, True), (optuna.samplers.NSGAIIISampler, True, True), ( lambda: optuna.samplers.PartialFixedSampler( fixed_params={"x": 0}, base_sampler=optuna.samplers.RandomSampler() ), False, True, ), (lambda: optuna.samplers.GridSampler(search_space={"x": [0]}), True, False), (lambda: optuna.samplers.QMCSampler(), False, True), (lambda: get_gp_sampler(n_startup_trials=0), True, True), ], ) def test_sampler_reseed_rng( sampler_class: Callable[[], BaseSampler], expected_has_rng: bool, expected_has_another_sampler: bool, ) -> None: def _extract_attr_name_from_sampler_by_cls(sampler: BaseSampler, cls: Any) -> str | None: for name, attr in sampler.__dict__.items(): if isinstance(attr, cls): return name return None sampler = sampler_class() rng_name = _extract_attr_name_from_sampler_by_cls(sampler, LazyRandomState) has_rng = rng_name is not None assert expected_has_rng == has_rng if has_rng: rng_name = str(rng_name) original_random_state = sampler.__dict__[rng_name].rng.get_state() sampler.reseed_rng() random_state = sampler.__dict__[rng_name].rng.get_state() if not isinstance(sampler, optuna.samplers.CmaEsSampler): assert str(original_random_state) != str(random_state) else: # CmaEsSampler has a RandomState that is not reseed by its reseed_rng method. assert str(original_random_state) == str(random_state) had_sampler_name = _extract_attr_name_from_sampler_by_cls(sampler, BaseSampler) has_another_sampler = had_sampler_name is not None assert expected_has_another_sampler == has_another_sampler if has_another_sampler: had_sampler_name = str(had_sampler_name) had_sampler = sampler.__dict__[had_sampler_name] had_sampler_rng_name = _extract_attr_name_from_sampler_by_cls(had_sampler, LazyRandomState) original_had_sampler_random_state = had_sampler.__dict__[ had_sampler_rng_name ].rng.get_state() with patch.object( had_sampler, "reseed_rng", wraps=had_sampler.reseed_rng, ) as mock_object: sampler.reseed_rng() assert mock_object.call_count == 1 had_sampler = sampler.__dict__[had_sampler_name] had_sampler_random_state = had_sampler.__dict__[had_sampler_rng_name].rng.get_state() assert str(original_had_sampler_random_state) != str(had_sampler_random_state) def parametrize_suggest_method(name: str) -> MarkDecorator: return pytest.mark.parametrize( f"suggest_method_{name}", [ lambda t: t.suggest_float(name, 0, 10), lambda t: t.suggest_int(name, 0, 10), lambda t: t.suggest_categorical(name, [0, 1, 2]), lambda t: t.suggest_float(name, 0, 10, step=0.5), lambda t: t.suggest_float(name, 1e-7, 10, log=True), lambda t: t.suggest_int(name, 1, 10, log=True), ], ) @pytest.mark.parametrize( "sampler_class", [ lambda: optuna.samplers.CmaEsSampler(n_startup_trials=0), ], ) def test_raise_error_for_samplers_during_multi_objectives( sampler_class: Callable[[], BaseSampler] ) -> None: study = optuna.study.create_study(directions=["maximize", "maximize"], sampler=sampler_class()) distribution = FloatDistribution(0.0, 1.0) with pytest.raises(ValueError): study.sampler.sample_independent(study, _create_new_trial(study), "x", distribution) with pytest.raises(ValueError): trial = _create_new_trial(study) study.sampler.sample_relative( study, trial, study.sampler.infer_relative_search_space(study, trial) ) @pytest.mark.parametrize("seed", [0, 169208]) def test_pickle_random_sampler(seed: int) -> None: sampler = optuna.samplers.RandomSampler(seed) restored_sampler = pickle.loads(pickle.dumps(sampler)) assert sampler._rng.rng.bytes(10) == restored_sampler._rng.rng.bytes(10) @parametrize_sampler @pytest.mark.parametrize( "distribution", [ FloatDistribution(-1.0, 1.0), FloatDistribution(0.0, 1.0), FloatDistribution(-1.0, 0.0), FloatDistribution(1e-7, 1.0, log=True), FloatDistribution(-10, 10, step=0.1), FloatDistribution(-10.2, 10.2, step=0.1), ], ) def test_float( sampler_class: Callable[[], BaseSampler], distribution: FloatDistribution, ) -> None: study = optuna.study.create_study(sampler=sampler_class()) points = np.array( [ study.sampler.sample_independent(study, _create_new_trial(study), "x", distribution) for _ in range(100) ] ) assert np.all(points >= distribution.low) assert np.all(points <= distribution.high) assert not isinstance( study.sampler.sample_independent(study, _create_new_trial(study), "x", distribution), np.floating, ) if distribution.step is not None: # Check all points are multiples of distribution.step. points -= distribution.low points /= distribution.step round_points = np.round(points) np.testing.assert_almost_equal(round_points, points) @parametrize_sampler @pytest.mark.parametrize( "distribution", [ IntDistribution(-10, 10), IntDistribution(0, 10), IntDistribution(-10, 0), IntDistribution(-10, 10, step=2), IntDistribution(0, 10, step=2), IntDistribution(-10, 0, step=2), IntDistribution(1, 100, log=True), ], ) def test_int(sampler_class: Callable[[], BaseSampler], distribution: IntDistribution) -> None: study = optuna.study.create_study(sampler=sampler_class()) points = np.array( [ study.sampler.sample_independent(study, _create_new_trial(study), "x", distribution) for _ in range(100) ] ) assert np.all(points >= distribution.low) assert np.all(points <= distribution.high) assert not isinstance( study.sampler.sample_independent(study, _create_new_trial(study), "x", distribution), np.integer, ) @parametrize_sampler @pytest.mark.parametrize("choices", [(1, 2, 3), ("a", "b", "c"), (1, "a")]) def test_categorical( sampler_class: Callable[[], BaseSampler], choices: Sequence[CategoricalChoiceType] ) -> None: distribution = CategoricalDistribution(choices) study = optuna.study.create_study(sampler=sampler_class()) def sample() -> float: trial = _create_new_trial(study) param_value = study.sampler.sample_independent(study, trial, "x", distribution) return float(distribution.to_internal_repr(param_value)) points = np.asarray([sample() for i in range(100)]) # 'x' value is corresponding to an index of distribution.choices. assert np.all(points >= 0) assert np.all(points <= len(distribution.choices) - 1) round_points = np.round(points) np.testing.assert_almost_equal(round_points, points) @parametrize_relative_sampler @pytest.mark.parametrize( "x_distribution", [ FloatDistribution(-1.0, 1.0), FloatDistribution(1e-7, 1.0, log=True), FloatDistribution(-10, 10, step=0.5), IntDistribution(3, 10), IntDistribution(1, 100, log=True), IntDistribution(3, 9, step=2), ], ) @pytest.mark.parametrize( "y_distribution", [ FloatDistribution(-1.0, 1.0), FloatDistribution(1e-7, 1.0, log=True), FloatDistribution(-10, 10, step=0.5), IntDistribution(3, 10), IntDistribution(1, 100, log=True), IntDistribution(3, 9, step=2), ], ) def test_sample_relative_numerical( relative_sampler_class: Callable[[], BaseSampler], x_distribution: BaseDistribution, y_distribution: BaseDistribution, ) -> None: search_space: dict[str, BaseDistribution] = dict(x=x_distribution, y=y_distribution) study = optuna.study.create_study(sampler=relative_sampler_class()) trial = study.ask(search_space) study.tell(trial, sum(trial.params.values())) def sample() -> list[int | float]: params = study.sampler.sample_relative(study, _create_new_trial(study), search_space) return [params[name] for name in search_space] points = np.array([sample() for _ in range(10)]) for i, distribution in enumerate(search_space.values()): assert isinstance( distribution, ( FloatDistribution, IntDistribution, ), ) assert np.all(points[:, i] >= distribution.low) assert np.all(points[:, i] <= distribution.high) for param_value, distribution in zip(sample(), search_space.values()): assert not isinstance(param_value, np.floating) assert not isinstance(param_value, np.integer) if isinstance(distribution, IntDistribution): assert isinstance(param_value, int) else: assert isinstance(param_value, float) @parametrize_relative_sampler def test_sample_relative_categorical(relative_sampler_class: Callable[[], BaseSampler]) -> None: search_space: dict[str, BaseDistribution] = dict( x=CategoricalDistribution([1, 10, 100]), y=CategoricalDistribution([-1, -10, -100]) ) study = optuna.study.create_study(sampler=relative_sampler_class()) trial = study.ask(search_space) study.tell(trial, sum(trial.params.values())) def sample() -> list[float]: params = study.sampler.sample_relative(study, _create_new_trial(study), search_space) return [params[name] for name in search_space] points = np.array([sample() for _ in range(10)]) for i, distribution in enumerate(search_space.values()): assert isinstance(distribution, CategoricalDistribution) assert np.all([v in distribution.choices for v in points[:, i]]) for param_value in sample(): assert not isinstance(param_value, np.floating) assert not isinstance(param_value, np.integer) assert isinstance(param_value, int) @parametrize_relative_sampler @pytest.mark.parametrize( "x_distribution", [ FloatDistribution(-1.0, 1.0), FloatDistribution(1e-7, 1.0, log=True), FloatDistribution(-10, 10, step=0.5), IntDistribution(1, 10), IntDistribution(1, 100, log=True), ], ) def test_sample_relative_mixed( relative_sampler_class: Callable[[], BaseSampler], x_distribution: BaseDistribution ) -> None: search_space: dict[str, BaseDistribution] = dict( x=x_distribution, y=CategoricalDistribution([-1, -10, -100]) ) study = optuna.study.create_study(sampler=relative_sampler_class()) trial = study.ask(search_space) study.tell(trial, sum(trial.params.values())) def sample() -> list[float]: params = study.sampler.sample_relative(study, _create_new_trial(study), search_space) return [params[name] for name in search_space] points = np.array([sample() for _ in range(10)]) assert isinstance( search_space["x"], ( FloatDistribution, IntDistribution, ), ) assert np.all(points[:, 0] >= search_space["x"].low) assert np.all(points[:, 0] <= search_space["x"].high) assert isinstance(search_space["y"], CategoricalDistribution) assert np.all([v in search_space["y"].choices for v in points[:, 1]]) for param_value, distribution in zip(sample(), search_space.values()): assert not isinstance(param_value, np.floating) assert not isinstance(param_value, np.integer) if isinstance( distribution, ( IntDistribution, CategoricalDistribution, ), ): assert isinstance(param_value, int) else: assert isinstance(param_value, float) @parametrize_sampler def test_conditional_sample_independent(sampler_class: Callable[[], BaseSampler]) -> None: # This test case reproduces the error reported in #2734. # See https://github.com/optuna/optuna/pull/2734#issuecomment-857649769. study = optuna.study.create_study(sampler=sampler_class()) categorical_distribution = CategoricalDistribution(choices=["x", "y"]) dependent_distribution = CategoricalDistribution(choices=["a", "b"]) study.add_trial( optuna.create_trial( params={"category": "x", "x": "a"}, distributions={"category": categorical_distribution, "x": dependent_distribution}, value=0.1, ) ) study.add_trial( optuna.create_trial( params={"category": "y", "y": "b"}, distributions={"category": categorical_distribution, "y": dependent_distribution}, value=0.1, ) ) _trial = _create_new_trial(study) category = study.sampler.sample_independent( study, _trial, "category", categorical_distribution ) assert category in ["x", "y"] value = study.sampler.sample_independent(study, _trial, category, dependent_distribution) assert value in ["a", "b"] def _create_new_trial(study: Study) -> FrozenTrial: trial_id = study._storage.create_new_trial(study._study_id) return study._storage.get_trial(trial_id) class FixedSampler(BaseSampler): def __init__( self, relative_search_space: dict[str, BaseDistribution], relative_params: dict[str, Any], unknown_param_value: Any, ) -> None: self.relative_search_space = relative_search_space self.relative_params = relative_params self.unknown_param_value = unknown_param_value def infer_relative_search_space( self, study: Study, trial: FrozenTrial ) -> dict[str, BaseDistribution]: return self.relative_search_space def sample_relative( self, study: Study, trial: FrozenTrial, search_space: dict[str, BaseDistribution] ) -> dict[str, Any]: return self.relative_params def sample_independent( self, study: Study, trial: FrozenTrial, param_name: str, param_distribution: BaseDistribution, ) -> Any: return self.unknown_param_value def test_sample_relative() -> None: relative_search_space: dict[str, BaseDistribution] = { "a": FloatDistribution(low=0, high=5), "b": CategoricalDistribution(choices=("foo", "bar", "baz")), "c": IntDistribution(low=20, high=50), # Not exist in `relative_params`. } relative_params = { "a": 3.2, "b": "baz", } unknown_param_value = 30 sampler = FixedSampler(relative_search_space, relative_params, unknown_param_value) study = optuna.study.create_study(sampler=sampler) def objective(trial: Trial) -> float: # Predefined parameters are sampled by `sample_relative()` method. assert trial.suggest_float("a", 0, 5) == 3.2 assert trial.suggest_categorical("b", ["foo", "bar", "baz"]) == "baz" # Other parameters are sampled by `sample_independent()` method. assert trial.suggest_int("c", 20, 50) == unknown_param_value assert trial.suggest_float("d", 1, 100, log=True) == unknown_param_value assert trial.suggest_float("e", 20, 40) == unknown_param_value return 0.0 study.optimize(objective, n_trials=10, catch=()) for trial in study.trials: assert trial.params == {"a": 3.2, "b": "baz", "c": 30, "d": 30, "e": 30} @parametrize_sampler def test_nan_objective_value(sampler_class: Callable[[], BaseSampler]) -> None: study = optuna.create_study(sampler=sampler_class()) def objective(trial: Trial, base_value: float) -> float: return trial.suggest_float("x", 0.1, 0.2) + base_value # Non NaN objective values. for i in range(10, 1, -1): study.optimize(lambda t: objective(t, i), n_trials=1, catch=()) assert int(study.best_value) == 2 # NaN objective values. study.optimize(lambda t: objective(t, float("nan")), n_trials=1, catch=()) assert int(study.best_value) == 2 # Non NaN objective value. study.optimize(lambda t: objective(t, 1), n_trials=1, catch=()) assert int(study.best_value) == 1 @parametrize_sampler def test_partial_fixed_sampling(sampler_class: Callable[[], BaseSampler]) -> None: study = optuna.create_study(sampler=sampler_class()) def objective(trial: Trial) -> float: x = trial.suggest_float("x", -1, 1) y = trial.suggest_int("y", -1, 1) z = trial.suggest_float("z", -1, 1) return x + y + z # First trial. study.optimize(objective, n_trials=1) # Second trial. Here, the parameter ``y`` is fixed as 0. fixed_params = {"y": 0} with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) study.sampler = optuna.samplers.PartialFixedSampler(fixed_params, study.sampler) study.optimize(objective, n_trials=1) trial_params = study.trials[-1].params assert trial_params["y"] == fixed_params["y"] @parametrize_multi_objective_sampler @pytest.mark.parametrize( "distribution", [ FloatDistribution(-1.0, 1.0), FloatDistribution(0.0, 1.0), FloatDistribution(-1.0, 0.0), FloatDistribution(1e-7, 1.0, log=True), FloatDistribution(-10, 10, step=0.1), FloatDistribution(-10.2, 10.2, step=0.1), IntDistribution(-10, 10), IntDistribution(0, 10), IntDistribution(-10, 0), IntDistribution(-10, 10, step=2), IntDistribution(0, 10, step=2), IntDistribution(-10, 0, step=2), IntDistribution(1, 100, log=True), CategoricalDistribution((1, 2, 3)), CategoricalDistribution(("a", "b", "c")), CategoricalDistribution((1, "a")), ], ) def test_multi_objective_sample_independent( multi_objective_sampler_class: Callable[[], BaseSampler], distribution: BaseDistribution ) -> None: study = optuna.study.create_study( directions=["minimize", "maximize"], sampler=multi_objective_sampler_class() ) for i in range(100): value = study.sampler.sample_independent( study, _create_new_trial(study), "x", distribution ) assert distribution._contains(distribution.to_internal_repr(value)) if not isinstance(distribution, CategoricalDistribution): # Please see https://github.com/optuna/optuna/pull/393 why this assertion is needed. assert not isinstance(value, np.floating) if isinstance(distribution, FloatDistribution): if distribution.step is not None: # Check the value is a multiple of `distribution.step` which is # the quantization interval of the distribution. value -= distribution.low value /= distribution.step round_value = np.round(value) np.testing.assert_almost_equal(round_value, value) def test_before_trial() -> None: n_calls = 0 n_trials = 3 class SamplerBeforeTrial(optuna.samplers.RandomSampler): def before_trial(self, study: Study, trial: FrozenTrial) -> None: assert len(study.trials) - 1 == trial.number assert trial.state == TrialState.RUNNING assert trial.values is None nonlocal n_calls n_calls += 1 sampler = SamplerBeforeTrial() study = optuna.create_study(directions=["minimize", "minimize"], sampler=sampler) study.optimize( lambda t: [t.suggest_float("y", -3, 3), t.suggest_int("x", 0, 10)], n_trials=n_trials ) assert n_calls == n_trials def test_after_trial() -> None: n_calls = 0 n_trials = 3 class SamplerAfterTrial(optuna.samplers.RandomSampler): def after_trial( self, study: Study, trial: FrozenTrial, state: TrialState, values: Sequence[float] | None, ) -> None: assert len(study.trials) - 1 == trial.number assert trial.state == TrialState.RUNNING assert trial.values is None assert state == TrialState.COMPLETE assert values is not None assert len(values) == 2 nonlocal n_calls n_calls += 1 sampler = SamplerAfterTrial() study = optuna.create_study(directions=["minimize", "minimize"], sampler=sampler) study.optimize(lambda t: [t.suggest_float("y", -3, 3), t.suggest_int("x", 0, 10)], n_trials=3) assert n_calls == n_trials def test_after_trial_pruning() -> None: n_calls = 0 n_trials = 3 class SamplerAfterTrial(optuna.samplers.RandomSampler): def after_trial( self, study: Study, trial: FrozenTrial, state: TrialState, values: Sequence[float] | None, ) -> None: assert len(study.trials) - 1 == trial.number assert trial.state == TrialState.RUNNING assert trial.values is None assert state == TrialState.PRUNED assert values is None nonlocal n_calls n_calls += 1 sampler = SamplerAfterTrial() study = optuna.create_study(directions=["minimize", "minimize"], sampler=sampler) study.optimize(pruned_objective, n_trials=n_trials) assert n_calls == n_trials def test_after_trial_failing() -> None: n_calls = 0 n_trials = 3 class SamplerAfterTrial(optuna.samplers.RandomSampler): def after_trial( self, study: Study, trial: FrozenTrial, state: TrialState, values: Sequence[float] | None, ) -> None: assert len(study.trials) - 1 == trial.number assert trial.state == TrialState.RUNNING assert trial.values is None assert state == TrialState.FAIL assert values is None nonlocal n_calls n_calls += 1 sampler = SamplerAfterTrial() study = optuna.create_study(directions=["minimize", "minimize"], sampler=sampler) with pytest.raises(ValueError): study.optimize(fail_objective, n_trials=n_trials) # Called once after the first failing trial before returning from optimize. assert n_calls == 1 def test_after_trial_failing_in_after_trial() -> None: n_calls = 0 n_trials = 3 class SamplerAfterTrialAlwaysFail(optuna.samplers.RandomSampler): def after_trial( self, study: Study, trial: FrozenTrial, state: TrialState, values: Sequence[float] | None, ) -> None: nonlocal n_calls n_calls += 1 raise NotImplementedError # Arbitrary error for testing purpose. sampler = SamplerAfterTrialAlwaysFail() study = optuna.create_study(sampler=sampler) with pytest.raises(NotImplementedError): study.optimize(lambda t: t.suggest_int("x", 0, 10), n_trials=n_trials) assert len(study.trials) == 1 assert n_calls == 1 sampler = SamplerAfterTrialAlwaysFail() study = optuna.create_study(sampler=sampler) # Not affected by `catch`. with pytest.raises(NotImplementedError): study.optimize( lambda t: t.suggest_int("x", 0, 10), n_trials=n_trials, catch=(NotImplementedError,) ) assert len(study.trials) == 1 assert n_calls == 2 def test_after_trial_with_study_tell() -> None: n_calls = 0 class SamplerAfterTrial(optuna.samplers.RandomSampler): def after_trial( self, study: Study, trial: FrozenTrial, state: TrialState, values: Sequence[float] | None, ) -> None: nonlocal n_calls n_calls += 1 sampler = SamplerAfterTrial() study = optuna.create_study(sampler=sampler) assert n_calls == 0 study.tell(study.ask(), 1.0) assert n_calls == 1 @parametrize_sampler def test_sample_single_distribution(sampler_class: Callable[[], BaseSampler]) -> None: relative_search_space = { "a": CategoricalDistribution([1]), "b": IntDistribution(low=1, high=1), "c": IntDistribution(low=1, high=1, log=True), "d": FloatDistribution(low=1.0, high=1.0), "e": FloatDistribution(low=1.0, high=1.0, log=True), "f": FloatDistribution(low=1.0, high=1.0, step=1.0), } with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = sampler_class() study = optuna.study.create_study(sampler=sampler) # We need to test the construction of the model, so we should set `n_trials >= 2`. for _ in range(2): trial = study.ask(fixed_distributions=relative_search_space) study.tell(trial, 1.0) for param_name in relative_search_space.keys(): assert trial.params[param_name] == 1 @parametrize_sampler @parametrize_suggest_method("x") def test_single_parameter_objective( sampler_class: Callable[[], BaseSampler], suggest_method_x: Callable[[Trial], float] ) -> None: def objective(trial: Trial) -> float: return suggest_method_x(trial) with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = sampler_class() study = optuna.study.create_study(sampler=sampler) study.optimize(objective, n_trials=10) assert len(study.trials) == 10 assert all(t.state == TrialState.COMPLETE for t in study.trials) @parametrize_sampler def test_conditional_parameter_objective(sampler_class: Callable[[], BaseSampler]) -> None: def objective(trial: Trial) -> float: x = trial.suggest_categorical("x", [True, False]) if x: return trial.suggest_float("y", 0, 1) return trial.suggest_float("z", 0, 1) with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = sampler_class() study = optuna.study.create_study(sampler=sampler) study.optimize(objective, n_trials=10) assert len(study.trials) == 10 assert all(t.state == TrialState.COMPLETE for t in study.trials) @parametrize_sampler @parametrize_suggest_method("x") @parametrize_suggest_method("y") def test_combination_of_different_distributions_objective( sampler_class: Callable[[], BaseSampler], suggest_method_x: Callable[[Trial], float], suggest_method_y: Callable[[Trial], float], ) -> None: def objective(trial: Trial) -> float: return suggest_method_x(trial) + suggest_method_y(trial) with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = sampler_class() study = optuna.study.create_study(sampler=sampler) study.optimize(objective, n_trials=3) assert len(study.trials) == 3 assert all(t.state == TrialState.COMPLETE for t in study.trials) @parametrize_sampler @pytest.mark.parametrize( "second_low,second_high", [ (0, 5), # Narrow range. (0, 20), # Expand range. (20, 30), # Set non-overlapping range. ], ) def test_dynamic_range_objective( sampler_class: Callable[[], BaseSampler], second_low: int, second_high: int ) -> None: def objective(trial: Trial, low: int, high: int) -> float: v = trial.suggest_float("x", low, high) v += trial.suggest_int("y", low, high) return v with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = sampler_class() study = optuna.study.create_study(sampler=sampler) study.optimize(lambda t: objective(t, 0, 10), n_trials=10) study.optimize(lambda t: objective(t, second_low, second_high), n_trials=10) assert len(study.trials) == 20 assert all(t.state == TrialState.COMPLETE for t in study.trials) # We add tests for constant objective functions to ensure the reproducibility of sorting. @parametrize_sampler_with_seed @pytest.mark.slow @pytest.mark.parametrize("objective_func", [lambda *args: sum(args), lambda *args: 0.0]) def test_reproducible(sampler_class: Callable[[int], BaseSampler], objective_func: Any) -> None: def objective(trial: Trial) -> float: a = trial.suggest_float("a", 1, 9) b = trial.suggest_float("b", 1, 9, log=True) c = trial.suggest_float("c", 1, 9, step=1) d = trial.suggest_int("d", 1, 9) e = trial.suggest_int("e", 1, 9, log=True) f = trial.suggest_int("f", 1, 9, step=2) g = trial.suggest_categorical("g", range(1, 10)) return objective_func(a, b, c, d, e, f, g) study = optuna.create_study(sampler=sampler_class(1)) study.optimize(objective, n_trials=15) study_same_seed = optuna.create_study(sampler=sampler_class(1)) study_same_seed.optimize(objective, n_trials=15) for i in range(15): assert study.trials[i].params == study_same_seed.trials[i].params study_different_seed = optuna.create_study(sampler=sampler_class(2)) study_different_seed.optimize(objective, n_trials=15) assert any( [study.trials[i].params != study_different_seed.trials[i].params for i in range(15)] ) @pytest.mark.slow @parametrize_sampler_with_seed def test_reseed_rng_change_sampling(sampler_class: Callable[[int], BaseSampler]) -> None: def objective(trial: Trial) -> float: a = trial.suggest_float("a", 1, 9) b = trial.suggest_float("b", 1, 9, log=True) c = trial.suggest_float("c", 1, 9, step=1) d = trial.suggest_int("d", 1, 9) e = trial.suggest_int("e", 1, 9, log=True) f = trial.suggest_int("f", 1, 9, step=2) g = trial.suggest_categorical("g", range(1, 10)) return a + b + c + d + e + f + g sampler = sampler_class(1) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=15) sampler_different_seed = sampler_class(1) sampler_different_seed.reseed_rng() study_different_seed = optuna.create_study(sampler=sampler_different_seed) study_different_seed.optimize(objective, n_trials=15) assert any( [study.trials[i].params != study_different_seed.trials[i].params for i in range(15)] ) # This function is used only in test_reproducible_in_other_process, but declared at top-level # because local function cannot be pickled, which occurs within multiprocessing. def run_optimize( k: int, sampler_name: str, sequence_dict: DictProxy, hash_dict: DictProxy, ) -> None: def objective(trial: Trial) -> float: a = trial.suggest_float("a", 1, 9) b = trial.suggest_float("b", 1, 9, log=True) c = trial.suggest_float("c", 1, 9, step=1) d = trial.suggest_int("d", 1, 9) e = trial.suggest_int("e", 1, 9, log=True) f = trial.suggest_int("f", 1, 9, step=2) g = trial.suggest_categorical("g", range(1, 10)) return a + b + c + d + e + f + g hash_dict[k] = hash("nondeterministic hash") sampler = sampler_class_with_seed[sampler_name](1) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=15) sequence_dict[k] = list(study.trials[-1].params.values()) @pytest.fixture def unset_seed_in_test(request: SubRequest) -> None: # Unset the hashseed at beginning and restore it at end regardless of an exception in the test. # See https://docs.pytest.org/en/stable/how-to/fixtures.html#adding-finalizers-directly # for details. hash_seed = os.getenv("PYTHONHASHSEED") if hash_seed is not None: del os.environ["PYTHONHASHSEED"] def restore_seed() -> None: if hash_seed is not None: os.environ["PYTHONHASHSEED"] = hash_seed request.addfinalizer(restore_seed) @pytest.mark.slow @parametrize_sampler_name_with_seed def test_reproducible_in_other_process(sampler_name: str, unset_seed_in_test: None) -> None: # This test should be tested without `PYTHONHASHSEED`. However, some tool such as tox # set the environmental variable "PYTHONHASHSEED" by default. # To do so, this test calls a finalizer: `unset_seed_in_test`. # Multiprocessing supports three way to start a process. # We use `spawn` option to create a child process as a fresh python process. # For more detail, see https://github.com/optuna/optuna/pull/3187#issuecomment-997673037. multiprocessing.set_start_method("spawn", force=True) manager = multiprocessing.Manager() sequence_dict: DictProxy = manager.dict() hash_dict: DictProxy = manager.dict() for i in range(3): p = multiprocessing.Process( target=run_optimize, args=(i, sampler_name, sequence_dict, hash_dict) ) p.start() p.join() # Hashes are expected to be different because string hashing is nondeterministic per process. assert not (hash_dict[0] == hash_dict[1] == hash_dict[2]) # But the sequences are expected to be the same. assert sequence_dict[0] == sequence_dict[1] == sequence_dict[2] @pytest.mark.parametrize("n_jobs", [1, 2]) @parametrize_relative_sampler def test_cache_is_invalidated( n_jobs: int, relative_sampler_class: Callable[[], BaseSampler] ) -> None: sampler = relative_sampler_class() study = optuna.study.create_study(sampler=sampler) def objective(trial: Trial) -> float: assert trial._relative_params is None assert study._thread_local.cached_all_trials is None trial.suggest_float("x", -10, 10) trial.suggest_float("y", -10, 10) assert trial._relative_params is not None return -1 study.optimize(objective, n_trials=10, n_jobs=n_jobs) optuna-4.1.0/tests/samplers_tests/tpe_tests/000077500000000000000000000000001471332314300212475ustar00rootroot00000000000000optuna-4.1.0/tests/samplers_tests/tpe_tests/__init__.py000066400000000000000000000000001471332314300233460ustar00rootroot00000000000000optuna-4.1.0/tests/samplers_tests/tpe_tests/test_multi_objective_sampler.py000066400000000000000000000441251471332314300275750ustar00rootroot00000000000000import random from typing import Callable from typing import Dict from typing import List from typing import Optional from typing import Union from unittest.mock import patch from unittest.mock import PropertyMock import numpy as np import pytest import optuna from optuna.samplers import _tpe from optuna.samplers import TPESampler class MockSystemAttr: def __init__(self) -> None: self.value: Dict[str, dict] = {} def set_trial_system_attr(self, _: int, key: str, value: dict) -> None: self.value[key] = value def suggest( sampler: optuna.samplers.BaseSampler, study: optuna.Study, trial: optuna.trial.FrozenTrial, distribution: optuna.distributions.BaseDistribution, past_trials: List[optuna.trial.FrozenTrial], ) -> float: attrs = MockSystemAttr() with patch.object(study._storage, "get_all_trials", return_value=past_trials), patch.object( study._storage, "set_trial_system_attr", side_effect=attrs.set_trial_system_attr ), patch.object(study._storage, "get_trial", return_value=trial), patch( "optuna.trial.Trial.system_attrs", new_callable=PropertyMock ) as mock1, patch( "optuna.trial.FrozenTrial.system_attrs", new_callable=PropertyMock, ) as mock2: mock1.return_value = attrs.value mock2.return_value = attrs.value suggestion = sampler.sample_independent(study, trial, "param-a", distribution) return suggestion def test_multi_objective_sample_independent_seed_fix() -> None: study = optuna.create_study(directions=["minimize", "maximize"]) dist = optuna.distributions.FloatDistribution(1.0, 100.0) random.seed(128) past_trials = [frozen_trial_factory(i, [random.random(), random.random()]) for i in range(16)] # Prepare a trial and a sample for later checks. trial = frozen_trial_factory(16, [0, 0]) sampler = TPESampler(seed=0) suggestion = suggest(sampler, study, trial, dist, past_trials) sampler = TPESampler(seed=0) assert suggest(sampler, study, trial, dist, past_trials) == suggestion sampler = TPESampler(seed=1) assert suggest(sampler, study, trial, dist, past_trials) != suggestion def test_multi_objective_sample_independent_prior() -> None: study = optuna.create_study(directions=["minimize", "maximize"]) dist = optuna.distributions.FloatDistribution(1.0, 100.0) random.seed(128) past_trials = [frozen_trial_factory(i, [random.random(), random.random()]) for i in range(16)] # Prepare a trial and a sample for later checks. trial = frozen_trial_factory(16, [0, 0]) sampler = TPESampler(seed=0) suggestion = suggest(sampler, study, trial, dist, past_trials) sampler = TPESampler(consider_prior=False, seed=0) assert suggest(sampler, study, trial, dist, past_trials) != suggestion sampler = TPESampler(prior_weight=0.5, seed=0) assert suggest(sampler, study, trial, dist, past_trials) != suggestion def test_multi_objective_sample_independent_n_startup_trial() -> None: study = optuna.create_study(directions=["minimize", "maximize"]) dist = optuna.distributions.FloatDistribution(1.0, 100.0) random.seed(128) past_trials = [frozen_trial_factory(i, [random.random(), random.random()]) for i in range(16)] trial = frozen_trial_factory(16, [0, 0]) def _suggest_and_return_call_count( sampler: optuna.samplers.BaseSampler, past_trials: List[optuna.trial.FrozenTrial], ) -> int: attrs = MockSystemAttr() with patch.object( study._storage, "get_all_trials", return_value=past_trials ), patch.object( study._storage, "set_trial_system_attr", side_effect=attrs.set_trial_system_attr ), patch.object( study._storage, "get_trial", return_value=trial ), patch( "optuna.trial.Trial.system_attrs", new_callable=PropertyMock ) as mock1, patch( "optuna.trial.FrozenTrial.system_attrs", new_callable=PropertyMock, ) as mock2, patch.object( optuna.samplers.RandomSampler, "sample_independent", return_value=1.0, ) as sample_method: mock1.return_value = attrs.value mock2.return_value = attrs.value sampler.sample_independent(study, trial, "param-a", dist) study._thread_local.cached_all_trials = None return sample_method.call_count sampler = TPESampler(n_startup_trials=16, seed=0) assert _suggest_and_return_call_count(sampler, past_trials[:-1]) == 1 sampler = TPESampler(n_startup_trials=16, seed=0) assert _suggest_and_return_call_count(sampler, past_trials) == 0 def test_multi_objective_sample_independent_misc_arguments() -> None: study = optuna.create_study(directions=["minimize", "maximize"]) dist = optuna.distributions.FloatDistribution(1.0, 100.0) random.seed(128) past_trials = [frozen_trial_factory(i, [random.random(), random.random()]) for i in range(32)] # Prepare a trial and a sample for later checks. trial = frozen_trial_factory(16, [0, 0]) sampler = TPESampler(seed=0) suggestion = suggest(sampler, study, trial, dist, past_trials) # Test misc. parameters. sampler = TPESampler(n_ei_candidates=13, seed=0) assert suggest(sampler, study, trial, dist, past_trials) != suggestion sampler = TPESampler(gamma=lambda _: 1, seed=0) assert suggest(sampler, study, trial, dist, past_trials) != suggestion @pytest.mark.parametrize("log, step", [(False, None), (True, None), (False, 0.1)]) def test_multi_objective_sample_independent_float_distributions( log: bool, step: Optional[float] ) -> None: # Prepare sample from float distribution for checking other distributions. study = optuna.create_study(directions=["minimize", "maximize"]) random.seed(128) float_dist = optuna.distributions.FloatDistribution(1.0, 100.0, log=log, step=step) if float_dist.step: value_fn: Optional[Callable[[int], float]] = ( lambda number: int(random.random() * 1000) * 0.1 ) else: value_fn = None past_trials = [ frozen_trial_factory( i, [random.random(), random.random()], dist=float_dist, value_fn=value_fn ) for i in range(16) ] trial = frozen_trial_factory(16, [0, 0]) sampler = TPESampler(seed=0) float_suggestion = suggest(sampler, study, trial, float_dist, past_trials) assert 1.0 <= float_suggestion < 100.0 if float_dist.step == 0.1: assert abs(int(float_suggestion * 10) - float_suggestion * 10) < 1e-3 # Test sample is different when `float_dist.log` is True or float_dist.step != 1.0. random.seed(128) dist = optuna.distributions.FloatDistribution(1.0, 100.0) past_trials = [frozen_trial_factory(i, [random.random(), random.random()]) for i in range(16)] trial = frozen_trial_factory(16, [0, 0]) sampler = TPESampler(seed=0) suggestion = suggest(sampler, study, trial, dist, past_trials) if float_dist.log or float_dist.step == 0.1: assert float_suggestion != suggestion else: assert float_suggestion == suggestion def test_multi_objective_sample_independent_categorical_distributions() -> None: """Test samples are drawn from the specified category.""" study = optuna.create_study(directions=["minimize", "maximize"]) random.seed(128) categories = [i * 0.3 + 1.0 for i in range(330)] def cat_value_fn(idx: int) -> float: return categories[random.randint(0, len(categories) - 1)] cat_dist = optuna.distributions.CategoricalDistribution(categories) past_trials = [ frozen_trial_factory( i, [random.random(), random.random()], dist=cat_dist, value_fn=cat_value_fn ) for i in range(16) ] trial = frozen_trial_factory(16, [0, 0]) sampler = TPESampler(seed=0) categorical_suggestion = suggest(sampler, study, trial, cat_dist, past_trials) assert categorical_suggestion in categories @pytest.mark.parametrize( "log, step", [ (False, 1), (True, 1), (False, 2), ], ) def test_multi_objective_sample_int_distributions(log: bool, step: int) -> None: """Test sampling from int distribution returns integer.""" study = optuna.create_study(directions=["minimize", "maximize"]) random.seed(128) def int_value_fn(idx: int) -> float: return random.randint(1, 99) int_dist = optuna.distributions.IntDistribution(1, 99, log, step) past_trials = [ frozen_trial_factory( i, [random.random(), random.random()], dist=int_dist, value_fn=int_value_fn ) for i in range(16) ] trial = frozen_trial_factory(16, [0, 0]) sampler = TPESampler(seed=0) int_suggestion = suggest(sampler, study, trial, int_dist, past_trials) assert 1 <= int_suggestion <= 99 assert isinstance(int_suggestion, int) @pytest.mark.parametrize( "state", [ (optuna.trial.TrialState.FAIL,), (optuna.trial.TrialState.PRUNED,), (optuna.trial.TrialState.RUNNING,), (optuna.trial.TrialState.WAITING,), ], ) def test_multi_objective_sample_independent_handle_unsuccessful_states( state: optuna.trial.TrialState, ) -> None: study = optuna.create_study(directions=["minimize", "maximize"]) dist = optuna.distributions.FloatDistribution(1.0, 100.0) random.seed(128) # Prepare sampling result for later tests. past_trials = [frozen_trial_factory(i, [random.random(), random.random()]) for i in range(32)] trial = frozen_trial_factory(32, [0, 0]) sampler = TPESampler(seed=0) all_success_suggestion = suggest(sampler, study, trial, dist, past_trials) study._thread_local.cached_all_trials = None # Test unsuccessful trials are handled differently. state_fn = build_state_fn(state) past_trials = [ frozen_trial_factory(i, [random.random(), random.random()], state_fn=state_fn) for i in range(32) ] trial = frozen_trial_factory(32, [0, 0]) sampler = TPESampler(seed=0) partial_unsuccessful_suggestion = suggest(sampler, study, trial, dist, past_trials) assert partial_unsuccessful_suggestion != all_success_suggestion def test_multi_objective_sample_independent_ignored_states() -> None: """Tests FAIL, RUNNING, and WAITING states are equally.""" study = optuna.create_study(directions=["minimize", "maximize"]) dist = optuna.distributions.FloatDistribution(1.0, 100.0) suggestions = [] for state in [ optuna.trial.TrialState.FAIL, optuna.trial.TrialState.RUNNING, optuna.trial.TrialState.WAITING, ]: random.seed(128) state_fn = build_state_fn(state) past_trials = [ frozen_trial_factory(i, [random.random(), random.random()], state_fn=state_fn) for i in range(32) ] trial = frozen_trial_factory(32, [0, 0]) sampler = TPESampler(seed=0) suggestions.append(suggest(sampler, study, trial, dist, past_trials)) assert len(set(suggestions)) == 1 @pytest.mark.parametrize("direction0", ["minimize", "maximize"]) @pytest.mark.parametrize("direction1", ["minimize", "maximize"]) def test_split_complete_trials_multi_objective(direction0: str, direction1: str) -> None: study = optuna.create_study(directions=(direction0, direction1)) for values in ([-2.0, -1.0], [3.0, 3.0], [0.0, 1.0], [-1.0, 0.0]): value0, value1 = values if direction0 == "maximize": value0 = -value0 if direction1 == "maximize": value1 = -value1 study.add_trial( optuna.create_trial( state=optuna.trial.TrialState.COMPLETE, values=(value0, value1), params={"x": 0}, distributions={"x": optuna.distributions.FloatDistribution(-1.0, 1.0)}, ) ) below_trials, above_trials = _tpe.sampler._split_complete_trials_multi_objective( study.trials, study, 2, ) assert [trial.number for trial in below_trials] == [0, 3] assert [trial.number for trial in above_trials] == [1, 2] def test_split_complete_trials_multi_objective_empty() -> None: study = optuna.create_study(directions=("minimize", "minimize")) assert _tpe.sampler._split_complete_trials_multi_objective([], study, 0) == ([], []) def test_calculate_weights_below_for_multi_objective() -> None: # No sample. study = optuna.create_study(directions=["minimize", "minimize"]) weights_below = _tpe.sampler._calculate_weights_below_for_multi_objective(study, [], None) assert len(weights_below) == 0 # One sample. study = optuna.create_study(directions=["minimize", "minimize"]) trial0 = optuna.create_trial(values=[0.2, 0.5]) study.add_trials([trial0]) weights_below = _tpe.sampler._calculate_weights_below_for_multi_objective( study, [trial0], None ) assert len(weights_below) == 1 assert sum(weights_below) > 0 # Two samples. study = optuna.create_study(directions=["minimize", "minimize"]) trial0 = optuna.create_trial(values=[0.2, 0.5]) trial1 = optuna.create_trial(values=[0.9, 0.4]) study.add_trials([trial0, trial1]) weights_below = _tpe.sampler._calculate_weights_below_for_multi_objective( study, [trial0, trial1], None, ) assert len(weights_below) == 2 assert weights_below[0] > weights_below[1] assert sum(weights_below) > 0 # Two equally contributed samples. study = optuna.create_study(directions=["minimize", "minimize"]) trial0 = optuna.create_trial(values=[0.2, 0.8]) trial1 = optuna.create_trial(values=[0.8, 0.2]) study.add_trials([trial0, trial1]) weights_below = _tpe.sampler._calculate_weights_below_for_multi_objective( study, [trial0, trial1], None, ) assert len(weights_below) == 2 assert weights_below[0] == weights_below[1] assert sum(weights_below) > 0 # Duplicated samples. study = optuna.create_study(directions=["minimize", "minimize"]) trial0 = optuna.create_trial(values=[0.2, 0.8]) trial1 = optuna.create_trial(values=[0.2, 0.8]) study.add_trials([trial0, trial1]) weights_below = _tpe.sampler._calculate_weights_below_for_multi_objective( study, [trial0, trial1], None, ) assert len(weights_below) == 2 assert weights_below[0] == weights_below[1] assert sum(weights_below) > 0 # Three samples. study = optuna.create_study(directions=["minimize", "minimize"]) trial0 = optuna.create_trial(values=[0.3, 0.3]) trial1 = optuna.create_trial(values=[0.2, 0.8]) trial2 = optuna.create_trial(values=[0.8, 0.2]) study.add_trials([trial0, trial1, trial2]) weights_below = _tpe.sampler._calculate_weights_below_for_multi_objective( study, [trial0, trial1, trial2], None, ) assert len(weights_below) == 3 assert weights_below[0] > weights_below[1] assert weights_below[0] > weights_below[2] assert weights_below[1] == weights_below[2] assert sum(weights_below) > 0 # Zero/negative objective values. study = optuna.create_study(directions=["minimize", "minimize"]) trial0 = optuna.create_trial(values=[-0.3, -0.3]) trial1 = optuna.create_trial(values=[0.0, -0.8]) trial2 = optuna.create_trial(values=[-0.8, 0.0]) study.add_trials([trial0, trial1, trial2]) weights_below = _tpe.sampler._calculate_weights_below_for_multi_objective( study, [trial0, trial1, trial2], None, ) assert len(weights_below) == 3 assert weights_below[0] > weights_below[1] assert weights_below[0] > weights_below[2] assert np.isclose(weights_below[1], weights_below[2]) assert sum(weights_below) > 0 # +/-inf objective values. study = optuna.create_study(directions=["minimize", "minimize"]) trial0 = optuna.create_trial(values=[-float("inf"), -float("inf")]) trial1 = optuna.create_trial(values=[0.0, -float("inf")]) trial2 = optuna.create_trial(values=[-float("inf"), 0.0]) study.add_trials([trial0, trial1, trial2]) weights_below = _tpe.sampler._calculate_weights_below_for_multi_objective( study, [trial0, trial1, trial2], None, ) assert len(weights_below) == 3 assert not any([np.isnan(w) for w in weights_below]) assert sum(weights_below) > 0 # Three samples with two infeasible trials. study = optuna.create_study(directions=["minimize", "minimize"]) trial0 = optuna.create_trial(values=[0.3, 0.3], system_attrs={"constraints": 2}) trial1 = optuna.create_trial(values=[0.2, 0.8], system_attrs={"constraints": 8}) trial2 = optuna.create_trial(values=[0.8, 0.2], system_attrs={"constraints": 0}) study.add_trials([trial0, trial1, trial2]) weights_below = _tpe.sampler._calculate_weights_below_for_multi_objective( study, [trial0, trial1, trial2], lambda trial: [trial.system_attrs["constraints"]], ) assert len(weights_below) == 3 assert weights_below[0] == _tpe.sampler.EPS assert weights_below[1] == _tpe.sampler.EPS assert weights_below[2] > 0 def frozen_trial_factory( number: int, values: List[float], dist: optuna.distributions.BaseDistribution = optuna.distributions.FloatDistribution( 1.0, 100.0 ), value_fn: Optional[Callable[[int], Union[int, float]]] = None, state_fn: Callable[ [int], optuna.trial.TrialState ] = lambda _: optuna.trial.TrialState.COMPLETE, ) -> optuna.trial.FrozenTrial: if value_fn is None: value = random.random() * 99.0 + 1.0 else: value = value_fn(number) trial = optuna.trial.FrozenTrial( number=number, trial_id=number, state=optuna.trial.TrialState.COMPLETE, value=None, datetime_start=None, datetime_complete=None, params={"param-a": value}, distributions={"param-a": dist}, user_attrs={}, system_attrs={}, intermediate_values={}, values=values, ) return trial def build_state_fn(state: optuna.trial.TrialState) -> Callable[[int], optuna.trial.TrialState]: def state_fn(idx: int) -> optuna.trial.TrialState: return [optuna.trial.TrialState.COMPLETE, state][idx % 2] return state_fn optuna-4.1.0/tests/samplers_tests/tpe_tests/test_parzen_estimator.py000066400000000000000000000341021471332314300262460ustar00rootroot00000000000000from typing import Callable from typing import Dict from typing import List import numpy as np import pytest from optuna import distributions from optuna.distributions import CategoricalChoiceType from optuna.samplers._tpe.parzen_estimator import _ParzenEstimator from optuna.samplers._tpe.parzen_estimator import _ParzenEstimatorParameters from optuna.samplers._tpe.probability_distributions import _BatchedCategoricalDistributions from optuna.samplers._tpe.probability_distributions import _BatchedDiscreteTruncNormDistributions from optuna.samplers._tpe.probability_distributions import _BatchedTruncNormDistributions from optuna.samplers._tpe.probability_distributions import _MixtureOfProductDistribution from optuna.samplers._tpe.sampler import default_weights def assert_distribution_almost_equal( d1: _MixtureOfProductDistribution, d2: _MixtureOfProductDistribution ) -> None: np.testing.assert_almost_equal(d1.weights, d2.weights) for d1_, d2_ in zip(d1.distributions, d2.distributions): assert type(d1_) is type(d2_) for field1, field2 in zip(d1_, d2_): np.testing.assert_almost_equal(np.array(field1), np.array(field2)) SEARCH_SPACE = { "a": distributions.FloatDistribution(1.0, 100.0), "b": distributions.FloatDistribution(1.0, 100.0, log=True), "c": distributions.FloatDistribution(1.0, 100.0, step=3.0), "d": distributions.IntDistribution(1, 100), "e": distributions.IntDistribution(1, 100, log=True), "f": distributions.CategoricalDistribution(["x", "y", "z"]), "g": distributions.CategoricalDistribution([0.0, float("inf"), float("nan"), None]), } MULTIVARIATE_SAMPLES = { "a": np.array([1.0]), "b": np.array([1.0]), "c": np.array([1.0]), "d": np.array([1]), "e": np.array([1]), "f": np.array([1]), "g": np.array([1]), } @pytest.mark.parametrize("consider_prior", [True, False]) @pytest.mark.parametrize("multivariate", [True, False]) def test_init_parzen_estimator(consider_prior: bool, multivariate: bool) -> None: parameters = _ParzenEstimatorParameters( consider_prior=consider_prior, prior_weight=1.0, consider_magic_clip=False, consider_endpoints=False, weights=lambda x: np.arange(x) + 1.0, multivariate=multivariate, categorical_distance_func={}, ) mpe = _ParzenEstimator(MULTIVARIATE_SAMPLES, SEARCH_SPACE, parameters) weights = np.array([1] + consider_prior * [1], dtype=float) weights /= weights.sum() expected_univariate = _MixtureOfProductDistribution( weights=weights, distributions=[ _BatchedTruncNormDistributions( mu=np.array([1.0] + consider_prior * [50.5]), sigma=np.array([49.5 if consider_prior else 99.0] + consider_prior * [99.0]), low=1.0, high=100.0, ), _BatchedTruncNormDistributions( mu=np.array([np.log(1.0)] + consider_prior * [np.log(100) / 2.0]), sigma=np.array( [np.log(100) / 2 if consider_prior else np.log(100.0)] + consider_prior * [np.log(100)] ), low=np.log(1.0), high=np.log(100.0), ), _BatchedDiscreteTruncNormDistributions( mu=np.array([1.0] + consider_prior * [50.5]), sigma=np.array([49.5 if consider_prior else 100.5] + consider_prior * [102.0]), low=1.0, high=100.0, step=3.0, ), _BatchedDiscreteTruncNormDistributions( mu=np.array([1.0] + consider_prior * [50.5]), sigma=np.array([49.5 if consider_prior else 99.5] + consider_prior * [100.0]), low=1, high=100, step=1, ), _BatchedTruncNormDistributions( mu=np.array( [np.log(1.0)] + consider_prior * [(np.log(100.5) + np.log(0.5)) / 2.0] ), sigma=np.array( [(np.log(100.5) + np.log(0.5)) / 2 if consider_prior else np.log(100.5)] + consider_prior * [np.log(100.5) - np.log(0.5)] ), low=np.log(0.5), high=np.log(100.5), ), _BatchedCategoricalDistributions( np.array([[0.2, 0.6, 0.2], [1.0 / 3.0, 1.0 / 3.0, 1.0 / 3.0]]) if consider_prior else np.array([[0.25, 0.5, 0.25]]) ), _BatchedCategoricalDistributions( np.array( [ [1.0 / 6.0, 0.5, 1.0 / 6.0, 1.0 / 6.0], [1.0 / 4.0, 1.0 / 4.0, 1.0 / 4.0, 1.0 / 4.0], ] ) if consider_prior else np.array([[0.2, 0.4, 0.2, 0.2]]) ), ], ) SIGMA0 = 0.2 expected_multivarite = _MixtureOfProductDistribution( weights=weights, distributions=[ _BatchedTruncNormDistributions( mu=np.array([1.0] + consider_prior * [50.5]), sigma=np.array([SIGMA0 * 99.0] + consider_prior * [99.0]), low=1.0, high=100.0, ), _BatchedTruncNormDistributions( mu=np.array([np.log(1.0)] + consider_prior * [np.log(100) / 2.0]), sigma=np.array([SIGMA0 * np.log(100)] + consider_prior * [np.log(100)]), low=np.log(1.0), high=np.log(100.0), ), _BatchedDiscreteTruncNormDistributions( mu=np.array([1.0] + consider_prior * [50.5]), sigma=np.array([SIGMA0 * 102.0] + consider_prior * [102.0]), low=1.0, high=100.0, step=3.0, ), _BatchedDiscreteTruncNormDistributions( mu=np.array([1.0] + consider_prior * [50.5]), sigma=np.array([SIGMA0 * 100.0] + consider_prior * [100.0]), low=1, high=100, step=1, ), _BatchedTruncNormDistributions( mu=np.array( [np.log(1.0)] + consider_prior * [(np.log(100.5) + np.log(0.5)) / 2.0] ), sigma=np.array( [SIGMA0 * (np.log(100.5) - np.log(0.5))] + consider_prior * [np.log(100.5) - np.log(0.5)] ), low=np.log(0.5), high=np.log(100.5), ), _BatchedCategoricalDistributions( np.array([[0.2, 0.6, 0.2], [1.0 / 3.0, 1.0 / 3.0, 1.0 / 3.0]]) if consider_prior else np.array([[0.25, 0.5, 0.25]]) ), _BatchedCategoricalDistributions( np.array( [ [1.0 / 6.0, 0.5, 1.0 / 6.0, 1.0 / 6.0], [1.0 / 4.0, 1.0 / 4.0, 1.0 / 4.0, 1.0 / 4.0], ] if consider_prior else np.array([[0.2, 0.4, 0.2, 0.2]]) ) ), ], ) expected = expected_multivarite if multivariate else expected_univariate # Test that the distribution is correct. assert_distribution_almost_equal(mpe._mixture_distribution, expected) # Test that the sampled values are valid. samples = mpe.sample(np.random.RandomState(0), 10) for param, values in samples.items(): for value in values: assert SEARCH_SPACE[param]._contains(value) @pytest.mark.parametrize("mus", (np.asarray([]), np.asarray([0.4]), np.asarray([-0.4, 0.4]))) @pytest.mark.parametrize("prior_weight", [1.0, 0.01, 100.0]) @pytest.mark.parametrize("prior", (True, False)) @pytest.mark.parametrize("magic_clip", (True, False)) @pytest.mark.parametrize("endpoints", (True, False)) @pytest.mark.parametrize("multivariate", (True, False)) def test_calculate_shape_check( mus: np.ndarray, prior_weight: float, prior: bool, magic_clip: bool, endpoints: bool, multivariate: bool, ) -> None: parameters = _ParzenEstimatorParameters( prior_weight=prior_weight, consider_prior=prior, consider_magic_clip=magic_clip, consider_endpoints=endpoints, weights=default_weights, multivariate=multivariate, categorical_distance_func={}, ) mpe = _ParzenEstimator( {"a": mus}, {"a": distributions.FloatDistribution(-1.0, 1.0)}, parameters ) assert len(mpe._mixture_distribution.weights) == max(len(mus) + int(prior), 1) @pytest.mark.parametrize("mus", (np.asarray([]), np.asarray([0.4]), np.asarray([-0.4, 0.4]))) @pytest.mark.parametrize("prior_weight", [1.0, 0.01, 100.0]) @pytest.mark.parametrize("prior", (True, False)) @pytest.mark.parametrize("categorical_distance_func", ({}, {"c": lambda x, y: abs(x - y)})) def test_calculate_shape_check_categorical( mus: np.ndarray, prior_weight: float, prior: bool, categorical_distance_func: Dict[ str, Callable[[CategoricalChoiceType, CategoricalChoiceType], float], ], ) -> None: parameters = _ParzenEstimatorParameters( prior_weight=prior_weight, consider_prior=prior, consider_magic_clip=True, consider_endpoints=False, weights=default_weights, multivariate=False, categorical_distance_func=categorical_distance_func, ) mpe = _ParzenEstimator( {"c": mus}, {"c": distributions.CategoricalDistribution([0.0, 1.0, 2.0])}, parameters ) assert len(mpe._mixture_distribution.weights) == max(len(mus) + int(prior), 1) @pytest.mark.parametrize("prior_weight", [None, -1.0, 0.0]) @pytest.mark.parametrize("mus", (np.asarray([]), np.asarray([0.4]), np.asarray([-0.4, 0.4]))) def test_invalid_prior_weight(prior_weight: float, mus: np.ndarray) -> None: parameters = _ParzenEstimatorParameters( prior_weight=prior_weight, consider_prior=True, consider_magic_clip=False, consider_endpoints=False, weights=default_weights, multivariate=False, categorical_distance_func={}, ) with pytest.raises(ValueError): _ParzenEstimator({"a": mus}, {"a": distributions.FloatDistribution(-1.0, 1.0)}, parameters) # TODO(ytsmiling): Improve test coverage for weights. @pytest.mark.parametrize( "mus, flags, expected", [ [ np.asarray([]), {"prior": False, "magic_clip": False, "endpoints": True}, {"weights": [1.0], "mus": [0.0], "sigmas": [2.0]}, ], [ np.asarray([]), {"prior": True, "magic_clip": False, "endpoints": True}, {"weights": [1.0], "mus": [0.0], "sigmas": [2.0]}, ], [ np.asarray([0.4]), {"prior": True, "magic_clip": False, "endpoints": True}, {"weights": [0.5, 0.5], "mus": [0.4, 0.0], "sigmas": [0.6, 2.0]}, ], [ np.asarray([-0.4]), {"prior": True, "magic_clip": False, "endpoints": True}, {"weights": [0.5, 0.5], "mus": [-0.4, 0.0], "sigmas": [0.6, 2.0]}, ], [ np.asarray([-0.4, 0.4]), {"prior": True, "magic_clip": False, "endpoints": True}, {"weights": [1.0 / 3] * 3, "mus": [-0.4, 0.4, 0.0], "sigmas": [0.6, 0.6, 2.0]}, ], [ np.asarray([-0.4, 0.4]), {"prior": True, "magic_clip": False, "endpoints": False}, {"weights": [1.0 / 3] * 3, "mus": [-0.4, 0.4, 0.0], "sigmas": [0.4, 0.4, 2.0]}, ], [ np.asarray([-0.4, 0.4]), {"prior": False, "magic_clip": False, "endpoints": True}, {"weights": [0.5, 0.5], "mus": [-0.4, 0.4], "sigmas": [0.8, 0.8]}, ], [ np.asarray([-0.4, 0.4, 0.41, 0.42]), {"prior": False, "magic_clip": False, "endpoints": True}, { "weights": [0.25, 0.25, 0.25, 0.25], "mus": [-0.4, 0.4, 0.41, 0.42], "sigmas": [0.8, 0.8, 0.01, 0.58], }, ], [ np.asarray([-0.4, 0.4, 0.41, 0.42]), {"prior": False, "magic_clip": True, "endpoints": True}, { "weights": [0.25, 0.25, 0.25, 0.25], "mus": [-0.4, 0.4, 0.41, 0.42], "sigmas": [0.8, 0.8, 0.4, 0.58], }, ], ], ) def test_calculate( mus: np.ndarray, flags: Dict[str, bool], expected: Dict[str, List[float]] ) -> None: parameters = _ParzenEstimatorParameters( prior_weight=1.0, consider_prior=flags["prior"], consider_magic_clip=flags["magic_clip"], consider_endpoints=flags["endpoints"], weights=default_weights, multivariate=False, categorical_distance_func={}, ) mpe = _ParzenEstimator( {"a": mus}, {"a": distributions.FloatDistribution(-1.0, 1.0)}, parameters ) expected_distribution = _MixtureOfProductDistribution( weights=np.asarray(expected["weights"]), distributions=[ _BatchedTruncNormDistributions( mu=np.asarray(expected["mus"]), sigma=np.asarray(expected["sigmas"]), low=-1.0, high=1.0, ) ], ) assert_distribution_almost_equal(mpe._mixture_distribution, expected_distribution) @pytest.mark.parametrize( "weights", [ lambda x: np.zeros(x), lambda x: -np.ones(x), lambda x: float("inf") * np.ones(x), lambda x: -float("inf") * np.ones(x), lambda x: np.asarray([float("nan") for _ in range(x)]), ], ) def test_invalid_weights(weights: Callable[[int], np.ndarray]) -> None: parameters = _ParzenEstimatorParameters( prior_weight=1.0, consider_prior=False, consider_magic_clip=False, consider_endpoints=False, weights=weights, multivariate=False, categorical_distance_func={}, ) with pytest.raises(ValueError): _ParzenEstimator( {"a": np.asarray([0.0])}, {"a": distributions.FloatDistribution(-1.0, 1.0)}, parameters ) optuna-4.1.0/tests/samplers_tests/tpe_tests/test_probability_distributions.py000066400000000000000000000063131471332314300301650ustar00rootroot00000000000000import warnings import numpy as np from optuna.samplers._tpe.probability_distributions import _BatchedCategoricalDistributions from optuna.samplers._tpe.probability_distributions import _BatchedDiscreteTruncNormDistributions from optuna.samplers._tpe.probability_distributions import _BatchedTruncNormDistributions from optuna.samplers._tpe.probability_distributions import _MixtureOfProductDistribution def test_mixture_of_product_distribution() -> None: dist0 = _BatchedTruncNormDistributions( mu=np.array([0.2, 3.0]), sigma=np.array([0.8, 1.0]), low=-1.0, high=1.0, ) dist1 = _BatchedDiscreteTruncNormDistributions( mu=np.array([0.0, 1.0]), sigma=np.array([1.0, 1.0]), low=-1.0, high=0.5, step=0.5, ) dist2 = _BatchedCategoricalDistributions(weights=np.array([[0.4, 0.6], [0.2, 0.8]])) mixture_distribution = _MixtureOfProductDistribution( weights=np.array([0.5, 0.5]), distributions=[dist0, dist1, dist2], ) samples = mixture_distribution.sample(np.random.RandomState(0), 5) # Test that the shapes are correct. assert samples.shape == (5, 3) # Test that the samples are in the valid range. assert np.all(dist0.low <= samples[:, 0]) assert np.all(samples[:, 0] <= dist0.high) assert np.all(dist1.low <= samples[:, 1]) assert np.all(samples[:, 1] <= dist1.high) np.testing.assert_almost_equal( np.fmod( samples[:, 1] - dist1.low, dist1.step, ), 0.0, ) assert np.all(0 <= samples[:, 2]) assert np.all(samples[:, 2] <= 1) assert np.all(np.fmod(samples[:, 2], 1.0) == 0.0) # Test reproducibility. assert np.all(samples == mixture_distribution.sample(np.random.RandomState(0), 5)) assert not np.all(samples == mixture_distribution.sample(np.random.RandomState(1), 5)) log_pdf = mixture_distribution.log_pdf(samples) assert log_pdf.shape == (5,) def test_mixture_of_product_distribution_extreme_case() -> None: rng = np.random.RandomState(0) mixture_distribution = _MixtureOfProductDistribution( weights=np.array([1.0, 0.0]), distributions=[ _BatchedTruncNormDistributions( mu=np.array([0.5, 0.3]), sigma=np.array([1e-10, 1.0]), low=-1.0, high=1.0, ), _BatchedDiscreteTruncNormDistributions( mu=np.array([-0.5, 1.0]), sigma=np.array([1e-10, 1.0]), low=-1.0, high=0.5, step=0.5, ), _BatchedCategoricalDistributions(weights=np.array([[0, 1], [0.2, 0.8]])), ], ) samples = mixture_distribution.sample(rng, 2) np.testing.assert_almost_equal(samples, np.array([[0.5, -0.5, 1.0]] * 2)) # The first point has the highest probability, # and all other points have probability almost zero. x = np.array([[0.5, 0.5, 1.0], [0.1, 0.5, 1.0], [0.5, 0.0, 1.0], [0.5, 0.5, 0.0]]) with warnings.catch_warnings(): warnings.simplefilter("ignore", RuntimeWarning) # Ignore log(0) warnings. log_pdf = mixture_distribution.log_pdf(x) assert np.all(log_pdf[1:] < -100) optuna-4.1.0/tests/samplers_tests/tpe_tests/test_sampler.py000066400000000000000000001357451471332314300243420ustar00rootroot00000000000000from __future__ import annotations import random from typing import Callable from typing import Dict from typing import Optional from typing import Union from unittest.mock import Mock from unittest.mock import patch import warnings import _pytest.capture import numpy as np import pytest import optuna from optuna import distributions from optuna.samplers import _tpe from optuna.samplers import TPESampler from optuna.samplers._base import _CONSTRAINTS_KEY from optuna.trial import Trial @pytest.mark.parametrize("use_hyperband", [False, True]) def test_hyperopt_parameters(use_hyperband: bool) -> None: sampler = TPESampler(**TPESampler.hyperopt_parameters()) study = optuna.create_study( sampler=sampler, pruner=optuna.pruners.HyperbandPruner() if use_hyperband else None ) study.optimize(lambda t: t.suggest_float("x", 10, 20), n_trials=50) def test_multivariate_experimental_warning() -> None: with pytest.warns(optuna.exceptions.ExperimentalWarning): optuna.samplers.TPESampler(multivariate=True) def test_constraints_func_experimental_warning() -> None: with pytest.warns(optuna.exceptions.ExperimentalWarning): optuna.samplers.TPESampler(constraints_func=lambda _: (0,)) def test_warn_independent_sampling(capsys: _pytest.capture.CaptureFixture) -> None: def objective(trial: Trial) -> float: x = trial.suggest_categorical("x", ["a", "b"]) if x == "a": return trial.suggest_float("y", 0, 1) else: return trial.suggest_float("z", 0, 1) # We need to reconstruct our default handler to properly capture stderr. optuna.logging._reset_library_root_logger() optuna.logging.enable_default_handler() optuna.logging.set_verbosity(optuna.logging.WARNING) sampler = TPESampler(multivariate=True, warn_independent_sampling=True, n_startup_trials=0) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=10) _, err = capsys.readouterr() assert err def test_warn_independent_sampling_group(capsys: _pytest.capture.CaptureFixture) -> None: def objective(trial: Trial) -> float: x = trial.suggest_categorical("x", ["a", "b"]) if x == "a": return trial.suggest_float("y", 0, 1) else: return trial.suggest_float("z", 0, 1) # We need to reconstruct our default handler to properly capture stderr. optuna.logging._reset_library_root_logger() optuna.logging.enable_default_handler() optuna.logging.set_verbosity(optuna.logging.WARNING) sampler = TPESampler( multivariate=True, warn_independent_sampling=True, group=True, n_startup_trials=0 ) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=10) _, err = capsys.readouterr() assert err == "" def test_infer_relative_search_space() -> None: sampler = TPESampler() search_space = { "a": distributions.FloatDistribution(1.0, 100.0), "b": distributions.FloatDistribution(1.0, 100.0, log=True), "c": distributions.FloatDistribution(1.0, 100.0, step=3.0), "d": distributions.IntDistribution(1, 100), "e": distributions.IntDistribution(0, 100, step=2), "f": distributions.IntDistribution(1, 100, log=True), "g": distributions.CategoricalDistribution(["x", "y", "z"]), } def obj(t: Trial) -> float: t.suggest_float("a", 1.0, 100.0) t.suggest_float("b", 1.0, 100.0, log=True) t.suggest_float("c", 1.0, 100.0, step=3.0) t.suggest_int("d", 1, 100) t.suggest_int("e", 0, 100, step=2) t.suggest_int("f", 1, 100, log=True) t.suggest_categorical("g", ["x", "y", "z"]) return 0.0 # Study and frozen-trial are not supposed to be accessed. study1 = Mock(spec=[]) frozen_trial = Mock(spec=[]) assert sampler.infer_relative_search_space(study1, frozen_trial) == {} study2 = optuna.create_study(sampler=sampler) study2.optimize(obj, n_trials=1) assert sampler.infer_relative_search_space(study2, study2.best_trial) == {} with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = TPESampler(multivariate=True) study3 = optuna.create_study(sampler=sampler) study3.optimize(obj, n_trials=1) assert sampler.infer_relative_search_space(study3, study3.best_trial) == search_space @pytest.mark.parametrize("multivariate", [False, True]) def test_sample_relative_empty_input(multivariate: bool) -> None: with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = TPESampler(multivariate=multivariate) # A frozen-trial is not supposed to be accessed. study = optuna.create_study() frozen_trial = Mock(spec=[]) assert sampler.sample_relative(study, frozen_trial, {}) == {} def test_sample_relative_prior() -> None: study = optuna.create_study() dist = optuna.distributions.FloatDistribution(1.0, 100.0) past_trials = [frozen_trial_factory(i, dist=dist) for i in range(1, 8)] # Prepare a trial and a sample for later checks. trial = frozen_trial_factory(8) with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = TPESampler(n_startup_trials=5, seed=0, multivariate=True) with patch.object(study._storage, "get_all_trials", return_value=past_trials): suggestion = sampler.sample_relative(study, trial, {"param-a": dist}) with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = TPESampler(consider_prior=False, n_startup_trials=5, seed=0, multivariate=True) with patch.object(study._storage, "get_all_trials", return_value=past_trials): assert sampler.sample_relative(study, trial, {"param-a": dist}) != suggestion with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = TPESampler(prior_weight=0.2, n_startup_trials=5, seed=0, multivariate=True) with patch.object(study._storage, "get_all_trials", return_value=past_trials): assert sampler.sample_relative(study, trial, {"param-a": dist}) != suggestion def test_sample_relative_n_startup_trial() -> None: study = optuna.create_study() dist = optuna.distributions.FloatDistribution(1.0, 100.0) past_trials = [frozen_trial_factory(i, dist=dist) for i in range(1, 8)] trial = frozen_trial_factory(8) # sample_relative returns {} for only 4 observations. with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = TPESampler(n_startup_trials=5, seed=0, multivariate=True) with patch.object(study._storage, "get_all_trials", return_value=past_trials[:4]): assert sampler.sample_relative(study, trial, {"param-a": dist}) == {} # sample_relative returns some value for only 7 observations. study._thread_local.cached_all_trials = None with patch.object(study._storage, "get_all_trials", return_value=past_trials): assert "param-a" in sampler.sample_relative(study, trial, {"param-a": dist}).keys() def test_sample_relative_misc_arguments() -> None: study = optuna.create_study() dist = optuna.distributions.FloatDistribution(1.0, 100.0) past_trials = [frozen_trial_factory(i, dist=dist) for i in range(1, 40)] # Prepare a trial and a sample for later checks. trial = frozen_trial_factory(40) with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = TPESampler(n_startup_trials=5, seed=0, multivariate=True) with patch.object(study._storage, "get_all_trials", return_value=past_trials): suggestion = sampler.sample_relative(study, trial, {"param-a": dist}) # Test misc. parameters. with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = TPESampler(n_ei_candidates=13, n_startup_trials=5, seed=0, multivariate=True) with patch.object(study._storage, "get_all_trials", return_value=past_trials): assert sampler.sample_relative(study, trial, {"param-a": dist}) != suggestion with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = TPESampler(gamma=lambda _: 5, n_startup_trials=5, seed=0, multivariate=True) with patch.object(study._storage, "get_all_trials", return_value=past_trials): assert sampler.sample_relative(study, trial, {"param-a": dist}) != suggestion with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = TPESampler( weights=lambda n: np.asarray([i**2 + 1 for i in range(n)]), n_startup_trials=5, seed=0, multivariate=True, ) with patch.object(study._storage, "get_all_trials", return_value=past_trials): assert sampler.sample_relative(study, trial, {"param-a": dist}) != suggestion def test_sample_relative_uniform_distributions() -> None: study = optuna.create_study() # Prepare sample from uniform distribution for cheking other distributions. uni_dist = optuna.distributions.FloatDistribution(1.0, 100.0) past_trials = [frozen_trial_factory(i, dist=uni_dist) for i in range(1, 8)] trial = frozen_trial_factory(8) with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = TPESampler(n_startup_trials=5, seed=0, multivariate=True) with patch.object(study._storage, "get_all_trials", return_value=past_trials): uniform_suggestion = sampler.sample_relative(study, trial, {"param-a": uni_dist}) assert 1.0 <= uniform_suggestion["param-a"] < 100.0 def test_sample_relative_log_uniform_distributions() -> None: """Prepare sample from uniform distribution for cheking other distributions.""" study = optuna.create_study() uni_dist = optuna.distributions.FloatDistribution(1.0, 100.0) past_trials = [frozen_trial_factory(i, dist=uni_dist) for i in range(1, 8)] trial = frozen_trial_factory(8) with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = TPESampler(n_startup_trials=5, seed=0, multivariate=True) with patch.object(study._storage, "get_all_trials", return_value=past_trials): uniform_suggestion = sampler.sample_relative(study, trial, {"param-a": uni_dist}) # Test sample from log-uniform is different from uniform. log_dist = optuna.distributions.FloatDistribution(1.0, 100.0, log=True) past_trials = [frozen_trial_factory(i, dist=log_dist) for i in range(1, 8)] trial = frozen_trial_factory(8) with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = TPESampler(n_startup_trials=5, seed=0, multivariate=True) with patch.object(study._storage, "get_all_trials", return_value=past_trials): loguniform_suggestion = sampler.sample_relative(study, trial, {"param-a": log_dist}) assert 1.0 <= loguniform_suggestion["param-a"] < 100.0 assert uniform_suggestion["param-a"] != loguniform_suggestion["param-a"] def test_sample_relative_disrete_uniform_distributions() -> None: """Test samples from discrete have expected intervals.""" study = optuna.create_study() disc_dist = optuna.distributions.FloatDistribution(1.0, 100.0, step=0.1) def value_fn(idx: int) -> float: random.seed(idx) return int(random.random() * 1000) * 0.1 past_trials = [frozen_trial_factory(i, dist=disc_dist, value_fn=value_fn) for i in range(1, 8)] trial = frozen_trial_factory(8) with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = TPESampler(n_startup_trials=5, seed=0, multivariate=True) with patch.object(study._storage, "get_all_trials", return_value=past_trials): discrete_uniform_suggestion = sampler.sample_relative(study, trial, {"param-a": disc_dist}) assert 1.0 <= discrete_uniform_suggestion["param-a"] <= 100.0 np.testing.assert_almost_equal( int(discrete_uniform_suggestion["param-a"] * 10), discrete_uniform_suggestion["param-a"] * 10, ) def test_sample_relative_categorical_distributions() -> None: """Test samples are drawn from the specified category.""" study = optuna.create_study() categories = [i * 0.3 + 1.0 for i in range(330)] def cat_value_fn(idx: int) -> float: random.seed(idx) return categories[random.randint(0, len(categories) - 1)] cat_dist = optuna.distributions.CategoricalDistribution(categories) past_trials = [ frozen_trial_factory(i, dist=cat_dist, value_fn=cat_value_fn) for i in range(1, 8) ] trial = frozen_trial_factory(8) with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = TPESampler(n_startup_trials=5, seed=0, multivariate=True) with patch.object(study._storage, "get_all_trials", return_value=past_trials): categorical_suggestion = sampler.sample_relative(study, trial, {"param-a": cat_dist}) assert categorical_suggestion["param-a"] in categories @pytest.mark.parametrize("step", [1, 2]) def test_sample_relative_int_uniform_distributions(step: int) -> None: """Test sampling from int distribution returns integer.""" study = optuna.create_study() def int_value_fn(idx: int) -> float: random.seed(idx) return step * random.randint(0, 100 // step) int_dist = optuna.distributions.IntDistribution(0, 100, step=step) past_trials = [ frozen_trial_factory(i, dist=int_dist, value_fn=int_value_fn) for i in range(1, 8) ] trial = frozen_trial_factory(8) with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = TPESampler(n_startup_trials=5, seed=0, multivariate=True) with patch.object(study._storage, "get_all_trials", return_value=past_trials): int_suggestion = sampler.sample_relative(study, trial, {"param-a": int_dist}) assert 1 <= int_suggestion["param-a"] <= 100 assert isinstance(int_suggestion["param-a"], int) assert int_suggestion["param-a"] % step == 0 def test_sample_relative_int_loguniform_distributions() -> None: """Test sampling from int distribution returns integer.""" study = optuna.create_study() def int_value_fn(idx: int) -> float: random.seed(idx) return random.randint(0, 100) intlog_dist = optuna.distributions.IntDistribution(1, 100, log=True) past_trials = [ frozen_trial_factory(i, dist=intlog_dist, value_fn=int_value_fn) for i in range(1, 8) ] trial = frozen_trial_factory(8) with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = TPESampler(n_startup_trials=5, seed=0, multivariate=True) with patch.object(study._storage, "get_all_trials", return_value=past_trials): intlog_suggestion = sampler.sample_relative(study, trial, {"param-a": intlog_dist}) assert 1 <= intlog_suggestion["param-a"] <= 100 assert isinstance(intlog_suggestion["param-a"], int) @pytest.mark.parametrize( "state", [ optuna.trial.TrialState.FAIL, optuna.trial.TrialState.PRUNED, optuna.trial.TrialState.RUNNING, optuna.trial.TrialState.WAITING, ], ) def test_sample_relative_handle_unsuccessful_states( state: optuna.trial.TrialState, ) -> None: dist = optuna.distributions.FloatDistribution(1.0, 100.0) # Prepare sampling result for later tests. study = optuna.create_study() for i in range(1, 100): trial = frozen_trial_factory(i, dist=dist) study._storage.create_new_trial(study._study_id, template_trial=trial) trial = frozen_trial_factory(100) with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = TPESampler(n_startup_trials=5, seed=0, multivariate=True) all_success_suggestion = sampler.sample_relative(study, trial, {"param-a": dist}) # Test unsuccessful trials are handled differently. study = optuna.create_study() state_fn = build_state_fn(state) for i in range(1, 100): trial = frozen_trial_factory(i, dist=dist, state_fn=state_fn) study._storage.create_new_trial(study._study_id, template_trial=trial) trial = frozen_trial_factory(100) with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = TPESampler(n_startup_trials=5, seed=0, multivariate=True) partial_unsuccessful_suggestion = sampler.sample_relative(study, trial, {"param-a": dist}) assert partial_unsuccessful_suggestion != all_success_suggestion def test_sample_relative_ignored_states() -> None: """Tests FAIL, RUNNING, and WAITING states are equally.""" dist = optuna.distributions.FloatDistribution(1.0, 100.0) suggestions = [] for state in [ optuna.trial.TrialState.FAIL, optuna.trial.TrialState.RUNNING, optuna.trial.TrialState.WAITING, ]: study = optuna.create_study() state_fn = build_state_fn(state) for i in range(1, 30): trial = frozen_trial_factory(i, dist=dist, state_fn=state_fn) study._storage.create_new_trial(study._study_id, template_trial=trial) with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = TPESampler(n_startup_trials=5, seed=0, multivariate=True) suggestions.append(sampler.sample_relative(study, trial, {"param-a": dist})["param-a"]) assert len(set(suggestions)) == 1 def test_sample_relative_pruned_state() -> None: """Tests PRUNED state is treated differently from both FAIL and COMPLETE.""" dist = optuna.distributions.FloatDistribution(1.0, 100.0) suggestions = [] for state in [ optuna.trial.TrialState.COMPLETE, optuna.trial.TrialState.FAIL, optuna.trial.TrialState.PRUNED, ]: study = optuna.create_study() state_fn = build_state_fn(state) for i in range(1, 40): trial = frozen_trial_factory(i, dist=dist, state_fn=state_fn) study._storage.create_new_trial(study._study_id, template_trial=trial) trial = frozen_trial_factory(40) with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = TPESampler(n_startup_trials=5, seed=0, multivariate=True) suggestions.append(sampler.sample_relative(study, trial, {"param-a": dist})["param-a"]) assert len(set(suggestions)) == 3 def test_sample_independent_prior() -> None: study = optuna.create_study() dist = optuna.distributions.FloatDistribution(1.0, 100.0) past_trials = [frozen_trial_factory(i, dist=dist) for i in range(1, 8)] # Prepare a trial and a sample for later checks. trial = frozen_trial_factory(8) sampler = TPESampler(n_startup_trials=5, seed=0) with patch.object(study._storage, "get_all_trials", return_value=past_trials): suggestion = sampler.sample_independent(study, trial, "param-a", dist) sampler = TPESampler(consider_prior=False, n_startup_trials=5, seed=0) with patch.object(study._storage, "get_all_trials", return_value=past_trials): assert sampler.sample_independent(study, trial, "param-a", dist) != suggestion sampler = TPESampler(prior_weight=0.1, n_startup_trials=5, seed=0) with patch.object(study._storage, "get_all_trials", return_value=past_trials): assert sampler.sample_independent(study, trial, "param-a", dist) != suggestion def test_sample_independent_n_startup_trial() -> None: study = optuna.create_study() dist = optuna.distributions.FloatDistribution(1.0, 100.0) past_trials = [frozen_trial_factory(i, dist=dist) for i in range(1, 8)] trial = frozen_trial_factory(8) sampler = TPESampler(n_startup_trials=5, seed=0) with patch.object(study._storage, "get_all_trials", return_value=past_trials[:4]): with patch.object( optuna.samplers.RandomSampler, "sample_independent", return_value=1.0 ) as sample_method: sampler.sample_independent(study, trial, "param-a", dist) assert sample_method.call_count == 1 sampler = TPESampler(n_startup_trials=5, seed=0) study._thread_local.cached_all_trials = None with patch.object(study._storage, "get_all_trials", return_value=past_trials): with patch.object( optuna.samplers.RandomSampler, "sample_independent", return_value=1.0 ) as sample_method: sampler.sample_independent(study, trial, "param-a", dist) assert sample_method.call_count == 0 def test_sample_independent_misc_arguments() -> None: study = optuna.create_study() dist = optuna.distributions.FloatDistribution(1.0, 100.0) past_trials = [frozen_trial_factory(i, dist=dist) for i in range(1, 8)] # Prepare a trial and a sample for later checks. trial = frozen_trial_factory(8) sampler = TPESampler(n_startup_trials=5, seed=0) with patch.object(study._storage, "get_all_trials", return_value=past_trials): suggestion = sampler.sample_independent(study, trial, "param-a", dist) # Test misc. parameters. sampler = TPESampler(n_ei_candidates=13, n_startup_trials=5, seed=0) with patch.object(study._storage, "get_all_trials", return_value=past_trials): assert sampler.sample_independent(study, trial, "param-a", dist) != suggestion sampler = TPESampler(gamma=lambda _: 5, n_startup_trials=5, seed=0) with patch.object(study._storage, "get_all_trials", return_value=past_trials): assert sampler.sample_independent(study, trial, "param-a", dist) != suggestion sampler = TPESampler( weights=lambda i: np.asarray([10 - j for j in range(i)]), n_startup_trials=5, seed=0 ) with patch("optuna.Study._get_trials", return_value=past_trials): assert sampler.sample_independent(study, trial, "param-a", dist) != suggestion def test_sample_independent_uniform_distributions() -> None: study = optuna.create_study() # Prepare sample from uniform distribution for cheking other distributions. uni_dist = optuna.distributions.FloatDistribution(1.0, 100.0) past_trials = [frozen_trial_factory(i, dist=uni_dist) for i in range(1, 8)] trial = frozen_trial_factory(8) sampler = TPESampler(n_startup_trials=5, seed=0) with patch.object(study._storage, "get_all_trials", return_value=past_trials): uniform_suggestion = sampler.sample_independent(study, trial, "param-a", uni_dist) assert 1.0 <= uniform_suggestion < 100.0 def test_sample_independent_log_uniform_distributions() -> None: """Prepare sample from uniform distribution for cheking other distributions.""" study = optuna.create_study() uni_dist = optuna.distributions.FloatDistribution(1.0, 100.0) past_trials = [frozen_trial_factory(i, dist=uni_dist) for i in range(1, 8)] trial = frozen_trial_factory(8) sampler = TPESampler(n_startup_trials=5, seed=0) with patch.object(study._storage, "get_all_trials", return_value=past_trials): uniform_suggestion = sampler.sample_independent(study, trial, "param-a", uni_dist) # Test sample from log-uniform is different from uniform. log_dist = optuna.distributions.FloatDistribution(1.0, 100.0, log=True) past_trials = [frozen_trial_factory(i, dist=log_dist) for i in range(1, 8)] trial = frozen_trial_factory(8) sampler = TPESampler(n_startup_trials=5, seed=0) with patch.object(study._storage, "get_all_trials", return_value=past_trials): loguniform_suggestion = sampler.sample_independent(study, trial, "param-a", log_dist) assert 1.0 <= loguniform_suggestion < 100.0 assert uniform_suggestion != loguniform_suggestion def test_sample_independent_discrete_uniform_distributions() -> None: """Test samples from discrete have expected intervals.""" study = optuna.create_study() disc_dist = optuna.distributions.FloatDistribution(1.0, 100.0, step=0.1) def value_fn(idx: int) -> float: random.seed(idx) return int(random.random() * 1000) * 0.1 past_trials = [frozen_trial_factory(i, dist=disc_dist, value_fn=value_fn) for i in range(1, 8)] trial = frozen_trial_factory(8) sampler = TPESampler(n_startup_trials=5, seed=0) with patch("optuna.Study.get_trials", return_value=past_trials): discrete_uniform_suggestion = sampler.sample_independent( study, trial, "param-a", disc_dist ) assert 1.0 <= discrete_uniform_suggestion <= 100.0 assert abs(int(discrete_uniform_suggestion * 10) - discrete_uniform_suggestion * 10) < 1e-3 def test_sample_independent_categorical_distributions() -> None: """Test samples are drawn from the specified category.""" study = optuna.create_study() categories = [i * 0.3 + 1.0 for i in range(330)] def cat_value_fn(idx: int) -> float: random.seed(idx) return categories[random.randint(0, len(categories) - 1)] cat_dist = optuna.distributions.CategoricalDistribution(categories) past_trials = [ frozen_trial_factory(i, dist=cat_dist, value_fn=cat_value_fn) for i in range(1, 8) ] trial = frozen_trial_factory(8) sampler = TPESampler(n_startup_trials=5, seed=0) with patch.object(study._storage, "get_all_trials", return_value=past_trials): categorical_suggestion = sampler.sample_independent(study, trial, "param-a", cat_dist) assert categorical_suggestion in categories def test_sample_independent_int_uniform_distributions() -> None: """Test sampling from int distribution returns integer.""" study = optuna.create_study() def int_value_fn(idx: int) -> float: random.seed(idx) return random.randint(0, 100) int_dist = optuna.distributions.IntDistribution(1, 100) past_trials = [ frozen_trial_factory(i, dist=int_dist, value_fn=int_value_fn) for i in range(1, 8) ] trial = frozen_trial_factory(8) sampler = TPESampler(n_startup_trials=5, seed=0) with patch.object(study._storage, "get_all_trials", return_value=past_trials): int_suggestion = sampler.sample_independent(study, trial, "param-a", int_dist) assert 1 <= int_suggestion <= 100 assert isinstance(int_suggestion, int) def test_sample_independent_int_loguniform_distributions() -> None: """Test sampling from int distribution returns integer.""" study = optuna.create_study() def int_value_fn(idx: int) -> float: random.seed(idx) return random.randint(0, 100) intlog_dist = optuna.distributions.IntDistribution(1, 100, log=True) past_trials = [ frozen_trial_factory(i, dist=intlog_dist, value_fn=int_value_fn) for i in range(1, 8) ] trial = frozen_trial_factory(8) sampler = TPESampler(n_startup_trials=5, seed=0) with patch.object(study._storage, "get_all_trials", return_value=past_trials): intlog_suggestion = sampler.sample_independent(study, trial, "param-a", intlog_dist) assert 1 <= intlog_suggestion <= 100 assert isinstance(intlog_suggestion, int) @pytest.mark.parametrize( "state", [ optuna.trial.TrialState.FAIL, optuna.trial.TrialState.PRUNED, optuna.trial.TrialState.RUNNING, optuna.trial.TrialState.WAITING, ], ) def test_sample_independent_handle_unsuccessful_states(state: optuna.trial.TrialState) -> None: dist = optuna.distributions.FloatDistribution(1.0, 100.0) # Prepare sampling result for later tests. study = optuna.create_study() for i in range(1, 30): trial = frozen_trial_factory(i, dist=dist) study._storage.create_new_trial(study._study_id, template_trial=trial) trial = frozen_trial_factory(30) sampler = TPESampler(n_startup_trials=5, seed=2) all_success_suggestion = sampler.sample_independent(study, trial, "param-a", dist) # Test unsuccessful trials are handled differently. state_fn = build_state_fn(state) study = optuna.create_study() for i in range(1, 30): trial = frozen_trial_factory(i, dist=dist, state_fn=state_fn) study._storage.create_new_trial(study._study_id, template_trial=trial) trial = frozen_trial_factory(30) sampler = TPESampler(n_startup_trials=5, seed=2) partial_unsuccessful_suggestion = sampler.sample_independent(study, trial, "param-a", dist) assert partial_unsuccessful_suggestion != all_success_suggestion def test_sample_independent_ignored_states() -> None: """Tests FAIL, RUNNING, and WAITING states are equally.""" dist = optuna.distributions.FloatDistribution(1.0, 100.0) suggestions = [] for state in [ optuna.trial.TrialState.FAIL, optuna.trial.TrialState.RUNNING, optuna.trial.TrialState.WAITING, ]: study = optuna.create_study() state_fn = build_state_fn(state) for i in range(1, 30): trial = frozen_trial_factory(i, dist=dist, state_fn=state_fn) study._storage.create_new_trial(study._study_id, template_trial=trial) trial = frozen_trial_factory(30) sampler = TPESampler(n_startup_trials=5, seed=0) suggestions.append(sampler.sample_independent(study, trial, "param-a", dist)) assert len(set(suggestions)) == 1 def test_sample_independent_pruned_state() -> None: """Tests PRUNED state is treated differently from both FAIL and COMPLETE.""" dist = optuna.distributions.FloatDistribution(1.0, 100.0) suggestions = [] for state in [ optuna.trial.TrialState.COMPLETE, optuna.trial.TrialState.FAIL, optuna.trial.TrialState.PRUNED, ]: study = optuna.create_study() state_fn = build_state_fn(state) for i in range(1, 30): trial = frozen_trial_factory(i, dist=dist, state_fn=state_fn) study._storage.create_new_trial(study._study_id, template_trial=trial) trial = frozen_trial_factory(30) sampler = TPESampler(n_startup_trials=5, seed=2) suggestions.append(sampler.sample_independent(study, trial, "param-a", dist)) assert len(set(suggestions)) == 3 def test_constrained_sample_independent_zero_startup() -> None: """Tests TPESampler with constrained option works when n_startup_trials=0.""" study = optuna.create_study() dist = optuna.distributions.FloatDistribution(1.0, 100.0) trial = frozen_trial_factory(30) sampler = TPESampler(n_startup_trials=0, seed=2, constraints_func=lambda _: (0,)) sampler.sample_independent(study, trial, "param-a", dist) @pytest.mark.parametrize("direction", ["minimize", "maximize"]) @pytest.mark.parametrize("constant_liar", [True, False]) @pytest.mark.parametrize("constraints", [True, False]) def test_split_trials(direction: str, constant_liar: bool, constraints: bool) -> None: study = optuna.create_study(direction=direction) for value in [-float("inf"), 0, 1, float("inf")]: study.add_trial( optuna.create_trial( state=optuna.trial.TrialState.COMPLETE, value=(value if direction == "minimize" else -value), params={"x": 0}, distributions={"x": optuna.distributions.FloatDistribution(-1.0, 1.0)}, system_attrs={_CONSTRAINTS_KEY: [-1]}, ) ) for step in [2, 1]: for value in [-float("inf"), 0, 1, float("inf"), float("nan")]: study.add_trial( optuna.create_trial( state=optuna.trial.TrialState.PRUNED, params={"x": 0}, distributions={"x": optuna.distributions.FloatDistribution(-1.0, 1.0)}, system_attrs={_CONSTRAINTS_KEY: [-1]}, intermediate_values={step: (value if direction == "minimize" else -value)}, ) ) study.add_trial( optuna.create_trial( state=optuna.trial.TrialState.PRUNED, params={"x": 0}, distributions={"x": optuna.distributions.FloatDistribution(-1.0, 1.0)}, system_attrs={_CONSTRAINTS_KEY: [-1]}, ) ) if constraints: for value in [1, 2, float("inf")]: study.add_trial( optuna.create_trial( state=optuna.trial.TrialState.COMPLETE, value=0, params={"x": 0}, distributions={"x": optuna.distributions.FloatDistribution(-1.0, 1.0)}, system_attrs={_CONSTRAINTS_KEY: [value]}, ) ) study.add_trial( optuna.create_trial( state=optuna.trial.TrialState.RUNNING, params={"x": 0}, distributions={"x": optuna.distributions.FloatDistribution(-1.0, 1.0)}, ) ) study.add_trial( optuna.create_trial( state=optuna.trial.TrialState.FAIL, ) ) study.add_trial( optuna.create_trial( state=optuna.trial.TrialState.WAITING, ) ) if constant_liar: states = [ optuna.trial.TrialState.COMPLETE, optuna.trial.TrialState.PRUNED, optuna.trial.TrialState.RUNNING, ] else: states = [optuna.trial.TrialState.COMPLETE, optuna.trial.TrialState.PRUNED] trials = study.get_trials(states=states) finished_trials = study.get_trials( states=(optuna.trial.TrialState.COMPLETE, optuna.trial.TrialState.PRUNED) ) for n_below in range(len(finished_trials) + 1): below_trials, above_trials = _tpe.sampler._split_trials( study, trials, n_below, constraints, ) below_trial_numbers = [trial.number for trial in below_trials] assert below_trial_numbers == list(range(n_below)) above_trial_numbers = [trial.number for trial in above_trials] assert above_trial_numbers == list(range(n_below, len(trials))) @pytest.mark.parametrize( "directions", [["minimize", "minimize"], ["maximize", "maximize"], ["minimize", "maximize"]] ) def test_split_trials_for_multiobjective_constant_liar(directions: list[str]) -> None: study = optuna.create_study(directions=directions) # 16 Trials (#0 -- #15) that should be sorted by non-dominated sort and HSSP. for obj1 in [-float("inf"), 0, 1, float("inf")]: val1 = obj1 if directions[0] == "minimize" else -obj1 for obj2 in [-float("inf"), 0, 1, float("inf")]: val2 = obj2 if directions[1] == "minimize" else -obj2 study.add_trial( optuna.create_trial( state=optuna.trial.TrialState.COMPLETE, values=[val1, val2], params={"x": 0}, distributions={"x": optuna.distributions.FloatDistribution(-1.0, 1.0)}, ) ) # 5 Trials (#16 -- # 20) that should come at the end of the sorting. n_running_trials = 5 for _ in range(n_running_trials): study.add_trial( optuna.create_trial( state=optuna.trial.TrialState.RUNNING, params={"x": 0}, distributions={"x": optuna.distributions.FloatDistribution(-1.0, 1.0)}, ) ) # 2 Trials (#21, #22) that should be ignored in the sorting. study.add_trial(optuna.create_trial(state=optuna.trial.TrialState.FAIL)) study.add_trial(optuna.create_trial(state=optuna.trial.TrialState.WAITING)) states = [optuna.trial.TrialState.COMPLETE, optuna.trial.TrialState.RUNNING] trials = study.get_trials(states=states) finished_trials = study.get_trials(states=(optuna.trial.TrialState.COMPLETE,)) # Below is the relation of `non-domination rank: trial_numbers`. # 0: [0], 1: [1,4], 2: [2,5,8], 3: [3,6,9,12], 4: [7,10,13], 5: [11,14], 6: [15] # NOTE(nabenabe0928): As each `values` includes `inf`, ref_point also includes `inf`, # leading to an arbitrary hypervolume contribution to be `inf`. That is why HSSP starts to pick # an earlier trial number in each non-domination rank. ground_truth = [0, 1, 4, 2, 8, 5, 3, 6, 9, 12, 7, 10, 13, 11, 14, 15] n_completed_trials = len(ground_truth) # NOTE(nabenabe0928): Running trials (#16 -- #20) must come at the end. ground_truth += [n_completed_trials + i for i in range(n_running_trials)] for n_below in range(1, len(finished_trials) + 1): below_trials, above_trials = _tpe.sampler._split_trials( study, trials, n_below, constraints_enabled=False ) below_trial_numbers = [trial.number for trial in below_trials] assert below_trial_numbers == np.sort(ground_truth[:n_below]).tolist() above_trial_numbers = [trial.number for trial in above_trials] assert above_trial_numbers == np.sort(ground_truth[n_below:]).tolist() @pytest.mark.parametrize("direction", ["minimize", "maximize"]) def test_split_complete_trials_single_objective(direction: str) -> None: study = optuna.create_study(direction=direction) for value in [-float("inf"), 0, 1, float("inf")]: study.add_trial( optuna.create_trial( state=optuna.trial.TrialState.COMPLETE, value=(value if direction == "minimize" else -value), params={"x": 0}, distributions={"x": optuna.distributions.FloatDistribution(-1.0, 1.0)}, ) ) for n_below in range(len(study.trials) + 1): below_trials, above_trials = _tpe.sampler._split_complete_trials_single_objective( study.trials, study, n_below, ) assert [trial.number for trial in below_trials] == list(range(n_below)) assert [trial.number for trial in above_trials] == list(range(n_below, len(study.trials))) def test_split_complete_trials_single_objective_empty() -> None: study = optuna.create_study() assert _tpe.sampler._split_complete_trials_single_objective([], study, 0) == ([], []) @pytest.mark.parametrize("direction", ["minimize", "maximize"]) def test_split_pruned_trials(direction: str) -> None: study = optuna.create_study(direction=direction) for step in [2, 1]: for value in [-float("inf"), 0, 1, float("inf"), float("nan")]: study.add_trial( optuna.create_trial( state=optuna.trial.TrialState.PRUNED, params={"x": 0}, distributions={"x": optuna.distributions.FloatDistribution(-1.0, 1.0)}, intermediate_values={step: (value if direction == "minimize" else -value)}, ) ) study.add_trial( optuna.create_trial( state=optuna.trial.TrialState.PRUNED, params={"x": 0}, distributions={"x": optuna.distributions.FloatDistribution(-1.0, 1.0)}, ) ) for n_below in range(len(study.trials) + 1): below_trials, above_trials = _tpe.sampler._split_pruned_trials( study.trials, study, n_below, ) assert [trial.number for trial in below_trials] == list(range(n_below)) assert [trial.number for trial in above_trials] == list(range(n_below, len(study.trials))) def test_split_pruned_trials_empty() -> None: study = optuna.create_study() assert _tpe.sampler._split_pruned_trials([], study, 0) == ([], []) @pytest.mark.parametrize("direction", ["minimize", "maximize"]) def test_split_infeasible_trials(direction: str) -> None: study = optuna.create_study(direction=direction) for value in [1, 2, float("inf")]: study.add_trial( optuna.create_trial( state=optuna.trial.TrialState.COMPLETE, value=0, params={"x": 0}, distributions={"x": optuna.distributions.FloatDistribution(-1.0, 1.0)}, system_attrs={_CONSTRAINTS_KEY: [value]}, ) ) for n_below in range(len(study.trials) + 1): below_trials, above_trials = _tpe.sampler._split_infeasible_trials(study.trials, n_below) assert [trial.number for trial in below_trials] == list(range(n_below)) assert [trial.number for trial in above_trials] == list(range(n_below, len(study.trials))) def test_split_infeasible_trials_empty() -> None: assert _tpe.sampler._split_infeasible_trials([], 0) == ([], []) def frozen_trial_factory( idx: int, dist: optuna.distributions.BaseDistribution = optuna.distributions.FloatDistribution( 1.0, 100.0 ), state_fn: Callable[ [int], optuna.trial.TrialState ] = lambda _: optuna.trial.TrialState.COMPLETE, value_fn: Optional[Callable[[int], Union[int, float]]] = None, target_fn: Callable[[float], float] = lambda val: (val - 20.0) ** 2, interm_val_fn: Callable[[int], Dict[int, float]] = lambda _: {}, ) -> optuna.trial.FrozenTrial: if value_fn is None: random.seed(idx) value = random.random() * 99.0 + 1.0 else: value = value_fn(idx) return optuna.trial.FrozenTrial( number=idx, state=state_fn(idx), value=target_fn(value), datetime_start=None, datetime_complete=None, params={"param-a": value}, distributions={"param-a": dist}, user_attrs={}, system_attrs={}, intermediate_values=interm_val_fn(idx), trial_id=idx, ) def build_state_fn(state: optuna.trial.TrialState) -> Callable[[int], optuna.trial.TrialState]: def state_fn(idx: int) -> optuna.trial.TrialState: return [optuna.trial.TrialState.COMPLETE, state][idx % 2] return state_fn def test_call_after_trial_of_random_sampler() -> None: sampler = TPESampler() study = optuna.create_study(sampler=sampler) with patch.object( sampler._random_sampler, "after_trial", wraps=sampler._random_sampler.after_trial ) as mock_object: study.optimize(lambda _: 1.0, n_trials=1) assert mock_object.call_count == 1 def test_mixed_relative_search_space_pruned_and_completed_trials() -> None: def objective(trial: Trial) -> float: if trial.number == 0: trial.suggest_float("param1", 0, 1) raise optuna.exceptions.TrialPruned() if trial.number == 1: trial.suggest_float("param2", 0, 1) return 0 return 0 sampler = TPESampler(n_startup_trials=1, multivariate=True) study = optuna.create_study(sampler=sampler) study.optimize(objective, 3) def test_group() -> None: with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = TPESampler(multivariate=True, group=True) study = optuna.create_study(sampler=sampler) with patch.object(sampler, "_sample_relative", wraps=sampler._sample_relative) as mock: study.optimize(lambda t: t.suggest_int("x", 0, 10), n_trials=2) assert mock.call_count == 1 assert study.trials[-1].distributions == {"x": distributions.IntDistribution(low=0, high=10)} with patch.object(sampler, "_sample_relative", wraps=sampler._sample_relative) as mock: study.optimize( lambda t: t.suggest_int("y", 0, 10) + t.suggest_float("z", -3, 3), n_trials=1 ) assert mock.call_count == 1 assert study.trials[-1].distributions == { "y": distributions.IntDistribution(low=0, high=10), "z": distributions.FloatDistribution(low=-3, high=3), } with patch.object(sampler, "_sample_relative", wraps=sampler._sample_relative) as mock: study.optimize( lambda t: t.suggest_int("y", 0, 10) + t.suggest_float("z", -3, 3) + t.suggest_float("u", 1e-2, 1e2, log=True) + bool(t.suggest_categorical("v", ["A", "B", "C"])), n_trials=1, ) assert mock.call_count == 2 assert study.trials[-1].distributions == { "u": distributions.FloatDistribution(low=1e-2, high=1e2, log=True), "v": distributions.CategoricalDistribution(choices=["A", "B", "C"]), "y": distributions.IntDistribution(low=0, high=10), "z": distributions.FloatDistribution(low=-3, high=3), } with patch.object(sampler, "_sample_relative", wraps=sampler._sample_relative) as mock: study.optimize(lambda t: t.suggest_float("u", 1e-2, 1e2, log=True), n_trials=1) assert mock.call_count == 3 assert study.trials[-1].distributions == { "u": distributions.FloatDistribution(low=1e-2, high=1e2, log=True) } with patch.object(sampler, "_sample_relative", wraps=sampler._sample_relative) as mock: study.optimize( lambda t: t.suggest_int("y", 0, 10) + t.suggest_int("w", 2, 8, log=True), n_trials=1 ) assert mock.call_count == 4 assert study.trials[-1].distributions == { "y": distributions.IntDistribution(low=0, high=10), "w": distributions.IntDistribution(low=2, high=8, log=True), } with patch.object(sampler, "_sample_relative", wraps=sampler._sample_relative) as mock: study.optimize(lambda t: t.suggest_int("x", 0, 10), n_trials=1) assert mock.call_count == 6 assert study.trials[-1].distributions == {"x": distributions.IntDistribution(low=0, high=10)} def test_invalid_multivariate_and_group() -> None: with pytest.raises(ValueError): _ = TPESampler(multivariate=False, group=True) def test_group_experimental_warning() -> None: with pytest.warns(optuna.exceptions.ExperimentalWarning): _ = TPESampler(multivariate=True, group=True) def test_constant_liar_experimental_warning() -> None: with pytest.warns(optuna.exceptions.ExperimentalWarning): _ = TPESampler(constant_liar=True) @pytest.mark.parametrize("multivariate", [True, False]) @pytest.mark.parametrize("multiobjective", [True, False]) def test_constant_liar_with_running_trial(multivariate: bool, multiobjective: bool) -> None: with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = TPESampler(multivariate=multivariate, constant_liar=True, n_startup_trials=0) directions = ["minimize"] * 2 if multiobjective else ["minimize"] study = optuna.create_study(sampler=sampler, directions=directions) # Add a complete trial. trial0 = study.ask() trial0.suggest_int("x", 0, 10) trial0.suggest_float("y", 0, 10) trial0.suggest_categorical("z", [0, 1, 2]) study.tell(trial0, [0, 0] if multiobjective else 0) # Add running trials. trial1 = study.ask() trial1.suggest_int("x", 0, 10) trial2 = study.ask() trial2.suggest_float("y", 0, 10) trial3 = study.ask() trial3.suggest_categorical("z", [0, 1, 2]) # Test suggestion with running trials. trial = study.ask() trial.suggest_int("x", 0, 10) trial.suggest_float("y", 0, 10) trial.suggest_categorical("z", [0, 1, 2]) study.tell(trial, [0, 0] if multiobjective else 0) def test_categorical_distance_func_experimental_warning() -> None: with pytest.warns(optuna.exceptions.ExperimentalWarning): _ = TPESampler(categorical_distance_func={"c": lambda x, y: 0.0}) optuna-4.1.0/tests/samplers_tests/tpe_tests/test_truncnorm.py000066400000000000000000000042451471332314300247140ustar00rootroot00000000000000import sys import numpy as np import pytest from scipy.stats import truncnorm as truncnorm_scipy from optuna._imports import try_import import optuna.samplers._tpe._truncnorm as truncnorm_ours with try_import() as _imports: from scipy.stats._continuous_distns import _log_gauss_mass as _log_gauss_mass_scipy @pytest.mark.filterwarnings("ignore::RuntimeWarning") @pytest.mark.parametrize( "a,b", [(-np.inf, np.inf), (-10, +10), (-1, +1), (-1e-3, +1e-3), (10, 100), (-100, -10), (0, 0)], ) def test_ppf(a: float, b: float) -> None: for x in np.concatenate( [np.linspace(0, 1, num=100), np.array([sys.float_info.min, 1 - sys.float_info.epsilon])] ): assert truncnorm_ours.ppf(x, a, b) == pytest.approx( truncnorm_scipy.ppf(x, a, b), nan_ok=True ), f"ppf(x={x}, a={a}, b={b})" @pytest.mark.filterwarnings("ignore::RuntimeWarning") @pytest.mark.parametrize( "a,b", [(-np.inf, np.inf), (-10, +10), (-1, +1), (-1e-3, +1e-3), (10, 100), (-100, -10), (0, 0)], ) @pytest.mark.parametrize("loc", [-10, 0, 10]) @pytest.mark.parametrize("scale", [0.1, 1, 10]) def test_logpdf(a: float, b: float, loc: float, scale: float) -> None: for x in np.concatenate( [np.linspace(np.max([a, -100]), np.min([b, 100]), num=1000), np.array([-2000.0, +2000.0])] ): assert truncnorm_ours.logpdf(x, a, b, loc, scale) == pytest.approx( truncnorm_scipy.logpdf(x, a, b, loc, scale), nan_ok=True ), f"logpdf(x={x}, a={a}, b={b})" @pytest.mark.skipif( not _imports.is_successful(), reason="Failed to import SciPy's internal function." ) @pytest.mark.parametrize( "a,b", # we don't test (0, 0) as SciPy returns the incorrect value. [(-np.inf, np.inf), (-10, +10), (-1, +1), (-1e-3, +1e-3), (10, 100), (-100, -10)], ) def test_log_gass_mass(a: float, b: float) -> None: for x in np.concatenate( [np.linspace(0, 1, num=100), np.array([sys.float_info.min, 1 - sys.float_info.epsilon])] ): assert truncnorm_ours._log_gauss_mass(np.array([a]), np.array([b])) == pytest.approx( np.atleast_1d(_log_gauss_mass_scipy(a, b)), nan_ok=True ), f"_log_gauss_mass(x={x}, a={a}, b={b})" optuna-4.1.0/tests/search_space_tests/000077500000000000000000000000001471332314300200275ustar00rootroot00000000000000optuna-4.1.0/tests/search_space_tests/__init__.py000066400000000000000000000000001471332314300221260ustar00rootroot00000000000000optuna-4.1.0/tests/search_space_tests/test_group_decomposed.py000066400000000000000000000200421471332314300247740ustar00rootroot00000000000000import pytest from optuna import create_study from optuna import TrialPruned from optuna.distributions import CategoricalDistribution from optuna.distributions import FloatDistribution from optuna.distributions import IntDistribution from optuna.search_space import _GroupDecomposedSearchSpace from optuna.search_space import _SearchSpaceGroup from optuna.testing.storages import StorageSupplier from optuna.trial import Trial def test_search_space_group() -> None: search_space_group = _SearchSpaceGroup() # No search space. assert search_space_group.search_spaces == [] # No distributions. search_space_group.add_distributions({}) assert search_space_group.search_spaces == [] # Add a single distribution. search_space_group.add_distributions({"x": IntDistribution(low=0, high=10)}) assert search_space_group.search_spaces == [{"x": IntDistribution(low=0, high=10)}] # Add a same single distribution. search_space_group.add_distributions({"x": IntDistribution(low=0, high=10)}) assert search_space_group.search_spaces == [{"x": IntDistribution(low=0, high=10)}] # Add disjoint distributions. search_space_group.add_distributions( { "y": IntDistribution(low=0, high=10), "z": FloatDistribution(low=-3, high=3), } ) assert search_space_group.search_spaces == [ {"x": IntDistribution(low=0, high=10)}, { "y": IntDistribution(low=0, high=10), "z": FloatDistribution(low=-3, high=3), }, ] # Add distributions, which include one of search spaces in the group. search_space_group.add_distributions( { "y": IntDistribution(low=0, high=10), "z": FloatDistribution(low=-3, high=3), "u": FloatDistribution(low=1e-2, high=1e2, log=True), "v": CategoricalDistribution(choices=["A", "B", "C"]), } ) assert search_space_group.search_spaces == [ {"x": IntDistribution(low=0, high=10)}, { "y": IntDistribution(low=0, high=10), "z": FloatDistribution(low=-3, high=3), }, { "u": FloatDistribution(low=1e-2, high=1e2, log=True), "v": CategoricalDistribution(choices=["A", "B", "C"]), }, ] # Add a distribution, which is included by one of search spaces in the group. search_space_group.add_distributions({"u": FloatDistribution(low=1e-2, high=1e2, log=True)}) assert search_space_group.search_spaces == [ {"x": IntDistribution(low=0, high=10)}, { "y": IntDistribution(low=0, high=10), "z": FloatDistribution(low=-3, high=3), }, {"u": FloatDistribution(low=1e-2, high=1e2, log=True)}, {"v": CategoricalDistribution(choices=["A", "B", "C"])}, ] # Add distributions whose intersection with one of search spaces in the group is not empty. search_space_group.add_distributions( { "y": IntDistribution(low=0, high=10), "w": IntDistribution(low=2, high=8, log=True), } ) assert search_space_group.search_spaces == [ {"x": IntDistribution(low=0, high=10)}, {"y": IntDistribution(low=0, high=10)}, {"z": FloatDistribution(low=-3, high=3)}, {"u": FloatDistribution(low=1e-2, high=1e2, log=True)}, {"v": CategoricalDistribution(choices=["A", "B", "C"])}, {"w": IntDistribution(low=2, high=8, log=True)}, ] # Add distributions which include some of search spaces in the group. search_space_group.add_distributions( { "y": IntDistribution(low=0, high=10), "w": IntDistribution(low=2, high=8, log=True), "t": FloatDistribution(low=10, high=100), } ) assert search_space_group.search_spaces == [ {"x": IntDistribution(low=0, high=10)}, {"y": IntDistribution(low=0, high=10)}, {"z": FloatDistribution(low=-3, high=3)}, {"u": FloatDistribution(low=1e-2, high=1e2, log=True)}, {"v": CategoricalDistribution(choices=["A", "B", "C"])}, {"w": IntDistribution(low=2, high=8, log=True)}, {"t": FloatDistribution(low=10, high=100)}, ] def test_group_decomposed_search_space() -> None: search_space = _GroupDecomposedSearchSpace() study = create_study() # No trial. assert search_space.calculate(study).search_spaces == [] # A single parameter. study.optimize(lambda t: t.suggest_int("x", 0, 10), n_trials=1) assert search_space.calculate(study).search_spaces == [{"x": IntDistribution(low=0, high=10)}] # Disjoint parameters. study.optimize(lambda t: t.suggest_int("y", 0, 10) + t.suggest_float("z", -3, 3), n_trials=1) assert search_space.calculate(study).search_spaces == [ {"x": IntDistribution(low=0, high=10)}, { "y": IntDistribution(low=0, high=10), "z": FloatDistribution(low=-3, high=3), }, ] # Parameters which include one of search spaces in the group. study.optimize( lambda t: t.suggest_int("y", 0, 10) + t.suggest_float("z", -3, 3) + t.suggest_float("u", 1e-2, 1e2, log=True) + bool(t.suggest_categorical("v", ["A", "B", "C"])), n_trials=1, ) assert search_space.calculate(study).search_spaces == [ {"x": IntDistribution(low=0, high=10)}, { "z": FloatDistribution(low=-3, high=3), "y": IntDistribution(low=0, high=10), }, { "u": FloatDistribution(low=1e-2, high=1e2, log=True), "v": CategoricalDistribution(choices=["A", "B", "C"]), }, ] # A parameter which is included by one of search spaces in thew group. study.optimize(lambda t: t.suggest_float("u", 1e-2, 1e2, log=True), n_trials=1) assert search_space.calculate(study).search_spaces == [ {"x": IntDistribution(low=0, high=10)}, { "y": IntDistribution(low=0, high=10), "z": FloatDistribution(low=-3, high=3), }, {"u": FloatDistribution(low=1e-2, high=1e2, log=True)}, {"v": CategoricalDistribution(choices=["A", "B", "C"])}, ] # Parameters whose intersection with one of search spaces in the group is not empty. study.optimize( lambda t: t.suggest_int("y", 0, 10) + t.suggest_int("w", 2, 8, log=True), n_trials=1 ) assert search_space.calculate(study).search_spaces == [ {"x": IntDistribution(low=0, high=10)}, {"y": IntDistribution(low=0, high=10)}, {"z": FloatDistribution(low=-3, high=3)}, {"u": FloatDistribution(low=1e-2, high=1e2, log=True)}, {"v": CategoricalDistribution(choices=["A", "B", "C"])}, {"w": IntDistribution(low=2, high=8, log=True)}, ] search_space = _GroupDecomposedSearchSpace() study = create_study() # Failed or pruned trials are not considered in the calculation of # an intersection search space. def objective(trial: Trial, exception: Exception) -> float: trial.suggest_float("a", 0, 1) raise exception study.optimize(lambda t: objective(t, RuntimeError()), n_trials=1, catch=(RuntimeError,)) study.optimize(lambda t: objective(t, TrialPruned()), n_trials=1) assert search_space.calculate(study).search_spaces == [] # If two parameters have the same name but different distributions, # the first one takes priority. study.optimize(lambda t: t.suggest_float("a", -1, 1), n_trials=1) study.optimize(lambda t: t.suggest_float("a", 0, 1), n_trials=1) assert search_space.calculate(study).search_spaces == [ {"a": FloatDistribution(low=-1, high=1)} ] def test_group_decomposed_search_space_with_different_studies() -> None: search_space = _GroupDecomposedSearchSpace() with StorageSupplier("sqlite") as storage: study0 = create_study(storage=storage) study1 = create_study(storage=storage) search_space.calculate(study0) with pytest.raises(ValueError): # `_GroupDecomposedSearchSpace` isn't supposed to be used for multiple studies. search_space.calculate(study1) optuna-4.1.0/tests/search_space_tests/test_intersection.py000066400000000000000000000070101471332314300241440ustar00rootroot00000000000000import pytest from optuna import create_study from optuna import TrialPruned from optuna.distributions import FloatDistribution from optuna.distributions import IntDistribution from optuna.search_space import intersection_search_space from optuna.search_space import IntersectionSearchSpace from optuna.testing.storages import StorageSupplier from optuna.trial import Trial def test_intersection_search_space() -> None: search_space = IntersectionSearchSpace() study = create_study() # No trial. assert search_space.calculate(study) == {} assert search_space.calculate(study) == intersection_search_space( study.get_trials(deepcopy=False) ) # Waiting trial. study.enqueue_trial( {"y": 0, "x": 5}, {"y": FloatDistribution(-3, 3), "x": IntDistribution(0, 10)} ) assert search_space.calculate(study) == {} assert search_space.calculate(study) == intersection_search_space( study.get_trials(deepcopy=False) ) # First trial. study.optimize(lambda t: t.suggest_float("y", -3, 3) + t.suggest_int("x", 0, 10), n_trials=1) assert search_space.calculate(study) == { "x": IntDistribution(low=0, high=10), "y": FloatDistribution(low=-3, high=3), } assert search_space.calculate(study) == intersection_search_space( study.get_trials(deepcopy=False) ) # Returned dict is sorted by parameter names. assert list(search_space.calculate(study).keys()) == ["x", "y"] # Second trial (only 'y' parameter is suggested in this trial). study.optimize(lambda t: t.suggest_float("y", -3, 3), n_trials=1) assert search_space.calculate(study) == {"y": FloatDistribution(low=-3, high=3)} assert search_space.calculate(study) == intersection_search_space( study.get_trials(deepcopy=False) ) # Failed or pruned trials are not considered in the calculation of # an intersection search space. def objective(trial: Trial, exception: Exception) -> float: trial.suggest_float("z", 0, 1) raise exception study.optimize(lambda t: objective(t, RuntimeError()), n_trials=1, catch=(RuntimeError,)) study.optimize(lambda t: objective(t, TrialPruned()), n_trials=1) assert search_space.calculate(study) == {"y": FloatDistribution(low=-3, high=3)} assert search_space.calculate(study) == intersection_search_space( study.get_trials(deepcopy=False) ) # If two parameters have the same name but different distributions, # those are regarded as different parameters. study.optimize(lambda t: t.suggest_float("y", -1, 1), n_trials=1) assert search_space.calculate(study) == {} assert search_space.calculate(study) == intersection_search_space( study.get_trials(deepcopy=False) ) # The search space remains empty once it is empty. study.optimize(lambda t: t.suggest_float("y", -3, 3) + t.suggest_int("x", 0, 10), n_trials=1) assert search_space.calculate(study) == {} assert search_space.calculate(study) == intersection_search_space( study.get_trials(deepcopy=False) ) def test_intersection_search_space_class_with_different_studies() -> None: search_space = IntersectionSearchSpace() with StorageSupplier("sqlite") as storage: study0 = create_study(storage=storage) study1 = create_study(storage=storage) search_space.calculate(study0) with pytest.raises(ValueError): # An `IntersectionSearchSpace` instance isn't supposed to be used for multiple studies. search_space.calculate(study1) optuna-4.1.0/tests/storages_tests/000077500000000000000000000000001471332314300172365ustar00rootroot00000000000000optuna-4.1.0/tests/storages_tests/__init__.py000066400000000000000000000000001471332314300213350ustar00rootroot00000000000000optuna-4.1.0/tests/storages_tests/journal_tests/000077500000000000000000000000001471332314300221325ustar00rootroot00000000000000optuna-4.1.0/tests/storages_tests/journal_tests/assets/000077500000000000000000000000001471332314300234345ustar00rootroot00000000000000optuna-4.1.0/tests/storages_tests/journal_tests/assets/4.0.0.dev.log000066400000000000000000000410451471332314300253570ustar00rootroot00000000000000{"op_code": 0, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "study_name": "single_empty", "directions": [1]} {"op_code": 0, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "study_name": "single_to_be_deleted", "directions": [1]} {"op_code": 1, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "study_id": 1} {"op_code": 0, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "study_name": "single_user_attr", "directions": [1]} {"op_code": 2, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "study_id": 2, "user_attr": {"a": 1}} {"op_code": 2, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "study_id": 2, "user_attr": {"b": 2}} {"op_code": 2, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "study_id": 2, "user_attr": {"c": 3}} {"op_code": 0, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "study_name": "single_system_attr", "directions": [1]} {"op_code": 3, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "study_id": 3, "system_attr": {"A": 1}} {"op_code": 3, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "study_id": 3, "system_attr": {"B": 2}} {"op_code": 3, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "study_id": 3, "system_attr": {"C": 3}} {"op_code": 0, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "study_name": "single_optimization", "directions": [1]} {"op_code": 4, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "study_id": 4, "datetime_start": "2024-06-07T17:27:27.682770"} {"op_code": 5, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 0, "param_name": "x", "param_value_internal": -3.501992262914262, "distribution": "{\"name\": \"FloatDistribution\", \"attributes\": {\"step\": null, \"low\": -5.0, \"high\": 5.0, \"log\": false}}"} {"op_code": 5, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 0, "param_name": "y", "param_value_internal": 3.0, "distribution": "{\"name\": \"IntDistribution\", \"attributes\": {\"log\": false, \"step\": 1, \"low\": 0, \"high\": 10}}"} {"op_code": 5, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 0, "param_name": "z", "param_value_internal": 1, "distribution": "{\"name\": \"CategoricalDistribution\", \"attributes\": {\"choices\": [-5, 0, 5]}}"} {"op_code": 7, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 0, "step": 0, "intermediate_value": 0.5} {"op_code": 8, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 0, "user_attr": {"a_0": 0}} {"op_code": 9, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 0, "system_attr": {"b_0": 1}} {"op_code": 6, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 0, "state": 1, "values": [21.263949809511352], "datetime_complete": "2024-06-07T17:27:27.689197"} {"op_code": 4, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "study_id": 4, "datetime_start": "2024-06-07T17:27:27.690779"} {"op_code": 5, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 1, "param_name": "x", "param_value_internal": -1.7233989164895638, "distribution": "{\"name\": \"FloatDistribution\", \"attributes\": {\"step\": null, \"low\": -5.0, \"high\": 5.0, \"log\": false}}"} {"op_code": 5, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 1, "param_name": "y", "param_value_internal": 3.0, "distribution": "{\"name\": \"IntDistribution\", \"attributes\": {\"log\": false, \"step\": 1, \"low\": 0, \"high\": 10}}"} {"op_code": 5, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 1, "param_name": "z", "param_value_internal": 2, "distribution": "{\"name\": \"CategoricalDistribution\", \"attributes\": {\"choices\": [-5, 0, 5]}}"} {"op_code": 7, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 1, "step": 0, "intermediate_value": 0.5} {"op_code": 8, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 1, "user_attr": {"a_1": 0}} {"op_code": 9, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 1, "system_attr": {"b_1": 1}} {"op_code": 6, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 1, "state": 1, "values": [36.9701038253574], "datetime_complete": "2024-06-07T17:27:27.697112"} {"op_code": 4, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "study_id": 4, "datetime_start": "2024-06-07T17:27:27.698716"} {"op_code": 5, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 2, "param_name": "x", "param_value_internal": 1.9294968260202507, "distribution": "{\"name\": \"FloatDistribution\", \"attributes\": {\"step\": null, \"low\": -5.0, \"high\": 5.0, \"log\": false}}"} {"op_code": 5, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 2, "param_name": "y", "param_value_internal": 10.0, "distribution": "{\"name\": \"IntDistribution\", \"attributes\": {\"log\": false, \"step\": 1, \"low\": 0, \"high\": 10}}"} {"op_code": 5, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 2, "param_name": "z", "param_value_internal": 0, "distribution": "{\"name\": \"CategoricalDistribution\", \"attributes\": {\"choices\": [-5, 0, 5]}}"} {"op_code": 7, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 2, "step": 0, "intermediate_value": 0.5} {"op_code": 8, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 2, "user_attr": {"a_2": 0}} {"op_code": 9, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 2, "system_attr": {"b_2": 1}} {"op_code": 6, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 2, "state": 1, "values": [128.7229580016222], "datetime_complete": "2024-06-07T17:27:27.704621"} {"op_code": 4, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "study_id": 4, "datetime_start": "2024-06-07T17:27:27.706478"} {"op_code": 5, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 3, "param_name": "x", "param_value_internal": -4.644437480216688, "distribution": "{\"name\": \"FloatDistribution\", \"attributes\": {\"step\": null, \"low\": -5.0, \"high\": 5.0, \"log\": false}}"} {"op_code": 5, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 3, "param_name": "y", "param_value_internal": 7.0, "distribution": "{\"name\": \"IntDistribution\", \"attributes\": {\"log\": false, \"step\": 1, \"low\": 0, \"high\": 10}}"} {"op_code": 5, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 3, "param_name": "z", "param_value_internal": 1, "distribution": "{\"name\": \"CategoricalDistribution\", \"attributes\": {\"choices\": [-5, 0, 5]}}"} {"op_code": 7, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 3, "step": 0, "intermediate_value": 0.5} {"op_code": 8, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 3, "user_attr": {"a_3": 0}} {"op_code": 9, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 3, "system_attr": {"b_3": 1}} {"op_code": 6, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 3, "state": 1, "values": [70.57079950764154], "datetime_complete": "2024-06-07T17:27:27.712770"} {"op_code": 4, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "study_id": 4, "datetime_start": "2024-06-07T17:27:27.714422"} {"op_code": 5, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 4, "param_name": "x", "param_value_internal": 2.771729259729552, "distribution": "{\"name\": \"FloatDistribution\", \"attributes\": {\"step\": null, \"low\": -5.0, \"high\": 5.0, \"log\": false}}"} {"op_code": 5, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 4, "param_name": "y", "param_value_internal": 3.0, "distribution": "{\"name\": \"IntDistribution\", \"attributes\": {\"log\": false, \"step\": 1, \"low\": 0, \"high\": 10}}"} {"op_code": 5, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 4, "param_name": "z", "param_value_internal": 2, "distribution": "{\"name\": \"CategoricalDistribution\", \"attributes\": {\"choices\": [-5, 0, 5]}}"} {"op_code": 7, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 4, "step": 0, "intermediate_value": 0.5} {"op_code": 8, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 4, "user_attr": {"a_4": 0}} {"op_code": 9, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 4, "system_attr": {"b_4": 1}} {"op_code": 6, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 4, "state": 1, "values": [41.68248308924093], "datetime_complete": "2024-06-07T17:27:27.721006"} {"op_code": 4, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "study_id": 4, "datetime_start": "2024-06-07T17:27:27.722644"} {"op_code": 5, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 5, "param_name": "x", "param_value_internal": -4.964423870465863, "distribution": "{\"name\": \"FloatDistribution\", \"attributes\": {\"step\": null, \"low\": -5.0, \"high\": 5.0, \"log\": false}}"} {"op_code": 5, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 5, "param_name": "y", "param_value_internal": 2.0, "distribution": "{\"name\": \"IntDistribution\", \"attributes\": {\"log\": false, \"step\": 1, \"low\": 0, \"high\": 10}}"} {"op_code": 5, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 5, "param_name": "z", "param_value_internal": 2, "distribution": "{\"name\": \"CategoricalDistribution\", \"attributes\": {\"choices\": [-5, 0, 5]}}"} {"op_code": 7, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 5, "step": 0, "intermediate_value": 0.5} {"op_code": 8, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 5, "user_attr": {"a_5": 0}} {"op_code": 9, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 5, "system_attr": {"b_5": 1}} {"op_code": 6, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 5, "state": 1, "values": [53.64550436565126], "datetime_complete": "2024-06-07T17:27:27.728591"} {"op_code": 4, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "study_id": 4, "datetime_start": "2024-06-07T17:27:27.730163"} {"op_code": 5, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 6, "param_name": "x", "param_value_internal": 0.05278977851666866, "distribution": "{\"name\": \"FloatDistribution\", \"attributes\": {\"step\": null, \"low\": -5.0, \"high\": 5.0, \"log\": false}}"} {"op_code": 5, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 6, "param_name": "y", "param_value_internal": 7.0, "distribution": "{\"name\": \"IntDistribution\", \"attributes\": {\"log\": false, \"step\": 1, \"low\": 0, \"high\": 10}}"} {"op_code": 5, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 6, "param_name": "z", "param_value_internal": 1, "distribution": "{\"name\": \"CategoricalDistribution\", \"attributes\": {\"choices\": [-5, 0, 5]}}"} {"op_code": 7, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 6, "step": 0, "intermediate_value": 0.5} {"op_code": 8, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 6, "user_attr": {"a_6": 0}} {"op_code": 9, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 6, "system_attr": {"b_6": 1}} {"op_code": 6, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 6, "state": 1, "values": [49.002786760715836], "datetime_complete": "2024-06-07T17:27:27.736569"} {"op_code": 4, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "study_id": 4, "datetime_start": "2024-06-07T17:27:27.738043"} {"op_code": 5, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 7, "param_name": "x", "param_value_internal": -2.604539735900806, "distribution": "{\"name\": \"FloatDistribution\", \"attributes\": {\"step\": null, \"low\": -5.0, \"high\": 5.0, \"log\": false}}"} {"op_code": 5, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 7, "param_name": "y", "param_value_internal": 1.0, "distribution": "{\"name\": \"IntDistribution\", \"attributes\": {\"log\": false, \"step\": 1, \"low\": 0, \"high\": 10}}"} {"op_code": 5, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 7, "param_name": "z", "param_value_internal": 0, "distribution": "{\"name\": \"CategoricalDistribution\", \"attributes\": {\"choices\": [-5, 0, 5]}}"} {"op_code": 7, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 7, "step": 0, "intermediate_value": 0.5} {"op_code": 8, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 7, "user_attr": {"a_7": 0}} {"op_code": 9, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 7, "system_attr": {"b_7": 1}} {"op_code": 6, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 7, "state": 1, "values": [32.78362723588624], "datetime_complete": "2024-06-07T17:27:27.744097"} {"op_code": 4, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "study_id": 4, "datetime_start": "2024-06-07T17:27:27.746062"} {"op_code": 5, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 8, "param_name": "x", "param_value_internal": -2.890244859605886, "distribution": "{\"name\": \"FloatDistribution\", \"attributes\": {\"step\": null, \"low\": -5.0, \"high\": 5.0, \"log\": false}}"} {"op_code": 5, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 8, "param_name": "y", "param_value_internal": 10.0, "distribution": "{\"name\": \"IntDistribution\", \"attributes\": {\"log\": false, \"step\": 1, \"low\": 0, \"high\": 10}}"} {"op_code": 5, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 8, "param_name": "z", "param_value_internal": 1, "distribution": "{\"name\": \"CategoricalDistribution\", \"attributes\": {\"choices\": [-5, 0, 5]}}"} {"op_code": 7, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 8, "step": 0, "intermediate_value": 0.5} {"op_code": 8, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 8, "user_attr": {"a_8": 0}} {"op_code": 9, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 8, "system_attr": {"b_8": 1}} {"op_code": 6, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 8, "state": 1, "values": [108.35351534847825], "datetime_complete": "2024-06-07T17:27:27.756142"} {"op_code": 4, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "study_id": 4, "datetime_start": "2024-06-07T17:27:27.758450"} {"op_code": 5, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 9, "param_name": "x", "param_value_internal": 3.6713526012998496, "distribution": "{\"name\": \"FloatDistribution\", \"attributes\": {\"step\": null, \"low\": -5.0, \"high\": 5.0, \"log\": false}}"} {"op_code": 5, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 9, "param_name": "y", "param_value_internal": 8.0, "distribution": "{\"name\": \"IntDistribution\", \"attributes\": {\"log\": false, \"step\": 1, \"low\": 0, \"high\": 10}}"} {"op_code": 5, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 9, "param_name": "z", "param_value_internal": 0, "distribution": "{\"name\": \"CategoricalDistribution\", \"attributes\": {\"choices\": [-5, 0, 5]}}"} {"op_code": 7, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 9, "step": 0, "intermediate_value": 0.5} {"op_code": 8, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 9, "user_attr": {"a_9": 0}} {"op_code": 9, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 9, "system_attr": {"b_9": 1}} {"op_code": 6, "worker_id": "a4572293-4260-45be-9d2a-489e532dc113-140704507377600", "trial_id": 9, "state": 1, "values": [102.47882992307117], "datetime_complete": "2024-06-07T17:27:27.767730"} optuna-4.1.0/tests/storages_tests/journal_tests/create_journal.py000066400000000000000000000033331471332314300255030ustar00rootroot00000000000000"""This script generates assets for testing backward compatibility of `JournalStorage`. """ from argparse import ArgumentParser import os import optuna from optuna.storages.journal import JournalFileBackend if __name__ == "__main__": parser = ArgumentParser() parser.add_argument( "--storage-url", default=f"{os.path.dirname(__file__)}/assets/{optuna.__version__}.log", ) args = parser.parse_args() storage = optuna.storages.JournalStorage(JournalFileBackend(args.storage_url)) # Empty study optuna.create_study(storage=storage, study_name="single_empty") # Delete the study optuna.create_study(storage=storage, study_name="single_to_be_deleted") optuna.delete_study(storage=storage, study_name="single_to_be_deleted") # Set study user attributes study = optuna.create_study(storage=storage, study_name="single_user_attr") study.set_user_attr("a", 1) study.set_user_attr("b", 2) study.set_user_attr("c", 3) # Set study system attributes study = optuna.create_study(storage=storage, study_name="single_system_attr") study.set_system_attr("A", 1) study.set_system_attr("B", 2) study.set_system_attr("C", 3) # Study for single-objective optimization def objective(trial: optuna.trial.Trial) -> float: x = trial.suggest_float("x", -5, 5) y = trial.suggest_int("y", 0, 10) z = trial.suggest_categorical("z", [-5, 0, 5]) trial.report(0.5, step=0) trial.set_user_attr(f"a_{trial.number}", 0) trial.set_system_attr(f"b_{trial.number}", 1) return x**2 + y**2 + z**2 study = optuna.create_study(storage=storage, study_name="single_optimization") study.optimize(objective, n_trials=10) optuna-4.1.0/tests/storages_tests/journal_tests/test_journal.py000066400000000000000000000236051471332314300252230ustar00rootroot00000000000000from __future__ import annotations from concurrent.futures import as_completed from concurrent.futures import ProcessPoolExecutor from concurrent.futures import ThreadPoolExecutor import pathlib import pickle from types import TracebackType from typing import Any from typing import IO from typing import Optional from typing import Type from unittest import mock import _pytest.capture from fakeredis import FakeStrictRedis import pytest import optuna from optuna import create_study from optuna.storages import BaseJournalLogStorage from optuna.storages import journal from optuna.storages import JournalFileOpenLock as DeprecatedJournalFileOpenLock from optuna.storages import JournalFileSymlinkLock as DeprecatedJournalFileSymlinkLock from optuna.storages import JournalStorage from optuna.storages.journal._base import BaseJournalSnapshot from optuna.storages.journal._file import BaseJournalFileLock from optuna.storages.journal._storage import JournalStorageReplayResult from optuna.testing.storages import StorageSupplier from optuna.testing.tempfile_pool import NamedTemporaryFilePool LOG_STORAGE = [ "file_with_open_lock", "file_with_link_lock", "redis_default", "redis_with_use_cluster", ] JOURNAL_STORAGE_SUPPORTING_SNAPSHOT = ["journal_redis"] class JournalLogStorageSupplier: def __init__(self, storage_type: str) -> None: self.storage_type = storage_type self.tempfile: Optional[IO[Any]] = None def __enter__(self) -> optuna.storages.journal.BaseJournalBackend: if self.storage_type.startswith("file"): self.tempfile = NamedTemporaryFilePool().tempfile() lock: BaseJournalFileLock if self.storage_type == "file_with_open_lock": lock = optuna.storages.journal.JournalFileOpenLock(self.tempfile.name) elif self.storage_type == "file_with_link_lock": lock = optuna.storages.journal.JournalFileSymlinkLock(self.tempfile.name) else: raise Exception("Must not reach here") return optuna.storages.journal.JournalFileBackend(self.tempfile.name, lock) elif self.storage_type.startswith("redis"): use_cluster = self.storage_type == "redis_with_use_cluster" journal_redis_storage = optuna.storages.journal.JournalRedisBackend( "redis://localhost", use_cluster ) journal_redis_storage._redis = FakeStrictRedis() # type: ignore[no-untyped-call] return journal_redis_storage else: raise RuntimeError("Unknown log storage type: {}".format(self.storage_type)) def __exit__( self, exc_type: Type[BaseException], exc_val: BaseException, exc_tb: TracebackType ) -> None: if self.tempfile: self.tempfile.close() @pytest.mark.parametrize("log_storage_type", LOG_STORAGE) def test_concurrent_append_logs_for_multi_processes(log_storage_type: str) -> None: if log_storage_type.startswith("redis"): pytest.skip("The `fakeredis` does not support multi process environments.") num_executors = 10 num_records = 200 record = {"key": "value"} with JournalLogStorageSupplier(log_storage_type) as storage: with ProcessPoolExecutor(num_executors) as pool: pool.map(storage.append_logs, [[record] for _ in range(num_records)], timeout=20) assert len(storage.read_logs(0)) == num_records assert all(record == r for r in storage.read_logs(0)) @pytest.mark.parametrize("log_storage_type", LOG_STORAGE) def test_concurrent_append_logs_for_multi_threads(log_storage_type: str) -> None: num_executors = 10 num_records = 200 record = {"key": "value"} with JournalLogStorageSupplier(log_storage_type) as storage: with ThreadPoolExecutor(num_executors) as pool: pool.map(storage.append_logs, [[record] for _ in range(num_records)], timeout=20) assert len(storage.read_logs(0)) == num_records assert all(record == r for r in storage.read_logs(0)) def pop_waiting_trial(file_path: str, study_name: str) -> Optional[int]: file_storage = optuna.storages.journal.JournalFileBackend(file_path) storage = optuna.storages.JournalStorage(file_storage) study = optuna.load_study(storage=storage, study_name=study_name) return study._pop_waiting_trial_id() def test_pop_waiting_trial_multiprocess_safe() -> None: with NamedTemporaryFilePool() as file: file_storage = optuna.storages.journal.JournalFileBackend(file.name) storage = optuna.storages.JournalStorage(file_storage) study = optuna.create_study(storage=storage) num_enqueued = 10 for i in range(num_enqueued): study.enqueue_trial({"i": i}) trial_id_set = set() with ProcessPoolExecutor(10) as pool: futures = [] for i in range(num_enqueued): future = pool.submit(pop_waiting_trial, file.name, study.study_name) futures.append(future) for future in as_completed(futures): trial_id = future.result() if trial_id is not None: trial_id_set.add(trial_id) assert len(trial_id_set) == num_enqueued @pytest.mark.parametrize("storage_mode", JOURNAL_STORAGE_SUPPORTING_SNAPSHOT) def test_save_snapshot_per_each_trial(storage_mode: str) -> None: def objective(trial: optuna.Trial) -> float: return trial.suggest_float("x", 0, 10) with StorageSupplier(storage_mode) as storage: assert isinstance(storage, JournalStorage) study = create_study(storage=storage) journal_log_storage = storage._backend assert isinstance(journal_log_storage, BaseJournalSnapshot) assert journal_log_storage.load_snapshot() is None with mock.patch("optuna.storages.journal._storage.SNAPSHOT_INTERVAL", 1, create=True): study.optimize(objective, n_trials=2) assert isinstance(journal_log_storage.load_snapshot(), bytes) @pytest.mark.parametrize("storage_mode", JOURNAL_STORAGE_SUPPORTING_SNAPSHOT) def test_save_snapshot_per_each_study(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: assert isinstance(storage, JournalStorage) journal_log_storage = storage._backend assert isinstance(journal_log_storage, BaseJournalSnapshot) assert journal_log_storage.load_snapshot() is None with mock.patch("optuna.storages.journal._storage.SNAPSHOT_INTERVAL", 1, create=True): for _ in range(2): create_study(storage=storage) assert isinstance(journal_log_storage.load_snapshot(), bytes) @pytest.mark.parametrize("storage_mode", JOURNAL_STORAGE_SUPPORTING_SNAPSHOT) def test_check_replay_result_restored_from_snapshot(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage1: with mock.patch("optuna.storages.journal._storage.SNAPSHOT_INTERVAL", 1, create=True): for _ in range(2): create_study(storage=storage1) assert isinstance(storage1, JournalStorage) storage2 = optuna.storages.JournalStorage(storage1._backend) assert len(storage1.get_all_studies()) == len(storage2.get_all_studies()) assert storage1._replay_result.log_number_read == storage2._replay_result.log_number_read @pytest.mark.parametrize("storage_mode", JOURNAL_STORAGE_SUPPORTING_SNAPSHOT) def test_snapshot_given(storage_mode: str, capsys: _pytest.capture.CaptureFixture) -> None: with StorageSupplier(storage_mode) as storage: assert isinstance(storage, JournalStorage) replay_result = JournalStorageReplayResult("") # Bytes object which is a valid pickled object. storage.restore_replay_result(pickle.dumps(replay_result)) assert replay_result.log_number_read == storage._replay_result.log_number_read # We need to reconstruct our default handler to properly capture stderr. optuna.logging._reset_library_root_logger() optuna.logging.enable_default_handler() optuna.logging.set_verbosity(optuna.logging.WARNING) # Bytes object which cannot be unpickled is passed. storage.restore_replay_result(b"hoge") _, err = capsys.readouterr() assert err # Bytes object which can be pickled but is not `JournalStorageReplayResult`. storage.restore_replay_result(pickle.dumps("hoge")) _, err = capsys.readouterr() assert err def test_if_future_warning_occurs() -> None: with NamedTemporaryFilePool() as file: with pytest.warns(FutureWarning): optuna.storages.JournalFileStorage(file.name) with pytest.warns(FutureWarning): optuna.storages.JournalRedisStorage("redis://localhost") class _CustomJournalBackendInheritingDeprecatedClass(BaseJournalLogStorage): def read_logs(self, log_number_from: int) -> list[dict[str, Any]]: return [{"": ""}] def append_logs(self, logs: list[dict[str, Any]]) -> None: return with pytest.warns(FutureWarning): _ = _CustomJournalBackendInheritingDeprecatedClass() @pytest.mark.parametrize( "lock_obj", (DeprecatedJournalFileOpenLock, DeprecatedJournalFileSymlinkLock) ) def test_future_warning_of_deprecated_file_lock_obj_paths( tmp_path: pathlib.PurePath, lock_obj: type[DeprecatedJournalFileOpenLock | DeprecatedJournalFileSymlinkLock], ) -> None: with pytest.warns(FutureWarning): lock_obj(filepath=str(tmp_path)) def test_raise_error_for_deprecated_class_import_from_journal() -> None: # TODO(nabenabe0928): Remove this test once deprecated objects, e.g., JournalFileStorage, # are removed. with pytest.raises(AttributeError): journal.JournalFileStorage # type: ignore[attr-defined] with pytest.raises(AttributeError): journal.JournalRedisStorage # type: ignore[attr-defined] with pytest.raises(AttributeError): journal.BaseJournalLogStorage # type: ignore[attr-defined] optuna-4.1.0/tests/storages_tests/journal_tests/test_log_compatibility.py000066400000000000000000000073171471332314300272650ustar00rootroot00000000000000import os import pytest import optuna from optuna.storages.journal import JournalFileBackend all_journal_files = [f"{os.path.dirname(__file__)}/assets/4.0.0.dev.log"] @pytest.mark.parametrize("journal_file", all_journal_files) def test_empty_study(journal_file: str) -> None: storage = optuna.storages.JournalStorage(JournalFileBackend(journal_file)) study = optuna.load_study(study_name="single_empty", storage=storage) assert study.directions == [optuna.study.StudyDirection.MINIMIZE] assert len(study.user_attrs) == 0 assert len(study.trials) == 0 @pytest.mark.parametrize("journal_file", all_journal_files) def test_create_and_delete_study(journal_file: str) -> None: storage = optuna.storages.JournalStorage(JournalFileBackend(journal_file)) assert len(storage.get_all_studies()) == 4 assert storage.get_study_id_from_name("single_empty") is not None with pytest.raises(KeyError): storage.get_study_id_from_name("single_to_be_deleted") @pytest.mark.parametrize("journal_file", all_journal_files) def test_set_study_user_and_system_attrs(journal_file: str) -> None: storage = optuna.storages.JournalStorage(JournalFileBackend(journal_file)) study_id = storage.get_study_id_from_name("single_user_attr") user_attrs = storage.get_study_user_attrs(study_id) assert user_attrs["a"] == 1 assert len(user_attrs) == 3 study_id = storage.get_study_id_from_name("single_system_attr") system_attrs = storage.get_study_system_attrs(study_id) assert system_attrs["A"] == 1 assert len(system_attrs) == 3 @pytest.mark.parametrize("journal_file", all_journal_files) def test_create_trial(journal_file: str) -> None: storage = optuna.storages.JournalStorage(JournalFileBackend(journal_file)) study_id = storage.get_study_id_from_name("single_optimization") trials = storage.get_all_trials(study_id) assert len(trials) == 10 @pytest.mark.parametrize("journal_file", all_journal_files) def test_set_trial_param(journal_file: str) -> None: storage = optuna.storages.JournalStorage(JournalFileBackend(journal_file)) study_id = storage.get_study_id_from_name("single_optimization") trials = storage.get_all_trials(study_id) for trial in trials: assert -5 <= trial.params["x"] <= 5 assert 0 <= trial.params["y"] <= 10 assert trial.params["z"] in [-5, 0, 5] @pytest.mark.parametrize("journal_file", all_journal_files) def test_set_trial_state_values(journal_file: str) -> None: storage = optuna.storages.JournalStorage(JournalFileBackend(journal_file)) study_id = storage.get_study_id_from_name("single_optimization") trials = storage.get_all_trials(study_id) for trial in trials: assert trial.state == optuna.trial.TrialState.COMPLETE assert trial.value is not None and 0 <= trial.value <= 150 @pytest.mark.parametrize("journal_file", all_journal_files) def test_set_trial_intermediate_value(journal_file: str) -> None: storage = optuna.storages.JournalStorage(JournalFileBackend(journal_file)) study_id = storage.get_study_id_from_name("single_optimization") trials = storage.get_all_trials(study_id) for trial in trials: assert len(trial.intermediate_values) == 1 assert trial.intermediate_values[0] == 0.5 @pytest.mark.parametrize("journal_file", all_journal_files) def test_set_trial_user_and_system_attrs(journal_file: str) -> None: storage = optuna.storages.JournalStorage(JournalFileBackend(journal_file)) study_id = storage.get_study_id_from_name("single_optimization") trials = storage.get_all_trials(study_id) for trial in trials: assert trial.user_attrs[f"a_{trial.number}"] == 0 assert trial.system_attrs[f"b_{trial.number}"] == 1 optuna-4.1.0/tests/storages_tests/rdb_tests/000077500000000000000000000000001471332314300212275ustar00rootroot00000000000000optuna-4.1.0/tests/storages_tests/rdb_tests/__init__.py000066400000000000000000000000001471332314300233260ustar00rootroot00000000000000optuna-4.1.0/tests/storages_tests/rdb_tests/create_db.py000066400000000000000000000075231471332314300235200ustar00rootroot00000000000000"""This script generates assets for testing schema migration. 1. Prepare Optuna If you want to generate a DB file for the latest version of Optuna, you have to edit `optuna/version.py` since we add a suffix to a version in the master branch. > cat optuna/version.py __version__ = "3.0.0b0.dev" Please temporarily remove the suffix when running this script. After generating an asset, the change should be reverted. If you want to generate a DB file for older versions of Optuna, you have to install it. I recommend you to create isolated environment using `venv` for this purpose. ```sh > deactivate # if you already use `venv` for development > python3 -m venv venv_gen > . venv_gen/bin/activate > pip install optuna==2.6.0 # install Optuna v2.6.0 ``` 2. Generate database ```sh > python3 create_db.py [I 2022-02-05 15:39:32,488] A new study created in RDB with name: single_empty ... > ``` 3. Switch Optuna version to the latest one If you use `venv`, simply `deactivate` and re-activate your development environment. """ from argparse import ArgumentParser from typing import Tuple from packaging import version import optuna def objective_test_upgrade(trial: optuna.trial.Trial) -> float: x = trial.suggest_float("x", -5, 5) # optuna==0.9.0 does not have suggest_float. y = trial.suggest_int("y", 0, 10) z = trial.suggest_categorical("z", [-5, 0, 5]) trial.storage.set_trial_system_attr(trial._trial_id, "a", 0) trial.set_user_attr("b", 1) trial.report(0.5, step=0) return x**2 + y**2 + z**2 def mo_objective_test_upgrade(trial: optuna.trial.Trial) -> Tuple[float, float]: x = trial.suggest_float("x", -5, 5) y = trial.suggest_int("y", 0, 10) z = trial.suggest_categorical("z", [-5, 0, 5]) trial.storage.set_trial_system_attr(trial._trial_id, "a", 0) trial.set_user_attr("b", 1) return x, x**2 + y**2 + z**2 def objective_test_upgrade_distributions(trial: optuna.trial.Trial) -> float: x1 = trial.suggest_float("x1", -5, 5) x2 = trial.suggest_float("x2", 1e-5, 1e-3, log=True) x3 = trial.suggest_float("x3", -6, 6, step=2) y1 = trial.suggest_int("y1", 0, 10) y2 = trial.suggest_int("y2", 1, 20, log=True) y3 = trial.suggest_int("y3", 5, 15, step=3) z = trial.suggest_categorical("z", [-5, 0, 5]) return x1**2 + x2**2 + x3**2 + y1**2 + y2**2 + y3**2 + z**2 if __name__ == "__main__": parser = ArgumentParser(description="Create SQLite database for schema upgrade tests.") parser.add_argument( "--storage-url", default=f"sqlite:///test_upgrade_assets/{optuna.__version__}.db" ) args = parser.parse_args() # Create an empty study. optuna.create_study(storage=args.storage_url, study_name="single_empty") # Create a study for single-objective optimization. study = optuna.create_study(storage=args.storage_url, study_name="single") study.set_user_attr("d", 3) study.optimize(objective_test_upgrade, n_trials=1) # Create a study for multi-objective optimization. try: optuna.create_study( storage=args.storage_url, study_name="multi_empty", directions=["minimize", "minimize"] ) study = optuna.create_study( storage=args.storage_url, study_name="multi", directions=["minimize", "minimize"] ) study.set_user_attr("d", 3) study.optimize(mo_objective_test_upgrade, n_trials=1) except TypeError: print(f"optuna=={optuna.__version__} does not support multi-objective optimization.") # Create a study for distributions upgrade. if version.parse(optuna.__version__) >= version.parse("2.4.0"): study = optuna.create_study(storage=args.storage_url, study_name="schema migration") study.optimize(objective_test_upgrade_distributions, n_trials=1) for s in optuna.get_all_study_summaries(args.storage_url): print(f"{s.study_name}, {s.n_trials}") optuna-4.1.0/tests/storages_tests/rdb_tests/test_models.py000066400000000000000000000407241471332314300241320ustar00rootroot00000000000000from datetime import datetime import pytest from sqlalchemy import create_engine from sqlalchemy.exc import IntegrityError from sqlalchemy.orm import Session from optuna.storages._rdb.models import BaseModel from optuna.storages._rdb.models import StudyDirectionModel from optuna.storages._rdb.models import StudyModel from optuna.storages._rdb.models import StudySystemAttributeModel from optuna.storages._rdb.models import TrialHeartbeatModel from optuna.storages._rdb.models import TrialIntermediateValueModel from optuna.storages._rdb.models import TrialModel from optuna.storages._rdb.models import TrialSystemAttributeModel from optuna.storages._rdb.models import TrialUserAttributeModel from optuna.storages._rdb.models import TrialValueModel from optuna.storages._rdb.models import VersionInfoModel from optuna.study._study_direction import StudyDirection from optuna.trial import TrialState @pytest.fixture def session() -> Session: engine = create_engine("sqlite:///:memory:") BaseModel.metadata.create_all(engine) return Session(bind=engine) class TestStudyDirectionModel: @staticmethod def _create_model(session: Session) -> StudyModel: study = StudyModel(study_id=1, study_name="test-study") dummy_study = StudyModel(study_id=2, study_name="dummy-study") session.add( StudyDirectionModel( study_id=study.study_id, direction=StudyDirection.MINIMIZE, objective=0 ) ) session.add( StudyDirectionModel( study_id=dummy_study.study_id, direction=StudyDirection.MINIMIZE, objective=0 ) ) session.commit() return study @staticmethod def test_where_study_id(session: Session) -> None: study = TestStudyDirectionModel._create_model(session) assert 1 == len(StudyDirectionModel.where_study_id(study.study_id, session)) assert 0 == len(StudyDirectionModel.where_study_id(-1, session)) @staticmethod def test_cascade_delete_on_study(session: Session) -> None: directions = [ StudyDirectionModel(study_id=1, direction=StudyDirection.MINIMIZE, objective=0), StudyDirectionModel(study_id=1, direction=StudyDirection.MAXIMIZE, objective=1), ] study = StudyModel(study_id=1, study_name="test-study", directions=directions) session.add(study) session.commit() assert 2 == len(StudyDirectionModel.where_study_id(study.study_id, session)) session.delete(study) session.commit() assert 0 == len(StudyDirectionModel.where_study_id(study.study_id, session)) class TestStudySystemAttributeModel: @staticmethod def test_find_by_study_and_key(session: Session) -> None: study = StudyModel(study_id=1, study_name="test-study") session.add( StudySystemAttributeModel(study_id=study.study_id, key="sample-key", value_json="1") ) session.commit() attr = StudySystemAttributeModel.find_by_study_and_key(study, "sample-key", session) assert attr is not None and "1" == attr.value_json assert StudySystemAttributeModel.find_by_study_and_key(study, "not-found", session) is None @staticmethod def test_where_study_id(session: Session) -> None: sample_study = StudyModel(study_id=1, study_name="test-study") empty_study = StudyModel(study_id=2, study_name="test-study") session.add( StudySystemAttributeModel( study_id=sample_study.study_id, key="sample-key", value_json="1" ) ) assert 1 == len(StudySystemAttributeModel.where_study_id(sample_study.study_id, session)) assert 0 == len(StudySystemAttributeModel.where_study_id(empty_study.study_id, session)) # Check the case of unknown study_id. assert 0 == len(StudySystemAttributeModel.where_study_id(-1, session)) @staticmethod def test_cascade_delete_on_study(session: Session) -> None: study_id = 1 direction = StudyDirectionModel(direction=StudyDirection.MINIMIZE, objective=0) study = StudyModel(study_id=study_id, study_name="test-study", directions=[direction]) study.system_attributes.append( StudySystemAttributeModel(study_id=study_id, key="sample-key1", value_json="1") ) study.system_attributes.append( StudySystemAttributeModel(study_id=study_id, key="sample-key2", value_json="2") ) session.add(study) session.commit() assert 2 == len(StudySystemAttributeModel.where_study_id(study_id, session)) session.delete(study) session.commit() assert 0 == len(StudySystemAttributeModel.where_study_id(study_id, session)) class TestTrialModel: @staticmethod def test_default_datetime(session: Session) -> None: # Regardless of the initial state the trial created here should have null datetime_start session.add(TrialModel(state=TrialState.WAITING)) session.commit() trial_model = session.query(TrialModel).first() assert trial_model is not None assert trial_model.datetime_start is None assert trial_model.datetime_complete is None @staticmethod def test_count(session: Session) -> None: study_1 = StudyModel(study_id=1, study_name="test-study-1") study_2 = StudyModel(study_id=2, study_name="test-study-2") session.add(TrialModel(study_id=study_1.study_id, state=TrialState.COMPLETE)) session.add(TrialModel(study_id=study_1.study_id, state=TrialState.RUNNING)) session.add(TrialModel(study_id=study_2.study_id, state=TrialState.RUNNING)) session.commit() assert 3 == TrialModel.count(session) assert 2 == TrialModel.count(session, study=study_1) assert 1 == TrialModel.count(session, state=TrialState.COMPLETE) @staticmethod def test_count_past_trials(session: Session) -> None: study_1 = StudyModel(study_id=1, study_name="test-study-1") study_2 = StudyModel(study_id=2, study_name="test-study-2") trial_1_1 = TrialModel(study_id=study_1.study_id, state=TrialState.COMPLETE) session.add(trial_1_1) session.commit() assert 0 == trial_1_1.count_past_trials(session) trial_1_2 = TrialModel(study_id=study_1.study_id, state=TrialState.RUNNING) session.add(trial_1_2) session.commit() assert 1 == trial_1_2.count_past_trials(session) trial_2_1 = TrialModel(study_id=study_2.study_id, state=TrialState.RUNNING) session.add(trial_2_1) session.commit() assert 0 == trial_2_1.count_past_trials(session) @staticmethod def test_cascade_delete_on_study(session: Session) -> None: study_id = 1 direction = StudyDirectionModel(direction=StudyDirection.MINIMIZE, objective=0) study = StudyModel(study_id=study_id, study_name="test-study", directions=[direction]) study.trials.append(TrialModel(study_id=study.study_id, state=TrialState.COMPLETE)) study.trials.append(TrialModel(study_id=study.study_id, state=TrialState.RUNNING)) session.add(study) session.commit() assert 2 == TrialModel.count(session, study) session.delete(study) session.commit() assert 0 == TrialModel.count(session, study) class TestTrialUserAttributeModel: @staticmethod def test_find_by_trial_and_key(session: Session) -> None: study = StudyModel(study_id=1, study_name="test-study") trial = TrialModel(study_id=study.study_id) session.add( TrialUserAttributeModel(trial_id=trial.trial_id, key="sample-key", value_json="1") ) session.commit() attr = TrialUserAttributeModel.find_by_trial_and_key(trial, "sample-key", session) assert attr is not None assert "1" == attr.value_json assert TrialUserAttributeModel.find_by_trial_and_key(trial, "not-found", session) is None @staticmethod def test_cascade_delete_on_trial(session: Session) -> None: trial_id = 1 direction = StudyDirectionModel(direction=StudyDirection.MINIMIZE, objective=0) study = StudyModel(study_id=1, study_name="test-study", directions=[direction]) trial = TrialModel(trial_id=trial_id, study_id=study.study_id, state=TrialState.COMPLETE) trial.user_attributes.append( TrialUserAttributeModel(trial_id=trial_id, key="sample-key1", value_json="1") ) trial.user_attributes.append( TrialUserAttributeModel(trial_id=trial_id, key="sample-key2", value_json="2") ) study.trials.append(trial) session.add(study) session.commit() assert 2 == len(TrialUserAttributeModel.where_trial_id(trial_id, session)) session.delete(trial) session.commit() assert 0 == len(TrialUserAttributeModel.where_trial_id(trial_id, session)) class TestTrialSystemAttributeModel: @staticmethod def test_find_by_trial_and_key(session: Session) -> None: study = StudyModel(study_id=1, study_name="test-study") trial = TrialModel(study_id=study.study_id) session.add( TrialSystemAttributeModel(trial_id=trial.trial_id, key="sample-key", value_json="1") ) session.commit() attr = TrialSystemAttributeModel.find_by_trial_and_key(trial, "sample-key", session) assert attr is not None assert "1" == attr.value_json assert TrialSystemAttributeModel.find_by_trial_and_key(trial, "not-found", session) is None @staticmethod def test_cascade_delete_on_trial(session: Session) -> None: trial_id = 1 direction = StudyDirectionModel(direction=StudyDirection.MINIMIZE, objective=0) study = StudyModel(study_id=1, study_name="test-study", directions=[direction]) trial = TrialModel(trial_id=trial_id, study_id=study.study_id, state=TrialState.COMPLETE) trial.system_attributes.append( TrialSystemAttributeModel(trial_id=trial_id, key="sample-key1", value_json="1") ) trial.system_attributes.append( TrialSystemAttributeModel(trial_id=trial_id, key="sample-key2", value_json="2") ) study.trials.append(trial) session.add(study) session.commit() assert 2 == len(TrialSystemAttributeModel.where_trial_id(trial_id, session)) session.delete(trial) session.commit() assert 0 == len(TrialSystemAttributeModel.where_trial_id(trial_id, session)) class TestTrialValueModel: @staticmethod def _create_model(session: Session) -> TrialModel: direction = StudyDirectionModel(direction=StudyDirection.MINIMIZE, objective=0) study = StudyModel(study_id=1, study_name="test-study", directions=[direction]) trial = TrialModel(trial_id=1, study_id=study.study_id, state=TrialState.COMPLETE) session.add(study) session.add(trial) session.add( TrialValueModel( trial_id=trial.trial_id, objective=0, value=10, value_type=TrialValueModel.TrialValueType.FINITE, ) ) session.commit() return trial @staticmethod def test_find_by_trial_and_objective(session: Session) -> None: trial = TestTrialValueModel._create_model(session) trial_value = TrialValueModel.find_by_trial_and_objective(trial, 0, session) assert trial_value is not None assert 10 == trial_value.value assert TrialValueModel.find_by_trial_and_objective(trial, 1, session) is None @staticmethod def test_where_trial_id(session: Session) -> None: trial = TestTrialValueModel._create_model(session) trial_values = TrialValueModel.where_trial_id(trial.trial_id, session) assert 1 == len(trial_values) assert 0 == trial_values[0].objective assert 10 == trial_values[0].value @staticmethod def test_cascade_delete_on_trial(session: Session) -> None: trial = TestTrialValueModel._create_model(session) trial.values.append( TrialValueModel( trial_id=1, objective=1, value=20, value_type=TrialValueModel.TrialValueType.FINITE ) ) session.commit() assert 2 == len(TrialValueModel.where_trial_id(trial.trial_id, session)) session.delete(trial) session.commit() assert 0 == len(TrialValueModel.where_trial_id(trial.trial_id, session)) class TestTrialIntermediateValueModel: @staticmethod def _create_model(session: Session) -> TrialModel: direction = StudyDirectionModel(direction=StudyDirection.MINIMIZE, objective=0) study = StudyModel(study_id=1, study_name="test-study", directions=[direction]) trial = TrialModel(trial_id=1, study_id=study.study_id, state=TrialState.COMPLETE) session.add(study) session.add(trial) session.add( TrialIntermediateValueModel( trial_id=trial.trial_id, step=0, intermediate_value=10, intermediate_value_type=TrialIntermediateValueModel.TrialIntermediateValueType.FINITE, # noqa: E501 ) ) session.commit() return trial @staticmethod def test_find_by_trial_and_step(session: Session) -> None: trial = TestTrialIntermediateValueModel._create_model(session) trial_intermediate_value = TrialIntermediateValueModel.find_by_trial_and_step( trial, 0, session ) assert trial_intermediate_value is not None assert 10 == trial_intermediate_value.intermediate_value assert TrialIntermediateValueModel.find_by_trial_and_step(trial, 1, session) is None @staticmethod def test_where_trial_id(session: Session) -> None: trial = TestTrialIntermediateValueModel._create_model(session) trial_intermediate_values = TrialIntermediateValueModel.where_trial_id( trial.trial_id, session ) assert 1 == len(trial_intermediate_values) assert 0 == trial_intermediate_values[0].step assert 10 == trial_intermediate_values[0].intermediate_value @staticmethod def test_cascade_delete_on_trial(session: Session) -> None: trial = TestTrialIntermediateValueModel._create_model(session) trial.intermediate_values.append( TrialIntermediateValueModel( trial_id=1, step=1, intermediate_value=20, intermediate_value_type=TrialIntermediateValueModel.TrialIntermediateValueType.FINITE, # noqa: E501 ) ) session.commit() assert 2 == len(TrialIntermediateValueModel.where_trial_id(trial.trial_id, session)) session.delete(trial) session.commit() assert 0 == len(TrialIntermediateValueModel.where_trial_id(trial.trial_id, session)) class TestTrialHeartbeatModel: @staticmethod def _create_model(session: Session) -> TrialModel: direction = StudyDirectionModel(direction=StudyDirection.MINIMIZE, objective=0) study = StudyModel(study_id=1, study_name="test-study", directions=[direction]) trial = TrialModel(trial_id=1, study_id=study.study_id, state=TrialState.COMPLETE) session.add(study) session.add(trial) session.add(TrialHeartbeatModel(trial_id=trial.trial_id)) session.commit() return trial @staticmethod def test_where_trial_id(session: Session) -> None: trial = TestTrialHeartbeatModel._create_model(session) trial_heartbeat = TrialHeartbeatModel.where_trial_id(trial.trial_id, session) assert trial_heartbeat is not None assert isinstance(trial_heartbeat.heartbeat, datetime) @staticmethod def test_cascade_delete_on_trial(session: Session) -> None: trial = TestTrialHeartbeatModel._create_model(session) session.commit() assert TrialHeartbeatModel.where_trial_id(trial.trial_id, session) is not None session.delete(trial) session.commit() assert TrialHeartbeatModel.where_trial_id(trial.trial_id, session) is None class TestVersionInfoModel: @staticmethod def test_version_info_id_constraint(session: Session) -> None: session.add(VersionInfoModel(schema_version=1, library_version="0.0.1")) session.commit() # Test check constraint of version_info_id. session.add(VersionInfoModel(version_info_id=2, schema_version=2, library_version="0.0.2")) pytest.raises(IntegrityError, lambda: session.commit()) optuna-4.1.0/tests/storages_tests/rdb_tests/test_storage.py000066400000000000000000000321711471332314300243100ustar00rootroot00000000000000from datetime import datetime import os import platform import shutil import sys import tempfile from typing import Any from typing import Dict from typing import Optional from unittest.mock import Mock from unittest.mock import patch import warnings import pytest import sqlalchemy.exc as sqlalchemy_exc from sqlalchemy.exc import IntegrityError import sqlalchemy.orm as sqlalchemy_orm import optuna from optuna import create_study from optuna import load_study from optuna.distributions import CategoricalDistribution from optuna.distributions import FloatDistribution from optuna.distributions import IntDistribution from optuna.storages import RDBStorage from optuna.storages._rdb import models from optuna.storages._rdb.models import SCHEMA_VERSION from optuna.storages._rdb.models import VersionInfoModel from optuna.storages._rdb.storage import _create_scoped_session from optuna.study import StudyDirection from optuna.testing.tempfile_pool import NamedTemporaryFilePool from optuna.trial import FrozenTrial from optuna.trial import TrialState from .create_db import mo_objective_test_upgrade from .create_db import objective_test_upgrade from .create_db import objective_test_upgrade_distributions def test_init() -> None: storage = create_test_storage() session = storage.scoped_session() version_info = session.query(VersionInfoModel).first() assert version_info is not None assert version_info.schema_version == SCHEMA_VERSION assert version_info.library_version == optuna.version.__version__ assert storage.get_current_version() == storage.get_head_version() assert storage.get_all_versions() == [ "v3.2.0.a", "v3.0.0.d", "v3.0.0.c", "v3.0.0.b", "v3.0.0.a", "v2.6.0.a", "v2.4.0.a", "v1.3.0.a", "v1.2.0.a", "v0.9.0.a", ] def test_init_url_template() -> None: with NamedTemporaryFilePool(suffix="{SCHEMA_VERSION}") as tf: storage = RDBStorage("sqlite:///" + tf.name) assert storage.engine.url.database is not None assert storage.engine.url.database.endswith(str(SCHEMA_VERSION)) def test_init_url_that_contains_percent_character() -> None: # Alembic's ini file regards '%' as the special character for variable expansion. # We checks `RDBStorage` does not raise an error even if a storage url contains the character. with NamedTemporaryFilePool(suffix="%") as tf: RDBStorage("sqlite:///" + tf.name) with NamedTemporaryFilePool(suffix="%foo") as tf: RDBStorage("sqlite:///" + tf.name) with NamedTemporaryFilePool(suffix="%foo%%bar") as tf: RDBStorage("sqlite:///" + tf.name) def test_init_db_module_import_error() -> None: expected_msg = ( "Failed to import DB access module for the specified storage URL. " "Please install appropriate one." ) with patch.dict(sys.modules, {"psycopg2": None}): with pytest.raises(ImportError, match=expected_msg): RDBStorage("postgresql://user:password@host/database") def test_engine_kwargs() -> None: create_test_storage(engine_kwargs={"pool_size": 5}) @pytest.mark.parametrize( "url,engine_kwargs,expected", [ ("mysql://localhost", {"pool_pre_ping": False}, False), ("mysql://localhost", {"pool_pre_ping": True}, True), ("mysql://localhost", {}, True), ("mysql+pymysql://localhost", {}, True), ("mysql://localhost", {"pool_size": 5}, True), ], ) def test_set_default_engine_kwargs_for_mysql( url: str, engine_kwargs: Dict[str, Any], expected: bool ) -> None: RDBStorage._set_default_engine_kwargs_for_mysql(url, engine_kwargs) assert engine_kwargs["pool_pre_ping"] is expected def test_set_default_engine_kwargs_for_mysql_with_other_rdb() -> None: # Do not change engine_kwargs if database is not MySQL. engine_kwargs: Dict[str, Any] = {} RDBStorage._set_default_engine_kwargs_for_mysql("sqlite:///example.db", engine_kwargs) assert "pool_pre_ping" not in engine_kwargs RDBStorage._set_default_engine_kwargs_for_mysql("postgres:///example.db", engine_kwargs) assert "pool_pre_ping" not in engine_kwargs def test_check_table_schema_compatibility() -> None: storage = create_test_storage() session = storage.scoped_session() # The schema version of a newly created storage is always up-to-date. storage._version_manager.check_table_schema_compatibility() # `SCHEMA_VERSION` has not been used for compatibility check since alembic was introduced. version_info = session.query(VersionInfoModel).one() version_info.schema_version = SCHEMA_VERSION - 1 session.commit() storage._version_manager.check_table_schema_compatibility() with pytest.raises(RuntimeError): storage._version_manager._set_alembic_revision( storage._version_manager._get_base_version() ) storage._version_manager.check_table_schema_compatibility() def create_test_storage(engine_kwargs: Optional[Dict[str, Any]] = None) -> RDBStorage: storage = RDBStorage("sqlite:///:memory:", engine_kwargs=engine_kwargs) return storage def test_create_scoped_session() -> None: storage = create_test_storage() # This object violates the unique constraint of version_info_id. v = VersionInfoModel(version_info_id=1, schema_version=1, library_version="0.0.1") with pytest.raises(IntegrityError): with _create_scoped_session(storage.scoped_session) as session: session.add(v) def test_upgrade_identity() -> None: storage = create_test_storage() # `upgrade()` has no effect because the storage version is already up-to-date. old_version = storage.get_current_version() storage.upgrade() new_version = storage.get_current_version() assert old_version == new_version @pytest.mark.skipif(platform.system() == "Windows", reason="Skip on Windows") @pytest.mark.parametrize( "optuna_version", [ "0.9.0.a", "1.2.0.a", "1.3.0.a", "2.4.0.a", "2.6.0.a", "3.0.0.a", "3.0.0.b", "3.0.0.c", "3.0.0.d", "3.2.0.a", ], ) def test_upgrade_single_objective_optimization(optuna_version: str) -> None: src_db_file = os.path.join( os.path.dirname(__file__), "test_upgrade_assets", f"{optuna_version}.db" ) with tempfile.TemporaryDirectory() as workdir: shutil.copyfile(src_db_file, f"{workdir}/sqlite.db") storage_url = f"sqlite:///{workdir}/sqlite.db" storage = RDBStorage( storage_url, skip_compatibility_check=True, skip_table_creation=True, ) assert storage.get_current_version() == f"v{optuna_version}" head_version = storage.get_head_version() with warnings.catch_warnings(): warnings.simplefilter("ignore", category=FutureWarning) storage.upgrade() assert head_version == storage.get_current_version() # Create a new study. study = create_study(storage=storage) assert len(study.trials) == 0 study.optimize(objective_test_upgrade, n_trials=1) assert len(study.trials) == 1 # Check empty study. study = load_study(storage=storage, study_name="single_empty") assert len(study.trials) == 0 study.optimize(objective_test_upgrade, n_trials=1) assert len(study.trials) == 1 # Resume single objective optimization. study = load_study(storage=storage, study_name="single") assert len(study.trials) == 1 study.optimize(objective_test_upgrade, n_trials=1) assert len(study.trials) == 2 for trial in study.trials: assert trial.user_attrs["b"] == 1 assert trial.intermediate_values[0] == 0.5 assert -5 <= trial.params["x"] <= 5 assert 0 <= trial.params["y"] <= 10 assert trial.params["z"] in (-5, 0, 5) assert trial.value is not None and 0 <= trial.value <= 150 assert study.user_attrs["d"] == 3 storage.engine.dispose() # Be sure to disconnect db @pytest.mark.skipif(platform.system() == "Windows", reason="Skip on Windows") @pytest.mark.parametrize( "optuna_version", ["2.4.0.a", "2.6.0.a", "3.0.0.a", "3.0.0.b", "3.0.0.c", "3.0.0.d", "3.2.0.a"] ) def test_upgrade_multi_objective_optimization(optuna_version: str) -> None: src_db_file = os.path.join( os.path.dirname(__file__), "test_upgrade_assets", f"{optuna_version}.db" ) with tempfile.TemporaryDirectory() as workdir: shutil.copyfile(src_db_file, f"{workdir}/sqlite.db") storage_url = f"sqlite:///{workdir}/sqlite.db" storage = RDBStorage(storage_url, skip_compatibility_check=True, skip_table_creation=True) assert storage.get_current_version() == f"v{optuna_version}" head_version = storage.get_head_version() with warnings.catch_warnings(): warnings.simplefilter("ignore", category=FutureWarning) storage.upgrade() assert head_version == storage.get_current_version() # Create a new study. study = create_study(storage=storage, directions=["minimize", "minimize"]) assert len(study.trials) == 0 study.optimize(mo_objective_test_upgrade, n_trials=1) assert len(study.trials) == 1 # Check empty study. study = load_study(storage=storage, study_name="multi_empty") assert len(study.trials) == 0 study.optimize(mo_objective_test_upgrade, n_trials=1) assert len(study.trials) == 1 # Resume multi-objective optimization. study = load_study(storage=storage, study_name="multi") assert len(study.trials) == 1 study.optimize(mo_objective_test_upgrade, n_trials=1) assert len(study.trials) == 2 for trial in study.trials: assert trial.user_attrs["b"] == 1 assert -5 <= trial.params["x"] <= 5 assert 0 <= trial.params["y"] <= 10 assert trial.params["z"] in (-5, 0, 5) assert -5 <= trial.values[0] < 5 assert 0 <= trial.values[1] <= 150 assert study.user_attrs["d"] == 3 storage.engine.dispose() # Be sure to disconnect db @pytest.mark.skipif(platform.system() == "Windows", reason="Skip on Windows") @pytest.mark.parametrize( "optuna_version", ["2.4.0.a", "2.6.0.a", "3.0.0.a", "3.0.0.b", "3.0.0.c", "3.0.0.d", "3.2.0.a"] ) def test_upgrade_distributions(optuna_version: str) -> None: src_db_file = os.path.join( os.path.dirname(__file__), "test_upgrade_assets", f"{optuna_version}.db" ) with tempfile.TemporaryDirectory() as workdir: shutil.copyfile(src_db_file, f"{workdir}/sqlite.db") storage_url = f"sqlite:///{workdir}/sqlite.db" storage = RDBStorage(storage_url, skip_compatibility_check=True, skip_table_creation=True) assert storage.get_current_version() == f"v{optuna_version}" head_version = storage.get_head_version() with warnings.catch_warnings(): warnings.simplefilter("ignore", category=FutureWarning) storage.upgrade() assert head_version == storage.get_current_version() new_study = load_study(storage=storage, study_name="schema migration") new_distribution_dict = new_study.trials[0]._distributions assert isinstance(new_distribution_dict["x1"], FloatDistribution) assert isinstance(new_distribution_dict["x2"], FloatDistribution) assert isinstance(new_distribution_dict["x3"], FloatDistribution) assert isinstance(new_distribution_dict["y1"], IntDistribution) assert isinstance(new_distribution_dict["y2"], IntDistribution) assert isinstance(new_distribution_dict["z"], CategoricalDistribution) # Check if Study.optimize can run on new storage. with warnings.catch_warnings(): warnings.simplefilter("ignore", category=UserWarning) new_study.optimize(objective_test_upgrade_distributions, n_trials=1) storage.engine.dispose() # Be sure to disconnect db def test_create_new_trial_with_retries() -> None: storage = RDBStorage("sqlite:///:memory:") study_id = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) n_retries = 0 def mock_func( study_id: int, template_trial: FrozenTrial, session: "sqlalchemy_orm.Session", ) -> FrozenTrial: nonlocal n_retries n_retries += 1 trial = models.TrialModel( study_id=study_id, number=None, state=TrialState.RUNNING, datetime_start=datetime.now(), ) session.add(trial) session.flush() trial.number = trial.count_past_trials(session) session.add(trial) if n_retries == 3: return trial raise sqlalchemy_exc.OperationalError("xxx", "yyy", Exception()) with patch( "optuna.storages._rdb.storage.RDBStorage._get_prepared_new_trial", new=Mock(side_effect=mock_func), ): _ = storage.create_new_trial(study_id) # Assert only one trial was created. # The added trials in the session were rollbacked. trials = storage.get_all_trials(study_id) assert len(trials) == 1 assert trials[0].number == 0 optuna-4.1.0/tests/storages_tests/rdb_tests/test_upgrade_assets/000077500000000000000000000000001471332314300252775ustar00rootroot00000000000000optuna-4.1.0/tests/storages_tests/rdb_tests/test_upgrade_assets/0.9.0.a.db000066400000000000000000002100001471332314300264620ustar00rootroot00000000000000SQLite format 3@  .0: P f   ^o6%%Qtabletrial_valuestrial_valuesCREATE TABLE trial_values ( trial_value_id INTEGER NOT NULL, trial_id INTEGER, step INTEGER, value FLOAT, PRIMARY KEY (trial_value_id), UNIQUE (trial_id, step), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_values_1trial_values5 %%-tabletrial_paramstrial_paramsCREATE TABLE trial_params ( param_id INTEGER NOT NULL, trial_id INTEGER, param_name VARCHAR(512), param_value FLOAT, distribution_json VARCHAR(2048), PRIMARY KEY (param_id), UNIQUE (trial_id, param_name), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_params_1trial_paramsR ;;;tabletrial_system_attributestrial_system_attributes CREATE TABLE trial_system_attributes ( trial_system_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json VARCHAR(2048), PRIMARY KEY (trial_system_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )M a;indexsqlite_autoindex_trial_system_attributes_1trial_system_attributes H 77/tabletrial_user_attributestrial_user_attributes CREATE TABLE trial_user_attributes ( trial_user_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json VARCHAR(2048), PRIMARY KEY (trial_user_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )I ]7indexsqlite_autoindex_trial_user_attributes_1trial_user_attributes tabletrialstrials CREATE TABLE trials ( trial_id INTEGER NOT NULL, study_id INTEGER, state VARCHAR(8) NOT NULL, value FLOAT, datetime_start DATETIME, datetime_complete DATETIME, PRIMARY KEY (trial_id), FOREIGN KEY(study_id) REFERENCES studies (study_id) )S;;=tablestudy_system_attributesstudy_system_attributesCREATE TABLE study_system_attributes ( study_system_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json VARCHAR(2048), PRIMARY KEY (study_system_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )Ma;indexsqlite_autoindex_study_system_attributes_1study_system_attributesI771tablestudy_user_attributesstudy_user_attributesCREATE TABLE study_user_attributes ( study_user_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json VARCHAR(2048), PRIMARY KEY (study_user_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )I]7indexsqlite_autoindex_study_user_attributes_1study_user_attributes\%%{tableversion_infoversion_infoCREATE TABLE version_info ( version_info_id INTEGER NOT NULL, schema_version INTEGER, library_version VARCHAR(256), PRIMARY KEY (version_info_id), CHECK (version_info_id=1) )j7indexix_studies_study_namestudiesCREATE UNIQUE INDEX ix_studies_study_name ON studies (study_name)-1tablestudiesstudiesCREATE TABLE studies ( study_id INTEGER NOT NULL, study_name VARCHAR(512) NOT NULL, direction VARCHAR(8) NOT NULL, PRIMARY KEY (study_id) ) singleMINIMIZE%single_emptyMINIMIZE  single% single_empty   0.9.0 d3  d c2  c LAACOMPLETE@R 2021-09-01 12:23:35.9490902021-09-01 12:23:36.010581  b1   b  a0  _number0  a   _number VS !z{"name": "CategoricalDistribution", "attributes": {"choices": [-5, 0, 5]}}Q y{"name": "IntUniformDistribution", "attributes": {"low": 0, "high": 10}}U x@kB{"name": "UniformDistribution", "attributes": {"low": -5, "high": 5}}  z y  x   ?   optuna-4.1.0/tests/storages_tests/rdb_tests/test_upgrade_assets/1.2.0.a.db000066400000000000000000002300001471332314300264560ustar00rootroot00000000000000SQLite format 3@  .0: P f   ^o6|)++ tablealembic_versionalembic_versionCREATE TABLE alembic_version ( version_num VARCHAR(32) NOT NULL, CONSTRAINT alembic_version_pkc PRIMARY KEY (version_num) )=Q+indexsqlite_autoindex_alembic_version_1alembic_version%%Qtabletrial_valuestrial_valuesCREATE TABLE trial_values ( trial_value_id INTEGER NOT NULL, trial_id INTEGER, step INTEGER, value FLOAT, PRIMARY KEY (trial_value_id), UNIQUE (trial_id, step), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_values_1trial_values5 %%-tabletrial_paramstrial_paramsCREATE TABLE trial_params ( param_id INTEGER NOT NULL, trial_id INTEGER, param_name VARCHAR(512), param_value FLOAT, distribution_json VARCHAR(2048), PRIMARY KEY (param_id), UNIQUE (trial_id, param_name), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_params_1trial_paramsR ;;;tabletrial_system_attributestrial_system_attributes CREATE TABLE trial_system_attributes ( trial_system_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json VARCHAR(2048), PRIMARY KEY (trial_system_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )M a;indexsqlite_autoindex_trial_system_attributes_1trial_system_attributes H 77/tabletrial_user_attributestrial_user_attributes CREATE TABLE trial_user_attributes ( trial_user_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json VARCHAR(2048), PRIMARY KEY (trial_user_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )I ]7indexsqlite_autoindex_trial_user_attributes_1trial_user_attributes tabletrialstrials CREATE TABLE trials ( trial_id INTEGER NOT NULL, study_id INTEGER, state VARCHAR(8) NOT NULL, value FLOAT, datetime_start DATETIME, datetime_complete DATETIME, PRIMARY KEY (trial_id), FOREIGN KEY(study_id) REFERENCES studies (study_id) )S;;=tablestudy_system_attributesstudy_system_attributesCREATE TABLE study_system_attributes ( study_system_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json VARCHAR(2048), PRIMARY KEY (study_system_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )Ma;indexsqlite_autoindex_study_system_attributes_1study_system_attributesI771tablestudy_user_attributesstudy_user_attributesCREATE TABLE study_user_attributes ( study_user_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json VARCHAR(2048), PRIMARY KEY (study_user_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )I]7indexsqlite_autoindex_study_user_attributes_1study_user_attributes\%%{tableversion_infoversion_infoCREATE TABLE version_info ( version_info_id INTEGER NOT NULL, schema_version INTEGER, library_version VARCHAR(256), PRIMARY KEY (version_info_id), CHECK (version_info_id=1) )j7indexix_studies_study_namestudiesCREATE UNIQUE INDEX ix_studies_study_name ON studies (study_name)-1tablestudiesstudiesCREATE TABLE studies ( study_id INTEGER NOT NULL, study_name VARCHAR(512) NOT NULL, direction VARCHAR(8) NOT NULL, PRIMARY KEY (study_id) ) singleMINIMIZE%single_emptyMINIMIZE  single% single_empty   1.2.0 d3  d c2  c LAACOMPLETE@ 2021-09-01 12:23:53.7428542021-09-01 12:23:53.835863  b1   b  a0  _number0  a   _number VR  !z{"name": "CategoricalDistribution", "attributes": {"choices": [-5, 0, 5]}}Q y{"name": "IntUniformDistribution", "attributes": {"low": 0, "high": 10}}U x)O {"name": "UniformDistribution", "attributes": {"low": -5, "high": 5}}  z y  x   ?    v1.2.0.a   v1.2.0.aoptuna-4.1.0/tests/storages_tests/rdb_tests/test_upgrade_assets/1.3.0.a.db000066400000000000000000002300001471332314300264570ustar00rootroot00000000000000SQLite format 3@  .0: P f   L]$j)++ tablealembic_versionalembic_versionCREATE TABLE alembic_version ( version_num VARCHAR(32) NOT NULL, CONSTRAINT alembic_version_pkc PRIMARY KEY (version_num) )=Q+indexsqlite_autoindex_alembic_version_1alembic_version%%Qtabletrial_valuestrial_valuesCREATE TABLE trial_values ( trial_value_id INTEGER NOT NULL, trial_id INTEGER, step INTEGER, value FLOAT, PRIMARY KEY (trial_value_id), UNIQUE (trial_id, step), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_values_1trial_values5 %%-tabletrial_paramstrial_paramsCREATE TABLE trial_params ( param_id INTEGER NOT NULL, trial_id INTEGER, param_name VARCHAR(512), param_value FLOAT, distribution_json VARCHAR(2048), PRIMARY KEY (param_id), UNIQUE (trial_id, param_name), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_params_1trial_paramsR ;;;tabletrial_system_attributestrial_system_attributes CREATE TABLE trial_system_attributes ( trial_system_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json VARCHAR(2048), PRIMARY KEY (trial_system_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )M a;indexsqlite_autoindex_trial_system_attributes_1trial_system_attributes H 77/tabletrial_user_attributestrial_user_attributes CREATE TABLE trial_user_attributes ( trial_user_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json VARCHAR(2048), PRIMARY KEY (trial_user_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )I ]7indexsqlite_autoindex_trial_user_attributes_1trial_user_attributes ')tabletrialstrials CREATE TABLE trials ( trial_id INTEGER NOT NULL, number INTEGER, study_id INTEGER, state VARCHAR(8) NOT NULL, value FLOAT, datetime_start DATETIME, datetime_complete DATETIME, PRIMARY KEY (trial_id), FOREIGN KEY(study_id) REFERENCES studies (study_id) )S;;=tablestudy_system_attributesstudy_system_attributesCREATE TABLE study_system_attributes ( study_system_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json VARCHAR(2048), PRIMARY KEY (study_system_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )Ma;indexsqlite_autoindex_study_system_attributes_1study_system_attributesI771tablestudy_user_attributesstudy_user_attributesCREATE TABLE study_user_attributes ( study_user_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json VARCHAR(2048), PRIMARY KEY (study_user_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )I]7indexsqlite_autoindex_study_user_attributes_1study_user_attributes\%%{tableversion_infoversion_infoCREATE TABLE version_info ( version_info_id INTEGER NOT NULL, schema_version INTEGER, library_version VARCHAR(256), PRIMARY KEY (version_info_id), CHECK (version_info_id=1) )j7indexix_studies_study_namestudiesCREATE UNIQUE INDEX ix_studies_study_name ON studies (study_name)-1tablestudiesstudiesCREATE TABLE studies ( study_id INTEGER NOT NULL, study_name VARCHAR(512) NOT NULL, direction VARCHAR(8) NOT NULL, PRIMARY KEY (study_id) ) singleMINIMIZE%single_emptyMINIMIZE  single% single_empty   1.3.0 d3  d c2  c MAACOMPLETE@C_0T2021-09-01 12:24:03.4611682021-09-01 12:24:03.548125  b1   b  a0   a KR !z{"name": "CategoricalDistribution", "attributes": {"choices": [-5, 0, 5]}}\ 3y{"name": "IntUniformDistribution", "attributes": {"low": 0, "high": 10, "step": 1}}U x t{"name": "UniformDistribution", "attributes": {"low": -5, "high": 5}}  z y  x   ?    v1.3.0.a   v1.3.0.aoptuna-4.1.0/tests/storages_tests/rdb_tests/test_upgrade_assets/2.4.0.a.db000066400000000000000000003100001471332314300264600ustar00rootroot00000000000000SQLite format 3@ 9 9.WJ E T 9T&UQe?indexsqlite_autoindex_trial_intermediate_values_1trial_intermediate_values,%%tabletrial_valuestrial_valuesCREATE TABLE trial_values ( trial_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, objective INTEGER NOT NULL, value FLOAT NOT NULL, PRIMARY KEY (trial_value_id), UNIQUE (trial_id, objective), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_values_1trial_values5%%-tabletrial_paramstrial_paramsCREATE TABLE trial_params ( param_id INTEGER NOT NULL, trial_id INTEGER, param_name VARCHAR(512), param_value FLOAT, distribution_json VARCHAR(2048), PRIMARY KEY (param_id), UNIQUE (trial_id, param_name), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_params_1trial_paramsR ;;;tabletrial_system_attributestrial_system_attributesCREATE TABLE trial_system_attributes ( trial_system_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json VARCHAR(2048), PRIMARY KEY (trial_system_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Ma;indexsqlite_autoindex_trial_system_attributes_1trial_system_attributesH 77/tabletrial_user_attributestrial_user_attributes CREATE TABLE trial_user_attributes ( trial_user_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json VARCHAR(2048), PRIMARY KEY (trial_user_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )I ]7indexsqlite_autoindex_trial_user_attributes_1trial_user_attributes   tabletrialstrials CREATE TABLE trials ( trial_id INTEGER NOT NULL, number INTEGER, study_id INTEGER, state VARCHAR(8) NOT NULL, datetime_start DATETIME, datetime_complete DATETIME, PRIMARY KEY (trial_id), FOREIGN KEY(study_id) REFERENCES studies (study_id) )S;;=tablestudy_system_attributesstudy_system_attributes CREATE TABLE study_system_attributes ( study_system_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json VARCHAR(2048), PRIMARY KEY (study_system_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )M a;indexsqlite_autoindex_study_system_attributes_1study_system_attributes I771tablestudy_user_attributesstudy_user_attributesCREATE TABLE study_user_attributes ( study_user_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json VARCHAR(2048), PRIMARY KEY (study_user_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )I]7indexsqlite_autoindex_study_user_attributes_1study_user_attributesJ--Gtablestudy_directionsstudy_directionsCREATE TABLE study_directions ( study_direction_id INTEGER NOT NULL, direction VARCHAR(8) NOT NULL, study_id INTEGER NOT NULL, objective INTEGER NOT NULL, PRIMARY KEY (study_direction_id), UNIQUE (study_id, objective), FOREIGN KEY(study_id) REFERENCES studies (study_id) )?S-indexsqlite_autoindex_study_directions_1study_directions\%%{tableversion_infoversion_infoCREATE TABLE version_info ( version_info_id INTEGER NOT NULL, schema_version INTEGER, library_version VARCHAR(256), PRIMARY KEY (version_info_id), CHECK (version_info_id=1) )j7indexix_studies_study_namestudiesCREATE UNIQUE INDEX ix_studies_study_name ON studies (study_name) otablestudiesstudiesCREATE TABLE studies ( study_id INTEGER NOT NULL, study_name VARCHAR(512) NOT NULL, PRIMARY KEY (study_ -schema migrationmulti#multi_empty single%single_empty -schema migration multi#multi_empty single% single_empty   2.4.0 MINIMIZE MINIMIZEMINIMIZE MINIMIZEMINIMIZEMINIMIZE  MINIMIZE     d3d3 d d c2c2 c c .t.DAACOMPLETE2022-02-16 15:46:59.3167642022-02-16 15:46:59.348160DAACOMPLETE2022-02-16 15:46:59.2792552022-02-16 15:46:59.295442DAACOMPLETE2022-02-16 15:46:59.2113912022-02-16 15:46:59.232018 b1b1 b1 bb  b a0a0-nsga2:generation0 a0 aa-nsga2:generation  a UK@ . m US !z{"name": "CategoricalDistribution", "attributes": {"choices": [-5, 0, 5]}}^ 3y3{"name": "IntUniformDistribution", "attributes": {"low": 5, "high": 14, "step": 3}}a 9y2{"name": "IntLogUniformDistribution", "attributes": {"low": 1, "high": 20, "step": 1}}^ 3y1 {"name": "IntUniformDistribution", "attributes": {"low": 0, "high": 10, "step": 1}}_ 7x3{"name": "DiscreteUniformDistribution", "attributes": {"low": -6, "high": 6, "q": 2}}a+x2?(OFr0{"name": "LogUniformDistribution", "attributes": {"low": 1e-05, "high": 0.001}}Wx1|{"name": "UniformDistribution", "attributes": {"low": -5, "high": 5}}T!z{"name": "CategoricalDistribution", "attributes": {"choices": [-5, 0, 5]}}]3y {"name": "IntUniformDistribution", "attributes": {"low": 0, "high": 10, "step": 1}}Vx@ ;$ {"name": "UniformDistribution", "attributes": {"low": -5, "high": 5}}R !z{"name": "CategoricalDistribution", "attributes": {"choices": [-5, 0, 5]}}\ 3y {"name": "IntUniformDistribution", "attributes": {"low": 0, "high": 10, "step": 1}}U x;\{"name": "UniformDistribution", "attributes": {"low": -5, "high": 5}} z y3 y2 y1 x3 x2x1zyx z y  x @s@S=r @\qt@ ;$  @_DUM    ?  ?    q& E T 9T&U`&s??otabletrial_intermediate_valuestrial_intermediate_valuesCREATE TABLE trial_intermediate_values ( trial_intermediate_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, step INTEGER NOT NULL, intermediate_value FLOAT NOT NULL, PRIMARY KEY (trial_intermediate_value_id), UNIQUE (trial_id, step), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )e?indexsqlite_autoindex_trial_intermediate_values_1trial_intermediate_values}%%tabletrial_valuestrial_valuesCREATE TABLE trial_values ( trial_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, objective INTEGER NOT NULL, value FLOAT NOT NULL, PRIMARY KEY (trial_value_id), UNIQUE (trial_id, objective), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_values_1trial_values%%-tabletrial_paramstrial_paramsCREATE TABLE trial_params ( param_id INTEGER NOT NULL, trial_id INTEGER, param_name VARCHAR(512), param_value FLOAT, distribution_json VARCHAR(2048), PRIMARY KEY (param_id), UNIQUE (trial_id, param_name), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_params_1trial_params;;;tabletrial_system_attributestrial_system_attributesCREATE TABLE trial_system_attributes ( trial_system_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json VARCHAR(2048), PRIMARY KEY (trial_system_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Ma;indexsqlite_autoindex_trial_system_attributes_1trial_system_attributesH 77/tabletrial_user_attributestrial_user_attributes CREATE TABLE trial_user_attributes ( trial_user_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json VARCHAR(2048), PRIMARY KEY (trial_user_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )K]7indexsqlite_autoindex_trial_user_attributes_1trial_user_attributes   tabletrialstrials CREATE TABLE trials ( trial_id INTEGER NOT NULL, number INTEGER, study_id INTEGER, state VARCHAR(8) NOT NULL, datetime_start DATETIME, datetime_complete DATETIME, PRIMARY KEY (trial_id), FOREIGN KEY(study_id) REFERENCES studies (study_id) )S;;=tablestudy_system_attributesstudy_system_attributes CREATE TABLE study_system_attributes ( study_system_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json VARCHAR(2048), PRIMARY KEY (study_system_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )M a;indexsqlite_autoindex_study_system_attributes_1study_system_attributes I771tablestudy_user_attributesstudy_user_attributesCREATE TABLE study_user_attributes ( study_user_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json VARCHAR(2048), PRIMARY KEY (study_user_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )I]7indexsqlite_autoindex_study_user_attributes_1study_user_attributesJ--Gtablestudy_directionsstudy_directionsCREATE TABLE study_directions ( study_direction_id INTEGER NOT NULL, direction VARCHAR(8) NOT NULL, study_id INTEGER NOT NULL, objective INTEGER NOT NULL, PRIMARY KEY (study_direction_id), UNIQUE (study_id, objective), FOREIGN KEY(study_id) REFERENCES studies (study_id) )?S-indexsqlite_autoindex_study_directions_1study_directions\%%{tableversion_infoversion_infoCREATE TABLE version_info ( version_info_id INTEGER NOT NULL, schema_version INTEGER, library_version VARCHAR(256), PRIMARY KEY (version_info_id), CHECK (version_info_id=1) )j7indexix_studies_study_namestudiesCREATE UNIQUE INDEX ix_studies_study_name ON studies (study_name) otablestudiesstudiesCREATE TABLE studies ( study_id INTEGER NOT NULL, study_name VARCHAR(512) NOT NULL, PRIMARY KEY (study_id) )  v2.4.0.a   v2.4.0.a j ` q 8 r +)++ tablealembic_versionalembic_versionCREATE TABLE alembic_version ( version_num VARCHAR(32) NOT NULL, CONSTRAINT alembic_version_pkc PRIMARY KEY (version_num) )=Q+indexsqlite_autoindex_alembic_version_1alembic_versionQe?indexsqlite_autoindex_trial_intermediate_values_1trial_intermediate_valuesp??otabletrial_intermediate_valuestrial_intermediate_valuesCREATE TABLE trial_intermediate_values ( trial_intermediate_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, step INTEGER NOT NULL, intermediate_value FLOAT NOT NULL, PRIMARY KEY (trial_intermediate_value_id), UNIQUE (trial_id, step), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_values_1trial_values,%%tabletrial_valuestrial_valuesCREATE TABLE trial_values ( trial_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, objective INTEGER NOT NULL, value FLOAT NOT NULL, PRIMARY KEY (trial_value_id), UNIQUE (trial_id, objective), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_params_1trial_params5%%-tabletrial_paramstrial_paramsCREATE TABLE trial_params ( param_id INTEGER NOT NULL, trial_id INTEGER, param_name VARCHAR(512), param_value FLOAT, distribution_json VARCHAR(2048), PRIMARY KEY (param_id), UNIQUE (trial_id, param_name), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Ma;indexsqlite_autoindex_trial_system_attributes_1trial_system_attributesR ;;;tabletrial_system_attributestrial_system_attributesCREATE TABLE trial_system_attributes ( trial_system_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json VARCHAR(2048), PRIMARY KEY (trial_system_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )I ]7indexsqlite_autoindex_trial_user_attributes_1trial_user_attributes optuna-4.1.0/tests/storages_tests/rdb_tests/test_upgrade_assets/2.6.0.a.db000066400000000000000000003300001471332314300264640ustar00rootroot00000000000000SQLite format 3@ ::.WJ M n S+w[p??otabletrial_intermediate_valuestrial_intermediate_valuesCREATE TABLE trial_intermediate_values ( trial_intermediate_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, step INTEGER NOT NULL, intermediate_value FLOAT NOT NULL, PRIMARY KEY (trial_intermediate_value_id), UNIQUE (trial_id, step), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Qe?indexsqlite_autoindex_trial_intermediate_values_1trial_intermediate_values,%%tabletrial_valuestrial_valuesCREATE TABLE trial_values ( trial_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, objective INTEGER NOT NULL, value FLOAT NOT NULL, PRIMARY KEY (trial_value_id), UNIQUE (trial_id, objective), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_values_1trial_values,%%tabletrial_paramstrial_paramsCREATE TABLE trial_params ( param_id INTEGER NOT NULL, trial_id INTEGER, param_name VARCHAR(512), param_value FLOAT, distribution_json TEXT, PRIMARY KEY (param_id), UNIQUE (trial_id, param_name), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_params_1trial_paramsI ;;)tabletrial_system_attributestrial_system_attributesCREATE TABLE trial_system_attributes ( trial_system_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_system_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Ma;indexsqlite_autoindex_trial_system_attributes_1trial_system_attributes? 77tabletrial_user_attributestrial_user_attributes CREATE TABLE trial_user_attributes ( trial_user_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_user_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )I ]7indexsqlite_autoindex_trial_user_attributes_1trial_user_attributes   tabletrialstrials CREATE TABLE trials ( trial_id INTEGER NOT NULL, number INTEGER, study_id INTEGER, state VARCHAR(8) NOT NULL, datetime_start DATETIME, datetime_complete DATETIME, PRIMARY KEY (trial_id), FOREIGN KEY(study_id) REFERENCES studies (study_id) )J;;+tablestudy_system_attributesstudy_system_attributes CREATE TABLE study_system_attributes ( study_system_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (study_system_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )M a;indexsqlite_autoindex_study_system_attributes_1study_system_attributes @77tablestudy_user_attributesstudy_user_attributesCREATE TABLE study_user_attributes ( study_user_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (study_user_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )I]7indexsqlite_autoindex_study_user_attributes_1study_user_attributesJ--Gtablestudy_directionsstudy_directionsCREATE TABLE study_directions ( study_direction_id INTEGER NOT NULL, direction VARCHAR(8) NOT NULL, study_id INTEGER NOT NULL, objective INTEGER NOT NULL, PRIMARY KEY (study_direction_id), UNIQUE (study_id, objective), FOREIGN KEY(study_id) REFERENCES studies (study_id) )?S-indexsqlite_autoindex_study_directions_1study_directions\%%{tableversion_infoversion_infoCREATE TABLE version_info ( version_info_id INTEGER NOT NULL, schema_version INTEGER, library_version VARCHAR(256), PRIMARY KEY (version_info_id), CHECK (version_info_id=1) )j7indexix_studies_study_namestudiesCREATE UNIQUE INDEX ix_studies_study_name ON studies (study_name) otablestudiesstudiesCREATE TABLE studies ( study_id INTEGER NOT NULL, study_name VARCHAR(512) NOT NULL, PRIMARY KEY (study_ -schema migrationmulti#multi_empty single%single_empty -schema migration multi#multi_empty single% single_empty   2.6.0 MINIMIZE MINIMIZEMINIMIZE MINIMIZEMINIMIZEMINIMIZE  MINIMIZE     d3d3 d d c2c2 c c .t.DAACOMPLETE2022-02-16 15:46:42.9495672022-02-16 15:46:42.979450DAACOMPLETE2022-02-16 15:46:42.9136282022-02-16 15:46:42.929011DAACOMPLETE2022-02-16 15:46:42.8481302022-02-16 15:46:42.869193 b1b1 b1 bb  b a0a0-nsga2:generation0 a0 aa-nsga2:generation  a DG9 $ \ DS !z{"name": "CategoricalDistribution", "attributes": {"choices": [-5, 0, 5]}}^ 3y3 {"name": "IntUniformDistribution", "attributes": {"low": 5, "high": 14, "step": 3}}a 9y2 {"name": "IntLogUniformDistribution", "attributes": {"low": 1, "high": 20, "step": 1}}^ 3y1{"name": "IntUniformDistribution", "attributes": {"low": 0, "high": 10, "step": 1}}f Cx3{"name": "DiscreteUniformDistribution", "attributes": {"low": -6.0, "high": 6.0, "q": 2.0}}a+x2?Bl{"name": "LogUniformDistribution", "attributes": {"low": 1e-05, "high": 0.001}}[x1@~Hcn{"name": "UniformDistribution", "attributes": {"low": -5.0, "high": 5.0}}S!z{"name": "CategoricalDistribution", "attributes": {"choices": [-5, 0, 5]}}\ 3y{"name": "IntUniformDistribution", "attributes": {"low": 0, "high": 10, "step": 1}}Zx@\Vi[|{"name": "UniformDistribution", "attributes": {"low": -5.0, "high": 5.0}}R !z{"name": "CategoricalDistribution", "attributes": {"choices": [-5, 0, 5]}}\ 3y{"name": "IntUniformDistribution", "attributes": {"low": 0, "high": 10, "step": 1}}Y x@'.]F{"name": "UniformDistribution", "attributes": {"low": -5.0, "high": 5.0}} z y3 y2 y1 x3 x2x1zyx z y  x @t՘ @Fk&AZ*@\Vi[|  @TS)    ?  ?     q& M n S+w[LL??otabletrial_intermzS-indexsqlite_autoindex_trial_heartbeats_1trial_heartbeats9p??otabletrial_intermediate_valuestrial_intermediate_valuesCREATE TABLE trial_intermediate_values ( trial_intermediate_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, step INTEGER NOT NULL, intermediate_value FLOAT NOT NULL, PRIMARY KEY (trial_intermediate_value_id), UNIQUE (trial_id, step), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Qe?indexsqlite_autoindex_trial_intermediate_values_1trial_intermediate_valuesk%%tabletrial_valuestrial_valuesCREATE TABLE trial_values ( trial_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, objective INTEGER NOT NULL, value FLOAT NOT NULL, PRIMARY KEY (trial_value_id), UNIQUE (trial_id, objective), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_values_1trial_values%%tabletrial_paramstrial_paramsCREATE TABLE trial_params ( param_id INTEGER NOT NULL, trial_id INTEGER, param_name VARCHAR(512), param_value FLOAT, distribution_json TEXT, PRIMARY KEY (param_id), UNIQUE (trial_id, param_name), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_params_1trial_params;;)tabletrial_system_attributestrial_system_attributesCREATE TABLE trial_system_attributes ( trial_system_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_system_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Ma;indexsqlite_autoindex_trial_system_attributes_1trial_system_attributes? 77tabletrial_user_attributestrial_user_attributes CREATE TABLE trial_user_attributes ( trial_user_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_user_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )K]7indexsqlite_autoindex_trial_user_attributes_1trial_user_attributes   tabletrialstrials CREATE TABLE trials ( trial_id INTEGER NOT NULL, number INTEGER, study_id INTEGER, state VARCHAR(8) NOT NULL, datetime_start DATETIME, datetime_complete DATETIME, PRIMARY KEY (trial_id), FOREIGN KEY(study_id) REFERENCES studies (study_id) )J;;+tablestudy_system_attributesstudy_system_attributes CREATE TABLE study_system_attributes ( study_system_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (study_system_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )M a;indexsqlite_autoindex_study_system_attributes_1study_system_attributes @77tablestudy_user_attributesstudy_user_attributesCREATE TABLE study_user_attributes ( study_user_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (study_user_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )I]7indexsqlite_autoindex_study_user_attributes_1study_user_attributesJ--Gtablestudy_directionsstudy_directionsCREATE TABLE study_directions ( study_direction_id INTEGER NOT NULL, direction VARCHAR(8) NOT NULL, study_id INTEGER NOT NULL, objective INTEGER NOT NULL, PRIMARY KEY (study_direction_id), UNIQUE (study_id, objective), FOREIGN KEY(study_id) REFERENCES studies (study_id) )?S-indexsqlite_autoindex_study_directions_1study_directions\%%{tableversion_infoversion_infoCREATE TABLE version_info ( version_info_id INTEGER NOT NULL, schema_version INTEGER, library_version VARCHAR(256), PRIMARY KEY (version_info_id), CHECK (version_info_id=1) )j7indexix_studies_study_namestudiesCREATE UNIQUE INDEX ix_studies_study_name ON studies (study_name) otablestudiesstudiesCREATE TABLE studies ( study_id INTEGER NOT NULL, study_name VARCHAR(512) NOT NULL, PRIMARY KEY (study_id) )   /i J c"/)++ tablealembic_versionalembic_versionCREATE TABLE alembic_version ( version_num VARCHAR(32) NOT NULL, CONSTRAINT alembic_version_pkc PRIMARY KEY (version_num) )=Q+indexsqlite_autoindex_alembic_version_1alembic_version?S-indexsqlite_autoindex_trial_heartbeats_1trial_heartbeats--otabletrial_heartbeatstrial_heartbeatsCREATE TABLE trial_heartbeats ( trial_heartbeat_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, heartbeat DATETIME NOT NULL, PRIMARY KEY (trial_heartbeat_id), UNIQUE (trial_id), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Qe?indexsqlite_autoindex_trial_intermediate_values_1trial_intermediate_valuesp??otabletrial_intermediate_valuestrial_intermediate_valuesCREATE TABLE trial_intermediate_values ( trial_intermediate_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, step INTEGER NOT NULL, intermediate_value FLOAT NOT NULL, PRIMARY KEY (trial_intermediate_value_id), UNIQUE (trial_id, step), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_values_1trial_values,%%tabletrial_valuestrial_valuesCREATE TABLE trial_values ( trial_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, objective INTEGER NOT NULL, value FLOAT NOT NULL, PRIMARY KEY (trial_value_id), UNIQUE (trial_id, objective), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_params_1trial_params,%%tabletrial_paramstrial_paramsCREATE TABLE trial_params ( param_id INTEGER NOT NULL, trial_id INTEGER, param_name VARCHAR(512), param_value FLOAT, distribution_json TEXT, PRIMARY KEY (param_id), UNIQUE (trial_id, param_name), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Ma;indexsqlite_autoindex_trial_system_attributes_1trial_system_attributesI ;;)tabletrial_system_attributestrial_system_attributesCREATE TABLE trial_system_attributes ( trial_system_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_system_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )I ]7indexsqlite_autoindex_trial_user_attributes_1trial_user_attributes  v2.6.0.a   v2.6.0.aoptuna-4.1.0/tests/storages_tests/rdb_tests/test_upgrade_assets/3.0.0.a.db000066400000000000000000003300001471332314300264570ustar00rootroot00000000000000SQLite format 3@ ==.WJ M n S+w[p??otabletrial_intermediate_valuestrial_intermediate_valuesCREATE TABLE trial_intermediate_values ( trial_intermediate_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, step INTEGER NOT NULL, intermediate_value FLOAT NOT NULL, PRIMARY KEY (trial_intermediate_value_id), UNIQUE (trial_id, step), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Qe?indexsqlite_autoindex_trial_intermediate_values_1trial_intermediate_values,%%tabletrial_valuestrial_valuesCREATE TABLE trial_values ( trial_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, objective INTEGER NOT NULL, value FLOAT NOT NULL, PRIMARY KEY (trial_value_id), UNIQUE (trial_id, objective), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_values_1trial_values,%%tabletrial_paramstrial_paramsCREATE TABLE trial_params ( param_id INTEGER NOT NULL, trial_id INTEGER, param_name VARCHAR(512), param_value FLOAT, distribution_json TEXT, PRIMARY KEY (param_id), UNIQUE (trial_id, param_name), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_params_1trial_paramsI ;;)tabletrial_system_attributestrial_system_attributesCREATE TABLE trial_system_attributes ( trial_system_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_system_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Ma;indexsqlite_autoindex_trial_system_attributes_1trial_system_attributes? 77tabletrial_user_attributestrial_user_attributes CREATE TABLE trial_user_attributes ( trial_user_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_user_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )I ]7indexsqlite_autoindex_trial_user_attributes_1trial_user_attributes   tabletrialstrials CREATE TABLE trials ( trial_id INTEGER NOT NULL, number INTEGER, study_id INTEGER, state VARCHAR(8) NOT NULL, datetime_start DATETIME, datetime_complete DATETIME, PRIMARY KEY (trial_id), FOREIGN KEY(study_id) REFERENCES studies (study_id) )J;;+tablestudy_system_attributesstudy_system_attributes CREATE TABLE study_system_attributes ( study_system_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (study_system_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )M a;indexsqlite_autoindex_study_system_attributes_1study_system_attributes @77tablestudy_user_attributesstudy_user_attributesCREATE TABLE study_user_attributes ( study_user_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (study_user_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )I]7indexsqlite_autoindex_study_user_attributes_1study_user_attributesJ--Gtablestudy_directionsstudy_directionsCREATE TABLE study_directions ( study_direction_id INTEGER NOT NULL, direction VARCHAR(8) NOT NULL, study_id INTEGER NOT NULL, objective INTEGER NOT NULL, PRIMARY KEY (study_direction_id), UNIQUE (study_id, objective), FOREIGN KEY(study_id) REFERENCES studies (study_id) )?S-indexsqlite_autoindex_study_directions_1study_directions\%%{tableversion_infoversion_infoCREATE TABLE version_info ( version_info_id INTEGER NOT NULL, schema_version INTEGER, library_version VARCHAR(256), PRIMARY KEY (version_info_id), CHECK (version_info_id=1) )j7indexix_studies_study_namestudiesCREATE UNIQUE INDEX ix_studies_study_name ON studies (study_name) otablestudiesstudiesCREATE TABLE studies ( study_id INTEGER NOT NULL, study_name VARCHAR(512) NOT NULL, PRIMARY KEY (study_ -schema migrationmulti#multi_empty single%single_empty -schema migration multi#multi_empty single% single_empty   3.0.0 MINIMIZE MINIMIZEMINIMIZE MINIMIZEMINIMIZEMINIMIZE  MINIMIZE     d3d3 d d c2c2 c c .t.DAACOMPLETE2022-02-23 20:30:55.7491452022-02-23 20:30:55.787565DAACOMPLETE2022-02-23 20:30:55.6933652022-02-23 20:30:55.719393DAACOMPLETE2022-02-23 20:30:55.5916912022-02-23 20:30:55.620748 b1b1 b1 bb  b a0a0-nsga2:generation0 a0 aa-nsga2:generation  a &\ * B u  T !z{"name": "CategoricalDistribution", "attributes": {"choices": [-5, 0, 5]}}e Ay3 {"name": "IntDistribution", "attributes": {"log": false, "step": 3, "low": 5, "high": 14}}d ?y2{"name": "IntDistribution", "attributes": {"log": true, "step": 1, "low": 1, "high": 20}}e Ay1{"name": "IntDistribution", "attributes": {"log": false, "step": 1, "low": 0, "high": 10}}m Qx3{"name": "FloatDistribution", "attributes": {"step": 2.0, "low": -6.0, "high": 6.0, "log": false}}wWx2?`v{"name": "FloatDistribution", "attributes": {"step": null, "low": 1e-05, "high": 0.001, "log": true}}uSx1@{"name": "FloatDistribution", "attributes": {"step": null, "low": -5.0, "high": 5.0, "log": false}}S !z{"name": "CategoricalDistribution", "attributes": {"choices": [-5, 0, 5]}}dAy{"name": "IntDistribution", "attributes": {"log": false, "step": 1, "low": 0, "high": 10}}tSx?q'T{"name": "FloatDistribution", "attributes": {"step": null, "low": -5.0, "high": 5.0, "log": false}}R  !z{"name": "CategoricalDistribution", "attributes": {"choices": [-5, 0, 5]}}c Ay{"name": "IntDistribution", "attributes": {"log": false, "step": 1, "low": 0, "high": 10}}s Sxi{"name": "FloatDistribution", "attributes": {"step": null, "low": -5.0, "high": 5.0, "log": false}} z y3 y2 y1 x3 x2x1zyx z y  x @g]4 @P .a?q'T  @%JY"7    ?  ?     q& M n S+w[LL??otabletrial_intermzS-indexsqlite_autoindex_trial_heartbeats_1trial_heartbeats9p??otabletrial_intermediate_valuestrial_intermediate_valuesCREATE TABLE trial_intermediate_values ( trial_intermediate_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, step INTEGER NOT NULL, intermediate_value FLOAT NOT NULL, PRIMARY KEY (trial_intermediate_value_id), UNIQUE (trial_id, step), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Qe?indexsqlite_autoindex_trial_intermediate_values_1trial_intermediate_valuesk%%tabletrial_valuestrial_valuesCREATE TABLE trial_values ( trial_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, objective INTEGER NOT NULL, value FLOAT NOT NULL, PRIMARY KEY (trial_value_id), UNIQUE (trial_id, objective), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_values_1trial_values%%tabletrial_paramstrial_paramsCREATE TABLE trial_params ( param_id INTEGER NOT NULL, trial_id INTEGER, param_name VARCHAR(512), param_value FLOAT, distribution_json TEXT, PRIMARY KEY (param_id), UNIQUE (trial_id, param_name), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_params_1trial_params;;)tabletrial_system_attributestrial_system_attributesCREATE TABLE trial_system_attributes ( trial_system_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_system_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Ma;indexsqlite_autoindex_trial_system_attributes_1trial_system_attributes? 77tabletrial_user_attributestrial_user_attributes CREATE TABLE trial_user_attributes ( trial_user_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_user_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )K]7indexsqlite_autoindex_trial_user_attributes_1trial_user_attributes   tabletrialstrials CREATE TABLE trials ( trial_id INTEGER NOT NULL, number INTEGER, study_id INTEGER, state VARCHAR(8) NOT NULL, datetime_start DATETIME, datetime_complete DATETIME, PRIMARY KEY (trial_id), FOREIGN KEY(study_id) REFERENCES studies (study_id) )J;;+tablestudy_system_attributesstudy_system_attributes CREATE TABLE study_system_attributes ( study_system_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (study_system_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )M a;indexsqlite_autoindex_study_system_attributes_1study_system_attributes @77tablestudy_user_attributesstudy_user_attributesCREATE TABLE study_user_attributes ( study_user_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (study_user_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )I]7indexsqlite_autoindex_study_user_attributes_1study_user_attributesJ--Gtablestudy_directionsstudy_directionsCREATE TABLE study_directions ( study_direction_id INTEGER NOT NULL, direction VARCHAR(8) NOT NULL, study_id INTEGER NOT NULL, objective INTEGER NOT NULL, PRIMARY KEY (study_direction_id), UNIQUE (study_id, objective), FOREIGN KEY(study_id) REFERENCES studies (study_id) )?S-indexsqlite_autoindex_study_directions_1study_directions\%%{tableversion_infoversion_infoCREATE TABLE version_info ( version_info_id INTEGER NOT NULL, schema_version INTEGER, library_version VARCHAR(256), PRIMARY KEY (version_info_id), CHECK (version_info_id=1) )j7indexix_studies_study_namestudiesCREATE UNIQUE INDEX ix_studies_study_name ON studies (study_name) otablestudiesstudiesCREATE TABLE studies ( study_id INTEGER NOT NULL, study_name VARCHAR(512) NOT NULL, PRIMARY KEY (study_id) )   /i J c"/)++ tablealembic_versionalembic_versionCREATE TABLE alembic_version ( version_num VARCHAR(32) NOT NULL, CONSTRAINT alembic_version_pkc PRIMARY KEY (version_num) )=Q+indexsqlite_autoindex_alembic_version_1alembic_version?S-indexsqlite_autoindex_trial_heartbeats_1trial_heartbeats--otabletrial_heartbeatstrial_heartbeatsCREATE TABLE trial_heartbeats ( trial_heartbeat_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, heartbeat DATETIME NOT NULL, PRIMARY KEY (trial_heartbeat_id), UNIQUE (trial_id), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Qe?indexsqlite_autoindex_trial_intermediate_values_1trial_intermediate_valuesp??otabletrial_intermediate_valuestrial_intermediate_valuesCREATE TABLE trial_intermediate_values ( trial_intermediate_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, step INTEGER NOT NULL, intermediate_value FLOAT NOT NULL, PRIMARY KEY (trial_intermediate_value_id), UNIQUE (trial_id, step), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_values_1trial_values,%%tabletrial_valuestrial_valuesCREATE TABLE trial_values ( trial_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, objective INTEGER NOT NULL, value FLOAT NOT NULL, PRIMARY KEY (trial_value_id), UNIQUE (trial_id, objective), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_params_1trial_params,%%tabletrial_paramstrial_paramsCREATE TABLE trial_params ( param_id INTEGER NOT NULL, trial_id INTEGER, param_name VARCHAR(512), param_value FLOAT, distribution_json TEXT, PRIMARY KEY (param_id), UNIQUE (trial_id, param_name), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Ma;indexsqlite_autoindex_trial_system_attributes_1trial_system_attributesI ;;)tabletrial_system_attributestrial_system_attributesCREATE TABLE trial_system_attributes ( trial_system_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_system_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )I ]7indexsqlite_autoindex_trial_user_attributes_1trial_user_attributes  v3.0.0.a   v3.0.0.aoptuna-4.1.0/tests/storages_tests/rdb_tests/test_upgrade_assets/3.0.0.b.db000066400000000000000000003300001471332314300264600ustar00rootroot00000000000000SQLite format 3@ 77.[2  E f K#oSg??]tabletrial_intermediate_valuestrial_intermediate_valuesCREATE TABLE trial_intermediate_values ( trial_intermediate_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, step INTEGER NOT NULL, intermediate_value FLOAT, PRIMARY KEY (trial_intermediate_value_id), UNIQUE (trial_id, step), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Qe?indexsqlite_autoindex_trial_intermediate_values_1trial_intermediate_values,%%tabletrial_valuestrial_valuesCREATE TABLE trial_values ( trial_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, objective INTEGER NOT NULL, value FLOAT NOT NULL, PRIMARY KEY (trial_value_id), UNIQUE (trial_id, objective), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_values_1trial_values,%%tabletrial_paramstrial_paramsCREATE TABLE trial_params ( param_id INTEGER NOT NULL, trial_id INTEGER, param_name VARCHAR(512), param_value FLOAT, distribution_json TEXT, PRIMARY KEY (param_id), UNIQUE (trial_id, param_name), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_params_1trial_paramsI ;;)tabletrial_system_attributestrial_system_attributesCREATE TABLE trial_system_attributes ( trial_system_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_system_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Ma;indexsqlite_autoindex_trial_system_attributes_1trial_system_attributes? 77tabletrial_user_attributestrial_user_attributes CREATE TABLE trial_user_attributes ( trial_user_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_user_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )I ]7indexsqlite_autoindex_trial_user_attributes_1trial_user_attributes   tabletrialstrials CREATE TABLE trials ( trial_id INTEGER NOT NULL, number INTEGER, study_id INTEGER, state VARCHAR(8) NOT NULL, datetime_start DATETIME, datetime_complete DATETIME, PRIMARY KEY (trial_id), FOREIGN KEY(study_id) REFERENCES studies (study_id) )J;;+tablestudy_system_attributesstudy_system_attributes CREATE TABLE study_system_attributes ( study_system_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (study_system_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )M a;indexsqlite_autoindex_study_system_attributes_1study_system_attributes @77tablestudy_user_attributesstudy_user_attributesCREATE TABLE study_user_attributes ( study_user_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (study_user_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )I]7indexsqlite_autoindex_study_user_attributes_1study_user_attributesJ--Gtablestudy_directionsstudy_directionsCREATE TABLE study_directions ( study_direction_id INTEGER NOT NULL, direction VARCHAR(8) NOT NULL, study_id INTEGER NOT NULL, objective INTEGER NOT NULL, PRIMARY KEY (study_direction_id), UNIQUE (study_id, objective), FOREIGN KEY(study_id) REFERENCES studies (study_id) )?S-indexsqlite_autoindex_study_directions_1study_directions\%%{tableversion_infoversion_infoCREATE TABLE version_info ( version_info_id INTEGER NOT NULL, schema_version INTEGER, library_version VARCHAR(256), PRIMARY KEY (version_info_id), CHECK (version_info_id=1) )j7indexix_studies_study_namestudiesCREATE UNIQUE INDEX ix_studies_study_name ON studies (study_name) otablestudiesstudiesCREATE TABLE studies ( study_id INTEGER NOT NULL, study_name VARCHAR(512) NOT NULL, PRIMARY KEY (study_ -schema migrationmulti#multi_empty single%single_empty -schema migration multi#multi_empty single% single_empty # 3.0.0b1.dev MINIMIZE MINIMIZEMINIMIZE MINIMIZEMINIMIZEMINIMIZE  MINIMIZE     d3d3 d d c2c2 c c .t.DAACOMPLETE2022-04-27 16:44:16.7912732022-04-27 16:44:16.815170DAACOMPLETE2022-04-27 16:44:16.7529482022-04-27 16:44:16.770557DAACOMPLETE2022-04-27 16:44:16.6854652022-04-27 16:44:16.705778 b1 b1 b  b a0-nsga2:generation0 a0 a-nsga2:generation  a &\ * B u  S  !z{"name": "CategoricalDistribution", "attributes": {"choices": [-5, 0, 5]}}e Ay3{"name": "IntDistribution", "attributes": {"log": false, "step": 3, "low": 5, "high": 14}}d ?y2{"name": "IntDistribution", "attributes": {"log": true, "step": 1, "low": 1, "high": 20}}e Ay1{"name": "IntDistribution", "attributes": {"log": false, "step": 1, "low": 0, "high": 10}}m Qx3{"name": "FloatDistribution", "attributes": {"step": 2.0, "low": -6.0, "high": 6.0, "log": false}}wWx2?9Ɇ\{"name": "FloatDistribution", "attributes": {"step": null, "low": 1e-05, "high": 0.001, "log": true}}uSx1={"name": "FloatDistribution", "attributes": {"step": null, "low": -5.0, "high": 5.0, "log": false}}S !z{"name": "CategoricalDistribution", "attributes": {"choices": [-5, 0, 5]}}dAy{"name": "IntDistribution", "attributes": {"log": false, "step": 1, "low": 0, "high": 10}}tSxh__{"name": "FloatDistribution", "attributes": {"step": null, "low": -5.0, "high": 5.0, "log": false}}R  !z{"name": "CategoricalDistribution", "attributes": {"choices": [-5, 0, 5]}}c Ay{"name": "IntDistribution", "attributes": {"log": false, "step": 1, "low": 0, "high": 10}}s Sx@W)GB{"name": "FloatDistribution", "attributes": {"step": null, "low": -5.0, "high": 5.0, "log": false}} z y3 y2 y1 x3 x2x1zyx z y  x @b#o @?Fp}h__  @4-h      ?      q&  E f K#oSUU??]tabletrial_intermediate_viS-indexsqlite_autoindex_trial_heartbeats_1trial_heartbeats(??]tabletrial_intermediate_valuestrial_intermediate_valuesCREATE TABLE trial_intermediate_values ( trial_intermediate_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, step INTEGER NOT NULL, intermediate_value FLOAT, PRIMARY KEY (trial_intermediate_value_id), UNIQUE (trial_id, step), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Qe?indexsqlite_autoindex_trial_intermediate_values_1trial_intermediate_valuesk%%tabletrial_valuestrial_valuesCREATE TABLE trial_values ( trial_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, objective INTEGER NOT NULL, value FLOAT NOT NULL, PRIMARY KEY (trial_value_id), UNIQUE (trial_id, objective), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_values_1trial_values%%tabletrial_paramstrial_paramsCREATE TABLE trial_params ( param_id INTEGER NOT NULL, trial_id INTEGER, param_name VARCHAR(512), param_value FLOAT, distribution_json TEXT, PRIMARY KEY (param_id), UNIQUE (trial_id, param_name), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_params_1trial_params;;)tabletrial_system_attributestrial_system_attributesCREATE TABLE trial_system_attributes ( trial_system_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_system_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Ma;indexsqlite_autoindex_trial_system_attributes_1trial_system_attributes? 77tabletrial_user_attributestrial_user_attributes CREATE TABLE trial_user_attributes ( trial_user_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_user_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )K]7indexsqlite_autoindex_trial_user_attributes_1trial_user_attributes   tabletrialstrials CREATE TABLE trials ( trial_id INTEGER NOT NULL, number INTEGER, study_id INTEGER, state VARCHAR(8) NOT NULL, datetime_start DATETIME, datetime_complete DATETIME, PRIMARY KEY (trial_id), FOREIGN KEY(study_id) REFERENCES studies (study_id) )J;;+tablestudy_system_attributesstudy_system_attributes CREATE TABLE study_system_attributes ( study_system_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (study_system_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )M a;indexsqlite_autoindex_study_system_attributes_1study_system_attributes @77tablestudy_user_attributesstudy_user_attributesCREATE TABLE study_user_attributes ( study_user_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (study_user_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )I]7indexsqlite_autoindex_study_user_attributes_1study_user_attributesJ--Gtablestudy_directionsstudy_directionsCREATE TABLE study_directions ( study_direction_id INTEGER NOT NULL, direction VARCHAR(8) NOT NULL, study_id INTEGER NOT NULL, objective INTEGER NOT NULL, PRIMARY KEY (study_direction_id), UNIQUE (study_id, objective), FOREIGN KEY(study_id) REFERENCES studies (study_id) )?S-indexsqlite_autoindex_study_directions_1study_directions\%%{tableversion_infoversion_infoCREATE TABLE version_info ( version_info_id INTEGER NOT NULL, schema_version INTEGER, library_version VARCHAR(256), PRIMARY KEY (version_info_id), CHECK (version_info_id=1) )j7indexix_studies_study_namestudiesCREATE UNIQUE INDEX ix_studies_study_name ON studies (study_name) otablestudiesstudiesCREATE TABLE studies ( study_id INTEGER NOT NULL, study_name VARCHAR(512) NOT NULL, PRIMARY KEY (study_id) ) # 8i J l+8)++ tablealembic_versionalembic_versionCREATE TABLE alembic_version ( version_num VARCHAR(32) NOT NULL, CONSTRAINT alembic_version_pkc PRIMARY KEY (version_num) )=Q+indexsqlite_autoindex_alembic_version_1alembic_version?S-indexsqlite_autoindex_trial_heartbeats_1trial_heartbeats--otabletrial_heartbeatstrial_heartbeatsCREATE TABLE trial_heartbeats ( trial_heartbeat_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, heartbeat DATETIME NOT NULL, PRIMARY KEY (trial_heartbeat_id), UNIQUE (trial_id), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Qe?indexsqlite_autoindex_trial_intermediate_values_1trial_intermediate_valuesg??]tabletrial_intermediate_valuestrial_intermediate_valuesCREATE TABLE trial_intermediate_values ( trial_intermediate_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, step INTEGER NOT NULL, intermediate_value FLOAT, PRIMARY KEY (trial_intermediate_value_id), UNIQUE (trial_id, step), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_values_1trial_values,%%tabletrial_valuestrial_valuesCREATE TABLE trial_values ( trial_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, objective INTEGER NOT NULL, value FLOAT NOT NULL, PRIMARY KEY (trial_value_id), UNIQUE (trial_id, objective), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_params_1trial_params,%%tabletrial_paramstrial_paramsCREATE TABLE trial_params ( param_id INTEGER NOT NULL, trial_id INTEGER, param_name VARCHAR(512), param_value FLOAT, distribution_json TEXT, PRIMARY KEY (param_id), UNIQUE (trial_id, param_name), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Ma;indexsqlite_autoindex_trial_system_attributes_1trial_system_attributesI ;;)tabletrial_system_attributestrial_system_attributesCREATE TABLE trial_system_attributes ( trial_system_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_system_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )I ]7indexsqlite_autoindex_trial_user_attributes_1trial_user_attributes  v3.0.0.b   v3.0.0.boptuna-4.1.0/tests/storages_tests/rdb_tests/test_upgrade_assets/3.0.0.c.db000066400000000000000000003300001471332314300264610ustar00rootroot00000000000000SQLite format 3@ 77.[5  E f K#oSQe?indexsqlite_autoindex_trial_intermediate_values_1trial_intermediate_values,%%tabletrial_valuestrial_valuesCREATE TABLE trial_values ( trial_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, objective INTEGER NOT NULL, value FLOAT NOT NULL, PRIMARY KEY (trial_value_id), UNIQUE (trial_id, objective), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_values_1trial_values,%%tabletrial_paramstrial_paramsCREATE TABLE trial_params ( param_id INTEGER NOT NULL, trial_id INTEGER, param_name VARCHAR(512), param_value FLOAT, distribution_json TEXT, PRIMARY KEY (param_id), UNIQUE (trial_id, param_name), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_params_1trial_paramsI ;;)tabletrial_system_attributestrial_system_attributesCREATE TABLE trial_system_attributes ( trial_system_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_system_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Ma;indexsqlite_autoindex_trial_system_attributes_1trial_system_attributes? 77tabletrial_user_attributestrial_user_attributes CREATE TABLE trial_user_attributes ( trial_user_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_user_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )I ]7indexsqlite_autoindex_trial_user_attributes_1trial_user_attributes   tabletrialstrials CREATE TABLE trials ( trial_id INTEGER NOT NULL, number INTEGER, study_id INTEGER, state VARCHAR(8) NOT NULL, datetime_start DATETIME, datetime_complete DATETIME, PRIMARY KEY (trial_id), FOREIGN KEY(study_id) REFERENCES studies (study_id) )J;;+tablestudy_system_attributesstudy_system_attributes CREATE TABLE study_system_attributes ( study_system_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (study_system_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )M a;indexsqlite_autoindex_study_system_attributes_1study_system_attributes @77tablestudy_user_attributesstudy_user_attributesCREATE TABLE study_user_attributes ( study_user_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (study_user_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )I]7indexsqlite_autoindex_study_user_attributes_1study_user_attributesJ--Gtablestudy_directionsstudy_directionsCREATE TABLE study_directions ( study_direction_id INTEGER NOT NULL, direction VARCHAR(8) NOT NULL, study_id INTEGER NOT NULL, objective INTEGER NOT NULL, PRIMARY KEY (study_direction_id), UNIQUE (study_id, objective), FOREIGN KEY(study_id) REFERENCES studies (study_id) )?S-indexsqlite_autoindex_study_directions_1study_directions\%%{tableversion_infoversion_infoCREATE TABLE version_info ( version_info_id INTEGER NOT NULL, schema_version INTEGER, library_version VARCHAR(256), PRIMARY KEY (version_info_id), CHECK (version_info_id=1) )j7indexix_studies_study_namestudiesCREATE UNIQUE INDEX ix_studies_study_name ON studies (study_name) otablestudiesstudiesCREATE TABLE studies ( study_id INTEGER NOT NULL, study_name VARCHAR(512) NOT NULL, PRIMARY KEY (study_ -schema migrationmulti#multi_empty single%single_empty -schema migration multi#multi_empty single% single_empty   3.0.0c MINIMIZE MINIMIZEMINIMIZE MINIMIZEMINIMIZEMINIMIZE  MINIMIZE     d3d3 d d c2c2 c c .t.DAACOMPLETE2022-05-31 18:18:17.6635512022-05-31 18:18:17.683868DAACOMPLETE2022-05-31 18:18:17.6268432022-05-31 18:18:17.642115DAACOMPLETE2022-05-31 18:18:17.5582862022-05-31 18:18:17.578216 b1 b1 b  b a0-nsga2:generation0 a0 a-nsga2:generation  a &[ ) A t S  !z{"name": "CategoricalDistribution", "attributes": {"choices": [-5, 0, 5]}}e Ay3{"name": "IntDistribution", "attributes": {"log": false, "step": 3, "low": 5, "high": 14}}d ?y2{"name": "IntDistribution", "attributes": {"log": true, "step": 1, "low": 1, "high": 20}}e Ay1 {"name": "IntDistribution", "attributes": {"log": false, "step": 1, "low": 0, "high": 10}}m Qx3{"name": "FloatDistribution", "attributes": {"step": 2.0, "low": -6.0, "high": 6.0, "log": false}}wWx2?${"name": "FloatDistribution", "attributes": {"step": null, "low": 1e-05, "high": 0.001, "log": true}}uSx1?%S0{"name": "FloatDistribution", "attributes": {"step": null, "low": -5.0, "high": 5.0, "log": false}}S!z{"name": "CategoricalDistribution", "attributes": {"choices": [-5, 0, 5]}}dAy {"name": "IntDistribution", "attributes": {"log": false, "step": 1, "low": 0, "high": 10}}tSx?` {"name": "FloatDistribution", "attributes": {"step": null, "low": -5.0, "high": 5.0, "log": false}}S !z{"name": "CategoricalDistribution", "attributes": {"choices": [-5, 0, 5]}}c Ay{"name": "IntDistribution", "attributes": {"log": false, "step": 1, "low": 0, "high": 10}}s Sx@vב.{"name": "FloatDistribution", "attributes": {"step": null, "low": -5.0, "high": 5.0, "log": false}} z y3 y2 y1 x3 x2x1zyx z y  x @Ջ @[Ooh?`  @Sm     ?FINITE    q&  E f K#oSgS??;tabletrial_intermediate_valuestrial_intermediate_valuesCREATE TABLE trial_intermediate_values ( trial_intermediate_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, step INTEGER NOT NULL, intermediate_value FLOAT, intermediate_value_type VARCHAR(7) NOT NULL, PRIMARY KEY (trial_intermediate_value_id), UNIQUE (trial_id, step), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )e?indexsqlite_autoindex_trial_intermediate_values_1trial_intermediate_valuesk%%tabletrial_valuestrial_valuesCREATE TABLE trial_values ( trial_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, objective INTEGER NOT NULL, value FLOAT NOT NULL, PRIMARY KEY (trial_value_id), UNIQUE (trial_id, objective), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_values_1trial_values%%tabletrial_paramstrial_paramsCREATE TABLE trial_params ( param_id INTEGER NOT NULL, trial_id INTEGER, param_name VARCHAR(512), param_value FLOAT, distribution_json TEXT, PRIMARY KEY (param_id), UNIQUE (trial_id, param_name), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_params_1trial_params;;)tabletrial_system_attributestrial_system_attributesCREATE TABLE trial_system_attributes ( trial_system_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_system_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Ma;indexsqlite_autoindex_trial_system_attributes_1trial_system_attributes? 77tabletrial_user_attributestrial_user_attributes CREATE TABLE trial_user_attributes ( trial_user_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_user_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )K]7indexsqlite_autoindex_trial_user_attributes_1trial_user_attributes   tabletrialstrials CREATE TABLE trials ( trial_id INTEGER NOT NULL, number INTEGER, study_id INTEGER, state VARCHAR(8) NOT NULL, datetime_start DATETIME, datetime_complete DATETIME, PRIMARY KEY (trial_id), FOREIGN KEY(study_id) REFERENCES studies (study_id) )J;;+tablestudy_system_attributesstudy_system_attributes CREATE TABLE study_system_attributes ( study_system_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (study_system_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )M a;indexsqlite_autoindex_study_system_attributes_1study_system_attributes @77tablestudy_user_attributesstudy_user_attributesCREATE TABLE study_user_attributes ( study_user_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (study_user_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )I]7indexsqlite_autoindex_study_user_attributes_1study_user_attributesJ--Gtablestudy_directionsstudy_directionsCREATE TABLE study_directions ( study_direction_id INTEGER NOT NULL, direction VARCHAR(8) NOT NULL, study_id INTEGER NOT NULL, objective INTEGER NOT NULL, PRIMARY KEY (study_direction_id), UNIQUE (study_id, objective), FOREIGN KEY(study_id) REFERENCES studies (study_id) )?S-indexsqlite_autoindex_study_directions_1study_directions\%%{tableversion_infoversion_infoCREATE TABLE version_info ( version_info_id INTEGER NOT NULL, schema_version INTEGER, library_version VARCHAR(256), PRIMARY KEY (version_info_id), CHECK (version_info_id=1) )j7indexix_studies_study_namestudiesCREATE UNIQUE INDEX ix_studies_study_name ON studies (study_name) otablestudiesstudiesCREATE TABLE studies ( study_id INTEGER NOT NULL, study_name VARCHAR(512) NOT NULL, PRIMARY KEY (study_id) )   V  i J ^  )++ tablealembic_versionalembic_versionCREATE TABLE alembic_version ( version_num VARCHAR(32) NOT NULL, CONSTRAINT alembic_version_pkc PRIMARY KEY (version_num) )=Q+indexsqlite_autoindex_alembic_version_1alembic_version--otabletrial_heartbeatstrial_heartbeatsCREATE TABLE trial_heartbeats ( trial_heartbeat_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, heartbeat DATETIME NOT NULL, PRIMARY KEY (trial_heartbeat_id), UNIQUE (trial_id), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )?S-indexsqlite_autoindex_trial_heartbeats_1trial_heartbeatsQe?indexsqlite_autoindex_trial_intermediate_values_1trial_intermediate_values??;tabletrial_intermediate_valuestrial_intermediate_valuesCREATE TABLE trial_intermediate_values ( trial_intermediate_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, step INTEGER NOT NULL, intermediate_value FLOAT, intermediate_value_type VARCHAR(7) NOT NULL, PRIMARY KEY (trial_intermediate_value_id), UNIQUE (trial_id, step), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_values_1trial_values,%%tabletrial_valuestrial_valuesCREATE TABLE trial_values ( trial_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, objective INTEGER NOT NULL, value FLOAT NOT NULL, PRIMARY KEY (trial_value_id), UNIQUE (trial_id, objective), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_params_1trial_params,%%tabletrial_paramstrial_paramsCREATE TABLE trial_params ( param_id INTEGER NOT NULL, trial_id INTEGER, param_name VARCHAR(512), param_value FLOAT, distribution_json TEXT, PRIMARY KEY (param_id), UNIQUE (trial_id, param_name), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Ma;indexsqlite_autoindex_trial_system_attributes_1trial_system_attributesI ;;)tabletrial_system_attributestrial_system_attributesCREATE TABLE trial_system_attributes ( trial_system_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_system_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )I ]7indexsqlite_autoindex_trial_user_attributes_1trial_user_attributes  v3.0.0.c   v3.0.0.coptuna-4.1.0/tests/storages_tests/rdb_tests/test_upgrade_assets/3.0.0.d.db000066400000000000000000003300001471332314300264620ustar00rootroot00000000000000SQLite format 3@ 77.[5  E f K#o:Qe?indexsqlite_autoindex_trial_intermediate_values_1trial_intermediate_valuesE%%Mtabletrial_valuestrial_valuesCREATE TABLE trial_values ( trial_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, objective INTEGER NOT NULL, value FLOAT, value_type VARCHAR(7) NOT NULL, PRIMARY KEY (trial_value_id), UNIQUE (trial_id, objective), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_values_1trial_values,%%tabletrial_paramstrial_paramsCREATE TABLE trial_params ( param_id INTEGER NOT NULL, trial_id INTEGER, param_name VARCHAR(512), param_value FLOAT, distribution_json TEXT, PRIMARY KEY (param_id), UNIQUE (trial_id, param_name), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_params_1trial_paramsI ;;)tabletrial_system_attributestrial_system_attributesCREATE TABLE trial_system_attributes ( trial_system_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_system_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Ma;indexsqlite_autoindex_trial_system_attributes_1trial_system_attributes? 77tabletrial_user_attributestrial_user_attributes CREATE TABLE trial_user_attributes ( trial_user_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_user_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )I ]7indexsqlite_autoindex_trial_user_attributes_1trial_user_attributes   tabletrialstrials CREATE TABLE trials ( trial_id INTEGER NOT NULL, number INTEGER, study_id INTEGER, state VARCHAR(8) NOT NULL, datetime_start DATETIME, datetime_complete DATETIME, PRIMARY KEY (trial_id), FOREIGN KEY(study_id) REFERENCES studies (study_id) )J;;+tablestudy_system_attributesstudy_system_attributes CREATE TABLE study_system_attributes ( study_system_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (study_system_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )M a;indexsqlite_autoindex_study_system_attributes_1study_system_attributes @77tablestudy_user_attributesstudy_user_attributesCREATE TABLE study_user_attributes ( study_user_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (study_user_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )I]7indexsqlite_autoindex_study_user_attributes_1study_user_attributesJ--Gtablestudy_directionsstudy_directionsCREATE TABLE study_directions ( study_direction_id INTEGER NOT NULL, direction VARCHAR(8) NOT NULL, study_id INTEGER NOT NULL, objective INTEGER NOT NULL, PRIMARY KEY (study_direction_id), UNIQUE (study_id, objective), FOREIGN KEY(study_id) REFERENCES studies (study_id) )?S-indexsqlite_autoindex_study_directions_1study_directions\%%{tableversion_infoversion_infoCREATE TABLE version_info ( version_info_id INTEGER NOT NULL, schema_version INTEGER, library_version VARCHAR(256), PRIMARY KEY (version_info_id), CHECK (version_info_id=1) )j7indexix_studies_study_namestudiesCREATE UNIQUE INDEX ix_studies_study_name ON studies (study_name) otablestudiesstudiesCREATE TABLE studies ( study_id INTEGER NOT NULL, study_name VARCHAR(512) NOT NULL, PRIMARY KEY (study_ -schema migrationmulti#multi_empty single%single_empty -schema migration multi#multi_empty single% single_empty # 3.0.0b1.dev MINIMIZE MINIMIZEMINIMIZE MINIMIZEMINIMIZEMINIMIZE  MINIMIZE     d3d3 d d c2c2 c c .t.DAACOMPLETE2022-06-02 21:07:53.1555082022-06-02 21:07:53.176120DAACOMPLETE2022-06-02 21:07:53.1188402022-06-02 21:07:53.133590DAACOMPLETE2022-06-02 21:07:53.0516352022-06-02 21:07:53.069650 b1 b1 b  b a0-nsga2:generation0 a0 a-nsga2:generation  a &[ ) A t S !z{"name": "CategoricalDistribution", "attributes": {"choices": [-5, 0, 5]}}e Ay3{"name": "IntDistribution", "attributes": {"log": false, "step": 3, "low": 5, "high": 14}}d ?y2{"name": "IntDistribution", "attributes": {"log": true, "step": 1, "low": 1, "high": 20}}e Ay1{"name": "IntDistribution", "attributes": {"log": false, "step": 1, "low": 0, "high": 10}}m Qx3{"name": "FloatDistribution", "attributes": {"step": 2.0, "low": -6.0, "high": 6.0, "log": false}}wWx2??B9⪊{"name": "FloatDistribution", "attributes": {"step": null, "low": 1e-05, "high": 0.001, "log": true}}uSx1G({"name": "FloatDistribution", "attributes": {"step": null, "low": -5.0, "high": 5.0, "log": false}}S !z{"name": "CategoricalDistribution", "attributes": {"choices": [-5, 0, 5]}}dAy{"name": "IntDistribution", "attributes": {"log": false, "step": 1, "low": 0, "high": 10}}tSxp.{"name": "FloatDistribution", "attributes": {"step": null, "low": -5.0, "high": 5.0, "log": false}}S !z{"name": "CategoricalDistribution", "attributes": {"choices": [-5, 0, 5]}}c Ay{"name": "IntDistribution", "attributes": {"log": false, "step": 1, "low": 0, "high": 10}}s Sx?Szl{"name": "FloatDistribution", "attributes": {"step": null, "low": -5.0, "high": 5.0, "log": false}} z y3 y2 y1 x3 x2x1zyx z y  x @{]FINITE @Df*4`FINITEp.FINITE @P/uFINITE     ?FINITE    q&  E f K#o:N:??;tabletrial_intermediate_valuestrial_intermediate_valuesCREATE TABLE trial_intermediate_values ( trial_intermediate_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, step INTEGER NOT NULL, intermediate_value FLOAT, intermediate_value_type VARCHAR(7) NOT NULL, PRIMARY KEY (trial_intermediate_value_id), UNIQUE (trial_id, step), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )e?indexsqlite_autoindex_trial_intermediate_values_1trial_intermediate_values%%Mtabletrial_valuestrial_valuesCREATE TABLE trial_values ( trial_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, objective INTEGER NOT NULL, value FLOAT, value_type VARCHAR(7) NOT NULL, PRIMARY KEY (trial_value_id), UNIQUE (trial_id, objective), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_values_1trial_values%%tabletrial_paramstrial_paramsCREATE TABLE trial_params ( param_id INTEGER NOT NULL, trial_id INTEGER, param_name VARCHAR(512), param_value FLOAT, distribution_json TEXT, PRIMARY KEY (param_id), UNIQUE (trial_id, param_name), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_params_1trial_params;;)tabletrial_system_attributestrial_system_attributesCREATE TABLE trial_system_attributes ( trial_system_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_system_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Ma;indexsqlite_autoindex_trial_system_attributes_1trial_system_attributes? 77tabletrial_user_attributestrial_user_attributes CREATE TABLE trial_user_attributes ( trial_user_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_user_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )K]7indexsqlite_autoindex_trial_user_attributes_1trial_user_attributes   tabletrialstrials CREATE TABLE trials ( trial_id INTEGER NOT NULL, number INTEGER, study_id INTEGER, state VARCHAR(8) NOT NULL, datetime_start DATETIME, datetime_complete DATETIME, PRIMARY KEY (trial_id), FOREIGN KEY(study_id) REFERENCES studies (study_id) )J;;+tablestudy_system_attributesstudy_system_attributes CREATE TABLE study_system_attributes ( study_system_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (study_system_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )M a;indexsqlite_autoindex_study_system_attributes_1study_system_attributes @77tablestudy_user_attributesstudy_user_attributesCREATE TABLE study_user_attributes ( study_user_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (study_user_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )I]7indexsqlite_autoindex_study_user_attributes_1study_user_attributesJ--Gtablestudy_directionsstudy_directionsCREATE TABLE study_directions ( study_direction_id INTEGER NOT NULL, direction VARCHAR(8) NOT NULL, study_id INTEGER NOT NULL, objective INTEGER NOT NULL, PRIMARY KEY (study_direction_id), UNIQUE (study_id, objective), FOREIGN KEY(study_id) REFERENCES studies (study_id) )?S-indexsqlite_autoindex_study_directions_1study_directions\%%{tableversion_infoversion_infoCREATE TABLE version_info ( version_info_id INTEGER NOT NULL, schema_version INTEGER, library_version VARCHAR(256), PRIMARY KEY (version_info_id), CHECK (version_info_id=1) )j7indexix_studies_study_namestudiesCREATE UNIQUE INDEX ix_studies_study_name ON studies (study_name) otablestudiesstudiesCREATE TABLE studies ( study_id INTEGER NOT NULL, study_name VARCHAR(512) NOT NULL, PRIMARY KEY (study_id) )   = i j 1 E)++ tablealembic_versionalembic_versionCREATE TABLE alembic_version ( version_num VARCHAR(32) NOT NULL, CONSTRAINT alembic_version_pkc PRIMARY KEY (version_num) )=Q+indexsqlite_autoindex_alembic_version_1alembic_version--otabletrial_heartbeatstrial_heartbeatsCREATE TABLE trial_heartbeats ( trial_heartbeat_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, heartbeat DATETIME NOT NULL, PRIMARY KEY (trial_heartbeat_id), UNIQUE (trial_id), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )?S-indexsqlite_autoindex_trial_heartbeats_1trial_heartbeatsQe?indexsqlite_autoindex_trial_intermediate_values_1trial_intermediate_values??;tabletrial_intermediate_valuestrial_intermediate_valuesCREATE TABLE trial_intermediate_values ( trial_intermediate_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, step INTEGER NOT NULL, intermediate_value FLOAT, intermediate_value_type VARCHAR(7) NOT NULL, PRIMARY KEY (trial_intermediate_value_id), UNIQUE (trial_id, step), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_values_1trial_valuesE%%Mtabletrial_valuestrial_valuesCREATE TABLE trial_values ( trial_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, objective INTEGER NOT NULL, value FLOAT, value_type VARCHAR(7) NOT NULL, PRIMARY KEY (trial_value_id), UNIQUE (trial_id, objective), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_params_1trial_params,%%tabletrial_paramstrial_paramsCREATE TABLE trial_params ( param_id INTEGER NOT NULL, trial_id INTEGER, param_name VARCHAR(512), param_value FLOAT, distribution_json TEXT, PRIMARY KEY (param_id), UNIQUE (trial_id, param_name), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Ma;indexsqlite_autoindex_trial_system_attributes_1trial_system_attributesI ;;)tabletrial_system_attributestrial_system_attributesCREATE TABLE trial_system_attributes ( trial_system_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_system_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )I ]7indexsqlite_autoindex_trial_user_attributes_1trial_user_attributes  v3.0.0.d   v3.0.0.doptuna-4.1.0/tests/storages_tests/rdb_tests/test_upgrade_assets/3.2.0.a.db000066400000000000000000003400001471332314300264620ustar00rootroot00000000000000SQLite format 3@ 11.WH -schema migrationmulti#multi_empty single%single_empty -schema migration multi#multi_empty single% single_empty   3.2.0.a MINIMIZE MINIMIZEMINIMIZE MINIMIZEMINIMIZEMINIMIZE  MINIMIZE     d3d3 d d   .t.DAACOMPLETE2023-03-13 13:23:06.5814252023-03-13 13:23:06.604608DAACOMPLETE2023-03-13 13:23:06.5440522023-03-13 13:23:06.559461DAACOMPLETE2023-03-13 13:23:06.4814672023-03-13 13:23:06.500233   b1 b1 b  b a0-nsga2:generation0 a0 a-nsga2:generation  a &[ * B u  S  !z{"name": "CategoricalDistribution", "attributes": {"choices": [-5, 0, 5]}}e Ay3{"name": "IntDistribution", "attributes": {"log": false, "step": 3, "low": 5, "high": 14}}d ?y2{"name": "IntDistribution", "attributes": {"log": true, "step": 1, "low": 1, "high": 20}}e Ay1 {"name": "IntDistribution", "attributes": {"log": false, "step": 1, "low": 0, "high": 10}}m Qx3{"name": "FloatDistribution", "attributes": {"step": 2.0, "low": -6.0, "high": 6.0, "log": false}}wWx2?JG?{"name": "FloatDistribution", "attributes": {"step": null, "low": 1e-05, "high": 0.001, "log": true}}uSx11>-D{"name": "FloatDistribution", "attributes": {"step": null, "low": -5.0, "high": 5.0, "log": false}}S !z{"name": "CategoricalDistribution", "attributes": {"choices": [-5, 0, 5]}}c Ay{"name": "IntDistribution", "attributes": {"log": false, "step": 1, "low": 0, "high": 10}}tSx@*i{"name": "FloatDistribution", "attributes": {"step": null, "low": -5.0, "high": 5.0, "log": false}}S !z{"name": "CategoricalDistribution", "attributes": {"choices": [-5, 0, 5]}}c Ay{"name": "IntDistribution", "attributes": {"log": false, "step": 1, "low": 0, "high": 10}}s Sxh{"name": "FloatDistribution", "attributes": {"step": null, "low": -5.0, "high": 5.0, "log": false}} z y3 y2 y1 x3 x2x1zyx z y  x @toEFINITE @5FINITE@*iFINITE @GEFINITE     ?FINITE    q&  E f Kda(dX 1uindexix_trials_study_idtrials CREATE INDEX ix_trials_study_id ON trials (study_id)  tabletrialstrials CREATE TABLE trials ( trial_id INTEGER NOT NULL, number INTEGER, study_id INTEGER, state VARCHAR(8) NOT NULL, datetime_start DATETIME, datetime_complete DATETIME, PRIMARY KEY (trial_id), FOREIGN KEY(study_id) REFERENCES studies (study_id) )J;;+tablestudy_system_attributesstudy_system_attributes CREATE TABLE study_system_attributes ( study_system_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (study_system_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )M a;indexsqlite_autoindex_study_system_attributes_1study_system_attributes @77tablestudy_user_attributesstudy_user_attributesCREATE TABLE study_user_attributes ( study_user_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (study_user_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )I]7indexsqlite_autoindex_study_user_attributes_1study_user_attributesJ--Gtablestudy_directionsstudy_directionsCREATE TABLE study_directions ( study_direction_id INTEGER NOT NULL, direction VARCHAR(8) NOT NULL, study_id INTEGER NOT NULL, objective INTEGER NOT NULL, PRIMARY KEY (study_direction_id), UNIQUE (study_id, objective), FOREIGN KEY(study_id) REFERENCES studies (study_id) )?S-indexsqlite_autoindex_study_directions_1study_directions\%%{tableversion_infoversion_infoCREATE TABLE version_info ( version_info_id INTEGER NOT NULL, schema_version INTEGER, library_version VARCHAR(256), PRIMARY KEY (version_info_id), CHECK (version_info_id=1) )j7indexix_studies_study_namestudiesCREATE UNIQUE INDEX ix_studies_study_name ON studies (study_name) otablestudiesstudiesCREATE TABLE studies ( study_id INTEGER NOT NULL, study_name VARCHAR(512) NOT NULL, PRIMARY KEY (study_id) ) s ' p ( VZ)++ tablealembic_versionalembic_versionCREATE TABLE alembic_version ( version_num VARCHAR(32) NOT NULL, CONSTRAINT alembic_version_pkc PRIMARY KEY (version_num) )=Q+indexsqlite_autoindex_alembic_version_1alembic_version--otabletrial_heartbeatstrial_heartbeatsCREATE TABLE trial_heartbeats ( trial_heartbeat_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, heartbeat DATETIME NOT NULL, PRIMARY KEY (trial_heartbeat_id), UNIQUE (trial_id), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )?S-indexsqlite_autoindex_trial_heartbeats_1trial_heartbeatsQe?indexsqlite_autoindex_trial_intermediate_values_1trial_intermediate_values??;tabletrial_intermediate_valuestrial_intermediate_valuesCREATE TABLE trial_intermediate_values ( trial_intermediate_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, step INTEGER NOT NULL, intermediate_value FLOAT, intermediate_value_type VARCHAR(7) NOT NULL, PRIMARY KEY (trial_intermediate_value_id), UNIQUE (trial_id, step), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_values_1trial_valuesE%%Mtabletrial_valuestrial_valuesCREATE TABLE trial_values ( trial_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, objective INTEGER NOT NULL, value FLOAT, value_type VARCHAR(7) NOT NULL, PRIMARY KEY (trial_value_id), UNIQUE (trial_id, objective), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_params_1trial_params,%%tabletrial_paramstrial_paramsCREATE TABLE trial_params ( param_id INTEGER NOT NULL, trial_id INTEGER, param_name VARCHAR(512), param_value FLOAT, distribution_json TEXT, PRIMARY KEY (param_id), UNIQUE (trial_id, param_name), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Ma;indexsqlite_autoindex_trial_system_attributes_1trial_system_attributesI;;)tabletrial_system_attributestrial_system_attributesCREATE TABLE trial_system_attributes ( trial_system_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_system_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )I ]7indexsqlite_autoindex_trial_user_attributes_1trial_user_attributes? 77tabletrial_user_attributestrial_user_attributes CREATE TABLE trial_user_attributes ( trial_user_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_user_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )    v3.2.0.a   v3.2.0.aoptuna-4.1.0/tests/storages_tests/test_cached_storage.py000066400000000000000000000124321471332314300236040ustar00rootroot00000000000000from unittest.mock import patch import pytest import optuna from optuna.storages._cached_storage import _CachedStorage from optuna.storages._rdb.storage import RDBStorage from optuna.study import StudyDirection from optuna.trial import TrialState def test_create_trial() -> None: base_storage = RDBStorage("sqlite:///:memory:") storage = _CachedStorage(base_storage) study_id = storage.create_new_study( directions=[StudyDirection.MINIMIZE], study_name="test-study" ) frozen_trial = optuna.trial.FrozenTrial( number=1, state=TrialState.RUNNING, value=None, datetime_start=None, datetime_complete=None, params={}, distributions={}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=1, ) with patch.object(base_storage, "_create_new_trial", return_value=frozen_trial): storage.create_new_trial(study_id) storage.create_new_trial(study_id) def test_set_trial_state_values() -> None: base_storage = RDBStorage("sqlite:///:memory:") storage = _CachedStorage(base_storage) study_id = storage.create_new_study( directions=[StudyDirection.MINIMIZE], study_name="test-study" ) trial_id = storage.create_new_trial(study_id) storage.set_trial_state_values(trial_id, state=TrialState.COMPLETE) cached_trial = storage.get_trial(trial_id) base_trial = base_storage.get_trial(trial_id) assert cached_trial == base_trial def test_uncached_set() -> None: """Test CachedStorage does flush to persistent storages. The CachedStorage flushes any modification of trials to a persistent storage immediately. """ base_storage = RDBStorage("sqlite:///:memory:") storage = _CachedStorage(base_storage) study_id = storage.create_new_study( directions=[StudyDirection.MINIMIZE], study_name="test-study" ) trial_id = storage.create_new_trial(study_id) trial = storage.get_trial(trial_id) with patch.object(base_storage, "set_trial_state_values", return_value=True) as set_mock: storage.set_trial_state_values(trial_id, state=trial.state, values=(0.3,)) assert set_mock.call_count == 1 trial_id = storage.create_new_trial(study_id) with patch.object(base_storage, "set_trial_param", return_value=True) as set_mock: storage.set_trial_param( trial_id, "paramA", 1.2, optuna.distributions.FloatDistribution(-0.2, 2.3) ) assert set_mock.call_count == 1 for state in [TrialState.COMPLETE, TrialState.PRUNED, TrialState.FAIL, TrialState.WAITING]: trial_id = storage.create_new_trial(study_id) with patch.object(base_storage, "set_trial_state_values", return_value=True) as set_mock: storage.set_trial_state_values(trial_id, state=state) assert set_mock.call_count == 1 trial_id = storage.create_new_trial(study_id) with patch.object(base_storage, "set_trial_intermediate_value", return_value=None) as set_mock: storage.set_trial_intermediate_value(trial_id, 3, 0.3) assert set_mock.call_count == 1 trial_id = storage.create_new_trial(study_id) with patch.object(base_storage, "set_trial_system_attr", return_value=None) as set_mock: storage.set_trial_system_attr(trial_id, "attrA", "foo") assert set_mock.call_count == 1 trial_id = storage.create_new_trial(study_id) with patch.object(base_storage, "set_trial_user_attr", return_value=None) as set_mock: storage.set_trial_user_attr(trial_id, "attrB", "bar") assert set_mock.call_count == 1 def test_read_trials_from_remote_storage() -> None: base_storage = RDBStorage("sqlite:///:memory:") storage = _CachedStorage(base_storage) study_id = storage.create_new_study( directions=[StudyDirection.MINIMIZE], study_name="test-study" ) storage._read_trials_from_remote_storage(study_id) # Non-existent study. with pytest.raises(KeyError): storage._read_trials_from_remote_storage(study_id + 1) # Create a trial via CachedStorage and update it via backend storage directly. trial_id = storage.create_new_trial(study_id) base_storage.set_trial_param( trial_id, "paramA", 1.2, optuna.distributions.FloatDistribution(-0.2, 2.3) ) base_storage.set_trial_state_values(trial_id, TrialState.COMPLETE, values=[0.0]) storage._read_trials_from_remote_storage(study_id) assert storage.get_trial(trial_id).state == TrialState.COMPLETE def test_delete_study() -> None: base_storage = RDBStorage("sqlite:///:memory:") storage = _CachedStorage(base_storage) study_id1 = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) trial_id1 = storage.create_new_trial(study_id1) storage.set_trial_state_values(trial_id1, state=TrialState.COMPLETE) study_id2 = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) trial_id2 = storage.create_new_trial(study_id2) storage.set_trial_state_values(trial_id2, state=TrialState.COMPLETE) # Update _StudyInfo.finished_trial_ids storage._read_trials_from_remote_storage(study_id1) storage._read_trials_from_remote_storage(study_id2) storage.delete_study(study_id1) assert storage._get_cached_trial(trial_id1) is None assert storage._get_cached_trial(trial_id2) is not None optuna-4.1.0/tests/storages_tests/test_heartbeat.py000066400000000000000000000335271471332314300226200ustar00rootroot00000000000000import itertools import multiprocessing import time from typing import Any from typing import Optional from unittest.mock import Mock from unittest.mock import patch import warnings import pytest import optuna from optuna import Study from optuna.storages import RDBStorage from optuna.storages._callbacks import RetryFailedTrialCallback from optuna.storages._heartbeat import BaseHeartbeat from optuna.storages._heartbeat import is_heartbeat_enabled from optuna.testing.storages import STORAGE_MODES_HEARTBEAT from optuna.testing.storages import StorageSupplier from optuna.testing.threading import _TestableThread from optuna.trial import FrozenTrial from optuna.trial import TrialState @pytest.mark.parametrize("storage_mode", STORAGE_MODES_HEARTBEAT) def test_repeatedly_called_record_heartbeat(storage_mode: str) -> None: heartbeat_interval = 1 grace_period = 2 with StorageSupplier( storage_mode, heartbeat_interval=heartbeat_interval, grace_period=grace_period ) as storage: assert is_heartbeat_enabled(storage) assert isinstance(storage, BaseHeartbeat) study1 = optuna.create_study(storage=storage) with pytest.warns(UserWarning): trial1 = study1.ask() storage.record_heartbeat(trial1._trial_id) storage.record_heartbeat(trial1._trial_id) @pytest.mark.parametrize("storage_mode", STORAGE_MODES_HEARTBEAT) def test_fail_stale_trials_with_optimize(storage_mode: str) -> None: heartbeat_interval = 1 grace_period = 2 with StorageSupplier( storage_mode, heartbeat_interval=heartbeat_interval, grace_period=grace_period ) as storage: assert is_heartbeat_enabled(storage) assert isinstance(storage, BaseHeartbeat) study1 = optuna.create_study(storage=storage) study2 = optuna.create_study(storage=storage) with pytest.warns(UserWarning): trial1 = study1.ask() trial2 = study2.ask() storage.record_heartbeat(trial1._trial_id) storage.record_heartbeat(trial2._trial_id) time.sleep(grace_period + 1) assert study1.trials[0].state is TrialState.RUNNING assert study2.trials[0].state is TrialState.RUNNING # Exceptions raised in spawned threads are caught by `_TestableThread`. with patch("optuna.storages._heartbeat.Thread", _TestableThread): study1.optimize(lambda _: 1.0, n_trials=1) assert study1.trials[0].state is TrialState.FAIL # type: ignore [comparison-overlap] assert study2.trials[0].state is TrialState.RUNNING @pytest.mark.parametrize("storage_mode", STORAGE_MODES_HEARTBEAT) def test_invalid_heartbeat_interval_and_grace_period(storage_mode: str) -> None: with pytest.raises(ValueError): with StorageSupplier(storage_mode, heartbeat_interval=-1): pass with pytest.raises(ValueError): with StorageSupplier(storage_mode, grace_period=-1): pass @pytest.mark.parametrize("storage_mode", STORAGE_MODES_HEARTBEAT) def test_failed_trial_callback(storage_mode: str) -> None: heartbeat_interval = 1 grace_period = 2 def _failed_trial_callback(study: Study, trial: FrozenTrial) -> None: assert study._storage.get_study_system_attrs(study._study_id)["test"] == "A" assert trial.system_attrs["test"] == "B" failed_trial_callback = Mock(wraps=_failed_trial_callback) with StorageSupplier( storage_mode, heartbeat_interval=heartbeat_interval, grace_period=grace_period, failed_trial_callback=failed_trial_callback, ) as storage: assert is_heartbeat_enabled(storage) assert isinstance(storage, BaseHeartbeat) study = optuna.create_study(storage=storage) study._storage.set_study_system_attr(study._study_id, "test", "A") with pytest.warns(UserWarning): trial = study.ask() trial.storage.set_trial_system_attr(trial._trial_id, "test", "B") storage.record_heartbeat(trial._trial_id) time.sleep(grace_period + 1) # Exceptions raised in spawned threads are caught by `_TestableThread`. with patch("optuna.storages._heartbeat.Thread", _TestableThread): study.optimize(lambda _: 1.0, n_trials=1) failed_trial_callback.assert_called_once() @pytest.mark.parametrize( "storage_mode,max_retry", itertools.product(STORAGE_MODES_HEARTBEAT, [None, 0, 1]) ) def test_retry_failed_trial_callback(storage_mode: str, max_retry: Optional[int]) -> None: heartbeat_interval = 1 grace_period = 2 with StorageSupplier( storage_mode, heartbeat_interval=heartbeat_interval, grace_period=grace_period, failed_trial_callback=RetryFailedTrialCallback(max_retry=max_retry), ) as storage: assert is_heartbeat_enabled(storage) assert isinstance(storage, BaseHeartbeat) study = optuna.create_study(storage=storage) with pytest.warns(UserWarning): trial = study.ask() trial.suggest_float("_", -1, -1) trial.report(0.5, 1) storage.record_heartbeat(trial._trial_id) time.sleep(grace_period + 1) # Exceptions raised in spawned threads are caught by `_TestableThread`. with patch("optuna.storages._heartbeat.Thread", _TestableThread): study.optimize(lambda _: 1.0, n_trials=1) # Test the last trial to see if it was a retry of the first trial or not. # Test max_retry=None to see if trial is retried. # Test max_retry=0 to see if no trials are retried. # Test max_retry=1 to see if trial is retried. assert RetryFailedTrialCallback.retried_trial_number(study.trials[1]) == ( None if max_retry == 0 else 0 ) # Test inheritance of trial fields. if max_retry != 0: assert study.trials[0].params == study.trials[1].params assert study.trials[0].distributions == study.trials[1].distributions assert study.trials[0].user_attrs == study.trials[1].user_attrs # Only `intermediate_values` are not inherited. assert study.trials[1].intermediate_values == {} @pytest.mark.parametrize( "storage_mode,max_retry", itertools.product(STORAGE_MODES_HEARTBEAT, [None, 0, 1]) ) def test_retry_failed_trial_callback_intermediate( storage_mode: str, max_retry: Optional[int] ) -> None: heartbeat_interval = 1 grace_period = 2 with StorageSupplier( storage_mode, heartbeat_interval=heartbeat_interval, grace_period=grace_period, failed_trial_callback=RetryFailedTrialCallback( max_retry=max_retry, inherit_intermediate_values=True ), ) as storage: assert is_heartbeat_enabled(storage) assert isinstance(storage, BaseHeartbeat) study = optuna.create_study(storage=storage) with warnings.catch_warnings(): warnings.simplefilter("ignore", category=UserWarning) trial = study.ask() trial.suggest_float("_", -1, -1) trial.report(0.5, 1) storage.record_heartbeat(trial._trial_id) time.sleep(grace_period + 1) # Exceptions raised in spawned threads are caught by `_TestableThread`. with patch("optuna.storages._heartbeat.Thread", _TestableThread): study.optimize(lambda _: 1.0, n_trials=1) # Test the last trial to see if it was a retry of the first trial or not. # Test max_retry=None to see if trial is retried. # Test max_retry=0 to see if no trials are retried. # Test max_retry=1 to see if trial is retried. assert RetryFailedTrialCallback.retried_trial_number(study.trials[1]) == ( None if max_retry == 0 else 0 ) # Test inheritance of trial fields. if max_retry != 0: assert study.trials[0].params == study.trials[1].params assert study.trials[0].distributions == study.trials[1].distributions assert study.trials[0].user_attrs == study.trials[1].user_attrs assert study.trials[0].intermediate_values == study.trials[1].intermediate_values @pytest.mark.parametrize("grace_period", [None, 2]) def test_fail_stale_trials(grace_period: Optional[int]) -> None: storage_mode = "sqlite" heartbeat_interval = 1 _grace_period = (heartbeat_interval * 2) if grace_period is None else grace_period def failed_trial_callback(study: "optuna.Study", trial: FrozenTrial) -> None: assert study._storage.get_study_system_attrs(study._study_id)["test"] == "A" assert trial.system_attrs["test"] == "B" def check_change_trial_state_to_fail(study: "optuna.Study") -> None: assert study.trials[0].state is TrialState.RUNNING optuna.storages.fail_stale_trials(study) assert study.trials[0].state is TrialState.FAIL # type: ignore [comparison-overlap] def check_keep_trial_state_in_running(study: "optuna.Study") -> None: assert study.trials[0].state is TrialState.RUNNING optuna.storages.fail_stale_trials(study) assert study.trials[0].state is TrialState.RUNNING with StorageSupplier(storage_mode) as storage: assert isinstance(storage, RDBStorage) storage.heartbeat_interval = heartbeat_interval storage.grace_period = grace_period storage.failed_trial_callback = failed_trial_callback study = optuna.create_study(storage=storage) study._storage.set_study_system_attr(study._study_id, "test", "A") with pytest.warns(UserWarning): trial = study.ask() trial.storage.set_trial_system_attr(trial._trial_id, "test", "B") time.sleep(_grace_period + 1) check_keep_trial_state_in_running(study) storage.record_heartbeat(trial._trial_id) check_keep_trial_state_in_running(study) time.sleep(_grace_period + 1) check_change_trial_state_to_fail(study) def run_fail_stale_trials(storage_url: str, sleep_time: int) -> None: heartbeat_interval = 1 grace_period = 2 storage = RDBStorage(storage_url) storage.heartbeat_interval = heartbeat_interval storage.grace_period = grace_period original_set_trial_state_values = storage.set_trial_state_values def _set_trial_state_values(*args: Any, **kwargs: Any) -> bool: # The second process fails to set state due to the race condition. time.sleep(sleep_time) return original_set_trial_state_values(*args, **kwargs) storage.set_trial_state_values = _set_trial_state_values # type: ignore[method-assign] study = optuna.load_study(study_name=None, storage=storage) optuna.storages.fail_stale_trials(study) def test_fail_stale_trials_with_race_condition() -> None: grace_period = 2 storage_mode = "sqlite" with StorageSupplier(storage_mode) as storage: assert isinstance(storage, RDBStorage) study = optuna.create_study(storage=storage) trial = study.ask() storage.record_heartbeat(trial._trial_id) time.sleep(grace_period + 1) p1 = multiprocessing.Process(target=run_fail_stale_trials, args=(storage.url, 1)) p1.start() p2 = multiprocessing.Process(target=run_fail_stale_trials, args=(storage.url, 2)) p2.start() p1.join() p2.join() assert p1.exitcode == 0 assert p2.exitcode == 0 assert study.trials[0].state is TrialState.FAIL def test_get_stale_trial_ids() -> None: storage_mode = "sqlite" heartbeat_interval = 1 grace_period = 2 with StorageSupplier(storage_mode) as storage: assert isinstance(storage, RDBStorage) storage.heartbeat_interval = heartbeat_interval storage.grace_period = grace_period study = optuna.create_study(storage=storage) with pytest.warns(UserWarning): trial = study.ask() storage.record_heartbeat(trial._trial_id) time.sleep(grace_period + 1) assert len(storage._get_stale_trial_ids(study._study_id)) == 1 assert storage._get_stale_trial_ids(study._study_id)[0] == trial._trial_id @pytest.mark.parametrize("storage_mode", STORAGE_MODES_HEARTBEAT) def test_retry_failed_trial_callback_repetitive_failure(storage_mode: str) -> None: heartbeat_interval = 1 grace_period = 2 max_retry = 3 n_trials = 5 with StorageSupplier( storage_mode, heartbeat_interval=heartbeat_interval, grace_period=grace_period, failed_trial_callback=RetryFailedTrialCallback(max_retry=max_retry), ) as storage: assert is_heartbeat_enabled(storage) assert isinstance(storage, BaseHeartbeat) study = optuna.create_study(storage=storage) # Make repeatedly failed and retried trials by heartbeat. for _ in range(n_trials): with pytest.warns(UserWarning): trial = study.ask() storage.record_heartbeat(trial._trial_id) time.sleep(grace_period + 1) optuna.storages.fail_stale_trials(study) trials = study.trials assert len(trials) == n_trials + 1 assert "failed_trial" not in trials[0].system_attrs assert "retry_history" not in trials[0].system_attrs # The trials 1-3 are retried ones originating from the trial 0. assert trials[1].system_attrs["failed_trial"] == 0 assert trials[1].system_attrs["retry_history"] == [0] assert trials[2].system_attrs["failed_trial"] == 0 assert trials[2].system_attrs["retry_history"] == [0, 1] assert trials[3].system_attrs["failed_trial"] == 0 assert trials[3].system_attrs["retry_history"] == [0, 1, 2] # Trials 4 and later are the newly started ones and # they are retried after exceeding max_retry. assert "failed_trial" not in trials[4].system_attrs assert "retry_history" not in trials[4].system_attrs assert trials[5].system_attrs["failed_trial"] == 4 assert trials[5].system_attrs["retry_history"] == [4] optuna-4.1.0/tests/storages_tests/test_storages.py000066400000000000000000001426251471332314300225100ustar00rootroot00000000000000from __future__ import annotations import copy from datetime import datetime import pickle import random from time import sleep from typing import Any from typing import Dict from typing import List from typing import Optional from typing import Tuple import numpy as np import pytest import optuna from optuna._typing import JSONSerializable from optuna.distributions import CategoricalDistribution from optuna.distributions import FloatDistribution from optuna.storages import _CachedStorage from optuna.storages import BaseStorage from optuna.storages import InMemoryStorage from optuna.storages import RDBStorage from optuna.storages._base import DEFAULT_STUDY_NAME_PREFIX from optuna.study._frozen import FrozenStudy from optuna.study._study_direction import StudyDirection from optuna.testing.storages import STORAGE_MODES from optuna.testing.storages import StorageSupplier from optuna.trial import FrozenTrial from optuna.trial import TrialState ALL_STATES = list(TrialState) EXAMPLE_ATTRS: Dict[str, JSONSerializable] = { "dataset": "MNIST", "none": None, "json_serializable": {"baseline_score": 0.001, "tags": ["image", "classification"]}, } def test_get_storage() -> None: assert isinstance(optuna.storages.get_storage(None), InMemoryStorage) assert isinstance(optuna.storages.get_storage("sqlite:///:memory:"), _CachedStorage) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_create_new_study(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study_id = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) frozen_studies = storage.get_all_studies() assert len(frozen_studies) == 1 assert frozen_studies[0]._study_id == study_id assert frozen_studies[0].study_name.startswith(DEFAULT_STUDY_NAME_PREFIX) study_id2 = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) # Study id must be unique. assert study_id != study_id2 frozen_studies = storage.get_all_studies() assert len(frozen_studies) == 2 assert {s._study_id for s in frozen_studies} == {study_id, study_id2} assert all(s.study_name.startswith(DEFAULT_STUDY_NAME_PREFIX) for s in frozen_studies) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_create_new_study_unique_id(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study_id = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) study_id2 = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) storage.delete_study(study_id2) study_id3 = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) # Study id must not be reused after deletion. if not isinstance(storage, (RDBStorage, _CachedStorage)): # TODO(ytsmiling) Fix RDBStorage so that it does not reuse study_id. assert len({study_id, study_id2, study_id3}) == 3 frozen_studies = storage.get_all_studies() assert {s._study_id for s in frozen_studies} == {study_id, study_id3} @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_create_new_study_with_name(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: # Generate unique study_name from the current function name and storage_mode. function_name = test_create_new_study_with_name.__name__ study_name = function_name + "/" + storage_mode study_id = storage.create_new_study( directions=[StudyDirection.MINIMIZE], study_name=study_name ) assert study_name == storage.get_study_name_from_id(study_id) with pytest.raises(optuna.exceptions.DuplicatedStudyError): storage.create_new_study(directions=[StudyDirection.MINIMIZE], study_name=study_name) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_delete_study(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study_id = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) storage.create_new_trial(study_id) trials = storage.get_all_trials(study_id) assert len(trials) == 1 with pytest.raises(KeyError): # Deletion of non-existent study. storage.delete_study(study_id + 1) storage.delete_study(study_id) study_id = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) trials = storage.get_all_trials(study_id) assert len(trials) == 0 storage.delete_study(study_id) with pytest.raises(KeyError): # Double free. storage.delete_study(study_id) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_delete_study_after_create_multiple_studies(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study_id1 = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) study_id2 = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) study_id3 = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) storage.delete_study(study_id2) studies = {s._study_id: s for s in storage.get_all_studies()} assert study_id1 in studies assert study_id2 not in studies assert study_id3 in studies @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_get_study_id_from_name_and_get_study_name_from_id(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: # Generate unique study_name from the current function name and storage_mode. function_name = test_get_study_id_from_name_and_get_study_name_from_id.__name__ study_name = function_name + "/" + storage_mode study_id = storage.create_new_study( directions=[StudyDirection.MINIMIZE], study_name=study_name ) # Test existing study. assert storage.get_study_name_from_id(study_id) == study_name assert storage.get_study_id_from_name(study_name) == study_id # Test not existing study. with pytest.raises(KeyError): storage.get_study_id_from_name("dummy-name") with pytest.raises(KeyError): storage.get_study_name_from_id(study_id + 1) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_set_and_get_study_directions(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: for target in [ (StudyDirection.MINIMIZE,), (StudyDirection.MAXIMIZE,), (StudyDirection.MINIMIZE, StudyDirection.MAXIMIZE), (StudyDirection.MAXIMIZE, StudyDirection.MINIMIZE), [StudyDirection.MINIMIZE, StudyDirection.MAXIMIZE], [StudyDirection.MAXIMIZE, StudyDirection.MINIMIZE], ]: study_id = storage.create_new_study(directions=target) def check_get() -> None: got_directions = storage.get_study_directions(study_id) assert got_directions == list( target ), "Direction of a study should be a tuple of `StudyDirection` objects." # Test setting value. check_get() # Test non-existent study. non_existent_study_id = study_id + 1 with pytest.raises(KeyError): storage.get_study_directions(non_existent_study_id) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_set_and_get_study_user_attrs(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study_id = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) def check_set_and_get(key: str, value: Any) -> None: storage.set_study_user_attr(study_id, key, value) assert storage.get_study_user_attrs(study_id)[key] == value # Test setting value. for key, value in EXAMPLE_ATTRS.items(): check_set_and_get(key, value) assert storage.get_study_user_attrs(study_id) == EXAMPLE_ATTRS # Test overwriting value. check_set_and_get("dataset", "ImageNet") # Non-existent study id. non_existent_study_id = study_id + 1 with pytest.raises(KeyError): storage.get_study_user_attrs(non_existent_study_id) # Non-existent study id. with pytest.raises(KeyError): storage.set_study_user_attr(non_existent_study_id, "key", "value") @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_set_and_get_study_system_attrs(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study_id = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) def check_set_and_get(key: str, value: Any) -> None: storage.set_study_system_attr(study_id, key, value) assert storage.get_study_system_attrs(study_id)[key] == value # Test setting value. for key, value in EXAMPLE_ATTRS.items(): check_set_and_get(key, value) assert storage.get_study_system_attrs(study_id) == EXAMPLE_ATTRS # Test overwriting value. check_set_and_get("dataset", "ImageNet") # Non-existent study id. non_existent_study_id = study_id + 1 with pytest.raises(KeyError): storage.get_study_system_attrs(non_existent_study_id) # Non-existent study id. with pytest.raises(KeyError): storage.set_study_system_attr(non_existent_study_id, "key", "value") @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_study_user_and_system_attrs_confusion(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study_id = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) for key, value in EXAMPLE_ATTRS.items(): storage.set_study_system_attr(study_id, key, value) assert storage.get_study_system_attrs(study_id) == EXAMPLE_ATTRS assert storage.get_study_user_attrs(study_id) == {} study_id = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) for key, value in EXAMPLE_ATTRS.items(): storage.set_study_user_attr(study_id, key, value) assert storage.get_study_user_attrs(study_id) == EXAMPLE_ATTRS assert storage.get_study_system_attrs(study_id) == {} @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_create_new_trial(storage_mode: str) -> None: def _check_trials( trials: List[FrozenTrial], idx: int, trial_id: int, time_before_creation: datetime, time_after_creation: datetime, ) -> None: assert len(trials) == idx + 1 assert len({t._trial_id for t in trials}) == idx + 1 assert trial_id in {t._trial_id for t in trials} assert {t.number for t in trials} == set(range(idx + 1)) assert all(t.state == TrialState.RUNNING for t in trials) assert all(t.params == {} for t in trials) assert all(t.intermediate_values == {} for t in trials) assert all(t.user_attrs == {} for t in trials) assert all(t.system_attrs == {} for t in trials) assert all( t.datetime_start < time_before_creation for t in trials if t._trial_id != trial_id and t.datetime_start is not None ) assert all( time_before_creation < t.datetime_start < time_after_creation for t in trials if t._trial_id == trial_id and t.datetime_start is not None ) assert all(t.datetime_complete is None for t in trials) assert all(t.value is None for t in trials) with StorageSupplier(storage_mode) as storage: study_id = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) n_trial_in_study = 3 for i in range(n_trial_in_study): time_before_creation = datetime.now() sleep(0.001) # Sleep 1ms to avoid faulty assertion on Windows OS. trial_id = storage.create_new_trial(study_id) sleep(0.001) time_after_creation = datetime.now() trials = storage.get_all_trials(study_id) _check_trials(trials, i, trial_id, time_before_creation, time_after_creation) # Create trial in non-existent study. with pytest.raises(KeyError): storage.create_new_trial(study_id + 1) study_id2 = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) for i in range(n_trial_in_study): storage.create_new_trial(study_id2) trials = storage.get_all_trials(study_id2) # Check that the offset of trial.number is zero. assert {t.number for t in trials} == set(range(i + 1)) trials = storage.get_all_trials(study_id) + storage.get_all_trials(study_id2) # Check trial_ids are unique across studies. assert len({t._trial_id for t in trials}) == 2 * n_trial_in_study @pytest.mark.parametrize("storage_mode", STORAGE_MODES) @pytest.mark.parametrize( "start_time,complete_time", [(datetime.now(), datetime.now()), (datetime(2022, 9, 1), datetime(2022, 9, 2))], ) def test_create_new_trial_with_template_trial( storage_mode: str, start_time: datetime, complete_time: datetime ) -> None: template_trial = FrozenTrial( state=TrialState.COMPLETE, value=10000, datetime_start=start_time, datetime_complete=complete_time, params={"x": 0.5}, distributions={"x": FloatDistribution(0, 1)}, user_attrs={"foo": "bar"}, system_attrs={"baz": 123}, intermediate_values={1: 10, 2: 100, 3: 1000}, number=55, # This entry is ignored. trial_id=-1, # dummy value (unused). ) def _check_trials(trials: List[FrozenTrial], idx: int, trial_id: int) -> None: assert len(trials) == idx + 1 assert len({t._trial_id for t in trials}) == idx + 1 assert trial_id in {t._trial_id for t in trials} assert {t.number for t in trials} == set(range(idx + 1)) assert all(t.state == template_trial.state for t in trials) assert all(t.params == template_trial.params for t in trials) assert all(t.distributions == template_trial.distributions for t in trials) assert all(t.intermediate_values == template_trial.intermediate_values for t in trials) assert all(t.user_attrs == template_trial.user_attrs for t in trials) assert all(t.system_attrs == template_trial.system_attrs for t in trials) assert all(t.datetime_start == template_trial.datetime_start for t in trials) assert all(t.datetime_complete == template_trial.datetime_complete for t in trials) assert all(t.value == template_trial.value for t in trials) with StorageSupplier(storage_mode) as storage: study_id = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) n_trial_in_study = 3 for i in range(n_trial_in_study): trial_id = storage.create_new_trial(study_id, template_trial=template_trial) trials = storage.get_all_trials(study_id) _check_trials(trials, i, trial_id) # Create trial in non-existent study. with pytest.raises(KeyError): storage.create_new_trial(study_id + 1) study_id2 = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) for i in range(n_trial_in_study): storage.create_new_trial(study_id2, template_trial=template_trial) trials = storage.get_all_trials(study_id2) assert {t.number for t in trials} == set(range(i + 1)) trials = storage.get_all_trials(study_id) + storage.get_all_trials(study_id2) # Check trial_ids are unique across studies. assert len({t._trial_id for t in trials}) == 2 * n_trial_in_study @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_get_trial_number_from_id(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: # Check if trial_number starts from 0. study_id = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) trial_id = storage.create_new_trial(study_id) assert storage.get_trial_number_from_id(trial_id) == 0 trial_id = storage.create_new_trial(study_id) assert storage.get_trial_number_from_id(trial_id) == 1 with pytest.raises(KeyError): storage.get_trial_number_from_id(trial_id + 1) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_set_trial_state_values_for_state(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study_id = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) trial_ids = [storage.create_new_trial(study_id) for _ in ALL_STATES] for trial_id, state in zip(trial_ids, ALL_STATES): if state == TrialState.WAITING: continue assert storage.get_trial(trial_id).state == TrialState.RUNNING datetime_start_prev = storage.get_trial(trial_id).datetime_start storage.set_trial_state_values( trial_id, state=state, values=(0.0,) if state.is_finished() else None ) assert storage.get_trial(trial_id).state == state # Repeated state changes to RUNNING should not trigger further datetime_start changes. if state == TrialState.RUNNING: assert storage.get_trial(trial_id).datetime_start == datetime_start_prev if state.is_finished(): assert storage.get_trial(trial_id).datetime_complete is not None else: assert storage.get_trial(trial_id).datetime_complete is None # Non-existent study. with pytest.raises(KeyError): non_existent_trial_id = max(trial_ids) + 1 storage.set_trial_state_values( non_existent_trial_id, state=TrialState.COMPLETE, ) for state in ALL_STATES: if not state.is_finished(): continue trial_id = storage.create_new_trial(study_id) storage.set_trial_state_values(trial_id, state=state, values=(0.0,)) for state2 in ALL_STATES: # Cannot update states of finished trials. with pytest.raises(RuntimeError): storage.set_trial_state_values(trial_id, state=state2) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_get_trial_param_and_get_trial_params(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: _, study_to_trials = _setup_studies(storage, n_study=2, n_trial=5, seed=1) for _, trial_id_to_trial in study_to_trials.items(): for trial_id, expected_trial in trial_id_to_trial.items(): assert storage.get_trial_params(trial_id) == expected_trial.params for key in expected_trial.params.keys(): assert storage.get_trial_param(trial_id, key) == expected_trial.distributions[ key ].to_internal_repr(expected_trial.params[key]) non_existent_trial_id = ( max(tid for ts in study_to_trials.values() for tid in ts.keys()) + 1 ) with pytest.raises(KeyError): storage.get_trial_params(non_existent_trial_id) with pytest.raises(KeyError): storage.get_trial_param(non_existent_trial_id, "paramA") existent_trial_id = non_existent_trial_id - 1 with pytest.raises(KeyError): storage.get_trial_param(existent_trial_id, "dummy-key") @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_set_trial_param(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: # Setup test across multiple studies and trials. study_id = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) trial_id_1 = storage.create_new_trial(study_id) trial_id_2 = storage.create_new_trial(study_id) trial_id_3 = storage.create_new_trial( storage.create_new_study(directions=[StudyDirection.MINIMIZE]) ) # Setup distributions. distribution_x = FloatDistribution(low=1.0, high=2.0) distribution_y_1 = CategoricalDistribution(choices=("Shibuya", "Ebisu", "Meguro")) distribution_y_2 = CategoricalDistribution(choices=("Shibuya", "Shinsen")) distribution_z = FloatDistribution(low=1.0, high=100.0, log=True) # Set new params. storage.set_trial_param(trial_id_1, "x", 0.5, distribution_x) storage.set_trial_param(trial_id_1, "y", 2, distribution_y_1) assert storage.get_trial_param(trial_id_1, "x") == 0.5 assert storage.get_trial_param(trial_id_1, "y") == 2 # Check set_param breaks neither get_trial nor get_trial_params. assert storage.get_trial(trial_id_1).params == {"x": 0.5, "y": "Meguro"} assert storage.get_trial_params(trial_id_1) == {"x": 0.5, "y": "Meguro"} # Set params to another trial. storage.set_trial_param(trial_id_2, "x", 0.3, distribution_x) storage.set_trial_param(trial_id_2, "z", 0.1, distribution_z) assert storage.get_trial_param(trial_id_2, "x") == 0.3 assert storage.get_trial_param(trial_id_2, "z") == 0.1 assert storage.get_trial(trial_id_2).params == {"x": 0.3, "z": 0.1} assert storage.get_trial_params(trial_id_2) == {"x": 0.3, "z": 0.1} # Set params with distributions that do not match previous ones. with pytest.raises(ValueError): storage.set_trial_param(trial_id_2, "y", 0.5, distribution_z) # Choices in CategoricalDistribution should match including its order. with pytest.raises(ValueError): storage.set_trial_param( trial_id_2, "y", 2, CategoricalDistribution(choices=("Meguro", "Shibuya", "Ebisu")) ) storage.set_trial_state_values(trial_id_2, state=TrialState.COMPLETE) # Cannot assign params to finished trial. with pytest.raises(RuntimeError): storage.set_trial_param(trial_id_2, "y", 2, distribution_y_1) # Check the previous call does not change the params. with pytest.raises(KeyError): storage.get_trial_param(trial_id_2, "y") # State should be checked prior to distribution compatibility. with pytest.raises(RuntimeError): storage.set_trial_param(trial_id_2, "y", 0.4, distribution_z) # Set params of trials in a different study. storage.set_trial_param(trial_id_3, "y", 1, distribution_y_2) assert storage.get_trial_param(trial_id_3, "y") == 1 assert storage.get_trial(trial_id_3).params == {"y": "Shinsen"} assert storage.get_trial_params(trial_id_3) == {"y": "Shinsen"} # Set params of non-existent trial. non_existent_trial_id = max([trial_id_1, trial_id_2, trial_id_3]) + 1 with pytest.raises(KeyError): storage.set_trial_param(non_existent_trial_id, "x", 0.1, distribution_x) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_set_trial_state_values_for_values(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: # Setup test across multiple studies and trials. study_id = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) trial_id_1 = storage.create_new_trial(study_id) trial_id_2 = storage.create_new_trial(study_id) trial_id_3 = storage.create_new_trial( storage.create_new_study(directions=[StudyDirection.MINIMIZE]) ) trial_id_4 = storage.create_new_trial(study_id) trial_id_5 = storage.create_new_trial(study_id) # Test setting new value. storage.set_trial_state_values(trial_id_1, state=TrialState.COMPLETE, values=(0.5,)) storage.set_trial_state_values( trial_id_3, state=TrialState.COMPLETE, values=(float("inf"),) ) storage.set_trial_state_values( trial_id_4, state=TrialState.WAITING, values=(0.1, 0.2, 0.3) ) storage.set_trial_state_values( trial_id_5, state=TrialState.WAITING, values=[0.1, 0.2, 0.3] ) assert storage.get_trial(trial_id_1).value == 0.5 assert storage.get_trial(trial_id_2).value is None assert storage.get_trial(trial_id_3).value == float("inf") assert storage.get_trial(trial_id_4).values == [0.1, 0.2, 0.3] assert storage.get_trial(trial_id_5).values == [0.1, 0.2, 0.3] non_existent_trial_id = max(trial_id_1, trial_id_2, trial_id_3, trial_id_4, trial_id_5) + 1 with pytest.raises(KeyError): storage.set_trial_state_values( non_existent_trial_id, state=TrialState.COMPLETE, values=(1,) ) # Cannot change values of finished trials. with pytest.raises(RuntimeError): storage.set_trial_state_values(trial_id_1, state=TrialState.COMPLETE, values=(1,)) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_set_trial_intermediate_value(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: # Setup test across multiple studies and trials. study_id = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) trial_id_1 = storage.create_new_trial(study_id) trial_id_2 = storage.create_new_trial(study_id) trial_id_3 = storage.create_new_trial( storage.create_new_study(directions=[StudyDirection.MINIMIZE]) ) trial_id_4 = storage.create_new_trial(study_id) # Test setting new values. storage.set_trial_intermediate_value(trial_id_1, 0, 0.3) storage.set_trial_intermediate_value(trial_id_1, 2, 0.4) storage.set_trial_intermediate_value(trial_id_3, 0, 0.1) storage.set_trial_intermediate_value(trial_id_3, 1, 0.4) storage.set_trial_intermediate_value(trial_id_3, 2, 0.5) storage.set_trial_intermediate_value(trial_id_3, 3, float("inf")) storage.set_trial_intermediate_value(trial_id_4, 0, float("nan")) assert storage.get_trial(trial_id_1).intermediate_values == {0: 0.3, 2: 0.4} assert storage.get_trial(trial_id_2).intermediate_values == {} assert storage.get_trial(trial_id_3).intermediate_values == { 0: 0.1, 1: 0.4, 2: 0.5, 3: float("inf"), } assert np.isnan(storage.get_trial(trial_id_4).intermediate_values[0]) # Test setting existing step. storage.set_trial_intermediate_value(trial_id_1, 0, 0.2) assert storage.get_trial(trial_id_1).intermediate_values == {0: 0.2, 2: 0.4} non_existent_trial_id = max(trial_id_1, trial_id_2, trial_id_3, trial_id_4) + 1 with pytest.raises(KeyError): storage.set_trial_intermediate_value(non_existent_trial_id, 0, 0.2) storage.set_trial_state_values(trial_id_1, state=TrialState.COMPLETE) # Cannot change values of finished trials. with pytest.raises(RuntimeError): storage.set_trial_intermediate_value(trial_id_1, 0, 0.2) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_get_trial_user_attrs(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: _, study_to_trials = _setup_studies(storage, n_study=2, n_trial=5, seed=10) assert all( storage.get_trial_user_attrs(trial_id) == trial.user_attrs for trials in study_to_trials.values() for trial_id, trial in trials.items() ) non_existent_trial = max(tid for ts in study_to_trials.values() for tid in ts.keys()) + 1 with pytest.raises(KeyError): storage.get_trial_user_attrs(non_existent_trial) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_set_trial_user_attr(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: trial_id_1 = storage.create_new_trial( storage.create_new_study(directions=[StudyDirection.MINIMIZE]) ) def check_set_and_get(trial_id: int, key: str, value: Any) -> None: storage.set_trial_user_attr(trial_id, key, value) assert storage.get_trial(trial_id).user_attrs[key] == value # Test setting value. for key, value in EXAMPLE_ATTRS.items(): check_set_and_get(trial_id_1, key, value) assert storage.get_trial(trial_id_1).user_attrs == EXAMPLE_ATTRS # Test overwriting value. check_set_and_get(trial_id_1, "dataset", "ImageNet") # Test another trial. trial_id_2 = storage.create_new_trial( storage.create_new_study(directions=[StudyDirection.MINIMIZE]) ) check_set_and_get(trial_id_2, "baseline_score", 0.001) assert len(storage.get_trial(trial_id_2).user_attrs) == 1 assert storage.get_trial(trial_id_2).user_attrs["baseline_score"] == 0.001 # Cannot set attributes of non-existent trials. non_existent_trial_id = max({trial_id_1, trial_id_2}) + 1 with pytest.raises(KeyError): storage.set_trial_user_attr(non_existent_trial_id, "key", "value") # Cannot set attributes of finished trials. storage.set_trial_state_values(trial_id_1, state=TrialState.COMPLETE) with pytest.raises(RuntimeError): storage.set_trial_user_attr(trial_id_1, "key", "value") @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_get_trial_system_attrs(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: _, study_to_trials = _setup_studies(storage, n_study=2, n_trial=5, seed=10) assert all( storage.get_trial_system_attrs(trial_id) == trial.system_attrs for trials in study_to_trials.values() for trial_id, trial in trials.items() ) non_existent_trial = max(tid for ts in study_to_trials.values() for tid in ts.keys()) + 1 with pytest.raises(KeyError): storage.get_trial_system_attrs(non_existent_trial) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_set_trial_system_attr(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study_id = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) trial_id_1 = storage.create_new_trial(study_id) def check_set_and_get(trial_id: int, key: str, value: Any) -> None: storage.set_trial_system_attr(trial_id, key, value) assert storage.get_trial_system_attrs(trial_id)[key] == value # Test setting value. for key, value in EXAMPLE_ATTRS.items(): check_set_and_get(trial_id_1, key, value) system_attrs = storage.get_trial(trial_id_1).system_attrs assert system_attrs == EXAMPLE_ATTRS # Test overwriting value. check_set_and_get(trial_id_1, "dataset", "ImageNet") # Test another trial. trial_id_2 = storage.create_new_trial(study_id) check_set_and_get(trial_id_2, "baseline_score", 0.001) system_attrs = storage.get_trial(trial_id_2).system_attrs assert system_attrs == {"baseline_score": 0.001} # Cannot set attributes of non-existent trials. non_existent_trial_id = max({trial_id_1, trial_id_2}) + 1 with pytest.raises(KeyError): storage.set_trial_system_attr(non_existent_trial_id, "key", "value") # Cannot set attributes of finished trials. storage.set_trial_state_values(trial_id_1, state=TrialState.COMPLETE) with pytest.raises(RuntimeError): storage.set_trial_system_attr(trial_id_1, "key", "value") @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_get_all_studies(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: expected_frozen_studies, _ = _setup_studies(storage, n_study=10, n_trial=10, seed=46) frozen_studies = storage.get_all_studies() assert len(frozen_studies) == len(expected_frozen_studies) for _, expected_frozen_study in expected_frozen_studies.items(): frozen_study: Optional[FrozenStudy] = None for s in frozen_studies: if s.study_name == expected_frozen_study.study_name: frozen_study = s break assert frozen_study is not None assert frozen_study.direction == expected_frozen_study.direction assert frozen_study.study_name == expected_frozen_study.study_name assert frozen_study.user_attrs == expected_frozen_study.user_attrs assert frozen_study.system_attrs == expected_frozen_study.system_attrs @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_get_trial(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: _, study_to_trials = _setup_studies(storage, n_study=2, n_trial=20, seed=47) for _, expected_trials in study_to_trials.items(): for expected_trial in expected_trials.values(): trial = storage.get_trial(expected_trial._trial_id) assert trial == expected_trial non_existent_trial_id = ( max(tid for ts in study_to_trials.values() for tid in ts.keys()) + 1 ) with pytest.raises(KeyError): storage.get_trial(non_existent_trial_id) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_get_all_trials(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: _, study_to_trials = _setup_studies(storage, n_study=2, n_trial=20, seed=48) for study_id, expected_trials in study_to_trials.items(): trials = storage.get_all_trials(study_id) for trial in trials: expected_trial = expected_trials[trial._trial_id] assert trial == expected_trial non_existent_study_id = max(study_to_trials.keys()) + 1 with pytest.raises(KeyError): storage.get_all_trials(non_existent_study_id) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) @pytest.mark.parametrize("param_names", [["a", "b"], ["b", "a"]]) def test_get_all_trials_params_order(storage_mode: str, param_names: list[str]) -> None: # We don't actually require that all storages to preserve the order of parameters, # but all current implementations do, so we test this property. with StorageSupplier(storage_mode) as storage: study_id = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) trial_id = storage.create_new_trial( study_id, optuna.trial.create_trial(state=TrialState.RUNNING) ) for param_name in param_names: storage.set_trial_param( trial_id, param_name, 1.0, distribution=FloatDistribution(0.0, 2.0) ) trials = storage.get_all_trials(study_id) assert list(trials[0].params.keys()) == param_names @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_get_all_trials_deepcopy_option(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: frozen_studies, study_to_trials = _setup_studies(storage, n_study=2, n_trial=5, seed=49) for study_id in frozen_studies: trials0 = storage.get_all_trials(study_id, deepcopy=True) assert len(trials0) == len(study_to_trials[study_id]) # Check modifying output does not break the internal state of the storage. trials0_original = copy.deepcopy(trials0) trials0[0].params["x"] = 0.1 trials1 = storage.get_all_trials(study_id, deepcopy=False) assert trials0_original == trials1 @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_get_all_trials_state_option(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study_id = storage.create_new_study(directions=[StudyDirection.MAXIMIZE]) generator = random.Random(51) states = ( TrialState.COMPLETE, TrialState.COMPLETE, TrialState.PRUNED, ) for state in states: t = _generate_trial(generator) t.state = state storage.create_new_trial(study_id, template_trial=t) trials = storage.get_all_trials(study_id, states=None) assert len(trials) == 3 trials = storage.get_all_trials(study_id, states=(TrialState.COMPLETE,)) assert len(trials) == 2 assert all(t.state == TrialState.COMPLETE for t in trials) trials = storage.get_all_trials(study_id, states=(TrialState.COMPLETE, TrialState.PRUNED)) assert len(trials) == 3 assert all(t.state in (TrialState.COMPLETE, TrialState.PRUNED) for t in trials) trials = storage.get_all_trials(study_id, states=()) assert len(trials) == 0 other_states = [ s for s in ALL_STATES if s != TrialState.COMPLETE and s != TrialState.PRUNED ] for state in other_states: trials = storage.get_all_trials(study_id, states=(state,)) assert len(trials) == 0 @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_get_all_trials_not_modified(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: _, study_to_trials = _setup_studies(storage, n_study=2, n_trial=20, seed=48) for study_id in study_to_trials.keys(): trials = storage.get_all_trials(study_id, deepcopy=False) deepcopied_trials = copy.deepcopy(trials) for trial in trials: if not trial.state.is_finished(): storage.set_trial_param(trial._trial_id, "paramX", 0, FloatDistribution(0, 1)) storage.set_trial_user_attr(trial._trial_id, "usr_attrX", 0) storage.set_trial_system_attr(trial._trial_id, "sys_attrX", 0) if trial.state == TrialState.RUNNING: if trial.number % 3 == 0: storage.set_trial_state_values(trial._trial_id, TrialState.COMPLETE, [0]) elif trial.number % 3 == 1: storage.set_trial_intermediate_value(trial._trial_id, 0, 0) storage.set_trial_state_values(trial._trial_id, TrialState.PRUNED, [0]) else: storage.set_trial_state_values(trial._trial_id, TrialState.FAIL) elif trial.state == TrialState.WAITING: storage.set_trial_state_values(trial._trial_id, TrialState.RUNNING) assert trials == deepcopied_trials @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_get_n_trials(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study_id_to_frozen_studies, _ = _setup_studies(storage, n_study=2, n_trial=7, seed=50) for study_id in study_id_to_frozen_studies: assert storage.get_n_trials(study_id) == 7 non_existent_study_id = max(study_id_to_frozen_studies.keys()) + 1 with pytest.raises(KeyError): assert storage.get_n_trials(non_existent_study_id) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_get_n_trials_state_option(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study_id = storage.create_new_study(directions=(StudyDirection.MAXIMIZE,)) generator = random.Random(51) states = [ TrialState.COMPLETE, TrialState.COMPLETE, TrialState.PRUNED, ] for s in states: t = _generate_trial(generator) t.state = s storage.create_new_trial(study_id, template_trial=t) assert storage.get_n_trials(study_id, TrialState.COMPLETE) == 2 assert storage.get_n_trials(study_id, TrialState.PRUNED) == 1 other_states = [ s for s in ALL_STATES if s != TrialState.COMPLETE and s != TrialState.PRUNED ] for s in other_states: assert storage.get_n_trials(study_id, s) == 0 @pytest.mark.parametrize("storage_mode", STORAGE_MODES) @pytest.mark.parametrize("direction", [StudyDirection.MAXIMIZE, StudyDirection.MINIMIZE]) @pytest.mark.parametrize( "values", [ [0.0, 1.0, 2.0], [0.0, float("inf"), 1.0], [0.0, float("-inf"), 1.0], [float("inf"), 0.0, 1.0, float("-inf")], [float("inf")], [float("-inf")], ], ) def test_get_best_trial(storage_mode: str, direction: StudyDirection, values: List[float]) -> None: with StorageSupplier(storage_mode) as storage: study_id = storage.create_new_study(directions=[direction]) with pytest.raises(ValueError): storage.get_best_trial(study_id) with pytest.raises(KeyError): storage.get_best_trial(study_id + 1) generator = random.Random(51) for v in values: template_trial = _generate_trial(generator) template_trial.state = TrialState.COMPLETE template_trial.value = v storage.create_new_trial(study_id, template_trial=template_trial) expected_value = max(values) if direction == StudyDirection.MAXIMIZE else min(values) assert storage.get_best_trial(study_id).value == expected_value def test_get_trials_included_trial_ids() -> None: storage_mode = "sqlite" with StorageSupplier(storage_mode) as storage: assert isinstance(storage, RDBStorage) study_id = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) trial_id = storage.create_new_trial(study_id) trial_id_greater_than = trial_id + 500000 trials = storage._get_trials( study_id, states=None, included_trial_ids=set(), trial_id_greater_than=trial_id_greater_than, ) assert len(trials) == 0 # A large exclusion list used to raise errors. Check that it is not an issue. # See https://github.com/optuna/optuna/issues/1457. trials = storage._get_trials( study_id, states=None, included_trial_ids=set(range(500000)), trial_id_greater_than=trial_id_greater_than, ) assert len(trials) == 1 def test_get_trials_trial_id_greater_than() -> None: storage_mode = "sqlite" with StorageSupplier(storage_mode) as storage: assert isinstance(storage, RDBStorage) study_id = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) storage.create_new_trial(study_id) trials = storage._get_trials( study_id, states=None, included_trial_ids=set(), trial_id_greater_than=-1 ) assert len(trials) == 1 trials = storage._get_trials( study_id, states=None, included_trial_ids=set(), trial_id_greater_than=500001 ) assert len(trials) == 0 def _setup_studies( storage: BaseStorage, n_study: int, n_trial: int, seed: int, direction: Optional[StudyDirection] = None, ) -> Tuple[Dict[int, FrozenStudy], Dict[int, Dict[int, FrozenTrial]]]: generator = random.Random(seed) study_id_to_frozen_study: Dict[int, FrozenStudy] = {} study_id_to_trials: Dict[int, Dict[int, FrozenTrial]] = {} for i in range(n_study): study_name = "test-study-name-{}".format(i) if direction is None: direction = generator.choice([StudyDirection.MINIMIZE, StudyDirection.MAXIMIZE]) study_id = storage.create_new_study(directions=(direction,), study_name=study_name) storage.set_study_user_attr(study_id, "u", i) storage.set_study_system_attr(study_id, "s", i) trials = {} for j in range(n_trial): trial = _generate_trial(generator) trial.number = j trial._trial_id = storage.create_new_trial(study_id, trial) trials[trial._trial_id] = trial study_id_to_trials[study_id] = trials study_id_to_frozen_study[study_id] = FrozenStudy( study_name=study_name, direction=direction, user_attrs={"u": i}, system_attrs={"s": i}, study_id=study_id, ) return study_id_to_frozen_study, study_id_to_trials def _generate_trial(generator: random.Random) -> FrozenTrial: example_params = { "paramA": (generator.uniform(0, 1), FloatDistribution(0, 1)), "paramB": (generator.uniform(1, 2), FloatDistribution(1, 2, log=True)), "paramC": ( generator.choice(["CatA", "CatB", "CatC"]), CategoricalDistribution(("CatA", "CatB", "CatC")), ), "paramD": (generator.uniform(-3, 0), FloatDistribution(-3, 0)), "paramE": (generator.choice([0.1, 0.2]), CategoricalDistribution((0.1, 0.2))), } example_attrs = { "attrA": "valueA", "attrB": 1, "attrC": None, "attrD": {"baseline_score": 0.001, "tags": ["image", "classification"]}, } state = generator.choice(ALL_STATES) params = {} distributions = {} user_attrs = {} system_attrs: Dict[str, Any] = {} intermediate_values = {} for key, (value, dist) in example_params.items(): if generator.choice([True, False]): params[key] = value distributions[key] = dist for key, value in example_attrs.items(): if generator.choice([True, False]): user_attrs["usr_" + key] = value if generator.choice([True, False]): system_attrs["sys_" + key] = value for i in range(generator.randint(4, 10)): if generator.choice([True, False]): intermediate_values[i] = generator.uniform(-10, 10) return FrozenTrial( number=0, # dummy state=state, value=generator.uniform(-10, 10), datetime_start=datetime.now(), datetime_complete=datetime.now() if state.is_finished() else None, params=params, distributions=distributions, user_attrs=user_attrs, system_attrs=system_attrs, intermediate_values=intermediate_values, trial_id=0, # dummy ) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_get_best_trial_for_multi_objective_optimization(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study_id = storage.create_new_study( directions=(StudyDirection.MAXIMIZE, StudyDirection.MINIMIZE) ) generator = random.Random(51) for i in range(3): template_trial = _generate_trial(generator) template_trial.state = TrialState.COMPLETE template_trial.values = [i, i + 1] storage.create_new_trial(study_id, template_trial=template_trial) with pytest.raises(RuntimeError): storage.get_best_trial(study_id) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_get_trial_id_from_study_id_trial_number(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: with pytest.raises(KeyError): # Matching study does not exist. storage.get_trial_id_from_study_id_trial_number(study_id=0, trial_number=0) study_id = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) with pytest.raises(KeyError): # Matching trial does not exist. storage.get_trial_id_from_study_id_trial_number(study_id, trial_number=0) trial_id = storage.create_new_trial(study_id) assert trial_id == storage.get_trial_id_from_study_id_trial_number( study_id, trial_number=0 ) # Trial IDs are globally unique within a storage but numbers are only unique within a # study. Create a second study within the same storage. study_id = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) trial_id = storage.create_new_trial(study_id) assert trial_id == storage.get_trial_id_from_study_id_trial_number( study_id, trial_number=0 ) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_pickle_storage(storage_mode: str) -> None: if "redis" in storage_mode: pytest.skip("The `fakeredis` does not support multi instances.") with StorageSupplier(storage_mode) as storage: study_id = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) storage.set_study_system_attr(study_id, "key", "pickle") restored_storage = pickle.loads(pickle.dumps(storage)) storage_system_attrs = storage.get_study_system_attrs(study_id) restored_storage_system_attrs = restored_storage.get_study_system_attrs(study_id) assert storage_system_attrs == restored_storage_system_attrs == {"key": "pickle"} if isinstance(storage, RDBStorage): assert storage.url == restored_storage.url assert storage.engine_kwargs == restored_storage.engine_kwargs assert storage.skip_compatibility_check == restored_storage.skip_compatibility_check assert storage.engine != restored_storage.engine assert storage.scoped_session != restored_storage.scoped_session assert storage._version_manager != restored_storage._version_manager @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_check_trial_is_updatable(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study_id = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) trial_id = storage.create_new_trial(study_id) storage.check_trial_is_updatable(trial_id, TrialState.RUNNING) storage.check_trial_is_updatable(trial_id, TrialState.WAITING) with pytest.raises(RuntimeError): storage.check_trial_is_updatable(trial_id, TrialState.FAIL) with pytest.raises(RuntimeError): storage.check_trial_is_updatable(trial_id, TrialState.PRUNED) with pytest.raises(RuntimeError): storage.check_trial_is_updatable(trial_id, TrialState.COMPLETE) optuna-4.1.0/tests/storages_tests/test_with_server.py000066400000000000000000000147131471332314300232160ustar00rootroot00000000000000from concurrent.futures import ProcessPoolExecutor from concurrent.futures import ThreadPoolExecutor import os import pickle from typing import Sequence import numpy as np import pytest import optuna from optuna.storages import BaseStorage from optuna.storages.journal import JournalRedisBackend from optuna.study import StudyDirection from optuna.trial import TrialState _STUDY_NAME = "_test_multiprocess" def f(x: float, y: float) -> float: return (x - 3) ** 2 + y def objective(trial: optuna.Trial) -> float: x = trial.suggest_float("x", -10, 10) y = trial.suggest_float("y", -10, 10) trial.report(x, 0) trial.report(y, 1) trial.set_user_attr("x", x) return f(x, y) def get_storage() -> BaseStorage: if "TEST_DB_URL" not in os.environ: pytest.skip("This test requires TEST_DB_URL.") storage_url = os.environ["TEST_DB_URL"] storage_mode = os.environ.get("TEST_DB_MODE", "") storage: BaseStorage if storage_mode == "": storage = optuna.storages.RDBStorage(url=storage_url) elif storage_mode == "journal-redis": journal_redis_storage = JournalRedisBackend(storage_url) storage = optuna.storages.JournalStorage(journal_redis_storage) else: assert False, f"The mode {storage_mode} is not supported." return storage def run_optimize(study_name: str, n_trials: int) -> None: storage = get_storage() # Create a study study = optuna.load_study(study_name=study_name, storage=storage) # Run optimization study.optimize(objective, n_trials=n_trials) def _check_trials(trials: Sequence[optuna.trial.FrozenTrial]) -> None: # Check trial states. assert all(trial.state == TrialState.COMPLETE for trial in trials) # Check trial values and params. assert all("x" in trial.params for trial in trials) assert all("y" in trial.params for trial in trials) assert all( np.isclose( np.asarray([trial.value for trial in trials]), [f(trial.params["x"], trial.params["y"]) for trial in trials], atol=1e-4, ).tolist() ) # Check intermediate values. assert all(len(trial.intermediate_values) == 2 for trial in trials) assert all(trial.params["x"] == trial.intermediate_values[0] for trial in trials) assert all(trial.params["y"] == trial.intermediate_values[1] for trial in trials) # Check attrs. assert all( np.isclose( [trial.user_attrs["x"] for trial in trials], [trial.params["x"] for trial in trials], atol=1e-4, ).tolist() ) def test_loaded_trials() -> None: # Please create the tables by placing this function before the multi-process tests. storage = get_storage() try: optuna.delete_study(study_name=_STUDY_NAME, storage=storage) except KeyError: pass N_TRIALS = 20 study = optuna.create_study(study_name=_STUDY_NAME, storage=storage) # Run optimization study.optimize(objective, n_trials=N_TRIALS) trials = study.trials assert len(trials) == N_TRIALS _check_trials(trials) # Create a new study to confirm the study can load trial properly. loaded_study = optuna.load_study(study_name=_STUDY_NAME, storage=storage) _check_trials(loaded_study.trials) @pytest.mark.parametrize( "input_value,expected", [ (float("inf"), float("inf")), (-float("inf"), -float("inf")), ], ) def test_store_infinite_values(input_value: float, expected: float) -> None: storage = get_storage() study_id = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) trial_id = storage.create_new_trial(study_id) storage.set_trial_intermediate_value(trial_id, 1, input_value) storage.set_trial_state_values(trial_id, state=TrialState.COMPLETE, values=(input_value,)) assert storage.get_trial(trial_id).value == expected assert storage.get_trial(trial_id).intermediate_values[1] == expected def test_store_nan_intermediate_values() -> None: storage = get_storage() study_id = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) trial_id = storage.create_new_trial(study_id) value = float("nan") storage.set_trial_intermediate_value(trial_id, 1, value) got_value = storage.get_trial(trial_id).intermediate_values[1] assert np.isnan(got_value) def test_multithread_create_study() -> None: storage = get_storage() with ThreadPoolExecutor(10) as pool: for _ in range(10): pool.submit( optuna.create_study, storage=storage, study_name="test-multithread-create-study", load_if_exists=True, ) def test_multiprocess_run_optimize() -> None: n_workers = 8 n_trials = 20 storage = get_storage() try: optuna.delete_study(study_name=_STUDY_NAME, storage=storage) except KeyError: pass optuna.create_study(storage=storage, study_name=_STUDY_NAME) with ProcessPoolExecutor(n_workers) as pool: pool.map(run_optimize, *zip(*[[_STUDY_NAME, n_trials]] * n_workers)) study = optuna.load_study(study_name=_STUDY_NAME, storage=storage) trials = study.trials assert len(trials) == n_workers * n_trials _check_trials(trials) def test_pickle_storage() -> None: storage = get_storage() study_id = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) storage.set_study_system_attr(study_id, "key", "pickle") restored_storage = pickle.loads(pickle.dumps(storage)) storage_system_attrs = storage.get_study_system_attrs(study_id) restored_storage_system_attrs = restored_storage.get_study_system_attrs(study_id) assert storage_system_attrs == restored_storage_system_attrs == {"key": "pickle"} @pytest.mark.parametrize("direction", [StudyDirection.MAXIMIZE, StudyDirection.MINIMIZE]) @pytest.mark.parametrize( "values", [ [0.0, 1.0, 2.0], [0.0, float("inf"), 1.0], [0.0, float("-inf"), 1.0], [float("inf"), 0.0, 1.0, float("-inf")], [float("inf")], [float("-inf")], ], ) def test_get_best_trial(direction: StudyDirection, values: Sequence[float]) -> None: storage = get_storage() study = optuna.create_study(direction=direction, storage=storage) study.add_trials( [optuna.create_trial(params={}, distributions={}, value=value) for value in values] ) expected_value = max(values) if direction == StudyDirection.MAXIMIZE else min(values) assert study.best_value == expected_value optuna-4.1.0/tests/study_tests/000077500000000000000000000000001471332314300165575ustar00rootroot00000000000000optuna-4.1.0/tests/study_tests/__init__.py000066400000000000000000000000001471332314300206560ustar00rootroot00000000000000optuna-4.1.0/tests/study_tests/test_constrained_optimization.py000066400000000000000000000011001471332314300252770ustar00rootroot00000000000000from optuna.study._constrained_optimization import _CONSTRAINTS_KEY from optuna.study._constrained_optimization import _get_feasible_trials from optuna.trial import create_trial def test_get_feasible_trials() -> None: trials = [] trials.append(create_trial(value=0.0, system_attrs={_CONSTRAINTS_KEY: [0.0]})) trials.append(create_trial(value=0.0, system_attrs={_CONSTRAINTS_KEY: [1.0]})) trials.append(create_trial(value=0.0)) feasible_trials = _get_feasible_trials(trials) assert len(feasible_trials) == 1 assert feasible_trials[0] == trials[0] optuna-4.1.0/tests/study_tests/test_dataframe.py000066400000000000000000000172061471332314300221220ustar00rootroot00000000000000from __future__ import annotations import pandas as pd import pytest from optuna import create_study from optuna import create_trial from optuna import Trial from optuna.testing.storages import STORAGE_MODES from optuna.testing.storages import StorageSupplier from optuna.trial import TrialState def test_study_trials_dataframe_with_no_trials() -> None: study_with_no_trials = create_study() trials_df = study_with_no_trials.trials_dataframe() assert trials_df.empty @pytest.mark.parametrize("storage_mode", STORAGE_MODES) @pytest.mark.parametrize( "attrs", [ ( "number", "value", "datetime_start", "datetime_complete", "params", "user_attrs", "system_attrs", "state", ), ( "number", "value", "datetime_start", "datetime_complete", "duration", "params", "user_attrs", "system_attrs", "state", "intermediate_values", "_trial_id", "distributions", ), ], ) @pytest.mark.parametrize("multi_index", [True, False]) def test_trials_dataframe(storage_mode: str, attrs: tuple[str, ...], multi_index: bool) -> None: def f(trial: Trial) -> float: x = trial.suggest_int("x", 1, 1) y = trial.suggest_categorical("y", (2.5,)) trial.set_user_attr("train_loss", 3) trial.storage.set_trial_system_attr(trial._trial_id, "foo", "bar") value = x + y # 3.5 # Test reported intermediate values, although it in practice is not "intermediate". trial.report(value, step=0) return value with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) study.optimize(f, n_trials=3) df = study.trials_dataframe(attrs=attrs, multi_index=multi_index) # Change index to access rows via trial number. if multi_index: df.set_index(("number", ""), inplace=True, drop=False) else: df.set_index("number", inplace=True, drop=False) assert len(df) == 3 # Number columns are as follows (total of 13): # non-nested: 6 (number, value, state, datetime_start, datetime_complete, duration) # params: 2 # distributions: 2 # user_attrs: 1 # system_attrs: 1 # intermediate_values: 1 expected_n_columns = len(attrs) if "params" in attrs: expected_n_columns += 1 if "distributions" in attrs: expected_n_columns += 1 assert len(df.columns) == expected_n_columns for i in range(3): assert df.number[i] == i assert df.state[i] == "COMPLETE" assert df.value[i] == 3.5 assert isinstance(df.datetime_start[i], pd.Timestamp) assert isinstance(df.datetime_complete[i], pd.Timestamp) if multi_index: if "distributions" in attrs: assert ("distributions", "x") in df.columns assert ("distributions", "y") in df.columns if "_trial_id" in attrs: assert ("trial_id", "") in df.columns # trial_id depends on other tests. if "duration" in attrs: assert ("duration", "") in df.columns assert df.params.x[i] == 1 assert df.params.y[i] == 2.5 assert df.user_attrs.train_loss[i] == 3 assert df.system_attrs.foo[i] == "bar" else: if "distributions" in attrs: assert "distributions_x" in df.columns assert "distributions_y" in df.columns if "_trial_id" in attrs: assert "trial_id" in df.columns # trial_id depends on other tests. if "duration" in attrs: assert "duration" in df.columns assert df.params_x[i] == 1 assert df.params_y[i] == 2.5 assert df.user_attrs_train_loss[i] == 3 assert df.system_attrs_foo[i] == "bar" @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_trials_dataframe_with_failure(storage_mode: str) -> None: def f(trial: Trial) -> float: x = trial.suggest_int("x", 1, 1) y = trial.suggest_categorical("y", (2.5,)) trial.set_user_attr("train_loss", 3) raise ValueError() return x + y # 3.5 with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) study.optimize(f, n_trials=3, catch=(ValueError,)) df = study.trials_dataframe() # Change index to access rows via trial number. df.set_index("number", inplace=True, drop=False) assert len(df) == 3 # non-nested: 6, params: 2, user_attrs: 1 system_attrs: 0 assert len(df.columns) == 9 for i in range(3): assert df.number[i] == i assert df.state[i] == "FAIL" assert df.value[i] is None assert isinstance(df.datetime_start[i], pd.Timestamp) assert isinstance(df.datetime_complete[i], pd.Timestamp) assert isinstance(df.duration[i], pd.Timedelta) assert df.params_x[i] == 1 assert df.params_y[i] == 2.5 assert df.user_attrs_train_loss[i] == 3 @pytest.mark.parametrize("attrs", [("value",), ("values",)]) @pytest.mark.parametrize("multi_index", [True, False]) def test_trials_dataframe_with_multi_objective_optimization( attrs: tuple[str, ...], multi_index: bool ) -> None: def f(trial: Trial) -> tuple[float, float]: x = trial.suggest_float("x", 1, 1) y = trial.suggest_float("y", 2, 2) return x + y, x**2 + y**2 # 3, 5 # without set_metric_names() study = create_study(directions=["minimize", "maximize"]) study.optimize(f, n_trials=1) df = study.trials_dataframe(attrs=attrs, multi_index=multi_index) if multi_index: assert df.get("values")[0][0] == 3 assert df.get("values")[1][0] == 5 else: assert df.values_0[0] == 3 assert df.values_1[0] == 5 # with set_metric_names() study.set_metric_names(["v0", "v1"]) df = study.trials_dataframe(attrs=attrs, multi_index=multi_index) if multi_index: assert df.get("values")["v0"][0] == 3 assert df.get("values")["v1"][0] == 5 else: assert df.get("values_v0")[0] == 3 assert df.get("values_v1")[0] == 5 @pytest.mark.parametrize("attrs", [("value",), ("values",)]) @pytest.mark.parametrize("multi_index", [True, False]) def test_trials_dataframe_with_multi_objective_optimization_with_fail_and_pruned( attrs: tuple[str, ...], multi_index: bool ) -> None: study = create_study(directions=["minimize", "maximize"]) study.add_trial(create_trial(state=TrialState.FAIL)) study.add_trial(create_trial(state=TrialState.PRUNED)) df = study.trials_dataframe(attrs=attrs, multi_index=multi_index) # without set_metric_names() if multi_index: for i in range(2): assert df.get("values")[0][i] is None assert df.get("values")[1][i] is None else: for i in range(2): assert df.values_0[i] is None assert df.values_1[i] is None # with set_metric_names() study.set_metric_names(["v0", "v1"]) df = study.trials_dataframe(attrs=attrs, multi_index=multi_index) if multi_index: assert df.get("values")["v0"][0] is None assert df.get("values")["v1"][0] is None else: assert df.get("values_v0")[0] is None assert df.get("values_v1")[0] is None optuna-4.1.0/tests/study_tests/test_multi_objective.py000066400000000000000000000165321471332314300233630ustar00rootroot00000000000000from __future__ import annotations import numpy as np import pytest from optuna.study import StudyDirection from optuna.study._multi_objective import _dominates from optuna.study._multi_objective import _fast_non_domination_rank from optuna.study._multi_objective import _normalize_value from optuna.trial import create_trial from optuna.trial import TrialState @pytest.mark.parametrize( ("v1", "v2"), [(-1, 1), (-float("inf"), 0), (0, float("inf")), (-float("inf"), float("inf"))] ) def test_dominates_1d_not_equal(v1: float, v2: float) -> None: t1 = create_trial(values=[v1]) t2 = create_trial(values=[v2]) assert _dominates(t1, t2, [StudyDirection.MINIMIZE]) assert not _dominates(t2, t1, [StudyDirection.MINIMIZE]) assert _dominates(t2, t1, [StudyDirection.MAXIMIZE]) assert not _dominates(t1, t2, [StudyDirection.MAXIMIZE]) @pytest.mark.parametrize("v", [0, -float("inf"), float("inf")]) @pytest.mark.parametrize("direction", [StudyDirection.MINIMIZE, StudyDirection.MAXIMIZE]) def test_dominates_1d_equal(v: float, direction: StudyDirection) -> None: assert not _dominates(create_trial(values=[v]), create_trial(values=[v]), [direction]) def test_dominates_2d() -> None: directions = [StudyDirection.MINIMIZE, StudyDirection.MAXIMIZE] # Check all pairs of trials consisting of these values, i.e., # [-inf, -inf], [-inf, -1], [-inf, 1], [-inf, inf], [-1, -inf], ... # These values should be specified in ascending order. vals = [-float("inf"), -1, 1, float("inf")] # The following table illustrates an example of dominance relations. # "d" cells in the table dominates the "t" cell in (MINIMIZE, MAXIMIZE) setting. # # value1 # ╔═════╤═════╤═════╤═════╤═════╗ # ║ │ -∞ │ -1 │ 1 │ ∞ ║ # ╟─────┼─────┼─────┼─────┼─────╢ # ║ -∞ │ │ │ d │ d ║ # ╟─────┼─────┼─────┼─────┼─────╢ # ║ -1 │ │ │ d │ d ║ # value0 ╟─────┼─────┼─────┼─────┼─────╢ # ║ 1 │ │ │ t │ d ║ # ╟─────┼─────┼─────┼─────┼─────╢ # ║ ∞ │ │ │ │ ║ # ╚═════╧═════╧═════╧═════╧═════╝ # # In the following code, we check that for each position of "t" cell, the relation # above holds. # Generate the set of all possible indices. all_indices = set((i, j) for i in range(len(vals)) for j in range(len(vals))) for t_i, t_j in all_indices: # Generate the set of all indices that dominates the current index. dominating_indices = set( (d_i, d_j) for d_i in range(t_i + 1) for d_j in range(t_j, len(vals)) ) dominating_indices -= {(t_i, t_j)} for d_i, d_j in dominating_indices: trial1 = create_trial(values=[vals[t_i], vals[t_j]]) trial2 = create_trial(values=[vals[d_i], vals[d_j]]) assert _dominates(trial2, trial1, directions) for d_i, d_j in all_indices - dominating_indices: trial1 = create_trial(values=[vals[t_i], vals[t_j]]) trial2 = create_trial(values=[vals[d_i], vals[d_j]]) assert not _dominates(trial2, trial1, directions) def test_dominates_invalid() -> None: directions = [StudyDirection.MINIMIZE, StudyDirection.MAXIMIZE] # The numbers of objectives for `t1` and `t2` don't match. t1 = create_trial(values=[1]) # One objective. t2 = create_trial(values=[1, 2]) # Two objectives. with pytest.raises(ValueError): _dominates(t1, t2, directions) # The numbers of objectives and directions don't match. t1 = create_trial(values=[1]) # One objective. t2 = create_trial(values=[1]) # One objective. with pytest.raises(ValueError): _dominates(t1, t2, directions) @pytest.mark.parametrize("t1_state", [TrialState.FAIL, TrialState.WAITING, TrialState.PRUNED]) @pytest.mark.parametrize("t2_state", [TrialState.FAIL, TrialState.WAITING, TrialState.PRUNED]) def test_dominates_incomplete_vs_incomplete(t1_state: TrialState, t2_state: TrialState) -> None: directions = [StudyDirection.MINIMIZE, StudyDirection.MAXIMIZE] t1 = create_trial(values=None, state=t1_state) t2 = create_trial(values=None, state=t2_state) assert not _dominates(t2, t1, list(directions)) assert not _dominates(t1, t2, list(directions)) @pytest.mark.parametrize("t1_state", [TrialState.FAIL, TrialState.WAITING, TrialState.PRUNED]) def test_dominates_complete_vs_incomplete(t1_state: TrialState) -> None: directions = [StudyDirection.MINIMIZE, StudyDirection.MAXIMIZE] t1 = create_trial(values=None, state=t1_state) t2 = create_trial(values=[1, 1], state=TrialState.COMPLETE) assert _dominates(t2, t1, list(directions)) assert not _dominates(t1, t2, list(directions)) @pytest.mark.parametrize( ("trial_values", "trial_ranks"), [ ([[10], [20], [20], [30]], [0, 1, 1, 2]), # Single objective ([[10, 30], [10, 10], [20, 20], [30, 10], [15, 15]], [1, 0, 2, 1, 1]), # Two objectives ( [[5, 5, 4], [5, 5, 5], [9, 9, 0], [5, 7, 5], [0, 0, 9], [0, 9, 9]], [0, 1, 0, 2, 0, 1], ), # Three objectives ( [[-5, -5, -4], [-5, -5, 5], [-9, -9, 0], [5, 7, 5], [0, 0, -9], [0, -9, 9]], [0, 1, 0, 2, 0, 1], ), # Negative values are included. ( [[1, 1], [1, float("inf")], [float("inf"), 1], [float("inf"), float("inf")]], [0, 1, 1, 2], ), # +infs are included. ( [[1, 1], [1, -float("inf")], [-float("inf"), 1], [-float("inf"), -float("inf")]], [2, 1, 1, 0], ), # -infs are included. ( [[1, 1], [1, 1], [1, 2], [2, 1], [0, 1.5], [1.5, 0], [0, 1.5]], [0, 0, 1, 1, 0, 0, 0], ), # Two objectives with duplicate values are included. ( [[1, 1], [1, 1], [1, 2], [2, 1], [1, 1], [0, 1.5], [0, 1.5]], [0, 0, 1, 1, 0, 0, 0], ), # Two objectives with duplicate values are included. ( [[1, 1, 1], [1, 1, 1], [1, 1, 2], [1, 2, 1], [2, 1, 1], [0, 1.5, 1.5], [0, 1.5, 1.5]], [0, 0, 1, 1, 1, 0, 0], ), # Three objectives with duplicate values are included. ], ) def test_fast_non_domination_rank(trial_values: list[float], trial_ranks: list[int]) -> None: ranks = list(_fast_non_domination_rank(np.array(trial_values))) assert np.array_equal(ranks, trial_ranks) def test_fast_non_domination_rank_invalid() -> None: with pytest.raises(ValueError): _fast_non_domination_rank( np.array([[1.0, 2.0], [3.0, 4.0]]), penalty=np.array([1.0, 2.0, 3.0]) ) def test_normalize_value() -> None: assert _normalize_value(1.0, StudyDirection.MINIMIZE) == 1.0 assert _normalize_value(1.0, StudyDirection.MAXIMIZE) == -1.0 assert _normalize_value(None, StudyDirection.MINIMIZE) == float("inf") assert _normalize_value(None, StudyDirection.MAXIMIZE) == float("inf") optuna-4.1.0/tests/study_tests/test_optimize.py000066400000000000000000000141351471332314300220340ustar00rootroot00000000000000from typing import Callable from typing import Generator from typing import Optional from unittest import mock from _pytest.logging import LogCaptureFixture import pytest from optuna import create_study from optuna import logging from optuna import Trial from optuna import TrialPruned from optuna.study import _optimize from optuna.study._tell import _tell_with_warning from optuna.study._tell import STUDY_TELL_WARNING_KEY from optuna.testing.objectives import fail_objective from optuna.testing.storages import STORAGE_MODES from optuna.testing.storages import StorageSupplier from optuna.trial import TrialState @pytest.fixture(autouse=True) def logging_setup() -> Generator[None, None, None]: # We need to reconstruct our default handler to properly capture stderr. logging._reset_library_root_logger() logging.enable_default_handler() logging.set_verbosity(logging.INFO) logging.enable_propagation() yield # After testing, restore default propagation setting. logging.disable_propagation() @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_run_trial(storage_mode: str, caplog: LogCaptureFixture) -> None: with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) caplog.clear() frozen_trial = _optimize._run_trial(study, lambda _: 1.0, catch=()) assert frozen_trial.state == TrialState.COMPLETE assert frozen_trial.value == 1.0 assert "Trial 0 finished with value: 1.0 and parameters" in caplog.text caplog.clear() frozen_trial = _optimize._run_trial(study, lambda _: float("inf"), catch=()) assert frozen_trial.state == TrialState.COMPLETE assert frozen_trial.value == float("inf") assert "Trial 1 finished with value: inf and parameters" in caplog.text caplog.clear() frozen_trial = _optimize._run_trial(study, lambda _: -float("inf"), catch=()) assert frozen_trial.state == TrialState.COMPLETE assert frozen_trial.value == -float("inf") assert "Trial 2 finished with value: -inf and parameters" in caplog.text @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_run_trial_automatically_fail(storage_mode: str, caplog: LogCaptureFixture) -> None: with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) frozen_trial = _optimize._run_trial(study, lambda _: float("nan"), catch=()) assert frozen_trial.state == TrialState.FAIL assert frozen_trial.value is None frozen_trial = _optimize._run_trial(study, lambda _: None, catch=()) # type: ignore[arg-type,return-value] # noqa: E501 assert frozen_trial.state == TrialState.FAIL assert frozen_trial.value is None frozen_trial = _optimize._run_trial(study, lambda _: object(), catch=()) # type: ignore[arg-type,return-value] # noqa: E501 assert frozen_trial.state == TrialState.FAIL assert frozen_trial.value is None frozen_trial = _optimize._run_trial(study, lambda _: [0, 1], catch=()) assert frozen_trial.state == TrialState.FAIL assert frozen_trial.value is None @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_run_trial_pruned(storage_mode: str, caplog: LogCaptureFixture) -> None: def gen_func(intermediate: Optional[float] = None) -> Callable[[Trial], float]: def func(trial: Trial) -> float: if intermediate is not None: trial.report(step=1, value=intermediate) raise TrialPruned return func with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) caplog.clear() frozen_trial = _optimize._run_trial(study, gen_func(), catch=()) assert frozen_trial.state == TrialState.PRUNED assert frozen_trial.value is None assert "Trial 0 pruned." in caplog.text caplog.clear() frozen_trial = _optimize._run_trial(study, gen_func(intermediate=1), catch=()) assert frozen_trial.state == TrialState.PRUNED assert frozen_trial.value == 1 assert "Trial 1 pruned." in caplog.text caplog.clear() frozen_trial = _optimize._run_trial(study, gen_func(intermediate=float("nan")), catch=()) assert frozen_trial.state == TrialState.PRUNED assert frozen_trial.value is None assert "Trial 2 pruned." in caplog.text @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_run_trial_catch_exception(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) frozen_trial = _optimize._run_trial(study, fail_objective, catch=(ValueError,)) assert frozen_trial.state == TrialState.FAIL assert STUDY_TELL_WARNING_KEY not in frozen_trial.system_attrs @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_run_trial_exception(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) with pytest.raises(ValueError): _optimize._run_trial(study, fail_objective, ()) # Test trial with unacceptable exception. with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) with pytest.raises(ValueError): _optimize._run_trial(study, fail_objective, (ArithmeticError,)) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_run_trial_invoke_tell_with_suppressing_warning(storage_mode: str) -> None: def func_numerical(trial: Trial) -> float: return trial.suggest_float("v", 0, 10) with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) with mock.patch( "optuna.study._optimize._tell_with_warning", side_effect=_tell_with_warning ) as mock_obj: _optimize._run_trial(study, func_numerical, ()) mock_obj.assert_called_once_with( study=mock.ANY, trial=mock.ANY, value_or_values=mock.ANY, state=mock.ANY, suppress_warning=True, ) optuna-4.1.0/tests/study_tests/test_study.py000066400000000000000000001626721471332314300213560ustar00rootroot00000000000000from __future__ import annotations from concurrent.futures import as_completed from concurrent.futures import ThreadPoolExecutor import copy import multiprocessing import pickle import platform import threading import time from typing import Any from typing import Callable from unittest.mock import Mock from unittest.mock import patch import uuid import warnings import _pytest.capture import pytest from optuna import copy_study from optuna import create_study from optuna import create_trial from optuna import delete_study from optuna import distributions from optuna import get_all_study_names from optuna import get_all_study_summaries from optuna import load_study from optuna import logging from optuna import Study from optuna import Trial from optuna import TrialPruned from optuna.exceptions import DuplicatedStudyError from optuna.exceptions import ExperimentalWarning from optuna.study import StudyDirection from optuna.study._constrained_optimization import _CONSTRAINTS_KEY from optuna.study.study import _SYSTEM_ATTR_METRIC_NAMES from optuna.testing.objectives import fail_objective from optuna.testing.storages import STORAGE_MODES from optuna.testing.storages import StorageSupplier from optuna.trial import FrozenTrial from optuna.trial import TrialState CallbackFuncType = Callable[[Study, FrozenTrial], None] def func(trial: Trial) -> float: x = trial.suggest_float("x", -10.0, 10.0) y = trial.suggest_float("y", 20, 30, log=True) z = trial.suggest_categorical("z", (-1.0, 1.0)) return (x - 2) ** 2 + (y - 25) ** 2 + z class Func: def __init__(self, sleep_sec: float | None = None) -> None: self.n_calls = 0 self.sleep_sec = sleep_sec self.lock = threading.Lock() def __call__(self, trial: Trial) -> float: with self.lock: self.n_calls += 1 # Sleep for testing parallelism. if self.sleep_sec is not None: time.sleep(self.sleep_sec) value = func(trial) check_params(trial.params) return value def check_params(params: dict[str, Any]) -> None: assert sorted(params.keys()) == ["x", "y", "z"] def check_value(value: float | None) -> None: assert isinstance(value, float) assert -1.0 <= value <= 12.0**2 + 5.0**2 + 1.0 def check_frozen_trial(frozen_trial: FrozenTrial) -> None: if frozen_trial.state == TrialState.COMPLETE: check_params(frozen_trial.params) check_value(frozen_trial.value) def check_study(study: Study) -> None: for trial in study.trials: check_frozen_trial(trial) assert not study._is_multi_objective() complete_trials = study.get_trials(deepcopy=False, states=(TrialState.COMPLETE,)) if len(complete_trials) == 0: with pytest.raises(ValueError): study.best_params with pytest.raises(ValueError): study.best_value with pytest.raises(ValueError): study.best_trial else: check_params(study.best_params) check_value(study.best_value) check_frozen_trial(study.best_trial) def stop_objective(threshold_number: int) -> Callable[[Trial], float]: def objective(trial: Trial) -> float: if trial.number >= threshold_number: trial.study.stop() return trial.number return objective def test_optimize_trivial_in_memory_new() -> None: study = create_study() study.optimize(func, n_trials=10) check_study(study) def test_optimize_trivial_in_memory_resume() -> None: study = create_study() study.optimize(func, n_trials=10) study.optimize(func, n_trials=10) check_study(study) def test_optimize_trivial_rdb_resume_study() -> None: study = create_study(storage="sqlite:///:memory:") study.optimize(func, n_trials=10) check_study(study) def test_optimize_with_direction() -> None: study = create_study(direction="minimize") study.optimize(func, n_trials=10) assert study.direction == StudyDirection.MINIMIZE check_study(study) study = create_study(direction="maximize") study.optimize(func, n_trials=10) assert study.direction == StudyDirection.MAXIMIZE check_study(study) with pytest.raises(ValueError): create_study(direction="test") @pytest.mark.parametrize("n_trials", (0, 1, 20)) @pytest.mark.parametrize("n_jobs", (1, 2, -1)) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_optimize_parallel(n_trials: int, n_jobs: int, storage_mode: str) -> None: f = Func() with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) study.optimize(f, n_trials=n_trials, n_jobs=n_jobs) assert f.n_calls == len(study.trials) == n_trials check_study(study) def test_optimize_with_thread_pool_executor() -> None: def objective(t: Trial) -> float: return t.suggest_float("x", -10, 10) study = create_study() with ThreadPoolExecutor(max_workers=5) as pool: for _ in range(10): pool.submit(study.optimize, objective, n_trials=10) assert len(study.trials) == 100 @pytest.mark.parametrize("n_trials", (0, 1, 20, None)) @pytest.mark.parametrize("n_jobs", (1, 2, -1)) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_optimize_parallel_timeout(n_trials: int, n_jobs: int, storage_mode: str) -> None: sleep_sec = 0.1 timeout_sec = 1.0 f = Func(sleep_sec=sleep_sec) with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) study.optimize(f, n_trials=n_trials, n_jobs=n_jobs, timeout=timeout_sec) assert f.n_calls == len(study.trials) if n_trials is not None: assert f.n_calls <= n_trials # A thread can process at most (timeout_sec / sleep_sec + 1) trials. n_jobs_actual = n_jobs if n_jobs != -1 else multiprocessing.cpu_count() max_calls = (timeout_sec / sleep_sec + 1) * n_jobs_actual assert f.n_calls <= max_calls check_study(study) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_optimize_with_catch(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) # Test default exceptions. with pytest.raises(ValueError): study.optimize(fail_objective, n_trials=20) assert len(study.trials) == 1 assert all(trial.state == TrialState.FAIL for trial in study.trials) # Test acceptable exception. study.optimize(fail_objective, n_trials=20, catch=(ValueError,)) assert len(study.trials) == 21 assert all(trial.state == TrialState.FAIL for trial in study.trials) # Test trial with unacceptable exception. with pytest.raises(ValueError): study.optimize(fail_objective, n_trials=20, catch=(ArithmeticError,)) assert len(study.trials) == 22 assert all(trial.state == TrialState.FAIL for trial in study.trials) @pytest.mark.parametrize("catch", [ValueError, (ValueError,), [ValueError], {ValueError}]) def test_optimize_with_catch_valid_type(catch: Any) -> None: study = create_study() study.optimize(fail_objective, n_trials=20, catch=catch) @pytest.mark.parametrize("catch", [None, 1]) def test_optimize_with_catch_invalid_type(catch: Any) -> None: study = create_study() with pytest.raises(TypeError): study.optimize(fail_objective, n_trials=20, catch=catch) @pytest.mark.parametrize("n_jobs", (2, -1)) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_optimize_with_reseeding(n_jobs: int, storage_mode: str) -> None: f = Func() with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) sampler = study.sampler with patch.object(sampler, "reseed_rng", wraps=sampler.reseed_rng) as mock_object: study.optimize(f, n_trials=1, n_jobs=2) assert mock_object.call_count == 1 def test_call_another_study_optimize_in_optimize() -> None: def inner_objective(t: Trial) -> float: return t.suggest_float("x", -10, 10) def objective(t: Trial) -> float: inner_study = create_study() inner_study.enqueue_trial({"x": t.suggest_int("initial_point", -10, 10)}) inner_study.optimize(inner_objective, n_trials=10) return inner_study.best_value study = create_study() study.optimize(objective, n_trials=10) assert len(study.trials) == 10 @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_study_set_and_get_user_attrs(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) study.set_user_attr("dataset", "MNIST") assert study.user_attrs["dataset"] == "MNIST" @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_trial_set_and_get_user_attrs(storage_mode: str) -> None: def f(trial: Trial) -> float: trial.set_user_attr("train_accuracy", 1) assert trial.user_attrs["train_accuracy"] == 1 return 0.0 with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) study.optimize(f, n_trials=1) frozen_trial = study.trials[0] assert frozen_trial.user_attrs["train_accuracy"] == 1 @pytest.mark.parametrize("storage_mode", STORAGE_MODES) @pytest.mark.parametrize("include_best_trial", [True, False]) def test_get_all_study_summaries(storage_mode: str, include_best_trial: bool) -> None: with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) study.optimize(func, n_trials=5) summaries = get_all_study_summaries(study._storage, include_best_trial) summary = [s for s in summaries if s._study_id == study._study_id][0] assert summary.study_name == study.study_name assert summary.n_trials == 5 if include_best_trial: assert summary.best_trial is not None else: assert summary.best_trial is None @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_get_all_study_summaries_with_no_trials(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) summaries = get_all_study_summaries(study._storage) summary = [s for s in summaries if s._study_id == study._study_id][0] assert summary.study_name == study.study_name assert summary.n_trials == 0 assert summary.datetime_start is None @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_get_all_study_names(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: n_studies = 5 studies = [create_study(storage=storage) for _ in range(n_studies)] study_names = get_all_study_names(storage) assert len(study_names) == n_studies for study, study_name in zip(studies, study_names): assert study_name == study.study_name def test_study_pickle() -> None: study_1 = create_study() study_1.optimize(func, n_trials=10) check_study(study_1) assert len(study_1.trials) == 10 dumped_bytes = pickle.dumps(study_1) study_2 = pickle.loads(dumped_bytes) check_study(study_2) assert len(study_2.trials) == 10 study_2.optimize(func, n_trials=10) check_study(study_2) assert len(study_2.trials) == 20 @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_create_study(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: # Test creating a new study. study = create_study(storage=storage, load_if_exists=False) # Test `load_if_exists=True` with existing study. create_study(study_name=study.study_name, storage=storage, load_if_exists=True) with pytest.raises(DuplicatedStudyError): create_study(study_name=study.study_name, storage=storage, load_if_exists=False) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_load_study(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: if storage is None: # :class:`~optuna.storages.InMemoryStorage` can not be used with `load_study` function. return study_name = str(uuid.uuid4()) with pytest.raises(KeyError): # Test loading an unexisting study. load_study(study_name=study_name, storage=storage) # Create a new study. created_study = create_study(study_name=study_name, storage=storage) # Test loading an existing study. loaded_study = load_study(study_name=study_name, storage=storage) assert created_study._study_id == loaded_study._study_id @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_load_study_study_name_none(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: if storage is None: # :class:`~optuna.storages.InMemoryStorage` can not be used with `load_study` function. return study_name = str(uuid.uuid4()) _ = create_study(study_name=study_name, storage=storage) loaded_study = load_study(study_name=None, storage=storage) assert loaded_study.study_name == study_name study_name = str(uuid.uuid4()) _ = create_study(study_name=study_name, storage=storage) # Ambiguous study. with pytest.raises(ValueError): load_study(study_name=None, storage=storage) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_delete_study(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: # Test deleting a non-existing study. with pytest.raises(KeyError): delete_study(study_name="invalid-study-name", storage=storage) # Test deleting an existing study. study = create_study(storage=storage, load_if_exists=False) delete_study(study_name=study.study_name, storage=storage) # Test failed to delete the study which is already deleted. with pytest.raises(KeyError): delete_study(study_name=study.study_name, storage=storage) @pytest.mark.parametrize("from_storage_mode", STORAGE_MODES) @pytest.mark.parametrize("to_storage_mode", STORAGE_MODES) def test_copy_study(from_storage_mode: str, to_storage_mode: str) -> None: with StorageSupplier(from_storage_mode) as from_storage, StorageSupplier( to_storage_mode ) as to_storage: from_study = create_study(storage=from_storage, directions=["maximize", "minimize"]) from_study._storage.set_study_system_attr(from_study._study_id, "foo", "bar") from_study.set_user_attr("baz", "qux") from_study.optimize( lambda t: (t.suggest_float("x0", 0, 1), t.suggest_float("x1", 0, 1)), n_trials=3 ) copy_study( from_study_name=from_study.study_name, from_storage=from_storage, to_storage=to_storage, ) to_study = load_study(study_name=from_study.study_name, storage=to_storage) assert to_study.study_name == from_study.study_name assert to_study.directions == from_study.directions to_study_system_attrs = to_study._storage.get_study_system_attrs(to_study._study_id) from_study_system_attrs = from_study._storage.get_study_system_attrs(from_study._study_id) assert to_study_system_attrs == from_study_system_attrs assert to_study.user_attrs == from_study.user_attrs assert len(to_study.trials) == len(from_study.trials) @pytest.mark.parametrize("from_storage_mode", STORAGE_MODES) @pytest.mark.parametrize("to_storage_mode", STORAGE_MODES) def test_copy_study_to_study_name(from_storage_mode: str, to_storage_mode: str) -> None: with StorageSupplier(from_storage_mode) as from_storage, StorageSupplier( to_storage_mode ) as to_storage: from_study = create_study(study_name="foo", storage=from_storage) _ = create_study(study_name="foo", storage=to_storage) with pytest.raises(DuplicatedStudyError): copy_study( from_study_name=from_study.study_name, from_storage=from_storage, to_storage=to_storage, ) copy_study( from_study_name=from_study.study_name, from_storage=from_storage, to_storage=to_storage, to_study_name="bar", ) _ = load_study(study_name="bar", storage=to_storage) def test_nested_optimization() -> None: def objective(trial: Trial) -> float: with pytest.raises(RuntimeError): trial.study.optimize(lambda _: 0.0, n_trials=1) return 1.0 study = create_study() study.optimize(objective, n_trials=10, catch=()) def test_stop_in_objective() -> None: # Test stopping the optimization: it should stop once the trial number reaches 4. study = create_study() study.optimize(stop_objective(4), n_trials=10) assert len(study.trials) == 5 # Test calling `optimize` again: it should stop once the trial number reaches 11. study.optimize(stop_objective(11), n_trials=10) assert len(study.trials) == 12 def test_stop_in_callback() -> None: def callback(study: Study, trial: FrozenTrial) -> None: if trial.number >= 4: study.stop() # Test stopping the optimization inside a callback. study = create_study() study.optimize(lambda _: 1.0, n_trials=10, callbacks=[callback]) assert len(study.trials) == 5 def test_stop_n_jobs() -> None: def callback(study: Study, trial: FrozenTrial) -> None: if trial.number >= 4: study.stop() study = create_study() study.optimize(lambda _: 1.0, n_trials=None, callbacks=[callback], n_jobs=2) assert 5 <= len(study.trials) <= 6 def test_stop_outside_optimize() -> None: # Test stopping outside the optimization: it should raise `RuntimeError`. study = create_study() with pytest.raises(RuntimeError): study.stop() # Test calling `optimize` after the `RuntimeError` is caught. study.optimize(lambda _: 1.0, n_trials=1) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_add_trial(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) assert len(study.trials) == 0 trial = create_trial(value=0.8) study.add_trial(trial) assert len(study.trials) == 1 assert study.trials[0].number == 0 assert study.best_value == 0.8 def test_add_trial_invalid_values_length() -> None: study = create_study() trial = create_trial(values=[0, 0]) with pytest.raises(ValueError): study.add_trial(trial) study = create_study(directions=["minimize", "minimize"]) trial = create_trial(value=0) with pytest.raises(ValueError): study.add_trial(trial) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_add_trials(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) assert len(study.trials) == 0 study.add_trials([]) assert len(study.trials) == 0 trials = [create_trial(value=i) for i in range(3)] study.add_trials(trials) assert len(study.trials) == 3 for i, trial in enumerate(study.trials): assert trial.number == i assert trial.value == i other_study = create_study(storage=storage) other_study.add_trials(study.trials) assert len(other_study.trials) == 3 for i, trial in enumerate(other_study.trials): assert trial.number == i assert trial.value == i @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_enqueue_trial_properly_sets_param_values(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) assert len(study.trials) == 0 study.enqueue_trial(params={"x": -5, "y": 5}) study.enqueue_trial(params={"x": -1, "y": 0}) def objective(trial: Trial) -> float: x = trial.suggest_int("x", -10, 10) y = trial.suggest_int("y", -10, 10) return x**2 + y**2 study.optimize(objective, n_trials=2) t0 = study.trials[0] assert t0.params["x"] == -5 assert t0.params["y"] == 5 t1 = study.trials[1] assert t1.params["x"] == -1 assert t1.params["y"] == 0 @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_enqueue_trial_with_unfixed_parameters(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) assert len(study.trials) == 0 study.enqueue_trial(params={"x": -5}) def objective(trial: Trial) -> float: x = trial.suggest_int("x", -10, 10) y = trial.suggest_int("y", -10, 10) return x**2 + y**2 study.optimize(objective, n_trials=1) t = study.trials[0] assert t.params["x"] == -5 assert -10 <= t.params["y"] <= 10 @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_enqueue_trial_properly_sets_user_attr(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) assert len(study.trials) == 0 study.enqueue_trial(params={"x": -5, "y": 5}, user_attrs={"is_optimal": False}) study.enqueue_trial(params={"x": 0, "y": 0}, user_attrs={"is_optimal": True}) def objective(trial: Trial) -> float: x = trial.suggest_int("x", -10, 10) y = trial.suggest_int("y", -10, 10) return x**2 + y**2 study.optimize(objective, n_trials=2) t0 = study.trials[0] assert t0.user_attrs == {"is_optimal": False} t1 = study.trials[1] assert t1.user_attrs == {"is_optimal": True} @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_enqueue_trial_with_non_dict_parameters(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) assert len(study.trials) == 0 with pytest.raises(TypeError): study.enqueue_trial(params=[17, 12]) # type: ignore[arg-type] @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_enqueue_trial_with_out_of_range_parameters(storage_mode: str) -> None: fixed_value = 11 with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) assert len(study.trials) == 0 study.enqueue_trial(params={"x": fixed_value}) def objective(trial: Trial) -> float: return trial.suggest_int("x", -10, 10) with pytest.warns(UserWarning): study.optimize(objective, n_trials=1) t = study.trials[0] assert t.params["x"] == fixed_value # Internal logic might differ when distribution contains a single element. # Test it explicitly. with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) assert len(study.trials) == 0 study.enqueue_trial(params={"x": fixed_value}) def objective(trial: Trial) -> float: return trial.suggest_int("x", 1, 1) # Single element. with pytest.warns(UserWarning): study.optimize(objective, n_trials=1) t = study.trials[0] assert t.params["x"] == fixed_value @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_enqueue_trial_skips_existing_finished(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) assert len(study.trials) == 0 def objective(trial: Trial) -> float: x = trial.suggest_int("x", -10, 10) y = trial.suggest_int("y", -10, 10) return x**2 + y**2 study.enqueue_trial({"x": -5, "y": 5}) study.optimize(objective, n_trials=1) t0 = study.trials[0] assert t0.params["x"] == -5 assert t0.params["y"] == 5 before_enqueue = len(study.trials) study.enqueue_trial({"x": -5, "y": 5}, skip_if_exists=True) after_enqueue = len(study.trials) assert before_enqueue == after_enqueue @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_enqueue_trial_skips_existing_waiting(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) assert len(study.trials) == 0 def objective(trial: Trial) -> float: x = trial.suggest_int("x", -10, 10) y = trial.suggest_int("y", -10, 10) return x**2 + y**2 study.enqueue_trial({"x": -5, "y": 5}) before_enqueue = len(study.trials) study.enqueue_trial({"x": -5, "y": 5}, skip_if_exists=True) after_enqueue = len(study.trials) assert before_enqueue == after_enqueue study.optimize(objective, n_trials=1) t0 = study.trials[0] assert t0.params["x"] == -5 assert t0.params["y"] == 5 @pytest.mark.parametrize("storage_mode", STORAGE_MODES) @pytest.mark.parametrize( "new_params", [{"x": -5, "y": 5, "z": 5}, {"x": -5}, {"x": -5, "z": 5}, {"x": -5, "y": 6}] ) def test_enqueue_trial_skip_existing_allows_unfixed( storage_mode: str, new_params: dict[str, int] ) -> None: with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) assert len(study.trials) == 0 def objective(trial: Trial) -> float: x = trial.suggest_int("x", -10, 10) y = trial.suggest_int("y", -10, 10) if trial.number == 1: z = trial.suggest_int("z", -10, 10) return x**2 + y**2 + z**2 return x**2 + y**2 study.enqueue_trial({"x": -5, "y": 5}) study.optimize(objective, n_trials=1) t0 = study.trials[0] assert t0.params["x"] == -5 assert t0.params["y"] == 5 study.enqueue_trial(new_params, skip_if_exists=True) study.optimize(objective, n_trials=1) unfixed_params = {"x", "y", "z"} - set(new_params) t1 = study.trials[1] assert all(t1.params[k] == new_params[k] for k in new_params) assert all(-10 <= t1.params[k] <= 10 for k in unfixed_params) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) @pytest.mark.parametrize( "param", ["foo", 1, 1.1, 1e17, 1e-17, float("inf"), float("-inf"), float("nan"), None] ) def test_enqueue_trial_skip_existing_handles_common_types(storage_mode: str, param: Any) -> None: with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) study.enqueue_trial({"x": param}) before_enqueue = len(study.trials) study.enqueue_trial({"x": param}, skip_if_exists=True) after_enqueue = len(study.trials) assert before_enqueue == after_enqueue @patch("optuna.study._optimize.gc.collect") def test_optimize_with_gc(collect_mock: Mock) -> None: study = create_study() study.optimize(func, n_trials=10, gc_after_trial=True) check_study(study) assert collect_mock.call_count == 10 @patch("optuna.study._optimize.gc.collect") def test_optimize_without_gc(collect_mock: Mock) -> None: study = create_study() study.optimize(func, n_trials=10, gc_after_trial=False) check_study(study) assert collect_mock.call_count == 0 @pytest.mark.parametrize("n_jobs", [1, 2]) def test_optimize_with_progbar(n_jobs: int, capsys: _pytest.capture.CaptureFixture) -> None: study = create_study() study.optimize(lambda _: 1.0, n_trials=10, n_jobs=n_jobs, show_progress_bar=True) _, err = capsys.readouterr() # Search for progress bar elements in stderr. assert "Best trial: 0" in err assert "Best value: 1" in err assert "10/10" in err if platform.system() != "Windows": # Skip this assertion because the progress bar sometimes stops at 99% on Windows. assert "100%" in err @pytest.mark.parametrize("n_jobs", [1, 2]) def test_optimize_without_progbar(n_jobs: int, capsys: _pytest.capture.CaptureFixture) -> None: study = create_study() study.optimize(lambda _: 1.0, n_trials=10, n_jobs=n_jobs) _, err = capsys.readouterr() assert "Best trial: 0" not in err assert "Best value: 1" not in err assert "10/10" not in err if platform.system() != "Windows": # Skip this assertion because the progress bar sometimes stops at 99% on Windows. assert "100%" not in err def test_optimize_with_progbar_timeout(capsys: _pytest.capture.CaptureFixture) -> None: study = create_study() study.optimize(lambda _: 1.0, timeout=2.0, show_progress_bar=True) _, err = capsys.readouterr() assert "Best trial: 0" in err assert "Best value: 1" in err assert "00:02/00:02" in err if platform.system() != "Windows": # Skip this assertion because the progress bar sometimes stops at 99% on Windows. assert "100%" in err def test_optimize_with_progbar_parallel_timeout(capsys: _pytest.capture.CaptureFixture) -> None: study = create_study() with pytest.warns( UserWarning, match="The timeout-based progress bar is not supported with n_jobs != 1." ): study.optimize(lambda _: 1.0, timeout=2.0, show_progress_bar=True, n_jobs=2) _, err = capsys.readouterr() # Testing for a character that forms progress bar borders. assert "|" not in err @pytest.mark.parametrize( "timeout,expected", [ (59.0, "/00:59"), (60.0, "/01:00"), (60.0 * 60, "/1:00:00"), (60.0 * 60 * 24, "/24:00:00"), (60.0 * 60 * 24 * 10, "/240:00:00"), ], ) def test_optimize_with_progbar_timeout_formats( timeout: float, expected: str, capsys: _pytest.capture.CaptureFixture ) -> None: study = create_study() study.optimize(stop_objective(5), timeout=timeout, show_progress_bar=True) _, err = capsys.readouterr() assert expected in err @pytest.mark.parametrize("n_jobs", [1, 2]) def test_optimize_without_progbar_timeout( n_jobs: int, capsys: _pytest.capture.CaptureFixture ) -> None: study = create_study() study.optimize(lambda _: 1.0, timeout=2.0, n_jobs=n_jobs) _, err = capsys.readouterr() assert "Best trial: 0" not in err assert "Best value: 1.0" not in err assert "00:02/00:02" not in err if platform.system() != "Windows": # Skip this assertion because the progress bar sometimes stops at 99% on Windows. assert "100%" not in err @pytest.mark.parametrize("n_jobs", [1, 2]) def test_optimize_progbar_n_trials_prioritized( n_jobs: int, capsys: _pytest.capture.CaptureFixture ) -> None: study = create_study() study.optimize(lambda _: 1.0, n_trials=10, n_jobs=n_jobs, timeout=10.0, show_progress_bar=True) _, err = capsys.readouterr() assert "Best trial: 0" in err assert "Best value: 1" in err assert "10/10" in err if platform.system() != "Windows": # Skip this assertion because the progress bar sometimes stops at 99% on Windows. assert "100%" in err assert "it" in err @pytest.mark.parametrize("n_jobs", [1, 2]) def test_optimize_without_progbar_n_trials_prioritized( n_jobs: int, capsys: _pytest.capture.CaptureFixture ) -> None: study = create_study() study.optimize(lambda _: 1.0, n_trials=10, n_jobs=n_jobs, timeout=10.0) _, err = capsys.readouterr() # Testing for a character that forms progress bar borders. assert "|" not in err @pytest.mark.parametrize("n_jobs", [1, 2]) def test_optimize_progbar_no_constraints( n_jobs: int, capsys: _pytest.capture.CaptureFixture ) -> None: study = create_study() with warnings.catch_warnings(): warnings.simplefilter("ignore", category=UserWarning) study.optimize(stop_objective(5), n_jobs=n_jobs, show_progress_bar=True) _, err = capsys.readouterr() # We can't simply test if stderr is empty, since we're not sure # what else could write to it. Instead, we are testing for a character # that forms progress bar borders. assert "|" not in err @pytest.mark.parametrize("n_jobs", [1, 2]) def test_optimize_without_progbar_no_constraints( n_jobs: int, capsys: _pytest.capture.CaptureFixture ) -> None: study = create_study() study.optimize(stop_objective(5), n_jobs=n_jobs) _, err = capsys.readouterr() # Testing for a character that forms progress bar borders. assert "|" not in err @pytest.mark.parametrize("n_jobs", [1, 4]) def test_callbacks(n_jobs: int) -> None: lock = threading.Lock() def with_lock(f: CallbackFuncType) -> CallbackFuncType: def callback(study: Study, trial: FrozenTrial) -> None: with lock: f(study, trial) return callback study = create_study() def objective(trial: Trial) -> float: return trial.suggest_int("x", 1, 1) # Empty callback list. study.optimize(objective, callbacks=[], n_trials=10, n_jobs=n_jobs) # One callback. values = [] callbacks = [with_lock(lambda study, trial: values.append(trial.value))] study.optimize(objective, callbacks=callbacks, n_trials=10, n_jobs=n_jobs) assert values == [1] * 10 # Two callbacks. values = [] params = [] callbacks = [ with_lock(lambda study, trial: values.append(trial.value)), with_lock(lambda study, trial: params.append(trial.params)), ] study.optimize(objective, callbacks=callbacks, n_trials=10, n_jobs=n_jobs) assert values == [1] * 10 assert params == [{"x": 1}] * 10 # If a trial is failed with an exception and the exception is caught by the study, # callbacks are invoked. states = [] callbacks = [with_lock(lambda study, trial: states.append(trial.state))] study.optimize( lambda t: 1 / 0, callbacks=callbacks, n_trials=10, n_jobs=n_jobs, catch=(ZeroDivisionError,), ) assert states == [TrialState.FAIL] * 10 # If a trial is failed with an exception and the exception isn't caught by the study, # callbacks aren't invoked. states = [] callbacks = [with_lock(lambda study, trial: states.append(trial.state))] with pytest.raises(ZeroDivisionError): study.optimize(lambda t: 1 / 0, callbacks=callbacks, n_trials=10, n_jobs=n_jobs, catch=()) assert states == [] def test_optimize_infinite_budget_progbar() -> None: def terminate_study(study: Study, trial: FrozenTrial) -> None: study.stop() study = create_study() with pytest.warns(UserWarning): study.optimize( func, n_trials=None, timeout=None, show_progress_bar=True, callbacks=[terminate_study] ) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_get_trials(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) study.optimize(lambda t: t.suggest_int("x", 1, 5), n_trials=5) with patch("copy.deepcopy", wraps=copy.deepcopy) as mock_object: trials0 = study.get_trials(deepcopy=False) assert mock_object.call_count == 0 assert len(trials0) == 5 trials1 = study.get_trials(deepcopy=True) assert mock_object.call_count > 0 assert trials0 == trials1 # `study.trials` is equivalent to `study.get_trials(deepcopy=True)`. old_count = mock_object.call_count trials2 = study.trials assert mock_object.call_count > old_count assert trials0 == trials2 @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_get_trials_state_option(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) def objective(trial: Trial) -> float: if trial.number == 0: return 0.0 # TrialState.COMPLETE. elif trial.number == 1: return 0.0 # TrialState.COMPLETE. elif trial.number == 2: raise TrialPruned # TrialState.PRUNED. else: assert False study.optimize(objective, n_trials=3) trials = study.get_trials(states=None) assert len(trials) == 3 trials = study.get_trials(states=(TrialState.COMPLETE,)) assert len(trials) == 2 assert all(t.state == TrialState.COMPLETE for t in trials) trials = study.get_trials(states=(TrialState.COMPLETE, TrialState.PRUNED)) assert len(trials) == 3 assert all(t.state in (TrialState.COMPLETE, TrialState.PRUNED) for t in trials) trials = study.get_trials(states=()) assert len(trials) == 0 other_states = [ s for s in list(TrialState) if s != TrialState.COMPLETE and s != TrialState.PRUNED ] for s in other_states: trials = study.get_trials(states=(s,)) assert len(trials) == 0 def test_log_completed_trial(capsys: _pytest.capture.CaptureFixture) -> None: # We need to reconstruct our default handler to properly capture stderr. logging._reset_library_root_logger() logging.set_verbosity(logging.INFO) study = create_study() study.optimize(lambda _: 1.0, n_trials=1) _, err = capsys.readouterr() assert "Trial 0" in err logging.set_verbosity(logging.WARNING) study.optimize(lambda _: 1.0, n_trials=1) _, err = capsys.readouterr() assert "Trial 1" not in err logging.set_verbosity(logging.DEBUG) study.optimize(lambda _: 1.0, n_trials=1) _, err = capsys.readouterr() assert "Trial 2" in err def test_log_completed_trial_skip_storage_access() -> None: study = create_study() # Create a trial to retrieve it as the `study.best_trial`. study.optimize(lambda _: 0.0, n_trials=1) frozen_trial = study.best_trial storage = study._storage with patch.object(storage, "get_best_trial", wraps=storage.get_best_trial) as mock_object: study._log_completed_trial(frozen_trial) assert mock_object.call_count == 1 logging.set_verbosity(logging.WARNING) with patch.object(storage, "get_best_trial", wraps=storage.get_best_trial) as mock_object: study._log_completed_trial(frozen_trial) assert mock_object.call_count == 0 logging.set_verbosity(logging.DEBUG) with patch.object(storage, "get_best_trial", wraps=storage.get_best_trial) as mock_object: study._log_completed_trial(frozen_trial) assert mock_object.call_count == 1 def test_create_study_with_multi_objectives() -> None: study = create_study(directions=["maximize"]) assert study.direction == StudyDirection.MAXIMIZE assert not study._is_multi_objective() study = create_study(directions=["maximize", "minimize"]) assert study.directions == [StudyDirection.MAXIMIZE, StudyDirection.MINIMIZE] assert study._is_multi_objective() with pytest.raises(ValueError): # Empty `direction` isn't allowed. _ = create_study(directions=[]) with pytest.raises(ValueError): _ = create_study(direction="minimize", directions=["maximize"]) with pytest.raises(ValueError): _ = create_study(direction="minimize", directions=[]) def test_create_study_with_direction_object() -> None: study = create_study(direction=StudyDirection.MAXIMIZE) assert study.direction == StudyDirection.MAXIMIZE study = create_study(directions=[StudyDirection.MAXIMIZE, StudyDirection.MINIMIZE]) assert study.directions == [StudyDirection.MAXIMIZE, StudyDirection.MINIMIZE] @pytest.mark.parametrize("n_objectives", [2, 3]) def test_optimize_with_multi_objectives(n_objectives: int) -> None: directions = ["minimize" for _ in range(n_objectives)] study = create_study(directions=directions) def objective(trial: Trial) -> list[float]: return [trial.suggest_float("v{}".format(i), 0, 5) for i in range(n_objectives)] study.optimize(objective, n_trials=10) assert len(study.trials) == 10 for trial in study.trials: assert trial.values assert len(trial.values) == n_objectives @pytest.mark.parametrize("direction", [StudyDirection.MINIMIZE, StudyDirection.MAXIMIZE]) def test_best_trial_constrained_optimization(direction: StudyDirection) -> None: study = create_study(direction=direction) storage = study._storage with pytest.raises(ValueError): # No trials. study.best_trial trial = study.ask() storage.set_trial_system_attr(trial._trial_id, _CONSTRAINTS_KEY, [1]) study.tell(trial, 0) with pytest.raises(ValueError): # No feasible trials. study.best_trial trial = study.ask() storage.set_trial_system_attr(trial._trial_id, _CONSTRAINTS_KEY, [0]) study.tell(trial, 0) assert study.best_trial.number == 1 trial = study.ask() storage.set_trial_system_attr(trial._trial_id, _CONSTRAINTS_KEY, [1]) study.tell(trial, -1 if direction == StudyDirection.MINIMIZE else 1) assert study.best_trial.number == 1 trial = study.ask() storage.set_trial_system_attr(trial._trial_id, _CONSTRAINTS_KEY, [0]) study.tell(trial, -1 if direction == StudyDirection.MINIMIZE else 1) assert study.best_trial.number == 3 def test_best_trials() -> None: study = create_study(directions=["minimize", "maximize"]) study.optimize(lambda t: [2, 2], n_trials=1) study.optimize(lambda t: [1, 1], n_trials=1) study.optimize(lambda t: [3, 1], n_trials=1) assert {tuple(t.values) for t in study.best_trials} == {(1, 1), (2, 2)} def test_best_trials_constrained_optimization() -> None: study = create_study(directions=["minimize", "maximize"]) storage = study._storage assert study.best_trials == [] trial = study.ask() storage.set_trial_system_attr(trial._trial_id, _CONSTRAINTS_KEY, [1]) study.tell(trial, [0, 0]) assert study.best_trials == [] trial = study.ask() storage.set_trial_system_attr(trial._trial_id, _CONSTRAINTS_KEY, [0]) study.tell(trial, [0, 0]) assert study.best_trials == [study.trials[1]] trial = study.ask() storage.set_trial_system_attr(trial._trial_id, _CONSTRAINTS_KEY, [1]) study.tell(trial, [-1, 1]) assert study.best_trials == [study.trials[1]] trial = study.ask() storage.set_trial_system_attr(trial._trial_id, _CONSTRAINTS_KEY, [0]) study.tell(trial, [1, 1]) assert {t.number for t in study.best_trials} == {1, 3} def test_wrong_n_objectives() -> None: n_objectives = 2 directions = ["minimize" for _ in range(n_objectives)] study = create_study(directions=directions) def objective(trial: Trial) -> list[float]: return [trial.suggest_float("v{}".format(i), 0, 5) for i in range(n_objectives + 1)] study.optimize(objective, n_trials=10) for trial in study.trials: assert trial.state is TrialState.FAIL def test_ask() -> None: study = create_study() trial = study.ask() assert isinstance(trial, Trial) def test_ask_enqueue_trial() -> None: study = create_study() study.enqueue_trial({"x": 0.5}, user_attrs={"memo": "this is memo"}) trial = study.ask() assert trial.suggest_float("x", 0, 1) == 0.5 assert trial.user_attrs == {"memo": "this is memo"} def test_ask_fixed_search_space() -> None: fixed_distributions = { "x": distributions.FloatDistribution(0, 1), "y": distributions.CategoricalDistribution(["bacon", "spam"]), } study = create_study() trial = study.ask(fixed_distributions=fixed_distributions) params = trial.params assert len(trial.params) == 2 assert 0 <= params["x"] < 1 assert params["y"] in ["bacon", "spam"] # Deprecated distributions are internally converted to corresponding distributions. @pytest.mark.filterwarnings("ignore::FutureWarning") def test_ask_distribution_conversion() -> None: fixed_distributions = { "ud": distributions.UniformDistribution(low=0, high=10), "dud": distributions.DiscreteUniformDistribution(low=0, high=10, q=2), "lud": distributions.LogUniformDistribution(low=1, high=10), "id": distributions.IntUniformDistribution(low=0, high=10), "idd": distributions.IntUniformDistribution(low=0, high=10, step=2), "ild": distributions.IntLogUniformDistribution(low=1, high=10), } study = create_study() with pytest.warns( FutureWarning, match="See https://github.com/optuna/optuna/issues/2941", ) as record: trial = study.ask(fixed_distributions=fixed_distributions) assert len(record) == 6 expected_distributions = { "ud": distributions.FloatDistribution(low=0, high=10, log=False, step=None), "dud": distributions.FloatDistribution(low=0, high=10, log=False, step=2), "lud": distributions.FloatDistribution(low=1, high=10, log=True, step=None), "id": distributions.IntDistribution(low=0, high=10, log=False, step=1), "idd": distributions.IntDistribution(low=0, high=10, log=False, step=2), "ild": distributions.IntDistribution(low=1, high=10, log=True, step=1), } assert trial.distributions == expected_distributions # It confirms that ask doesn't convert non-deprecated distributions. def test_ask_distribution_conversion_noop() -> None: fixed_distributions = { "ud": distributions.FloatDistribution(low=0, high=10, log=False, step=None), "dud": distributions.FloatDistribution(low=0, high=10, log=False, step=2), "lud": distributions.FloatDistribution(low=1, high=10, log=True, step=None), "id": distributions.IntDistribution(low=0, high=10, log=False, step=1), "idd": distributions.IntDistribution(low=0, high=10, log=False, step=2), "ild": distributions.IntDistribution(low=1, high=10, log=True, step=1), "cd": distributions.CategoricalDistribution(choices=["a", "b", "c"]), } study = create_study() trial = study.ask(fixed_distributions=fixed_distributions) # Check fixed_distributions doesn't change. assert trial.distributions == fixed_distributions def test_tell() -> None: study = create_study() assert len(study.trials) == 0 trial = study.ask() assert len(study.trials) == 1 assert len(study.get_trials(states=(TrialState.COMPLETE,))) == 0 study.tell(trial, 1.0) assert len(study.trials) == 1 assert len(study.get_trials(states=(TrialState.COMPLETE,))) == 1 study.tell(study.ask(), [1.0]) assert len(study.trials) == 2 assert len(study.get_trials(states=(TrialState.COMPLETE,))) == 2 # `trial` could be int. study.tell(study.ask().number, 1.0) assert len(study.trials) == 3 assert len(study.get_trials(states=(TrialState.COMPLETE,))) == 3 # Inf is supported as values. study.tell(study.ask(), float("inf")) assert len(study.trials) == 4 assert len(study.get_trials(states=(TrialState.COMPLETE,))) == 4 study.tell(study.ask(), state=TrialState.PRUNED) assert len(study.trials) == 5 assert len(study.get_trials(states=(TrialState.PRUNED,))) == 1 study.tell(study.ask(), state=TrialState.FAIL) assert len(study.trials) == 6 assert len(study.get_trials(states=(TrialState.FAIL,))) == 1 def test_tell_pruned() -> None: study = create_study() study.tell(study.ask(), state=TrialState.PRUNED) assert study.trials[-1].value is None assert study.trials[-1].state == TrialState.PRUNED # Store the last intermediates as value. trial = study.ask() trial.report(2.0, step=1) study.tell(trial, state=TrialState.PRUNED) assert study.trials[-1].value == 2.0 assert study.trials[-1].state == TrialState.PRUNED # Inf is also supported as a value. trial = study.ask() trial.report(float("inf"), step=1) study.tell(trial, state=TrialState.PRUNED) assert study.trials[-1].value == float("inf") assert study.trials[-1].state == TrialState.PRUNED # NaN is not supported as a value. trial = study.ask() trial.report(float("nan"), step=1) study.tell(trial, state=TrialState.PRUNED) assert study.trials[-1].value is None assert study.trials[-1].state == TrialState.PRUNED def test_tell_automatically_fail() -> None: study = create_study() # Check invalid values, e.g. str cannot be cast to float. with pytest.warns(UserWarning): study.tell(study.ask(), "a") # type: ignore assert len(study.trials) == 1 assert study.trials[-1].state == TrialState.FAIL assert study.trials[-1].values is None # Check invalid values, e.g. `None` that cannot be cast to float. with pytest.warns(UserWarning): study.tell(study.ask(), None) assert len(study.trials) == 2 assert study.trials[-1].state == TrialState.FAIL assert study.trials[-1].values is None # Check number of values. with pytest.warns(UserWarning): study.tell(study.ask(), []) assert len(study.trials) == 3 assert study.trials[-1].state == TrialState.FAIL assert study.trials[-1].values is None # Check wrong number of values, e.g. two values for single direction. with pytest.warns(UserWarning): study.tell(study.ask(), [1.0, 2.0]) assert len(study.trials) == 4 assert study.trials[-1].state == TrialState.FAIL assert study.trials[-1].values is None # Both state and values are not specified. with pytest.warns(UserWarning): study.tell(study.ask()) assert len(study.trials) == 5 assert study.trials[-1].state == TrialState.FAIL assert study.trials[-1].values is None # Nan is not supported. with pytest.warns(UserWarning): study.tell(study.ask(), float("nan")) assert len(study.trials) == 6 assert study.trials[-1].state == TrialState.FAIL assert study.trials[-1].values is None def test_tell_multi_objective() -> None: study = create_study(directions=["minimize", "maximize"]) study.tell(study.ask(), [1.0, 2.0]) assert len(study.trials) == 1 def test_tell_multi_objective_automatically_fail() -> None: # Number of values doesn't match the length of directions. study = create_study(directions=["minimize", "maximize"]) with pytest.warns(UserWarning): study.tell(study.ask(), []) assert len(study.trials) == 1 assert study.trials[-1].state == TrialState.FAIL assert study.trials[-1].values is None with pytest.warns(UserWarning): study.tell(study.ask(), [1.0]) assert len(study.trials) == 2 assert study.trials[-1].state == TrialState.FAIL assert study.trials[-1].values is None with pytest.warns(UserWarning): study.tell(study.ask(), [1.0, 2.0, 3.0]) assert len(study.trials) == 3 assert study.trials[-1].state == TrialState.FAIL assert study.trials[-1].values is None with pytest.warns(UserWarning): study.tell(study.ask(), [1.0, None]) # type: ignore assert len(study.trials) == 4 assert study.trials[-1].state == TrialState.FAIL assert study.trials[-1].values is None with pytest.warns(UserWarning): study.tell(study.ask(), [None, None]) # type: ignore assert len(study.trials) == 5 assert study.trials[-1].state == TrialState.FAIL assert study.trials[-1].values is None with pytest.warns(UserWarning): study.tell(study.ask(), 1.0) assert len(study.trials) == 6 assert study.trials[-1].state == TrialState.FAIL assert study.trials[-1].values is None def test_tell_invalid() -> None: study = create_study() # Missing values for completions. with pytest.raises(ValueError): study.tell(study.ask(), state=TrialState.COMPLETE) # Invalid values for completions. with pytest.raises(ValueError): study.tell(study.ask(), "a", state=TrialState.COMPLETE) # type: ignore with pytest.raises(ValueError): study.tell(study.ask(), None, state=TrialState.COMPLETE) with pytest.raises(ValueError): study.tell(study.ask(), [], state=TrialState.COMPLETE) with pytest.raises(ValueError): study.tell(study.ask(), [1.0, 2.0], state=TrialState.COMPLETE) with pytest.raises(ValueError): study.tell(study.ask(), float("nan"), state=TrialState.COMPLETE) # `state` must be None or finished state. with pytest.raises(ValueError): study.tell(study.ask(), state=TrialState.RUNNING) # `state` must be None or finished state. with pytest.raises(ValueError): study.tell(study.ask(), state=TrialState.WAITING) # `value` must be None for `TrialState.PRUNED`. with pytest.raises(ValueError): study.tell(study.ask(), values=1, state=TrialState.PRUNED) # `value` must be None for `TrialState.FAIL`. with pytest.raises(ValueError): study.tell(study.ask(), values=1, state=TrialState.FAIL) # Trial that has not been asked for cannot be told. with pytest.raises(ValueError): study.tell(study.ask().number + 1, 1.0) # Waiting trial cannot be told. with pytest.raises(ValueError): study.enqueue_trial({}) study.tell(study.trials[-1].number, 1.0) # It must be Trial or int for trial. with pytest.raises(TypeError): study.tell("1", 1.0) # type: ignore def test_tell_duplicate_tell() -> None: study = create_study() trial = study.ask() study.tell(trial, 1.0) # Should not panic when passthrough is enabled. study.tell(trial, 1.0, skip_if_finished=True) with pytest.raises(ValueError): study.tell(trial, 1.0, skip_if_finished=False) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_enqueued_trial_datetime_start(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) def objective(trial: Trial) -> float: time.sleep(1) x = trial.suggest_int("x", -10, 10) return x study.enqueue_trial(params={"x": 1}) assert study.trials[0].datetime_start is None study.optimize(objective, n_trials=1) assert study.trials[0].datetime_start is not None @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_study_summary_datetime_start_calculation(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: def objective(trial: Trial) -> float: x = trial.suggest_int("x", -10, 10) return x # StudySummary datetime_start tests. study = create_study(storage=storage) study.enqueue_trial(params={"x": 1}) # Study summary with only enqueued trials should have null datetime_start. summaries = get_all_study_summaries(study._storage, include_best_trial=True) assert summaries[0].datetime_start is None # Study summary with completed trials should have nonnull datetime_start. study.optimize(objective, n_trials=1) study.enqueue_trial(params={"x": 1}, skip_if_exists=False) summaries = get_all_study_summaries(study._storage, include_best_trial=True) assert summaries[0].datetime_start is not None def _process_tell(study: Study, trial: Trial | int, values: float) -> None: study.tell(trial, values) def test_tell_from_another_process() -> None: pool = multiprocessing.Pool() with StorageSupplier("sqlite") as storage: # Create a study and ask for a new trial. study = create_study(storage=storage) trial0 = study.ask() # Test normal behaviour. pool.starmap(_process_tell, [(study, trial0, 1.2)]) assert len(study.trials) == 1 assert study.best_trial.state == TrialState.COMPLETE assert study.best_value == 1.2 # Test study.tell using trial number. trial = study.ask() pool.starmap(_process_tell, [(study, trial.number, 1.5)]) assert len(study.trials) == 2 assert study.best_trial.state == TrialState.COMPLETE assert study.best_value == 1.2 # Should fail because the trial0 is already finished. with pytest.raises(ValueError): pool.starmap(_process_tell, [(study, trial0, 1.2)]) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_pop_waiting_trial_thread_safe(storage_mode: str) -> None: if "sqlite" == storage_mode or "cached_sqlite" == storage_mode: pytest.skip("study._pop_waiting_trial is not thread-safe on SQLite3") num_enqueued = 10 with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) for i in range(num_enqueued): study.enqueue_trial({"i": i}) trial_id_set = set() with ThreadPoolExecutor(10) as pool: futures = [] for i in range(num_enqueued): future = pool.submit(study._pop_waiting_trial_id) futures.append(future) for future in as_completed(futures): trial_id_set.add(future.result()) assert len(trial_id_set) == num_enqueued def test_set_metric_names() -> None: metric_names = ["v0", "v1"] study = create_study(directions=["minimize", "minimize"]) study.set_metric_names(metric_names) got_metric_names = study._storage.get_study_system_attrs(study._study_id).get( _SYSTEM_ATTR_METRIC_NAMES ) assert got_metric_names is not None assert metric_names == got_metric_names def test_set_metric_names_experimental_warning() -> None: study = create_study() with pytest.warns(ExperimentalWarning): study.set_metric_names(["v0"]) def test_set_invalid_metric_names() -> None: metric_names = ["v0", "v1", "v2"] study = create_study(directions=["minimize", "minimize"]) with pytest.raises(ValueError): study.set_metric_names(metric_names) def test_get_metric_names() -> None: study = create_study() assert study.metric_names is None study.set_metric_names(["v0"]) assert study.metric_names == ["v0"] study.set_metric_names(["v1"]) assert study.metric_names == ["v1"] optuna-4.1.0/tests/study_tests/test_study_summary.py000066400000000000000000000026471471332314300231260ustar00rootroot00000000000000import copy import pytest from optuna import create_study from optuna import get_all_study_summaries from optuna.storages import RDBStorage def test_study_summary_eq_ne() -> None: storage = RDBStorage("sqlite:///:memory:") create_study(storage=storage) study = create_study(storage=storage) summaries = get_all_study_summaries(study._storage, include_best_trial=True) assert len(summaries) == 2 assert summaries[0] == copy.deepcopy(summaries[0]) assert not summaries[0] != copy.deepcopy(summaries[0]) assert not summaries[0] == summaries[1] assert summaries[0] != summaries[1] assert not summaries[0] == 1 assert summaries[0] != 1 def test_study_summary_lt_le() -> None: storage = RDBStorage("sqlite:///:memory:") create_study(storage=storage) study = create_study(storage=storage) summaries = get_all_study_summaries(study._storage, include_best_trial=True) assert len(summaries) == 2 summary_0 = summaries[0] summary_1 = summaries[1] assert summary_0 < summary_1 assert not summary_1 < summary_0 with pytest.raises(TypeError): summary_0 < 1 assert summary_0 <= summary_0 assert not summary_1 <= summary_0 with pytest.raises(TypeError): summary_0 <= 1 # A list of StudySummaries is sortable. summaries.reverse() summaries.sort() assert summaries[0] == summary_0 assert summaries[1] == summary_1 optuna-4.1.0/tests/terminator_tests/000077500000000000000000000000001471332314300175735ustar00rootroot00000000000000optuna-4.1.0/tests/terminator_tests/improvement_tests/000077500000000000000000000000001471332314300233625ustar00rootroot00000000000000optuna-4.1.0/tests/terminator_tests/improvement_tests/test_emmr_evaluator.py000066400000000000000000000033671471332314300300260ustar00rootroot00000000000000from __future__ import annotations import math import sys import numpy as np import pytest from optuna.distributions import FloatDistribution from optuna.study import StudyDirection from optuna.terminator import EMMREvaluator from optuna.terminator.improvement.emmr import MARGIN_FOR_NUMARICAL_STABILITY from optuna.trial import create_trial from optuna.trial import FrozenTrial def test_emmr_evaluate() -> None: evaluator = EMMREvaluator(min_n_trials=3) trials = [ create_trial( value=i, distributions={"a": FloatDistribution(-1.0, 10.0)}, params={"a": float(i)}, ) for i in range(3) ] criterion = evaluator.evaluate(trials=trials, study_direction=StudyDirection.MAXIMIZE) assert np.isfinite(criterion) assert criterion < sys.float_info.max def test_emmr_evaluate_with_invalid_argument() -> None: with pytest.raises(ValueError): EMMREvaluator(min_n_trials=1) def test_emmr_evaluate_with_insufficient_trials() -> None: evaluator = EMMREvaluator() trials: list[FrozenTrial] = [] for _ in range(2): criterion = evaluator.evaluate(trials=trials, study_direction=StudyDirection.MAXIMIZE) assert math.isclose(criterion, sys.float_info.max * MARGIN_FOR_NUMARICAL_STABILITY) trials.append(create_trial(value=0, distributions={}, params={})) def test_emmr_evaluate_with_empty_intersection_search_space() -> None: evaluator = EMMREvaluator() trials = [create_trial(value=0, distributions={}, params={}) for _ in range(3)] with pytest.warns(UserWarning): criterion = evaluator.evaluate(trials=trials, study_direction=StudyDirection.MAXIMIZE) assert math.isclose(criterion, sys.float_info.max * MARGIN_FOR_NUMARICAL_STABILITY) optuna-4.1.0/tests/terminator_tests/improvement_tests/test_evaluator.py000066400000000000000000000054661471332314300270100ustar00rootroot00000000000000import pytest from optuna.distributions import FloatDistribution from optuna.study import StudyDirection from optuna.terminator import BestValueStagnationEvaluator from optuna.terminator import RegretBoundEvaluator from optuna.terminator.improvement.evaluator import BaseImprovementEvaluator from optuna.trial import create_trial # TODO(g-votte): test the following edge cases # TODO(g-votte): - the user specifies non-default top_trials_ratio or min_n_trials def test_regret_bound_evaluate() -> None: trials = [ create_trial( value=0, distributions={"a": FloatDistribution(-1.0, 1.0)}, params={"a": 0.0}, ) ] evaluator = RegretBoundEvaluator() evaluator.evaluate(trials, study_direction=StudyDirection.MAXIMIZE) def test_best_value_stagnation_invalid_argument() -> None: with pytest.raises(ValueError): # Test that a negative `max_stagnation_trials` raises ValueError. BestValueStagnationEvaluator(max_stagnation_trials=-1) def test_best_value_stagnation_evaluate() -> None: evaluator = BestValueStagnationEvaluator(max_stagnation_trials=1) # A case of monotonical improvement (best step is the latest element). trials = [create_trial(value=value) for value in [0, 1, 2]] assert evaluator.evaluate(trials=trials, study_direction=StudyDirection.MAXIMIZE) == 1 trials = [create_trial(value=value) for value in [2, 1, 0]] assert evaluator.evaluate(trials=trials, study_direction=StudyDirection.MINIMIZE) == 1 # A case of jagged improvement (best step is the second element). trials = [create_trial(value=value) for value in [0, 1, 0]] assert evaluator.evaluate(trials=trials, study_direction=StudyDirection.MAXIMIZE) == 0 trials = [create_trial(value=value) for value in [1, 0, 1]] assert evaluator.evaluate(trials=trials, study_direction=StudyDirection.MINIMIZE) == 0 # A case of flat improvement (best step is the first element). trials = [create_trial(value=value) for value in [0, 0, 0]] assert evaluator.evaluate(trials=trials, study_direction=StudyDirection.MAXIMIZE) == -1 assert evaluator.evaluate(trials=trials, study_direction=StudyDirection.MINIMIZE) == -1 @pytest.mark.parametrize("evaluator", [RegretBoundEvaluator(), BestValueStagnationEvaluator()]) def test_evaluate_with_no_trial(evaluator: BaseImprovementEvaluator) -> None: with pytest.raises(ValueError): evaluator.evaluate(trials=[], study_direction=StudyDirection.MAXIMIZE) def test_evaluate_with_empty_intersection_search_space() -> None: evaluator = RegretBoundEvaluator() trials = [ create_trial( value=0, distributions={}, params={}, ) ] with pytest.raises(ValueError): evaluator.evaluate(trials=trials, study_direction=StudyDirection.MAXIMIZE) optuna-4.1.0/tests/terminator_tests/test_callback.py000066400000000000000000000022441471332314300227420ustar00rootroot00000000000000from optuna.study import create_study from optuna.study import Study from optuna.terminator import BaseTerminator from optuna.terminator import TerminatorCallback from optuna.trial import TrialState class _DeterministicTerminator(BaseTerminator): def __init__(self, termination_trial_number: int) -> None: self._termination_trial_number = termination_trial_number def should_terminate(self, study: Study) -> bool: trials = study.get_trials(states=[TrialState.COMPLETE]) latest_number = max([t.number for t in trials]) if latest_number >= self._termination_trial_number: return True else: return False def test_terminator_callback_terminator() -> None: # This test case validates that the study is stopped when the `should_terminate` method of the # terminator returns `True` for the first time. termination_trial_number = 10 callback = TerminatorCallback( terminator=_DeterministicTerminator(termination_trial_number), ) study = create_study() study.optimize(lambda _: 0.0, callbacks=[callback], n_trials=100) assert len(study.trials) == termination_trial_number + 1 optuna-4.1.0/tests/terminator_tests/test_erroreval.py000066400000000000000000000056441471332314300232160ustar00rootroot00000000000000from __future__ import annotations import math import pytest from optuna.study.study import create_study from optuna.terminator import CrossValidationErrorEvaluator from optuna.terminator import report_cross_validation_scores from optuna.terminator import StaticErrorEvaluator from optuna.terminator.erroreval import _CROSS_VALIDATION_SCORES_KEY from optuna.trial import create_trial from optuna.trial import FrozenTrial def _create_trial(value: float, cv_scores: list[float]) -> FrozenTrial: return create_trial( params={}, distributions={}, value=value, system_attrs={_CROSS_VALIDATION_SCORES_KEY: cv_scores}, ) @pytest.mark.parametrize("direction", ["minimize", "maximize"]) def test_cross_validation_evaluator(direction: str) -> None: study = create_study(direction=direction) sign = 1 if direction == "minimize" else -1 study.add_trials( [ _create_trial( value=sign * 2.0, cv_scores=[1.0, -1.0] ), # Second best trial with 1.0 var. _create_trial(value=sign * 1.0, cv_scores=[2.0, -2.0]), # Best trial with 4.0 var. ] ) evaluator = CrossValidationErrorEvaluator() serror = evaluator.evaluate(study.trials, study.direction) expected_scale = 1.5 assert serror == math.sqrt(4.0 * expected_scale) @pytest.mark.parametrize("direction", ["minimize", "maximize"]) def test_cross_validation_evaluator_without_cv_scores(direction: str) -> None: study = create_study(direction=direction) study.add_trial( # Note that the CV score is not reported with the system attr. create_trial(params={}, distributions={}, value=0.0) ) evaluator = CrossValidationErrorEvaluator() with pytest.raises(ValueError): evaluator.evaluate(study.trials, study.direction) @pytest.mark.parametrize("direction", ["minimize", "maximize"]) def test_report_cross_validation_scores(direction: str) -> None: scores = [1.0, 2.0] study = create_study(direction=direction) trial = study.ask() report_cross_validation_scores(trial, scores) study.tell(trial, 0.0) assert study.trials[0].system_attrs[_CROSS_VALIDATION_SCORES_KEY] == scores @pytest.mark.parametrize("direction", ["minimize", "maximize"]) def test_report_cross_validation_scores_with_illegal_scores_length(direction: str) -> None: scores = [1.0] study = create_study(direction=direction) trial = study.ask() with pytest.raises(ValueError): report_cross_validation_scores(trial, scores) @pytest.mark.parametrize("direction", ["minimize", "maximize"]) def test_static_evaluator(direction: str) -> None: study = create_study(direction=direction) study.add_trials( [ _create_trial(value=2.0, cv_scores=[1.0, -1.0]), ] ) evaluator = StaticErrorEvaluator(constant=100.0) serror = evaluator.evaluate(study.trials, study.direction) assert serror == 100.0 optuna-4.1.0/tests/terminator_tests/test_median_erroreval.py000066400000000000000000000043601471332314300245250ustar00rootroot00000000000000from __future__ import annotations import numpy as np import pytest from optuna.distributions import FloatDistribution from optuna.study import StudyDirection from optuna.terminator import EMMREvaluator from optuna.terminator import MedianErrorEvaluator from optuna.trial import create_trial from optuna.trial import FrozenTrial def test_validation_ratio_to_initial_median_evaluator() -> None: with pytest.raises(ValueError): MedianErrorEvaluator(paired_improvement_evaluator=EMMREvaluator(), warm_up_trials=-1) with pytest.raises(ValueError): MedianErrorEvaluator(paired_improvement_evaluator=EMMREvaluator(), n_initial_trials=0) with pytest.raises(ValueError): MedianErrorEvaluator(paired_improvement_evaluator=EMMREvaluator(), threshold_ratio=0.0) @pytest.mark.parametrize("direction", [StudyDirection.MINIMIZE, StudyDirection.MAXIMIZE]) def test_ratio_to_initial_median_evaluator( direction: StudyDirection, ) -> None: trials: list[FrozenTrial] = [] evaluator = MedianErrorEvaluator( paired_improvement_evaluator=EMMREvaluator(min_n_trials=3), warm_up_trials=1, n_initial_trials=3, threshold_ratio=2.0, ) trials.append( create_trial( value=1.0, distributions={"a": FloatDistribution(-1.0, 10000.0)}, params={"a": 1.0}, ) ) criterion = evaluator.evaluate(trials, direction) assert criterion <= 0.0 trials.append( create_trial( value=22.0, distributions={"a": FloatDistribution(-1.0, 10000.0)}, params={"a": 22.0}, ) ) criterion = evaluator.evaluate(trials, direction) assert criterion <= 0.0 trials.append( create_trial( value=333.0, distributions={"a": FloatDistribution(-1.0, 10000.0)}, params={"a": 333.0}, ) ) criterion = evaluator.evaluate(trials, direction) assert criterion <= 0.0 trials.append( create_trial( value=4444.0, distributions={"a": FloatDistribution(-1.0, 10000.0)}, params={"a": 4444.0}, ) ) criterion = evaluator.evaluate(trials, direction) assert np.isfinite(criterion) assert 0.0 <= criterion optuna-4.1.0/tests/terminator_tests/test_terminator.py000066400000000000000000000044251471332314300233750ustar00rootroot00000000000000from __future__ import annotations import pytest from optuna.study._study_direction import StudyDirection from optuna.study.study import create_study from optuna.terminator import BaseImprovementEvaluator from optuna.terminator import StaticErrorEvaluator from optuna.terminator import Terminator from optuna.terminator.improvement.evaluator import BestValueStagnationEvaluator from optuna.trial import FrozenTrial class _StaticImprovementEvaluator(BaseImprovementEvaluator): def __init__(self, constant: float) -> None: super().__init__() self._constant = constant def evaluate(self, trials: list[FrozenTrial], study_direction: StudyDirection) -> float: return self._constant def test_init() -> None: # Test that a positive `min_n_trials` does not raise any error. Terminator(min_n_trials=1) with pytest.raises(ValueError): # Test that a non-positive `min_n_trials` raises ValueError. Terminator(min_n_trials=0) Terminator(BestValueStagnationEvaluator(), min_n_trials=1) def test_should_terminate() -> None: study = create_study() # Add dummy trial because Terminator needs at least one trial. trial = study.ask() study.tell(trial, 0.0) # Improvement is greater than error. terminator = Terminator( improvement_evaluator=_StaticImprovementEvaluator(3), error_evaluator=StaticErrorEvaluator(0), min_n_trials=1, ) assert not terminator.should_terminate(study) # Improvement is less than error. terminator = Terminator( improvement_evaluator=_StaticImprovementEvaluator(-1), error_evaluator=StaticErrorEvaluator(0), min_n_trials=1, ) assert terminator.should_terminate(study) # Improvement is less than error. However, the number of trials is less than `min_n_trials`. terminator = Terminator( improvement_evaluator=_StaticImprovementEvaluator(-1), error_evaluator=StaticErrorEvaluator(0), min_n_trials=2, ) assert not terminator.should_terminate(study) # Improvement is equal to error. terminator = Terminator( improvement_evaluator=_StaticImprovementEvaluator(0), error_evaluator=StaticErrorEvaluator(0), min_n_trials=1, ) assert not terminator.should_terminate(study) optuna-4.1.0/tests/test_callbacks.py000066400000000000000000000013001471332314300175070ustar00rootroot00000000000000from optuna import create_study from optuna.study import MaxTrialsCallback from optuna.testing.objectives import pruned_objective from optuna.trial import TrialState def test_stop_with_MaxTrialsCallback() -> None: # Test stopping the optimization with MaxTrialsCallback. study = create_study() study.optimize(lambda _: 1.0, n_trials=10, callbacks=[MaxTrialsCallback(5)]) assert len(study.trials) == 5 # Test stopping the optimization with MaxTrialsCallback with pruned trials study = create_study() study.optimize( pruned_objective, n_trials=10, callbacks=[MaxTrialsCallback(5, states=(TrialState.PRUNED,))], ) assert len(study.trials) == 5 optuna-4.1.0/tests/test_cli.py000066400000000000000000001565641471332314300163660ustar00rootroot00000000000000import json import os import platform import re import subprocess from subprocess import CalledProcessError import tempfile from typing import Any from typing import Callable from typing import Optional from typing import Tuple from unittest.mock import MagicMock from unittest.mock import patch import fakeredis import numpy as np from pandas import Timedelta from pandas import Timestamp import pytest import yaml import optuna import optuna.cli from optuna.exceptions import CLIUsageError from optuna.exceptions import ExperimentalWarning from optuna.storages import JournalStorage from optuna.storages import RDBStorage from optuna.storages.journal import JournalFileBackend from optuna.storages.journal import JournalRedisBackend from optuna.study import StudyDirection from optuna.testing.storages import StorageSupplier from optuna.testing.tempfile_pool import NamedTemporaryFilePool from optuna.trial import Trial from optuna.trial import TrialState # An example of objective functions def objective_func(trial: Trial) -> float: x = trial.suggest_float("x", -10, 10) return (x + 5) ** 2 # An example of objective functions for branched search spaces def objective_func_branched_search_space(trial: Trial) -> float: c = trial.suggest_categorical("c", ("A", "B")) if c == "A": x = trial.suggest_float("x", -10, 10) return (x + 5) ** 2 else: y = trial.suggest_float("y", -10, 10) return (y + 5) ** 2 # An example of objective functions for multi-objective optimization def objective_func_multi_objective(trial: Trial) -> Tuple[float, float]: x = trial.suggest_float("x", -10, 10) return (x + 5) ** 2, (x - 5) ** 2 def _parse_output(output: str, output_format: str) -> Any: """Parse CLI output. Args: output: The output of command. output_format: The format of output specified by command. Returns: For table format, a list of dict formatted rows. For JSON or YAML format, a list or a dict corresponding to ``output``. """ if output_format == "value": # Currently, _parse_output with output_format="value" is used only for # `study-names` command. return [{"name": values} for values in output.split(os.linesep)] elif output_format == "table": rows = output.split(os.linesep) assert all(len(rows[0]) == len(row) for row in rows) # Check ruled lines. assert rows[0] == rows[2] == rows[-1] keys = [r.strip() for r in rows[1].split("|")[1:-1]] ret = [] for record in rows[3:-1]: attrs = {} for key, attr in zip(keys, record.split("|")[1:-1]): attrs[key] = attr.strip() ret.append(attrs) return ret elif output_format == "json": return json.loads(output) elif output_format == "yaml": return yaml.safe_load(output) else: assert False @pytest.mark.skip_coverage def test_create_study_command() -> None: with StorageSupplier("sqlite") as storage: assert isinstance(storage, RDBStorage) storage_url = str(storage.engine.url) # Create study. command = ["optuna", "create-study", "--storage", storage_url] subprocess.check_call(command) # Command output should be in name string format (no-name + UUID). study_name = str(subprocess.check_output(command).decode().strip()) name_re = r"^no-name-[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$" assert re.match(name_re, study_name) is not None # study_name should be stored in storage. study_id = storage.get_study_id_from_name(study_name) assert study_id == 2 @pytest.mark.skip_coverage def test_create_study_command_with_study_name() -> None: with StorageSupplier("sqlite") as storage: assert isinstance(storage, RDBStorage) storage_url = str(storage.engine.url) study_name = "test_study" # Create study with name. command = ["optuna", "create-study", "--storage", storage_url, "--study-name", study_name] study_name = str(subprocess.check_output(command).decode().strip()) # Check if study_name is stored in the storage. study_id = storage.get_study_id_from_name(study_name) assert storage.get_study_name_from_id(study_id) == study_name @pytest.mark.skip_coverage def test_create_study_command_without_storage_url() -> None: with pytest.raises(subprocess.CalledProcessError) as err: subprocess.check_output( ["optuna", "create-study"], env={k: v for k, v in os.environ.items() if k != "OPTUNA_STORAGE"}, ) usage = err.value.output.decode() assert usage.startswith("usage:") @pytest.mark.skip_coverage def test_create_study_command_with_storage_env() -> None: with StorageSupplier("sqlite") as storage: assert isinstance(storage, RDBStorage) storage_url = str(storage.engine.url) # Create study. command = ["optuna", "create-study"] env = {**os.environ, "OPTUNA_STORAGE": storage_url} subprocess.check_call(command, env=env) # Command output should be in name string format (no-name + UUID). study_name = str(subprocess.check_output(command, env=env).decode().strip()) name_re = r"^no-name-[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$" assert re.match(name_re, study_name) is not None # study_name should be stored in storage. study_id = storage.get_study_id_from_name(study_name) assert study_id == 2 @pytest.mark.skip_coverage def test_create_study_command_with_direction() -> None: with StorageSupplier("sqlite") as storage: assert isinstance(storage, RDBStorage) storage_url = str(storage.engine.url) command = ["optuna", "create-study", "--storage", storage_url, "--direction", "minimize"] study_name = str(subprocess.check_output(command).decode().strip()) study_id = storage.get_study_id_from_name(study_name) assert storage.get_study_directions(study_id) == [StudyDirection.MINIMIZE] command = ["optuna", "create-study", "--storage", storage_url, "--direction", "maximize"] study_name = str(subprocess.check_output(command).decode().strip()) study_id = storage.get_study_id_from_name(study_name) assert storage.get_study_directions(study_id) == [StudyDirection.MAXIMIZE] command = ["optuna", "create-study", "--storage", storage_url, "--direction", "test"] # --direction should be either 'minimize' or 'maximize'. with pytest.raises(subprocess.CalledProcessError): subprocess.check_call(command) @pytest.mark.skip_coverage def test_create_study_command_with_multiple_directions() -> None: with StorageSupplier("sqlite") as storage: assert isinstance(storage, RDBStorage) storage_url = str(storage.engine.url) command = [ "optuna", "create-study", "--storage", storage_url, "--directions", "minimize", "maximize", ] study_name = str(subprocess.check_output(command).decode().strip()) study_id = storage.get_study_id_from_name(study_name) expected_directions = [StudyDirection.MINIMIZE, StudyDirection.MAXIMIZE] assert storage.get_study_directions(study_id) == expected_directions command = [ "optuna", "create-study", "--storage", storage_url, "--directions", "minimize", "maximize", "test", ] # Each direction in --directions should be either `minimize` or `maximize`. with pytest.raises(subprocess.CalledProcessError): subprocess.check_call(command) command = [ "optuna", "create-study", "--storage", storage_url, "--direction", "minimize", "--directions", "minimize", "maximize", "test", ] # It can't specify both --direction and --directions with pytest.raises(subprocess.CalledProcessError): subprocess.check_call(command) @pytest.mark.skip_coverage def test_delete_study_command() -> None: with StorageSupplier("sqlite") as storage: assert isinstance(storage, RDBStorage) storage_url = str(storage.engine.url) study_name = "delete-study-test" # Create study. command = ["optuna", "create-study", "--storage", storage_url, "--study-name", study_name] subprocess.check_call(command) assert study_name in {s.study_name: s for s in storage.get_all_studies()} # Delete study. command = ["optuna", "delete-study", "--storage", storage_url, "--study-name", study_name] subprocess.check_call(command) assert study_name not in {s.study_name: s for s in storage.get_all_studies()} @pytest.mark.skip_coverage def test_delete_study_command_without_storage_url() -> None: with pytest.raises(subprocess.CalledProcessError): subprocess.check_output( ["optuna", "delete-study", "--study-name", "dummy_study"], env={k: v for k, v in os.environ.items() if k != "OPTUNA_STORAGE"}, ) @pytest.mark.skip_coverage def test_study_set_user_attr_command() -> None: with StorageSupplier("sqlite") as storage: assert isinstance(storage, RDBStorage) storage_url = str(storage.engine.url) # Create study. study_name = storage.get_study_name_from_id( storage.create_new_study(directions=[StudyDirection.MINIMIZE]) ) base_command = [ "optuna", "study", "set-user-attr", "--study-name", study_name, "--storage", storage_url, ] example_attrs = {"architecture": "ResNet", "baselen_score": "0.002"} for key, value in example_attrs.items(): subprocess.check_call(base_command + ["--key", key, "--value", value]) # Attrs should be stored in storage. study_id = storage.get_study_id_from_name(study_name) study_user_attrs = storage.get_study_user_attrs(study_id) assert len(study_user_attrs) == 2 assert all(study_user_attrs[k] == v for k, v in example_attrs.items()) @pytest.mark.skip_coverage @pytest.mark.parametrize("output_format", (None, "table", "json", "yaml")) def test_study_names_command(output_format: Optional[str]) -> None: with StorageSupplier("sqlite") as storage: assert isinstance(storage, RDBStorage) storage_url = str(storage.engine.url) expected_study_names = ["study-names-test1", "study-names-test2"] expected_column_name = "name" # Create a study. command = [ "optuna", "create-study", "--storage", storage_url, "--study-name", expected_study_names[0], ] subprocess.check_output(command) # Get study names. command = ["optuna", "study-names", "--storage", storage_url] if output_format is not None: command += ["--format", output_format] output = str(subprocess.check_output(command).decode().strip()) study_names = _parse_output(output, output_format or "value") # Check user_attrs are not printed. assert len(study_names) == 1 assert study_names[0]["name"] == expected_study_names[0] # Create another study. command = [ "optuna", "create-study", "--storage", storage_url, "--study-name", expected_study_names[1], ] subprocess.check_output(command) # Get study names. command = ["optuna", "study-names", "--storage", storage_url] if output_format is not None: command += ["--format", output_format] output = str(subprocess.check_output(command).decode().strip()) study_names = _parse_output(output, output_format or "value") assert len(study_names) == 2 for i, study_name in enumerate(study_names): assert list(study_name.keys()) == [expected_column_name] assert study_name["name"] == expected_study_names[i] @pytest.mark.skip_coverage def test_study_names_command_without_storage_url() -> None: with pytest.raises(subprocess.CalledProcessError): subprocess.check_output( ["optuna", "study-names", "--study-name", "dummy_study"], env={k: v for k, v in os.environ.items() if k != "OPTUNA_STORAGE"}, ) @pytest.mark.skip_coverage @pytest.mark.parametrize("output_format", (None, "table", "json", "yaml")) def test_studies_command(output_format: Optional[str]) -> None: with StorageSupplier("sqlite") as storage: assert isinstance(storage, RDBStorage) storage_url = str(storage.engine.url) # First study. study_1 = optuna.create_study(storage=storage) # Run command. command = ["optuna", "studies", "--storage", storage_url] if output_format is not None: command += ["--format", output_format] output = str(subprocess.check_output(command).decode().strip()) studies = _parse_output(output, output_format or "table") expected_keys = ["name", "direction", "n_trials", "datetime_start"] # Check user_attrs are not printed. if output_format is None or output_format == "table": assert list(studies[0].keys()) == expected_keys else: assert set(studies[0].keys()) == set(expected_keys) # Add a second study. study_2 = optuna.create_study( storage=storage, study_name="study_2", directions=["minimize", "maximize"] ) study_2.optimize(objective_func_multi_objective, n_trials=10) study_2.set_user_attr("key_1", "value_1") study_2.set_user_attr("key_2", "value_2") # Run command again to include second study. output = str(subprocess.check_output(command).decode().strip()) studies = _parse_output(output, output_format or "table") expected_keys = ["name", "direction", "n_trials", "datetime_start", "user_attrs"] assert len(studies) == 2 for study in studies: if output_format is None or output_format == "table": assert list(study.keys()) == expected_keys else: assert set(study.keys()) == set(expected_keys) # Check study_name, direction, n_trials and user_attrs for the first study. assert studies[0]["name"] == study_1.study_name if output_format is None or output_format == "table": assert studies[0]["n_trials"] == "0" assert eval(studies[0]["direction"]) == ("MINIMIZE",) assert eval(studies[0]["user_attrs"]) == {} else: assert studies[0]["n_trials"] == 0 assert studies[0]["direction"] == ["MINIMIZE"] assert studies[0]["user_attrs"] == {} # Check study_name, direction, n_trials and user_attrs for the second study. assert studies[1]["name"] == study_2.study_name if output_format is None or output_format == "table": assert studies[1]["n_trials"] == "10" assert eval(studies[1]["direction"]) == ("MINIMIZE", "MAXIMIZE") assert eval(studies[1]["user_attrs"]) == {"key_1": "value_1", "key_2": "value_2"} else: assert studies[1]["n_trials"] == 10 assert studies[1]["direction"] == ["MINIMIZE", "MAXIMIZE"] assert studies[1]["user_attrs"] == {"key_1": "value_1", "key_2": "value_2"} @pytest.mark.skip_coverage @pytest.mark.parametrize("output_format", (None, "table", "json", "yaml")) def test_studies_command_flatten(output_format: Optional[str]) -> None: with StorageSupplier("sqlite") as storage: assert isinstance(storage, RDBStorage) storage_url = str(storage.engine.url) # First study. study_1 = optuna.create_study(storage=storage) # Run command. command = ["optuna", "studies", "--storage", storage_url, "--flatten"] if output_format is not None: command += ["--format", output_format] output = str(subprocess.check_output(command).decode().strip()) studies = _parse_output(output, output_format or "table") expected_keys_1 = [ "name", "direction_0", "n_trials", "datetime_start", ] # Check user_attrs are not printed. if output_format is None or output_format == "table": assert list(studies[0].keys()) == expected_keys_1 else: assert set(studies[0].keys()) == set(expected_keys_1) # Add a second study. study_2 = optuna.create_study( storage=storage, study_name="study_2", directions=["minimize", "maximize"] ) study_2.optimize(objective_func_multi_objective, n_trials=10) study_2.set_user_attr("key_1", "value_1") study_2.set_user_attr("key_2", "value_2") # Run command again to include second study. output = str(subprocess.check_output(command).decode().strip()) studies = _parse_output(output, output_format or "table") if output_format is None or output_format == "table": expected_keys_1 = expected_keys_2 = [ "name", "direction_0", "direction_1", "n_trials", "datetime_start", "user_attrs", ] else: expected_keys_1 = ["name", "direction_0", "n_trials", "datetime_start", "user_attrs"] expected_keys_2 = [ "name", "direction_0", "direction_1", "n_trials", "datetime_start", "user_attrs", ] assert len(studies) == 2 if output_format is None or output_format == "table": assert list(studies[0].keys()) == expected_keys_1 assert list(studies[1].keys()) == expected_keys_2 else: assert set(studies[0].keys()) == set(expected_keys_1) assert set(studies[1].keys()) == set(expected_keys_2) # Check study_name, direction, n_trials and user_attrs for the first study. assert studies[0]["name"] == study_1.study_name if output_format is None or output_format == "table": assert studies[0]["n_trials"] == "0" assert studies[0]["user_attrs"] == "{}" else: assert studies[0]["n_trials"] == 0 assert studies[0]["user_attrs"] == {} assert studies[0]["direction_0"] == "MINIMIZE" # Check study_name, direction, n_trials and user_attrs for the second study. assert studies[1]["name"] == study_2.study_name if output_format is None or output_format == "table": assert studies[1]["n_trials"] == "10" assert studies[1]["user_attrs"] == "{'key_1': 'value_1', 'key_2': 'value_2'}" else: assert studies[1]["n_trials"] == 10 assert studies[1]["user_attrs"] == {"key_1": "value_1", "key_2": "value_2"} assert studies[1]["direction_0"] == "MINIMIZE" assert studies[1]["direction_1"] == "MAXIMIZE" @pytest.mark.skip_coverage @pytest.mark.parametrize("objective", (objective_func, objective_func_branched_search_space)) @pytest.mark.parametrize("output_format", (None, "table", "json", "yaml")) def test_trials_command(objective: Callable[[Trial], float], output_format: Optional[str]) -> None: with StorageSupplier("sqlite") as storage: assert isinstance(storage, RDBStorage) storage_url = str(storage.engine.url) study_name = "test_study" n_trials = 10 study = optuna.create_study(storage=storage, study_name=study_name) study.optimize(objective, n_trials=n_trials) attrs = ( "number", "value", "datetime_start", "datetime_complete", "duration", "params", "user_attrs", "state", ) # Run command. command = [ "optuna", "trials", "--storage", storage_url, "--study-name", study_name, ] if output_format is not None: command += ["--format", output_format] output = str(subprocess.check_output(command).decode().strip()) trials = _parse_output(output, output_format or "table") assert len(trials) == n_trials df = study.trials_dataframe(attrs, multi_index=True) for i, trial in enumerate(trials): for key in df.columns: expected_value = df.loc[i][key] # The param may be NaN when the objective function has branched search space. if ( key[0] == "params" and isinstance(expected_value, float) and np.isnan(expected_value) ): if output_format is None or output_format == "table": assert key[1] not in eval(trial["params"]) else: assert key[1] not in trial["params"] continue if key[1] == "": value = trial[key[0]] else: if output_format is None or output_format == "table": value = eval(trial[key[0]])[key[1]] else: value = trial[key[0]][key[1]] if isinstance(value, (int, float)): if np.isnan(expected_value): assert np.isnan(value) else: assert value == expected_value elif isinstance(expected_value, Timestamp): assert value == expected_value.strftime("%Y-%m-%d %H:%M:%S") elif isinstance(expected_value, Timedelta): assert value == str(expected_value.to_pytimedelta()) else: assert value == str(expected_value) @pytest.mark.skip_coverage @pytest.mark.parametrize("objective", (objective_func, objective_func_branched_search_space)) @pytest.mark.parametrize("output_format", (None, "table", "json", "yaml")) def test_trials_command_flatten( objective: Callable[[Trial], float], output_format: Optional[str] ) -> None: with StorageSupplier("sqlite") as storage: assert isinstance(storage, RDBStorage) storage_url = str(storage.engine.url) study_name = "test_study" n_trials = 10 study = optuna.create_study(storage=storage, study_name=study_name) study.optimize(objective, n_trials=n_trials) attrs = ( "number", "value", "datetime_start", "datetime_complete", "duration", "params", "user_attrs", "state", ) # Run command. command = [ "optuna", "trials", "--storage", storage_url, "--study-name", study_name, "--flatten", ] if output_format is not None: command += ["--format", output_format] output = str(subprocess.check_output(command).decode().strip()) trials = _parse_output(output, output_format or "table") assert len(trials) == n_trials df = study.trials_dataframe(attrs) for i, trial in enumerate(trials): assert set(trial.keys()) <= set(df.columns) for key in df.columns: expected_value = df.loc[i][key] # The param may be NaN when the objective function has branched search space. if ( key.startswith("params_") and isinstance(expected_value, float) and np.isnan(expected_value) ): if output_format is None or output_format == "table": assert trial[key] == "" else: assert key not in trial continue value = trial[key] if isinstance(value, (int, float)): if np.isnan(expected_value): assert np.isnan(value) else: assert value == expected_value elif isinstance(expected_value, Timestamp): assert value == expected_value.strftime("%Y-%m-%d %H:%M:%S") elif isinstance(expected_value, Timedelta): assert value == str(expected_value.to_pytimedelta()) else: assert value == str(expected_value) @pytest.mark.skip_coverage @pytest.mark.parametrize("objective", (objective_func, objective_func_branched_search_space)) @pytest.mark.parametrize("output_format", (None, "table", "json", "yaml")) def test_best_trial_command( objective: Callable[[Trial], float], output_format: Optional[str] ) -> None: with StorageSupplier("sqlite") as storage: assert isinstance(storage, RDBStorage) storage_url = str(storage.engine.url) study_name = "test_study" n_trials = 10 study = optuna.create_study(storage=storage, study_name=study_name) study.optimize(objective, n_trials=n_trials) attrs = ( "number", "value", "datetime_start", "datetime_complete", "duration", "params", "user_attrs", "state", ) # Run command. command = [ "optuna", "best-trial", "--storage", storage_url, "--study-name", study_name, ] if output_format is not None: command += ["--format", output_format] output = str(subprocess.check_output(command).decode().strip()) best_trial = _parse_output(output, output_format or "table") if output_format is None or output_format == "table": assert len(best_trial) == 1 best_trial = best_trial[0] df = study.trials_dataframe(attrs, multi_index=True) for key in df.columns: expected_value = df.loc[study.best_trial.number][key] # The param may be NaN when the objective function has branched search space. if ( key[0] == "params" and isinstance(expected_value, float) and np.isnan(expected_value) ): if output_format is None or output_format == "table": assert key[1] not in eval(best_trial["params"]) else: assert key[1] not in best_trial["params"] continue if key[1] == "": value = best_trial[key[0]] else: if output_format is None or output_format == "table": value = eval(best_trial[key[0]])[key[1]] else: value = best_trial[key[0]][key[1]] if isinstance(value, (int, float)): if np.isnan(expected_value): assert np.isnan(value) else: assert value == expected_value elif isinstance(expected_value, Timestamp): assert value == expected_value.strftime("%Y-%m-%d %H:%M:%S") elif isinstance(expected_value, Timedelta): assert value == str(expected_value.to_pytimedelta()) else: assert value == str(expected_value) @pytest.mark.skip_coverage @pytest.mark.parametrize("objective", (objective_func, objective_func_branched_search_space)) @pytest.mark.parametrize("output_format", (None, "table", "json", "yaml")) def test_best_trial_command_flatten( objective: Callable[[Trial], float], output_format: Optional[str] ) -> None: with StorageSupplier("sqlite") as storage: assert isinstance(storage, RDBStorage) storage_url = str(storage.engine.url) study_name = "test_study" n_trials = 10 study = optuna.create_study(storage=storage, study_name=study_name) study.optimize(objective, n_trials=n_trials) attrs = ( "number", "value", "datetime_start", "datetime_complete", "duration", "params", "user_attrs", "state", ) # Run command. command = [ "optuna", "best-trial", "--storage", storage_url, "--study-name", study_name, "--flatten", ] if output_format is not None: command += ["--format", output_format] output = str(subprocess.check_output(command).decode().strip()) best_trial = _parse_output(output, output_format or "table") if output_format is None or output_format == "table": assert len(best_trial) == 1 best_trial = best_trial[0] df = study.trials_dataframe(attrs) assert set(best_trial.keys()) <= set(df.columns) for key in df.columns: expected_value = df.loc[study.best_trial.number][key] # The param may be NaN when the objective function has branched search space. if ( key.startswith("params_") and isinstance(expected_value, float) and np.isnan(expected_value) ): if output_format is None or output_format == "table": assert best_trial[key] == "" else: assert key not in best_trial continue value = best_trial[key] if isinstance(value, (int, float)): if np.isnan(expected_value): assert np.isnan(value) else: assert value == expected_value elif isinstance(expected_value, Timestamp): assert value == expected_value.strftime("%Y-%m-%d %H:%M:%S") elif isinstance(expected_value, Timedelta): assert value == str(expected_value.to_pytimedelta()) else: assert value == str(expected_value) @pytest.mark.skip_coverage @pytest.mark.parametrize("output_format", (None, "table", "json", "yaml")) def test_best_trials_command(output_format: Optional[str]) -> None: with StorageSupplier("sqlite") as storage: assert isinstance(storage, RDBStorage) storage_url = str(storage.engine.url) study_name = "test_study" n_trials = 10 study = optuna.create_study( storage=storage, study_name=study_name, directions=("minimize", "minimize") ) study.optimize(objective_func_multi_objective, n_trials=n_trials) attrs = ( "number", "values", "datetime_start", "datetime_complete", "duration", "params", "user_attrs", "state", ) # Run command. command = [ "optuna", "best-trials", "--storage", storage_url, "--study-name", study_name, ] if output_format is not None: command += ["--format", output_format] output = str(subprocess.check_output(command).decode().strip()) trials = _parse_output(output, output_format or "table") best_trials = [trial.number for trial in study.best_trials] assert len(trials) == len(best_trials) df = study.trials_dataframe(attrs, multi_index=True) for trial in trials: number = int(trial["number"]) if output_format in (None, "table") else trial["number"] assert number in best_trials for key in df.columns: expected_value = df.loc[number][key] # The param may be NaN when the objective function has branched search space. if ( key[0] == "params" and isinstance(expected_value, float) and np.isnan(expected_value) ): if output_format is None or output_format == "table": assert key[1] not in eval(trial["params"]) else: assert key[1] not in trial["params"] continue if key[1] == "": value = trial[key[0]] else: if output_format is None or output_format == "table": value = eval(trial[key[0]])[key[1]] else: value = trial[key[0]][key[1]] if isinstance(value, (int, float)): if np.isnan(expected_value): assert np.isnan(value) else: assert value == expected_value elif isinstance(expected_value, Timestamp): assert value == expected_value.strftime("%Y-%m-%d %H:%M:%S") elif isinstance(expected_value, Timedelta): assert value == str(expected_value.to_pytimedelta()) else: assert value == str(expected_value) @pytest.mark.skip_coverage @pytest.mark.parametrize("output_format", (None, "table", "json", "yaml")) def test_best_trials_command_flatten(output_format: Optional[str]) -> None: with StorageSupplier("sqlite") as storage: assert isinstance(storage, RDBStorage) storage_url = str(storage.engine.url) study_name = "test_study" n_trials = 10 study = optuna.create_study( storage=storage, study_name=study_name, directions=("minimize", "minimize") ) study.optimize(objective_func_multi_objective, n_trials=n_trials) attrs = ( "number", "values", "datetime_start", "datetime_complete", "duration", "params", "user_attrs", "state", ) # Run command. command = [ "optuna", "best-trials", "--storage", storage_url, "--study-name", study_name, "--flatten", ] if output_format is not None: command += ["--format", output_format] output = str(subprocess.check_output(command).decode().strip()) trials = _parse_output(output, output_format or "table") best_trials = [trial.number for trial in study.best_trials] assert len(trials) == len(best_trials) df = study.trials_dataframe(attrs) for trial in trials: assert set(trial.keys()) <= set(df.columns) number = int(trial["number"]) if output_format in (None, "table") else trial["number"] for key in df.columns: expected_value = df.loc[number][key] # The param may be NaN when the objective function has branched search space. if ( key.startswith("params_") and isinstance(expected_value, float) and np.isnan(expected_value) ): if output_format is None or output_format == "table": assert trial[key] == "" else: assert key not in trial continue value = trial[key] if isinstance(value, (int, float)): if np.isnan(expected_value): assert np.isnan(value) else: assert value == expected_value elif isinstance(expected_value, Timestamp): assert value == expected_value.strftime("%Y-%m-%d %H:%M:%S") elif isinstance(expected_value, Timedelta): assert value == str(expected_value.to_pytimedelta()) else: assert value == str(expected_value) @pytest.mark.skip_coverage def test_create_study_command_with_skip_if_exists() -> None: with StorageSupplier("sqlite") as storage: assert isinstance(storage, RDBStorage) storage_url = str(storage.engine.url) study_name = "test_study" # Create study with name. command = ["optuna", "create-study", "--storage", storage_url, "--study-name", study_name] study_name = str(subprocess.check_output(command).decode().strip()) # Check if study_name is stored in the storage. study_id = storage.get_study_id_from_name(study_name) assert storage.get_study_name_from_id(study_id) == study_name # Try to create the same name study without `--skip-if-exists` flag (error). command = ["optuna", "create-study", "--storage", storage_url, "--study-name", study_name] with pytest.raises(subprocess.CalledProcessError): subprocess.check_output(command) # Try to create the same name study with `--skip-if-exists` flag (OK). command = [ "optuna", "create-study", "--storage", storage_url, "--study-name", study_name, "--skip-if-exists", ] study_name = str(subprocess.check_output(command).decode().strip()) new_study_id = storage.get_study_id_from_name(study_name) assert study_id == new_study_id # The existing study instance is reused. @pytest.mark.skip_coverage def test_empty_argv() -> None: command_empty = ["optuna"] command_empty_output = str(subprocess.check_output(command_empty)) command_help = ["optuna", "help"] command_help_output = str(subprocess.check_output(command_help)) assert command_empty_output == command_help_output def test_check_storage_url() -> None: storage_in_args = "sqlite:///args.db" assert storage_in_args == optuna.cli._check_storage_url(storage_in_args) with pytest.warns(ExperimentalWarning): with patch.dict("optuna.cli.os.environ", {"OPTUNA_STORAGE": "sqlite:///args.db"}): optuna.cli._check_storage_url(None) with pytest.raises(CLIUsageError): optuna.cli._check_storage_url(None) @pytest.mark.skipif(platform.system() == "Windows", reason="Skip on Windows") @patch("optuna.storages.journal._redis.redis") def test_get_storage_without_storage_class(mock_redis: MagicMock) -> None: with tempfile.NamedTemporaryFile(suffix=".db") as fp: storage = optuna.cli._get_storage(f"sqlite:///{fp.name}", storage_class=None) assert isinstance(storage, RDBStorage) with tempfile.NamedTemporaryFile(suffix=".log") as fp: storage = optuna.cli._get_storage(fp.name, storage_class=None) assert isinstance(storage, JournalStorage) assert isinstance(storage._backend, JournalFileBackend) mock_redis.Redis = fakeredis.FakeRedis storage = optuna.cli._get_storage("redis://localhost:6379", storage_class=None) assert isinstance(storage, JournalStorage) assert isinstance(storage._backend, JournalRedisBackend) with pytest.raises(CLIUsageError): optuna.cli._get_storage("./file-not-found.log", storage_class=None) @pytest.mark.skipif(platform.system() == "Windows", reason="Skip on Windows") @patch("optuna.storages.journal._redis.redis") def test_get_storage_with_storage_class(mock_redis: MagicMock) -> None: with tempfile.NamedTemporaryFile(suffix=".db") as fp: storage = optuna.cli._get_storage(f"sqlite:///{fp.name}", storage_class=None) assert isinstance(storage, RDBStorage) with tempfile.NamedTemporaryFile(suffix=".log") as fp: storage = optuna.cli._get_storage(fp.name, storage_class="JournalFileBackend") assert isinstance(storage, JournalStorage) assert isinstance(storage._backend, JournalFileBackend) mock_redis.Redis = fakeredis.FakeRedis storage = optuna.cli._get_storage( "redis:///localhost:6379", storage_class="JournalRedisBackend" ) assert isinstance(storage, JournalStorage) assert isinstance(storage._backend, JournalRedisBackend) with pytest.raises(CLIUsageError): with tempfile.NamedTemporaryFile(suffix=".db") as fp: optuna.cli._get_storage(f"sqlite:///{fp.name}", storage_class="InMemoryStorage") @pytest.mark.skip_coverage def test_storage_upgrade_command() -> None: with StorageSupplier("sqlite") as storage: assert isinstance(storage, RDBStorage) storage_url = str(storage.engine.url) command = ["optuna", "storage", "upgrade"] with pytest.raises(CalledProcessError): subprocess.check_call( command, env={k: v for k, v in os.environ.items() if k != "OPTUNA_STORAGE"}, ) command.extend(["--storage", storage_url]) subprocess.check_call(command) @pytest.mark.skip_coverage def test_storage_upgrade_command_with_invalid_url() -> None: with StorageSupplier("sqlite") as storage: assert isinstance(storage, RDBStorage) command = ["optuna", "storage", "upgrade", "--storage", "invalid-storage-url"] with pytest.raises(CalledProcessError): subprocess.check_call(command) parametrize_for_ask = pytest.mark.parametrize( "sampler,sampler_kwargs,output_format", [ (None, None, None), ("RandomSampler", None, None), ("TPESampler", '{"multivariate": true}', None), (None, None, "json"), (None, None, "yaml"), ], ) @pytest.mark.skip_coverage @parametrize_for_ask def test_ask( sampler: Optional[str], sampler_kwargs: Optional[str], output_format: Optional[str], ) -> None: study_name = "test_study" search_space = ( '{"x": {"name": "FloatDistribution", "attributes": {"low": 0.0, "high": 1.0}}, ' '"y": {"name": "CategoricalDistribution", "attributes": {"choices": ["foo"]}}}' ) with NamedTemporaryFilePool() as tf: db_url = "sqlite:///{}".format(tf.name) args = ["optuna", "create-study", "--storage", db_url, "--study-name", study_name] subprocess.run(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE) args = [ "optuna", "ask", "--storage", db_url, "--study-name", study_name, "--search-space", search_space, ] if sampler is not None: args += ["--sampler", sampler] if sampler_kwargs is not None: args += ["--sampler-kwargs", sampler_kwargs] if output_format is not None: args += ["--format", output_format] result = subprocess.run(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE) output = str(result.stdout.decode().strip()) trial = _parse_output(output, output_format or "json") if output_format == "table": assert len(trial) == 1 trial = trial[0] assert trial["number"] == "0" params = eval(trial["params"]) assert len(params) == 2 assert 0 <= params["x"] <= 1 assert params["y"] == "foo" else: assert trial["number"] == 0 assert 0 <= trial["params"]["x"] <= 1 assert trial["params"]["y"] == "foo" @pytest.mark.skip_coverage @parametrize_for_ask def test_ask_flatten( sampler: Optional[str], sampler_kwargs: Optional[str], output_format: Optional[str], ) -> None: study_name = "test_study" search_space = ( '{"x": {"name": "FloatDistribution", "attributes": {"low": 0.0, "high": 1.0}}, ' '"y": {"name": "CategoricalDistribution", "attributes": {"choices": ["foo"]}}}' ) with NamedTemporaryFilePool() as tf: db_url = "sqlite:///{}".format(tf.name) args = ["optuna", "create-study", "--storage", db_url, "--study-name", study_name] subprocess.run(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE) args = [ "optuna", "ask", "--storage", db_url, "--study-name", study_name, "--search-space", search_space, "--flatten", ] if sampler is not None: args += ["--sampler", sampler] if sampler_kwargs is not None: args += ["--sampler-kwargs", sampler_kwargs] if output_format is not None: args += ["--format", output_format] result = subprocess.run(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE) output = str(result.stdout.decode().strip()) trial = _parse_output(output, output_format or "json") if output_format == "table": assert len(trial) == 1 trial = trial[0] assert trial["number"] == "0" assert 0 <= float(trial["params_x"]) <= 1 assert trial["params_y"] == "foo" else: assert trial["number"] == 0 assert 0 <= trial["params_x"] <= 1 assert trial["params_y"] == "foo" @pytest.mark.skip_coverage @pytest.mark.parametrize("output_format", (None, "table", "json", "yaml")) def test_ask_empty_search_space(output_format: str) -> None: study_name = "test_study" with NamedTemporaryFilePool() as tf: db_url = "sqlite:///{}".format(tf.name) args = ["optuna", "create-study", "--storage", db_url, "--study-name", study_name] subprocess.run(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE) args = [ "optuna", "ask", "--storage", db_url, "--study-name", study_name, ] if output_format is not None: args += ["--format", output_format] output = str(subprocess.check_output(args).decode().strip()) trial = _parse_output(output, output_format or "json") if output_format == "table": assert len(trial) == 1 trial = trial[0] assert trial["number"] == "0" assert trial["params"] == "{}" else: assert trial["number"] == 0 assert trial["params"] == {} @pytest.mark.skip_coverage @pytest.mark.parametrize("output_format", (None, "table", "json", "yaml")) def test_ask_empty_search_space_flatten(output_format: str) -> None: study_name = "test_study" with NamedTemporaryFilePool() as tf: db_url = "sqlite:///{}".format(tf.name) args = ["optuna", "create-study", "--storage", db_url, "--study-name", study_name] subprocess.run(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE) args = [ "optuna", "ask", "--storage", db_url, "--study-name", study_name, "--flatten", ] if output_format is not None: args += ["--format", output_format] output = str(subprocess.check_output(args).decode().strip()) trial = _parse_output(output, output_format or "json") if output_format == "table": assert len(trial) == 1 trial = trial[0] assert trial["number"] == "0" assert "params" not in trial else: assert trial["number"] == 0 assert "params" not in trial @pytest.mark.skip_coverage def test_ask_sampler_kwargs_without_sampler() -> None: study_name = "test_study" search_space = ( '{"x": {"name": "FloatDistribution", "attributes": {"low": 0.0, "high": 1.0}}, ' '"y": {"name": "CategoricalDistribution", "attributes": {"choices": ["foo"]}}}' ) with NamedTemporaryFilePool() as tf: db_url = "sqlite:///{}".format(tf.name) args = ["optuna", "create-study", "--storage", db_url, "--study-name", study_name] subprocess.run(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE) args = [ "optuna", "ask", "--storage", db_url, "--study-name", study_name, "--search-space", search_space, "--sampler-kwargs", '{"multivariate": true}', ] result = subprocess.run(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE) error_message = result.stderr.decode() assert "`--sampler_kwargs` is set without `--sampler`." in error_message @pytest.mark.skip_coverage def test_ask_without_create_study_beforehand() -> None: study_name = "test_study" search_space = ( '{"x": {"name": "FloatDistribution", "attributes": {"low": 0.0, "high": 1.0}}, ' '"y": {"name": "CategoricalDistribution", "attributes": {"choices": ["foo"]}}}' ) with NamedTemporaryFilePool() as tf: db_url = "sqlite:///{}".format(tf.name) args = [ "optuna", "ask", "--storage", db_url, "--study-name", study_name, "--search-space", search_space, ] result = subprocess.run(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE) error_message = result.stderr.decode() assert ( "Implicit study creation within the 'ask' command was dropped in Optuna v4.0.0." in error_message ) @pytest.mark.skip_coverage @pytest.mark.parametrize( "direction,directions,sampler,sampler_kwargs", [ (None, None, None, None), ("minimize", None, None, None), (None, "minimize maximize", None, None), (None, None, "RandomSampler", None), (None, None, "TPESampler", '{"multivariate": true}'), ], ) def test_create_study_and_ask( direction: Optional[str], directions: Optional[str], sampler: Optional[str], sampler_kwargs: Optional[str], ) -> None: study_name = "test_study" search_space = ( '{"x": {"name": "FloatDistribution", "attributes": {"low": 0.0, "high": 1.0}}, ' '"y": {"name": "CategoricalDistribution", "attributes": {"choices": ["foo"]}}}' ) with NamedTemporaryFilePool() as tf: db_url = "sqlite:///{}".format(tf.name) create_study_args = [ "optuna", "create-study", "--storage", db_url, "--study-name", study_name, ] if direction is not None: create_study_args += ["--direction", direction] if directions is not None: create_study_args += ["--directions"] + directions.split() subprocess.check_call(create_study_args) args = [ "optuna", "ask", "--storage", db_url, "--study-name", study_name, "--search-space", search_space, ] if sampler is not None: args += ["--sampler", sampler] if sampler_kwargs is not None: args += ["--sampler-kwargs", sampler_kwargs] output = str(subprocess.check_output(args).decode().strip()) trial = _parse_output(output, "json") assert trial["number"] == 0 assert 0 <= trial["params"]["x"] <= 1 assert trial["params"]["y"] == "foo" @pytest.mark.skip_coverage def test_tell() -> None: study_name = "test_study" with NamedTemporaryFilePool() as tf: db_url = "sqlite:///{}".format(tf.name) args = ["optuna", "create-study", "--storage", db_url, "--study-name", study_name] subprocess.run(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE) output: Any = subprocess.check_output( [ "optuna", "ask", "--storage", db_url, "--study-name", study_name, "--format", "json", ] ) output = output.decode("utf-8") output = json.loads(output) trial_number = output["number"] subprocess.check_output( [ "optuna", "tell", "--storage", db_url, "--trial-number", str(trial_number), "--values", "1.2", ] ) study = optuna.load_study(storage=db_url, study_name=study_name) assert len(study.trials) == 1 assert study.trials[0].state == TrialState.COMPLETE assert study.trials[0].values == [1.2] # Error when updating a finished trial. ret = subprocess.run( [ "optuna", "tell", "--storage", db_url, "--trial-number", str(trial_number), "--values", "1.2", ] ) assert ret.returncode != 0 # Passing `--skip-if-finished` to a finished trial for a noop. subprocess.check_output( [ "optuna", "tell", "--storage", db_url, "--trial-number", str(trial_number), "--values", "1.3", # Setting a different value and make sure it's not persisted. "--skip-if-finished", ] ) study = optuna.load_study(storage=db_url, study_name=study_name) assert len(study.trials) == 1 assert study.trials[0].state == TrialState.COMPLETE assert study.trials[0].values == [1.2] @pytest.mark.skip_coverage def test_tell_with_nan() -> None: study_name = "test_study" with NamedTemporaryFilePool() as tf: db_url = "sqlite:///{}".format(tf.name) args = ["optuna", "create-study", "--storage", db_url, "--study-name", study_name] subprocess.run(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE) output: Any = subprocess.check_output( [ "optuna", "ask", "--storage", db_url, "--study-name", study_name, "--format", "json", ] ) output = output.decode("utf-8") output = json.loads(output) trial_number = output["number"] subprocess.check_output( [ "optuna", "tell", "--storage", db_url, "--trial-number", str(trial_number), "--values", "nan", ] ) study = optuna.load_study(storage=db_url, study_name=study_name) assert len(study.trials) == 1 assert study.trials[0].state == TrialState.FAIL assert study.trials[0].values is None @pytest.mark.skip_coverage @pytest.mark.parametrize( "verbosity, expected", [ ("--verbose", True), ("--quiet", False), ], ) def test_configure_logging_verbosity(verbosity: str, expected: bool) -> None: with StorageSupplier("sqlite") as storage: assert isinstance(storage, RDBStorage) storage_url = str(storage.engine.url) # Create study. args = ["optuna", "create-study", "--storage", storage_url, verbosity] # `--verbose` makes the log level DEBUG. # `--quiet` makes the log level WARNING. result = subprocess.run(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE) error_message = result.stderr.decode() assert ("A new study created in RDB with name" in error_message) == expected optuna-4.1.0/tests/test_convert_positional_args.py000066400000000000000000000103311471332314300225310ustar00rootroot00000000000000import re from typing import List import pytest from optuna._convert_positional_args import convert_positional_args def _sample_func(*, a: int, b: int, c: int) -> int: return a + b + c class _SimpleClass: @convert_positional_args(previous_positional_arg_names=["self", "a", "b"]) def simple_method(self, a: int, *, b: int, c: int = 1) -> None: pass def test_convert_positional_args_decorator() -> None: previous_positional_arg_names: List[str] = [] decorator_converter = convert_positional_args( previous_positional_arg_names=previous_positional_arg_names ) decorated_func = decorator_converter(_sample_func) assert decorated_func.__name__ == _sample_func.__name__ def test_convert_positional_args_future_warning_for_methods() -> None: simple_class = _SimpleClass() with pytest.warns(FutureWarning) as record: simple_class.simple_method(1, 2, c=3) # type: ignore simple_class.simple_method(1, b=2, c=3) # No warning. simple_class.simple_method(a=1, b=2, c=3) # No warning. assert len(record) == 1 for warn in record.list: assert isinstance(warn.message, FutureWarning) assert "simple_method" in str(warn.message) def test_convert_positional_args_future_warning() -> None: previous_positional_arg_names: List[str] = ["a", "b"] decorator_converter = convert_positional_args( previous_positional_arg_names=previous_positional_arg_names ) assert callable(decorator_converter) decorated_func = decorator_converter(_sample_func) with pytest.warns(FutureWarning) as record: decorated_func(1, 2, c=3) # type: ignore decorated_func(1, b=2, c=3) # type: ignore decorated_func(a=1, b=2, c=3) # No warning. assert len(record) == 2 for warn in record.list: assert isinstance(warn.message, FutureWarning) assert _sample_func.__name__ in str(warn.message) def test_convert_positional_args_mypy_type_inference() -> None: previous_positional_arg_names: List[str] = [] decorator_converter = convert_positional_args( previous_positional_arg_names=previous_positional_arg_names ) assert callable(decorator_converter) class _Sample: def __init__(self) -> None: pass def method(self) -> bool: return True def _func_sample() -> _Sample: return _Sample() def _func_none() -> None: pass ret_none = decorator_converter(_func_none)() assert ret_none is None ret_sample = decorator_converter(_func_sample)() assert isinstance(ret_sample, _Sample) assert ret_sample.method() @pytest.mark.parametrize( "previous_positional_arg_names, raise_error", [(["a", "b", "c", "d"], True), (["a", "d"], True), (["b", "a"], False)], ) def test_convert_positional_args_invalid_previous_positional_arg_names( previous_positional_arg_names: List[str], raise_error: bool ) -> None: decorator_converter = convert_positional_args( previous_positional_arg_names=previous_positional_arg_names ) assert callable(decorator_converter) if raise_error: with pytest.raises(AssertionError) as record: decorator_converter(_sample_func) res = re.findall(r"({.+?}|set\(\))", str(record.value)) assert len(res) == 2 assert eval(res[0]) == set(previous_positional_arg_names) assert eval(res[1]) == set(["a", "b", "c"]) else: decorator_converter(_sample_func) def test_convert_positional_args_invalid_positional_args() -> None: previous_positional_arg_names: List[str] = ["a", "b"] decorator_converter = convert_positional_args( previous_positional_arg_names=previous_positional_arg_names ) assert callable(decorator_converter) decorated_func = decorator_converter(_sample_func) with pytest.warns(FutureWarning): with pytest.raises(TypeError) as record: decorated_func(1, 2, 3) # type: ignore assert str(record.value) == "_sample_func() takes 2 positional arguments but 3 were given." with pytest.raises(TypeError) as record: decorated_func(1, 3, b=2) # type: ignore assert str(record.value) == "_sample_func() got multiple values for arguments {'b'}." optuna-4.1.0/tests/test_deprecated.py000066400000000000000000000150221471332314300176760ustar00rootroot00000000000000from typing import Any from typing import Optional import pytest from optuna import _deprecated class _Sample: def __init__(self, a: Any, b: Any, c: Any) -> None: pass def _method(self) -> None: """summary detail """ pass def _method_expected(self) -> None: """summary detail .. warning:: Deprecated in v1.1.0. This feature will be removed in the future. The removal of this feature is currently scheduled for v3.0.0, but this schedule is subject to change. See https://github.com/optuna/optuna/releases/tag/v1.1.0. """ pass @pytest.mark.parametrize("deprecated_version", ["1.1", 100, None, "2.0.0"]) @pytest.mark.parametrize("removed_version", ["1.1", 10, "1.0.0"]) def test_deprecation_raises_error_for_invalid_version( deprecated_version: Any, removed_version: Any ) -> None: with pytest.raises(ValueError): _deprecated.deprecated_func(deprecated_version, removed_version) with pytest.raises(ValueError): _deprecated.deprecated_class(deprecated_version, removed_version) def test_deprecation_decorator() -> None: deprecated_version = "1.1.0" removed_version = "3.0.0" decorator_deprecation = _deprecated.deprecated_func(deprecated_version, removed_version) assert callable(decorator_deprecation) def _func() -> int: return 10 decorated_func = decorator_deprecation(_func) assert decorated_func.__name__ == _func.__name__ assert decorated_func.__doc__ == _deprecated._DEPRECATION_NOTE_TEMPLATE.format( d_ver=deprecated_version, r_ver=removed_version ) with pytest.warns(FutureWarning): decorated_func() def test_deprecation_instance_method_decorator() -> None: deprecated_version = "1.1.0" removed_version = "3.0.0" decorator_deprecation = _deprecated.deprecated_func(deprecated_version, removed_version) assert callable(decorator_deprecation) decorated_method = decorator_deprecation(_Sample._method) assert decorated_method.__name__ == _Sample._method.__name__ assert decorated_method.__doc__ == _Sample._method_expected.__doc__ with pytest.warns(FutureWarning): decorated_method(None) # type: ignore def test_deprecation_class_decorator() -> None: deprecated_version = "1.1.0" removed_version = "3.0.0" decorator_deprecation = _deprecated.deprecated_class(deprecated_version, removed_version) assert callable(decorator_deprecation) decorated_class = decorator_deprecation(_Sample) assert decorated_class.__name__ == "_Sample" assert decorated_class.__init__.__name__ == "__init__" assert decorated_class.__doc__ == _deprecated._DEPRECATION_NOTE_TEMPLATE.format( d_ver=deprecated_version, r_ver=removed_version ) with pytest.warns(FutureWarning): decorated_class("a", "b", "c") def test_deprecation_class_decorator_name() -> None: name = "foo" decorator_deprecation = _deprecated.deprecated_class("1.1.0", "3.0.0", name=name) decorated_sample = decorator_deprecation(_Sample) with pytest.warns(FutureWarning) as record: decorated_sample("a", "b", "c") assert isinstance(record.list[0].message, Warning) assert name in record.list[0].message.args[0] def test_deprecation_decorator_name() -> None: def _func() -> int: return 10 name = "bar" decorator_deprecation = _deprecated.deprecated_func("1.1.0", "3.0.0", name=name) decorated_sample_func = decorator_deprecation(_func) with pytest.warns(FutureWarning) as record: decorated_sample_func() assert isinstance(record.list[0].message, Warning) assert name in record.list[0].message.args[0] @pytest.mark.parametrize("text", [None, "", "test", "test" * 100]) def test_deprecation_text_specified(text: Optional[str]) -> None: def _func() -> int: return 10 decorator_deprecation = _deprecated.deprecated_func("1.1.0", "3.0.0", text=text) decorated_func = decorator_deprecation(_func) expected_func_doc = _deprecated._DEPRECATION_NOTE_TEMPLATE.format(d_ver="1.1.0", r_ver="3.0.0") if text is None: pass elif len(text) > 0: expected_func_doc += "\n\n " + text + "\n" else: expected_func_doc += "\n\n\n" assert decorated_func.__name__ == _func.__name__ assert decorated_func.__doc__ == expected_func_doc with pytest.warns(FutureWarning) as record: decorated_func() assert isinstance(record.list[0].message, Warning) expected_warning_message = _deprecated._DEPRECATION_WARNING_TEMPLATE.format( name="_func", d_ver="1.1.0", r_ver="3.0.0" ) if text is not None: expected_warning_message += " " + text assert record.list[0].message.args[0] == expected_warning_message @pytest.mark.parametrize("text", [None, "", "test", "test" * 100]) def test_deprecation_class_text_specified(text: Optional[str]) -> None: class _Class: def __init__(self, a: Any, b: Any, c: Any) -> None: pass decorator_deprecation = _deprecated.deprecated_class("1.1.0", "3.0.0", text=text) decorated_class = decorator_deprecation(_Class) expected_class_doc = _deprecated._DEPRECATION_NOTE_TEMPLATE.format( d_ver="1.1.0", r_ver="3.0.0" ) if text is None: pass elif len(text) > 0: expected_class_doc += "\n\n " + text + "\n" else: expected_class_doc += "\n\n\n" assert decorated_class.__name__ == _Class.__name__ assert decorated_class.__doc__ == expected_class_doc with pytest.warns(FutureWarning) as record: decorated_class(None, None, None) assert isinstance(record.list[0].message, Warning) expected_warning_message = _deprecated._DEPRECATION_WARNING_TEMPLATE.format( name="_Class", d_ver="1.1.0", r_ver="3.0.0" ) if text is not None: expected_warning_message += " " + text assert record.list[0].message.args[0] == expected_warning_message def test_deprecation_decorator_default_removed_version() -> None: deprecated_version = "1.1.0" removed_version = "3.0.0" decorator_deprecation = _deprecated.deprecated_func(deprecated_version, removed_version) assert callable(decorator_deprecation) def _func() -> int: return 10 decorated_func = decorator_deprecation(_func) assert decorated_func.__name__ == _func.__name__ assert decorated_func.__doc__ == _deprecated._DEPRECATION_NOTE_TEMPLATE.format( d_ver=deprecated_version, r_ver=removed_version ) with pytest.warns(FutureWarning): decorated_func() optuna-4.1.0/tests/test_distributions.py000066400000000000000000000560521471332314300205100ustar00rootroot00000000000000import copy import json from typing import Any from typing import cast from typing import Dict from typing import Optional import warnings import numpy as np import pytest from optuna import distributions _choices = (None, True, False, 0, 1, 0.0, 1.0, float("nan"), float("inf"), -float("inf"), "", "a") _choices_json = '[null, true, false, 0, 1, 0.0, 1.0, NaN, Infinity, -Infinity, "", "a"]' EXAMPLE_DISTRIBUTIONS: Dict[str, Any] = { "i": distributions.IntDistribution(low=1, high=9, log=False), # i2 and i3 are identical to i, and tested for cases when `log` and `step` are omitted in json. "i2": distributions.IntDistribution(low=1, high=9, log=False), "i3": distributions.IntDistribution(low=1, high=9, log=False), "il": distributions.IntDistribution(low=2, high=12, log=True), "il2": distributions.IntDistribution(low=2, high=12, log=True), "id": distributions.IntDistribution(low=1, high=9, log=False, step=2), "id2": distributions.IntDistribution(low=1, high=9, log=False, step=2), "f": distributions.FloatDistribution(low=1.0, high=2.0, log=False), "fl": distributions.FloatDistribution(low=0.001, high=100.0, log=True), "fd": distributions.FloatDistribution(low=1.0, high=9.0, log=False, step=2.0), "c1": distributions.CategoricalDistribution(choices=_choices), "c2": distributions.CategoricalDistribution(choices=("Roppongi", "Azabu")), "c3": distributions.CategoricalDistribution(choices=["Roppongi", "Azabu"]), } EXAMPLE_JSONS = { "i": '{"name": "IntDistribution", "attributes": {"low": 1, "high": 9}}', "i2": '{"name": "IntDistribution", "attributes": {"low": 1, "high": 9, "log": false}}', "i3": '{"name": "IntDistribution", ' '"attributes": {"low": 1, "high": 9, "log": false, "step": 1}}', "il": '{"name": "IntDistribution", ' '"attributes": {"low": 2, "high": 12, "log": true}}', "il2": '{"name": "IntDistribution", ' '"attributes": {"low": 2, "high": 12, "log": true, "step": 1}}', "id": '{"name": "IntDistribution", ' '"attributes": {"low": 1, "high": 9, "step": 2}}', "id2": '{"name": "IntDistribution", ' '"attributes": {"low": 1, "high": 9, "log": false, "step": 2}}', "f": '{"name": "FloatDistribution", ' '"attributes": {"low": 1.0, "high": 2.0, "log": false, "step": null}}', "fl": '{"name": "FloatDistribution", ' '"attributes": {"low": 0.001, "high": 100.0, "log": true, "step": null}}', "fd": '{"name": "FloatDistribution", ' '"attributes": {"low": 1.0, "high": 9.0, "step": 2.0, "log": false}}', "c1": f'{{"name": "CategoricalDistribution", "attributes": {{"choices": {_choices_json}}}}}', "c2": '{"name": "CategoricalDistribution", "attributes": {"choices": ["Roppongi", "Azabu"]}}', "c3": '{"name": "CategoricalDistribution", "attributes": {"choices": ["Roppongi", "Azabu"]}}', } EXAMPLE_ABBREVIATED_JSONS = { "i": '{"type": "int", "low": 1, "high": 9}', "i2": '{"type": "int", "low": 1, "high": 9, "log": false}', "i3": '{"type": "int", "low": 1, "high": 9, "log": false, "step": 1}', "il": '{"type": "int", "low": 2, "high": 12, "log": true}', "il2": '{"type": "int", "low": 2, "high": 12, "log": true, "step": 1}', "id": '{"type": "int", "low": 1, "high": 9, "step": 2}', "id2": '{"type": "int", "low": 1, "high": 9, "log": false, "step": 2}', "f": '{"type": "float", "low": 1.0, "high": 2.0, "log": false, "step": null}', "fl": '{"type": "float", "low": 0.001, "high": 100, "log": true, "step": null}', "fd": '{"type": "float", "low": 1.0, "high": 9.0, "log": false, "step": 2.0}', "c1": f'{{"type": "categorical", "choices": {_choices_json}}}', "c2": '{"type": "categorical", "choices": ["Roppongi", "Azabu"]}', "c3": '{"type": "categorical", "choices": ["Roppongi", "Azabu"]}', } def test_json_to_distribution() -> None: for key in EXAMPLE_JSONS: distribution_actual = distributions.json_to_distribution(EXAMPLE_JSONS[key]) assert distribution_actual == EXAMPLE_DISTRIBUTIONS[key] unknown_json = '{"name": "UnknownDistribution", "attributes": {"low": 1.0, "high": 2.0}}' pytest.raises(ValueError, lambda: distributions.json_to_distribution(unknown_json)) def test_abbreviated_json_to_distribution() -> None: for key in EXAMPLE_ABBREVIATED_JSONS: distribution_actual = distributions.json_to_distribution(EXAMPLE_ABBREVIATED_JSONS[key]) assert distribution_actual == EXAMPLE_DISTRIBUTIONS[key] unknown_json = '{"type": "unknown", "low": 1.0, "high": 2.0}' pytest.raises(ValueError, lambda: distributions.json_to_distribution(unknown_json)) invalid_distribution = ( '{"type": "float", "low": 0.0, "high": -100.0}', '{"type": "float", "low": 7.3, "high": 7.2, "log": true}', '{"type": "float", "low": -30.0, "high": -40.0, "step": 3.0}', '{"type": "float", "low": 1.0, "high": 100.0, "step": 0.0}', '{"type": "float", "low": 1.0, "high": 100.0, "step": -1.0}', '{"type": "int", "low": 123, "high": 100}', '{"type": "int", "low": 123, "high": 100, "step": 2}', '{"type": "int", "low": 123, "high": 100, "log": true}', '{"type": "int", "low": 1, "high": 100, "step": 0}', '{"type": "int", "low": 1, "high": 100, "step": -1}', '{"type": "categorical", "choices": []}', ) for distribution in invalid_distribution: pytest.raises(ValueError, lambda: distributions.json_to_distribution(distribution)) def test_distribution_to_json() -> None: for key in EXAMPLE_JSONS: json_actual = json.loads(distributions.distribution_to_json(EXAMPLE_DISTRIBUTIONS[key])) json_expect = json.loads(EXAMPLE_JSONS[key]) if json_expect["name"] == "IntDistribution" and "step" not in json_expect["attributes"]: json_expect["attributes"]["step"] = 1 if json_expect["name"] == "IntDistribution" and "log" not in json_expect["attributes"]: json_expect["attributes"]["log"] = False assert json_actual == json_expect def test_check_distribution_compatibility() -> None: # Test the same distribution. for key in EXAMPLE_JSONS: distributions.check_distribution_compatibility( EXAMPLE_DISTRIBUTIONS[key], EXAMPLE_DISTRIBUTIONS[key] ) # We need to create new objects to compare NaNs. # See https://github.com/optuna/optuna/pull/3567#pullrequestreview-974939837. distributions.check_distribution_compatibility( EXAMPLE_DISTRIBUTIONS[key], distributions.json_to_distribution(EXAMPLE_JSONS[key]) ) # Test different distribution classes. pytest.raises( ValueError, lambda: distributions.check_distribution_compatibility( EXAMPLE_DISTRIBUTIONS["i"], EXAMPLE_DISTRIBUTIONS["fl"] ), ) # Test compatibility between IntDistributions. distributions.check_distribution_compatibility( EXAMPLE_DISTRIBUTIONS["id"], EXAMPLE_DISTRIBUTIONS["i"] ) with pytest.raises(ValueError): distributions.check_distribution_compatibility( EXAMPLE_DISTRIBUTIONS["i"], EXAMPLE_DISTRIBUTIONS["il"] ) with pytest.raises(ValueError): distributions.check_distribution_compatibility( EXAMPLE_DISTRIBUTIONS["il"], EXAMPLE_DISTRIBUTIONS["id"] ) # Test compatibility between FloatDistributions. distributions.check_distribution_compatibility( EXAMPLE_DISTRIBUTIONS["fd"], EXAMPLE_DISTRIBUTIONS["f"] ) with pytest.raises(ValueError): distributions.check_distribution_compatibility( EXAMPLE_DISTRIBUTIONS["f"], EXAMPLE_DISTRIBUTIONS["fl"] ) with pytest.raises(ValueError): distributions.check_distribution_compatibility( EXAMPLE_DISTRIBUTIONS["fl"], EXAMPLE_DISTRIBUTIONS["fd"] ) # Test dynamic value range (CategoricalDistribution). pytest.raises( ValueError, lambda: distributions.check_distribution_compatibility( EXAMPLE_DISTRIBUTIONS["c2"], distributions.CategoricalDistribution(choices=("Roppongi", "Akasaka")), ), ) # Test dynamic value range (except CategoricalDistribution). distributions.check_distribution_compatibility( EXAMPLE_DISTRIBUTIONS["i"], distributions.IntDistribution(low=-3, high=2) ) distributions.check_distribution_compatibility( EXAMPLE_DISTRIBUTIONS["il"], distributions.IntDistribution(low=1, high=13, log=True) ) distributions.check_distribution_compatibility( EXAMPLE_DISTRIBUTIONS["id"], distributions.IntDistribution(low=-3, high=1, step=2) ) distributions.check_distribution_compatibility( EXAMPLE_DISTRIBUTIONS["f"], distributions.FloatDistribution(low=-3.0, high=-2.0) ) distributions.check_distribution_compatibility( EXAMPLE_DISTRIBUTIONS["fl"], distributions.FloatDistribution(low=0.1, high=1.0, log=True) ) distributions.check_distribution_compatibility( EXAMPLE_DISTRIBUTIONS["fd"], distributions.FloatDistribution(low=-1.0, high=11.0, step=0.5) ) @pytest.mark.parametrize("value", (0, 1, 4, 10, 11, 1.1, "1", "1.1", "-1.0", True, False)) def test_int_internal_representation(value: Any) -> None: i = distributions.IntDistribution(low=1, high=10) if isinstance(value, int): expected_value = value else: expected_value = int(float(value)) assert i.to_external_repr(i.to_internal_repr(value)) == expected_value @pytest.mark.parametrize( "value, kwargs", [ ("foo", {}), ((), {}), ([], {}), ({}, {}), (set(), {}), (np.ones(2), {}), (np.nan, {}), (0, dict(log=True)), (-1, dict(log=True)), ], ) def test_int_internal_representation_error(value: Any, kwargs: Dict[str, Any]) -> None: i = distributions.IntDistribution(low=1, high=10, **kwargs) with pytest.raises(ValueError): i.to_internal_repr(value) @pytest.mark.parametrize( "value", (1.99, 2.0, 4.5, 7, 7.1, 1, "1", "1.1", "-1.0", True, False), ) def test_float_internal_representation(value: Any) -> None: f = distributions.FloatDistribution(low=2.0, high=7.0) if isinstance(value, float): expected_value = value else: expected_value = float(value) assert f.to_external_repr(f.to_internal_repr(value)) == expected_value @pytest.mark.parametrize( "value, kwargs", [ ("foo", {}), ((), {}), ([], {}), ({}, {}), (set(), {}), (np.ones(2), {}), (np.nan, {}), (0.0, dict(log=True)), (-1.0, dict(log=True)), ], ) def test_float_internal_representation_error(value: Any, kwargs: Dict[str, Any]) -> None: f = distributions.FloatDistribution(low=2.0, high=7.0, **kwargs) with pytest.raises(ValueError): f.to_internal_repr(value) def test_categorical_internal_representation() -> None: c = EXAMPLE_DISTRIBUTIONS["c1"] for choice in c.choices: if isinstance(choice, float) and np.isnan(choice): assert np.isnan(c.to_external_repr(c.to_internal_repr(choice))) else: assert c.to_external_repr(c.to_internal_repr(choice)) == choice # We need to create new objects to compare NaNs. # See https://github.com/optuna/optuna/pull/3567#pullrequestreview-974939837. c_ = distributions.json_to_distribution(EXAMPLE_JSONS["c1"]) for choice in cast(distributions.CategoricalDistribution, c_).choices: if isinstance(choice, float) and np.isnan(choice): assert np.isnan(c.to_external_repr(c.to_internal_repr(choice))) else: assert c.to_external_repr(c.to_internal_repr(choice)) == choice @pytest.mark.parametrize( ("expected", "value", "step"), [ (False, 0.9, 1), (True, 1, 1), (False, 1.5, 1), (True, 4, 1), (True, 10, 1), (False, 11, 1), (False, 10, 2), (True, 1, 3), (False, 5, 3), (True, 10, 3), ], ) def test_int_contains(expected: bool, value: float, step: int) -> None: with warnings.catch_warnings(): # When `step` is 2, UserWarning will be raised since the range is not divisible by 2. # The range will be replaced with [1, 9]. warnings.simplefilter("ignore", category=UserWarning) i = distributions.IntDistribution(low=1, high=10, step=step) assert i._contains(value) == expected @pytest.mark.parametrize( ("expected", "value", "step"), [ (False, 1.99, None), (True, 2.0, None), (True, 2.5, None), (True, 7, None), (False, 7.1, None), (False, 0.99, 2.0), (True, 2.0, 2.0), (False, 3.0, 2.0), (True, 6, 2.0), (False, 6.1, 2.0), ], ) def test_float_contains(expected: bool, value: float, step: Optional[float]) -> None: with warnings.catch_warnings(): # When `step` is 2.0, UserWarning will be raised since the range is not divisible by 2. # The range will be replaced with [2.0, 6.0]. warnings.simplefilter("ignore", category=UserWarning) f = distributions.FloatDistribution(low=2.0, high=7.0, step=step) assert f._contains(value) == expected def test_categorical_contains() -> None: c = distributions.CategoricalDistribution(choices=("Roppongi", "Azabu")) assert not c._contains(-1) assert c._contains(0) assert c._contains(1) assert c._contains(1.5) assert not c._contains(3) def test_empty_range_contains() -> None: i = distributions.IntDistribution(low=1, high=1) assert not i._contains(0) assert i._contains(1) assert not i._contains(2) iq = distributions.IntDistribution(low=1, high=1, step=2) assert not iq._contains(0) assert iq._contains(1) assert not iq._contains(2) il = distributions.IntDistribution(low=1, high=1, log=True) assert not il._contains(0) assert il._contains(1) assert not il._contains(2) f = distributions.FloatDistribution(low=1.0, high=1.0) assert not f._contains(0.9) assert f._contains(1.0) assert not f._contains(1.1) fd = distributions.FloatDistribution(low=1.0, high=1.0, step=2.0) assert not fd._contains(0.9) assert fd._contains(1.0) assert not fd._contains(1.1) fl = distributions.FloatDistribution(low=1.0, high=1.0, log=True) assert not fl._contains(0.9) assert fl._contains(1.0) assert not fl._contains(1.1) @pytest.mark.parametrize( ("expected", "low", "high", "log", "step"), [ (True, 1, 1, False, 1), (True, 3, 3, False, 2), (True, 2, 2, True, 1), (False, -123, 0, False, 1), (False, -123, 0, False, 123), (False, 2, 4, True, 1), ], ) def test_int_single(expected: bool, low: int, high: int, log: bool, step: int) -> None: distribution = distributions.IntDistribution(low=low, high=high, log=log, step=step) assert distribution.single() == expected @pytest.mark.parametrize( ("expected", "low", "high", "log", "step"), [ (True, 2.0, 2.0, False, None), (True, 2.0, 2.0, True, None), (True, 2.22, 2.22, False, 0.1), (True, 2.22, 2.24, False, 0.3), (False, 1.0, 1.001, False, None), (False, 7.3, 10.0, True, None), (False, -30, -20, False, 2), (False, -30, -20, False, 10), # In Python, "0.3 - 0.2 != 0.1" is True. (False, 0.2, 0.3, False, 0.1), (False, 0.7, 0.8, False, 0.1), ], ) def test_float_single( expected: bool, low: float, high: float, log: bool, step: Optional[float] ) -> None: with warnings.catch_warnings(): # When `step` is 0.3, UserWarning will be raised since the range is not divisible by 0.3. # The range will be replaced with [2.22, 2.24]. warnings.simplefilter("ignore", category=UserWarning) distribution = distributions.FloatDistribution(low=low, high=high, log=log, step=step) assert distribution.single() == expected def test_categorical_single() -> None: assert distributions.CategoricalDistribution(choices=("foo",)).single() assert not distributions.CategoricalDistribution(choices=("foo", "bar")).single() def test_invalid_distribution() -> None: with pytest.warns(UserWarning): distributions.CategoricalDistribution(choices=({"foo": "bar"},)) # type: ignore def test_eq_ne_hash() -> None: # Two instances of a class are regarded as equivalent if the fields have the same values. for d in EXAMPLE_DISTRIBUTIONS.values(): d_copy = copy.deepcopy(d) assert d == d_copy assert hash(d) == hash(d_copy) # Different field values. d0 = distributions.FloatDistribution(low=1, high=2) d1 = distributions.FloatDistribution(low=1, high=3) assert d0 != d1 # Different distribution classes. d2 = distributions.IntDistribution(low=1, high=2) assert d0 != d2 def test_repr() -> None: # The following variables are needed to apply `eval` to distribution # instances that contain `float('nan')` or `float('inf')` as a field value. nan = float("nan") # NOQA inf = float("inf") # NOQA for d in EXAMPLE_DISTRIBUTIONS.values(): assert d == eval("distributions." + repr(d)) @pytest.mark.parametrize( ("key", "low", "high", "log", "step"), [ ("i", 1, 9, False, 1), ("il", 2, 12, True, 1), ("id", 1, 9, False, 2), ], ) def test_int_distribution_asdict(key: str, low: int, high: int, log: bool, step: int) -> None: expected_dict = {"low": low, "high": high, "log": log, "step": step} assert EXAMPLE_DISTRIBUTIONS[key]._asdict() == expected_dict @pytest.mark.parametrize( ("key", "low", "high", "log", "step"), [ ("f", 1.0, 2.0, False, None), ("fl", 0.001, 100.0, True, None), ("fd", 1.0, 9.0, False, 2.0), ], ) def test_float_distribution_asdict( key: str, low: float, high: float, log: bool, step: Optional[float] ) -> None: expected_dict = {"low": low, "high": high, "log": log, "step": step} assert EXAMPLE_DISTRIBUTIONS[key]._asdict() == expected_dict def test_int_init_error() -> None: # Empty distributions cannot be instantiated. with pytest.raises(ValueError): distributions.IntDistribution(low=123, high=100) with pytest.raises(ValueError): distributions.IntDistribution(low=100, high=10, log=True) with pytest.raises(ValueError): distributions.IntDistribution(low=123, high=100, step=2) # 'step' must be 1 when 'log' is True. with pytest.raises(ValueError): distributions.IntDistribution(low=1, high=100, log=True, step=2) # 'step' should be positive. with pytest.raises(ValueError): distributions.IntDistribution(low=1, high=100, step=0) with pytest.raises(ValueError): distributions.IntDistribution(low=1, high=10, step=-1) def test_float_init_error() -> None: # Empty distributions cannot be instantiated. with pytest.raises(ValueError): distributions.FloatDistribution(low=0.0, high=-100.0) with pytest.raises(ValueError): distributions.FloatDistribution(low=7.3, high=7.2, log=True) with pytest.raises(ValueError): distributions.FloatDistribution(low=-30.0, high=-40.0, step=2.5) # 'step' must be None when 'log' is True. with pytest.raises(ValueError): distributions.FloatDistribution(low=1.0, high=100.0, log=True, step=0.5) # 'step' should be positive. with pytest.raises(ValueError): distributions.FloatDistribution(low=1.0, high=10.0, step=0) with pytest.raises(ValueError): distributions.FloatDistribution(low=1.0, high=100.0, step=-1) def test_categorical_init_error() -> None: with pytest.raises(ValueError): distributions.CategoricalDistribution(choices=()) def test_categorical_distribution_different_sequence_types() -> None: c1 = distributions.CategoricalDistribution(choices=("Roppongi", "Azabu")) c2 = distributions.CategoricalDistribution(choices=["Roppongi", "Azabu"]) assert c1 == c2 @pytest.mark.filterwarnings("ignore::FutureWarning") def test_convert_old_distribution_to_new_distribution() -> None: ud = distributions.UniformDistribution(low=0, high=10) assert distributions._convert_old_distribution_to_new_distribution( ud ) == distributions.FloatDistribution(low=0, high=10, log=False, step=None) dud = distributions.DiscreteUniformDistribution(low=0, high=10, q=2) assert distributions._convert_old_distribution_to_new_distribution( dud ) == distributions.FloatDistribution(low=0, high=10, log=False, step=2) lud = distributions.LogUniformDistribution(low=1, high=10) assert distributions._convert_old_distribution_to_new_distribution( lud ) == distributions.FloatDistribution(low=1, high=10, log=True, step=None) id = distributions.IntUniformDistribution(low=0, high=10) assert distributions._convert_old_distribution_to_new_distribution( id ) == distributions.IntDistribution(low=0, high=10, log=False, step=1) idd = distributions.IntUniformDistribution(low=0, high=10, step=2) assert distributions._convert_old_distribution_to_new_distribution( idd ) == distributions.IntDistribution(low=0, high=10, log=False, step=2) ild = distributions.IntLogUniformDistribution(low=1, high=10) assert distributions._convert_old_distribution_to_new_distribution( ild ) == distributions.IntDistribution(low=1, high=10, log=True, step=1) def test_convert_old_distribution_to_new_distribution_noop() -> None: # No conversion happens for CategoricalDistribution. cd = distributions.CategoricalDistribution(choices=["a", "b", "c"]) assert distributions._convert_old_distribution_to_new_distribution(cd) == cd # No conversion happens for new distributions. fd = distributions.FloatDistribution(low=0, high=10, log=False, step=None) assert distributions._convert_old_distribution_to_new_distribution(fd) == fd dfd = distributions.FloatDistribution(low=0, high=10, log=False, step=2) assert distributions._convert_old_distribution_to_new_distribution(dfd) == dfd lfd = distributions.FloatDistribution(low=1, high=10, log=True, step=None) assert distributions._convert_old_distribution_to_new_distribution(lfd) == lfd id = distributions.IntDistribution(low=0, high=10) assert distributions._convert_old_distribution_to_new_distribution(id) == id idd = distributions.IntDistribution(low=0, high=10, step=2) assert distributions._convert_old_distribution_to_new_distribution(idd) == idd ild = distributions.IntDistribution(low=1, high=10, log=True) assert distributions._convert_old_distribution_to_new_distribution(ild) == ild def test_is_distribution_log() -> None: lfd = distributions.FloatDistribution(low=1, high=10, log=True) assert distributions._is_distribution_log(lfd) lid = distributions.IntDistribution(low=1, high=10, log=True) assert distributions._is_distribution_log(lid) fd = distributions.FloatDistribution(low=0, high=10, log=False) assert not distributions._is_distribution_log(fd) id = distributions.IntDistribution(low=0, high=10, log=False) assert not distributions._is_distribution_log(id) cd = distributions.CategoricalDistribution(choices=["a", "b", "c"]) assert not distributions._is_distribution_log(cd) optuna-4.1.0/tests/test_experimental.py000066400000000000000000000063161471332314300203010ustar00rootroot00000000000000from typing import Any import pytest from optuna import _experimental from optuna.exceptions import ExperimentalWarning def _sample_func() -> int: return 10 class _Sample: def __init__(self, a: Any, b: Any, c: Any) -> None: pass def _method(self) -> None: """summary detail """ pass def _method_expected(self) -> None: """summary detail .. note:: Added in v1.1.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v1.1.0. """ pass @pytest.mark.parametrize("version", ["1.1", 100, None]) def test_experimental_raises_error_for_invalid_version(version: Any) -> None: with pytest.raises(ValueError): _experimental.experimental_func(version) with pytest.raises(ValueError): _experimental.experimental_class(version) def test_experimental_func_decorator() -> None: version = "1.1.0" decorator_experimental = _experimental.experimental_func(version) assert callable(decorator_experimental) decorated_func = decorator_experimental(_sample_func) assert decorated_func.__name__ == _sample_func.__name__ assert decorated_func.__doc__ == _experimental._EXPERIMENTAL_NOTE_TEMPLATE.format(ver=version) with pytest.warns(ExperimentalWarning): decorated_func() def test_experimental_instance_method_decorator() -> None: version = "1.1.0" decorator_experimental = _experimental.experimental_func(version) assert callable(decorator_experimental) decorated_method = decorator_experimental(_Sample._method) assert decorated_method.__name__ == _Sample._method.__name__ assert decorated_method.__doc__ == _Sample._method_expected.__doc__ with pytest.warns(ExperimentalWarning): decorated_method(None) # type: ignore def test_experimental_class_decorator() -> None: version = "1.1.0" decorator_experimental = _experimental.experimental_class(version) assert callable(decorator_experimental) decorated_class = decorator_experimental(_Sample) assert decorated_class.__name__ == "_Sample" assert decorated_class.__init__.__name__ == "__init__" assert decorated_class.__doc__ == _experimental._EXPERIMENTAL_NOTE_TEMPLATE.format(ver=version) with pytest.warns(ExperimentalWarning): decorated_class("a", "b", "c") def test_experimental_class_decorator_name() -> None: name = "foo" decorator_experimental = _experimental.experimental_class("1.1.0", name=name) decorated_sample = decorator_experimental(_Sample) with pytest.warns(ExperimentalWarning) as record: decorated_sample("a", "b", "c") assert isinstance(record.list[0].message, Warning) assert name in record.list[0].message.args[0] def test_experimental_decorator_name() -> None: name = "bar" decorator_experimental = _experimental.experimental_func("1.1.0", name=name) decorated_sample_func = decorator_experimental(_sample_func) with pytest.warns(ExperimentalWarning) as record: decorated_sample_func() assert isinstance(record.list[0].message, Warning) assert name in record.list[0].message.args[0] optuna-4.1.0/tests/test_imports.py000066400000000000000000000015301471332314300172720ustar00rootroot00000000000000import pytest from optuna._imports import try_import def test_try_import_is_successful() -> None: with try_import() as imports: pass assert imports.is_successful() imports.check() def test_try_import_is_successful_other_error() -> None: with pytest.raises(NotImplementedError): with try_import() as imports: raise NotImplementedError assert imports.is_successful() # No imports failed so `imports` is successful. imports.check() def test_try_import_not_successful() -> None: with try_import() as imports: raise ImportError assert not imports.is_successful() with pytest.raises(ImportError): imports.check() with try_import() as imports: raise SyntaxError assert not imports.is_successful() with pytest.raises(ImportError): imports.check() optuna-4.1.0/tests/test_logging.py000066400000000000000000000062021471332314300172240ustar00rootroot00000000000000import logging import _pytest.capture import _pytest.logging import optuna.logging def test_get_logger(caplog: _pytest.logging.LogCaptureFixture) -> None: # Log propagation is necessary for caplog to capture log outputs. optuna.logging.enable_propagation() logger = optuna.logging.get_logger("optuna.foo") with caplog.at_level(logging.INFO, logger="optuna.foo"): logger.info("hello") assert "hello" in caplog.text def test_default_handler(capsys: _pytest.capture.CaptureFixture) -> None: # We need to reconstruct our default handler to properly capture stderr. optuna.logging._reset_library_root_logger() optuna.logging.set_verbosity(optuna.logging.INFO) library_root_logger = optuna.logging._get_library_root_logger() example_logger = optuna.logging.get_logger("optuna.bar") # Default handler enabled optuna.logging.enable_default_handler() assert library_root_logger.handlers example_logger.info("hey") _, err = capsys.readouterr() assert "hey" in err # Default handler disabled optuna.logging.disable_default_handler() assert not library_root_logger.handlers example_logger.info("yoyo") _, err = capsys.readouterr() assert "yoyo" not in err def test_verbosity(capsys: _pytest.capture.CaptureFixture) -> None: # We need to reconstruct our default handler to properly capture stderr. optuna.logging._reset_library_root_logger() library_root_logger = optuna.logging._get_library_root_logger() example_logger = optuna.logging.get_logger("optuna.hoge") optuna.logging.enable_default_handler() # level INFO optuna.logging.set_verbosity(optuna.logging.INFO) assert library_root_logger.getEffectiveLevel() == logging.INFO example_logger.warning("hello-warning") example_logger.info("hello-info") example_logger.debug("hello-debug") _, err = capsys.readouterr() assert "hello-warning" in err assert "hello-info" in err assert "hello-debug" not in err # level WARNING optuna.logging.set_verbosity(optuna.logging.WARNING) assert library_root_logger.getEffectiveLevel() == logging.WARNING example_logger.warning("bye-warning") example_logger.info("bye-info") example_logger.debug("bye-debug") _, err = capsys.readouterr() assert "bye-warning" in err assert "bye-info" not in err assert "bye-debug" not in err def test_propagation(caplog: _pytest.logging.LogCaptureFixture) -> None: optuna.logging._reset_library_root_logger() logger = optuna.logging.get_logger("optuna.foo") # Propagation is disabled by default. with caplog.at_level(logging.INFO, logger="optuna"): logger.info("no-propagation") assert "no-propagation" not in caplog.text # Enable propagation. optuna.logging.enable_propagation() with caplog.at_level(logging.INFO, logger="optuna"): logger.info("enable-propagate") assert "enable-propagate" in caplog.text # Disable propagation. optuna.logging.disable_propagation() with caplog.at_level(logging.INFO, logger="optuna"): logger.info("disable-propagation") assert "disable-propagation" not in caplog.text optuna-4.1.0/tests/test_multi_objective.py000066400000000000000000000056211471332314300207660ustar00rootroot00000000000000from __future__ import annotations from typing import Sequence import pytest from optuna import create_study from optuna import trial from optuna.study import StudyDirection from optuna.study._multi_objective import _get_pareto_front_trials_by_trials from optuna.trial import FrozenTrial def _trial_to_values(t: FrozenTrial) -> tuple[float, ...]: assert t.values is not None return tuple(t.values) def assert_is_output_equal_to_ans( trials: list[FrozenTrial], directions: Sequence[StudyDirection], ans: set[tuple[int]] ) -> None: res = {_trial_to_values(t) for t in _get_pareto_front_trials_by_trials(trials, directions)} assert res == ans @pytest.mark.parametrize( "directions,values_set,ans_set", [ ( ["minimize", "maximize"], [[2, 2], [1, 1], [3, 1], [3, 2], [1, 3]], [{(2, 2)}] + [{(1, 1), (2, 2)}] * 3 + [{(1, 3)}], ), ( ["minimize", "maximize", "minimize"], [[2, 2, 2], [1, 1, 1], [3, 1, 3], [3, 2, 3], [1, 3, 1]], [{(2, 2, 2)}] + [{(1, 1, 1), (2, 2, 2)}] * 3 + [{(1, 3, 1)}], ), ], ) def test_get_pareto_front_trials( directions: list[str], values_set: list[list[int]], ans_set: list[set[tuple[int]]] ) -> None: study = create_study(directions=directions) assert_is_output_equal_to_ans(study.trials, study.directions, set()) for values, ans in zip(values_set, ans_set): study.optimize(lambda t: values, n_trials=1) assert_is_output_equal_to_ans(study.trials, study.directions, ans) assert len(_get_pareto_front_trials_by_trials(study.trials, study.directions)) == 1 # The trial result is the same as the last one. study.optimize(lambda t: values_set[-1], n_trials=1) assert_is_output_equal_to_ans(study.trials, study.directions, ans_set[-1]) assert len(_get_pareto_front_trials_by_trials(study.trials, study.directions)) == 2 @pytest.mark.parametrize( "directions,values_set,ans_set", [ (["minimize", "maximize"], [[1, 1], [2, 2], [3, 2]], [{(1, 1), (2, 2)}, {(1, 1), (3, 2)}]), ( ["minimize", "maximize", "minimize"], [[1, 1, 1], [2, 2, 2], [3, 2, 3]], [{(1, 1, 1), (2, 2, 2)}, {(1, 1, 1), (3, 2, 3)}], ), ], ) def test_get_pareto_front_trials_with_constraint( directions: list[str], values_set: list[list[int]], ans_set: list[set[tuple[int]]] ) -> None: study = create_study(directions=directions) trials = [ trial.create_trial(values=values, system_attrs={"constraints": [i % 2]}) for i, values in enumerate(values_set) ] study.add_trials(trials) for consider_constraint, ans in zip([False, True], ans_set): trials = study.trials study_dirs = study.directions assert ans == { _trial_to_values(t) for t in _get_pareto_front_trials_by_trials(trials, study_dirs, consider_constraint) } optuna-4.1.0/tests/test_transform.py000066400000000000000000000177341471332314300176250ustar00rootroot00000000000000import math from typing import Any import numpy as np import pytest from optuna._transform import _SearchSpaceTransform from optuna._transform import _untransform_numerical_param from optuna.distributions import BaseDistribution from optuna.distributions import CategoricalDistribution from optuna.distributions import FloatDistribution from optuna.distributions import IntDistribution @pytest.mark.parametrize( "param,distribution", [ (0, IntDistribution(0, 3)), (1, IntDistribution(1, 10, log=True)), (2, IntDistribution(0, 10, step=2)), (0.0, FloatDistribution(0, 3)), (1.0, FloatDistribution(1, 10, log=True)), (0.2, FloatDistribution(0, 1, step=0.2)), ("foo", CategoricalDistribution(["foo"])), ("bar", CategoricalDistribution(["foo", "bar", "baz"])), ], ) def test_search_space_transform_shapes_dtypes(param: Any, distribution: BaseDistribution) -> None: trans = _SearchSpaceTransform({"x0": distribution}) trans_params = trans.transform({"x0": param}) if isinstance(distribution, CategoricalDistribution): expected_bounds_shape = (len(distribution.choices), 2) expected_params_shape = (len(distribution.choices),) else: expected_bounds_shape = (1, 2) expected_params_shape = (1,) assert trans.bounds.shape == expected_bounds_shape assert trans.bounds.dtype == np.float64 assert trans_params.shape == expected_params_shape assert trans_params.dtype == np.float64 def test_search_space_transform_encoding() -> None: trans = _SearchSpaceTransform({"x0": IntDistribution(0, 3)}) assert len(trans.column_to_encoded_columns) == 1 np.testing.assert_equal(trans.column_to_encoded_columns[0], np.array([0])) np.testing.assert_equal(trans.encoded_column_to_column, np.array([0])) trans = _SearchSpaceTransform({"x0": CategoricalDistribution(["foo", "bar", "baz"])}) assert len(trans.column_to_encoded_columns) == 1 np.testing.assert_equal(trans.column_to_encoded_columns[0], np.array([0, 1, 2])) np.testing.assert_equal(trans.encoded_column_to_column, np.array([0, 0, 0])) trans = _SearchSpaceTransform( { "x0": FloatDistribution(0, 3), "x1": CategoricalDistribution(["foo", "bar", "baz"]), "x3": FloatDistribution(0, 1, step=0.2), } ) assert len(trans.column_to_encoded_columns) == 3 np.testing.assert_equal(trans.column_to_encoded_columns[0], np.array([0])) np.testing.assert_equal(trans.column_to_encoded_columns[1], np.array([1, 2, 3])) np.testing.assert_equal(trans.column_to_encoded_columns[2], np.array([4])) np.testing.assert_equal(trans.encoded_column_to_column, np.array([0, 1, 1, 1, 2])) @pytest.mark.parametrize("transform_log", [True, False]) @pytest.mark.parametrize("transform_step", [True, False]) @pytest.mark.parametrize("transform_0_1", [True, False]) @pytest.mark.parametrize( "param,distribution", [ (0, IntDistribution(0, 3)), (3, IntDistribution(0, 3)), (1, IntDistribution(1, 10, log=True)), (10, IntDistribution(1, 10, log=True)), (2, IntDistribution(0, 10, step=2)), (10, IntDistribution(0, 10, step=2)), (0.0, FloatDistribution(0, 3)), (3.0, FloatDistribution(0, 3)), (1.0, FloatDistribution(1, 10, log=True)), (10.0, FloatDistribution(1, 10, log=True)), (0.2, FloatDistribution(0, 1, step=0.2)), (1.0, FloatDistribution(0, 1, step=0.2)), ], ) def test_search_space_transform_numerical( transform_log: bool, transform_step: bool, transform_0_1: bool, param: Any, distribution: BaseDistribution, ) -> None: trans = _SearchSpaceTransform( {"x0": distribution}, transform_log=transform_log, transform_step=transform_step, transform_0_1=transform_0_1, ) if transform_0_1: expected_low = 0.0 expected_high = 1.0 else: expected_low = distribution.low # type: ignore expected_high = distribution.high # type: ignore if isinstance(distribution, FloatDistribution): if transform_log and distribution.log: expected_low = math.log(expected_low) expected_high = math.log(expected_high) if transform_step and distribution.step is not None: half_step = 0.5 * distribution.step expected_low -= half_step expected_high += half_step elif isinstance(distribution, IntDistribution): if transform_step: half_step = 0.5 * distribution.step expected_low -= half_step expected_high += half_step if distribution.log and transform_log: expected_low = math.log(expected_low) expected_high = math.log(expected_high) for bound in trans.bounds: assert bound[0] == expected_low assert bound[1] == expected_high trans_params = trans.transform({"x0": param}) assert trans_params.size == 1 assert expected_low <= trans_params <= expected_high assert np.isclose(param, trans.untransform(trans_params)["x0"]) @pytest.mark.parametrize( "param,distribution", [ ("foo", CategoricalDistribution(["foo"])), ("bar", CategoricalDistribution(["foo", "bar", "baz"])), ], ) def test_search_space_transform_values_categorical( param: Any, distribution: CategoricalDistribution ) -> None: trans = _SearchSpaceTransform({"x0": distribution}) for bound in trans.bounds: assert bound[0] == 0.0 assert bound[1] == 1.0 trans_params = trans.transform({"x0": param}) for trans_param in trans_params: assert trans_param in (0.0, 1.0) def test_search_space_transform_untransform_params() -> None: search_space = { "x0": CategoricalDistribution(["corge"]), "x1": CategoricalDistribution(["foo", "bar", "baz", "qux"]), "x2": CategoricalDistribution(["quux", "quuz"]), "x3": FloatDistribution(2, 3), "x4": FloatDistribution(-2, 2), "x5": FloatDistribution(1, 10, log=True), "x6": FloatDistribution(1, 1, log=True), "x7": FloatDistribution(0, 1, step=0.2), "x8": IntDistribution(2, 4), "x9": IntDistribution(1, 10, log=True), "x10": IntDistribution(1, 9, step=2), } params = { "x0": "corge", "x1": "qux", "x2": "quux", "x3": 2.0, "x4": -2, "x5": 1.0, "x6": 1.0, "x7": 0.2, "x8": 2, "x9": 1, "x10": 3, } trans = _SearchSpaceTransform(search_space) trans_params = trans.transform(params) untrans_params = trans.untransform(trans_params) for name in params.keys(): assert untrans_params[name] == params[name] @pytest.mark.parametrize("transform_log", [True, False]) @pytest.mark.parametrize("transform_step", [True, False]) @pytest.mark.parametrize( "distribution", [ FloatDistribution(0, 1, step=0.2), IntDistribution(2, 4), IntDistribution(1, 10, log=True), ], ) def test_transform_untransform_params_at_bounds( transform_log: bool, transform_step: bool, distribution: BaseDistribution ) -> None: EPS = 1e-12 # Skip the following two conditions that do not clip in `_untransform_numerical_param`: # 1. `IntDistribution(log=True)` without `transform_log` if not transform_log and (isinstance(distribution, IntDistribution) and distribution.log): return trans = _SearchSpaceTransform({"x0": distribution}, transform_log, transform_step) # Manually create round-off errors. lower_bound = trans.bounds[0][0] - EPS upper_bound = trans.bounds[0][1] + EPS trans_lower_param = _untransform_numerical_param(lower_bound, distribution, transform_log) trans_upper_param = _untransform_numerical_param(upper_bound, distribution, transform_log) assert trans_lower_param == distribution.low # type: ignore assert trans_upper_param == distribution.high # type: ignore optuna-4.1.0/tests/trial_tests/000077500000000000000000000000001471332314300165225ustar00rootroot00000000000000optuna-4.1.0/tests/trial_tests/__init__.py000066400000000000000000000000001471332314300206210ustar00rootroot00000000000000optuna-4.1.0/tests/trial_tests/test_fixed.py000066400000000000000000000017721471332314300212410ustar00rootroot00000000000000from __future__ import annotations import pytest from optuna.trial import FixedTrial def test_params() -> None: params = {"x": 1} trial = FixedTrial(params) assert trial.params == {} assert trial.suggest_float("x", 0, 10) == 1 assert trial.params == params @pytest.mark.filterwarnings("ignore::FutureWarning") @pytest.mark.parametrize("positional_args_names", [[], ["step"], ["step", "log"]]) def test_suggest_int_positional_args(positional_args_names: list[str]) -> None: # If log is specified as positional, step must also be provided as positional. params = {"x": 1} trial = FixedTrial(params) kwargs = dict(step=1, log=False) args = [kwargs[name] for name in positional_args_names] # No error should not be raised even if the coding style is old. trial.suggest_int("x", -1, 1, *args) def test_number() -> None: params = {"x": 1} trial = FixedTrial(params, 2) assert trial.number == 2 trial = FixedTrial(params) assert trial.number == 0 optuna-4.1.0/tests/trial_tests/test_frozen.py000066400000000000000000000334301471332314300214410ustar00rootroot00000000000000from __future__ import annotations import copy import datetime from typing import Any from typing import Tuple import pytest from optuna import create_study from optuna.distributions import BaseDistribution from optuna.distributions import FloatDistribution from optuna.distributions import IntDistribution from optuna.testing.storages import STORAGE_MODES from optuna.testing.storages import StorageSupplier import optuna.trial from optuna.trial import BaseTrial from optuna.trial import create_trial from optuna.trial import FrozenTrial from optuna.trial import TrialState def _create_trial( *, value: float = 0.2, params: dict[str, Any] = {"x": 10}, distributions: dict[str, BaseDistribution] = {"x": FloatDistribution(5, 12)}, ) -> FrozenTrial: trial = optuna.trial.create_trial(value=value, params=params, distributions=distributions) trial.number = 0 return trial def test_eq_ne() -> None: trial = _create_trial() trial_other = copy.copy(trial) assert trial == trial_other trial_other.value = 0.3 assert trial != trial_other def test_lt() -> None: trial = _create_trial() trial_other = copy.copy(trial) assert not trial < trial_other trial_other.number = trial.number + 1 assert trial < trial_other assert not trial_other < trial with pytest.raises(TypeError): trial < 1 assert trial <= trial_other assert not trial_other <= trial with pytest.raises(TypeError): trial <= 1 # A list of FrozenTrials is sortable. trials = [trial_other, trial] trials.sort() assert trials[0] is trial assert trials[1] is trial_other def test_repr() -> None: trial = _create_trial() assert trial == eval(repr(trial)) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_sampling(storage_mode: str) -> None: def objective(trial: BaseTrial) -> float: a = trial.suggest_float("a", 0.0, 10.0) b = trial.suggest_float("b", 0.1, 10.0, log=True) c = trial.suggest_float("c", 0.0, 10.0, step=1.0) d = trial.suggest_int("d", 0, 10) e = trial.suggest_categorical("e", [0, 1, 2]) f = trial.suggest_int("f", 1, 10, log=True) return a + b + c + d + e + f with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) study.optimize(objective, n_trials=1) best_trial = study.best_trial # re-evaluate objective with the best hyperparameters v = objective(best_trial) assert v == best_trial.value def test_set_value() -> None: trial = _create_trial() trial.value = 0.1 assert trial.value == 0.1 def test_set_values() -> None: trial = _create_trial() trial.values = (0.1, 0.2) assert trial.values == [0.1, 0.2] # type: ignore[comparison-overlap] trial = _create_trial() trial.values = [0.1, 0.2] assert trial.values == [0.1, 0.2] def test_validate() -> None: # Valid. valid_trial = _create_trial() valid_trial._validate() # Invalid: `datetime_start` is not set when the trial is not in the waiting state. for state in [ TrialState.RUNNING, TrialState.COMPLETE, TrialState.PRUNED, TrialState.FAIL, ]: invalid_trial = copy.copy(valid_trial) invalid_trial.state = state invalid_trial.datetime_start = None with pytest.raises(ValueError): invalid_trial._validate() # Invalid: `state` is `RUNNING` and `datetime_complete` is set. invalid_trial = copy.copy(valid_trial) invalid_trial.state = TrialState.RUNNING with pytest.raises(ValueError): invalid_trial._validate() # Invalid: `state` is not `RUNNING` and `datetime_complete` is not set. for state in [TrialState.COMPLETE, TrialState.PRUNED, TrialState.FAIL]: invalid_trial = copy.copy(valid_trial) invalid_trial.state = state invalid_trial.datetime_complete = None with pytest.raises(ValueError): invalid_trial._validate() # Invalid: `state` is `FAIL`, and `value` is set. invalid_trial = copy.copy(valid_trial) invalid_trial.state = TrialState.FAIL invalid_trial.value = 1.0 with pytest.raises(ValueError): invalid_trial._validate() # Invalid: `state` is `COMPLETE` and `value` is not set. invalid_trial = copy.copy(valid_trial) invalid_trial.value = None with pytest.raises(ValueError): invalid_trial._validate() # Invalid: `state` is `COMPLETE` and `value` is NaN. invalid_trial = copy.copy(valid_trial) invalid_trial.value = float("nan") with pytest.raises(ValueError): invalid_trial._validate() # Invalid: `state` is `COMPLETE` and `values` includes NaN. invalid_trial = copy.copy(valid_trial) invalid_trial.values = [0.0, float("nan")] with pytest.raises(ValueError): invalid_trial._validate() # Invalid: Inconsistent `params` and `distributions` inconsistent_pairs: list[Tuple[dict[str, Any], dict[str, BaseDistribution]]] = [ # `params` has an extra element. ({"x": 0.1, "y": 0.5}, {"x": FloatDistribution(0, 1)}), # `distributions` has an extra element. ({"x": 0.1}, {"x": FloatDistribution(0, 1), "y": FloatDistribution(0.1, 1.0, log=True)}), # The value of `x` isn't contained in the distribution. ({"x": -0.5}, {"x": FloatDistribution(0, 1)}), ] for params, distributions in inconsistent_pairs: invalid_trial = copy.copy(valid_trial) invalid_trial.params = params invalid_trial.distributions = distributions with pytest.raises(ValueError): invalid_trial._validate() def test_number() -> None: trial = _create_trial() assert trial.number == 0 trial.number = 2 assert trial.number == 2 def test_params() -> None: params = {"x": 1} trial = _create_trial( value=0.2, params=params, distributions={"x": FloatDistribution(0, 10)}, ) assert trial.suggest_float("x", 0, 10) == 1 assert trial.params == params params = {"x": 2} trial.params = params assert trial.suggest_float("x", 0, 10) == 2 assert trial.params == params def test_distributions() -> None: distributions = {"x": FloatDistribution(0, 10)} trial = _create_trial( value=0.2, params={"x": 1}, distributions=dict(distributions), ) assert trial.distributions == distributions distributions = {"x": FloatDistribution(1, 9)} trial.distributions = dict(distributions) assert trial.distributions == distributions def test_user_attrs() -> None: trial = _create_trial() assert trial.user_attrs == {} user_attrs = {"data": "MNIST"} trial.user_attrs = user_attrs assert trial.user_attrs == user_attrs def test_system_attrs() -> None: trial = _create_trial() assert trial.system_attrs == {} system_attrs = {"system_message": "test"} trial.system_attrs = system_attrs assert trial.system_attrs == system_attrs def test_called_single_methods_when_multi() -> None: state = TrialState.COMPLETE values = (0.2, 0.3) params = {"x": 10} distributions: dict[str, BaseDistribution] = {"x": FloatDistribution(5, 12)} user_attrs = {"foo": "bar"} system_attrs = {"baz": "qux"} intermediate_values = {0: 0.0, 1: 0.1, 2: 0.1} trial = optuna.trial.create_trial( state=state, values=values, params=params, distributions=distributions, user_attrs=user_attrs, system_attrs=system_attrs, intermediate_values=intermediate_values, ) with pytest.raises(RuntimeError): trial.value with pytest.raises(RuntimeError): trial.value = 0.1 with pytest.raises(RuntimeError): trial.value = [0.1] # type: ignore def test_init() -> None: def _create_trial(value: float | None, values: list[float] | None) -> FrozenTrial: return FrozenTrial( number=0, trial_id=0, state=TrialState.COMPLETE, value=value, values=values, datetime_start=datetime.datetime.now(), datetime_complete=datetime.datetime.now(), params={}, distributions={"x": FloatDistribution(0, 10)}, user_attrs={}, system_attrs={}, intermediate_values={}, ) _ = _create_trial(0.2, None) _ = _create_trial(None, [0.2]) with pytest.raises(ValueError): _ = _create_trial(0.2, [0.2]) with pytest.raises(ValueError): _ = _create_trial(0.2, []) # TODO(hvy): Write exhaustive test include invalid combinations when feature is no longer # experimental. @pytest.mark.parametrize("state", [TrialState.COMPLETE, TrialState.FAIL]) def test_create_trial(state: TrialState) -> None: value: float | None = 0.2 params = {"x": 10} distributions: dict[str, BaseDistribution] = {"x": FloatDistribution(5, 12)} user_attrs = {"foo": "bar"} system_attrs = {"baz": "qux"} intermediate_values = {0: 0.0, 1: 0.1, 2: 0.1} if state == TrialState.FAIL: value = None trial = create_trial( state=state, value=value if state == TrialState.COMPLETE else None, params=params, distributions=distributions, user_attrs=user_attrs, system_attrs=system_attrs, intermediate_values=intermediate_values, ) assert isinstance(trial, FrozenTrial) assert trial.state == state assert trial.value == value assert trial.params == params assert trial.distributions == distributions assert trial.user_attrs == user_attrs assert trial.system_attrs == system_attrs assert trial.intermediate_values == intermediate_values assert trial.datetime_start is not None assert (trial.datetime_complete is not None) == state.is_finished() with pytest.raises(ValueError): create_trial( state=state, value=0.2 if state != TrialState.COMPLETE else None, params=params, distributions=distributions, user_attrs=user_attrs, system_attrs=system_attrs, intermediate_values=intermediate_values, ) # Deprecated distributions are internally converted to corresponding distributions. @pytest.mark.filterwarnings("ignore::FutureWarning") def test_create_trial_distribution_conversion() -> None: fixed_params = { "ud": 0, "dud": 2, "lud": 1, "id": 0, "idd": 2, "ild": 1, } fixed_distributions = { "ud": optuna.distributions.UniformDistribution(low=0, high=10), "dud": optuna.distributions.DiscreteUniformDistribution(low=0, high=10, q=2), "lud": optuna.distributions.LogUniformDistribution(low=1, high=10), "id": optuna.distributions.IntUniformDistribution(low=0, high=10), "idd": optuna.distributions.IntUniformDistribution(low=0, high=10, step=2), "ild": optuna.distributions.IntLogUniformDistribution(low=1, high=10), } with pytest.warns( FutureWarning, match="See https://github.com/optuna/optuna/issues/2941", ) as record: trial = create_trial(params=fixed_params, distributions=fixed_distributions, value=1) assert len(record) == 6 expected_distributions = { "ud": optuna.distributions.FloatDistribution(low=0, high=10, log=False, step=None), "dud": optuna.distributions.FloatDistribution(low=0, high=10, log=False, step=2), "lud": optuna.distributions.FloatDistribution(low=1, high=10, log=True, step=None), "id": optuna.distributions.IntDistribution(low=0, high=10, log=False, step=1), "idd": optuna.distributions.IntDistribution(low=0, high=10, log=False, step=2), "ild": optuna.distributions.IntDistribution(low=1, high=10, log=True, step=1), } assert trial.distributions == expected_distributions # It confirms that ask doesn't convert non-deprecated distributions. def test_create_trial_distribution_conversion_noop() -> None: fixed_params = { "ud": 0, "dud": 2, "lud": 1, "id": 0, "idd": 2, "ild": 1, "cd": "a", } fixed_distributions = { "ud": optuna.distributions.FloatDistribution(low=0, high=10, log=False, step=None), "dud": optuna.distributions.FloatDistribution(low=0, high=10, log=False, step=2), "lud": optuna.distributions.FloatDistribution(low=1, high=10, log=True, step=None), "id": optuna.distributions.IntDistribution(low=0, high=10, log=False, step=1), "idd": optuna.distributions.IntDistribution(low=0, high=10, log=False, step=2), "ild": optuna.distributions.IntDistribution(low=1, high=10, log=True, step=1), "cd": optuna.distributions.CategoricalDistribution(choices=["a", "b", "c"]), } trial = create_trial(params=fixed_params, distributions=fixed_distributions, value=1) # Check fixed_distributions doesn't change. assert trial.distributions == fixed_distributions @pytest.mark.filterwarnings("ignore::FutureWarning") @pytest.mark.parametrize("positional_args_names", [[], ["step"], ["step", "log"]]) def test_suggest_int_positional_args(positional_args_names: list[str]) -> None: # If log is specified as positional, step must also be provided as positional. trial = FrozenTrial( number=0, trial_id=0, state=TrialState.COMPLETE, value=0.0, values=None, datetime_start=datetime.datetime.now(), datetime_complete=datetime.datetime.now(), params={"x": 1}, distributions={"x": IntDistribution(-1, 1)}, user_attrs={}, system_attrs={}, intermediate_values={}, ) kwargs = dict(step=1, log=False) args = [kwargs[name] for name in positional_args_names] # No error should not be raised even if the coding style is old. trial.suggest_int("x", -1, 1, *args) optuna-4.1.0/tests/trial_tests/test_trial.py000066400000000000000000000717411471332314300212600ustar00rootroot00000000000000from __future__ import annotations import datetime import math from typing import Any from typing import Dict from typing import List from typing import Optional from typing import Tuple from unittest.mock import Mock from unittest.mock import patch import warnings import numpy as np import pytest import optuna from optuna import create_study from optuna import distributions from optuna import load_study from optuna import samplers from optuna import storages from optuna.distributions import CategoricalDistribution from optuna.distributions import FloatDistribution from optuna.distributions import IntDistribution from optuna.testing.pruners import DeterministicPruner from optuna.testing.samplers import DeterministicSampler from optuna.testing.storages import STORAGE_MODES from optuna.testing.storages import StorageSupplier from optuna.testing.tempfile_pool import NamedTemporaryFilePool from optuna.trial import Trial from optuna.trial._trial import _LazyTrialSystemAttrs @pytest.mark.filterwarnings("ignore::FutureWarning") @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_check_distribution_suggest_float(storage_mode: str) -> None: sampler = samplers.RandomSampler() with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage, sampler=sampler) trial = Trial(study, study._storage.create_new_trial(study._study_id)) x1 = trial.suggest_float("x1", 10, 20) x2 = trial.suggest_uniform("x1", 10, 20) assert x1 == x2 x3 = trial.suggest_float("x2", 1e-5, 1e-3, log=True) x4 = trial.suggest_loguniform("x2", 1e-5, 1e-3) assert x3 == x4 x5 = trial.suggest_float("x3", 10, 20, step=1.0) x6 = trial.suggest_discrete_uniform("x3", 10, 20, 1.0) assert x5 == x6 with pytest.raises(ValueError): trial.suggest_float("x4", 1e-5, 1e-2, step=1e-5, log=True) with pytest.raises(ValueError): trial.suggest_int("x1", 10, 20) trial = Trial(study, study._storage.create_new_trial(study._study_id)) with pytest.raises(ValueError): trial.suggest_int("x1", 10, 20) @pytest.mark.filterwarnings("ignore::FutureWarning") @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_check_distribution_suggest_uniform(storage_mode: str) -> None: sampler = samplers.RandomSampler() with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage, sampler=sampler) trial = Trial(study, study._storage.create_new_trial(study._study_id)) with pytest.warns() as record: trial.suggest_uniform("x", 10, 20) trial.suggest_uniform("x", 10, 20) trial.suggest_uniform("x", 10, 30) # we expect exactly one warning (not counting ones caused by deprecation) assert len([r for r in record if r.category is not FutureWarning]) == 1 with pytest.raises(ValueError): trial.suggest_int("x", 10, 20) trial = Trial(study, study._storage.create_new_trial(study._study_id)) with pytest.raises(ValueError): trial.suggest_int("x", 10, 20) @pytest.mark.filterwarnings("ignore::FutureWarning") @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_check_distribution_suggest_loguniform(storage_mode: str) -> None: sampler = samplers.RandomSampler() with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage, sampler=sampler) trial = Trial(study, study._storage.create_new_trial(study._study_id)) with pytest.warns() as record: trial.suggest_loguniform("x", 10, 20) trial.suggest_loguniform("x", 10, 20) trial.suggest_loguniform("x", 10, 30) # We expect exactly one warning (not counting ones caused by deprecation). assert len([r for r in record if r.category is not FutureWarning]) == 1 with pytest.raises(ValueError): trial.suggest_int("x", 10, 20) trial = Trial(study, study._storage.create_new_trial(study._study_id)) with pytest.raises(ValueError): trial.suggest_int("x", 10, 20) @pytest.mark.filterwarnings("ignore::FutureWarning") @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_check_distribution_suggest_discrete_uniform(storage_mode: str) -> None: sampler = samplers.RandomSampler() with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage, sampler=sampler) trial = Trial(study, study._storage.create_new_trial(study._study_id)) with pytest.warns() as record: trial.suggest_discrete_uniform("x", 10, 20, 2) trial.suggest_discrete_uniform("x", 10, 20, 2) trial.suggest_discrete_uniform("x", 10, 22, 2) # We expect exactly one warning (not counting ones caused by deprecation). assert len([r for r in record if r.category is not FutureWarning]) == 1 with pytest.raises(ValueError): trial.suggest_int("x", 10, 20, step=2) trial = Trial(study, study._storage.create_new_trial(study._study_id)) with pytest.raises(ValueError): trial.suggest_int("x", 10, 20, step=2) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) @pytest.mark.parametrize("enable_log", [False, True]) def test_check_distribution_suggest_int(storage_mode: str, enable_log: bool) -> None: sampler = samplers.RandomSampler() with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage, sampler=sampler) trial = Trial(study, study._storage.create_new_trial(study._study_id)) with pytest.warns() as record: trial.suggest_int("x", 10, 20, log=enable_log) trial.suggest_int("x", 10, 20, log=enable_log) trial.suggest_int("x", 10, 22, log=enable_log) # We expect exactly one warning (not counting ones caused by deprecation). assert len([r for r in record if r.category is not FutureWarning]) == 1 with pytest.raises(ValueError): trial.suggest_float("x", 10, 20, log=enable_log) trial = Trial(study, study._storage.create_new_trial(study._study_id)) with pytest.raises(ValueError): trial.suggest_float("x", 10, 20, log=enable_log) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_check_distribution_suggest_categorical(storage_mode: str) -> None: sampler = samplers.RandomSampler() with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage, sampler=sampler) trial = Trial(study, study._storage.create_new_trial(study._study_id)) trial.suggest_categorical("x", [10, 20, 30]) with pytest.raises(ValueError): trial.suggest_categorical("x", [10, 20]) with pytest.raises(ValueError): trial.suggest_int("x", 10, 20) trial = Trial(study, study._storage.create_new_trial(study._study_id)) with pytest.raises(ValueError): trial.suggest_int("x", 10, 20) @pytest.mark.filterwarnings("ignore::FutureWarning") @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_suggest_uniform(storage_mode: str) -> None: sampler = DeterministicSampler({"x": 1.0, "y": 2.0}) with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage, sampler=sampler) trial = Trial(study, study._storage.create_new_trial(study._study_id)) assert trial.suggest_uniform("x", 0.0, 3.0) == 1.0 # Test suggesting a param. assert trial.suggest_uniform("x", 0.0, 3.0) == 1.0 # Test suggesting the same param. assert trial.suggest_uniform("y", 0.0, 3.0) == 2.0 # Test suggesting a different param. assert trial.params == {"x": 1.0, "y": 2.0} @pytest.mark.filterwarnings("ignore::FutureWarning") @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_suggest_loguniform(storage_mode: str) -> None: with pytest.raises(ValueError): FloatDistribution(low=1.0, high=0.9, log=True) with pytest.raises(ValueError): FloatDistribution(low=0.0, high=0.9, log=True) sampler = DeterministicSampler({"x": 1.0, "y": 2.0}) with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage, sampler=sampler) trial = Trial(study, study._storage.create_new_trial(study._study_id)) assert trial.suggest_loguniform("x", 0.1, 4.0) == 1.0 # Test suggesting a param. assert trial.suggest_loguniform("x", 0.1, 4.0) == 1.0 # Test suggesting the same param. assert trial.suggest_loguniform("y", 0.1, 4.0) == 2.0 # Test suggesting a different param. assert trial.params == {"x": 1.0, "y": 2.0} @pytest.mark.filterwarnings("ignore::FutureWarning") @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_suggest_discrete_uniform(storage_mode: str) -> None: sampler = DeterministicSampler({"x": 1.0, "y": 2.0}) with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage, sampler=sampler) trial = Trial(study, study._storage.create_new_trial(study._study_id)) assert ( trial.suggest_discrete_uniform("x", 0.0, 3.0, 1.0) == 1.0 ) # Test suggesting a param. assert ( trial.suggest_discrete_uniform("x", 0.0, 3.0, 1.0) == 1.0 ) # Test suggesting the same param. assert ( trial.suggest_discrete_uniform("y", 0.0, 3.0, 1.0) == 2.0 ) # Test suggesting a different param. assert trial.params == {"x": 1.0, "y": 2.0} @pytest.mark.filterwarnings("ignore::FutureWarning") @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_suggest_low_equals_high(storage_mode: str) -> None: with patch.object( distributions, "_get_single_value", wraps=distributions._get_single_value ) as mock_object, StorageSupplier(storage_mode) as storage: study = create_study(storage=storage, sampler=samplers.TPESampler(n_startup_trials=0)) trial = Trial(study, study._storage.create_new_trial(study._study_id)) assert trial.suggest_uniform("a", 1.0, 1.0) == 1.0 # Suggesting a param. assert mock_object.call_count == 1 assert trial.suggest_uniform("a", 1.0, 1.0) == 1.0 # Suggesting the same param. assert mock_object.call_count == 1 assert trial.suggest_loguniform("b", 1.0, 1.0) == 1.0 # Suggesting a param. assert mock_object.call_count == 2 assert trial.suggest_loguniform("b", 1.0, 1.0) == 1.0 # Suggesting the same param. assert mock_object.call_count == 2 assert trial.suggest_discrete_uniform("c", 1.0, 1.0, 1.0) == 1.0 # Suggesting a param. assert mock_object.call_count == 3 assert ( trial.suggest_discrete_uniform("c", 1.0, 1.0, 1.0) == 1.0 ) # Suggesting the same param. assert mock_object.call_count == 3 assert trial.suggest_int("d", 1, 1) == 1 # Suggesting a param. assert mock_object.call_count == 4 assert trial.suggest_int("d", 1, 1) == 1 # Suggesting the same param. assert mock_object.call_count == 4 assert trial.suggest_float("e", 1.0, 1.0) == 1.0 # Suggesting a param. assert mock_object.call_count == 5 assert trial.suggest_float("e", 1.0, 1.0) == 1.0 # Suggesting the same param. assert mock_object.call_count == 5 assert trial.suggest_float("f", 0.5, 0.5, log=True) == 0.5 # Suggesting a param. assert mock_object.call_count == 6 assert trial.suggest_float("f", 0.5, 0.5, log=True) == 0.5 # Suggesting the same param. assert mock_object.call_count == 6 assert trial.suggest_float("g", 0.5, 0.5, log=False) == 0.5 # Suggesting a param. assert mock_object.call_count == 7 assert trial.suggest_float("g", 0.5, 0.5, log=False) == 0.5 # Suggesting the same param. assert mock_object.call_count == 7 assert trial.suggest_float("h", 0.5, 0.5, step=1.0) == 0.5 # Suggesting a param. assert mock_object.call_count == 8 assert trial.suggest_float("h", 0.5, 0.5, step=1.0) == 0.5 # Suggesting the same param. assert mock_object.call_count == 8 assert trial.suggest_int("i", 1, 1, log=True) == 1 # Suggesting a param. assert mock_object.call_count == 9 assert trial.suggest_int("i", 1, 1, log=True) == 1 # Suggesting the same param. assert mock_object.call_count == 9 @pytest.mark.filterwarnings("ignore::FutureWarning") @pytest.mark.parametrize("storage_mode", STORAGE_MODES) @pytest.mark.parametrize( "range_config", [ {"low": 0.0, "high": 10.0, "q": 3.0, "mod_high": 9.0}, {"low": 1.0, "high": 11.0, "q": 3.0, "mod_high": 10.0}, {"low": 64.0, "high": 1312.0, "q": 160.0, "mod_high": 1184.0}, {"low": 0.0, "high": 10.0, "q": math.pi, "mod_high": 3 * math.pi}, {"low": 0.0, "high": 3.45, "q": 0.1, "mod_high": 3.4}, ], ) def test_suggest_discrete_uniform_range(storage_mode: str, range_config: Dict[str, float]) -> None: sampler = samplers.RandomSampler() # Check upper endpoints. mock = Mock() mock.side_effect = lambda study, trial, param_name, distribution: distribution.high with patch.object(sampler, "sample_independent", mock) as mock_object, StorageSupplier( storage_mode ) as storage: study = create_study(storage=storage, sampler=sampler) trial = Trial(study, study._storage.create_new_trial(study._study_id)) with pytest.warns(UserWarning): x = trial.suggest_discrete_uniform( "x", range_config["low"], range_config["high"], range_config["q"] ) assert x == range_config["mod_high"] assert mock_object.call_count == 1 # Check lower endpoints. mock = Mock() mock.side_effect = lambda study, trial, param_name, distribution: distribution.low with patch.object(sampler, "sample_independent", mock) as mock_object, StorageSupplier( storage_mode ) as storage: study = create_study(storage=storage, sampler=sampler) trial = Trial(study, study._storage.create_new_trial(study._study_id)) with pytest.warns(UserWarning): x = trial.suggest_discrete_uniform( "x", range_config["low"], range_config["high"], range_config["q"] ) assert x == range_config["low"] assert mock_object.call_count == 1 def test_suggest_float_invalid_step() -> None: study = create_study() trial = study.ask() with pytest.raises(ValueError): trial.suggest_float("x1", 10, 20, step=0) with pytest.raises(ValueError): trial.suggest_float("x2", 10, 20, step=-1) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_suggest_int(storage_mode: str) -> None: sampler = DeterministicSampler({"x": 1, "y": 2}) with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage, sampler=sampler) trial = Trial(study, study._storage.create_new_trial(study._study_id)) assert trial.suggest_int("x", 0, 3) == 1 # Test suggesting a param. assert trial.suggest_int("x", 0, 3) == 1 # Test suggesting the same param. assert trial.suggest_int("y", 0, 3) == 2 # Test suggesting a different param. assert trial.params == {"x": 1, "y": 2} @pytest.mark.parametrize("storage_mode", STORAGE_MODES) @pytest.mark.parametrize( "range_config", [ {"low": 0, "high": 10, "step": 3, "mod_high": 9}, {"low": 1, "high": 11, "step": 3, "mod_high": 10}, {"low": 64, "high": 1312, "step": 160, "mod_high": 1184}, ], ) def test_suggest_int_range(storage_mode: str, range_config: Dict[str, int]) -> None: sampler = samplers.RandomSampler() # Check upper endpoints. mock = Mock() mock.side_effect = lambda study, trial, param_name, distribution: distribution.high with patch.object(sampler, "sample_independent", mock) as mock_object, StorageSupplier( storage_mode ) as storage: study = create_study(storage=storage, sampler=sampler) trial = Trial(study, study._storage.create_new_trial(study._study_id)) with pytest.warns(UserWarning): x = trial.suggest_int( "x", range_config["low"], range_config["high"], step=range_config["step"] ) assert x == range_config["mod_high"] assert mock_object.call_count == 1 # Check lower endpoints. mock = Mock() mock.side_effect = lambda study, trial, param_name, distribution: distribution.low with patch.object(sampler, "sample_independent", mock) as mock_object, StorageSupplier( storage_mode ) as storage: study = create_study(storage=storage, sampler=sampler) trial = Trial(study, study._storage.create_new_trial(study._study_id)) with pytest.warns(UserWarning): x = trial.suggest_int( "x", range_config["low"], range_config["high"], step=range_config["step"] ) assert x == range_config["low"] assert mock_object.call_count == 1 def test_suggest_int_invalid_step() -> None: study = create_study() trial = study.ask() with pytest.raises(ValueError): trial.suggest_int("x1", 10, 20, step=0) with pytest.raises(ValueError): trial.suggest_int("x2", 10, 20, step=-1) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_suggest_int_log(storage_mode: str) -> None: sampler = DeterministicSampler({"x": 1, "y": 2}) with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage, sampler=sampler) trial = Trial(study, study._storage.create_new_trial(study._study_id)) assert trial.suggest_int("x", 1, 3, log=True) == 1 # Test suggesting a param. assert trial.suggest_int("x", 1, 3, log=True) == 1 # Test suggesting the same param. assert trial.suggest_int("y", 1, 3, log=True) == 2 # Test suggesting a different param. assert trial.params == {"x": 1, "y": 2} @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_suggest_int_log_invalid_range(storage_mode: str) -> None: sampler = samplers.RandomSampler() with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage, sampler=sampler) trial = Trial(study, study._storage.create_new_trial(study._study_id)) with warnings.catch_warnings(): # UserWarning will be raised since [0.5, 10] is not divisible by 1. warnings.simplefilter("ignore", category=UserWarning) with pytest.raises(ValueError): trial.suggest_int("z", 0.5, 10, log=True) # type: ignore with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage, sampler=sampler) trial = Trial(study, study._storage.create_new_trial(study._study_id)) with pytest.raises(ValueError): trial.suggest_int("w", 1, 3, step=2, log=True) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_distributions(storage_mode: str) -> None: def objective(trial: Trial) -> float: trial.suggest_float("a", 0, 10) trial.suggest_float("b", 0.1, 10, log=True) trial.suggest_float("c", 0, 10, step=1) trial.suggest_int("d", 0, 10) trial.suggest_categorical("e", ["foo", "bar", "baz"]) trial.suggest_int("f", 1, 10, log=True) return 1.0 with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) study.optimize(objective, n_trials=1) assert study.best_trial.distributions == { "a": FloatDistribution(low=0, high=10), "b": FloatDistribution(low=0.1, high=10, log=True), "c": FloatDistribution(low=0, high=10, step=1), "d": IntDistribution(low=0, high=10), "e": CategoricalDistribution(choices=("foo", "bar", "baz")), "f": IntDistribution(low=1, high=10, log=True), } def test_should_prune() -> None: pruner = DeterministicPruner(True) study = create_study(pruner=pruner) trial = Trial(study, study._storage.create_new_trial(study._study_id)) trial.report(1, 1) assert trial.should_prune() @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_relative_parameters(storage_mode: str) -> None: class SamplerStubForTestRelativeParameters(samplers.BaseSampler): def infer_relative_search_space( self, study: "optuna.study.Study", trial: "optuna.trial.FrozenTrial" ) -> Dict[str, distributions.BaseDistribution]: return { "x": FloatDistribution(low=5, high=6), "y": FloatDistribution(low=5, high=6), } def sample_relative( self, study: "optuna.study.Study", trial: "optuna.trial.FrozenTrial", search_space: Dict[str, distributions.BaseDistribution], ) -> Dict[str, Any]: return {"x": 5.5, "y": 5.5, "z": 5.5} def sample_independent( self, study: "optuna.study.Study", trial: "optuna.trial.FrozenTrial", param_name: str, param_distribution: distributions.BaseDistribution, ) -> Any: return 5.0 sampler = SamplerStubForTestRelativeParameters() with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage, sampler=sampler) def create_trial() -> Trial: return Trial(study, study._storage.create_new_trial(study._study_id)) # Suggested by `sample_relative`. trial0 = create_trial() distribution0 = FloatDistribution(low=0, high=100) assert trial0._suggest("x", distribution0) == 5.5 # Not suggested by `sample_relative` (due to unknown parameter name). trial1 = create_trial() distribution1 = distribution0 assert trial1._suggest("w", distribution1) != 5.5 # Not suggested by `sample_relative` (due to incompatible value range). trial2 = create_trial() distribution2 = FloatDistribution(low=0, high=5) assert trial2._suggest("x", distribution2) != 5.5 # Error (due to incompatible distribution class). trial3 = create_trial() distribution3 = IntDistribution(low=1, high=100) with pytest.raises(ValueError): trial3._suggest("y", distribution3) # Error ('z' is included in `sample_relative` but not in `infer_relative_search_space`). trial4 = create_trial() distribution4 = FloatDistribution(low=0, high=10) with pytest.raises(ValueError): trial4._suggest("z", distribution4) # Error (due to incompatible distribution class). trial5 = create_trial() distribution5 = IntDistribution(low=1, high=100, log=True) with pytest.raises(ValueError): trial5._suggest("y", distribution5) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_datetime_start(storage_mode: str) -> None: trial_datetime_start: List[Optional[datetime.datetime]] = [None] def objective(trial: Trial) -> float: trial_datetime_start[0] = trial.datetime_start return 1.0 with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) study.optimize(objective, n_trials=1) assert study.trials[0].datetime_start == trial_datetime_start[0] def test_report_value() -> None: study = create_study() trial = Trial(study, study._storage.create_new_trial(study._study_id)) # Report values that can be cast to `float` (OK). trial.report(1.23, 1) trial.report(float("nan"), 2) trial.report("1.23", 3) # type: ignore trial.report("inf", 4) # type: ignore trial.report(1, 5) trial.report(np.array([1], dtype=np.float32)[0], 6) # Report values that cannot be cast to `float`. with pytest.raises(TypeError): trial.report(None, 7) # type: ignore with pytest.raises(TypeError): trial.report("foo", 7) # type: ignore with pytest.raises(TypeError): trial.report([1, 2, 3], 7) # type: ignore with pytest.raises(TypeError): trial.report("foo", -1) # type: ignore def test_report_step() -> None: study = create_study() trial = study.ask() value = 1.0 # Report values whose steps can be cast to `int` (OK). trial.report(value, 0) trial.report(value, 1.0) # type: ignore trial.report(value, np.int64(2)) # type: ignore # Report values whose steps cannot be cast to `int` (Error). with pytest.raises(TypeError): trial.report(value, None) # type: ignore with pytest.raises(TypeError): trial.report(value, "foo") # type: ignore with pytest.raises(TypeError): trial.report(value, [1, 2, 3]) # type: ignore # Report a value whose step is negative (Error). with pytest.raises(ValueError): trial.report(value, -1) def test_report_warning() -> None: study = create_study() trial = study.ask() trial.report(1.23, 1) # Warn if multiple times call report method at the same step with pytest.warns(UserWarning): trial.report(1, 1) def test_suggest_with_multi_objectives() -> None: study = create_study(directions=["maximize", "maximize"]) def objective(trial: Trial) -> Tuple[float, float]: p0 = trial.suggest_float("p0", -10, 10) p1 = trial.suggest_float("p1", 3, 5) p2 = trial.suggest_float("p2", 0.00001, 0.1, log=True) p3 = trial.suggest_float("p3", 100, 200, step=5) p4 = trial.suggest_int("p4", -20, -15) p5 = trial.suggest_categorical("p5", [7, 1, 100]) p6 = trial.suggest_float("p6", -10, 10, step=1.0) p7 = trial.suggest_int("p7", 1, 7, log=True) return ( p0 + p1 + p2, p3 + p4 + p5 + p6 + p7, ) study.optimize(objective, n_trials=10) def test_raise_error_for_report_with_multi_objectives() -> None: study = create_study(directions=["maximize", "maximize"]) def objective(trial: Trial) -> Tuple[float, float]: with pytest.raises(NotImplementedError): trial.report(1.0, 0) return 1.0, 1.0 study.optimize(objective, n_trials=1) def test_raise_error_for_should_prune_multi_objectives() -> None: study = create_study(directions=["maximize", "maximize"]) def objective(trial: Trial) -> Tuple[float, float]: with pytest.raises(NotImplementedError): trial.should_prune() return 1.0, 1.0 study.optimize(objective, n_trials=1) def test_persisted_param() -> None: study_name = "my_study" with NamedTemporaryFilePool() as fp: storage = f"sqlite:///{fp.name}" study = create_study(storage=storage, study_name=study_name) assert isinstance(study._storage, storages._CachedStorage), "Pre-condition." # Test more than one trial. The `_CachedStorage` does a cache miss for the first trial and # thus behaves differently for the first trial in comparisons to the following. for _ in range(3): trial = study.ask() trial.suggest_float("x", 0, 1) study = load_study(storage=storage, study_name=study_name) assert all("x" in t.params for t in study.trials) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_lazy_trial_system_attrs(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study = optuna.create_study(storage=storage) trial = study.ask() storage.set_trial_system_attr(trial._trial_id, "int", 0) storage.set_trial_system_attr(trial._trial_id, "str", "A") # _LazyTrialSystemAttrs gets attrs the first time it is needed. # Then, we create the instance for each method, and test the first and second use. system_attrs = _LazyTrialSystemAttrs(trial._trial_id, storage) assert system_attrs == {"int": 0, "str": "A"} assert system_attrs == {"int": 0, "str": "A"} system_attrs = _LazyTrialSystemAttrs(trial._trial_id, storage) assert len(system_attrs) == 2 assert len(system_attrs) == 2 system_attrs = _LazyTrialSystemAttrs(trial._trial_id, storage) assert set(system_attrs.keys()) == {"int", "str"} assert set(system_attrs.keys()) == {"int", "str"} system_attrs = _LazyTrialSystemAttrs(trial._trial_id, storage) assert set(system_attrs.values()) == {0, "A"} assert set(system_attrs.values()) == {0, "A"} system_attrs = _LazyTrialSystemAttrs(trial._trial_id, storage) assert set(system_attrs.items()) == {("int", 0), ("str", "A")} assert set(system_attrs.items()) == {("int", 0), ("str", "A")} @pytest.mark.filterwarnings("ignore::FutureWarning") @pytest.mark.parametrize("positional_args_names", [[], ["step"], ["step", "log"]]) def test_suggest_int_positional_args(positional_args_names: list[str]) -> None: # If log is specified as positional, step must also be provided as positional. study = optuna.create_study() trial = study.ask() kwargs = dict(step=1, log=False) args = [kwargs[name] for name in positional_args_names] # No error should not be raised even if the coding style is old. trial.suggest_int("x", -1, 1, *args) optuna-4.1.0/tests/trial_tests/test_trials.py000066400000000000000000000156341471332314300214420ustar00rootroot00000000000000import datetime import time from typing import Any from typing import Dict from typing import Optional import pytest from optuna.distributions import BaseDistribution from optuna.distributions import CategoricalDistribution from optuna.distributions import FloatDistribution from optuna.distributions import IntDistribution from optuna.study.study import create_study from optuna.trial import BaseTrial from optuna.trial import create_trial from optuna.trial import FixedTrial from optuna.trial import FrozenTrial from optuna.trial import Trial parametrize_trial_type = pytest.mark.parametrize("trial_type", [FixedTrial, FrozenTrial, Trial]) def _create_trial( trial_type: type, params: Optional[Dict[str, Any]] = None, distributions: Optional[Dict[str, BaseDistribution]] = None, ) -> BaseTrial: if params is None: params = {"x": 10} assert params is not None if distributions is None: distributions = {"x": FloatDistribution(5, 12)} assert distributions is not None if trial_type == FixedTrial: return FixedTrial(params) elif trial_type == FrozenTrial: trial = create_trial(value=0.2, params=params, distributions=distributions) trial.number = 0 return trial elif trial_type == Trial: study = create_study() study.enqueue_trial(params) return study.ask() else: assert False @pytest.mark.parametrize("trial_type", [FixedTrial, FrozenTrial]) def test_suggest_float(trial_type: type) -> None: trial = _create_trial( trial_type=trial_type, params={"x": 0.2}, distributions={"x": FloatDistribution(0.0, 1.0)} ) assert trial.suggest_float("x", 0.0, 1.0) == 0.2 with pytest.raises(ValueError): trial.suggest_float("x", 0.0, 1.0, step=10, log=True) with pytest.raises(ValueError): trial.suggest_float("y", 0.0, 1.0) @pytest.mark.filterwarnings("ignore::FutureWarning") @pytest.mark.parametrize("trial_type", [FixedTrial, FrozenTrial]) def test_suggest_uniform(trial_type: type) -> None: trial = _create_trial( trial_type=trial_type, params={"x": 0.2}, distributions={"x": FloatDistribution(0.0, 1.0)}, ) assert trial.suggest_uniform("x", 0.0, 1.0) == 0.2 with pytest.raises(ValueError): trial.suggest_uniform("y", 0.0, 1.0) @pytest.mark.filterwarnings("ignore::FutureWarning") @pytest.mark.parametrize("trial_type", [FixedTrial, FrozenTrial]) def test_suggest_loguniform(trial_type: type) -> None: trial = _create_trial( trial_type=trial_type, params={"x": 0.99}, distributions={"x": FloatDistribution(0.1, 1.0, log=True)}, ) assert trial.suggest_loguniform("x", 0.1, 1.0) == 0.99 with pytest.raises(ValueError): trial.suggest_loguniform("y", 0.0, 1.0) @pytest.mark.filterwarnings("ignore::FutureWarning") @pytest.mark.parametrize("trial_type", [FixedTrial, FrozenTrial]) def test_suggest_discrete_uniform(trial_type: type) -> None: trial = _create_trial( trial_type=trial_type, params={"x": 0.9}, distributions={"x": FloatDistribution(0.0, 1.0, step=0.1)}, ) assert trial.suggest_discrete_uniform("x", 0.0, 1.0, 0.1) == 0.9 with pytest.raises(ValueError): trial.suggest_discrete_uniform("y", 0.0, 1.0, 0.1) @pytest.mark.parametrize("trial_type", [FixedTrial, FrozenTrial]) def test_suggest_int_log(trial_type: type) -> None: trial = _create_trial( trial_type=trial_type, params={"x": 1}, distributions={"x": IntDistribution(1, 10, log=True)}, ) assert trial.suggest_int("x", 1, 10, log=True) == 1 with pytest.raises(ValueError): trial.suggest_int("x", 1, 10, step=2, log=True) with pytest.raises(ValueError): trial.suggest_int("y", 1, 10, log=True) @pytest.mark.parametrize("trial_type", [FixedTrial, FrozenTrial]) def test_suggest_categorical(trial_type: type) -> None: # Integer categories. trial = _create_trial( trial_type=trial_type, params={"x": 1}, distributions={"x": CategoricalDistribution((0, 1, 2, 3))}, ) assert trial.suggest_categorical("x", (0, 1, 2, 3)) == 1 with pytest.raises(ValueError): trial.suggest_categorical("y", [0, 1, 2, 3]) # String categories. trial = _create_trial( trial_type=trial_type, params={"x": "baz"}, distributions={"x": CategoricalDistribution(("foo", "bar", "baz"))}, ) assert trial.suggest_categorical("x", ("foo", "bar", "baz")) == "baz" # Unknown parameter. with pytest.raises(ValueError): trial.suggest_categorical("y", ["foo", "bar", "baz"]) # Not in choices. with pytest.raises(ValueError): trial.suggest_categorical("x", ["foo", "bar"]) # Unknown parameter and bad category type. with pytest.warns(UserWarning): with pytest.raises(ValueError): # Must come after `pytest.warns` to catch failures. trial.suggest_categorical("x", [{"foo": "bar"}]) # type: ignore @parametrize_trial_type @pytest.mark.parametrize( ("suggest_func", "distribution"), [ (lambda trial, *args: trial.suggest_int(*args), IntDistribution(1, 10)), ( lambda trial, *args: trial.suggest_int(*args, log=True), IntDistribution(1, 10, log=True), ), (lambda trial, *args: trial.suggest_int(*args, step=2), IntDistribution(1, 9, step=2)), (lambda trial, *args: trial.suggest_float(*args), FloatDistribution(1, 10)), ( lambda trial, *args: trial.suggest_float(*args, log=True), FloatDistribution(1, 10, log=True), ), ( lambda trial, *args: trial.suggest_float(*args, step=1), FloatDistribution(1, 10, step=1), ), ], ) def test_not_contained_param( trial_type: type, suggest_func: Any, distribution: BaseDistribution ) -> None: trial = _create_trial( trial_type=trial_type, params={"x": 1.0}, distributions={"x": distribution}, ) with pytest.warns(UserWarning): assert suggest_func(trial, "x", 10, 100) == 1 @parametrize_trial_type def test_set_user_attrs(trial_type: type) -> None: trial = _create_trial(trial_type) trial.set_user_attr("data", "MNIST") assert trial.user_attrs["data"] == "MNIST" @parametrize_trial_type def test_report(trial_type: type) -> None: # Ignores reported values. trial = _create_trial(trial_type) trial.report(1.0, 1) trial.report(2.0, 2) @parametrize_trial_type def test_should_prune(trial_type: type) -> None: # Never prunes trials. assert not _create_trial(trial_type).should_prune() @parametrize_trial_type def test_datetime_start(trial_type: type) -> None: trial = _create_trial(trial_type) assert trial.datetime_start is not None old_date_time_start = trial.datetime_start time.sleep(0.001) # Sleep 1ms to avoid faulty assertion on Windows OS. assert datetime.datetime.now() != old_date_time_start optuna-4.1.0/tests/visualization_tests/000077500000000000000000000000001471332314300203105ustar00rootroot00000000000000optuna-4.1.0/tests/visualization_tests/__init__.py000066400000000000000000000000001471332314300224070ustar00rootroot00000000000000optuna-4.1.0/tests/visualization_tests/matplotlib_tests/000077500000000000000000000000001471332314300237015ustar00rootroot00000000000000optuna-4.1.0/tests/visualization_tests/matplotlib_tests/__init__.py000066400000000000000000000000001471332314300260000ustar00rootroot00000000000000optuna-4.1.0/tests/visualization_tests/matplotlib_tests/test_contour.py000066400000000000000000000020001471332314300267730ustar00rootroot00000000000000import numpy as np from optuna.visualization.matplotlib._contour import _create_zmap from optuna.visualization.matplotlib._contour import _interpolate_zmap def test_create_zmap() -> None: x_values = np.arange(10) y_values = np.arange(10) z_values = list(np.random.rand(10)) # we are testing for exact placement of z_values # so also passing x_values and y_values as xi and yi zmap = _create_zmap(x_values.tolist(), y_values.tolist(), z_values, x_values, y_values) assert len(zmap) == len(z_values) for coord, value in zmap.items(): # test if value under coordinate # still refers to original trial value xidx = coord[0] yidx = coord[1] assert xidx == yidx assert z_values[xidx] == value def test_interpolate_zmap() -> None: contour_point_num = 2 zmap = {(0, 0): 1.0, (1, 1): 4.0} expected = np.array([[1.0, 2.5], [2.5, 4.0]]) actual = _interpolate_zmap(zmap, contour_point_num) assert np.allclose(expected, actual) optuna-4.1.0/tests/visualization_tests/matplotlib_tests/test_optimization_history.py000066400000000000000000000022721471332314300316240ustar00rootroot00000000000000from __future__ import annotations from io import BytesIO import pytest from optuna.visualization._optimization_history import _OptimizationHistoryInfo from optuna.visualization.matplotlib._matplotlib_imports import plt from optuna.visualization.matplotlib._optimization_history import _get_optimization_history_plot from tests.visualization_tests.test_optimization_history import optimization_history_info_lists @pytest.mark.parametrize("target_name", ["Objective Value", "Target Name"]) @pytest.mark.parametrize("info_list", optimization_history_info_lists) def test_get_optimization_history_plot( target_name: str, info_list: list[_OptimizationHistoryInfo] ) -> None: figure = _get_optimization_history_plot(info_list, target_name=target_name) assert figure.get_ylabel() == target_name expected_legends = [] for info in info_list: expected_legends.append(info.values_info.label_name) if info.best_values_info is not None: expected_legends.append(info.best_values_info.label_name) legends = [legend.get_text() for legend in figure.legend().get_texts()] assert sorted(legends) == sorted(expected_legends) plt.savefig(BytesIO()) plt.close() optuna-4.1.0/tests/visualization_tests/test_contour.py000066400000000000000000000534331471332314300234220ustar00rootroot00000000000000from __future__ import annotations from io import BytesIO import math from typing import Any from typing import Callable import numpy as np import pytest from optuna.distributions import BaseDistribution from optuna.distributions import CategoricalDistribution from optuna.distributions import FloatDistribution from optuna.study import create_study from optuna.study import Study from optuna.testing.objectives import fail_objective from optuna.testing.visualization import prepare_study_with_trials from optuna.trial import create_trial from optuna.visualization import plot_contour as plotly_plot_contour from optuna.visualization._contour import _AxisInfo from optuna.visualization._contour import _ContourInfo from optuna.visualization._contour import _get_contour_info from optuna.visualization._contour import _SubContourInfo from optuna.visualization._plotly_imports import go from optuna.visualization._utils import COLOR_SCALE from optuna.visualization.matplotlib import plot_contour as plt_plot_contour from optuna.visualization.matplotlib._matplotlib_imports import Axes from optuna.visualization.matplotlib._matplotlib_imports import plt parametrize_plot_contour = pytest.mark.parametrize( "plot_contour", [plotly_plot_contour, plt_plot_contour] ) def _create_study_with_failed_trial() -> Study: study = create_study() study.optimize(fail_objective, n_trials=1, catch=(ValueError,)) return study def _create_study_with_log_scale_and_str_category_2d() -> Study: study = create_study() distributions = { "param_a": FloatDistribution(1e-7, 1e-2, log=True), "param_b": CategoricalDistribution(["100", "101"]), } study.add_trial( create_trial( value=0.0, params={"param_a": 1e-6, "param_b": "101"}, distributions=distributions ) ) study.add_trial( create_trial( value=1.0, params={"param_a": 1e-5, "param_b": "100"}, distributions=distributions ) ) return study def _create_study_with_log_scale_and_str_category_3d() -> Study: study = create_study() distributions = { "param_a": FloatDistribution(1e-7, 1e-2, log=True), "param_b": CategoricalDistribution(["100", "101"]), "param_c": CategoricalDistribution(["one", "two"]), } study.add_trial( create_trial( value=0.0, params={"param_a": 1e-6, "param_b": "101", "param_c": "one"}, distributions=distributions, ) ) study.add_trial( create_trial( value=1.0, params={"param_a": 1e-5, "param_b": "100", "param_c": "two"}, distributions=distributions, ) ) return study def _create_study_mixture_category_types() -> Study: study = create_study() distributions: dict[str, BaseDistribution] = { "param_a": CategoricalDistribution([None, "100"]), "param_b": CategoricalDistribution([101, 102.0]), } study.add_trial( create_trial( value=0.0, params={"param_a": None, "param_b": 101}, distributions=distributions ) ) study.add_trial( create_trial( value=0.5, params={"param_a": "100", "param_b": 102.0}, distributions=distributions ) ) return study def _create_study_with_overlapping_params(direction: str) -> Study: study = create_study(direction=direction) distributions = { "param_a": FloatDistribution(1.0, 2.0), "param_b": CategoricalDistribution(["100", "101"]), "param_c": CategoricalDistribution(["foo", "bar"]), } study.add_trial( create_trial( value=0.0, params={"param_a": 1.0, "param_b": "101", "param_c": "foo"}, distributions=distributions, ) ) study.add_trial( create_trial( value=1.0, params={"param_a": 1.0, "param_b": "101", "param_c": "bar"}, distributions=distributions, ) ) study.add_trial( create_trial( value=1.0, params={"param_a": 2.0, "param_b": "100", "param_c": "foo"}, distributions=distributions, ) ) return study @parametrize_plot_contour def test_plot_contour_customized_target_name(plot_contour: Callable[..., Any]) -> None: params = ["param_a", "param_b"] study = prepare_study_with_trials() figure = plot_contour(study, params=params, target_name="Target Name") if isinstance(figure, go.Figure): assert figure.data[0]["colorbar"].title.text == "Target Name" elif isinstance(figure, Axes): assert figure.figure.axes[-1].get_ylabel() == "Target Name" @parametrize_plot_contour @pytest.mark.parametrize( "specific_create_study, params", [ [create_study, []], [create_study, ["param_a"]], [create_study, ["param_a", "param_b"]], [create_study, ["param_a", "param_b", "param_c"]], [create_study, ["param_a", "param_b", "param_c", "param_d"]], [create_study, None], [_create_study_with_failed_trial, []], [_create_study_with_failed_trial, ["param_a"]], [_create_study_with_failed_trial, ["param_a", "param_b"]], [_create_study_with_failed_trial, ["param_a", "param_b", "param_c"]], [_create_study_with_failed_trial, ["param_a", "param_b", "param_c", "param_d"]], [_create_study_with_failed_trial, None], [prepare_study_with_trials, []], [prepare_study_with_trials, ["param_a"]], [prepare_study_with_trials, ["param_a", "param_b"]], [prepare_study_with_trials, ["param_a", "param_b", "param_c"]], [prepare_study_with_trials, ["param_a", "param_b", "param_c", "param_d"]], [prepare_study_with_trials, None], [_create_study_with_log_scale_and_str_category_2d, None], [_create_study_with_log_scale_and_str_category_3d, None], [_create_study_mixture_category_types, None], ], ) def test_plot_contour( plot_contour: Callable[..., Any], specific_create_study: Callable[[], Study], params: list[str] | None, ) -> None: study = specific_create_study() figure = plot_contour(study, params=params) if isinstance(figure, go.Figure): figure.write_image(BytesIO()) else: plt.savefig(BytesIO()) plt.close() def test_target_is_none_and_study_is_multi_obj() -> None: study = create_study(directions=["minimize", "minimize"]) with pytest.raises(ValueError): _get_contour_info(study) @pytest.mark.parametrize( "specific_create_study", [create_study, _create_study_with_failed_trial], ) @pytest.mark.parametrize( "params", [ [], ["param_a"], ["param_a", "param_b"], ["param_a", "param_b", "param_c"], ["param_a", "param_b", "param_c", "param_d"], None, ], ) def test_get_contour_info_empty( specific_create_study: Callable[[], Study], params: list[str] | None ) -> None: study = specific_create_study() info = _get_contour_info(study, params=params) assert len(info.sorted_params) == 0 assert len(info.sub_plot_infos) == 0 def test_get_contour_info_non_exist_param_error() -> None: study = prepare_study_with_trials() with pytest.raises(ValueError): _get_contour_info(study, ["optuna"]) @pytest.mark.parametrize("params", [[], ["param_a"]]) def test_get_contour_info_too_short_params(params: list[str]) -> None: study = prepare_study_with_trials() info = _get_contour_info(study, params=params) assert len(info.sorted_params) == len(params) assert len(info.sub_plot_infos) == len(params) def test_get_contour_info_2_params() -> None: params = ["param_a", "param_b"] study = prepare_study_with_trials() info = _get_contour_info(study, params=params) assert info == _ContourInfo( sorted_params=params, sub_plot_infos=[ [ _SubContourInfo( xaxis=_AxisInfo( name="param_a", range=(0.925, 2.575), is_log=False, is_cat=False, indices=[0.925, 1.0, 2.5, 2.575], values=[1.0, None, 2.5], ), yaxis=_AxisInfo( name="param_b", range=(-0.1, 2.1), is_log=False, is_cat=False, indices=[-0.1, 0.0, 1.0, 2.0, 2.1], values=[2.0, 0.0, 1.0], ), z_values={(1, 3): 0.0, (2, 2): 1.0}, constraints=[True, True, True], ) ] ], reverse_scale=True, target_name="Objective Value", ) @pytest.mark.parametrize( "params", [ ["param_a", "param_b", "param_c"], ["param_a", "param_b", "param_c", "param_d"], None, ], ) def test_get_contour_info_more_than_2_params(params: list[str] | None) -> None: study = prepare_study_with_trials() n_params = len(params) if params is not None else 4 info = _get_contour_info(study, params=params) assert len(info.sorted_params) == n_params assert np.shape(np.asarray(info.sub_plot_infos, dtype=object)) == (n_params, n_params, 4) @pytest.mark.parametrize( "params", [ ["param_a", "param_b"], ["param_a", "param_b", "param_c"], ], ) def test_get_contour_info_customized_target(params: list[str]) -> None: study = prepare_study_with_trials() info = _get_contour_info( study, params=params, target=lambda t: t.params["param_d"], target_name="param_d" ) n_params = len(params) assert len(info.sorted_params) == n_params plot_shape = (1, 1, 4) if n_params == 2 else (n_params, n_params, 4) assert np.shape(np.asarray(info.sub_plot_infos, dtype=object)) == plot_shape @pytest.mark.parametrize( "params", [ ["param_a", "param_b"], # `x_axis` has one observation. ["param_b", "param_a"], # `y_axis` has one observation. ], ) def test_generate_contour_plot_for_few_observations(params: list[str]) -> None: study = create_study(direction="minimize") study.add_trial( create_trial( values=[0.0], params={"param_a": 1.0, "param_b": 2.0}, distributions={ "param_a": FloatDistribution(0.0, 3.0), "param_b": FloatDistribution(0.0, 3.0), }, ) ) study.add_trial( create_trial( values=[2.0], params={"param_b": 0.0}, distributions={"param_b": FloatDistribution(0.0, 3.0)}, ) ) info = _get_contour_info(study, params=params) assert info == _ContourInfo( sorted_params=sorted(params), sub_plot_infos=[ [ _SubContourInfo( xaxis=_AxisInfo( name=sorted(params)[0], range=(1.0, 1.0), is_log=False, is_cat=False, indices=[1.0], values=[1.0, None], ), yaxis=_AxisInfo( name=sorted(params)[1], range=(-0.1, 2.1), is_log=False, is_cat=False, indices=[-0.1, 0.0, 2.0, 2.1], values=[2.0, 0.0], ), z_values={}, constraints=[], ) ] ], reverse_scale=True, target_name="Objective Value", ) def test_get_contour_info_log_scale_and_str_category_2_params() -> None: # If the search space has two parameters, plot_contour generates a single plot. study = _create_study_with_log_scale_and_str_category_2d() info = _get_contour_info(study) assert info == _ContourInfo( sorted_params=["param_a", "param_b"], sub_plot_infos=[ [ _SubContourInfo( xaxis=_AxisInfo( name="param_a", range=(math.pow(10, -6.05), math.pow(10, -4.95)), is_log=True, is_cat=False, indices=[math.pow(10, -6.05), 1e-6, 1e-5, math.pow(10, -4.95)], values=[1e-6, 1e-5], ), yaxis=_AxisInfo( name="param_b", range=(-0.05, 1.05), is_log=False, is_cat=True, indices=["100", "101"], values=["101", "100"], ), z_values={(1, 1): 0.0, (2, 0): 1.0}, constraints=[True, True], ) ] ], reverse_scale=True, target_name="Objective Value", ) def test_get_contour_info_log_scale_and_str_category_more_than_2_params() -> None: # If the search space has three parameters, plot_contour generates nine plots. study = _create_study_with_log_scale_and_str_category_3d() info = _get_contour_info(study) params = ["param_a", "param_b", "param_c"] assert info.sorted_params == params assert np.shape(np.asarray(info.sub_plot_infos, dtype=object)) == (3, 3, 4) ranges = { "param_a": (math.pow(10, -6.05), math.pow(10, -4.95)), "param_b": (-0.05, 1.05), "param_c": (-0.05, 1.05), } is_log = {"param_a": True, "param_b": False, "param_c": False} is_cat = {"param_a": False, "param_b": True, "param_c": True} indices: dict[str, list[str | float]] = { "param_a": [math.pow(10, -6.05), 1e-6, 1e-5, math.pow(10, -4.95)], "param_b": ["100", "101"], "param_c": ["one", "two"], } values = {"param_a": [1e-6, 1e-5], "param_b": ["101", "100"], "param_c": ["one", "two"]} def _check_axis(axis: _AxisInfo, name: str) -> None: assert axis.name == name assert axis.range == ranges[name] assert axis.is_log == is_log[name] assert axis.is_cat == is_cat[name] assert axis.indices == indices[name] assert axis.values == values[name] for yi in range(3): for xi in range(3): xaxis = info.sub_plot_infos[yi][xi].xaxis yaxis = info.sub_plot_infos[yi][xi].yaxis x_param = params[xi] y_param = params[yi] _check_axis(xaxis, x_param) _check_axis(yaxis, y_param) z_values = info.sub_plot_infos[yi][xi].z_values if xi == yi: assert z_values == {} else: for i, v in enumerate([0.0, 1.0]): x_value = xaxis.values[i] y_value = yaxis.values[i] assert x_value is not None assert y_value is not None xi = xaxis.indices.index(x_value) yi = yaxis.indices.index(y_value) assert z_values[(xi, yi)] == v def test_get_contour_info_mixture_category_types() -> None: study = _create_study_mixture_category_types() info = _get_contour_info(study) assert info == _ContourInfo( sorted_params=["param_a", "param_b"], sub_plot_infos=[ [ _SubContourInfo( xaxis=_AxisInfo( name="param_a", range=(-0.05, 1.05), is_log=False, is_cat=True, indices=["100", "None"], values=["None", "100"], ), yaxis=_AxisInfo( name="param_b", range=(100.95, 102.05), is_log=False, is_cat=False, indices=[100.95, 101, 102, 102.05], values=[101.0, 102.0], ), z_values={(0, 2): 0.5, (1, 1): 0.0}, constraints=[True, True], ) ] ], reverse_scale=True, target_name="Objective Value", ) @pytest.mark.parametrize("value", [float("inf"), -float("inf")]) def test_get_contour_info_nonfinite_removed(value: float) -> None: study = prepare_study_with_trials(value_for_first_trial=value) info = _get_contour_info(study, params=["param_b", "param_d"]) assert info == _ContourInfo( sorted_params=["param_b", "param_d"], sub_plot_infos=[ [ _SubContourInfo( xaxis=_AxisInfo( name="param_b", range=(-0.05, 1.05), is_log=False, is_cat=False, indices=[-0.05, 0.0, 1.0, 1.05], values=[0.0, 1.0], ), yaxis=_AxisInfo( name="param_d", range=(1.9, 4.1), is_log=False, is_cat=False, indices=[1.9, 2.0, 4.0, 4.1], values=[4.0, 2.0], ), z_values={(1, 2): 2.0, (2, 1): 1.0}, constraints=[True, True], ) ] ], reverse_scale=True, target_name="Objective Value", ) @pytest.mark.parametrize("objective", (0, 1)) @pytest.mark.parametrize("value", (float("inf"), -float("inf"))) def test_get_contour_info_nonfinite_multiobjective(objective: int, value: float) -> None: study = prepare_study_with_trials(n_objectives=2, value_for_first_trial=value) info = _get_contour_info( study, params=["param_b", "param_d"], target=lambda t: t.values[objective], target_name="Target Name", ) assert info == _ContourInfo( sorted_params=["param_b", "param_d"], sub_plot_infos=[ [ _SubContourInfo( xaxis=_AxisInfo( name="param_b", range=(-0.05, 1.05), is_log=False, is_cat=False, indices=[-0.05, 0.0, 1.0, 1.05], values=[0.0, 1.0], ), yaxis=_AxisInfo( name="param_d", range=(1.9, 4.1), is_log=False, is_cat=False, indices=[1.9, 2.0, 4.0, 4.1], values=[4.0, 2.0], ), z_values={(1, 2): 2.0, (2, 1): 1.0}, constraints=[True, True], ) ] ], reverse_scale=True, target_name="Target Name", ) @pytest.mark.parametrize("direction,expected", (("minimize", 0.0), ("maximize", 1.0))) def test_get_contour_info_overlapping_params(direction: str, expected: float) -> None: study = _create_study_with_overlapping_params(direction) info = _get_contour_info(study, params=["param_a", "param_b"]) assert info == _ContourInfo( sorted_params=["param_a", "param_b"], sub_plot_infos=[ [ _SubContourInfo( xaxis=_AxisInfo( name="param_a", range=(0.95, 2.05), is_log=False, is_cat=False, indices=[0.95, 1.0, 2.0, 2.05], values=[1.0, 1.0, 2.0], ), yaxis=_AxisInfo( name="param_b", range=(-0.05, 1.05), is_log=False, is_cat=True, indices=["100", "101"], values=["101", "101", "100"], ), z_values={(1, 1): expected, (2, 0): 1.0}, constraints=[True, True, True], ) ] ], reverse_scale=False if direction == "maximize" else True, target_name="Objective Value", ) @pytest.mark.parametrize("direction", ["minimize", "maximize"]) def test_color_map(direction: str) -> None: study = create_study(direction=direction) for i in range(3): study.add_trial( create_trial( value=float(i), params={"param_a": float(i), "param_b": float(i)}, distributions={ "param_a": FloatDistribution(0.0, 3.0), "param_b": FloatDistribution(0.0, 3.0), }, ) ) # `target` is `None`. contour = plotly_plot_contour(study).data[0] assert COLOR_SCALE == [v[1] for v in contour["colorscale"]] if direction == "minimize": assert contour["reversescale"] else: assert not contour["reversescale"] # When `target` is not `None`, `reversescale` is always `True`. contour = plotly_plot_contour(study, target=lambda t: t.number, target_name="Number").data[0] assert COLOR_SCALE == [v[1] for v in contour["colorscale"]] assert contour["reversescale"] # Multi-objective optimization. study = create_study(directions=[direction, direction]) for i in range(3): study.add_trial( create_trial( values=[float(i), float(i)], params={"param_a": float(i), "param_b": float(i)}, distributions={ "param_a": FloatDistribution(0.0, 3.0), "param_b": FloatDistribution(0.0, 3.0), }, ) ) contour = plotly_plot_contour(study, target=lambda t: t.number, target_name="Number").data[0] assert COLOR_SCALE == [v[1] for v in contour["colorscale"]] assert contour["reversescale"] optuna-4.1.0/tests/visualization_tests/test_edf.py000066400000000000000000000154451471332314300224700ustar00rootroot00000000000000from __future__ import annotations from io import BytesIO from typing import Any from typing import Callable import numpy as np import pytest import optuna from optuna import Study from optuna.study import create_study from optuna.testing.visualization import prepare_study_with_trials from optuna.trial import create_trial from optuna.visualization import plot_edf as plotly_plot_edf from optuna.visualization._edf import _EDFInfo from optuna.visualization._edf import _get_edf_info from optuna.visualization._edf import NUM_SAMPLES_X_AXIS from optuna.visualization._plotly_imports import _imports as plotly_imports from optuna.visualization.matplotlib import plot_edf as plt_plot_edf from optuna.visualization.matplotlib._matplotlib_imports import _imports as plt_imports if plotly_imports.is_successful(): from optuna.visualization._plotly_imports import go if plt_imports.is_successful(): from optuna.visualization.matplotlib._matplotlib_imports import Axes from optuna.visualization.matplotlib._matplotlib_imports import plt parametrized_plot_edf = pytest.mark.parametrize("plot_edf", [plotly_plot_edf, plt_plot_edf]) def save_static_image(figure: go.Figure | Axes | np.ndarray) -> None: if isinstance(figure, go.Figure): figure.write_image(BytesIO()) else: plt.savefig(BytesIO()) plt.close() @parametrized_plot_edf def _validate_edf_values(edf_values: np.ndarray) -> None: np_values = np.array(edf_values) # Confirms that the values are monotonically non-decreasing. assert np.all(np.diff(np_values) >= 0.0) # Confirms that the values are in [0,1]. assert np.all((0 <= np_values) & (np_values <= 1)) @parametrized_plot_edf def test_target_is_none_and_study_is_multi_obj(plot_edf: Callable[..., Any]) -> None: study = create_study(directions=["minimize", "minimize"]) with pytest.raises(ValueError): plot_edf(study) @parametrized_plot_edf @pytest.mark.parametrize("direction", ["minimize", "maximize"]) def test_edf_plot_no_trials(plot_edf: Callable[..., Any], direction: str) -> None: figure = plot_edf(create_study(direction=direction)) save_static_image(figure) @parametrized_plot_edf @pytest.mark.parametrize("direction", ["minimize", "maximize"]) @pytest.mark.parametrize("num_studies", [0, 1, 2]) def test_edf_plot_no_trials_studies( plot_edf: Callable[..., Any], direction: str, num_studies: int ) -> None: studies = [create_study(direction=direction) for _ in range(num_studies)] figure = plot_edf(studies) save_static_image(figure) @parametrized_plot_edf @pytest.mark.parametrize("direction", ["minimize", "maximize"]) @pytest.mark.parametrize("num_studies", [0, 1, 2]) def test_plot_edf_with_multiple_studies( plot_edf: Callable[..., Any], direction: str, num_studies: int ) -> None: studies = [] for _ in range(num_studies): study = create_study(direction=direction) study.optimize(lambda t: t.suggest_float("x", 0, 5), n_trials=10) studies.append(study) figure = plot_edf(studies) save_static_image(figure) @parametrized_plot_edf def test_plot_edf_with_target(plot_edf: Callable[..., Any]) -> None: study = create_study() study.optimize(lambda t: t.suggest_float("x", 0, 5), n_trials=10) with pytest.warns(UserWarning): figure = plot_edf(study, target=lambda t: t.params["x"]) save_static_image(figure) @parametrized_plot_edf @pytest.mark.parametrize("target_name", [None, "Target Name"]) def test_plot_edf_with_target_name(plot_edf: Callable[..., Any], target_name: str | None) -> None: study = create_study() study.optimize(lambda t: t.suggest_float("x", 0, 5), n_trials=10) if target_name is None: figure = plot_edf(study) else: figure = plot_edf(study, target_name=target_name) expected = target_name if target_name is not None else "Objective Value" if isinstance(figure, go.Figure): assert figure.layout.xaxis.title.text == expected elif isinstance(figure, Axes): assert figure.xaxis.label.get_text() == expected save_static_image(figure) def test_empty_edf_info() -> None: def _assert_empty(info: _EDFInfo) -> None: assert info.lines == [] np.testing.assert_array_equal(info.x_values, np.array([])) edf_info = _get_edf_info([]) _assert_empty(edf_info) study = create_study() edf_info = _get_edf_info(study) _assert_empty(edf_info) trial = study.ask() study.tell(trial, state=optuna.trial.TrialState.PRUNED) edf_info = _get_edf_info(study) _assert_empty(edf_info) @pytest.mark.parametrize("n_studies", [1, 2, 3]) @pytest.mark.parametrize("target", [None, lambda t: t.params["x"]]) def test_get_edf_info(n_studies: int, target: Callable[[optuna.trial.FrozenTrial], float]) -> None: studies = [] n_trials = 3 max_target = 5 for i in range(n_studies): study = create_study(study_name=str(i)) study.optimize(lambda t: t.suggest_float("x", 0, max_target), n_trials=n_trials) studies.append(study) info = _get_edf_info(studies, target=target, target_name="Target Name") assert info.x_values.shape == (NUM_SAMPLES_X_AXIS,) assert all([0 <= x <= max_target for x in info.x_values]) assert len(info.lines) == n_studies for i, line in enumerate(info.lines): assert str(i) == line.study_name assert line.y_values.shape == (NUM_SAMPLES_X_AXIS,) _validate_edf_values(line.y_values) @pytest.mark.parametrize("value", [float("inf"), -float("inf")]) def test_nonfinite_removed(value: float) -> None: study = prepare_study_with_trials(value_for_first_trial=value) edf_info = _get_edf_info(study) assert all(np.isfinite(edf_info.x_values)) @pytest.mark.parametrize("objective", (0, 1)) @pytest.mark.parametrize("value", (float("inf"), -float("inf"))) def test_nonfinite_multiobjective(objective: int, value: float) -> None: study = prepare_study_with_trials(n_objectives=2, value_for_first_trial=value) edf_info = _get_edf_info( study, target=lambda t: t.values[objective], target_name="Target Name" ) assert all(np.isfinite(edf_info.x_values)) def test_inconsistent_number_of_trial_values() -> None: studies: list[Study] = [] n_studies = 5 for i in range(n_studies): study = prepare_study_with_trials() if i % 2 == 0: study.add_trial(create_trial(value=1.0)) studies.append(study) edf_info = _get_edf_info(studies) x_values = edf_info.x_values min_objective = 0.0 max_objective = 2.0 assert np.min(x_values) == min_objective assert np.max(x_values) == max_objective assert len(x_values) == NUM_SAMPLES_X_AXIS lines = edf_info.lines assert len(lines) == n_studies for line, study in zip(lines, studies): assert line.study_name == study.study_name _validate_edf_values(line.y_values) optuna-4.1.0/tests/visualization_tests/test_hypervolume_history.py000066400000000000000000000036331471332314300260660ustar00rootroot00000000000000from typing import Sequence import numpy as np import pytest from optuna.samplers import NSGAIISampler from optuna.study import create_study from optuna.trial import FrozenTrial from optuna.trial import Trial from optuna.visualization._hypervolume_history import _get_hypervolume_history_info from optuna.visualization._hypervolume_history import _HypervolumeHistoryInfo @pytest.mark.parametrize( "directions", [ ["minimize", "minimize"], ["minimize", "maximize"], ["maximize", "minimize"], ["maximize", "maximize"], ], ) def test_get_optimization_history_info(directions: str) -> None: signs = [1 if d == "minimize" else -1 for d in directions] def objective(trial: Trial) -> Sequence[float]: def impl(trial: Trial) -> Sequence[float]: if trial.number == 0: return 1.5, 1.5 # dominated by the reference_point elif trial.number == 1: return 0.75, 0.75 elif trial.number == 2: return 0.5, 0.5 # dominates Trial #1 elif trial.number == 3: return 0.5, 0.5 # dominates Trial #1 elif trial.number == 4: return 0.75, 0.25 # incomparable return 0.0, 0.0 # dominates all values = impl(trial) return signs[0] * values[0], signs[1] * values[1] def constraints(trial: FrozenTrial) -> Sequence[float]: if trial.number == 2: return (1,) # infeasible return (0,) # feasible sampler = NSGAIISampler(constraints_func=constraints) study = create_study(directions=directions, sampler=sampler) study.optimize(objective, n_trials=6) reference_point = np.asarray(signs) info = _get_hypervolume_history_info(study, reference_point) assert info == _HypervolumeHistoryInfo( trial_numbers=[0, 1, 2, 3, 4, 5], values=[0.0, 0.0625, 0.0625, 0.25, 0.3125, 1.0] ) optuna-4.1.0/tests/visualization_tests/test_intermediate_plot.py000066400000000000000000000107261471332314300254370ustar00rootroot00000000000000from io import BytesIO from typing import Any from typing import Callable from typing import Sequence import pytest from optuna.study import create_study from optuna.testing.objectives import fail_objective from optuna.trial import FrozenTrial from optuna.trial import Trial import optuna.visualization._intermediate_values from optuna.visualization._intermediate_values import _get_intermediate_plot_info from optuna.visualization._intermediate_values import _IntermediatePlotInfo from optuna.visualization._intermediate_values import _TrialInfo from optuna.visualization._plotly_imports import go import optuna.visualization.matplotlib._intermediate_values from optuna.visualization.matplotlib._matplotlib_imports import plt def test_intermediate_plot_info() -> None: # Test with no trials. study = create_study(direction="minimize") assert _get_intermediate_plot_info(study) == _IntermediatePlotInfo(trial_infos=[]) # Test with a trial with intermediate values. def objective(trial: Trial, report_intermediate_values: bool) -> float: if report_intermediate_values: trial.report(1.0, step=0) trial.report(2.0, step=1) return 0.0 study = create_study() study.optimize(lambda t: objective(t, True), n_trials=1) assert _get_intermediate_plot_info(study) == _IntermediatePlotInfo( trial_infos=[ _TrialInfo( trial_number=0, sorted_intermediate_values=[(0, 1.0), (1, 2.0)], feasible=True ) ] ) # Test a study with one trial with intermediate values and # one trial without intermediate values. # Expect the trial with no intermediate values to be ignored. study.optimize(lambda t: objective(t, False), n_trials=1) assert _get_intermediate_plot_info(study) == _IntermediatePlotInfo( trial_infos=[ _TrialInfo( trial_number=0, sorted_intermediate_values=[(0, 1.0), (1, 2.0)], feasible=True ) ] ) # Test a study of only one trial that has no intermediate values. study = create_study() study.optimize(lambda t: objective(t, False), n_trials=1) assert _get_intermediate_plot_info(study) == _IntermediatePlotInfo(trial_infos=[]) # Ignore failed trials. study = create_study() study.optimize(fail_objective, n_trials=1, catch=(ValueError,)) assert _get_intermediate_plot_info(study) == _IntermediatePlotInfo(trial_infos=[]) # Test a study with constraints def objective_with_constraints(trial: Trial) -> float: trial.set_user_attr("constraint", [trial.number % 2]) trial.report(1.0, step=0) trial.report(2.0, step=1) return 0.0 def constraints(trial: FrozenTrial) -> Sequence[float]: return trial.user_attrs["constraint"] study = create_study(sampler=optuna.samplers.NSGAIIISampler(constraints_func=constraints)) study.optimize(objective_with_constraints, n_trials=2) assert _get_intermediate_plot_info(study) == _IntermediatePlotInfo( trial_infos=[ _TrialInfo( trial_number=0, sorted_intermediate_values=[(0, 1.0), (1, 2.0)], feasible=True ), _TrialInfo( trial_number=1, sorted_intermediate_values=[(0, 1.0), (1, 2.0)], feasible=False ), ] ) @pytest.mark.parametrize( "plotter", [ optuna.visualization._intermediate_values._get_intermediate_plot, optuna.visualization.matplotlib._intermediate_values._get_intermediate_plot, ], ) @pytest.mark.parametrize( "info", [ _IntermediatePlotInfo(trial_infos=[]), _IntermediatePlotInfo( trial_infos=[ _TrialInfo( trial_number=0, sorted_intermediate_values=[(0, 1.0), (1, 2.0)], feasible=True ) ] ), _IntermediatePlotInfo( trial_infos=[ _TrialInfo( trial_number=0, sorted_intermediate_values=[(0, 1.0), (1, 2.0)], feasible=True ), _TrialInfo( trial_number=1, sorted_intermediate_values=[(1, 2.0), (0, 1.0)], feasible=False ), ] ), ], ) def test_plot_intermediate_values( plotter: Callable[[_IntermediatePlotInfo], Any], info: _IntermediatePlotInfo ) -> None: figure = plotter(info) if isinstance(figure, go.Figure): figure.write_image(BytesIO()) else: plt.savefig(BytesIO()) plt.close() optuna-4.1.0/tests/visualization_tests/test_optimization_history.py000066400000000000000000000354061471332314300262400ustar00rootroot00000000000000from __future__ import annotations from io import BytesIO import math from typing import Sequence import numpy as np import pytest from optuna import samplers from optuna import TrialPruned from optuna.study import create_study from optuna.testing.objectives import fail_objective from optuna.testing.objectives import pruned_objective from optuna.trial import FrozenTrial from optuna.trial import Trial from optuna.visualization._optimization_history import _get_optimization_history_info_list from optuna.visualization._optimization_history import _get_optimization_history_plot from optuna.visualization._optimization_history import _OptimizationHistoryInfo from optuna.visualization._optimization_history import _ValuesInfo from optuna.visualization._optimization_history import _ValueState def test_target_is_none_and_study_is_multi_obj() -> None: study = create_study(directions=["minimize", "minimize"]) with pytest.raises(ValueError): _get_optimization_history_info_list( study, target=None, target_name="Objective Value", error_bar=False ) @pytest.mark.parametrize("direction", ["minimize", "maximize"]) @pytest.mark.parametrize("error_bar", [False, True]) def test_warn_default_target_name_with_customized_target(direction: str, error_bar: bool) -> None: # Single study. study = create_study(direction=direction) with pytest.warns(UserWarning): _get_optimization_history_info_list( study, target=lambda t: t.number, target_name="Objective Value", error_bar=error_bar ) # Multiple studies. studies = [create_study(direction=direction) for _ in range(10)] with pytest.warns(UserWarning): _get_optimization_history_info_list( studies, target=lambda t: t.number, target_name="Objective Value", error_bar=error_bar ) @pytest.mark.parametrize("direction", ["minimize", "maximize"]) @pytest.mark.parametrize("error_bar", [False, True]) def test_info_with_no_trials(direction: str, error_bar: bool) -> None: # Single study. study = create_study(direction=direction) info_list = _get_optimization_history_info_list( study, target=None, target_name="Objective Value", error_bar=error_bar ) assert info_list == [] # Multiple studies. studies = [create_study(direction=direction) for _ in range(10)] info_list = _get_optimization_history_info_list( studies, target=None, target_name="Objective Value", error_bar=error_bar ) assert info_list == [] @pytest.mark.parametrize("direction", ["minimize", "maximize"]) @pytest.mark.parametrize("error_bar", [False, True]) def test_ignore_failed_trials(direction: str, error_bar: bool) -> None: # Single study. study = create_study(direction=direction) study.optimize(fail_objective, n_trials=1, catch=(ValueError,)) info_list = _get_optimization_history_info_list( study, target=None, target_name="Objective Value", error_bar=error_bar ) assert info_list == [] # Multiple studies. studies = [create_study(direction=direction) for _ in range(10)] for study in studies: study.optimize(fail_objective, n_trials=1, catch=(ValueError,)) info_list = _get_optimization_history_info_list( studies, target=None, target_name="Objective Value", error_bar=error_bar ) assert info_list == [] @pytest.mark.parametrize("direction", ["minimize", "maximize"]) @pytest.mark.parametrize("error_bar", [False, True]) def test_ignore_pruned_trials(direction: str, error_bar: bool) -> None: # Single study. study = create_study(direction=direction) study.optimize(pruned_objective, n_trials=1, catch=(ValueError,)) info_list = _get_optimization_history_info_list( study, target=None, target_name="Objective Value", error_bar=error_bar ) assert info_list == [] # Multiple studies. studies = [create_study(direction=direction) for _ in range(10)] for study in studies: study.optimize(pruned_objective, n_trials=1, catch=(ValueError,)) info_list = _get_optimization_history_info_list( studies, target=None, target_name="Objective Value", error_bar=error_bar ) assert info_list == [] @pytest.mark.parametrize("direction", ["minimize", "maximize"]) def test_get_optimization_history_info_list(direction: str) -> None: target_name = "Target Name" def objective(trial: Trial) -> float: if trial.number == 0: return 1.0 elif trial.number == 1: return 2.0 elif trial.number == 2: return 0.0 return 0.0 # Test with a trial. study = create_study(direction=direction) study.optimize(objective, n_trials=3) info_list = _get_optimization_history_info_list( study, target=None, target_name=target_name, error_bar=False ) best_values = [1.0, 1.0, 0.0] if direction == "minimize" else [1.0, 2.0, 2.0] assert info_list == [ _OptimizationHistoryInfo( [0, 1, 2], _ValuesInfo([1.0, 2.0, 0.0], None, target_name, [_ValueState.Feasible] * 3), _ValuesInfo(best_values, None, "Best Value", [_ValueState.Feasible] * 3), ) ] # Test customized target. info_list = _get_optimization_history_info_list( study, target=lambda t: t.number, target_name=target_name, error_bar=False ) assert info_list == [ _OptimizationHistoryInfo( [0, 1, 2], _ValuesInfo([0.0, 1.0, 2.0], None, target_name, [_ValueState.Feasible] * 3), None, ) ] @pytest.mark.parametrize("direction", ["minimize", "maximize"]) def test_get_optimization_history_info_list_with_multiple_studies(direction: str) -> None: n_studies = 10 base_values = [1.0, 2.0, 0.0] base_best_values = [1.0, 1.0, 0.0] if direction == "minimize" else [1.0, 2.0, 2.0] target_name = "Target Name" value_states = [_ValueState.Feasible] * 3 # Test with trials. studies = [create_study(direction=direction) for _ in range(n_studies)] for i, study in enumerate(studies): study.optimize(lambda t: base_values[t.number] + i, n_trials=3) info_list = _get_optimization_history_info_list( studies, target=None, target_name=target_name, error_bar=False ) for i, info in enumerate(info_list): values_i = [v + i for v in base_values] best_values_i = [v + i for v in base_best_values] assert info == _OptimizationHistoryInfo( [0, 1, 2], _ValuesInfo(values_i, None, f"{target_name} of {studies[i].study_name}", value_states), _ValuesInfo( best_values_i, None, f"Best Value of {studies[i].study_name}", value_states ), ) # Test customized target. info_list = _get_optimization_history_info_list( studies, target=lambda t: t.number, target_name=target_name, error_bar=False ) for i, info in enumerate(info_list): assert info == _OptimizationHistoryInfo( [0, 1, 2], _ValuesInfo( [0.0, 1.0, 2.0], None, f"{target_name} of {studies[i].study_name}", value_states ), None, ) @pytest.mark.parametrize("direction", ["minimize", "maximize"]) def test_get_optimization_history_info_list_with_infeasible(direction: str) -> None: target_name = "Target Name" def objective(trial: Trial) -> float: if trial.number == 0: trial.set_user_attr("constraint", [0]) return 1.0 elif trial.number == 1: trial.set_user_attr("constraint", [0]) return 2.0 elif trial.number == 2: trial.set_user_attr("constraint", [1]) return 3.0 return 0.0 def constraints_func(trial: FrozenTrial) -> Sequence[float]: return trial.user_attrs["constraint"] sampler = samplers.TPESampler(constraints_func=constraints_func) study = create_study(sampler=sampler, direction=direction) study.optimize(objective, n_trials=3) info_list = _get_optimization_history_info_list( study, target=None, target_name=target_name, error_bar=False ) best_values = [1.0, 1.0, 1.0] if direction == "minimize" else [1.0, 2.0, 2.0] states = [_ValueState.Feasible, _ValueState.Feasible, _ValueState.Infeasible] assert info_list == [ _OptimizationHistoryInfo( [0, 1, 2], _ValuesInfo( [1.0, 2.0, 3.0], None, target_name, states, ), _ValuesInfo(best_values, None, "Best Value", states), ) ] @pytest.mark.parametrize("direction", ["minimize", "maximize"]) def test_get_optimization_history_info_list_with_pruned_trial(direction: str) -> None: target_name = "Target Name" def objective(trial: Trial) -> float: if trial.number == 0: return 1.0 elif trial.number == 1: return 2.0 elif trial.number == 2: raise TrialPruned() return 0.0 study = create_study(direction=direction) study.optimize(objective, n_trials=3) info_list = _get_optimization_history_info_list( study, target=None, target_name=target_name, error_bar=False ) assert info_list[0].values_info.states == [ _ValueState.Feasible, _ValueState.Feasible, _ValueState.Incomplete, ] assert math.isnan(info_list[0].values_info.values[2]) if info_list[0].best_values_info is not None: assert ( info_list[0].best_values_info.values == [1.0, 1.0, 1.0] if direction == "minimize" else [1.0, 2.0, 2.0] ) @pytest.mark.parametrize("direction", ["minimize", "maximize"]) def test_get_optimization_history_info_list_with_error_bar(direction: str) -> None: n_studies = 10 target_name = "Target Name" def objective(trial: Trial) -> float: if trial.number == 0: return 1.0 elif trial.number == 1: return 2.0 elif trial.number == 2: return 0.0 return 0.0 # Test with trials. studies = [create_study(direction=direction) for _ in range(n_studies)] for study in studies: study.optimize(objective, n_trials=3) info_list = _get_optimization_history_info_list( study, target=None, target_name=target_name, error_bar=True ) best_values = [1.0, 1.0, 0.0] if direction == "minimize" else [1.0, 2.0, 2.0] assert info_list == [ _OptimizationHistoryInfo( [0, 1, 2], _ValuesInfo([1.0, 2.0, 0.0], [0.0, 0.0, 0.0], target_name, [_ValueState.Feasible] * 3), _ValuesInfo(best_values, [0.0, 0.0, 0.0], "Best Value", [_ValueState.Feasible] * 3), ) ] # Test customized target. info_list = _get_optimization_history_info_list( study, target=lambda t: t.number, target_name=target_name, error_bar=True ) assert info_list == [ _OptimizationHistoryInfo( [0, 1, 2], _ValuesInfo([0.0, 1.0, 2.0], [0.0, 0.0, 0.0], target_name, [_ValueState.Feasible] * 3), None, ) ] @pytest.mark.parametrize("direction", ["minimize", "maximize"]) def test_error_bar_in_optimization_history(direction: str) -> None: def objective(trial: Trial) -> float: return trial.suggest_float("x", 0, 1) studies = [create_study(direction=direction) for _ in range(3)] suggested_params = [0.1, 0.3, 0.2] for x, study in zip(suggested_params, studies): study.enqueue_trial({"x": x}) study.optimize(objective, n_trials=1) info_list = _get_optimization_history_info_list( studies, target=None, target_name="Objective Value", error_bar=True ) mean = np.mean(suggested_params).item() std = np.std(suggested_params).item() value_states = [_ValueState.Feasible] assert info_list == [ _OptimizationHistoryInfo( [0], _ValuesInfo([mean], [std], "Objective Value", value_states), _ValuesInfo([mean], [std], "Best Value", value_states), ) ] optimization_history_info_lists = [ [], # Empty info. [ # Vanilla. _OptimizationHistoryInfo( [0, 1, 2], _ValuesInfo([1.0, 2.0, 0.0], None, "Dummy", [_ValueState.Feasible] * 3), None, ) ], [ # with infeasible trial. _OptimizationHistoryInfo( [0, 1, 2], _ValuesInfo([1.0, 2.0, 0.0], None, "Dummy", [_ValueState.Infeasible] * 3), None, ) ], [ # with incomplete trial. _OptimizationHistoryInfo( [0, 1, 2], _ValuesInfo([1.0, 2.0, 0.0], None, "Dummy", [_ValueState.Incomplete] * 3), None, ) ], [ # Multiple. _OptimizationHistoryInfo( [0, 1, 2], _ValuesInfo([1.0, 2.0, 0.0], None, "Dummy", [_ValueState.Feasible] * 3), None, ), _OptimizationHistoryInfo( [0, 1, 2], _ValuesInfo([1.0, 1.0, 1.0], None, "Dummy", [_ValueState.Feasible] * 3), None, ), _OptimizationHistoryInfo( [0, 1, 2], _ValuesInfo([1.0, 1.0, 1.0], None, "Dummy", [_ValueState.Infeasible] * 3), None, ), ], [ # With best values. _OptimizationHistoryInfo( [0, 1, 2], _ValuesInfo([1.0, 2.0, 0.0], None, "Dummy", [_ValueState.Feasible] * 3), _ValuesInfo([1.0, 1.0, 1.0], None, "Best Value", [_ValueState.Feasible] * 3), ) ], [ # With error bar. _OptimizationHistoryInfo( [0, 1, 2], _ValuesInfo([1.0, 2.0, 0.0], [1.0, 2.0, 0.0], "Dummy", [_ValueState.Feasible] * 3), None, ) ], [ # With best values and error bar. _OptimizationHistoryInfo( [0, 1, 2], _ValuesInfo([1.0, 2.0, 0.0], [1.0, 2.0, 0.0], "Dummy", [_ValueState.Feasible] * 3), _ValuesInfo( [1.0, 1.0, 1.0], [1.0, 2.0, 0.0], "Best Value", [_ValueState.Feasible] * 3 ), ) ], ] @pytest.mark.parametrize("target_name", ["Objective Value", "Target Name"]) @pytest.mark.parametrize("info_list", optimization_history_info_lists) def test_get_optimization_history_plot( target_name: str, info_list: list[_OptimizationHistoryInfo] ) -> None: figure = _get_optimization_history_plot(info_list, target_name=target_name) assert figure.layout.yaxis.title.text == target_name expected_legends = [] for info in info_list: expected_legends.append(info.values_info.label_name) expected_legends.append("Infeasible Trial") if info.best_values_info is not None: expected_legends.append(info.best_values_info.label_name) legends = [scatter.name for scatter in figure.data if scatter.name is not None] assert sorted(legends) == sorted(expected_legends) figure.write_image(BytesIO()) optuna-4.1.0/tests/visualization_tests/test_parallel_coordinate.py000066400000000000000000000604341471332314300257330ustar00rootroot00000000000000from __future__ import annotations from io import BytesIO import math from typing import Any from typing import Callable import numpy as np import pytest from optuna.distributions import BaseDistribution from optuna.distributions import CategoricalDistribution from optuna.distributions import FloatDistribution from optuna.study import create_study from optuna.study import Study from optuna.testing.objectives import fail_objective from optuna.testing.visualization import prepare_study_with_trials from optuna.trial import create_trial from optuna.visualization import matplotlib from optuna.visualization import plot_parallel_coordinate as plotly_plot_parallel_coordinate from optuna.visualization._parallel_coordinate import _DimensionInfo from optuna.visualization._parallel_coordinate import _get_parallel_coordinate_info from optuna.visualization._parallel_coordinate import _ParallelCoordinateInfo from optuna.visualization._plotly_imports import go from optuna.visualization._utils import COLOR_SCALE from optuna.visualization.matplotlib._matplotlib_imports import plt parametrize_plot_parallel_coordinate = pytest.mark.parametrize( "plot_parallel_coordinate", [plotly_plot_parallel_coordinate, matplotlib.plot_parallel_coordinate], ) def _create_study_with_failed_trial() -> Study: study = create_study() study.optimize(fail_objective, n_trials=1, catch=(ValueError,)) return study def _create_study_with_categorical_params() -> Study: study_categorical_params = create_study() distributions: dict[str, BaseDistribution] = { "category_a": CategoricalDistribution(("preferred", "opt")), "category_b": CategoricalDistribution(("net", "una")), } study_categorical_params.add_trial( create_trial( value=0.0, params={"category_a": "preferred", "category_b": "net"}, distributions=distributions, ) ) study_categorical_params.add_trial( create_trial( value=2.0, params={"category_a": "opt", "category_b": "una"}, distributions=distributions, ) ) return study_categorical_params def _create_study_with_numeric_categorical_params() -> Study: study_categorical_params = create_study() distributions: dict[str, BaseDistribution] = { "category_a": CategoricalDistribution((1, 2)), "category_b": CategoricalDistribution((10, 20, 30)), } study_categorical_params.add_trial( create_trial( value=0.0, params={"category_a": 2, "category_b": 20}, distributions=distributions, ) ) study_categorical_params.add_trial( create_trial( value=1.0, params={"category_a": 1, "category_b": 30}, distributions=distributions, ) ) study_categorical_params.add_trial( create_trial( value=2.0, params={"category_a": 2, "category_b": 10}, distributions=distributions, ) ) return study_categorical_params def _create_study_with_log_params() -> Study: study_log_params = create_study() distributions: dict[str, BaseDistribution] = { "param_a": FloatDistribution(1e-7, 1e-2, log=True), "param_b": FloatDistribution(1, 1000, log=True), } study_log_params.add_trial( create_trial( value=0.0, params={"param_a": 1e-6, "param_b": 10}, distributions=distributions, ) ) study_log_params.add_trial( create_trial( value=1.0, params={"param_a": 2e-5, "param_b": 200}, distributions=distributions, ) ) study_log_params.add_trial( create_trial( value=0.1, params={"param_a": 1e-4, "param_b": 30}, distributions=distributions, ) ) return study_log_params def _create_study_with_log_scale_and_str_and_numeric_category() -> Study: study_multi_distro_params = create_study() distributions: dict[str, BaseDistribution] = { "param_a": CategoricalDistribution(("preferred", "opt")), "param_b": CategoricalDistribution((1, 2, 10)), "param_c": FloatDistribution(1, 1000, log=True), "param_d": CategoricalDistribution((1, -1, 2)), } study_multi_distro_params.add_trial( create_trial( value=0.0, params={"param_a": "preferred", "param_b": 2, "param_c": 30, "param_d": 2}, distributions=distributions, ) ) study_multi_distro_params.add_trial( create_trial( value=1.0, params={"param_a": "opt", "param_b": 1, "param_c": 200, "param_d": 2}, distributions=distributions, ) ) study_multi_distro_params.add_trial( create_trial( value=2.0, params={"param_a": "preferred", "param_b": 10, "param_c": 10, "param_d": 1}, distributions=distributions, ) ) study_multi_distro_params.add_trial( create_trial( value=3.0, params={"param_a": "opt", "param_b": 2, "param_c": 10, "param_d": -1}, distributions=distributions, ) ) return study_multi_distro_params def test_target_is_none_and_study_is_multi_obj() -> None: study = create_study(directions=["minimize", "minimize"]) with pytest.raises(ValueError): _get_parallel_coordinate_info(study) def test_plot_parallel_coordinate_customized_target_name() -> None: study = prepare_study_with_trials() figure = plotly_plot_parallel_coordinate(study, target_name="Target Name") assert figure.data[0]["dimensions"][0]["label"] == "Target Name" figure = matplotlib.plot_parallel_coordinate(study, target_name="Target Name") assert figure.get_figure().axes[1].get_ylabel() == "Target Name" @parametrize_plot_parallel_coordinate @pytest.mark.parametrize( "specific_create_study, params", [ [create_study, None], [prepare_study_with_trials, ["param_a", "param_b"]], [prepare_study_with_trials, ["param_a", "param_b", "param_c"]], [prepare_study_with_trials, ["param_a", "param_b", "param_c", "param_d"]], [_create_study_with_failed_trial, None], [_create_study_with_categorical_params, None], [_create_study_with_numeric_categorical_params, None], [_create_study_with_log_params, None], [_create_study_with_log_scale_and_str_and_numeric_category, None], ], ) def test_plot_parallel_coordinate( plot_parallel_coordinate: Callable[..., Any], specific_create_study: Callable[[], Study], params: list[str] | None, ) -> None: study = specific_create_study() figure = plot_parallel_coordinate(study, params=params) if isinstance(figure, go.Figure): figure.write_image(BytesIO()) else: plt.savefig(BytesIO()) plt.close() def test_get_parallel_coordinate_info() -> None: # Test with no trial. study = create_study() info = _get_parallel_coordinate_info(study) assert info == _ParallelCoordinateInfo( dim_objective=_DimensionInfo( label="Objective Value", values=(), range=(0, 0), is_log=False, is_cat=False, tickvals=[], ticktext=[], ), dims_params=[], reverse_scale=True, target_name="Objective Value", ) study = prepare_study_with_trials() # Test with no parameters. info = _get_parallel_coordinate_info(study, params=[]) assert info == _ParallelCoordinateInfo( dim_objective=_DimensionInfo( label="Objective Value", values=(0.0, 2.0, 1.0), range=(0.0, 2.0), is_log=False, is_cat=False, tickvals=[], ticktext=[], ), dims_params=[], reverse_scale=True, target_name="Objective Value", ) # Test with a trial. info = _get_parallel_coordinate_info(study, params=["param_a", "param_b"]) assert info == _ParallelCoordinateInfo( dim_objective=_DimensionInfo( label="Objective Value", values=(0.0, 1.0), range=(0.0, 1.0), is_log=False, is_cat=False, tickvals=[], ticktext=[], ), dims_params=[ _DimensionInfo( label="param_a", values=(1.0, 2.5), range=(1.0, 2.5), is_log=False, is_cat=False, tickvals=[], ticktext=[], ), _DimensionInfo( label="param_b", values=(2.0, 1.0), range=(1.0, 2.0), is_log=False, is_cat=False, tickvals=[], ticktext=[], ), ], reverse_scale=True, target_name="Objective Value", ) # Test with a trial to select parameter. info = _get_parallel_coordinate_info(study, params=["param_a"]) assert info == _ParallelCoordinateInfo( dim_objective=_DimensionInfo( label="Objective Value", values=(0.0, 1.0), range=(0.0, 1.0), is_log=False, is_cat=False, tickvals=[], ticktext=[], ), dims_params=[ _DimensionInfo( label="param_a", values=(1.0, 2.5), range=(1.0, 2.5), is_log=False, is_cat=False, tickvals=[], ticktext=[], ), ], reverse_scale=True, target_name="Objective Value", ) # Test with a customized target value. with pytest.warns(UserWarning): info = _get_parallel_coordinate_info( study, params=["param_a"], target=lambda t: t.params["param_b"] ) assert info == _ParallelCoordinateInfo( dim_objective=_DimensionInfo( label="Objective Value", values=(2.0, 1.0), range=(1.0, 2.0), is_log=False, is_cat=False, tickvals=[], ticktext=[], ), dims_params=[ _DimensionInfo( label="param_a", values=(1.0, 2.5), range=(1.0, 2.5), is_log=False, is_cat=False, tickvals=[], ticktext=[], ), ], reverse_scale=True, target_name="Objective Value", ) # Test with a customized target name. info = _get_parallel_coordinate_info( study, params=["param_a", "param_b"], target_name="Target Name" ) assert info == _ParallelCoordinateInfo( dim_objective=_DimensionInfo( label="Target Name", values=(0.0, 1.0), range=(0.0, 1.0), is_log=False, is_cat=False, tickvals=[], ticktext=[], ), dims_params=[ _DimensionInfo( label="param_a", values=(1.0, 2.5), range=(1.0, 2.5), is_log=False, is_cat=False, tickvals=[], ticktext=[], ), _DimensionInfo( label="param_b", values=(2.0, 1.0), range=(1.0, 2.0), is_log=False, is_cat=False, tickvals=[], ticktext=[], ), ], reverse_scale=True, target_name="Target Name", ) # Test with wrong params that do not exist in trials. with pytest.raises(ValueError, match="Parameter optuna does not exist in your study."): _get_parallel_coordinate_info(study, params=["optuna", "optuna"]) # Ignore failed trials. study = _create_study_with_failed_trial() info = _get_parallel_coordinate_info(study) assert info == _ParallelCoordinateInfo( dim_objective=_DimensionInfo( label="Objective Value", values=(), range=(0, 0), is_log=False, is_cat=False, tickvals=[], ticktext=[], ), dims_params=[], reverse_scale=True, target_name="Objective Value", ) def test_get_parallel_coordinate_info_categorical_params() -> None: # Test with categorical params that cannot be converted to numeral. study = _create_study_with_categorical_params() info = _get_parallel_coordinate_info(study) assert info == _ParallelCoordinateInfo( dim_objective=_DimensionInfo( label="Objective Value", values=(0.0, 2.0), range=(0.0, 2.0), is_log=False, is_cat=False, tickvals=[], ticktext=[], ), dims_params=[ _DimensionInfo( label="category_a", values=(0, 1), range=(0, 1), is_log=False, is_cat=True, tickvals=[0, 1], ticktext=["preferred", "opt"], ), _DimensionInfo( label="category_b", values=(0, 1), range=(0, 1), is_log=False, is_cat=True, tickvals=[0, 1], ticktext=["net", "una"], ), ], reverse_scale=True, target_name="Objective Value", ) def test_get_parallel_coordinate_info_categorical_numeric_params() -> None: # Test with categorical params that can be interpreted as numeric params. study = _create_study_with_numeric_categorical_params() # Trials are sorted by using param_a and param_b, i.e., trial#1, trial#2, and trial#0. info = _get_parallel_coordinate_info(study) assert info == _ParallelCoordinateInfo( dim_objective=_DimensionInfo( label="Objective Value", values=(1.0, 2.0, 0.0), range=(0.0, 2.0), is_log=False, is_cat=False, tickvals=[], ticktext=[], ), dims_params=[ _DimensionInfo( label="category_a", values=(0, 1, 1), range=(0, 1), is_log=False, is_cat=True, tickvals=[0, 1], ticktext=["1", "2"], ), _DimensionInfo( label="category_b", values=(2, 0, 1), range=(0, 2), is_log=False, is_cat=True, tickvals=[0, 1, 2], ticktext=["10", "20", "30"], ), ], reverse_scale=True, target_name="Objective Value", ) def test_get_parallel_coordinate_info_log_params() -> None: # Test with log params. study = _create_study_with_log_params() info = _get_parallel_coordinate_info(study) assert info == _ParallelCoordinateInfo( dim_objective=_DimensionInfo( label="Objective Value", values=(0.0, 1.0, 0.1), range=(0.0, 1.0), is_log=False, is_cat=False, tickvals=[], ticktext=[], ), dims_params=[ _DimensionInfo( label="param_a", values=(-6, math.log10(2e-5), -4), range=(-6.0, -4.0), is_log=True, is_cat=False, tickvals=[-6, -5, -4.0], ticktext=["1e-06", "1e-05", "0.0001"], ), _DimensionInfo( label="param_b", values=(1.0, math.log10(200), math.log10(30)), range=(1.0, math.log10(200)), is_log=True, is_cat=False, tickvals=[1.0, 2.0, math.log10(200)], ticktext=["10", "100", "200"], ), ], reverse_scale=True, target_name="Objective Value", ) def test_get_parallel_coordinate_info_unique_param() -> None: # Test case when one unique value is suggested during the optimization. study_categorical_params = create_study() distributions: dict[str, BaseDistribution] = { "category_a": CategoricalDistribution(("preferred", "opt")), "param_b": FloatDistribution(1, 1000, log=True), } study_categorical_params.add_trial( create_trial( value=0.0, params={"category_a": "preferred", "param_b": 30}, distributions=distributions, ) ) # Both parameters contain unique values. info = _get_parallel_coordinate_info(study_categorical_params) assert info == _ParallelCoordinateInfo( dim_objective=_DimensionInfo( label="Objective Value", values=(0.0,), range=(0.0, 0.0), is_log=False, is_cat=False, tickvals=[], ticktext=[], ), dims_params=[ _DimensionInfo( label="category_a", values=(0.0,), range=(0, 0), is_log=False, is_cat=True, tickvals=[0], ticktext=["preferred"], ), _DimensionInfo( label="param_b", values=(math.log10(30),), range=(math.log10(30), math.log10(30)), is_log=True, is_cat=False, tickvals=[math.log10(30)], ticktext=["30"], ), ], reverse_scale=True, target_name="Objective Value", ) study_categorical_params.add_trial( create_trial( value=2.0, params={"category_a": "preferred", "param_b": 20}, distributions=distributions, ) ) # Still "category_a" contains unique suggested value during the optimization. info = _get_parallel_coordinate_info(study_categorical_params) assert info == _ParallelCoordinateInfo( dim_objective=_DimensionInfo( label="Objective Value", values=(0.0, 2.0), range=(0.0, 2.0), is_log=False, is_cat=False, tickvals=[], ticktext=[], ), dims_params=[ _DimensionInfo( label="category_a", values=(0, 0), range=(0, 0), is_log=False, is_cat=True, tickvals=[0], ticktext=["preferred"], ), _DimensionInfo( label="param_b", values=(math.log10(30), math.log10(20)), range=(math.log10(20), math.log10(30)), is_log=True, is_cat=False, tickvals=[math.log10(20), math.log10(30)], ticktext=["20", "30"], ), ], reverse_scale=True, target_name="Objective Value", ) def test_get_parallel_coordinate_info_with_log_scale_and_str_and_numeric_category() -> None: # Test with sample from multiple distributions including categorical params # that can be interpreted as numeric params. study = _create_study_with_log_scale_and_str_and_numeric_category() info = _get_parallel_coordinate_info(study) assert info == _ParallelCoordinateInfo( dim_objective=_DimensionInfo( label="Objective Value", values=(1.0, 3.0, 0.0, 2.0), range=(0.0, 3.0), is_log=False, is_cat=False, tickvals=[], ticktext=[], ), dims_params=[ _DimensionInfo( label="param_a", values=(1, 1, 0, 0), range=(0, 1), is_log=False, is_cat=True, tickvals=[0, 1], ticktext=["preferred", "opt"], ), _DimensionInfo( label="param_b", values=(0, 1, 1, 2), range=(0, 2), is_log=False, is_cat=True, tickvals=[0, 1, 2], ticktext=["1", "2", "10"], ), _DimensionInfo( label="param_c", values=(math.log10(200), 1.0, math.log10(30), 1.0), range=(1.0, math.log10(200)), is_log=True, is_cat=False, tickvals=[1, 2, math.log10(200)], ticktext=["10", "100", "200"], ), _DimensionInfo( label="param_d", values=(2, 0, 2, 1), range=(0, 2), is_log=False, is_cat=True, tickvals=[0, 1, 2], ticktext=["-1", "1", "2"], ), ], reverse_scale=True, target_name="Objective Value", ) @pytest.mark.parametrize("direction", ["minimize", "maximize"]) def test_color_map(direction: str) -> None: study = create_study(direction=direction) for i in range(3): study.add_trial( create_trial( value=float(i), params={"param_a": float(i), "param_b": float(i)}, distributions={ "param_a": FloatDistribution(0.0, 3.0), "param_b": FloatDistribution(0.0, 3.0), }, ) ) # `target` is `None`. line = plotly_plot_parallel_coordinate(study).data[0]["line"] assert COLOR_SCALE == [v[1] for v in line["colorscale"]] if direction == "minimize": assert line["reversescale"] else: assert not line["reversescale"] # When `target` is not `None`, `reversescale` is always `True`. line = plotly_plot_parallel_coordinate( study, target=lambda t: t.number, target_name="Target Name" ).data[0]["line"] assert COLOR_SCALE == [v[1] for v in line["colorscale"]] assert line["reversescale"] # Multi-objective optimization. study = create_study(directions=[direction, direction]) for i in range(3): study.add_trial( create_trial( values=[float(i), float(i)], params={"param_a": float(i), "param_b": float(i)}, distributions={ "param_a": FloatDistribution(0.0, 3.0), "param_b": FloatDistribution(0.0, 3.0), }, ) ) line = plotly_plot_parallel_coordinate( study, target=lambda t: t.number, target_name="Target Name" ).data[0]["line"] assert COLOR_SCALE == [v[1] for v in line["colorscale"]] assert line["reversescale"] def test_get_parallel_coordinate_info_only_missing_params() -> None: # When all trials contain only a part of parameters, # the plot returns an empty figure. study = create_study() study.add_trial( create_trial( value=0.0, params={"param_a": 1e-6}, distributions={ "param_a": FloatDistribution(1e-7, 1e-2, log=True), }, ) ) study.add_trial( create_trial( value=1.0, params={"param_b": 200}, distributions={ "param_b": FloatDistribution(1, 1000, log=True), }, ) ) info = _get_parallel_coordinate_info(study) assert info == _ParallelCoordinateInfo( dim_objective=_DimensionInfo( label="Objective Value", values=(), range=(0, 0), is_log=False, is_cat=False, tickvals=[], ticktext=[], ), dims_params=[], reverse_scale=True, target_name="Objective Value", ) @pytest.mark.parametrize("value", [float("inf"), -float("inf")]) def test_nonfinite_removed(value: float) -> None: study = prepare_study_with_trials(value_for_first_trial=value) info = _get_parallel_coordinate_info(study) assert all(np.isfinite(info.dim_objective.values)) @pytest.mark.parametrize("objective", (0, 1)) @pytest.mark.parametrize("value", (float("inf"), -float("inf"))) def test_nonfinite_multiobjective(objective: int, value: float) -> None: study = prepare_study_with_trials(n_objectives=2, value_for_first_trial=value) info = _get_parallel_coordinate_info( study, target=lambda t: t.values[objective], target_name="Target Name" ) assert all(np.isfinite(info.dim_objective.values)) optuna-4.1.0/tests/visualization_tests/test_param_importances.py000066400000000000000000000243251471332314300254330ustar00rootroot00000000000000from __future__ import annotations from io import BytesIO from typing import Any from typing import Callable import pytest from optuna.distributions import FloatDistribution from optuna.importance import FanovaImportanceEvaluator from optuna.importance import MeanDecreaseImpurityImportanceEvaluator from optuna.importance._base import BaseImportanceEvaluator from optuna.samplers import RandomSampler from optuna.study import create_study from optuna.study import Study from optuna.testing.objectives import fail_objective from optuna.testing.visualization import prepare_study_with_trials from optuna.trial import create_trial from optuna.trial import Trial from optuna.visualization import plot_param_importances as plotly_plot_param_importances from optuna.visualization._param_importances import _get_importances_info from optuna.visualization._param_importances import _get_importances_infos from optuna.visualization._param_importances import _ImportancesInfo from optuna.visualization._plotly_imports import go from optuna.visualization.matplotlib import plot_param_importances as plt_plot_param_importances from optuna.visualization.matplotlib._matplotlib_imports import Axes from optuna.visualization.matplotlib._matplotlib_imports import plt parametrize_plot_param_importances = pytest.mark.parametrize( "plot_param_importances", [plotly_plot_param_importances, plt_plot_param_importances] ) def _create_study_with_failed_trial() -> Study: study = create_study() study.optimize(fail_objective, n_trials=1, catch=(ValueError,)) return study def _create_multiobjective_study_with_failed_trial() -> Study: study = create_study(directions=["minimize", "minimize"]) study.optimize(fail_objective, n_trials=1, catch=(ValueError,)) return study def _create_multiobjective_study() -> Study: return prepare_study_with_trials(n_objectives=2) def test_target_is_none_and_study_is_multi_obj() -> None: study = create_study(directions=["minimize", "minimize"]) with pytest.raises(ValueError): _get_importances_info( study=study, evaluator=None, params=None, target=None, target_name="Objective Value" ) @parametrize_plot_param_importances def test_plot_param_importances_customized_target_name( plot_param_importances: Callable[..., Any] ) -> None: params = ["param_a", "param_b"] study = prepare_study_with_trials() figure = plot_param_importances(study, params=params, target_name="Target Name") if isinstance(figure, go.Figure): assert figure.layout.xaxis.title.text == "Hyperparameter Importance" elif isinstance(figure, Axes): assert figure.figure.axes[0].get_xlabel() == "Hyperparameter Importance" @parametrize_plot_param_importances def test_plot_param_importances_multiobjective_all_objectives_displayed( plot_param_importances: Callable[..., Any] ) -> None: n_objectives = 2 params = ["param_a"] study = prepare_study_with_trials(n_objectives) figure = plot_param_importances(study, params=params) if isinstance(figure, go.Figure): assert len(figure.data) == n_objectives elif isinstance(figure, Axes): assert len(figure.patches) == n_objectives * len(params) @parametrize_plot_param_importances @pytest.mark.parametrize( "specific_create_study", [ create_study, _create_multiobjective_study, _create_study_with_failed_trial, _create_multiobjective_study_with_failed_trial, prepare_study_with_trials, ], ) @pytest.mark.parametrize( "params", [ [], ["param_a"], None, ], ) def test_plot_param_importances( plot_param_importances: Callable[..., Any], specific_create_study: Callable[[], Study], params: list[str] | None, ) -> None: study = specific_create_study() figure = plot_param_importances(study, params=params) if isinstance(figure, go.Figure): figure.write_image(BytesIO()) else: plt.savefig(BytesIO()) plt.close() @pytest.mark.parametrize( "specific_create_study", [create_study, _create_study_with_failed_trial], ) @pytest.mark.parametrize( "params", [ [], ["param_a"], None, ], ) def test_get_param_importances_info_empty( specific_create_study: Callable[[], Study], params: list[str] | None ) -> None: study = specific_create_study() info = _get_importances_info( study, None, params=params, target=None, target_name="Objective Value" ) assert info == _ImportancesInfo( importance_values=[], param_names=[], importance_labels=[], target_name="Objective Value" ) @pytest.mark.parametrize( "specific_create_study,objective_names", [(create_study, ["Foo"]), (_create_multiobjective_study, ["Foo", "Bar"])], ) def test_get_param_importances_infos_custom_objective_names( specific_create_study: Callable[[], Study], objective_names: list[str] ) -> None: study = specific_create_study() study.set_metric_names(objective_names) infos = _get_importances_infos( study, evaluator=None, params=["param_a"], target=None, target_name="Objective Value" ) assert len(infos) == len(study.directions) assert all(info.target_name == expected for info, expected in zip(infos, objective_names)) @pytest.mark.parametrize( "specific_create_study,objective_names", [ (create_study, ["Objective Value"]), (_create_multiobjective_study, ["Objective Value 0", "Objective Value 1"]), ], ) def test_get_param_importances_infos_default_objective_names( specific_create_study: Callable[[], Study], objective_names: list[str] ) -> None: study = specific_create_study() infos = _get_importances_infos( study, evaluator=None, params=["param_a"], target=None, target_name="Objective Value" ) assert len(infos) == len(study.directions) assert all(info.target_name == expected for info, expected in zip(infos, objective_names)) def test_switch_label_when_param_insignificant() -> None: def _objective(trial: Trial) -> int: x = trial.suggest_int("x", 0, 2) _ = trial.suggest_int("y", -1, 1) return x**2 study = create_study() for x in range(1, 3): study.enqueue_trial({"x": x, "y": 0}) study.optimize(_objective, n_trials=2) info = _get_importances_info(study, None, None, None, "Objective Value") # Test if label for `y` param has been switched to `<0.01`. assert info.importance_labels == ["<0.01", "1.00"] @pytest.mark.parametrize("inf_value", [float("inf"), -float("inf")]) @pytest.mark.parametrize( "evaluator", [MeanDecreaseImpurityImportanceEvaluator(seed=10), FanovaImportanceEvaluator(seed=10)], ) @pytest.mark.parametrize("n_trials", [0, 10]) def test_get_info_importances_nonfinite_removed( inf_value: float, evaluator: BaseImportanceEvaluator, n_trials: int ) -> None: def _objective(trial: Trial) -> float: x1 = trial.suggest_float("x1", 0.1, 3) x2 = trial.suggest_float("x2", 0.1, 3, log=True) x3 = trial.suggest_float("x3", 2, 4, log=True) return x1 + x2 * x3 seed = 13 target_name = "Objective Value" study = create_study(sampler=RandomSampler(seed=seed)) study.optimize(_objective, n_trials=n_trials) # Create param importances info without inf value. info_without_inf = _get_importances_info( study, evaluator=evaluator, params=None, target=None, target_name=target_name ) # A trial with an inf value is added into the study manually. study.add_trial( create_trial( value=inf_value, params={"x1": 1.0, "x2": 1.0, "x3": 3.0}, distributions={ "x1": FloatDistribution(low=0.1, high=3), "x2": FloatDistribution(low=0.1, high=3, log=True), "x3": FloatDistribution(low=2, high=4, log=True), }, ) ) # Create param importances info with inf value. info_with_inf = _get_importances_info( study, evaluator=evaluator, params=None, target=None, target_name=target_name ) # Obtained info instances should be the same between with inf and without inf, # because the last trial whose objective value is an inf is ignored. assert info_with_inf == info_without_inf @pytest.mark.parametrize("target_idx", [0, 1]) @pytest.mark.parametrize("inf_value", [float("inf"), -float("inf")]) @pytest.mark.parametrize( "evaluator", [MeanDecreaseImpurityImportanceEvaluator(seed=10), FanovaImportanceEvaluator(seed=10)], ) @pytest.mark.parametrize("n_trial", [0, 10]) def test_multi_objective_trial_with_infinite_value_ignored( target_idx: int, inf_value: float, evaluator: BaseImportanceEvaluator, n_trial: int ) -> None: def _multi_objective_function(trial: Trial) -> tuple[float, float]: x1 = trial.suggest_float("x1", 0.1, 3) x2 = trial.suggest_float("x2", 0.1, 3, log=True) x3 = trial.suggest_float("x3", 2, 4, log=True) return x1, x2 * x3 seed = 13 target_name = "Target Name" study = create_study(directions=["minimize", "minimize"], sampler=RandomSampler(seed=seed)) study.optimize(_multi_objective_function, n_trials=n_trial) # Create param importances info without inf value. info_without_inf = _get_importances_info( study, evaluator=evaluator, params=None, target=lambda t: t.values[target_idx], target_name=target_name, ) # A trial with an inf value is added into the study manually. study.add_trial( create_trial( values=[inf_value, inf_value], params={"x1": 1.0, "x2": 1.0, "x3": 3.0}, distributions={ "x1": FloatDistribution(low=0.1, high=3), "x2": FloatDistribution(low=0.1, high=3, log=True), "x3": FloatDistribution(low=2, high=4, log=True), }, ) ) # Create param importances info with inf value. info_with_inf = _get_importances_info( study, evaluator=evaluator, params=None, target=lambda t: t.values[target_idx], target_name=target_name, ) # Obtained info instances should be the same between with inf and without inf, # because the last trial whose objective value is an inf is ignored. assert info_with_inf == info_without_inf optuna-4.1.0/tests/visualization_tests/test_pareto_front.py000066400000000000000000000373601471332314300244340ustar00rootroot00000000000000from __future__ import annotations from io import BytesIO from typing import Any from typing import Callable from typing import Sequence import warnings import pytest import optuna from optuna import create_study from optuna import create_trial from optuna.distributions import FloatDistribution from optuna.study.study import Study from optuna.trial import FrozenTrial from optuna.visualization import plot_pareto_front import optuna.visualization._pareto_front from optuna.visualization._pareto_front import _get_pareto_front_info from optuna.visualization._pareto_front import _ParetoFrontInfo from optuna.visualization._plotly_imports import go from optuna.visualization._utils import COLOR_SCALE from optuna.visualization.matplotlib._matplotlib_imports import plt import optuna.visualization.matplotlib._pareto_front def test_get_pareto_front_info_infer_n_targets() -> None: study = optuna.create_study(directions=["minimize", "minimize"]) assert _get_pareto_front_info(study).n_targets == 2 study = optuna.create_study(directions=["minimize"] * 5) assert ( _get_pareto_front_info( study, target_names=["target1", "target2"], targets=lambda _: [0.0, 1.0] ).n_targets == 2 ) study = optuna.create_study(directions=["minimize"] * 5) study.optimize(lambda _: [0] * 5, n_trials=1) assert _get_pareto_front_info(study, targets=lambda _: [0.0, 1.0]).n_targets == 2 study = optuna.create_study(directions=["minimize"] * 2) with pytest.raises(ValueError): _get_pareto_front_info(study, targets=lambda _: [0.0, 1.0]) def create_study_2d( constraints_func: Callable[[FrozenTrial], Sequence[float]] | None = None ) -> Study: sampler = optuna.samplers.TPESampler(seed=0, constraints_func=constraints_func) study = optuna.create_study(directions=["minimize", "minimize"], sampler=sampler) study.enqueue_trial({"x": 1, "y": 2}) study.enqueue_trial({"x": 1, "y": 1}) study.enqueue_trial({"x": 0, "y": 2}) study.enqueue_trial({"x": 1, "y": 0}) study.optimize(lambda t: [t.suggest_int("x", 0, 2), t.suggest_int("y", 0, 2)], n_trials=4) return study def create_study_3d() -> Study: study = optuna.create_study(directions=["minimize", "minimize", "minimize"]) study.enqueue_trial({"x": 1, "y": 2}) study.enqueue_trial({"x": 1, "y": 1}) study.enqueue_trial({"x": 0, "y": 2}) study.enqueue_trial({"x": 1, "y": 0}) study.optimize( lambda t: [t.suggest_int("x", 0, 2), t.suggest_int("y", 0, 2), 1.0], n_trials=4, ) return study @pytest.mark.parametrize("include_dominated_trials", [False, True]) @pytest.mark.parametrize("axis_order", [None, [0, 1], [1, 0]]) @pytest.mark.parametrize("targets", [None, lambda t: (t.values[0], t.values[1])]) @pytest.mark.parametrize("target_names", [None, ["Foo", "Bar"]]) @pytest.mark.parametrize("metric_names", [None, ["v0", "v1"]]) def test_get_pareto_front_info_unconstrained( include_dominated_trials: bool, axis_order: list[int] | None, targets: Callable[[FrozenTrial], Sequence[float]] | None, target_names: list[str] | None, metric_names: list[str] | None, ) -> None: if axis_order is not None and targets is not None: pytest.skip( "skip using both axis_order and targets because they cannot be used at the same time." ) study = create_study_2d() if metric_names is not None: study.set_metric_names(metric_names) trials = study.get_trials(deepcopy=False) with warnings.catch_warnings(): warnings.filterwarnings("ignore", category=FutureWarning) info = _get_pareto_front_info( study=study, include_dominated_trials=include_dominated_trials, axis_order=axis_order, targets=targets, target_names=target_names, ) assert info == _ParetoFrontInfo( n_targets=2, target_names=target_names or metric_names or ["Objective 0", "Objective 1"], best_trials_with_values=[(trials[2], [0, 2]), (trials[3], [1, 0])], non_best_trials_with_values=( [(trials[0], [1, 2]), (trials[1], [1, 1])] if include_dominated_trials else [] ), infeasible_trials_with_values=[], axis_order=axis_order or [0, 1], include_dominated_trials=include_dominated_trials, has_constraints=False, ) @pytest.mark.parametrize("include_dominated_trials", [False, True]) @pytest.mark.parametrize("axis_order", [None, [0, 1], [1, 0]]) @pytest.mark.parametrize("targets", [None, lambda t: (t.values[0], t.values[1])]) @pytest.mark.parametrize("target_names", [None, ["Foo", "Bar"]]) @pytest.mark.parametrize("metric_names", [None, ["v0", "v1"]]) @pytest.mark.parametrize("use_study_with_constraints", [True, False]) def test_get_pareto_front_info_constrained( include_dominated_trials: bool, axis_order: list[int] | None, targets: Callable[[FrozenTrial], Sequence[float]] | None, target_names: list[str] | None, metric_names: list[str] | None, use_study_with_constraints: bool, ) -> None: if axis_order is not None and targets is not None: pytest.skip( "skip using both axis_order and targets because they cannot be used at the same time." ) # (x, y) = (1, 0) is infeasible; others are feasible. def constraints_func(t: FrozenTrial) -> Sequence[float]: return [1.0] if t.params["x"] == 1 and t.params["y"] == 0 else [-1.0] if use_study_with_constraints: study = create_study_2d(constraints_func=constraints_func) else: study = create_study_2d() if metric_names is not None: study.set_metric_names(metric_names) trials = study.get_trials(deepcopy=False) with warnings.catch_warnings(): warnings.filterwarnings("ignore", category=FutureWarning) info = _get_pareto_front_info( study=study, include_dominated_trials=include_dominated_trials, axis_order=axis_order, targets=targets, target_names=target_names, constraints_func=None if use_study_with_constraints else constraints_func, ) assert info == _ParetoFrontInfo( n_targets=2, target_names=target_names or metric_names or ["Objective 0", "Objective 1"], best_trials_with_values=[(trials[1], [1, 1]), (trials[2], [0, 2])], non_best_trials_with_values=[(trials[0], [1, 2])] if include_dominated_trials else [], infeasible_trials_with_values=[(trials[3], [1, 0])], axis_order=axis_order or [0, 1], include_dominated_trials=include_dominated_trials, has_constraints=True, ) @pytest.mark.parametrize("include_dominated_trials", [False, True]) @pytest.mark.parametrize("axis_order", [None, [0, 1], [1, 0]]) @pytest.mark.parametrize("targets", [None, lambda t: (t.values[0], t.values[1])]) @pytest.mark.parametrize("target_names", [None, ["Foo", "Bar"]]) @pytest.mark.parametrize("metric_names", [None, ["v0", "v1"]]) @pytest.mark.parametrize("use_study_with_constraints", [True, False]) def test_get_pareto_front_info_all_infeasible( include_dominated_trials: bool, axis_order: list[int] | None, targets: Callable[[FrozenTrial], Sequence[float]] | None, target_names: list[str] | None, metric_names: list[str] | None, use_study_with_constraints: bool, ) -> None: if axis_order is not None and targets is not None: pytest.skip( "skip using both axis_order and targets because they cannot be used at the same time." ) # all trials are infeasible. def constraints_func(t: FrozenTrial) -> Sequence[float]: return [1.0] if use_study_with_constraints: study = create_study_2d(constraints_func=constraints_func) else: study = create_study_2d() if metric_names is not None: study.set_metric_names(metric_names) trials = study.get_trials(deepcopy=False) with warnings.catch_warnings(): warnings.filterwarnings("ignore", category=FutureWarning) info = _get_pareto_front_info( study=study, include_dominated_trials=include_dominated_trials, axis_order=axis_order, targets=targets, target_names=target_names, constraints_func=None if use_study_with_constraints else constraints_func, ) assert info == _ParetoFrontInfo( n_targets=2, target_names=target_names or metric_names or ["Objective 0", "Objective 1"], best_trials_with_values=[], non_best_trials_with_values=[], infeasible_trials_with_values=[ (trials[0], [1, 2]), (trials[1], [1, 1]), (trials[2], [0, 2]), (trials[3], [1, 0]), ], axis_order=axis_order or [0, 1], include_dominated_trials=include_dominated_trials, has_constraints=True, ) @pytest.mark.parametrize("include_dominated_trials", [False, True]) @pytest.mark.parametrize("axis_order", [None, [0, 1, 2], [2, 1, 0]]) @pytest.mark.parametrize("targets", [None, lambda t: (t.values[0], t.values[1], t.values[2])]) @pytest.mark.parametrize("target_names", [None, ["Foo", "Bar", "Baz"]]) def test_get_pareto_front_info_3d( include_dominated_trials: bool, axis_order: list[int] | None, targets: Callable[[FrozenTrial], Sequence[float]] | None, target_names: list[str] | None, ) -> None: if axis_order is not None and targets is not None: pytest.skip( "skip using both axis_order and targets because they cannot be used at the same time." ) study = create_study_3d() trials = study.get_trials(deepcopy=False) with warnings.catch_warnings(): warnings.filterwarnings("ignore", category=FutureWarning) info = _get_pareto_front_info( study=study, include_dominated_trials=include_dominated_trials, axis_order=axis_order, targets=targets, target_names=target_names, ) assert info == _ParetoFrontInfo( n_targets=3, target_names=target_names or ["Objective 0", "Objective 1", "Objective 2"], best_trials_with_values=[(trials[2], [0, 2, 1]), (trials[3], [1, 0, 1])], non_best_trials_with_values=( [(trials[0], [1, 2, 1]), (trials[1], [1, 1, 1])] if include_dominated_trials else [] ), infeasible_trials_with_values=[], axis_order=axis_order or [0, 1, 2], include_dominated_trials=include_dominated_trials, has_constraints=False, ) def test_get_pareto_front_info_invalid_number_of_target_names() -> None: study = optuna.create_study(directions=["minimize", "minimize"]) with pytest.raises(ValueError): _get_pareto_front_info(study=study, target_names=["Foo"]) @pytest.mark.filterwarnings("ignore::FutureWarning") @pytest.mark.parametrize("n_dims", [1, 4]) @pytest.mark.parametrize("include_dominated_trials", [False, True]) @pytest.mark.parametrize("constraints_func", [None, lambda _: [-1.0]]) def test_get_pareto_front_info_unsupported_dimensions( n_dims: int, include_dominated_trials: bool, constraints_func: Callable[[FrozenTrial], Sequence[float]] | None, ) -> None: study = optuna.create_study(directions=["minimize"] * n_dims) with pytest.raises(ValueError): _get_pareto_front_info( study=study, include_dominated_trials=include_dominated_trials, constraints_func=constraints_func, ) @pytest.mark.filterwarnings("ignore::FutureWarning") @pytest.mark.parametrize("axis_order", [[0, 1, 1], [0, 0], [0, 2], [-1, 1]]) @pytest.mark.parametrize("include_dominated_trials", [False, True]) @pytest.mark.parametrize("constraints_func", [None, lambda _: [-1.0]]) def test_get_pareto_front_info_invalid_axis_order( axis_order: list[int], include_dominated_trials: bool, constraints_func: Callable[[FrozenTrial], Sequence[float]] | None, ) -> None: study = optuna.create_study(directions=["minimize", "minimize"]) with pytest.raises(ValueError): _get_pareto_front_info( study=study, include_dominated_trials=include_dominated_trials, axis_order=axis_order, constraints_func=constraints_func, ) @pytest.mark.filterwarnings("ignore::FutureWarning") @pytest.mark.parametrize("include_dominated_trials", [False, True]) @pytest.mark.parametrize("constraints_func", [None, lambda _: [-1.0]]) def test_get_pareto_front_info_invalid_target_values( include_dominated_trials: bool, constraints_func: Callable[[FrozenTrial], Sequence[float]] | None, ) -> None: study = optuna.create_study(directions=["minimize", "minimize"]) study.optimize(lambda _: [0, 0], n_trials=3) with pytest.raises(ValueError): _get_pareto_front_info( study=study, targets=lambda t: t.values[0], include_dominated_trials=include_dominated_trials, constraints_func=constraints_func, ) @pytest.mark.filterwarnings("ignore::FutureWarning") @pytest.mark.parametrize("include_dominated_trials", [False, True]) @pytest.mark.parametrize("constraints_func", [None, lambda _: [-1.0]]) def test_get_pareto_front_info_using_axis_order_and_targets( include_dominated_trials: bool, constraints_func: Callable[[FrozenTrial], Sequence[float]] | None, ) -> None: study = optuna.create_study(directions=["minimize", "minimize", "minimize"]) with pytest.raises(ValueError): _get_pareto_front_info( study=study, axis_order=[0, 1, 2], targets=lambda t: (t.values[0], t.values[1], t.values[2]), include_dominated_trials=include_dominated_trials, constraints_func=constraints_func, ) def test_constraints_func_future_warning() -> None: study = optuna.create_study(directions=["minimize", "minimize"]) with pytest.warns(FutureWarning): _get_pareto_front_info( study=study, constraints_func=lambda _: [1.0], ) @pytest.mark.parametrize( "plotter", [ optuna.visualization._pareto_front._get_pareto_front_plot, optuna.visualization.matplotlib._pareto_front._get_pareto_front_plot, ], ) @pytest.mark.parametrize( "info_template", [ _get_pareto_front_info(create_study_2d()), _get_pareto_front_info(create_study_3d()), ], ) @pytest.mark.parametrize("include_dominated_trials", [True, False]) @pytest.mark.parametrize("has_constraints", [True, False]) def test_get_pareto_front_plot( plotter: Callable[[_ParetoFrontInfo], Any], info_template: _ParetoFrontInfo, include_dominated_trials: bool, has_constraints: bool, ) -> None: info = info_template if not include_dominated_trials: info = info._replace(include_dominated_trials=False, non_best_trials_with_values=[]) if not has_constraints: info = info._replace(has_constraints=False, infeasible_trials_with_values=[]) figure = plotter(info) if isinstance(figure, go.Figure): figure.write_image(BytesIO()) else: plt.savefig(BytesIO()) plt.close() @pytest.mark.parametrize("direction", ["minimize", "maximize"]) def test_color_map(direction: str) -> None: study = create_study(directions=[direction, direction]) for i in range(3): study.add_trial( create_trial( values=[float(i), float(i)], params={"param_a": 1.0, "param_b": 2.0}, distributions={ "param_a": FloatDistribution(0.0, 3.0), "param_b": FloatDistribution(0.0, 3.0), }, ) ) # Since `plot_pareto_front`'s colormap depends on only trial.number, # `reversecale` is not in the plot. marker = plot_pareto_front(study).data[0]["marker"] assert COLOR_SCALE == [v[1] for v in marker["colorscale"]] assert "reversecale" not in marker optuna-4.1.0/tests/visualization_tests/test_rank.py000066400000000000000000000575651471332314300226760ustar00rootroot00000000000000from __future__ import annotations from io import BytesIO import math from typing import Any from typing import Callable import numpy as np import pytest from optuna.distributions import BaseDistribution from optuna.distributions import CategoricalDistribution from optuna.distributions import FloatDistribution from optuna.samplers._base import _CONSTRAINTS_KEY from optuna.study import create_study from optuna.study import Study from optuna.testing.objectives import fail_objective from optuna.testing.visualization import prepare_study_with_trials from optuna.trial import create_trial from optuna.visualization import plot_rank as plotly_plot_rank from optuna.visualization._plotly_imports import go from optuna.visualization._rank import _AxisInfo from optuna.visualization._rank import _convert_color_idxs_to_scaled_rgb_colors from optuna.visualization._rank import _get_axis_info from optuna.visualization._rank import _get_order_with_same_order_averaging from optuna.visualization._rank import _get_rank_info from optuna.visualization._rank import _RankPlotInfo from optuna.visualization._rank import _RankSubplotInfo parametrize_plot_rank = pytest.mark.parametrize("plot_rank", [plotly_plot_rank]) def _create_study_with_failed_trial() -> Study: study = create_study() study.optimize(fail_objective, n_trials=1, catch=(ValueError,)) return study def _create_study_with_log_scale_and_str_category_2d() -> Study: study = create_study() distributions = { "param_a": FloatDistribution(1e-7, 1e-2, log=True), "param_b": CategoricalDistribution(["100", "101"]), } study.add_trial( create_trial( value=0.0, params={"param_a": 1e-6, "param_b": "101"}, distributions=distributions ) ) study.add_trial( create_trial( value=1.0, params={"param_a": 1e-5, "param_b": "100"}, distributions=distributions ) ) return study def _create_study_with_log_scale_and_str_category_3d() -> Study: study = create_study() distributions = { "param_a": FloatDistribution(1e-7, 1e-2, log=True), "param_b": CategoricalDistribution(["100", "101"]), "param_c": CategoricalDistribution(["one", "two"]), } study.add_trial( create_trial( value=0.0, params={"param_a": 1e-6, "param_b": "101", "param_c": "one"}, distributions=distributions, ) ) study.add_trial( create_trial( value=1.0, params={"param_a": 1e-5, "param_b": "100", "param_c": "two"}, distributions=distributions, ) ) return study def _create_study_with_constraints() -> Study: study = create_study() distributions: dict[str, BaseDistribution] = { "param_a": FloatDistribution(0.1, 0.2), "param_b": FloatDistribution(0.3, 0.4), } study.add_trial( create_trial( value=0.0, params={"param_a": 0.11, "param_b": 0.31}, distributions=distributions, system_attrs={_CONSTRAINTS_KEY: [-0.1, 0.0]}, ) ) study.add_trial( create_trial( value=1.0, params={"param_a": 0.19, "param_b": 0.34}, distributions=distributions, system_attrs={_CONSTRAINTS_KEY: [0.1, 0.0]}, ) ) return study def _create_study_mixture_category_types() -> Study: study = create_study() distributions: dict[str, BaseDistribution] = { "param_a": CategoricalDistribution([None, "100"]), "param_b": CategoricalDistribution([101, 102.0]), } study.add_trial( create_trial( value=0.0, params={"param_a": None, "param_b": 101}, distributions=distributions ) ) study.add_trial( create_trial( value=0.5, params={"param_a": "100", "param_b": 102.0}, distributions=distributions ) ) return study def _named_tuple_equal(t1: Any, t2: Any) -> bool: if isinstance(t1, np.ndarray): return bool(np.all(t1 == t2)) elif isinstance(t1, tuple) or isinstance(t1, list): if len(t1) != len(t2): return False for x, y in zip(t1, t2): if not _named_tuple_equal(x, y): return False return True else: return t1 == t2 def _get_nested_list_shape(nested_list: list[list[Any]]) -> tuple[int, int]: assert all(len(nested_list[0]) == len(row) for row in nested_list) return len(nested_list), len(nested_list[0]) @parametrize_plot_rank @pytest.mark.parametrize( "specific_create_study, params", [ [create_study, []], [create_study, ["param_a"]], [create_study, ["param_a", "param_b"]], [create_study, ["param_a", "param_b", "param_c"]], [create_study, ["param_a", "param_b", "param_c", "param_d"]], [create_study, None], [_create_study_with_failed_trial, []], [_create_study_with_failed_trial, ["param_a"]], [_create_study_with_failed_trial, ["param_a", "param_b"]], [_create_study_with_failed_trial, ["param_a", "param_b", "param_c"]], [_create_study_with_failed_trial, ["param_a", "param_b", "param_c", "param_d"]], [_create_study_with_failed_trial, None], [prepare_study_with_trials, []], [prepare_study_with_trials, ["param_a"]], [prepare_study_with_trials, ["param_a", "param_b"]], [prepare_study_with_trials, ["param_a", "param_b", "param_c"]], [prepare_study_with_trials, ["param_a", "param_b", "param_c", "param_d"]], [prepare_study_with_trials, None], [_create_study_with_log_scale_and_str_category_2d, None], [_create_study_with_log_scale_and_str_category_3d, None], [_create_study_mixture_category_types, None], [_create_study_with_constraints, None], ], ) def test_plot_rank( plot_rank: Callable[..., Any], specific_create_study: Callable[[], Study], params: list[str] | None, ) -> None: study = specific_create_study() figure = plot_rank(study, params=params) if isinstance(figure, go.Figure): figure.write_image(BytesIO()) def test_target_is_none_and_study_is_multi_obj() -> None: study = create_study(directions=["minimize", "minimize"]) with pytest.raises(ValueError): _get_rank_info(study, params=None, target=None, target_name="Objective Value") @pytest.mark.parametrize( "specific_create_study", [create_study, _create_study_with_failed_trial], ) @pytest.mark.parametrize( "params", [ [], ["param_a"], ["param_a", "param_b"], ["param_a", "param_b", "param_c"], ["param_a", "param_b", "param_c", "param_d"], None, ], ) def test_get_rank_info_empty( specific_create_study: Callable[[], Study], params: list[str] | None ) -> None: study = specific_create_study() info = _get_rank_info(study, params=params, target=None, target_name="Objective Value") assert len(info.params) == 0 assert len(info.sub_plot_infos) == 0 def test_get_rank_info_non_exist_param_error() -> None: study = prepare_study_with_trials() with pytest.raises(ValueError): _get_rank_info(study, ["optuna"], target=None, target_name="Objective Value") @pytest.mark.parametrize("params", [[], ["param_a"]]) def test_get_rank_info_too_short_params(params: list[str]) -> None: study = prepare_study_with_trials() info = _get_rank_info(study, params=params, target=None, target_name="Objective Value") assert len(info.params) == len(params) assert len(info.sub_plot_infos) == len(params) def test_get_rank_info_2_params() -> None: params = ["param_a", "param_b"] study = prepare_study_with_trials() info = _get_rank_info(study, params=params, target=None, target_name="Objective Value") assert _named_tuple_equal( info, _RankPlotInfo( params=params, sub_plot_infos=[ [ _RankSubplotInfo( xaxis=_AxisInfo( name="param_a", range=(0.925, 2.575), is_log=False, is_cat=False, ), yaxis=_AxisInfo( name="param_b", range=(-0.1, 2.1), is_log=False, is_cat=False, ), xs=[1.0, 2.5], ys=[2.0, 1.0], trials=[study.trials[0], study.trials[2]], zs=np.array([0.0, 1.0]), colors=_convert_color_idxs_to_scaled_rgb_colors(np.array([0.0, 0.5])), ) ] ], target_name="Objective Value", zs=np.array([0.0, 2.0, 1.0]), colors=_convert_color_idxs_to_scaled_rgb_colors(np.array([0.0, 1.0, 0.5])), has_custom_target=False, ), ) @pytest.mark.parametrize( "params", [ ["param_a", "param_b", "param_c"], ["param_a", "param_b", "param_c", "param_d"], None, ], ) def test_get_rank_info_more_than_2_params(params: list[str] | None) -> None: study = prepare_study_with_trials() n_params = len(params) if params is not None else 4 info = _get_rank_info(study, params=params, target=None, target_name="Objective Value") assert len(info.params) == n_params assert _get_nested_list_shape(info.sub_plot_infos) == (n_params, n_params) @pytest.mark.parametrize( "params", [ ["param_a", "param_b"], ["param_a", "param_b", "param_c"], ], ) def test_get_rank_info_customized_target(params: list[str]) -> None: study = prepare_study_with_trials() info = _get_rank_info( study, params=params, target=lambda t: t.params["param_d"], target_name="param_d" ) n_params = len(params) assert len(info.params) == n_params plot_shape = (1, 1) if n_params == 2 else (n_params, n_params) assert _get_nested_list_shape(info.sub_plot_infos) == plot_shape @pytest.mark.parametrize( "params", [ ["param_a", "param_b"], # `x_axis` has one observation. ["param_b", "param_a"], # `y_axis` has one observation. ], ) def test_generate_rank_plot_for_no_plots(params: list[str]) -> None: study = create_study(direction="minimize") study.add_trial( create_trial( values=[0.0], params={"param_a": 1.0}, distributions={ "param_a": FloatDistribution(0.0, 3.0), }, ) ) study.add_trial( create_trial( values=[2.0], params={"param_b": 0.0}, distributions={"param_b": FloatDistribution(0.0, 3.0)}, ) ) info = _get_rank_info(study, params=params, target=None, target_name="Objective Value") axis_infos = { "param_a": _AxisInfo( name="param_a", range=(1.0, 1.0), is_log=False, is_cat=False, ), "param_b": _AxisInfo( name="param_b", range=(0.0, 0.0), is_log=False, is_cat=False, ), } assert _named_tuple_equal( info, _RankPlotInfo( params=params, sub_plot_infos=[ [ _RankSubplotInfo( xaxis=axis_infos[params[0]], yaxis=axis_infos[params[1]], xs=[], ys=[], trials=[], zs=np.array([]), colors=_convert_color_idxs_to_scaled_rgb_colors(np.array([])).reshape( -1, 3 ), ) ] ], target_name="Objective Value", zs=np.array([0.0, 2.0]), colors=_convert_color_idxs_to_scaled_rgb_colors(np.array([0.0, 1.0])), has_custom_target=False, ), ) @pytest.mark.parametrize( "params", [ ["param_a", "param_b"], # `x_axis` has one observation. ["param_b", "param_a"], # `y_axis` has one observation. ], ) def test_generate_rank_plot_for_few_observations(params: list[str]) -> None: study = create_study(direction="minimize") study.add_trial( create_trial( values=[0.0], params={"param_a": 1.0, "param_b": 2.0}, distributions={ "param_a": FloatDistribution(0.0, 3.0), "param_b": FloatDistribution(0.0, 3.0), }, ) ) study.add_trial( create_trial( values=[2.0], params={"param_b": 0.0}, distributions={"param_b": FloatDistribution(0.0, 3.0)}, ) ) info = _get_rank_info(study, params=params, target=None, target_name="Objective Value") axis_infos = { "param_a": _AxisInfo( name="param_a", range=(1.0, 1.0), is_log=False, is_cat=False, ), "param_b": _AxisInfo( name="param_b", range=(-0.1, 2.1), is_log=False, is_cat=False, ), } assert _named_tuple_equal( info, _RankPlotInfo( params=params, sub_plot_infos=[ [ _RankSubplotInfo( xaxis=axis_infos[params[0]], yaxis=axis_infos[params[1]], xs=[study.get_trials()[0].params[params[0]]], ys=[study.get_trials()[0].params[params[1]]], trials=[study.get_trials()[0]], zs=np.array([0.0]), colors=_convert_color_idxs_to_scaled_rgb_colors(np.array([0.0])), ) ] ], target_name="Objective Value", zs=np.array([0.0, 2.0]), colors=_convert_color_idxs_to_scaled_rgb_colors(np.array([0.0, 1.0])), has_custom_target=False, ), ) def test_get_rank_info_log_scale_and_str_category_2_params() -> None: # If the search space has two parameters, plot_rank generates a single plot. study = _create_study_with_log_scale_and_str_category_2d() info = _get_rank_info(study, params=None, target=None, target_name="Objective Value") assert _named_tuple_equal( info, _RankPlotInfo( params=["param_a", "param_b"], sub_plot_infos=[ [ _RankSubplotInfo( xaxis=_AxisInfo( name="param_a", range=(math.pow(10, -6.05), math.pow(10, -4.95)), is_log=True, is_cat=False, ), yaxis=_AxisInfo( name="param_b", range=(-0.05, 1.05), is_log=False, is_cat=True, ), xs=[1e-6, 1e-5], ys=["101", "100"], trials=[study.trials[0], study.trials[1]], zs=np.array([0.0, 1.0]), colors=_convert_color_idxs_to_scaled_rgb_colors(np.array([0.0, 1.0])), ) ] ], target_name="Objective Value", zs=np.array([0.0, 1.0]), colors=_convert_color_idxs_to_scaled_rgb_colors(np.array([0.0, 1.0])), has_custom_target=False, ), ) def test_get_rank_info_log_scale_and_str_category_more_than_2_params() -> None: # If the search space has three parameters, plot_rank generates nine plots. study = _create_study_with_log_scale_and_str_category_3d() info = _get_rank_info(study, params=None, target=None, target_name="Objective Value") params = ["param_a", "param_b", "param_c"] assert info.params == params assert _get_nested_list_shape(info.sub_plot_infos) == (3, 3) ranges = { "param_a": (math.pow(10, -6.05), math.pow(10, -4.95)), "param_b": (-0.05, 1.05), "param_c": (-0.05, 1.05), } is_log = {"param_a": True, "param_b": False, "param_c": False} is_cat = {"param_a": False, "param_b": True, "param_c": True} param_values = {"param_a": [1e-6, 1e-5], "param_b": ["101", "100"], "param_c": ["one", "two"]} zs = np.array([0.0, 1.0]) colors = _convert_color_idxs_to_scaled_rgb_colors(np.array([0.0, 1.0])) def _check_axis(axis: _AxisInfo, name: str) -> None: assert axis.name == name assert axis.range == ranges[name] assert axis.is_log == is_log[name] assert axis.is_cat == is_cat[name] for yi in range(3): for xi in range(3): xaxis = info.sub_plot_infos[yi][xi].xaxis yaxis = info.sub_plot_infos[yi][xi].yaxis x_param = params[xi] y_param = params[yi] _check_axis(xaxis, x_param) _check_axis(yaxis, y_param) assert info.sub_plot_infos[yi][xi].xs == param_values[x_param] assert info.sub_plot_infos[yi][xi].ys == param_values[y_param] assert info.sub_plot_infos[yi][xi].trials == study.trials assert np.all(info.sub_plot_infos[yi][xi].zs == zs) assert np.all(info.sub_plot_infos[yi][xi].colors == colors) info.target_name == "Objective Value" assert np.all(info.zs == zs) assert np.all(info.colors == colors) assert not info.has_custom_target def test_get_rank_info_mixture_category_types() -> None: study = _create_study_mixture_category_types() info = _get_rank_info(study, params=None, target=None, target_name="Objective Value") assert _named_tuple_equal( info, _RankPlotInfo( params=["param_a", "param_b"], sub_plot_infos=[ [ _RankSubplotInfo( xaxis=_AxisInfo( name="param_a", range=(-0.05, 1.05), is_log=False, is_cat=True, ), yaxis=_AxisInfo( name="param_b", range=(100.95, 102.05), is_log=False, is_cat=False, ), xs=[None, "100"], ys=[101, 102.0], trials=study.trials, zs=np.array([0.0, 0.5]), colors=_convert_color_idxs_to_scaled_rgb_colors(np.array([0.0, 1.0])), ) ] ], target_name="Objective Value", zs=np.array([0.0, 0.5]), colors=_convert_color_idxs_to_scaled_rgb_colors(np.array([0.0, 1.0])), has_custom_target=False, ), ) @pytest.mark.parametrize("value", [float("inf"), float("-inf")]) def test_get_rank_info_nonfinite(value: float) -> None: study = prepare_study_with_trials(value_for_first_trial=value) info = _get_rank_info( study, params=["param_b", "param_d"], target=None, target_name="Objective Value" ) colors = ( _convert_color_idxs_to_scaled_rgb_colors(np.array([0.0, 1.0, 0.5])) if value == float("-inf") else _convert_color_idxs_to_scaled_rgb_colors(np.array([1.0, 0.5, 0.0])) ) assert _named_tuple_equal( info, _RankPlotInfo( params=["param_b", "param_d"], sub_plot_infos=[ [ _RankSubplotInfo( xaxis=_AxisInfo( name="param_b", range=(-0.1, 2.1), is_log=False, is_cat=False, ), yaxis=_AxisInfo( name="param_d", range=(1.9, 4.1), is_log=False, is_cat=False, ), xs=[2.0, 0.0, 1.0], ys=[4.0, 4.0, 2.0], trials=study.trials, zs=np.array([value, 2.0, 1.0]), colors=colors, ) ] ], target_name="Objective Value", zs=np.array([value, 2.0, 1.0]), colors=colors, has_custom_target=False, ), ) @pytest.mark.parametrize("objective", (0, 1)) @pytest.mark.parametrize("value", (float("inf"), float("-inf"))) def test_get_rank_info_nonfinite_multiobjective(objective: int, value: float) -> None: study = prepare_study_with_trials(n_objectives=2, value_for_first_trial=value) info = _get_rank_info( study, params=["param_b", "param_d"], target=lambda t: t.values[objective], target_name="Target Name", ) colors = ( _convert_color_idxs_to_scaled_rgb_colors(np.array([0.0, 1.0, 0.5])) if value == float("-inf") else _convert_color_idxs_to_scaled_rgb_colors(np.array([1.0, 0.5, 0.0])) ) assert _named_tuple_equal( info, _RankPlotInfo( params=["param_b", "param_d"], sub_plot_infos=[ [ _RankSubplotInfo( xaxis=_AxisInfo( name="param_b", range=(-0.1, 2.1), is_log=False, is_cat=False, ), yaxis=_AxisInfo( name="param_d", range=(1.9, 4.1), is_log=False, is_cat=False, ), xs=[2.0, 0.0, 1.0], ys=[4.0, 4.0, 2.0], trials=study.trials, zs=np.array([value, 2.0, 1.0]), colors=colors, ) ] ], target_name="Target Name", zs=np.array([value, 2.0, 1.0]), colors=colors, has_custom_target=True, ), ) def test_generate_rank_info_with_constraints() -> None: study = _create_study_with_constraints() info = _get_rank_info(study, params=None, target=None, target_name="Objective Value") expected_color = _convert_color_idxs_to_scaled_rgb_colors(np.array([0.0, 1.0])) expected_color[1] = [204, 204, 204] assert _named_tuple_equal( info, _RankPlotInfo( params=["param_a", "param_b"], sub_plot_infos=[ [ _RankSubplotInfo( xaxis=_get_axis_info(study.trials, "param_a"), yaxis=_get_axis_info(study.trials, "param_b"), xs=[0.11, 0.19], ys=[0.31, 0.34], trials=study.trials, zs=np.array([0.0, 1.0]), colors=expected_color, ) ] ], target_name="Objective Value", zs=np.array([0.0, 1.0]), colors=expected_color, has_custom_target=False, ), ) def test_get_order_with_same_order_averaging() -> None: x = np.array([6.0, 2.0, 3.0, 1.0, 4.5, 4.5, 8.0, 8.0, 0.0, 8.0]) assert np.all(x == _get_order_with_same_order_averaging(x)) def test_convert_color_idxs_to_scaled_rgb_colors() -> None: x1 = np.array([0.1, 0.2]) result1 = _convert_color_idxs_to_scaled_rgb_colors(x1) np.testing.assert_array_equal(result1, [[69, 117, 180], [116, 173, 209]]) x2 = np.array([]) result2 = _convert_color_idxs_to_scaled_rgb_colors(x2) np.testing.assert_array_equal(result2, []) optuna-4.1.0/tests/visualization_tests/test_slice.py000066400000000000000000000342201471332314300230210ustar00rootroot00000000000000from __future__ import annotations from io import BytesIO from typing import Any from typing import Callable import pytest from optuna.distributions import BaseDistribution from optuna.distributions import CategoricalDistribution from optuna.distributions import FloatDistribution from optuna.study import create_study from optuna.study import Study from optuna.testing.objectives import fail_objective from optuna.testing.visualization import prepare_study_with_trials from optuna.trial import create_trial from optuna.visualization import plot_slice as plotly_plot_slice from optuna.visualization._plotly_imports import go from optuna.visualization._slice import _get_slice_plot_info from optuna.visualization._slice import _SlicePlotInfo from optuna.visualization._slice import _SliceSubplotInfo from optuna.visualization._utils import COLOR_SCALE from optuna.visualization.matplotlib import plot_slice as plt_plot_slice from optuna.visualization.matplotlib._matplotlib_imports import Axes from optuna.visualization.matplotlib._matplotlib_imports import plt parametrize_plot_slice = pytest.mark.parametrize("plot_slice", [plotly_plot_slice, plt_plot_slice]) def _create_study_with_failed_trial() -> Study: study = create_study() study.optimize(fail_objective, n_trials=1, catch=(ValueError,)) return study def _create_study_with_log_scale_and_str_category_2d() -> Study: study = create_study() distributions = { "param_a": FloatDistribution(1e-7, 1e-2, log=True), "param_b": CategoricalDistribution(["100", "101"]), } study.add_trial( create_trial( value=0.0, params={"param_a": 1e-6, "param_b": "101"}, distributions=distributions ) ) study.add_trial( create_trial( value=1.0, params={"param_a": 1e-5, "param_b": "100"}, distributions=distributions ) ) return study def _create_study_mixture_category_types() -> Study: study = create_study() distributions: dict[str, BaseDistribution] = { "param_a": CategoricalDistribution([None, "100"]), "param_b": CategoricalDistribution([101, 102.0]), } study.add_trial( create_trial( value=0.0, params={"param_a": None, "param_b": 101}, distributions=distributions ) ) study.add_trial( create_trial( value=0.5, params={"param_a": "100", "param_b": 102.0}, distributions=distributions ) ) return study @parametrize_plot_slice def test_plot_slice_customized_target_name(plot_slice: Callable[..., Any]) -> None: params = ["param_a", "param_b"] study = prepare_study_with_trials() figure = plot_slice(study, params=params, target_name="Target Name") if isinstance(figure, go.Figure): figure.layout.yaxis.title.text == "Target Name" elif isinstance(figure, Axes): assert figure[0].yaxis.label.get_text() == "Target Name" @parametrize_plot_slice @pytest.mark.parametrize( "specific_create_study, params", [ [create_study, []], [create_study, ["param_a"]], [create_study, None], [prepare_study_with_trials, []], [prepare_study_with_trials, ["param_a"]], [prepare_study_with_trials, None], [_create_study_with_log_scale_and_str_category_2d, None], [_create_study_mixture_category_types, None], ], ) def test_plot_slice( plot_slice: Callable[..., Any], specific_create_study: Callable[[], Study], params: list[str] | None, ) -> None: study = specific_create_study() figure = plot_slice(study, params=params) if isinstance(figure, go.Figure): figure.write_image(BytesIO()) else: plt.savefig(BytesIO()) plt.close() def test_target_is_none_and_study_is_multi_obj() -> None: study = create_study(directions=["minimize", "minimize"]) with pytest.raises(ValueError): _get_slice_plot_info(study, None, target=None, target_name="Objective Value") @pytest.mark.parametrize( "specific_create_study", [create_study, _create_study_with_failed_trial], ) @pytest.mark.parametrize( "params", [ [], ["param_a"], ["param_a", "param_b"], ["param_a", "param_b", "param_c"], ["param_a", "param_b", "param_c", "param_d"], None, ], ) def test_get_slice_plot_info_empty( specific_create_study: Callable[[], Study], params: list[str] | None ) -> None: study = specific_create_study() info = _get_slice_plot_info(study, params=params, target=None, target_name="Objective Value") assert len(info.subplots) == 0 def test_get_slice_plot_info_non_exist_param_error() -> None: study = prepare_study_with_trials() with pytest.raises(ValueError): _get_slice_plot_info(study, params=["optuna"], target=None, target_name="Objective Value") @pytest.mark.parametrize( "params", [ ["param_a"], ["param_a", "param_b"], ["param_a", "param_b", "param_c"], ["param_a", "param_b", "param_c", "param_d"], None, ], ) def test_get_slice_plot_info_params(params: list[str] | None) -> None: study = prepare_study_with_trials() params = ["param_a", "param_b", "param_c", "param_d"] if params is None else params expected_subplot_infos = { "param_a": _SliceSubplotInfo( param_name="param_a", x=[1.0, 2.5], y=[0.0, 1.0], trial_numbers=[0, 2], is_log=False, is_numerical=True, x_labels=None, constraints=[True, True], ), "param_b": _SliceSubplotInfo( param_name="param_b", x=[2.0, 0.0, 1.0], y=[0.0, 2.0, 1.0], trial_numbers=[0, 1, 2], is_log=False, is_numerical=True, x_labels=None, constraints=[True, True, True], ), "param_c": _SliceSubplotInfo( param_name="param_c", x=[3.0, 4.5], y=[0.0, 1.0], trial_numbers=[0, 2], is_log=False, is_numerical=True, x_labels=None, constraints=[True, True], ), "param_d": _SliceSubplotInfo( param_name="param_d", x=[4.0, 4.0, 2.0], y=[0.0, 2.0, 1.0], trial_numbers=[0, 1, 2], is_log=False, is_numerical=True, x_labels=None, constraints=[True, True, True], ), } info = _get_slice_plot_info(study, params=params, target=None, target_name="Objective Value") assert info == _SlicePlotInfo( target_name="Objective Value", subplots=[expected_subplot_infos[p] for p in params], ) def test_get_slice_plot_info_customized_target() -> None: params = ["param_a"] study = prepare_study_with_trials() info = _get_slice_plot_info( study, params=params, target=lambda t: t.params["param_d"], target_name="param_d", ) assert info == _SlicePlotInfo( target_name="param_d", subplots=[ _SliceSubplotInfo( param_name="param_a", x=[1.0, 2.5], y=[4.0, 2.0], trial_numbers=[0, 2], is_log=False, is_numerical=True, x_labels=None, constraints=[True, True], ), ], ) @pytest.mark.parametrize( "params", [ ["param_a", "param_b"], # First column has 1 observation. ["param_b", "param_a"], # Second column has 1 observation ], ) def test_get_slice_plot_info_for_few_observations(params: list[str]) -> None: study = create_study(direction="minimize") study.add_trial( create_trial( values=[0.0], params={"param_a": 1.0, "param_b": 2.0}, distributions={ "param_a": FloatDistribution(0.0, 3.0), "param_b": FloatDistribution(0.0, 3.0), }, ) ) study.add_trial( create_trial( values=[2.0], params={"param_b": 0.0}, distributions={"param_b": FloatDistribution(0.0, 3.0)}, ) ) info = _get_slice_plot_info(study, params, None, "Objective Value") assert info == _SlicePlotInfo( target_name="Objective Value", subplots=[ _SliceSubplotInfo( param_name="param_a", x=[1.0], y=[0.0], trial_numbers=[0], is_log=False, is_numerical=True, x_labels=None, constraints=[True], ), _SliceSubplotInfo( param_name="param_b", x=[2.0, 0.0], y=[0.0, 2.0], trial_numbers=[0, 1], is_log=False, is_numerical=True, x_labels=None, constraints=[True, True], ), ], ) def test_get_slice_plot_info_log_scale_and_str_category_2_params() -> None: study = _create_study_with_log_scale_and_str_category_2d() info = _get_slice_plot_info(study, None, None, "Objective Value") distribution_b = study.trials[0].distributions["param_b"] assert isinstance(distribution_b, CategoricalDistribution) assert info == _SlicePlotInfo( target_name="Objective Value", subplots=[ _SliceSubplotInfo( param_name="param_a", x=[1e-6, 1e-5], y=[0.0, 1.0], trial_numbers=[0, 1], is_log=True, is_numerical=True, x_labels=None, constraints=[True, True], ), _SliceSubplotInfo( param_name="param_b", x=["101", "100"], y=[0.0, 1.0], trial_numbers=[0, 1], is_log=False, is_numerical=False, x_labels=distribution_b.choices, constraints=[True, True], ), ], ) def test_get_slice_plot_info_mixture_category_types() -> None: study = _create_study_mixture_category_types() info = _get_slice_plot_info(study, None, None, "Objective Value") distribution_a = study.trials[0].distributions["param_a"] distribution_b = study.trials[0].distributions["param_b"] assert isinstance(distribution_a, CategoricalDistribution) assert isinstance(distribution_b, CategoricalDistribution) assert info == _SlicePlotInfo( target_name="Objective Value", subplots=[ _SliceSubplotInfo( param_name="param_a", x=[None, "100"], y=[0.0, 0.5], trial_numbers=[0, 1], is_log=False, is_numerical=False, x_labels=distribution_a.choices, constraints=[True, True], ), _SliceSubplotInfo( param_name="param_b", x=[101, 102.0], y=[0.0, 0.5], trial_numbers=[0, 1], is_log=False, is_numerical=False, x_labels=distribution_b.choices, constraints=[True, True], ), ], ) @pytest.mark.parametrize("value", [float("inf"), -float("inf")]) def test_get_slice_plot_info_nonfinite_removed(value: float) -> None: study = prepare_study_with_trials(value_for_first_trial=value) info = _get_slice_plot_info( study, params=["param_b", "param_d"], target=None, target_name="Objective Value" ) assert info == _SlicePlotInfo( target_name="Objective Value", subplots=[ _SliceSubplotInfo( param_name="param_b", x=[0.0, 1.0], y=[2.0, 1.0], trial_numbers=[1, 2], is_log=False, is_numerical=True, x_labels=None, constraints=[True, True], ), _SliceSubplotInfo( param_name="param_d", x=[4.0, 2.0], y=[2.0, 1.0], trial_numbers=[1, 2], is_log=False, is_numerical=True, x_labels=None, constraints=[True, True], ), ], ) @pytest.mark.parametrize("objective", (0, 1)) @pytest.mark.parametrize("value", (float("inf"), -float("inf"))) def test_get_slice_plot_info_nonfinite_multiobjective(objective: int, value: float) -> None: study = prepare_study_with_trials(n_objectives=2, value_for_first_trial=value) info = _get_slice_plot_info( study, params=["param_b", "param_d"], target=lambda t: t.values[objective], target_name="Target Name", ) assert info == _SlicePlotInfo( target_name="Target Name", subplots=[ _SliceSubplotInfo( param_name="param_b", x=[0.0, 1.0], y=[2.0, 1.0], trial_numbers=[1, 2], is_log=False, is_numerical=True, x_labels=None, constraints=[True, True], ), _SliceSubplotInfo( param_name="param_d", x=[4.0, 2.0], y=[2.0, 1.0], trial_numbers=[1, 2], is_log=False, is_numerical=True, x_labels=None, constraints=[True, True], ), ], ) @pytest.mark.parametrize("direction", ["minimize", "maximize"]) def test_color_map(direction: str) -> None: study = create_study(direction=direction) for i in range(3): study.add_trial( create_trial( value=float(i), params={"param_a": float(i), "param_b": float(i)}, distributions={ "param_a": FloatDistribution(0.0, 3.0), "param_b": FloatDistribution(0.0, 3.0), }, ) ) # Since `plot_slice`'s colormap depends on only trial.number, `reversecale` is not in the plot. marker = plotly_plot_slice(study).data[0]["marker"] assert COLOR_SCALE == [v[1] for v in marker["colorscale"]] assert "reversecale" not in marker optuna-4.1.0/tests/visualization_tests/test_terminator_improvement.py000066400000000000000000000115261471332314300265370ustar00rootroot00000000000000from __future__ import annotations from io import BytesIO from typing import Any from typing import Callable import pytest from optuna.distributions import FloatDistribution from optuna.study import create_study from optuna.study import Study from optuna.terminator import BaseErrorEvaluator from optuna.terminator import BaseImprovementEvaluator from optuna.terminator import CrossValidationErrorEvaluator from optuna.terminator import RegretBoundEvaluator from optuna.terminator import report_cross_validation_scores from optuna.terminator import StaticErrorEvaluator from optuna.testing.objectives import fail_objective from optuna.testing.visualization import prepare_study_with_trials from optuna.trial import create_trial from optuna.trial import TrialState from optuna.visualization import plot_terminator_improvement as plotly_plot_terminator_improvement from optuna.visualization._terminator_improvement import _get_improvement_info from optuna.visualization._terminator_improvement import _get_y_range from optuna.visualization._terminator_improvement import _ImprovementInfo parametrize_plot_terminator_improvement = pytest.mark.parametrize( "plot_terminator_improvement", [plotly_plot_terminator_improvement] ) def _create_study_with_failed_trial() -> Study: study = create_study() study.optimize(fail_objective, n_trials=1, catch=(ValueError,)) return study def _prepare_study_with_cross_validation_scores() -> Study: study = create_study() for _ in range(3): trial = study.ask({"x": FloatDistribution(0, 1)}) report_cross_validation_scores(trial, [1.0, 2.0]) study.tell(trial, 0) return study def test_study_is_multi_objective() -> None: study = create_study(directions=["minimize", "minimize"]) with pytest.raises(ValueError): _get_improvement_info(study=study) @parametrize_plot_terminator_improvement @pytest.mark.parametrize( "specific_create_study, plot_error", [ (create_study, False), (_create_study_with_failed_trial, False), (prepare_study_with_trials, False), (_prepare_study_with_cross_validation_scores, False), (_prepare_study_with_cross_validation_scores, True), ], ) def test_plot_terminator_improvement( plot_terminator_improvement: Callable[..., Any], specific_create_study: Callable[[], Study], plot_error: bool, ) -> None: study = specific_create_study() figure = plot_terminator_improvement(study, plot_error) figure.write_image(BytesIO()) @pytest.mark.parametrize( "specific_create_study", [create_study, _create_study_with_failed_trial], ) @pytest.mark.parametrize("plot_error", [False, True]) def test_get_terminator_improvement_info_empty( specific_create_study: Callable[[], Study], plot_error: bool ) -> None: study = specific_create_study() info = _get_improvement_info(study, plot_error) assert info == _ImprovementInfo(trial_numbers=[], improvements=[], errors=None) @pytest.mark.parametrize("get_error", [False, True]) @pytest.mark.parametrize( "improvement_evaluator_class", [lambda: RegretBoundEvaluator(), lambda: None] ) @pytest.mark.parametrize( "error_evaluator_class", [ lambda: CrossValidationErrorEvaluator(), lambda: StaticErrorEvaluator(0), lambda: None, ], ) def test_get_improvement_info( get_error: bool, improvement_evaluator_class: Callable[[], BaseImprovementEvaluator | None], error_evaluator_class: Callable[[], BaseErrorEvaluator | None], ) -> None: study = _prepare_study_with_cross_validation_scores() info = _get_improvement_info( study, get_error, improvement_evaluator_class(), error_evaluator_class() ) assert info.trial_numbers == [0, 1, 2] assert len(info.improvements) == 3 if get_error: assert info.errors is not None assert len(info.errors) == 3 assert info.errors[0] == info.errors[1] == info.errors[2] else: assert info.errors is None def test_get_improvement_info_started_with_failed_trials() -> None: study = create_study() for _ in range(3): study.add_trial(create_trial(state=TrialState.FAIL)) trial = study.ask({"x": FloatDistribution(0, 1)}) study.tell(trial, 0) info = _get_improvement_info(study) assert info.trial_numbers == [3] assert len(info.improvements) == 1 assert info.errors is None @pytest.mark.parametrize( "info", [ _ImprovementInfo(trial_numbers=[0], improvements=[0], errors=None), _ImprovementInfo(trial_numbers=[0], improvements=[0], errors=[0]), _ImprovementInfo(trial_numbers=[0, 1], improvements=[0, 1], errors=[0, 1]), ], ) @pytest.mark.parametrize("min_n_trials", [1, 2]) def test_get_y_range(info: _ImprovementInfo, min_n_trials: int) -> None: y_range = _get_y_range(info, min_n_trials) assert len(y_range) == 2 assert y_range[0] <= y_range[1] optuna-4.1.0/tests/visualization_tests/test_timeline.py000066400000000000000000000150421471332314300235310ustar00rootroot00000000000000from __future__ import annotations import datetime from io import BytesIO import time from typing import Any from typing import Callable import _pytest.capture import pytest import optuna from optuna.samplers._base import _CONSTRAINTS_KEY from optuna.study.study import Study from optuna.trial import TrialState from optuna.visualization import plot_timeline as plotly_plot_timeline from optuna.visualization._plotly_imports import _imports as plotly_imports from optuna.visualization._timeline import _get_timeline_info from optuna.visualization.matplotlib import plot_timeline as plt_plot_timeline from optuna.visualization.matplotlib._matplotlib_imports import _imports as plt_imports if plotly_imports.is_successful(): from optuna.visualization._plotly_imports import go if plt_imports.is_successful(): from optuna.visualization.matplotlib._matplotlib_imports import plt parametrize_plot_timeline = pytest.mark.parametrize( "plot_timeline", [plotly_plot_timeline, plt_plot_timeline], ) def _create_study( trial_states: list[TrialState], trial_sys_attrs: dict[str, Any] | None = None, ) -> Study: study = optuna.create_study() fmax = float(len(trial_states)) for i, s in enumerate(trial_states): study.add_trial( optuna.trial.create_trial( params={"x": float(i)}, distributions={"x": optuna.distributions.FloatDistribution(-1.0, fmax)}, value=0.0 if s == TrialState.COMPLETE else None, state=s, system_attrs=trial_sys_attrs, ) ) return study def _create_study_negative_elapsed_time() -> Study: start = datetime.datetime.now() complete = start - datetime.timedelta(seconds=1.0) study = optuna.create_study() study.add_trial( optuna.trial.FrozenTrial( number=-1, trial_id=-1, state=TrialState.COMPLETE, value=0.0, values=None, datetime_start=start, datetime_complete=complete, params={}, distributions={}, user_attrs={}, system_attrs={}, intermediate_values={}, ) ) return study def test_get_timeline_info_empty() -> None: study = optuna.create_study() info = _get_timeline_info(study) assert len(info.bars) == 0 @pytest.mark.parametrize( "trial_sys_attrs, infeasible", [ (None, False), ({_CONSTRAINTS_KEY: [1.0]}, True), ({_CONSTRAINTS_KEY: [-1.0]}, False), ], ) def test_get_timeline_info(trial_sys_attrs: dict[str, Any] | None, infeasible: bool) -> None: states = [TrialState.COMPLETE, TrialState.RUNNING, TrialState.WAITING] study = _create_study(states, trial_sys_attrs) info = _get_timeline_info(study) assert len(info.bars) == len(study.get_trials()) for bar, trial in zip(info.bars, study.get_trials()): assert bar.number == trial.number assert bar.state == trial.state assert type(bar.hovertext) is str assert isinstance(bar.start, datetime.datetime) assert isinstance(bar.complete, datetime.datetime) assert bar.start <= bar.complete assert bar.infeasible == infeasible def test_get_timeline_info_negative_elapsed_time(capsys: _pytest.capture.CaptureFixture) -> None: # We need to reconstruct our default handler to properly capture stderr. optuna.logging._reset_library_root_logger() optuna.logging.enable_default_handler() optuna.logging.set_verbosity(optuna.logging.WARNING) study = _create_study_negative_elapsed_time() info = _get_timeline_info(study) _, err = capsys.readouterr() assert err != "" assert len(info.bars) == len(study.get_trials()) for bar, trial in zip(info.bars, study.get_trials()): assert bar.number == trial.number assert bar.state == trial.state assert type(bar.hovertext) is str assert isinstance(bar.start, datetime.datetime) assert isinstance(bar.complete, datetime.datetime) assert bar.complete < bar.start @parametrize_plot_timeline @pytest.mark.parametrize( "trial_states", [ [], [TrialState.COMPLETE, TrialState.PRUNED, TrialState.FAIL, TrialState.RUNNING], [TrialState.RUNNING, TrialState.FAIL, TrialState.PRUNED, TrialState.COMPLETE], ], ) def test_get_timeline_plot( plot_timeline: Callable[..., Any], trial_states: list[TrialState] ) -> None: study = _create_study(trial_states) figure = plot_timeline(study) if isinstance(figure, go.Figure): figure.write_image(BytesIO()) else: plt.savefig(BytesIO()) plt.close() @parametrize_plot_timeline @pytest.mark.parametrize("waiting_time", [0.0, 1.5]) def test_get_timeline_plot_with_killed_running_trials( plot_timeline: Callable[..., Any], waiting_time: float ) -> None: def _objective_with_sleep(trial: optuna.Trial) -> float: sleep_start_datetime = datetime.datetime.now() # Spin waiting is used here because high accuracy is necessary even in weak VM. # Please check the motivation of the bugfix in https://github.com/optuna/optuna/pull/5549/ while datetime.datetime.now() - sleep_start_datetime < datetime.timedelta(seconds=0.1): # `sleep(0.1)` is only guaranteed to rest for more than 0.1 second; the actual time # depends on the OS. spin waiting is used here to rest for 0.1 second as precisely as # possible without voluntarily releasing the context. pass assert datetime.datetime.now() - sleep_start_datetime < datetime.timedelta(seconds=0.19) trial.suggest_float("x", -1.0, 1.0) return 1.0 study = optuna.create_study() trial = optuna.trial.create_trial( params={"x": 0.0}, distributions={"x": optuna.distributions.FloatDistribution(-1.0, 1.0)}, value=None, state=TrialState.RUNNING, ) study.add_trial(trial) study.optimize(_objective_with_sleep, n_trials=2) time.sleep(waiting_time) figure = plot_timeline(study) if isinstance(figure, go.Figure): bar_colors = [d["marker"]["color"] for d in figure["data"]] assert "green" in bar_colors, "Running trial, i.e. green color, must be included." bar_length_in_milliseconds = figure["data"][1]["x"][0] # If the waiting time is too long, stop the timeline plots for running trials. assert waiting_time < 1.0 or bar_length_in_milliseconds < waiting_time * 1000 figure.write_image(BytesIO()) else: pytest.skip("Matplotlib test is unimplemented.") optuna-4.1.0/tests/visualization_tests/test_utils.py000066400000000000000000000173551471332314300230740ustar00rootroot00000000000000import datetime import logging from textwrap import dedent from typing import cast import numpy as np import pytest from pytest import LogCaptureFixture import optuna from optuna.distributions import FloatDistribution from optuna.study import create_study from optuna.testing.visualization import prepare_study_with_trials from optuna.trial import create_trial from optuna.trial import FrozenTrial from optuna.trial import TrialState from optuna.visualization import is_available from optuna.visualization._utils import _check_plot_args from optuna.visualization._utils import _filter_nonfinite from optuna.visualization._utils import _is_log_scale from optuna.visualization._utils import _make_hovertext def test_is_log_scale() -> None: study = create_study() study.add_trial( create_trial( value=0.0, params={"param_linear": 1.0}, distributions={"param_linear": FloatDistribution(0.0, 3.0)}, ) ) study.add_trial( create_trial( value=2.0, params={"param_linear": 2.0, "param_log": 1e-3}, distributions={ "param_linear": FloatDistribution(0.0, 3.0), "param_log": FloatDistribution(1e-5, 1.0, log=True), }, ) ) assert _is_log_scale(study.trials, "param_log") assert not _is_log_scale(study.trials, "param_linear") def _is_plotly_available() -> bool: try: import plotly # NOQA available = True except Exception: available = False return available def test_visualization_is_available() -> None: assert is_available() == _is_plotly_available() def test_check_plot_args() -> None: study = create_study(directions=["minimize", "minimize"]) with pytest.raises(ValueError): _check_plot_args(study, None, "Objective Value") with pytest.warns(UserWarning): _check_plot_args(study, lambda t: cast(float, t.value), "Objective Value") @pytest.mark.parametrize("value, expected", [(float("inf"), 1), (-float("inf"), 1), (0.0, 2)]) def test_filter_inf_trials(value: float, expected: int) -> None: study = create_study() study.add_trial( create_trial( value=0.0, params={"x": 1.0}, distributions={"x": FloatDistribution(0.0, 1.0)}, ) ) study.add_trial( create_trial( value=value, params={"x": 0.0}, distributions={"x": FloatDistribution(0.0, 1.0)}, ) ) trials = _filter_nonfinite(study.get_trials(states=(TrialState.COMPLETE,))) assert len(trials) == expected assert all([t.number == num for t, num in zip(trials, range(expected))]) @pytest.mark.parametrize( "value,objective_selected,expected", [ (float("inf"), 0, 2), (-float("inf"), 0, 2), (0.0, 0, 3), (float("inf"), 1, 1), (-float("inf"), 1, 1), (0.0, 1, 3), ], ) def test_filter_inf_trials_multiobjective( value: float, objective_selected: int, expected: int ) -> None: study = create_study(directions=["minimize", "maximize"]) study.add_trial( create_trial( values=[0.0, 1.0], params={"x": 1.0}, distributions={"x": FloatDistribution(0.0, 1.0)}, ) ) study.add_trial( create_trial( values=[0.0, value], params={"x": 0.0}, distributions={"x": FloatDistribution(0.0, 1.0)}, ) ) study.add_trial( create_trial( values=[value, value], params={"x": 0.0}, distributions={"x": FloatDistribution(0.0, 1.0)}, ) ) def _target(t: FrozenTrial) -> float: return t.values[objective_selected] trials = _filter_nonfinite(study.get_trials(states=(TrialState.COMPLETE,)), target=_target) assert len(trials) == expected assert all([t.number == num for t, num in zip(trials, range(expected))]) @pytest.mark.parametrize("with_message", [True, False]) def test_filter_inf_trials_message(caplog: LogCaptureFixture, with_message: bool) -> None: study = create_study() study.add_trial( create_trial( value=0.0, params={"x": 1.0}, distributions={"x": FloatDistribution(0.0, 1.0)}, ) ) study.add_trial( create_trial( value=float("inf"), params={"x": 0.0}, distributions={"x": FloatDistribution(0.0, 1.0)}, ) ) optuna.logging.enable_propagation() _filter_nonfinite(study.get_trials(states=(TrialState.COMPLETE,)), with_message=with_message) msg = "Trial 1 is omitted in visualization because its objective value is inf or nan." if with_message: assert msg in caplog.text n_filtered_as_inf = 0 for record in caplog.records: if record.msg == msg: assert record.levelno == logging.WARNING n_filtered_as_inf += 1 assert n_filtered_as_inf == 1 else: assert msg not in caplog.text @pytest.mark.filterwarnings("ignore::UserWarning") def test_filter_nonfinite_with_invalid_target() -> None: study = prepare_study_with_trials() trials = study.get_trials(states=(TrialState.COMPLETE,)) with pytest.raises(ValueError): _filter_nonfinite(trials, target=lambda t: "invalid target") # type: ignore def test_make_hovertext() -> None: trial_no_user_attrs = FrozenTrial( number=0, trial_id=0, state=TrialState.COMPLETE, value=0.2, datetime_start=datetime.datetime.now(), datetime_complete=datetime.datetime.now(), params={"x": 10}, distributions={"x": FloatDistribution(5, 12)}, user_attrs={}, system_attrs={}, intermediate_values={}, ) assert ( _make_hovertext(trial_no_user_attrs) == dedent( """ { "number": 0, "values": [ 0.2 ], "params": { "x": 10 } } """ ) .strip() .replace("\n", "
") ) trial_user_attrs_valid_json = FrozenTrial( number=0, trial_id=0, state=TrialState.COMPLETE, value=0.2, datetime_start=datetime.datetime.now(), datetime_complete=datetime.datetime.now(), params={"x": 10}, distributions={"x": FloatDistribution(5, 12)}, user_attrs={"a": 42, "b": 3.14}, system_attrs={}, intermediate_values={}, ) assert ( _make_hovertext(trial_user_attrs_valid_json) == dedent( """ { "number": 0, "values": [ 0.2 ], "params": { "x": 10 }, "user_attrs": { "a": 42, "b": 3.14 } } """ ) .strip() .replace("\n", "
") ) trial_user_attrs_invalid_json = FrozenTrial( number=0, trial_id=0, state=TrialState.COMPLETE, value=0.2, datetime_start=datetime.datetime.now(), datetime_complete=datetime.datetime.now(), params={"x": 10}, distributions={"x": FloatDistribution(5, 12)}, user_attrs={"a": 42, "b": 3.14, "c": np.zeros(1), "d": np.nan}, system_attrs={}, intermediate_values={}, ) assert ( _make_hovertext(trial_user_attrs_invalid_json) == dedent( """ { "number": 0, "values": [ 0.2 ], "params": { "x": 10 }, "user_attrs": { "a": 42, "b": 3.14, "c": "[0.]", "d": NaN } } """ ) .strip() .replace("\n", "
") ) optuna-4.1.0/tests/visualization_tests/test_visualizations.py000066400000000000000000000061641471332314300250140ustar00rootroot00000000000000from __future__ import annotations from typing import Callable from matplotlib.axes._axes import Axes import matplotlib.pyplot as plt import plotly.graph_objects as go import pytest import optuna from optuna.study.study import ObjectiveFuncType from optuna.visualization import plot_contour from optuna.visualization import plot_edf from optuna.visualization import plot_optimization_history from optuna.visualization import plot_parallel_coordinate from optuna.visualization import plot_param_importances from optuna.visualization import plot_rank from optuna.visualization import plot_slice from optuna.visualization import plot_timeline from optuna.visualization.matplotlib import ( plot_optimization_history as matplotlib_plot_optimization_history, ) from optuna.visualization.matplotlib import ( plot_parallel_coordinate as matplotlib_plot_parallel_coordinate, ) from optuna.visualization.matplotlib import ( plot_param_importances as matplotlib_plot_param_importances, ) from optuna.visualization.matplotlib import plot_contour as matplotlib_plot_contour from optuna.visualization.matplotlib import plot_edf as matplotlib_plot_edf from optuna.visualization.matplotlib import plot_rank as matplotlib_plot_rank from optuna.visualization.matplotlib import plot_slice as matplotlib_plot_slice from optuna.visualization.matplotlib import plot_timeline as matplotlib_plot_timeline parametrize_visualization_functions_for_single_objective = pytest.mark.parametrize( "plot_func", [ plot_optimization_history, plot_edf, plot_contour, plot_parallel_coordinate, plot_rank, plot_slice, plot_timeline, plot_param_importances, matplotlib_plot_optimization_history, matplotlib_plot_edf, matplotlib_plot_contour, matplotlib_plot_parallel_coordinate, matplotlib_plot_rank, matplotlib_plot_slice, matplotlib_plot_timeline, matplotlib_plot_param_importances, ], ) def objective_single_dynamic_with_categorical(trial: optuna.Trial) -> float: category = trial.suggest_categorical("category", ["foo", "bar"]) if category == "foo": return (trial.suggest_float("x1", 0, 10) - 2) ** 2 else: return -((trial.suggest_float("x2", -10, 0) + 5) ** 2) def objective_single_none_categorical(trial: optuna.Trial) -> float: x = trial.suggest_float("x", -100, 100) trial.suggest_categorical("y", ["foo", None]) return x**2 parametrize_single_objective_functions = pytest.mark.parametrize( "objective_func", [ objective_single_dynamic_with_categorical, objective_single_none_categorical, ], ) @parametrize_visualization_functions_for_single_objective @parametrize_single_objective_functions def test_visualizations_with_single_objectives( plot_func: Callable[[optuna.study.Study], go.Figure | Axes], objective_func: ObjectiveFuncType ) -> None: study = optuna.create_study(sampler=optuna.samplers.RandomSampler()) study.optimize(objective_func, n_trials=20) fig = plot_func(study) # Must not raise an exception here. if isinstance(fig, Axes): plt.close() optuna-4.1.0/tutorial/000077500000000000000000000000001471332314300146665ustar00rootroot00000000000000optuna-4.1.0/tutorial/10_key_features/000077500000000000000000000000001471332314300176545ustar00rootroot00000000000000optuna-4.1.0/tutorial/10_key_features/001_first.py000066400000000000000000000116141471332314300217400ustar00rootroot00000000000000""" .. _first: Lightweight, versatile, and platform agnostic architecture ========================================================== Optuna is entirely written in Python and has few dependencies. This means that we can quickly move to the real example once you get interested in Optuna. Quadratic Function Example -------------------------- Usually, Optuna is used to optimize hyperparameters, but as an example, let's optimize a simple quadratic function: :math:`(x - 2)^2`. """ ################################################################################################### # First of all, import :mod:`optuna`. import optuna ################################################################################################### # In optuna, conventionally functions to be optimized are named `objective`. def objective(trial): x = trial.suggest_float("x", -10, 10) return (x - 2) ** 2 ################################################################################################### # This function returns the value of :math:`(x - 2)^2`. Our goal is to find the value of ``x`` # that minimizes the output of the ``objective`` function. This is the "optimization." # During the optimization, Optuna repeatedly calls and evaluates the objective function with # different values of ``x``. # # A :class:`~optuna.trial.Trial` object corresponds to a single execution of the objective # function and is internally instantiated upon each invocation of the function. # # The `suggest` APIs (for example, :func:`~optuna.trial.Trial.suggest_float`) are called # inside the objective function to obtain parameters for a trial. # :func:`~optuna.trial.Trial.suggest_float` selects parameters uniformly within the range # provided. In our example, from :math:`-10` to :math:`10`. # # To start the optimization, we create a study object and pass the objective function to method # :func:`~optuna.study.Study.optimize` as follows. study = optuna.create_study() study.optimize(objective, n_trials=100) ################################################################################################### # You can get the best parameter as follows. best_params = study.best_params found_x = best_params["x"] print("Found x: {}, (x - 2)^2: {}".format(found_x, (found_x - 2) ** 2)) ################################################################################################### # We can see that the ``x`` value found by Optuna is close to the optimal value of ``2``. ################################################################################################### # .. note:: # When used to search for hyperparameters in machine learning, # usually the objective function would return the loss or accuracy # of the model. ################################################################################################### # Study Object # ------------ # # Let us clarify the terminology in Optuna as follows: # # * **Trial**: A single call of the objective function # * **Study**: An optimization session, which is a set of trials # * **Parameter**: A variable whose value is to be optimized, such as ``x`` in the above example # # In Optuna, we use the study object to manage optimization. # Method :func:`~optuna.study.create_study` returns a study object. # A study object has useful properties for analyzing the optimization outcome. ################################################################################################### # To get the dictionary of parameter name and parameter values: study.best_params ################################################################################################### # To get the best observed value of the objective function: study.best_value ################################################################################################### # To get the best trial: study.best_trial ################################################################################################### # To get all trials: study.trials for trial in study.trials[:2]: # Show first two trials print(trial) ################################################################################################### # To get the number of trials: len(study.trials) ################################################################################################### # By executing :func:`~optuna.study.Study.optimize` again, we can continue the optimization. study.optimize(objective, n_trials=100) ################################################################################################### # To get the updated number of trials: len(study.trials) ################################################################################################### # As the objective function is so easy that the last 100 trials don't improve the result. # However, we can check the result again: best_params = study.best_params found_x = best_params["x"] print("Found x: {}, (x - 2)^2: {}".format(found_x, (found_x - 2) ** 2)) optuna-4.1.0/tutorial/10_key_features/002_configurations.py000066400000000000000000000064571471332314300236550ustar00rootroot00000000000000""" .. _configurations: Pythonic Search Space ===================== For hyperparameter sampling, Optuna provides the following features: - :func:`optuna.trial.Trial.suggest_categorical` for categorical parameters - :func:`optuna.trial.Trial.suggest_int` for integer parameters - :func:`optuna.trial.Trial.suggest_float` for floating point parameters With optional arguments of ``step`` and ``log``, we can discretize or take the logarithm of integer and floating point parameters. """ import optuna def objective(trial): # Categorical parameter optimizer = trial.suggest_categorical("optimizer", ["MomentumSGD", "Adam"]) # Integer parameter num_layers = trial.suggest_int("num_layers", 1, 3) # Integer parameter (log) num_channels = trial.suggest_int("num_channels", 32, 512, log=True) # Integer parameter (discretized) num_units = trial.suggest_int("num_units", 10, 100, step=5) # Floating point parameter dropout_rate = trial.suggest_float("dropout_rate", 0.0, 1.0) # Floating point parameter (log) learning_rate = trial.suggest_float("learning_rate", 1e-5, 1e-2, log=True) # Floating point parameter (discretized) drop_path_rate = trial.suggest_float("drop_path_rate", 0.0, 1.0, step=0.1) ################################################################################################### # Defining Parameter Spaces # ------------------------- # # In Optuna, we define search spaces using familiar Python syntax including conditionals and loops. # # Also, you can use branches or loops depending on the parameter values. # # For more various use, see `examples `__. ################################################################################################### # - Branches: import sklearn.ensemble import sklearn.svm def objective(trial): classifier_name = trial.suggest_categorical("classifier", ["SVC", "RandomForest"]) if classifier_name == "SVC": svc_c = trial.suggest_float("svc_c", 1e-10, 1e10, log=True) classifier_obj = sklearn.svm.SVC(C=svc_c) else: rf_max_depth = trial.suggest_int("rf_max_depth", 2, 32, log=True) classifier_obj = sklearn.ensemble.RandomForestClassifier(max_depth=rf_max_depth) ################################################################################################### # - Loops: # # .. code-block:: python # # import torch # import torch.nn as nn # # # def create_model(trial, in_size): # n_layers = trial.suggest_int("n_layers", 1, 3) # # layers = [] # for i in range(n_layers): # n_units = trial.suggest_int("n_units_l{}".format(i), 4, 128, log=True) # layers.append(nn.Linear(in_size, n_units)) # layers.append(nn.ReLU()) # in_size = n_units # layers.append(nn.Linear(in_size, 10)) # # return nn.Sequential(*layers) ################################################################################################### # Note on the Number of Parameters # ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ # # The difficulty of optimization increases roughly exponentially with regard to the number of parameters. That is, the number of necessary trials increases exponentially when you increase the number of parameters, so it is recommended to not add unimportant parameters. optuna-4.1.0/tutorial/10_key_features/003_efficient_optimization_algorithms.py000066400000000000000000000226011471332314300276040ustar00rootroot00000000000000""" .. _pruning: Efficient Optimization Algorithms ================================= Optuna enables efficient hyperparameter optimization by adopting state-of-the-art algorithms for sampling hyperparameters and pruning efficiently unpromising trials. Sampling Algorithms ------------------- Samplers basically continually narrow down the search space using the records of suggested parameter values and evaluated objective values, leading to an optimal search space which giving off parameters leading to better objective values. More detailed explanation of how samplers suggest parameters is in :class:`~optuna.samplers.BaseSampler`. Optuna provides the following sampling algorithms: - Grid Search implemented in :class:`~optuna.samplers.GridSampler` - Random Search implemented in :class:`~optuna.samplers.RandomSampler` - Tree-structured Parzen Estimator algorithm implemented in :class:`~optuna.samplers.TPESampler` - CMA-ES based algorithm implemented in :class:`~optuna.samplers.CmaEsSampler` - Gaussian process-based algorithm implemented in :class:`~optuna.samplers.GPSampler` - Algorithm to enable partial fixed parameters implemented in :class:`~optuna.samplers.PartialFixedSampler` - Nondominated Sorting Genetic Algorithm II implemented in :class:`~optuna.samplers.NSGAIISampler` - A Quasi Monte Carlo sampling algorithm implemented in :class:`~optuna.samplers.QMCSampler` The default sampler is :class:`~optuna.samplers.TPESampler`. Switching Samplers ------------------ """ import optuna ################################################################################################### # By default, Optuna uses :class:`~optuna.samplers.TPESampler` as follows. study = optuna.create_study() print(f"Sampler is {study.sampler.__class__.__name__}") ################################################################################################### # If you want to use different samplers for example :class:`~optuna.samplers.RandomSampler` # and :class:`~optuna.samplers.CmaEsSampler`, study = optuna.create_study(sampler=optuna.samplers.RandomSampler()) print(f"Sampler is {study.sampler.__class__.__name__}") study = optuna.create_study(sampler=optuna.samplers.CmaEsSampler()) print(f"Sampler is {study.sampler.__class__.__name__}") ################################################################################################### # Pruning Algorithms # ------------------ # # ``Pruners`` automatically stop unpromising trials at the early stages of the training (a.k.a., automated early-stopping). # Currently :mod:`~optuna.pruners` module is expected to be used only for single-objective optimization. # # Optuna provides the following pruning algorithms: # # - Median pruning algorithm implemented in :class:`~optuna.pruners.MedianPruner` # # - Non-pruning algorithm implemented in :class:`~optuna.pruners.NopPruner` # # - Algorithm to operate pruner with tolerance implemented in :class:`~optuna.pruners.PatientPruner` # # - Algorithm to prune specified percentile of trials implemented in :class:`~optuna.pruners.PercentilePruner` # # - Asynchronous Successive Halving algorithm implemented in :class:`~optuna.pruners.SuccessiveHalvingPruner` # # - Hyperband algorithm implemented in :class:`~optuna.pruners.HyperbandPruner` # # - Threshold pruning algorithm implemented in :class:`~optuna.pruners.ThresholdPruner` # # - A pruning algorithm based on `Wilcoxon signed-rank test `__ implemented in :class:`~optuna.pruners.WilcoxonPruner` # # We use :class:`~optuna.pruners.MedianPruner` in most examples, # though basically it is outperformed by :class:`~optuna.pruners.SuccessiveHalvingPruner` and # :class:`~optuna.pruners.HyperbandPruner` as in `this benchmark result `__. # # # Activating Pruners # ------------------ # To turn on the pruning feature, you need to call :func:`~optuna.trial.Trial.report` and :func:`~optuna.trial.Trial.should_prune` after each step of the iterative training. # :func:`~optuna.trial.Trial.report` periodically monitors the intermediate objective values. # :func:`~optuna.trial.Trial.should_prune` decides termination of the trial that does not meet a predefined condition. # # We would recommend using integration modules for major machine learning frameworks. # Exclusive list is :mod:`~optuna.integration` and usecases are available in `optuna-examples `__. import logging import sys import sklearn.datasets import sklearn.linear_model import sklearn.model_selection def objective(trial): iris = sklearn.datasets.load_iris() classes = list(set(iris.target)) train_x, valid_x, train_y, valid_y = sklearn.model_selection.train_test_split( iris.data, iris.target, test_size=0.25, random_state=0 ) alpha = trial.suggest_float("alpha", 1e-5, 1e-1, log=True) clf = sklearn.linear_model.SGDClassifier(alpha=alpha) for step in range(100): clf.partial_fit(train_x, train_y, classes=classes) # Report intermediate objective value. intermediate_value = 1.0 - clf.score(valid_x, valid_y) trial.report(intermediate_value, step) # Handle pruning based on the intermediate value. if trial.should_prune(): raise optuna.TrialPruned() return 1.0 - clf.score(valid_x, valid_y) ################################################################################################### # Set up the median stopping rule as the pruning condition. # Add stream handler of stdout to show the messages optuna.logging.get_logger("optuna").addHandler(logging.StreamHandler(sys.stdout)) study = optuna.create_study(pruner=optuna.pruners.MedianPruner()) study.optimize(objective, n_trials=20) ################################################################################################### # As you can see, several trials were pruned (stopped) before they finished all of the iterations. # The format of message is ``"Trial pruned."``. ################################################################################################### # Which Sampler and Pruner Should be Used? # ---------------------------------------- # # From the benchmark results which are available at `optuna/optuna - wiki "Benchmarks with Kurobako" `__, at least for not deep learning tasks, we would say that # # * For :class:`~optuna.samplers.RandomSampler`, :class:`~optuna.pruners.MedianPruner` is the best. # * For :class:`~optuna.samplers.TPESampler`, :class:`~optuna.pruners.HyperbandPruner` is the best. # # However, note that the benchmark is not deep learning. # For deep learning tasks, # consult the below table. # This table is from the `Ozaki et al., Hyperparameter Optimization Methods: Overview and Characteristics, in IEICE Trans, Vol.J103-D No.9 pp.615-631, 2020 `__ paper, # which is written in Japanese. # # +---------------------------+-----------------------------------------+---------------------------------------------------------------+ # | Parallel Compute Resource | Categorical/Conditional Hyperparameters | Recommended Algorithms | # +===========================+=========================================+===============================================================+ # | Limited | No | TPE. GP-EI if search space is low-dimensional and continuous. | # + +-----------------------------------------+---------------------------------------------------------------+ # | | Yes | TPE. GP-EI if search space is low-dimensional and continuous | # +---------------------------+-----------------------------------------+---------------------------------------------------------------+ # | Sufficient | No | CMA-ES, Random Search | # + +-----------------------------------------+---------------------------------------------------------------+ # | | Yes | Random Search or Genetic Algorithm | # +---------------------------+-----------------------------------------+---------------------------------------------------------------+ # ################################################################################################### # Integration Modules for Pruning # ------------------------------- # To implement pruning mechanism in much simpler forms, Optuna provides integration modules for the following libraries. # # For the complete list of Optuna's integration modules, see :mod:`~optuna.integration`. # # For example, `LightGBMPruningCallback `__ introduces pruning without directly changing the logic of training iteration. # (See also `example `__ for the entire script.) # # .. code-block:: text # # import optuna.integration # # pruning_callback = optuna.integration.LightGBMPruningCallback(trial, 'validation-error') # gbm = lgb.train(param, dtrain, valid_sets=[dvalid], callbacks=[pruning_callback]) optuna-4.1.0/tutorial/10_key_features/004_distributed.py000066400000000000000000000067701471332314300231450ustar00rootroot00000000000000""" .. _distributed: Easy Parallelization ==================== It's straightforward to parallelize :func:`optuna.study.Study.optimize`. If you want to manually execute Optuna optimization: 1. start an RDB server (this example uses MySQL) 2. create a study with ``--storage`` argument 3. share the study among multiple nodes and processes Of course, you can use Kubernetes as in `the kubernetes examples `__. To just see how parallel optimization works in Optuna, check the below video. .. raw:: html Create a Study -------------- You can create a study using ``optuna create-study`` command. Alternatively, in Python script you can use :func:`optuna.create_study`. .. code-block:: bash $ mysql -u root -e "CREATE DATABASE IF NOT EXISTS example" $ optuna create-study --study-name "distributed-example" --storage "mysql://root@localhost/example" [I 2020-07-21 13:43:39,642] A new study created with name: distributed-example Then, write an optimization script. Let's assume that ``foo.py`` contains the following code. .. code-block:: python import optuna def objective(trial): x = trial.suggest_float("x", -10, 10) return (x - 2) ** 2 if __name__ == "__main__": study = optuna.load_study( study_name="distributed-example", storage="mysql://root@localhost/example" ) study.optimize(objective, n_trials=100) Share the Study among Multiple Nodes and Processes -------------------------------------------------- Finally, run the shared study from multiple processes. For example, run ``Process 1`` in a terminal, and do ``Process 2`` in another one. They get parameter suggestions based on shared trials' history. Process 1: .. code-block:: bash $ python foo.py [I 2020-07-21 13:45:02,973] Trial 0 finished with value: 45.35553104173011 and parameters: {'x': 8.73465151598285}. Best is trial 0 with value: 45.35553104173011. [I 2020-07-21 13:45:04,013] Trial 2 finished with value: 4.6002397305938905 and parameters: {'x': 4.144816945707463}. Best is trial 1 with value: 0.028194513284051464. ... Process 2 (the same command as process 1): .. code-block:: bash $ python foo.py [I 2020-07-21 13:45:03,748] Trial 1 finished with value: 0.028194513284051464 and parameters: {'x': 1.8320877810162361}. Best is trial 1 with value: 0.028194513284051464. [I 2020-07-21 13:45:05,783] Trial 3 finished with value: 24.45966755098074 and parameters: {'x': 6.945671597566982}. Best is trial 1 with value: 0.028194513284051464. ... .. note:: ``n_trials`` is the number of trials each process will run, not the total number of trials across all processes. For example, the script given above runs 100 trials for each process, 100 trials * 2 processes = 200 trials. :class:`optuna.study.MaxTrialsCallback` can ensure how many times trials will be performed across all processes. .. note:: We do not recommend SQLite for distributed optimizations at scale because it may cause deadlocks and serious performance issues. Please consider to use another database engine like PostgreSQL or MySQL. .. note:: Please avoid putting the SQLite database on NFS when running distributed optimizations. See also: https://www.sqlite.org/faq.html#q5 """ optuna-4.1.0/tutorial/10_key_features/005_visualization.py000066400000000000000000000227431471332314300235230ustar00rootroot00000000000000""" .. _visualization: Quick Visualization for Hyperparameter Optimization Analysis ============================================================ Optuna provides various visualization features in :mod:`optuna.visualization` to analyze optimization results visually. Note that this tutorial requires `Plotly `__ to be installed: .. code-block:: console $ pip install plotly # Required if you are running this tutorial in Jupyter Notebook. $ pip install nbformat If you prefer to use `Matplotlib `__ instead of Plotly, please run the following command: .. code-block:: console $ pip install matplotlib This tutorial walks you through this module by visualizing the optimization results of PyTorch model for FashionMNIST dataset. For visualizing multi-objective optimization (i.e., the usage of :func:`optuna.visualization.plot_pareto_front`), please refer to the tutorial of :ref:`multi_objective`. .. note:: By using `Optuna Dashboard `__, you can also check the optimization history, hyperparameter importances, hyperparameter relationships, etc. in graphs and tables. Please make your study persistent using :ref:`RDB backend ` and execute following commands to run Optuna Dashboard. .. code-block:: console $ pip install optuna-dashboard $ optuna-dashboard sqlite:///example-study.db Please check out `the GitHub repository `__ for more details. .. list-table:: :header-rows: 1 * - Manage Studies - Visualize with Interactive Graphs * - .. image:: https://user-images.githubusercontent.com/5564044/205545958-305f2354-c7cd-4687-be2f-9e46e7401838.gif - .. image:: https://user-images.githubusercontent.com/5564044/205545965-278cd7f4-da7d-4e2e-ac31-6d81b106cada.gif """ ################################################################################################### import torch import torch.nn as nn import torch.nn.functional as F import torchvision import optuna # You can use Matplotlib instead of Plotly for visualization by simply replacing `optuna.visualization` with # `optuna.visualization.matplotlib` in the following examples. from optuna.visualization import plot_contour from optuna.visualization import plot_edf from optuna.visualization import plot_intermediate_values from optuna.visualization import plot_optimization_history from optuna.visualization import plot_parallel_coordinate from optuna.visualization import plot_param_importances from optuna.visualization import plot_rank from optuna.visualization import plot_slice from optuna.visualization import plot_timeline SEED = 13 torch.manual_seed(SEED) DEVICE = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu") DIR = ".." BATCHSIZE = 128 N_TRAIN_EXAMPLES = BATCHSIZE * 30 N_VALID_EXAMPLES = BATCHSIZE * 10 def define_model(trial): n_layers = trial.suggest_int("n_layers", 1, 2) layers = [] in_features = 28 * 28 for i in range(n_layers): out_features = trial.suggest_int("n_units_l{}".format(i), 64, 512) layers.append(nn.Linear(in_features, out_features)) layers.append(nn.ReLU()) in_features = out_features layers.append(nn.Linear(in_features, 10)) layers.append(nn.LogSoftmax(dim=1)) return nn.Sequential(*layers) # Defines training and evaluation. def train_model(model, optimizer, train_loader): model.train() for batch_idx, (data, target) in enumerate(train_loader): data, target = data.view(-1, 28 * 28).to(DEVICE), target.to(DEVICE) optimizer.zero_grad() F.nll_loss(model(data), target).backward() optimizer.step() def eval_model(model, valid_loader): model.eval() correct = 0 with torch.no_grad(): for batch_idx, (data, target) in enumerate(valid_loader): data, target = data.view(-1, 28 * 28).to(DEVICE), target.to(DEVICE) pred = model(data).argmax(dim=1, keepdim=True) correct += pred.eq(target.view_as(pred)).sum().item() accuracy = correct / N_VALID_EXAMPLES return accuracy ################################################################################################### # Define the objective function. def objective(trial): train_dataset = torchvision.datasets.FashionMNIST( DIR, train=True, download=True, transform=torchvision.transforms.ToTensor() ) train_loader = torch.utils.data.DataLoader( torch.utils.data.Subset(train_dataset, list(range(N_TRAIN_EXAMPLES))), batch_size=BATCHSIZE, shuffle=True, ) val_dataset = torchvision.datasets.FashionMNIST( DIR, train=False, transform=torchvision.transforms.ToTensor() ) val_loader = torch.utils.data.DataLoader( torch.utils.data.Subset(val_dataset, list(range(N_VALID_EXAMPLES))), batch_size=BATCHSIZE, shuffle=True, ) model = define_model(trial).to(DEVICE) optimizer = torch.optim.Adam( model.parameters(), trial.suggest_float("lr", 1e-5, 1e-1, log=True) ) for epoch in range(10): train_model(model, optimizer, train_loader) val_accuracy = eval_model(model, val_loader) trial.report(val_accuracy, epoch) if trial.should_prune(): raise optuna.exceptions.TrialPruned() return val_accuracy ################################################################################################### study = optuna.create_study( direction="maximize", sampler=optuna.samplers.TPESampler(seed=SEED), pruner=optuna.pruners.MedianPruner(), ) study.optimize(objective, n_trials=30, timeout=300) ################################################################################################### # Plot functions # -------------- # Visualize the optimization history. See :func:`~optuna.visualization.plot_optimization_history` for the details. plot_optimization_history(study) ################################################################################################### # Visualize the learning curves of the trials. See :func:`~optuna.visualization.plot_intermediate_values` for the details. plot_intermediate_values(study) ################################################################################################### # Visualize high-dimensional parameter relationships. See :func:`~optuna.visualization.plot_parallel_coordinate` for the details. plot_parallel_coordinate(study) ################################################################################################### # Select parameters to visualize. plot_parallel_coordinate(study, params=["lr", "n_layers"]) ################################################################################################### # Visualize hyperparameter relationships. See :func:`~optuna.visualization.plot_contour` for the details. plot_contour(study) ################################################################################################### # Select parameters to visualize. plot_contour(study, params=["lr", "n_layers"]) ################################################################################################### # Visualize individual hyperparameters as slice plot. See :func:`~optuna.visualization.plot_slice` for the details. plot_slice(study) ################################################################################################### # Select parameters to visualize. plot_slice(study, params=["lr", "n_layers"]) ################################################################################################### # Visualize parameter importances. See :func:`~optuna.visualization.plot_param_importances` for the details. plot_param_importances(study) ################################################################################################### # Learn which hyperparameters are affecting the trial duration with hyperparameter importance. optuna.visualization.plot_param_importances( study, target=lambda t: t.duration.total_seconds(), target_name="duration" ) ################################################################################################### # Visualize empirical distribution function. See :func:`~optuna.visualization.plot_edf` for the details. plot_edf(study) ################################################################################################### # Visualize parameter relations with scatter plots colored by objective values. See :func:`~optuna.visualization.plot_rank` for the details. plot_rank(study) ################################################################################################### # Visualize the optimization timeline of performed trials. See :func:`~optuna.visualization.plot_timeline` for the details. plot_timeline(study) ################################################################################################### # Customize generated figures # --------------------------- # In :mod:`optuna.visualization` and :mod:`optuna.visualization.matplotlib`, a function returns an editable figure object: # :class:`plotly.graph_objects.Figure` or :class:`matplotlib.axes.Axes` depending on the module. # This allows users to modify the generated figure for their demand by using API of the visualization library. # The following example replaces figure titles drawn by Plotly-based :func:`~optuna.visualization.plot_intermediate_values` manually. fig = plot_intermediate_values(study) fig.update_layout( title="Hyperparameter optimization for FashionMNIST classification", xaxis_title="Epoch", yaxis_title="Validation Accuracy", ) optuna-4.1.0/tutorial/10_key_features/README.rst000066400000000000000000000002311471332314300213370ustar00rootroot00000000000000.. _key_features: Key Features ------------ Showcases Optuna's `Key Features `__. optuna-4.1.0/tutorial/20_recipes/000077500000000000000000000000001471332314300166215ustar00rootroot00000000000000optuna-4.1.0/tutorial/20_recipes/001_rdb.py000066400000000000000000000100701471332314300203200ustar00rootroot00000000000000""" .. _rdb: Saving/Resuming Study with RDB Backend ========================================== An RDB backend enables persistent experiments (i.e., to save and resume a study) as well as access to history of studies. In addition, we can run multi-node optimization tasks with this feature, which is described in :ref:`distributed`. In this section, let's try simple examples running on a local environment with SQLite DB. .. note:: You can also utilize other RDB backends, e.g., PostgreSQL or MySQL, by setting the storage argument to the DB's URL. Please refer to `SQLAlchemy's document `__ for how to set up the URL. New Study --------- We can create a persistent study by calling :func:`~optuna.study.create_study` function as follows. An SQLite file ``example.db`` is automatically initialized with a new study record. """ import logging import sys import optuna # Add stream handler of stdout to show the messages optuna.logging.get_logger("optuna").addHandler(logging.StreamHandler(sys.stdout)) study_name = "example-study" # Unique identifier of the study. storage_name = "sqlite:///{}.db".format(study_name) study = optuna.create_study(study_name=study_name, storage=storage_name) ################################################################################################### # To run a study, call :func:`~optuna.study.Study.optimize` method passing an objective function. def objective(trial): x = trial.suggest_float("x", -10, 10) return (x - 2) ** 2 study.optimize(objective, n_trials=3) ################################################################################################### # Resume Study # ------------ # # To resume a study, instantiate a :class:`~optuna.study.Study` object # passing the study name ``example-study`` and the DB URL ``sqlite:///example-study.db``. study = optuna.create_study(study_name=study_name, storage=storage_name, load_if_exists=True) study.optimize(objective, n_trials=3) ################################################################################################### # Note that the storage doesn't store the state of the instance of :mod:`~optuna.samplers` # and :mod:`~optuna.pruners`. # When we resume a study with a sampler whose ``seed`` argument is specified for # reproducibility, you need to restore the sampler with using ``pickle`` as follows:: # # import pickle # # # Save the sampler with pickle to be loaded later. # with open("sampler.pkl", "wb") as fout: # pickle.dump(study.sampler, fout) # # restored_sampler = pickle.load(open("sampler.pkl", "rb")) # study = optuna.create_study( # study_name=study_name, storage=storage_name, load_if_exists=True, sampler=restored_sampler # ) # study.optimize(objective, n_trials=3) # ################################################################################################### # Experimental History # -------------------- # # Note that this section requires the installation of `Pandas `__: # # .. code-block:: bash # # $ pip install pandas # # We can access histories of studies and trials via the :class:`~optuna.study.Study` class. # For example, we can get all trials of ``example-study`` as: study = optuna.create_study(study_name=study_name, storage=storage_name, load_if_exists=True) df = study.trials_dataframe(attrs=("number", "value", "params", "state")) ################################################################################################### # The method :func:`~optuna.study.Study.trials_dataframe` returns a pandas dataframe like: print(df) ################################################################################################### # A :class:`~optuna.study.Study` object also provides properties # such as :attr:`~optuna.study.Study.trials`, :attr:`~optuna.study.Study.best_value`, # :attr:`~optuna.study.Study.best_params` (see also :ref:`first`). print("Best params: ", study.best_params) print("Best value: ", study.best_value) print("Best Trial: ", study.best_trial) print("Trials: ", study.trials) optuna-4.1.0/tutorial/20_recipes/002_multi_objective.py000066400000000000000000000127521471332314300227470ustar00rootroot00000000000000""" .. _multi_objective: Multi-objective Optimization with Optuna ======================================== This tutorial showcases Optuna's multi-objective optimization feature by optimizing the validation accuracy of Fashion MNIST dataset and the FLOPS of the model implemented in PyTorch. We use `fvcore `__ to measure FLOPS. """ import torch import torch.nn as nn import torch.nn.functional as F import torchvision from fvcore.nn import FlopCountAnalysis import optuna DEVICE = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu") DIR = ".." BATCHSIZE = 128 N_TRAIN_EXAMPLES = BATCHSIZE * 30 N_VALID_EXAMPLES = BATCHSIZE * 10 def define_model(trial): n_layers = trial.suggest_int("n_layers", 1, 3) layers = [] in_features = 28 * 28 for i in range(n_layers): out_features = trial.suggest_int("n_units_l{}".format(i), 4, 128) layers.append(nn.Linear(in_features, out_features)) layers.append(nn.ReLU()) p = trial.suggest_float("dropout_{}".format(i), 0.2, 0.5) layers.append(nn.Dropout(p)) in_features = out_features layers.append(nn.Linear(in_features, 10)) layers.append(nn.LogSoftmax(dim=1)) return nn.Sequential(*layers) # Defines training and evaluation. def train_model(model, optimizer, train_loader): model.train() for batch_idx, (data, target) in enumerate(train_loader): data, target = data.view(-1, 28 * 28).to(DEVICE), target.to(DEVICE) optimizer.zero_grad() F.nll_loss(model(data), target).backward() optimizer.step() def eval_model(model, valid_loader): model.eval() correct = 0 with torch.no_grad(): for batch_idx, (data, target) in enumerate(valid_loader): data, target = data.view(-1, 28 * 28).to(DEVICE), target.to(DEVICE) pred = model(data).argmax(dim=1, keepdim=True) correct += pred.eq(target.view_as(pred)).sum().item() accuracy = correct / N_VALID_EXAMPLES flops = FlopCountAnalysis(model, inputs=(torch.randn(1, 28 * 28).to(DEVICE),)).total() return flops, accuracy ################################################################################################### # Define multi-objective objective function. # Objectives are FLOPS and accuracy. def objective(trial): train_dataset = torchvision.datasets.FashionMNIST( DIR, train=True, download=True, transform=torchvision.transforms.ToTensor() ) train_loader = torch.utils.data.DataLoader( torch.utils.data.Subset(train_dataset, list(range(N_TRAIN_EXAMPLES))), batch_size=BATCHSIZE, shuffle=True, ) val_dataset = torchvision.datasets.FashionMNIST( DIR, train=False, transform=torchvision.transforms.ToTensor() ) val_loader = torch.utils.data.DataLoader( torch.utils.data.Subset(val_dataset, list(range(N_VALID_EXAMPLES))), batch_size=BATCHSIZE, shuffle=True, ) model = define_model(trial).to(DEVICE) optimizer = torch.optim.Adam( model.parameters(), trial.suggest_float("lr", 1e-5, 1e-1, log=True) ) for epoch in range(10): train_model(model, optimizer, train_loader) flops, accuracy = eval_model(model, val_loader) return flops, accuracy ################################################################################################### # Run multi-objective optimization # -------------------------------- # # If your optimization problem is multi-objective, # Optuna assumes that you will specify the optimization direction for each objective. # Specifically, in this example, we want to minimize the FLOPS (we want a faster model) # and maximize the accuracy. So we set ``directions`` to ``["minimize", "maximize"]``. study = optuna.create_study(directions=["minimize", "maximize"]) study.optimize(objective, n_trials=30, timeout=300) print("Number of finished trials: ", len(study.trials)) ################################################################################################### # Note that the following sections requires the installation of `Plotly `__ for visualization # and `scikit-learn `__ for hyperparameter importance calculation: # # .. code-block:: console # # $ pip install plotly # $ pip install scikit-learn # $ pip install nbformat # Required if you are running this tutorial in Jupyter Notebook. # # Check trials on Pareto front visually. optuna.visualization.plot_pareto_front(study, target_names=["FLOPS", "accuracy"]) ################################################################################################### # Fetch the list of trials on the Pareto front with :attr:`~optuna.study.Study.best_trials`. # # For example, the following code shows the number of trials on the Pareto front and picks the trial with the highest accuracy. print(f"Number of trials on the Pareto front: {len(study.best_trials)}") trial_with_highest_accuracy = max(study.best_trials, key=lambda t: t.values[1]) print("Trial with highest accuracy: ") print(f"\tnumber: {trial_with_highest_accuracy.number}") print(f"\tparams: {trial_with_highest_accuracy.params}") print(f"\tvalues: {trial_with_highest_accuracy.values}") ################################################################################################### # Learn which hyperparameters are affecting the flops most with hyperparameter importance. optuna.visualization.plot_param_importances( study, target=lambda t: t.values[0], target_name="flops" ) optuna-4.1.0/tutorial/20_recipes/003_attributes.py000066400000000000000000000055451471332314300217540ustar00rootroot00000000000000""" .. _attributes: User Attributes =============== This feature is to annotate experiments with user-defined attributes. """ ################################################################################################### # Adding User Attributes to Studies # --------------------------------- # # A :class:`~optuna.study.Study` object provides :func:`~optuna.study.Study.set_user_attr` method # to register a pair of key and value as an user-defined attribute. # A key is supposed to be a ``str``, and a value be any object serializable with ``json.dumps``. import sklearn.datasets import sklearn.model_selection import sklearn.svm import optuna study = optuna.create_study(storage="sqlite:///example.db") study.set_user_attr("contributors", ["Akiba", "Sano"]) study.set_user_attr("dataset", "MNIST") ################################################################################################### # We can access annotated attributes with :attr:`~optuna.study.Study.user_attrs` property. study.user_attrs # {'contributors': ['Akiba', 'Sano'], 'dataset': 'MNIST'} ################################################################################################### # :class:`~optuna.study.StudySummary` object, which can be retrieved by # :func:`~optuna.study.get_all_study_summaries`, also contains user-defined attributes. study_summaries = optuna.get_all_study_summaries("sqlite:///example.db") study_summaries[0].user_attrs # {"contributors": ["Akiba", "Sano"], "dataset": "MNIST"} ################################################################################################### # .. seealso:: # ``optuna study set-user-attr`` command, which sets an attribute via command line interface. ################################################################################################### # Adding User Attributes to Trials # -------------------------------- # # As with :class:`~optuna.study.Study`, a :class:`~optuna.trial.Trial` object provides # :func:`~optuna.trial.Trial.set_user_attr` method. # Attributes are set inside an objective function. def objective(trial): iris = sklearn.datasets.load_iris() x, y = iris.data, iris.target svc_c = trial.suggest_float("svc_c", 1e-10, 1e10, log=True) clf = sklearn.svm.SVC(C=svc_c) accuracy = sklearn.model_selection.cross_val_score(clf, x, y).mean() trial.set_user_attr("accuracy", accuracy) return 1.0 - accuracy # return error for minimization study.optimize(objective, n_trials=1) ################################################################################################### # We can access annotated attributes as: study.trials[0].user_attrs ################################################################################################### # Note that, in this example, the attribute is not annotated to a :class:`~optuna.study.Study` # but a single :class:`~optuna.trial.Trial`. optuna-4.1.0/tutorial/20_recipes/004_cli.py000066400000000000000000000060031471332314300203240ustar00rootroot00000000000000""" .. _cli: Command-Line Interface ====================== .. csv-table:: :header: Command, Description :widths: 20, 40 :escape: \\ ask, Create a new trial and suggest parameters. best-trial, Show the best trial. best-trials, Show a list of trials located at the Pareto front. create-study, Create a new study. delete-study, Delete a specified study. storage upgrade, Upgrade the schema of a storage. studies, Show a list of studies. study set-user-attr, Set a user attribute to a study. tell, Finish a trial\\, which was created by the ask command. trials, Show a list of trials. Optuna provides command-line interface as shown in the above table. Let us assume you are not in IPython shell and writing Python script files instead. It is totally fine to write scripts like the following: """ import optuna def objective(trial): x = trial.suggest_float("x", -10, 10) return (x - 2) ** 2 if __name__ == "__main__": study = optuna.create_study() study.optimize(objective, n_trials=100) print("Best value: {} (params: {})\n".format(study.best_value, study.best_params)) ################################################################################################### # However, if we cannot write ``objective`` explicitly in Python code such as developing a new # drug in a lab, an interactive way is suitable. # In Optuna CLI, :ref:`ask_and_tell` style commands provide such an interactive and flexible interface. # # Let us assume we minimize the objective value depending on a parameter ``x`` in :math:`[-10, 10]` # and objective value is calculated via some experiments by hand. # Even so, we can invoke the optimization as follows. # Don't care about ``--storage sqlite:///example.db`` for now, which is described in :ref:`rdb`. # # .. code-block:: bash # # $ STUDY_NAME=`optuna create-study --storage sqlite:///example.db` # $ optuna ask --storage sqlite:///example.db --study-name $STUDY_NAME --sampler TPESampler \ # --search-space '{"x": {"name": "FloatDistribution", "attributes": {"step": null, "low": -10.0, "high": 10.0, "log": false}}}' # # # [I 2022-08-20 06:08:53,158] Asked trial 0 with parameters {'x': 2.512238141966016}. # {"number": 0, "params": {"x": 2.512238141966016}} # # The argument of ``--search-space`` option can be generated by using # :func:`optuna.distributions.distribution_to_json`, for example, # ``optuna.distributions.distribution_to_json(optuna.distributions.FloatDistribution(-10, 10))``. # Please refer to :class:`optuna.distributions.FloatDistribution` and # :class:`optuna.distributions.IntDistribution` for detailed explanations of their arguments. # # After conducting an experiment using the suggested parameter in the lab, # we store the result to Optuna's study as follows: # # .. code-block:: bash # # $ optuna tell --storage sqlite:///example.db --study-name $STUDY_NAME --trial-number 0 --values 0.7 --state complete # [I 2022-08-20 06:22:50,888] Told trial 0 with values [0.7] and state TrialState.COMPLETE. # optuna-4.1.0/tutorial/20_recipes/005_user_defined_sampler.py000066400000000000000000000150611471332314300237410ustar00rootroot00000000000000""" .. _user_defined_sampler: User-Defined Sampler ==================== Thanks to user-defined samplers, you can: - experiment your own sampling algorithms, - implement task-specific algorithms to refine the optimization performance, or - wrap other optimization libraries to integrate them into Optuna pipelines (e.g., `BoTorchSampler `__). This section describes the internal behavior of sampler classes and shows an example of implementing a user-defined sampler. Overview of Sampler ------------------- A sampler has the responsibility to determine the parameter values to be evaluated in a trial. When a `suggest` API (e.g., :func:`~optuna.trial.Trial.suggest_float`) is called inside an objective function, the corresponding distribution object (e.g., :class:`~optuna.distributions.FloatDistribution`) is created internally. A sampler samples a parameter value from the distribution. The sampled value is returned to the caller of the `suggest` API and evaluated in the objective function. To create a new sampler, you need to define a class that inherits :class:`~optuna.samplers.BaseSampler`. The base class has three abstract methods; :meth:`~optuna.samplers.BaseSampler.infer_relative_search_space`, :meth:`~optuna.samplers.BaseSampler.sample_relative`, and :meth:`~optuna.samplers.BaseSampler.sample_independent`. As the method names imply, Optuna supports two types of sampling: one is **relative sampling** that can consider the correlation of the parameters in a trial, and the other is **independent sampling** that samples each parameter independently. At the beginning of a trial, :meth:`~optuna.samplers.BaseSampler.infer_relative_search_space` is called to provide the relative search space for the trial. Then, :meth:`~optuna.samplers.BaseSampler.sample_relative` is invoked to sample relative parameters from the search space. During the execution of the objective function, :meth:`~optuna.samplers.BaseSampler.sample_independent` is used to sample parameters that don't belong to the relative search space. .. note:: Please refer to the document of :class:`~optuna.samplers.BaseSampler` for further details. An Example: Implementing SimulatedAnnealingSampler -------------------------------------------------- For example, the following code defines a sampler based on `Simulated Annealing (SA) `__: """ import numpy as np import optuna class SimulatedAnnealingSampler(optuna.samplers.BaseSampler): def __init__(self, temperature=100): self._rng = np.random.RandomState() self._temperature = temperature # Current temperature. self._current_trial = None # Current state. def sample_relative(self, study, trial, search_space): if search_space == {}: return {} # Simulated Annealing algorithm. # 1. Calculate transition probability. prev_trial = study.trials[-2] if self._current_trial is None or prev_trial.value <= self._current_trial.value: probability = 1.0 else: probability = np.exp( (self._current_trial.value - prev_trial.value) / self._temperature ) self._temperature *= 0.9 # Decrease temperature. # 2. Transit the current state if the previous result is accepted. if self._rng.uniform(0, 1) < probability: self._current_trial = prev_trial # 3. Sample parameters from the neighborhood of the current point. # The sampled parameters will be used during the next execution of # the objective function passed to the study. params = {} for param_name, param_distribution in search_space.items(): if ( not isinstance(param_distribution, optuna.distributions.FloatDistribution) or (param_distribution.step is not None and param_distribution.step != 1) or param_distribution.log ): msg = ( "Only suggest_float() with `step` `None` or 1.0 and" " `log` `False` is supported" ) raise NotImplementedError(msg) current_value = self._current_trial.params[param_name] width = (param_distribution.high - param_distribution.low) * 0.1 neighbor_low = max(current_value - width, param_distribution.low) neighbor_high = min(current_value + width, param_distribution.high) params[param_name] = self._rng.uniform(neighbor_low, neighbor_high) return params # The rest are unrelated to SA algorithm: boilerplate def infer_relative_search_space(self, study, trial): return optuna.search_space.intersection_search_space(study.get_trials(deepcopy=False)) def sample_independent(self, study, trial, param_name, param_distribution): independent_sampler = optuna.samplers.RandomSampler() return independent_sampler.sample_independent(study, trial, param_name, param_distribution) ################################################################################################### # .. note:: # In favor of code simplicity, the above implementation doesn't support some features (e.g., maximization). # If you're interested in how to support those features, please see # `examples/samplers/simulated_annealing.py # `__. # # # You can use ``SimulatedAnnealingSampler`` in the same way as built-in samplers as follows: def objective(trial): x = trial.suggest_float("x", -10, 10) y = trial.suggest_float("y", -5, 5) return x**2 + y sampler = SimulatedAnnealingSampler() study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=100) best_trial = study.best_trial print("Best value: ", best_trial.value) print("Parameters that achieve the best value: ", best_trial.params) ################################################################################################### # In this optimization, the values of ``x`` and ``y`` parameters are sampled by using # ``SimulatedAnnealingSampler.sample_relative`` method. # # .. note:: # Strictly speaking, in the first trial, # ``SimulatedAnnealingSampler.sample_independent`` method is used to sample parameter values. # Because :func:`~optuna.search_space.intersection_search_space` used in # ``SimulatedAnnealingSampler.infer_relative_search_space`` cannot infer the search space # if there are no complete trials. optuna-4.1.0/tutorial/20_recipes/006_user_defined_pruner.py000066400000000000000000000127061471332314300236150ustar00rootroot00000000000000""" .. _user_defined_pruner: User-Defined Pruner =================== In :mod:`optuna.pruners`, we described how an objective function can optionally include calls to a pruning feature which allows Optuna to terminate an optimization trial when intermediate results do not appear promising. In this document, we describe how to implement your own pruner, i.e., a custom strategy for determining when to stop a trial. Overview of Pruning Interface ----------------------------- The :func:`~optuna.study.create_study` constructor takes, as an optional argument, a pruner inheriting from :class:`~optuna.pruners.BasePruner`. The pruner should implement the abstract method :func:`~optuna.pruners.BasePruner.prune`, which takes arguments for the associated :class:`~optuna.study.Study` and :class:`~optuna.trial.Trial` and returns a boolean value: :obj:`True` if the trial should be pruned and :obj:`False` otherwise. Using the Study and Trial objects, you can access all other trials through the :func:`~optuna.study.Study.get_trials` method and, and from a trial, its reported intermediate values through the :func:`~optuna.trial.FrozenTrial.intermediate_values` (a dictionary which maps an integer ``step`` to a float value). You can refer to the source code of the built-in Optuna pruners as templates for building your own. In this document, for illustration, we describe the construction and usage of a simple (but aggressive) pruner which prunes trials that are in last place compared to completed trials at the same step. .. note:: Please refer to the documentation of :class:`~optuna.pruners.BasePruner` or, for example, :class:`~optuna.pruners.ThresholdPruner` or :class:`~optuna.pruners.PercentilePruner` for more robust examples of pruner implementation, including error checking and complex pruner-internal logic. An Example: Implementing ``LastPlacePruner`` -------------------------------------------- We aim to optimize the ``loss`` and ``alpha`` hyperparameters for a stochastic gradient descent classifier (``SGDClassifier``) run on the sklearn iris dataset. We implement a pruner which terminates a trial at a certain step if it is in last place compared to completed trials at the same step. We begin considering pruning after a "warmup" of 1 training step and 5 completed trials. For demonstration purposes, we :func:`print` a diagnostic message from ``prune`` when it is about to return :obj:`True` (indicating pruning). It may be important to note that the ``SGDClassifier`` score, as it is evaluated on a holdout set, decreases with enough training steps due to overfitting. This means that a trial could be pruned even if it had a favorable (high) value on a previous training set. After pruning, Optuna will take the intermediate value last reported as the value of the trial. """ import numpy as np from sklearn.datasets import load_iris from sklearn.model_selection import train_test_split from sklearn.linear_model import SGDClassifier import optuna from optuna.pruners import BasePruner from optuna.trial._state import TrialState class LastPlacePruner(BasePruner): def __init__(self, warmup_steps, warmup_trials): self._warmup_steps = warmup_steps self._warmup_trials = warmup_trials def prune(self, study: "optuna.study.Study", trial: "optuna.trial.FrozenTrial") -> bool: # Get the latest score reported from this trial step = trial.last_step if step: # trial.last_step == None when no scores have been reported yet this_score = trial.intermediate_values[step] # Get scores from other trials in the study reported at the same step completed_trials = study.get_trials(deepcopy=False, states=(TrialState.COMPLETE,)) other_scores = [ t.intermediate_values[step] for t in completed_trials if step in t.intermediate_values ] other_scores = sorted(other_scores) # Prune if this trial at this step has a lower value than all completed trials # at the same step. Note that steps will begin numbering at 0 in the objective # function definition below. if step >= self._warmup_steps and len(other_scores) > self._warmup_trials: if this_score < other_scores[0]: print(f"prune() True: Trial {trial.number}, Step {step}, Score {this_score}") return True return False ################################################################################################### # Lastly, let's confirm the implementation is correct with the simple hyperparameter optimization. def objective(trial): iris = load_iris() classes = np.unique(iris.target) X_train, X_valid, y_train, y_valid = train_test_split( iris.data, iris.target, train_size=100, test_size=50, random_state=0 ) loss = trial.suggest_categorical("loss", ["hinge", "log_loss", "perceptron"]) alpha = trial.suggest_float("alpha", 0.00001, 0.001, log=True) clf = SGDClassifier(loss=loss, alpha=alpha, random_state=0) score = 0 for step in range(0, 5): clf.partial_fit(X_train, y_train, classes=classes) score = clf.score(X_valid, y_valid) trial.report(score, step) if trial.should_prune(): raise optuna.TrialPruned() return score pruner = LastPlacePruner(warmup_steps=1, warmup_trials=5) study = optuna.create_study(direction="maximize", pruner=pruner) study.optimize(objective, n_trials=50) optuna-4.1.0/tutorial/20_recipes/007_optuna_callback.py000066400000000000000000000047651471332314300227170ustar00rootroot00000000000000""" .. _optuna_callback: Callback for Study.optimize =========================== This tutorial showcases how to use & implement Optuna ``Callback`` for :func:`~optuna.study.Study.optimize`. ``Callback`` is called after every evaluation of ``objective``, and it takes :class:`~optuna.study.Study` and :class:`~optuna.trial.FrozenTrial` as arguments, and does some work. `MLflowCallback `__ is a great example. """ ################################################################################################### # Stop optimization after some trials are pruned in a row # ------------------------------------------------------- # # This example implements a stateful callback which stops the optimization # if a certain number of trials are pruned in a row. # The number of trials pruned in a row is specified by ``threshold``. import optuna class StopWhenTrialKeepBeingPrunedCallback: def __init__(self, threshold: int): self.threshold = threshold self._consequtive_pruned_count = 0 def __call__(self, study: optuna.study.Study, trial: optuna.trial.FrozenTrial) -> None: if trial.state == optuna.trial.TrialState.PRUNED: self._consequtive_pruned_count += 1 else: self._consequtive_pruned_count = 0 if self._consequtive_pruned_count >= self.threshold: study.stop() ################################################################################################### # This objective prunes all the trials except for the first 5 trials (``trial.number`` starts with 0). def objective(trial): if trial.number > 4: raise optuna.TrialPruned return trial.suggest_float("x", 0, 1) ################################################################################################### # Here, we set the threshold to ``2``: optimization finishes once two trials are pruned in a row. # So, we expect this study to stop after 7 trials. import logging import sys # Add stream handler of stdout to show the messages optuna.logging.get_logger("optuna").addHandler(logging.StreamHandler(sys.stdout)) study_stop_cb = StopWhenTrialKeepBeingPrunedCallback(2) study = optuna.create_study() study.optimize(objective, n_trials=10, callbacks=[study_stop_cb]) ################################################################################################### # As you can see in the log above, the study stopped after 7 trials as expected. optuna-4.1.0/tutorial/20_recipes/008_specify_params.py000066400000000000000000000122551471332314300225740ustar00rootroot00000000000000""" .. _specify_params: Specify Hyperparameters Manually ================================ It's natural that you have some specific sets of hyperparameters to try first such as initial learning rate values and the number of leaves. Also, it's possible that you've already tried those sets before having Optuna find better sets of hyperparameters. Optuna provides two APIs to support such cases: 1. Passing those sets of hyperparameters and let Optuna evaluate them - :func:`~optuna.study.Study.enqueue_trial` 2. Adding the results of those sets as completed ``Trial``\\s - :func:`~optuna.study.Study.add_trial` .. _enqueue_trial_tutorial: --------------------------------------------------------- First Scenario: Have Optuna evaluate your hyperparameters --------------------------------------------------------- In this scenario, let's assume you have some out-of-box sets of hyperparameters but have not evaluated them yet and decided to use Optuna to find better sets of hyperparameters. Optuna has :func:`optuna.study.Study.enqueue_trial` which lets you pass those sets of hyperparameters to Optuna and Optuna will evaluate them. This section walks you through how to use this lit API with `LightGBM `__. """ import lightgbm as lgb import numpy as np import sklearn.datasets import sklearn.metrics from sklearn.model_selection import train_test_split import optuna ################################################################################################### # Define the objective function. def objective(trial): data, target = sklearn.datasets.load_breast_cancer(return_X_y=True) train_x, valid_x, train_y, valid_y = train_test_split(data, target, test_size=0.25) dtrain = lgb.Dataset(train_x, label=train_y) dvalid = lgb.Dataset(valid_x, label=valid_y) param = { "objective": "binary", "metric": "auc", "verbosity": -1, "boosting_type": "gbdt", "bagging_fraction": min(trial.suggest_float("bagging_fraction", 0.4, 1.0 + 1e-12), 1), "bagging_freq": trial.suggest_int("bagging_freq", 0, 7), "min_child_samples": trial.suggest_int("min_child_samples", 5, 100), } gbm = lgb.train(param, dtrain, valid_sets=[dvalid]) preds = gbm.predict(valid_x) pred_labels = np.rint(preds) accuracy = sklearn.metrics.accuracy_score(valid_y, pred_labels) return accuracy ################################################################################################### # Then, construct ``Study`` for hyperparameter optimization. study = optuna.create_study(direction="maximize", pruner=optuna.pruners.MedianPruner()) ################################################################################################### # Here, we get Optuna evaluate some sets with larger ``"bagging_fraq"`` value and # the default values. study.enqueue_trial( { "bagging_fraction": 1.0, "bagging_freq": 0, "min_child_samples": 20, } ) study.enqueue_trial( { "bagging_fraction": 0.75, "bagging_freq": 5, "min_child_samples": 20, } ) import logging import sys # Add stream handler of stdout to show the messages to see Optuna works expectedly. optuna.logging.get_logger("optuna").addHandler(logging.StreamHandler(sys.stdout)) study.optimize(objective, n_trials=100, timeout=600) ################################################################################################### # .. _add_trial_tutorial: # # ---------------------------------------------------------------------- # Second scenario: Have Optuna utilize already evaluated hyperparameters # ---------------------------------------------------------------------- # # In this scenario, let's assume you have some out-of-box sets of hyperparameters and # you have already evaluated them but the results are not desirable so that you are thinking of # using Optuna. # # Optuna has :func:`optuna.study.Study.add_trial` which lets you register those results # to Optuna and then Optuna will sample hyperparameters taking them into account. # # In this section, the ``objective`` is the same as the first scenario. study = optuna.create_study(direction="maximize", pruner=optuna.pruners.MedianPruner()) study.add_trial( optuna.trial.create_trial( params={ "bagging_fraction": 1.0, "bagging_freq": 0, "min_child_samples": 20, }, distributions={ "bagging_fraction": optuna.distributions.FloatDistribution(0.4, 1.0 + 1e-12), "bagging_freq": optuna.distributions.IntDistribution(0, 7), "min_child_samples": optuna.distributions.IntDistribution(5, 100), }, value=0.94, ) ) study.add_trial( optuna.trial.create_trial( params={ "bagging_fraction": 0.75, "bagging_freq": 5, "min_child_samples": 20, }, distributions={ "bagging_fraction": optuna.distributions.FloatDistribution(0.4, 1.0 + 1e-12), "bagging_freq": optuna.distributions.IntDistribution(0, 7), "min_child_samples": optuna.distributions.IntDistribution(5, 100), }, value=0.95, ) ) study.optimize(objective, n_trials=100, timeout=600) optuna-4.1.0/tutorial/20_recipes/009_ask_and_tell.py000066400000000000000000000213301471332314300222020ustar00rootroot00000000000000""" .. _ask_and_tell: Ask-and-Tell Interface ======================= Optuna has an `Ask-and-Tell` interface, which provides a more flexible interface for hyperparameter optimization. This tutorial explains three use-cases when the ask-and-tell interface is beneficial: - :ref:`Apply-optuna-to-an-existing-optimization-problem-with-minimum-modifications` - :ref:`Define-and-Run` - :ref:`Batch-Optimization` .. _Apply-optuna-to-an-existing-optimization-problem-with-minimum-modifications: ---------------------------------------------------------------------------- Apply Optuna to an existing optimization problem with minimum modifications ---------------------------------------------------------------------------- Let's consider the traditional supervised classification problem; you aim to maximize the validation accuracy. To do so, you train `LogisticRegression` as a simple model. """ import numpy as np from sklearn.datasets import make_classification from sklearn.linear_model import LogisticRegression from sklearn.model_selection import train_test_split import optuna X, y = make_classification(n_features=10) X_train, X_test, y_train, y_test = train_test_split(X, y) C = 0.01 clf = LogisticRegression(C=C) clf.fit(X_train, y_train) val_accuracy = clf.score(X_test, y_test) # the objective ################################################################################################### # Then you try to optimize hyperparameters ``C`` and ``solver`` of the classifier by using optuna. # When you introduce optuna naively, you define an ``objective`` function # such that it takes ``trial`` and calls ``suggest_*`` methods of ``trial`` to sample the hyperparameters: def objective(trial): X, y = make_classification(n_features=10) X_train, X_test, y_train, y_test = train_test_split(X, y) C = trial.suggest_float("C", 1e-7, 10.0, log=True) solver = trial.suggest_categorical("solver", ("lbfgs", "saga")) clf = LogisticRegression(C=C, solver=solver) clf.fit(X_train, y_train) val_accuracy = clf.score(X_test, y_test) return val_accuracy study = optuna.create_study(direction="maximize") study.optimize(objective, n_trials=10) ################################################################################################### # This interface is not flexible enough. # For example, if ``objective`` requires additional arguments other than ``trial``, # you need to define a class as in # `How to define objective functions that have own arguments? <../../faq.html#how-to-define-objective-functions-that-have-own-arguments>`__. # The ask-and-tell interface provides a more flexible syntax to optimize hyperparameters. # The following example is equivalent to the previous code block. study = optuna.create_study(direction="maximize") n_trials = 10 for _ in range(n_trials): trial = study.ask() # `trial` is a `Trial` and not a `FrozenTrial`. C = trial.suggest_float("C", 1e-7, 10.0, log=True) solver = trial.suggest_categorical("solver", ("lbfgs", "saga")) clf = LogisticRegression(C=C, solver=solver) clf.fit(X_train, y_train) val_accuracy = clf.score(X_test, y_test) study.tell(trial, val_accuracy) # tell the pair of trial and objective value ################################################################################################### # The main difference is to use two methods: :func:`optuna.study.Study.ask` # and :func:`optuna.study.Study.tell`. # :func:`optuna.study.Study.ask` creates a trial that can sample hyperparameters, and # :func:`optuna.study.Study.tell` finishes the trial by passing ``trial`` and an objective value. # You can apply Optuna's hyperparameter optimization to your original code # without an ``objective`` function. # # If you want to make your optimization faster with a pruner, you need to explicitly pass the state of trial # to the argument of :func:`optuna.study.Study.tell` method as follows: # # .. code-block:: python # # import numpy as np # from sklearn.datasets import load_iris # from sklearn.linear_model import SGDClassifier # from sklearn.model_selection import train_test_split # # import optuna # # # X, y = load_iris(return_X_y=True) # X_train, X_valid, y_train, y_valid = train_test_split(X, y) # classes = np.unique(y) # n_train_iter = 100 # # # define study with hyperband pruner. # study = optuna.create_study( # direction="maximize", # pruner=optuna.pruners.HyperbandPruner( # min_resource=1, max_resource=n_train_iter, reduction_factor=3 # ), # ) # # for _ in range(20): # trial = study.ask() # # alpha = trial.suggest_float("alpha", 0.0, 1.0) # # clf = SGDClassifier(alpha=alpha) # pruned_trial = False # # for step in range(n_train_iter): # clf.partial_fit(X_train, y_train, classes=classes) # # intermediate_value = clf.score(X_valid, y_valid) # trial.report(intermediate_value, step) # # if trial.should_prune(): # pruned_trial = True # break # # if pruned_trial: # study.tell(trial, state=optuna.trial.TrialState.PRUNED) # tell the pruned state # else: # score = clf.score(X_valid, y_valid) # study.tell(trial, score) # tell objective value ################################################################################################### # .. note:: # # :func:`optuna.study.Study.tell` method can take a trial number rather than the trial object. # ``study.tell(trial.number, y)`` is equivalent to ``study.tell(trial, y)``. ################################################################################################### # .. _Define-and-Run: # # --------------- # Define-and-Run # --------------- # The ask-and-tell interface supports both `define-by-run` and `define-and-run` APIs. # This section shows the example of the `define-and-run` API # in addition to the define-by-run example above. # # Define distributions for the hyperparameters before calling the # :func:`optuna.study.Study.ask` method for define-and-run API. # For example, distributions = { "C": optuna.distributions.FloatDistribution(1e-7, 10.0, log=True), "solver": optuna.distributions.CategoricalDistribution(("lbfgs", "saga")), } ################################################################################################### # Pass ``distributions`` to :func:`optuna.study.Study.ask` method at each call. # The retuned ``trial`` contains the suggested hyperparameters. study = optuna.create_study(direction="maximize") n_trials = 10 for _ in range(n_trials): trial = study.ask(distributions) # pass the pre-defined distributions. # two hyperparameters are already sampled from the pre-defined distributions C = trial.params["C"] solver = trial.params["solver"] clf = LogisticRegression(C=C, solver=solver) clf.fit(X_train, y_train) val_accuracy = clf.score(X_test, y_test) study.tell(trial, val_accuracy) ################################################################################################### # .. _Batch-Optimization: # # ------------------- # Batch Optimization # ------------------- # The ask-and-tell interface enables us to optimize a batched objective for faster optimization. # For example, parallelizable evaluation, operation over vectors, etc. ################################################################################################### # The following objective takes batched hyperparameters ``xs`` and ``ys`` instead of a single # pair of hyperparameters ``x`` and ``y`` and calculates the objective over the full vectors. def batched_objective(xs: np.ndarray, ys: np.ndarray): return xs**2 + ys ################################################################################################### # In the following example, the number of pairs of hyperparameters in a batch is :math:`10`, # and ``batched_objective`` is evaluated three times. # Thus, the number of trials is :math:`30`. # Note that you need to store either ``trial_numbers`` or ``trial`` to call # :func:`optuna.study.Study.tell` method after the batched evaluations. batch_size = 10 study = optuna.create_study(sampler=optuna.samplers.CmaEsSampler()) for _ in range(3): # create batch trial_numbers = [] x_batch = [] y_batch = [] for _ in range(batch_size): trial = study.ask() trial_numbers.append(trial.number) x_batch.append(trial.suggest_float("x", -10, 10)) y_batch.append(trial.suggest_float("y", -10, 10)) # evaluate batched objective x_batch = np.array(x_batch) y_batch = np.array(y_batch) objectives = batched_objective(x_batch, y_batch) # finish all trials in the batch for trial_number, objective in zip(trial_numbers, objectives): study.tell(trial_number, objective) optuna-4.1.0/tutorial/20_recipes/010_reuse_best_trial.py000066400000000000000000000076541471332314300231220ustar00rootroot00000000000000""" .. _reuse_best_trial: Re-use the best trial ====================== In some cases, you may want to re-evaluate the objective function with the best hyperparameters again after the hyperparameter optimization. For example, - You have found good hyperparameters with Optuna and want to run a similar `objective` function using the best hyperparameters found so far to further analyze the results, or - You have optimized with Optuna using a partial dataset to reduce training time. After the hyperparameter tuning, you want to train the model using the whole dataset with the best hyperparameter values found. :class:`~optuna.study.Study.best_trial` provides an interface to re-evaluate the objective function with the current best hyperparameter values. This tutorial shows an example of how to re-run a different `objective` function with the current best values, like the first example above. Investigating the best model further ------------------------------------- Let's consider a classical supervised classification problem with Optuna as follows: """ from sklearn import metrics from sklearn.datasets import make_classification from sklearn.linear_model import LogisticRegression from sklearn.model_selection import train_test_split import optuna def objective(trial): X, y = make_classification(n_features=10, random_state=1) X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=1) C = trial.suggest_float("C", 1e-7, 10.0, log=True) clf = LogisticRegression(C=C) clf.fit(X_train, y_train) return clf.score(X_test, y_test) study = optuna.create_study(direction="maximize") study.optimize(objective, n_trials=10) print(study.best_trial.value) # Show the best value. ################################################################################################### # Suppose after the hyperparameter optimization, you want to calculate other evaluation metrics # such as recall, precision, and f1-score on the same dataset. # You can define another objective function that shares most of the ``objective`` # function to reproduce the model with the best hyperparameters. def detailed_objective(trial): # Use same code objective to reproduce the best model X, y = make_classification(n_features=10, random_state=1) X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=1) C = trial.suggest_float("C", 1e-7, 10.0, log=True) clf = LogisticRegression(C=C) clf.fit(X_train, y_train) # calculate more evaluation metrics pred = clf.predict(X_test) acc = metrics.accuracy_score(pred, y_test) recall = metrics.recall_score(pred, y_test) precision = metrics.precision_score(pred, y_test) f1 = metrics.f1_score(pred, y_test) return acc, f1, recall, precision ################################################################################################### # Pass ``study.best_trial`` as the argument of ``detailed_objective``. detailed_objective(study.best_trial) # calculate acc, f1, recall, and precision ################################################################################################### # The difference between :class:`~optuna.study.Study.best_trial` and ordinal trials # ---------------------------------------------------------------------------------- # # This uses :class:`~optuna.study.Study.best_trial`, which returns the `best_trial` as a # :class:`~optuna.trial.FrozenTrial`. # The :class:`~optuna.trial.FrozenTrial` is different from an active trial # and behaves differently from :class:`~optuna.trial.Trial` in some situations. # For example, pruning does not work because :class:`~optuna.trial.FrozenTrial.should_prune` # always returns ``False``. # # .. note:: # For multi-objective optimization as demonstrated by :ref:`multi_objective`, # :attr:`~optuna.study.Study.best_trials` returns a list of :class:`~optuna.trial.FrozenTrial` # on Pareto front. So we can re-use each trial in the list by the similar way above. optuna-4.1.0/tutorial/20_recipes/011_journal_storage.py000066400000000000000000000031631471332314300227550ustar00rootroot00000000000000""" .. _journal_storage: (File-based) Journal Storage ============================ Optuna provides :class:`~optuna.storages.JournalStorage`. With this feature, you can easily run a distributed optimization over network using NFS as the shared storage, without need for setting up RDB or Redis. """ import logging import sys import optuna # Add stream handler of stdout to show the messages optuna.logging.get_logger("optuna").addHandler(logging.StreamHandler(sys.stdout)) study_name = "example-study" # Unique identifier of the study. file_path = "./optuna_journal_storage.log" storage = optuna.storages.JournalStorage( optuna.storages.journal.JournalFileBackend(file_path), # NFS path for distributed optimization ) study = optuna.create_study(study_name=study_name, storage=storage) def objective(trial): x = trial.suggest_float("x", -10, 10) return (x - 2) ** 2 study.optimize(objective, n_trials=3) ################################################################################################### # Although the optimization in this example is too short to run in parallel, you can extend this # example to write a optimization script which can be run in parallel. # ################################################################################################### # .. note:: # In a Windows environment, an error message "A required privilege is not held by the client" # may appear. In this case, you can solve the problem with creating storage by specifying # :class:`~optuna.storages.journal.JournalFileOpenLock`. See the reference of # :class:`~optuna.storages.JournalStorage` for any details. optuna-4.1.0/tutorial/20_recipes/012_artifact_tutorial.py000066400000000000000000000517471471332314300233130ustar00rootroot00000000000000""" .. _artifact_tutorial: Optuna Artifacts Tutorial ========================= .. contents:: Table of Contents :depth: 2 The artifact module of Optuna is a module designed for saving comparatively large attributes on a trial-by-trial basis in forms such as files. Introduced from Optuna v3.3, this module finds a broad range of applications, such as utilizing snapshots of large size models for hyperparameter tuning, optimizing massive chemical structures, and even human-in-the-loop optimization employing images or sounds. Use of Optuna's artifact module allows you to handle data that would be too large to store in a database. Furthermore, by integrating with `optuna-dashboard `__, saved artifacts can be automatically visualized with the web UI, which significantly reduces the effort of experiment management. TL;DR ----- - The artifact module provides a simple way to save and use large data associated with trials. - Saved artifacts can be visualized just by accessing the web page using optuna-dashboard, and downloading is also easy. - Thanks to the abstraction of the artifact module, the backend (file system, AWS S3) can be easily switched. - As the artifact module is tightly linked with Optuna, experiment management can be completed with the Optuna ecosystem alone, simplifying the code base. Concepts -------- .. list-table:: :header-rows: 1 * - Fig 1. Concepts of the "artifact". * - .. image:: https://github.com/optuna/optuna/assets/38826298/112e0b75-9d22-474b-85ea-9f3e0d75fa8d An "artifact" is associated with an Optuna trial. In Optuna, the objective function is evaluated sequentially to search for the maximum (or minimum) value. Each evaluation of the sequentially repeated objective function is called a trial. Normally, trials and their associated attributes are saved via storage objects to files or RDBs, etc. For experiment management, you can also save and use :attr:`optuna.trial.Trial.user_attrs` for each trial. However, these attributes are assumed to be integers, short strings, or other small data, which are not suitable for storing large data. With Optuna's artifact module, users can save large data (such as model snapshots, chemical structures, image and audio data, etc.) for each trial. Also, while this tutorial does not touch upon it, it's possible to manage artifacts associated not only with trials but also with studies. Please refer to the `official documentation `__ if you are interested in. Situations where artifacts are useful ------------------------------------- Artifacts are useful when you want to save data that is too large to be stored in RDB for each trial. For example, the artifact module would be handy in situations like the following: - Saving snapshots of machine learning models: Suppose you are tuning hyperparameters for a large-scale machine learning model like an LLM. The model is very large, and each round of learning (which corresponds to one trial in Optuna) takes time. To prepare for unexpected incidents during training (such as blackouts at the data center or a preemption of computation jobs by the scheduler), you may want to save snapshots of the model in the middle of training for each trial. These snapshots often tend to be large and are more suitable to be saved as some kinds of files than to be stored in RDB. In such cases, the artifact module is useful. - Optimizing chemical structures: Suppose you are formulating and exploring a problem of finding stable chemical structures as a black-box optimization problem. Evaluating one chemical structure corresponds to one trial in Optuna, and that chemical structure is a complex and large one. It is not appropriate to store such chemical structure data in RDB. It is conceivable to save the chemical structure data in a specific file format, and in such a case, the artifact module is useful. - Human-in-the-loop optimization of images: Suppose you are optimizing prompts for a generative model that outputs images. You sample the prompts using Optuna, output images using the generative model, and let humans rate the images for a Human-in-the-loop optimization process. Since the output images are large data, it is not appropriate to use RDB to store them, and in such cases, using the artifact module is well suited. How Trials and Artifacts are Recorded ------------------------------------- As explained so far, the artifact module is useful when you want to save large data for each trial. In this section, we explain how artifacts work in the following two scenarios: first when SQLite + local file system-based artifact backend is used (suitable when the entire optimization cycle is completed locally), and second when MySQL + AWS S3-based artifact backend is used (suitable when you want to keep the data in a remote location). Scenario 1: SQLite + file system-based artifact store ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. list-table:: :header-rows: 1 * - Fig 2. SQLite + file system-based artifact store. * - .. image:: https://github.com/optuna/optuna/assets/38826298/d41d042e-6b78-4615-bf96-05f73a47e9ea First, we explain a simple case where the optimization is completed locally. Normally, Optuna's optimization history is persisted into some kind of a database via storage objects. Here, let's consider a method using SQLite, a lightweight RDB management system, as the backend. With SQLite, data is stored in a single file (e.g., ``./example.db``). The optimization history comprises what parameters were sampled in each trial, what the evaluation values for those parameters were, when each trial started and ended, etc. This file is in the SQLite format, and it is not suitable for storing large data. Writing large data entries may cause performance degradation. Note that SQLite is not suitable for distributed parallel optimization. If you want to perform that, please use MySQL as we will explain later, or :class:`~optuna.storages.JournalStorage`. So, let's use the artifact module to save large data in a different format. Suppose the data is generated for each trial and you want to save it in some format (e.g., png format if it's an image). The specific destination for saving the artifacts can be any directory on the local file system (e.g., the ``./artifacts`` directory). When defining the objective function, you only need to save and reference the data using the artifact module. The simple pseudocode for the above case would look something like this: .. code-block:: python import os import optuna from optuna.artifacts import FileSystemArtifactStore from optuna.artifacts import upload_artifact from optuna.artifacts import download_artifact base_path = "./artifacts" os.makedirs(base_path, exist_ok=True) artifact_store = FileSystemArtifactStore(base_path=base_path) def objective(trial: optuna.Trial) -> float: ... = trial.suggest_float("x", -10, 10) # Creating and writing an artifact. file_path = generate_example(...) # This function returns some kind of file. artifact_id = upload_artifact( artifact_store=artifact_store, file_path=file_path, study_or_trial=trial, ) # The return value is the artifact ID. trial.set_user_attr( "artifact_id", artifact_id ) # Save the ID in RDB so that it can be referenced later. return ... study = optuna.create_study(study_name="test_study", storage="sqlite:///example.db") study.optimize(objective, n_trials=100) # Downloading artifacts associated with the best trial. best_artifact_id = study.best_trial.user_attrs.get("artifact_id") download_file_path = ... # Set the path to save the downloaded artifact. download_artifact( artifact_store=artifact_store, file_path=download_file_path, artifact_id=best_artifact_id ) with open(download_file_path, "rb") as f: content = f.read().decode("utf-8") print(content) Scenario 2: Remote MySQL RDB server + AWS S3 artifact store ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. list-table:: :header-rows: 1 * - Fig 3. Remote MySQL RDB server + AWS S3 artifact store. * - .. image:: https://github.com/optuna/optuna/assets/38826298/067efc85-1fad-4b46-a2be-626c64439d7b Next, we explain the case where data is read and written remotely. As the scale of optimization increases, it becomes difficult to complete all calculations locally. Optuna's storage objects can persist data remotely by specifying a URL, enabling distributed optimization. Here, we will use MySQL as a remote relational database server. MySQL is an open-source relational database management system and a well-known software used for various purposes. For using MySQL with Optuna, the `tutorial `__ can be a good reference. However, it is also not appropriate to read and write large data in a relational database like MySQL. In Optuna, it is common to use the artifact module when you want to read and write such data for each trial. Unlike Scenario 1, we distribute the optimization across computation nodes, so local file system-based backends will not work. Instead, we will use AWS S3, an online cloud storage service, and Boto3, a framework for interacting with it from Python. As of v3.3, Optuna has a built-in artifact store with this Boto3 backend. The flow of data is shown in Figure 3. The information calculated in each trial, which corresponds to the optimization history (excluding artifact information), is written to the MySQL server. On the other hand, the artifact information is written to AWS S3. All workers conducting distributed optimization can read and write in parallel to each, and issues such as race conditions are automatically resolved by Optuna's storage module and artifact module. As a result, although the actual data location changes between artifact information and non-artifact information (the former is in AWS S3, the latter is in the MySQL RDB), users can read and write data transparently. Translating the above process into simple pseudocode would look something like this: .. code-block:: python import os import boto3 from botocore.config import Config import optuna from optuna.artifact import upload_artifact from optuna.artifact import download_artifact from optuna.artifact.boto3 import Boto3ArtifactStore artifact_store = Boto3ArtifactStore( client=boto3.client( "s3", aws_access_key_id=os.environ[ "AWS_ACCESS_KEY_ID" ], # Assume that these environment variables are set up properly. The same applies below. aws_secret_access_key=os.environ["AWS_SECRET_ACCESS_KEY"], endpoint_url=os.environ["S3_ENDPOINT"], config=Config(connect_timeout=30, read_timeout=30), ), bucket_name="example_bucket", ) def objective(trial: optuna.Trial) -> float: ... = trial.suggest_float("x", -10, 10) # Creating and writing an artifact. file_path = generate_example(...) # This function returns some kind of file. artifact_id = upload_artifact( artifact_store=artifact_store, file_path=file_path, study_or_trial=trial, ) # The return value is the artifact ID. trial.set_user_attr( "artifact_id", artifact_id ) # Save the ID in RDB so that it can be referenced later. return ... study = optuna.create_study( study_name="test_study", storage="mysql://USER:PASS@localhost:3306/test", # Set the appropriate URL. ) study.optimize(objective, n_trials=100) # Downloading artifacts associated with the best trial. best_artifact_id = study.best_trial.user_attrs.get("artifact_id") download_file_path = ... # Set the path to save the downloaded artifact. download_artifact( artifact_store=artifact_store, file_path=download_file_path, artifact_id=best_artifact_id ) with open(download_file_path, "rb") as f: content = f.read().decode("utf-8") print(content) Example: Optimization of Chemical Structures -------------------------------------------- In this section, we introduce an example of optimizing chemical structure using Optuna by utilizing the artifact module. We will target relatively small structures, but the approach remains the same even for complex structures. Consider the process of a specific molecule adsorbing onto another substance. In this process, the ease of adsorption reaction changes depending on the position of the adsorbing molecule to the substance it is adsorbed onto. The ease of adsorption reaction can be evaluated by the adsorption energy (the difference between the energy of the system after adsorption and before). By formulating the problem as a minimization problem of an objective function that takes the positional relationship of the adsorbing molecule as input and outputs the adsorption energy, this problem is solved as a black-box optimization problem. First, let's import the necessary modules and define some helper functions. You need to install the ASE library for handling chemical structures in addition to Optuna, so please install it with `pip install ase`. """ from __future__ import annotations import io import logging import os import sys import tempfile from ase import Atoms from ase.build import bulk, fcc111, molecule, add_adsorbate from ase.calculators.emt import EMT from ase.io import write, read from ase.optimize import LBFGS import numpy as np from optuna.artifacts import FileSystemArtifactStore from optuna.artifacts import upload_artifact from optuna.artifacts import download_artifact from optuna.logging import get_logger from optuna import create_study from optuna import Trial # Add stream handler of stdout to show the messages get_logger("optuna").addHandler(logging.StreamHandler(sys.stdout)) def get_opt_energy(atoms: Atoms, fmax: float = 0.001) -> float: calculator = EMT() atoms.set_calculator(calculator) opt = LBFGS(atoms, logfile=None) opt.run(fmax=fmax) return atoms.get_total_energy() def create_slab() -> tuple[Atoms, float]: calculator = EMT() bulk_atoms = bulk("Pt", cubic=True) bulk_atoms.calc = calculator a = np.mean(np.diag(bulk_atoms.cell)) slab = fcc111("Pt", a=a, size=(4, 4, 4), vacuum=40.0, periodic=True) slab.calc = calculator E_slab = get_opt_energy(slab, fmax=1e-4) return slab, E_slab def create_mol() -> tuple[Atoms, float]: calculator = EMT() mol = molecule("CO") mol.calc = calculator E_mol = get_opt_energy(mol, fmax=1e-4) return mol, E_mol def atoms_to_json(atoms: Atoms) -> str: f = io.StringIO() write(f, atoms, format="json") return f.getvalue() def json_to_atoms(atoms_str: str) -> Atoms: return read(io.StringIO(atoms_str), format="json") def file_to_atoms(file_path: str) -> Atoms: return read(file_path, format="json") ################################################################################################### # Each function is as follows. # # - ``get_opt_energy``: Takes a chemical structure, transitions it to a locally stable structure, and returns the energy after the transition. # - ``create_slab``: Constructs the substance being adsorbed. # - ``create_mol``: Constructs the molecule being adsorbed. # - ``atoms_to_json``: Converts the chemical structure to a string. # - ``json_to_atoms``: Converts the string to a chemical structure. # - ``file_to_atoms``: Reads the string from a file and converts it to a chemical structure. # # Using these functions, the code to search for adsorption structures using Optuna is as follows. The objective function is defined # as class ``Objective`` in order to carry the artifact store. In its ``__call__`` method, it retrieves the substance being adsorbed # (``slab``) and the molecule being adsorbed (``mol``), then after sampling their positional relationship using Optuna (multiple # ``trial.suggest_xxx`` methods), it triggers an adsorption reaction with the ``add_adsorbate`` function, transitions to a locally # stable structure, then saves the structure in the artifact store and returns the adsorption energy. # # The ``main`` function contains the code to create a ``Study`` and execute optimization. When creating a ``Study``, a storage is # specified using SQLite, and a back end using the local file system is used for the artifact store. In other words, it corresponds # to Scenario 1 explained in the previous section. After performing 100 trials of optimization, it displays the information for the # best trial, and finally saves the chemical structure as ``best_atoms.png``. The obtained ``best_atoms.png``` is shown in Figure 4. class Objective: def __init__(self, artifact_store: FileSystemArtifactStore) -> None: self._artifact_store = artifact_store def __call__(self, trial: Trial) -> float: slab = json_to_atoms(trial.study.user_attrs["slab"]) E_slab = trial.study.user_attrs["E_slab"] mol = json_to_atoms(trial.study.user_attrs["mol"]) E_mol = trial.study.user_attrs["E_mol"] phi = 180.0 * trial.suggest_float("phi", -1, 1) theta = np.arccos(trial.suggest_float("theta", -1, 1)) * 180.0 / np.pi psi = 180 * trial.suggest_float("psi", -1, 1) x_pos = trial.suggest_float("x_pos", 0, 0.5) y_pos = trial.suggest_float("y_pos", 0, 0.5) z_hig = trial.suggest_float("z_hig", 1, 5) xy_position = np.matmul([x_pos, y_pos, 0], slab.cell)[:2] mol.euler_rotate(phi=phi, theta=theta, psi=psi) add_adsorbate(slab, mol, z_hig, xy_position) E_slab_mol = get_opt_energy(slab, fmax=1e-2) write(f"./tmp/{trial.number}.json", slab, format="json") artifact_id = upload_artifact( artifact_store=self._artifact_store, file_path=f"./tmp/{trial.number}.json", study_or_trial=trial, ) trial.set_user_attr("structure", artifact_id) return E_slab_mol - E_slab - E_mol def main(): study = create_study( study_name="test_study", storage="sqlite:///example.db", load_if_exists=True, ) slab, E_slab = create_slab() study.set_user_attr("slab", atoms_to_json(slab)) study.set_user_attr("E_slab", E_slab) mol, E_mol = create_mol() study.set_user_attr("mol", atoms_to_json(mol)) study.set_user_attr("E_mol", E_mol) os.makedirs("./tmp", exist_ok=True) base_path = "./artifacts" os.makedirs(base_path, exist_ok=True) artifact_store = FileSystemArtifactStore(base_path=base_path) study.optimize(Objective(artifact_store), n_trials=3) print( f"Best trial is #{study.best_trial.number}\n" f" Its adsorption energy is {study.best_value}\n" f" Its adsorption position is\n" f" phi : {study.best_params['phi']}\n" f" theta: {study.best_params['theta']}\n" f" psi. : {study.best_params['psi']}\n" f" x_pos: {study.best_params['x_pos']}\n" f" y_pos: {study.best_params['y_pos']}\n" f" z_hig: {study.best_params['z_hig']}" ) best_artifact_id = study.best_trial.user_attrs["structure"] with tempfile.TemporaryDirectory() as tmpdir_name: download_file_path = os.path.join(tmpdir_name, f"{best_artifact_id}.json") download_artifact( artifact_store=artifact_store, file_path=download_file_path, artifact_id=best_artifact_id, ) best_atoms = file_to_atoms(download_file_path) print(best_atoms) write("best_atoms.png", best_atoms, rotation=("315x,0y,0z")) if __name__ == "__main__": main() ################################################################################################### # .. list-table:: # :header-rows: 1 # # * - Fig 4. The chemical structure obtained by the above code. # * - .. image:: https://github.com/optuna/optuna/assets/38826298/c6bd62fd-599a-424e-8c2c-ca88af85cc63 # # As shown above, it is convenient to use the artifact module when performing the optimization of chemical structures with Optuna. # In the case of small structures or fewer trial numbers, it's fine to convert it to a string and save it directly in the RDB. # However, when dealing with complex structures or performing large-scale searches, it's better to save it outside the RDB to # avoid overloading it, such as in an external file system or AWS S3. # # Conclusion # ---------- # # The artifact module is a useful feature when you want to save relatively large data for each trial. It can be used for various # purposes such as saving snapshots of machine learning models, optimizing chemical structures, and human-in-the-loop optimization # of images and sounds. It's a powerful assistant for black-box optimization with Optuna. Also, if there are ways to use it that # we, the Optuna committers, haven't noticed, please let us know on GitHub discussions. Have a great optimization life with Optuna! optuna-4.1.0/tutorial/20_recipes/013_wilcoxon_pruner.py000066400000000000000000000320701471332314300230150ustar00rootroot00000000000000""" .. _wilcoxon_pruner: Early-stopping independent evaluations by Wilcoxon pruner ============================================================ This tutorial showcases Optuna's :class:`~optuna.pruners.WilcoxonPruner`. This pruner is effective for objective functions that averages multiple evaluations. We solve `Traveling Salesman Problem (TSP) `__ by `Simulated Annealing (SA) `__. Overview: Solving Traveling Salesman Problem with Simulated Annealing ---------------------------------------------------------------------------- Traveling Salesman Problem (TSP) is a classic problem in combinatorial optimization that involves finding the shortest possible route for a salesman who needs to visit a set of cities, each exactly once, and return to the starting city. TSP has been extensively studied in fields such as mathematics, computer science, and operations research, and has numerous practical applications in logistics, manufacturing, and DNA sequencing, among others. The problem is classified as NP-hard, so approximation algorithms or heuristic methods are commonly employed for larger instances. One simple heuristic method applicable to TSP is simulated annealing (SA). SA starts with an initial solution (it can be constructed by a simpler heuristic like greedy method), and it randomly checks the neighborhood (defined later) of the solution. If a neighbor is better, the solution is updated to the neighbor. If the neighbor is worse, SA still updates the solution to the neighbor with probability :math:`e^{-\\Delta c / T}`, where :math:`\\Delta c (> 0)` is the difference of the cost (sum of the distance) between the new solution and the old one and :math:`T` is a parameter called "temperature". The temperature controls how much worsening of the solution is tolerated to escape from the local minimum (high means more tolerant). If the temperature is too low, SA will quickly fall into a local minimum; if the temperature is too high, SA will be like a random walk and the optimization will be inefficient. Typically, we set a "temperature schedule" that starts from a high temperature and gradually decreases to zero. There are several ways to define neighborhood for TSP, but we use a simple neighborhood called `2-opt `__. 2-opt neighbor chooses a path in the current solution and reverses the visiting order in the path. For example, if the initial solution is `a→b→c→d→e→a`, `a→d→c→b→e→a` is a 2-opt neighbor (the path from `b` to `d` is reversed). This neighborhood is good because computing the difference of the cost can be done in constant time (we only need to care about the start and the end of the chosen path). Main Tutorial: Tuning SA Parameters for TSP ==================================================== First, let's import some packages and define the parameters setting of SA and the cost function of TSP. """ # NOQA from dataclasses import dataclass import math import numpy as np import optuna import plotly.graph_objects as go from numpy.linalg import norm @dataclass class SAOptions: max_iter: int = 10000 T0: float = 1.0 alpha: float = 2.0 patience: int = 50 def tsp_cost(vertices: np.ndarray, idxs: np.ndarray) -> float: return norm(vertices[idxs] - vertices[np.roll(idxs, 1)], axis=-1).sum() ################################################################################################### # Greedy solution for initial guess. def tsp_greedy(vertices: np.ndarray) -> np.ndarray: idxs = [0] for _ in range(len(vertices) - 1): dists_from_last = norm(vertices[idxs[-1], None] - vertices, axis=-1) dists_from_last[idxs] = np.inf idxs.append(np.argmin(dists_from_last)) return np.array(idxs) ################################################################################################### # .. note:: # For simplicity of implementation, we use SA with the 2-opt neighborhood to solve TSP, # but note that this is far from the "best" way to solve TSP. There are significantly more # advanced methods than this method. ################################################################################################### # The implementation of SA with 2-opt neighborhood is following. def tsp_simulated_annealing(vertices: np.ndarray, options: SAOptions) -> np.ndarray: def temperature(t: float): assert 0.0 <= t and t <= 1.0 return options.T0 * (1 - t) ** options.alpha N = len(vertices) idxs = tsp_greedy(vertices) cost = tsp_cost(vertices, idxs) best_idxs = idxs.copy() best_cost = cost remaining_patience = options.patience for iter in range(options.max_iter): i = np.random.randint(0, N) j = (i + 2 + np.random.randint(0, N - 3)) % N i, j = min(i, j), max(i, j) # Reverse the order of vertices between range [i+1, j]. # cost difference by 2-opt reversal delta_cost = ( -norm(vertices[idxs[(i + 1) % N]] - vertices[idxs[i]]) - norm(vertices[idxs[j]] - vertices[idxs[(j + 1) % N]]) + norm(vertices[idxs[i]] - vertices[idxs[j]]) + norm(vertices[idxs[(i + 1) % N]] - vertices[idxs[(j + 1) % N]]) ) temp = temperature(iter / options.max_iter) if delta_cost <= 0.0 or np.random.random() < math.exp(-delta_cost / temp): # accept the 2-opt reversal cost += delta_cost idxs[i + 1 : j + 1] = idxs[i + 1 : j + 1][::-1] if cost < best_cost: best_idxs[:] = idxs best_cost = cost remaining_patience = options.patience if cost > best_cost: # If the best solution is not updated for "patience" iteratoins, # restart from the best solution. remaining_patience -= 1 if remaining_patience == 0: idxs[:] = best_idxs cost = best_cost remaining_patience = options.patience return best_idxs ################################################################################################### # We make a random dataset of TSP. def make_dataset(num_vertex: int, num_problem: int, seed: int = 0) -> np.ndarray: rng = np.random.default_rng(seed=seed) return rng.random((num_problem, num_vertex, 2)) dataset = make_dataset( num_vertex=100, num_problem=50, ) N_TRIALS = 50 ################################################################################################### # We set a very small number of SA iterations for demonstration purpose. # In practice, you should set a larger number of iterations (e.g., 1000000). N_SA_ITER = 10000 ################################################################################################### # We counts the number of evaluation to know how many problems is pruned. num_evaluation = 0 ################################################################################################### # In this tutorial, we optimize three parameters: ``T0``, ``alpha``, and ``patience``. # # ``T0`` and ``alpha`` defining the temperature schedule # --------------------------------------------------------------------------------------- # # In simulated annealing, it is important to determine a good temperature scheduling, but # there is no "silver schedule" that is good for all problems, so we must tune the schedule # for this problem. # This code parametrizes the temperature as a monomial function ``T0 * (1 - t) ** alpha``, where # `t` progresses from 0 to 1. We try to optimize the two parameters ``T0`` and ``alpha``. # # ``patience`` # ----------------------------- # # This parameter specifies a threshold of how many iterations we allow the annealing process # continue without updating the best value. Practically, simulated annealing often drives # the solution far away from the current best solution, and rolling back to the best solution # periodically often improves optimization efficiency a lot. However, if the rollback happens # too often, the optimization may get stuck in a local optimum, so we must tune the threshold # to a sensible amount. # # .. note:: # Some samplers, including the default ``TPESampler``, currently cannot utilize the # information of pruned trials effectively (especially when the last intermediate value # is not the best approximation to the final objective function). # As a workaround for this issue, you can return an estimation of the final value # (e.g., the average of all evaluated values) when ``trial.should_prune()`` returns ``True``, # instead of `raise optuna.TrialPruned()`. # This will improve the sampler performance. ################################################################################################### # We define the objective function to be optimized as follows. # We early stop the evaluation by using the pruner. def objective(trial: optuna.Trial) -> float: global num_evaluation options = SAOptions( max_iter=N_SA_ITER, T0=trial.suggest_float("T0", 0.01, 10.0, log=True), alpha=trial.suggest_float("alpha", 1.0, 10.0, log=True), patience=trial.suggest_int("patience", 10, 1000, log=True), ) results = [] # For best results, shuffle the evaluation order in each trial. instance_ids = np.random.permutation(len(dataset)) for instance_id in instance_ids: num_evaluation += 1 result_idxs = tsp_simulated_annealing(vertices=dataset[instance_id], options=options) result_cost = tsp_cost(dataset[instance_id], result_idxs) results.append(result_cost) trial.report(result_cost, instance_id) if trial.should_prune(): # Return the current predicted value instead of raising `TrialPruned`. # This is a workaround to tell the Optuna about the evaluation # results in pruned trials. return sum(results) / len(results) return sum(results) / len(results) ################################################################################################### # We use ``TPESampler`` with ``WilcoxonPruner``. np.random.seed(0) sampler = optuna.samplers.TPESampler(seed=1) pruner = optuna.pruners.WilcoxonPruner(p_threshold=0.1) study = optuna.create_study(direction="minimize", sampler=sampler, pruner=pruner) study.enqueue_trial({"T0": 1.0, "alpha": 2.0, "patience": 50}) # default params study.optimize(objective, n_trials=N_TRIALS) ################################################################################################### # We can show the optimization results as: print(f"The number of trials: {len(study.trials)}") print(f"Best value: {study.best_value} (params: {study.best_params})") print(f"Number of evaluation: {num_evaluation} / {len(dataset) * N_TRIALS}") ################################################################################################### # Visualize the optimization history. # Note that this plot shows both completed and pruned trials in same ways. optuna.visualization.plot_optimization_history(study) ################################################################################################### # Visualize the number of evaluations in each trial. x_values = [x for x in range(len(study.trials)) if x != study.best_trial.number] y_values = [ len(t.intermediate_values) for t in study.trials if t.number != study.best_trial.number ] best_trial_y = [len(study.best_trial.intermediate_values)] best_trial_x = [study.best_trial.number] fig = go.Figure() fig.add_trace(go.Bar(x=x_values, y=y_values, name="Evaluations")) fig.add_trace(go.Bar(x=best_trial_x, y=best_trial_y, name="Best Trial", marker_color="red")) fig.update_layout( title="Number of evaluations in each trial", xaxis_title="Trial number", yaxis_title="Number of evaluations before pruned", ) fig ################################################################################################### # Visualize the greedy solution (used by initial guess) of a TSP problem. d = dataset[0] result_idxs = tsp_greedy(d) result_idxs = np.append(result_idxs, result_idxs[0]) fig = go.Figure() fig.add_trace(go.Scatter(x=d[result_idxs, 0], y=d[result_idxs, 1], mode="lines+markers")) fig.update_layout( title=f"greedy solution (initial guess), cost: {tsp_cost(d, result_idxs):.3f}", xaxis=dict(scaleanchor="y", scaleratio=1), ) fig ################################################################################################### # Visualize the solution found by ``tsp_simulated_annealing`` of the same TSP problem. params = study.best_params options = SAOptions( max_iter=N_SA_ITER, patience=params["patience"], T0=params["T0"], alpha=params["alpha"], ) result_idxs = tsp_simulated_annealing(d, options) result_idxs = np.append(result_idxs, result_idxs[0]) fig = go.Figure() fig.add_trace(go.Scatter(x=d[result_idxs, 0], y=d[result_idxs, 1], mode="lines+markers")) fig.update_layout( title=f"n_iter: {options.max_iter}, cost: {tsp_cost(d, result_idxs):.3f}", xaxis=dict(scaleanchor="y", scaleratio=1), ) fig optuna-4.1.0/tutorial/20_recipes/README.rst000066400000000000000000000001441471332314300203070ustar00rootroot00000000000000.. _recipes: Recipes ------- Showcases the recipes that might help you using Optuna with comfort. optuna-4.1.0/tutorial/README.rst000066400000000000000000000006131471332314300163550ustar00rootroot00000000000000Tutorial ======== If you are new to Optuna or want a general introduction, we highly recommend the below video. .. raw:: html