Skip to main content

Browse the archive

https://forge.softwareheritage.org/source/swh-scheduler.git
13 March 2024, 03:48:39 UTC
Revision a76bb02f0e94bf1c61124c9133d48a03f3d1a05f authored by David Douard on 15 April 2022, 16:08:49 UTC, committed by David Douard on 20 April 2022, 14:34:23 UTC 
Make scheduling policy used in schedule_recurrent configurable
 
Add support for a configuration option "scheduling_policy" in the config
file loaded by the 'swh scheduler schedule-recurrent' command. This
config entry allows to specify the scheduling policies used by the
schedule-recurrent tool, instead of having them hardcoded in the source
code.

A visit type policy config entry should have at least a 'weight' value
for each policy.

Default values are unchanged.

Eg.:

  scheduling_policy:
    git:
      - policy: already_visited_order_by_lag
        weight: 55
        tablesample: 0.5
      - policy: never_visited_oldest_update_first
        weight: 45
        tablesample: 0.5

Note: there may not be configuration entries for all visit types, but if
      a visit type policy is configured, the config entry should be complete
      (in other words, the merging of the configuration with the default
      values is only done at first config level).
1 parent 5302efd
Raw File
Tip revision: a76bb02f0e94bf1c61124c9133d48a03f3d1a05f authored by David Douard on 15 April 2022, 16:08:49 UTC
Make scheduling policy used in schedule_recurrent configurable
Tip revision: a76bb02
setup.py 
#!/usr/bin/env python3
# Copyright (C) 2015-2019  The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information

from io import open
from os import path

from setuptools import find_packages, setup

here = path.abspath(path.dirname(__file__))

# Get the long description from the README file
with open(path.join(here, "README.md"), encoding="utf-8") as f:
    long_description = f.read()


def parse_requirements(*names):
    requirements = []
    for name in names:
        if name:
            reqf = "requirements-%s.txt" % name
        else:
            reqf = "requirements.txt"

        if not path.exists(reqf):
            return requirements

        with open(reqf) as f:
            for line in f.readlines():
                line = line.strip()
                if not line or line.startswith("#"):
                    continue
                requirements.append(line)
    return requirements


setup(
    name="swh.scheduler",
    description="Software Heritage Scheduler",
    long_description=long_description,
    long_description_content_type="text/markdown",
    python_requires=">=3.7",
    author="Software Heritage developers",
    author_email="swh-devel@inria.fr",
    url="https://forge.softwareheritage.org/diffusion/DSCH/",
    packages=find_packages(),
    setup_requires=["setuptools-scm"],
    use_scm_version=True,
    install_requires=parse_requirements(None, "swh"),
    extras_require={
        "testing": parse_requirements("test", "journal", "simulator"),
        "journal": parse_requirements("journal"),
        "simulator": parse_requirements("simulator"),
    },
    include_package_data=True,
    entry_points="""
        [swh.cli.subcommands]
        scheduler=swh.scheduler.cli
        scheduler-journal=swh.scheduler.cli.journal
    """,
    classifiers=[
        "Programming Language :: Python :: 3",
        "Intended Audience :: Developers",
        "License :: OSI Approved :: GNU General Public License v3 (GPLv3)",
        "Operating System :: OS Independent",
        "Development Status :: 5 - Production/Stable",
    ],
    project_urls={
        "Bug Reports": "https://forge.softwareheritage.org/maniphest",
        "Funding": "https://www.softwareheritage.org/donate",
        "Source": "https://forge.softwareheritage.org/source/swh-scheduler",
        "Documentation": "https://docs.softwareheritage.org/devel/swh-scheduler/",
    },
)
back to top