https://forge.softwareheritage.org/source/swh-scheduler.git
For each known visit type, we run a loop which: - monitors the size of the relevant celery queue - schedules more visits of the relevant type once the number of available slots goes over a given threshold (currently set to 5% of the max queue size). The scheduling of visits combines multiple scheduling policies, for now using static ratios set in the `POLICY_RATIOS` dict. We emit a warning if the ratio of origins fetched for each policy is skewed with respect to the original request (allowing, for now, manual adjustement of the ratios). The CLI endpoint spawns one thread for each visit type, which all handle connections to RabbitMQ and the scheduler backend separately. For now, we handle exceptions in the visit scheduling threads by (stupidly) respawning the relevant thread directly. We should probably improve this to give up after a specific number of tries. Co-authored-by: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
- HEAD
- refs/heads/add-forge-now-cli-evolution
- refs/heads/add_forge_now_cli
- refs/heads/debian/buster-swh
- refs/heads/debian/stretch-swh
- refs/heads/debian/unstable-swh
- refs/heads/debian/upstream
- refs/heads/generated-differential-D4928-source
- refs/heads/generated-differential-D4981-source
- refs/heads/improve-cli-check-ingestion
- refs/heads/master
- refs/heads/oneshot-task
- refs/heads/pristine-tar
- refs/merge-requests/262/head
- refs/merge-requests/262/merge
- refs/merge-requests/263/head
- refs/merge-requests/263/merge
- refs/merge-requests/264/head
- refs/merge-requests/264/merge
- refs/merge-requests/265/head
- refs/merge-requests/265/merge
- refs/merge-requests/266/head
- refs/merge-requests/266/merge
- refs/merge-requests/267/head
- refs/merge-requests/267/merge
- refs/merge-requests/268/head
- refs/merge-requests/268/merge
- refs/merge-requests/269/head
- refs/merge-requests/269/merge
- refs/merge-requests/270/head
- refs/merge-requests/270/merge
- refs/merge-requests/271/head
- refs/merge-requests/271/merge
- refs/merge-requests/272/head
- refs/merge-requests/272/merge
- refs/merge-requests/273/head
- refs/merge-requests/273/merge
- refs/merge-requests/274/head
- refs/merge-requests/274/merge
- refs/merge-requests/275/head
- refs/merge-requests/275/merge
- refs/merge-requests/276/head
- refs/merge-requests/276/merge
- refs/merge-requests/277/head
- refs/merge-requests/277/merge
- refs/merge-requests/278/head
- refs/merge-requests/278/merge
- refs/merge-requests/279/head
- refs/merge-requests/279/merge
- refs/merge-requests/280/head
- refs/merge-requests/280/merge
- refs/merge-requests/281/head
- refs/merge-requests/281/merge
- refs/merge-requests/282/head
- refs/merge-requests/282/merge
- refs/merge-requests/283/head
- refs/merge-requests/283/merge
- refs/merge-requests/284/head
- refs/merge-requests/284/merge
- refs/merge-requests/285/head
- refs/merge-requests/285/merge
- refs/merge-requests/286/head
- refs/merge-requests/286/merge
- refs/merge-requests/287/head
- refs/merge-requests/287/merge
- refs/merge-requests/288/head
- refs/merge-requests/288/merge
- refs/merge-requests/289/head
- refs/merge-requests/289/merge
- refs/merge-requests/290/head
- refs/merge-requests/290/merge
- refs/merge-requests/291/head
- refs/merge-requests/291/merge
- refs/merge-requests/292/head
- refs/merge-requests/292/merge
- refs/merge-requests/307/head
- refs/merge-requests/307/merge
- refs/merge-requests/332/head
- refs/merge-requests/332/merge
- refs/merge-requests/333/head
- refs/merge-requests/333/merge
- refs/merge-requests/353/head
- refs/merge-requests/353/merge
- refs/merge-requests/360/head
- refs/merge-requests/360/merge
- refs/merge-requests/365/head
- refs/merge-requests/365/merge
- refs/merge-requests/369/head
- refs/merge-requests/369/merge
- refs/merge-requests/372/head
- refs/merge-requests/372/merge
- refs/merge-requests/373/head
- refs/merge-requests/373/merge
- refs/merge-requests/374/head
- refs/merge-requests/374/merge
- refs/tags/debian/upstream/0.0.1
- refs/tags/v0.0.47
- refs/tags/v0.0.56
- 50d7fd7ff49b02654c1416f80e786acac3a980d5
- v2.2.1
- v2.2.0
- v2.1.0
- v2.0.0
- v1.9.2
- v1.9.1
- v1.9.0
- v1.8.0
- v1.7.0
- v1.6.0
- v1.5.1
- v1.5.0
- v1.4.0
- v1.3.0
- v1.2.3
- v1.2.2
- v1.2.1
- v1.2.0
- v1.13.0
- v1.12.1
- v1.12.0
- v1.11.0
- v1.10.2
- v1.10.1
- v1.10.0
- v1.1.2
- v1.1.1
- v1.1.0
- v1.0.0
- v0.9.2
- v0.9.1
- v0.9.0
- v0.8.2
- v0.8.1
- v0.8.0
- v0.7.0
- v0.6.0
- v0.5.3
- v0.5.2
- v0.5.1
- v0.5.0
- v0.4.0
- v0.3.0
- v0.23.0
- v0.22.0
- v0.21.0
- v0.20.0
- v0.2.2
- v0.2.1
- v0.2.0
- v0.19.0
- v0.18.2
- v0.18.1
- v0.18.0
- v0.17.1
- v0.17.0
- v0.16.0
- v0.15.0
- v0.14.2
- v0.14.1
- v0.14.0
- v0.13.0
- v0.12.0
- v0.11.0
- v0.10.0
- v0.1.1
- v0.1.0
- v0.0.9
- v0.0.72
- v0.0.71
- v0.0.70
- v0.0.7
- v0.0.69
- v0.0.68
- v0.0.67
- v0.0.66
- v0.0.65
- v0.0.64
- v0.0.63
- v0.0.62
- v0.0.61
- v0.0.60
- v0.0.6
- v0.0.59
- v0.0.58
- v0.0.57
- v0.0.55
- v0.0.54
- v0.0.53
- v0.0.52
- v0.0.51
- v0.0.50
- v0.0.5
- v0.0.49
- v0.0.48
- v0.0.46
- v0.0.45
- v0.0.44
- v0.0.43
- v0.0.42
- v0.0.41
- v0.0.40
- v0.0.4
- v0.0.39
- v0.0.38
- v0.0.37
- v0.0.36
- v0.0.35
- v0.0.34
- v0.0.33
- v0.0.32
- v0.0.31
- v0.0.30
- v0.0.3
- v0.0.29
- v0.0.28
- v0.0.27
- v0.0.26
- v0.0.25
- v0.0.24
- v0.0.23
- v0.0.22
- v0.0.21
- v0.0.20
- v0.0.2
- v0.0.19
- v0.0.18
- v0.0.17
- v0.0.16
- v0.0.15
- v0.0.14
- v0.0.13
- v0.0.12
- v0.0.11
- v0.0.10
- v0.0.1
- debian/upstream/1.9.2
- debian/upstream/1.9.1
- debian/upstream/1.9.0
- debian/upstream/1.8.0
- debian/upstream/1.7.0
- debian/upstream/1.6.0
- debian/upstream/1.5.1
- debian/upstream/1.5.0
- debian/upstream/1.4.0
- debian/upstream/1.3.0
- debian/upstream/1.2.3
- debian/upstream/1.2.2
- debian/upstream/1.2.1
- debian/upstream/1.2.0
- debian/upstream/1.13.0
- debian/upstream/1.12.1
- debian/upstream/1.12.0
- debian/upstream/1.11.0
- debian/upstream/1.10.2
- debian/upstream/1.10.1
- debian/upstream/1.10.0
- debian/upstream/1.1.2
- debian/upstream/1.1.1
- debian/upstream/1.1.0
- debian/upstream/1.0.0
- debian/upstream/0.9.2
- debian/upstream/0.9.1
- debian/upstream/0.9.0
- debian/upstream/0.8.2
- debian/upstream/0.8.1
- debian/upstream/0.8.0
- debian/upstream/0.7.0
- debian/upstream/0.6.0
- debian/upstream/0.5.3
- debian/upstream/0.5.2
- debian/upstream/0.5.1
- debian/upstream/0.5.0
- debian/upstream/0.4.0
- debian/upstream/0.3.0
- debian/upstream/0.23.0
- debian/upstream/0.22.0
- debian/upstream/0.21.0
- debian/upstream/0.20.0
- debian/upstream/0.2.2
- debian/upstream/0.2.1
- debian/upstream/0.2.0
- debian/upstream/0.19.0
- debian/upstream/0.18.2
- debian/upstream/0.18.1
- debian/upstream/0.18.0
- debian/upstream/0.17.1
- debian/upstream/0.17.0
- debian/upstream/0.16.0
- debian/upstream/0.15.0
- debian/upstream/0.14.2
- debian/upstream/0.14.1
- debian/upstream/0.13.0
- debian/upstream/0.12.0
- debian/upstream/0.11.0
- debian/upstream/0.10.0
- debian/upstream/0.1.1
- debian/upstream/0.1.0
- debian/upstream/0.0.9
- debian/upstream/0.0.72
- debian/upstream/0.0.71
- debian/upstream/0.0.70
- debian/upstream/0.0.7
- debian/upstream/0.0.69
- debian/upstream/0.0.68
- debian/upstream/0.0.67
- debian/upstream/0.0.66
- debian/upstream/0.0.65
- debian/upstream/0.0.64
- debian/upstream/0.0.63
- debian/upstream/0.0.62
- debian/upstream/0.0.61
- debian/upstream/0.0.60
- debian/upstream/0.0.6
- debian/upstream/0.0.59
- debian/upstream/0.0.58
- debian/upstream/0.0.57
- debian/upstream/0.0.56
- debian/upstream/0.0.55
- debian/upstream/0.0.54
- debian/upstream/0.0.53
- debian/upstream/0.0.52
- debian/upstream/0.0.51
- debian/upstream/0.0.50
- debian/upstream/0.0.5
- debian/upstream/0.0.49
- debian/upstream/0.0.48
- debian/upstream/0.0.47
- debian/upstream/0.0.46
- debian/upstream/0.0.45
- debian/upstream/0.0.44
- debian/upstream/0.0.43
- debian/upstream/0.0.42
- debian/upstream/0.0.41
- debian/upstream/0.0.40
- debian/upstream/0.0.4
- debian/upstream/0.0.39
- debian/upstream/0.0.38
- debian/upstream/0.0.35
- debian/upstream/0.0.34
- debian/upstream/0.0.33
- debian/upstream/0.0.32
- debian/upstream/0.0.31
- debian/upstream/0.0.30
- debian/upstream/0.0.3
- debian/upstream/0.0.29
- debian/upstream/0.0.28
- debian/upstream/0.0.27
- debian/upstream/0.0.26
- debian/upstream/0.0.25
- debian/upstream/0.0.24
- debian/upstream/0.0.23
- debian/upstream/0.0.22
- debian/upstream/0.0.21
- debian/upstream/0.0.20
- debian/upstream/0.0.2
- debian/upstream/0.0.19
- debian/upstream/0.0.18
- debian/upstream/0.0.17
- debian/upstream/0.0.16
- debian/upstream/0.0.15
- debian/upstream/0.0.14
- debian/upstream/0.0.13
- debian/upstream/0.0.12
- debian/upstream/0.0.11
- debian/upstream/0.0.10
- debian/1.9.2-1_swh1_bpo10+1
- debian/1.9.2-1_swh1
- debian/1.9.1-1_swh1_bpo10+1
- debian/1.9.1-1_swh1
- debian/1.9.0-1_swh1_bpo10+1
- debian/1.9.0-1_swh1
- debian/1.8.0-1_swh1_bpo10+1
- debian/1.8.0-1_swh1
- debian/1.7.0-1_swh1_bpo10+1
- debian/1.7.0-1_swh1
- debian/1.6.0-1_swh1_bpo10+1
- debian/1.6.0-1_swh1
- debian/1.5.1-1_swh1_bpo10+1
- debian/1.5.1-1_swh1
- debian/1.5.0-1_swh1_bpo10+1
- debian/1.5.0-1_swh1
- debian/1.4.0-1_swh1_bpo10+1
- debian/1.4.0-1_swh1
- debian/1.3.0-1_swh1
- debian/1.2.3-1_swh1_bpo10+1
- debian/1.2.3-1_swh1
- debian/1.2.2-1_swh1_bpo10+1
- debian/1.2.2-1_swh1
- debian/1.2.1-1_swh1_bpo10+1
- debian/1.2.1-1_swh1
- debian/1.2.0-1_swh1_bpo10+1
- debian/1.2.0-1_swh1
- debian/1.13.0-1_swh1
- debian/1.12.1-1_swh1_bpo10+1
- debian/1.12.1-1_swh1
- debian/1.12.0-2_swh1_bpo10+1
- debian/1.12.0-2_swh1
- debian/1.12.0-1_swh1
- debian/1.11.0-1_swh1_bpo10+1
- debian/1.11.0-1_swh1
- debian/1.10.2-1_swh1_bpo10+1
- debian/1.10.2-1_swh1
- debian/1.10.1-1_swh1_bpo10+1
- debian/1.10.1-1_swh1
- debian/1.10.0-1_swh1_bpo10+1
- debian/1.10.0-1_swh1
- debian/1.1.2-1_swh1_bpo10+1
- debian/1.1.2-1_swh1
- debian/1.1.1-1_swh1_bpo10+1
- debian/1.1.1-1_swh1
- debian/1.1.0-1_swh1_bpo10+1
- debian/1.1.0-1_swh1
- debian/1.0.0-1_swh1_bpo10+1
- debian/1.0.0-1_swh1
- debian/0.9.2-1_swh1_bpo10+1
- debian/0.9.2-1_swh1
- debian/0.9.1-1_swh1_bpo10+1
- debian/0.9.1-1_swh1
- debian/0.9.0-1_swh2_bpo10+1
- debian/0.9.0-1_swh2
- debian/0.9.0-1_swh1
- debian/0.8.2-1_swh2_bpo10+1
- debian/0.8.2-1_swh2
- debian/0.8.2-1_swh1
- debian/0.8.1-1_swh1_bpo10+1
- debian/0.8.1-1_swh1
- debian/0.8.0-1_swh1_bpo10+1
- debian/0.8.0-1_swh1
- debian/0.7.0-1_swh1_bpo10+1
- debian/0.7.0-1_swh1
- debian/0.6.0-1_swh1_bpo10+1
- debian/0.6.0-1_swh1
- debian/0.5.3-1_swh1_bpo10+1
- debian/0.5.3-1_swh1
- debian/0.5.2-1_swh1_bpo10+1
- debian/0.5.2-1_swh1
- debian/0.5.1-1_swh1_bpo10+1
- debian/0.5.1-1_swh1
- debian/0.5.0-1_swh1_bpo10+1
- debian/0.5.0-1_swh1
- debian/0.4.0-1_swh1
- debian/0.3.0-1_swh1_bpo10+1
- debian/0.3.0-1_swh1
- debian/0.23.0-1_swh1_bpo10+1
- debian/0.23.0-1_swh1
- debian/0.22.0-1_swh1_bpo10+1
- debian/0.22.0-1_swh1
- debian/0.21.0-1_swh1_bpo10+1
- debian/0.21.0-1_swh1
- debian/0.20.0-1_swh1_bpo10+1
- debian/0.20.0-1_swh1
- debian/0.2.2-1_swh1_bpo10+1
- debian/0.2.2-1_swh1
- debian/0.2.1-1_swh1_bpo10+1
- debian/0.2.1-1_swh1
- debian/0.2.0-1_swh1_bpo10+1
- debian/0.2.0-1_swh1
- debian/0.19.0-1_swh1_bpo10+1
- debian/0.19.0-1_swh1
- debian/0.18.2-1_swh1_bpo10+1
- debian/0.18.2-1_swh1
- debian/0.18.1-1_swh1
- debian/0.18.0-1_swh1_bpo10+1
- debian/0.18.0-1_swh1
- debian/0.17.1-1_swh1_bpo10+1
- debian/0.17.1-1_swh1
- debian/0.17.0-1_swh1_bpo10+1
- debian/0.17.0-1_swh1
- debian/0.16.0-1_swh1_bpo10+1
- debian/0.16.0-1_swh1
- debian/0.15.0-1_swh1_bpo10+1
- debian/0.15.0-1_swh1
- debian/0.14.2-1_swh1_bpo10+1
- debian/0.14.2-1_swh1
- debian/0.14.1-1_swh1_bpo10+1
- debian/0.14.1-1_swh1
- debian/0.13.0-1_swh1_bpo10+1
- debian/0.13.0-1_swh1
- debian/0.12.0-1_swh1_bpo10+1
- debian/0.12.0-1_swh1
- debian/0.11.0-1_swh1_bpo10+1
- debian/0.11.0-1_swh1
- debian/0.10.0-1_swh1_bpo10+1
- debian/0.10.0-1_swh1
- debian/0.1.1-1_swh1_bpo10+1
- debian/0.1.1-1_swh1
- debian/0.1.0-1_swh1_bpo10+1
- debian/0.1.0-1_swh1
- debian/0.0.72-1_swh1_bpo10+1
- debian/0.0.72-1_swh1
- debian/0.0.71-1_swh1_bpo10+1
- debian/0.0.71-1_swh1
- debian/0.0.70-1_swh1_bpo10+1
- debian/0.0.70-1_swh1
- debian/0.0.69-1_swh1_bpo10+1
- debian/0.0.69-1_swh1
- debian/0.0.68-1_swh1
- debian/0.0.67-1_swh1
- debian/0.0.66-1_swh1_bpo10+1
- debian/0.0.66-1_swh1
- debian/0.0.65-1_swh2_bpo10+1
- debian/0.0.65-1_swh2
- debian/0.0.65-1_swh1
- debian/0.0.64-1_swh1_bpo10+1
- debian/0.0.64-1_swh1
- debian/0.0.63-1_swh2_bpo10+1
- debian/0.0.63-1_swh2
- debian/0.0.63-1_swh1
- debian/0.0.62-1_swh1_bpo9+1
- debian/0.0.62-1_swh1_bpo10+1
- debian/0.0.62-1_swh1
- debian/0.0.61-1_swh1_bpo9+1
- debian/0.0.61-1_swh1_bpo10+1
- debian/0.0.61-1_swh1
- debian/0.0.60-1_swh2_bpo9+1
- debian/0.0.60-1_swh2_bpo10+1
- debian/0.0.60-1_swh2
- debian/0.0.60-1_swh1
- debian/0.0.59-1_swh1_bpo9+1
- debian/0.0.59-1_swh1_bpo10+1
- debian/0.0.59-1_swh1
- debian/0.0.58-1_swh1_bpo9+1
- debian/0.0.58-1_swh1_bpo10+1
- debian/0.0.58-1_swh1
- debian/0.0.57-1_swh1_bpo9+1
- debian/0.0.57-1_swh1_bpo10+1
- debian/0.0.57-1_swh1
- debian/0.0.56-1_swh1_bpo9+1
- debian/0.0.56-1_swh1
- debian/0.0.55-1_swh1
- debian/0.0.54-1_swh1_bpo9+1
- debian/0.0.54-1_swh1
- debian/0.0.53-1_swh1_bpo9+1
- debian/0.0.53-1_swh1
- debian/0.0.52-1_swh1_bpo9+1
- debian/0.0.52-1_swh1
- debian/0.0.51-1_swh1_bpo9+1
- debian/0.0.51-1_swh1
- debian/0.0.50-1_swh2_bpo9+1
- debian/0.0.50-1_swh2
- debian/0.0.50-1_swh1
- debian/0.0.49-1_swh2_bpo9+1
- debian/0.0.49-1_swh2
- debian/0.0.49-1_swh1
- debian/0.0.48-1_swh1_bpo9+1
- debian/0.0.48-1_swh1
- debian/0.0.47-1_swh2_bpo9+1
- debian/0.0.47-1_swh2
- debian/0.0.47-1_swh1_bpo9+1
- debian/0.0.47-1_swh1
- debian/0.0.46-1_swh1_bpo9+1
- debian/0.0.46-1_swh1
- debian/0.0.45-1_swh1_bpo9+1
- debian/0.0.45-1_swh1
- debian/0.0.44-1_swh1_bpo9+1
- debian/0.0.44-1_swh1
- debian/0.0.43-1_swh1_bpo9+1
- debian/0.0.43-1_swh1
- debian/0.0.42-1_swh1_bpo9+1
- debian/0.0.42-1_swh1
- debian/0.0.41-1_swh1_bpo9+1
- debian/0.0.40-1_swh1_bpo9+1
- debian/0.0.40-1_swh1
- debian/0.0.38-1_swh1_bpo9+1
- debian/0.0.38-1_swh1
Cook and download a directory from the Software Heritage Vault
You have requested the cooking of the directory with identifier swh:1:dir:c90fcf5faa32fbe1d6766311cb6e4f95723d298c into a standard tar.gz archive
.
Are you sure you want to continue ?
Download a directory from the Software Heritage Vault
You have requested the download of the directory with identifier swh:1:dir:c90fcf5faa32fbe1d6766311cb6e4f95723d298c as a standard tar.gz archive
.
Are you sure you want to continue ?
Cook and download a revision from the Software Heritage Vault
You have requested the cooking of the history heading to revision with identifier swh:1:rev:50d7fd7ff49b02654c1416f80e786acac3a980d5 into a bare git archive
.
Are you sure you want to continue ?
Download a revision from the Software Heritage Vault
You have requested the download of the history heading to revision with identifier swh:1:rev:50d7fd7ff49b02654c1416f80e786acac3a980d5 as a bare git archive
.
Are you sure you want to continue ?
Invalid Email !
The provided email is not well-formed.
Download link has expired
The requested archive is no longer available for download from the Software Heritage Vault.
Do you want to cook it again ?
Take a new snapshot of a software origin
If the archived software origin currently browsed is not synchronized with its upstream version (for instance when new commits have been issued), you can explicitly request Software Heritage to take a new snapshot of it.
Use the form below to proceed. Once a request has been submitted and accepted, it will be processed as soon as possible. You can then check its processing state by visiting this dedicated page.Processing "take a new snapshot" request ...
To reference or cite the objects present in the Software Heritage archive, permalinks based on SoftWare Hash IDentifiers (SWHIDs) must be used.
Select below a type of object currently browsed in order to display its associated SWHID and permalink.
Add a new cli endpoint to schedule recurrent visits in Celery
File | Mode | Size |
---|---|---|
data | ||
docs | ||
sql | ||
swh | ||
.gitignore | -rw-r--r-- | 129 bytes |
.pre-commit-config.yaml | -rw-r--r-- | 1.0 KB |
AUTHORS | -rw-r--r-- | 112 bytes |
CODE_OF_CONDUCT.md | -rw-r--r-- | 3.3 KB |
CONTRIBUTORS | -rw-r--r-- | 14 bytes |
LICENSE | -rw-r--r-- | 34.3 KB |
LICENSE.Celery | -rw-r--r-- | 2.6 KB |
MANIFEST.in | -rw-r--r-- | 154 bytes |
Makefile | -rw-r--r-- | 163 bytes |
README.md | -rw-r--r-- | 292 bytes |
conftest.py | -rw-r--r-- | 664 bytes |
mypy.ini | -rw-r--r-- | 750 bytes |
pyproject.toml | -rw-r--r-- | 237 bytes |
pytest.ini | -rw-r--r-- | 33 bytes |
requirements-journal.txt | -rw-r--r-- | 12 bytes |
requirements-simulator.txt | -rw-r--r-- | 21 bytes |
requirements-swh.txt | -rw-r--r-- | 50 bytes |
requirements-test.txt | -rw-r--r-- | 155 bytes |
requirements.txt | -rw-r--r-- | 384 bytes |
setup.cfg | -rw-r--r-- | 247 bytes |
setup.py | -rwxr-xr-x | 2.5 KB |
tox.ini | -rw-r--r-- | 1.5 KB |
Computing file changes ...