Skip to main content
  • Home
  • Development
  • Documentation
  • Donate
  • Operational login
  • Browse the archive

swh logo
SoftwareHeritage
Software
Heritage
Archive
Features
  • Search

  • Downloads

  • Save code now

  • Add forge now

  • Help

https://github.com/Ramy-Badr-Ahmed/swh-client
24 July 2024, 17:59:46 UTC
  • Code
  • Branches (3)
  • Releases (0)
  • Visits
Revision 453153760e2761537d3df9a6930be7cb0883071a authored by Moritz Schubotz on 24 July 2024, 16:51:29 UTC, committed by GitHub on 24 July 2024, 16:51:29 UTC
Fix typo in SWH_TOKEN_PROD
1 parent 8d1239c
  • Files
  • Changes
    • Branches
    • Releases
    • HEAD
    • refs/heads/main
    • refs/tags/v1.0-beta
    • refs/tags/v1.5
    • 453153760e2761537d3df9a6930be7cb0883071a
    No releases to show
  • 1c4ca38
  • /
  • Module
  • /
  • HTTPConnector
  • /
  • HTTPClient.php
Raw File Download
Take a new snapshot of a software origin

If the archived software origin currently browsed is not synchronized with its upstream version (for instance when new commits have been issued), you can explicitly request Software Heritage to take a new snapshot of it.

Use the form below to proceed. Once a request has been submitted and accepted, it will be processed as soon as possible. You can then check its processing state by visiting this dedicated page.
swh spinner

Processing "take a new snapshot" request ...

To reference or cite the objects present in the Software Heritage archive, permalinks based on SoftWare Hash IDentifiers (SWHIDs) must be used.
Select below a type of object currently browsed in order to display its associated SWHID and permalink.

  • revision
  • directory
  • content
  • snapshot
origin badgerevision badge
swh:1:rev:453153760e2761537d3df9a6930be7cb0883071a
origin badgedirectory badge
swh:1:dir:c632bcaf28d6f51ddf7fe080f1e7759478063892
origin badgecontent badge
swh:1:cnt:4c60dde8b8236b584445962c7605f7839684e676
origin badgesnapshot badge
swh:1:snp:6e93bd2e1f019500326fed8dde08c617dffb20d7

This interface enables to generate software citations, provided that the root directory of browsed objects contains a citation.cff or codemeta.json file.
Select below a type of object currently browsed in order to generate citations for them.

  • revision
  • directory
  • content
  • snapshot
Generate software citation in BibTex format (requires biblatex-software package)
Generating citation ...
Generate software citation in BibTex format (requires biblatex-software package)
Generating citation ...
Generate software citation in BibTex format (requires biblatex-software package)
Generating citation ...
Generate software citation in BibTex format (requires biblatex-software package)
Generating citation ...
Tip revision: 453153760e2761537d3df9a6930be7cb0883071a authored by Moritz Schubotz on 24 July 2024, 16:51:29 UTC
Fix typo in SWH_TOKEN_PROD
Tip revision: 4531537
HTTPClient.php
<?php

/**
 * @Author: Ramy-Badr-Ahmed
 * @Desc: SWH API Client
 * @Repo: https://github.com/Ramy-Badr-Ahmed/swh-client
 */

namespace Module\HTTPConnector;

use Module\DataType\SwhCoreID;
use Module\Logging\Logger;
use Module\Globals\HTTP;
use GuzzleHttp\Exception\RequestException as GuzzleRequestException;
use Illuminate\Http\Client\ConnectionException;
use Illuminate\Http\Client\PendingRequest;
use Illuminate\Http\Client\RequestException;
use Illuminate\Http\Client\Response;
use Illuminate\Support\Arr;
use Illuminate\Support\Collection;
use Illuminate\Support\Str;
use Illuminate\Translation\ArrayLoader;
use Illuminate\Translation\Translator;
use Illuminate\Validation\ValidationException;
use Illuminate\Validation\Validator;
use Throwable;
use TypeError;
use Exception;

abstract class HTTPClient
{
    use Logger;
    private const API_Version = "/api/1/";
    protected const API_ENDPOINTS = [
        "origin"       => ['expects' => 'URL' , 'route' => self::API_Version . "origin/~/get/"],
        "visit"        => ['expects' => 'URL' , 'route' => self::API_Version . "origin/~/visit/~"],
        "visits"       => ['expects' => 'URL' , 'route' => self::API_Version . "origin/~/visits/~"],
        "save"         => ['expects' => 'URL' , 'route' => self::API_Version . "origin/save/~/url/~/"],
        "saveWithID"   => ['expects' => 'int' , 'route' => self::API_Version . "origin/save/~/"],
        "resolve"      => ['expects' => 'SWHID', 'route' => self::API_Version . "resolve/~/"],
        "snapshot"     => ['expects' => 'SHA1', 'route' => self::API_Version . "snapshot/~/"],
        "release"      => ['expects' => 'SHA1', 'route' => self::API_Version . "release/~/"],

        "revision"     => ['expects' => 'SHA1', 'route' => self::API_Version . "revision/~/"],
        "revisionLog"  => ['expects' => 'SHA1', 'route' => self::API_Version . "revision/~/log/~"],
        "revisionPath" => ['expects' => 'SHA1', 'route' => self::API_Version . "revision/~/directory/~/"],

        "directoryPath"=> ['expects' => 'SHA1', 'route' => self::API_Version . "directory/~/~/"],
        "directory"    => ['expects' => 'SHA1', 'route' => self::API_Version . "directory/~/"],
        "content"      => ['expects' => 'SHA1', 'route' => self::API_Version . "content/sha1_git:~/"],   // todo: sha256, blake2s256
    ];
    private const SUPPORTED_METHODS = ['get', 'post', 'head'];
    public const SUPPORTED_OPTIONS = ['delay', 'debug'];
    protected const CLIENT_OPTIONS = ['responseType', 'apiURL'];
    public const LOG_OPTIONS = ['isVerbose', 'fileDatestamp'];
    protected const PENDING_REQUEST_OPTIONS = ['connectTimeout', 'timeout', 'retry', 'sleepMilliseconds', 'serverType'];
    protected static array $serverErrorCodes = [500, 501, 502, 503, 504, 505, 506, 507, 508, 510, 511];

    public const RESPONSE_TYPE_ARRAY = 'json';
    public const RESPONSE_TYPE_OBJECT = 'object';
    public const RESPONSE_TYPE_COLLECT = 'collect';

    public static string $responseType = self::RESPONSE_TYPE_ARRAY;
    public static string $serverType = 'production';
    public static ?string $apiURL = Null;
    private static int $connectTimeout;
    private static int $timeout;
    private static int $retry;
    private static int $sleepMilliseconds;
    private static array $swhConfigs;
    protected PendingRequest $HTTPRequest;

    /**
     * @param string $method
     * @param string $endpointName
     * @param Collection $append2Url
     * @param ...$options
     * @return Response|iterable|Throwable
     */
    abstract protected function invokeEndpoint(string $method, string $endpointName, Collection $append2Url, ...$options): Response|iterable|Throwable;

    abstract protected static function request(string $method, string $uri, ...$options): Response;

    /**
     * @param ...$options
     * @return void
     */
    public static function setOptions(...$options) : void
    {
        $irrelevantOptions = array_diff(array_keys($options), array_merge(self::CLIENT_OPTIONS, self::PENDING_REQUEST_OPTIONS, self::LOG_OPTIONS));
        if($irrelevantOptions){
            self::addLogs("Undefined Option(s). Ignoring: ".implode(", ", $irrelevantOptions));
            //return;
        }
        if(isset($options['responseType'])){
            self::$responseType = match($options['responseType']){
                'collect'       => self::RESPONSE_TYPE_COLLECT,
                'object'        => self::RESPONSE_TYPE_OBJECT,
                default         => self::RESPONSE_TYPE_ARRAY
            };
        }
        if(isset($options['apiURL'])){
            self::$apiURL = $options['apiURL'];
        }

        $setPendingRequestOptions = Arr::only($options, self::PENDING_REQUEST_OPTIONS);

        if(!empty($setPendingRequestOptions)){
            foreach ($setPendingRequestOptions as $key => $value){
                self::${$key} = $value;
            }
        }

        self::setLogOptions(...$options);
    }

    public function __construct()
    {
        self::$swhConfigs = require 'swhConfigs.php';

        self::$apiURL = self::$apiURL ?? self::$swhConfigs[self::$serverType]['api-url'];

        self::openLog();

        $this->HTTPRequest = HTTP::withToken(self::$swhConfigs[self::$serverType]['token'])
            ->connectTimeout(self::$connectTimeout ?? 5)
            ->timeout(self::$timeout ?? 5)
            ->throw(function ($response, $e) {
                if($response->serverError()){
                    throw new Exception("Server-side Error Status", $response->status());
                }
            })
            ->retry(self::$retry ?? 5, self::$sleepMilliseconds ?? 5000,
                function ($e, $request) {
                    $retryMessage = "Retrying ?";

                    if(isset($e) && !$e instanceof GuzzleRequestException) {
                        switch (true) {
                            case $e instanceof ConnectionException || $e->response->serverError():

                                self::addLogs($retryMessage . " : Yes. Reason --> {$e->getMessage()}");
                                return true;
                            case $e->response->status() === 406:

                                self::addLogs($retryMessage . " : Yes. Reason --> {$e->getMessage()}");
                                $request->acceptJson();
                                return true;
                            case $e->response->status() === 403:

                                self::$serverType = 'staging';

                                if(self::$apiURL === self::$swhConfigs[self::$serverType]['api-url']) {
                                    self::addLogs($retryMessage . " : Yes. Reason --> {$e->getMessage()}");

                                    $request->withToken(self::$swhConfigs[self::$serverType]['token']);
                                    return true;
                                }
                                break;
                            case $e instanceof RequestException || $e->response->clientError():

                                self::addLogs($retryMessage . " : No. Reason --> {$e->getMessage()}");
                                return false;
                        }
                    }
                    return false;
                })
            ->accept('application/json')
            ->withOptions([
                'debug' => false,
                'allow_redirects' => ['max' => 1, 'strict'=> true, 'protocols' => ['https'], 'track_redirects' => true],
                'force_ip_resolve' => 'v4',
                'http_errors' => false,
                'verify' => true,
                'headers' => ['User-Agent' => 'swh-client/1.0'],
                'decode_content' => 'gzip',
                'version' => '1.1'
            ]);
    }

    /**
     * @param string $method
     * @param string $endPointName
     * @param Collection $append2Url
     * @return Void
     * @throws Exception
     */
    protected function prepareForInvoke(string $method, string $endPointName, Collection &$append2Url): Void
    {
        if(isset(parse_url($endPointName)["host"])){
            $append2Url = Null;
            return;
        }
        if(!in_array($method, self::SUPPORTED_METHODS)){
            throw new Exception("Method Mismatch. Unsupported HTTP method passed. Supported methods: ".implode(", ", self::SUPPORTED_METHODS), 980);
        }

        if(self::isValidEndpoint($endPointName) && self::isValidPattern($endPointName, $append2Url, [self::class, $endPointName==='resolve' ? 'isExpectedSwhIdPattern' : 'isExpectedPattern'])){
            $this->addLogs('Tests passed without errors, proceeding...');
        }
        if(self::API_ENDPOINTS[$endPointName]['expects'] ==='URL'){
            $append2Url[0] = preg_replace('#/$#','', $append2Url[0]);

            if($endPointName==='save') $append2Url = $append2Url->reverse();

        }
    }

    /**
     * @param string $endPointName
     * @return bool
     * @throws Exception
     */
    private static function isValidEndpoint(string $endPointName): bool
    {
        if(Arr::exists(self::API_ENDPOINTS, $endPointName) === false) {
            throw new Exception("Error in isValidEndpoint(): Unrecognised Endpoint", 980);
        }
        return true;
    }
    /**
     * @param string $endPointName
     * @param Collection $append2Url
     * @param callable $patternCallback
     * @return bool
     * @throws Exception
     */
    private static function isValidPattern(string $endPointName, Collection $append2Url, callable $patternCallback) : bool
    {
        if($append2Url->count() !== Str::substrCount(self::API_ENDPOINTS[$endPointName]['route'], '~')){
            throw new Exception("Error in isValidPattern(): Incompatible URL substitution pattern", 980);
        }
        return $patternCallback($endPointName, $append2Url);
    }

    /**
     * @param string $endPointName
     * @param Collection $append2Url
     * @return bool
     * @throws Exception
     * @throws Throwable
     */
    private static function isExpectedPattern(string $endPointName, Collection $append2Url) : bool
    {
        if($endPointName === 'saveWithID'){
            throw_unless(is_int($append2Url[0]), new Exception("Validation failed in isExpectedPattern(): The route '$endPointName' expects Integer", 980));
            return true;
        }

        $validator = new Validator(new Translator(new ArrayLoader(), 'en'), $append2Url->toArray(),
            [
                0 => ['url', 'max:255', 'regex: /^[a-f0-9]{40}(?:\/\??\S*)?$/i']     // ex: SHA1(/?branches_from=v2.6.37-rc6&branches_count=1000)?  SHA1(/license)? // includes all other queries
            ],
            [
                'url'   => ':input is a non-valid URL',
                'max'   => ':input too long',
                'regex' => ':input is not a valid sha_1',
            ],
        );

        $errors = $validator->errors()->all();
        $rulesKey = $validator->failed()[0];

        switch (true){
            case count($errors)>=2:
                throw ValidationException::withMessages([implode('/', array_keys($rulesKey)) => "Validation failed in isExpectedPattern(): Non-valid '{$validator->getData()[0]}' URL/SHA_1"]);

            case Arr::has($rulesKey , 'Url'):
                self::addLogs("Validator Note: Non-valid URL. Pass: '$endPointName' endpoint expects --> ". self::API_ENDPOINTS[$endPointName]['expects']);

                if(self::API_ENDPOINTS[$endPointName]['expects'] === 'URL'){
                    throw ValidationException::withMessages([implode(array_keys($rulesKey)) => "Validation failed in isExpectedPattern(): The route '$endPointName' doesn't expect SHA1 entry. $errors[0]"]);
                };
                break;

            case Arr::has($rulesKey , "Regex"):
                self::addLogs("Validator Note: Non-valid SHA1. Pass: '$endPointName' endpoint expects --> ". self::API_ENDPOINTS[$endPointName]['expects']);

                if(self::API_ENDPOINTS[$endPointName]['expects'] === 'SHA1'){
                    throw ValidationException::withMessages([implode(array_keys($rulesKey)) => "Validation failed in isExpectedPattern(): The route '$endPointName' doesn't expect URL entry.'{$validator->getData()[0]}' is a non-valid SHA1"]);
                };
                break;
        }
        return true;
    }

    /**
     * @param string $endPointName
     * @param Collection $append2Url
     * @return bool
     * @throws Exception
     */
    private static function isExpectedSwhIdPattern(string $endPointName, Collection $append2Url): bool
    {
        try{
            $swhID = Str::of($append2Url->toArray()[0])->match('/^([^;]+)/')->value();
            new SwhCoreID($swhID);
            return true;
        }catch (TypeError $e){

            self::addLogs("Validator Note: Non-valid SWHID. Pass: '$endPointName' endpoint expects --> ". self::API_ENDPOINTS[$endPointName]['expects']);

            throw new Exception('Validation failed in isExpectedSwhIdPattern(): provided swhID seems incorrectly formatted. Correct format is
             --> swh:1:swhInitials:40-bit-hexString', 980);
        }
    }
}
The diff you're trying to view is too large. Only the first 1000 changed files have been loaded.
Showing with 0 additions and 0 deletions (0 / 0 diffs computed)
swh spinner

Computing file changes ...

back to top

Software Heritage — Copyright (C) 2015–2025, The Software Heritage developers. License: GNU AGPLv3+.
The source code of Software Heritage itself is available on our development forge.
The source code files archived by Software Heritage are available under their own copyright and licenses.
Terms of use: Archive access, API— Content policy— Contact— JavaScript license information— Web API