Revision b7f4e2d37664955425117babcc08690ee5a505dd authored by Dataliberate on 18 February 2019, 17:01:37 UTC, committed by Dataliberate on 18 February 2019, 17:01:37 UTC
Add build termconfig script
1 parent cd2b745
RELEASING.TXT
0-a) Technical pre-conditions for release.
Overview
Code, schema, examples and supporting documentation should be in a stable state.
Pay particular attention to release-specific structures and docs/releases.html
(e.g. no stray '2.x', '2.*' etc. in hidden markup or tables)
In sdoapp.py we declare the current version. This should match the number
chosen in /docs/releases.html (check markup assigns an HTML ID too).
The following steps assume a general healthy freeze (tested / QA as below).
in sdoapp.py:
SCHEMA_VERSION=2.2 # or 2.x, as above.
in sdoapp.py:
releaselog = { "2.0": "2015-05-13", "2.1": "2015-08-06" }
becomes
releaselog = { "2.0": "2015-05-13", "2.1": "2015-08-06", "2.2": "2015-11-05" }
... this should have a release date for the current release and all
previous releases that are archived under data/releases/{version}/*
TODO: Add a unit test to ensure this.
TODO: Re-order this document to avoid forward references.
Under data/releases/{version}/ we need a versioned-snapshot structure:
e.g.
data/releases/2.1
data/releases/2.1/all-layers.nq
data/releases/2.1/core.nq
data/releases/2.1/ext-auto.nq
data/releases/2.1/ext-bib.nq
data/releases/2.1/README.md
data/releases/2.1/schema-all.html
data/releases/2.1/schema.nt
data/releases/2.1/schema.rdfa
Of these,
* schema.rdfa is a frozen one-time copy of data/schema.rdfa
* schema-all.html is generated from this via running the Web app and
capturing via curl of /version/build-latest,
e.g. http://localhost:8080/version/build-latest
when served locally via "dev_appserver ."
For now, you need to edit this html to remove webschemas.org from heading section.
* To create .nt, .nq, etc files use scripts/buildreleasefiles.sh
* README.md should be copied/updated accordingly. TODO: simplify this file.
Example usage:
First copy snapshot into place:
* cp data/schema.rdfa data/releases/2.2/schema.rdfa
* Now the HTML snapshot itself. Assuming a local server (e.g. from 'dev_appserver .') we can do:
* curl http://localhost:8080/version/build-latest > data/releases/3.2/schema-all.html (taking care with the version number, of course)
or this, to use webschemas: curl https://webschemas.org/version/build-latest > data/releases/3.4/schema-all.html
* clean up the html including the bit that says you're in a developer view etc.
Then generate N-Triples etc. versions:
* Ensure that the EXTENSIONS variable at the top of the script 'scripts/buildreleasefiles.sh' contains all the extensions for this release.
* scripts/buildreleasefiles.sh 3.1 - Check listed extensions is correct update if not
* Run './scripts/buildsitemap.py' make sure sitemap is up to date
* Inspect snapshot, add to git and push to repo.
* ls -l data/releases/2.2/
* git add data/releases/2.2/*
* git push
* update sites: e.g. ./scripts/updateAppEngine.sh to update sdo-phobos.appspot.com, webschemas.org
* Inspect /version/{x.y}/ on those sites manually; check for newly added terms etc.
* TODO: more QA and automation would be highly appropriate here.
TODO: list optional tasks post-launch (blog, email, refresh dydra sparql db etc.).
1) General preconditions / process and QA for release.
1-a) Steering Group have signed off on changes and release plan,
and no active and unresolved disputes in the Community Group.
1-b) All code is commited to appropriate release branch at Github (usually
configured as the current default github branch for /schemaorg/ project).
1-c) All tests pass.
It is best to test against a fresh checkout to avoid depending on uncommitted
files. The unit tests depend upon a local installation of Google AppEngine, e.g.
PYTHONPATH=/usr/local/google_appengine ./scripts/run_tests.py
To run the full unit tests (graph test of schema integrity) the 'rdflib'
python library is also needed: 'sudo easy_install rdflib' should do it.
Example transcript:
git clone https://github.com/schemaorg/schemaorg.git
Cloning into 'schemaorg'...
remote: Counting objects: 6375, done.
remote: Compressing objects: 100% (80/80), done.
remote: Total 6375 (delta 43), reused 0 (delta 0), pack-reused 6295
Receiving objects: 100% (6375/6375), 27.62 MiB | 1.01 MiB/s, done.
Resolving deltas: 100% (4074/4074), done.
Checking connectivity... done.
danbri-macbookpro2:tmp danbri$ cd schemaorg/
[...]
Ran 70 tests in 21.168s
OK (expected failures=3)
1-d) Latest candidate release branch is pushed to per-branch appspot instance
(e.g. sdo-phobos.appspot.com) and the generic unstable upstream site
(i.e. webschemas.org).
1-e) The manual QA page /docs/qa.html has been reviewed to ensure
representative pages of each type appear to be in a healthy state.
e.g. see http://webschemas.org/docs/qa.html
2.) Final checks
There is often some final tweaking, bugs, etc. When things settle down, a
final round of sanity-check tests is useful:
* Run again './scripts/buildsitemap.py' make sure sitemap is up to date
* Run again: PYTHONPATH=/usr/local/google_appengine ./scripts/run_tests.py
* Run again 'git status', confirm no uncommited files or changes.
* Run again: './scripts/updateAppEngine.sh' for development/upstream site sync.
3.) Previously: "Merge release branch into master"; now we develop in "master."
4.) Publish from a clean checkout of 'master' branch to schema.org.
Using gcloud app deploy
~~~~~~~~~~~~~~~~~~~~~~~
From 3.4.1 onwards use app deploy script (does a bit of housekeeping):
scripts/appdeploy.sh --project schemaorgae --no-promote --version=3-4-1 schemaorg.yaml
##### original deploy: gcloud app deploy --no-promote --project schemaorgae --version=3.4 schemaorg.yaml
Note: the --no-promote flage deploys the version but does not transfer traffic to it.
This can then be done when required via the appengine console, versions screen:
https://console.cloud.google.com/appengine/versions?project=schemaorgae&serviceId=default&versionssize=50
===========OLD============
Either by editing app.yaml to make sure 'application: schemaorgae' is active,
or by passing the name of the live appengine ID on the commandline.
===========OLD============
Currently @rvguha (as Steering Group chair) and @danbri (as Community Group chair and
webmaster / dogsbody) have write access to the corresponding appengine account.
===========OLD============
* e.g. update schemaorg.yaml then: appcfg.py update schemaorg.yaml -A schemaorgae
* Count to thirty. Log into appengine and you should see a new version.
* Send 5% of traffic to this version to catch any errors, then move to 100%.
===========OLD============
Note that appengine versions are visible using URLs with this pattern:
http://3-3.schemaorgae.appspot.com/docs/releases.html
5.) Related post-release tasks.
* Let the Community Group and Steering Group know.
* Blog post.
* There's a SPARQL database at DYDRA that can load NQuad dumps from
https://github.com/schemaorg/schemaorg/blob/sdo-phobos/data/releases/2.2/all-layers.nq
* Make a new per-release branch, reserve an appspot.com account for it.
* Configure GitHub with the new default branch.
See https://github.com/schemaorg/schemaorg/settings/branches
* Update issue tracker moving issues to new releases.
* Close issues that were addressed in this release.
* Make careful considered adjustment to expected release for the rest. Or
just tag them all for the next release.
* Update git release tags (TODO: we have no history of handling this yet).
For example, if we are in 'master' and 'git status' shows it in a clean state,
* git checkout -b sdo-deimos
* This tells us we can do:
* git push --set-upstream origin sdo-deimos
> To https://github.com/schemaorg/schemaorg.git
> * [new branch] sdo-deimos -> sdo-deimos
> Branch sdo-deimos set up to track remote branch sdo-deimos from origin.
TODO:
* clean up release tags
* see also: git log --pretty=fuller --since="2018-06-15"
![swh spinner](/static/img/swh-spinner.gif)
Computing file changes ...