| 7cadd4c | Boud Roukema | 21 July 2020, 16:36:11 UTC | Update default calculation checksums This commit updates the checksums for the default (high accuracy) calculations for the 21 July 2020 downloaded version of the source data files. | 21 July 2020, 16:36:11 UTC |
| 2fadf4b | Boud Roukema | 21 July 2020, 16:01:22 UTC | Printing location when downloaded input data checksum is different There are many different directory trees involved in Maneage system: the top directory, the 'reproduce/' directory and its sub-directories, '.build/' (that point to a user-defined build area), and a possibly user-defined input directory. Until now, in the case of a download checksum failure, it was not immediately obvious [1] to the user *where* the file with a failed checksum is. To clarify to the user *where* the suspicious file is now located, this commit adds a line to 'reproduce/analysis/make/download.mk' to print out this full path location: '$$unchecked' along with the expected and calculated checksums. [1] Euphemism for me spending lots of time debugging and being confused. | 21 July 2020, 16:30:47 UTC |
| 919d8bd | Boud Roukema | 21 July 2020, 15:18:22 UTC | Update WHO+WPC19CCTF data for 21 July - step 1 This commit does the first step in updating the input data files to those of 21 July (today). The verification checksums are done for the `enable_dev_override` mode in this commit. The second step will be to update the checksums from the default (higher accuracy) calculations. | 21 July 2020, 15:18:22 UTC |
| b1823f5 | Boud Roukema | 20 July 2020, 23:29:36 UTC | Tables += poisson probability; discussion + conclusion In this commit: Tables: poisson probablities are added; total counts per subsequence are removed - this is because the latter are not referred to in the text while the former are needed for frequentist interpretations. The discussion and conclusion sections have been generally reorganised, substructured, and had some media references added. The WP C19CCTF data download date is added to the system, so `verify.mk` was updated. A bash bug with testing `enable_dev_override` is fixed. | 20 July 2020, 23:29:36 UTC |
| 2bfe5e1 | Boud Roukema | 20 July 2020, 16:15:12 UTC | README-hacking.md: clarify Zenodo usage in publication checklist This commit clarifies the initial usage of Zenodo for reserving a Zenodo identifier and starting an 'unpublished' upload. Some other minor wording changes are done here. | 20 July 2020, 21:36:08 UTC |
| 2ed8a2d | Boud Roukema | 20 July 2020, 16:31:43 UTC | make dist: only archive files that are under version control Until this commit, the '$(project-package-contents)' rules in 'reproduce/analysis/make/initialize.mk' included a line to provide all contents, recursively, of the directory 'reproduce/' in the package for further distribution. This could potentially lead to the distribution of private working files that are used during development and not intended for general distribution. With this commit, only those files in 'reproduce/' and 'tex/src' that are under version control are copied to the temporary directory (that is later used for creating an archive). With this change, the archiving commands actually became more clean (we don't have to manually remove 'LOCAL.conf' or other temporary files). Extensive comments have also been added above each step to clarify each step's purpose and method. | 20 July 2020, 20:43:45 UTC |
| 0eb85e8 | Boud Roukema | 20 July 2020, 13:51:09 UTC | Update poisson.tex checksum without extra comma The previous commit fixed an unnecessary comma in the second list of countries for the abstract. This modifies the poisson.tex include file, and so the checksum for `verify.mk` needs to be updated. This is done in this commit. | 20 July 2020, 13:51:09 UTC |
| 9842b61 | Boud Roukema | 20 July 2020, 02:46:09 UTC | Figure legends; list in abstract This commit should stop the legends from overshadowing points too much in the counts curves; and should fix the extra comma that appeared in the list of countries with phi_i^{28} between 1.0 and 3.0. | 20 July 2020, 02:46:09 UTC |
| abe1dba | Boud Roukema | 20 July 2020, 01:17:31 UTC | Adapt README.md; update TODO The README.md is adapted for this project in this commit: only a few quick changes were needed. The TODO is updated. The project seems essentially ready. Only polishing is needed. | 20 July 2020, 01:17:31 UTC |
| cb32347 | Boud Roukema | 20 July 2020, 00:50:03 UTC | Merge branch 'subpoisson' of codeberg:boud/subpoisson into subpoisson | 20 July 2020, 00:50:03 UTC |
| f07c7f2 | Boud Roukema | 20 July 2020, 00:45:00 UTC | EJE/ArXiv/Zenodo/SWF submittable formats This commit adjusts the rules, especially in `reproduce/analysis/make/initialize.mk`, for making the various versions of the package available to Zenodo, to SWF, to ArXiv, and to the journal. TODO has been updated to summarise the new rule names. This is more hardwired than done modularly. Possibly the most useful line to contribute to upstream maneage is: ```` tar -cv -f - $$(git ls-files reproduce) | (cd $$dir ; tar -xv -f -) ```` so that only git-managed files - and not untracked working files - are stored in the .tar.gz or equivalent package. Texlive has `latexpand` added, so `./project configure --existing-conf` is needed with this commit. | 20 July 2020, 00:45:00 UTC |
| 8694bd8 | Boud Roukema | 19 July 2020, 21:54:59 UTC | Update output file md5sums This commit updates the output file m5sums for the verify step, for the high resolution calculation. | 19 July 2020, 21:54:59 UTC |
| 28e174f | Boud Roukema | 19 July 2020, 20:54:19 UTC | Disable enable_dev_override | 19 July 2020, 20:54:19 UTC |
| 8842dad | Boud Roukema | 19 July 2020, 20:49:06 UTC | LaTeX tidying; some content tidying Mostly minor LaTeX tidying; \usepackage[hyphenbreaks]{breakurl} was needed for long URLs. | 19 July 2020, 20:49:06 UTC |
| 5fe6a85 | Boud Roukema | 19 July 2020, 16:41:07 UTC | Reproducibility/zenodo/codeberg links; fig font This commit implements and tidies up links to reproducibilty and archiving of the source package. The counts curves fonts are made bigger. | 19 July 2020, 16:41:07 UTC |
| 66bb0f0 | Boud Roukema | 19 July 2020, 15:06:16 UTC | Remove duplicate RU and TR from C19CCTF data; reproducibility Since RU and TR can be considered to be both in Europe and Asia, they were included twice in the list of countries taken from the C19CCTF template listing countries by geographical region. This commit removes the superfluous copies of RU and TR data. Reproducibility of the plain text data files requires countries to be output in a fixed order. The python implementation of 'set' seems to consider order arbitrary (as in ordinary mathematics), and the Pool.apply_async parallel processes do not consider order important. This commit does sorting to generate outputs in a fixed order independent of CPU calculation order. | 19 July 2020, 15:06:16 UTC |
| 324e5ab | Boud Roukema | 19 July 2020, 13:11:07 UTC | Verify checksums for standard calculation In this commit, the verify checksums are calculated for either the `enable_dev_override` mode or the default production (publishable) mode, checking automatically which is currently activated. A warning is given if the user is in `enable_dev_override` mode. | 19 July 2020, 13:11:07 UTC |
| 2e48505 | Boud Roukema | 19 July 2020, 09:14:28 UTC | Start verifying the outputs This commit starts the `verify` system for output data files and output .tex files. The md5sums in this commit are for the `enable_dev_override` fast, inaccurate option and are not meant to stay in place. | 19 July 2020, 09:14:28 UTC |
| f14189f | Boud Roukema | 19 July 2020, 01:01:52 UTC | Copyright declarations; rm irrelevant files This commit adds a few copyright declarations and removes some gnuastro/ files that are irrelevant to this project. | 19 July 2020, 01:01:52 UTC |
| 5842e77 | Boud Roukema | 19 July 2020, 00:46:57 UTC | Tidy up the six publishable data files Six data files for publishing on Zenodo (and in the ArXiv source) are now placed in `.build/data-to-publish` in a moderately friendly and data lineage traceable way. The aim is that readers should be able to easily replot and reuse the data whichever way they wish. | 19 July 2020, 00:46:57 UTC |
| aaff072 | Boud Roukema | 18 July 2020, 03:13:16 UTC | Python enable_dev_override fix The variable `enable_dev_override` cannot be left undefined. Python conventions differ from `make` conventions. This commit should fix a bug caused by `enable_dev_override` being left undefined. | 18 July 2020, 03:13:16 UTC |
| 2fb8c69 | Boud Roukema | 18 July 2020, 03:06:21 UTC | Fixed rng seed values with parallel processes This commit sets the random number generator seed centrally from `subpoisson.conf` through to `subpoisson.py`, passing through to `theil_sen_robust_stderr`. Minor corrections are done in `paper.tex`. The logic of `poisson.mk` is fixed so that $(outdir) is an order-only parameter, since updating a directory should not cause targets needing that directory to be reperformed. Developer overrides are now done with a single parameter in `subpoisson.conf`, enable_dev_override. If n_cpus is left undefined, then the available number of cpus is now max(1,min(ntot-2, 20)), where ntot is the total number available. | 18 July 2020, 03:06:21 UTC |
| b243835 | Boud Roukema | 17 July 2020, 21:03:07 UTC | High resolution phi_i calculation This commit increases the phi_i calculation resolution, making the results more accurate but increasing the calculation time. If you have many threads/cores, then increasing the value of `n_cpus` will speed up the calculation significantly. During the remaining few days of continued development, these values may be switched on and off several times. Check carefully if you are concerned about seeing publishable quality values in your pdf. | 17 July 2020, 21:03:07 UTC |
| 36b72f0 | Boud Roukema | 17 July 2020, 20:39:31 UTC | Add missing script count_jumps_mod.py This commit adds `reproduce/analysis/python/count_jumps_mod.py`, which is needed to compare the numbers of jumps present in the WHO and C19CCTF data. | 17 July 2020, 20:39:31 UTC |
| 90cd9d1 | Boud Roukema | 17 July 2020, 20:26:17 UTC | Compare WHO to Wikipedia Case Count Task Force data With this commit, the WHO and Wikipedia Case Count Task Force (C19CCTF) data in the medical cases chart templates are compared to see how many disruptive day-to-day 'jumps' (jumps or drops) occur. This way, the choice of using the C19CCTF data is justified quantitatively rather than forcing the reader to wonder if the claim about data quality is true or not. The text is updated, and a figure added. A plain text file with the jump counts is added. | 17 July 2020, 20:26:17 UTC |
| c18c170 | Mohammad Akhlaghi | 17 July 2020, 12:38:02 UTC | README.md now has description of building project in Docker Docker is a "container" technology that allows an almost independent operating system run on the host. It is useful when the host OS doesn't support some features or has internal problems (for example its C library or C compiler have problems). Fortunately a Maneaged project can easily be built within a Docker image and a minimal image operating system. With this commit, a section has been added to 'README.md' to describe this process. Each step of the Dockerfile is explined, to help users that may not be too familiar with Docker, or help Docker user who are not familiar with Maneage. | 17 July 2020, 12:38:02 UTC |
| 87c03fe | Boud Roukema | 16 July 2020, 18:47:39 UTC | Minor text improvements to be done: TODO += Marius pointed out that explaining a little more - even just one sentence - what the KS actually *is* would help the reader unfamiliar with XX C statistical tests. | 16 July 2020, 18:47:39 UTC |
| 21d7314 | Boud Roukema | 16 July 2020, 03:00:58 UTC | Fix constitutional refs typo | 16 July 2020, 03:00:58 UTC |
| 2513d9a | Boud Roukema | 16 July 2020, 02:59:24 UTC | Fix cross-ref label; constitutional refs Minor fix. | 16 July 2020, 02:59:24 UTC |
| 03c4b09 | Boud Roukema | 16 July 2020, 02:44:12 UTC | Do all countries; do all 6 counts curves This commit does two minor changes for bigger calculation levels. | 16 July 2020, 02:44:12 UTC |
| 1f0bf9d | Boud Roukema | 16 July 2020, 02:38:37 UTC | First complete version This version should be the first complete version that could, in principle, be submitted, apart from any minor bugs that may have been missed. The `maneage` verification steps in `verify.mk` have not (yet?) been implemented. The first priority is to make sure that the plain text input and output data files are ready for the ArXiv package. A secondary priority will be to try to follow the `maneage` verification mechanism. See TODO for other items that could be useful to do. | 16 July 2020, 02:39:05 UTC |
| 097f8cc | Boud Roukema | 16 July 2020, 00:23:42 UTC | Lots of text changes Changes in this commit include: * EJE format changes; * general title, abstract, introduction, method improvements; * references matching the introduction updates | 16 July 2020, 00:23:42 UTC |
| 671eeb8 | Boud Roukema | 15 July 2020, 17:51:06 UTC | EJE style - sffamily, namerefs EJE uses the sans serif style for the title and section headers rather than numbers. This commit adjusts the paper style to closer match the EJE style. | 15 July 2020, 17:51:06 UTC |
| 0188c2d | Boud Roukema | 15 July 2020, 03:01:18 UTC | Improve comments in subpoisson.conf This commit aims to make the comments in `reproduce/analysis/config/subpoisson.conf` clearer. | 15 July 2020, 03:01:18 UTC |
| c0f8b84 | Boud Roukema | 15 July 2020, 02:50:52 UTC | Update abstract to better match content The automatic results in the abstract piped from the calculations and text were somewhat misleading. The abstract should make more sense with this commit. Abstract word estimate about 240 (emacs) or 210 (wc on pdf + mouse cut/paste). This commit requires at least 6 countries enabled in the `for country in country_list:` line in `subpoisson.py` and should compile OK. | 15 July 2020, 02:50:52 UTC |
| 4bebc6a | Boud Roukema | 15 July 2020, 01:05:13 UTC | Include counts curves in paper This commit includes some of the least noisy daily counts curves, and some median-phi_i counts curves, in the paper. The aim is to make the result visually clear to the reader. | 15 July 2020, 01:05:13 UTC |
| 16bbeef | Boud Roukema | 14 July 2020, 23:12:15 UTC | Poisson and phi_i consistent curves Plot n_i(j) curves for the N_plot_lowest_phi countries in each of the four cases, comparing to the 68% Poisson band, and, if phi_i > 1, to the phi_i band. Do the same thing for the N_plot_median_phi countries at the median of the phi distribution for comparison, in the same way; these should all have phi_i > 1 and show the phi_i model. | 14 July 2020, 23:12:15 UTC |
| d027060 | Boud Roukema | 14 July 2020, 17:14:56 UTC | N-day subsequence tables: choose correct column This commit corrects the column selection in `reproduce/analysis/make/poisson.mk` for the N-day subsequence tables. | 14 July 2020, 17:14:56 UTC |
| 7a645da | Boud Roukema | 14 July 2020, 14:00:12 UTC | Subsequence means and totals to tables The low-phi_i countries appear to favour 10k in either total or mean counts. This commit automates this and adds these values to the subsequence tables, and omits psi from them, since psi is of less interest. | 14 July 2020, 14:00:12 UTC |
| 4142391 | Boud Roukema | 14 July 2020, 02:28:58 UTC | Text/figs/tables tidying; simpler defaults The presentation of the article is starting to converge. This commit does several small steps in this tidying in terms of the presentation of the results that are the most relevant. The speedup default parameters are set mostly left unset, apart from the `delta_log10_*` parameters which are kept at low resolution for speed. With these two parameters set at 0.5 and 6 cpus a full calculation took about 10 minutes to run on an ordinary few-years-old desktop computer. | 14 July 2020, 02:28:58 UTC |
| cf83f3b | Boud Roukema | 14 July 2020, 01:41:33 UTC | Parallel processing: minor fixes This commit removes a testing parameter and adds a j_OK estimate. | 14 July 2020, 01:41:33 UTC |
| 9b1fb3d | Boud Roukema | 14 July 2020, 01:26:00 UTC | Parallel processing per country seems to work This commit implements python asychronous parallel processing across countries. It appears to work correctly. The parameter `n_cpus` is added to `reproduce/analysis/config/subpoisson.conf`. The speedup is significant. | 14 July 2020, 01:26:00 UTC |
| eefbc9d | Boud Roukema | 13 July 2020, 21:40:51 UTC | Solve make rules bug Until this commit, `reproduce/analysis/make/poisson.mk` had the target + prerequisite `$(outdir)/done-check-poisson: $(outdir)`. The motivation was to make sure that the directory is created if it doesn't yet exist. But the problem is that whenever a file is added to or removed from the directory, the directory last-modified timestamp is updated. Since `./project make clean-poisson` removes old output files from `$(outdir)`, this updated the timestamp of that directory. Thus, the `make` rule said that the rules for the target `$(outdir)/done-check-poisson` had to be updated, because the prerequisite (the directory) was newer than the target (the zero-byte file `done-check-poisson`). This is solved by converting the prerequisite to an `order-only` prerequisite: `$(outdir)/done-check-poisson: | $(outdir)`. This bug appears to be solved. | 13 July 2020, 21:40:51 UTC |
| d76a9e1 | Boud Roukema | 13 July 2020, 12:37:52 UTC | LaTeX source minor tidying Two minor fixes: fig references and \sloppy in data availability. | 13 July 2020, 12:37:52 UTC |
| 78c31e3 | Boud Roukema | 13 July 2020, 03:20:17 UTC | Very rough discussion + conclusion | 13 July 2020, 03:20:17 UTC |
| 0324b06 | Boud Roukema | 13 July 2020, 02:12:02 UTC | Fix minor LaTeX bug Fix in newcommand in `reproduce/analysis/make/download.mk`. | 13 July 2020, 02:12:02 UTC |
| 29f9941 | Boud Roukema | 13 July 2020, 01:17:08 UTC | Centralise fixed vs clock rng seed With this commit, the boolean parameter `fixed_rng_seed` in `reproduce/analysis/config/subpoisson.conf` decides whether the calculations for the paper should be run with a fixed pseudo-random number generator. For reproducibility, the default is `True`. For checking that the results are not especially sensitive to the seed, `fixed_rng_seed` should be set to `False`. | 13 July 2020, 01:17:08 UTC |
| 37ba3d8 | Boud Roukema | 12 July 2020, 22:57:00 UTC | s/WHO/Wikipedia medical cases chart/g This commit updates the text from WHO to Wikipedia medical cases chart data, leaving the WHO bugs as an unfortunate problem. | 12 July 2020, 22:57:00 UTC |
| 05fbba8 | Boud Roukema | 12 July 2020, 22:07:36 UTC | Fix directory for input data - not output dir The input data directory is not the output data directory. This commit should fix that in `reproduce/analysis/make/poisson.mk`. | 12 July 2020, 22:07:36 UTC |
| 48d5f9c | Boud Roukema | 12 July 2020, 22:01:05 UTC | Try to fix make rule for medical cases file This commit hopefully fixes a missing target in `reproduce/analysis/make/download.mk` - for the Wikipedia medical cases file. The `TODO` file is also updated. | 12 July 2020, 22:01:05 UTC |
| bcb11b5 | Boud Roukema | 12 July 2020, 21:43:56 UTC | Switch to Wikipedia medical cases charts This commit seems to work correctly for the Wikipedia medical cases charts, which are better curated data than the WHO official data. Minor title improvement. | 12 July 2020, 21:43:56 UTC |
| 779035a | Boud Roukema | 12 July 2020, 18:32:41 UTC | Add Wikipedia data to repository In previous commits, obvious errors in the WHO national daily SARS-CoV-2 data were partially corrected using the `replace_pairs` algorithm. This is not an ideal data curation method, since it requires guesswork and adds some noise to the data. With this commit, a script to manually download the Wikipedia `medical cases chart` data, along with the data as of today, are added. The `make` rules in this commit have not yet been tested. | 12 July 2020, 18:32:41 UTC |
| 0ba49f9 | Boud Roukema | 12 July 2020, 04:13:11 UTC | Plots of least noisy counts curves Counts curves for the least noisy countries are partly implemented in this commit. | 12 July 2020, 04:13:11 UTC |
| c271289 | Boud Roukema | 12 July 2020, 00:53:49 UTC | Subsequence figures, tables, start dates The three types of subsequences (28, 14, 7 days) appear to work correctly. Figures of \psi_i and tables including start dates are given. | 12 July 2020, 00:53:49 UTC |
| e067f5d | Boud Roukema | 11 July 2020, 23:16:56 UTC | Merge branch 'subpoisson' of codeberg:boud/subpoisson into subpoisson | 11 July 2020, 23:16:56 UTC |
| eb2b074 | Boud Roukema | 11 July 2020, 23:15:03 UTC | Subsequence starting figs+tables This development-step commit starts to add figures and tables for the subsequences. | 11 July 2020, 23:15:03 UTC |
| 43ebb7c | Boud Roukema | 11 July 2020, 22:03:42 UTC | Fix EJE format patch file error The patch file patched against a patched file instead of the original file. This commit fixes that and should work. | 11 July 2020, 22:03:42 UTC |
| 5833232 | Boud Roukema | 11 July 2020, 20:26:42 UTC | Subseq start; covid-19 refs; EJE style In this commit, the routine for searching for optimal subsequences is added. Some COVID-19 references are added. Some improvements to bring the citation style closer to EJE format are done, though it doesn't quite match exactly. | 11 July 2020, 20:26:42 UTC |
| e089981 | Boud Roukema | 11 July 2020, 16:22:54 UTC | Add psi figure; fix error bars This commit adds the psi_N figure to the paper and corrects the error bar, whose values didn't follow the correct matplotlib convention. | 11 July 2020, 16:22:54 UTC |
| 2471624 | Boud Roukema | 11 July 2020, 15:31:17 UTC | Table improvements; phi refinement In this commit, the first table is improved in style. The phi uncertainty column is sacrificed in favour of the raw Poisson probability, because some countries have values that are not rejections. A one-stage refinement in the accuracy of estimating phi is added. More powerful algorithms could be used for further speedup. | 11 July 2020, 15:31:17 UTC |
| bc5df79 | Boud Roukema | 11 July 2020, 04:14:10 UTC | Make rules, plot tick label annoyance In this commit, some more `clean-*` rules are added in `reproduce/analysis/make/paper.mk` and documented in `project`, for the `./project --help` command. The difficulty in convincing `matplotlib` to label logarithmic axes nicely like `plotutils` does is briefly commented in `reproduce/analysis/python/subpoisson.py`. | 11 July 2020, 04:14:10 UTC |
| fc49776 | Boud Roukema | 11 July 2020, 01:57:48 UTC | Add table of lowest phi, psi countries In this commit, the Theil-Sen fit to phi_i(N_i) is plotted and explained in the text. TODO: The psi_i(N_i) plot is started but not ready, and not yet for inclusion in `paper.tex`. A table with the key country criteria for the lowest values of phi and psi is included. The default values in this commit (and other recent ones) in `reproduce/analysis/config/subpoisson.conf` are for fast development and are inaccurate. Stronger values, for slower but more accurate calculations, are needed for proper results. TODO: The subsequence results have not yet been started. A table of the lowest phi and psi countries | 11 July 2020, 01:57:48 UTC |
| fbcbde8 | Boud Roukema | 10 July 2020, 22:49:12 UTC | Start on results section; first two figures. With this commit, the results section with two figures has been started. A plain text file `.build/data-to-publish/phi_N.dat` is now created automatically. This will have to be included in ArXiv and maybe the publisher version of the data - with the appropriate copyright declarations. Plotutils is removed from `reproduce/software/config/TARGETS.conf`. Copyright declarations are added to the .py scripts. | 10 July 2020, 22:49:12 UTC |
| b4ca5b5 | Boud Roukema | 10 July 2020, 19:04:47 UTC | Add plotutils to maneage This commit adds the Debian 2.6-11 patched version of plotutils 2.6 . Debian distributes the original and patched versions of software separately - it's up to the user to apply the patches if s/he wishes to compile from source. To satisfy the maneage system, until plotutils is added to the maneage zenodo archive, this commit uses a codeberg git repository which provides a downloadable .tar.gz file. A third parameter `tarball_download_name="$(strip $(3))";` is added to the `make` level `import-source` macro in `reproduce/software/make/build-rules.mk` in order to allow for non-standard URLs. | 10 July 2020, 19:04:47 UTC |
| 5218ffb | Boud Roukema | 10 July 2020, 15:12:56 UTC | Analysis section of method An initial rough draft of the analysis section is added in this commit, including the subsequences, and an appropriate Bonferonni-Sidak caveat. The names and indices of the variables in the science text have been made more specific. Consistency of these may still need to be checked. | 10 July 2020, 15:12:56 UTC |
| d4ead90 | Boud Roukema | 10 July 2020, 02:44:17 UTC | Theil-Sen module; min_days A Theil-Sen python module is added for robust linear fitting, adapted from my octave routine for this. The min_days starting sequence is now written in the paper.tex text. | 10 July 2020, 02:44:17 UTC |
| 1ca644f | Boud Roukema | 10 July 2020, 00:38:22 UTC | Rewrite introduction, method 2.1; simpler thresholds; arXiv refs This commit rewrites the introduction to better match the abstract. The data treatment part of the method (2.1) is updated. The threshold is set to a single threshold for both starting and ending, but the minimum days requirement is also set at the beginning, to avoid initial fluctuations artificially cutting off a sequence. Refs for burstiness are added. A patch to the EJE/Springer .bst file is applied for ArXiv IDs. Irrelevant maneage template 'delete-me' files are deleted. | 10 July 2020, 00:38:22 UTC |
| c852727 | Marius Peper | 07 July 2020, 20:58:07 UTC | Fixed typo that lead to crash when building healpy Until now, if a project needed the healpy software package, Maneage would crash with the following error message (abridged for full name in build directory). This was caused by a typo in the version of 'healpix' (the dependency of 'healpy'). make: *** No rule to make target '.../version-info/proglib/healpix-' With this commit, the typo in line 334 of 'python.mk' is fixed, so that when '$(ipydir)/healpy-$(healpy-version)' gets called it correctly searches for a rule to make '$(ibidir)/healpix-$(healpix-version)'. | 07 July 2020, 21:34:04 UTC |
| e1f10ac | Mohammad Akhlaghi | 07 July 2020, 12:58:04 UTC | Project distribution tarball can account for no PDFs in tex/tikz Until now the './project make dist' command implicitly assumed that the 'tex/tikz' directory always contains PDF files (because of the 'cp tex/tikz/*.pdf $$dir/tex/tikz' line). This was annoying for projects that don't use TiKZ or PGFPlots to generate their plots, and they had to manually comment this line. With this commit a check has been placed to see if any PDF files exist in there at all. If there aren't PDF files, the 'cp' command above is ignored. | 07 July 2020, 12:58:04 UTC |
| 32a6ea7 | Boud Roukema | 05 July 2020, 19:03:26 UTC | Add missing EJE patch file This commit adds the patch file `reproduce/analysis/patches/20200703_EJE_abstract.patch` which was missing. | 05 July 2020, 19:03:26 UTC |
| 0e4d4b3 | Mohammad Akhlaghi | 05 July 2020, 18:17:05 UTC | Configure script prefers clang for macOS systems In the previous commit (Commit 1bc00c9: Only using clang in macOS systems that also have GCC) we set the used C compiler for high-level programs to be 'clang' on macOS systems. But I forgot to do the same kind of change in the configure script (to prefer 'clang' when we are testing for a C compiler on the host). With this commit, the compiler checking phases of the configure script have been improved, so on macOS systems, we now first search for 'clang', then search for 'gcc'. While doing this, I also noticed that the 'rpath' checking command was done before we actually define 'instdir'!!! So in effect, the 'rpath' directory was being set to '/lib'! So with this commit, this test has been taken to after defining 'instdir'. | 05 July 2020, 18:17:05 UTC |
| 5d97210 | Mohammad Akhlaghi | 05 July 2020, 15:18:45 UTC | Removing possibly existing paper.bbl before remaking it Until now, when the bibliography file ('paper.bbl') had a LaTeX-related error (for example the journal name was a LaTeX macro that isn't defined), the first 'pdflatex' command that is run before 'biber' would crash, not allowing the project to reach 'biber'. So the user would have to manually remove 'paper.bbl' before running './project make'. With this commit, we remove any possibly existing 'paper.bbl' file before rebuilding it. Generally, this also helps in keeping things clean during the generation of the new bibliography. This bug was found by Mahdieh Nabavi. | 05 July 2020, 15:18:45 UTC |
| 1bc00c9 | Mohammad Akhlaghi | 02 July 2020, 16:56:46 UTC | Only using clang in macOS systems that also have GCC Until now, when Maneage was built on a macOS that had both a clang and GCC, we would make links to both. But this cause many conflicts in some high-level programs (for example Numpy and etc, all the programs where we have explicity set 'export CC=clang' before the build recipe). This happens because the GCC that is built on a macOS isn't complete for some operations. To fix this problem, when we are on a macOS, we explicity set 'gcc' to point to 'clang' and 'g++' to point to 'clang++'. We also don't link to the host's C-preprocessor ('cpp') on macOS systems because this is only a GNU feature and using the GNU CPP is also known to have some basic problems. For example this was reported by Mahdieh Nabavi (which was the main trigger for this work): ld: Symbol not found: ___keymgr_global Referenced from: /Users/Mahdieh/build/software/installed/bin/cpp Expected in: /usr/lib/libSystem.B.dylib Also, to avoid linking to another link on the host tools (in the 'makelink' function of 'basic.mk'), we are now using 'realpath'. | 05 July 2020, 14:46:32 UTC |
| cedea21 | Mohammad Akhlaghi | 04 July 2020, 21:15:45 UTC | Commit hash of Maneage branch used to build project as LaTeX macro To help in the documentation, the Git hash of the Maneage branch commit that the project has most recently merged with (or branched from) is now also provided as a LaTeX macro ('\maneageversion'). It is calculated in 'reproduce/analysis/make/initialize.mk' (in the recipe to 'initialize.tex'). | 04 July 2020, 21:15:45 UTC |
| 6322d4f | Mohammad Akhlaghi | 04 July 2020, 20:48:10 UTC | Better names and comments in INPUTS.conf Until now, the dataset's configuration names had a 'WFPC2' prefix. But this very alien to anyone that is not familiar with the history of the Hubble Space Telescope (the camera is no longer used! Its just used here since its one of the standard FITS files from the FITS standard webpage). With this commit the variable names have been modified to be more readable and clear (having a 'DEMO-' prefix). Also the comments of 'INPUTS.conf' (describing the purpose of each variable) were edited and made more clear. | 04 July 2020, 20:58:05 UTC |
| 8e13095 | Boud Roukema | 04 July 2020, 02:37:39 UTC | Abstract: complete update of method + early results This commit changes many files. It should generate a reasonable looking pdf for two reasonable choices of the start (and stop) thresholds, and give the basic results in the abstract. Runtime is a few minutes. Minor *.mk bug: if the pdf is not fully made, then `./project make` sometimes causes the `poisson` rule to be run too. | 04 July 2020, 02:37:39 UTC |
| 27e9ade | Mohammad Akhlaghi | 04 July 2020, 01:16:16 UTC | Some command line messages of ./project didn't mention shell Until now, the 'shell' mode of the './project' script was missing in the top output of './project --help' and in the error message printed when no operation was given, or when more than one operation was given. This is now corrected. | 04 July 2020, 01:19:37 UTC |
| 318b73e | Mohammad Akhlaghi | 04 July 2020, 00:36:22 UTC | Citing Maneage paper in acknowledgments In the previous commit, the modified abstract of the acknowledgments only included the URL of Maneage, but its more formal to cite the Maneage paper, the URL is already present in the paper. | 04 July 2020, 00:36:22 UTC |
| 5c91a50 | Mohammad Akhlaghi | 03 July 2020, 23:12:07 UTC | Corrected Acknowledgments section in default paper Until now, the acknowledgment section didn't contain the new name of Maneage and it also included an acknowledgment of Gnuastro (which is not appropriate for a general project which may not use Gnuastro). With this commit this is fixed. | 03 July 2020, 23:15:53 UTC |
| 24de267 | Boud Roukema | 03 July 2020, 16:48:56 UTC | EJE style This commit does many changes to generate the first version which is more or less in Springer/EJE style, using the Springer LaTeX files. The abstract is only one column wide, in contrast to official EJE publications. This is a minor problem only - though it would look nice prior to submission to match the official style. | 03 July 2020, 16:48:56 UTC |
| 5ab9d71 | Boud Roukema | 03 July 2020, 10:46:06 UTC | Remove tikz line from rule for paper.pdf | 03 July 2020, 10:46:06 UTC |
| 8ea2ad8 | Boud Roukema | 03 July 2020, 10:37:17 UTC | Add missing preamble-eje.tex The parameter file for EJE bibtexing was missing. This is still a stub. | 03 July 2020, 10:37:17 UTC |
| 905e8dd | Boud Roukema | 03 July 2020, 09:10:51 UTC | Clustering generalisation of Poisson test; KS; possible refs This commit adds (temporarily: replaces) the 1 (2) sigma test by a more general test - find the best fit "cluster" size where the infection count is assumed to follow a Poisson distribution after both the model mean and the observed value are divided by the value of `cluster`. A Kolmogorov-Smirnov test is used to check if the full resulting cumulative probability distribution is consistent with a uniform distribution, and to optimise over this to find the most likely value of `cluster` (least distant resulting distribution). Some references that somewhat relate to the topic are added to `TODO.refs` . | 03 July 2020, 09:10:51 UTC |
| a9cc330 | Boud Roukema | 02 July 2020, 06:17:18 UTC | + arXiv:1908.06822 ILMs | 02 July 2020, 06:17:18 UTC |
| 12c8ceb | Boud Roukema | 02 July 2020, 06:08:35 UTC | TODO.refs - recent references that may be useful new file: TODO.refs | 02 July 2020, 06:08:35 UTC |
| e1096f0 | Boud Roukema | 01 July 2020, 23:36:18 UTC | TODO: allow toggling of reproducible random number generator Python (numpy) has a setup for not only serial, but also parallel reproducible rng generation based on a fixed seed. This is useful for reproducibility, but a bad idea when developing the calculations, since the difference between random fluctuations and real properties of the data is hidden. Before submitting the paper, the reproducible option should be made the default. This is added as a TODO item. | 01 July 2020, 23:36:18 UTC |
| deba07c | Mohammad Akhlaghi | 01 July 2020, 17:14:44 UTC | Properly accounting for space characters in host's PATH Until now, when reading the host's PATH environment variable we weren't accounting for directory names with a space character. This was most prominently visible in the 'low-level-links' step where we put links to some core system components into the project's build directory (mainly for prorietary systems like macOS). To address the problem, double quotations have been placed around the part that we extract 'ccache' from the PATH, and the part where we make the symbolic link. In the process the comments above 'makelink' were made more clear and 'low-level-links' now depends on 'grep' (which is the highest-level program it uses). This bug was reported by Mahdieh Navabi. | 01 July 2020, 17:18:53 UTC |
| 122aaac | Boud Roukema | 01 July 2020, 16:42:19 UTC | Initial version of get_noise.py An initial version of a Poisson-based statistic is implemented in `reproduce/analysis/python/get_noise.py`. Together with corresponding updates especially in the main calling program, `reproduce/analysis/python/subpoisson.py`, some preliminary results appear: RS, BY, AE, AZ appear to form a distribution of points sharply separated from those of other countries/territories, in the sense of being "too normal". | 01 July 2020, 16:42:19 UTC |
| aee6d61 | Raul Infante-Sainz | 01 July 2020, 12:19:23 UTC | Minor typo corrected in referencing Libidn Until this commit, once Libidn was installed, insted of its own name and version, the name and version of Libjpeg were saved (in the target if Libidn). This robably come from a copy/paste of the rule. With this commit, this minor bug has been corrected. I also added my name as an author of `reproduce/software/make/xorg.mk' Makefile since I added some code there. | 01 July 2020, 15:06:06 UTC |
| 953cdaa | Boud Roukema | 01 July 2020, 14:28:12 UTC | The replace_pairs function works This commit provides a unit-testable version of `reproduce/analysis/python/replace_pairs_mod.py`, which is now called by `reproduce/analysis/python/subpoisson.py`. The latter also now reports the total number of counts, countries, and negative counts, and feeds them to the paper. | 01 July 2020, 14:28:12 UTC |
| 2da14c9 | Boud Roukema | 01 July 2020, 10:40:39 UTC | Describe method; start fixing pairs This commit adds and modifies many files in relation to describing the method and feeding through analysis variables from the configure file `reproduce/analysis/config/subpoisson.conf` through to `paper.tex` using the `maneage` mechanisms of automatic consistency. Part of the transition to EJE journal style is implemented here, but is incomplete. An algorithm to replace pairs (0,m2) by (k,m2-k) where k is a Poisson random variable drawn from a distribution of mean m2/2 is partly written but still incomplete: `reproduce/analysis/python/replace_pairs_mod.py` | 01 July 2020, 10:40:39 UTC |
| 9d7d04a | Boud Roukema | 01 July 2020, 08:19:20 UTC | Allow for single non-report or low fluctuation days Given the range of time zones in the world, and the need for WHO to quantise the daily reports per 24-hour period, it is unsurprising that sometimes two days' worth of detected infections may be listed by WHO as a sequence of two days in which all the infections are allocated to the second of the two days. It is also reasonable to avoid having the cutoff date for the analysable sequence to be too sensitive to individual days of low fluctuations. For both these reasons, this commit introduces the parameter stop_days, for which a value of 2 is reasonable: the analysable sequence is only stopped after *stop_days* consecutive days of the daily infection rate dropping below *threshold_stop*. This commit also tidies python conventions for filenames/def's. | 01 July 2020, 08:19:35 UTC |
| ea7bfaa | Boud Roukema | 01 July 2020, 02:24:29 UTC | Analysis: read data; find valid range Make rules are added in this commit to check a routine `reproduce/analysis/python/find_high_enough_range.py` for finding a valid subrange of data for a given country for reasonable Poisson errors to be viable. The main analysis routine is started, reading in the input file and calling the find_high_enough_range routine. Lines in `reproduce/analysis/make/poisson.mk` do some minor cleaning: commas in BQ and PS are changed to semi-colons, and the international conveyances `country` is given the fake two-letter acronym of XX. Help info is added to `project`, so that `./project --help` lists the `make` sub-commands `clean-poisson`, `check-poisson`, and `poisson` with brief descriptions of what they do. | 01 July 2020, 02:24:29 UTC |
| 6339d46 | Boud Roukema | 30 June 2020, 22:19:47 UTC | Start introduction, general paper.tex components; +scipy Commit 505eaac4 *did* correctly solve the bug reported in commit 225547c9187 - this shifted to the bottom of the TODO file in the 'done' section (which will probably later be removed to avoid distracting users). Remove stencil components of `paper.tex`: add affiliations, introduction first two paragraphs; acknowledgments; start references (import from general .bib file from astro project). TODO list: EJE journal style remains on the TODO list. The basic python science tools (scipy, numpy, matplotlib) are added to `reproduce/software/config/TARGETS.conf` in this commit. | 30 June 2020, 22:19:47 UTC |
| db31bce | Mohammad Akhlaghi | 30 June 2020, 16:00:25 UTC | Proper deletion of util-linux source after successfully building it After recently adding util-linux to Maneage build-tree, we had forgot to delete the unpacked and built source directory after it was installed! This has been corrected with this commit. | 30 June 2020, 16:00:25 UTC |
| 505eaac | Boud Roukema | 30 June 2020, 15:13:13 UTC | This commit might fix the bug reported in 225547c9187 This fix - restoring tabs in `reproduce/analysis/make/top-make.mk` which had been accidentally replaced by spaces - may solve the bug reported in commit 225547c9187, but still has to be tested in a clean setting. | 30 June 2020, 15:13:13 UTC |
| f32b509 | Mohammad Akhlaghi | 30 June 2020, 14:45:17 UTC | Entered data and software directories stored as absolute addresses Until now, when the user specified an input and software directory, the raw string they entered was used. But when this string was a relative location, this could be problematic in general scenarios. With this commit, the same function that finds the absolute location of the build directory is used to find the absolute address of the data and software directories. | 30 June 2020, 14:45:17 UTC |
| 3c05ca1 | Mohammad Akhlaghi | 30 June 2020, 14:26:04 UTC | The distclean target accounts for non-existance of git hooks Until now, when the user wanted to complete remove all built files (including software), the './project make distclean' command would fail if the git hooks weren't installed. They are present when the project's configuration has been successfully finished, but this bug can happen when trying to re-do an incomplete build. With this commit, this is fixed by adding an '-f' has been added before the 'rm' command for the Git hooks. | 30 June 2020, 14:26:04 UTC |
| 225547c | Boud Roukema | 30 June 2020, 13:07:24 UTC | Do not try this commit from scratch Commit 5d2839f (and this one, with only an edit to TODO) gives an infinite loop on the initial install. Do not waste CPU power and internet bandwidth: do not try it except if you wish to study the bug! | 30 June 2020, 13:07:24 UTC |
| 9ddff8b | Mohammad Akhlaghi | 29 June 2020, 00:11:21 UTC | Core Xorg libraries necessary for Ghostscript now included Until now, in order to build Ghostscript, the project used the host's Xorg libraries. This was because we hadn't yet added the necessary build rules for them. With this commit, the instructions to build the necessary Xorg libraries for Ghostscript have also been added. Also, the shared Ghostscript library has been built with this commit and two sets of standard fonts are also included, setting us on the path to build TeXLive from source later. This task was done with the help and support of Raul Infante-Sainz. | 30 June 2020, 02:17:17 UTC |
