sort by:
Revision Author Date Message Commit Date
72242ca Add dist-journal option for packaging This commit adds a 'dist-journal' .tar.gz packaging option in `reproduce/analysis/make/initialize.mk` and solves a LaTeX/bibTeX bug related to escaping of '%'. 16 May 2021, 21:11:25 UTC
2e07534 More tidying; update Zenodo ID This commit does some more minor tidying (e.g. ISO 3166-1 codes are now listed in the captions of Figs 8--10). A new Zenodo ID that is now reserved for this paper replaces the old one. The SWH ID - the full git hash of the commit of interest - is that of the previous commit. 16 May 2021, 14:23:32 UTC
27ac91a Smoothing of text This commit makes many small changes that remove some typos, clarify some text, and tidy up some parts of the discussion that were not fully updated from the July 2020 data analysis (now we have May 2021). This version should be ready for submission, apart from needing an updated Zenodo ID. 16 May 2021, 01:23:49 UTC
81da1ae Merge branch 'maneage' into postreferee_fixes This commit should compile correctly - it should merge upstream Maneage commit 5a6e6e6 into the postreferee_fixes branch of subpoisson. 10 May 2021, 20:03:31 UTC
ba62f48 Minor fix - pfi_correlations This commit makes two lines of changes for the pfi output files for the jhu case. 10 May 2021, 15:11:57 UTC
91b0452 Initial freeze of new verification md5sums This commit adds an initial set of verification md5sums for the May 2021 data set. The JHU AIC/BIC and PFI tables are also added to the official output datasets and to the appendix for completeness (it happens that the JHU data find a stronger correlation between the RSF PFI and the clustering indices than the WP C19CCTF data do, so the C19CCTF results as presented are conservative). 10 May 2021, 12:22:13 UTC
f3d2469 Update discussion This commit updates the discussion following the changes made to satisfy the referees, in particular the update to new data, i.e. to the multiple waves of the epidemic. The conclusion and abstract are also updated in this commit. 10 May 2021, 03:24:07 UTC
99b38d6 RSF/PFI subsection This commit shifts the RSF PFI results to the results section, since it is more significant than just a discussion. Some discussion about it *is* also needed. 09 May 2021, 23:17:11 UTC
dc96691 Start updating results/discussion/abstract Updating the data from July 2020 to May 2021, as well as including the recommendations from the reviewers, requires some changes in the description of the results, as well updating the discussion, conclusion and abstract. This commit starts this updating process by updating the results section and the corresponding part of the abstract. Next = discussion... 09 May 2021, 02:11:25 UTC
ce75420 Minor adjustments; clustering threshold This commit fixes a few minor bugs and introduces the clustering_threshold for checking if any countries with low clustering fail to be modelled by the phi_i' model; those with high clustering help understand the overall population, but are not the main question of interest in the paper. The parameter is added to: `reproduce/analysis/config/subpoisson.conf` 07 May 2021, 21:12:58 UTC
877a917 Merge branch 'postreferee_fixes' of codeberg:boud/subpoisson into postreferee_fixes 06 May 2021, 20:35:53 UTC
0fdb3c1 Reviewer 2 point 6: update the data to May 2021 Reviewer 2 literally asked in point 6 if this method applies to "multiple epidemic waves". I interpret this as a request to update to the current primary data set (WP C19CCTF) and supplementary data sets (WHO, JHU) from early July 2021 to (now) early May 2021 (i.e. from about 3 months' analysable data for many countries to about a year's data for many more countries). This commit updates to 6 May 2021 in terms of upload scripts, sha512sum checksums and WP C19CCTF and JHU data provided in plain text column format. The results are largely compatible with the earlier results. 06 May 2021, 20:28:58 UTC
da3ec37 Add Kendall vs Spearman reference CrouxDehon2010 This commit adds the reference Croux+Dehon2010, which explains why Kendall tau is moderately better than Spearman rho, for several statistical reasons. 06 May 2021, 00:19:56 UTC
075987a Minor tidying This commit adds a missing hline in a table and bold-red-highlights the column headers for new columns in Tables 1 and 7 to make sure that the reviewers+editor are aware that these are new. 05 May 2021, 10:08:00 UTC
087f80e Reviewer 3 Point 1 RSF PFI vs psi_i This commit starts to include a table, figures and text to continue on Reviewer 3's point 1 - the need for non-parametric analysis. Kendall's tau test indeed shows a highly significant ranking correlation between lack of media freedom and lack of noise in the daily SARS-CoV-2 counts. This gives strong credibility to the hypothesis of artificial intervention in the official counts of several countries in association with low levels of media freedom. It is difficult to imagine other ways in which a lack of media freedom would have an epidemiological effect. The text is still only written very roughly, and this commit may still have bugs. 05 May 2021, 03:37:57 UTC
46f5719 Add missing script rsf_pfi.py This commit adds the script an initial draft of the script: `reproduce/analysis/python/rsf_pfi.py` 04 May 2021, 21:31:48 UTC
c9cfd67 Reviewer 3 point 1 - use non-parametric statistics This commit starts responding to Reviewer 3's point 1 about using non-parametric statistics. The Kendall tau ranking correlation is introduced to compare the results of the paper with the RSF Press Freedom Index (PFI). So far this commit (not yet fully tested) should confirm that the local copy of the RSF 2020 PFI data satisfies the expected sha512sum and make some plots and calculate Kendall tau. These have not yet been integrated into the narrative - the text. 04 May 2021, 21:15:25 UTC
2cd5d8b Reviewer 3 point 2 This commit adds some text to the discussion with Reviewer 3's suggestion of using an autoregressive model. It's not clear how this would be relevant to the current analysis, but it can't hurt to speculate about why it might be interesting for followup work by the reader. 04 May 2021, 01:17:48 UTC
da671d5 Reviewer 2 point 10 This commit responds to Reviewer 2's unnumbered summary comment "10": a caveat subsection is added to the discussion section; and the numbers of Poisson and super-Poisson rejections at the 0.01 and 0.05 levels are quantified in 3.2.1 04 May 2021, 00:35:58 UTC
e8aec83 Tidy up new figures The new figures make more sense square than non-square. The absence of Algeria from the primary versus negative binomial analysis has to be mentioned. These and related issues are mostly fixed in this commit. (TODO: The values for Algeria still need to be fed through from the analysis through to paper.tex.) 03 May 2021, 14:02:11 UTC
2c0100d Run plot_alternatives.py in make sequence This commit aims to insert 'plot_alternatives.py' into the 'make' sequence in `reproduce/analysis/make/poisson.mk` and highlights the captions of the new figures. 03 May 2021, 04:08:09 UTC
bd64a37 Reviewer 2 point 7 graphs, results This commit adds graphs for Reviewer 2's point 7 and discusses the results. 03 May 2021, 03:24:24 UTC
a8f02d0 Minor edits: weekly dip Minor edits for the text describing the weekly dip. 30 April 2021, 20:20:40 UTC
bd296fc Debug 2-day, 3-day orderings This commit (hopefully) fixes the ordering of printing out the 2-day and 3-day results, with -99.9 as a dummy value to be ignored for countries for which the grouped data set is too small. 30 April 2021, 16:25:51 UTC
0e42122 Reviewer 2 point 7 This commit adds the code to switch from daily counts to 2-day or 3-day counts, along the lines of Reviwer 2's point 7. 29 April 2021, 23:38:30 UTC
4a01bd9 Reviewer 2 Point 5 tidy up This commit should tidy up the response to point 5 of Reviewer 2 - a weekly dip index. 29 April 2021, 20:31:07 UTC
6bb900f Reviewer 2 point 5: weekly cycle This commit adds an analysis of the daily counts to check for dips on weekends, as requested by Reviewer 2 in point 5. 29 April 2021, 03:09:20 UTC
4e05411 Add alt models to Table 1; expand ISO acronyms This commit expands ISO 3166-1 acronyms as requested by Reviewer 2 in point 8; and adds the results of the alternative models to Table 1. 28 April 2021, 00:45:38 UTC
a80b43d Minor text edits, adding refs Minor: this commit adds a few statistical literature references, acknowledges the help from the reviewers, highlights new references as changed in this version, and changes R0 to R_0. 27 April 2021, 20:55:19 UTC
f10686e Reviewer 2: point 1 done This commit includes the calculations for point 1 of Reviewer 2 and starts dealing with model comparison with the AIC + BIC, responding to some degree to all three reviewers. The phi_i' model appears preferred based on the small set of countries used for fast testing of the script. 26 April 2021, 22:35:15 UTC
caa94bc Minor tidying Minor tidying of the text update from a few minutes ago. 26 April 2021, 01:02:12 UTC
85e85d5 Reviewer 2: start explaining alternatives This commit starts adding text to explain updates in response to Reviewer 2. The use of logarithmic medians instead of linear medians is added as an optional toggle in `reproduce/analysis/python/get_noise.py`. The negative binomial method is formalised as a more restrictive alternative to the phi_i' method. The AIC and BIC are briefly defined in the text for model selection. TODO: The top-level coding in subpoisson.py for looping through each alternative, calculating the AIC+BIC, and putting the new data in appropriate tables remain to be done. Several other items suggested by Reviewer 2 remain to be done. 26 April 2021, 00:33:22 UTC
5a6e6e6 README.md: edited steps to only build software env in Docker image Until now, while the series of steps mentioned in 'README.md' were complete, they had some implicit thing in them that made it a little hard to run as a checklist (the commands to do some basic things weren't included). Also, it was recommending to run a long 'docker run ...' command, which wasn't too user friendly. With this commit, the series of steps is now a complete checklist, containing every step. Also, the checklist now recommends putting the long 'docker run' command inside a script called 'docker-run' that will also do a 'sudo' internally (thus making things very easy for a first-time user). Also, since the 'docker-run' script contains host OS-specific directory names, it should not be under control, so it has been added to the '.gitignore' file in case users decide to keep this same name (which is recommended). 25 April 2021, 22:59:41 UTC
476143f Reviewer 1: text updates This commit mostly edits the introduction, in response to Reviewer 1's requests to better explain the motivation for the choice of distributions and to consider an overdispersed distribution to add value to the analysis. The method section needs to be updated to explain the negative binomial method, which is now in the script `reproduce/analysis/python/get_noise.py`; the results need to be updated; and discussion and conclusions will need to be updated in turn. 22 April 2021, 01:11:12 UTC
6e4ec9a IMPORTANT: print-general-metadata new name for print-copyright Summary: - Use the new name of this variable in your Makefiles. - In 'metadata.conf', remove fixed URL prefixes for DOIs ('https://doi.org/') or arXiv ('https://arxiv.org/abs'). Until now, the Make variable that would print the general metadata (of whole project) into each to-be-published dataset was called 'print-copyright'! But it now does much more than simply printing the copyright, it will also print a lot of metadata like arXiv ID, Zenodo DOI and etc into plain-text outputs. The out-dated name could thus be misleading and cause confusions. With this commit, the variable is therefore called 'print-general-metadata'. After merging your project with the Maneage branch, please replace any usage of 'print-copyright' to 'print-general-metadata'. Also with this commit, 'README-hacking.md' mentions 'metadata.conf' and 'print-general-metadata' in the "Publication checklist" section and reminds you to keep the first up to date, and use the second in your to-be-published datasets. 17 April 2021, 03:31:31 UTC
3d5ace2 Negative binomial seems to work With this commit, the negative binomial model as a non-intuitive alternative to the phi' model is nearly ready. It does not (yet) output correctly to the tables for the paper. 14 April 2021, 01:52:31 UTC
ededbdb Start to implement neg binomial This commit starts to implement the negative binomial distribution. It is currently in a test phase and unlikely to work correctly. 13 April 2021, 01:30:30 UTC
182f0e3 Text changes for referees This commit implements a few text changes to start responding to reviewers' concerns; fixes the URL of the Abdi+ reference; the main Maneage reference is partially updated (DOI + in press). 12 April 2021, 21:19:09 UTC
20b6273 Configuration: corrected check of group name When built in 'group' mode, the write permissions of all created files will be activated for a certain group of users in the host operating system. The user specifies the name of the group with the '--group' option at configure time. At the very start, the './project' script checks to see if the given group name actually exists or not (to avoid hard-to-debug errors popping up later). Until now, the checking 'sg' command (that was used to build the project with group-writable permissions) would always fail due to the excessive number of redirections. Therefore, it would always print the error message and abort. With this commit, the output of 'sg' is no longer re-directed (which also helps users in debuggin). If the group does actually exist, it will just print a small statement saying so, and if it fails, the error message is printed. This fixed the problem, allowing maneage to be built in group-mode. I also noticed that the variable name keeping the group name ('reproducible_paper_group_name') used the old name for the project (which was "Reproducible paper template"! So it has been changed/corrected to 'maneage_group_name'. 28 March 2021, 11:55:03 UTC
611c2f1 Initialization: removed other Gnuastro-specific features In the previous commit, some Gnuastro-specific initializations were removed but a few more cases remained that are removed with this commit. 26 March 2021, 20:39:48 UTC
ac8890d ./project: unused --minmapsize option is removed Until now, the './project' script included an '--minmapsize' option which is an option to one of the original programs that was used in Maneage (Gnuastro). Such an option doesn't exist in many other programs, so it is not a suitable option for the generic Maneage project (and can just cause confusion). It was also not used in any part of Maneage any more! With this commit, this option is removed from the core Maneage './project' script and if any project uses it, they can implement it in their own branch. 26 March 2021, 18:19:22 UTC
66453b4 Maneage installation: removed TCL as a dependency of SWIG Until now the SWIG software would use the host operating system's packages to find the TCL configuraiton (which we don't install yet in Maneage). In particular, you can see the error during its configuration here: .... checking for pkg-config... pkg-config checking for Tcl configuration... found /usr/lib/tclConfig.sh /usr/lib/tclConfig.sh: line 2: dpkg-architecture: command not found /usr/lib//tcl8.6/tclConfig.sh: line 2: dpkg-architecture: com. not found With this commit, TCL has been disabled when building SWIG with the '--without-tcl' option. Later, when we add TCL in Maneage, we can remove this option. 24 March 2021, 21:01:35 UTC
a981196 Configuration: nullability-completeness warnings suppressed With a recent update of macOS systems (macOS Big Sur 11.2.3 and Xcode 12.4), there are many warnings when building C programs (for example the simple program we compile to check the compiler, or some of the software like `gzip'). It prints hundreds of warning lines for every source file that are irrelevant for our builds, but really clutters the output. With this commit, these warnings are disabled by adding `-Wno-nullability-completeness' to the 'CPPFLAGS' environment variable. This has also been added to the very first check of the C compiler in the configure step. 20 March 2021, 01:08:21 UTC
c3e82b1 Configuration: --debug option available in this phase also Until now, each time there was a problem in the configuration of Maneage'd projects and debugging was necessary, we had to take the following changes: - Run the configuration on a single thread ('-j1') to see the building of only the problematic software. - Disable the Zenodo check manually by commenting those parts of 'reproduce/software/shell/configure.sh'. Because the internet connection wastes a few seconds and is thus very annoying during repeated runs! - Manually remove the '-k' option that was passed to Make (when building the software). With the '-k', Make keeps going with the execution of other targets if something crashes and this usually causes confusions during the debugging. Doing the manual changes within the code was both very annoying and prone to errors (forgetting to correct it!). With this commit, the existing '--debug' option has been generalized to the software configuration phase of Maneage also. Until now, it was only available in the analysis phase (and would directly be passed to the 'make' command that would run the analysis). When this option is used, and the project is in the software configuration phase, the Zenodo check won't be done, it will use one single thread ('-j1'), and it will stop the execution as soon as an error occurs (Make is not run with '-k'). 20 March 2021, 00:53:37 UTC
1524213 Installation: minor correction in links to system libraries Until now when making a link to the system's 'dl' and 'pthread' libraries we were simply linking the installed location on the system (in '/usr/lib'). However, in some systems, these may themselves be links to other locations and this could cause linking problems. With this commit, we now use 'realpath' to extract the absolute address of the final file that the libraries may link to, and directly link to them. A minor cosmetic correction was also made in the build rule for CFITSIO: the long line was broken into two! 12 February 2021, 23:49:16 UTC
585585d Open access DOIs This commit fixes mnras_openaccess_titles.bst to preferentially link to open-access versions of articles through DOIs, using the alias 'oadoi.org' of Unpaywall.org . URL bases of the ADS are (partially) updated too. Rouabah's DOI is added, on the assumption that at some time in the future it will be validated. 23 January 2021, 21:37:08 UTC
ed16608 First try for PeerJ LaTeX style LaTeX style change. The acknowledgments, accidentally deleted, have been restored. 23 January 2021, 06:31:56 UTC
ecbaadc Default LaTeX preamble: some packages moved to preamble-project.tex Until now, important LaTeX packages like 'caption' (for managing figure captions), 'hyperref' (for managing links) and 'xcolor' (for managing colors) were being loaded inside the optional 'tex/src/preamble-maneagge-defualt-style.tex' file. We recommend to remove this file from loading when you use custom journal sytels. However, these packages will often be necessary after loading special journal styles also. With this commit, these packages are now loaded into LaTeX as part of the 'tex/src/preamble-project.tex' file. This file is in charge of LaTeX settings that are custom to the project and independent of its style. Several other small corrections are made with this commit: - I noticed that './project make texclean' crashes if no PDF exists in the working directory! So a '-f' was added to the 'rm' command of the 'texclean' rule. - As part of the LaTeX Hyperref, we can set general metadata or properties for the PDF (that aren't written into the printable PDF, but into the file metadata). They can be viewed in many PDF viewers as PDF properties. Until now, we were only using the '\projecttitle' macro here to write the paper's title. However, thanks to the recently added 'reproduce/analysis/config/metadata.conf', we now have a lot of useful information that can also go here. So the 'metadata-copyright-owner' is now used to define the PDF author, and the project's 'metadata-git-repository' and commit hash are written into the PDF subject. But to import these, it was necessary to define them as LaTeX macros, hence the addition of these macros in 'initialize.mk'. - Some extra packages that aren't necessary to build the default PDF were removed in 'preamble-project.tex'. 12 January 2021, 16:17:18 UTC
f7e7b46 Remove template archive today link from ref 11 January 2021, 11:59:59 UTC
da756b2 Update Zenodo ID to 4432080 11 January 2021, 11:54:21 UTC
b8505ee Add ref to open gov data paper This commit adds a reference to an open government data (OGD) research paper: SARS-CoV-2 daily counts are, in principle, open government data, so a gateway article into studies of the way that governmental agencies really handle OGD should help the reader. 11 January 2021, 11:46:43 UTC
9c13f35 Merge branch 'maneage' into tandf This is a merge of upstream maneage commit d9a6855 plus a few minor fixes. I check the instructions in the commit description, and did these steps: * cd .build/ * mv -iv * analysis * mv -iv analysis/software . * mv -iv analysis/locks software/ * ./project configure --existing-conf * ./project make clean-paper # or make texclean for the upstream name After this, ./project make seems to work correctly. The 'verify' step passes successfully. 10 January 2021, 20:31:49 UTC
0e2659f Minor updates in re-finalising paper Add machine properties to text of paper; check/tidy rules for producing archival files for Zenodo; clarify that the two new papers are not about a GDP/fraud anti-correlation. 10 January 2021, 19:57:05 UTC
cbe3030 make dist: removing temp files moved after project-specific files Until now, when you ran './project make dist', first it would delete the temporary files (like files ending in '~' or '.swp' created by some editors), then it had a place to add project-specific operations for the distribution. However, in the process of cleaning the temporary files, it would 'cd' into the directory that would later be packaged. So project-specific operations would first have to 'cd' back into the top source directory. This was prone to hard-to-find bugs. With this commit, to avoid the problem the project-specific operations are now placed before the cleaning phase. This is also technically good because in the project-specific operations there may also be temporary files that shouldn't go into the distribution tarball. 10 January 2021, 03:32:54 UTC
fcf9ec9 Finish maneage update; change md5sums for some results This commit should hopefully finish off the Maneage update. Some fixes of files in conflict that were edited but apparently not committed are now done. In this commit, formal verification, using md5sums, of the key results files in .tex format failed. Inspecting the numerical differences shows that these appear in the uncertainties estimated using a pseudo-random number generator by python libraries. The random seed is unchanged, but there have been several updates in python libraries: -python-version = 3.7.4 +python-version = 3.8.5 -h5py-version = 2.9.0 +h5py-version = 2.10.0 -mpi4py-version = 3.0.2 +mpi4py-version = 3.0.3 -numpy-version = 1.17.2 +numpy-version = 1.19.1 -scipy-version = 1.3.1 +scipy-version = 1.5.2 Results obtained with the old libraries (e.g. commit a59436b) successfully pass the verification, remaining bytewise reproducible. Since the more recent libraries can reasonably expected to be improved (e.g. with a better choice of default pseudorandom number generator), and since the results do not appear to be significantly different, the updated version is now considered to be correct, and the md5sums for the results files are updated in this commit. LaTeX trivia: the \input command does not appear to strictly insert the characters from the input file into the outer file. A bug with 'misplaced noalign' for an hline required putting the \hline in the input file instead of the outer file (paper.tex), despite the apparent resulting byte content being identical. 09 January 2021, 14:40:19 UTC
d9a6855 IMPORTANT: analysis outputs written in BDIR/analysis Until now, the build directory contained a 'software/' directory (that hosted all the built software), a 'tex/' subdirectory for the final building of the paper, and many other directories containing intermediate/final data of the specific project. But this mixing of built software and data is against our modularity and minimal complexity principles: built software and built data are separate things and keeping them separate will enable many optimizations. With this commit, the build directory of the core Maneage branch will only contain two sub-directories: 'software/' and 'analysis/'. The 'software/' directory has the same contents as before and is not touched in this commit. However, the 'analysis/' directory is new and everything created in the './project make' phase of the project will be created inside of this directory. To facilitate easy access to these top-level built directories, two new variables are defined at the top of 'initialize.mk': 'badir', which is short for "built-analysis directory" and 'bsdir', which is short for "built-software directory". HOW TO IMPLEMENT THIS CHANGE IN YOUR PROJECT. It is easy: simply replace all occurances of '$(BDIR)' in your project's subMakefiles (except the ones below) to '$(badir)'. To confirm if everything is fine before building your project from scratch after merging, you can run the following command to see where 'BDIR' is used and confirm the only remaning cases. $ grep -r BDIR reproduce/analysis/* --> make/verify.mk: innobdir=$$(echo $$infile | sed -e's|$(BDIR)/||g'); \ --> make/initialize.mk:badir=$(BDIR)/analysis --> make/initialize.mk:bsdir=$(BDIR)/software --> make/initialize.mk: $$sys_rm -rf $(BDIR) --> make/top-prepare.mk:all: $(BDIR)/software/preparation-done.mk 'BDIR' should only be present in lines of the files above. If you see '$(BDIR)' used anywhere else, simply change it to '$(badir)'. Ofcourse, if your project assumes BDIR in other contexts, feel free to keep it, it will not conflict. If anything un-expected happens, please post a comment on the link below (you need to be registered on Savannah to post a comment): https://savannah.nongnu.org/task/?15855 One consequence of this change is that the 'analysis/' subdirectory can be optionally mounted on a separate partition. The need for this actually came up for some new users of Maneage in a Docker image. Docker can fix portability problems on systems that we haven't yet supported (even Windows!), or had a chance to fix low-level issues on. However, Docker doesn't have a GUI interface. So to see the built PDF or intermediate data, it was necessary to copy the built data to the host system after every change, which is annoying during working on a project. It would also need two copies of the source: one in the host, one in the container. All these frustrations can be fixed with this new feature. To describe this scenario, README.md now has a new section titled "Only software environment in the Docker image". It explains step-by-step how you can make a Docker image to only host the built software environment. While your project's source, software tarballs and 'BDIR/analysis' directories are on your host operating system. It has been tested before this commit and works very nicely. 09 January 2021, 03:00:15 UTC
22aa8a2 Merge branch 'maneage' into tandf This commit aims to update the subpoisson project to a more recent version of Maneage, which should make it more likely to work 'out of the box' for people on a wider variety of systems, including Darwin/Xnu (macOS, iOS, ...) systems. This commit has not yet been tested. 08 January 2021, 19:58:06 UTC
39d3e7e Fix table EOLs; add two COVID-19 1BL papers This commit adds '\\' before \hline in tables where LaTeX is now a bit fussier than before; add brief citations to two COVID-19 1BL papers; updates the bst file and and removes the ^Ms from it. 08 January 2021, 17:46:48 UTC
a59436b Shorten abstract; country full names; acknowledgments Reduce abstract word length to around 200 words; replace a few of the iso 2-letter country codes in the abstract by the full country names; acknowledge helpful comments by Taha Rouabah. 08 January 2021, 04:16:19 UTC
3845c89 Switch to tandf style: first try These commits should enable switching to the style of TandF journals. 08 January 2021, 02:03:31 UTC
b413260 Merge branch 'subpoisson' of codeberg:boud/subpoisson into subpoisson 07 January 2021, 18:15:47 UTC
499281a Minor: merge branch 'wiley' into subpoisson - one keyword change 07 January 2021, 18:14:37 UTC
b91af98 Configuration: GNU Binutils linking bug on some systems fixed Until now, when building GNU Binutils on GNU Linux operating systems, we would simply put a link to the host's core C library components (the '*crt*' files). However, the symbolic link wasn't "forced"! So if it already existed in the build directory, it would crash. With this commit a '-f' option has been added to the 'ln' command and this fixed the problem. This bug was reported by Zahra Sharbaf. 05 January 2021, 18:01:19 UTC
a1a966a Building of Less program now uses patchelf to ensure good linking After correctly setting Less to depend on 'ncurses', I noticed its still not linking to Maneage's 'ncurses', but pointing to my host system's 'ncurses' (that happens to have the same version! So it would crash on a system with a different version). This shows that like some other software, we need to manually correct the RPATH inside Less. With this command, the necessary call to 'patchelf' has been added and with it, the installed 'less' command properly linked to Maneage's internal build of 'ncurses'. 04 January 2021, 03:32:38 UTC
dc4aa8c README-hacking.md: edits and improvements to publication checklist After going through the publication checklist, some edits were made to make things more clear. Also, an item was added to remind the project author that the commit hashes on the uploaded data files should be the same. 04 January 2021, 03:21:03 UTC
02e53b9 README.md: summary Dockerfile with all necessary lines in one step Until now, the description in 'README.md' to build the Dockerfile in 'README.md' had one item per line, thoroughly describing the reason behind that line. But in many cases, the user is already familiar with Docker (or has already read through the items) and just wants to have the Dockerfile ready fast. In these cases, all those extra explanations are annoying. With this commit, an item '0' has been added at the start of the item list for summary. It only contains the necessary Dockerfile contents with no extra explanation. 04 January 2021, 02:58:05 UTC
31f4ea3 Building of less software depends on ncurses Until now, the 'less' software package (used to view large files easily on the command-line and used by Git for things like 'git diff' or 'git log') only depended on 'patchelf' (which is a very low-level software). However, as Boud reported in bug #59811 [1], building less would crash with an error saying "Cannot find terminal libraries" in some systems (including the proposed Docker image of 'README.md' which I confirmed afterwards). Looking into the 'configure' script of 'less', I noticed that 'less' is actually just checking for some functions provided by the ncurses library! With this commit, 'less' depends on 'ncurses'. I was able to confirm that with this change, 'less' successfully builds within the Docker image. [1] https://savannah.nongnu.org/bugs/?59811 04 January 2021, 01:52:25 UTC
b1bd282 ./project make: new texclean target Until now there was only a 'clean' (to delete all files created during the 'make' phase) and the 'distclean' (to delete all files during configuration and make). But sometimes we don't want to delete all the files created during the full 'make' phase, we only want to delete the files that were created by LaTeX for building the paper. Witht this commit, a new target has been added for this job. You can now run the following command for this job: ./project make texclean Only the files in '$(BDIR)/tex/build' will be deleted (and the 'tikz' directory under that location is recreated, ready for a future build). 02 January 2021, 15:59:49 UTC
ff43476 Copyright year updated in all source files Having entered 2021, it was necessary to update the copyright years at the top of the source files. We recommend that you do this for all your project-specific source files also. 02 January 2021, 15:52:31 UTC
b910b2e Better warnings when maneage branch not present and PDF not built Until now, there was no warning when the 'maneage' branch didn't exist in the Git history. This can happen when you forget to push the 'maneage' branch to a remote for your project, and you later clone your project from that remote (for example on another computer). We use the 'maneage' branch to report the latest commit hash and date in the final paper (which can greatly help future readers). Since we check the 'maneage' branch on every run of './project make' (in 'initialize.mk') this would result in a printed statement like this: fatal: Not a valid object name maneage Also until now, the description of what to do when TeXLive wasn't installed properly wasn't complete: it didn't mention that it is necessary to delete the TeXLive target files. This could confuse users (they would re-run './project configure -e', but with no effect). With this commit, for the 'maneage' branch issue a complete warning will be printed. Telling the user what to do to get the 'maneage' branch (and thus fix this warning). Also, the LaTeX macros that go in the paper are now red when the 'maneage' branch doesn't exist, telling the user to see the printed warning (thus encouraging the user to get the branch). For the TeXLive issue, the necessary commands to run are now also printed in the warning. 14 December 2020, 02:28:14 UTC
cce4016 Configuration: not settting C_INCLUDE_PATH on macOS Until now, when building the high-level (optional) software, we would give both 'CPPFLAGS' and 'C_INCLUDE_PATH' the same value/directory in 'high-level.mk'. But we recently found that on macOS's C compiler ('clang'), if a directory is included in both 'CPPFLAGS' and 'C_INCLUDE_PATH', then that directory is ignored in 'CPPFLAGS' (which has higher priority). This caused linking problems when the version of a software on the host was different from the Maneage version. With this commit, 'C_INCLUDE_PATH' is not set on macOS any more and this fixed the problem on the reported systems. This bug was fixed with the help of Mohammad Akhlaghi and Mahdieh Navabi. 09 December 2020, 22:44:13 UTC
021ff34 Less is now built as a basic software Less is rarely used in non-interactive mode and is primarily intended for interactively viewing large files. So its need within Maneage (for batch processing) wasn't often felt until now. However, when running './project shell' (which completely closes-off the outside environment), or building a Maneage'd project within a minimal container that doesn't have less, it becomes hard to use Git (and in particular its 'diff' output which depends on 'less'). With this commit, Less has been added as a dependency of Git in 'basic.mk'. In total its built product is roughly 800KB and builds within a second or two. So it isn't a burden on any project. But it can be very useful when the projects are being developed within the Maneage environment itself. 02 December 2020, 01:01:45 UTC
1ee1d66 Installation: m4 no longer depends on Texinfo In a recent build on a macOS, we recognized that Texinfo needs the 'libintl.h' headers of Gettext. However, Gettext depends on M4, and until now we had set M4 to depend on Texinfo. Therefore adding Gettext as a dependency of Texinfo would cause a circular dependency. On the macOS, we temporarily disabled M4's Texinfo dependency, and the build went through. I also checked on my GNU/Linux system: temporarily renamed all Texinfo built files from my system and done a clean build of M4 and it succeeded. To be further safe, I built Maneage from this commit (where M4 doesn't depend on Texinfo) in a Docker container, and it went through with no problems. So the current M4 version indeed doesn't need Texinfo. I think adding Texinfo as a dependency of M4 was a historic issue from the early days. In the process, I also cleaned 'basic.mk' a little: - A "# Level N" comment was added on top of each group of software that can be built in parallel (generally). - GNU Nano was moved to the end of the file (to be "Level 6"). - Some comments were edited in some places. 01 December 2020, 18:53:22 UTC
e796e6f README-hacking.md: recommended to push maneage after merging Until now at the end of the updating process, we hadn't explicity talked about pushing the branches. So people would usually only push their 'master' branch to their remote. While the merged 'master' branch does contain the commits from the core Maneage branch, having a no-updated 'maneage' branch reference on their remote can be confusing. With this commit, at the end of the process to merge with the 'maneage' branch we explicitly recommend to push both the 'master' and 'maneage' branches. 01 December 2020, 16:47:31 UTC
ac8eee9 Default paper: macros available for date of commits cited Until now, Maneage only provided the commit hashes (of the project and Maneage) as LaTeX macros to use in your paper. However, they are too cryptic and not really human friendly (unless you have access to the Git history on a computer). With this commit, to make things easier for the readers, the date of both commits are also available as LaTeX macros for use in the paper. The date of the Maneage commit is also included in the acknowledgements. Also, the paragraph above the acknowledgements has been updated with better explanation on why adding this acknowledgement in the science papers is good/necessary. 01 December 2020, 11:43:04 UTC
6ca2fbf IMPORTANT: organizational improvements in Maneage TeX sources This only concerns the TeX sources in the default branch. In case you don't use them, there should only be a clean conflict in 'paper.tex' (that is obvious and easy to fix). Conflicts may only happen in some of the 'tex/src/preamble-*.tex' files if you have actually changed them for your project. But generally any conflict that does arise by this commit with your project branch should be very clear and easy to fix and test. In short, from now on things will even be easier: any LaTeX configuration that you want to do for your project can be done in 'tex/src/preamble-project.tex', so you don't have to worry about any other LaTeX preamble file. They are either templates (like the ones for PGFPlots and BibLaTeX) or low-level things directly related to Maneage. Until now, this distinction wasn't too clear. Here is a summary of the improvements: - Two new options to './project make': with '--highlight-new' and '--highlight-notes' it is now possible to activate highlighting on the command-line. Until now, there was a LaTeX macro for this at the start of 'paper.tex' (\highlightchanges). But changing that line would change the Git commit hash, making it hard for the readers to trust that this is the same PDF. With these two new run-time options, the printed commit hash will not changed. - paper.tex: the sentences are formatted as one sentence per line (and one line per sentence). This helps in version controlling narrative and following the changes per sentence. A description of this format (and its advantages) is also included in the default text. - The internal Maneage preambles have been modified: - 'tex/src/preamble-header.tex' and 'tex/src/preamble-style.tex' have been merged into one preamble file called 'tex/src/preamble-maneage-default-style.tex'. This helps a lot in simply removing it when you use a journal style file for example. - Things like the options to highlight parts of the text are now put in a special 'tex/src/preamble-maneage.tex'. This helps highlight that these are Maneage-specific features that are independent of the style used in the paper. - There is a new 'tex/src/preamble-project.tex' that is the place you can add your project-specific customizations. 01 December 2020, 00:12:11 UTC
c312f4e README-hacking.md: updated paper to cite for using Maneage Until now, we were asking the users of Maneage to cite the first paper that used its primoridal version (arXiv:1505:01664). But there is now a paper that fully describes the concept (arXiv:2006.03018). With this commit, in the 'citation' section of 'README-hacking.md' we now ask to cite the new paper. 28 November 2020, 22:18:36 UTC
fcc77a7 Building final PDF: pdf-build-final has to be given an explicit yes Until now, when the 'pdf-build-final' configuration variable (defined in 'reproduce/analysis/config/pdf-build.conf') was given any string a PDF would be built. This was very confusing, because people could put a 'no' and the PDF would still be built! With this commit, only when this variable has a value of 'yes' will the PDF be built. If given any other string (or no string at all), it will not produce a PDF. This issue was reported by Zahra Sharbaf. 15 November 2020, 13:32:57 UTC
9cfb8d3 README.md: added commands to delete all Docker images Until now we had described the basic commands on how to create and use Docker images, but we hadn't mentioned how you can delete them. With this commit the commands necessary for deleting Docker images have also been added at the bottom of the section on Docker. 13 November 2020, 19:11:18 UTC
2b39a67 Recipes for final initialize and verify targets not on stdout The LaTeX macro files for these two subMakefiles are created on every run of './project make'. So their commands are also printed every time and hardly ever will a normal user want to modify or change these. So to avoid populating the standard output of a Maneaged project with all these extra lines every time (possibly getting mixed with the important analysis or LaTeX outputs), an '@' has been placed at the start of the recipes. With an '@' at the start of the recipe, Make is instructed to not print the commands it wants to run in the standard output. 18 October 2020, 13:37:16 UTC
04bda5a Update README-hacking.md with elaphrocentre ArXiv:2010.03742 This commit updates README-hacking.md with the URIs for the 'elaphrocentre' galaxy formation pipeline paper arXiv:2010.03742. This makes three papers currently in the peer review pipeline: arXiv:2006.03018, arXiv:2007.11779, and arXiv:2010.03742, each chronologically corresponding to various stages of the review process. 09 October 2020, 13:28:21 UTC
b677efb TexLive's xstring package is now necessary After a fresh build of Maneage with a newly downloaded TeXLive, I noticed that it is complaining about not finding 'xstring.sty', apparently some package that depeneded on it is no longer including it itself! It is thus now added to the packages that are built by Maneage's TeXLive. 02 October 2020, 00:12:35 UTC
ee8c6be Gnuastro's analysis configuration files removed Until now, the core Maneage branch included some configuration files for Gnuastro's programs. This was actually a remnant of the distant past when Maneage didn't actually build its own software and we had to rely on the host's software versions. This file contained the configuration files specific to Gnuastro for this project and also had a feature to avoid checking the host's own configuration files. However, we now build all our software ourselves with fixed configuration files (for the version that is being installed and its version is stored). So those extra configuration files were just extra and caused confusion and problems in some scenarios. With this commit, those extra files are now removed. Also, two small issues are also addressed in parallel with this commit: - When running './project make clean', the 'hardware-parameters.tex' macro file (which is created by './project configure' is not deleted. - The project title is now written into the default output's PDF's properties (through 'hypersetup' in 'tex/src/preamble-header.tex') through the LaTeX macro. All these issues were found and fixed with the help of Samane Raji. 24 September 2020, 01:15:40 UTC
0c882a4 Checking Xcode installation for macOS systems Until now, during the configure step it was checked if the host Operative System were GNU/Linux, and if not, we assumed it is macOS. However, it can be any other different OS! With this commit, now we explicity check if the system is GNU/Linux or Darwin (macOS). If it is not any of them, a warning message says to the user that the host system is different from which we have checked so far (and invite to contact us if there is any problem). In addition to this, if the system is macOS, now it checks if Xcode is already installed in the host system. If it is not installed, a warning message informs the user to do that in case a problem/crash in the configure step occurs. We have found that it is convenient to have Xcode installed in order to avoid some problems. 15 September 2020, 17:49:42 UTC
11b11e9 R is built without tcl/tk (for GUI) dependence Tcl/Tk are a set of tools to provide Graphic User Interface (GUI) support in some software. But they are not yet natively built within Maneage, primarily because we have higher-priority work right now. GUI tools in general aren't high on our priority list right now because GUI tools are generally good for human interaction (which is contrary to the reproducible philosophy), not automatic analysis (a core concept in reproducibility). So even later, when we do include Tcl/Tk in Maneage, their direct usage will be discouraged. Until this commit, because we don't yet build Tcl/Tk, the default maneage install of the statistical package R failed on a Debian Stretch, with 6227 repeats of the line: '/usr/lib//tcl8.5/tclConfig.sh: line 2: dpkg-architecture: command not found' To fix this problem (atleast until Tcl/Tk is installed within Maneage), R is now configured with the '--without-tcltk' option which fixed the problem. Please see the description above the R installation instructions in 'reproduce/software/make/high-level.mk' for more. 09 September 2020, 11:46:15 UTC
24d01f5 Removed all occurances of IFS in low-level scripts Following the previous commit, we recognized that the 'IFS' terms are not necessary and can be even cause problems. So all their occurances in the scripts of Maneage have been removed with this commit. 08 September 2020, 00:25:29 UTC
6727f84 Software installation: removed IFS statements in pre-make-build.sh Until a recent commit, the IFS='"' was added at the start of the variables in this shell script and as a result, the SPACE character wasn't being used as a delimiter. This caused a major problem when downloading the tarballs (all the backup servers were considered as the top link). With this commit we removed these 'IFS' statements). Because we now check for the existance of meta-characters in the build directory name, there is no more problem, and also generally both the calling command and internally, we have double-qutations around the variable names. So removal of IFS will not affect the result in this scenario. This bug was found by Mohammadreza Khellat. 07 September 2020, 18:42:03 UTC
f5fe8cf Clarify which graph the data came from; fix absent dates This commit adds the name of the graph from which the data came; this will not help much in archival terms, since the URL is unarchivable, but for a short period may be useful. A minor fix is made to the early dates in March, for which CDTA does not provide data. 31 August 2020, 18:48:39 UTC
57b1a90 Verify that the WP C19CCTF data match the CDTA DZ data This commit adds the file `reproduce/inputs/Wikipedia_SARSCoV2_charts_DZ_cdta_dz.dat` which includes a fresh download of WP C19CCTF data for Algeria, and a manually verified check on the official Algerian data provided by the Center for Development of Advanced Technologies (CDTA) at https://covid19.cdta.dz/dashboard/production/index.php . The fifth column indicates the CDTA data at all dates provided, and 0 in dates not provided. The cdta.dz website provides an http response that requires javascript mouse events for extraction of the data. These data were extracted manually, although in principle a script could be written for automatic extraction. Almost no differences were found, except where the CDTA data appears to disagree with the WP C19CCTF version of the official data: * on 2020-04-08, CDTA has 1578 instead of 1572; * on 2020-04-19, CDTA has 2534 (identical to 2020-04-18) instead of 2629; * on 2020-06-23, CDTA has 12077 instead of 12077; * seven dates in early July are missing in the CDTA data. 31 August 2020, 18:31:13 UTC
79aa59a Fix superscripts in pdf 30 August 2020, 20:31:30 UTC
79a3c67 Minor: better keyword - data validation 30 August 2020, 20:02:52 UTC
80e3202 Changes to shift journal style to Wiley; algeria.sh This commit makes a whole bunch of style changes to shift to the Wiley journal style. A script to compare the WP C19CCTF and JHU Algerian daily SARS-CoV-2 counts with the Rouabah+2020 counts is added for running by hand. 30 August 2020, 19:12:47 UTC
4254c80 Add a sentence on JHU to first para of method section Since some readers convinced by the JHU name might not read far enough to notice that the JHU data are analysed in an appendix and give almost identical results (apart from India), a sentence is added earlier to increase the chance that the reader knows that JHU data is considered. 30 August 2020, 19:06:34 UTC
290b80d Minor reference fix 30 August 2020, 01:15:42 UTC
b4be72e Remove eps hack needed for old hyperref; include .sty files This commit removes the eps file hack from `reproduce/analysis/make/paper.mk` since the hack does not seem to be needed any more with a recent hyperref/texlive setup. In this commit, style files (*.sty) are added to 'package-contents' in `reproduce/analysis/make/initialize.mk`. Hint: if compiling in an old texlive system, remove `NJDnatbib.sty` by hand from the distribution in order to avoid superscript links that have almost zero height. 30 August 2020, 01:09:14 UTC
f1c88e4 Fix doubled word Email 30 August 2020, 00:10:55 UTC
ca8d5ae Do a proper LaTeX n-accent-aigu in Torun 30 August 2020, 00:07:53 UTC
21dd1f1 Do journal patches properly; package-contents: eps files With this commit, the journal patches are put into their correct places and applied correctly. The 'package-contents' rule in `reproduce/analysis/make/initialize.mk` is changed so that only the .eps files used in the paper are included in the package. While these figures may be interesting to people wanting more info, beyond the formally published paper, they will be annoying and confusing to journal typesetting staff, who for some journals say that figures have to be numbered from 1 to N. Giving them N+M figures with only N to be really used could lead to errors that waste the authors' time to clean up later, or errors that the authors don't notice in the proofs. 29 August 2020, 23:59:17 UTC
1c16987 Fix unremoved old Wiley .log file 29 August 2020, 04:06:04 UTC
59ba0b1 Fix confusing percent symbol; full run This commit fixes a '%' symbol that was creating problems in the latexpand inclusion of the .bbl file into paper-full.tex (the % was protected, creating \%, which was unexpected in a list of \bibitem's). This commit disables the dev mode. 29 August 2020, 03:52:21 UTC
back to top