| d8f8ff1 | yakutovicha | 31 January 2019, 17:15:17 UTC | Merge pull request #2434 from ltalirz/issue_2355_verdi_help_exit_status let verdi help return exit status 0 | 31 January 2019, 17:15:17 UTC |
| 2d3a80a | Leopold Talirz | 31 January 2019, 16:20:33 UTC | Merge branch 'release_v0.12.3' into issue_2355_verdi_help_exit_status | 31 January 2019, 16:20:33 UTC |
| 9b7654f | Leopold Talirz | 31 January 2019, 16:04:01 UTC | let verdi help return exit status 0 | 31 January 2019, 16:04:01 UTC |
| d4ddb6b | Leopold Talirz | 31 January 2019, 15:54:59 UTC | Add example for pytest fixtures with processes (#2432) Example kindly provided by @chrisjsewell | 31 January 2019, 15:54:59 UTC |
| 040b34e | Leopold Talirz | 30 January 2019, 22:47:37 UTC | add instructions when verdi import fails (#2420) * when verdi import fails because of an old/new export file version, add a hint on what to do * fix issue with pip 19.x (go back to pip 18.1) * fix issue with pre-commit failing to install. discussed in #2362 * fix nasty travis issue with build of pymatgen (built using incompatible numpy version provided by travis) * upgrade pymatgen to 2018.12.12 (last version with py2 support) | 30 January 2019, 22:47:37 UTC |
| ff7af8c | Leopold Talirz | 29 January 2019, 09:09:19 UTC | Update RabbitMQ docs for macports (#2411) | 29 January 2019, 09:09:19 UTC |
| 9b0b8a3 | Casper Welzel Andersen | 23 January 2019, 10:43:25 UTC | Freeze pip version to 18.1 in Travis (#2415) Hack-fix for issue #2414 | 23 January 2019, 10:43:25 UTC |
| c61f212 | ConradJohnston | 21 January 2019, 16:19:36 UTC | Merge pull request #2405 from giovannipizzi/fix_sqlalchemy_migration_tests Fix blocking bug on new SQLAlchemy migration tests | 21 January 2019, 16:19:36 UTC |
| 982bcea | Giovanni Pizzi | 21 January 2019, 15:45:38 UTC | Fix blocking bug on new SQLAlchemy migration tests The problem was that the session was not properly closed. I am also moving some logic around and having a couple of useful methods that are reused in all tests. | 21 January 2019, 15:45:38 UTC |
| be717b6 | Sebastiaan Huber | 19 January 2019, 07:45:15 UTC | Further simplification of node type string definition and loading (#2401) With the recent introduction of the `CalcJob` process, which now ensures that also for job calculations, it is the `CalcJobNode` that gets stored in the database, which has a type string just like all the other node types. This made a lot of the old logic `aiida.plugins.loader` to determine the correct `type` string for a `Node` (sub) class obsolete as now all node classes (except Data) are per definition internal to `aiida-core` and have a type string that is based directly on their module path. The few functions that remain, `get_type_string_from_class` that formats the correct type string for a given `Node` class and the `load_node_class` which can reload that class from the string have been moved to `aiida.orm.utils.node` since it has no longer anything to do with a "plugin". Finally, by moving the `WorkflowNode`, `CalculationNode` and `ProcessNode` classes in the `__init__` files of their respective modules allows us to add further simplification to `load_node_class` as well as making all these classes and their subclasses importable from the `aiida.orm.node` module. Eventually, when `aiida.orm.data` is also moved there, all classes can even be exposed directly on `aiida.orm` and that is the deepest a user should ever go for they should need or be allowed to import. | 19 January 2019, 07:45:15 UTC |
| c0fba2e | szoupanos | 18 January 2019, 08:22:53 UTC | Implement functionality to export and import `Log` entries. (#2393) The native logging of python is extended with a `DbLogHandler` that will, when it is configured for a specific AiiDA entity (a Node, Group or any other database entity), store a copy of the record in the `DbLog` table with a reference to the entity it belongs to. Since the entities do not necessarily live in the same table, a foreign key could not be used, so instead the primary key was stored as well. However, when exporting and importing entities and their log messages, the entities would get new primary keys, which would orphan the log messages. The attribute that is persistent across exporting and importing is the UUID. Therefore the reference attribute for log entries to their entities has been changed from the `pk` to the `uuid`. | 18 January 2019, 08:22:53 UTC |
| 423523c | Sebastiaan Huber | 18 January 2019, 07:45:48 UTC | Redesign the Parser class (#2397) Now that job calculations are directly implemented as processes, by sub classing the `CalcJob` process, a lot of functionality has been removed from the `CalcJobNode` that did not belong there. However, quite a bit of attributes and methods remain there that are used by the `Parser` that parses results attached to the node, such as names for retrieved links. These should, just like the input and output nodes, be specified on the `Process` class. We redesign the `Parser` class to get this the information it needs from the `Process` class which can be gotten through the `CalcJobNode` as a proxy. Sub sequently, it can directly get information like default output node labels as well as exit codes from the process class. The base method that will be called by the engine to trigger parsing of a retrieved job calculation is the `parse` method. The engine will pass the nodes that the parser needs for parsing as keyword arguments. The engine knows which nodes to pass as they have been marked as such by the `CalcJob` process spec. The `Parser` class exposes a convenience method `parse_from_node` that can be called outside of the engine, passing in a `CalcJobNode`. This will trigger the `parse` method to be called again, but it will be wrapped in a `calcfunction` ensuring that the produced outputs will be connected to a `ProcessNode` in the provenance graph. Finally, what was the `CalculationResultManager` has been renamed to `CalcJobResultManager` and has been moved to `aiida.orm.utils`. The interface has also been adapted and it now no longer goes through the parser class to get the information it needs but rather through the `CalcJob` class that can be retrieved from the `CalcJobNode` for which it is constructed. | 18 January 2019, 07:45:48 UTC |
| 2c961cb | Sebastiaan Huber | 17 January 2019, 14:48:36 UTC | Implement `CalcJob` process class (#2389) This commit can be summarized in three steps: * Reimplementation of a job calculation as `Process` called `CalcJob` * Changing job calculation to be purely informational * Remove the old job calculation mechanics and business logic * Reimplementation of a job calculation as `Process` called `CalcJob` The old way of creating a job calculation, was to subclass the `JobCalculation` class and override the `_use_methods` class method to define the input nodes and the `_prepare_for_submission` to setup the input files for the calculation. The problem was that these methods were implemented on the `Node` class, thus mixing the responsabilities of running and introspecting the results of a completed calculation. Here we define the `CalcJob` class, a subclass of `Process`. This class replaces the old `JobCalculation` and allows a user to defined the inputs and outputs through the `ProcessSpec`, just as they would do for a `WorkChain`. Except, instead of defining an `outline`, one should implement the `prepare_for_submission`, which fulfills the exact same function as before, only it is now a public method of the `CalcJob` process class. * Changing job calculation to be purely informational Finally, the role of the job calculation state, stored as an attribute with the key `state` on the `CalcJobNode` has changed significantly. The original job calculations had a calculation state that controlled the logic during its lifetime. This was already superceded a long time ago by the process wrapper that now fully governs the progression of the calculation. Despite the calculation state no longer being authoritative during the calculation's lifetime, it was still present. Here we finally fully remove and only leave a stripped down version. The remaining state is stored as an attribute and is a sub state while the `CalcJob` process is in an active state and serves as a more granual state that can be queried for. This is useful, because the process status, which also keeps similar information is human readable and doesn't allow for easy querying. * Remove methods of `Node` to replace or remove links The `replace_*link_from` methods where only implemented because they were needed by the old job calculation created. Since that system is now removed, this functionality is no longer needed. Likewise, the methods to remove links, either from the cache or the database were not used nor tested and so they are removed. * Prevent adding links from or to sealed nodes When a `Node` instance is sealed, it should be impossible to add links to or from it. To enforce this, the `validate_incoming` and `validate_outgoing` methods are overridden in the `Sealable` mixin that will raise `ModificationNotAllowed` if the node is sealed. | 17 January 2019, 14:48:36 UTC |
| de14357 | Sebastiaan Huber | 17 January 2019, 12:13:31 UTC | Fix bug in the setup of a profile through `verdi setup/quicksetup` (#2395) When setting up a profile that already exists, `verdi` is supposed prompt the user whether they want to reuse the information of the existing profile or if they want to change it. However, instead of asking to confirm, the command was prompting, causing the user to always have to reenter the information. | 17 January 2019, 12:13:31 UTC |
| 487c112 | Giovanni Pizzi | 16 January 2019, 06:38:59 UTC | Add framework for migration tests in SQLAlchemy (#2392) Add a framework for testing migrations in SQLAlchemy/Alembic and ports one test from Django, also to act as an example and to verify that the migration testing is properly working. Also removed old files that were running some tests but actually were just running on a set of example migrations and not on those of AiiDA. | 16 January 2019, 06:38:59 UTC |
| a2456ab | Sebastiaan Huber | 15 January 2019, 09:04:01 UTC | Add a transaction context manager to backend (#2387) This allows to group operations that will be rolled back if the context is exited with an exception. This is laying the groundwork for implementing `Node` as part of the new backend system as links, caches, etc will have to be done in a transaction. | 15 January 2019, 09:04:01 UTC |
| 69f3bd5 | Leopold Talirz | 14 January 2019, 12:55:00 UTC | Modularize `verdi profile delete` (#2151) Modularized the code to delete a profile by splitting it into functions that have been moved to `aiida.manage.configuration.setup` module. Then, three new options are added to `verdi profile delete` * `--include-db/--skip-db` * `--include-repository/--skip-repository` * `--include-config/--skip-config` The unittest for `verdi profile delete` now makes use of `--skip-db` such that the test no longer prompts for a sudo password locally. The full functionality is tested on Travis in `.ci/test_profile.py`. | 14 January 2019, 12:55:00 UTC |
| cd2b051 | Sebastiaan Huber | 11 January 2019, 16:54:38 UTC | Remove implementation of legacy workflows (#2379) Note that the database models are kept in place as they will be removed later, because it will require a migration that will drop the tables. However, we want to include some functionality that allows the user to dump the content for storage, before having the migration executed. | 11 January 2019, 16:54:38 UTC |
| 1e339c3 | Sebastiaan Huber | 10 January 2019, 07:09:48 UTC | Simplify loading of node class from type string (#2376) The type string of a `CalcJobNode` should now be just that of the node class and not the sub class. Since the users are currently still sub classing the node class and not a process class, we have to make a special exception when generating the type string for the node. Conversely, this type string, stored in the `type` column of the node, should be used to load the `CalcJobNode` class when loading from the database. The only classes that can legally exist in a database, and therefore be loaded, are defined in `aiida-core`. Therefore, using the entry point system to map the type string onto an actual ORM class is no longer necessary. We rename the `aiida.plugins.loader.load_plugin` function to the more correct `load_node_class`, which given a type string, will return the corresponding ORM node sub class. Note that the whole machinery around generating type and query strings and loading the nodes based on them is still somewhat convoluted and contains hacks for two reasons: 1) Data is not yet moved within the `aiida.orm.node` sub module and as a result gets the `data.Data.` type string, which will not match the `node.Node.` type when sub classing in queries. 2) CalcJobProcesses are defined by sub classing JobCalculation Until the user directly define a Process sub class that uses the `CalcJobNode` as its node class, exceptions will have to be made. If these two issues are addressed, a lot of the code around type strings can be simplified and cleaned up. | 10 January 2019, 07:09:48 UTC |
| dd0b1b9 | Sebastiaan Huber | 09 January 2019, 15:33:13 UTC | Homogenize use of boolean properties for `Kind` and `StructureData` The `Kind` and `StructureData` were using an inconsisten mixture of methods and properties for boolean attributes, such as `has_vacancies` and `is_alloy`. To conform to `CifData` these are now all turned into properties. | 09 January 2019, 16:20:37 UTC |
| e078852 | Sebastiaan Huber | 08 January 2019, 15:40:57 UTC | Implement the `CifData.has_unknown_atomic_sites` property This property will verify that all coordinates of all atomic sites can be properly converted into float values. If just one coordinate cannot be successfully converted to a float, the cif is said to have unknown atomic sites, as it will not be able to be parsed into a usable `StructureData` object for further calculations. | 09 January 2019, 16:20:37 UTC |
| 11d41e9 | Sebastiaan Huber | 08 January 2019, 15:10:18 UTC | Consider `X` to be an unknown species in `CifData.has_unknown_species` The `CifData.has_unknown_species` return whether the chemical formula contains any unknown species, which were defined as those species that are not listed in `aiida.common.constants.elements`. This was intended to filter out those cif files that contained species that could not be parsed into a `StructureData` that could actually be used for a condensed matter calculation, since the presence of an unknown species would render that impossible. However, recently, the specie `X` was added to the internal dictionary of elements, breaking the method. Here we explicitly ignore `X` in the `has_unknown_species` method, returning its implementation to its original intention. | 09 January 2019, 16:20:37 UTC |
| a31e90c | Sebastiaan Huber | 08 January 2019, 14:47:23 UTC | Fix bug in `CifData.has_partial_occupancies` for non-float values For `CifData` with values of the `_atom_site_occupancy` tag that cannot be parsed into a float, the `has_partial_occupancies` property would except. Here we properly check whether the value can be cast to a float after removing any parentheses that may be present. | 09 January 2019, 16:20:37 UTC |
| 65c60d5 | Sebastiaan Huber | 09 January 2019, 15:38:33 UTC | Ensure that configuration options are considered for logging config (#2375) The configuration knows various options to change the logging configuration, however, these were not respected for two reasons: * Logging configuration was not lazily evaluated * Globally configured options were not agglomerated The first problem was caused by the fact that the logging configuration is evaluated upon loading the `aiida` module, at which point the profile is not necessarily loaded yet, causing the `get_config_option` functions to return the option defaults. The solution is to have the dictionary lazily evaluated by using lambdas, which are resolved when `configure_logging` is called. Finally, we make sure this function is called each time the profile is set. The second problem arose from the fact that if a profile is defined, the `get_config_option` only returned the config value if explicitly set for that profile and otherwise it would return the option default. This means that if the option was defined globally for the configuration it was ignored. This is now corrected where if the current profile does not explicitly define a value for the option but it is globally defined, the global value is returned. | 09 January 2019, 15:38:33 UTC |
| 9b6b41a | Sebastiaan Huber | 08 January 2019, 11:56:59 UTC | Add support for compound projection `state` in `verdi process list` (#2367) Also set it as one of the default projections. | 08 January 2019, 11:56:59 UTC |
| 6c1117a | Sebastiaan Huber | 08 January 2019, 11:18:26 UTC | Add functionality to delete individual Log entries to collection (#2369) Also fixed a bug in the `delete` method of the `AuthInfo` and `Comment` collections that were not caught because there were no tests. If the entry that was asked to be deleted did not exist, no exception was raised, however, that was expected by the implementation. | 08 January 2019, 11:18:26 UTC |
| 9cee73f | Sebastiaan Huber | 08 January 2019, 10:40:37 UTC | Add support for indexing and slicing in `orm.Group.nodes` iterator (#2371) This allows one to directly get just a single or a slice of all the nodes contained within a `Group` instead of having to iterate over it to get the nodes of interest. | 08 January 2019, 10:40:37 UTC |
| db31d44 | Sebastiaan Huber | 06 January 2019, 13:25:14 UTC | Replace call to deprecated `set_` methods with `set_option` (#2361) The `Node._set_internal` method was still calling deprecated `set_` methods for options of the `CalcJobNode` sub class. These should now be set through the explicit `set_option` method. The `JobProcess` was likewise violating this rule, when setting the options of the underlying `CalcJobNode` it is using as a storage record. | 06 January 2019, 13:25:14 UTC |
| 097fa6a | Sebastiaan Huber | 04 January 2019, 14:38:17 UTC | Merge `develop` into `provenance_redesign` and update pre-commit dependencies | 04 January 2019, 14:38:17 UTC |
| 6424ba1 | Sebastiaan Huber | 04 January 2019, 11:25:03 UTC | Use different version for `pylint-django` for python 2 and 3 Starting from `pylint-django>=2.0` only `pylint>=2.0` is supported which has dropped support for python 2. As a result we need to depend on different version of `pylint-django` based on the version of python. With the new version of `pylint` a lot of new checks are added that had to be either corrected or disabled. Given that some of these checks only are valid for `pylint>=2.0`, they will trigger a warning in the pre-commit for python 2. For this reason we have to temporarily disable the `bad-option-value` globally in the `.pylintrc`. When support for python 2 is dropped, this should be removed. | 04 January 2019, 14:10:39 UTC |
| 616f0d8 | Sebastiaan Huber | 04 January 2019, 11:23:58 UTC | Remove `pylint` as explicit requirement The requirement `prospector` already indirectly requires `pylint` and it needs a different version based on the version of python. By fixing the version of `pylint` ourselves, we will have the wrong version for the version of `prospector` for either one of python 2 or python 3. | 04 January 2019, 11:23:58 UTC |
| 0b78a3d | Sebastiaan Huber | 04 January 2019, 11:14:40 UTC | Call `pip install` with the `--no-cache-dir` option This prevents a mismatch between for example `numpy` and `scipy`. | 04 January 2019, 11:14:40 UTC |
| 45b4b40 | Sebastiaan Huber | 04 January 2019, 08:49:20 UTC | Merge `develop` into `provenance_redesign` | 04 January 2019, 09:33:43 UTC |
| a4bbfd1 | Sebastiaan Huber | 03 January 2019, 16:15:30 UTC | Reenable the Travis linter pre-commit test (#2364) The release of ruby `gem==3.0.0` in December 2018 broke the `pre-commit` package because it used the `--no-ri / --no-rdoc` flags that were deprecated in that gem release. These flags were replaced by the new `--no-document` flag in `pre-commit==1.13.0`, which allows us to reenable the Travis linter that had been temporarily disabled. | 03 January 2019, 16:15:30 UTC |
| 19110f9 | Bonan Zhu | 17 December 2018, 20:12:25 UTC | Removed content related to the old plugin system Removed contents implying putting plugin files in the aiida_core subpackages. I also added example plugin structures to give the read examples where the code included in the documentation should go. Also emphasises the importance of declearing the entry points and re-install/refresh when there is any change. | 21 December 2018, 20:27:46 UTC |
| 692eaf3 | Sebastiaan Huber | 20 December 2018, 10:33:12 UTC | Reorganization of some top level modules (#2357) There are a few top-level modules whose purpose was not fully clear and partially as well as partially overlapping, which caused the content to be quite unorganized and scattered. Here we reorganize these modules after having clearly defined their purpose: * aiida.common: for generic utility functions and data structures * aiida.manage: for code that manages an AiiDA instance * aiida.tools: for code that interfaces with third-party libraries The module `aiida.control` has been removed. Code that interacted with internal components have been moved into `cmdline` or `manage` where applicable. The `aiida.control.postgres` has been moved to the `aiida.manage.external` module which is reserved for code that have to interact with system wide components, such as the postgres database. The module `aiida.utils` has been removed. Generic functionality has been placed in `aiida.common` and more specific functionality has been placed in the appropriate sub folders. | 20 December 2018, 10:33:12 UTC |
| 14ecf15 | Sebastiaan Huber | 18 December 2018, 08:20:10 UTC | Implement `verdi config` (#2354) The new command `verdi config` replaces the various commands under `verdi devel` that were used to list, get, set and unset configuration options, that were used to be called "properties". Since the term "property" is a reserved keyword, it was decided to rename them to "options", which is also the term used by "git". The interface of `verdi config` also mirrors that of `git config`. That is to say to get the value of an option simply call `git config <option_name>` and to set it `git config <option_name> <option_value>`. To unset it, the `--unset` flag can be used. Finally, the getting, setting or unsetting can be applied to a certain "scope", meaning to be configuration wide or profile specific. To make the implementation of this command simple and clean, the bulk of the work was in refactoring the definition, construction and operation on the configuration of an AiiDA instance. This is now represented by the `Config` class, through which these configuration options can be set or unset as well as retrieved. | 18 December 2018, 08:20:10 UTC |
| 3e96d16 | Dominik Gresch | 17 December 2018, 16:40:46 UTC | Disable caching for `InlineCalculation`. (#1872) | 17 December 2018, 16:40:46 UTC |
| a86dfed | Leopold Talirz | 17 December 2018, 10:25:03 UTC | remove broken verdi-plug entry point (#2356) verdi-plug entry point does not work (likely a leftover) $ verdi-plug Traceback (most recent call last): File "/Users/leopold/Applications/miniconda3/envs/aiida_master/bin/verdi-plug", line 5, in <module> from aiida.cmdline.verdi_plug import verdi_plug ImportError: No module named verdi_plug | 17 December 2018, 10:25:03 UTC |
| 1e7e25e | Sebastiaan Huber | 14 December 2018, 14:53:02 UTC | Fix bug in `CalcJobNode.get_desc` (#2353) The function still called `CalcJobNode.get_state` with the argument `from_attribute`, which has recently been removed. Originally, the state was stored in a separate database table and had a proxy as an attribute in the attribute table. The calc state table has been removed, leaving the attribute as the only source of the state. | 14 December 2018, 14:53:02 UTC |
| f1db958 | Sebastiaan Huber | 14 December 2018, 13:47:32 UTC | Docs: update the example provenance graph images in concepts section (#2350) The original .svg files are also included. They were generated in Inkspace but saved as plain SVG files. After that they were exported as PNG files. With the `convert` Unix utility, the images were rescaled to 800 px as follows: convert filename.png -resize 800 filename.png This will rescale the image to 800px wide, maintaining the aspect ratio and writing it to the same file. | 14 December 2018, 13:47:32 UTC |
| 53feb9a | Martin Uhrin | 14 December 2018, 12:58:40 UTC | Fix issue with wrong file mode for python3 (#2351) The log file to report an integrity violation was not correct and would complain about expecting a bytes type instead of str. Fixed by setting the mode to `w+`. | 14 December 2018, 12:58:40 UTC |
| bd91e88 | Sebastiaan Huber | 14 December 2018, 11:53:18 UTC | Refactor the management of the configuration file (#2349) The configuration file is now abstracted by the `Config` class. Loading the configuration through `aiida.manage.load_config` will now return a `Config` instance which internally keep the contents of the configuration file of the AiiDA instace. Altering the config will go through this object. All configuration related code, such as settings and methods to setup the configuration folder for a new AiiDA instance, are organized in the `aiida.manage.configuration` module. | 14 December 2018, 11:53:18 UTC |
| 65dfecb | yakutovicha | 13 December 2018, 16:56:36 UTC | Various name changes, improvements and fixes to `Group` (#2329) * `aiida.orm.importexport` - While importing data from the export file allow to specify user-defined group and to put all the imported data in this group * `aiida.common.utils` - Remove `get_group_type_mapping` function which was mapping machine-specific group names with the user-friendly ones - Add escape_for_sql_like that escapes `%` or `_` symbols provided by user * `aiida.orm.groups` - Add `GroupTypeString` enum which contains all allowed group types: data.upf (was `data.upf.family`) auto.import (was `aiida.import`) auto.run (was `autogroup.run`) user (was empty string) - Remove `Group.query` and `Group.group_query` methods, as they are redundant * `aiida.orm.data.upf`: - Set `UPFGROUP_TYPE` to `GroupTypeString.UPFGROUP_TYPE` - Replace the usage of `Group.query` by `QueryBuilder` in `get_upf_groups` and `get_upf_family_names` methods * `aiida.orm.autogroup`: - set `VERDIAUTOGROUP_TYPE` to `GroupTypeString.VERDIAUTOGROUP_TYPE` * `aiida.cmdline.commands.cmd_group` - Add `verdi group copy` - Add option to show all available group types - Add defaulf for group_type option - Replace `Group.query` with `QueryBuilder` in `verdi group list` - Remove usage of the get_group_type_mapping() function * `aiida.cmdline.commands.cmd_import` - Add the possibility to define a group that will contain all imported nodes. * `aiida.cmdline.params.types.group` - Add the possibility to the `GroupParamType` to create groups if they don't exist * `aiida.backend*`: - Rename `type` and `name` to `type_string` and `label` for the database models * Improve documentation for django and sqla backends migration | 13 December 2018, 16:56:36 UTC |
| f9b9b8b | Martin Uhrin | 13 December 2018, 09:27:19 UTC | Move interface of raw SQL execution to the Backend class (#2346) Moved from `QueryManager` to the corresponding backend class. The longer term view will probably be that `QueryManager` gets dropped completely and more backend specific methods will be moved into the `Backend` class itself. This way we have a single point where we know to find such methods. | 13 December 2018, 09:27:19 UTC |
| a13919c | Martin Uhrin | 12 December 2018, 16:05:52 UTC | Allow deepcopy of stored data nodes (#2347) The `expose_inputs` of the `ProcessSpec` is implemented by `plumpy` and it recursively deep copies the ports of the original spec to the target spec. This includes deep copying any default values that those ports may have, which may also be a stored `Data` node. Originally we prohibited the deep copying of stored `Data` nodes, because it may suggest that an actual copy of the node would also be created in the database, while that is not the case. The decision was made to simply raise when trying to deep copy a stored data node. However, this breaks the `exposed_inputs` functionality as it is perfectly legal to set a stored data node as a port default. Since we have to allow users to set stored data nodes as a port default, we have to lift the ban of deep copying them. | 12 December 2018, 16:05:52 UTC |
| 7ff016d | Sebastiaan Huber | 12 December 2018, 13:09:04 UTC | Use `ProfileParamType` in `verdi` and expose in `ctx.obj` (#2345) The `verdi` entry point now uses the `ProfileParamType` for its `-p` option, which enables tab-completion for the profile option. In addition, we add the loaded profile to the `obj` attribute of the click context `ctx`. This way, when a `verdi` sub commands needs the active profile, one can simply use the `@click.pass_context` decorator to access the profile through `ctx.obj.profile`. | 12 December 2018, 13:09:04 UTC |
| 11e6988 | Leopold Talirz | 12 December 2018, 12:17:53 UTC | Docs reorganization installation section (#2330) * quick install now contains all necessary information to actually set up AiiDA * foldable sections with instructions for different operating systems keep quick install page short * detailed instructions for different OS's retained under "prerequisites" section and linked directly from quick install * OS-specific sections on postgres and rabbitmq merged into corresponding "prerequisites" sections | 12 December 2018, 12:17:53 UTC |
| 4d0c787 | yakutovicha | 12 December 2018, 11:27:23 UTC | Merge pull request #2339 from ltalirz/add_license_to_manifest add license file to manifest | 12 December 2018, 11:27:23 UTC |
| c87a0db | Sebastiaan Huber | 12 December 2018, 11:25:36 UTC | Implement `verdi database integrity` endpoints for links and nodes (#2343) These two new end points will scan the database for invalid links and nodes, by running prepared SQL statements. If any violations are found, they will be printed to the terminal. For now there are no options implemented to apply a patch to remove the violations. | 12 December 2018, 11:25:36 UTC |
| 1673ec2 | Sebastiaan Huber | 12 December 2018, 07:59:04 UTC | Update export schema version and temporarily disable export (#2341) The migration after the provenance redesign has been merged into `provenance_redesign`. To avoid people from importing "dirty" export files into a cleanly migrated database, the export schema version is upped. Additionally, creating new export archives with the latest version is disabled, until the export procedure has been verified to abide by the new rules in place after the provenance redesign. | 12 December 2018, 07:59:04 UTC |
| 42bc788 | Sebastiaan Huber | 07 December 2018, 18:04:41 UTC | Add checks for unexpected links in provenance redesign migration The migration for the provenance redesign will delete illegal links such as `return` and `call` links outgoing from calculation nodes. Since the migration will simply delete them, before we do so, we detect these illegal links, and if found will print a warning to the console and write the offending links to a migration log file. This way, the user could potentially restore the deleted links. | 11 December 2018, 15:35:05 UTC |
| 6a1d7f3 | Giovanni Pizzi | 07 December 2018, 14:33:00 UTC | Add migrations to properly deal with `ProcessCalculation` and `WorkCalculation` There has been a period, at the very beginning of the move to processes where the node classes had the type `calculation.process.ProcessCalculation.` These could represent either the execution of a workfunction or a workchain. Therefore, we first convert all `ProcessCalculations` to `WorkCalculations` and in a second step, we convert all `WorkCalculations` that have an attribute called `function_name` to `WorkFunctionNode`. The only remaining `WorkCalculations` then have to be `WorkChainNodes`. | 11 December 2018, 15:35:05 UTC |
| ad89d51 | Sebastiaan Huber | 04 December 2018, 21:26:19 UTC | Implement the database migrations after the provenance redesign The migrations consist of two steps: * Inferring the node process type of the node type * Migrating the node type, deleting illegal links and renaming links The process type column was added sometime ago, which is used to store the entry point string of the process class that led to the creation of the node. This is mostly applicable for job calculations and workflows. Since the node type is the same for all calculations and workflows, the process type is necessary to keep the information of the process class that ran it. Older databases may contain job calculations and workchains that were run before the code was updated to include the process type. There the information for what sub class, of especially the JobCalculation, was contained in the node type. Since in this migration, the node type will be migrated to be just the base CalcJobNode, with type string `node.process.calculation.calcjob.CalcJobNode.`, the information of the calculation sub class would be lost. Therefore, before migrating the node types, we attempt to infer the entry point string from the node type string and store that in the process type column. Node type strings that cannot be mapped on a known entry point string will get a fallback process type string and are logged to a file. After this is completd, a normal database migration is applied, where illegal links are deleted, the remaining renamed according to the new rules and names and the node types are renamed according to the new node ORM class type strings. | 11 December 2018, 15:35:05 UTC |
| bcdb46c | Leopold Talirz | 11 December 2018, 14:22:36 UTC | add license file to manifest required by license and by conda-forge | 11 December 2018, 14:22:36 UTC |
| 454bfc3 | ConradJohnston | 11 December 2018, 11:03:16 UTC | Add test for availability of sudo in quicksetup (#2333) Quicksetup needs to access the Postgres database, but does not know anything about the system setup. It initially tries to see if it is the postgres superuser, but if it is not, it tries to become the postgres superuser via sudo. This is the backstop position and relies on the user having a modern Linux-like system. If, however, the sudo command is not available to this user at all (for example, perhaps on an HPC cluser), an error will be raised and quicksetup will fail ungracefully. This commits add a test to see if sudo can be found before attempting to use it. If it is not, no attempt is made, and the final manual setup message is printed. - Add simple check of the availability of `sudo` command to the user. - Add warning message if `sudo` cannot be found. - Update `click.echo` calls to the AiiDA `echo` function. - Update selected messages to 'warning' type messages. | 11 December 2018, 11:03:16 UTC |
| 64bf428 | Sebastiaan Huber | 11 December 2018, 09:13:08 UTC | Implement `verdi database migrate` (#2334) Up till now, the triggering of a database migration happened through separate interfaces for the two different backends. For Django, if the database schema did not match the code's, a warning message would be printed asking the user to execute a stand alone python script that was a modified Django `manage.py` script. For SqlAlchemy, the user was automatically prompted with an Alembic migration. Here we unify the migration operation by making it go through `verdi` through the endpoint `verdi database migrate`. The error message upon an outdated database schema and the to be executed commands are now identical for both backends. | 11 December 2018, 09:13:08 UTC |
| c9a1d7f | Sebastiaan Huber | 10 December 2018, 09:10:18 UTC | Add `CalculationTools` base and entry point `aiida.tools.calculations` (#2331) With the migration to the new provenance design, the type string of calculation nodes, no longer refer to the actual sub class of the `JobCalculation` that was run, but just the base `CalcJobNode`. The actual class of the calculation process was stored in the `process_type` if it could be mapped onto a known entry point. However, this change means that when a user now loads a node of a completed calculation node, let's say `PwCalculation`, the loaded node will be an instance of `CalcJobNode` and not `PwCalculation`, which means that any utility methods defined on the `PwCalculation` class are inaccessible. We can return this functionality through the concept of calculation tools, which will get exposed through the `CalcJobNode` class and will load a specifically registered entry point. | 10 December 2018, 09:10:18 UTC |
| a64d5c1 | Martin Uhrin | 08 December 2018, 12:54:09 UTC | Ensure QueryBuilder return frontend entity of DbLog and DbComment (#2325) Also renamed the ORM moduels `logs` and `comments` to be plurals | 08 December 2018, 12:54:09 UTC |
| ada95a1 | Martin Uhrin | 08 December 2018, 09:10:04 UTC | Delete process queues when process terminates (#2324) The fixes in `plumpy` and `kiwipy` should now ensure that response queues for a specific process should now be closed properly when the process is stopped. Also changed the process tests to use close() because it is illegal to have two active process instances that refer to the same underlying process instance (i.e. with the same id) Finally, change the way that validation of ports is done which now also shows which port a validation error came from | 08 December 2018, 09:10:04 UTC |
| a6bbe32 | Martin Uhrin | 05 December 2018, 22:04:17 UTC | Removed `get_aiida_class` Replaced all instances with either `get_backend_entity` or `get_orm_entity` (as appropriate). Also changed backend group to be independent of the ORM. Because the iterator was getting the ORM class and returning that, which required it to access up to the ORM level. Now it just returns backend entities which are in turn converted to ORM entities by the `orm.Group`. This also means you can't call `len(group.nodes)` because this is not allowed for iterators but the same can be achieved with `len(list(group.nodes))` | 07 December 2018, 18:58:51 UTC |
| 331eb2b | Spyros Zoupanos | 05 December 2018, 15:39:33 UTC | Removal of `get_aiida_class` from `QueryBuilder` Added convertors to go from a DbModel BackendEntity and from BackendEntity to OrmEntity. | 07 December 2018, 18:58:51 UTC |
| 05fae57 | Leopold Talirz | 07 December 2018, 18:10:50 UTC | Document how to enable tab completion in conda (#2316) | 07 December 2018, 18:10:50 UTC |
| 77a5dd3 | zhubonan | 07 December 2018, 13:58:33 UTC | Document the Windows Subsystem for Linux (#2319) Added guide for installing AiiDA on Windows sub-system for Linux | 07 December 2018, 13:58:33 UTC |
| ed17a36 | Leopold Talirz | 07 December 2018, 11:17:30 UTC | Upgrade prospector (#2317) Dependency `prospector==1.1.5` finally works with `pylint<2.0.0` | 07 December 2018, 11:17:30 UTC |
| 453f500 | Giovanni Pizzi | 07 December 2018, 10:15:03 UTC | Various speedups for `verdi` command line (#2315) In particular when completing tests, thanks to @ltalirz. - Using a new instance of the reentry manager that prevents automatic reentry scans - Removed pointless `with_dbenv` decorators from `verdi devel tests` and from the `TestModuleParamType`. - Prevent unnecessary imports of `plumpy` that is a bit slow to import - Changed point where `AIIDADB_PROFILE` is set in the main `verdi` command, to set it also when a default profile is used. - The plugin manager is now always using the same global reentry manager (that does NOT automatically re-run `reentry scan`, which slows things down). - Ignoring `-p` flag in `verdi setup` when using profile name as argument - A default profile is now set, when none is set yet, when running non-interactively, which is what `verdi quicksetup` does. | 07 December 2018, 10:15:03 UTC |
| 5057580 | lekah | 07 December 2018, 08:44:03 UTC | Check for the existence of groups on import (#2270) The import will now check if a group that is to be imported has a name that already exist in the database. Instead of raising a unique name is generated, by iteratively incrementing a counter at the end of the name, until a unique name is found. | 07 December 2018, 08:44:03 UTC |
| c62e63a | ConradJohnston | 07 December 2018, 08:06:32 UTC | Document the purpose and use of ``clean_value`` (#2295) - Update the docs to explain the use of `clean_value` and its behaviour. - Update `clean_value` doc string to better explain its purpose. - Change collection imports from, for example, `import collections.Iterable` to `from collections import Iterable` to comply with a forthcoming deprecation that will break these imports in Python 3.8. | 07 December 2018, 08:06:32 UTC |
| 7a78f8a | Andreas Stamminger | 07 December 2018, 07:17:11 UTC | Improve robustness of parsing versions and element names from UPF files (#2296) * Change regex for element matching to match case insensitive * Remove case-insensitive matching Just realized this is not necessary * Change regex matching to match whole string instead of lines. * Improve comment * Add tests for UPF parser | 07 December 2018, 07:17:11 UTC |
| c96b78e | zhubonan | 06 December 2018, 23:47:25 UTC | Add documentation for plugin fixtures (#2314) * WIP created plugin testing guide * Improve the code snippet Also correction of type errors. * Fix typo * Small change of wording | 06 December 2018, 23:47:25 UTC |
| 818888f | Martin Uhrin | 06 December 2018, 22:15:47 UTC | Removed pika log level (#2312) This was added before because of issues #1249 but this no longer happens as we're using separate a thread for the pika event loop and what's more the issue has, I believe, been resolved with pika 1.0 which we now use. | 06 December 2018, 22:15:47 UTC |
| d0a9ad9 | Sebastiaan Huber | 06 December 2018, 20:28:44 UTC | Migrate the type and process type string of built in calculation plugins (#2308) The built in calculation entry points were recently moved and renamed but no migration was put in. This requires renaming the node types, process types and the input plugin attribute of code instances. | 06 December 2018, 20:28:44 UTC |
| d4dce2d | Leopold Talirz | 06 December 2018, 16:46:44 UTC | Move requirements from setup_requirements.py to setup.json (#2307) | 06 December 2018, 16:46:44 UTC |
| 782f38b | Snehal Kumbhar | 06 December 2018, 14:53:28 UTC | REST API: fix structure visualisation end point. (#2305) | 06 December 2018, 14:53:28 UTC |
| ff4c8d3 | Leopold Talirz | 06 December 2018, 13:46:30 UTC | Move practical developer docs to AiiDA wiki (#2297) * move git cheatsheet * move documentation on writing tests * merged sphinx cheatsheet from aiida wiki with sphinx cheatsheet in docs (note: this contains some useful hints and would not work on the github wiki) * remove outdated data_cmdline replaced by `developer_guide/devel_tutorial/cmdline_plugin.rst` | 06 December 2018, 13:46:30 UTC |
| 06d06bc | Giovanni Pizzi | 06 December 2018, 13:08:27 UTC | Disabling the possibility to store base Nodes and ProcessNodes (#2301) The main work is actually fixing all tests to avoid to use Node directly. This also allowed to spot a few bugs and inconsistencies in the tests. | 06 December 2018, 13:08:26 UTC |
| 7bab498 | Elsa Passaro | 06 December 2018, 11:59:37 UTC | Merge pull request #2302 from asle85/add_provenance_docs Add provenance docs | 06 December 2018, 11:59:37 UTC |
| d4530b4 | epassaro | 06 December 2018, 11:17:08 UTC | Merge branch 'provenance_redesign' into add_provenance_docs | 06 December 2018, 11:17:08 UTC |
| b18e6d9 | epassaro | 06 December 2018, 11:14:53 UTC | added latest changes to provenance docs | 06 December 2018, 11:14:53 UTC |
| 9891508 | zhubonan | 06 December 2018, 10:06:47 UTC | Updated documentation about the usage of `CodeInfo` (#2276) Both `CalcInfo` and `CodeInfo` are disscussed already in the two examples (QE and integer summation). Note about the support of multiple codes for a single calculation is added. This feature was not clearly documented earlier. A leftover QE example in the plugin development guide is removed. Also improved the integer/float summation examples with links to direct user to general plugin system documentation and online examples. | 06 December 2018, 10:06:47 UTC |
| 18ec77c | epassaro | 06 December 2018, 09:38:27 UTC | Merge branch 'provenance_redesign' into add_provenance_docs | 06 December 2018, 09:38:27 UTC |
| 85d7309 | zhubonan | 05 December 2018, 17:47:46 UTC | Improved setup of ipython line magic %aiida (#2272) Instead of telling users to edit the `ipython_config.py`, which does not exist by default. We now suggest simply add a python file in the startup folder of ipython to register the line magic `%aiida`. A new file is added serving as an example. | 05 December 2018, 17:47:46 UTC |
| c306854 | epassaro | 05 December 2018, 17:15:09 UTC | added provenance concept docs | 05 December 2018, 17:15:09 UTC |
| 8f3f5b5 | Andreas Stamminger | 05 December 2018, 16:19:57 UTC | Make test for unicode folders skippable. (#2283) Skip test for unicode folder names if the filesystem encoding is not set to UTF-8, because otherwise it will always fail. | 05 December 2018, 16:19:57 UTC |
| 6623718 | Snehal Kumbhar | 05 December 2018, 15:33:16 UTC | Remove wsgi file/configuration/documentation for django application (#2293) | 05 December 2018, 15:33:16 UTC |
| 86b4a13 | Sebastiaan Huber | 05 December 2018, 14:34:16 UTC | Ensure correct types of `LinkTriples` returned by `Node.get_stored_link_triples` (#2292) The `Node.get_stored_link_triples` was using `LinkTriple` instances where the type of the `link_type` element was not of `LinkType` but of string, as it was directly using the string value returned by the query builder for the `type` column of the `DbLink` table. By wrapping this value in `LinkType` the returned `LinkTriples` once again have the correct type for all its elements. | 05 December 2018, 14:34:16 UTC |
| 44774b2 | epassaro | 05 December 2018, 13:37:09 UTC | added graph examples | 05 December 2018, 13:37:09 UTC |
| 88026c7 | Conrad Johnston | 04 December 2018, 16:06:37 UTC | Replace use of os.system with subprocess.call - Replaces the use of os.system in aiida.common.graph with subprocess.call The former was dangerous as it was passed arguments directly from the cmdline via verdi graph and so would execute and valid shell string, The solution is to use subprocess.call() which sanitises the input automatically. - Also prints a more usful message if graphviz is possibly not installed. | 04 December 2018, 22:11:24 UTC |
| c40a9b8 | Conrad Johnston | 04 December 2018, 14:39:56 UTC | Document graphviz dependency The use of verdi graph to generate a plot of part of the provenance graph requires the graphviz package, which was not documented other than in the source code. This commit increases the visibility of this dependency in the docs. | 04 December 2018, 22:11:24 UTC |
| 074c577 | Sebastiaan Huber | 04 December 2018, 20:29:21 UTC | Temporarily lock versions of `kiwipy` and `pika` (#2290) A new patch version v0.3.11 of `kiwipy` was released that breaks functionality, so until that is fixed, we lock the version. We do the same for `pika` which requires `v1.0.0b1`. | 04 December 2018, 20:29:21 UTC |
| af61105 | ConradJohnston | 04 December 2018, 11:18:01 UTC | Change `ancestor_of/descendant_of` to `with_descendants/with_ancestors` (#2278) * Deprecate `ancestor_of` and `descendant_of` * Replace all occurrences of `ancestor_of` with `with_descendants` * Replace all occurrences of `descendant_of` with `with_ancestors` * Remove `_beta` version of recursive `ancestor_of` and `descendant_of` joins * Update docs with this convention | 04 December 2018, 11:18:01 UTC |
| 1adafc4 | Leopold Talirz | 04 December 2018, 10:40:02 UTC | Reorganization of the documentation structure (#2279) Top-level reorganization as first step of restructuring the documentation. | 04 December 2018, 10:40:02 UTC |
| 13dcc03 | lekah | 04 December 2018, 10:05:59 UTC | Force SQLAlchemy to not de-duplicate rows (#2281) For the original SqlAlchemy implementation the following equation did not hold: qb.count() == len(qb.all()) This was because SqlAlchemy was de-duplicating the query set in the `all()` call. The breaking of this equality was inconsistent with the Django implementation and the result when operating directly on the database. The postgres behavior is now restored by setting `self._query._has_mapper_entities = False` on the SqlA query object. This is a temporary fix and will soon be released as a public option in a new version of SqlAlchemy. | 04 December 2018, 10:05:59 UTC |
| da6acac | yakutovicha | 04 December 2018, 09:13:37 UTC | Raise exception when exporting a node without `path` sub folder in repository (#2274) This most likely occurred when a user manually deleted directories from the repository and then recreated them, without also creating the sub folder that is currently used by default. | 04 December 2018, 09:13:37 UTC |
| 6c6b890 | Leopold Talirz | 03 December 2018, 22:51:58 UTC | Update supported versions of python (#2277) Specify exactly which versions of python `aiida` is supposed to run on. Also see page 6 of presentation by @dev-zero on python3 compatibility in `aiida-core` from Sep 13th 2018. | 03 December 2018, 22:51:58 UTC |
| df42835 | lekah | 03 December 2018, 22:14:48 UTC | Changed type of uuids returned by the QueryBuilder to be unicode (#2259) | 03 December 2018, 22:14:48 UTC |
| 8293da3 | Andreas Stamminger | 03 December 2018, 21:29:42 UTC | Improve backward compatability of `verdi graph generate` (#2275) Escaped newline chars added to labels during graph generation such that newline characters are translated verbatim to the dot file. This fixes the "string ran past end of line" error encountered when using older versions of graphviz. | 03 December 2018, 21:29:42 UTC |
| a7fa45c | Andreas Stamminger | 03 December 2018, 20:38:42 UTC | Remove preceding '$' chars from code-blocks. (#2273) Simplify copy-and-pasting commands from the documentation by removing the preceding `$` chars indicating bash-commands. Note that `$` are kept in the command if it is not meant for copy-and-pasting, i.e. if it is prefixed by some python environment name, indicating that the command is to be run inside the environment. | 03 December 2018, 20:38:41 UTC |
| d015237 | Sebastiaan Huber | 03 December 2018, 17:38:39 UTC | Add `--group` option to `verdi process list` (#2254) This option allows a user to narrow down the query set to only those nodes that are within the specified group. | 03 December 2018, 17:38:39 UTC |
| 66fbf10 | Sebastiaan Huber | 03 December 2018, 15:50:31 UTC | Remove work around for postgres on Travis (#2266) | 03 December 2018, 15:50:31 UTC |
