https://github.com/aiidateam/aiida_core

sort by:
Revision Author Date Message Commit Date
0060786 Release `v1.0.0b1` (#2593) 12 March 2019, 14:29:13 UTC
d2c1493 Merge pull request #2590 from aiidateam/merge_master_into_develop This merges the changes of `v0.12.3` into `develop` 12 March 2019, 12:49:11 UTC
44dcf1b adapt creation of environment.yml * skip requirements specified for specific versions * switch to OrderedDict in order to have sensible order of keys in environment.yml 12 March 2019, 11:26:35 UTC
fe76c17 Fix version comparison in import export The export version number in newer `aiida-core` versions is no longer a string but a float. Converting both to `StrictVersion` should solve the problem. The import of `StrictVersion` from `distutils` triggered an error in `pylint` and therefore `distutils` is currently ignored until the issue is addressed: https://github.com/PyCQA/pylint/issues/73 12 March 2019, 09:22:49 UTC
7ef21e6 Refactor `validate_pyproject.py` * rename to `validate_consistency.py` * merge with `validate_version.py` * fix unicode issue for `environment.yml` 12 March 2019, 09:22:38 UTC
45ac3c7 Merge branch 'master' of github.com:aiidateam/aiida_core into merge_master_into_develop 12 March 2019, 09:11:42 UTC
ca18340 Merge pull request #2589 from aiidateam/provenance_redesign Provenance redesign 11 March 2019, 16:14:13 UTC
f8b0941 Fix typo in `QueryBuilder` docs for "ilike" operator (#2587) 11 March 2019, 13:17:03 UTC
0260a1d Fix bug in `TemplatereplacerCalculation` input check (#2583) Variable `input_through_stdin` was not considered correctly as it was always a boolean, and the conditional was incorrectly checking for `is not None`. 11 March 2019, 12:12:26 UTC
cc08280 Fix test for multiple log imports (#2576) * Import/Export tests: Add `@with_temp_dir` decorator. * Fix `test_multiple_import_for_single_node` in `TestLogs`: Do not rely on ordering of log uuids 07 March 2019, 15:13:53 UTC
e64a607 Allow the definition of `None` as default in process functions (#2582) It was impossible to define a process function with a keyword argument that had `None` as default, because the dynamically created ports always only specified `orm.Data` as valid types. So the validation of the default value would fail during the spec definition. Here we detect if `None` is passed as the default in the function signature, in which case we define the tuple of `orm.Data` and `type(None)` to be a valid type. Note that we need to use `type(None)` because the port validation later on will call `isinstance(input, valid_types)` which will not work if one of the values in `valid_types` is simply `None`. The `Process._setup_inputs` had to be adjusted to skip values of `None` because they cannot be linked to obviously but they can now potentially be passed as inputs to a `Process`. 07 March 2019, 08:06:38 UTC
838c1dd Redefine the structure of the `CalcInfo.local_copy_list` (#2581) With the upcoming change of the node repository, files within it can no longer be assumed to necessarily be stored on the local file system. It is therefore impossible to address files within it through an absolute file path, however, that is exactly what the `local_copy_list` expects as the first item of its tuples. Now, the `Node` class only exposes methods to get either the content of a repository file or an filelike object. To comply with this change in design the structure of the `local_copy_list` tuples is changed to tuples of length three where each element represents: * UUID of the node * Relative key of the file within the node repository * Relative path within the working directory where to copy the file The `upload_calculation` function of the execmanager has been updated to this interface change, but because our `Transport` interface does not yet provide a method to put an object from a filelike object, we have to create a temporary file on the local file system whose absolute filepath can be passed to the `Transport.put` call. Once the transport is updated to provide a `put_object_from_filelike`, this inefficiency can be removed. 07 March 2019, 01:10:31 UTC
277a4d8 Remove deprecated `verdi calculation` and `verdi work` (#2574) Their shared functionality was moved to `verdi process` and the remaining specific commands for `CalcJobNodes` have found a home in the `verdi calcjob` command. 05 March 2019, 20:39:56 UTC
8b4eed0 Add page to documenation with justification for design changes (#2571) In this case particularly, add justifications for the provenance redesign the calculation job redesign and the new module hierarchy along with its API stability policy. 05 March 2019, 12:40:49 UTC
b45c6c7 Add tab-navigation methods to (some) nodes This fixes #2230. In particular, now: - CalculationNode have a `.inputs.<LABEL>` property that returns a manager to navigate incoming INPUT_CALC nodes - CalculationNode have a `.outputs.<LABEL>` property that returns a manager to navigate outgoing CREATE nodes - WorkflowNode have a `.inputs.<LABEL>` property that returns a manager to navigate incoming INPUT_WORK nodes - WorkflowNode have a `.outputs.<LABEL>` property that returns a manager to navigate outgoing RETURN nodes - Node *does not have anymore* `.inp` and `.out` as these were referring to incoming and outgoing and were ambiguous Moreover, `Data.created_by` and `ProcessNode.called_by` methods were already implemented and have been renamed to `Data.creator` and `ProcessNode.caller`. Also tests added for all these methods, including the already existing ones. 05 March 2019, 11:56:07 UTC
97dbda7 Add LinkType to the classes pre-loaded in the shell Fixes #2560 05 March 2019, 11:56:07 UTC
6272507 Ensuring that get_node_by_label raises In the case there is more than one matching node with the same label. This fixes #2558 05 March 2019, 11:56:07 UTC
d39007d Importing `metadata` for `Log`-entries properly. (#2486) The test tries to test the case in issue #2448, where a specific Node has a few Log messages. Then one wants to import a file that has more Log messages, pertaining to the exact same Node. The test checks that all Log messages are imported according to their UUID. Added ex-/import test: `test_reimport_of_logs_for_single_node` Added simple test: `test_export_of_imported_logs`. The test checks if one can re-export imported Logs. Correct import of Logs into sqla from django. Check a Log's metadata is JSON serializable upon construction. Add test of Logs to check correct metadata format. Add type check of metadata when constructing new Log. Make sure uuid is a string when retrieving it for comparisons, during import with SQLA. This was already the case for import with django. This is crucial, since Log.uuid returns a UUID() type. To make a proper comparison and ensure we do not try to insert an existing Log, this MUST be converted to a string, i.e. str(UUID()) Tests for importing existing comments. Added `comment_mode` to import, which may be: - `'newest'`: Will keep the Comment with most recent mtime. - `'overwrite'`: Will overwrite existing Comments with the ones from the import file. Added `delete_many` method to comments. Acts within a session for SQLA. Minor fixes to import of extras. Changed default `extras_mode_new` from `'ignore'` to `'import'` in `import_data_sqla`. This is to make it the same as for `import_data_dj`. Furthermore, `'ignore'` was not a valid value. Update docstrings of import_data* functions Check `mtime` is the same before and after export/import. Both for Node and Comment. Centralized import check of Comments in `_merge_comment` function. It will update existing Comments if needed, NOT delete entries. Updated `delete_many` method to accept filters. Both for Logs and Comments and both for django and SQLA. Add `comment_mode` to `verdi import`. Add `metadata` check for exprot/import of Log. Replace all mentions of issue #2342 with issue #2426. Redesign `test_reimport_of_logs_for_single_node` and `test_reimport_of_comments_for_single_node`. Create separate two new tests: `test_multiple_imports_for_single_node` for both Logs and Comments. Co-authored-by: Spyros Zoupanos <spyros.zoupanos@epfl.ch> 05 March 2019, 07:25:53 UTC
2c54be6 Add efficient `count` method for `aiida.orm.groups.Group` (#2567) Before the only way to get the number of entries in a `Group` was through the iterator `nodes` and call `len()` on it. However, this is almost always not the fastest way. Given that the count is used often in for example `verdi group list -C` it is worthwhile to have a `count` method that operates as direct on the database layer as possible. 05 March 2019, 06:40:50 UTC
28b0c82 Prevent launching or construction of `WorkChain` and `CalcJob` base class (#2564) The `WorkChain` and `CalcJob` classes are base classes that are meant to be sub classed as they miss the necessary functionality to successfully run. However, it was still possible to launch these base classes through the launchers or straight up constructing them. They would immediately fail when launched, but they would create a node in the provenance graph nonetheless. To prevent this from happening, a check is placed in the constructors to make sure that it is actually a sub class that is being constructed. 04 March 2019, 22:50:38 UTC
76c9037 Add `uuid` property to the `Process` class (#2566) This serves as a proxy to the UUID of the underlying `ProcessNode`. 04 March 2019, 22:08:31 UTC
2c89388 Add an index to columns of `DbLink` for SqlAlchemy (#2561) Indices are added for the `input_id`, `output_id` and `type` column of the `DbLink` model for SqlAlchemy. This makes it congruent with the Django model and ensures that the engine will not grind to a halt for bigger production databases which often needs to get all incoming or outgoing links for a node. 04 March 2019, 21:24:41 UTC
5592f81 Merge pull request #2556 from aiidateam/support/0.12.x Release v0.12.3 04 March 2019, 11:54:21 UTC
1b5b269 Remove any use of our ORM in the migrations and tests Many of the migrations and their respective unit tests used the ORM of `aiida-core` or the database model classes directly. However, as the database models evolve and change, during migrations one cannot rely on the current implementation of the database models or the ORM as it may not be commensurate of their state at the time of the migration. This means that the migrations absolutely have to be written relying only on raw SQL or on the model classes that are provided through the migration mechanism of the various backends, as they will make sure that those fake classes reflect the state of the models at the time of the migration. 04 March 2019, 10:59:01 UTC
dbcb7f8 Renaming `DbNode.type` to `DbNode.node_type` 04 March 2019, 10:59:01 UTC
1a757c7 Re-enable export functionality with exception of migrations. (#2554) This re-enables the possibility to create export archives in the new format with version `v0.4`. The migration of existing archives with an older version is not yet implemented. Tests that rely on this functionality have been temporarily disabled. The logic of the `verdi export migrate` is, however, already adapted to support automatic recursive migration from any version to the latest. 03 March 2019, 13:38:37 UTC
8bf2445 Update install docs (#2555) Backport streamlined quick install instructions from `provenance_redesign` 02 March 2019, 19:30:31 UTC
b40b2ca Add option to circumvent ORM in `Group.add_nodes` for SQLA (#2518) This option is only available at the `BackendGroup` level and therefore only to users who need this speedup (the ORM level users should not be aware of such options). This was originally added to speed up the import of archives for SQLA backends whose implementation was modified to use this optimization. 02 March 2019, 16:12:03 UTC
2db295e Update authors and bump version to `0.12.3` (#2553) 01 March 2019, 17:58:15 UTC
da6bb4d Implement support for process classes to `QueryBuilder.append` (#2421) Any `Process` class will have an associated `ProcessNode` sub class associated with it that it will use to store its provenance when ran. In addition with the `process_type`, determined by its specific process sub class entry point, defines a set of nodes that correspond to that process class. For example: the `ArithmeticAddCalculation` will use a `CalcJobNode` to store its executions and has a process type according to its entry point: `aiida.calculations:arithmetic.add` This commit implements the functionality necessary that allows a user to append the `ArithmeticAddCalculation` process class to a query builder instance directly. 01 March 2019, 16:15:06 UTC
4806ba1 Update changelog for release `0.12.3` (#2455) 01 March 2019, 14:26:54 UTC
29498f2 bug fix for `verdi node delete` (#2545) Fix a bug such that if no argument is passed to `verdi node delete`. It will try to delete all notes in the database. This is because `QueryBuilder().append(Node, filters={'or': []}, project='id').all()` match to every single node and projects its id. Perhaps this behaviour of `QueryBuilder` is not as expected? 28 February 2019, 20:33:49 UTC
f757e55 Fix BandsData issue related to the k-points labels (#2492) fixes #2491 * fix problem with empty labels array * Fix bands plotting If the first and/or the last kpoint label does not point to the first and/or the last kpoint - the plotted band structure was cut on the left and/or right sides. To avoid this I add empty labels to the first and/or the last kpoints if they are not labeled. * Add documentation on how to use the BandsData object 28 February 2019, 19:28:40 UTC
56344da Make various protected `Node` methods public (#2544) These methods are intended to be called from outside the class hierarchy but were made protected (by prefixing them with an underscore) merely to hide them from auto-completion and signal to users that they should not be called by the user. However, this is the wrong usage of this feature and methods that should be callable from the outside should not be protected. 27 February 2019, 18:18:56 UTC
acba3bd Temporarily pin `pylint==2.2.2` and `astroid==2.1.0` for python 3 (#2543) Version `2.3.0` of `pylint` that was released two hours ago, breaks on python 3 with their `pylint-plugin-utils==0.4`. See `pylint-plugin-utils` issue: https://github.com/PyCQA/pylint-plugin-utils/issues/12 Similarly, `astroid==2.2.0` was just released which also causes pylint to fail on python 3 so we pin this to `astroid==2.1.0`. See `astroid` issue: https://github.com/PyCQA/astroid/issues/649 27 February 2019, 14:53:51 UTC
80a0aa9 Improve the efficiency of `DbLog` migration for SqlAlchemy (#2540) The old implementation was using a `NOT IN` with the list of all node ids to filter all log entries that had an `objpk` that did not correspond to a node entry. This is extremely inefficient as it loads the entire node pk column in memory and has to traverse it for each log entry. In this commit, we change this to a construct that will simply search for the `objpk` in the node table, which is indexed and simply returns 1 if it exists, which is than checked in the filter of the logs. 27 February 2019, 10:24:10 UTC
59b05ad Simplify imports from the `aiida.orm` (#2534) All the important classes are now exposed on the `aiida.orm` level which allows to simplify import statements from: `from aiida.orm.nodes.data.structure import StructureData` to `from aiida.orm import StructureData` Also the repository was scanned for instance of the pattern `" "` in string literals, which occurred due to `yapf` joining a multiline string onto a single line. 26 February 2019, 23:09:02 UTC
85fc1c4 Rename `ParameterData` to `Dict` (#2530) 26 February 2019, 21:28:26 UTC
32be995 Fix and complement the list of `verdi shell` auto imports (#2533) 26 February 2019, 07:55:18 UTC
6315c6e Remove the `FrozenDict` data sub class (#2532) The `FrozenDict` class was added to allow `ProcessNode` instances to be passed to `Process` classes. However, its implementation is flawed as it uses the pks of the nodes it wraps, which are not persistent across databases and as such provenance will be lost when it is imported. We remove the source code here. In a future commit, it will have to be decided how to migrate existing data. 26 February 2019, 06:53:11 UTC
a3e36e5 Reenable transport plugin tests (#2531) A few transport plugin tests had been commented out because due to a change on Travis they started failing. The reason was that Travis added files to the `.bashrc` that depended on the `TRAVIS_HOME` environment variable to be set, which was not the case if one SSH'ed to localhost. This caused the sourcing of the `.bashrc` to fail during the SSH transport tests in which we SSH to localhost where this environment variable is not set. As a workaround we add a step in the before install section of the `.travis.yml` where we use `sed` to replace the `TRAVIS_HOME` variable for `HOME` which will in fact be set. 26 February 2019, 06:10:20 UTC
562935c Remove the `Error` data sub class (#2529) This was added at one point for the reimplementation of legacy workflows and started out as an internal exception wrapper but was ultimately never used. 25 February 2019, 18:39:12 UTC
c7de4f7 Formalizing importable resources on second level modules (#2528) The convention for those resources, e.g. classes and functions, that should be usable by users of `aiida-core`, is that they should be importable directly from a second level module in the hierarchy. That is to say, anything that can be imported directly from `aiida.orm` is intended to be used by users and therefore these resources come with the guarantee that if they will be changed or moved, this will be done with a decent deprecation path. 25 February 2019, 17:54:30 UTC
221f308 Reorganize the `aiida.work` module (#2524) The `aiida.work` module was conceived when the `WorkChain` class was first implemented. Ever since, the `Process` class on which it is based starting filling are more prominent role and slowly all running processes were migrated to be a `Process` sub class. All of the logic was added to the `aiida.work` module, which turned into the module that implemented all the functionality of the engine of `aiida-core`. Therefore we rename the `aiida.work` module to `aiida.engine` and reorganize the internal structure to clarify the role of the different components. 25 February 2019, 14:38:22 UTC
1a3426e Update python dependencies (#2520) This PR updates all python dependencies of `aiida-core` to the latest version, except for: * `pycifrw` (version 4.4 still has an issue with python2, pinged maintainer about new release) * `ipython` (6 drops python2 compatibility) * `pg8000` (<1.13.0 was hardcoded) * `tornado` (5 drops python2 compatibility) * `pymatgen` (2019 drops python2 compatibility) * `prospector` (we're currently using later versions of `pylint-django` than `prospector` demands because the versions demanded by `prospector` make pylint crash. While `prospector` 1.1.5 was fine with us increasing the `pylint-django` version, `prospector` 1.1.6.2 gets touchy and refuses to run, see [here](https://travis-ci.org/ltalirz/aiida_core/jobs/497709384#L1514)) I.e. we're pretty much up to date, except for dependencies that drop python2 compatibility. 25 February 2019, 13:31:26 UTC
2e4ab93 Add `verdi status` command (#2487) New command that shows the status of the various services required by `aiida-core` such as Postgres and RabbitMQ. 25 February 2019, 11:19:32 UTC
f8c0191 Make the node repository API backend agnostic (#2506) Initially, the only backend supported for the file repository of nodes was the local file system. In the near future, additional backends such as an object store or a remote file system will be implemented. For this to work the API of the node repository will have to be adapted because now it is geared to work with a local file system. To prevent too many changes after the release of the beta, we already define the API here that should be able to support the different repository backends that will be implemented in the future. 22 February 2019, 17:30:50 UTC
7b1c90a Fixing bug related to deleted nodes in recent log migration (#2507) In a recent database migration, `0024_dblog_update` for Django and `041a79fc615f` for SqlAlchemy, the `DbLog` table was updated where the `objpk` was replaced by a proper foreign key to the `DbNode` row to which the log belongs. This required the deletion of logs that did not correspond to node entities, which were typically legacy workflows. The migration accounted for this by detecting these logs, exporting them to a file for safekeeping, before deleting them. However, the scenario did not consider that for the logs that remained, the corresponding node may have been deleted. Since the relationship was not formalized through a foreign key this would not have cascaded onto the related log entries. The migration would then fail as a result when trying to the set the `dbnode_id` foreign key to the `ojbpk` in the case that the node row no longer existed. In this commit, the migrations are updated to detect orphaned log messages and write them to a file before deleting them, just as is done for logs that belong to legacy workflows or unknown entities. 22 February 2019, 15:16:32 UTC
2482689 Improve efficiency when exporting `Logs` and `Comments` (#2512) Moving `QueryBuilder` that retrieves `Logs` and `Comments` outside the graph traversal loop. This will significantly speed up the export process. 22 February 2019, 14:06:55 UTC
28eea8c Providing a non-ORM way to add nodes to a group. This speeds up a lot the group addition procedure and import method of SQLA should benefit from it (#2471) Fast group addition, skipping the ORM This PR solves issue #1319 and provides a fast way to add nodes to a group that circumvents the ORM. Import/export benefit from this; particularly import under SQLA. In order to achieve this, it uses RAW SQL statements. It was agreed by @giovannipizzi @sphuber and @szoupanos that this is acceptable for aiida 0.X. sqlalchemy >= 1.1 provides better solutions for this in aiida 1.0 (see comments in the code). Furthermore, it adds a new migration `7a6587e16f4c`. For consistency, PR #2514 inserts this migration also in the upstream branch for aiida 1.0 (and handles collisions with other following migrations). 22 February 2019, 12:53:02 UTC
8e03e32 Backport migration `7a6587e16f4c` of release `v0.12.3` (#2514) Migration `7a6587e16f4c` was added to the `v0.12.3` release to improve the speed of adding nodes to groups by adding a uniqueness constraint, with implied index, on the `dbgroup_dbnodes` join table for SqlAlchemy. However, a similar change had already been introduced in migration `59edaf8a8b79` to be released in `v1.0.0` but in a slightly differently way. This would pose problems for people who would migrate their database from `v0.12.3` to any version based on `v1.*`. Therefore, special care needed to be taken to resolve this branching of revisisons. To solve this branching in the migration revisions for the SqlAlchemy backend, the new migration `7a6587e16f4c` is backported and added directly after revision `35d4ee9a1b0e`, which is the point of divergence between `v0.12.*` and `v1.0.0`. Revision `59edaf8a8b79` is then adapted to first check for the presence of the changes introduced by the backported revision `7a6587e16f4c` and when found drop them, before applying the changes it would have done originally. This method guarantees that the migrations will work whether the database contains the changes of the backported migration `7a6587e16f4c` or not. 22 February 2019, 12:18:56 UTC
e2501a6 `verdi quicksetup`: move check for existing profile (#2473) The check whether a profile of the same name already exists was happening too late, after finding the database of the same name. This should be the first thing to check. 22 February 2019, 10:59:59 UTC
bbfc73c Added node label in the tree end point response (#2511) * Added node label in tree end point response * Returned number of passed incomings and outgoings in tree rest end point * updated tree rest endpoint test to check the returned node attributes 22 February 2019, 10:26:14 UTC
3d592eb Add deprecation warning for python 2 to `aiida/__init__.py` (#2509) Currently, it is being marked deprecated for `aiida-core>=2.0.0` even though it is not yet known when that will be released. But it is very likely that `aiida-core==v1.*` will be the last version to support python 2. 22 February 2019, 09:45:21 UTC
d76d07b Restore the import speed of main package to keep `verdi` snappy (#2513) * Make the `DBLogHandler` log configuration lazy upon loading database env The logging needs to be configured upon loading the `aiida` package, but we want to prevent the database environment or the `aiida.orm` module from being loaded as this will slow down the initialization. This is especially detrimental to the performance of `verdi` which everytime it is loaded, i.e. also each time tab-completion is triggered, the whole package needs to be loaded. To prevent this, the implementation of the `DBLogHandler` was already moved from `aiida.common` to `aiida.orm.utils` but since it is still being configured in `aiida.common.log`, the logging module will import the `aiida.orm` module. To fix this, we remove the handler from the default logging configuration and reinsert it dynamically when the database environment is loaded, without which the handler would be non-functional anyway. * Copy implementation `plumpy` language tools instead of importing The `aiida.common.lang` module will be imported when loading `verdi` and importing the `protected` and `override` decorators from the `plumpy` library were causing the whole library to be imported, slowing everything down. By copying the implementation, the time for executing `verdi` is roughly reduced by 25%. 22 February 2019, 07:55:32 UTC
b22f551 Move classes and functions needing OMR out of `aiida.common` (#2501) The module `aiida.common` should be importable without having to load the ORM, so the following modules, which need the ORM have been moved within `aiida.orm`: * Move `aiida.common.serialize` to `aiida.orm.utils.serialize` * Move `DBLogHandler` to `aiida.orm.utils.log` 18 February 2019, 15:15:28 UTC
5395aaa Move all factories into `aiida.plugins.factories` The are exposed and importable from `aiida.plugins`. 18 February 2019, 10:54:53 UTC
feeb8f0 Rename the `aiida.scheduler` and `aiida.transport` modules To keep in line with the naming scheme of the other modules the respective modules have been renamed to their plurals: * `aiida.scheduler` -> `aiida.schedulers` * `aiida.transport` -> `aiida.transports` 18 February 2019, 10:54:53 UTC
685be54 Rename `authinfo` and `computer` backend modules to plurals Also refactored the `computer` and `authinfo` modules for both backend implementations to get rid of the pre-commit exceptions. 18 February 2019, 10:54:53 UTC
7eb5d77 Move all test files to start with `test_` prefix This will make it easier to switch to generic testing frameworks as well as make it easier to distinguish them from implementation files in an editor that without the prefix often share the same name. 18 February 2019, 10:54:53 UTC
6791557 Add data migrations to remove `node.` prefix from node types Now node type strings will either start with `data.` or `process.` 16 February 2019, 18:49:14 UTC
de48161 Add exception in `load_node_class` for backwards compatibility The `load_node_class` serves to turn a node type string into the corresponding `Node` sub class. This is based on how those type strings are constructed. Currently, they will start either with `data.` or `process.` since the `node.` prefix has been dropped. However, some recent database migrations targeted databases where the `node.` prefix was still present and therefore so did the tests. However, when the migration tests are run, they will do so with the current ORM, which no longer considers type strings starting with `node.` to be valid. To make the migration tests work without changing the migrations themselves, which cannot be done, we need to add an exception in the node class loader for type strings starting with `node.`. By just stripping it, it should once again conform to the current node type spec and the logic can be continued. 16 February 2019, 18:49:14 UTC
073f321 Move `aiida.orm.node` to `aiida.orm.nodes` In addition to the move of the node module, the import definitions of the internal modules were cleaned up. This makes it possible to expose all the `Node` sub classes that should be importable by the user, on the top level, i.e. after calling `from aiida import orm` one can access all important node classes directly through the `orm` module. Certain data modules defined top level utility functions to convert the node from its current type into another, often using external libraries such as `ase` and `pymatgen`. These functions typically belong into the `aiida.tools` module. Especially so because these were decorated with the `calcfunction` decorator, requiring the entire `aiida.work` module to be imported when importing the `aiida.orm` module, causing a cyclic dependency. These functions for `cif`, `structure` and `array.trajectory` have been moved to their respective modules within `aiida.tools`. 16 February 2019, 18:49:14 UTC
1ed73d8 Documentation: create placeholder for v1.0 breaking changes (#2483) Created a placeholder section in `install/updating_installation` where the finalised prose describing all of the important behavior changing and backwards-incompatible change should be placed before the v1.0.0b release. The content of this section is being drafted on the wiki at: https://github.com/aiidateam/aiida_core/wiki/Backward-incompatible-changes-in-1.0.0 Before the beta release, the finalised content from the wiki page should be converted using `pandoc` and then pasted in this section. Moved the section about converting workchains to the wiki. This section should be included in the larger v1.0 changes text so that we don't have to maintain two separate versions of it. 16 February 2019, 15:07:10 UTC
dace59e Add annoying prompt for `verdi database migrate` (#2488) - Replace the existing yes/no confirmation with an annoying and scary three-stage prompt. The goal is to prevent casual use of verdi database migrate as it may not be possible to revert a database back to the pre v1.0 state. - Add `NotImplementedError` when using `--force``. We want to keep the option available in the cmdline as a placeholder, but disable its use until the v1.0 are well tested and proven to be revertable. 16 February 2019, 08:32:17 UTC
9489070 Merge pull request #2481 from aiidateam/fix_2233_node_to_new_backend Migrate `Node` to the new backend interface 15 February 2019, 15:48:32 UTC
08e422c Finalizing work of moving `Node` to new backend interface A new abstract interface has been defined for `BackendNode` in the module `aiida.orm.implementation.nodes`. This has subsequently been implemented for the two current database backends Django and SqlAlchemy. The role of the `BackendNode` class is to provide a single interface, independent on the database backend implementation, to store and retrieve data from `Node` instances in the database. The front end `Node` class will then only ever interact with the backend entity that it encapsulates and shall never directly interact with the database model that is encapsulated within the `BackendNode` class. This commit deprecates quite a few methods from the old node class, notably all methods of the variant `*_db_*` that retrieved or set data directly from the database. These are now part of the `BackendNode` interface, but since all its methods will operate directly on the database, the `db` infix has been dropped. All business logic that does not operate directly on database data, such as the caching of links and attributes of unstored nodes, is implemented in the front end ORM class `aiida.orm.node.Node`. Also the biggest part of the business logic is implemented here, to keep the backend layer thin and lean. 15 February 2019, 14:40:04 UTC
07f8fd8 Migrating parts of the old `Node` class to new interface The following parts have been moved: * Methods interacting with attributes * Methods interacting with the label and description * Methods related to links * Methods related to storing a node Additionally, actual business logic is removed from the legacy workflow database models, because they were referencing the node code that is to be migrated, but the model classes are only kept such that a migration to drop the tables is not yet necessary. The API to run legacy workflows had already been dropped in a previous commit and is no longer supported. 15 February 2019, 13:31:57 UTC
0e561aa Migrating parts of the old `Node` class to new interface The following parts have been moved: * Utility functions * Methods interacting with extras * Methods interacting with users * Methods interacting with computer * Methods interacting with comments 15 February 2019, 13:31:57 UTC
2455328 Groundwork for moving Node to new backend system 15 February 2019, 13:31:57 UTC
3e77ad3 Merge pull request #2472 from CasperWA/fix_2342_update_export_import_to_prov_redesign Ex-/import process_type column - Tests for correct export/import of v0.4 export-files 15 February 2019, 09:59:37 UTC
e00c6dc Column `process_type` in table `dbnode` is now exported and imported. Added tests for v0.4 db schema migrations. This is registered as being from revision 8 to 27 (in django migration numbers). The major changes include those made by the provenance redesign. Ordered export_and_import.py file with more classes. Adding new, complementary to existing ones, while rearranging and dividing existing tests. Class `TestLinks` has been updated and reviewed, removing redundant tests. Class `TestProvenanceRedesign` have tests related to correct ex-/import with the new migrations. Update asserts to relevant non-deprecated functions. This means if sets are compared, update to `assertSetEqual`. Otherwise, use `assertEqual` instead of depracated `assertEquals`. Removed test: `test_recursive_export_input_and_create_links_proper` Following tests are essentially redoing text_complex_workflow_graph_links: - `test_links_for_workflows` - `test_double_return_links_for_workflows` But are kept in, due to the specificity check of links. In other words, checks of link count and correct export/import of special cases. Moved general imports to top of file (and out of classes/functions). Import correctly all Data entities (they are now `node.data.*` instead of `data.*`) Added test of Links using high-level workflow and calculation nodes: - `CalcJobNode` - `CalcFunctionNode` - `WorkChainNode` - `WorkFunctionNode` Renamed generic test function names. Added comments to classes and functions where missing. 14 February 2019, 18:20:18 UTC
b1b34f6 Add support for the export/import of `Node` extras (#2416) The extras of `Nodes` can now be exported together with other data. When importing an archive that contains extras for nodes, the behavior depends on whether that node was already present in the database into which the data is imported, namely: * The `Node` did not yet exist: In this case, the extras will be imported wholesale or not at all. The behavior can be toggled from `verdi import` with the flag `--extras-mode-new`. * The `Node` already exists: In this case, the various possibilities are a lot more complex, because the existing and archived node can have overlapping and non-overlapping extras. What to do in case of conflicts and in general can be controlled by the flag `--extras-mode-existing`. The currently implemented options are as follows: - ask: import all extras and prompt what to do for existing extras. - keep_existing: import all extras and keep original value of existing. - update_existing: import all extras and overwrite value of existing. - mirror: remove any pre-existing extras and import all archive extras. - none: do not import any extras. 13 February 2019, 19:14:02 UTC
c6426c0 rest api: iotree limit (#2458) * added tree_in_limit and tree_out_limit filters to tree endpoint of REST API * tree endpoint now returns total no. of incoming and outgoing nodes 11 February 2019, 19:09:26 UTC
6c01cae Disable some SSH transport tests until Travis issue is solved (#2470) Backport of #2464 11 February 2019, 14:34:58 UTC
b85ac3e Remove unserializable data from metadata in `Log` records (#2469) If unserializable data, often found in the `exc_info` or `args` keys, is not removed from the metadata of a log record, a database exception will be thrown. 09 February 2019, 13:13:23 UTC
3f0365d Improve robustness of `verdi` in absence of profiles (#2425) It can occur, due to a problem during the first time setup of an AiiDA instance, that the configuration file is left incomplete, i.e. without any valid profiles or a default profile. This could render the `verdi` command unusable as it assumed the configuration to always be complete. We add a check to the `verdi` main command and add extract checks and information to `verdi profile list` and `verdi profile show` in the case of absent profiles or a default profile. 09 February 2019, 10:59:50 UTC
5c65039 Add the `verdi data upf content` command (#2468) This will print the bare contents of the file behind `CifData` nodes 09 February 2019, 08:15:35 UTC
69dbdb2 Implement support for process functions in `verdi process report` (#2466) The nodes used by process functions, i.e. `CalcFunctionNode` and `WorkFunctionNode` also can have log messages attached that should be displayable through `verdi process report`. This is implemented in a helper function `get_process_function_report`. 09 February 2019, 07:35:37 UTC
7420655 Fix description of 'contains' filter in QB docs (#2429) The 'contains' filter operator in the QueryBuilder takes a list of values, not an individual value. 08 February 2019, 17:27:27 UTC
0c9e0b8 Add support for the export/import of `Comment` entities (#2413) To make the export of `Comment` entities possible, the `QueryBuilder` had to be extended to support retrieving comments for a given `Node`. The following changes were applied: * Added support to join user to comment (`with_user`) * Added support to join comment to user (`with_comment`) The `verdi export create` has a new flag to include or exclude the export of comments for nodes that are to be exported, defaulting to include: `--include-comments/--exclude-comments` Documentation has been updated to include new `QueryBuilder` join args in table and the `metadata.json` example has been updated in documentation to include correct Comment info. 08 February 2019, 16:09:08 UTC
a000899 Add data migrations for `TrajectoryData` nodes In the previous commit, the ORM for `TrajectoryData` was changed to start storing the `symbols` as a list in the attributes of the node instead of a `numpy` array in the repository, to make it possible to query for the symbols. This requires a data migration which is implemented in this commit. Since it involves the deletion of the data from the repository after it has been copied to the database, the migration is performed in two stages. The first will copy the data from the repository to the attributes on the node and the second will delete the data in the repository. In this way, if an exception occurs during the migration, no data will be lost. 08 February 2019, 15:01:08 UTC
072f5dc Changes to `Data` class attributes and `TrajectoryData` data storage Methods changed to properties: * `Code.is_hidden` * `RemoteData.is_empty` Name changes: * `ArrayData.iterarrays` to `ArrayData.get_iterarrays` * `Code.full_text_info` to `Code.get_full_text_info` * `Code.is_hidden` to `Code.hidden` * `StructureData._get_cif` to `StructureData.get_cif` * `TrajectoryData._get_aiida_structure` to * `TrajectoryData.get_structure` * `TrajectoryData._get_cif` to `TrajectoryData.get_cif` Data structure storage changes: * Symbols in `TrajectoryData` are now passed as a list instead of * `numpy.ndarray` * Symbols in `TrajectoryData` are now stored as an attribute instead of * an array in the repository * `TrajectoryData.set_trajectory` arguments `stepid` and `cells` are now optional Change `TrajectoryData` to make passing stepids and cells optional. While nothing will be stored for cells if not given a consecutive sequence will be automatically assigned to stepids if missing from the inputs. 08 February 2019, 15:01:08 UTC
c6580e6 Ensure nested `calcfunctions` work as long `store_provenance=False` (#2464) In principle `calcfunctions` are not allowed to call other `calcfunctions` because a calculation is not allowed to call other calculations. However, `calcfunctions` can be called as normal functions by setting the metadata variable `store_provenance` to `False`. Therefore, as long as `store_provenance` is turned off for nested functions, a `calcfunction` is allowed to call other `calcfunctions`. To make this work, the `Process._setup_db_record` method had to be adapted to only add a call link to the parent process, if and only if `store_provenance` for the child node is enabled. Otherwise, the link validation would fail since a `CALL` link outgoing of a `CalculationNode` is forbidden. 08 February 2019, 12:48:37 UTC
bdb7983 Fix bug in `delete_nodes` when passing pks of non-existing nodes (#2440) The `aiida.manage.database.delete.nodes.delete_nodes` function was not checking whether the pks that were passed correspond to existing nodes which would cause the line that fetches the repository folders, after the graph traversal queries had been completed, to except. To fix this we first verify that the passed pks exist and if not print a warning and discard them. 08 February 2019, 11:35:59 UTC
4378d52 Catch `UnroutableError` for orphaned processes in `verdi process` calls (#2445) If the RabbitMQ task of a running process is lost before it is completed, it will never complete, but any `verdi process` commands that try to reach it such as `pause`, `play` and `kill` will throw an `UnroutableError`. This exception should be caught and an error message should be printed instead. 08 February 2019, 10:47:55 UTC
d4996f1 Fix incorrect call in `StructuresCollection.find` of `MpdsImporter` (#2442) The `find` method requires a `query` as a first argument which was not being passed. 08 February 2019, 10:06:10 UTC
c121070 Fix import error of `md5_file` in `CifData` and `UpfData` (#2438) The recent move of `md5_file` from `aiida.common.utils` to `aiida.common.files` was not caught in some of the methods of these classes because it imported the whole module `aiida.common.utils` and used the function as `aiida.common.utils.md5_file`. 08 February 2019, 09:25:00 UTC
8f5d0b6 Add exception for `calculation.job.JobCalculation` process type inference (#2428) In the migration following the provenance redesign, the process type had to be inferred from the existing type string for `JobCalculation` nodes. The function `infer_calculation_entry_point` did not account for literal `calculation.job.JobCalculation.` type strings being present and would return `.JobCalculation` as the fallback process type. However, it makes more sense to just have an empty process type string in this case, as would be the case for a base `CalcJobNode` instance. 08 February 2019, 08:31:24 UTC
ba4b3e7 Generalize the UUID deduplication for more database tables The original deduplication code was implemented only for the `DbNode` table but has now been extended to `DbComputer` and `DbGroup`. Note that this has not been done for `DbComment`, because it does not yet have an entity loader. If `load_comment` were to be implemented, support for deduplication can be added to the `deduplicate_uuids` function. 08 February 2019, 07:38:03 UTC
dc7e125 Add uniqueness constraints on all UUID fields in SqlAlchemy models This adds the uniqueness constraint on the UUID fields of the models: * Computer * Comment * Group * Node * Workflow A check is performed before the migration is applied, to see whether the affected tables actually contain any entries with duplicate UUIDs. In this case a warning is emitted and the migration is stopped. The infrastructure to deduplicate, that is present all ready for nodes, has not yet been generalized to arbitrary tables. 08 February 2019, 07:38:03 UTC
dcdfbf2 Move `aiida.orm.data` within the `aiida.orm.node` module (#2402) This final move makes the hierarchy of all node modules and its sub classes consistent. The `Data` and `ProcessNode` classes who are both sub classes of the `Node` base class and are direct siblings live on the same level in the hierarchy, direct within the `aiida.orm.node` module. Note that because the type string of all `Data` nodes now will start with `node.` instead of `data.`, a data migration had to be added. 08 February 2019, 06:26:55 UTC
631ff01 Pin the version of dependency `pika==1.0.0b1` (#2463) The second beta was just released which breaks our implementation. This will have to be adapted later. 08 February 2019, 05:48:41 UTC
93bdb16 Add support for the export/import of `Log` entities To make the export of `Log` entities possible, the `QueryBuilder` had to be extended to support retrieving log records for a given `Node`. The following changes were applied: * Internal `log_model_class` has been removed and replaced by `orm.Log` * Added support to join node to log (`with_node`) * Added support to join log to node (`with_log`) The `verdi export create` has a new flag to include or exclude the export of logs for nodes that are to be exported, defaulting to include: `--include-logs/--exclude-logs` Documentation has been updated to include new `QueryBuilder` join args in table and the `metadata.json` example has been updated in documentation to include correct Log info. 07 February 2019, 21:23:06 UTC
78248e1 Fix the wrong migration for the `DbLog` table In commit `c0fba2e38557fe1ea535d1dc24076cba99212616`, a migration was introduced for the `DbLog` table in order to support the export and import of log entries. However, this migration was flawed and would lose valuable information. The old `DbLog` was designed to support adding log messages to various entities. Because these entities were not restricted to just nodes, and therefore could live in different database tables, a foreign key could not be used for the relationship, but instead an `objpk` and `objname` column were used to allow multiple table references. However, for the import to know whether an exported log record is already present in the database and therefore should not be imported again, the object pk could not be used as these do remain the same between databases. To this end the migration added a UUID column to the `DbLog` table and dropped the `objpk` column. However, the data was not migrated before doing so and so the connection between logs and their entities was lost. After the faulty commit, we realized that the reason for not using the foreign key was that logs needed to be able to be associated with both nodes as well as legacy workflows, but since support for the latter had already been dropped this was no longer necessary. Given that now only nodes can have associated logs, we can revert the `objpk` to be a proper foreign key and the `objname` can be dropped. Since the logs for legacy workflows (and any other unexpected entities) will be dropped from the database, they should be exported to a file for archival purposed. In summary, the migrations perform the following logical actions: * Export existing logs for legacy worklows and other entities to file * Delete records from legacy workflows and other entities from `dblog` * Create a foreign key `dbnode_id` * Migrate data from `objpk` to `dbnode_id` * Create a new column `uuid` for `DbLog` that is unique * Delete the `objpk` and `ojbname` columns from `DbLog` * Delete `objpk` and `objname` from the `metadata` of `DbLog` records Note that the addition of the unique `uuid` column had to be done in three steps. The problem is that the default value for the UUID is generated by a python function which cannot therefore be done on a database level but the values have to be set after the creation of the column. However, as soon as the column is created the uniqueness constraint is violated since it will be empty for all the existing records. For that reason the migration is split in three steps. In the first, the column is created and set as nullable. The second step will populate the records using the python function to generate the UUIDs and finally, the third migration will alter the column to be unique. Note that for Django this migration could be written in a single file, but for SqlAlchemy it had to be broken into several migrations. This is because of Alembic, the framework used to perform SqlAlchemy migrations, which had trouble with setting the UUIDs after creation of the column in a single transaction. To circumvent this problem, the migration was split into three separate migrations and Alembic is now instructed to use a single transaction per revision, mirroring the migration behavior of Django. 07 February 2019, 21:23:06 UTC
309b2f2 Disable some SSH transport tests until Travis issue is solved (#2461) Problems described in issue #2460 07 February 2019, 14:29:43 UTC
dd25fd1 Fix upper limit for dependency to `pg8000<1.13.0` (#2456) The library `pg8000` which is used by `pgtest` to interface with Postgres and is used by us for testing purposed released a new minor version `v1.13.0` on Feb 1 2019, in which it dropped `py2` support causing our tests to fail miserably. For the time being, we put an upper bound on that dependency. 06 February 2019, 09:38:31 UTC
9199702 Fix upper limit for dependency to `pg8000<1.13.0` (#2450) The library `pg8000` which is used by `pgtest` to interface with Postgres and is used by us for testing purposed released a new minor version `v1.13.0` on Feb 1 2019, in which it dropped `py2` support causing our tests to fail miserably. For the time being, we put an upper bound on that dependency. 04 February 2019, 09:35:08 UTC
e9d4e03 decode dict keys only if strings (#2436) backport fix #2017 01 February 2019, 09:53:58 UTC
dde378e Backport postgres improvements to deal with setups without sudo (#2433) Also updated `yapf` to recent version 01 February 2019, 08:02:56 UTC
back to top