| 93757dc | Thomas B. Mooney | 23 April 2021, 18:22:15 UTC | Merge pull request #1999 from tmooney/usage_log_dsn Update connection string for usage log. | 23 April 2021, 18:22:15 UTC |
| 6d70e33 | Thomas B. Mooney | 22 April 2021, 22:30:57 UTC | Update connection string for usage log. It makes me sad this string is hardcoded here, but I guess updating it once every several years isn't so bad :) | 22 April 2021, 22:30:57 UTC |
| 44d575e | Thomas B. Mooney | 14 April 2021, 18:26:30 UTC | Merge pull request #1998 from tmooney/monitor_jobs_when_call_caching When call-caching is enabled, check the health of jobs. | 14 April 2021, 18:26:30 UTC |
| f4035f5 | Thomas B. Mooney | 14 April 2021, 15:16:23 UTC | When call-caching is enabled, check the health of jobs. Before we were letting jobs fail and manually running `script.submit` for replacement jobs in the event of LSF failure due to the high cost of starting a workflow from the beginning over such a trivial failure. With call-caching that cost is reduced, so it's better to have quicker notification of issues. | 14 April 2021, 15:16:23 UTC |
| f19d9c1 | Thomas B. Mooney | 12 April 2021, 20:00:27 UTC | Merge pull request #1997 from tmooney/use_filetest_access Enable ACL support for filetests. | 12 April 2021, 20:00:27 UTC |
| f2e6974 | Thomas B. Mooney | 24 March 2021, 18:42:19 UTC | Enable ACL support for filetests. Less performant, sure, but actually works with a lovely non-posix-compliant filesystem, too! See also: https://perldoc.perl.org/filetest | 24 March 2021, 18:42:19 UTC |
| 96de3cf | Thomas B. Mooney | 22 March 2021, 20:59:53 UTC | Merge pull request #1996 from tmooney/du_apparent_size Use the apparent size for allocations. | 22 March 2021, 20:59:53 UTC |
| 29930af | Thomas B. Mooney | 22 March 2021, 20:59:45 UTC | Merge pull request #1995 from tmooney/requeue_for_build_retry Add ability to queue rebuilds for CwlPipeline builds. | 22 March 2021, 20:59:45 UTC |
| 288c115 | Thomas B. Mooney | 17 March 2021, 19:57:21 UTC | Use the apparent size for allocations. With the fancy fancy cache layer we're using these days, the actual size of files that haven't been pulled into the cache is not detemrined correctly by `du`. But the apparent size is good enough! | 17 March 2021, 19:57:21 UTC |
| 7487982 | Thomas B. Mooney | 15 March 2021, 20:43:03 UTC | Add ability to queue rebuilds for CwlPipeline builds. This functions like `genome model build queue` but for restarting builds rather than creating new ones. Since we're moving away from supporting the queue for arbitrary users, this is set up to only allowing the queue of failed "production" builds. | 15 March 2021, 20:43:03 UTC |
| fc62358 | Thomas B. Mooney | 09 February 2021, 16:03:09 UTC | Merge pull request #1992 from tmooney/merge_with_liftover Need to liftOver each track individually. | 09 February 2021, 16:03:09 UTC |
| 9186b05 | Thomas B. Mooney | 09 February 2021, 16:02:00 UTC | Merge pull request #1994 from tmooney/volume_and_group_create Create commands for disk groups and volumes | 09 February 2021, 16:02:00 UTC |
| 617f61d | Thomas B. Mooney | 26 January 2021, 18:28:18 UTC | Add help/doc sections to new volume/group create commands. | 26 January 2021, 18:28:18 UTC |
| c87f212 | Thomas B. Mooney | 26 January 2021, 16:24:46 UTC | New command to help create volumes. | 26 January 2021, 16:24:46 UTC |
| 1f88462 | Thomas B. Mooney | 25 January 2021, 23:20:33 UTC | New command to facilitate creating disk groups. | 25 January 2021, 23:20:33 UTC |
| cb71a24 | Thomas B. Mooney | 01 December 2020, 19:59:37 UTC | Merge pull request #1990 from tmooney/possible_volumes Add true value to end of module. | 01 December 2020, 19:59:37 UTC |
| deb9ee3 | Thomas B. Mooney | 12 November 2020, 16:44:01 UTC | Merge pull request #1991 from tmooney/bqm_remote Make the "prod-builder" run_as configurable. | 12 November 2020, 16:44:01 UTC |
| 6e282f4 | Thomas B. Mooney | 12 November 2020, 16:43:39 UTC | Merge pull request #1993 from tmooney/move_cwlpipelines Use more UR-centric approach to test for owner class existence. | 12 November 2020, 16:43:39 UTC |
| 751dbc9 | Thomas B. Mooney | 09 November 2020, 19:20:02 UTC | Use more UR-centric approach to test for owner class existence. UR classes don't necessarily have defined perl module files, and we don't support auto-updating any non-UR classes, so this should be good enough! | 09 November 2020, 19:20:02 UTC |
| 2d3cf39 | Thomas B. Mooney | 27 October 2020, 20:23:23 UTC | Need to liftOver each track individually. This assumes that the two existing track names are the ones that exist for multi-tracked files. "Weird" tracks will not work right. | 27 October 2020, 20:23:23 UTC |
| ee980b3 | Thomas B. Mooney | 19 October 2020, 17:30:48 UTC | Merge pull request #1988 from tmooney/auto_remote_sync_admin Auto-connect production imports to remote volumes. | 19 October 2020, 17:30:48 UTC |
| 0d73a55 | Thomas B. Mooney | 15 October 2020, 19:02:13 UTC | Make the "prod-builder" run_as configurable. Now that we're operating on multiple clusters we've run into a situation where it'd be nice to distinguish between then with different "run_as" indications. This will let us change it when run on the new cluster while leaving it the same on the old. (Or someone enterprising could set the appropriate environment variable to setup for the "other" cluster for the one they're on!) | 15 October 2020, 19:02:13 UTC |
| 1584b35 | Thomas B. Mooney | 14 October 2020, 19:16:50 UTC | Add true value to end of module. | 14 October 2020, 19:16:50 UTC |
| f9d5142 | Thomas B. Mooney | 14 October 2020, 17:28:09 UTC | Merge pull request #1989 from tmooney/possible_volumes New command to help identify which volumes to mount. | 14 October 2020, 17:28:09 UTC |
| 072763f | Thomas B. Mooney | 14 October 2020, 14:48:54 UTC | Merge pull request #1983 from tmooney/prod-builder_for_wo_projects Stop hardcoding "apipe-builder" when determining applicable models | 14 October 2020, 14:48:54 UTC |
| 2a0beb1 | Thomas B. Mooney | 13 October 2020, 21:41:39 UTC | New command to help identify which volumes to mount. This command lists the possible volumes configured for an AnP. This could be used for a safe list of volumes to mount when launching a container to work with the AnP. | 13 October 2020, 21:41:39 UTC |
| 91121e8 | Thomas B. Mooney | 12 October 2020, 15:43:04 UTC | Keep dbh connection open on short-lived shellcmd runs. The one in the cleanup guard is especially important, since on a successful run it'll be running as the system shuts down. In that circumstance calling the disconnect process may fail. | 12 October 2020, 15:43:04 UTC |
| fed4496 | Thomas B. Mooney | 08 October 2020, 19:37:11 UTC | Auto-connect production imports to remote volumes. | 12 October 2020, 15:30:35 UTC |
| c6f04ff | Thomas B. Mooney | 30 September 2020, 15:41:21 UTC | Merge pull request #1984 from tmooney/import_data_from_lims_to_remote Support importing LIMS data into the system directly onto a remote volume. | 30 September 2020, 15:41:21 UTC |
| 1c318d9 | Thomas B. Mooney | 17 September 2020, 20:23:27 UTC | Support importing LIMS data into the GMS on a remote volume. | 24 September 2020, 20:31:29 UTC |
| 33b5387 | Thomas B. Mooney | 16 September 2020, 22:20:23 UTC | Make test more generic about users, too. | 24 September 2020, 14:27:14 UTC |
| 24f0ddf | Thomas B. Mooney | 22 September 2020, 21:29:00 UTC | Merge pull request #1986 from tmooney/ssl_for_dgidb Use HTTPS to connect to dgidb. | 22 September 2020, 21:29:00 UTC |
| a88cbb1 | Thomas B. Mooney | 22 September 2020, 21:28:51 UTC | Merge pull request #1985 from tmooney/splitpath_typo Fix typo: s/split_path/splitpath/. | 22 September 2020, 21:28:51 UTC |
| cfd78bf | Thomas B. Mooney | 22 September 2020, 17:16:11 UTC | Add testcase for updating instrument data bam path on allocation move. | 22 September 2020, 17:16:11 UTC |
| 54b3b2f | Thomas B. Mooney | 21 September 2020, 20:07:58 UTC | Use HTTPS to connect to dgidb. The new release of dgidb requires SSL. The HTTP version redirects. This code is pretty old and is likely a good candidate for outright removal, but these changes make the tests pass again (albeit mostly in a lame way). | 21 September 2020, 20:07:58 UTC |
| 2d85d58 | Thomas B. Mooney | 17 September 2020, 22:09:10 UTC | Fix typo: s/split_path/splitpath/. | 17 September 2020, 22:09:10 UTC |
| 41bb3fd | Thomas B. Mooney | 16 September 2020, 22:05:46 UTC | Stop hardcoding "apipe-builder" when determining applicable models We've locally been using a different production user for many years, so it'd be nice if this worked again! | 16 September 2020, 22:05:46 UTC |
| 8216cbc | Thomas B. Mooney | 15 September 2020, 21:11:07 UTC | Merge pull request #1981 from tmooney/allocation_transfer New command `genome disk allocation transfer`. | 15 September 2020, 21:11:07 UTC |
| b00190a | Thomas B. Mooney | 15 September 2020, 21:10:51 UTC | Merge pull request #1978 from tmooney/file_database_call_caching Cromwell Call Caching support | 15 September 2020, 21:10:51 UTC |
| 391184c | Thomas B. Mooney | 15 September 2020, 21:10:37 UTC | Merge pull request #1974 from tmooney/remove_extra_subcall Remove extra layer of method calls. | 15 September 2020, 21:10:37 UTC |
| 819df31 | Thomas B. Mooney | 31 August 2020, 17:24:05 UTC | Switch to xxh64 instead of fingerprint for call-caching. The "fingerprint" method only works with hard-links, but sometimes we have to resort to copying from the original storage to our working volume, which busts the cache since the original file metadata isn't maintained for the copy. | 31 August 2020, 17:24:05 UTC |
| 24bf893 | Thomas B. Mooney | 10 June 2020, 21:46:58 UTC | Mitigate having multiple workflows per build. The "right" answer here would be to link them to the corresponding Genome::Process. The issue right now is that the process is created *after* the workflow inputs are assigned, so we have no way to accurately determine the correct process for our current run. There are some strategies for trying to figure it out. For example: - look up our current LSF Job ID and check the database for a matching process. - look for the one process that is currently marked as "running" and die if there's a second one. For now this assumes only one process ever "succeeds" for a build and the build view displays all the workflows in an arbitrary order. | 11 August 2020, 15:39:39 UTC |
| 42b2262 | Thomas B. Mooney | 27 July 2020, 19:41:07 UTC | Merge pull request #1982 from tmooney/no_lsf_tmp_modifications Remove modifications of pre-provided LSF temp directory. | 27 July 2020, 19:41:07 UTC |
| 365caf9 | Thomas B. Mooney | 24 July 2020, 21:49:23 UTC | Remove modifications of pre-provided LSF temp directory. On the new cluster this directory is locked down so we lack permissions to change it. | 24 July 2020, 21:49:23 UTC |
| 6cae9c6 | Thomas B. Mooney | 23 July 2020, 19:43:45 UTC | Test execution of the command. We can make sure we updated the database entries, even if testing the rsync portion is impractical! | 23 July 2020, 19:43:45 UTC |
| 97cff7d | Thomas B. Mooney | 15 July 2020, 21:28:28 UTC | Make sure directory exists before attempting `rsync`. | 15 July 2020, 21:28:28 UTC |
| 4a5d6db | Thomas B. Mooney | 29 June 2020, 20:26:14 UTC | New `genome disk allocation transfer` for sending to a remote volume. | 15 July 2020, 21:18:12 UTC |
| 5705880 | Thomas B. Mooney | 29 June 2020, 20:24:20 UTC | Add instrument data to update_owner_for_move. I'm not a big fan of the way this one spot has to be added to for any potential allocation owner to get updated, but it works and it's fairly straightforward! So let's stick with it for now. | 29 June 2020, 20:24:20 UTC |
| b363008 | Thomas B. Mooney | 22 June 2020, 19:28:30 UTC | Merge pull request #1979 from tmooney/imported_data_more_tolerant_of_bam_paths Allow imported BAMs with arbitrary filenames | 22 June 2020, 19:28:30 UTC |
| 80f6afe | Thomas B. Mooney | 22 June 2020, 19:27:49 UTC | Merge pull request #1980 from tmooney/listdockerimages_t_truevalue `test-tracker` in `--track` mode requires tests to return a true value in order to pass. | 22 June 2020, 19:27:49 UTC |
| d0023c8 | Thomas B. Mooney | 18 June 2020, 18:14:10 UTC | `test-tracker` requires tests to return a true value in order to pass. But "skip" doesn't do that--it returns undef, so we'll add our own true value! | 18 June 2020, 18:14:10 UTC |
| aca76c9 | Thomas B. Mooney | 17 June 2020, 17:03:03 UTC | Try to find any single BAM in the directory if expected path not found. The "trusted data" importer makes it easier to import a BAM with any arbitrary filename. It should be fine to accept this as long as it's the only one. | 17 June 2020, 17:03:03 UTC |
| 1c5281f | Thomas B. Mooney | 17 June 2020, 17:01:10 UTC | Fix bug in no-allocation error message. See that comma there in the old version? That would eat the second half of the message! | 17 June 2020, 17:02:29 UTC |
| c4273f7 | Thomas B. Mooney | 16 June 2020, 19:30:44 UTC | Merge pull request #1976 from tmooney/describe_cwl_pipeline_pps New command to list docker images for a CWL pipeline build. | 16 June 2020, 19:30:44 UTC |
| ff6a3f7 | Thomas B. Mooney | 10 June 2020, 19:47:45 UTC | Don't regenerate config and labels on build restart. | 10 June 2020, 21:57:15 UTC |
| fddaad0 | Thomas B. Mooney | 10 June 2020, 19:39:48 UTC | Configuration option to enable cromwell call caching. | 10 June 2020, 19:39:48 UTC |
| 1d679cc | Thomas B. Mooney | 28 May 2020, 20:43:00 UTC | Merge pull request #1977 from tmooney/no_df_of_remote_volumes For now, don't try to `df` remote volumes. | 28 May 2020, 20:43:00 UTC |
| 3631187 | Thomas B. Mooney | 28 May 2020, 17:59:25 UTC | Merge pull request #1975 from tmooney/skip_copycat_linelength_tests Skip non-deterministic tests. | 28 May 2020, 17:59:25 UTC |
| 77daed3 | Thomas B. Mooney | 28 May 2020, 15:41:54 UTC | Also use physical path for remote volumes in volume lister. | 28 May 2020, 15:41:54 UTC |
| 1f5eaba | Thomas B. Mooney | 28 May 2020, 14:32:38 UTC | For now, don't try to `df` remote volumes. For volumes that are mounted via a cache layer, df doesn't work properly (among other non-compliant aspects of the filesystem). In our case we could possibly use SMB to connect to the remote server and find out the canonical used and total KB, but for now let's trust our database to be accurate. | 28 May 2020, 14:32:38 UTC |
| 45e459a | Thomas B. Mooney | 21 May 2020, 20:47:56 UTC | New command to list docker images for a CWL pipeline build. Someone was asking me what versions of things ran in a given model, and it seemed like there should be an easier way to answer this question! | 21 May 2020, 20:47:56 UTC |
| 1ceffce | Thomas B. Mooney | 21 May 2020, 17:12:06 UTC | Remove old debug statement. | 21 May 2020, 17:12:06 UTC |
| 6ec2901 | Thomas B. Mooney | 21 May 2020, 16:47:51 UTC | Merge pull request #1973 from tmooney/uuid_generator_for_disk_tables Use UUID generator for creating disk groups. | 21 May 2020, 16:47:51 UTC |
| 4ba8b8e | Thomas B. Mooney | 21 May 2020, 16:45:39 UTC | Skip non-deterministic tests. Sometimes they pass. Sometimes they fail. But the code hasn't changed in a long time, so it's clear these tests are not particularly useful. | 21 May 2020, 16:45:39 UTC |
| 8123730 | Thomas B. Mooney | 21 May 2020, 15:21:31 UTC | Merge pull request #1928 from tmooney/only_count_active_allocations_in_cqid_verification Only consider active allocations when deciding whether to import. | 21 May 2020, 15:21:31 UTC |
| 6f88458 | Thomas B. Mooney | 21 May 2020, 15:20:51 UTC | Merge pull request #1972 from tmooney/include_docker_volumes_env_when_setting_new Make `docker_volumes` config additive to the existing environment. | 21 May 2020, 15:20:51 UTC |
| ed40892 | Thomas B. Mooney | 21 May 2020, 15:16:47 UTC | Remove extra layer of method calls. In 461d694dae993e839e8fee2dd77c7eb6a3b2533a we removed instrumentation, which left the outer call here as a mostly pointless wrapper. So now let's just consolidate them. | 21 May 2020, 15:16:47 UTC |
| be544fa | Thomas B. Mooney | 20 May 2020, 21:12:58 UTC | Use UUID generator for creating disk groups. We've heretofore been importing them from an external canonical source that determines their IDs, but now we need to be able to make them directly. | 20 May 2020, 21:12:58 UTC |
| 4e2e675 | Thomas B. Mooney | 06 April 2020, 21:21:20 UTC | Make `docker_volumes` config additive to the existing environment. The environment will have been set up with necessary mounts to get things started (e.g., via `gsub`). We'll need to keep those along in addition to whatever project-specific mounts someone chooses to add. | 06 April 2020, 21:28:45 UTC |
| 632d242 | Thomas B. Mooney | 11 March 2020, 14:02:57 UTC | Merge pull request #1971 from tmooney/add_missing_config_spec_lsf_user_group Add missing lsf_user_group config spec file. | 11 March 2020, 14:02:57 UTC |
| 0c2c931 | Thomas B. Mooney | 10 March 2020, 19:39:11 UTC | Add missing lsf_user_group config spec file. | 10 March 2020, 19:39:11 UTC |
| 1096ca1 | Thomas B. Mooney | 10 March 2020, 18:03:19 UTC | Merge pull request #1970 from tmooney/fix_workflowbuilder_block Pass the inputs along to the outputs in Block operation. | 10 March 2020, 18:03:19 UTC |
| 7fbcecd | Thomas B. Mooney | 09 March 2020, 21:00:24 UTC | Pass the inputs along to the outputs in Block operation. | 09 March 2020, 21:00:24 UTC |
| f6262bd | Thomas B. Mooney | 05 March 2020, 21:11:52 UTC | Merge pull request #1968 from tmooney/site_ris Have ris.wustl.edu domains use the "TGI" site, too. | 05 March 2020, 21:11:52 UTC |
| 3dd2f28 | Thomas B. Mooney | 05 March 2020, 21:11:37 UTC | Merge pull request #1967 from tmooney/dont_assume_mount_path_structure Don't assume mount_path structure. | 05 March 2020, 21:11:37 UTC |
| 98c37c4 | Thomas B. Mooney | 05 March 2020, 21:11:22 UTC | Merge pull request #1966 from tmooney/lsf_compute_group Support setting an LSF user_group. | 05 March 2020, 21:11:22 UTC |
| af8841e | Thomas B. Mooney | 04 March 2020, 23:19:59 UTC | Have ris.wustl.edu domains use the "TGI" site, too. Maybe at some point we'll split these up since they represent two different clusters. It might even be a good opportunity to pull things from TGI that really belong in the main classes! But for now, the two clusters are similar enough... | 04 March 2020, 23:19:59 UTC |
| 71e0a0d | Thomas B. Mooney | 27 February 2020, 20:23:53 UTC | Use standard "get_allocation_for_path" instead of rolling our own. | 27 February 2020, 20:23:53 UTC |
| b1e09e4 | Thomas B. Mooney | 27 February 2020, 19:58:07 UTC | Look up the mount path instead of presuming how many parts to remove. | 27 February 2020, 19:58:07 UTC |
| 3478932 | Thomas B. Mooney | 27 February 2020, 18:52:50 UTC | Merge pull request #1965 from tmooney/hsqldb_cromwell_db Support using file-based database for Cromwell workflows | 27 February 2020, 18:52:50 UTC |
| a8bfd94 | Thomas B. Mooney | 27 February 2020, 18:52:17 UTC | Merge pull request #1964 from tmooney/disk_volume_uuids Use UUIDs for new disk volumes. | 27 February 2020, 18:52:17 UTC |
| 0cdfeb7 | Thomas B. Mooney | 27 February 2020, 18:48:25 UTC | Use new lsf_user_group config for workflow bsubs. | 27 February 2020, 18:48:25 UTC |
| 537c44c | Thomas B. Mooney | 27 February 2020, 15:40:57 UTC | Support passing the "user group" to bsub. | 27 February 2020, 15:40:57 UTC |
| ccdc5c1 | Thomas B. Mooney | 26 February 2020, 20:02:09 UTC | Merge pull request #1963 from tmooney/use_same_interpreter_instead_of_hardcoded_genome_perl Use the current interpreter rather than assuming genome-perl. | 26 February 2020, 20:02:09 UTC |
| 3d04fa2 | Thomas B. Mooney | 26 February 2020, 15:11:24 UTC | Better messages about skipping WF; only show when requested. | 26 February 2020, 15:11:24 UTC |
| c4efdb1 | Thomas B. Mooney | 25 February 2020, 23:03:11 UTC | Don't try to load a local database while the build is running. The running Cromwell process will have a lock on the database, so we can't look. (This view remains primarily useful for looking at failed builds.) | 25 February 2020, 23:03:11 UTC |
| 6cb38de | Thomas B. Mooney | 25 February 2020, 22:08:29 UTC | Shell out to write the note so it's saved immediately. Hopefully we don't encounter that sudo message, since we can't fix it from here... | 25 February 2020, 22:08:29 UTC |
| 60d8328 | Thomas B. Mooney | 25 February 2020, 21:43:09 UTC | Save timing report when workflow completes. | 25 February 2020, 21:43:09 UTC |
| 914a412 | Thomas B. Mooney | 25 February 2020, 21:18:22 UTC | API endpoint to generate a timing report. | 25 February 2020, 21:18:22 UTC |
| d3cb71b | Thomas B. Mooney | 18 February 2020, 22:51:21 UTC | Retry server requests up to five times. | 25 February 2020, 20:29:15 UTC |
| 8b4da7f | Thomas B. Mooney | 18 February 2020, 22:50:52 UTC | Now that we're setting up local servers, can't assume constant here. | 25 February 2020, 20:29:14 UTC |
| 228db08 | Thomas B. Mooney | 18 February 2020, 20:55:54 UTC | Use local server to get build information when necessary. Maybe this should always do this? | 25 February 2020, 20:29:14 UTC |
| 31772ac | Thomas B. Mooney | 18 February 2020, 20:55:14 UTC | Use a local server to get the outputs information for Cromwell. | 25 February 2020, 20:29:13 UTC |
| f105802 | Thomas B. Mooney | 18 February 2020, 20:53:32 UTC | Use new cmdline generator. | 25 February 2020, 20:29:13 UTC |
| e4182d3 | Thomas B. Mooney | 18 February 2020, 20:52:40 UTC | Standardize getting Cromwell path; new method to spawn a local server. | 25 February 2020, 20:29:13 UTC |
| 8b66ff7 | Thomas B. Mooney | 18 February 2020, 18:03:19 UTC | Support specifying hsqldb server for Cromwell. | 25 February 2020, 20:29:12 UTC |
| b143ece | Thomas B. Mooney | 24 February 2020, 18:19:42 UTC | Use UUIDs for new disk volumes. | 24 February 2020, 18:19:42 UTC |
| 594a51d | Thomas B. Mooney | 20 February 2020, 22:11:27 UTC | Use the current interpreter rather than assuming genome-perl. This usually doesn't matter at all, but if someone is trying to test different perl versions they'll be glad it's here :) | 20 February 2020, 22:11:27 UTC |
| 4915e2f | Thomas B. Mooney | 18 February 2020, 19:24:59 UTC | Merge pull request #1959 from tmooney/no_more_ptero Remove PTero support. | 18 February 2020, 19:24:59 UTC |
| e8d3e88 | Thomas B. Mooney | 18 February 2020, 19:24:16 UTC | Merge pull request #1961 from genome/export Clone of PR#1960 | 18 February 2020, 19:24:16 UTC |
