| refs/tags/r1982 | r1982 | a168653 | grid-control '82 edition MERGE: Merging r1941 through r1981 into STABLE General: * Code review changes * PEP-8 transition * Updated NOTICE and documentation Backend: * Bulk submission and delegate usage for cream ce backend (#38) * GridEngine: don't crash on return of empty status string * GridEngine: Fixed job number parsing * Started moving to the new task interface * Moved SE dataset transfer stuff around CMSSW: * Automate CMS proxy init * Dumping hash to gc.stdout in case of failure * Removed dependency of CMSSW_Advanced on parameter factory instance Config: * Added API to get shared GCEvent instances from config interface * Added get_fn / get_dn config API * Added option to prevent loading of previous config file * Fixing issue with += repeating on subsequent go.py calls * New domain config for EKP due to network migration * Proper join_opt_locations implementation * Relaxed whitespace interpretation in config options Dataset: * Added option to change variable names of partition processor * Added the parameter datasource_name to the constructor of all dataset plugins * Added threaded MultiDatasetProvider * Don't load old config when creating new config instance for ListProvider * Dropping support for ParaMod in ScanProvider * Extensions to use 'srm' modifier in LFNPartitionProcessor * Improved binding of datasource * Improved resync behaviour - avoiding unnecessary resyncs * Moved default dataset provider config query from bind into DataTask * Reduce wait time when grid-control is aborted during dataset queries * Refactored DataSplitterIO - increased code sharing between V1 and V2 * Refactored DataSplitter - resync is now done by separate class GUI: * Added drawing objects to cherrypy GUI * Added Fallback for getmaxyx * GUI subsystem rewrite * Improved activity monitoring * Improved ANSI GUI update mechanism JobDB / JobManager: * Automated zip file recovery in zipped job db * Improved encapsulation of job object * Show runtime for transition to FAILED Logging: * Added ProgressActivity to track progress of some activity * Adjusted debug levels * Improved logging option parser and using it in central script parser * Removed explicit 'user' logger usage Monitoring: * Refactored EventHandler to distinguish between local and remote event handlers Parameters: * Added a GitParameterParser (git), extracting the hash of HEAD of a given path within a git repository * Added a SvnParameterParser (svn) * Added dependency based reordering of parameter sources * Added TrunkedParameterSource - use in repeat and chain * Ensure consistent hash in csv parameter source * Fixed regex_transform parameter source * Improved csv error message * Improved diagnosis of parameter source errors * Improved separation between str and ParameterMetadata instances * ParameterFactory: Removed need for instance variable with the parameter source repository * Parameter source elevation works now with a placeholder psrc and a resolver psrc * SimpleParameterFactory: Simplified parameter parser code * Use chain instead of cross to connect unused datasets to the current parameter space Plugins: * Added more class aliases, __repr__ * Added option to configure plugin bind arguments * Improved instance args binding * Report close matches for mistyped plugin names Reports: * Report inheritance is now based on output method * Enforced uniform job status color scheme in report classes * GUIReport: allow to override height using config string * GUIReport: fixed length of progress bar Runtime: * glite search path fix to avoid centos packages * Moved SE out timestamp to the proper location * Removed some bash-isms * Using proper identation in dataset srm copy cleanup * Using specific directory path instead of using the stack Scripts: * datasetDBS3Add.py: parent_logical_file_name logical file name is not working (at least for me), but it is not need for a DBS3 migration. So comment out at the moment * datasetListFromLS.py: Added option to recurse into local directories Storage: * Added new gc-storage-tool with gfal + fallbacks and new protocols (eg. http, ftp, S3) Tasks: * Added GC_TASK_TIME similar to GC_TASK_DATE * DataTask: Allow to initialize multiple datasources Misc: * Added option "workdir space timeout" * Added unspecified and when_unspecified * Improved debug console * Improved process run error message * Possible fix for non-threadsafe shutils implementation * Removed LoggedProcess from main backends * Renamed API and settings modules * Replaced raw file handles with SafeFile instances -----BEGIN PGP SIGNATURE----- iQJDBAABCAAtFiEEkMDxtSFO/IZ82i5JcNxbYKTQMR0FAllkpkQPHHN0b2JlckBj ZXJuLmNoAAoJEHDcW2Ck0DEdtg0P/jqqFmorNhpcaTmp6smH0EpUrG90NyP4UP/6 HV52ZwWwNufTHJJjHnafCXqt2vIyGSs7yYtM2NPykzq3oQCueIFoGVaa/ioxrOhO Ixhvbs7iGdY4x7mvXjibz3rxLNfoJdETgviv5eZ0zropjw9caZtVq8nMDuGEwDkq dW90IqsWbJ8QpVnyvfPyjG6ghE8q0sz9xog17rwmXUOlEzg7SSAi6ZkIwcWuF0+Q S6VsqOsNyr8TbDz4m5mqhBlhsnsvU+H3rjzzbWpTTmkVvQwMsQ+QDmcoOuN+dMa5 v2Ue3ysbA8dFfe7PmGG5XSymm7c42j7WV5l8Ng/01Wd6Wj+D3aHe9ck/nE4Mo5Ay V2K+KMIEDdkPxuV83CA0mvG/dosWdBSXkSi3oR0hiO0Nhv6PKoKPgnU28LdKJ2RW qVy1JDBW+gxf98P9y7QCw20jbzWqT5cGeqiCJf3AGLaxrmi43RfHD1hQcPn+cg// j9+cY91S6ANnKuYh9vfeH1PX7ekiAnE+qxN/wU9fhfsVdnIWlHImnfurIdb2neTc 9OTLW6jJUqjNFU1Emw7JuWiL2ZEgnKoNgd9Bj4Fs1IGBmhELdoMrmvdiZuRCS6Tj A4dityLBRQahVyZb3OTvHlSJVIpMb386KhUuLsnwD4cPhstl8wUfyzg+88NkARW+ O3w9GYko =C+eK -----END PGP SIGNATURE----- | 11 July 2017, 10:18:51 UTC |
| refs/tags/r1941 | r1941 | 59fccda | grid-control - Bozeman Edition General: * Increased test suite code coverage to >91% * Updated configuration, copyrights and examples * Added API examples * Added codeclimate config file * Code review and convention changes * Removed dead code, added comments, fixed small issues * Add global config file for KIT (default backend, map GC walltime to condor walltime, map HDPA caching file list) * Performance improvements * Reduced code size Compatibility: * Fixed some python compatibility issues (2.3-3.5) * Fixed pypy segfault * Fixed python3 import issue * Rewritten functions to avoid StopIteration (due to PEP 479 changes in python 3.5) Runtime: * Implemented shellcheck suggestions Core: * Added environment variable to disable the gc logo * Fixed output processor API and inheritance * Matcher case sensitivity, fixed display of matcher infos * Removed global state / registry variables * Renamed Workflow.jobCycle to Workflow.process Exceptions: * Added more exception handling * Avoid using handleException in LoggingHandler * Fixed cleanup of ignored exceptions * Fixed file paths shown in exceptions * Refactored hpfwk exception tools * Removed handling of impossible exception Config: * Added config option aliases * Added getMatcher to config interface, fixed interactivity checks * Added option to make matcher settings mutable * Adding flag to control config registration with the error handler * Allow config option resolving for all fillers * Allowed to call getConfigDict from instances of Settings * Ensure options are sorted in python-config-interface * Ensure workdir is in current.conf, reduced nesting of config interface instances * Fixed issue with inheriting sections from TaggedConfigView without set classes * Fixed multiple config option retrieval * Fixed new config interpolation * Improved error message for config filler * Improved ListFilter interface, logging output * Refactored config file parser * Renamed createConfig to create_config and changed default value for useDefaultFiles * Renamed filter plugin option * Unified interactivity checks and added hook for testing Utils: * Enabled default enum hashing * Improved error messages in DictFormat parser * Moved split_* functions into utils.parsing * Removed eprint/vprint/verbosity functions * Rewritten thread creation functions to store parentage in thread name * Removed gc_itertools and moved functions into python_compat / thread_tools * Using ensureDirExists and exitWithUsage where needed Process: * Added ability to set environment of LocalProcess (used for thread-safe monitor script calls) * Fixed double free of file descriptor * Fixed LocalProcess restart * Moved deprecated LoggedProcess into separate file Datasets: * Added DataProvider.bName * Allow to stream dataset information *through* the dataset I/O * Fixed dataset provider and processor API to improve clarity, fixed handling of missing metadata * Fixed issues with dataset providers (in particular ScanProvider and InfoScanners) and dataset / partition processors uncovered by new testsuite * Fixed PartitionEstimator setup * Implemented RequirementsPartitionProcessor to derive wall time / cpu time requirements from the number of entries * Moved shared triggerResync instance out of DataProcessor * Refactored DataProvider, made many options mutable, fixed resync of file based partitions * Refactored (dataset/partition)processor filter * Refactored source file of PartitionProcessor * Removed unused DatasetID * Renamed some variables in DataSplitter * Reorganized dataset processors * Replaced GCScanProvider setup variables with dynamically loaded GCProviderSetup classes * Simplified phedex site selection and unified site name handling in CMS DataProviders Parameters: * Added API to get list of used sources, fixed display of some parameter sources * Added NullParameterSource * Added SubSpaceParameterSource and support for subspace references to SimpleParameterFactory * Added TransformParameterSource * Allow nesting ParameterFactories * Changed inheritance of DataParameterSource * Changed ParameterInfo to use makeEnum * Fixed frange implementation * Implemented automatic simplification of parameter space trees * Improved csv error reporting * Introducing a repository variable to parameter management to replace the dataset registry * Moved canFinish into ParameterSources * Moved CombineParameterSource into development file * Public export of ParameterAdapter * Refactored parameter resync logic * Removed direct imports from ParameterFactory * Removed ParameterSource.managerMap and replaced it with general purpose plugin accessors * Reworked unused datasource discovery and made default parameters configurable in ParameterFactory * Simplified config handing in ParameterFactory GUI: * Added NullGUI * Fixed CherryPy GUI and refactored plugin graph creation * Fixed some GUI output issue * Fixed wait activity output * Removed dead code (ANSI.getyx) * Rewritten Activity logger - using new thread facilities * Rewritten GUI to fixed display issues - using new activity logger * Untangled BasicReport and BarReport, added new ColorBarReport CMSSW: * Added option to use run based lumi filter * Added SiteDB cache * Fixed api access in SiteDB * Fixed display of nickname dependent variables in CMSSW_Advanced * Fixed issue with CMS dataset providers * Fix for pure pro/epilog CMSSW Tasks * Refactored scram stuff into SCRAMTask * Refactored SiteDB * Renamed module with SiteDB implementation * Using GridJSONWebService in dbs3_lite_client Tasks: * Added indirection via config option between task walltime setting and wms.canSubmit check * Removed unused report and getSubmitInfo function from task Monitoring: * Fixed monitoring calls (for dashboard) Report: * Added JSON table output * Fixed BasicReport handling of workflows with 0 jobs * Removed magic number from BasicReport Backend: * Continued backend refactoring * Added promiscous logging mode * Create access token after the storage manager * Explicit parameters for job cancel function * Fixed access token retrieval * Fixed backend job state results * Improved OutputProcessor exception handling * Condor: Docker universe needs another JDL file * Condor: Fix HDPA filelist (Don't create empty files.txt) * Condor: Patch for docker * Condor: Prevent Job.DONE for missing jobs if the job status program returned an error state * CREAM: Fixed purge * GridEngine: Fixed regressions * GridWMS: Refactored JDL writing into separate class * MultiWMS: Ensure reproducible WMS selection Scripts: * Fixed API issues in scripts * Unified display of available plugins * datasetDBS3Add.py: Fixed API regressions * go.py: Removed deprecated command line option, updated help message * lumiInfo.py: Fixed opts.job_events * parameterList.py: Added option to change the parameter factory * parameterList.py: Fixed user selected parameter factory Logging: * Fixed logging deadlock (python 2.3) * Improved and simplified logging * Make log_process messages customizable Webservices: * Added webservice session aliases * Improved error message in VOMS GridServices * Refactorized RestClient HPFWK: * Added base class iterator to Plugin to support this functionality * Changed behaviour of init_hpf_packages * Refactored plugin class discovery -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAABCAAGBQJXzMWeAAoJEHDcW2Ck0DEdH20QANuS0awZuHLQj4Kw5iMufROv Ukms3cG+f8/o0t6geG+i5G/JOiLTYib56U1z8FLwekRZiYfvOtjxmo+vb/Dc3C1t Ednc+lQD8PBfNtsptWEDy97ExkKRWpVXnygmLPwDSd5oaRb7/UTgEdcTZ6vULNpN OpXSh31XJkxh9YG4hjag+va8S4iOzyyerD4MQuXromG0vEoRfz6lewtdfsPs/spC JYn8sl1dCWXO++QI9k/RCwaZbfK7pWuEXR0xFqeO8bfNlQNJRvKJ+uTgk0F80KGe SeaHJjUrPNUGGfc0pVm64X/gnISeUrdLD+tYSIukSAvuVp35IELQeIvUn7ZpFzgO QrX5NnZnewCYeycfJGQATbEJQwloWklOvnyAk7PukY611P+h6EYq07+qHXX0l/UH 7z68fq3bpPi0vbvF3P3rRWOttVXfbfn+p/gH0EcOt2UUNDkn3EraJfn78nXOsdzH T3/sElZJGnriFSvQypK7ofEXmw9uu3prIsctvEr5aal78UrSnyuwECxpzfywFaG2 4ofp5LUx/lxZtwUbWCvyzgydOAhu5IhWHeOzwZWDP1RKBtKUrakUSceuvX/Mdsg7 ifoBkrWitpO3AJkGmKBPlvLXrBsLOUrD7MnTbFpCAv0++hHtGvVTsbSXNeSTY9Cy t5WLMOYulGuoHQRjPqYx =OVv+ -----END PGP SIGNATURE----- | 05 September 2016, 01:08:33 UTC |
| refs/tags/r1864 | r1864 | 0a15cdd | grid-control - Reliant Edition General: * Added CMSSW testsuite configuration files modelling a complete workflow * Added codecov.yml * Fixed code review issues and improved error messages * Updated examples Backends: * Added CREAM CE backend * Moved PoolType to makeEnum and the config getter to getEnum * Split access token plugins separate files * Condor: Forward cpu and memory requirements CMSSW: * Added SCRAMTask as base class of the CMSSW task plugin * Allow environment variables in lumi filter path * Ensure that only a unique list of config files is added to the sandbox * Transform value of "vo software dir" option into absolute path if it exists * Changed "phedex t1 mode" config option to an enum option * Made prolog and epilog scripts uniform - therby fixing prolog script issues * Fixed CMSSW software directory discovery * Fixed DAS provider * Fixed config file issue with CMSSW_Advanced * Fixed instrumentation of MC config files * Fixed adding VO_CMS_SW_DIR option * Fixed multiple script issues relating to CMSSW tasks * Ensure that the basename is used in several places of gc-run.cmssw.sh Core: * Added job name generator plugins * Check existance of copied file, since copy commands sometimes return 0 regardless of problems * Ensure all GC variables are fully uppercase * Removed arbitary access to first job to get list of variables * Removed readURL and readJSON calls * Replace environment variables in getCommand calls * Return error code in case of exception Runtime: * Added GC_GLITE_LOCATION to force a glite location Monitoring: * Added onFinish handler to monitoring class * Allow monitoring scripts with multiple arguments Datasets: * Added DataProcessor to sort by dataset, block and file URL * Added DebugJobProcessor to display error logs in case of job failures * Added "enabled" method to allow skipping inactive dataset and partition processors in data pipeline * Added MetaPartitionProcessor to default partition processor list * Added several aliases to DatasetProcessors * Allow changing of all DataProcessor options - now they automatically trigger a resync of the dataset and parameter space * Allow to use multiple ScanProviders * Ensure correct data type for stripMetadata keyword option in dataset writer * Ensure unchanged ordering in flattened config file list * Fixed lumi processor issue triggered by missing metadata * Infoscanner now disables user output before creating external workflow instance * Use getPath to query workdir in InfoScanner * Prevent running lumi filter twice (leading to errors with strict lumi filter enabled) Configuration: * Added content of all config instances to the GC debug log file * Added GC_CONFIG_FILE and GC_CONFIG_DIR to reference the current config file/dir in options given in the config file * Added triggerResync Handler * Changed config API calls to conform with the needs of the upcoming automatic config documentation generator * Display replaced string instead of original string in substVars check * Fixed interaction of dynamically set "+=" and "-=" with default options * Fixed path options with variables and relative path specifiers * Improved config entry processing Parameters: * Added 'depends' member function to parameter source to get dependent variables * Added __repr__ of constant parameter source * Fixed CSVParameterSource * Fixed default value access in lookup parameter source * Fixed inconsistencies in lookup between variables defined in "constants", "parameters" or the task section * Use ConstParameterSource in case of single value parameters Scripts: * datasetListFromX.py: Added option to dump config for the used command line options * lumiInfo.py: Fixed to allow working with MC workflows * parameterList.py: Use dataset information from work directory if it exists * parameterList.py: Using full partition processor list for DatasetParameterSource -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAABCAAGBQJXQJAIAAoJEHDcW2Ck0DEdppIQAMqwM6//X3a80HZ9Czc2AlZG aXZzi8b0RPCusp8mTyKdeyMrbaZRM+QW+xBp0rL6MBk5wYHy/F9zxbzstN9lvHq8 5W+grAvuYC0SXs97dTgWjKeMuiaplAwiXqnAhnIq6Cm90yXbzFK8Pc/YY2Y65LvQ sJPba2eySU71qF8j5loIImZA6zxvHIk0Qqh0+9bWKo54LldBoR7vSFclCO1ODXTo 8bBylgPfF9Pow+k512IydKxbZQHdCGbNKUsrk8Vh/TzXZ9iUZvONNZjld2Ht4HYF kwzNp4TIRFNi6YV+aaHZwTzrgWm8Nb8dibLcCQgEcRkrZ+V6YHjE7DrQ8dQJX5uW aqjdGR/yd7rTESUvCr0LuIcdKoAsZIPqQ3i984SH+GmSyPYsIL+hfiAIJ2cMqghy Gs9UXht95ufigBMNz2R05K8rwb6ULvQY2UeXAgUPDNLr75J+a620fX3iLwpjettL SsZ6KeV7JL8jNcPJ4uTp7QfpPnNBqdUHSNVy4BXhy938OtN6Yyl7LovEQ8Jxnx8b 4R/dyd/UA81HOFWpb1swnhOvxhj5tsPJ6CFCI7tLnMp+dUBNxP7S9WI2Evjjp2qa y5nr3IKMjfZtViuLpRmj+uIyul075Sl01pmVaSr4kH1ryADqvb3hR5vhRTFqbveM o4CkTnj4un0p58rWe/rO =vrE2 -----END PGP SIGNATURE----- | 21 May 2016, 16:42:03 UTC |
| refs/tags/r1764 | r1764 | dfb284e | grid-control - Defiant edition General: * Fixed source review issues * Fixed documentation * Added many aliases for more conventient access to functionality Packaging: * Fix for pip installations * Updated packaging script to take python version into account * Updated travisCI config file * Added codecov config file * Updated NOTICE file Core: * Added ability to query multiple plugins to hpfwk * Replaced instances of LoggedProcess (using bundled popen2) with multi-threaded LocalProcess (using only stdlib) * Made WMS job status parser configurable * Catch exceptions in threaded tchain and made raise_any exception parameter optional * Improved exception information in case the plugin is not found Config: * fix for lone "-=" in config options * Improved ListFilter to access items through a key function and allow to negate matcher output * changeImpossible handler: Display raw objects in case the string representations are identical * Improved error message about using nested variables Logging: * Fixed debug.log output in the presence of the --debug option GUI: * Added ANSI strip function * Added new visualization to cherrypy gui * Added NullReport * Added backend report module (fixing #42) Runtime: * Fix for grid UI discovery script Backend: * Added experimental SLURM backend * Moved JMS (WMS by KIT based on SLURM) to own file * Added grid class lookup redirection to prevent loading grid plugins * Use logging *print functions in LocalWMS * Refactored job.info parser into OutputProcessor * Added "old style" local backend selection via "[local] wms" option * Properly setup wms broker in MultiWMS * Fixed sandbox creating race condition Datasets: * Added ConfigProvider to specify datasets from within the config file * Added TFCPartitionProcessor * Added NullScanner to output final entry of info scanner pipeline * Wrap dataset retrieval error in DatasetError for improved exception information * Reset dataset statistics after query * Moved url filter DataProcessor to filter infrastructure and added ability to use other datasources as filter entry sources * Allow running CMSSW Metadata info scanner without cmssw.log.gz (for CMSSW epilog tasks) * Improved CSV file format sniffer * Improved bounds checking of dataset metadata * Fix to prevent abort on broken output dir content in OutputDirsFromWork info scanner CMSSW: * Split CMSPartitionProcessor into LFNPartitionProcessor and CMSSWPartitionProcessor * Fixed issue with CMS partition lfn prefix Scripts: * Fixed several issues with scripts due to latest version change * plugin.py: Added script to query available plugins * lumiInfo.py: Added option to change json output directory * getDatasetFromGC.py: Added option to configure list of info scanners -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAABCAAGBQJXLmcoAAoJEHDcW2Ck0DEdSWsQAMFRBEEkPQpjnflTnZhGNNyy 4CxBsFVKksgzNaLqCyQEhqw3HXXONMneCP8Pz7k9zIK4V47tpZCauITJhDNldnrD PBxVPcHKce73/XsZ6EeXBcRIi6ZktvQV8rw4CFo02HPTByuZl4s5HxuWHb7FMyUj ChzTJdotIGWN72LK+Qcz8bIA/OCzFiYyFCxNIU5oClYNK8VmyCQ7OEiR9uH0zDAE mp207eufhm7Mx32Gfjp2A1tNJm/EsfA8BmX/lmpHfTAxvAa/93KdztBNoEAfAirC z0ATiKPcTwElJteu8gfc52tls4ib2WA9EXSVDobTnfzLvBxfXsSedoNKXHQTd1/6 FcLHQD5IGbGfkNjlo/p/Htzf7bcS+8S/9QFM3m2nDrfHEb52C6zsxKzw++vHg/+2 ywz+Yj8iSYWgA0vIMHlWtBxxYx8a6xbkXSmqCEC5t7OWHTOxvWWuNPO/dA4Nq92x FO4gHQPM2aGey7yZx7sC7HUK47IB6bCD3YEFk+ybLfJMC+e5rTQK4HfHGL81jseW EYpOFChMj94eNcJWvi1QikR1xcjlAjhGts6bH+zgnPLXbwvA9sdjQZ0OtQJ0xvOK +DscUZ/TCt5FZ7ZuBip1Uqem/clP73pWANizSEKTYzDzN9U+WSzZLApfFEdNi6wD clXInW/pu5u6nElOvfmm =KVbs -----END PGP SIGNATURE----- | 07 May 2016, 22:07:18 UTC |
| refs/tags/r1701 | r1701 | 141c0c5 | grid-control - Enterprise edition General: * MAJOR: Added pypy and Python 3.x support * Continuing support of Python 2.3 - 2.7 * Added pypi entry for grid-control * Fixed some issues when running under windows * Fixed code review issues and reduced code complexity * Integrated with Travis CI / codecov / landscape.io services * Updated examples Configuration: * MAJOR: Completely new config subsystem * Support for python config files * Support environment variable expansion in paths * Removed $PWD and GC directory from default path search list * Added new command line parser * Added matcher, list lookup and dict lookup classes * Added $GC_CONFIG as new default config file path * Added path of parent config file to resolve list of include files Dataset: * MAJOR: Redesigned flow of dataset information to allow processing steps at all levels ( (InfoScanner ->) DataProvider -> DataProcessor -> DataSplitter -> PartitionProcessor -> DataParameterSource ) * Moved dataset metadata format to JSON * Enabled user requested resync of dataset * Changed default nickname to preserve processed dataset name * Added DataProcessor to apply a lumi filter * Added DataProcessor to configure events per job / files per job according to partition targets * Added DataProcessor to ensure uniqueness of lfns * Added DataProcessor to protect against duplicate nicknames * Added DataSplitter to split datasets into pieces with the same (subset of) metadata * Added getDatasets function to DataProvider * Added global tag retrieval to ObjectsFromCMSSW info scanner * Added InfoScanner JobInfoFromOutputDir to check job exit code and store job info dictionary in objStore * Added option to choose location format in CMS Dataset providers * Added option to RunSplitter to have N runs in a single partition * Added option to switch to old nickname algorithm * Added ConfigDataProvider to get dataset information directly from config file * Added PartitionProcessor to allow nickname specific lumi filters and include them as transient variable * Added PartitionProcessor to filter on dataset locations after splitting - ie. for running tasks * Added PartitionProcessor to forward dataset metadata to the job environment * Changed name of 'SEList' member to more neutral 'Locations' * DataProvider getState should load as-is - therefore removed ability to change settings for ListProvider and disabled additional checks * NickNameProducer is now a DataProcessor * ScanProvider: Added option to estimate number of events in dataset from scaled variables (eg. file size) Backends: * Added groundwork for HTCondor backend * Added AFSAccessToken (used by default on naf machines) * Added check about the sandbox size * Added many common aliases to backends * Added memory option to local WMS, cleanup of NAF options * Added option to translate software requirements into batch submit options * Added requirement parsing to WMS * GliteWMS: Disable wms endpoint discovery by default * Made backend timing configurable * Renamed *Proxy classes to *AccessToken * VomsProxy: Added proxy path option * GliteWMS: Added option to perform full WMS discovery instead of incremental search Runtime: * Changed how timing information and file metadata is stored in job.info * Improved search for grid UI * Display LSB release info * Fallback to lcg-cp if srmcp fails * Log received signals in trap function * Bash environment scripts now return error code 1 in case of failure * Build debug_helper string in variable before printing it to avoid crosstalk by background processes Parameters: * Moved parameter metadata format to JSON * Parameter default change: 'equal' is the new default lookup matcher * Added option to specify formatting variables * Added option to repeat particular parameter values * Added option to repeat a specific parameter value * Fixed general purpose lookup parameters in BasicParameterFactory * Avoid repeated resyncs by updating config file Monitoring: * Refactored Monitoring and added dedicated thread pool class * Removed config file from monitoring environment - use data from work directory instead * dashboard: Avoid reporting tasktype 'production' - even if the task is a MC production task * dashboard: Fixed user name reporting * scripts: Enforcing uniform GC_ prefix for internal environment variables * scripts: Event handler will wait until timeout for external processes to finish GUI / Report: * Main report in GUI overview and JobManager are now customizable * Completely new Report interface - consumes optional task information * Added BasicBarReport: Display normal task status with progress bar * Added CategoryReport: Base class for reporting about task parameter categories * Added GUIReport: Display adaptive graphical category report * Added ModuleReport: Display simple text category report * Added PlotReport: Generate a large variety of plots (cpu / wall time, bandwidth, etc.) * Added TimeReport: Display consumed wall time (and estimate cost) * Changed GUI + workflow interaction * Enforcing ANSI GUI update in case of vanishing activity CMSSW: * DBS3 is the new default CMS data provider * Removed DBS2 support * Using CVMFS if available (and removed AFS fallback) * Added option to change lfn prefixes of running tasks (eg. to give lfns a dcap/xrootd prefix) * Added option to define lfn prefix shortcuts * Added option to select the DBS instance for DBS3/DAS * Added sadditional phedex parameters * DBS registration: Derive parent LFN from parent PFN if LFN not given in framework job report * DBS registration: The instrumentation file now prints the GlobalTag (to avoiding parsing the config file) * Fixed problems with CMSSW prolog/epilog scripts * Generate CMSSW tarball only on request * Renamed local CMSSW tarball from runtime.tar.gz to cmssw-project-area.tar.gz * Updated Lexicon.py from DMWM repo and fixed python 3 issues * Use SiteDBv2 to lookup cmssitename from given se fqdn * Fixed instrumentation of config files * Improved error messages in lumi filter processing * Refactored CMSSW Advanced to use existing facilities Scripts: * Refactored all scripts to use new option parser and reduce code complexity * Added datasetDBS3Add.py: New script to register DBS3 datasets * go.py: Refactored grid-control main user script * go.py: Added --debug to enable direct exception logging * go.py: Added --logging to quickly set log levels * go.py: Added --reset to reset resubmit attempts * go.py: Removed help-vars option for now * datasetInfo.py: Added option to specify default data provider * datasetInfo.py: Improved output of metadata related infos * datasetListFromCMSSW.sh: Added protection against running without CMSSW environment * downloadFromSE.py: fixed output in threading mode * downloadFromSE.py: report job number of failing download * lumiTool.py: Added option to get lumi json file from parameterised tasks * report.py: Added option to get list of known plugins * report.py: Running with task information is now optional Core: * Added JobDB migration class * Added option to run SandboxProcessor instances on successful jobs * Added option to mark variable substitutions * Renamed option 'kick offender' to 'defect tries' * Added preliminary implementation of 'output processor' class * Added signal handler to trigger a debugging session (SIGURG) * Allow non-interactive workdir creation * Changed interface of enum implementation and allow arbitarty enum values * JobDB: Added safeguard against broken job state * Refactored job manager and split out kickoffender and chunk verification into new (default) SimpleJobManager * ZippedJobDB: Improved auto-recovery of broken files * Unified version information * Fixed incomplete readin of job database * Refactored table printing Plugins: * Refactored dynamic class loader and changed all imports to use absolute specifiers * Allow plugins to directly define aliases (removed many alias classes) * Added class function getClassNames to get name of classes and aliases * Added general purpose InstanceFactory * Added infrastructure to retrieve config section from the classes Utilities: * Moved base functionality into new HPFramework library (hpfwk) * Started splitting utils.py into multiple files * Improved exception handling (including advice how to contact developers) * Added library with interruptible Event, Lock and Queue implementation * Added multi-threaded LocalProcess (supports timeouts, interactive, all python versions) * Added REST client (using bundled requests library for modern python versions, fallback to urllib2) * Added 'bind' API that allows classes to specify how they are constructed from strings * Added hang_protection function - used to determine free space and to resolve hostnames * Added Result class (small named tuple) * resolveInstallPath now takes the first matching executable file and also handles multiple matches Logging * Added default settings for default, user and user.time logging streams * Added logging handler to store process log infos in archive * Added manual finish method to ActivityLog * Added process logging function * Added uniquefilter class to user log streams * Changed exception settings to make them consistent with default log setup * Enable exception output when logging setup is not yet called * Made logging setup configurable from config file * Refactored logging setup * Removed check from logging setup to cope with [DEFAULT] options unrelated to logging * Using python default logging infrastructure for almost all user output * Made logging streams compatible with GUI streams -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAABCAAGBQJXLmY/AAoJEHDcW2Ck0DEdem4P/jr3wG9CFhC8zaxjToq3R0qt viioY1+Pc71oEdztOgCX7+q3LJYxq5TLVRNLolenGet5NtCuk7qYkPwF/Z6W+Knt Jr8Pq6fhVjf21/qQzgxJZuWjqjcVBS7CnY+9to3+wqVhzng6Ylj00B9/CQk+OBTh KECnpZSlLZHCDKVS1g1Ch4Uc7gJtsYC+1xdBIALmVWbgBswiYlMW0QRNyzlvMHCk OZ4P4BbaokkUn7+dWhIibrZ6XzIyiILVAH1i2hhDCtvV9hZVQ8nOB3lcMfi46A65 A+qfipclYCdX2Qe3gHQUYKFJcdD2whpW1mWpJBPFFGldsBGNSvSOzlgC6PitXEJb mRyuIRvakVGXq/IRFoZ2CQwhAFbDMG8KmL7Lm5VKTC3xUbjVts4RLdnekkBi3TPm BET9l9zTk04jt4guArF6XsnL2uPGvnRGOYTXhLvXzFjRo6pzqImaNaZ0Qojwb1UV 8ub4mjtkBjo0xKIpFptP3JX8U/2JrNUwoYm7XaERLYIkWawssxJDi1usbgGGZNJi hmx+AT3T4lId6i2UF98hUpPs6aiiZRZcY3h3SA4OJemlW6JmyHKxTTKFyyG18gQ0 CQBYvxU3/eLrEnWrR74L/c87r/JQm4afgU5WRRJ2IZkual0KQTKCO1CyCNFGLPgy XfwUpa3/Vyw+iZ2H5np7 =SlSI -----END PGP SIGNATURE----- | 07 May 2016, 22:03:05 UTC |
