Release r1701 created by Fred Stober on 07 May 2016, 22:03:05 UTC
grid-control - Enterprise edition
General: * MAJOR: Added pypy and Python 3.x support * Continuing support of Python 2.3 - 2.7 * Added pypi entry for grid-control * Fixed some issues when running under windows * Fixed code review issues and reduced code complexity * Integrated with Travis CI / codecov / landscape.io services * Updated examples Configuration: * MAJOR: Completely new config subsystem * Support for python config files * Support environment variable expansion in paths * Removed $PWD and GC directory from default path search list * Added new command line parser * Added matcher, list lookup and dict lookup classes * Added $GC_CONFIG as new default config file path * Added path of parent config file to resolve list of include files Dataset: * MAJOR: Redesigned flow of dataset information to allow processing steps at all levels ( (InfoScanner ->) DataProvider -> DataProcessor -> DataSplitter -> PartitionProcessor -> DataParameterSource ) * Moved dataset metadata format to JSON * Enabled user requested resync of dataset * Changed default nickname to preserve processed dataset name * Added DataProcessor to apply a lumi filter * Added DataProcessor to configure events per job / files per job according to partition targets * Added DataProcessor to ensure uniqueness of lfns * Added DataProcessor to protect against duplicate nicknames * Added DataSplitter to split datasets into pieces with the same (subset of) metadata * Added getDatasets function to DataProvider * Added global tag retrieval to ObjectsFromCMSSW info scanner * Added InfoScanner JobInfoFromOutputDir to check job exit code and store job info dictionary in objStore * Added option to choose location format in CMS Dataset providers * Added option to RunSplitter to have N runs in a single partition * Added option to switch to old nickname algorithm * Added ConfigDataProvider to get dataset information directly from config file * Added PartitionProcessor to allow nickname specific lumi filters and include them as transient variable * Added PartitionProcessor to filter on dataset locations after splitting - ie. for running tasks * Added PartitionProcessor to forward dataset metadata to the job environment * Changed name of 'SEList' member to more neutral 'Locations' * DataProvider getState should load as-is - therefore removed ability to change settings for ListProvider and disabled additional checks * NickNameProducer is now a DataProcessor * ScanProvider: Added option to estimate number of events in dataset from scaled variables (eg. file size) Backends: * Added groundwork for HTCondor backend * Added AFSAccessToken (used by default on naf machines) * Added check about the sandbox size * Added many common aliases to backends * Added memory option to local WMS, cleanup of NAF options * Added option to translate software requirements into batch submit options * Added requirement parsing to WMS * GliteWMS: Disable wms endpoint discovery by default * Made backend timing configurable * Renamed *Proxy classes to *AccessToken * VomsProxy: Added proxy path option * GliteWMS: Added option to perform full WMS discovery instead of incremental search Runtime: * Changed how timing information and file metadata is stored in job.info * Improved search for grid UI * Display LSB release info * Fallback to lcg-cp if srmcp fails * Log received signals in trap function * Bash environment scripts now return error code 1 in case of failure * Build debug_helper string in variable before printing it to avoid crosstalk by background processes Parameters: * Moved parameter metadata format to JSON * Parameter default change: 'equal' is the new default lookup matcher * Added option to specify formatting variables * Added option to repeat particular parameter values * Added option to repeat a specific parameter value * Fixed general purpose lookup parameters in BasicParameterFactory * Avoid repeated resyncs by updating config file Monitoring: * Refactored Monitoring and added dedicated thread pool class * Removed config file from monitoring environment - use data from work directory instead * dashboard: Avoid reporting tasktype 'production' - even if the task is a MC production task * dashboard: Fixed user name reporting * scripts: Enforcing uniform GC_ prefix for internal environment variables * scripts: Event handler will wait until timeout for external processes to finish GUI / Report: * Main report in GUI overview and JobManager are now customizable * Completely new Report interface - consumes optional task information * Added BasicBarReport: Display normal task status with progress bar * Added CategoryReport: Base class for reporting about task parameter categories * Added GUIReport: Display adaptive graphical category report * Added ModuleReport: Display simple text category report * Added PlotReport: Generate a large variety of plots (cpu / wall time, bandwidth, etc.) * Added TimeReport: Display consumed wall time (and estimate cost) * Changed GUI + workflow interaction * Enforcing ANSI GUI update in case of vanishing activity CMSSW: * DBS3 is the new default CMS data provider * Removed DBS2 support * Using CVMFS if available (and removed AFS fallback) * Added option to change lfn prefixes of running tasks (eg. to give lfns a dcap/xrootd prefix) * Added option to define lfn prefix shortcuts * Added option to select the DBS instance for DBS3/DAS * Added sadditional phedex parameters * DBS registration: Derive parent LFN from parent PFN if LFN not given in framework job report * DBS registration: The instrumentation file now prints the GlobalTag (to avoiding parsing the config file) * Fixed problems with CMSSW prolog/epilog scripts * Generate CMSSW tarball only on request * Renamed local CMSSW tarball from runtime.tar.gz to cmssw-project-area.tar.gz * Updated Lexicon.py from DMWM repo and fixed python 3 issues * Use SiteDBv2 to lookup cmssitename from given se fqdn * Fixed instrumentation of config files * Improved error messages in lumi filter processing * Refactored CMSSW Advanced to use existing facilities Scripts: * Refactored all scripts to use new option parser and reduce code complexity * Added datasetDBS3Add.py: New script to register DBS3 datasets * go.py: Refactored grid-control main user script * go.py: Added --debug to enable direct exception logging * go.py: Added --logging to quickly set log levels * go.py: Added --reset to reset resubmit attempts * go.py: Removed help-vars option for now * datasetInfo.py: Added option to specify default data provider * datasetInfo.py: Improved output of metadata related infos * datasetListFromCMSSW.sh: Added protection against running without CMSSW environment * downloadFromSE.py: fixed output in threading mode * downloadFromSE.py: report job number of failing download * lumiTool.py: Added option to get lumi json file from parameterised tasks * report.py: Added option to get list of known plugins * report.py: Running with task information is now optional Core: * Added JobDB migration class * Added option to run SandboxProcessor instances on successful jobs * Added option to mark variable substitutions * Renamed option 'kick offender' to 'defect tries' * Added preliminary implementation of 'output processor' class * Added signal handler to trigger a debugging session (SIGURG) * Allow non-interactive workdir creation * Changed interface of enum implementation and allow arbitarty enum values * JobDB: Added safeguard against broken job state * Refactored job manager and split out kickoffender and chunk verification into new (default) SimpleJobManager * ZippedJobDB: Improved auto-recovery of broken files * Unified version information * Fixed incomplete readin of job database * Refactored table printing Plugins: * Refactored dynamic class loader and changed all imports to use absolute specifiers * Allow plugins to directly define aliases (removed many alias classes) * Added class function getClassNames to get name of classes and aliases * Added general purpose InstanceFactory * Added infrastructure to retrieve config section from the classes Utilities: * Moved base functionality into new HPFramework library (hpfwk) * Started splitting utils.py into multiple files * Improved exception handling (including advice how to contact developers) * Added library with interruptible Event, Lock and Queue implementation * Added multi-threaded LocalProcess (supports timeouts, interactive, all python versions) * Added REST client (using bundled requests library for modern python versions, fallback to urllib2) * Added 'bind' API that allows classes to specify how they are constructed from strings * Added hang_protection function - used to determine free space and to resolve hostnames * Added Result class (small named tuple) * resolveInstallPath now takes the first matching executable file and also handles multiple matches Logging * Added default settings for default, user and user.time logging streams * Added logging handler to store process log infos in archive * Added manual finish method to ActivityLog * Added process logging function * Added uniquefilter class to user log streams * Changed exception settings to make them consistent with default log setup * Enable exception output when logging setup is not yet called * Made logging setup configurable from config file * Refactored logging setup * Removed check from logging setup to cope with [DEFAULT] options unrelated to logging * Using python default logging infrastructure for almost all user output * Made logging streams compatible with GUI streams -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAABCAAGBQJXLmY/AAoJEHDcW2Ck0DEdem4P/jr3wG9CFhC8zaxjToq3R0qt viioY1+Pc71oEdztOgCX7+q3LJYxq5TLVRNLolenGet5NtCuk7qYkPwF/Z6W+Knt Jr8Pq6fhVjf21/qQzgxJZuWjqjcVBS7CnY+9to3+wqVhzng6Ylj00B9/CQk+OBTh KECnpZSlLZHCDKVS1g1Ch4Uc7gJtsYC+1xdBIALmVWbgBswiYlMW0QRNyzlvMHCk OZ4P4BbaokkUn7+dWhIibrZ6XzIyiILVAH1i2hhDCtvV9hZVQ8nOB3lcMfi46A65 A+qfipclYCdX2Qe3gHQUYKFJcdD2whpW1mWpJBPFFGldsBGNSvSOzlgC6PitXEJb mRyuIRvakVGXq/IRFoZ2CQwhAFbDMG8KmL7Lm5VKTC3xUbjVts4RLdnekkBi3TPm BET9l9zTk04jt4guArF6XsnL2uPGvnRGOYTXhLvXzFjRo6pzqImaNaZ0Qojwb1UV 8ub4mjtkBjo0xKIpFptP3JX8U/2JrNUwoYm7XaERLYIkWawssxJDi1usbgGGZNJi hmx+AT3T4lId6i2UF98hUpPs6aiiZRZcY3h3SA4OJemlW6JmyHKxTTKFyyG18gQ0 CQBYvxU3/eLrEnWrR74L/c87r/JQm4afgU5WRRJ2IZkual0KQTKCO1CyCNFGLPgy XfwUpa3/Vyw+iZ2H5np7 =SlSI -----END PGP SIGNATURE-----
Target: 141c0c565ee85857c67b39f66c0f02bce1a88bfa
Directory: 3cd65fe1269fb58e85b9a05f3a08eda91a63805e
Directory: 3cd65fe1269fb58e85b9a05f3a08eda91a63805e
