https://github.com/twitter/heron

sort by:
Revision Author Date Message Commit Date
ae16822 remove the copying of script file 10 September 2017, 01:11:25 UTC
120643f remove startup scripts 10 September 2017, 00:59:27 UTC
3d18239 healthmgr: aurora heron shell controller (#2283) * healthmgr add aurora heron-shell controller * fix style * wrapper class * address maosong comment * address maosong comment: throw excetion to external logic * update example yaml * fix style 06 September 2017, 03:18:17 UTC
c309ecc Add config mapping for gcs bucket in api server deploy. (#2281) * Add config mapping for gcs bucket in api server deploy. * User heron image. 03 September 2017, 08:22:03 UTC
44e2cd6 cleanup docker scripts (#2277) * add scripts for startup docker and include client install * start a few topologies when you start the docker * added a conf file to start essential services in docker in sandbox mode * docker changes * streamlined docker stuff * remove start services * add supported platforms 03 September 2017, 08:18:33 UTC
21c2378 Add kubernetes service for apiserver. (#2280) * Add kubernetes service for apiserver. * Add comment to service component. 03 September 2017, 05:45:48 UTC
bd77cdf Notify both producer and consumer when init the Communicator (#2279) Currently, when we init the Communicator, the changes will be picked up only when the producer/consumer are notified by other events. It can lead to the race condition. This pull request fixes the issue by notifying both producer/consumer immediately to run with new values. 03 September 2017, 03:07:58 UTC
5a5912b add error log for physical plan check in tmaster (#2267) 01 September 2017, 21:07:00 UTC
bda9ef0 Add kubernetes deploy files. (#2265) * Add kubernetes deploy files. * Make default zookeeper deployment have only 1 container. * Move apiserver yaml config to gcp subdirectory. 01 September 2017, 20:29:54 UTC
da081a7 consistent behavior about when to shade. Shading is done when the final artifacts are generated (#2273) 01 September 2017, 20:29:01 UTC
40a5447 Normalize Shading and fix simulator build (#2271) 01 September 2017, 15:36:50 UTC
110ebd0 Calculate the hostname during constructor rather than during register (#2270) 01 September 2017, 15:36:35 UTC
ca8a05f Make shading uniform accross instance and api (#2266) 31 August 2017, 19:03:39 UTC
23dcad4 Add topology name validation for kubernetes scheduler. (#2264) 31 August 2017, 00:55:56 UTC
aedeab4 [ISSUE-1124] - Windows Bolt support (#2241) https://github.com/twitter/heron/issues/1124 31 August 2017, 00:07:26 UTC
bdd66b3 Read http response for all http response codes. (#2263) * Read http response for all http response codes. * Remove unused import. 30 August 2017, 22:20:55 UTC
0020bd3 ensure empty function with unique names (#2262) 30 August 2017, 20:39:03 UTC
fadd863 Add zookeeper to the heron dist docker ubuntu14.04 (#2259) * Add zookeeper to the heron dist docker ubuntu14.04 * Remove zookeeper home env variable from container. * remove versioned zookeeper directory. 30 August 2017, 19:45:19 UTC
fb441ef changes to make heron compile with bazel 0.5.4 (#2207) * changes to make heron compile with bazel 0.5.1 * changes to compile with bazel 0.5.2 * added bazel 0.5.2 for travis * remove crosstool_top for travis with bazel 0.5.2 * remove crosstool_top for applatix with bazel 0.5.2 * added support for bazel 0.5.4 rc4 candidate * fixed warnings in pkg_tar and heron downloader * upgrade to new bazel 0.5.4 release * changed the path for heron-python-instance target * updated docs to use bazel 0.5.4 and updated heron version to be 0.15.1 30 August 2017, 18:30:46 UTC
c44c4f0 [ISSUE-2170] fix tracker crash when zk root topologies path does not … (#2260) * [ISSUE-2170] fix tracker crash when zk root topologies path does not exist. * Fix checksytle. 30 August 2017, 17:54:43 UTC
b094a62 fix the executor unit test (#2261) 30 August 2017, 07:31:46 UTC
d9176f0 healthmgr check null return value (#2258) * fix null exception * add verbose 29 August 2017, 23:36:14 UTC
9e74bf5 Support components name generated by SummingBird. (#2256) 29 August 2017, 17:32:10 UTC
9df0e59 Renamed Exactly Once to Effectively Once (#2254) * Renamed Exactly Once to Effectively Once * Substitute exactly once with effectively once * Replaced exactly once with effectively once 29 August 2017, 16:58:36 UTC
e931d49 add `restartcontainer` resolver to healthmgr (#2230) * add restartcontainer resolver * add physicalplanprovider * update healthmgr.yaml for local test * fix styple and compile * fix maosong comment * fix compile * GrowingWaitQueueDetector removed * fix ashvin comment * fix compile * change min to ms * add `final` * fix 1. `final` ,2. `empty list`, 3. `synchronized`. 4. `toString()->getName() * add `sync` * refactor PhysicalPlanProvider.get() 29 August 2017, 00:07:48 UTC
8b98ebd Add src id in exception message. (#2252) 28 August 2017, 20:07:09 UTC
15a4621 Move system_config to from common to instance since only instance uses it (#2251) * Move system_config to from common to instance since only instance uses it * Added system config to the init file 27 August 2017, 22:47:55 UTC
57daad4 Move EventLooper and GatewayLooper to instance since no one else uses them (#2250) * Move EventLooper and GatewayLooper to instance since no one else uses them * Fix path * Fix path. Attempt 2 27 August 2017, 21:15:39 UTC
af2a0c6 Moved system_constants from common to instance since only instance uses it (#2249) 27 August 2017, 18:54:38 UTC
4e4bbb1 Deleted unused file (#2248) 27 August 2017, 16:41:36 UTC
fed3aff Include only Topology proto in heron java api jar (#2247) 26 August 2017, 23:52:00 UTC
987975d Remove unused proto (#2246) 26 August 2017, 17:57:51 UTC
d816f44 Upgraded zookeeper to the latest stable version (#2245) 26 August 2017, 16:04:18 UTC
6610ca9 Make metrics cache client in healthMgr fetch location from statemgr (#2232) * Make metrics cache client in healthMgr fetch location from statemgr The metrics cache port is selected at runtime. So a user cannot provide it as a configuration. The cache client needs to fetch it. * Fix checkstyle errors 26 August 2017, 01:05:25 UTC
60f9fff Fix heronpy packaging (#2244) 26 August 2017, 01:02:56 UTC
0ad8e87 Remove unnecessary sub module BUILD files (#2240) * Remove unnecessary sub module BUILD files * More rejiggering of BUILD * More regiggering * Fixed unittest * Fixed the build 26 August 2017, 00:01:31 UTC
41aef74 Removing legacy kakfa connector (#2242) This is dead code that no longer compiles and is being deprecated 25 August 2017, 23:50:13 UTC
fd40efb Move api, dsl, tools, proto and connectors (#2229) * Move api, dsl, connectors and proto to heronpy * Separate heronpy/proto dependency 25 August 2017, 21:54:08 UTC
81c6957 Moved common/python/network to instance/python/network since no one else uses them (#2237) * Moved common/python/network to instance/python/network since no one else uses them * Fixed module path 25 August 2017, 06:28:00 UTC
5cd67de Fix bazel warning (#2239) 25 August 2017, 05:24:19 UTC
57ee7ec Move utils to heron instance (#2233) * Move utils to heron instance This change to remove heron-api dependency for heron-common * Move Pex loader back to common 24 August 2017, 20:30:19 UTC
1a97d45 Added an option to bazel build/test without doing checkstyle. (#2236) If you invoke bazel build heron/... or bazel test heron/..., it will skip checkstyle Also ensured that bazel options are in alphatetical order 24 August 2017, 17:21:13 UTC
8888b59 Convert kubernetes scheduler to use heron downloader. (#2223) * Convert kubernetes scheduler to use heron downloader. * Add comment for heron-conf directory configuration setting. 23 August 2017, 01:33:29 UTC
be1cb38 Make bazel configure python3 friendly (#2228) 22 August 2017, 19:35:50 UTC
716470b Limit size of a single data tuple. (#2224) * Limit size of a single data tuple. * Break line. 22 August 2017, 04:43:15 UTC
1d4e016 Fix yaml health manager config parsing error (#2226) Fixes Issue #2225 21 August 2017, 22:56:52 UTC
2604bc6 0.15.0 documentation release (#2221) * update versions in the website's config.yaml * begin adding brew install instructions to quick start guide * fix python doc issue * finish homebrew instructions * get heronpy version from website YAML config 21 August 2017, 22:55:36 UTC
5deeede Fix Join Bolt implementation (#2215) * Fix Join Bolt implementation * Added some comments to clarify the logic 21 August 2017, 20:55:52 UTC
723ef86 Ability to launch health manager on container-0 (#2203) * Fix long type parsing errors in yaml * Add health manager mode config * Fix checkstyle errors * Add Health Mgr changes to Aurora scheduler * Fix unit test failures 20 August 2017, 03:52:55 UTC
1f2003a Add a heron downloader to download and extract topology packages. (#2216) * Add a heron downloader to download and extract topology packages. * Return instead of calling exit. 20 August 2017, 00:19:39 UTC
96465b3 upgrade gflags and glog to newer versions (#2211) * upgrade gflags and glog * remove old gflags 19 August 2017, 19:27:55 UTC
d4a5cf5 remove inputs from streamlet since the core streamlet class does not use it (#2214) 18 August 2017, 22:17:37 UTC
8b8de79 Removed operation since it is unused (#2213) 18 August 2017, 20:49:44 UTC
02f979e All of dsl methods need to output the same stream. This pr (#2212) makes DslBoltBase as the refactored place where the output is defined and the others make use of it by deriving from it 18 August 2017, 19:30:29 UTC
fe1bbf9 Make the exception style compatible with both python2 and python3 (#2210) 18 August 2017, 05:03:51 UTC
f946fe1 Fixed libtool install command in ubuntu 16.04 (#2208) * Fixed libtool install command in ubuntu 16.04 * Fixed path in docker readme 18 August 2017, 04:52:42 UTC
21d65d0 Upgrade libevent to a more recent stable version (#2209) 17 August 2017, 21:46:51 UTC
f24444e Replace all localhost by 127.0.0.1 (#2204) * Replace all localhost by 127.0.0.1 * removed unneeded change * Fix unittest 17 August 2017, 00:29:11 UTC
e85b232 Documentation changes for 0.15.0 release (#2201) * update versions in the website's config.yaml * begin adding brew install instructions to quick start guide * fix python doc issue 16 August 2017, 00:14:04 UTC
00f81db replace hard coded stmgr port in stmgr_unittest (#1852) * add target to CountDownLatch wait() * check intermediate result * update countDown() * add timeout to CountDownLatch wait() * add test for CountDownLatch * fix mark's comment * change to bind 0 * rephrase comment and refactor d->duration * add debug log * add tuple debug print * remove duplicate common.stmgr_ports_.reserve(common.num_stmgrs_); * add debug print * register new port to tmaster * sync port for tmaster client in dummy stmgr * fix code style * remove print to save test time * fix looper location in the vector * recover dummy instance cpp * clean dummy stmgr * clean debug print in stmgr unit test * clean debug print in stmgr unit test again * fix mtmgr checking * add publisher port setter for metrics mgr client * update checking * update comment * fix bug after merge master * move setPort() to constructor * update comment and style * fix test * move FetchTMasterLocation() after StartStmgrServer() so that the tmaster_client_ will be set actual stmgr_port * print stmgr_port * add log print * update and try again friday * fix tmaster restart on new location test * remove redundant debug log * clean unnecessary change * turn on all tests * address sanjeev comment 15 August 2017, 23:12:34 UTC
fd966ed Fix requests version, fix default env. (#2202) 15 August 2017, 23:11:24 UTC
7a54db0 cleanup the dependencies for generating heron storm jar (#2191) * cleanup the dependencies for generating heron storm jar * added basics-java dependency to heron-storm jar 15 August 2017, 21:49:24 UTC
aa04322 Fix the stateful component overloaded functions in window bolt (#2199) 15 August 2017, 05:00:24 UTC
33d61fe Remove unused third party packages (#2198) 15 August 2017, 04:34:37 UTC
62b8214 Expose all API related classes in Python Packaging (#2197) * Move python packaging to be based on connectors instead of spouts * Removed debug messages * Fix the right version for HERON_VERSION 15 August 2017, 01:28:00 UTC
4f05db6 Update cpplint to the latest release. Minor fixes in the code along the way (#2196) * Update cpplint to the latest release. Minor fixes in the code along the way * Removed excessive printing 14 August 2017, 23:32:41 UTC
c77d3b4 Add health manager operation mode (#2181) * Add health manager operation mode * Add healthMgr to heron core package 14 August 2017, 21:15:49 UTC
ba816dd Fix log level and parameters (#2195) 14 August 2017, 21:09:24 UTC
b9cfd6e Move over from using iteritems to items since its more python3 friendly (#2194) * Move over from using iteritems to items since its more python3 friendly * One missed iteritems in cpplint 14 August 2017, 20:30:07 UTC
d8c9a3a Fix python examples now that spouts have moved to connectors (#2193) 14 August 2017, 19:48:50 UTC
0782fa0 Upgrade to gtest 1.8 (#2192) * Upgrade to gtest 1.8 and it includes gmock 1.8 Remove the old gmock 1.7 directory in third_party * include the googletest 1.8 zip 14 August 2017, 15:08:30 UTC
c7fa60b Follow pythonic idiom of _ instead of camelCase in DSL (#2190) * Follow pythonic idiom of _ instead of camelCase in DSL * Fix build 13 August 2017, 01:58:14 UTC
ff36602 Follow Python idiom of using _ instead of camelCase (#2189) * Follow Python idiom of using _ instead of camelCase * Fixed dsl to follow stateful changes 12 August 2017, 16:58:40 UTC
b5d3291 HeronTupleSet needs src task id (#2187) 12 August 2017, 04:07:03 UTC
9714095 Certain information should be printed only in debug mode (#2186) 12 August 2017, 03:00:52 UTC
bc3d6d2 Fix import name in python examples (#2185) 12 August 2017, 03:00:06 UTC
afde0ac Create a connectors directory for spouts and bolts (sources/sinks) (#2176) * Create a connectors directory for spouts and bolts (sources/sinks) * Add subdirectories for connectors. * Moved from spouts to connectors 12 August 2017, 02:25:56 UTC
3af8305 Verify cluster name matches when submitting through apiserver. (#2184) 12 August 2017, 01:47:26 UTC
bfcf50d Add topology resource unit tests. (#2179) 11 August 2017, 22:00:24 UTC
0c10b1f Remove spi dependency from simulator since TopologyUtils moved from spi (#2182) 11 August 2017, 21:00:51 UTC
7c585ad Moved TopologyUtils from spi to common (#2180) * Moved TopologyUtils from spi to common * Fixed unittest 11 August 2017, 17:20:30 UTC
e1ed629 Add base framework for api server. (#2094) * Add base framework for api server. * Fix build issues. * Remove extra newlines. * add config files as resources into api server jar * package heron api server into tools * Implement submit/kill/activate/deactivate for the rest api server. * Remove excess logging and print statements from SubmitterMain. * Add help flag and remove println and hardcoded directory. * Add apiserver tests for the ConfigUtils. * Add user and verion params for submit topology. * Add release file flag for apiserver. The api server will read the, build version, build user and build tag from the release file and use those values for submitting topologies. * Set the submit_user when submitting a topology. * adding URL based handling in heron submit * Fix activate/deactivate/kill api paths in cli. * support multiple deployment modes * Return status 200 for a successful kill topology. * Catch exceptions as opposed to runtime exceptions in topology resources. * Update topology actions to check for status code 200. kill/activate/deactivate endpoints will return a http status code of 200 when successful. * fix printing debug information * Add script to start the api server. * package heron api server * get the actual binary since heron-apiserver is a symbolic link * Add base-template flag and method to find where the jar is installed. * Print error message if there is a parsing exception. * Add error message when server fails to start. * Update bind exception message. * Add port flag for api server. * added the service-url parameter * - Remove HERON_VERSION from docker files - Add error message reporting for service mode deployment * remove debug print out * Fix how yaml files are written to files. * Pass extra parameters to update and restart extra parameters Added exception handling for requests call * Apply submit overrides and add version endpoint. * Add restart rest endpoint. * append a list element instead of extending it * Add printing of args in debug mode * Add rest update endpoint. * Add verbose flag to apiserver. * Allow update to accept overrides. * Add more scheduler implementations. * sort the version items and print it * Add dryrun response for submit and update endpoints. * fix python command line errors * Fix api server activate handler. * fixed submission to local cluster * fix spelling mistake of function invocation * Remove api depenedency from apiserver build file. 10 August 2017, 19:13:00 UTC
414969c Force destroy Marathon group on topology kill (#2168) * Move uploader init above PackingPlan creation to fix NPE in case of bad PackingPlan * Alwayse use force=true on marathon group deletes * Fix style issues 10 August 2017, 16:58:01 UTC
e62daed Heron Python Code Cleanup (#2173) * First cut for packaging changes * Made all instance changes adhere to the new way of referring things * Fixed errors * Added dsl and spouts in instance * Undid unneeded changes * Added changes to heronpy generation * Removed unneeded files * Fixed build * Fixed generation * Fix integration tests * Fix unittest * Fix unittest 10 August 2017, 14:56:59 UTC
f9299d7 Fix JavaDoc for heron api Config (#2172) 10 August 2017, 07:26:31 UTC
5bcd357 Update pulsar to apache client. (#2171) 10 August 2017, 06:19:21 UTC
6b90739 Let Python instance die if it is not able to handle stmgr messages (#2167) 09 August 2017, 07:49:55 UTC
70212a4 Added docstrings to python dsl (#2166) 09 August 2017, 06:43:01 UTC
d7a68dc Refactor DSL by trimming streamlet class and pushing more code into individual operators (#2165) 09 August 2017, 01:23:05 UTC
07a5b01 Add kubernetes stateful config. (#2164) * Add kubernetes stateful config. * Add stateful key to heron internals. * Indent stateful config. 08 August 2017, 23:41:19 UTC
80b55e6 added update certificates for java (#2163) 08 August 2017, 20:45:20 UTC
80275f3 Fix NPE when logging exceptions that have no message using SLF4J (#2161) * Move uploader init above PackingPlan creation to fix NPE in case of bad PackingPlan * Untested (but probably working) fix for NPE crash * Set to empty string 07 August 2017, 23:55:45 UTC
fc73aa4 Dhalion: Add implementation for resolvers and health policy (#2162) * First Heron-Dhalion commit to enable topology auto-health-management * Add dhalion dependencies * Build and deploy health manager library * Add utility class to pull metrics from tracker * Connect PackingPlan and Topology providers to the StateMgr * Minor refactor for readability * Refactor code to use latest dhalion api * Add health policy config yaml * Add Heron main class for HealthManager * Fix any issues running HealthManager as a process * Assume all health policy configs to be strings * Isolate policy config specific guice modules * Add event handlers and invoke resolver actions * Fix dhalion package name change related errros * Use JsonPath to parse tracker response * Add bazel dependencies for json path * Replace tracker's metrics call with timeline call * Fix checkstyle errors * Use the Dhalion project's repo as binary source * Update dependencies as per update heron-common * Add sensors for BackPressure and ExecuteCount * Add BufferSizeSensor and refactor sensors * Minor refactor for readability * Refactor sensor code to use latest dhalion api * Make BackPressureSensor configurable, add high-pass-filter * Fix checkstyle errors * Replace tracker's metrics call with timeline call * Added metric name to sensors * Fixed checkstyle errors * Mark HealthManager evolving, remove dead code, improve javadoc * Add sample config file for the HealthManager * Address review comment, use ConfigReader for yaml files * Fix checkstyle errors * Add detector for backpressure and basic diagnoser logic * Improve diagnoser accuracy using instance buffer sizes * Refactor code to use updated diagnoser api improve readability * Make BackPressure detector configurable and add tests * Fix names of symptoms and diagnosis instances * Add load disparity detector and refactor diagnosers * Add pending buffer size disparity anomaly detector * Add large pending buffers anomaly detector * Fix checkstyle errors * Replace tracker's metrics call with timeline call * Add GrowingWaitQDetector * Fixed checkstyle errors * Refactored skew detectors, changed naming, added appropriate stats * Updated diagnosers to take into account more metrics and added new tests * Fixed checkstyle errors * Add DynamicResourceAllocation Policy * Isolate policy config specific guice modules * Add ScaleUpResolver * Connect DynamicResourceAllocationPolicy to resolvers * Add event handlers and invoke resolver actions * Remove blocking sleep call after any resolver action * Reuse Dhalions HealthPolicy to reduce code duplication * Fix dhalion package name change related errros * Add large pending buffers anomaly detector to policy * Fix checkstyle errors * Replace tracker's metrics call with timeline call * Fixed checkstyle errors * Add metrics duration configuration for all sensors * Compute and provide buffer growth rate for all instances * Decouple HealthManager constructor and tracker Resolves a review comment * Remove interface for constants * Rename healthm so that it matches other components * Address review comments * Add dependency on full proto package instead of sub package * Refactor method consuming command line arguments * Avoid building health manager package Wait for the dhalion jar to be available on maven central before enabling health manager. Also fix healthmgr path in build scripts * Address review comments * Refactor code based on review comments in PR-1980 * Enable health manager build since dhalion is in maven now * Address review comments PR 2101 * Refactor after merging reviewed code * Fix checkstyle errors * Re-enable a detector test * Fix errors introduced by merging detector branch * Fix errors and refactor after merging master branch 07 August 2017, 22:53:37 UTC
4165cdf Added Pulsar Spout and Pulsar Streamlet (#2160) * Addded PulsarStreamlet for composing pulsar sourced python dsl * More changes to BUILD * Removed pulsar dep for now 06 August 2017, 16:32:45 UTC
5a3a8d7 Added exactly once semantics for python dsl (#2159) * Added exactly once semantics for python dsl * Fixed stateful stuff in window bolt 05 August 2017, 19:27:58 UTC
f001b7d Don't print topology context since it just prints a class pointer info (#2157) * Don't print topology context since it just prints a class pointer info * Fixed build 05 August 2017, 06:21:26 UTC
a6023c9 Misc Python fixes (#2158) * Misc Python fixes * Fix unittest 05 August 2017, 06:21:07 UTC
b015bc9 If an instance is not connected, stmgr server should return false while sending restore message (#2156) 05 August 2017, 03:57:12 UTC
7b6228c Sort the task ids to ensure that identical fields grouping tuples (#2155) end up in same task ids even accross runs 04 August 2017, 22:29:26 UTC
9b0e408 Make sure that metric registration and adding tasks are done once. (#2152) * Refactor spout/bolt code and make sure that metric registration and adding tasks are done once * Changed inited to initialized 04 August 2017, 19:12:45 UTC
back to top