https://github.com/twitter/heron

sort by:
Revision Author Date Message Commit Date
86c64eb fix unit tests 02 August 2017, 07:00:58 UTC
37c63e5 fix trailing space 02 August 2017, 06:04:55 UTC
ababc47 topology submitting user is captured in the cli rather than in submitter main 02 August 2017, 05:26:27 UTC
e055ae4 update sendOutState (#2131) 01 August 2017, 01:47:46 UTC
d764d79 While unzipping pex files, don't overwrite existing files. (#2129) * While unzipping pex files, don't overwrite existing files. This is useful while running multi container topologies in local mode where multiple executors are unzipping the same file * Clean the working dir * Fixed unittest 31 July 2017, 23:00:02 UTC
22cd815 Use Java ver 8 as source and target version (#2130) * Use Java ver 8 as source and target version Signed-off-by: Ali Ahmed <a.ahmed@streaml.io> * Use Java ver 8 as source and target version Signed-off-by: Ali Ahmed <a.ahmed@streaml.io> * Use Java ver 8 as source and target version Signed-off-by: Ali Ahmed <a.ahmed@streaml.io> 31 July 2017, 22:37:32 UTC
18d6cab Add Dhalion-sensor implementation for heron (#2101) * First Heron-Dhalion commit to enable topology auto-health-management * Add dhalion dependencies * Build and deploy health manager library * Add utility class to pull metrics from tracker * Connect PackingPlan and Topology providers to the StateMgr * Minor refactor for readability * Refactor code to use latest dhalion api * Add health policy config yaml * Add Heron main class for HealthManager * Fix any issues running HealthManager as a process * Assume all health policy configs to be strings * Isolate policy config specific guice modules * Add event handlers and invoke resolver actions * Fix dhalion package name change related errros * Use JsonPath to parse tracker response * Add bazel dependencies for json path * Replace tracker's metrics call with timeline call * Fix checkstyle errors * Use the Dhalion project's repo as binary source * Update dependencies as per update heron-common * Add sensors for BackPressure and ExecuteCount * Add BufferSizeSensor and refactor sensors * Minor refactor for readability * Refactor sensor code to use latest dhalion api * Make BackPressureSensor configurable, add high-pass-filter * Fix checkstyle errors * Replace tracker's metrics call with timeline call * Added metric name to sensors * Fixed checkstyle errors * Mark HealthManager evolving, remove dead code, improve javadoc * Add sample config file for the HealthManager * Address review comment, use ConfigReader for yaml files * Fix checkstyle errors * Add metrics duration configuration for all sensors * Decouple HealthManager constructor and tracker Resolves a review comment * Remove interface for constants * Rename healthm so that it matches other components * Address review comments * Add dependency on full proto package instead of sub package * Refactor method consuming command line arguments * Avoid building health manager package Wait for the dhalion jar to be available on maven central before enabling health manager. Also fix healthmgr path in build scripts * Address review comments * Refactor code based on review comments in PR-1980 * Enable health manager build since dhalion is in maven now * Address review comments PR 2101 31 July 2017, 22:19:11 UTC
94556d7 limit stmgr client reconnect attempts (#2125) 31 July 2017, 18:31:42 UTC
f8eff84 Remove duplicate variable (#2127) 31 July 2017, 14:45:29 UTC
c94b0bf Print Command details only in verbose mode (#2126) * Print Command details only in verbose mode * Added more log statements under debug 30 July 2017, 05:45:03 UTC
eeb6285 Make instance send out state (#2120) 28 July 2017, 23:02:25 UTC
1c860dd register stateful processing messages with StreamManagerClient (#2119) 28 July 2017, 23:02:07 UTC
df0dd74 organize the docker directory (#2122) 28 July 2017, 18:02:11 UTC
d0de5d0 Added a new socket option to take the maximum packet size accepted (#2118) * Added a new socket option to take the maximum packet size accepted * Fixed unittest * Fixed var name 28 July 2017, 05:57:40 UTC
1fd6d7f add stateful support into Slave (#2106) 28 July 2017, 00:53:53 UTC
9724d44 Rename TupleStreamMessage2 to TupleStreamMessage for better comprehension (#2117) 27 July 2017, 18:53:00 UTC
f85b7a1 Remove TupleMessage2 since no one is using it (#2116) 27 July 2017, 17:37:23 UTC
b39b0fb Removed TupleStream proto structure since no one uses it (#2115) 27 July 2017, 16:21:44 UTC
e49c796 Removed TupleMessage since its obsolete (#2114) * Removed TupleMessage since its obsolete * Fix unit test 27 July 2017, 07:10:09 UTC
caf163f StreamManager never deals with TupleMessage (#2110) 27 July 2017, 00:21:43 UTC
6e16ce6 Removed TupleMessage usage in python instance because stmgr does not supports it (#2111) 27 July 2017, 00:21:27 UTC
412827b Fix previous AbstractTupleRoutingTest refactor (#2112) 26 July 2017, 19:14:29 UTC
7e389ed Removed dummy since no one uses it (#2108) 26 July 2017, 11:39:59 UTC
990b481 Renamed controller to tmaster_controller (#2107) * Renamed controller to tmaster_controller to avoid confusion with stateful_controller * Changed NULL to nullptr 26 July 2017, 06:41:29 UTC
f8f6a77 Introduce StatefulController in TMaster and handle all stateful messages (#2105) * Introduce StatefulController in TMaster and handle all stateful related messages * Address pr feedback 26 July 2017, 05:35:25 UTC
f558fae update queue with type Message (#2103) 25 July 2017, 22:37:14 UTC
94aebd8 Added ability in tmaster to accept stateful processing messages (#2102) from stmgr. 25 July 2017, 19:42:06 UTC
555d346 For stateful topologies, TMaster should connect with checkpoint manager (#2100) * For stateful topologies, TMaster should connect with checkpoint manager * Fixed executor unittest 25 July 2017, 18:15:04 UTC
936cebf packet size check (#2098) * add packet size check * add comment * re-use function 25 July 2017, 02:49:08 UTC
a9f68ea Dhalion health manager (#1980) * First Heron-Dhalion commit to enable topology auto-health-management * Add dhalion dependencies * Build and deploy health manager library * Add utility class to pull metrics from tracker * Connect PackingPlan and Topology providers to the StateMgr * Minor refactor for readability * Refactor code to use latest dhalion api * Add health policy config yaml * Add Heron main class for HealthManager * Fix any issues running HealthManager as a process * Assume all health policy configs to be strings * Isolate policy config specific guice modules * Add event handlers and invoke resolver actions * Fix dhalion package name change related errros * Use JsonPath to parse tracker response * Add bazel dependencies for json path * Replace tracker's metrics call with timeline call * Fix checkstyle errors * Use the Dhalion project's repo as binary source * Update dependencies as per update heron-common * Mark HealthManager evolving, remove dead code, improve javadoc * Add sample config file for the HealthManager * Address review comment, use ConfigReader for yaml files * Fix checkstyle errors * Decouple HealthManager constructor and tracker Resolves a review comment * Remove interface for constants * Rename healthm so that it matches other components * Address review comments * Add dependency on full proto package instead of sub package * Refactor method consuming command line arguments * Avoid building health manager package Wait for the dhalion jar to be available on maven central before enabling health manager. Also fix healthmgr path in build scripts * Address review comments 25 July 2017, 02:45:44 UTC
8a4336e add stateful controller into tmaster dir (#1973) * add stateful helper into tmaster * add unit test for stateful restorer * update * update * address comments * update metric name 25 July 2017, 01:26:08 UTC
223e4fa add clean stateful checkpoint handler into tcontroller (#2096) * add clean stateful checkpoint handler into tcontroller * address comments 24 July 2017, 23:42:51 UTC
f9bddff Introduce stateful restorer in stmgr. (#2091) * Introduce stateful restorer in stmgr. This allows stmgr to handle all checkpoint save/restore as well as exactly once messages * Added metrics to keep track of bytes/tuples discarded during restore 24 July 2017, 20:43:48 UTC
eefcd08 New API for topology reliability modes (#2089) * Added topology.reliability.mode to specify what reliability the user wants to run the topology in. This will deprecate the topology.acking, topology.stateful.enabled and topology.stateful.exactonce settings, thereby providing user with a single switch to control the topology behaviour * Fixed build * Made the value of topology.reliability.mode to be mirroring the names * Fixed integration build * Fixed integration test * Brought back setEnableAcking interface back to keep compat with Heron native topologies inside Twitter * Fixed old values of reliability mode * Marked deprecated fields/methods with annotations * Removed STATEFUL mode since there is still lack of clarity on what that means * Added comments that the suppresswarnings will go away once TOPOLOGY_ENABLE_ACKING goes away 21 July 2017, 01:51:48 UTC
e394660 Correct heap memory usage of stream manager (#2086) 20 July 2017, 17:43:31 UTC
a2ab64f Remove website build from Travis CI build (#2088) 20 July 2017, 00:43:21 UTC
c1facd5 Remove unused function (#2081) * Remove unused function * Deleted the whole file because no one uses it 19 July 2017, 22:43:55 UTC
2a1cf77 Move uploader init above PackingPlan creation to fix NPE in case of bad PackingPlan (#2083) 19 July 2017, 21:34:14 UTC
68d8d0e comment instead of removing (#2085) * try 1 * comment instead of removing it 19 July 2017, 21:23:48 UTC
4acd367 try 1 (#2084) 19 July 2017, 20:02:44 UTC
8fc4455 Limit number of messages in mempool, instead of total byte sizes (#2073) 19 July 2017, 05:55:32 UTC
f537a31 Do not log exception when OOM. (#2080) 19 July 2017, 05:44:22 UTC
b63478b Add Pulsar spout. (#2069) * Add Pulsar spout. * Reset pulsar metrics when getValueAndReset() is called. * Create a default client and consumer configuration. The default consumer configuration uses a shared subscription with a 60 second ack timeout so messages are replayed. 19 July 2017, 04:24:15 UTC
966a1d6 Instantiate ClientMgr in constructor (#2078) 18 July 2017, 20:25:28 UTC
6837b8e Instantiate checkpoint manager client in stmgr (#2076) * Instantiate checkpoint manager in stmgr * Fixed build * Fixed unittest 18 July 2017, 18:12:47 UTC
db60686 Added Stateful processing related message handling in TMasterClient (#2075) 18 July 2017, 05:10:14 UTC
a663d64 fix kafka spout HeapByteBuffer not serializable problem (#2063) 17 July 2017, 23:16:55 UTC
120cb2d Added stateful processing restore state machine to stmgr. (#2070) * Added stateful processing restore state machine to stmgr. * Added unittest for stateful restorer * Added config file as arg to the test * Added case to test dead instances while in restore * Added test case to check for ckptmgr restart * Added more comments to classes and methods 17 July 2017, 22:19:21 UTC
98e9c69 packaging python with all the modules (#2072) * python packaging * remove tree utility that shows the contents of the python package 17 July 2017, 16:18:47 UTC
20eb2b6 fix the comparison to be string rather than numerical (#2071) 15 July 2017, 01:04:39 UTC
0ec2490 change ci-docker.sh to use sh instead of bash (#2068) 14 July 2017, 22:26:39 UTC
7a1af1e Added stateful processing message handlers in stmgr-server (#2067) * Added stateful processing message handlers in stmgr-server * Made changes based on the feedback 14 July 2017, 22:13:53 UTC
32cf16d Remove API dep on c.t.h.common.basics.TypeUtils (#2061) 14 July 2017, 18:24:25 UTC
5680770 Add hooks in stmgr client/clientmgr to send checkpoint markers. (#2060) Also add hook into stmgr to get notifications about dead client connections 14 July 2017, 17:05:13 UTC
856d9d2 added a script to build/publish docker images (#2065) 14 July 2017, 06:26:36 UTC
b4659fc Added ability in tuple cache to take in checkpoint messages to be sent to downstream tasks (#2058) * Added ability in tuple cache to take in checkpoint messages to be sent to downstream tasks * Added unittest to test the checkpoint message handling in tuple cache * Added more comments detailing the diff between clear and drain 13 July 2017, 20:10:06 UTC
e0f2d28 Mempool acquire should always return a Clean protobuf (#2057) 13 July 2017, 01:56:14 UTC
bff8367 Adding debugging to integration test spout, increasing timeouts for get condition (#2055) * Adding debugging to integration test spout, increasing timeouts for get condition * reverting the move of tuplesToAck 13 July 2017, 00:02:09 UTC
631471b Added Checkpoint Gateway Layer while sending data to instances (#2047) * added applatix scripts (#2) * change the docker build image to use the latest * remove applatix scripts * Added Checkpoint Gateway Layer while sending data to instances * Clear the protobuf before using it * Address feedback 13 July 2017, 00:00:55 UTC
430b15f Use Tier. (#2056) 12 July 2017, 21:57:09 UTC
7103a30 Shell refactor (#2054) 12 July 2017, 21:33:16 UTC
b7954f6 Increase timeout for integration test result checking (#2053) 11 July 2017, 23:03:08 UTC
a31659b retain heron core tar.gz file (#2049) 11 July 2017, 19:21:52 UTC
57d1aec Add countdown latch to runnable in testRegisterTimerEventInNanoSeconds (#2050) 11 July 2017, 17:34:10 UTC
e44197b Added Spout for Apache Pulsar (#2048) * added applatix scripts (#2) * change the docker build image to use the latest * remove applatix scripts * Added Spout for Apache Pulsar 11 July 2017, 01:12:11 UTC
2559540 Created heron/spouts to house all well-known spouts. (#2046) * added applatix scripts (#2) * change the docker build image to use the latest * Created heron/spouts as the location to house all well known spouts Moved TextFile/FixedLines spouts/streamlets to seed it * Revert "Created heron/spouts as the location to house all well known spouts" This reverts commit 891523b63d67bcceb5e88d2d3658253d2fb50f38. * Created heron/spouts as the location to house all well known spouts Moved TextFile/FixedLines spouts/streamlets to seed it * remove applatix scripts 10 July 2017, 23:13:12 UTC
3b6b729 added applatix scripts and tools (#2045) 10 July 2017, 21:19:28 UTC
2a7d53f remove the hard coded path in dsl (#2044) 10 July 2017, 05:26:11 UTC
365cef6 package python dsl package (#2043) 10 July 2017, 03:33:17 UTC
9dddbd8 move scripts and tools around (#2041) 08 July 2017, 23:43:37 UTC
4d4b173 Python DSL (#2033) * Added first cut of python dsl * Added first cut of text input * Made everything a Streamlet * Added BUILD file and did the first set of fixes * Added MapBolt * Added FlatMapBolt * Added FilterBolt and FlatMapBolt * Added partial sample bolt * Added JoinBolt and JoinGrouping * Added RepartitionBolt * Made everything TIMEWINDOW * Join now support sliding interval * Added ReduceByWindow Bolt * Switched everything to reducebykeyandwindow bolt * Fixed all compilation issues * Added StaticLines Streamlet * Merged spouts inside the streamlet files * Changed function name * Added a dsl example * reduce takes timewindow * Fixed the format of Stream constructor * Fixed bugs while composing dag * Fixed more bugs * Fixed the way to specify custom grouping * Fixed generation of stage names * Specify classpath rather than instances * Fixed the Reduce/Join Custom grouping * Fixed logic for join/reduce tuple * Fixed the format of output tupled * Addressed review comments 08 July 2017, 21:21:38 UTC
afbcfab Extract HeronTuple from current_tuples while acking it (#2032) * Current tuples is a tuple of (HeronTuple, time). While acking we need to extract the HeronTuple * Do not reuse tm * Ignore itime 07 July 2017, 08:44:48 UTC
e032dee Applatix workflow with build followed by test (#2015) * applatix parallel builds * try test build followed by test workflow * change the container to applatix container * add the actual test * use artifacts for test containers * map the output to input * fix applatix errors * debugging applatix * update the path to be applatix * due to unicode from slack typed the entire command * moved common.sh to only dir above and change where it is being used * bind the user variable * move save-logs.py one level above * remove tar, zip files to make artifact transfer faster. Include tests * export scripts and its subdirectories * generate test package * include the path in test.sh * cleanup the travis and applatix scripts * fix spelling mistake * parallelize language tests * fix applatix yaml to invoke the appropriate test script * remove starting of http server twice in tests 06 July 2017, 22:23:10 UTC
adfea16 CustomGrouping can be either a CustomGrouping instance or a class path of that instance (#2031) 06 July 2017, 18:07:40 UTC
0ccb2c5 Made cloudpickle the default serializer (#2030) * Added cloudpickler for better serialization * remove style check warnings * orrected name * Removed pyspark stuff 05 July 2017, 14:36:39 UTC
a9bdab8 Window tuples should take care of handling acks (#2028) 04 July 2017, 03:06:40 UTC
88a73c9 Changed the custom grouping to take in an actual instance of (#2027) ICustomGrouping than a class path. 02 July 2017, 13:37:34 UTC
ed2451a Check if any unshaded third party libraries are introduced into `heron-instance.jar` (#2023) 30 June 2017, 21:31:02 UTC
4732947 Added WindowBolt interface in python (#2021) * Added window bolt to python api * Added a sample topology for WindowBolt * Added Tumbling Window Bolt * Changed WindowInfo to WindowContext * Changed copyright 30 June 2017, 20:33:51 UTC
1511845 Added Checkpoint Gateway (#2024) * Added Checkpoint Gateway, a class that buffers tuples until upstream markers have all been received. * Took Neng's feedback into consideration 30 June 2017, 19:52:06 UTC
af932a0 add security check for shell-kill-executor (#2013) * add security check for shell-kill-executor * remove unused code * update log * add space in test 30 June 2017, 19:47:47 UTC
6fefa59 set init value getTotalMemoryUsed (#2026) 30 June 2017, 19:40:10 UTC
e819e17 flatten the directory structure for integration tests (#2018) * flatten the directory structure for integration tests * flatten the directory structure for python integration test topologies * changing path names using integration-test to integration_test and also ensure module imports are correct * flatten the python integration tests and corresponding changes * package tests into tarballs * package tests to self extractable binary * Change travis script to use packaged integration tests * after installation, we should invoke test-runner instead of test-runner.pex 30 June 2017, 18:30:49 UTC
043548e Updated Python documentation (#2017) * modified CSS and python docs content * modify doc gen script to use PyPI pyheron package instead of local sources * fix issue in custom grouping topology BUILD file * fix pydoc gen script once again * fix scrolling offset issue * some additions to the python docstrings * more re-org for the python topologies doc * update link to bazel dev environment * remove applatix dir from this branch * merge applatix material with master * undelete mistakenly removed kubernetes doc * undo mistaken changes to graphite doc * merge applatix scripts from master * undo more extraneous changes * remove errant parens in config.yaml * remove unnecessary ls from python doc gen script * remove references to streamparse * change upgrade language to migrate * change booleans to uppercase for python * finish descriptions for example topologies * s/pyheron/heronpy * replace raw version numbers with shortcodes * fix broken shortcode reference * small commit to re-run CI * fix python lint issues * change website build script * add pdoc install to setup script * upgrade to new version of hugo * remove website generation from CI * fix hugo installation command in .travis.yml * another adjustment to hugo install URL * remove references to streamparse in python docs 30 June 2017, 01:41:59 UTC
a2967c8 add libtool-bin in doc compiling on linux (#2025) * add libtool-bin * add setuptools 30 June 2017, 00:11:55 UTC
bd7bb2d add shard number to log name (#2022) * add shard number to log name * fix unit test * update func name 29 June 2017, 22:18:49 UTC
9ae5739 clean common dependency (#2019) 29 June 2017, 21:07:55 UTC
673ed2c limit the mem pool size (#2011) 29 June 2017, 17:45:34 UTC
089cff0 Added Neighbour Calculator, a utility class to calculate (#2016) * Added Neighbour Calculator, a utility class to calculate upstream and downstream components on a per task basis * neighbour calculator unittest is not flaky * Moved neighbour calculator from manager to util 29 June 2017, 03:38:39 UTC
1687703 Expose TopologyContext in heronpy (#2014) 28 June 2017, 06:46:30 UTC
2bf4626 Moved Herons pypi to heronpy. Updated requirements as well (#2012) 28 June 2017, 05:33:54 UTC
5d3f0fa add kill-executor handler (#1997) * add kill executor handler * new line at the end * remove unused import * remove unused import 28 June 2017, 04:33:58 UTC
12a79ec add pmap to shell (#1995) * add pmap to shell * add PmapHandler to init 28 June 2017, 00:36:35 UTC
c4ba267 Remove pyheron since now python is first class api in Heron (#2006) 27 June 2017, 22:28:29 UTC
4fb6622 Fix applatix scripts (#2010) * fix the docker images for source code checkout * fixed the applatix scripts to pass the ci environ 27 June 2017, 20:46:43 UTC
932331f fix the docker images for source code checkout (#2009) 27 June 2017, 20:39:31 UTC
5f39435 changes for applatix ci (#2008) 27 June 2017, 20:33:48 UTC
330f33a rename tools/travis-ci to tools/travis and update the references to it (#2007) 27 June 2017, 20:33:08 UTC
05d3bc1 Restart subprocess with shell_env. (#2004) 27 June 2017, 20:01:04 UTC
77035b4 changed travis scripts to take an input arg that indicates the CI env (#2003) 27 June 2017, 19:13:32 UTC
back to top