https://github.com/twitter/heron

sort by:
Revision Author Date Message Commit Date
b6dddd3 Fix Java checkstyle Remove trailing spaces to fix the java checkstyle. 10 February 2018, 01:07:13 UTC
f554b79 fix null in printout during submission (#2722) * fix null warning * revert function sig * fix unit 08 February 2018, 06:28:45 UTC
7cab3f3 use actual path (#2725) 08 February 2018, 02:46:54 UTC
81a75f4 Simplify website setup (#2721) * remove nodejs install from macos setup * update README 07 February 2018, 20:07:48 UTC
75c25a0 deprecate RRPacking in default config (#2717) * deprecate RRPacking in default config * update to the same alg * fix typo 07 February 2018, 17:41:21 UTC
6cba81f adding some confs for heron nomad and standalone (#2716) * adding some confs for heron nomad and standalone * fxing nomad client.yaml 07 February 2018, 15:26:33 UTC
ad3ecfc fix a RRrepacking bug (#2720) * fix a bug * update unit 07 February 2018, 02:46:49 UTC
a8a4a3c mitigate prompt (#2719) 07 February 2018, 00:30:34 UTC
ca7d5ca [Streamlet Scala API] Add FunSuite Support (#2695) * Add FunSuite Support to Streamlet Scala API * Remove unused spec2 rule * Review comments are addressed. 06 February 2018, 22:49:00 UTC
8f36a93 Move stmgr quitting logic from stmgr client to tmaster client (#2711) * Move stmgr quitting logic from stmgr client to tmaster client We found out that when a bad stmgr is experiencing connection issue to all other stmgrs, it takes the stmgr a lot longer to reach the per stmgr client max attempt and quite becaus bad connections cause extra delays. In our case, instead of quitting after 5 minutes (1s reconnect timeout, 300 max attempts), the bad stmgr quitted after more than 2 hours. At the same time, all other good stmgrs were quitting every 5 minutes. Here is a more detailed document about the issue: https://docs.google.com/document/d/1WHNc2NEp2gVL9ge2QVKp9t4Hpd4U9sAbzBqCu4-iDUM/edit# 06 February 2018, 05:55:34 UTC
9ffeb39 fixes the helm chart, config and api server memory restriction for k8s (#2714) 06 February 2018, 05:35:01 UTC
c62ec91 Expose the component config through the tracker. (#2709) 06 February 2018, 03:33:23 UTC
bc5236e Add Streamlet Trait Support (#2696) 05 February 2018, 19:39:14 UTC
6ec53b3 HttpUploader is being refactored. (#2713) 05 February 2018, 18:10:47 UTC
391afe4 [metricscache] align metricscache toggle switch (#2707) * algin metricscache and dhalion toggle switch * add comment 03 February 2018, 23:50:24 UTC
c530424 [metricscache] reduce exception stack trace to first 2 lines (#2706) * reduce metricscache exception stack trace to first 2 lines * fix style 03 February 2018, 23:50:09 UTC
b72e106 [metricscache] make default log level to INFO (#2705) * make default log level to info for metricscache * fix unit test * change info to fine for each received metrics 03 February 2018, 23:49:52 UTC
8bdf82f Fix sandbox script (#2710) 03 February 2018, 03:45:37 UTC
6d07270 modify language in sandbox.sh script (#2708) 02 February 2018, 00:44:35 UTC
8f5a1c8 added sandbox script to deploy heron in a single docker (#2704) * added sandbox script to deploy heron in a single docker * fix fonts with bold faces * add readme file for sandbox * added an example session * fix README docker requirements 01 February 2018, 22:32:40 UTC
c6ec8a7 Fix src and java doc generation for heron-api (#2702) * Fix src and java doc generation for heron-api * Update Name 01 February 2018, 19:24:41 UTC
6634b8f Necessary software for single docker image (#2701) * added executables for single docker image * make a single docker image 31 January 2018, 23:12:58 UTC
1a831ec make heron core distribution - symbolic linkable (#2684) * make heron core linkable * use a global config whether to use core URI or core directory * fix nomad scheduler test * fix nomad scheduler tests * fix the AuroraSchedulerTest 30 January 2018, 18:39:51 UTC
debcfe6 depend on java protobuf target from protobuf sources (#2640) * depend on java protobuf target from protobuf sources * fix to use protobuf the version in WORKSPACE * skip the checkstyle checking for external directories used by bazel 29 January 2018, 22:17:32 UTC
3e7fc59 Clean up simulator/utils (#2674) * Add support for custom evictor and trigger policies Add support in both the streamlet and topology apis for the usage of custom eviction policies extending com.twitter.heron.api.windowing.EvictionPolicy and trigger policies extending com.twitter.heron.api.windowing.TriggerPolicy to enable user-defined windowing schemes. * Fixed Checkstyle Issues Fixed Checkstyle Issues * Fixed extra line errors, reorganized if statement in WIndowedBoltExecutor to require both Policies to be provided if one of them is * Reversed policy order to fix null pointer exception * Trial push (local compilation not working for some reason, going to push then pull clean copy) * Added Changes necessary to allow custom trigger policies to attach WindowManager, Config object, TriggerHandler and EvictionPolicy * Fixed test errors due to lack of setting necessary paramters and defining "started" again for CountTriggerPolicy * Remove flags and checks on AbstractBaseTriggerPolicy * Clean up simulator/utils This is the first commit i'll be making to the simulator package, as I attempt to add stateful processing to the simulator pursuant to issue #2346. This cleanup first removes a couple of unused classes from simulator/instance. It also removes some odd programming paradigms which were used for the simulator utils, like a pseudo-static class that acts on an object and objects derived from it (PhysicalPlanUtil), a class that can only be instantiated by a static method which creates maps of itself (StreamConsumers), a class with an entirely unnecessary private constructor (XORManager) and general passing of directly derived objects instead of encapsulation. I've merged the PhysicalPlanUtil and StreamConsumers functionality into TopologyManager, a class which manages additional functionality for a single topology and its directly derived objects. * Addressing comments * Added back runnable. Should be push-ready * Actually used the runnable 29 January 2018, 06:04:05 UTC
f42236d [Streamlet Scala API] Add initial Build Files (#2688) * Heron Scala Support Build Files * Indentation is fixed. 26 January 2018, 03:08:43 UTC
10b6238 Add tolerations to tools and apiserver deploymens for kubernetes. (#2687) 25 January 2018, 22:00:00 UTC
8f0b5df expose heron tracker as a service (#2686) 25 January 2018, 17:32:06 UTC
e4ea17f no need to check the existence of env_exec.sh in scripts/setup-intellij.sh (#2683) 25 January 2018, 06:14:04 UTC
8874b23 Add Download hostname override config (#2680) * Add Download hostname override config * Fix Checkstyle errors * Move to regular command line flag * Refactor to method * Remove static var manipulation * Fix whitespace 24 January 2018, 01:51:49 UTC
90a615d fix mismatch container id between packing alg and aurora (#2582) * fix packing container id and aurroa container id mismatch * fix import * fix import * fix import 2 * add import * add import 3 * fix unit tests * fix style * fix style 2 * fix style 3 * add fix for aurora json parser * debug * update debug * add debug * fix optional ornul * add debug * add debug print * update container diff calc * fix stderr * fix update packing plan proto * clean code * clean shell util * address comment and fix unit test * addresscomments * fix unit test * add import * update spi according to ashvin suggestion * fix style * fix style 2 * fix ning comment * fix ning comment 23 January 2018, 20:07:28 UTC
5e47399 add section on setting the service URL for a standalone cluster (#2682) 23 January 2018, 19:55:20 UTC
fe51f4a Heron standalone docs update (WIP) (#2676) * add instructions on stopping the cluster * clarify EDITOR setting * document standalone 'get' command 23 January 2018, 19:09:05 UTC
729e393 JSON output for heron-admin standalone cluster info command (#2675) * add json pretty print for cluster info * fix output falsely pointing to apiserver * fix get service-url command 23 January 2018, 19:07:24 UTC
c0ec563 Don't drop tuples during backpressure in DrainTupleStream (cross stream manager traffic) (#2679) 23 January 2018, 17:42:38 UTC
d1f72fe Remove cmake requirement in bazel configure (#2661) 23 January 2018, 00:51:07 UTC
89fdb37 Add k8s pod tolerations for topologies. (#2677) 23 January 2018, 00:20:02 UTC
c88e815 Heron standalone mode documentation (WIP) (#2614) * add new doc and add doc to TOC * add some example commands * more standalone instructions * change syntax of inventory.yaml * finish draft of doc 22 January 2018, 19:44:04 UTC
e2de6a4 move scripts/docker to scripts/images (#2672) * build docker images in build-artifacts.sh * rename scripts/docker to scripts/images for building docker images 20 January 2018, 18:59:58 UTC
a78ef88 Kubernetes installation with Helm docs (#2606) * add new doc plus modify TOC * begin adding instructions for different platforms * add more specific minikube startup instructions * more small changes * update instructions for changed heron charts URL * add new GKE instructions for resources * fix helm charts repo URL * add links to GCE machine type docs * add gke-small resource config to heron repo * finishing touches on helm doc * remove reference to SSDs on GKE machines * add section on RBAC permissions * change machine type for dev cluster 19 January 2018, 21:23:41 UTC
15af8a5 Use the minimal ram size for components instead of the last one (#2659) This is needed because in packing algorithms there is no guarantee that the ram size for each instance works across whole topology. 19 January 2018, 17:54:38 UTC
bec0da9 Fix for running topologies with DC/OS 1.10 or Marathon 1.5 (#2670) 18 January 2018, 21:12:03 UTC
da26e85 moved eco to the top level (#2669) * moved eco to the top level and packaged to be shipped with storm compatibility layer * move eco yaml files into the eco-examples directory * moved eco example yaml files 18 January 2018, 21:09:56 UTC
2d86042 Add support for custom evictor and trigger policies (#2653) * Add support for custom evictor and trigger policies Add support in both the streamlet and topology apis for the usage of custom eviction policies extending com.twitter.heron.api.windowing.EvictionPolicy and trigger policies extending com.twitter.heron.api.windowing.TriggerPolicy to enable user-defined windowing schemes. * Fixed Checkstyle Issues Fixed Checkstyle Issues * Fixed extra line errors, reorganized if statement in WIndowedBoltExecutor to require both Policies to be provided if one of them is * Reversed policy order to fix null pointer exception * Trial push (local compilation not working for some reason, going to push then pull clean copy) * Added Changes necessary to allow custom trigger policies to attach WindowManager, Config object, TriggerHandler and EvictionPolicy * Fixed test errors due to lack of setting necessary paramters and defining "started" again for CountTriggerPolicy * Remove flags and checks on AbstractBaseTriggerPolicy 18 January 2018, 05:39:34 UTC
850461c modified the helm chart (#2668) 18 January 2018, 00:12:04 UTC
a02d4cb Update k8s stateful sets termination and pod management. (#2666) - use parallel for pod management - use 0 seconds for a terminmation grace period for pods 17 January 2018, 19:26:34 UTC
dbad3de Add no checkstyle build targets (#2660) 16 January 2018, 19:14:24 UTC
1ac7341 update README.md [https://github.com/twitter/heron/issues/2664] (#2665) * update README.md [https://github.com/twitter/heron/issues/2664] * update README.md further [https://github.com/twitter/heron/issues/2664] 16 January 2018, 08:38:37 UTC
e5c96b0 RoundRobinPacking implements IRepacking (#2600) * from maosong diff * revert format * refine code * format code * refactor func name * address comments * add unit test * fix style and import * fix unit test * add parallelism postive check * fix style * adopt @ning comment * revert param check to pass unit test 09 January 2018, 23:56:38 UTC
400caa4 Joshfischer/eco config (#2658) * initial addition of directory structure * clean up * adding base config for build file \ fixing linting issues * saving progress. Still need to fix missing class definition on ParseException * fixing shaded dependency issue * starting experimentation on how to approach loading config from yaml file * experimenting with snake yaml to extract topology defintion * adding start of topology and component definitions * refactor * successfully populating ecoTopologyDefinition from yaml * clean up * clean up * fixed versioning error * saving progress, still working through building topology from yaml * saving progress * cleaning up styles * checking default config values * adding test for customized config map * transitioning to support heron api * removing last of streamlet api from eco * making a pivot to match storm flux behavior better * saving before chaos * saving progress, running into issue with verifying the toplogy * adding some logging for debugging * printing topology definition. Adding tests * saving with failing test, still working through investigation * back to working * saving progress * finishing eco parser test * handling exception * package change * package change * adding missed id * MVP of ECO at this commit * fixing styles, pulling topology name from topology definition file * fixing spelling * clean up * clean up * refatoring eco builder package * start to removing static calls * removing static calls * still working through linting errors * fixing lint errors * fixing setter * adding test * start to verfiying bean references * verifying components * fixing tests * worked passed constructor argument issue. Now need to deal with backtypes from Storm * saving before restructure * building kafka topology * fixing imports for test * fixing folder structure * refactor for testability * removing more statics * cleaning up eco builders * confirmation config builder maps as expected * finishing configBuilder test * cleaning up BoltBuilder * verfiying component builder * replacing java logger * adding tests * adding verification of building implementations of IRichBolts * clean up * adding mocks for testing * fixing classNotFound errors * coverage on all of stream bulder * need to refactor ObjectBuilder * moving static call * adding verifcation to test * clean up * clean up * verifying path of building objects with args * covering other path * starting on builder utility * verifying className * adding test * adding test * clean up * adding test * clean up * clean up * fixing checkstyle errors * fixing linting issues * fixing linting issues * fixing linting issues * fixing linting issues * linting issues * fixing checkstyle * fixing checkstyle * saving progress * adding simple wordcount eco example * adding examples * checkstyles 09 January 2018, 17:09:24 UTC
d609b7c Fixing the slurm scheduler (#2657) * Fixing the slurm scheduler to work with api changes * Fixing the slurm scheduler to work with api changes * Fixing an error with the tests 08 January 2018, 22:22:38 UTC
fde69b7 Updated to use brew delivered bazel (#2655) 06 January 2018, 20:16:21 UTC
0e06ad7 Fix a typo in comment (#2654) 03 January 2018, 21:01:47 UTC
a1953b9 Add k8s support for aws ebs volume mounts. (#2650) 03 January 2018, 00:41:04 UTC
adb3cf9 Increase wait time in MetricsCacheSinkTest and TMasterSinkTest (#2646) * Increase wait time in MetricsCacheSinkTest and TMasterSinkTest Restart wait time is changed to 3 times of reconnect time from 2. - One reconnect time at beginning and - another delay of reconnect time caused by exception handler and - an extra second. 02 January 2018, 23:43:32 UTC
e6c7c12 added a quick fix (#2652) 02 January 2018, 23:11:11 UTC
0ae3f3a Decouple increment and log to avoid potential error (#2644) 02 January 2018, 21:54:17 UTC
5011cd9 bookkeeper using emptyDir (#2649) * bookkeeper using emptyDir * volumes should be aligning with containers 02 January 2018, 21:45:45 UTC
b7a2486 Fix aurora scheduler (#2645) * Fix aurora scheduler (added a few binding values missed in PR #2632) 02 January 2018, 20:59:54 UTC
0f55e1e make the yaml cpp tarball (#2648) * make the yaml cpp tarball downloadable * no need for script file env file generation 02 January 2018, 00:07:45 UTC
c42c87d Refactoring and improving standalone mode (#2633) * Refactoring and improving standalone mode * cleaning up * fixing formatting * adding running heron-ui and heron-tracker * fixing test * fixing build file 31 December 2017, 01:39:56 UTC
c32eaa7 Add python encoding (#2631) * add python encoding * more clean up * Change python2.7 to python We hope the python code to be version independent, so remove most 2.7 version number and keep only those that had specified 2.7 before. * missed 3 files 27 December 2017, 18:37:17 UTC
fdb8d27 keep topology defn during update (#2602) This PR keeps topology defn during update. 26 December 2017, 23:58:36 UTC
63b9497 Refactor aurora scheduler to use SchedulerUtils to construct arguments (#2632) Refactor aurora scheduler code to be more consistent with other schedulers. Topology arguments are hardcoded in aurora config file currently instead of using the shared functions in SchedulerUtils. It is hard to maintain and it is causing compatibility issue (since config file is in client and the setup code is in core). 26 December 2017, 23:53:32 UTC
a7a0b99 Update docker build images with bazel 0.9.0 (#2641) 25 December 2017, 19:15:58 UTC
407ff8e Move the logical plan format to the tracker. (#2639) 23 December 2017, 23:36:20 UTC
086b2d7 move to bazel 0.9.0 and cleanups (#2637) * move to bazel 0.9.0 * lots of cleanup * fix the pex issue for bazel in linux 23 December 2017, 17:45:34 UTC
c569ba1 Add instance groups to the physical plan response. (#2636) 23 December 2017, 17:35:58 UTC
bb36167 added targets for heron tools, heron and heron-core (#2635) 22 December 2017, 20:49:04 UTC
77904c2 [Streamlet] Reinforce usage of StreamletNamePrefixes for more type-safety (#2627) * [Streamlet] Reinforce using StreamletNamePrefixes for more type-safety * [Streamlet] Fix documentation issues * Review comment is addressed. 22 December 2017, 07:11:18 UTC
9b17e30 Add cluster role and environment info to sink context. (#2634) * Add cluster role and environment info to sink context. This patch exposes cluster, role and environment to the SinkContext so metric sinks can export this information. * Remove log. 22 December 2017, 00:06:47 UTC
c591fda Multi node standalone cluster for heron (#2611) * adding Heron standalone cluster mode * cleaning up * addressing comments * cleaning up * correcting logic * improving method to check if ip or hostname is self * cleaning up * resolving merge conflicts * addressing comments 20 December 2017, 00:47:03 UTC
843dbd4 Add new state-manager-config-file argument for aurora (#2630) * Add new state-manager-config-file argument for aurora * Add environment variables for aurora * format 19 December 2017, 22:03:56 UTC
f4aeae9 Fix unit tests (MetricsCacheSinkTest, TMasterSinkTest) (#2629) 19 December 2017, 04:42:16 UTC
9c55749 remove problematic link (#2628) 18 December 2017, 19:59:16 UTC
f2caa54 Fix website asset pipeline + add publish script (#2626) * small fix to make clean command * add publish script * add publish script * update website version * add python six installation step * add git push step 15 December 2017, 19:47:39 UTC
d1c28de Revert "Fix website asset pipeline + add publish script (WIP) (#2624)" (#2625) This reverts commit 081435408ebb82177a74cda0fee0919f73f845bb. 15 December 2017, 19:46:32 UTC
0814354 Fix website asset pipeline + add publish script (WIP) (#2624) Fix website asset pipeline + add publish script 15 December 2017, 19:46:03 UTC
061ef99 use gtest as a artifact rather than a tarball (#2623) 14 December 2017, 01:31:45 UTC
097c7bc use cereal from artifact rather than a tarball (#2622) * use cereal as an artifact rather than a tarball * remove protobuf include directory 13 December 2017, 15:52:03 UTC
2ac72ee use protobuf artifactory rather than tarball (#2621) 13 December 2017, 07:09:19 UTC
e374885 use gflags as a artifact instead of tar ball (#2620) * use gflags as a artifact instead of tar ball * make sure that check style skips external/ 13 December 2017, 01:16:30 UTC
300cb2f Fix api server create config cluster mode. (#2617) * Fix api server create config cluster mode. - Use config cluster mode if we are not running locally. * Fix unit test. 13 December 2017, 01:15:57 UTC
58cfc5b generate helm charts (#2619) * generate helm charts * cleanup echo statements 12 December 2017, 22:31:45 UTC
be5a36b generate docker image and push for development (#2613) * added test docker image push * change publish to dev publish * modified helm charts * added alignment based on Chris's comments * update the service for heron-apiserver * change the path for release.yaml in apiserver * move zookeeper wait to script file in the container * print zk host and zk port correctly * update the base docker image to version 0.4.0 * name of the cluster should not contain zookeeper * remove heron-core-release target 12 December 2017, 19:41:49 UTC
8d0fcfe Refactor heron client main.py (#2607) * Refactor heron client main.py Refactor main function of heron client into two steps, 1. set up command handlers 2. execute command and make it possible to customize heron client by creating new handlers. 12 December 2017, 18:43:52 UTC
5cdbfa0 convert git repo for bazel dockers rules to artifiact (#2618) 12 December 2017, 14:32:06 UTC
49567c6 Fix bookkeeper statefulset yaml. (#2609) 11 December 2017, 00:14:05 UTC
7c4d1a5 make a single heron package for each platform, removed api, tools, cl… (#2608) * make a single heron package for each platform, removed api, tools, client and instead a single package called heron * prepare the docker image using only tools and core only 08 December 2017, 03:42:06 UTC
86bcac2 update java API docs (#2604) 07 December 2017, 22:02:20 UTC
02020a7 Kubernetes configuration (#2507) * begin adding configs list * begin adding kubernetes configs * add config list shortcode plus more kubernetes configs * more k8s configs * finish adding k8s configs and modify shortcode template * add brief intro sentence 07 December 2017, 22:01:01 UTC
b5759c2 Delete index.yaml Not needed since Helm needs to host the charts. 07 December 2017, 03:53:06 UTC
b045ad8 change version and add index to package (#2603) 07 December 2017, 00:17:39 UTC
20bd753 use bazel docker rules (#2545) * use bazel docker rules * Update WORKSPACE * fix the installation * update tests * change pex source * removed commented lines 06 December 2017, 23:37:48 UTC
b99a75a adding Nomad scheduler support for Heron (#2601) * adding Nomad scheduler support for Heron * adding unit tests * cleaning up * addressing comments * addressing comments 05 December 2017, 23:39:38 UTC
ef22068 Add nfs volumes to kubernetes. (#2598) 02 December 2017, 19:02:02 UTC
e2fee5d Assign pplan helper before doing state activate. (#2599) 02 December 2017, 00:11:22 UTC
db120ea easy install for kubernetes (#2590) * easy install for kubernetes * remove depedency charts * added notes 30 November 2017, 19:54:32 UTC
f42ce24 Add ability for users to allow tuple drops upon backpressure detection (#2593) * Add ability for users to allow tuple drops upon backpressure detection * Undo examples change * Address feedback 30 November 2017, 09:17:28 UTC
c24ba96 Add null protection in config translation code (#2597) This change contains only null pointer checks. check job hasn't finished yet but it has been running for awhile so compiling should be successful now. 29 November 2017, 23:00:37 UTC
back to top