https://github.com/twitter/heron

sort by:
Revision Author Date Message Commit Date
aadbd19 added a quick fix 02 January 2018, 22:48:12 UTC
0ae3f3a Decouple increment and log to avoid potential error (#2644) 02 January 2018, 21:54:17 UTC
5011cd9 bookkeeper using emptyDir (#2649) * bookkeeper using emptyDir * volumes should be aligning with containers 02 January 2018, 21:45:45 UTC
b7a2486 Fix aurora scheduler (#2645) * Fix aurora scheduler (added a few binding values missed in PR #2632) 02 January 2018, 20:59:54 UTC
0f55e1e make the yaml cpp tarball (#2648) * make the yaml cpp tarball downloadable * no need for script file env file generation 02 January 2018, 00:07:45 UTC
c42c87d Refactoring and improving standalone mode (#2633) * Refactoring and improving standalone mode * cleaning up * fixing formatting * adding running heron-ui and heron-tracker * fixing test * fixing build file 31 December 2017, 01:39:56 UTC
c32eaa7 Add python encoding (#2631) * add python encoding * more clean up * Change python2.7 to python We hope the python code to be version independent, so remove most 2.7 version number and keep only those that had specified 2.7 before. * missed 3 files 27 December 2017, 18:37:17 UTC
fdb8d27 keep topology defn during update (#2602) This PR keeps topology defn during update. 26 December 2017, 23:58:36 UTC
63b9497 Refactor aurora scheduler to use SchedulerUtils to construct arguments (#2632) Refactor aurora scheduler code to be more consistent with other schedulers. Topology arguments are hardcoded in aurora config file currently instead of using the shared functions in SchedulerUtils. It is hard to maintain and it is causing compatibility issue (since config file is in client and the setup code is in core). 26 December 2017, 23:53:32 UTC
a7a0b99 Update docker build images with bazel 0.9.0 (#2641) 25 December 2017, 19:15:58 UTC
407ff8e Move the logical plan format to the tracker. (#2639) 23 December 2017, 23:36:20 UTC
086b2d7 move to bazel 0.9.0 and cleanups (#2637) * move to bazel 0.9.0 * lots of cleanup * fix the pex issue for bazel in linux 23 December 2017, 17:45:34 UTC
c569ba1 Add instance groups to the physical plan response. (#2636) 23 December 2017, 17:35:58 UTC
bb36167 added targets for heron tools, heron and heron-core (#2635) 22 December 2017, 20:49:04 UTC
77904c2 [Streamlet] Reinforce usage of StreamletNamePrefixes for more type-safety (#2627) * [Streamlet] Reinforce using StreamletNamePrefixes for more type-safety * [Streamlet] Fix documentation issues * Review comment is addressed. 22 December 2017, 07:11:18 UTC
9b17e30 Add cluster role and environment info to sink context. (#2634) * Add cluster role and environment info to sink context. This patch exposes cluster, role and environment to the SinkContext so metric sinks can export this information. * Remove log. 22 December 2017, 00:06:47 UTC
c591fda Multi node standalone cluster for heron (#2611) * adding Heron standalone cluster mode * cleaning up * addressing comments * cleaning up * correcting logic * improving method to check if ip or hostname is self * cleaning up * resolving merge conflicts * addressing comments 20 December 2017, 00:47:03 UTC
843dbd4 Add new state-manager-config-file argument for aurora (#2630) * Add new state-manager-config-file argument for aurora * Add environment variables for aurora * format 19 December 2017, 22:03:56 UTC
f4aeae9 Fix unit tests (MetricsCacheSinkTest, TMasterSinkTest) (#2629) 19 December 2017, 04:42:16 UTC
9c55749 remove problematic link (#2628) 18 December 2017, 19:59:16 UTC
f2caa54 Fix website asset pipeline + add publish script (#2626) * small fix to make clean command * add publish script * add publish script * update website version * add python six installation step * add git push step 15 December 2017, 19:47:39 UTC
d1c28de Revert "Fix website asset pipeline + add publish script (WIP) (#2624)" (#2625) This reverts commit 081435408ebb82177a74cda0fee0919f73f845bb. 15 December 2017, 19:46:32 UTC
0814354 Fix website asset pipeline + add publish script (WIP) (#2624) Fix website asset pipeline + add publish script 15 December 2017, 19:46:03 UTC
061ef99 use gtest as a artifact rather than a tarball (#2623) 14 December 2017, 01:31:45 UTC
097c7bc use cereal from artifact rather than a tarball (#2622) * use cereal as an artifact rather than a tarball * remove protobuf include directory 13 December 2017, 15:52:03 UTC
2ac72ee use protobuf artifactory rather than tarball (#2621) 13 December 2017, 07:09:19 UTC
e374885 use gflags as a artifact instead of tar ball (#2620) * use gflags as a artifact instead of tar ball * make sure that check style skips external/ 13 December 2017, 01:16:30 UTC
300cb2f Fix api server create config cluster mode. (#2617) * Fix api server create config cluster mode. - Use config cluster mode if we are not running locally. * Fix unit test. 13 December 2017, 01:15:57 UTC
58cfc5b generate helm charts (#2619) * generate helm charts * cleanup echo statements 12 December 2017, 22:31:45 UTC
be5a36b generate docker image and push for development (#2613) * added test docker image push * change publish to dev publish * modified helm charts * added alignment based on Chris's comments * update the service for heron-apiserver * change the path for release.yaml in apiserver * move zookeeper wait to script file in the container * print zk host and zk port correctly * update the base docker image to version 0.4.0 * name of the cluster should not contain zookeeper * remove heron-core-release target 12 December 2017, 19:41:49 UTC
8d0fcfe Refactor heron client main.py (#2607) * Refactor heron client main.py Refactor main function of heron client into two steps, 1. set up command handlers 2. execute command and make it possible to customize heron client by creating new handlers. 12 December 2017, 18:43:52 UTC
5cdbfa0 convert git repo for bazel dockers rules to artifiact (#2618) 12 December 2017, 14:32:06 UTC
49567c6 Fix bookkeeper statefulset yaml. (#2609) 11 December 2017, 00:14:05 UTC
7c4d1a5 make a single heron package for each platform, removed api, tools, cl… (#2608) * make a single heron package for each platform, removed api, tools, client and instead a single package called heron * prepare the docker image using only tools and core only 08 December 2017, 03:42:06 UTC
86bcac2 update java API docs (#2604) 07 December 2017, 22:02:20 UTC
02020a7 Kubernetes configuration (#2507) * begin adding configs list * begin adding kubernetes configs * add config list shortcode plus more kubernetes configs * more k8s configs * finish adding k8s configs and modify shortcode template * add brief intro sentence 07 December 2017, 22:01:01 UTC
b5759c2 Delete index.yaml Not needed since Helm needs to host the charts. 07 December 2017, 03:53:06 UTC
b045ad8 change version and add index to package (#2603) 07 December 2017, 00:17:39 UTC
20bd753 use bazel docker rules (#2545) * use bazel docker rules * Update WORKSPACE * fix the installation * update tests * change pex source * removed commented lines 06 December 2017, 23:37:48 UTC
b99a75a adding Nomad scheduler support for Heron (#2601) * adding Nomad scheduler support for Heron * adding unit tests * cleaning up * addressing comments * addressing comments 05 December 2017, 23:39:38 UTC
ef22068 Add nfs volumes to kubernetes. (#2598) 02 December 2017, 19:02:02 UTC
e2fee5d Assign pplan helper before doing state activate. (#2599) 02 December 2017, 00:11:22 UTC
db120ea easy install for kubernetes (#2590) * easy install for kubernetes * remove depedency charts * added notes 30 November 2017, 19:54:32 UTC
f42ce24 Add ability for users to allow tuple drops upon backpressure detection (#2593) * Add ability for users to allow tuple drops upon backpressure detection * Undo examples change * Address feedback 30 November 2017, 09:17:28 UTC
c24ba96 Add null protection in config translation code (#2597) This change contains only null pointer checks. check job hasn't finished yet but it has been running for awhile so compiling should be successful now. 29 November 2017, 23:00:37 UTC
1791577 imporve print (#2594) 29 November 2017, 16:36:52 UTC
6690eb2 Use yaml lib to generate override.yaml (#2596) 29 November 2017, 08:36:34 UTC
972e8c9 [Streamlet] Support abstraction and type safety more (#2589) * [Streamlet] Support abstraction and type safety more * Add UT coverage for default Streamlet name creation * PrefixType enum is removed because of addressing via other PR 28 November 2017, 18:04:26 UTC
fe0d624 Add config translations for getComponentConfiguration() (#2592) 28 November 2017, 02:55:46 UTC
2847855 [work in progress] fix a bug for `deactivated heron-instance` during `heron update` (#2580) fix a bug for `deactivated heron-instance` during `heron update` 27 November 2017, 22:55:00 UTC
e9aee44 Standardize builder interfaces and remove Resources class (#2559) * add second useKryoSerializer option * add setters for all configs in Builder * add enum for prefix names * use prefix name enum in streamlet class constructors * change streamlet examples to use new builder * remove trailing space caught by checkstyle * fix checkstyle issues * fix javadoc variable names to satisfy checkstyle * fix starred import statement and remove unnecessary java List import * reverse static and final in Builder class designation * fix order of imports in FilesystemSinkTopology * remove ByteAmount dependency from streamlet API * fix example topology that uses ByteAmount * add missing javadoc for methods 27 November 2017, 22:44:22 UTC
bd2056f Fix gcs bookkeeper apiserver yaml. (#2591) 27 November 2017, 18:47:42 UTC
75b495d Reduce the docker image size (#2578) * reduce the docker image size * reduce the docker size image using open jdk 8 slim image * incorporated comments 27 November 2017, 18:38:49 UTC
27c1d5e Add the ability to mount volumes for topologies in k8s. (#2588) * Add the ability to mount volumes for topologies in k8s. * Fix config property exception message. * Fix checkstyle and host path constant. 27 November 2017, 05:39:23 UTC
58031e2 Fix for "Marathon scheduler generates json based on "old method"" (#2583) * Corrected Marathon POST message to fit specification in https://github.com/mesosphere/marathon/commit/261880df99113f0e1b3e03791e0850b1da2457d1 * Update MarathonScheduler.java * Fix style issues Fixed travis style issues => "Checkstyle ends with 8 errors": Spaces around "+" (added 8 spaces) * fix style issues Fix: Line is longer than 100 characters (found 106). 24 November 2017, 04:07:53 UTC
d7ee1b5 [Streamlet] Small refactoring is proposed and UT coverage is extended (#2570) * Small refactoring is proposed and UT coverage is extended * Error message is updated * Review comments are addressed 24 November 2017, 02:29:32 UTC
dbf652c Add ConsumerStreamlet name to stageNames Set to avoid duplication (#2574) 24 November 2017, 02:29:10 UTC
343015a Add batch support for AuroraScheduler to remove containers (#2579) * Add batch support for AuroraScheduler to remove containers Currently, the AuroraScheduler will remove containers one by one, very slow. This pull request allows AuroraScheduler to remove containers in one batch. * Make add containers support verbose option 22 November 2017, 19:56:16 UTC
90dcfb3 Streamlet theoretical documentation (WIP) (#2452) * add diagram and some more explanation * add example fodder * re-arrange JS creation * s/functional/streamlet * begin adding section on join types * more operations docs * transform operation docs * clarify transform operations * s/CVS/CSV * add docs for sink operations * add docs for clone operations * split off non-kv-streamlet operations * split off material into separate doc * some more restructuring (beginning to eliminate KV streamlet discussion) * remove references to KV streamlets * finish draft of join docs * fix broken tables * begin adding section on repartitioning * add new diagram * finish draft 22 November 2017, 18:36:31 UTC
df62689 Refactor kubernetes scheduler to use statefulsets. (#2560) * Refactor kubernetes scheduler to use statefulsets. * Fix more conflict issues. * Revert minikube apiserver yaml. * Fix spelling mistake. * Find max number of instances in a container for opening debugging ports. 22 November 2017, 00:32:41 UTC
9e0d48f Fix wrong stmgr log (#2577) Currently, the stmgr logs incorrectly for state change: it logs the new state to old state. This pull requests fixes it by logging the old state to new state. 21 November 2017, 08:31:21 UTC
4e4b3de Log when stmgr broadcasts physical plan (#2573) Currently, it is hard to trace the destination instances that a stmgr sends new physical plan to, especially during run-time scaling. This pull request logs it down for better tracing and debug. Tested with LocalScheduler. 19 November 2017, 01:52:12 UTC
4d44f5b add exception handling for metricscache process (#2572) * add exception handling for quiting process * address comment * update comment * put halt in finally 19 November 2017, 01:02:41 UTC
d29a6b5 Streamlet API cleanup (#2567) * push name checking/setting logic into a single method * remove unnecessary return calls in void functions * use class variables to set default name * change variable name to satisfy checkstyle * switch variables to private 18 November 2017, 00:55:14 UTC
874feb3 remove unused kubernetes config (#2568) 17 November 2017, 23:23:24 UTC
70614ba bump version (#2569) 17 November 2017, 21:49:00 UTC
863038e Fix sample ComponentJVMOptionsTopology (#2565) * Fix sample topology Currently, ComponentJVMOptionsTopology specifies a NewSize greater than the heap size. This pull request: - fixes it - adds the sample to add component-level config for a particular component 17 November 2017, 20:30:44 UTC
1684c27 Fix the logging of System Config (#2566) Currently, when heron-instance tries to log the System Config, it will only log the reference value, since there is no correct override for SystemConfig.toString(). This pull request fixes this issue by overriding the SystemConfig.toString(). 17 November 2017, 08:25:19 UTC
89e9169 Log component's config when constructing the instance (#2564) Currently, people are having a hard time to figure out the actual configs applying to a component: - There are a lot of transformations and overrides - The actual configs applied are not directly visible This pull request logs the actual configs applying to the instance during the actual config construction for an instance. It makes debug easier. Furthermore, it is invoked only when having new PhysicalPlanHelper, so it does not hurt runtime performance. 17 November 2017, 08:20:56 UTC
9011208 enhance heron-shell /killexecutor (#2557) * add killing process * update killpg to kill * address comment * address comment * update comment * update log 17 November 2017, 06:08:10 UTC
c1e0c01 Beefed up the kryo check to catch during submission time (#2562) * Beefed up the kryo check to catch during submission time * Fixed unittest 17 November 2017, 01:16:37 UTC
8315ff3 Fix StreamletImplTest unit test (#2561) 16 November 2017, 19:22:17 UTC
fcac777 fix metricscache location bug (#2556) * fix metricscache location in zk after session reconnection * add corner case handling * address comment 2 * update comment * minor update * add comment 16 November 2017, 08:34:39 UTC
7c2f931 Streamlet name needs to be robust for blank values (#2558) 16 November 2017, 07:41:23 UTC
162b097 Refactoring windowing (#2547) * refactoring windowing to reduce complexity - change to using timers rather than tick tuples - removing unneccessary locks * cleaning up 16 November 2017, 03:00:20 UTC
ef3fc79 Correct the print_usage functionality in Heron Executor (#2548) 16 November 2017, 02:00:47 UTC
5267285 Retry when failing to kill an Aurora topology (#2554) A lot of Twitter internal topology owners complained heron failing to kill an aurora topology because the Aurora service can be unavailable or unstable for a while, or it can sometimes provide a false negative response. This pull request adds retry mechanism to kill an Aurora topology with more confidence. It can increase the robustness of AuroraScheduler regardless of the Aurora Service People can also specify the retry attempts and interval via config. This pull request also re-format some coding-styles for RuntimeManagerRunner.java. Tested with actual Twitter distributed env. 16 November 2017, 01:49:29 UTC
57e910f Make Kryo the default serializer for the Streamlet API (#2549) * add second useKryoSerializer option * use alternate system for setting kryo * fix error in useKryoSerializer function * remove unnecessary private variable from Config class * remove unnecessary import * change serializer setting interface and add tests for config builder * make variable private rather than protected * fix resource byte interface via ByteAmount * use ByteAmount in Resources class and add tests * fix rebase conflict with master * fix additional rebase confligt * fix error in useKryoSerializer function * remove unnecessary private variable from Config class * re-fix rebase conflict * make variable private rather than protected * travis cmake is already 3.x.x and no need to install it (#2552) * use ByteAmount in Resources class and add tests * fix variable naming issue in StreamletImplTest * fix too-long line in StreamletImplTest * fix checkstyle errors and change set serializer method * fix final checkstyle errors * fix naming error leading to variable obfuscation * re-fix naming error and fix too-long line * fix variable obfuscation error 16 November 2017, 01:48:20 UTC
9afe32e fix bugs in AuroraHeronShellController (#2553) * fix stmgr index * delegate to aurora client when shell/kill fails * delegate to aurora client * address commnet 15 November 2017, 23:27:24 UTC
57310fe adding remote debugging functionality (#2484) * adding remote debugging functionality * addressing comments 15 November 2017, 23:03:39 UTC
c37c5a3 fix null (#2555) 15 November 2017, 19:55:07 UTC
da234e9 Move topology config translations into a different function to avoid overwriting (#2551) 15 November 2017, 15:38:16 UTC
371853a Add example to set Container Ram Padding (#2550) Add the example to setContainerRamPadding in ComponentJVMOptionsTopology, since some customers are asking how to do it. 15 November 2017, 02:13:48 UTC
ee99068 travis cmake is already 3.x.x and no need to install it (#2552) 14 November 2017, 22:21:56 UTC
b0aee33 Extend LocalFileSystemUploader UT Coverage (#2542) 13 November 2017, 19:42:52 UTC
f540aab Introduce Kryo Serializer to Streamlet (#2540) * Add Kryo support to streamlet * Check the presence of Kryo at submit time * add builder pattern for Config and Resources (Streamlet API) and modify examples * add defaultResources() method * update test for resources builder * remove unnecessary import * Working version * Fixed java docs * Fixed build * remove unnecessary import * fix checkstyle errors * fix checkstyle errors (javadoc params) * Address comments 13 November 2017, 14:50:15 UTC
42e95fc improve passing of executor ports (#2516) * fix get executor args * improve passing of executor ports * shortening names * fixing minor bug * refactoring code based on comments * addressing comments * refactoring some code 13 November 2017, 14:48:31 UTC
fa5bfc9 Update README.md with service url config (#2544) 12 November 2017, 23:15:43 UTC
efcf42e Update README to use service url config (#2543) 12 November 2017, 23:09:23 UTC
d4ee296 update gitignore (#2539) 12 November 2017, 18:19:27 UTC
509cb94 Use builder pattern in Streamlet API (#2538) * add builder pattern for Config and Resources (Streamlet API) and modify examples * add defaultResources() method * update test for resources builder * remove unnecessary import * remove unnecessary import * fix checkstyle errors * fix checkstyle errors 12 November 2017, 00:06:29 UTC
cfadb1b Change heron api dependency (#2534) * Change hero api dependency * Fix naming issue in BUILD file * Update BUILD comments * Fix Build and Test failures 11 November 2017, 00:46:37 UTC
4e74d66 Add a config translation function for components and fix a typo. (#2536) * Add a config translation function for components and fix a typo. The previous translation function injects new topology level configs that are not good for components and caused serialization errors. New function skips the extra configs. 10 November 2017, 21:48:17 UTC
3a9cc10 Used for example (#2537) 10 November 2017, 21:12:48 UTC
1e4c480 Since Heron does not use kryo for anything, there is no need to shade it out (#2535) 10 November 2017, 20:16:16 UTC
757041a Explicity setting heron apiserver to run as cluster-admin (#2531) * Explicity setting heron apiserver to run as cluster-admin * addressing comments 09 November 2017, 20:14:04 UTC
c65e312 Allow users to register timer tasks in topologies (#2522) * allo topologies to register timer events * adding comments * fixing format * refactoring based on feedback * addressing comments 09 November 2017, 18:12:21 UTC
2115ca3 Generalize the interface of a Source by allowing it to emit a collection at once (#2526) * Generalize the interface of a Source by allowing it to emit a collection at once * Stylecheck fixes 09 November 2017, 18:09:30 UTC
5886a31 Fix log.error exception in check_direct_mode_cluster_definition() (#2532) 09 November 2017, 06:34:31 UTC
1de216b bump docs/website version (#2529) 08 November 2017, 19:57:42 UTC
back to top