https://github.com/twitter/heron

sort by:
Revision Author Date Message Commit Date
3270bb7 Refactor heron client main.py (#2607) * Refactor heron client main.py Refactor main function of heron client into two steps, 1. set up command handlers 2. execute command and make it possible to customize heron client by creating new handlers. 12 December 2017, 23:51:52 UTC
14945f4 Add null protection in config translation code (#2597) This change contains only null pointer checks. check job hasn't finished yet but it has been running for awhile so compiling should be successful now. 29 November 2017, 23:25:02 UTC
649a005 Add config translations for getComponentConfiguration() (#2592) 28 November 2017, 17:55:38 UTC
566aef8 [work in progress] fix a bug for `deactivated heron-instance` during `heron update` (#2580) fix a bug for `deactivated heron-instance` during `heron update` 28 November 2017, 17:55:34 UTC
d649b04 Fix wrong stmgr log (#2577) Currently, the stmgr logs incorrectly for state change: it logs the new state to old state. This pull requests fixes it by logging the old state to new state. 28 November 2017, 17:55:29 UTC
60bcfe0 Log when stmgr broadcasts physical plan (#2573) Currently, it is hard to trace the destination instances that a stmgr sends new physical plan to, especially during run-time scaling. This pull request logs it down for better tracing and debug. Tested with LocalScheduler. 28 November 2017, 17:55:25 UTC
2c9565b Fix the logging of System Config (#2566) Currently, when heron-instance tries to log the System Config, it will only log the reference value, since there is no correct override for SystemConfig.toString(). This pull request fixes this issue by overriding the SystemConfig.toString(). 28 November 2017, 17:55:20 UTC
6b6b140 Log component's config when constructing the instance (#2564) Currently, people are having a hard time to figure out the actual configs applying to a component: - There are a lot of transformations and overrides - The actual configs applied are not directly visible This pull request logs the actual configs applying to the instance during the actual config construction for an instance. It makes debug easier. Furthermore, it is invoked only when having new PhysicalPlanHelper, so it does not hurt runtime performance. 28 November 2017, 17:55:16 UTC
700afcf Fix StreamletImplTest unit test (#2561) 16 November 2017, 19:35:54 UTC
fcac777 fix metricscache location bug (#2556) * fix metricscache location in zk after session reconnection * add corner case handling * address comment 2 * update comment * minor update * add comment 16 November 2017, 08:34:39 UTC
7c2f931 Streamlet name needs to be robust for blank values (#2558) 16 November 2017, 07:41:23 UTC
162b097 Refactoring windowing (#2547) * refactoring windowing to reduce complexity - change to using timers rather than tick tuples - removing unneccessary locks * cleaning up 16 November 2017, 03:00:20 UTC
ef3fc79 Correct the print_usage functionality in Heron Executor (#2548) 16 November 2017, 02:00:47 UTC
5267285 Retry when failing to kill an Aurora topology (#2554) A lot of Twitter internal topology owners complained heron failing to kill an aurora topology because the Aurora service can be unavailable or unstable for a while, or it can sometimes provide a false negative response. This pull request adds retry mechanism to kill an Aurora topology with more confidence. It can increase the robustness of AuroraScheduler regardless of the Aurora Service People can also specify the retry attempts and interval via config. This pull request also re-format some coding-styles for RuntimeManagerRunner.java. Tested with actual Twitter distributed env. 16 November 2017, 01:49:29 UTC
57e910f Make Kryo the default serializer for the Streamlet API (#2549) * add second useKryoSerializer option * use alternate system for setting kryo * fix error in useKryoSerializer function * remove unnecessary private variable from Config class * remove unnecessary import * change serializer setting interface and add tests for config builder * make variable private rather than protected * fix resource byte interface via ByteAmount * use ByteAmount in Resources class and add tests * fix rebase conflict with master * fix additional rebase confligt * fix error in useKryoSerializer function * remove unnecessary private variable from Config class * re-fix rebase conflict * make variable private rather than protected * travis cmake is already 3.x.x and no need to install it (#2552) * use ByteAmount in Resources class and add tests * fix variable naming issue in StreamletImplTest * fix too-long line in StreamletImplTest * fix checkstyle errors and change set serializer method * fix final checkstyle errors * fix naming error leading to variable obfuscation * re-fix naming error and fix too-long line * fix variable obfuscation error 16 November 2017, 01:48:20 UTC
9afe32e fix bugs in AuroraHeronShellController (#2553) * fix stmgr index * delegate to aurora client when shell/kill fails * delegate to aurora client * address commnet 15 November 2017, 23:27:24 UTC
57310fe adding remote debugging functionality (#2484) * adding remote debugging functionality * addressing comments 15 November 2017, 23:03:39 UTC
c37c5a3 fix null (#2555) 15 November 2017, 19:55:07 UTC
da234e9 Move topology config translations into a different function to avoid overwriting (#2551) 15 November 2017, 15:38:16 UTC
371853a Add example to set Container Ram Padding (#2550) Add the example to setContainerRamPadding in ComponentJVMOptionsTopology, since some customers are asking how to do it. 15 November 2017, 02:13:48 UTC
ee99068 travis cmake is already 3.x.x and no need to install it (#2552) 14 November 2017, 22:21:56 UTC
b0aee33 Extend LocalFileSystemUploader UT Coverage (#2542) 13 November 2017, 19:42:52 UTC
f540aab Introduce Kryo Serializer to Streamlet (#2540) * Add Kryo support to streamlet * Check the presence of Kryo at submit time * add builder pattern for Config and Resources (Streamlet API) and modify examples * add defaultResources() method * update test for resources builder * remove unnecessary import * Working version * Fixed java docs * Fixed build * remove unnecessary import * fix checkstyle errors * fix checkstyle errors (javadoc params) * Address comments 13 November 2017, 14:50:15 UTC
42e95fc improve passing of executor ports (#2516) * fix get executor args * improve passing of executor ports * shortening names * fixing minor bug * refactoring code based on comments * addressing comments * refactoring some code 13 November 2017, 14:48:31 UTC
fa5bfc9 Update README.md with service url config (#2544) 12 November 2017, 23:15:43 UTC
efcf42e Update README to use service url config (#2543) 12 November 2017, 23:09:23 UTC
d4ee296 update gitignore (#2539) 12 November 2017, 18:19:27 UTC
509cb94 Use builder pattern in Streamlet API (#2538) * add builder pattern for Config and Resources (Streamlet API) and modify examples * add defaultResources() method * update test for resources builder * remove unnecessary import * remove unnecessary import * fix checkstyle errors * fix checkstyle errors 12 November 2017, 00:06:29 UTC
cfadb1b Change heron api dependency (#2534) * Change hero api dependency * Fix naming issue in BUILD file * Update BUILD comments * Fix Build and Test failures 11 November 2017, 00:46:37 UTC
4e74d66 Add a config translation function for components and fix a typo. (#2536) * Add a config translation function for components and fix a typo. The previous translation function injects new topology level configs that are not good for components and caused serialization errors. New function skips the extra configs. 10 November 2017, 21:48:17 UTC
3a9cc10 Used for example (#2537) 10 November 2017, 21:12:48 UTC
1e4c480 Since Heron does not use kryo for anything, there is no need to shade it out (#2535) 10 November 2017, 20:16:16 UTC
757041a Explicity setting heron apiserver to run as cluster-admin (#2531) * Explicity setting heron apiserver to run as cluster-admin * addressing comments 09 November 2017, 20:14:04 UTC
c65e312 Allow users to register timer tasks in topologies (#2522) * allo topologies to register timer events * adding comments * fixing format * refactoring based on feedback * addressing comments 09 November 2017, 18:12:21 UTC
2115ca3 Generalize the interface of a Source by allowing it to emit a collection at once (#2526) * Generalize the interface of a Source by allowing it to emit a collection at once * Stylecheck fixes 09 November 2017, 18:09:30 UTC
5886a31 Fix log.error exception in check_direct_mode_cluster_definition() (#2532) 09 November 2017, 06:34:31 UTC
1de216b bump docs/website version (#2529) 08 November 2017, 19:57:42 UTC
df878ba Apply config translation for Bolt and Spout (#2527) * Apply config translation for Bolt and Spout The translation in StormSubmitter processes topology level configs. Components have their own configs and component level configs are merged later when Spouts and Bolts are created. Without this change, the raw Storm configs for Spouts and Bolts are passed into Heron layer. As the result, TOPOLOGY_TICK_TUPLE_FREQ_MS is missing in Bolt level because it is generated in the translation by Config.setTickTupleFrequency(). 08 November 2017, 08:58:17 UTC
26b616d Fix the command line for heron executor for aurora. (#2525) 07 November 2017, 21:28:55 UTC
37a969d Java streamlet examples (#2501) * move examples from other directory * fix build errors; add serialVersionUid to serializable classes * rename to original topology name * continue adding comments to example topologies * comments for the ImpressionsAndClicks topology * finish adding comments * fix build error * add example clone topology * add comments for streamlet clone example * add formatted output topology * add smart watch topology (for reduce functions) * improve logging for impressions and clicks topology * updates to pulsar topology * add new repartition topology * fix broken examples build * fix indentation * remove star import statements * fix import order * fix remaining java syntax issues * more comments * add private constructor for StreamletUtils * more syntax fixes * fix final formatting issues * remove some unnecessary logging statements * re-add windowed word count topology * add newline at end of file * change time windows to stay within max pending value * Removed KVStreamlet * Fix build * Fixed checkstyle * Fix bug in JoinTest * Fixed jointest * modifications to ads/impressions topology * rework impressions/clicks topology for API changes * fix smart watch topology * fix windowed word count topology * fix unused imports * add comments to repartition topology * remove repartition from integer topology * change description in join operation 07 November 2017, 21:26:06 UTC
2acf4d1 Simplified Streamlet interface (#2519) * Removed KVStreamlet * Fix build * Fixed checkstyle * Fix bug in JoinTest * Fixed jointest 06 November 2017, 18:19:52 UTC
61e4aae fix broken docker build script (#2521) 05 November 2017, 22:04:41 UTC
62771c2 Remove unneeded dep when submitting a tar (#2520) Remove unneeded heron-instance.jar dep from the classpath to start the jvm to execute the topology's main method when submitting a tar. 04 November 2017, 04:40:15 UTC
b5162f0 Use heron home cluster defaults when running in a cluster. (#2517) 03 November 2017, 17:59:10 UTC
b34f389 Added in static files for HeronUI so that Offline Heron Deployment works (#2513) * Create react js For HeronUI to run in offline environment Will be used in heron/heron/tools/ui/resources/templates/application.html * Create JSXTransformer.js For HeronUI to run in offline environment Will be used in heron/heron/tools/ui/resources/templates/application.html * Create list.min.js For HeronUI to run in offline environment From: http://cdnjs.cloudflare.com/ajax/libs/list.js/1.1.1/list.min.js Will be used in: heron/heron/tools/ui/resources/templates/application.html * Changed cloudera links Now point to static links inside the heron project that were added * Changed name of file * Changed name of file Includes version number * Changed file name Corrected file name * Corrected file name Included version number * Fixed points to correct js files that were renamed with version number * Create d3.min.3.4.11.js Created js file with name and version * Create d3-tip.min.0.6.3.js correct name and file for UI fix * FixUI to point to internal js file * Fixed UI to point to static js file * FixedUI to point to static js file * Fixed UI to point to static js files * file reference error fixed fixed typo * changed out dash for dot * Rename JSXTransformer-0.10.0.js to JSXTransformer.0.10.0.js 02 November 2017, 19:04:58 UTC
7f0a602 fixes bookeeper statefulset's deployment url (#2514) should be https://raw.githubusercontent.com/twitter/heron/master/deploy/kubernetes/general/bookkeeper.statefulset.yaml otherwise 404 will occur 02 November 2017, 05:01:44 UTC
0ceb9b7 increasing memory for examples (#2512) 02 November 2017, 02:29:30 UTC
7a8b4a9 fixing formating in examples (#2511) 01 November 2017, 23:03:26 UTC
60936ae Stateful windowing (#2398) * Stateful windowing * cleaning up * fixing bugs * cleaning up * adding documentation * addressing comments 01 November 2017, 16:45:59 UTC
58c8bd3 Added a generalized reduceByKeyAndWindow operator inside the Streamlet (#2502) * Added a generalized reduceByKeyAndWindow operator inside the Streamlet * Make functions more generic * Fixed checkstyle * More checkstyle * Took feedback into consideration * Fixed checkstyle * More checkstyle * Added a new reduceByWindow function * Removed unused classes * Brought back GeneralReduce operators * Used Regular Reduce functions for simple reducers * Added a test for ReduceByWindowOperator * Streamlet typing changes (removal of superfluous super-typing) (#2505) * update KeyValue type signature to fix map issue * remove unnecessary 'super's from KVStreamlet * remove unnecessary 'super's from Streamlet * Add more tests * Added one more test * More tests * One more reduce test 01 November 2017, 04:06:06 UTC
8bd4e59 adding metricscache-sink entry to configs (#2506) * adding metricscache-sink entry to configs * adding metricscache-sink to sinks 31 October 2017, 22:27:14 UTC
06ceb01 Issue 2500: Provide a yaml file to deploy bookkeeper in a StatefulSet for k8s installation (#2503) * Issue 2500: Provide a yaml file to deploy bookkeeper in a StatefulSet for k8s installation * remove storage-class annotation, since it is using `ssd` storageClass * Add pod affinity * Add a reference to Persistent Volumes 31 October 2017, 20:22:21 UTC
fb505af Currently Storm and Heron have different encoding styles for Integer in api.Confg. (#2504) Storm will put the Integer object directly while Heron will put the Integer.toString() object. So if the config is put via Heron style, like: com.twitter.heron.api.Config.setMessageTimeoutSecs(conf: Storm Config, 1200), and submit with a StormSubmitter. The StormSubmitter fails to parse the Integer value from the Config and throws: Exception in thread "main" java.lang.ClassCastException: java.lang.String cannot be cast to java.lang.Integer at org.apache.storm.utils.ConfigUtils.translateConfig(ConfigUtils.java:70) This pull request uses a more generic Integer convention method in ConfigUtils.translateConfig(..), so the StormSubmitter could submit with different styles config. Tested with LocalScheduler. 31 October 2017, 07:22:40 UTC
7d660b3 add processingdataskew detector (#2498) 30 October 2017, 04:38:16 UTC
754cdb7 include heron.aurora file in the distribution (#2496) 28 October 2017, 18:57:28 UTC
5236e2b bump version (#2497) 27 October 2017, 23:34:58 UTC
e1101f5 fix map inconsistency (#2494) * revert map * update example topo accordingly 27 October 2017, 06:59:25 UTC
2e23cac Fix the correct python package for building heronpy (#2493) 27 October 2017, 01:36:52 UTC
d0f79d2 Add gflags to cpp instance to get rid of positional arguments (#2491) 27 October 2017, 00:09:13 UTC
5fda058 Make stmgr and tmaster use gflags. This removes the ugly positional arguments (#2490) * Make stmgr use gflags. This removes the ugly positional arguments * Convert tmaster from using positional arguments to using gflags 26 October 2017, 13:36:47 UTC
b13aed1 Make Streamlets and KVStreamets independent of each other (#2487) * Make all operators available on both kvstreamlets and streamlets * More changes * More changes * Fix compile issues * Fix compilation issues * Finally it compiles * Tests compile * Fix test build * More comments * Brought back ? super and ? extends * Added mapToStreamlet operator 26 October 2017, 03:42:43 UTC
0bc905b Add flags to the heron executor. (#2463) * Add flags to the heron executor. * Fix checkstyle. * Fix unknown variable name. 26 October 2017, 02:24:09 UTC
bb30139 Python Instance now takes key/value based arguments (#2489) 25 October 2017, 23:00:39 UTC
fd1d9ef validate topology before submitting (#2483) * verify topology before submitting moved TopologyUtils to heron api * cleaning up * cleaning up for checkstyle 25 October 2017, 17:34:34 UTC
198eae6 remove unnecessary import statement (#2488) 25 October 2017, 13:02:17 UTC
146f2d8 Add dual license to api files (#2435) 25 October 2017, 02:42:49 UTC
2469302 Updated the workspace after we upgraded to newer requests (#2486) * Updated the workspace after we upgraded to newer requests * Set the right url 24 October 2017, 21:21:44 UTC
4ec7203 Acking topologies for Java documentation (#2485) * begin adding section on acking * more material * finish draft * add failing and timeout sections * add setMessageTimeoutSecs to AckingTopology configuration * remove sleep slow down from topology * add discussion of unique IDs * add note in AckingTopology about unique message IDs 24 October 2017, 20:28:56 UTC
c904982 Kill both containers when zombie host happens (#2480) 24 October 2017, 18:54:35 UTC
e1ddb7f Removed dependency on psutil since its not manylinux compliant (#2479) * Removed dependency on psutil since its not manylinux compliant * Fixed unittest 24 October 2017, 06:50:34 UTC
e05c060 Update k8s/GKE docs for multiple storage systems (#2478) * update GKE instructions for multiple storage systems * fix naming of bookkeeper-apiserver.yaml 23 October 2017, 20:57:25 UTC
1f41d34 Documentation for the Prometheus metrics sink (#2457) * add new prometheus doc and modify other metrics docs * change description of port * add port-file description * remove flat-metrics parameter 23 October 2017, 20:54:15 UTC
ee323a4 Update python requests version (#2456) * Update python requests version * Set version to 2.12.3 23 October 2017, 18:53:23 UTC
5dce4af changing commandline arguments to be passed in a key value fashion (#2471) * changing commandline arguments to be passed in a key value fashion allow override config in heron_executor.py to actually override heron internal configs in HeronInstance * fixing unittest * shading dep 23 October 2017, 18:45:26 UTC
c31eb91 fix bugs and simplifying join API (#2476) * fix bugs and simplifying join API * extract join type 23 October 2017, 13:45:44 UTC
4b3f18f Make cli python3 compliant (#2475) * Make cli python3 compliant * Make heronpy python3 compliant * More python3 stuff * More python3 stuff * More python3 * Ignore 2.7 warnings * Ignore python2.7 pylint warnings * Ignore pylint 2.7 warnings 23 October 2017, 06:25:56 UTC
288662f Add gcp bookkeeper deploy file. (#2477) * Add gcp bookkeeper deploy file. * Add api server configs for bk and gcs. * Rename apiserver file for bookkeeper. 23 October 2017, 02:31:52 UTC
da00c85 Heron on Kubernetes docs (#2444) * fix syntax highlighting in bash blocks * begin adding updated instructions * add minikube instructions * move currently unnecessary material to snippet * add GKE instructions * add note on 3-node clusters vs minikube * finish general k8s cluster instructions 22 October 2017, 21:14:46 UTC
a40006d fixing bugs in python streamlet join (#2474) 21 October 2017, 14:09:54 UTC
c5cb0a5 Change logging interface to accept ostringstream to be outputted to log (#2469) 21 October 2017, 00:01:36 UTC
f1681ba Clear bookkeeper data, when redeploying minikube (#2466) 20 October 2017, 20:54:26 UTC
b6a2b34 Guard empty execute state (#2453) * Guard empty execution state. * remove trailing whitespace. 20 October 2017, 20:42:46 UTC
4a93037 bump version (#2470) 20 October 2017, 17:15:53 UTC
30329fe Added a CPP example topology (#2465) * Added a CPP example topology. Also misc fixes in api to not use glog * Dont link shared * Pass absolute path for dlopen 20 October 2017, 04:48:59 UTC
2ba084e Make container ram padding configurable for round robin packing. (#2464) * Make container ram padding configurable for round robin packing. * Remove extra logging statements. * Add container ram padding to config in cpp. 20 October 2017, 03:28:19 UTC
c0e6899 rename link in TOC (#2467) 20 October 2017, 00:34:38 UTC
3c844c9 Documentation for the Scribe metrics sink (#2459) * add new prometheus doc and modify other metrics docs * change description of port * add port-file description * add new Scribe doc * finish draft of scribe doc * change CSS for docs header section * add new index doc for observability 19 October 2017, 22:30:13 UTC
8ce680d Added cli/scheduler support for submitting/launching cpp topologies (#2462) * Added ability to launch cpp topologies from cli * Include cpp instance binary in the list of packaged files * Pass CPP binary to aurora instance * Fix unittest * Fix executor unittest 19 October 2017, 17:22:44 UTC
aef7402 add prometheus-sinks configs to the remaining base templates (#2458) 19 October 2017, 17:21:49 UTC
30f2913 Added tests for cpp api (#2461) 19 October 2017, 05:59:33 UTC
63a1940 Enforce Memory limit for Python Instance (#2460) 19 October 2017, 03:29:00 UTC
787aaba Heron C++ Instance (#2449) * CPP Instance * remove using emptyString now that EMPTY_STRING is a std::string * Add ldl flag * Renamed ReConnect to Reconnect 18 October 2017, 21:41:54 UTC
038db9e Rename "DSL" to "Streamlet API" (#2424) * general s/dsl/streamlet * address python name changes * additional changes to fix broken tests * fix two python import issues * fix bazel reference * one more naming fix for failing test * fix naming problems leading to broken tests * make constructor on example streamlet topology private * fix indentation issue setting off Java checkstyle errors * remove unnecessary and out-of-order java imports * rename files to fix breaking integration tests * rename file and references * add missing import * g/Steamlet/Streamlet * remove superfluous files resulting from merge * re-add JoinOperatorTest * switch from heron-python-streamlet-api to heron-python-streamlet * fix package name * remove unnecessary import 18 October 2017, 19:45:59 UTC
d8b1f7e Heron API server documentation (#2405) * add new docs for Heron API server * add new topology submission explanation * center doc images * finishing touches * add section on configuration overrides * fix broken merge * remove docs for --release-file flag 18 October 2017, 16:58:11 UTC
4139423 add heron config section (#2417) 18 October 2017, 16:57:44 UTC
3d7bf68 Change EMPTY_STRING defn to be a plain string so that we are ok getting its references (#2446) 18 October 2017, 02:13:09 UTC
2ae8644 Revert "Guard empty execution state. (#2447)" (#2448) This reverts commit eeb0f9352a5776c4f412c368083ab72ebd6c83ac. 18 October 2017, 02:10:54 UTC
eeb0f93 Guard empty execution state. (#2447) 17 October 2017, 23:24:04 UTC
91fd289 Increase state manager reachability timeout. (#2395) * Increase state manager reachability timeout. * Fix checkstyle error. 17 October 2017, 21:28:35 UTC
373158e Theoretical documentation for windowing (#2414) * more changes to theory section * finish up theory explanation * comment out unfinished snippet * remove comment about window intervals in seconds * rename necessary files and indicators * fix remaining mentions of 'dsl' 17 October 2017, 21:14:49 UTC
back to top