https://github.com/twitter/heron

sort by:
Revision Author Date Message Commit Date
f540aab Introduce Kryo Serializer to Streamlet (#2540) * Add Kryo support to streamlet * Check the presence of Kryo at submit time * add builder pattern for Config and Resources (Streamlet API) and modify examples * add defaultResources() method * update test for resources builder * remove unnecessary import * Working version * Fixed java docs * Fixed build * remove unnecessary import * fix checkstyle errors * fix checkstyle errors (javadoc params) * Address comments 13 November 2017, 14:50:15 UTC
42e95fc improve passing of executor ports (#2516) * fix get executor args * improve passing of executor ports * shortening names * fixing minor bug * refactoring code based on comments * addressing comments * refactoring some code 13 November 2017, 14:48:31 UTC
fa5bfc9 Update README.md with service url config (#2544) 12 November 2017, 23:15:43 UTC
efcf42e Update README to use service url config (#2543) 12 November 2017, 23:09:23 UTC
d4ee296 update gitignore (#2539) 12 November 2017, 18:19:27 UTC
509cb94 Use builder pattern in Streamlet API (#2538) * add builder pattern for Config and Resources (Streamlet API) and modify examples * add defaultResources() method * update test for resources builder * remove unnecessary import * remove unnecessary import * fix checkstyle errors * fix checkstyle errors 12 November 2017, 00:06:29 UTC
cfadb1b Change heron api dependency (#2534) * Change hero api dependency * Fix naming issue in BUILD file * Update BUILD comments * Fix Build and Test failures 11 November 2017, 00:46:37 UTC
4e74d66 Add a config translation function for components and fix a typo. (#2536) * Add a config translation function for components and fix a typo. The previous translation function injects new topology level configs that are not good for components and caused serialization errors. New function skips the extra configs. 10 November 2017, 21:48:17 UTC
3a9cc10 Used for example (#2537) 10 November 2017, 21:12:48 UTC
1e4c480 Since Heron does not use kryo for anything, there is no need to shade it out (#2535) 10 November 2017, 20:16:16 UTC
757041a Explicity setting heron apiserver to run as cluster-admin (#2531) * Explicity setting heron apiserver to run as cluster-admin * addressing comments 09 November 2017, 20:14:04 UTC
c65e312 Allow users to register timer tasks in topologies (#2522) * allo topologies to register timer events * adding comments * fixing format * refactoring based on feedback * addressing comments 09 November 2017, 18:12:21 UTC
2115ca3 Generalize the interface of a Source by allowing it to emit a collection at once (#2526) * Generalize the interface of a Source by allowing it to emit a collection at once * Stylecheck fixes 09 November 2017, 18:09:30 UTC
5886a31 Fix log.error exception in check_direct_mode_cluster_definition() (#2532) 09 November 2017, 06:34:31 UTC
1de216b bump docs/website version (#2529) 08 November 2017, 19:57:42 UTC
df878ba Apply config translation for Bolt and Spout (#2527) * Apply config translation for Bolt and Spout The translation in StormSubmitter processes topology level configs. Components have their own configs and component level configs are merged later when Spouts and Bolts are created. Without this change, the raw Storm configs for Spouts and Bolts are passed into Heron layer. As the result, TOPOLOGY_TICK_TUPLE_FREQ_MS is missing in Bolt level because it is generated in the translation by Config.setTickTupleFrequency(). 08 November 2017, 08:58:17 UTC
26b616d Fix the command line for heron executor for aurora. (#2525) 07 November 2017, 21:28:55 UTC
37a969d Java streamlet examples (#2501) * move examples from other directory * fix build errors; add serialVersionUid to serializable classes * rename to original topology name * continue adding comments to example topologies * comments for the ImpressionsAndClicks topology * finish adding comments * fix build error * add example clone topology * add comments for streamlet clone example * add formatted output topology * add smart watch topology (for reduce functions) * improve logging for impressions and clicks topology * updates to pulsar topology * add new repartition topology * fix broken examples build * fix indentation * remove star import statements * fix import order * fix remaining java syntax issues * more comments * add private constructor for StreamletUtils * more syntax fixes * fix final formatting issues * remove some unnecessary logging statements * re-add windowed word count topology * add newline at end of file * change time windows to stay within max pending value * Removed KVStreamlet * Fix build * Fixed checkstyle * Fix bug in JoinTest * Fixed jointest * modifications to ads/impressions topology * rework impressions/clicks topology for API changes * fix smart watch topology * fix windowed word count topology * fix unused imports * add comments to repartition topology * remove repartition from integer topology * change description in join operation 07 November 2017, 21:26:06 UTC
2acf4d1 Simplified Streamlet interface (#2519) * Removed KVStreamlet * Fix build * Fixed checkstyle * Fix bug in JoinTest * Fixed jointest 06 November 2017, 18:19:52 UTC
61e4aae fix broken docker build script (#2521) 05 November 2017, 22:04:41 UTC
62771c2 Remove unneeded dep when submitting a tar (#2520) Remove unneeded heron-instance.jar dep from the classpath to start the jvm to execute the topology's main method when submitting a tar. 04 November 2017, 04:40:15 UTC
b5162f0 Use heron home cluster defaults when running in a cluster. (#2517) 03 November 2017, 17:59:10 UTC
b34f389 Added in static files for HeronUI so that Offline Heron Deployment works (#2513) * Create react js For HeronUI to run in offline environment Will be used in heron/heron/tools/ui/resources/templates/application.html * Create JSXTransformer.js For HeronUI to run in offline environment Will be used in heron/heron/tools/ui/resources/templates/application.html * Create list.min.js For HeronUI to run in offline environment From: http://cdnjs.cloudflare.com/ajax/libs/list.js/1.1.1/list.min.js Will be used in: heron/heron/tools/ui/resources/templates/application.html * Changed cloudera links Now point to static links inside the heron project that were added * Changed name of file * Changed name of file Includes version number * Changed file name Corrected file name * Corrected file name Included version number * Fixed points to correct js files that were renamed with version number * Create d3.min.3.4.11.js Created js file with name and version * Create d3-tip.min.0.6.3.js correct name and file for UI fix * FixUI to point to internal js file * Fixed UI to point to static js file * FixedUI to point to static js file * Fixed UI to point to static js files * file reference error fixed fixed typo * changed out dash for dot * Rename JSXTransformer-0.10.0.js to JSXTransformer.0.10.0.js 02 November 2017, 19:04:58 UTC
7f0a602 fixes bookeeper statefulset's deployment url (#2514) should be https://raw.githubusercontent.com/twitter/heron/master/deploy/kubernetes/general/bookkeeper.statefulset.yaml otherwise 404 will occur 02 November 2017, 05:01:44 UTC
0ceb9b7 increasing memory for examples (#2512) 02 November 2017, 02:29:30 UTC
7a8b4a9 fixing formating in examples (#2511) 01 November 2017, 23:03:26 UTC
60936ae Stateful windowing (#2398) * Stateful windowing * cleaning up * fixing bugs * cleaning up * adding documentation * addressing comments 01 November 2017, 16:45:59 UTC
58c8bd3 Added a generalized reduceByKeyAndWindow operator inside the Streamlet (#2502) * Added a generalized reduceByKeyAndWindow operator inside the Streamlet * Make functions more generic * Fixed checkstyle * More checkstyle * Took feedback into consideration * Fixed checkstyle * More checkstyle * Added a new reduceByWindow function * Removed unused classes * Brought back GeneralReduce operators * Used Regular Reduce functions for simple reducers * Added a test for ReduceByWindowOperator * Streamlet typing changes (removal of superfluous super-typing) (#2505) * update KeyValue type signature to fix map issue * remove unnecessary 'super's from KVStreamlet * remove unnecessary 'super's from Streamlet * Add more tests * Added one more test * More tests * One more reduce test 01 November 2017, 04:06:06 UTC
8bd4e59 adding metricscache-sink entry to configs (#2506) * adding metricscache-sink entry to configs * adding metricscache-sink to sinks 31 October 2017, 22:27:14 UTC
06ceb01 Issue 2500: Provide a yaml file to deploy bookkeeper in a StatefulSet for k8s installation (#2503) * Issue 2500: Provide a yaml file to deploy bookkeeper in a StatefulSet for k8s installation * remove storage-class annotation, since it is using `ssd` storageClass * Add pod affinity * Add a reference to Persistent Volumes 31 October 2017, 20:22:21 UTC
fb505af Currently Storm and Heron have different encoding styles for Integer in api.Confg. (#2504) Storm will put the Integer object directly while Heron will put the Integer.toString() object. So if the config is put via Heron style, like: com.twitter.heron.api.Config.setMessageTimeoutSecs(conf: Storm Config, 1200), and submit with a StormSubmitter. The StormSubmitter fails to parse the Integer value from the Config and throws: Exception in thread "main" java.lang.ClassCastException: java.lang.String cannot be cast to java.lang.Integer at org.apache.storm.utils.ConfigUtils.translateConfig(ConfigUtils.java:70) This pull request uses a more generic Integer convention method in ConfigUtils.translateConfig(..), so the StormSubmitter could submit with different styles config. Tested with LocalScheduler. 31 October 2017, 07:22:40 UTC
7d660b3 add processingdataskew detector (#2498) 30 October 2017, 04:38:16 UTC
754cdb7 include heron.aurora file in the distribution (#2496) 28 October 2017, 18:57:28 UTC
5236e2b bump version (#2497) 27 October 2017, 23:34:58 UTC
e1101f5 fix map inconsistency (#2494) * revert map * update example topo accordingly 27 October 2017, 06:59:25 UTC
2e23cac Fix the correct python package for building heronpy (#2493) 27 October 2017, 01:36:52 UTC
d0f79d2 Add gflags to cpp instance to get rid of positional arguments (#2491) 27 October 2017, 00:09:13 UTC
5fda058 Make stmgr and tmaster use gflags. This removes the ugly positional arguments (#2490) * Make stmgr use gflags. This removes the ugly positional arguments * Convert tmaster from using positional arguments to using gflags 26 October 2017, 13:36:47 UTC
b13aed1 Make Streamlets and KVStreamets independent of each other (#2487) * Make all operators available on both kvstreamlets and streamlets * More changes * More changes * Fix compile issues * Fix compilation issues * Finally it compiles * Tests compile * Fix test build * More comments * Brought back ? super and ? extends * Added mapToStreamlet operator 26 October 2017, 03:42:43 UTC
0bc905b Add flags to the heron executor. (#2463) * Add flags to the heron executor. * Fix checkstyle. * Fix unknown variable name. 26 October 2017, 02:24:09 UTC
bb30139 Python Instance now takes key/value based arguments (#2489) 25 October 2017, 23:00:39 UTC
fd1d9ef validate topology before submitting (#2483) * verify topology before submitting moved TopologyUtils to heron api * cleaning up * cleaning up for checkstyle 25 October 2017, 17:34:34 UTC
198eae6 remove unnecessary import statement (#2488) 25 October 2017, 13:02:17 UTC
146f2d8 Add dual license to api files (#2435) 25 October 2017, 02:42:49 UTC
2469302 Updated the workspace after we upgraded to newer requests (#2486) * Updated the workspace after we upgraded to newer requests * Set the right url 24 October 2017, 21:21:44 UTC
4ec7203 Acking topologies for Java documentation (#2485) * begin adding section on acking * more material * finish draft * add failing and timeout sections * add setMessageTimeoutSecs to AckingTopology configuration * remove sleep slow down from topology * add discussion of unique IDs * add note in AckingTopology about unique message IDs 24 October 2017, 20:28:56 UTC
c904982 Kill both containers when zombie host happens (#2480) 24 October 2017, 18:54:35 UTC
e1ddb7f Removed dependency on psutil since its not manylinux compliant (#2479) * Removed dependency on psutil since its not manylinux compliant * Fixed unittest 24 October 2017, 06:50:34 UTC
e05c060 Update k8s/GKE docs for multiple storage systems (#2478) * update GKE instructions for multiple storage systems * fix naming of bookkeeper-apiserver.yaml 23 October 2017, 20:57:25 UTC
1f41d34 Documentation for the Prometheus metrics sink (#2457) * add new prometheus doc and modify other metrics docs * change description of port * add port-file description * remove flat-metrics parameter 23 October 2017, 20:54:15 UTC
ee323a4 Update python requests version (#2456) * Update python requests version * Set version to 2.12.3 23 October 2017, 18:53:23 UTC
5dce4af changing commandline arguments to be passed in a key value fashion (#2471) * changing commandline arguments to be passed in a key value fashion allow override config in heron_executor.py to actually override heron internal configs in HeronInstance * fixing unittest * shading dep 23 October 2017, 18:45:26 UTC
c31eb91 fix bugs and simplifying join API (#2476) * fix bugs and simplifying join API * extract join type 23 October 2017, 13:45:44 UTC
4b3f18f Make cli python3 compliant (#2475) * Make cli python3 compliant * Make heronpy python3 compliant * More python3 stuff * More python3 stuff * More python3 * Ignore 2.7 warnings * Ignore python2.7 pylint warnings * Ignore pylint 2.7 warnings 23 October 2017, 06:25:56 UTC
288662f Add gcp bookkeeper deploy file. (#2477) * Add gcp bookkeeper deploy file. * Add api server configs for bk and gcs. * Rename apiserver file for bookkeeper. 23 October 2017, 02:31:52 UTC
da00c85 Heron on Kubernetes docs (#2444) * fix syntax highlighting in bash blocks * begin adding updated instructions * add minikube instructions * move currently unnecessary material to snippet * add GKE instructions * add note on 3-node clusters vs minikube * finish general k8s cluster instructions 22 October 2017, 21:14:46 UTC
a40006d fixing bugs in python streamlet join (#2474) 21 October 2017, 14:09:54 UTC
c5cb0a5 Change logging interface to accept ostringstream to be outputted to log (#2469) 21 October 2017, 00:01:36 UTC
f1681ba Clear bookkeeper data, when redeploying minikube (#2466) 20 October 2017, 20:54:26 UTC
b6a2b34 Guard empty execute state (#2453) * Guard empty execution state. * remove trailing whitespace. 20 October 2017, 20:42:46 UTC
4a93037 bump version (#2470) 20 October 2017, 17:15:53 UTC
30329fe Added a CPP example topology (#2465) * Added a CPP example topology. Also misc fixes in api to not use glog * Dont link shared * Pass absolute path for dlopen 20 October 2017, 04:48:59 UTC
2ba084e Make container ram padding configurable for round robin packing. (#2464) * Make container ram padding configurable for round robin packing. * Remove extra logging statements. * Add container ram padding to config in cpp. 20 October 2017, 03:28:19 UTC
c0e6899 rename link in TOC (#2467) 20 October 2017, 00:34:38 UTC
3c844c9 Documentation for the Scribe metrics sink (#2459) * add new prometheus doc and modify other metrics docs * change description of port * add port-file description * add new Scribe doc * finish draft of scribe doc * change CSS for docs header section * add new index doc for observability 19 October 2017, 22:30:13 UTC
8ce680d Added cli/scheduler support for submitting/launching cpp topologies (#2462) * Added ability to launch cpp topologies from cli * Include cpp instance binary in the list of packaged files * Pass CPP binary to aurora instance * Fix unittest * Fix executor unittest 19 October 2017, 17:22:44 UTC
aef7402 add prometheus-sinks configs to the remaining base templates (#2458) 19 October 2017, 17:21:49 UTC
30f2913 Added tests for cpp api (#2461) 19 October 2017, 05:59:33 UTC
63a1940 Enforce Memory limit for Python Instance (#2460) 19 October 2017, 03:29:00 UTC
787aaba Heron C++ Instance (#2449) * CPP Instance * remove using emptyString now that EMPTY_STRING is a std::string * Add ldl flag * Renamed ReConnect to Reconnect 18 October 2017, 21:41:54 UTC
038db9e Rename "DSL" to "Streamlet API" (#2424) * general s/dsl/streamlet * address python name changes * additional changes to fix broken tests * fix two python import issues * fix bazel reference * one more naming fix for failing test * fix naming problems leading to broken tests * make constructor on example streamlet topology private * fix indentation issue setting off Java checkstyle errors * remove unnecessary and out-of-order java imports * rename files to fix breaking integration tests * rename file and references * add missing import * g/Steamlet/Streamlet * remove superfluous files resulting from merge * re-add JoinOperatorTest * switch from heron-python-streamlet-api to heron-python-streamlet * fix package name * remove unnecessary import 18 October 2017, 19:45:59 UTC
d8b1f7e Heron API server documentation (#2405) * add new docs for Heron API server * add new topology submission explanation * center doc images * finishing touches * add section on configuration overrides * fix broken merge * remove docs for --release-file flag 18 October 2017, 16:58:11 UTC
4139423 add heron config section (#2417) 18 October 2017, 16:57:44 UTC
3d7bf68 Change EMPTY_STRING defn to be a plain string so that we are ok getting its references (#2446) 18 October 2017, 02:13:09 UTC
2ae8644 Revert "Guard empty execution state. (#2447)" (#2448) This reverts commit eeb0f9352a5776c4f412c368083ab72ebd6c83ac. 18 October 2017, 02:10:54 UTC
eeb0f93 Guard empty execution state. (#2447) 17 October 2017, 23:24:04 UTC
91fd289 Increase state manager reachability timeout. (#2395) * Increase state manager reachability timeout. * Fix checkstyle error. 17 October 2017, 21:28:35 UTC
373158e Theoretical documentation for windowing (#2414) * more changes to theory section * finish up theory explanation * comment out unfinished snippet * remove comment about window intervals in seconds * rename necessary files and indicators * fix remaining mentions of 'dsl' 17 October 2017, 21:14:49 UTC
6560ae2 Updated architecture and design goals documentation (WIP) (#2387) * begin revamping architecture page * more restructuring * begin updating design goals * some small changes * new draft of design goals doc * add heron vs storm comparison table * small mods to comparison table: * add note on containerization 17 October 2017, 21:14:04 UTC
2eb82ff Added Heron Instance related vars in cpp. This will be used by CPP Instance (#2442) 17 October 2017, 13:31:41 UTC
6432b92 Add helper methods in Config Helper Functions that will be used by CPP Instance (#2445) 17 October 2017, 07:48:07 UTC
ec39933 Do minor Metrics Cleanup (#2441) 1. Rename TaskId vars to instance id since thats what they are 2. Make MetricsMgrClient take instance_index as well 17 October 2017, 07:45:58 UTC
c299f93 Log an error message when we dont recognize message type (#2440) 17 October 2017, 07:44:48 UTC
46f4d6a C++ api bindings for Heron (#2429) * C++ API for Heron * make cereal boost independent * Patched cereal * Added Logger * Removed logger and replaced by ostream * Fix travis issue * fix cereal compilation * install linux-libc-dev for asm/errno.h * Missing package * skip the portability tests * Added a patch file to make cereal compile on High Sierra * Removed ACKING usages 16 October 2017, 18:57:50 UTC
a8c4e2f bump version (#2439) 16 October 2017, 17:43:03 UTC
91c8fa1 Updated README.md (#2437) Adding `start` parameter to minikube command. 16 October 2017, 03:45:09 UTC
e06251d fixing join operator (#2436) 16 October 2017, 03:03:35 UTC
68c2310 adding BaseTickTupleAwareRichBolt for Storm compatibility (#2434) 14 October 2017, 04:12:19 UTC
2c91f5f adding StormTopology compatibility (#2433) 14 October 2017, 01:35:02 UTC
9216451 fix travis to use new targets and also some checkstyle (#2432) 14 October 2017, 00:16:05 UTC
61df897 bump version (#2430) 13 October 2017, 20:28:02 UTC
2aaaf74 Add kubernetes deploy scripts for minikube. (#2426) 13 October 2017, 06:49:57 UTC
3ba4503 Add apiserver.yaml and move bookkeeper.yaml for Kubernetes baremetal and VM deployments (#2418) * Add generic apiserver.yaml and move bookkeeper.yaml to generic. * Add README to kubernetes general deploy. 13 October 2017, 06:49:38 UTC
d4d9065 fix heron submit examples (#2423) 12 October 2017, 23:22:38 UTC
57b10db General Cleanup of DSL (#2420) * General Cleanup of DSL 1. Renamed GeneratorSource to simply Source 2. Added Sink definition for complex sinks * Updated comment 12 October 2017, 22:26:48 UTC
593a0b9 Move examples to updated config system (#2425) * move to updated config system * fix remaining instance 12 October 2017, 21:29:44 UTC
7314aa5 allow users to set topology environment properties (#2416) * allow users to set topology environment properties * adding storm compatibility * fixing warning * fixing format * changing config description 12 October 2017, 21:22:11 UTC
5334130 update auroraheronshellcontroller (#2419) * delegate to aurora client controller * update heron-shell aurora controller 12 October 2017, 19:54:37 UTC
90ca0fb update restartcontainerresolver (#2415) * add backpressure * debug print * debug print 2 * update metricscache query main to more general accepting meticscachemgr url * change metrics suffix * add metricscachemgr lib to client * debug print metrics type * change metrics type reader item * System.out.println("check null " + iMetrics); * debug print diagnose * add log to identify resolver null * remove debug print * remote backpressure trigger * fix code style * fix mark comment * update log message * update topo temp..ly * update topology serial version id * remove typo * update topo config * add debug print * add debug print * add debug print * change backpressure sensor * add debug print * add metricscacheprovider parsing route * add debug print * check null val * unique metric name * fix buffer size sensor * add debug print * fix style * update test topo * add debug print * update debug print * add debug print * fix type * fix symptom name * fix restart contaienr resolver * fix style * add topo name * add debug print, fix class name * update topo id * update container index * fix container 0 * remove unused imports * filterfiles 12 October 2017, 18:01:38 UTC
dc882bf fix spelling mistakes (#2413) * fix type * fix typo 12 October 2017, 01:18:30 UTC
back to top