https://github.com/twitter/heron

sort by:
Revision Author Date Message Commit Date
1f7f55f fix travis to use new targets and also some checkstyle 13 October 2017, 22:25:28 UTC
61df897 bump version (#2430) 13 October 2017, 20:28:02 UTC
2aaaf74 Add kubernetes deploy scripts for minikube. (#2426) 13 October 2017, 06:49:57 UTC
3ba4503 Add apiserver.yaml and move bookkeeper.yaml for Kubernetes baremetal and VM deployments (#2418) * Add generic apiserver.yaml and move bookkeeper.yaml to generic. * Add README to kubernetes general deploy. 13 October 2017, 06:49:38 UTC
d4d9065 fix heron submit examples (#2423) 12 October 2017, 23:22:38 UTC
57b10db General Cleanup of DSL (#2420) * General Cleanup of DSL 1. Renamed GeneratorSource to simply Source 2. Added Sink definition for complex sinks * Updated comment 12 October 2017, 22:26:48 UTC
593a0b9 Move examples to updated config system (#2425) * move to updated config system * fix remaining instance 12 October 2017, 21:29:44 UTC
7314aa5 allow users to set topology environment properties (#2416) * allow users to set topology environment properties * adding storm compatibility * fixing warning * fixing format * changing config description 12 October 2017, 21:22:11 UTC
5334130 update auroraheronshellcontroller (#2419) * delegate to aurora client controller * update heron-shell aurora controller 12 October 2017, 19:54:37 UTC
90ca0fb update restartcontainerresolver (#2415) * add backpressure * debug print * debug print 2 * update metricscache query main to more general accepting meticscachemgr url * change metrics suffix * add metricscachemgr lib to client * debug print metrics type * change metrics type reader item * System.out.println("check null " + iMetrics); * debug print diagnose * add log to identify resolver null * remove debug print * remote backpressure trigger * fix code style * fix mark comment * update log message * update topo temp..ly * update topology serial version id * remove typo * update topo config * add debug print * add debug print * add debug print * change backpressure sensor * add debug print * add metricscacheprovider parsing route * add debug print * check null val * unique metric name * fix buffer size sensor * add debug print * fix style * update test topo * add debug print * update debug print * add debug print * fix type * fix symptom name * fix restart contaienr resolver * fix style * add topo name * add debug print, fix class name * update topo id * update container index * fix container 0 * remove unused imports * filterfiles 12 October 2017, 18:01:38 UTC
dc882bf fix spelling mistakes (#2413) * fix type * fix typo 12 October 2017, 01:18:30 UTC
1cb3652 Refactory kubernetes deploy yamls. (#2412) 11 October 2017, 23:31:03 UTC
7b2df95 Add bookkeeper kubernetes yaml script. (#2390) 11 October 2017, 23:30:15 UTC
f4b4a22 update healthmgr (#2406) * add backpressure * debug print * debug print 2 * update metricscache query main to more general accepting meticscachemgr url * change metrics suffix * add metricscachemgr lib to client * debug print metrics type * change metrics type reader item * System.out.println("check null " + iMetrics); * debug print diagnose * add log to identify resolver null * remove debug print * remote backpressure trigger * fix code style * fix mark comment * update log message * update topo temp..ly * update topology serial version id * remove typo * update topo config * add debug print * add debug print * add debug print * change backpressure sensor * add debug print * add metricscacheprovider parsing route * add debug print * check null val * unique metric name * fix buffer size sensor * add debug print * fix style * update test topo * add debug print * update debug print * add debug print * fix type * separate healthmgr files * remove logs * use continue 11 October 2017, 19:03:52 UTC
784ad25 Add basic zookeeper yaml for kubernetes. (#2411) Add basic zookeeper yaml for kubernetes for baremetal or VM deployments 11 October 2017, 18:41:35 UTC
5e08783 update aurora heron shell controller (#2410) * add backpressure * debug print * debug print 2 * update metricscache query main to more general accepting meticscachemgr url * change metrics suffix * add metricscachemgr lib to client * debug print metrics type * change metrics type reader item * System.out.println("check null " + iMetrics); * debug print diagnose * add log to identify resolver null * remove debug print * remote backpressure trigger * fix code style * fix mark comment * update log message * update topo temp..ly * update topology serial version id * remove typo * update topo config * add debug print * add debug print * add debug print * change backpressure sensor * add debug print * add metricscacheprovider parsing route * add debug print * check null val * unique metric name * fix buffer size sensor * add debug print * fix style * update test topo * add debug print * update debug print * add debug print * fix type * fix symptom name * fix restart contaienr resolver * fix style * add topo name * add debug print, fix class name * update topo id * update container index * fix container 0 * update container 0 exception * separate sheduler files * update imports 11 October 2017, 18:37:36 UTC
867bebb Update Heron Topology API docs for Java (#2399) * remove outdated instructions * remove unnecessary instructions for Storm-to-Heron migration * remove EO doc from TOC * fix broken alert * fix broken links in /bolts and /spouts and add announcement about the functional API 11 October 2017, 06:57:15 UTC
f8435ca fixing SerializableGenerator (#2408) 11 October 2017, 06:50:49 UTC
2f5565d fixing flapMap signature (#2407) * fixing flapMap signature * make return type more generic 11 October 2017, 06:50:09 UTC
b4bb9a2 update metricscachemgr (#2401) * add backpressure * debug print * debug print 2 * update metricscache query main to more general accepting meticscachemgr url * change metrics suffix * add metricscachemgr lib to client * debug print metrics type * change metrics type reader item * System.out.println("check null " + iMetrics); * debug print diagnose * add log to identify resolver null * remove debug print * remote backpressure trigger * fix code style * fix mark comment * update log message * update topo temp..ly * update topology serial version id * remove typo * update topo config * add debug print * add debug print * add debug print * change backpressure sensor * add debug print * add metricscacheprovider parsing route * add debug print * check null val * unique metric name * fix buffer size sensor * add debug print * fix style * revert healthmgr changes, keep metricscahe changes 10 October 2017, 22:06:10 UTC
35735ed Rename toSink to consume since thats more representative of what it ds (#2404) 10 October 2017, 14:48:39 UTC
77ddcbd s/setDeliverySemantics/set_delivery_semantics (#2402) 10 October 2017, 08:11:15 UTC
ca019c8 Fix code block and collapse behavior (website) (#2403) * fix collapse behavior * fix small issue in java doc 09 October 2017, 23:15:37 UTC
62330d1 Heron Explorer documentation (#2227) * add Heron Explorer doc * add discussion on heron CLI vs heron explorer * begin adding list of capabilities * finish draft of new doc * finish filling out commands list * fix syntax highlighting 09 October 2017, 21:17:39 UTC
20a154f Effectively-once semantics for the Heron API (#2367) * begin adding section on delivery semantics * add new EO doc * remove unnecessary Sass * fix syntax highlighting * more spout and bolt logic explanation * fill out submission section * fix syntax highlighting * finish up draft * add link in Java topologies doc * fix methods table * s/all/both * fix syntax highlighting in log block 09 October 2017, 21:06:19 UTC
6432920 Enhance Python DSL to bring it par with Java (#2391) * First cut for new java like dsl * More fixes * First cut of enhanced streamlet * Moved all impl details to impl dir * Made several streamlets compliant with new semantics * Added joins/reduces * Added builder and various sources * Added Config/Resources/WindowConfig classes * Added Runner/Window/KeyedWindow definition * Fixed compile issues * Modified mock to match current signature * moved text file connector to obey new interface * Fix compile issues * Fix compilation issues * More compilation issues * More compile fixes * More compile fixes * Almost there * Mess with state correctly * Fixed emissions * Fixed paths * Fixed transform operator * Added Log Streamlet * Added ConsumeBolt * Updated integration tests * Update integration test * Removed unused file * Fixed integration test * Fixed integration tests * Fixed lint * Fixed integration tests * Moved helper looper into its own common pool * Use ArrayLooper instead of defining our own * Removed generator since we are no longer using it * Move the dsl example away from Window * Fixed UnionStreamlet * New result for new topology * The right json 08 October 2017, 07:02:54 UTC
511348b Add fallback for proto type mismatch (#2397) This is temporary measure till we enforce the right protobuf classes on the client and server 06 October 2017, 23:12:21 UTC
5d237b6 Enable python checkstyle for heronpy files (#2393) * Enable python checkstyle for heronpy files * Took feedback 06 October 2017, 06:29:49 UTC
88c5515 Issue 2286: Implement bookkeeper based stateful storage (#2287) * BK Uploader * Add a BK downloader * remove local repo * Add all dependencies * Fix * Fix errors after bumping guava to 20 * Add more tests * Address checkstyle problems * fix typo * Move heron/dlog to `heron/io/dlog` * Use `shaded` distributedlog jar * Include dlog uploader in apiserver and tools * Allow configuring num-replicas for topologies stored in bookkeeper * Remove redundant dependency * Implement bookkeeper (via distributedlog library) as the stateful storage * Address checkstyle issue * Address checkstyle issues 05 October 2017, 23:37:53 UTC
5301deb Log exception when scheduler fails to deploy config. (#2378) * Log exception when scheduler fails to deploy config. * Throw exception when container fails to deploy. This allows to cli to print a meaningful error message to the user. * Fix failing unit test. * Catch ConnectException. * Only catch io exception. * Remove unused import. 05 October 2017, 22:41:43 UTC
c75ce41 Modify local scheduler to work without a core uri. (#2376) * Modify local scheduler to work without a core uri. This adds a config option to use an installed version of heron so we don't need to package the heron client within a docker image to run the sandbox. * Revert kubernetes controller. * Return false if we fail to create symlink to heron core. * Rename scheudler util method. 05 October 2017, 20:23:23 UTC
0a21bbc Colorful dry-run output when we output to TTY device. (#2385) 04 October 2017, 22:50:43 UTC
32e0cc3 For topologies submitted with older jars, topologuy reliability mode (#2383) 04 October 2017, 19:06:47 UTC
f08512f introduce sandbox config (#2373) 04 October 2017, 15:22:28 UTC
2f2ae49 remove remark about heron in beta (#2381) 03 October 2017, 23:33:45 UTC
af6d5d9 Be explicit about subprocess pipe's EOF. (#2380) 03 October 2017, 21:15:02 UTC
a3de75e Fix bazel 6.0 issues (#2379) 03 October 2017, 19:56:36 UTC
a73551a Move zookeeper yaml to gcp. (#2377) Move zookeeper yaml to gcp since it uses a gcp persistent volume. 03 October 2017, 14:02:33 UTC
1c2990d Adding new metrics from Apache Storm (#2375) * adding CountStatAndMetric and LatencyStatAndMetric metrics * removing log message * removing unneccessary file * fixing checkstyle issues 03 October 2017, 08:57:59 UTC
603cfb8 Make bazel updates for 6.0 (#2372) This maintains backward compatibility with 0.54 03 October 2017, 03:10:42 UTC
a5d1e60 fix typo (#2374) 02 October 2017, 21:19:15 UTC
8924a3a fixing docker build script for heron (#2350) 02 October 2017, 03:39:17 UTC
bf9fe93 changing watermark generator to be triggered by tick tuples for event time based windowing (#2370) * changing watermark generator to be triggered by tick tuples for event time based windowing * fixing unit test 01 October 2017, 15:32:49 UTC
f8cd589 Fix operator to support python 2 and 3. (#2369) * Fix operator to support python 2 and 3. * Allow query operators to compile on python 2 and 3. 01 October 2017, 03:33:34 UTC
72d5bfb Add component type label for prometheus metrics. (#2364) * Add component type label for prometheus metrics. * Remove unused variable. * Do not assume default for __execute-time-ns metric. * Fix code style error. 30 September 2017, 23:23:50 UTC
02312d5 Create connection buffer and metrics map every time the connection comes up (#2366) 30 September 2017, 01:21:31 UTC
5f1b8bd Add helper method to specify ram (#2365) * Add helper method to specify ram * Made naming more java friendly 29 September 2017, 16:11:03 UTC
af70b45 Seperate out the low level api examples and higher level dsl examples (#2361) * Seperate out the low level api examples and higher level dsl examples * Fix package targets 29 September 2017, 03:44:53 UTC
d51021c Changing TimeTriggerPolicy to use tick tuples as a timer (#2357) Changed Tick Tuple to frequency to milliseconds 29 September 2017, 01:41:34 UTC
6d27c3c remove note on production usage (#2363) 29 September 2017, 00:47:57 UTC
8fa07fc Make it clear that the Functional API is in beta (#2362) * merge website with master * add beta mode alerts 29 September 2017, 00:18:53 UTC
4e56512 re-address submodule issue (#2360) 28 September 2017, 23:08:33 UTC
492acf4 Heron DSL documentation (#2352) * add generated CSS files to gitignore for website * add benefits of DSL * add more info on partitioning, why the DSL, etc * add link to delivery semantics image * adjust CSS for blockquote elements * add description of KV streamlets * finish rough draft * remove unnecessary image for now * bump version and add note on type safety 28 September 2017, 23:02:22 UTC
a402049 Fix website submodule issue (#2359) * revert to older commit * revert website * remove stray references to DSL 28 September 2017, 20:58:25 UTC
35a351a Small fixes for new Java DSL docs (#2358) * adjust heron version * s/declarative/functional * s/dsl/api * remove usage of DSL * fix example repo URL * fix broken url 28 September 2017, 19:15:42 UTC
2b18743 Heron Java DSL docs (WIP) (#2342) Heron Java DSL docs 28 September 2017, 18:38:50 UTC
a471aff Fix src collection for java api (#2356) 27 September 2017, 23:10:21 UTC
eb90566 More Python3 Friendliness (#2355) * Second round of python3 * More Python3 * long to int * convert reduce to use functools * More python3 * more python3 27 September 2017, 23:02:09 UTC
cde3cf5 fix intellij setup script (#2354) 27 September 2017, 18:30:26 UTC
022d645 Move java dsl to be part of heron api (#2353) 27 September 2017, 01:31:11 UTC
1c94687 Added a generic source that has open/cleanup. Also renamed a few functions (#2351) 26 September 2017, 15:44:10 UTC
1b37539 Fix mktemp issue and revert ci-docker (#2344) 25 September 2017, 22:49:50 UTC
d8049f4 fix simulator for not initializing instance (#2347) * fix simulator * fix unit test 25 September 2017, 19:31:18 UTC
a55490d Fix website asset pipeline (#2348) * move img folder into static * update gitignore * remove static folders from git * update gitignore * fix make clean command * remove gulp clean command * add font awesome CSS 25 September 2017, 18:42:05 UTC
9e8ebfe Delivery semantics documentation (WIP) (#2324) * add new delivery semantics doc * add stateful processing doc * fix CSS setup and add delivery semantics image * remove stateful processing doc for now 25 September 2017, 17:18:55 UTC
0214a6b heronpy now needs protobuf 3.4.0 (#2345) 25 September 2017, 13:53:40 UTC
3f1c55c Checkpoint manager does not support override (#2343) 25 September 2017, 01:58:26 UTC
9ccedee Fix missing artifact for docker build (#2341) 24 September 2017, 22:39:43 UTC
d9ef89d Add heron cli config setting for service_url. (#2326) * Add heron cli config setting for service_url. * Remove commented out code. * Change cli config to yaml format. * Update help messages. * More help message changes. 23 September 2017, 20:15:30 UTC
9342b14 clean heron java instance for stateful processing (#2336) * initial commits to fix instnace with stateful processing functionality * fix instance stateful processing * fix typo and refine example * Fixed bug * clean debug code * clean comments * address comments 23 September 2017, 14:36:22 UTC
c5214b6 After building make sure that all the components have been built (#2340) 23 September 2017, 02:53:01 UTC
124511b Consider override yaml (#2339) 23 September 2017, 02:31:38 UTC
d568f71 Fix default acking (#2338) 22 September 2017, 23:53:49 UTC
5b6bb2f Make delivery semantics public (#2337) 22 September 2017, 21:41:44 UTC
72cff11 Add prometheus scrape annotations to pods in kubernetes scheduler. (#2331) * Add prometheus scrape annotations to pods in kubernetes scheduler. * Remove unused code. 22 September 2017, 17:31:18 UTC
a1705f7 Added a Config class to remove api.Config depedency (#2334) * Introduced Config into the dsl to remove dep on heron.api.Config * Updated comments * Removed unneeded interface * Made transform function more general * Fix checkstyle 22 September 2017, 03:16:42 UTC
6cdfd35 Python3 compatibility (#2335) * Switched from 2.7 to generic python * Removed 2.7 * made all print statements python2 and python3 friendly * Made it python3 friendly * Upgraded unittest to use 1.1.0 * Made print statements compliant with both python2 and python3 * Made statemgrs python2 and 3 compatible * Make raise and print python3 compativle * Made python3 compatible * More python3 compat * Move to beginning of file * python3 fying * Python3fy * Fixed compile bug * Style changes * More python3fy * Python3fy * Pythonify * More Pythonify3 * Unwind some changes for now * More python3fy * More python3fy * Python3fy 22 September 2017, 03:14:54 UTC
44586a5 pex_pytest migration (#2333) 21 September 2017, 03:03:54 UTC
c84ec53 Revert "Take override.yaml in stmgr, metricsmgr, metricsmgr-cache, tmaster, instance (#2311)" (#2332) This reverts commit 3df0a4db2cb51a6bd96df7afd0b9d9eb3b6ada96. 20 September 2017, 23:42:08 UTC
3df0a4d Take override.yaml in stmgr, metricsmgr, metricsmgr-cache, tmaster, instance (#2311) 20 September 2017, 17:14:14 UTC
839649a clean python state manager (#2327) * clean python state manager * update tracker 20 September 2017, 17:06:26 UTC
07713e9 Add prometheus sink to default kubernetes metrics config. (#2329) 20 September 2017, 07:03:25 UTC
389b3f5 Java DSL for Heron (#2285) * Placeholder for Java DSL * More changes to address comments * Compiling version * Added FlatMap * Add filter * Added repartition * Added WindowConfig * First cut of JoinStreamlet * Fixed comparison bug * Added reduce * Added ability to create KVStreamlet from Streamlet * Added java docs * Added Union and reduceByWindow functions * Introduced clone and generalized repartition function * Make Window bolt conform to the proposed window interface * Finally compiling * Fix style checks * Moved Bolts to bolts dir and Streamlets to streamlets dir * Convert Streamlet/KVStreamlet into interfaces * moved window config to base * Moved all impl classes into impl * Made the Streamlet interface generics based * Added inner/left/outer joins * Add support for atleast once semantics by doing acking * Assign WindowConfig parameters to the WindowBolt * Made non windowed bolts stateful * Introducing Builder and keeping track of all children * Calculate children for kvstreamlets as well * Moved StreamletImpl and KVStreamletImpl to more intuitive locations * Moved groupings to their own dir * Adjusted the build into the new semantics * Introduced a Context class that allows us to run dsl streamlets * Added a createStreamlet interface to have an easy way to create Streamlets * WordCount Topology using Java DSL. * Fixed logic while building * Added SerializableSupplier concept to make Supplier Spouts Serializable * Introduced SerializablePredicate * Added SerializableBinaryOperator * Added SerializableBiFunction * Introduced SerializableFunction * Made KeyValue Serializable. With this the word count topology is now running * Added a trivial sink log * Reduced Operations should return WindowInformation * Updated java docs * Added JavaDocs for groupings * Added JavaDocs to spouts * Added Javadocs for all streamlet implementations * Added a ContextImpl class and made Context an interface * Added WindowConfigImpl to hide implementation details * Renamed as per feedback * Fixed build * Took feedback * Took feedback * Took feedback * Took feedback * Added a generic to function as a sink * Renamed spouts to sources and bolts to operators * Added sinks folder * Moved Spout references to Source * Moved Bolts to operators * Renamed Bolts to Sinks * Renamed Impl to Base * Took Bill's suggestion * Renamed Context as Runner * Added a transform operator that gives users more flexible transforms * Renamed to newSource * Renamed window functions based on feedback * Renamed to toSink * added count info to windows * Added check for name * Fixdd misplaced comments * Fixed comments * Refactored common code to single place * Took feedback into account * Fixed comments * Took feedback * Moved the examples to the newly formed example * Added unittest for BaseStreamlet * Added unittest for KVBaseStreamlet 20 September 2017, 05:48:20 UTC
9a4d39e upgrade to protobuf3.4 (#2316) 19 September 2017, 20:40:44 UTC
a507947 take into account feedback from Neng (#2322) 19 September 2017, 18:06:46 UTC
3d8aed1 clean tmater client (#2321) * clean tmater client 19 September 2017, 17:15:11 UTC
a0d80a3 Issue 2219: Provide a bookkeeper based uploader and downloader (#2220) * BK Uploader * Add a BK downloader * remove local repo * Add all dependencies * Fix * Fix errors after bumping guava to 20 * Add more tests * Address checkstyle problems * fix typo * Move heron/dlog to `heron/io/dlog` * Use `shaded` distributedlog jar * Include dlog uploader in apiserver and tools * Allow configuring num-replicas for topologies stored in bookkeeper * Remove redundant dependency * Revert "Use `shaded` distributedlog jar" This reverts commit c63ffbb9034d50211a649cbdca4859b6e079de95. * Revert "Revert "Use `shaded` distributedlog jar"" This reverts commit 9595aca4e62dcc129def11309b06947e5bda0301. * use the new shaded jar * update * Provide a dlog tool for verifying downloader and uploader * Add a README instruction for how to use the dlog tool for verifying downloader and uploader * Disable write lock * Use latest 0.5.0 release * Fix the guava dependency in tests * Fix checkstyle problems * private constructor * Revert the Extractor from singleton to static methods * Make extractor final * Fix checkstyle 19 September 2017, 05:22:02 UTC
dd5f2d7 fix python state manager (#2319) * heron executor exits when statemanager raises exception 19 September 2017, 05:12:38 UTC
481064e Added a Stateful example topology. (#2318) * Added a Stateful example topology. Also fixed a bug while running this * Added the example topology 19 September 2017, 04:39:37 UTC
f00a9f1 Add prometheus metrics web sink. (#2243) * Add prometheus metrics web sink. Provides a simple web server that exposes and endpoint for prometheus to scrape metrics. * revert local metrics_sinks config and remove commented out code. * Remove instance id from stmgr metrices when exporting to prometheus. * Revert local metrics_sink.yaml config. * Rename Metrics to MetricsUtil. 19 September 2017, 03:07:40 UTC
71676ab Seperate Stmgr and Instance Serving in Stmgr (#2313) * Refactored stmgr and instance related communication to seperate classes * First compiling version * Tests compile * More compile time fixes * Pass local data port to stmgr * ensure that we preserve backwards compat with metrics 19 September 2017, 00:58:16 UTC
9ff9889 Upgrade pex (#2314) * upgrade pex in place 18 September 2017, 20:21:07 UTC
f3ea101 Add image pull policy param for kubernetes scheduler. (#2315) 18 September 2017, 05:17:22 UTC
f87d05e Always show links button (logs, job, exceptions, etc..) for instances. (#2312) 15 September 2017, 20:12:00 UTC
2f65be5 Use bufferevent to do Network I/O (#2297) 14 September 2017, 19:12:38 UTC
2d96ddc Fixed the path of the python examples (#2305) 14 September 2017, 06:07:40 UTC
1a4dd55 Bug fixes and integration test for Python DSL (#2304) * Bug fixes and integration tests for python dsl * cleaning up * addressing comments 14 September 2017, 06:07:23 UTC
b12c7e6 fix bugs in stmgr unit test (#2306) * fix stmgr unit test * fix reconnect timer 14 September 2017, 03:57:31 UTC
b06fd91 Split executor args in aurora file so easy to read diff. (#2303) 13 September 2017, 21:18:06 UTC
3843924 Refactoring storm api (#2300) * refactoring storm compatability code and api - Moved all the Storm API to storm-compatibility/ - Modified heron/examples to use storm api only and move to storm-compatibility-examples/. Also removed all dependencies of the twitter API from these examples - Modified heron/examples to use the heron api only and moved to examples/ - Made sure the correct headers are on the storm api related files * fixing storm compatibility api paths * adding back logic to use simulator * fixing examples paths 12 September 2017, 21:48:34 UTC
back to top