https://github.com/twitter/heron

sort by:
Revision Author Date Message Commit Date
f78c7ed remove the hard coded path in dsl 10 July 2017, 05:23:28 UTC
365cef6 package python dsl package (#2043) 10 July 2017, 03:33:17 UTC
9dddbd8 move scripts and tools around (#2041) 08 July 2017, 23:43:37 UTC
4d4b173 Python DSL (#2033) * Added first cut of python dsl * Added first cut of text input * Made everything a Streamlet * Added BUILD file and did the first set of fixes * Added MapBolt * Added FlatMapBolt * Added FilterBolt and FlatMapBolt * Added partial sample bolt * Added JoinBolt and JoinGrouping * Added RepartitionBolt * Made everything TIMEWINDOW * Join now support sliding interval * Added ReduceByWindow Bolt * Switched everything to reducebykeyandwindow bolt * Fixed all compilation issues * Added StaticLines Streamlet * Merged spouts inside the streamlet files * Changed function name * Added a dsl example * reduce takes timewindow * Fixed the format of Stream constructor * Fixed bugs while composing dag * Fixed more bugs * Fixed the way to specify custom grouping * Fixed generation of stage names * Specify classpath rather than instances * Fixed the Reduce/Join Custom grouping * Fixed logic for join/reduce tuple * Fixed the format of output tupled * Addressed review comments 08 July 2017, 21:21:38 UTC
afbcfab Extract HeronTuple from current_tuples while acking it (#2032) * Current tuples is a tuple of (HeronTuple, time). While acking we need to extract the HeronTuple * Do not reuse tm * Ignore itime 07 July 2017, 08:44:48 UTC
e032dee Applatix workflow with build followed by test (#2015) * applatix parallel builds * try test build followed by test workflow * change the container to applatix container * add the actual test * use artifacts for test containers * map the output to input * fix applatix errors * debugging applatix * update the path to be applatix * due to unicode from slack typed the entire command * moved common.sh to only dir above and change where it is being used * bind the user variable * move save-logs.py one level above * remove tar, zip files to make artifact transfer faster. Include tests * export scripts and its subdirectories * generate test package * include the path in test.sh * cleanup the travis and applatix scripts * fix spelling mistake * parallelize language tests * fix applatix yaml to invoke the appropriate test script * remove starting of http server twice in tests 06 July 2017, 22:23:10 UTC
adfea16 CustomGrouping can be either a CustomGrouping instance or a class path of that instance (#2031) 06 July 2017, 18:07:40 UTC
0ccb2c5 Made cloudpickle the default serializer (#2030) * Added cloudpickler for better serialization * remove style check warnings * orrected name * Removed pyspark stuff 05 July 2017, 14:36:39 UTC
a9bdab8 Window tuples should take care of handling acks (#2028) 04 July 2017, 03:06:40 UTC
88a73c9 Changed the custom grouping to take in an actual instance of (#2027) ICustomGrouping than a class path. 02 July 2017, 13:37:34 UTC
ed2451a Check if any unshaded third party libraries are introduced into `heron-instance.jar` (#2023) 30 June 2017, 21:31:02 UTC
4732947 Added WindowBolt interface in python (#2021) * Added window bolt to python api * Added a sample topology for WindowBolt * Added Tumbling Window Bolt * Changed WindowInfo to WindowContext * Changed copyright 30 June 2017, 20:33:51 UTC
1511845 Added Checkpoint Gateway (#2024) * Added Checkpoint Gateway, a class that buffers tuples until upstream markers have all been received. * Took Neng's feedback into consideration 30 June 2017, 19:52:06 UTC
af932a0 add security check for shell-kill-executor (#2013) * add security check for shell-kill-executor * remove unused code * update log * add space in test 30 June 2017, 19:47:47 UTC
6fefa59 set init value getTotalMemoryUsed (#2026) 30 June 2017, 19:40:10 UTC
e819e17 flatten the directory structure for integration tests (#2018) * flatten the directory structure for integration tests * flatten the directory structure for python integration test topologies * changing path names using integration-test to integration_test and also ensure module imports are correct * flatten the python integration tests and corresponding changes * package tests into tarballs * package tests to self extractable binary * Change travis script to use packaged integration tests * after installation, we should invoke test-runner instead of test-runner.pex 30 June 2017, 18:30:49 UTC
043548e Updated Python documentation (#2017) * modified CSS and python docs content * modify doc gen script to use PyPI pyheron package instead of local sources * fix issue in custom grouping topology BUILD file * fix pydoc gen script once again * fix scrolling offset issue * some additions to the python docstrings * more re-org for the python topologies doc * update link to bazel dev environment * remove applatix dir from this branch * merge applatix material with master * undelete mistakenly removed kubernetes doc * undo mistaken changes to graphite doc * merge applatix scripts from master * undo more extraneous changes * remove errant parens in config.yaml * remove unnecessary ls from python doc gen script * remove references to streamparse * change upgrade language to migrate * change booleans to uppercase for python * finish descriptions for example topologies * s/pyheron/heronpy * replace raw version numbers with shortcodes * fix broken shortcode reference * small commit to re-run CI * fix python lint issues * change website build script * add pdoc install to setup script * upgrade to new version of hugo * remove website generation from CI * fix hugo installation command in .travis.yml * another adjustment to hugo install URL * remove references to streamparse in python docs 30 June 2017, 01:41:59 UTC
a2967c8 add libtool-bin in doc compiling on linux (#2025) * add libtool-bin * add setuptools 30 June 2017, 00:11:55 UTC
bd7bb2d add shard number to log name (#2022) * add shard number to log name * fix unit test * update func name 29 June 2017, 22:18:49 UTC
9ae5739 clean common dependency (#2019) 29 June 2017, 21:07:55 UTC
673ed2c limit the mem pool size (#2011) 29 June 2017, 17:45:34 UTC
089cff0 Added Neighbour Calculator, a utility class to calculate (#2016) * Added Neighbour Calculator, a utility class to calculate upstream and downstream components on a per task basis * neighbour calculator unittest is not flaky * Moved neighbour calculator from manager to util 29 June 2017, 03:38:39 UTC
1687703 Expose TopologyContext in heronpy (#2014) 28 June 2017, 06:46:30 UTC
2bf4626 Moved Herons pypi to heronpy. Updated requirements as well (#2012) 28 June 2017, 05:33:54 UTC
5d3f0fa add kill-executor handler (#1997) * add kill executor handler * new line at the end * remove unused import * remove unused import 28 June 2017, 04:33:58 UTC
12a79ec add pmap to shell (#1995) * add pmap to shell * add PmapHandler to init 28 June 2017, 00:36:35 UTC
c4ba267 Remove pyheron since now python is first class api in Heron (#2006) 27 June 2017, 22:28:29 UTC
4fb6622 Fix applatix scripts (#2010) * fix the docker images for source code checkout * fixed the applatix scripts to pass the ci environ 27 June 2017, 20:46:43 UTC
932331f fix the docker images for source code checkout (#2009) 27 June 2017, 20:39:31 UTC
5f39435 changes for applatix ci (#2008) 27 June 2017, 20:33:48 UTC
330f33a rename tools/travis-ci to tools/travis and update the references to it (#2007) 27 June 2017, 20:33:08 UTC
05d3bc1 Restart subprocess with shell_env. (#2004) 27 June 2017, 20:01:04 UTC
77035b4 changed travis scripts to take an input arg that indicates the CI env (#2003) 27 June 2017, 19:13:32 UTC
38c4552 Misc fixes in Python. Remove pyheron in docs plus remove common dep from examples (#2002) 27 June 2017, 15:16:30 UTC
9b65d5b Discover the platform in travis scripts (#2001) * applatix parallel builds * Autodiscover the platform on travis and accordingly pass it to bazel * move the discover platform code to common.sh * invoke discover_platform function, print the platform 27 June 2017, 15:05:56 UTC
cad3e1a change the default gcc 4.8 to just gcc (#1999) * applatix parallel builds * added applatix for parameter os * revert the change to applatix scm checkout container * remove the space in applatix array parameters * changed the defaults of gcc-4.8 to just gcc 27 June 2017, 05:06:47 UTC
0ffef32 remove the space in applatix array parameters (#1998) * applatix parallel builds * added applatix for parameter os * revert the change to applatix scm checkout container * remove the space in applatix array parameters 27 June 2017, 04:54:07 UTC
d9d4e69 Made heron instance and examples depend on api and not on pyheron (#1992) * Made heron instance and examples depend on api and not on pyheron * Changed the name of the topology bin for integration tests 27 June 2017, 04:15:20 UTC
18b9f05 Fix applatix parameter os (#1996) * applatix parallel builds * added applatix for parameter os * revert the change to applatix scm checkout container 27 June 2017, 02:48:00 UTC
1df8c7e use of array parameters in applatix (#1994) * applatix parallel builds * use applatix containers * use array parameters 27 June 2017, 02:40:39 UTC
5acd537 applatix builds parallely for centos7 and ubuntu14.04 (#1993) * applatix parallel builds * use applatix containers 27 June 2017, 01:35:19 UTC
bf56c75 change the applatix docker image (#1991) 26 June 2017, 23:38:23 UTC
9905895 Applatix CI integration (#1986) * applatix ci integration * install git in build container * remove Dockerfile.build and build-docker-build.sh 26 June 2017, 23:20:36 UTC
d70bf37 Made pyheron independent of common. It should only depend on the api (#1990) * Made pyheron independent of common. It should only depend on the api * Added common dep to integration test because pyheron no longer has it 26 June 2017, 23:04:31 UTC
c4fe05f Add support for src task id in the storm api (#1938) * Added src task id information of tuples flowing between stmgrs and between stmgrs and instance * StreamManager changes to support src_task_id. We now break tuple batches based on src_task_ids * Heron-instance changes to support src_task_id. It can pass the compliation. But unit tests will fail. Heron-simulator has not been changed yet. So src_task_id will not work in heron-simulator. * Fix a NPE in heron-instance * Added the support of src_task_id in heron-simulator. Fixed all unit tests and coding style. * Set src_task_id in python instance as well 26 June 2017, 19:39:59 UTC
1ca533b Kubernetes Manual Scaling (#1970) * Initial configuration docs for kubernetes deployment * Changed docker image example to streamlio * Support for manual scaling using update command * Convert back to getScheduledResource * Changed controller methods to protected * Added a configuration for namespaces * Refactoring of controller code * Further refactoring of kubernetes controller. Added utility for talking to json-based scheduler apis. * Added default namespace * Throwing exception in case of not being able to read response code * Moved api client to common package. Refactored to rely more on exception logic * Style update * Using base path in controller to cut down on duplication * Another quick optimization to base path * Added tests for Kubernetes Controller * Added tests for adding and removing containers in k8s scheduler * Removed kubebrnetesURI from class variables * Removed one-liner * Removing some redundant logging * Added some clarification comments and logging * Fixing a build issue * Added some more comments and documentation. Consolidated some error logging into an exception * Small javadoc updates * Removed javadoc returns * Style fixes * Typo fix * Small bug fix for getting needed container resources. Bug fix for print statement counter 26 June 2017, 19:38:37 UTC
8185a00 Use TopologyContext as the base for TopologyContextImpl (#1989) 25 June 2017, 18:44:26 UTC
4f5f75c Refactored constants (#1988) * Refactored constants * Fixed constants references in tracker * Fixed constants * Fixed build 25 June 2017, 07:40:06 UTC
f08e67a Remove CustomGrouping and ITaskHooks from common.misc since they are now in the api (#1987) * Removed task hooks and custom grouping defn since they are duplicates * Fixed merge 24 June 2017, 05:44:14 UTC
4c50a8b Removed heron/common serializer and metrics and use heron/api defn (#1985) * Removed heron/common serializer and metrics and use heron/api defn * Fixed path * Fixed path * Fixed path * Fixed path * Fixed path * Fixed path * Fixed imports * Fixed path * Corrected path * Removed unnecessary exports * Fixed paths * Fixed path * Fixed paths 24 June 2017, 00:34:31 UTC
9c6e3af Add GcsUploader for uploading topologies to Google Cloud Storage. (#1976) * Add GcsUploader for uploading topologies to Google Cloud Storage. * Fix failing unit test. * Change method names in controller and remove uploader subclass in tests. * Throw real excpetions from createGcsController method. * Fix javadoc comment. * Add google_client_version variable to workspace. 23 June 2017, 22:24:52 UTC
c1ec2a0 Heron client unit test (#1983) 23 June 2017, 21:08:59 UTC
89de141 Rephrase references to Storm (#1979) * Rephrase references to Storm * update overview, remove storm reference * Updating missed links 23 June 2017, 15:59:13 UTC
2a8f465 Native Python API for Heron. (#1982) * Native Python API for Heron. * Copied unittests over 23 June 2017, 13:14:15 UTC
5dfd4d6 add checking server starting return (#1984) 23 June 2017, 05:14:38 UTC
aee7824 Change TMaster to watch the packing plan for changes (#1955) * TMaster to fetch packing plan from state manager * fix typo * updating the stream manger tests * fixing executor test and only killing packing plan if we're confident we're master * We need to wait before reactivating * Change init sequence to fetch not register packing plan watch until we know we're master * need to future chain fetching packing plan and becoming tmaster 21 June 2017, 17:53:16 UTC
f544344 Added get topology name to context (#1978) 20 June 2017, 21:04:23 UTC
a071204 Pex files need to be unzipped so that depdencies can be set/found (#1977) * Pex files need to be unzipped so that depdencies can be set/found * Fixed unittest 20 June 2017, 14:49:57 UTC
d44d591 import metrics from pyheron module (#1975) 20 June 2017, 04:09:11 UTC
b2f5d26 added the python requirements to be >= (#1974) 19 June 2017, 20:43:02 UTC
6be67ce change reqs pyyaml to PyYAML (#1972) 18 June 2017, 02:05:19 UTC
d41971e Allow the tracker to take command line args for a state manager. (#1967) 16 June 2017, 23:14:31 UTC
adc2020 Fix inaccurate execute-latency metric (#1969) * Fix inaccurate execute-latency metric Currently in bolt instance, we would use an inaccurate start of time, causing the execute-latency higher than expected, especially when the actual execute-latency is small. This pull request fixes this issue by recording a more accurate start of time. 16 June 2017, 21:54:58 UTC
343d56e Changed the python requirements for pyheron so that it doesn't force to downgrade (#1966) 16 June 2017, 06:37:52 UTC
4cdd068 fix a typo (#1968) 16 June 2017, 01:27:35 UTC
0a2bce3 Topic Real Time Streaming does not exist as list of allowed Topic Names (#1965) 15 June 2017, 22:14:42 UTC
ee7c8bf Streamline preparation of docker images for distribution (#1961) * Scripts to build docker artifacts * add curl to docker images 15 June 2017, 11:46:15 UTC
325c06d add stateful restorer (#1940) 15 June 2017, 01:27:55 UTC
dd59676 add stateful checkpointer into tmaster (#1929) 14 June 2017, 23:56:40 UTC
639f4cc Remove extra initialize method from ContainerFileDownloadHandler. (#1962) 14 June 2017, 21:37:43 UTC
61f18be Allow any type of file to be downloaded from a container through the shell. (#1959) 14 June 2017, 20:20:48 UTC
8084402 Increase delta in Python loopers test (#1958) 14 June 2017, 20:17:19 UTC
7546124 Fix ui download file handler. (#1949) Adds a constructor that accepts a baseUrl. 14 June 2017, 18:23:46 UTC
da5e320 Add support for watching packing plan to c++ state managers (#1953) 14 June 2017, 00:15:16 UTC
2a8ae0c Fix typos and improve log messages (#1954) 13 June 2017, 23:49:39 UTC
9fa3fad print bazel log directory for apps (#1950) * print bazel log directory for apps * Create stmgr_unittest.cpp * Create tmaster_unittest.cpp * fix style 13 June 2017, 18:24:24 UTC
f5790ca Scripts to build docker artifacts (#1948) 10 June 2017, 20:41:30 UTC
393c28e add note on hostname in /etc/hosts (#1945) 10 June 2017, 00:16:08 UTC
41e6f38 add a readme file for pyheron (#1944) * add a readme file for pyheron * added the URL for streamparse 08 June 2017, 21:23:48 UTC
babd3f9 Update documentation regarding topology_bin in the SubmitterMain (#1939) * Cli passes only the file name of the topology_bin. Thus it's not necessary for the schedulers to get the basename. * Removed FileUtil since it's no longer used * Fixed the unittest * Fixed comments based on feedback 08 June 2017, 21:01:13 UTC
16a8f25 Kubernetes Scheduler Support (#1928) 08 June 2017, 17:31:58 UTC
6d5f59b UpdateTopologyManager using stale topology (#1937) * UpdateTopologyManager using stale topology * Update UpdateTopologyManager to pull Topology again after deactivating topology * Refactor deactivateTopology to return new Topology * deactivateTopology() now returns new Topology * Modified UpdateTopologyManagerTest to allow build 08 June 2017, 17:04:13 UTC
d0da9fa Moved the docker builds and docker images to use open-jdk-8 (#1941) 08 June 2017, 05:50:39 UTC
f1dfcc9 Add simple script to compile heron and package it in a docker image. (#1934) * Add simple script to compile heron and package it in a docker image. The image created can be used for running heron topologies in environments that support docker containers (DCOS, Kubernetes, etc..). * Add docker image building to the build artifacts script. * Remove platform check when building docker image. * Remove check from setup_scratch_dir(). * Rename run_build_docker to build_docker_image. 08 June 2017, 00:00:08 UTC
c2fda3b Remove heron core fetching for MarathonScheduler. (#1894) * Remove heron core fetching for MarathonScheduler. The heron core will be installed within the docker image. * Fix style error in MarathonScheduler. * Remove heron core fetching for MarathonScheduler. The heron core will be installed within the docker image. * Fix style error in MarathonScheduler. * Fix ubuntu spelling mistake in website dcos docs. 07 June 2017, 23:58:16 UTC
24dd431 When the python pex file is copied over to the container, only (#1926) the base name should be referred 07 June 2017, 21:55:16 UTC
858dd7d Python packaging (#1425) * pyheron packaging * python packaging * python packaging * python packaging * Fixed BUILD and requirements * version is set to 0.0.0 for pyheron if you are building out of a branch * removed the backed up BUILD file * python packaging using wheel files * upgrade to python 2.7.10 in travis * upgrade python 2.7 to 2.7.10 * use different command to upgrade to 2.7.10 in travis * switch the order of packages sources * add sudo pip install * print python versions * python packaging * remove pyheron library creation script * setup.py is automatically generated based on the version 07 June 2017, 19:13:04 UTC
adbb9ee UpdateTopologyMaster using stale Topology from initial submission (#1924) * UpdateTopologyManager to use Topology from PhysicalPlan * UpdateTopologyManager to use Topology from PhysicalPlan * Adding comments 05 June 2017, 23:36:24 UTC
6dcefbc Adding sleep to TestSpout to throttle down cpu usage (#1925) 05 June 2017, 23:35:54 UTC
efdee1f Fixing logging bug in UpdateTopologyManager (#1931) 05 June 2017, 22:04:47 UTC
b8616e9 Fix heron-shell dependency (#1930) 05 June 2017, 21:36:01 UTC
53048df Decode serialized value in heron-tracker. (#1927) 02 June 2017, 21:32:45 UTC
3a9af24 Asyncly handle subprocess stdout/stderr in client. (#1923) 01 June 2017, 17:36:51 UTC
97c7c40 Removed per client/server specific acquire/release and consolidated them in one place (#1914) 31 May 2017, 20:08:06 UTC
3c3b176 Add back missing arguments. (#1921) 31 May 2017, 17:57:08 UTC
399cbdd Remove unused heron.instance.slave.fetch.pplan.interval.sec (#1920) 31 May 2017, 00:09:16 UTC
88391ba Clean up exception handling in common tests (#1919) 30 May 2017, 21:56:25 UTC
42a6bb1 Remove unused heron.instance.state.check.interval.sec config (#1916) 30 May 2017, 21:56:11 UTC
0568bfa update STATEFUL_YAML to STATEFUL_CONFIG_YAML (#1917) 30 May 2017, 21:25:44 UTC
db9e08a Add authentication for Marathon scheduler (#1871) 30 May 2017, 17:35:53 UTC
back to top