https://github.com/twitter/heron

sort by:
Revision Author Date Message Commit Date
6de734e Currently heron-instances exports most of metrics scoped by different streams. But a lot of customers expressed their interest on the value of all streams aggregated and for now they have to sum them up by themselves. This pull request exports metrics aggregated on differetn streams too. Tested with LocalScheduler. 29 June 2017, 00:37:50 UTC
1687703 Expose TopologyContext in heronpy (#2014) 28 June 2017, 06:46:30 UTC
2bf4626 Moved Herons pypi to heronpy. Updated requirements as well (#2012) 28 June 2017, 05:33:54 UTC
5d3f0fa add kill-executor handler (#1997) * add kill executor handler * new line at the end * remove unused import * remove unused import 28 June 2017, 04:33:58 UTC
12a79ec add pmap to shell (#1995) * add pmap to shell * add PmapHandler to init 28 June 2017, 00:36:35 UTC
c4ba267 Remove pyheron since now python is first class api in Heron (#2006) 27 June 2017, 22:28:29 UTC
4fb6622 Fix applatix scripts (#2010) * fix the docker images for source code checkout * fixed the applatix scripts to pass the ci environ 27 June 2017, 20:46:43 UTC
932331f fix the docker images for source code checkout (#2009) 27 June 2017, 20:39:31 UTC
5f39435 changes for applatix ci (#2008) 27 June 2017, 20:33:48 UTC
330f33a rename tools/travis-ci to tools/travis and update the references to it (#2007) 27 June 2017, 20:33:08 UTC
05d3bc1 Restart subprocess with shell_env. (#2004) 27 June 2017, 20:01:04 UTC
77035b4 changed travis scripts to take an input arg that indicates the CI env (#2003) 27 June 2017, 19:13:32 UTC
38c4552 Misc fixes in Python. Remove pyheron in docs plus remove common dep from examples (#2002) 27 June 2017, 15:16:30 UTC
9b65d5b Discover the platform in travis scripts (#2001) * applatix parallel builds * Autodiscover the platform on travis and accordingly pass it to bazel * move the discover platform code to common.sh * invoke discover_platform function, print the platform 27 June 2017, 15:05:56 UTC
cad3e1a change the default gcc 4.8 to just gcc (#1999) * applatix parallel builds * added applatix for parameter os * revert the change to applatix scm checkout container * remove the space in applatix array parameters * changed the defaults of gcc-4.8 to just gcc 27 June 2017, 05:06:47 UTC
0ffef32 remove the space in applatix array parameters (#1998) * applatix parallel builds * added applatix for parameter os * revert the change to applatix scm checkout container * remove the space in applatix array parameters 27 June 2017, 04:54:07 UTC
d9d4e69 Made heron instance and examples depend on api and not on pyheron (#1992) * Made heron instance and examples depend on api and not on pyheron * Changed the name of the topology bin for integration tests 27 June 2017, 04:15:20 UTC
18b9f05 Fix applatix parameter os (#1996) * applatix parallel builds * added applatix for parameter os * revert the change to applatix scm checkout container 27 June 2017, 02:48:00 UTC
1df8c7e use of array parameters in applatix (#1994) * applatix parallel builds * use applatix containers * use array parameters 27 June 2017, 02:40:39 UTC
5acd537 applatix builds parallely for centos7 and ubuntu14.04 (#1993) * applatix parallel builds * use applatix containers 27 June 2017, 01:35:19 UTC
bf56c75 change the applatix docker image (#1991) 26 June 2017, 23:38:23 UTC
9905895 Applatix CI integration (#1986) * applatix ci integration * install git in build container * remove Dockerfile.build and build-docker-build.sh 26 June 2017, 23:20:36 UTC
d70bf37 Made pyheron independent of common. It should only depend on the api (#1990) * Made pyheron independent of common. It should only depend on the api * Added common dep to integration test because pyheron no longer has it 26 June 2017, 23:04:31 UTC
c4fe05f Add support for src task id in the storm api (#1938) * Added src task id information of tuples flowing between stmgrs and between stmgrs and instance * StreamManager changes to support src_task_id. We now break tuple batches based on src_task_ids * Heron-instance changes to support src_task_id. It can pass the compliation. But unit tests will fail. Heron-simulator has not been changed yet. So src_task_id will not work in heron-simulator. * Fix a NPE in heron-instance * Added the support of src_task_id in heron-simulator. Fixed all unit tests and coding style. * Set src_task_id in python instance as well 26 June 2017, 19:39:59 UTC
1ca533b Kubernetes Manual Scaling (#1970) * Initial configuration docs for kubernetes deployment * Changed docker image example to streamlio * Support for manual scaling using update command * Convert back to getScheduledResource * Changed controller methods to protected * Added a configuration for namespaces * Refactoring of controller code * Further refactoring of kubernetes controller. Added utility for talking to json-based scheduler apis. * Added default namespace * Throwing exception in case of not being able to read response code * Moved api client to common package. Refactored to rely more on exception logic * Style update * Using base path in controller to cut down on duplication * Another quick optimization to base path * Added tests for Kubernetes Controller * Added tests for adding and removing containers in k8s scheduler * Removed kubebrnetesURI from class variables * Removed one-liner * Removing some redundant logging * Added some clarification comments and logging * Fixing a build issue * Added some more comments and documentation. Consolidated some error logging into an exception * Small javadoc updates * Removed javadoc returns * Style fixes * Typo fix * Small bug fix for getting needed container resources. Bug fix for print statement counter 26 June 2017, 19:38:37 UTC
8185a00 Use TopologyContext as the base for TopologyContextImpl (#1989) 25 June 2017, 18:44:26 UTC
4f5f75c Refactored constants (#1988) * Refactored constants * Fixed constants references in tracker * Fixed constants * Fixed build 25 June 2017, 07:40:06 UTC
f08e67a Remove CustomGrouping and ITaskHooks from common.misc since they are now in the api (#1987) * Removed task hooks and custom grouping defn since they are duplicates * Fixed merge 24 June 2017, 05:44:14 UTC
4c50a8b Removed heron/common serializer and metrics and use heron/api defn (#1985) * Removed heron/common serializer and metrics and use heron/api defn * Fixed path * Fixed path * Fixed path * Fixed path * Fixed path * Fixed path * Fixed imports * Fixed path * Corrected path * Removed unnecessary exports * Fixed paths * Fixed path * Fixed paths 24 June 2017, 00:34:31 UTC
9c6e3af Add GcsUploader for uploading topologies to Google Cloud Storage. (#1976) * Add GcsUploader for uploading topologies to Google Cloud Storage. * Fix failing unit test. * Change method names in controller and remove uploader subclass in tests. * Throw real excpetions from createGcsController method. * Fix javadoc comment. * Add google_client_version variable to workspace. 23 June 2017, 22:24:52 UTC
c1ec2a0 Heron client unit test (#1983) 23 June 2017, 21:08:59 UTC
89de141 Rephrase references to Storm (#1979) * Rephrase references to Storm * update overview, remove storm reference * Updating missed links 23 June 2017, 15:59:13 UTC
2a8f465 Native Python API for Heron. (#1982) * Native Python API for Heron. * Copied unittests over 23 June 2017, 13:14:15 UTC
5dfd4d6 add checking server starting return (#1984) 23 June 2017, 05:14:38 UTC
aee7824 Change TMaster to watch the packing plan for changes (#1955) * TMaster to fetch packing plan from state manager * fix typo * updating the stream manger tests * fixing executor test and only killing packing plan if we're confident we're master * We need to wait before reactivating * Change init sequence to fetch not register packing plan watch until we know we're master * need to future chain fetching packing plan and becoming tmaster 21 June 2017, 17:53:16 UTC
f544344 Added get topology name to context (#1978) 20 June 2017, 21:04:23 UTC
a071204 Pex files need to be unzipped so that depdencies can be set/found (#1977) * Pex files need to be unzipped so that depdencies can be set/found * Fixed unittest 20 June 2017, 14:49:57 UTC
d44d591 import metrics from pyheron module (#1975) 20 June 2017, 04:09:11 UTC
b2f5d26 added the python requirements to be >= (#1974) 19 June 2017, 20:43:02 UTC
6be67ce change reqs pyyaml to PyYAML (#1972) 18 June 2017, 02:05:19 UTC
d41971e Allow the tracker to take command line args for a state manager. (#1967) 16 June 2017, 23:14:31 UTC
adc2020 Fix inaccurate execute-latency metric (#1969) * Fix inaccurate execute-latency metric Currently in bolt instance, we would use an inaccurate start of time, causing the execute-latency higher than expected, especially when the actual execute-latency is small. This pull request fixes this issue by recording a more accurate start of time. 16 June 2017, 21:54:58 UTC
343d56e Changed the python requirements for pyheron so that it doesn't force to downgrade (#1966) 16 June 2017, 06:37:52 UTC
4cdd068 fix a typo (#1968) 16 June 2017, 01:27:35 UTC
0a2bce3 Topic Real Time Streaming does not exist as list of allowed Topic Names (#1965) 15 June 2017, 22:14:42 UTC
ee7c8bf Streamline preparation of docker images for distribution (#1961) * Scripts to build docker artifacts * add curl to docker images 15 June 2017, 11:46:15 UTC
325c06d add stateful restorer (#1940) 15 June 2017, 01:27:55 UTC
dd59676 add stateful checkpointer into tmaster (#1929) 14 June 2017, 23:56:40 UTC
639f4cc Remove extra initialize method from ContainerFileDownloadHandler. (#1962) 14 June 2017, 21:37:43 UTC
61f18be Allow any type of file to be downloaded from a container through the shell. (#1959) 14 June 2017, 20:20:48 UTC
8084402 Increase delta in Python loopers test (#1958) 14 June 2017, 20:17:19 UTC
7546124 Fix ui download file handler. (#1949) Adds a constructor that accepts a baseUrl. 14 June 2017, 18:23:46 UTC
da5e320 Add support for watching packing plan to c++ state managers (#1953) 14 June 2017, 00:15:16 UTC
2a8ae0c Fix typos and improve log messages (#1954) 13 June 2017, 23:49:39 UTC
9fa3fad print bazel log directory for apps (#1950) * print bazel log directory for apps * Create stmgr_unittest.cpp * Create tmaster_unittest.cpp * fix style 13 June 2017, 18:24:24 UTC
f5790ca Scripts to build docker artifacts (#1948) 10 June 2017, 20:41:30 UTC
393c28e add note on hostname in /etc/hosts (#1945) 10 June 2017, 00:16:08 UTC
41e6f38 add a readme file for pyheron (#1944) * add a readme file for pyheron * added the URL for streamparse 08 June 2017, 21:23:48 UTC
babd3f9 Update documentation regarding topology_bin in the SubmitterMain (#1939) * Cli passes only the file name of the topology_bin. Thus it's not necessary for the schedulers to get the basename. * Removed FileUtil since it's no longer used * Fixed the unittest * Fixed comments based on feedback 08 June 2017, 21:01:13 UTC
16a8f25 Kubernetes Scheduler Support (#1928) 08 June 2017, 17:31:58 UTC
6d5f59b UpdateTopologyManager using stale topology (#1937) * UpdateTopologyManager using stale topology * Update UpdateTopologyManager to pull Topology again after deactivating topology * Refactor deactivateTopology to return new Topology * deactivateTopology() now returns new Topology * Modified UpdateTopologyManagerTest to allow build 08 June 2017, 17:04:13 UTC
d0da9fa Moved the docker builds and docker images to use open-jdk-8 (#1941) 08 June 2017, 05:50:39 UTC
f1dfcc9 Add simple script to compile heron and package it in a docker image. (#1934) * Add simple script to compile heron and package it in a docker image. The image created can be used for running heron topologies in environments that support docker containers (DCOS, Kubernetes, etc..). * Add docker image building to the build artifacts script. * Remove platform check when building docker image. * Remove check from setup_scratch_dir(). * Rename run_build_docker to build_docker_image. 08 June 2017, 00:00:08 UTC
c2fda3b Remove heron core fetching for MarathonScheduler. (#1894) * Remove heron core fetching for MarathonScheduler. The heron core will be installed within the docker image. * Fix style error in MarathonScheduler. * Remove heron core fetching for MarathonScheduler. The heron core will be installed within the docker image. * Fix style error in MarathonScheduler. * Fix ubuntu spelling mistake in website dcos docs. 07 June 2017, 23:58:16 UTC
24dd431 When the python pex file is copied over to the container, only (#1926) the base name should be referred 07 June 2017, 21:55:16 UTC
858dd7d Python packaging (#1425) * pyheron packaging * python packaging * python packaging * python packaging * Fixed BUILD and requirements * version is set to 0.0.0 for pyheron if you are building out of a branch * removed the backed up BUILD file * python packaging using wheel files * upgrade to python 2.7.10 in travis * upgrade python 2.7 to 2.7.10 * use different command to upgrade to 2.7.10 in travis * switch the order of packages sources * add sudo pip install * print python versions * python packaging * remove pyheron library creation script * setup.py is automatically generated based on the version 07 June 2017, 19:13:04 UTC
adbb9ee UpdateTopologyMaster using stale Topology from initial submission (#1924) * UpdateTopologyManager to use Topology from PhysicalPlan * UpdateTopologyManager to use Topology from PhysicalPlan * Adding comments 05 June 2017, 23:36:24 UTC
6dcefbc Adding sleep to TestSpout to throttle down cpu usage (#1925) 05 June 2017, 23:35:54 UTC
efdee1f Fixing logging bug in UpdateTopologyManager (#1931) 05 June 2017, 22:04:47 UTC
b8616e9 Fix heron-shell dependency (#1930) 05 June 2017, 21:36:01 UTC
53048df Decode serialized value in heron-tracker. (#1927) 02 June 2017, 21:32:45 UTC
3a9af24 Asyncly handle subprocess stdout/stderr in client. (#1923) 01 June 2017, 17:36:51 UTC
97c7c40 Removed per client/server specific acquire/release and consolidated them in one place (#1914) 31 May 2017, 20:08:06 UTC
3c3b176 Add back missing arguments. (#1921) 31 May 2017, 17:57:08 UTC
399cbdd Remove unused heron.instance.slave.fetch.pplan.interval.sec (#1920) 31 May 2017, 00:09:16 UTC
88391ba Clean up exception handling in common tests (#1919) 30 May 2017, 21:56:25 UTC
42a6bb1 Remove unused heron.instance.state.check.interval.sec config (#1916) 30 May 2017, 21:56:11 UTC
0568bfa update STATEFUL_YAML to STATEFUL_CONFIG_YAML (#1917) 30 May 2017, 21:25:44 UTC
db9e08a Add authentication for Marathon scheduler (#1871) 30 May 2017, 17:35:53 UTC
e242f53 Override stmgr max packet size config (#1912) 30 May 2017, 17:29:55 UTC
554ccc9 Update grouping tests to not use Thread.sleep() (#1909) * Update grouping tests to not use Thread.sleep() * Update grouping tests to not use Thread.sleep() * Update grouping tests to not use Thread.sleep() 30 May 2017, 17:15:25 UTC
8e5c6dc Refactor network tests to use NIO selectors instead of sleep() (#1910) * Refactor network tests to use NIO selectors instead of sleep() * Refactor network tests to use NIO selectors instead of sleep() 30 May 2017, 17:15:06 UTC
b5de046 Fixed render issue for max back pressure (#1913) 28 May 2017, 05:36:35 UTC
990ae5e Added helper methods to physial plan helper (#1905) 27 May 2017, 03:56:23 UTC
9fd3da2 Add API support for backpressure metric (instance level) & display backpressure (instance level) in Heron UI (#1791) 27 May 2017, 00:45:05 UTC
6858133 Removes the last 2 instances of Thread.sleep() from the tests (#1911) * refactor metrics manager tests to not use THread.sleep * Also refactoring MetricsCacheSinkTest * Remove the last 2 instances of Thread.sleep() * fix javadocs * merge from master 26 May 2017, 23:14:18 UTC
6d81d8c Refactor BoltInstanceTest and SpoutInstanceTest to not use Thread.sleep() (#1900) * Add SlaveTester to centralize duplicate slave, looper, communicator test logic * Decoupled CommunicatorTester from SlaveTester * consolidate duplicate initialization logic * Removing sleeps from bolt/spout tests, clean up communicator * Moving field into Constants and removing unused Constant values * Update AbstractNetworkTest to use the new CommunicatorTester constructor * Move countdown latch past the static increment logic * Make expectedTuples final 26 May 2017, 20:52:01 UTC
f64afa0 Merge pull request #1865 from huijunwu/order_unittest improve order_unittest 26 May 2017, 17:55:56 UTC
093278e Merge branch 'master' of https://github.com/twitter/heron into order_unittest 26 May 2017, 17:50:36 UTC
07067e6 fix unit test (#1906) 26 May 2017, 17:46:49 UTC
cc5a5b9 add ckptmgr client in tmaster (#1901) 26 May 2017, 00:59:10 UTC
f71ada3 add ckptmgr client in stmgr (#1902) 26 May 2017, 00:58:49 UTC
a9cf457 Added helper methods for stateful operations (#1898) * Added helper methods for stateful operations * Took feedback into consideration * Use common method to elminate repetive code 26 May 2017, 00:24:47 UTC
233b54f replace 5 seconds to const val 26 May 2017, 00:24:05 UTC
595e709 Refactor common netwrok test code into a superclass (#1899) * Add SlaveTester to centralize duplicate slave, looper, communicator test logic * Decoupled CommunicatorTester from SlaveTester * Refactoring out duplicate code in network tests * consolidate duplicate initialization logic * fix typo * clean-up per feedback 26 May 2017, 00:12:28 UTC
d60c52c cleanup stateful configs and statefulstorage package (#1903) 25 May 2017, 23:23:57 UTC
9ea9e85 Add SlaveTester to centralize duplicate slave, looper, communicator test logic (#1893) * Add SlaveTester to centralize duplicate slave, looper, communicator test logic * Decoupled CommunicatorTester from SlaveTester * consolidate duplicate initialization logic 25 May 2017, 19:53:21 UTC
a4b9da2 refactor metrics manager tests to not use Thread.sleep (#1887) * refactor metrics manager tests to not use THread.sleep * Also refactoring MetricsCacheSinkTest 25 May 2017, 19:08:30 UTC
f8a5cf9 Another attempt to fix flakey HeronServerTest (#1897) 25 May 2017, 18:04:24 UTC
4bdaab3 Added config variable/helper function that govern the amount of (#1896) data stmgr can buffer waiting for checkpoint markers before giving up 25 May 2017, 17:43:28 UTC
back to top