https://github.com/apache/flink

sort by:
Revision Author Date Message Commit Date
e4d308d Commit for release 1.0.0 24 February 2016, 21:07:53 UTC
b94d99d [hotfix] Expose current watermark to Triggers This closes #1706 24 February 2016, 21:01:14 UTC
ceb6424 [FLINK-3435] [streaming] Proparly separate IngestionTime and EventTime - IngestionTime now only auto-generates watermarks - EventTime does not auto-generate ingestion watermarks This also removes the "ExecutionConfig.areTimestampsEnabled()" flag. This closes #1699 24 February 2016, 21:00:41 UTC
004eb00 [docs] Move state and windows guide to separate pages - Move "Windowing" to separate page - Make "Working with Time" sub page of "Windowing" - Fix link in "Working with Time" page - Move "Working with State" page to "Fault Tolerance" - Add Table of Contents to State Backends page - Hierarchical restart strategies sections - CEP => Event Processing (CEP) This closes #1705. 24 February 2016, 20:00:15 UTC
af2091c [FLINK-3499] [runtime] Clear ZooKeeper references on recovery This closes #1707. 24 February 2016, 19:56:00 UTC
be68b17 [hotfix] Add logging for RocksDB backup durations 24 February 2016, 14:55:39 UTC
cec7fbb [FLINK-3490] Bump Chill version to 0.7.4 This closes #1701 24 February 2016, 11:28:27 UTC
4e35d2c [FLINK-3483] [runtime-web] Bump dagre-d3 version to 0.4.17 24 February 2016, 11:05:28 UTC
eaa9050 [FLINK-3478] [runtime-web] Don't serve files outside of web folder This closes #1697 24 February 2016, 10:46:58 UTC
ac0135a [dist] Link plan visualizer to project website 24 February 2016, 10:32:14 UTC
0629e25 [FLINK-2021] Rework examples to use ParameterTool This closes #1581 23 February 2016, 18:34:47 UTC
c40cfd3 [FLINK-3315] Fix Slot Sharing in Streaming API This changes slot sharing settings to single method slotSharingGroup(String) on DataStream. Operations inherit the slot sharing group of the input if all input operations are in the same slot sharing group. The default slot sharing group is "default" this can also be explicitly set using slotSharingGroup("default"). This overrides the inheriting behaviour. This closes #1641. 23 February 2016, 17:06:13 UTC
dc4d147 [FLINK-3426] Fix JobManagerLeader[Re]ElectionTest.testleaderElection on Windows removed unused import use new Path().toUri() added missing braces This closes #1651. 23 February 2016, 16:50:57 UTC
7ddc078 [FLINK-3418] Don't run RocksDB copy utils in external process This was causing to many problems with security tokens and yarn. Now, let the RocksDB backup run in a thread but don't interrupt these Threads anymore on closing. The Threads will close themselves because the copy operation will fail because of a FileNotFoundException when the state directories are being cleaned up. This also removes the ExternalProcessRunner because it is not needed anymore and using it causes too many headaches. This closes #1687. 23 February 2016, 16:50:57 UTC
271a715 [hotfix][release script] make sure the quickstart depends on scala 2.10 23 February 2016, 14:44:15 UTC
eb19c67 [docs] Fix typos in jobmanager HA documentation 23 February 2016, 14:42:49 UTC
ff4fbf9 [FLINK-3425] FileOutputFormat closes outStream in case of failure This closes #1652. 23 February 2016, 14:20:08 UTC
ff8535c [FLINK-3464] [docs] Add SBT template documentation to the quickstart documentation This closes #1688. 23 February 2016, 14:20:08 UTC
6323ed4 [FLINK-3410] [restart] Choose NoRestart strategy if the number of retries is set to 0 Add test case This closes #1643. 23 February 2016, 14:20:08 UTC
77c1dee [FLINK-3433] AvroOutputFormat#readObject uses readFully() This closes #1666 23 February 2016, 14:20:07 UTC
8b29c14 [FLINK-3438] ExternalProcessRunner fails to detect ClassNotFound exception because of locale settings [FLINK-3438] Improved solution, no workaround [FLINK-3438] Change: a faulty process now causes a RuntimeException to be thrown This closes #1665. 23 February 2016, 14:20:07 UTC
939768a [FLINK-3460] [build] Set Flink dependencies in flink-streaming-connectors, flink-batch-connectors, cep, gelly and flink-ml modules to provided The flink-streaming-connectors all depend on flink-streaming-java in compile scope. This entails that this dependency is always pulled in, when you build a fat jar. By setting this dependency to provided, this will be avoided. Furthermore, the connectors are always used in conjunction with flink-streaming-java. This means that you will always have an explicit dependency import of flink-streaming-java in your build script. This allows you to run your program locally but it is also easy to exclude the dependency from being included in the fat jar by setting it to provided, as well. This closes #1683. 23 February 2016, 14:20:07 UTC
3b47ad2 [FLINK-3459] [build] Fix conflicting dependencies commons-collections, commons-beanutils and commons-beanutils-core The Hadoop dependencies have a dependency on commons-configuration which pulls in transitively the commons-collection, commons-beanutils and common-beanutils-core depedencies. Commons-beanutils and commons-collection contain classes which live in the same namespace. They are also binary compatible but not binary identical. This is a problem for the sbt assembly plugin which checks for binary identity. In order to solve the problem, we bump the commons-configuration version to 1.7 so that only commons-beanutils is pulled in. This is necessary, because the transitive promotion of dependencies of the shade plugin only excludes the commons-beanutils dependency only from the directly depending dependency. All parent dependencies won't have the exclusion. This is a problem for SBT which will pull the dependency as part of one of the parents, then. Moreover, we replace commons-beanutils by commons-beanutils-bean-collections which contains only the non-conflicting classes wrt commons-collections. This closes #1682. 23 February 2016, 14:20:07 UTC
b204888 [FLINK-3458] [build] Disable shade-flink execution in flink-shaded-hadoop The shade-flink execution of the parent pom caused the problem that the guava dependencies were relocated twice in the flink-shaded-hadoop jar. In order to avoid this, this patch disables the shade-flink execution. This closes #1681. 23 February 2016, 14:20:07 UTC
407c285 [FLINK-3416] [py] Support for spaces in flink path This closes #1674. 23 February 2016, 14:20:06 UTC
dedcd2d [runtime] Log restart strategy on job submission 23 February 2016, 13:54:09 UTC
1bb8013 [tests] Add log statements to JobManagerHAJobGraphRecoveryITCase Adds further log statements in order to improve debuggability of JobManagerHAJobGraphRecoveryITCase. Hope to help for debugging: - hpick://s3.amazonaws.com/archive.travis-ci.org/jobs/110095304/log.txt - https://s3.amazonaws.com/archive.travis-ci.org/jobs/110085371/log.txt 23 February 2016, 13:54:06 UTC
8626971 [FLINK-3440][Kafka 0.8] Commit also offsets retrieved from Kafka into the OffsetStore (ZK) This closes #1692 23 February 2016, 13:46:54 UTC
de21630 FLINK-2213: Makes the number of vcores per YARN container configurable. This closes #1588 23 February 2016, 08:56:36 UTC
1c48e34 [FLINK-3385] [runtime] Fix outer join skipping unprobed partitions This closes #1680 23 February 2016, 00:18:42 UTC
03923c3 [hotfix] [runtime] Fix processing unmatched build-side when all buckets spilled 23 February 2016, 00:18:18 UTC
8656ed5 [FLINK-3455] Bump Kafka to 0.9.0.1 and 0.8.2.2 This closes #1684 22 February 2016, 20:19:47 UTC
bfc14eb [FLINK-3450] Duplicate TypeSerializer in StateDescriptor.writeObject The StateDescriptor can be serializer asynchronously in case of asynchronous checkpoints. In that case two threads would try to concurrently use the TypeSerializer: The normal state updating and the checkpoint serialization. If the TypeSerializer is a KryoSerializer this can lead to problems. Therefore the need to duplicate it before using in "writeObject". 22 February 2016, 13:59:22 UTC
80c0c65 [FLINK-3439] Remove final Long.MAX_VALUE Watermark in StreamSource 22 February 2016, 09:46:23 UTC
9691d95 [FLINK-3453] [runtime] Fix TaskManagerTest stability 20 February 2016, 21:45:26 UTC
60d65ce [cleanup] remove RMQTopology file 20 February 2016, 18:13:37 UTC
9bdee1b [FLINK-3248] add constructor params and generic ConnectionFactory This adds more default constructor parameters to the RMQSource. In addition, users may override the setupConnectionFactory() method to return their onwn configured factory. This closes #1670. 20 February 2016, 18:13:37 UTC
7f9ade4 [FLINK-3453] [runtime, runtime-web] Report partial stack trace sample for cleared tasks This closes #1678. 20 February 2016, 18:02:56 UTC
5117bc6 [hotfox] Fix javadoc of getState to comply with constructor changes 20 February 2016, 13:58:39 UTC
5aa7c25 [FLINK-3446] [runtime-web] Don't trigger back pressure sample for archived job This closes #1673. 19 February 2016, 23:15:41 UTC
a3b9d97 [tests] Try to improve CI test stability Squashes the following commits: - [tests] Wait for task managers in JobManagerFailsITCase Possible fix for: https://s3.amazonaws.com/archive.travis-ci.org/jobs/110235128/log.txt - [tests] Move ITCase from runtime to tests - [tests] Merge abstract and sub type class - [tests] Determine JobManagerProcess ports from logs This closes #1676. 19 February 2016, 23:12:38 UTC
59f978f [FLINK-3381] [runtime] Handle ConnectionLoss in ZooKeeper test util 19 February 2016, 23:10:29 UTC
8912fb0 [FLINK-3386][FLINK-3368][Kafka 0.8] Cleanup Kafka Connector - Remove Guava dependency - Remove unused methods - Move ZKString Serializer - Add user-friendly error messages when parsing arguments This closes #1623 and closes #1672 19 February 2016, 18:39:13 UTC
0b7c2db [FLINK-3368][Kafka 0.8] Some improvements to the Legacy Fetcher 19 February 2016, 10:09:21 UTC
3a8717f [FLINK-3386] Ensure Flink Kafka 0.8 is correctly using 'auto.offset.reset' 19 February 2016, 10:08:35 UTC
d48bb59 [FLINK-3368][Kafka 0.8] Handle leader changes in Kafka Consumer. 19 February 2016, 10:08:35 UTC
0c6c723 [docs] fix typos in quickstart and CEP docs This closes #1663 18 February 2016, 19:28:57 UTC
138334a [hotfix] [tests] Increase timeouts in WebFrontendITCase for stability on CI infrastructure 18 February 2016, 11:39:04 UTC
7278e87 [FLINK-3424] [tests] Close InputStream in FileStateBackendTest#validateBytesInStream This closes #1653 18 February 2016, 11:39:04 UTC
06e3dd1 [FLINK-3423] [tests] Fixed ExternalProcessRunnerTest on Windows This closes #1650 18 February 2016, 11:39:04 UTC
41b826b [FLINK-3436] Remove ComplexIntegrationITCase 17 February 2016, 16:47:58 UTC
c97a63e [FLINK-3400] Move RocksDB Copy Utils to flink-streaming-java They are not specific to RocksDB, just utilities for copying local folders to/from HDFS. Moving them to flink-streaming-java means that they are always in the classpath of the TaskManager, not only in the user-code jar when using RocksDB. If they are only in the user-code jar the external process runner cannot find the class files, leading to ClassNotFoundExceptions. 17 February 2016, 16:37:43 UTC
cee135a [FLINK-3430] Remove "no POJO" warning in TypeAnalyzer This closes #1655 17 February 2016, 14:18:21 UTC
a4e8fcb [hotfix] Auto-select actor system port in TestingCluster to prevent port collisions 17 February 2016, 14:09:13 UTC
06a42bf [FLINK-3419] [streaming] Drop 'partitionByHash' function, subsumed by 'keyBy()' 17 February 2016, 14:09:13 UTC
a8afec3 [FLINK-3421] [streaming scala] Remove unneeded ClassTag context bounds 17 February 2016, 14:09:13 UTC
6dda531 [FLINK-3413] [streaming] Make implicit conversions from Java DataStream to Scala DataStream explicit This also clean up a lot of JavaDocs in various Scala DataStream API classes. 17 February 2016, 14:09:13 UTC
bd137ae [hotfix] Properly declare ComplexIntegrationTest as an Integration Test Case 17 February 2016, 14:09:13 UTC
97d1f4a [hotfix] Fixes to flink-annotations JavaDocs 17 February 2016, 14:09:12 UTC
50a166d [FLINK-3299] Remove ApplicationID from Environment This closes #1642. 17 February 2016, 13:46:04 UTC
e08d7a6 [FLINK-3401] [streaming] [api breaking] AscendingTimestampExtractor only logs violations of ascending timestamp order. The user can also explicitly set an 'IgnoringHandler' or a 'FailingHandler', which do nothing on violations, respectively fail hard. 17 February 2016, 10:35:36 UTC
74c2b80 [FLINK-3379] [FLINK-3415] [streaming] Refactor TimestampExtractor into two separate classes - one class handled periodic watermarks - the other class handled watermarks triggered by elements This also makes sure that any timestamp assigner / watermark generators cannot generate negative watermarks This closes #1646 17 February 2016, 10:29:03 UTC
c1412dd Add (basic) RocksDB State Backend Documentation 17 February 2016, 09:57:44 UTC
be4601e [FLINK-3402] Refactor Common Parts of Stream/Batch Documentation 17 February 2016, 09:57:44 UTC
134b5c2 [FLINK-3403] Create Section "Working with Time" in Streaming Guide 17 February 2016, 09:57:44 UTC
ce8f966 [FLINK-3243] Fix Interplay of TimeCharacteristic and Time Windows This adds dedicated WindowAssigners for processing time and event time. timeWindow() and timeWindowAll() respect the TimeCharacteristic set on the StreamExecutionEnvironment. This will make the easy stuff easy, i.e. using time windows and quickly switching the time characteristic. Users will then have the flexibility to mix different kinds of window assigners in their job. This also expands the translation tests to verify that the correct window operators are instantiated. 17 February 2016, 09:57:44 UTC
ebab470 [FLINK-3420] [api-breaking] Remove utility functions 'readTextFileWithValue' and 'readFileOfPrimitives' from StreamExecutionEnvironment These methods are highly specific for very niche cases of bounded data stream processing. As such, the disadvantages (bloat and lock the API, lock the development into support) outweigh the benefit. This closes #1648 17 February 2016, 09:32:05 UTC
401837b [FLINK-3308] [py] Remove debug mode 17 February 2016, 08:29:33 UTC
a66458e [hotfix] [streaming] Add check in StreamRecord for reserved timestamp values 16 February 2016, 16:42:18 UTC
8cce136 [FLINK-3413] [streaming] Remove implicit Seq -> DataStream conversion Because the implicit conversion creates a new ExecutionEnvironment, it leads to strange errors when used withing programs with more than one source. 16 February 2016, 16:42:18 UTC
5b1231d [hotfix] [streaming] Remove some unused internal classes 16 February 2016, 16:42:18 UTC
75dcfc0 [hotfix] [rocksdb] Update predefined RocksDB option profiles. 16 February 2016, 16:42:18 UTC
489e653 [hotfix] [streaming] Remove unused code fragment from DataStream.union() function. 16 February 2016, 16:42:18 UTC
932a49d [hotfix] [docs] Complete sentences in cluster_execution.md 16 February 2016, 16:27:29 UTC
2714aaf [FLINK-3296] Remove 'flushing' behavior of the OutputFormat support of the DataStream API This closes #1563 16 February 2016, 14:52:53 UTC
8486c3c [hotfix] Add Scala version suffix to flink-cep module 16 February 2016, 14:35:51 UTC
76d3a63 FLINK-2380: allow the specification of a default filesystem scheme in the flink configuration file. This closes #1524 16 February 2016, 14:23:44 UTC
c658763 [FLINK-3304] Making the Avro Schema serializable. This closes #1635 16 February 2016, 13:47:43 UTC
65d0805 [FLINK-3158] Enforce maven version to be < 3.3 when building a release 15 February 2016, 18:39:43 UTC
ca11c1c [hotfix] Annotate StoppableFunction as PublicEvolving 15 February 2016, 15:44:59 UTC
9053254 [hotfix] Add better error reporting in case of wrongly specified retry delay 15 February 2016, 15:17:53 UTC
f60f8fb [FLINK-2111] Make stoppable stream task and stoppable stream source operator type safe Update index.js and reset vendor.css and vendor.js to master version Update web-dashboard Remove duplicate flink-runtime-web dependency from flink-tests Remove not used ProgramStopException Change stopping behaviour to only work in job status RUNNING This closes #750. 15 February 2016, 15:17:36 UTC
bdd4024 [FLINK-2111] Add "stop" signal to cleanly shutdown streaming jobs - added JobType to JobGraph and ExecutionGraph - added interface Stoppable, applied to SourceStreamTask - added STOP signal logic to JobManager, TaskManager, ExecutionGraph - extended Client to support stop - extended Cli frontend, JobManager frontend - updated documenation Fix JobManagerTest.testStopSignal and testStopSignalFail The StoppableInvokable could not be instantiated by Task because it was declared as a private class. Adds additional checks to verify that the stop signal behaves correctly. Auto-detect if job is stoppable A job is stoppable iff all sources are stoppable - Replace JobType by stoppable flag - Add StoppableFunction and StoppableInvokable to support the optional stop operation - added REST get/delete test (no extra YARN test -- think not required as get/delete is both tested) - bug fix: job got canceld instead of stopped in web interface - Add StoppingException - Allow to stop jobs when they are not in state RUNNING Second round of Till's comments 15 February 2016, 15:16:13 UTC
5eae47f [FLINK-3187] [restart] Introduce RestartStrategy to ExecutionGraph A RestartStrategy defines how the ExecutionGraph reacts in case of a restart. Different strategies are conceivable. For example, no restart, fixed delay restart, exponential backoff restart, scaling in/out restart, etc. Expose RestartStrategy to user API This removes the setNumberExecutionRetries and the setDelayBetweenRetries on the ExecutionEnvironment and the ExecutionConfig. Instead the more general RestartStrategy can be set. In order to maintain the separation between the runtime and api module, one sets a RestartStrategyConfiguration which is transformed into a RestartStrategy on the JobManager. Replace old execution-retries configuration parameters by restart-strategy. Add FixedDelayRestartStrategy test case Reintroduce old configuration values and API calls for the deprecated restart mechanism The old configuration values and API calls will be respected if no explicit RestartStrategy has been set. The values, if correct, are used to instantiate a FixedDelayRestartStrategy. Add deprecation comments to the JavaDocs Add logging statement for job recovery Fix JobManagerProcessFailureBatchRecoveryITCase by introducing a job recovery timeout Add proper annotations to RestartStrategies Let ExecutionGraphRestartTest extend TestLogger This closes #1470. 15 February 2016, 14:58:05 UTC
b17632d [FLINK-3354] Determine correct size for RocksDB snapshots Closes #1638 15 February 2016, 13:08:41 UTC
73bc35f [FLINK-3160] [web-dashboard] Aggregate operator statistics by TaskManager Adds a new per-job tab displaying subtask statistics aggregated by TaskManager. This closes #1564. 15 February 2016, 12:55:00 UTC
babf84c [hotfix] [web-dashboard] Make build idempotent 15 February 2016, 12:55:00 UTC
b2f6f8c Revert "[FLINK-3369] [runtime] Make RemoteTransportException instance of CancelTaskException" This reverts commit cf3ae88b73e30a2d69ac1cc6009a8304ea3f53cc. The reason is that we cannot be sure that a failing sender will always be the root cause why the receiver cannot read the data. A TM can also fail after a sender is finished leading to a loss of the intermediate result partitions. Then the receiver will no longer be able to read the data. 15 February 2016, 11:13:45 UTC
c9db63b [FLINK-3367] Add PublicEvolving and Internal annotations to flink-streaming-java and flink-streaming-scala This closes #1606 14 February 2016, 21:51:30 UTC
53f8d77 [FLINK-3367] Add PublicEvolving and Internal annotations to flink-core 14 February 2016, 19:49:18 UTC
e5f33b6 [FLINK-3367] Add PublicEvolving and Internal annotations to flink-java and flink-scala 14 February 2016, 19:26:26 UTC
937963e [FLINK-3107] [runtime] Defer start of checkpoint ID counter 12 February 2016, 22:17:04 UTC
8e55bbd [FLINK-3393] [core] ExternalProcessRunner wait to finish copying error stream 12 February 2016, 22:04:06 UTC
8b7caaa [hotfix] Fix field names in RocksDBStateBackend 12 February 2016, 21:59:14 UTC
f5d7190 [FLINK-3352] Use HDFS Config in RocksDB Copy Utilities This also moves the utilities (HDFSCopyFromLocal and HDFSCopyToLocal) to the RocksDB package because we would need a HDFS dependency in flink-core otherwise. 12 February 2016, 21:59:14 UTC
31310cd [FLINK-3359] Make RocksDB File Copies Asynchronous 12 February 2016, 21:59:14 UTC
c47cb7a [FLINK-3392] [kafka] Fix unsynchronized access in ClosableBlockingQueue 12 February 2016, 18:30:22 UTC
e927ec0 [FLINK-2991] Adjust RocksDB Folding State to latest RocksDBStateBackend 12 February 2016, 17:51:01 UTC
94cba89 [FLINK-2991] Add Folding State and use in WindowOperator This enables efficient incremental aggregation of fold window. This also adds: - WindowedStream.apply(initVal, foldFunction, windowFunction) - AllWindowedStream.apply(initVal, foldFunction, windowFunction) This closes #1605 12 February 2016, 17:51:01 UTC
d93b154 [hotfix] Minor code cleanups in AbstractStateBackend 12 February 2016, 17:51:01 UTC
back to top