https://github.com/apache/flink

sort by:
Revision Author Date Message Commit Date
1d7d542 Commit for release 1.11.4 21 July 2021, 02:20:58 UTC
ed0df86 [FLINK-23312][ci] speed up compilation for e2e tests The "compile" builder already applies all checks so we can use -Dfast here; also, the web UI is not actually needed in the E2E tests. 14 July 2021, 08:07:17 UTC
ab4e3e3 [FLINK-23164][tests] Harden JobMasterTest.testMultipleStartsWork The JobMasterTest.testMultipleStartsWork fails because we don't wait for the JobMaster to finish its suspension before restarting the JobMaster. This can lead to ignoring the underlying control START control message. This again can lead to a situation where we are in the STOPPED state and try to send UnfencedMessage(startJobExecution) in JobMaster.start. This commit fixes this problem by waiting on the suspension future to complete before restarting the JobMaster. This closes #16443. 12 July 2021, 07:54:22 UTC
ba3e0d4 [FLINK-18182][kinesis] Updating guava to 29.0-jre for Kinesis connector 29 June 2021, 07:25:34 UTC
22614f4 [FLINK-18182][kinesis] Updating to latest AWS SDK for Kinesis connector 29 June 2021, 07:25:34 UTC
d280b22 [FLINK-23166][python] Fix ZipUtils to handle properly for softlinks This closes #16309. 28 June 2021, 23:17:57 UTC
6eb6ca5 [FLINK-22470][python] Make sure that the root cause of the exception encountered during compiling the job was exposed to users in all cases This closes #15765. 25 June 2021, 05:44:29 UTC
aa7baa5 [FLINK-22946][runtime] Recycle floating buffer outside the lock to avoid deadlock 18 June 2021, 12:56:46 UTC
92b41c5 [FLINK-21945][streaming] Add the possibility to force certain exchanges to be aligned. Introduces the AlignmentType to CheckpointOptions that includes a FORCED_ALIGNED. This barrier acts as an aligned barrier for certain exchanges. The flag is then later used to restore the unaligned characteristic in SubtaskCheckpointCoordinatorImpl for downstream exchanges. 08 June 2021, 06:29:44 UTC
8f75828 [FLINK-22815][checkpointing] Disable unaligned checkpoints for broadcast partitioning Broadcast partitioning can not work with unaligned checkpointing. There is no guarantees that records are consumed at the same rate in all channels. This can result in some tasks applying state changes corresponding to a certain broadcasted event while others don't. In turn upon restore it may lead to an unconsistent state. 08 June 2021, 06:29:44 UTC
0c8a0bf [FLINK-22597] Make JobMaster restartable The problem is that the DefaultLeaderRetrievalService for the ResourceManager is reused by the JobMaster. Since the DefaultLeaderRetrievalService is only usable once, this means that the JobMaster can only be started once even though the JobManagerRunnerImpl restarts it in case of a regained leadership. 07 May 2021, 15:17:33 UTC
3fc8a2e [FLINK-20383][runtime] Fix race condition in notification. Notification happens outside of lock, so between non-null check and actual notification, readView could have been released. 04 May 2021, 15:55:31 UTC
c53a715 [FLINK-22109][table-planner-blink] Resolve misleading exception message in invalid nested function This closes #15523. 29 April 2021, 16:35:43 UTC
7a89994 [FLINK-22489][webui] Fix displaying individual subtasks backpressure-level Previously (incorrectly) backpressure-level from the whole task was being displayed for each of the subtasks. 28 April 2021, 13:23:16 UTC
0c1af6c [FLINK-22424][network] Prevent releasing PipelinedSubpartition while Task can still write to it This bug was happening when a downstream tasks were failing over or being cancelled. If all of the downstream tasks released subpartition views, underlying memory segments/buffers could have been recycled, while upstream task was still writting some records. The problem is fixed by adding the writer (result partition) itself as one more reference counted user of the result partition 28 April 2021, 13:18:51 UTC
603a3c3 [FLINK-20654] Fix double recycling of Buffers in case of an exception on persisting Exception can be thrown for example if task is being cancelled. This was leading to same buffer being recycled twice. Most of the times that was just leading to an IllegalReferenceCount being thrown, which was ignored, as this task was being cancelled. However on rare occasions this buffer could have been picked up by another task after being recycled for the first time, recycled second time and being picked up by another (third task). In that case we had two users of the same buffer, which could lead to all sort of data corruptions. 22 April 2021, 05:59:07 UTC
4dfef1b [FLINK-21986][State Backends] fix native memory used by RocksDB not be released timely after job restart [FLINK-21986][State Backends] fix spotless-check error This closes #15619. 20 April 2021, 08:28:51 UTC
2e645cb [FLINK-22208][build] Bump snappy-java to 1.1.8.3 19 April 2021, 12:27:13 UTC
685a216 [hotfix][table-api-java] Improve built-in functions as parameters of user-defined functions 19 April 2021, 05:43:32 UTC
2766d30 [FLINK-22184][client] Shutdown client outside of netty thread 12 April 2021, 08:42:58 UTC
abdf8c0 [hotfix][table-common] Support negative decimal scale in fromValues 12 April 2021, 08:33:51 UTC
35babaf [FLINK-21008][coordination] Register ClusterEntrypoint#closeAsync as shutdown hook for the cleanup This closes #15536. 09 April 2021, 10:14:32 UTC
278839b [hotfix][yarn-tests] Check the applicationId in verifyStringsInNamedLogFiles 08 April 2021, 15:48:59 UTC
6f1e1b5 [FLINK-21148][test] Refactors test code to use CommonTestUtils for loops 08 April 2021, 15:48:59 UTC
9250c4a [FLINK-21148][test] Increases timeout The actual problem of the test failure was that YARN was not fast enough to spin up the Flink cluster. The test waited for 10secs for the job to finish. The logs indicate that there was no issue. The problems occurred when the test initiated the killing of the YARN application after 10 seconds. Increasing the timeout should fix this issue. 08 April 2021, 15:48:58 UTC
7c7f7fe [FLINK-21148][test] Makes the test fail early when running into a timeout 08 April 2021, 15:48:58 UTC
e3066e4 [FLINK-21148][test] Removes runTest in runDetachedModeTest runDetachedModeTest is a utility method called by actual tests. These tests are taking care of setting up the test context for cleanup. Hence, it does not have to be included in runDetachedModeTest 08 April 2021, 15:48:58 UTC
b653222 [hotfix][test] Add error message for CommonTestUtils#waitUntilCondition 08 April 2021, 15:48:57 UTC
f90493f [FLINK-22142][ci] Don't log Kafka connector output to the AZP console This commit removes the console logging of the Kafka connector to the AZP console. This closes #15512. 08 April 2021, 09:00:02 UTC
3bd44e0 [FLINK-22081][core] Fix entropy injection metadata path in pluggable HadoopS3FileSystem 08 April 2021, 07:20:27 UTC
2a89f8d [FLINK-21980][zk] ZooKeeperRunningJobsRegistry creates an empty znode This closes #15393. 07 April 2021, 16:58:09 UTC
9faa553 [FLINK-22014][ha] Make sure that AbstractHaServices delete first the HA data before deleting blobs on closeAndCleanupAllData The AbstractHaServices should always first delete the HA data (pointers to jobs and checkpoints) before deleting the referenced objects such as blobs. That way we ensure that we don't leave the system in an invalid state if the process stops abruptly because either all the data is still available or the pointers have been cleared and Flink only leaves some orphaned blobs. The case where the pointers are still there but not the blobs will no longer be possible with this fix. This closes #15470. 07 April 2021, 16:30:59 UTC
87717bb [hotfix] Let SimpleRecoveryITCaseBase extend from TestLogger 01 April 2021, 07:46:43 UTC
ec33747 [hotfix] Make SimpleRecoveryITCaseBase.testRestartMultipleTimes restart strategy agnostic 01 April 2021, 07:46:43 UTC
f61c3da [hotfix][tests] Use MiniClusterWithClientResource in SimpleRecoveryITCaseBase Using the resource allows to reuse it across tests and ensures a proper shut down at the end. 01 April 2021, 07:46:43 UTC
efd694a [FLINK-21609][tests] Remove usage of LocalCollectionOutpuFormat from SimpleRecoveryITCaseBase The LocalCollectionOutputFormat does not work well together with restarts because it records results from different attempts. Hence, instead of using this format we now use the collect() call in the SimpleRecoveryITCaseBase. This closes #15453. 01 April 2021, 07:46:32 UTC
a6d802d [FLINK-20752][coordination] Properly respect max-failures-per-interval 31 March 2021, 11:19:04 UTC
311a2f3 [FLINK-21515][tests] Fix testStopWithSavepointShouldNotInterruptTheSource instability SourceStreamTaskTest.testStopWithSavepointShouldNotInterruptTheSource is unstable because the latch can be reset after triggering (while it`s supposed to be reset before the test starts). 30 March 2021, 19:50:00 UTC
1d5e9b8 Revert "[FLINK-5717][datastream] Fix NPE and lost timer during window merging for ContinuousProcessingTimeTrigger" This reverts commit 6c4dae80211a65d1b1219bd53e38855ed4f0e7e5. This reverts commit 8d113a2f41440a06d6c5321e767f78671a65ce93. 25 March 2021, 09:50:23 UTC
8d113a2 [hotfix] Fix compilation of ContinuousProcessingTimeTriggerTest 25 March 2021, 09:25:28 UTC
6c4dae8 [FLINK-5717][datastream] Fix NPE and lost timer during window merging for ContinuousProcessingTimeTrigger This fixes: * NPE in clear() due to state merged out. * Timer lost due to no timer registration for new window. This closes #15241 25 March 2021, 08:32:26 UTC
c3992df [FLINK-21550][tests] Harden ZooKeeperHaServicesTest.testSimpleClose This commit hardens the ZooKeeperHaServicesTest by not busy loop waiting with a timeout on a leader election but instead to use LeaderRetrievalUtils.LeaderConnectionInfoListener which offers a future which is completed when the leader is elected. This closes #15334. 23 March 2021, 13:38:47 UTC
6f19a9f [hotfix][test] Adds initialization of flag that triggers cancellation 19 March 2021, 11:38:16 UTC
9e29a22 [FLINK-21654][tests] Adds retry loop to YarnClient.getApplications This fix should cover the issue with YARN-7007. The corresponding discussion is happening in FLINK-15534. This closes #15134. 19 March 2021, 11:38:15 UTC
a0eeac3 [FLINK-21753][runtime] Don't cycle reference between memory manager and gc cleaner action This closes #15224 16 March 2021, 09:55:15 UTC
f28db07 [FLINK-21735][coordination] Harden JobMaster#updateTaskExecutionState() This closes #15196. 15 March 2021, 12:57:14 UTC
90c70e1 [FLINK-21725][table] Add tests for large tuples 12 March 2021, 22:54:52 UTC
d5dd1d7 [FLINK-21725][core] Sync tuples with updated TupleGenerator 12 March 2021, 22:51:24 UTC
085a0f3 [FLINK-21725][core] Update TupleGenerator to name setter/constructor arguments like fields 12 March 2021, 22:51:23 UTC
7edd724 [FLINK-21606] Fail TaskExecutor hard if the ResourceManager rejects its registration. The ResourceManager rejects the connection attempt of a TaskExecutor if the TaskExecutor is not known to the ResourceManager. In this case, we don't wait for the max registration timeout but instead fail hard and fast. This closes #15107. 12 March 2021, 13:36:24 UTC
2f094da [FLINK-21606] Fail hard if the ResourceManager rejects a JobMaster registration 12 March 2021, 13:34:35 UTC
db0a5f5 [FLINK-21606] Release job resources on TaskExecutor if the JobMaster rejects connection attempts The JobMaster can reject a connection attempt from the TaskExecutor if it is not responsible for the expected job. In this case, the TaskExecutor has some old connection information for another job and should release all the job related resources (slots, tracked partitions). 12 March 2021, 13:34:35 UTC
969fda2 [FLINK-21606] Introduce RegistrationResponse.Rejection and RegisteredRpcConnection.onRegistrationRejection Introduce RegisteredRpcConnection.onRegistrationRejection to support rejection of regisration attempts. 12 March 2021, 13:34:35 UTC
258f54d [hotfix] Remove Mocking from RegisteredRpcConnectionTest Introduce TestRegistrationGateway interface and DefaultTestRegistrationGateway for easier testing. 12 March 2021, 13:34:34 UTC
172fdfe [FLINK-21606] Add JobID to JobMasterGateway.registerTaskExecutor The newly introduce parameter of type JobID can be used to verify that the target JobMaster is really responsible for the job the TaskExecutor thinks the JobMaster is responsible for. 08 March 2021, 15:01:01 UTC
b61e01a [FLINK-21497][coordination] Only complete leader future with valid leader 02 March 2021, 09:30:19 UTC
c912488 [FLINK-21030][runtime] Adds trigger for global failover We cannot assume that the termination future is finishing when triggering a synchronous savepoint. It might be that the savepoint creation succeeds but the graceful shutdown of the job fails. In this case, the job termination does not complete. The job might end up in an inconsistent state. That's why, a global fail over is triggered if the scheduler observes one of the current executions ending up in a non-finished state. Unit tests and integration tests are added to cover this use-case. Cherry-picked ea9ac12 This closes #15035. 26 February 2021, 13:48:56 UTC
2accc7e [hotfix][test] Adds unit test for local and global failure happened concurrently This test should verify that the scheduler works as expected when a local failure succeeds a global fail-over operation before the restarting happened. Cherry-picked 1d08484 26 February 2021, 11:03:47 UTC
637aa13 [hotfix][task] Interrupt source legacy thread on failure. If a legacy source task fails outside of the legacy thread, the legacy thread blocks proper cancellation (completion future never completed). 26 February 2021, 10:59:58 UTC
c8bd850 [hotfix][tests] Adds log message to MiniClusterWithClientResource shutdown 26 February 2021, 10:59:41 UTC
c6b10f6 [FLINK-21486] Throw exception when restoring Rocks timers with Heap timers enabled This closes #15022 26 February 2021, 08:16:18 UTC
1196dff [FLINK-21453][checkpointing] Do not ignore endOfInput when terminating a job with savepoint When job is stopping with savepoint WITH drain flag (terminating), there is no intention to resume the job ever from that savepoint. Hence we have to make sure that we flush all of the buffered records from the job. To do that, we need to invoke endOfInput. 25 February 2021, 10:30:58 UTC
7cb2154 [FLINK-21453][checkpointing][refactor] Replace advanceToEndOfTime with new CheckpointType.SAVEPOINT_TERMINATE 25 February 2021, 10:30:58 UTC
017b33d [hotfix][task] Rename isStoppingBySyncSavepoint to ignoreEndOfInput 25 February 2021, 10:30:58 UTC
463d9e3 [FLINK-21028][task] Do not interrupt the source thread on stop with savepoint Currently stop with savepoint relies on the EndOfPartitionEvents propagation and performs clean shutdown after the stop with savepoint (which can produce some records to process after the savepoint while stopping). If we interrupt source thread, we might leave the newtork stack in an inconsitent state. So, if we want to relay on the clean shutdown, we can not interrupt the source thread. 23 February 2021, 17:47:02 UTC
9bc2fba [FLINK-21213][task] Degrade log level to INFO when ignore to decline checkpoint as task not running 20 February 2021, 06:08:51 UTC
9946891 [FLINK-21274] Block main thread when running the TaskManagerRunner In order to ensure that the TaskManager properly shuts down we need to let the main thread block on the execution of the TaskManager. This will ensure that there is always a non-daemon thread as long as the TaskManager runs. This closes #14915. 11 February 2021, 18:02:41 UTC
b4e3b49 [FLINK-21274][runtime] Change the ClusterEntrypoint.runClusterEntrypoint to wait on the result of clusterEntrypoint.getTerminationFuture().get() and do the System.exit outside of the future callback 11 February 2021, 18:02:23 UTC
30351d6 [FLINK-21138] Explicit Classloader in QueryableStateClient 11 February 2021, 17:47:33 UTC
95281a7 [FLINK-21138] Fix classloader on QueryableStateClient 11 February 2021, 17:47:32 UTC
5f6a804 [FLINK-21138][tests] Register serializer instance in AbstractQueryableStateTestBase.testCustomKryoSerializerHandling The test AbstractQueryableStateTestBase.testCustomKryoSerializerHandling only works if one registers an actual serializer instance. 11 February 2021, 17:47:32 UTC
d69d134 [FLINK-21138] - User ClassLoader in KvStateServerHandler 11 February 2021, 17:47:30 UTC
3ea8653 [FLINK-21312][checkpointing] Unset IsStoppingBySyncSavepoint from the LegacySourceThread 1. Unset IsStoppingBySyncSavepoint from the LegacySourceThread 2. Also unset it in StreamSource so that endInput is consistent for head and tail operators 3. Replace ITCase with a lower level test. Legacy Source Thread needs checkpoint lock after it exits run() and before it re-sets OperatorChain.StoppingBySyncSavepoint. This creates a race condition because that lock is held by the Task thread performing sync-savepoint. On a lower level there is no such problem because there is no checkpoint completion notification. 4. Fix init of SavepointITCase.testStopSavepointWithBoundedInput - Only one (main) thread counts down BoundedPassThroughOperator.snapshotAllowedLatch so it is set to 1 - InfiniteTestSource.createdSources is updated from run() - Propagate the exception from InfiniteTestSource.run() 11 February 2021, 14:57:19 UTC
d342679 [hotfix][tests] Import static constant in SourceStreamTaskTest 11 February 2021, 14:57:19 UTC
9984b58 [FLINK-21013][table-planner-blink] Ingest row time into StreamRecord in Blink planner This closes #14785. 09 February 2021, 08:45:35 UTC
6f7fbb3 [FLINK-21208][python] Make Arrow Coder serialize schema info in every batch (#14859) * [FLINK-21208][python] Make Arrow Coder serialize schema info in every batch * fix * fix2 08 February 2021, 01:49:56 UTC
d7d0bca [FLINK-21210][coordination] ApplicationClusterEntryPoint explicitly closes PackagedProgram 05 February 2021, 11:40:16 UTC
d01de29 [FLINK-21175][tests] Fix unpredictable Thread.getState in StreamTaskTestHarness due to concurrent class loading 04 February 2021, 14:04:06 UTC
299820a [hotfix][task] Rename SourceStreamTask.isFinished to wasStoppedExternally 03 February 2021, 19:52:48 UTC
a996fc0 [FLINK-21132][runtime][tests] Test StopWith Savepoint against concurrent EndOfInput 03 February 2021, 19:52:48 UTC
df04d5a [FLINK-21132][runtime][tests] Parameterize StopWithSavepoint test with chaining strategy 03 February 2021, 19:52:48 UTC
c030d31 [FLINK-21132][runtime][tests] Stop with savepoint shouldn't end input 03 February 2021, 19:52:48 UTC
ff3b9c3 [FLINK-21132][runtime] Don't end input on stop with savepoint EndOfInput was used to handle any stopping of the job. When stopping with savepoint the input is not actually ended. This causes issues with some sinks (e.g. Iceberg). With this change, endInput is not call for stop-with-savepoint. To differentiate stop-with-savepoint from other cases only checkpoint (RPC/barriers) are considered and not network EOP. That's enough because EOP is only injected after the CP completion (i.e. when the downstream is also notified by sync savepoint by CP barriers). 03 February 2021, 19:52:48 UTC
ce03675 [hotfix][clients] ClientUtils return URLClassLoader 03 February 2021, 11:38:24 UTC
772e8fb [FLINK-21215][task] Do not overwrite the original CheckpointFailureReason in AsyncCheckpointRunnable Before this change, original failure reason would be hidden and replaced with CHECKPOINT_ASYNC_EXCEPTION 03 February 2021, 10:42:41 UTC
a1d1e86 [FLINK-9844][client] PackagedProgram#close() closes ClassLoader 03 February 2021, 10:11:13 UTC
9c837f1 [FLINK-21216][python] Limit the numpy version < 1.20.0 (#14825) 01 February 2021, 09:15:42 UTC
1aae7f7 [FLINK-21164][rest] Delete temporary jars 29 January 2021, 18:27:11 UTC
4ad5c81 [FLINK-20675][checkpointing] Ensure asynchronous checkpoint failure could fail the job by default 29 January 2021, 08:07:11 UTC
43fa9bf [FLINK-20675][checkpointing] Only decline async checkpoint failure when task is still running 29 January 2021, 08:07:06 UTC
20c6ed1 [FLINK-16443][checkpointing] Make sure that CheckpointException are also serialized in DeclineCheckpoint. 29 January 2021, 08:06:59 UTC
a3da0ef [FLINK-21123][fs] Bump beanutils to 1.9.4 26 January 2021, 19:22:08 UTC
ceb15b3 [FLINK-21020][build] Bump Jackson to 2.10.5.1 26 January 2021, 15:43:44 UTC
cece43b [FLINK-19436][tests] Properly shutdown cluster in e2e tests This closes #14747 25 January 2021, 13:59:57 UTC
ca3f080 [FLINK-20680][table-common] Fix invalid validation of var-arg function with no parameters 25 January 2021, 10:19:25 UTC
bb9e0f5 [FLINK-21070][table-runtime-blink] Fix invalid reuse of generated code CompileUtils reuses code based on only the class name. This could be error-prone as code that is slightly different would be ignored. The wrong behavior was visible during code generation of structured types. This closes #14720. 22 January 2021, 20:33:15 UTC
428c673 [FLINK-19771][connector-jdbc] Fix NPE when accessing array using postgres Switching to handle arrays in createInternalConverter rather than createNullableInternalConverter, so nulls are handled properly. This closes #13787. 22 January 2021, 09:37:15 UTC
e9a33bb [FLINK-21071][docker][tests] Run docker tests against 1.11 images 21 January 2021, 17:17:01 UTC
0767fd8 [FLINK-20913][hive] Do not use HiveConf constructor to avoid overriding properties with default values This closes #14707 Co-authored-by: dixingxing <dixingxing@autohome.com.cn> 21 January 2021, 03:05:02 UTC
11c2adb [FLINK-21009] Can not disable certain options in Elasticsearch 7 connector 19 January 2021, 10:12:21 UTC
back to top