https://github.com/elastic/elasticsearch

sort by:
Revision Author Date Message Commit Date
5ad0236 Update license override list (#85940) (#85944) This commit updates the list of license expiration overrides as requested by our compliance team. (cherry picked from commit e4d9ecc50ba15dc19e42f26faa1c141b454a784c) 15 April 2022, 22:29:12 UTC
a3e4313 Upgrade bytebuddy to latest version with Java 19 support (#85881) (#85915) 14 April 2022, 21:10:58 UTC
3eb5c9d Grant kibana_system role read access to APM data streams (#85744) (#85906) Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> 14 April 2022, 14:57:30 UTC
c53c4c0 Release notes for 7.17.3 (#85861) Release notes for 7.17.3 14 April 2022, 13:10:47 UTC
59a632d Upgrade to JDK 18.0.0+36 (#85376) (#85875) Closes #85357. Upgrade to the latest Adoptium JDK for bundling with Elasticsearch. 14 April 2022, 07:59:01 UTC
f9d77a2 Add environment variables for Java 18 and 19 Java home 13 April 2022, 21:15:20 UTC
d02c9a7 Prevent ThreadContext header leak when sending response backport(#68649) (#85865) We need to stash thread context in DefaultRestChannel before we call channel.sendResponse because when calling this method, our execution might be delayed and the thread be reused for another task - like sending another response. And it would see thread context from the initial “delayed” work. This commit also expands the assertions on the empty thread context to make sure it does not contains response headers. closes #68278 13 April 2022, 10:30:52 UTC
b622c83 Reduce port range re-use in tests (#85777) Today we select the port range to use in tests by taking the Gradle worker ID mod 223. We now use significantly more than 223 Gradle workers in each test run which means port ranges are re-used, resulting in spurious failures. This commit replaces the magic number 223 with a computation defined in terms of the available ports and the number of ports to allocate per worker. It also reduces the number of ports per worker to 30 to try and reduce the frequency of port clashes. 13 April 2022, 08:12:24 UTC
3b81473 Fix TranslogTests#testStats (#85828) Today these tests assume that the last-modified time on the translog file is at least one millisecond before the time at which stats are computed, but this isn't always true. With this commit we add a (bounded) wait for the clock to advance beyond the file's last-modified time to ensure that we observe a nonzero age. Closes #85717 13 April 2022, 07:56:50 UTC
750bb1e Mute MigrationIT.testGetDeprecationInfo (#85813) (#85814) Relates #85743 12 April 2022, 11:02:44 UTC
3efc890 AwaitsFix for #85760 (#85810) (#85811) Co-authored-by: David Turner <david.turner@elastic.co> Co-authored-by: David Turner <david.turner@elastic.co> 12 April 2022, 10:09:52 UTC
a63ecde Update URL used for build scan CI metrics links 11 April 2022, 19:27:34 UTC
037f56c Remove special rejection handling in Cold/Frozen cache services (#85775) (#85779) Now #85217 is merged the special handling introduced by #85239 to fail Gaps when a task is rejected can be removed. The `onRejection` method of the AbstractRunnable task will be executed if the task is rejected or if an unexpected exception is thrown before the task is added to the thread pool executor queue. 11 April 2022, 11:45:17 UTC
629d979 Relax docs about using new repos across major versions (#85740) (#85752) Today the docs say that new major versions need new snapshot repositories or else corruption may occur. This isn't true, we support using the same repository across upgrades. 07 April 2022, 16:39:49 UTC
2803d46 [DOCS] Adds size param to evaluate DFA API docs (#85735) (#85739) 07 April 2022, 08:28:43 UTC
c7f08ed [7.17] [ML] Allow retrieving `boolean` fields from _source in DFA jobs (#85672) (#85723) This commit fixes a bug where `boolean` mapped fields may only be retrieved from doc values in Data Frame Analytics jobs. While this problem may occur for `boolean` fields with disabled doc values, a more common way of being affected by this bug is running an analysis against data with a very high number of fields. If there are more fields than the `index.max_docvalue_fields_search` setting, we fall back to retrieving the fields from _source. Then `boolean` mapped fields would fail with the error message "cannot apply boolean mapping to field [<field>]". 06 April 2022, 15:25:05 UTC
a86e504 Mute testShardStatusStaysCompleteAfterNodeLeavesIfRegisteredWhileNodeOffline (#76804) (#85722) See #76804 06 April 2022, 13:00:24 UTC
c85aad3 Test mute (#85699) See https://github.com/elastic/elasticsearch/issues/85698 05 April 2022, 14:34:44 UTC
8578c5e [7.17] Enable overrides of license expiration dates (#85351) (#85687) * Enable overrides of license expiration dates (#85351) In rare cases, we are required to adjust the expiration date of a license that already exists for legal or accounting reasons, rather than simply generating a new one. This commit introduces a basic mechanism for overriding licenses by hash of license ID, by way of a list currently hardcoded in the repo. This approach is fairly naive, and should be replaced as noted in the code if the frequency we are required to use this mechanism increases. This PR adds Java-based REST tests for the Put License API to help validate this behavior. (cherry picked from commit 4ea21033737b8dcaa42590f3fc6459e97a8b63ca) * Fix compilation for 7.17 branch 05 April 2022, 01:40:20 UTC
acf13e0 [7.17] Bump commons-compress to 1.21 (#85581) (#85681) * Bump commons-compress to 1.21 (#85581) This updates the dependency version for the Apache commons compress project to 1.21 # Conflicts: # build-tools/build.gradle # distribution/tools/geoip-cli/build.gradle # test/x-content/build.gradle * Fix backport 04 April 2022, 15:45:31 UTC
0b41037 Reduce logging for cache file synchronization failures (#85331) (#85667) The searchable snapshot cache on cold tier synchronizes cache files every 60 seconds on disk. But if the synchronization fails due to disk being full (or any other problem that can arise at the file system level) the scheduled task logs a failure for every cache file to synchronize, filling up the disk even more since that be hundreds of files. This pull request reduce the logs emitted in such cases. Only one error message is logged per shard (actually per cache directory, each shard has its own cache dir) every 10 minutes. 04 April 2022, 10:17:02 UTC
1100f5a Fix skip caching factor with indices.queries.cache.all_segments (#85510) (#85646) The indices.queries.cache.all_segments setting should not be used in production environments. And we should also use the default skipCachingFactor (i.e, 10) for when that setting is enabled. 01 April 2022, 20:05:25 UTC
dea100e [ML] Simplify shortcircuiting scores renormalizer (#85625) (#85644) In #85555 a Phaser was used to track ongoing renormalizations and wait for them to complete. However, this is a very complex way to do things. It turns out there's an easier way, as submitting a task to a threadpool returns a Future that can be used to track when the submitted task completes. There was also a bug in the ShortCircuitingRenormalizer class where the possibility of RejectedExecutionException was not being considered. 01 April 2022, 19:23:40 UTC
66d5e65 [ML] Avoid multiple queued quantiles documents in renormalizer (#85555) (#85598) It's possible that an anomaly detection job can output a flurry of quantiles documents for use in renormalization in the time that a single renormalization takes. When this happens we only use the most recent quantiles document for the next renormalization, but the previous code queued up all the quantiles documents until the next renormalization started. The new approach that this PR implements is to only keep the most recent new quantiles document while the current renormalization is in progress. This avoids wasting memory on a queue that will be largely discarded (apart from one element) later on. Fixes #85539 01 April 2022, 18:15:20 UTC
a759992 Update gradle wrapper to 7.4.2 (#85608) (#85620) 01 April 2022, 15:53:44 UTC
fd3774c Fix fields wildcard support to vector tile search API (#85595) (#85615) This commit changes the strategy on how fields are fetched by iterating over the returned map. In addition, the geo field is added the last one on the fields list so it does not get overridden. 01 April 2022, 13:03:15 UTC
9f5a883 [7.17] Fix wildcard highlighting on match_only_text (#85500) (#85593) Fixes a bug where match_only_text fields were ignored during highlighting when a field name with wildcard was specified. Closes #85493 01 April 2022, 08:14:42 UTC
a1b3bef Bump versions for 7.17.2 release 31 March 2022, 19:01:38 UTC
d133702 SQL: Update DbVisualizer connection steps (#85167) (#85559) Update DbVisualizer connection steps to reflect version 13.0's integrated Elasticsearch support. 31 March 2022, 15:02:11 UTC
9268ebc [7.17][DOCS] Removes coming tag from release notes. (#85530) 31 March 2022, 14:01:33 UTC
ddcf22e Update build convention dependencies (#85444) (#85486) - Update spotless to potentially fix invalid git worktree handling - Update shadow plugin to be in sync with version used in build-tools 31 March 2022, 13:13:33 UTC
437d42a fix: remove tests from 7.17 (#85534) (#85540) Backport PR #85467 added two tests: * Multi-value date histogram * Multi-value date histogram docvalues only as a result of a mistake while doing a conflict resolution. With this PR I will remove them temporarily from master, backport the fix to 7.17.3 and later make another PR to master to add them back. This is a fix for #85509. (cherry picked from commit b4ff756beced1de35d355ed3ac200e27219722d2) # Conflicts: # rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/search.aggregation/360_date_histogram.yml 31 March 2022, 11:29:13 UTC
debaab0 [Test] Fix FollowIndexSecurityIT by granting needed previleges (#84467) (#85514) CCR user on the leader cluster needs more privileges than what are documented (#61308). Specifically it needs to renew the retention lease at a fixed time interval. This PR fixes it by granting the "manage" index privilege to the CCR user on the leader cluster. Note we still want to revisit privileges required CCR or at least fix our documentation. This will be tracked with #61308. Resolves: #84156 31 March 2022, 01:06:46 UTC
5d498a1 DOCS: Add 7.17.2 release notes (#85392) Remove the `:coming` tag 30 March 2022, 18:49:32 UTC
7d169fc [7.17] test: timezone with data_histogram and composite date_histogram (#85176) (#85467) (cherry picked from commit 3be4d6b9355e131546d7aa934613fef7bc4d12ee) # Conflicts: # rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/search.aggregation/360_date_histogram.yml 30 March 2022, 14:44:39 UTC
884d9be [7.17] Ensure the cleanup in testAutoFollowPatterns is always happening (#85482) * Assert actual follow tasks instead of count (#85151) Currently, the test is flaky and sometimes fails with expected 1 but was 2 failure. Updating the code to assert against collection content to collect additional information on the cause of the failure. (cherry picked from commit e31be1b570091ca8b29f924bf1fbae99937e44db) * Ensure the cleanup in testAutoFollowPatterns is always happening (#85401) In case of failure testAutoFollowPatterns will leave auto-following pattern running that would case assertion failure in other test cases (cherry picked from commit 92c45385f128ccdd872b93122af77c3bd6847d03) * revert java 17 usage 30 March 2022, 07:55:14 UTC
e5f9996 [DOCS] Removed extraneous end tag (#85430) 29 March 2022, 01:19:56 UTC
22c9eed [DOCS] Reuse data tier content in node role docs (#84346) (#85426) (#85428) * [DOCS] Reuse data tier content in node role docs (#84346) Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> * [DOCS] Fix duplicate anchor (#85424) * [DOCS] Fix duplicate anchor * Removed misplaced end tag Co-authored-by: James Rodewig <james.rodewig@elastic.co> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: James Rodewig <james.rodewig@elastic.co> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> 28 March 2022, 23:59:18 UTC
1ed056d [DOCS] Reuse data tier content in node role docs (#84346) (#85419) Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: James Rodewig <james.rodewig@elastic.co> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> 28 March 2022, 23:47:40 UTC
3cee793 Fall back to naive parallel forks calculation in non-standard cases (#85410) (#85421) Some architectures provide non-standard output in /proc/cpuinfo. Specifically, AWS graviton. In case we can't properly calculate physical cpu cores via the standard method just fallback to using the JDK-backed implementation. 28 March 2022, 22:34:26 UTC
e9b1a1e Fix calculation of max parallel forks for test execution (#85360) (#85404) 28 March 2022, 18:35:53 UTC
1f1774b Replace failing PolicyStepsRegistry sanity check assert with a test (#85346) 28 March 2022, 14:54:58 UTC
216fefc [7.17] Better logging when EsThreadPoolExecutor throws unexpected exceptions (#85217) (#85379) * Better logging when EsThreadPoolExecutor throws unexpected exceptions (#85217) We recently had issues with EsThreadPoolExecutor throwing unexpected exceptions (in our case an AccessControlException) before the Runnable or AbstractRunnable has been submitted for execution. In the case of AbstractRunnable the task is not rejected and it caused some resources to leak. This commit catches unexpected exceptions thrown when calling the EsThreadPoolExecutor#execute() method with AbstractRunnable and logs a message. It also assert that this situation should never happen. * fix 28 March 2022, 09:54:56 UTC
de7261d Use new changelog process in 7.17 (#85315) Backport of #85279. * Split the 7.17 release notes into a page per patch version * Sync the release notes task code with master * Various adjustments to align the generated docs with the existing docs 28 March 2022, 07:15:57 UTC
38b3bc4 Limit concurrent shard requests in disk usage API (#84900) (#85366) The disk_usage API fails on indices that have more than 16 shards. The reason is that we broadcast all shard requests simultaneously, and the handling thread pool - the analyzer thread pool - whose queue size is only 16. That means some shard requests will be rejected. This PR limits the number of concurrent shard requests sent to each node when we broadcast them. Closes #84779 26 March 2022, 19:49:29 UTC
12c32fa Increase store ref before snapshotting index commit (#84776) (#85112) (#85365) Snapshotted commits should also hold a reference to the store, so they are always usable; otherwise, callers need to manage the store's references manually. This change applies only to InternalEngine as we already do this in ReadOnlyEngine. 26 March 2022, 18:37:12 UTC
eac9031 TransportBroadcastAction should always set response for each shard (#84926) The TransportBroadcastAction can leave the responses of some group unset if all shards of that group hit shard_unavailable_exception. 26 March 2022, 17:58:58 UTC
55b5663 Relax error handling in transport disk usage API (#84774) Relax the failure handling to cover other exceptions such as BroadcastShardOperationFailedException. 26 March 2022, 17:57:37 UTC
ba3b1e3 Redirect testclusters ES process stdout/stderr to log file (#85349) (#85350) Although most of the time the stdout and stderr of the Java process is not needed since ES has its own log file, there are some cases where it is very important to have the output before ES has initialized its logging system. This commit changes the stdout/stderr of the ES process created by testclusters to use the same log file that ES will write to. There should not be contention with ProcessBuilder and ES trying to write at the same time because ES swaps out the stream handles to write to its logging system, so after that point ProcessBuilder will never see anymore output. relates #68333 25 March 2022, 01:33:06 UTC
2c174a1 [ML] Do not fetch source when finding index of last state docs (#85334) (#85343) There are two places where the ML code needs to find out which index in the rolling sequence of state indices the previous state document is in. Previously the search to do this was needlessly retrieving _source, which wastes memory. The change tells the two searches not to fetch _source, only the metadata. Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> 24 March 2022, 22:28:40 UTC
8e33912 Clarify where force merge will occur in ILM (#85187) (#85339) This commit clarifies the force merge behavior for the `forcemerge` action either as a standalone ILM action, or as part of the searchable snapshot action. When using a searchable snapshot it can be confusing especially because the force merge can occur in a phase prior to the phase configured in the ILM policy (since ILM doesn't migrate the index prior to the searchable snapshot mounting). 24 March 2022, 17:42:40 UTC
e3154a1 [Transform] Revert surfacing script deprecations warnings in deprecation info API (#85317) This largely reverts the change made in #84040. (A few bits of infrastructure like the extra option on OriginSettingClient are not reverted.) The reason is that the transform validation action that was used to validate the search syntax (including Painless scripts) actually runs the search. This can take a long time for transforms with big source indices, and leads to an unacceptably long response time for the deprecation info API. For example, if as a result of validating transform searches the deprecation info API takes longer than 30 seconds to respond then the Kibana upgrade assistant will time out and think it's never going to respond. Removing this functionality that was added in 7.17.1 leaves us no worse off than we were in 7.17.0 in terms of surfacing deprecated search syntax used in transforms, so we should do this for 7.17.2 to avoid the worse problem of the upgrade assistant hanging. Then in a later release we should try to find a way to surface deprecations in the Painless scripts embedded in transform searches without running the entire search. 24 March 2022, 11:28:04 UTC
57c9b57 Fix restart with auto-expand replicas (#85277) (#85306) No longer contract `number_of_replicas` when a node is marked restarting, since this poses a risk of the shard/index becoming unavailable. 24 March 2022, 10:01:17 UTC
1f03d5b [7.17] Fix leaking listeners bug on frozen tier (#85239) (#85300) * Fix leaking listeners bug on frozen tier (#85239) We identified a bug in searchable snapshot code that could prevent some search queries to be completed. This bug results in search tasks lingering on a data_frozen node for hours or days, filling the search thread pool and blocking other search requests to be processed. This bug can be reproduced on a data_frozen node with: multiple search queries requesting the same region from the shared cache the region corresponds to cached data not yet available in cache the searchable snapshot's thread pool that fetches cached data has no active workers the thread that will write the cached data (ie, the thread that picks up the Gap to fill) has restrictive permissions that prevents new thread to be created When such a situation occurs, the thread pool in charge of executing the AbstractRunnable that fills the cache throws an exception (in our case, an AccessControlException) that is not caught and not logged. In consequence the Gap that should have been written in cache is never filled with data and the listeners waiting for the cached data to be available won't be called back, blocking the execution of the searches waiting for the data. This pull request introduces an integration test that reproduce this situation: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "modifyThreadGroup") at java.security.AccessControlContext.checkPermission(AccessControlContext.java:485) ~[?:?] at java.security.AccessController.checkPermission(AccessController.java:1068) ~[?:?] at java.lang.SecurityManager.checkPermission(SecurityManager.java:416) ~[?:?] at org.elasticsearch.secure_sm.SecureSM.checkThreadGroupAccess(SecureSM.java:186) ~[?:8.2.0-SNAPSHOT] at org.elasticsearch.secure_sm.SecureSM.checkAccess(SecureSM.java:135) ~[?:8.2.0-SNAPSHOT] at java.lang.ThreadGroup.checkAccess(ThreadGroup.java:335) ~[?:?] at java.lang.Thread.<init>(Thread.java:422) ~[?:?] at java.lang.Thread.<init>(Thread.java:708) ~[?:?] at org.elasticsearch.common.util.concurrent.EsExecutors$EsThreadFactory.newThread(EsExecutors.java:274) ~[?:8.2.0-SNAPSHOT] at java.util.concurrent.ThreadPoolExecutor$Worker.<init>(ThreadPoolExecutor.java:630) ~[?:?] at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:920) ~[?:?] at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1364) ~[?:?] at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.execute(EsThreadPoolExecutor.java:66) ~[?:8.2.0-SNAPSHOT] at org.elasticsearch.xpack.searchablesnapshots.cache.shared.FrozenCacheService$CacheFileRegion.populateAndRead(FrozenCacheService.java:769) [?:8.2.0-SNAPSHOT] at org.elasticsearch.xpack.searchablesnapshots.cache.shared.FrozenCacheService$FrozenCacheFile.populateAndRead(FrozenCacheService.java:896) [?:8.2.0-SNAPSHOT] at org.elasticsearch.xpack.searchablesnapshots.store.input.FrozenIndexInput.readWithoutBlobCache(FrozenIndexInput.java:143) [?:8.2.0-SNAPSHOT] at org.elasticsearch.xpack.searchablesnapshots.store.input.MetadataCachingIndexInput.doReadInternal(MetadataCachingIndexInput.java:104) [?:8.2.0-SNAPSHOT] at org.elasticsearch.xpack.searchablesnapshots.store.input.BaseSearchableSnapshotIndexInput.readInternal(BaseSearchableSnapshotIndexInput.java:112) [?:8.2.0-SNAPSHOT] at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:291) [?:9.1.0 5b522487ba8e0f1002b50a136817ca037aec9686 - jtibs - 2022-03-16 10:32:40] at org.apache.lucene.store.BufferedIndexInput.readShort(BufferedIndexInput.java:243) [?:9.1.0 5b522487ba8e0f1002b50a136817ca037aec9686 - jtibs - 2022-03-16 10:32:40] at org.apache.lucene.util.packed.DirectReader$DirectPackedReader16.get(DirectReader.java:302) [?:9.1.0 5b522487ba8e0f1002b50a136817ca037aec9686 - jtibs - 2022-03-16 10:32:40] at org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$20.ordValue(Lucene90DocValuesProducer.java:852) [?:9.1.0 5b522487ba8e0f1002b50a136817ca037aec9686 - jtibs - 2022-03-16 10:32:40] at org.apache.lucene.index.SingletonSortedSetDocValues.advanceExact(SingletonSortedSetDocValues.java:81) [?:9.1.0 5b522487ba8e0f1002b50a136817ca037aec9686 - jtibs - 2022-03-16 10:32:40] at org.elasticsearch.index.fielddata.FieldData$10.advanceExact(FieldData.java:336) [?:8.2.0-SNAPSHOT] at org.elasticsearch.script.field.AbstractKeywordDocValuesField.setNextDocId(AbstractKeywordDocValuesField.java:44) [?:8.2.0-SNAPSHOT] at org.elasticsearch.search.lookup.LeafDocLookup.getScriptFieldFactory(LeafDocLookup.java:74) [?:8.2.0-SNAPSHOT] at org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:89) [?:8.2.0-SNAPSHOT] at org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:27) [?:8.2.0-SNAPSHOT] at org.elasticsearch.painless.PainlessScript$Script.execute(doc['text.raw'].value.toString().length() > 0:5) [?:?] at org.elasticsearch.index.query.ScriptQueryBuilder$ScriptQuery$1$1.matches(ScriptQueryBuilder.java:193) [?:8.2.0-SNAPSHOT] at org.apache.lucene.search.Weight$DefaultBulkScorer.scoreRange(Weight.java:281) [?:9.1.0 5b522487ba8e0f1002b50a136817ca037aec9686 - jtibs - 2022-03-16 10:32:40] at org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:254) [?:9.1.0 5b522487ba8e0f1002b50a136817ca037aec9686 - jtibs - 2022-03-16 10:32:40] at org.elasticsearch.search.internal.CancellableBulkScorer.score(CancellableBulkScorer.java:45) [?:8.2.0-SNAPSHOT] at org.apache.lucene.search.BulkScorer.score(BulkScorer.java:38) [?:9.1.0 5b522487ba8e0f1002b50a136817ca037aec9686 - jtibs - 2022-03-16 10:32:40] at org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:194) [?:8.2.0-SNAPSHOT] at org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:167) [?:8.2.0-SNAPSHOT] at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:555) [?:9.1.0 5b522487ba8e0f1002b50a136817ca037aec9686 - jtibs - 2022-03-16 10:32:40] at org.elasticsearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:232) [?:8.2.0-SNAPSHOT] at org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:187) [?:8.2.0-SNAPSHOT] at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:88) [?:8.2.0-SNAPSHOT] at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:462) [?:8.2.0-SNAPSHOT] at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:625) [?:8.2.0-SNAPSHOT] at org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:487) [?:8.2.0-SNAPSHOT] at org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:47) [?:8.2.0-SNAPSHOT] at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) [?:8.2.0-SNAPSHOT] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [?:8.2.0-SNAPSHOT] at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33) [?:8.2.0-SNAPSHOT] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773) [?:8.2.0-SNAPSHOT] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [?:8.2.0-SNAPSHOT] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?] at java.lang.Thread.run(Thread.java:833) [?:?] This permission issue has been fixed yesterday by #85180 but I still think it is valuable to have an integration test that combin scripts and searchable snapshot. In addition to this, this pull request changes the searchable snapshot code to always fail any Gap that was not correctly submitted for execution: this way the current search (and any other searches that are waiting for the cached data to be available) returns a search failure and not block indefinitely. In order to catch any future similar issue we also log and assert any exception that could be thrown when submitting the task to fill the required gaps. * Adjust for 7.17 24 March 2022, 09:42:46 UTC
fdb2c8f Validate index format agreement for system index descriptors (#85173) (#85290) * Validate index format agreement for system index descriptors (#85173) * Validate index format for system indices * Add bwc test for watcher meta version issue * Update docs/changelog/85173.yaml * Add include_type_name warning handler for bwc tests 23 March 2022, 20:21:39 UTC
b767d12 Wait longer for CCR auto-follow stats to appear in monitoring index (#85278) (#85288) The test `AutoFollowIT.testAutoFollowPatterns()` failed multiple times when waiting for CCR's auto-follow stats to be collected and indexed in the `.monitoring-es-*` index. This is possibly because monitoring data are collected every 10 seconds and some of them can take more time to be collected, exceeding the 30s timeout in the test. This pull request re-uses the `assertLongBusy()` so that the test will wait more time for the data to show up and in case the data are not here it will prints out the current CCR's auto-follow stats for debugging purpose. Closes #84403 23 March 2022, 17:45:44 UTC
0ef101c Return empty version instead of blowing up if we cannot find it (#85244) (#85271) SystemIndexManager.readMappingVersion() throws an exception in a few cases if we cannot find a valid version. For example in some really old versions of certain system indices we see version numbers like 7090399. This commit changes that method to return an empty version instead. 23 March 2022, 14:01:07 UTC
8278308 Improve DeprecationHttpIT robustness (#85235) (#85261) Backports the following commits to 7.17: Improve DeprecationHttpIT robustness (Improve DeprecationHttpIT robustness #85235) 23 March 2022, 12:44:27 UTC
ceefed4 [DOCS] Searchable snapshot data persistence (#85218) (#85269) Clarified that fully mounted indices persists on-disk data across restarts whereas partially mounted indices are cleared upon restarts. 23 March 2022, 12:09:34 UTC
c7988aa Fix atomic writes in HDFS (#85265) Ensure that the temporary file stream is closed before renaming it to the final name. This issue can impact HDFS repository operations, as we use the atomic writes primitive to ensure that writes to metadata files are atomic. Backport of #85210 23 March 2022, 11:41:56 UTC
194386e [7.17] Cleanup SystemIndexMigration tests (#84281) (#85259) Backports the following commits to 8.0: Cleanup SystemIndexMigration tests (#84281) and Fix randomness in SystemIndexMigrationIT #85224 23 March 2022, 10:22:28 UTC
568dc6b [7.17] Disable deprecation log indexing in docker test (#85150) (#85154) Backports the following commits to 7.17: Disable deprecation log indexing in docker test (Disable deprecation log indexing in docker test #85150) 23 March 2022, 08:51:05 UTC
66efe2f Re-add content from removed Security-specific snapshot pages (#85084) (#85251) This commit adds back some notes which were lost when we consolidated the snapshot/restore documentation into a single location. The notes in question are that: 1) If a Snapshot repository contains Security's feature state, then that repository contains security-sensitive information. This may be obvious to some, but is good to state explicitly. 2) Some files, such as the keystore and TLS keys, are not included in snapshots and are important to back up via other means. 22 March 2022, 22:56:32 UTC
b6b30f3 Wrap thread creation in doPrivileged call (#85180) (#85182) EsExecutors has a thread factory for thread construction, and both creates a thread in a given thread group and sets it as a daemon thread. Currently that thread creation happens in the access control context of the calling code, but this could happen from anywhere inside Elasticsearch. Since the point of EsExecutors is be the one place handling thread creation (for the most part), this should happen in the context of server, without caring about the whatever code triggered the thread pool to expand. 21 March 2022, 19:28:25 UTC
98e5f06 Rewrite match and match_phrase queries to term queries on keyword fields (#85165) Term queries can in certain circumstances (eg when run against constant keyword fields) rewrite themselves to match_no_docs queries, which is very useful for filtering out shards from searches and field_caps requests. But match and match_phrase queries can reduce down to simple term queries when there is no fuzziness defined on them, and when they are run using a keyword analyzer. This commit makes simple match and match_phrase rewrite themselves to term queries when run against keyword fields. Fixes #82515 21 March 2022, 17:45:42 UTC
f0e0727 REST tests for variable width historam (#84836) (#85116) This expands on the super basic REST tests for variable width histogram with some ever so slightly less basic tests. They are still pretty basic, but they do a good job of making sure that we don't break backwards compatibility with the API. Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> 21 March 2022, 15:25:39 UTC
ac3e38e Add isolated test coverage for example plugins 18 March 2022, 20:18:02 UTC
8583c9d [ML] Fixes for multi-line start patterns in text structure endpoint (#85109) This PR contains 3 fixes for the way multi-line start patterns are created in the text structure endpoint: 1. For delimited files the multi-line start pattern used to be based on the first field that was a timestamp, boolean or number. This PR adds the option of a low cardinality keyword field as an alternative (i.e. an enum field effectively). It means there is more chance of a field early in each record being chosen as the mechanism for determining whether a line is the first line of a record. 2. The multi-line start pattern for delimited files now only permits the delimiter character between fields, not within quoted fields. Previously it was possible for the multi-line start pattern to match continuation lines. Unfortunately this may mean we can no longer determine a multi-line start pattern for files whose only suitable field is to the right of fields that sometimes contain commas, and the only solution in this case will be to reorder the columns before importing the data. Hopefully this problem will be very rare. 3. For semi-structured text log files there is now a cap on the complexity of the multi-line start pattern. It has been observed that the patterns generated for slightly malformed CSV files could run for days against the malformed lines of those files - the classic problem of a regex that doesn't match but nearly does doing lots of backtracking. We now throw an error in this situation and suggest overriding the format to delimited. Backport of #85066 18 March 2022, 15:19:23 UTC
d68a509 Make sure the random generated shape can be indexed in GeoTilerTests (#84986) (#85108) Make sure the random generated shape can be indexed # Conflicts: # x-pack/plugin/spatial/src/test/java/org/elasticsearch/xpack/spatial/search/aggregations/bucket/geogrid/GeoGridTilerTestCase.java 18 March 2022, 15:05:08 UTC
f771030 Fix GeoGridTestCase#toXContent (#85093) This change make sure we create a valid random instance for GeoGridTestCase. 18 March 2022, 14:14:54 UTC
3b6af88 Change deprecationIssue level to critical for slow log level (#85062) index.indexing.slowlog.level or index.search.slowlog.level settings have been removed in 8.0 and should be marked as CRITICAL to prevent upgrade relates #84992 18 March 2022, 08:19:24 UTC
db2d446 Disable truststore test in FIPS mode (#84745) FIPS JSSE cannot read the common truststore formats (because the underlying crypto algorithms are not FIPS compliant). That means the `testDeprecationOfTruststoreWithNoTrustedEntries` test is not applicable on FIPS Resolves: #84703 18 March 2022, 04:02:22 UTC
7d15172 [DOCS] Fix Vector tiles search docs for features.id (#85067) (#85078) Removes the `features.id` property from the response body. This property was actually generated by the tool used to decode the mvt file to JSON. (cherry picked from commit a780558e4c1812d324c759d1f230576f6413e3bf) # Conflicts: # docs/reference/search/search-vector-tile-api.asciidoc Co-authored-by: Ignacio Vera <ivera@apache.org> 17 March 2022, 20:30:49 UTC
2ac9395 [7.17] Add list of allowed domains for Watcher email action (#84894) (#85031) * Add list of allowed domains for Watcher email action (#84894) This adds the `xpack.notification.email.account.domain_allowlist` dynamic cluster setting that allows an administrator to specify a list of domains to which emails are allowed to be sent. The default value for this setting is `["*"]` which means all domains are allowed. It supports rudimentary globbing (`*`) in the domain name, so `*.company.com` will work as a valid option. Resolves https://github.com/elastic/elasticsearch/issues/84739 * Fix compilation for older Java version 16 March 2022, 15:17:17 UTC
eaeb20f CompoundProcessor should also catch exceptions when executing a processor (#84838) Currently, CompoundProcessor does not catch Exception and if a processor throws an error and a method higher in the call stack doesn't catch the exception then pipeline execution stalls and bulk requests may not complete. Usually these exceptions are caught by IngestService#executePipelines(...) method, but when a processor executes async (for example: enrich processor) and the thread that executes enrich is no longer the original write thread then there is no logic that deals with failing pipeline execution and cleaning resources up. This then leads to memory leaks. Closes #84781 Also change how 'pipeline doesn't exist' error is thrown in TrackingResultProcessor. With the change to CompoundProcessor thrown exceptions are caught and delegated to handler. SimulateExecutionService in verbose mode ignores exceptions delegated to its handler, since it assumes that processorResultList contains the result (successful or not successful) of every processor in the pipeline. In case TrackingResultProcessor for PipelineProcessor couldn't find the mentioned pipeline then it just throws an error without updating the processorResultList. This commit addresses that. 16 March 2022, 14:39:57 UTC
c31dbf3 Revert enrich cache lookup optimisation (#85000) This PR reverts the optimisation that was added via #77259. This optimisation cleverly ensures no duplicate searches happen if multiple threads concurrently execute the same search. However there are issues with the implementation that cause issues like #84781. The optimisation make use of CompletableFuture and in this case we don't check whether the result has completed exceptionally. Which causes the callback not being invoked and this leads to bulk request not being completed and hanging around. The ingest framework due to its asynchronous nature is already complex and adding CompletableFuture into the mix makes debugging these issues very time consuming. This is the main reason why we like to revert this commit. 16 March 2022, 13:49:06 UTC
76eb914 Mark GeoIpDownloaderTask as completed after cancellation #84028 (#85014) (#85023) Persistent task framework requires tasks to be marked either completed or failed after cancellation. This was missing in GeoIpDownloader. 16 March 2022, 12:40:10 UTC
566de9e Increment version on system index settings change (#84994) (#85019) The test was failing on assertion around index settings version change in case actual settings were changed. This change adds missing version increment and readable messages to the related assertions. (cherry picked from commit 351a410ebfdbaf97d15971ad0a438f17d1126573) 16 March 2022, 11:04:06 UTC
c831a61 [7.17] Add BWC test for field-caps (#84455) (#84990) * Add BWC test for field-caps (#84455) Relates #83494 * fix tests * fix tests 15 March 2022, 21:42:13 UTC
a1645af Remove EOL platforms from testing matrix 15 March 2022, 19:26:26 UTC
5b227c5 Role descriptor link (#83753) Add a reference to the role descriptor 15 March 2022, 18:30:34 UTC
690e1db Remove usages of elasticsearch.build plugin in non-production projects (#84961) (#84964) (cherry picked from commit dc3d3b5e1242693e429aa003942ecfa0404a5b30) 15 March 2022, 16:55:44 UTC
a9fb2fc [ML] Fix Kibana date format and similar overrides in text structure endpoint (#84967) (#84974) This PR fixes two problems. The first caused the second to be noticed: 1. The timestamp format pattern for Kibana's default date format was wrong - it was designed to match double digit day of month only, when Kibana uses a single digit for 1-9. 2. Trying to use a timestamp format override to work around the first problem didn't work. This exposed a second, more general problem. When a timestamp format override is supplied we try to work out if a built in timestamp format would always include it, and use that in preference. (This allows friendly format names like "iso8601" to replace complex patterns.) However, the way this ability to substitute was being determined was flawed, leading to incorrect substitutions. In the case of the Kibana timestamp format it resulted in the correct format supplied as an override being replaced with the buggy built in timestamp format. To fix this, substitutions are now only performed if the correct number of digits are present as well as the Grok pattern matching. 15 March 2022, 11:41:47 UTC
846d803 GeoIP update logging (#84494) The only place we download the database (in the traditional pull from the configured endpoint sense) is in the `GeoIpDownloader`. The `DatabaseNodeService` is looking up the chunks of data we save in the `.geoip_databases` index. This updates the logging to be clear about who does what. 14 March 2022, 14:18:11 UTC
e822f3c Lower severity of IDE Checkstyle rules for JavaDoc (#84877) Lower the severity from "warning" to "info", and raise the length of methods that we ignore from 2 lines to 5. 14 March 2022, 10:16:52 UTC
a4edcff Stop checking non-asciidoc files for doc snippets (#84390) When SnippetsTask looks for doc snippets, the list of files it checks includes roughly 350 files that aren't asciidoc files. Image files (both png and jpg), yaml files, and so on. Set an explicit include pattern so that Gradle skips these files instead of trying to read them. 14 March 2022, 08:35:11 UTC
56feb22 Fix point visitor in DiskUsage API (#84909) (#84917) * Fix point visitor in DiskUsage API (#84909) We should use numIndexDimensions instead of numDimensions when comparing packedValues of PointValues. Relates #84816 * Fix tests 13 March 2022, 17:15:13 UTC
45d35aa Fix classpath for example build integration tests (#81837) This fixes two classpath issues for integration tests in the example build 1. adds log4j to the runtime classpath where needed 2. removes exclude of jakarta json which was added as a workaround for https://github.com/elastic/elasticsearch-java/issues/47 (cherry picked from commit 350fe2d21f4c5dce876de9423beaaefb61ec7319) 10 March 2022, 23:44:49 UTC
2d1a8c9 Bump timeout for feature migration in system index migration test (#84828) (#84893) * Bump timeout on assertBusy loop 10 March 2022, 22:54:23 UTC
840f020 Update Gradle wrapper to 7.4.1 (#84841) (#84884) 10 March 2022, 18:13:41 UTC
4c41c7a Do not send default ignore_throttled parameter since it is deprecated (#84827) 10 March 2022, 17:14:15 UTC
e7fa335 EQL: Clean any used memory by the sequence matcher and circuit breaker used bytes in case of exception (#84451) (#84821) (cherry picked from commit 4c243d69500bb983230434b828bfd798b68a202b) 10 March 2022, 07:13:54 UTC
cf52fc0 Note that S3 compat includes performance (#84798) Today the note in the docs about S3-compatible repositories notes that the repo must behave correctly, but it's also important that it has the same performance profile. This commit extends the docs to include this info. 09 March 2022, 12:36:02 UTC
03fbe90 [7.17] Remove existing indices/datastreams/aliases before simulating index template (#84675) (#84769) * Remove existing indices/datastreams/aliases before simulating index template (#84675) # Conflicts: # rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/indices.simulate_index_template/10_basic.yml # x-pack/plugin/src/yamlRestTest/resources/rest-api-spec/test/data_stream/180_simulate_existing_data_stream.yml * Fix backport typo issue 08 March 2022, 16:37:17 UTC
a59df06 [ML] Return all Datafeeds with GET Anomaly Detector (#84759) (#84767) Fixes a bug where only the first 10 datafeeds were included 08 March 2022, 16:33:58 UTC
be2fbc9 Pin specific pip py modules used by connector SDK (#84757) (#84762) This pins certain Python modules used by Tableau's connector SDK to certain versions. This will ensure build success consistency. 08 March 2022, 14:57:31 UTC
b6349f9 Remove Eclipse IDE contributor documentation (#84724) (#84732) # Conflicts: # CONTRIBUTING.md 07 March 2022, 20:58:59 UTC
17f5cc8 Backport doc fixes to 7.17 (#84722) * Update painless-reindex-context.asciidoc (#84444) (#84712) ctx['op'] should be set to 'noop', not 'none' when specifying no operation. Elasticsearch error when using 'none': ```json { "error" : { "root_cause" : [ { "type" : "illegal_argument_exception", "reason" : "Operation type [none] not allowed, only [noop, index, delete] are allowed" } ], "type" : "illegal_argument_exception", "reason" : "Operation type [none] not allowed, only [noop, index, delete] are allowed" }, "status" : 400 } ``` Co-authored-by: jalvar08 <jeovanny.alvarez@gmail.com> * [DOCS] Update install instructions for Debian/Ubuntu (#84645) (#84714) The use of `apt-key` is deprecated and will no longer be available after Debian 11 and Ubuntu 22.04. This updates the installation instructions for Debian-based distributions. Closes #84644 Co-authored-by: er0k <er0k@users.noreply.github.com> * Fix some typos in plugins & reference docs (#84667) (#84717) This pull request removes a few instances of duplicate words or punctuation and erroneous spelling from the docs. Co-authored-by: Abele Mălan <6689720+AbeleMM@users.noreply.github.com> Co-authored-by: jalvar08 <jeovanny.alvarez@gmail.com> Co-authored-by: er0k <er0k@users.noreply.github.com> Co-authored-by: Abele Mălan <6689720+AbeleMM@users.noreply.github.com> 07 March 2022, 19:04:12 UTC
e5243e7 Configure required IDEA plugins (#84674) Closes #53812. The CheckStyle-IDEA and EcliseCodeFormatter IDEA plugins are to all intents and purposes required when using IntelliJ to work on Elasticsearch, so add configuration that marks then as required. 07 March 2022, 17:02:48 UTC
2a46f1d Update intellij security policy for later versions of IDEA (#84681) (#84686) 07 March 2022, 16:25:18 UTC
back to top