1.3.15

2024-03-01 Version 1.3.15 Release Notes

Upgrades

• Bump netty from 4.1.100.Final to 4.1.104.Final (#11294, #11775), #12034)
• Bump netty from 4.1.106.Final to 4.1.107.Final (#12372)
• Bump org.apache.kerby:kerb-admin from 1.0.1 to 2.0.3 (#12194)

2.12.0

2024-02-09 Version 2.12.0 Release Notes

[2.12.0]

Added

• [Admission control] Add Resource usage collector service and resource usage tracker (#9890)
• [Admission control] Add enhancements to FS stats to include read/write time, queue size and IO time (#10541)
• [Remote cluster state] Change file names for remote cluster state (#10557)
• [Search Pipelines] Add request-scoped state shared between processors (and three new processors) (#9405)
• Per request phase latency (#10351)
• [Remote Store] Add repository stats for remote store(#10567)
• [Remote cluster state] Upload global metadata in cluster state to remote store(#10404)
• [Remote cluster state] Download functionality of global metadata from remote store (#10535)
• [Remote cluster state] Restore global metadata from remote store when local state is lost after quorum loss (#10404)
• [Remote cluster state] Make index and global metadata upload timeout dynamic cluster settings (#10814)
• Add search query categorizer (#10255)
• Per request phase latency (#10351)
• Add cluster state stats (#10670)
• Remove ingest processor supports excluding fields (#10967, #11983)
• [Tiered caching] Enabling serialization for IndicesRequestCache key object (#10275)
• [Tiered caching] Defining interfaces, listeners and extending IndicesRequestCache with Tiered cache support (#10753)
• [Remote cluster state] Restore cluster state version during remote state auto restore (#10853)
• Update the indexRandom function to create more segments for concurrent search tests (10247)
• Add support for query profiler with concurrent aggregation (#9248)
• Introduce ConcurrentQueryProfiler to profile query using concurrent segment search path and support concurrency during rewrite and create weight (10352)
• Implement on behalf of token passing for extensions (#8679)
• Provide service accounts tokens to extensions (#9618)
• [Streaming Indexing] Introduce new experimental server HTTP transport based on Netty 4 and Project Reactor (Reactor Netty) (#9672)
• Enable must_exist parameter for update aliases API (#11210)
• Add back half_float BKD based sort query optimization (#11024)
• Request level coordinator slow logs (#10650)
• Add template snippets support for field and target_field in KV ingest processor (#10040)
• Allowing pipeline processors to access index mapping info by passing ingest service ref as part of the processor factory parameters (#10307)
• Add experimental SIMD implementation of B-tree to round down dates (#11194)
• Make number of segment metadata files in remote segment store configurable (#11329)
• Allow changing number of replicas of searchable snapshot index (#11317)
• Adding slf4j license header to LoggerMessageFormat.java (#11069)
• [BWC and API enforcement] Introduce checks for enforcing the API restrictions (#11175)
• Maintainer approval check (#11378)
• Create separate transport action for render search template action (#11170)
• Add additional handling in SearchTemplateRequest when simulate is set to true (#11591)
• Introduce cluster level setting cluster.index.restrict.replication.type to prevent replication type setting override during index creations(#11583)
• Add match_only_text field that is optimized for storage by trading off positional queries performance (#6836)
• Add copy ingest processor (#11870)
• Introduce new feature flag "WRITEABLE_REMOTE_INDEX" to gate the writeable remote index functionality (#11717)
• Bump OpenTelemetry from 1.32.0 to 1.34.1 (#11891)
• Add remove_by_pattern ingest processor (#11920)
• Support index level allocation filtering for searchable snapshot index (#11563)
• Add org.opensearch.rest.MethodHandlers and RestController#getAllHandlers (11876)
• New DateTime format for RFC3339 compatible date fields (#11465)
• Add community_id ingest processor (#12121)
• Add support for Google Application Default Credentials in repository-gcs (#8394)
• Remove concurrent segment search feature flag for GA launch (#12074)
• Enable Fuzzy codec for doc id fields using a bloom filter (#11022)
• [Metrics Framework] Adds support for Histogram metric (#12062)
• Support for returning scores in matched queries (#11626)

Dependencies

• Bumps jetty version to 9.4.52.v20230823 to fix GMS-2023-1857 (#9822)
• Bump Lucene from 9.7.0 to 9.8.0 (10276)
• Bump commons-io:commons-io from 2.13.0 to 2.15.1 (#10294, #11001, #11002, #11446, #11554, #11560, #11796)
• Bump com.google.api.grpc:proto-google-common-protos from 2.10.0 to 2.25.1 (#10208, #10298)
• Bump com.netflix.nebula.ospackage-base from 11.4.0 to 11.8.0 (#10295, #11630, #12167)
• Bump org.apache.zookeeper:zookeeper from 3.9.0 to 3.9.1 (#10506)
• Bump de.thetaphi:forbiddenapis from 3.5.1 to 3.6 (#10508)
• Bump org.codehaus.woodstox:stax2-api from 4.2.1 to 4.2.2 (#10639)
• Bump org.bouncycastle:bc-fips from 1.0.2.3 to 1.0.2.4 (#10297)
• Bump com.google.http-client:google-http-client from 1.43.2 to 1.43.3 (#10635)
• Bump com.squareup.okio:okio from 3.5.0 to 3.7.0 (#10637, #11632)
• Bump org.apache.logging.log4j:log4j-core from 2.20.0 to 2.22.1 (#10858, #11000, #11270, #11695)
• Bump `aws-actions/configure-aws-creden...

2.11.1

2023-11-20 Version 2.11.1 Release Notes

[2.11.1]

Changed

• Use iterative approach to evaluate Regex.simpleMatch (#11060)

Fixed

• [BUG] Disable sort optimization for HALF_FLOAT (#10999)
• Adding version condition while adding geoshape doc values to the index, to ensure backward compatibility.(#11095)
• Remove shadowJar from lang-painless module publication (#11369)

2.11.0

2023-10-12 Version 2.11.0 Release Notes

[2.11]

Added

• Add coordinator level stats for search latency (#8386)
• Add metrics for thread_pool task wait time (#9681)
• Async blob read support for S3 plugin (#9694)
• [Telemetry-Otel] Added support for OtlpGrpcSpanExporter exporter (#9666)
• Async blob read support for encrypted containers (#10131)
• Implement Visitor Design pattern in QueryBuilder to enable the capability to traverse through the complex QueryBuilder tree. (#10110)
• Add capability to restrict async durability mode for remote indexes (#10189)
• Add Doc Status Counter for Indexing Engine (#4562)
• Add unreferenced file cleanup count to merge stats (#10204)
• Configurable merge policy for index with an option to choose from LogByteSize and Tiered merge policy (#9992)
• [Remote Store] Add support to restrict creation & deletion if system repository and mutation of immutable settings of system repository (#9839)
• Improve compressed request handling (#10261)

Dependencies

• Bump JNA version from 5.5 to 5.13 (#9963)
• Bump peter-evans/create-or-update-comment from 2 to 3 (#9575)
• Bump actions/checkout from 2 to 4 (#9968)
• Bump OpenTelemetry from 1.26.0 to 1.30.1 (#9950)
• Bump org.apache.commons:commons-compress from 1.23.0 to 1.24.0 ([#9973, #9972](#9973, #9972))
• Bump com.google.cloud:google-cloud-core-http from 2.21.1 to 2.23.0 (#9971)
• Bump mockito from 5.4.0 to 5.5.0 (#10022)
• Bump bytebuddy from 1.14.3 to 1.14.7 (#10022)
• Bump com.zaxxer:SparseBitSet from 1.2 to 1.3 (#10098)
• Bump tibdex/github-app-token from 1.5.0 to 2.1.0 (#10125)
• Bump org.wiremock:wiremock-standalone from 2.35.0 to 3.1.0 (#9752)
• Bump org.eclipse.jgit from 6.5.0 to 6.7.0 (#10147)
• Bump codecov/codecov-action from 2 to 3 (#10209)
• Bump com.google.http-client:google-http-client-jackson2 from 1.43.2 to 1.43.3 (#10126)
• Bump org.xerial.snappy:snappy-java from 1.1.10.3 to 1.1.10.5 (#10206, #10299)
• Bump org.bouncycastle:bcpkix-jdk15to18 from 1.75 to 1.76 (10219)`
• Bump org.bouncycastle:bcprov-jdk15to18 from 1.75 to 1.76 (10219)`
• Bump org.bouncycastle:bcmail-jdk15to18 from 1.75 to 1.76 (10219)`
• Bump asm from 9.5 to 9.6 (#10302)
• Bump netty from 4.1.97.Final to 4.1.99.Final (#10303)
• Bump peter-evans/create-pull-request from 3 to 5 (#10301)
• Bump org.apache.avro:avro from 1.11.2 to 1.11.3 (#10210)
• Bump netty from 4.1.99.Final to 4.1.100.Final (#10564)

Changed

• Add instrumentation in rest and network layer. (#9415)
• Allow parameterization of tests with OpenSearchIntegTestCase.SuiteScopeTestCase annotation (#9916)
• Add instrumentation in transport service. (#10042)
• [Tracing Framework] Add support for SpanKind. (#10122)
• Pass parent filter to inner query in nested query (#10246)
• Disable concurrent segment search when terminate_after is used (#10200)
• Add instrumentation in Inbound Handler. (#100143)
• Enable remote segment upload backpressure by default (#10356)
• [Remote Store] Add support to reload repository metadata inplace (#9569)
• [Metrics Framework] Add Metrics framework. (#10241)
• Updating the separator for RemoteStoreLockManager since underscore is allowed in base64UUID url charset (#10379)

Removed

• Remove spurious SGID bit on directories (#9447)

Fixed

• Fix ignore_missing parameter has no effect when using template snippet in rename ingest processor (#9725)
• Fix broken backward compatibility from 2.7 for IndexSorted field indices (#10045)
• Fix concurrent search NPE when track_total_hits, terminate_after and size=0 are used (#10082)
• Fix remove ingest processor handing ignore_missing parameter not correctly (10089)
• Fix registration and initialization of multiple extensions (10256)
• Fix circular dependency in Settings initialization (10194)
• Fix Segment Replication ShardLockObtainFailedException bug during index corruption (10370)

2.10.0

2023-09-08 Version 2.10.0 Release Notes

[2.10]

Added

• Add server version as REST response header #6583
• Start replication checkpointTimers on primary before segments upload to remote store. (#8221#8221)
• Introduce new static cluster setting to control slice computation for concurrent segment search. (#8847)
• Add configuration for file cache size to max remote data ratio to prevent oversubscription of file cache (#8606)
• Disallow compression level to be set for default and best_compression index codecs (#8737#8737)
• [distribution/archives] [Linux] [x64] Provide the variant of the distributions bundled with JRE (#8195#8195)
• Prioritize replica shard movement during shard relocation (#8875)
• Introducing Default and Best Compression codecs as their algorithm name (#9123)
• Make SearchTemplateRequest implement IndicesRequest.Replaceable (#9122)
• [BWC and API enforcement] Define the initial set of annotations, their meaning and relations between them (#9223)
• [Remote Store] Add Segment download stats to remotestore stats API (#8718)
• [Remote Store] Add remote segment transfer stats on NodesStats API (#9168 #9393 #9454)
• [Segment Replication] Support realtime reads for GET requests (#9212)
• Allow test clusters to run with TLS (#8900)
• Add jdk.incubator.vector module support for JDK 20+ (#8601)
• [Feature] Expose term frequency in Painless script score context (#9081)
• Add support for reading partial files to HDFS repository (#9513)
• [Remote Store] Rate limiter integration for remote store uploads and downloads(#9448)
• [BWC and API enforcement] Decorate the existing APIs with proper annotations (part 1) (#9520)
• Add support for extensions to search responses using SearchExtBuilder (#9379)
• [Remote State] Create service to publish cluster state to remote store (#9160)
• Core crypto library to perform encryption and decryption of source content (#8466)
• Expose DelimitedTermFrequencyTokenFilter to allow providing term frequencies along with terms (#9479)
• APIs for performing async blob reads and async downloads from the repository using multiple streams (#9592)
• Add concurrent segment search related metrics to node and index stats (#9622)
• Add average concurrency metric for concurrent segment search (#9670)
• Introduce cluster default remote translog buffer interval setting (#9584)
• Added encryption-sdk lib to provide encryption and decryption capabilities (#8466 #9289)
• [Segment Replication] Adding segment replication statistics rolled up at index, node and cluster level (#9709)
• Added crypto-kms plugin to provide AWS KMS based key providers for encryption/decryption. (#8465)
• [Remote state] Integrate remote cluster state in publish/commit flow (#9665)
• [Remote Store] Changes to introduce repository registration during bootstrap via node attributes. (#9105)
• [Remote state] Auto restore index metadata from last known cluster state (#9831)

Dependencies

• Bump org.apache.logging.log4j:log4j-core from 2.17.1 to 2.20.0 (#8307)
• Bump io.grpc:grpc-context from 1.46.0 to 1.57.1 (#8726, #9145)
• Bump com.netflix.nebula:gradle-info-plugin from 12.1.5 to 12.1.6 (#8724)
• Bump commons-codec:commons-codec from 1.15 to 1.16.0 (#8725)
• Bump org.apache.zookeeper:zookeeper from 3.8.1 to 3.9.0 (#8844, #9146)
• Bump org.gradle.test-retry from 1.5.3 to 1.5.4 (#8842)
• Bump com.netflix.nebula.ospackage-base from 11.3.0 to 11.4.0 (#8838)
• Bump com.google.http-client:google-http-client-gson from 1.43.2 to 1.43.3 (#8840)
• OpenJDK Update (July 2023 Patch releases) (#8869)
• Bump hadoop libraries from 3.3.4 to 3.3.6 (#6995)
• Bump com.gradle.enterprise from 3.13.3 to 3.14.1 (#8996)
• Bump org.apache.commons:commons-lang3 from 3.12.0 to 3.13.0 (#8995)
• Bump com.google.cloud:google-cloud-core-http from 2.21.0 to 2.21.1 (#8999)
• Bump com.maxmind.geoip2:geoip2 from 4.0.1 to 4.1.0 (#8998)
• Bump org.apache.commons:commons-lang3 from 3.12.0 to 3.13.0 in /plugins/repository-hdfs (#8997)
• Bump netty from 4.1.94.Final to 4.1.96.Final (#9030)
• Bump com.google.jimfs:jimfs from 1.2 to 1.3.0 (#9080)
• Bump io.projectreactor.netty:reactor-netty-http from 1.1.8 to 1.1.9 (#9147)
• Bump org.apache.maven:maven-model from 3.9.3 to 3.9.4 (#9148)
• Bump com.azure:azure-storage-blob from 12.22.3 to 12.23.0 (#9231)
• Bump com.diffplug.spotless from 6.19.0 to 6.20.0 (#9227)
• Bump org.xerial.snappy:snappy-java from 1.1.8.2 to 1.1.10.3 (#9252)
• Bump com.squareup.okhttp3:okhttp from 4.9.3 to 4.11.0 (#9252)
• Bump com.squareup.okio:okio from 2.8.0 to 3.5.0 (#9252)
• Bump com.google.code.gson:gson from 2.9.0 to 2.10.1 (#9230)
• Bump lycheeverse/lychee-action from 1.2.0 to 1.8.0 (#9228)
• Bump snakeyaml from 2.0 to 2.1 (#9269)
• Bump aws-actions/configure-aws-credentials from 1 to 2 (#9302)
• Bump com.github.luben:zstd-jni from 1.5.5-3 to 1.5.5-5 (#9431
• Bump netty from 4.1.96.Final to 4.1.97.Final (#9553)
• Bump io.grpc:grpc-api from 1.57.1 to 1.57.2 (#9578)
• Add Encryption SDK dependencies (#8466)

Changed

• Default to mmapfs within hybridfs (#8508)
• Perform aggregation postCollection in ContextIndexSearcher after searching leaves (#8303)
• Make Span exporter configurable (#8620)
• Perform aggregation postCollection in ContextIndexSearcher after searching leaves (#8303)
• [Refactor] StreamIO from common to core.common namespace in core lib ([#8157](htt...

1.3.13

2023-09-14 Version 1.3.13 Release Notes

Upgrades

• Bump netty from 4.1.96.Final to 4.1.97.Final (#9553)
• Bump org.xerial.snappy:snappy-java from 1.1.8.2 to 1.1.10.3 (#9252)
• Bump com.squareup.okhttp3:okhttp from 4.9.3 to 4.11.0 (#9252)
• Bump com.squareup.okio:okio from 2.8.0 to 3.5.0 (#9252)

2.9.0

2023-07-13 Version 2.9.0 Release Notes

[2.9]

Added

• [SearchPipeline] Add new search pipeline processor type, SearchPhaseResultsProcessor, that can modify the result of one search phase before starting the next phase.(#7283)
• Add task cancellation monitoring service (#7642)
• Add TokenManager Interface (#7452)
• Add Remote store as a segment replication source (#7653)
• Implement concurrent aggregations support without profile option (#7514)
• Add dynamic index and cluster setting for concurrent segment search (#7956)
• Add descending order search optimization through reverse segment read. (#7967)
• [Search pipelines] Added search pipelines output to node stats (#8053)
• Update components of segrep backpressure to support remote store. (#8020)
• Make remote cluster connection setup in async (#8038)
• Add API to initialize extensions (#8029#8029)
• Add distributed tracing framework (#7543)
• Enable Point based optimization for custom comparators (#8168)
• [Extensions] Support extension additional settings with extension REST initialization (#8414)
• Adds mock implementation for TelemetryPlugin (#7545)
• Support transport action names when registering NamedRoutes (#7957)
• Create concept of persistent ThreadContext headers that are unstashable (#8291#8291)
• [Search pipelines] Add Global Ignore_failure options for Processors (#8373)
• Enable Partial Flat Object (#7997)
• Add jdk.incubator.vector module support for JDK 20+ (#8601)
• Introduce full support for Search Pipeline (#8613)
• Add partial results support for concurrent segment search (#8306)

Dependencies

• Bump com.azure:azure-storage-common from 12.21.0 to 12.21.1 (#7566, #7814)
• Bump com.google.guava:guava from 30.1.1-jre to 32.1.1-jre (#7565, #7811, #7807, #7808, #8402, #8400, #8401, #8581)
• Bump net.minidev:json-smart from 2.4.10 to 2.4.11 (#7660, #7812)
• Bump org.gradle.test-retry from 1.5.2 to 1.5.3 (#7810)
• Bump com.diffplug.spotless from 6.17.0 to 6.18.0 (#7896)
• Bump jackson from 2.15.1 to 2.15.2 (#7897)
• Add com.github.luben:zstd-jni version 1.5.5-3 (#2996)
• Bump netty from 4.1.91.Final to 4.1.93.Final (#7901)
• Bump com.amazonaws 1.12.270 to software.amazon.awssdk 2.20.55 (7372)
• Add org.reactivestreams 1.0.4 (7372)
• Bump com.networknt:json-schema-validator from 1.0.81 to 1.0.85 ([7968], #8255)
• Bump com.netflix.nebula:gradle-extra-configurations-plugin from 9.0.0 to 10.0.0 in /buildSrc (#7068)
• Bump com.google.guava:guava from 32.0.0-jre to 32.0.1-jre (#8009)
• Bump commons-io:commons-io from 2.12.0 to 2.13.0 (#8014, #8013, #8010)
• Bump com.diffplug.spotless from 6.18.0 to 6.19.0 (#8007)
• Bump 'com.azure:azure-storage-blob to 12.22.2 from 12.21.1 (#8043)
• Bump org.jruby.joni:joni from 2.1.48 to 2.2.1 (#8015, #8254)
• Bump com.google.guava:guava from 32.0.0-jre to 32.0.1-jre (#8011, #8012, #8107)
• Bump io.projectreactor:reactor-core from 3.4.18 to 3.5.6 in /plugins/repository-azure (#8016)
• Bump spock-core from 2.1-groovy-3.0 to 2.3-groovy-3.0 (#8122)
• Bump com.networknt:json-schema-validator from 1.0.83 to 1.0.84 (#8141)
• Bump com.netflix.nebula:gradle-info-plugin from 12.1.3 to 12.1.5 (#8139, #8568)
• Bump commons-io:commons-io from 2.12.0 to 2.13.0 in /plugins/discovery-azure-classic (#8140)
• Bump mockito from 5.2.0 to 5.4.0 (#8181)
• Bump netty from 4.1.93.Final to 4.1.94.Final (#8191)
• Bump org.apache.hadoop:hadoop-minicluster from 3.3.5 to 3.3.6 (#8257)
• Bump io.projectreactor.netty:reactor-netty-http from 1.1.7 to 1.1.8 (#8256)
• [Upgrade] Lucene 9.7.0 release (#8272)
• Bump org.jboss.resteasy:resteasy-jackson2-provider from 3.0.26.Final to 6.2.4.Final in /qa/wildfly (#8209)
• Bump com.google.api-client:google-api-client from 1.34.0 to 2.2.0 (#8276)
• Update Apache HttpCore/ HttpClient and Apache HttpCore5 / HttpClient5 dependencies (#8434)
• Bump org.apache.maven:maven-model from 3.9.2 to 3.9.3 (#8403)
• Bump io.projectreactor.netty:reactor-netty and io.projectreactor.netty:reactor-netty-core from 1.1.7 to 1.1.8 (#8405)
• Bump com.azure:azure-storage-blob from 12.22.2 to 12.22.3 (#8572)
• Bump net.minidev:json-smart from 2.4.11 to 2.5.0 (#8575, #8576)
• Bump com.google.jimfs:jimfs from 1.2 to 1.3.0 (#8577, #8571)
• Bump com.networknt:json-schema-validator from 1.0.85 to 1.0.86 (#8573)
• Bump com.google.cloud:google-cloud-core-http from 2.17.0 to 2.21.0 (#8586)
• Bump com.google.jimfs:jimfs from 1.2 to 1.3.0 (#8585)

Changed

• Replace jboss-annotations-api_1.2_spec with jakarta.annotation-api (#7836)
• Reduce memory copy in zstd compression (#7681)
• Add min, max, average and thread info to resource stats in tasks API (#7673)
• Add ZSTD compression for snapshotting (#2996)
• Change com.amazonaws.sdk.ec2MetadataServiceEndpointOverride to aws.ec2MetadataServiceEndpoint (7372)
• Change com.amazonaws.sdk.stsEndpointOverride to aws.stsEndpointOverride (7372)
• Add new query profile collector fields with concurrent search execution (#7898)
• Align range and default value for deletes_pct_allowed in merge policy (#7730)
• Rename QueryPhase actors like Suggest, Rescore to be processors rather than phase (#8025)
• Compress and cache cluster state during validate join request (#7321)
• [Snapshot Interop] Add Changes in Create Snapshot Flow for remote store interoperability. (#7118)
• Allow insecure string settings to warn-log usage and advise to migration of a newer secure variant (#5496)
• Add self-organizing hash table to improve the performance of bucket aggregations (#7652)
• Check UTF16 string size before converting to String to avoid OOME (#7963)
• Move ZSTD compression codecs out of the sandbox (#7908)
• Update ZSTD default compression level (#8471)
• [Search Pipelines] Pass pipeline creation context to processor factories (#8164)
• Enabling compression levels for zstd and zstd_no_dict (#8312)
• Optimize Metadata build() to skip redundant computations as part of ClusterState build (#7853)
• Add safeguard limits for file cache during node level allocation (#8208)
• Move span actions t...

2.8.0

2023-06-06 Version 2.8.0 Release Notes

[2.8]

Added

• [Extensions] Moving Extensions APIs to support cross versions via protobuf. (#7402)
• [Extensions] Add IdentityPlugin into core to support Extension identities (#7246)
• Add connectToNodeAsExtension in TransportService (#6866)
• [Search Pipelines] Accept pipelines defined in search source (#7253)
• [Search Pipelines] Add default_search_pipeline index setting (#7470)
• [Search Pipelines] Add RenameFieldResponseProcessor for Search Pipelines (#7377)
• [Search Pipelines] Split search pipeline processor factories by type (#7597)
• [Search Pipelines] Add script processor (#7607)
• Add 'unsigned_long' numeric field type (#6237)
• Add back primary shard preference for queries (#7375)
• Add task cancellation timestamp in task API (#7455)
• Adds ExtensionsManager.lookupExtensionSettingsById (#7466)
• SegRep with Remote: Add hook for publishing checkpoint notifications after segment upload to remote store (#7394)
• Add search_after query optimizations with shard/segment short cutting (#7453)
• Provide mechanism to configure XContent parsing constraints (after update to Jackson 2.15.0 and above) (#7550)
• Support to clear filecache using clear indices cache API (#7498)
• Create NamedRoute to map extension routes to a shortened name (#6870)
• Added @dbwiddis as on OpenSearch maintainer (#7665)
• [Extensions] Add ExtensionAwarePlugin extension point to add custom settings for extensions (#7526)
• Add new cluster setting to set default index replication type (#7420)

Dependencies

• Bump com.netflix.nebula:gradle-info-plugin from 12.0.0 to 12.1.3 (#7564)
• Bump com.netflix.nebula:nebula-publishing-plugin from 19.2.0 to 20.2.0
• Bump com.google.protobuf:protobuf-java from 3.22.2 to 3.22.3
• Bump jackson from 2.14.2 to 2.15.0 (#7286)
• Bump com.netflix.nebula:nebula-publishing-plugin from 20.2.0 to 20.3.0
• Bump com.netflix.nebula.ospackage-base from 11.0.0 to 11.3.0
• Bump gradle.plugin.com.github.johnrengelman:shadow from 7.1.2 to 8.0.0
• Bump jna from 5.11.0 to 5.13.0
• Bump commons-io:commons-io from 2.7 to 2.12.0 (#7661, #7658, #7656)
• Bump org.apache.shiro:shiro-core from 1.9.1 to 1.11.0 (#7397)
• Bump jetty-server in hdfs-fixture from 9.4.49.v20220914 to 9.4.51.v20230217 (#7405)
• OpenJDK Update (April 2023 Patch releases) (#7448
• Bump org.apache.commons:commons-compress from 1.22 to 1.23.0 (#7462)
• Bump com.azure:azure-core from 1.34.0 to 1.39.0
• Bump com.networknt:json-schema-validator from 1.0.78 to 1.0.81 (#7460)
• Bump Apache Lucene to 9.6.0 (#7505)
• Bump com.google.cloud:google-cloud-core-http from 1.93.3 to 2.17.0 (#7488)
• Bump com.google.guava:guava from 30.1.1-jre to 32.0.0-jre (#7565, #7811, #7808, #7807)
• Bump com.azure:azure-storage-common from 12.20.0 to 12.21.1 (#7566, #7814)
• Bump org.apache.commons:commons-compress from 1.22 to 1.23.0 (#7563)
• Bump jackson from 2.15.0 to 2.15.1 (#7603)
• Bump net.minidev:json-smart from 2.4.10 to 2.4.11 (#7660, #7812)
• Bump io.projectreactor.netty:reactor-netty-core from 1.1.5 to 1.1.7 (#7657)
• Bump org.apache.maven:maven-model from 3.9.1 to 3.9.2 (#7655)
• Bump com.google.api:gax from 2.17.0 to 2.27.0 (#7697)
• Bump io.projectreactor.netty:reactor-netty from 1.1.4 to 1.1.7 (#7725)
• Bump io.projectreactor.netty:reactor-netty-http from 1.1.4 to 1.1.7 (#7725)
• Bump com.google.http-client:google-http-client-appengine from 1.41.8 to 1.43.2 (#7813)
• Bump org.gradle.test-retry from 1.5.2 to 1.5.3 (#7810)

Changed

• Enable ./gradlew build on MacOS by disabling bcw tests (#7303)
• Moved concurrent-search from sandbox plugin to server module behind feature flag (#7203)
• Allow access to indices cache clear APIs for read only indexes (#7303)
• Changed concurrent-search threadpool type to be resizable and support task resource tracking (#7502)
• Default search preference to _primary for searchable snapshot indices (#7628)
• [Segment Replication] Remove codec name string match check for checkpoints (#7741)

Fixed

• Add more index blocks check for resize APIs (#6774)
• Replaces ZipInputStream with ZipFile to fix Zip Slip vulnerability (#7230)
• Add missing validation/parsing of SearchBackpressureMode of SearchBackpressureSettings (#7541)
• [Search Pipelines] Better exception handling in search pipelines (#7735)
• Fix input validation in segments and delete pit request (#6645)

1.3.10

Release Notes: https://github.com/opensearch-project/OpenSearch/blob/main/release-notes/opensearch.release-notes-1.3.10.md

2.7.0

2023-04-18 Version 2.7.0 Release Notes

[2.7]

Added

• Add GeoTile and GeoHash Grid aggregations on GeoShapes. (#5589)
• Disallow multiple data paths for search nodes (#6427)
• [Segment Replication] Allocation and rebalancing based on average primary shard count per index (#6422)
• The truncation limit of the OpenSearchJsonLayout logger is now configurable (#6569)
• Add 'base_path' setting to File System Repository (#6558)
• Return success on DeletePits when no PITs exist. (#6544)
• Add initial search pipelines (#6587)
• Add node repurpose command for search nodes (#6517)
• Add wait_for_completion parameter to resize, open, and forcemerge APIs (#6434)
• [Segment Replication] Apply backpressure when replicas fall behind (#6563)
• [Remote Store] Integrate remote segment store in peer recovery flow (#6664)
• Enable sort optimization for all NumericTypes (#6464
• Remove 'cluster_manager' role attachment when using 'node.master' deprecated setting (#6331)
• Add new cluster settings to ignore weighted round-robin routing and fallback to default behaviour. (#6834)
• Add experimental support for ZSTD compression. (#3577)
• [Segment Replication] Add point in time and scroll query compatibility. (#6644)
• Add retry delay as dynamic setting for cluster maanger throttling. (#6998)
• Introduce full support for searchable snapshots (#5087)
• Introduce full support for Segment Replication (#5147)

Dependencies

• Bump org.apache.logging.log4j:log4j-core from 2.18.0 to 2.20.0 (#6490)
• Bump com.azure:azure-storage-common from 12.19.3 to 12.20.0 (#6492
• Bump snakeyaml from 1.33 to 2.0 (#6511)
• Bump io.projectreactor.netty:reactor-netty from 1.1.3 to 1.1.4
• Bump com.avast.gradle:gradle-docker-compose-plugin from 0.15.2 to 0.16.11
• Bump net.minidev:json-smart from 2.4.8 to 2.4.9
• Bump com.google.protobuf:protobuf-java to 3.22.2 (#6994)
• Bump Netty to 4.1.90.Final (#6677
• Bump com.diffplug.spotless from 6.15.0 to 6.17.0
• Bump org.apache.zookeeper:zookeeper from 3.8.0 to 3.8.1
• Bump net.minidev:json-smart from 2.4.7 to 2.4.10
• Bump org.apache.maven:maven-model from 3.6.2 to 3.9.1
• Bump org.codehaus.jettison:jettison from 1.5.3 to 1.5.4 (#6878)
• Add com.github.luben:zstd-jni:1.5.5-1 (#3577)
• Bump: Netty from 4.1.90.Final to 4.1.91.Final , ASM 9.4 to ASM 9.5, ByteBuddy 1.14.2 to 1.14.3 (#6981)
• Bump com.azure:azure-storage-blob from 12.15.0 to 12.21.1
• Bump org.gradle.test-retry from 1.5.1 to 1.5.2
• Bump org.apache.hadoop:hadoop-minicluster from 3.3.4 to 3.3.5

Changed

• Require MediaType in Strings.toString API (#6009)
• [Refactor] XContent base classes from xcontent to core library (#5902)
• Added a new field type: flat_object (#6507)
• Changed opensearch-env to respect already set OPENSEARCH_HOME environment variable (#6956)
• Increased visibility of BaseRestHandler’s unrecognized method using a new public unrecognizedStrings method. (#7125)

Deprecated

• Map, List, and Set in org.opensearch.common.collect (#6609)

Fixed

• Added depth check in doc parser for deep nested document (#5199)
• Added equals/hashcode for named DocValueFormat.DateTime inner class (#6357)
• Fixed bug for searchable snapshot to take 'base_path' of blob into account (#6558)
• Fix fuzziness validation (#5805)
• Avoid negative memory result in IndicesQueryCache stats calculation (#6917)
• Fix GetSnapshots to not return non-existent snapshots with ignore_unavailable=true (#6839)
• Fix GlobalAggregation with profile option enabled returns incorrect result (#7114)