module Elasticsearch::API::Actions
Public Instance Methods
Source
# File lib/elasticsearch/api/actions/bulk.rb, line 135 def bulk(arguments = {}) request_opts = { endpoint: arguments[:endpoint] || 'bulk' } defined_params = [:index].each_with_object({}) do |variable, set_variables| set_variables[variable] = arguments[variable] if arguments.key?(variable) end request_opts[:defined_params] = defined_params unless defined_params.empty? raise ArgumentError, "Required argument 'body' missing" unless arguments[:body] arguments = arguments.clone headers = arguments.delete(:headers) || {} body = arguments.delete(:body) _index = arguments.delete(:index) method = Elasticsearch::API::HTTP_POST path = if _index "#{Utils.listify(_index)}/_bulk" else '_bulk' end params = Utils.process_params(arguments) payload = if body.is_a? Array Elasticsearch::API::Utils.bulkify(body) else body end headers.merge!({ 'Content-Type' => 'application/vnd.elasticsearch+x-ndjson; compatible-with=9' }) Elasticsearch::API::Response.new( perform_request(method, path, params, payload, headers, request_opts) ) end
Bulk index or delete documents. Perform multiple index
, create
, delete
, and update
actions in a single request. This reduces overhead and can greatly increase indexing speed. If the Elasticsearch
security features are enabled, you must have the following index privileges for the target data stream, index, or index alias:
-
To use the
create
action, you must have thecreate_doc
,create
,index
, orwrite
index privilege. Data streams support only thecreate
action. -
To use the
index
action, you must have thecreate
,index
, orwrite
index privilege. -
To use the
delete
action, you must have thedelete
orwrite
index privilege. -
To use the
update
action, you must have theindex
orwrite
index privilege. -
To automatically create a data stream or index with a bulk
API
request, you must have theauto_configure
,create_index
, ormanage
index privilege. -
To make the result of a bulk operation visible to search using the
refresh
parameter, you must have themaintenance
ormanage
index privilege.
Automatic data stream creation requires a matching index template with data stream enabled. The actions are specified in the request body using a newline delimited JSON (NDJSON) structure: + action_and_meta_datan optional_sourcen action_and_meta_datan optional_sourcen .… action_and_meta_datan optional_sourcen + The index
and create
actions expect a source on the next line and have the same semantics as the op_type
parameter in the standard index API
. A create
action fails if a document with the same ID already exists in the target An index
action adds or replaces a document as necessary. NOTE: Data streams support only the create
action. To update or delete a document in a data stream, you must target the backing index containing the document. An update
action expects that the partial doc, upsert, and script and its options are specified on the next line. A delete
action does not expect a source on the next line and has the same semantics as the standard delete API
. NOTE: The final line of data must end with a newline character (\n
). Each newline character may be preceded by a carriage return (\r
). When sending NDJSON data to the _bulk
endpoint, use a Content-Type
header of application/json
or application/x-ndjson
. Because this format uses literal newline characters (\n
) as delimiters, make sure that the JSON actions and sources are not pretty printed. If you provide a target in the request path, it is used for any actions that don’t explicitly specify an _index
argument. A note on the format: the idea here is to make processing as fast as possible. As some of the actions are redirected to other shards on other nodes, only action_meta_data
is parsed on the receiving node side. Client libraries using this protocol should try and strive to do something similar on the client side, and reduce buffering as much as possible. There is no “correct” number of actions to perform in a single bulk request. Experiment with different settings to find the optimal size for your particular workload. Note that Elasticsearch
limits the maximum size of a HTTP request to 100mb by default so clients must ensure that no request exceeds this size. It is not possible to index a single document that exceeds the size limit, so you must pre-process any such documents into smaller pieces before sending them to Elasticsearch
. For instance, split documents into pages or chapters before indexing them, or store raw binary data in a system outside Elasticsearch
and replace the raw data with a link to the external system in the documents that you send to Elasticsearch
. **Client suppport for bulk requests** Some of the officially supported clients provide helpers to assist with bulk requests and reindexing:
-
Go: Check out
esutil.BulkIndexer
-
Perl: Check out
Search::Elasticsearch::Client::5_0::Bulk
andSearch::Elasticsearch::Client::5_0::Scroll
-
Python: Check out
elasticsearch.helpers.*
-
JavaScript: Check out
client.helpers.*
-
.NET: Check out
BulkAllObservable
-
PHP: Check out bulk indexing.
**Submitting bulk requests with cURL** If you’re providing text file input to curl
, you must use the --data-binary
flag instead of plain -d
. The latter doesn’t preserve newlines. For example: + $ cat requests { “index” : { “_index” : “test”, “_id” : “1” } } { “field1” : “value1” } $ curl -s -H “Content-Type: application/x-ndjson” -XPOST localhost:9200/_bulk –data-binary “@requests”; echo {“took”:7, “errors”: false, “items”:} + **Optimistic concurrency control** Each index
and delete
action within a bulk API
call may include the if_seq_no
and if_primary_term
parameters in their respective action and meta data lines. The if_seq_no
and if_primary_term
parameters control how operations are run, based on the last modification to existing documents. See Optimistic concurrency control for more details. Versioning Each bulk item can include the version value using the version
field. It automatically follows the behavior of the index or delete operation based on the _version
mapping. It also support the version_type
. Routing Each bulk item can include the routing value using the routing
field. It automatically follows the behavior of the index or delete operation based on the _routing
mapping. NOTE: Data streams do not support custom routing unless they were created with the allow_custom_routing
setting enabled in the template. **Wait for active shards** When making bulk calls, you can set the wait_for_active_shards
parameter to require a minimum number of shard copies to be active before starting to process the bulk request. Refresh Control when the changes made by this request are visible to search. NOTE: Only the shards that receive the bulk request will be affected by refresh. Imagine a +_bulk?refresh=wait_for+ request with three documents in it that happen to be routed to different shards in an index with five shards. The request will only wait for those three shards to refresh. The other two shards that make up the index do not participate in the _bulk
request at all.
@option arguments [String] :index The name of the data stream, index, or index alias to perform bulk actions on. @option arguments [Boolean] :include_source_on_error True or false if to include the document source in the error message in case of parsing errors. Server default: true. @option arguments [Boolean] :list_executed_pipelines If true
, the response will include the ingest pipelines that were run for each index or create. @option arguments [String] :pipeline The pipeline identifier to use to preprocess incoming documents.
If the index has a default ingest pipeline specified, setting the value to +_none+ turns off the default ingest pipeline for this request. If a final pipeline is configured, it will always run regardless of the value of this parameter.
@option arguments [String] :refresh If true
, Elasticsearch
refreshes the affected shards to make this operation visible to search.
If +wait_for+, wait for a refresh to make this operation visible to search. If +false+, do nothing with refreshes. Valid values: +true+, +false+, +wait_for+. Server default: false.
@option arguments [String] :routing A custom value that is used to route operations to a specific shard. @option arguments [Boolean, String, Array<String>] :_source Indicates whether to return the _source
field (true
or false
) or contains a list of fields to return. @option arguments [String, Array<String>] :_source_excludes A comma-separated list of source fields to exclude from the response.
You can also use this parameter to exclude fields from the subset specified in +_source_includes+ query parameter. If the +_source+ parameter is +false+, this parameter is ignored.
@option arguments [String, Array<String>] :_source_includes A comma-separated list of source fields to include in the response.
If this parameter is specified, only these source fields are returned. You can exclude fields from this subset using the +_source_excludes+ query parameter. If the +_source+ parameter is +false+, this parameter is ignored.
@option arguments [Time] :timeout The period each action waits for the following operations: automatic index creation, dynamic mapping updates, and waiting for active shards.
The default is +1m+ (one minute), which guarantees Elasticsearch waits for at least the timeout before failing. The actual wait time could be longer, particularly when multiple waits occur. Server default: 1m.
@option arguments [Integer, String] :wait_for_active_shards The number of shard copies that must be active before proceeding with the operation.
Set to +all+ or any positive integer up to the total number of shards in the index (+number_of_replicas+1+). The default is +1+, which waits for each primary shard to be active. Server default: 1.
@option arguments [Boolean] :require_alias If true
, the request’s actions must target an index alias. @option arguments [Boolean] :require_data_stream If true
, the request’s actions must target a data stream (existing or to be created). @option arguments [Hash] :headers Custom HTTP headers @option arguments [String|Array] :body operations. Array of Strings, Header/Data pairs, or the conveniency “combined” format can be passed, refer to Elasticsearch::API::Utils.bulkify
documentation.
@see www.elastic.co/docs/api/doc/elasticsearch/v9/operation/operation-bulk
Source
# File lib/elasticsearch/api/actions/clear_scroll.rb, line 40 def clear_scroll(arguments = {}) request_opts = { endpoint: arguments[:endpoint] || 'clear_scroll' } arguments = arguments.clone headers = arguments.delete(:headers) || {} body = arguments.delete(:body) _scroll_id = arguments.delete(:scroll_id) method = Elasticsearch::API::HTTP_DELETE path = '_search/scroll' params = Utils.process_params(arguments) if Array(arguments[:ignore]).include?(404) Utils.rescue_from_not_found do Elasticsearch::API::Response.new( perform_request(method, path, params, body, headers, request_opts) ) end else Elasticsearch::API::Response.new( perform_request(method, path, params, body, headers, request_opts) ) end end
Clear a scrolling search. Clear the search context and results for a scrolling search.
@option arguments [String, Array] :scroll_id A comma-separated list of scroll IDs to clear.
To clear all scroll IDs, use +_all+. IMPORTANT: Scroll IDs can be long. It is recommended to specify scroll IDs in the request body parameter.
@option arguments [Hash] :headers Custom HTTP headers @option arguments [Hash] :body request body
*Deprecation notice*: A scroll id can be quite large and should be specified as part of the body Deprecated since version 7.0.0
@see www.elastic.co/docs/api/doc/elasticsearch/v9/operation/operation-clear-scroll
Source
# File lib/elasticsearch/api/actions/close_point_in_time.rb, line 35 def close_point_in_time(arguments = {}) request_opts = { endpoint: arguments[:endpoint] || 'close_point_in_time' } arguments = arguments.clone headers = arguments.delete(:headers) || {} body = arguments.delete(:body) method = Elasticsearch::API::HTTP_DELETE path = '_pit' params = {} Elasticsearch::API::Response.new( perform_request(method, path, params, body, headers, request_opts) ) end
Close a point in time. A point in time must be opened explicitly before being used in search requests. The keep_alive
parameter tells Elasticsearch
how long it should persist. A point in time is automatically closed when the keep_alive
period has elapsed. However, keeping points in time has a cost; close them as soon as they are no longer required for search requests.
@option arguments [Hash] :headers Custom HTTP headers @option arguments [Hash] :body request body
@see www.elastic.co/docs/api/doc/elasticsearch/v9/operation/operation-open-point-in-time
Source
# File lib/elasticsearch/api/actions/count.rb, line 70 def count(arguments = {}) request_opts = { endpoint: arguments[:endpoint] || 'count' } defined_params = [:index].each_with_object({}) do |variable, set_variables| set_variables[variable] = arguments[variable] if arguments.key?(variable) end request_opts[:defined_params] = defined_params unless defined_params.empty? arguments = arguments.clone headers = arguments.delete(:headers) || {} body = arguments.delete(:body) _index = arguments.delete(:index) method = if body Elasticsearch::API::HTTP_POST else Elasticsearch::API::HTTP_GET end path = if _index "#{Utils.listify(_index)}/_count" else '_count' end params = Utils.process_params(arguments) Elasticsearch::API::Response.new( perform_request(method, path, params, body, headers, request_opts) ) end
Count search results. Get the number of documents matching a query. The query can be provided either by using a simple query string as a parameter, or by defining Query DSL within the request body. The query is optional. When no query is provided, the API
uses match_all
to count all the documents. The count API
supports multi-target syntax. You can run a single count API
search across multiple data streams and indices. The operation is broadcast across all shards. For each shard ID group, a replica is chosen and the search is run against it. This means that replicas increase the scalability of the count.
@option arguments [String, Array] :index A comma-separated list of data streams, indices, and aliases to search.
It supports wildcards (+*+). To search all data streams and indices, omit this parameter or use +*+ or +_all+.
@option arguments [Boolean] :allow_no_indices If false
, the request returns an error if any wildcard expression, index alias, or _all
value targets only missing or closed indices.
This behavior applies even if the request targets other open indices. For example, a request targeting +foo*,bar*+ returns an error if an index starts with +foo+ but no index starts with +bar+. Server default: true.
@option arguments [String] :analyzer The analyzer to use for the query string.
This parameter can be used only when the +q+ query string parameter is specified.
@option arguments [Boolean] :analyze_wildcard If true
, wildcard and prefix queries are analyzed.
This parameter can be used only when the +q+ query string parameter is specified.
@option arguments [String] :default_operator The default operator for query string query: AND
or OR
.
This parameter can be used only when the +q+ query string parameter is specified. Server default: OR.
@option arguments [String] :df The field to use as a default when no field prefix is given in the query string.
This parameter can be used only when the +q+ query string parameter is specified.
@option arguments [String, Array<String>] :expand_wildcards The type of index that wildcard patterns can match.
If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams. It supports comma-separated values, such as +open,hidden+. Server default: open.
@option arguments [Boolean] :ignore_throttled If true
, concrete, expanded, or aliased indices are ignored when frozen. Server default: true. @option arguments [Boolean] :ignore_unavailable If false
, the request returns an error if it targets a missing or closed index. @option arguments [Boolean] :lenient If true
, format-based query failures (such as providing text to a numeric field) in the query string will be ignored.
This parameter can be used only when the +q+ query string parameter is specified.
@option arguments [Float] :min_score The minimum _score
value that documents must have to be included in the result. @option arguments [String] :preference The node or shard the operation should be performed on.
By default, it is random.
@option arguments [String] :routing A custom value used to route operations to a specific shard. @option arguments [Integer] :terminate_after The maximum number of documents to collect for each shard.
If a query reaches this limit, Elasticsearch terminates the query early. Elasticsearch collects documents before sorting.IMPORTANT: Use with caution. Elasticsearch applies this parameter to each shard handling the request. When possible, let Elasticsearch perform early termination automatically. Avoid specifying this parameter for requests that target data streams with backing indices across multiple data tiers.
@option arguments [String] :q The query in Lucene query string syntax. This parameter cannot be used with a request body. @option arguments [Hash] :headers Custom HTTP headers @option arguments [Hash] :body request body
@see www.elastic.co/docs/api/doc/elasticsearch/v9/operation/operation-count
Source
# File lib/elasticsearch/api/actions/create.rb, line 115 def create(arguments = {}) request_opts = { endpoint: arguments[:endpoint] || 'create' } defined_params = [:index, :id].each_with_object({}) do |variable, set_variables| set_variables[variable] = arguments[variable] if arguments.key?(variable) end request_opts[:defined_params] = defined_params unless defined_params.empty? raise ArgumentError, "Required argument 'body' missing" unless arguments[:body] raise ArgumentError, "Required argument 'index' missing" unless arguments[:index] raise ArgumentError, "Required argument 'id' missing" unless arguments[:id] arguments = arguments.clone headers = arguments.delete(:headers) || {} body = arguments.delete(:body) _id = arguments.delete(:id) _index = arguments.delete(:index) method = Elasticsearch::API::HTTP_PUT path = "#{Utils.listify(_index)}/_create/#{Utils.listify(_id)}" params = Utils.process_params(arguments) Elasticsearch::API::Response.new( perform_request(method, path, params, body, headers, request_opts) ) end
Create a new document in the index. You can index a new JSON document with the +/<target>/_doc/+ or +/<target>/_create/<_id>+ APIs Using _create
guarantees that the document is indexed only if it does not already exist. It returns a 409 response when a document with a same ID already exists in the index. To update an existing document, you must use the +/<target>/_doc/+ API
. If the Elasticsearch
security features are enabled, you must have the following index privileges for the target data stream, index, or index alias:
-
To add a document using the +PUT /<target>/_create/<_id>+ or +POST /<target>/_create/<_id>+ request formats, you must have the
create_doc
,create
,index
, orwrite
index privilege. -
To automatically create a data stream or index with this
API
request, you must have theauto_configure
,create_index
, ormanage
index privilege.
Automatic data stream creation requires a matching index template with data stream enabled. **Automatically create data streams and indices** If the request’s target doesn’t exist and matches an index template with a data_stream
definition, the index operation automatically creates the data stream. If the target doesn’t exist and doesn’t match a data stream template, the operation automatically creates the index and applies any matching index templates. NOTE: Elasticsearch
includes several built-in index templates. To avoid naming collisions with these templates, refer to index pattern documentation. If no mapping exists, the index operation creates a dynamic mapping. By default, new fields and objects are automatically added to the mapping if needed. Automatic index creation is controlled by the action.auto_create_index
setting. If it is true
, any index can be created automatically. You can modify this setting to explicitly allow or block automatic creation of indices that match specified patterns or set it to false
to turn off automatic index creation entirely. Specify a comma-separated list of patterns you want to allow or prefix each pattern with +++ or -
to indicate whether it should be allowed or blocked. When a list is specified, the default behaviour is to disallow. NOTE: The action.auto_create_index
setting affects the automatic creation of indices only. It does not affect the creation of data streams. Routing By default, shard placement — or routing — is controlled by using a hash of the document’s ID value. For more explicit control, the value fed into the hash function used by the router can be directly specified on a per-operation basis using the routing
parameter. When setting up explicit mapping, you can also use the _routing
field to direct the index operation to extract the routing value from the document itself. This does come at the (very minimal) cost of an additional document parsing pass. If the _routing
mapping is defined and set to be required, the index operation will fail if no routing value is provided or extracted. NOTE: Data streams do not support custom routing unless they were created with the allow_custom_routing
setting enabled in the template. Distributed The index operation is directed to the primary shard based on its route and performed on the actual node containing this shard. After the primary shard completes the operation, if needed, the update is distributed to applicable replicas. **Active shards** To improve the resiliency of writes to the system, indexing operations can be configured to wait for a certain number of active shard copies before proceeding with the operation. If the requisite number of active shard copies are not available, then the write operation must wait and retry, until either the requisite shard copies have started or a timeout occurs. By default, write operations only wait for the primary shards to be active before proceeding (that is to say wait_for_active_shards
is 1
). This default can be overridden in the index settings dynamically by setting index.write.wait_for_active_shards
. To alter this behavior per operation, use the +wait_for_active_shards request+ parameter. Valid values are all or any positive integer up to the total number of configured copies per shard in the index (which is +number_of_replicas++1). Specifying a negative value or a number greater than the number of shard copies will throw an error. For example, suppose you have a cluster of three nodes, A, B, and C and you create an index index with the number of replicas set to 3 (resulting in 4 shard copies, one more copy than there are nodes). If you attempt an indexing operation, by default the operation will only ensure the primary copy of each shard is available before proceeding. This means that even if B and C went down and A hosted the primary shard copies, the indexing operation would still proceed with only one copy of the data. If wait_for_active_shards
is set on the request to 3
(and all three nodes are up), the indexing operation will require 3 active shard copies before proceeding. This requirement should be met because there are 3 active nodes in the cluster, each one holding a copy of the shard. However, if you set wait_for_active_shards
to all
(or to 4
, which is the same in this situation), the indexing operation will not proceed as you do not have all 4 copies of each shard active in the index. The operation will timeout unless a new node is brought up in the cluster to host the fourth copy of the shard. It is important to note that this setting greatly reduces the chances of the write operation not writing to the requisite number of shard copies, but it does not completely eliminate the possibility, because this check occurs before the write operation starts. After the write operation is underway, it is still possible for replication to fail on any number of shard copies but still succeed on the primary. The _shards
section of the API
response reveals the number of shard copies on which replication succeeded and failed.
@option arguments [String] :id A unique identifier for the document.
To automatically generate a document ID, use the +POST /<target>/_doc/+ request format. (*Required*)
@option arguments [String] :index The name of the data stream or index to target.
If the target doesn't exist and matches the name or wildcard (+*+) pattern of an index template with a +data_stream+ definition, this request creates the data stream. If the target doesn't exist and doesn’t match a data stream template, this request creates the index. (*Required*)
@option arguments [Integer] :if_primary_term Only perform the operation if the document has this primary term. @option arguments [Integer] :if_seq_no Only perform the operation if the document has this sequence number. @option arguments [Boolean] :include_source_on_error True or false if to include the document source in the error message in case of parsing errors. Server default: true. @option arguments [String] :op_type Set to create
to only index the document if it does not already exist (put if absent).
If a document with the specified +_id+ already exists, the indexing operation will fail. The behavior is the same as using the +<index>/_create+ endpoint. If a document ID is specified, this paramater defaults to +index+. Otherwise, it defaults to +create+. If the request targets a data stream, an +op_type+ of +create+ is required.
@option arguments [String] :pipeline The ID of the pipeline to use to preprocess incoming documents.
If the index has a default ingest pipeline specified, setting the value to +_none+ turns off the default ingest pipeline for this request. If a final pipeline is configured, it will always run regardless of the value of this parameter.
@option arguments [String] :refresh If true
, Elasticsearch
refreshes the affected shards to make this operation visible to search.
If +wait_for+, it waits for a refresh to make this operation visible to search. If +false+, it does nothing with refreshes. Server default: false.
@option arguments [Boolean] :require_alias If true
, the destination must be an index alias. @option arguments [Boolean] :require_data_stream If true
, the request’s actions must target a data stream (existing or to be created). @option arguments [String] :routing A custom value that is used to route operations to a specific shard. @option arguments [Time] :timeout The period the request waits for the following operations: automatic index creation, dynamic mapping updates, waiting for active shards.
Elasticsearch waits for at least the specified timeout period before failing. The actual wait time could be longer, particularly when multiple waits occur.This parameter is useful for situations where the primary shard assigned to perform the operation might not be available when the operation runs. Some reasons for this might be that the primary shard is currently recovering from a gateway or undergoing relocation. By default, the operation will wait on the primary shard to become available for at least 1 minute before failing and responding with an error. The actual wait time could be longer, particularly when multiple waits occur. Server default: 1m.
@option arguments [Integer] :version The explicit version number for concurrency control.
It must be a non-negative long number.
@option arguments [String] :version_type The version type. @option arguments [Integer, String] :wait_for_active_shards The number of shard copies that must be active before proceeding with the operation.
You can set it to +all+ or any positive integer up to the total number of shards in the index (+number_of_replicas+1+). The default value of +1+ means it waits for each primary shard to be active. Server default: 1.
@option arguments [Hash] :headers Custom HTTP headers @option arguments [Hash] :body document
@see www.elastic.co/docs/api/doc/elasticsearch/v9/operation/operation-create
Source
# File lib/elasticsearch/api/actions/delete.rb, line 71 def delete(arguments = {}) request_opts = { endpoint: arguments[:endpoint] || 'delete' } defined_params = [:index, :id].each_with_object({}) do |variable, set_variables| set_variables[variable] = arguments[variable] if arguments.key?(variable) end request_opts[:defined_params] = defined_params unless defined_params.empty? raise ArgumentError, "Required argument 'index' missing" unless arguments[:index] raise ArgumentError, "Required argument 'id' missing" unless arguments[:id] arguments = arguments.clone headers = arguments.delete(:headers) || {} body = nil _id = arguments.delete(:id) _index = arguments.delete(:index) method = Elasticsearch::API::HTTP_DELETE path = "#{Utils.listify(_index)}/_doc/#{Utils.listify(_id)}" params = Utils.process_params(arguments) if Array(arguments[:ignore]).include?(404) Utils.rescue_from_not_found do Elasticsearch::API::Response.new( perform_request(method, path, params, body, headers, request_opts) ) end else Elasticsearch::API::Response.new( perform_request(method, path, params, body, headers, request_opts) ) end end
Delete a document. Remove a JSON document from the specified index. NOTE: You cannot send deletion requests directly to a data stream. To delete a document in a data stream, you must target the backing index containing the document. **Optimistic concurrency control** Delete operations can be made conditional and only be performed if the last modification to the document was assigned the sequence number and primary term specified by the if_seq_no
and if_primary_term
parameters. If a mismatch is detected, the operation will result in a VersionConflictException
and a status code of 409
. Versioning Each document indexed is versioned. When deleting a document, the version can be specified to make sure the relevant document you are trying to delete is actually being deleted and it has not changed in the meantime. Every write operation run on a document, deletes included, causes its version to be incremented. The version number of a deleted document remains available for a short time after deletion to allow for control of concurrent operations. The length of time for which a deleted document’s version remains available is determined by the index.gc_deletes
index setting. Routing If routing is used during indexing, the routing value also needs to be specified to delete a document. If the _routing
mapping is set to required
and no routing value is specified, the delete API
throws a RoutingMissingException
and rejects the request. For example: + DELETE /my-index-000001/_doc/1?routing=shard-1 + This request deletes the document with ID 1, but it is routed based on the user. The document is not deleted if the correct routing is not specified. Distributed The delete operation gets hashed into a specific shard ID. It then gets redirected into the primary shard within that ID group and replicated (if needed) to shard replicas within that ID group.
@option arguments [String] :id A unique identifier for the document. (Required) @option arguments [String] :index The name of the target index. (Required) @option arguments [Integer] :if_primary_term Only perform the operation if the document has this primary term. @option arguments [Integer] :if_seq_no Only perform the operation if the document has this sequence number. @option arguments [String] :refresh If true
, Elasticsearch
refreshes the affected shards to make this operation visible to search.
If +wait_for+, it waits for a refresh to make this operation visible to search. If +false+, it does nothing with refreshes. Server default: false.
@option arguments [String] :routing A custom value used to route operations to a specific shard. @option arguments [Time] :timeout The period to wait for active shards.This parameter is useful for situations where the primary shard assigned to perform the delete operation might not be available when the delete operation runs.
Some reasons for this might be that the primary shard is currently recovering from a store or undergoing relocation. By default, the delete operation will wait on the primary shard to become available for up to 1 minute before failing and responding with an error. Server default: 1m.
@option arguments [Integer] :version An explicit version number for concurrency control.
It must match the current version of the document for the request to succeed.
@option arguments [String] :version_type The version type. @option arguments [Integer, String] :wait_for_active_shards The minimum number of shard copies that must be active before proceeding with the operation.
You can set it to +all+ or any positive integer up to the total number of shards in the index (+number_of_replicas+1+). The default value of +1+ means it waits for each primary shard to be active. Server default: 1.
@option arguments [Hash] :headers Custom HTTP headers
@see www.elastic.co/docs/api/doc/elasticsearch/v9/operation/operation-delete
Source
# File lib/elasticsearch/api/actions/delete_by_query.rb, line 143 def delete_by_query(arguments = {}) request_opts = { endpoint: arguments[:endpoint] || 'delete_by_query' } defined_params = [:index].each_with_object({}) do |variable, set_variables| set_variables[variable] = arguments[variable] if arguments.key?(variable) end request_opts[:defined_params] = defined_params unless defined_params.empty? raise ArgumentError, "Required argument 'body' missing" unless arguments[:body] raise ArgumentError, "Required argument 'index' missing" unless arguments[:index] arguments = arguments.clone headers = arguments.delete(:headers) || {} body = arguments.delete(:body) _index = arguments.delete(:index) method = Elasticsearch::API::HTTP_POST path = "#{Utils.listify(_index)}/_delete_by_query" params = Utils.process_params(arguments) Elasticsearch::API::Response.new( perform_request(method, path, params, body, headers, request_opts) ) end
Delete documents. Deletes documents that match the specified query. If the Elasticsearch
security features are enabled, you must have the following index privileges for the target data stream, index, or alias:
-
read
-
delete
orwrite
You can specify the query criteria in the request URI or the request body using the same syntax as the search API
. When you submit a delete by query request, Elasticsearch
gets a snapshot of the data stream or index when it begins processing the request and deletes matching documents using internal versioning. If a document changes between the time that the snapshot is taken and the delete operation is processed, it results in a version conflict and the delete operation fails. NOTE: Documents with a version equal to 0 cannot be deleted using delete by query because internal versioning does not support 0 as a valid version number. While processing a delete by query request, Elasticsearch
performs multiple search requests sequentially to find all of the matching documents to delete. A bulk delete request is performed for each batch of matching documents. If a search or bulk request is rejected, the requests are retried up to 10 times, with exponential back off. If the maximum retry limit is reached, processing halts and all failed requests are returned in the response. Any delete requests that completed successfully still stick, they are not rolled back. You can opt to count version conflicts instead of halting and returning by setting conflicts
to proceed
. Note that if you opt to count version conflicts the operation could attempt to delete more documents from the source than max_docs
until it has successfully deleted +max_docs documents+, or it has gone through every document in the source query. **Throttling delete requests** To control the rate at which delete by query issues batches of delete operations, you can set requests_per_second
to any positive decimal number. This pads each batch with a wait time to throttle the rate. Set requests_per_second
to -1
to disable throttling. Throttling uses a wait time between batches so that the internal scroll requests can be given a timeout that takes the request padding into account. The padding time is the difference between the batch size divided by the requests_per_second
and the time spent writing. By default the batch size is 1000
, so if requests_per_second
is set to 500
: + target_time = 1000 / 500 per second = 2 seconds wait_time = target_time - write_time = 2 seconds - .5 seconds = 1.5 seconds + Since the batch is issued as a single _bulk
request, large batch sizes cause Elasticsearch
to create many requests and wait before starting the next set. This is “bursty” instead of “smooth”. Slicing Delete by query supports sliced scroll to parallelize the delete process. This can improve efficiency and provide a convenient way to break the request down into smaller parts. Setting slices
to auto
lets Elasticsearch
choose the number of slices to use. This setting will use one slice per shard, up to a certain limit. If there are multiple source data streams or indices, it will choose the number of slices based on the index or backing index with the smallest number of shards. Adding slices to the delete by query operation creates sub-requests which means it has some quirks:
-
You can see these requests in the tasks APIs. These sub-requests are “child” tasks of the task for the request with slices.
-
Fetching the status of the task for the request with slices only contains the status of completed slices.
-
These sub-requests are individually addressable for things like cancellation and rethrottling.
-
Rethrottling the request with
slices
will rethrottle the unfinished sub-request proportionally. -
Canceling the request with
slices
will cancel each sub-request. -
Due to the nature of
slices
each sub-request won’t get a perfectly even portion of the documents. All documents will be addressed, but some slices may be larger than others. Expect larger slices to have a more even distribution. -
Parameters like
requests_per_second
andmax_docs
on a request withslices
are distributed proportionally to each sub-request. Combine that with the earlier point about distribution being uneven and you should conclude that usingmax_docs
withslices
might not result in exactlymax_docs
documents being deleted. -
Each sub-request gets a slightly different snapshot of the source data stream or index though these are all taken at approximately the same time.
If you’re slicing manually or otherwise tuning automatic slicing, keep in mind that:
-
Query performance is most efficient when the number of slices is equal to the number of shards in the index or backing index. If that number is large (for example, 500), choose a lower number as too many
slices
hurts performance. Settingslices
higher than the number of shards generally does not improve efficiency and adds overhead. -
Delete performance scales linearly across available resources with the number of slices.
Whether query or delete performance dominates the runtime depends on the documents being reindexed and cluster resources. **Cancel a delete by query operation** Any delete by query can be canceled using the task cancel API
. For example: + POST _tasks/r1A2WoRbTwKZ516z6NEs5A:36619/_cancel + The task ID can be found by using the get tasks API
. Cancellation should happen quickly but might take a few seconds. The get task status API
will continue to list the delete by query task until this task checks that it has been cancelled and terminates itself.
@option arguments [String, Array] :index A comma-separated list of data streams, indices, and aliases to search.
It supports wildcards (+*+). To search all data streams or indices, omit this parameter or use +*+ or +_all+. (*Required*)
@option arguments [Boolean] :allow_no_indices If false
, the request returns an error if any wildcard expression, index alias, or _all
value targets only missing or closed indices.
This behavior applies even if the request targets other open indices. For example, a request targeting +foo*,bar*+ returns an error if an index starts with +foo+ but no index starts with +bar+. Server default: true.
@option arguments [String] :analyzer Analyzer to use for the query string.
This parameter can be used only when the +q+ query string parameter is specified.
@option arguments [Boolean] :analyze_wildcard If true
, wildcard and prefix queries are analyzed.
This parameter can be used only when the +q+ query string parameter is specified.
@option arguments [String] :conflicts What to do if delete by query hits version conflicts: abort
or proceed
. Server default: abort. @option arguments [String] :default_operator The default operator for query string query: AND
or OR
.
This parameter can be used only when the +q+ query string parameter is specified. Server default: OR.
@option arguments [String] :df The field to use as default where no field prefix is given in the query string.
This parameter can be used only when the +q+ query string parameter is specified.
@option arguments [String, Array<String>] :expand_wildcards The type of index that wildcard patterns can match.
If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams. It supports comma-separated values, such as +open,hidden+. Server default: open.
@option arguments [Integer] :from Skips the specified number of documents. Server default: 0. @option arguments [Boolean] :ignore_unavailable If false
, the request returns an error if it targets a missing or closed index. @option arguments [Boolean] :lenient If true
, format-based query failures (such as providing text to a numeric field) in the query string will be ignored.
This parameter can be used only when the +q+ query string parameter is specified.
@option arguments [Integer] :max_docs The maximum number of documents to process.
Defaults to all documents. When set to a value less then or equal to +scroll_size+, a scroll will not be used to retrieve the results for the operation.
@option arguments [String] :preference The node or shard the operation should be performed on.
It is random by default.
@option arguments [Boolean] :refresh If true
, Elasticsearch
refreshes all shards involved in the delete by query after the request completes.
This is different than the delete API's +refresh+ parameter, which causes just the shard that received the delete request to be refreshed. Unlike the delete API, it does not support +wait_for+.
@option arguments [Boolean] :request_cache If true
, the request cache is used for this request.
Defaults to the index-level setting.
@option arguments [Float] :requests_per_second The throttle for this request in sub-requests per second. Server default: -1. @option arguments [String] :routing A custom value used to route operations to a specific shard. @option arguments [String] :q A query in the Lucene query string syntax. @option arguments [Time] :scroll The period to retain the search context for scrolling. @option arguments [Integer] :scroll_size The size of the scroll request that powers the operation. Server default: 1000. @option arguments [Time] :search_timeout The explicit timeout for each search request.
It defaults to no timeout.
@option arguments [String] :search_type The type of the search operation.
Available options include +query_then_fetch+ and +dfs_query_then_fetch+.
@option arguments [Integer, String] :slices The number of slices this task should be divided into. Server default: 1. @option arguments [Array<String>] :sort A comma-separated list of +<field>:<direction>+ pairs. @option arguments [Array<String>] :stats The specific tag
of the request for logging and statistical purposes. @option arguments [Integer] :terminate_after The maximum number of documents to collect for each shard.
If a query reaches this limit, Elasticsearch terminates the query early. Elasticsearch collects documents before sorting.Use with caution. Elasticsearch applies this parameter to each shard handling the request. When possible, let Elasticsearch perform early termination automatically. Avoid specifying this parameter for requests that target data streams with backing indices across multiple data tiers.
@option arguments [Time] :timeout The period each deletion request waits for active shards. Server default: 1m. @option arguments [Boolean] :version If true
, returns the document version as part of a hit. @option arguments [Integer, String] :wait_for_active_shards The number of shard copies that must be active before proceeding with the operation.
Set to +all+ or any positive integer up to the total number of shards in the index (+number_of_replicas+1+). The +timeout+ value controls how long each write request waits for unavailable shards to become available. Server default: 1.
@option arguments [Boolean] :wait_for_completion If true
, the request blocks until the operation is complete.
If +false+, Elasticsearch performs some preflight checks, launches the request, and returns a task you can use to cancel or get the status of the task. Elasticsearch creates a record of this task as a document at +.tasks/task/${taskId}+. When you are done with a task, you should delete the task document so Elasticsearch can reclaim the space. Server default: true.
@option arguments [Hash] :headers Custom HTTP headers @option arguments [Hash] :body request body
@see www.elastic.co/docs/api/doc/elasticsearch/v9/operation/operation-delete-by-query
Source
# File lib/elasticsearch/api/actions/delete_by_query_rethrottle.rb, line 35 def delete_by_query_rethrottle(arguments = {}) request_opts = { endpoint: arguments[:endpoint] || 'delete_by_query_rethrottle' } defined_params = [:task_id].each_with_object({}) do |variable, set_variables| set_variables[variable] = arguments[variable] if arguments.key?(variable) end request_opts[:defined_params] = defined_params unless defined_params.empty? raise ArgumentError, "Required argument 'task_id' missing" unless arguments[:task_id] arguments = arguments.clone headers = arguments.delete(:headers) || {} body = nil _task_id = arguments.delete(:task_id) method = Elasticsearch::API::HTTP_POST path = "_delete_by_query/#{Utils.listify(_task_id)}/_rethrottle" params = Utils.process_params(arguments) Elasticsearch::API::Response.new( perform_request(method, path, params, body, headers, request_opts) ) end
Throttle a delete by query operation. Change the number of requests per second for a particular delete by query operation. Rethrottling that speeds up the query takes effect immediately but rethrotting that slows down the query takes effect after completing the current batch to prevent scroll timeouts.
@option arguments [String, Integer] :task_id The ID for the task. (Required) @option arguments [Float] :requests_per_second The throttle for this request in sub-requests per second.
To disable throttling, set it to +-1+.
@option arguments [Hash] :headers Custom HTTP headers
@see www.elastic.co/docs/api/doc/elasticsearch/v9/operation/operation-delete-by-query-rethrottle
Source
# File lib/elasticsearch/api/actions/delete_script.rb, line 38 def delete_script(arguments = {}) request_opts = { endpoint: arguments[:endpoint] || 'delete_script' } defined_params = [:id].each_with_object({}) do |variable, set_variables| set_variables[variable] = arguments[variable] if arguments.key?(variable) end request_opts[:defined_params] = defined_params unless defined_params.empty? raise ArgumentError, "Required argument 'id' missing" unless arguments[:id] arguments = arguments.clone headers = arguments.delete(:headers) || {} body = nil _id = arguments.delete(:id) method = Elasticsearch::API::HTTP_DELETE path = "_scripts/#{Utils.listify(_id)}" params = Utils.process_params(arguments) Elasticsearch::API::Response.new( perform_request(method, path, params, body, headers, request_opts) ) end
Delete a script or search template. Deletes a stored script or search template.
@option arguments [String] :id The identifier for the stored script or search template. (Required) @option arguments [Time] :master_timeout The period to wait for a connection to the master node.
If no response is received before the timeout expires, the request fails and returns an error. It can also be set to +-1+ to indicate that the request should never timeout. Server default: 30s.
@option arguments [Time] :timeout The period to wait for a response.
If no response is received before the timeout expires, the request fails and returns an error. It can also be set to +-1+ to indicate that the request should never timeout. Server default: 30s.
@option arguments [Hash] :headers Custom HTTP headers
@see www.elastic.co/docs/api/doc/elasticsearch/v9/operation/operation-delete-script
Source
# File lib/elasticsearch/api/actions/exists.rb, line 68 def exists(arguments = {}) request_opts = { endpoint: arguments[:endpoint] || 'exists' } defined_params = [:index, :id].each_with_object({}) do |variable, set_variables| set_variables[variable] = arguments[variable] if arguments.key?(variable) end request_opts[:defined_params] = defined_params unless defined_params.empty? raise ArgumentError, "Required argument 'index' missing" unless arguments[:index] raise ArgumentError, "Required argument 'id' missing" unless arguments[:id] arguments = arguments.clone headers = arguments.delete(:headers) || {} body = nil _id = arguments.delete(:id) _index = arguments.delete(:index) method = Elasticsearch::API::HTTP_HEAD path = "#{Utils.listify(_index)}/_doc/#{Utils.listify(_id)}" params = Utils.process_params(arguments) Utils.rescue_from_not_found do perform_request(method, path, params, body, headers, request_opts).status == 200 end end
Check a document. Verify that a document exists. For example, check to see if a document with the _id
0 exists: + HEAD my-index-000001/_doc/0 + If the document exists, the API
returns a status code of +200 - OK+. If the document doesn’t exist, the API
returns +404 - Not Found+. **Versioning support** You can use the version
parameter to check the document only if its current version is equal to the specified one. Internally, Elasticsearch
has marked the old document as deleted and added an entirely new document. The old version of the document doesn’t disappear immediately, although you won’t be able to access it. Elasticsearch
cleans up deleted documents in the background as you continue to index more data.
@option arguments [String] :id A unique document identifier. (Required) @option arguments [String] :index A comma-separated list of data streams, indices, and aliases.
It supports wildcards (+*+). (*Required*)
@option arguments [String] :preference The node or shard the operation should be performed on.
By default, the operation is randomized between the shard replicas.If it is set to +_local+, the operation will prefer to be run on a local allocated shard when possible. If it is set to a custom value, the value is used to guarantee that the same shards will be used for the same custom value. This can help with "jumping values" when hitting different shards in different refresh states. A sample value can be something like the web session ID or the user name.
@option arguments [Boolean] :realtime If true
, the request is real-time as opposed to near-real-time. Server default: true. @option arguments [Boolean] :refresh If true
, the request refreshes the relevant shards before retrieving the document.
Setting it to +true+ should be done after careful thought and verification that this does not cause a heavy load on the system (and slow down indexing).
@option arguments [String] :routing A custom value used to route operations to a specific shard. @option arguments [Boolean, String, Array<String>] :_source Indicates whether to return the _source
field (true
or false
) or lists the fields to return. @option arguments [String, Array<String>] :_source_excludes A comma-separated list of source fields to exclude from the response.
You can also use this parameter to exclude fields from the subset specified in +_source_includes+ query parameter. If the +_source+ parameter is +false+, this parameter is ignored.
@option arguments [String, Array<String>] :_source_includes A comma-separated list of source fields to include in the response.
If this parameter is specified, only these source fields are returned. You can exclude fields from this subset using the +_source_excludes+ query parameter. If the +_source+ parameter is +false+, this parameter is ignored.
@option arguments [String, Array<String>] :stored_fields A comma-separated list of stored fields to return as part of a hit.
If no fields are specified, no stored fields are included in the response. If this field is specified, the +_source+ parameter defaults to +false+.
@option arguments [Integer] :version Explicit version number for concurrency control.
The specified version must match the current version of the document for the request to succeed.
@option arguments [String] :version_type The version type. @option arguments [Hash] :headers Custom HTTP headers
@see www.elastic.co/docs/api/doc/elasticsearch/v9/operation/operation-get
Source
# File lib/elasticsearch/api/actions/exists_source.rb, line 51 def exists_source(arguments = {}) request_opts = { endpoint: arguments[:endpoint] || 'exists_source' } defined_params = [:index, :id].each_with_object({}) do |variable, set_variables| set_variables[variable] = arguments[variable] if arguments.key?(variable) end request_opts[:defined_params] = defined_params unless defined_params.empty? raise ArgumentError, "Required argument 'index' missing" unless arguments[:index] raise ArgumentError, "Required argument 'id' missing" unless arguments[:id] arguments = arguments.clone headers = arguments.delete(:headers) || {} body = nil _id = arguments.delete(:id) _index = arguments.delete(:index) method = Elasticsearch::API::HTTP_HEAD path = "#{Utils.listify(_index)}/_source/#{Utils.listify(_id)}" params = Utils.process_params(arguments) Elasticsearch::API::Response.new( perform_request(method, path, params, body, headers, request_opts) ) end
Check for a document source. Check whether a document source exists in an index. For example: + HEAD my-index-000001/_source/1 + A document’s source is not available if it is disabled in the mapping.
@option arguments [String] :id A unique identifier for the document. (Required) @option arguments [String] :index A comma-separated list of data streams, indices, and aliases.
It supports wildcards (+*+). (*Required*)
@option arguments [String] :preference The node or shard the operation should be performed on.
By default, the operation is randomized between the shard replicas.
@option arguments [Boolean] :realtime If true
, the request is real-time as opposed to near-real-time. Server default: true. @option arguments [Boolean] :refresh If true
, the request refreshes the relevant shards before retrieving the document.
Setting it to +true+ should be done after careful thought and verification that this does not cause a heavy load on the system (and slow down indexing).
@option arguments [String] :routing A custom value used to route operations to a specific shard. @option arguments [Boolean, String, Array<String>] :_source Indicates whether to return the _source
field (true
or false
) or lists the fields to return. @option arguments [String, Array<String>] :_source_excludes A comma-separated list of source fields to exclude in the response. @option arguments [String, Array<String>] :_source_includes A comma-separated list of source fields to include in the response. @option arguments [Integer] :version The version number for concurrency control.
It must match the current version of the document for the request to succeed.
@option arguments [String] :version_type The version type. @option arguments [Hash] :headers Custom HTTP headers
@see www.elastic.co/docs/api/doc/elasticsearch/v9/operation/operation-get
Source
# File lib/elasticsearch/api/actions/explain.rb, line 59 def explain(arguments = {}) request_opts = { endpoint: arguments[:endpoint] || 'explain' } defined_params = [:index, :id].each_with_object({}) do |variable, set_variables| set_variables[variable] = arguments[variable] if arguments.key?(variable) end request_opts[:defined_params] = defined_params unless defined_params.empty? raise ArgumentError, "Required argument 'index' missing" unless arguments[:index] raise ArgumentError, "Required argument 'id' missing" unless arguments[:id] arguments = arguments.clone headers = arguments.delete(:headers) || {} body = arguments.delete(:body) _id = arguments.delete(:id) _index = arguments.delete(:index) method = if body Elasticsearch::API::HTTP_POST else Elasticsearch::API::HTTP_GET end path = "#{Utils.listify(_index)}/_explain/#{Utils.listify(_id)}" params = Utils.process_params(arguments) Elasticsearch::API::Response.new( perform_request(method, path, params, body, headers, request_opts) ) end
Explain a document match result. Get information about why a specific document matches, or doesn’t match, a query. It computes a score explanation for a query and a specific document.
@option arguments [String] :id The document identifier. (Required) @option arguments [String] :index Index names that are used to limit the request.
Only a single index name can be provided to this parameter. (*Required*)
@option arguments [String] :analyzer The analyzer to use for the query string.
This parameter can be used only when the +q+ query string parameter is specified.
@option arguments [Boolean] :analyze_wildcard If true
, wildcard and prefix queries are analyzed.
This parameter can be used only when the +q+ query string parameter is specified.
@option arguments [String] :default_operator The default operator for query string query: AND
or OR
.
This parameter can be used only when the +q+ query string parameter is specified. Server default: OR.
@option arguments [String] :df The field to use as default where no field prefix is given in the query string.
This parameter can be used only when the +q+ query string parameter is specified.
@option arguments [Boolean] :lenient If true
, format-based query failures (such as providing text to a numeric field) in the query string will be ignored.
This parameter can be used only when the +q+ query string parameter is specified.
@option arguments [String] :preference The node or shard the operation should be performed on.
It is random by default.
@option arguments [String] :routing A custom value used to route operations to a specific shard. @option arguments [Boolean, String, Array<String>] :_source True
or false
to return the _source
field or not or a list of fields to return. @option arguments [String, Array<String>] :_source_excludes A comma-separated list of source fields to exclude from the response.
You can also use this parameter to exclude fields from the subset specified in +_source_includes+ query parameter. If the +_source+ parameter is +false+, this parameter is ignored.
@option arguments [String, Array<String>] :_source_includes A comma-separated list of source fields to include in the response.
If this parameter is specified, only these source fields are returned. You can exclude fields from this subset using the +_source_excludes+ query parameter. If the +_source+ parameter is +false+, this parameter is ignored.
@option arguments [String, Array<String>] :stored_fields A comma-separated list of stored fields to return in the response. @option arguments [String] :q The query in the Lucene query string syntax. @option arguments [Hash] :headers Custom HTTP headers @option arguments [Hash] :body request body
@see www.elastic.co/docs/api/doc/elasticsearch/v9/operation/operation-explain
Source
# File lib/elasticsearch/api/actions/field_caps.rb, line 48 def field_caps(arguments = {}) request_opts = { endpoint: arguments[:endpoint] || 'field_caps' } defined_params = [:index].each_with_object({}) do |variable, set_variables| set_variables[variable] = arguments[variable] if arguments.key?(variable) end request_opts[:defined_params] = defined_params unless defined_params.empty? arguments = arguments.clone headers = arguments.delete(:headers) || {} body = arguments.delete(:body) _index = arguments.delete(:index) method = if body Elasticsearch::API::HTTP_POST else Elasticsearch::API::HTTP_GET end path = if _index "#{Utils.listify(_index)}/_field_caps" else '_field_caps' end params = Utils.process_params(arguments) Elasticsearch::API::Response.new( perform_request(method, path, params, body, headers, request_opts) ) end
Get the field capabilities. Get information about the capabilities of fields among multiple indices. For data streams, the API
returns field capabilities among the stream’s backing indices. It returns runtime fields like any other field. For example, a runtime field with a type of keyword is returned the same as any other field that belongs to the keyword
family.
@option arguments [String, Array] :index A comma-separated list of data streams, indices, and aliases used to limit the request. Supports wildcards (*). To target all data streams and indices, omit this parameter or use * or _all. @option arguments [Boolean] :allow_no_indices If false, the request returns an error if any wildcard expression, index alias,
or +_all+ value targets only missing or closed indices. This behavior applies even if the request targets other open indices. For example, a request targeting +foo*,bar*+ returns an error if an index starts with foo but no index starts with bar. Server default: true.
@option arguments [String, Array<String>] :expand_wildcards The type of index that wildcard patterns can match. If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams. Supports comma-separated values, such as +open,hidden+. Server default: open. @option arguments [String, Array<String>] :fields A comma-separated list of fields to retrieve capabilities for. Wildcard (+*+) expressions are supported. @option arguments [Boolean] :ignore_unavailable If true
, missing or closed indices are not included in the response. @option arguments [Boolean] :include_unmapped If true, unmapped fields are included in the response. @option arguments [String] :filters A comma-separated list of filters to apply to the response. @option arguments [Array<String>] :types A comma-separated list of field types to include.
Any fields that do not match one of these types will be excluded from the results. It defaults to empty, meaning that all field types are returned.
@option arguments [Boolean] :include_empty_fields If false, empty fields are not included in the response. Server default: true. @option arguments [Hash] :headers Custom HTTP headers @option arguments [Hash] :body request body
@see www.elastic.co/docs/api/doc/elasticsearch/v9/operation/operation-field-caps
Source
# File lib/elasticsearch/api/actions/get.rb, line 99 def get(arguments = {}) request_opts = { endpoint: arguments[:endpoint] || 'get' } defined_params = [:index, :id].each_with_object({}) do |variable, set_variables| set_variables[variable] = arguments[variable] if arguments.key?(variable) end request_opts[:defined_params] = defined_params unless defined_params.empty? raise ArgumentError, "Required argument 'index' missing" unless arguments[:index] raise ArgumentError, "Required argument 'id' missing" unless arguments[:id] arguments = arguments.clone headers = arguments.delete(:headers) || {} body = nil _id = arguments.delete(:id) _index = arguments.delete(:index) method = Elasticsearch::API::HTTP_GET path = "#{Utils.listify(_index)}/_doc/#{Utils.listify(_id)}" params = Utils.process_params(arguments) if Array(arguments[:ignore]).include?(404) Utils.rescue_from_not_found do Elasticsearch::API::Response.new( perform_request(method, path, params, body, headers, request_opts) ) end else Elasticsearch::API::Response.new( perform_request(method, path, params, body, headers, request_opts) ) end end
Get a document by its ID. Get a document and its source or stored fields from an index. By default, this API
is realtime and is not affected by the refresh rate of the index (when data will become visible for search). In the case where stored fields are requested with the stored_fields
parameter and the document has been updated but is not yet refreshed, the API
will have to parse and analyze the source to extract the stored fields. To turn off realtime behavior, set the realtime
parameter to false. **Source filtering** By default, the API
returns the contents of the _source
field unless you have used the stored_fields
parameter or the _source
field is turned off. You can turn off _source
retrieval by using the _source
parameter: + GET my-index-000001/_doc/0?_source=false + If you only need one or two fields from the _source
, use the _source_includes
or _source_excludes
parameters to include or filter out particular fields. This can be helpful with large documents where partial retrieval can save on network overhead Both parameters take a comma separated list of fields or wildcard expressions. For example: + GET my-index-000001/_doc/0?_source_includes=*.id&_source_excludes=entities + If you only want to specify includes, you can use a shorter notation: + GET my-index-000001/_doc/0?_source=*.id + Routing If routing is used during indexing, the routing value also needs to be specified to retrieve a document. For example: + GET my-index-000001/_doc/2?routing=user1 + This request gets the document with ID 2, but it is routed based on the user. The document is not fetched if the correct routing is not specified. Distributed The GET operation is hashed into a specific shard ID. It is then redirected to one of the replicas within that shard ID and returns the result. The replicas are the primary shard and its replicas within that shard ID group. This means that the more replicas you have, the better your GET scaling will be. **Versioning support** You can use the version
parameter to retrieve the document only if its current version is equal to the specified one. Internally, Elasticsearch
has marked the old document as deleted and added an entirely new document. The old version of the document doesn’t disappear immediately, although you won’t be able to access it. Elasticsearch
cleans up deleted documents in the background as you continue to index more data.
@option arguments [String] :id A unique document identifier. (Required) @option arguments [String] :index The name of the index that contains the document. (Required) @option arguments [Boolean] :force_synthetic_source Indicates whether the request forces synthetic _source
.
Use this paramater to test if the mapping supports synthetic +_source+ and to get a sense of the worst case performance. Fetches with this parameter enabled will be slower than enabling synthetic source natively in the index.
@option arguments [String] :preference The node or shard the operation should be performed on.
By default, the operation is randomized between the shard replicas.If it is set to +_local+, the operation will prefer to be run on a local allocated shard when possible. If it is set to a custom value, the value is used to guarantee that the same shards will be used for the same custom value. This can help with "jumping values" when hitting different shards in different refresh states. A sample value can be something like the web session ID or the user name.
@option arguments [Boolean] :realtime If true
, the request is real-time as opposed to near-real-time. Server default: true. @option arguments [Boolean] :refresh If true
, the request refreshes the relevant shards before retrieving the document.
Setting it to +true+ should be done after careful thought and verification that this does not cause a heavy load on the system (and slow down indexing).
@option arguments [String] :routing A custom value used to route operations to a specific shard. @option arguments [Boolean, String, Array<String>] :_source Indicates whether to return the _source
field (true
or false
) or lists the fields to return. @option arguments [String, Array<String>] :_source_excludes A comma-separated list of source fields to exclude from the response.
You can also use this parameter to exclude fields from the subset specified in +_source_includes+ query parameter. If the +_source+ parameter is +false+, this parameter is ignored.
@option arguments [String, Array<String>] :_source_includes A comma-separated list of source fields to include in the response.
If this parameter is specified, only these source fields are returned. You can exclude fields from this subset using the +_source_excludes+ query parameter. If the +_source+ parameter is +false+, this parameter is ignored.
@option arguments [String, Array<String>] :stored_fields A comma-separated list of stored fields to return as part of a hit.
If no fields are specified, no stored fields are included in the response. If this field is specified, the +_source+ parameter defaults to +false+. Only leaf fields can be retrieved with the +stored_field+ option. Object fields can't be returned;if specified, the request fails.
@option arguments [Integer] :version The version number for concurrency control.
It must match the current version of the document for the request to succeed.
@option arguments [String] :version_type The version type. @option arguments [Hash] :headers Custom HTTP headers
@see www.elastic.co/docs/api/doc/elasticsearch/v9/operation/operation-get
Source
# File lib/elasticsearch/api/actions/get_script.rb, line 35 def get_script(arguments = {}) request_opts = { endpoint: arguments[:endpoint] || 'get_script' } defined_params = [:id].each_with_object({}) do |variable, set_variables| set_variables[variable] = arguments[variable] if arguments.key?(variable) end request_opts[:defined_params] = defined_params unless defined_params.empty? raise ArgumentError, "Required argument 'id' missing" unless arguments[:id] arguments = arguments.clone headers = arguments.delete(:headers) || {} body = nil _id = arguments.delete(:id) method = Elasticsearch::API::HTTP_GET path = "_scripts/#{Utils.listify(_id)}" params = Utils.process_params(arguments) Elasticsearch::API::Response.new( perform_request(method, path, params, body, headers, request_opts) ) end
Get a script or search template. Retrieves a stored script or search template.
@option arguments [String] :id The identifier for the stored script or search template. (Required) @option arguments [Time] :master_timeout The period to wait for the master node.
If the master node is not available before the timeout expires, the request fails and returns an error. It can also be set to +-1+ to indicate that the request should never timeout. Server default: .
@option arguments [Hash] :headers Custom HTTP headers
@see www.elastic.co/docs/api/doc/elasticsearch/v9/operation/operation-get-script
Source
# File lib/elasticsearch/api/actions/get_script_context.rb, line 31 def get_script_context(arguments = {}) request_opts = { endpoint: arguments[:endpoint] || 'get_script_context' } arguments = arguments.clone headers = arguments.delete(:headers) || {} body = nil method = Elasticsearch::API::HTTP_GET path = '_script_context' params = {} Elasticsearch::API::Response.new( perform_request(method, path, params, body, headers, request_opts) ) end
Get script contexts. Get a list of supported script contexts and their methods.
@option arguments [Hash] :headers Custom HTTP headers
@see www.elastic.co/docs/api/doc/elasticsearch/v9/operation/operation-get-script-context
Source
# File lib/elasticsearch/api/actions/get_script_languages.rb, line 31 def get_script_languages(arguments = {}) request_opts = { endpoint: arguments[:endpoint] || 'get_script_languages' } arguments = arguments.clone headers = arguments.delete(:headers) || {} body = nil method = Elasticsearch::API::HTTP_GET path = '_script_language' params = {} Elasticsearch::API::Response.new( perform_request(method, path, params, body, headers, request_opts) ) end
Get script languages. Get a list of available script types, languages, and contexts.
@option arguments [Hash] :headers Custom HTTP headers
@see www.elastic.co/docs/api/doc/elasticsearch/v9/operation/operation-get-script-languages
Source
# File lib/elasticsearch/api/actions/get_source.rb, line 54 def get_source(arguments = {}) request_opts = { endpoint: arguments[:endpoint] || 'get_source' } defined_params = [:index, :id].each_with_object({}) do |variable, set_variables| set_variables[variable] = arguments[variable] if arguments.key?(variable) end request_opts[:defined_params] = defined_params unless defined_params.empty? raise ArgumentError, "Required argument 'index' missing" unless arguments[:index] raise ArgumentError, "Required argument 'id' missing" unless arguments[:id] arguments = arguments.clone headers = arguments.delete(:headers) || {} body = nil _id = arguments.delete(:id) _index = arguments.delete(:index) method = Elasticsearch::API::HTTP_GET path = "#{Utils.listify(_index)}/_source/#{Utils.listify(_id)}" params = Utils.process_params(arguments) Elasticsearch::API::Response.new( perform_request(method, path, params, body, headers, request_opts) ) end
Get a document’s source. Get the source of a document. For example: + GET my-index-000001/_source/1 + You can use the source filtering parameters to control which parts of the _source
are returned: + GET my-index-000001/_source/1/?_source_includes=*.id&_source_excludes=entities +
@option arguments [String] :id A unique document identifier. (Required) @option arguments [String] :index The name of the index that contains the document. (Required) @option arguments [String] :preference The node or shard the operation should be performed on.
By default, the operation is randomized between the shard replicas.
@option arguments [Boolean] :realtime If true
, the request is real-time as opposed to near-real-time. Server default: true. @option arguments [Boolean] :refresh If true
, the request refreshes the relevant shards before retrieving the document.
Setting it to +true+ should be done after careful thought and verification that this does not cause a heavy load on the system (and slow down indexing).
@option arguments [String] :routing A custom value used to route operations to a specific shard. @option arguments [Boolean, String, Array<String>] :_source Indicates whether to return the _source
field (true
or false
) or lists the fields to return. @option arguments [String, Array<String>] :_source_excludes A comma-separated list of source fields to exclude in the response. @option arguments [String, Array<String>] :_source_includes A comma-separated list of source fields to include in the response. @option arguments [String, Array<String>] :stored_fields A comma-separated list of stored fields to return as part of a hit. @option arguments [Integer] :version The version number for concurrency control.
It must match the current version of the document for the request to succeed.
@option arguments [String] :version_type The version type. @option arguments [Hash] :headers Custom HTTP headers
@see www.elastic.co/docs/api/doc/elasticsearch/v9/operation/operation-get
Source
# File lib/elasticsearch/api/actions/health_report.rb, line 46 def health_report(arguments = {}) request_opts = { endpoint: arguments[:endpoint] || 'health_report' } defined_params = [:feature].each_with_object({}) do |variable, set_variables| set_variables[variable] = arguments[variable] if arguments.key?(variable) end request_opts[:defined_params] = defined_params unless defined_params.empty? arguments = arguments.clone headers = arguments.delete(:headers) || {} body = nil _feature = arguments.delete(:feature) method = Elasticsearch::API::HTTP_GET path = if _feature "_health_report/#{Utils.listify(_feature)}" else '_health_report' end params = Utils.process_params(arguments) Elasticsearch::API::Response.new( perform_request(method, path, params, body, headers, request_opts) ) end
Get the cluster health. Get a report with the health status of an Elasticsearch
cluster. The report contains a list of indicators that compose Elasticsearch
functionality. Each indicator has a health status of: green, unknown, yellow or red. The indicator will provide an explanation and metadata describing the reason for its current health status. The cluster’s status is controlled by the worst indicator status. In the event that an indicator’s status is non-green, a list of impacts may be present in the indicator result which detail the functionalities that are negatively affected by the health issue. Each impact carries with it a severity level, an area of the system that is affected, and a simple description of the impact on the system. Some health indicators can determine the root cause of a health problem and prescribe a set of steps that can be performed in order to improve the health of the system. The root cause and remediation steps are encapsulated in a diagnosis. A diagnosis contains a cause detailing a root cause analysis, an action containing a brief description of the steps to take to fix the problem, the list of affected resources (if applicable), and a detailed step-by-step troubleshooting guide to fix the diagnosed problem. NOTE: The health indicators perform root cause analysis of non-green health statuses. This can be computationally expensive when called frequently. When setting up automated polling of the API
for health status, set verbose to false to disable the more expensive analysis logic.
@option arguments [String] :feature A feature of the cluster, as returned by the top-level health report API
. @option arguments [Time] :timeout Explicit operation timeout. @option arguments [Boolean] :verbose Opt-in for more information about the health of the system. Server default: true. @option arguments [Integer] :size Limit the number of affected resources the health report API
returns. Server default: 1000. @option arguments [Hash] :headers Custom HTTP headers
@see www.elastic.co/docs/api/doc/elasticsearch/v9/operation/operation-health-report
Source
# File lib/elasticsearch/api/actions/index.rb, line 146 def index(arguments = {}) request_opts = { endpoint: arguments[:endpoint] || 'index' } defined_params = [:index, :id].each_with_object({}) do |variable, set_variables| set_variables[variable] = arguments[variable] if arguments.key?(variable) end request_opts[:defined_params] = defined_params unless defined_params.empty? raise ArgumentError, "Required argument 'body' missing" unless arguments[:body] raise ArgumentError, "Required argument 'index' missing" unless arguments[:index] arguments = arguments.clone headers = arguments.delete(:headers) || {} body = arguments.delete(:body) _id = arguments.delete(:id) _index = arguments.delete(:index) method = _id ? Elasticsearch::API::HTTP_PUT : Elasticsearch::API::HTTP_POST path = if _index && _id "#{Utils.listify(_index)}/_doc/#{Utils.listify(_id)}" else "#{Utils.listify(_index)}/_doc" end params = Utils.process_params(arguments) Elasticsearch::API::Response.new( perform_request(method, path, params, body, headers, request_opts) ) end
Create or update a document in an index. Add a JSON document to the specified data stream or index and make it searchable. If the target is an index and the document already exists, the request updates the document and increments its version. NOTE: You cannot use this API
to send update requests for existing documents in a data stream. If the Elasticsearch
security features are enabled, you must have the following index privileges for the target data stream, index, or index alias:
-
To add or overwrite a document using the +PUT /<target>/_doc/<_id>+ request format, you must have the
create
,index
, orwrite
index privilege. -
To add a document using the +POST /<target>/_doc/+ request format, you must have the
create_doc
,create
,index
, orwrite
index privilege. -
To automatically create a data stream or index with this
API
request, you must have theauto_configure
,create_index
, ormanage
index privilege.
Automatic data stream creation requires a matching index template with data stream enabled. NOTE: Replica shards might not all be started when an indexing operation returns successfully. By default, only the primary is required. Set wait_for_active_shards
to change this default behavior. **Automatically create data streams and indices** If the request’s target doesn’t exist and matches an index template with a data_stream
definition, the index operation automatically creates the data stream. If the target doesn’t exist and doesn’t match a data stream template, the operation automatically creates the index and applies any matching index templates. NOTE: Elasticsearch
includes several built-in index templates. To avoid naming collisions with these templates, refer to index pattern documentation. If no mapping exists, the index operation creates a dynamic mapping. By default, new fields and objects are automatically added to the mapping if needed. Automatic index creation is controlled by the action.auto_create_index
setting. If it is true
, any index can be created automatically. You can modify this setting to explicitly allow or block automatic creation of indices that match specified patterns or set it to false
to turn off automatic index creation entirely. Specify a comma-separated list of patterns you want to allow or prefix each pattern with +++ or -
to indicate whether it should be allowed or blocked. When a list is specified, the default behaviour is to disallow. NOTE: The action.auto_create_index
setting affects the automatic creation of indices only. It does not affect the creation of data streams. **Optimistic concurrency control** Index operations can be made conditional and only be performed if the last modification to the document was assigned the sequence number and primary term specified by the if_seq_no
and if_primary_term
parameters. If a mismatch is detected, the operation will result in a VersionConflictException
and a status code of 409
. Routing By default, shard placement — or routing — is controlled by using a hash of the document’s ID value. For more explicit control, the value fed into the hash function used by the router can be directly specified on a per-operation basis using the routing
parameter. When setting up explicit mapping, you can also use the _routing
field to direct the index operation to extract the routing value from the document itself. This does come at the (very minimal) cost of an additional document parsing pass. If the _routing
mapping is defined and set to be required, the index operation will fail if no routing value is provided or extracted. NOTE: Data streams do not support custom routing unless they were created with the allow_custom_routing
setting enabled in the template. Distributed The index operation is directed to the primary shard based on its route and performed on the actual node containing this shard. After the primary shard completes the operation, if needed, the update is distributed to applicable replicas. **Active shards** To improve the resiliency of writes to the system, indexing operations can be configured to wait for a certain number of active shard copies before proceeding with the operation. If the requisite number of active shard copies are not available, then the write operation must wait and retry, until either the requisite shard copies have started or a timeout occurs. By default, write operations only wait for the primary shards to be active before proceeding (that is to say wait_for_active_shards
is 1
). This default can be overridden in the index settings dynamically by setting index.write.wait_for_active_shards
. To alter this behavior per operation, use the +wait_for_active_shards request+ parameter. Valid values are all or any positive integer up to the total number of configured copies per shard in the index (which is +number_of_replicas++1). Specifying a negative value or a number greater than the number of shard copies will throw an error. For example, suppose you have a cluster of three nodes, A, B, and C and you create an index index with the number of replicas set to 3 (resulting in 4 shard copies, one more copy than there are nodes). If you attempt an indexing operation, by default the operation will only ensure the primary copy of each shard is available before proceeding. This means that even if B and C went down and A hosted the primary shard copies, the indexing operation would still proceed with only one copy of the data. If wait_for_active_shards
is set on the request to 3
(and all three nodes are up), the indexing operation will require 3 active shard copies before proceeding. This requirement should be met because there are 3 active nodes in the cluster, each one holding a copy of the shard. However, if you set wait_for_active_shards
to all
(or to 4
, which is the same in this situation), the indexing operation will not proceed as you do not have all 4 copies of each shard active in the index. The operation will timeout unless a new node is brought up in the cluster to host the fourth copy of the shard. It is important to note that this setting greatly reduces the chances of the write operation not writing to the requisite number of shard copies, but it does not completely eliminate the possibility, because this check occurs before the write operation starts. After the write operation is underway, it is still possible for replication to fail on any number of shard copies but still succeed on the primary. The _shards
section of the API
response reveals the number of shard copies on which replication succeeded and failed. **No operation (noop) updates** When updating a document by using this API
, a new version of the document is always created even if the document hasn’t changed. If this isn’t acceptable use the _update
API
with detect_noop
set to true
. The detect_noop
option isn’t available on this API
because it doesn’t fetch the old source and isn’t able to compare it against the new source. There isn’t a definitive rule for when noop updates aren’t acceptable. It’s a combination of lots of factors like how frequently your data source sends updates that are actually noops and how many queries per second Elasticsearch
runs on the shard receiving the updates. Versioning Each indexed document is given a version number. By default, internal versioning is used that starts at 1 and increments with each update, deletes included. Optionally, the version number can be set to an external value (for example, if maintained in a database). To enable this functionality, version_type
should be set to external
. The value provided must be a numeric, long value greater than or equal to 0, and less than around +9.2e+18+. NOTE: Versioning is completely real time, and is not affected by the near real time aspects of search operations. If no version is provided, the operation runs without any version checks. When using the external version type, the system checks to see if the version number passed to the index request is greater than the version of the currently stored document. If true, the document will be indexed and the new version number used. If the value provided is less than or equal to the stored document’s version number, a version conflict will occur and the index operation will fail. For example: “‘ PUT my-index-000001/_doc/1?version=2&version_type=external {
"user": { "id": "elkbee" }
} In this example, the operation will succeed since the supplied version of 2 is higher than the current document version of 1. If the document was already updated and its version was set to 2 or higher, the indexing command will fail and result in a conflict (409 HTTP status code). A nice side effect is that there is no need to maintain strict ordering of async indexing operations run as a result of changes to a source database, as long as version numbers from the source database are used. Even the simple case of updating the Elasticsearch
index using data from a database is simplified if external versioning is used, as only the latest version will be used if the index operations arrive out of order.
@option arguments [String] :id A unique identifier for the document.
To automatically generate a document ID, use the +POST /<target>/_doc/+ request format and omit this parameter.
@option arguments [String] :index The name of the data stream or index to target.
If the target doesn't exist and matches the name or wildcard (+*+) pattern of an index template with a +data_stream+ definition, this request creates the data stream. If the target doesn't exist and doesn't match a data stream template, this request creates the index. You can check for existing targets with the resolve index API. (*Required*)
@option arguments [Integer] :if_primary_term Only perform the operation if the document has this primary term. @option arguments [Integer] :if_seq_no Only perform the operation if the document has this sequence number. @option arguments [Boolean] :include_source_on_error True or false if to include the document source in the error message in case of parsing errors. Server default: true. @option arguments [String] :op_type Set to create
to only index the document if it does not already exist (put if absent).
If a document with the specified +_id+ already exists, the indexing operation will fail. The behavior is the same as using the +<index>/_create+ endpoint. If a document ID is specified, this paramater defaults to +index+. Otherwise, it defaults to +create+. If the request targets a data stream, an +op_type+ of +create+ is required.
@option arguments [String] :pipeline The ID of the pipeline to use to preprocess incoming documents.
If the index has a default ingest pipeline specified, then setting the value to +_none+ disables the default ingest pipeline for this request. If a final pipeline is configured it will always run, regardless of the value of this parameter.
@option arguments [String] :refresh If true
, Elasticsearch
refreshes the affected shards to make this operation visible to search.
If +wait_for+, it waits for a refresh to make this operation visible to search. If +false+, it does nothing with refreshes. Server default: false.
@option arguments [String] :routing A custom value that is used to route operations to a specific shard. @option arguments [Time] :timeout The period the request waits for the following operations: automatic index creation, dynamic mapping updates, waiting for active shards.This parameter is useful for situations where the primary shard assigned to perform the operation might not be available when the operation runs.
Some reasons for this might be that the primary shard is currently recovering from a gateway or undergoing relocation. By default, the operation will wait on the primary shard to become available for at least 1 minute before failing and responding with an error. The actual wait time could be longer, particularly when multiple waits occur. Server default: 1m.
@option arguments [Integer] :version An explicit version number for concurrency control.
It must be a non-negative long number.
@option arguments [String] :version_type The version type. @option arguments [Integer, String] :wait_for_active_shards The number of shard copies that must be active before proceeding with the operation.
You can set it to +all+ or any positive integer up to the total number of shards in the index (+number_of_replicas+1+). The default value of +1+ means it waits for each primary shard to be active. Server default: 1.
@option arguments [Boolean] :require_alias If true
, the destination must be an index alias. @option arguments [Hash] :headers Custom HTTP headers @option arguments [Hash] :body document
@see www.elastic.co/docs/api/doc/elasticsearch/v9/operation/operation-create
Source
# File lib/elasticsearch/api/actions/info.rb, line 31 def info(arguments = {}) request_opts = { endpoint: arguments[:endpoint] || 'info' } arguments = arguments.clone headers = arguments.delete(:headers) || {} body = nil method = Elasticsearch::API::HTTP_GET path = '' params = {} Elasticsearch::API::Response.new( perform_request(method, path, params, body, headers, request_opts) ) end
Get cluster info. Get basic build, version, and cluster information.
@option arguments [Hash] :headers Custom HTTP headers
@see www.elastic.co/docs/api/doc/elasticsearch/v9/group/endpoint-info
Source
# File lib/elasticsearch/api/actions/mget.rb, line 57 def mget(arguments = {}) request_opts = { endpoint: arguments[:endpoint] || 'mget' } defined_params = [:index].each_with_object({}) do |variable, set_variables| set_variables[variable] = arguments[variable] if arguments.key?(variable) end request_opts[:defined_params] = defined_params unless defined_params.empty? raise ArgumentError, "Required argument 'body' missing" unless arguments[:body] arguments = arguments.clone headers = arguments.delete(:headers) || {} body = arguments.delete(:body) _index = arguments.delete(:index) method = Elasticsearch::API::HTTP_POST path = if _index "#{Utils.listify(_index)}/_mget" else '_mget' end params = Utils.process_params(arguments) Elasticsearch::API::Response.new( perform_request(method, path, params, body, headers, request_opts) ) end
Get multiple documents. Get multiple JSON documents by ID from one or more indices. If you specify an index in the request URI, you only need to specify the document IDs in the request body. To ensure fast responses, this multi get (mget) API
responds with partial results if one or more shards fail. **Filter source fields** By default, the _source
field is returned for every document (if stored). Use the _source
and _source_include
or source_exclude
attributes to filter what fields are returned for a particular document. You can include the _source
, _source_includes
, and _source_excludes
query parameters in the request URI to specify the defaults to use when there are no per-document instructions. **Get stored fields** Use the stored_fields
attribute to specify the set of stored fields you want to retrieve. Any requested fields that are not stored are ignored. You can include the stored_fields
query parameter in the request URI to specify the defaults to use when there are no per-document instructions.
@option arguments [String] :index Name of the index to retrieve documents from when ids
are specified, or when a document in the docs
array does not specify an index. @option arguments [Boolean] :force_synthetic_source Should this request force synthetic _source?
Use this to test if the mapping supports synthetic _source and to get a sense of the worst case performance. Fetches with this enabled will be slower the enabling synthetic source natively in the index.
@option arguments [String] :preference Specifies the node or shard the operation should be performed on. Random by default. @option arguments [Boolean] :realtime If true
, the request is real-time as opposed to near-real-time. Server default: true. @option arguments [Boolean] :refresh If true
, the request refreshes relevant shards before retrieving documents. @option arguments [String] :routing Custom value used to route operations to a specific shard. @option arguments [Boolean, String, Array<String>] :_source True or false to return the _source
field or not, or a list of fields to return. @option arguments [String, Array<String>] :_source_excludes A comma-separated list of source fields to exclude from the response.
You can also use this parameter to exclude fields from the subset specified in +_source_includes+ query parameter.
@option arguments [String, Array<String>] :_source_includes A comma-separated list of source fields to include in the response.
If this parameter is specified, only these source fields are returned. You can exclude fields from this subset using the +_source_excludes+ query parameter. If the +_source+ parameter is +false+, this parameter is ignored.
@option arguments [String, Array<String>] :stored_fields If true
, retrieves the document fields stored in the index rather than the document _source
. Server default: false. @option arguments [Hash] :headers Custom HTTP headers @option arguments [Hash] :body request body
@see www.elastic.co/docs/api/doc/elasticsearch/v9/operation/operation-mget
Source
# File lib/elasticsearch/api/actions/msearch.rb, line 63 def msearch(arguments = {}) request_opts = { endpoint: arguments[:endpoint] || 'msearch' } defined_params = [:index].each_with_object({}) do |variable, set_variables| set_variables[variable] = arguments[variable] if arguments.key?(variable) end request_opts[:defined_params] = defined_params unless defined_params.empty? raise ArgumentError, "Required argument 'body' missing" unless arguments[:body] arguments = arguments.clone headers = arguments.delete(:headers) || {} body = arguments.delete(:body) _index = arguments.delete(:index) method = Elasticsearch::API::HTTP_POST path = if _index "#{Utils.listify(_index)}/_msearch" else '_msearch' end params = Utils.process_params(arguments) if body.is_a?(Array) && body.any? { |d| d.key? :search } payload = body.each_with_object([]) do |item, sum| meta = item data = meta.delete(:search) sum << meta sum << data end.map { |item| Elasticsearch::API.serializer.dump(item) } payload << '' unless payload.empty? payload = payload.join("\n") elsif body.is_a?(Array) payload = body.map { |d| d.is_a?(String) ? d : Elasticsearch::API.serializer.dump(d) } payload << '' unless payload.empty? payload = payload.join("\n") else payload = body end headers.merge!({ 'Content-Type' => 'application/vnd.elasticsearch+x-ndjson; compatible-with=9' }) Elasticsearch::API::Response.new( perform_request(method, path, params, payload, headers, request_opts) ) end
Run multiple searches. The format of the request is similar to the bulk API
format and makes use of the newline delimited JSON (NDJSON) format. The structure is as follows: + headern bodyn headern bodyn + This structure is specifically optimized to reduce parsing if a specific search ends up redirected to another node. IMPORTANT: The final line of data must end with a newline character \n
. Each newline character may be preceded by a carriage return \r
. When sending requests to this endpoint the Content-Type
header should be set to application/x-ndjson
.
@option arguments [String, Array] :index Comma-separated list of data streams, indices, and index aliases to search. @option arguments [Boolean] :allow_no_indices If false, the request returns an error if any wildcard expression, index alias, or _all value targets only missing or closed indices. This behavior applies even if the request targets other open indices. For example, a request targeting foo*,bar* returns an error if an index starts with foo but no index starts with bar. @option arguments [Boolean] :ccs_minimize_roundtrips If true, network roundtrips between the coordinating node and remote clusters are minimized for cross-cluster search requests. Server default: true. @option arguments [String, Array<String>] :expand_wildcards Type of index that wildcard expressions can match. If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams. @option arguments [Boolean] :ignore_throttled If true, concrete, expanded or aliased indices are ignored when frozen. @option arguments [Boolean] :ignore_unavailable If true, missing or closed indices are not included in the response. @option arguments [Boolean] :include_named_queries_score Indicates whether hit.matched_queries should be rendered as a map that includes
the name of the matched query associated with its score (true) or as an array containing the name of the matched queries (false) This functionality reruns each named query on every hit in a search response. Typically, this adds a small overhead to a request. However, using computationally expensive named queries on a large number of hits may add significant overhead.
@option arguments [Integer] :max_concurrent_searches Maximum number of concurrent searches the multi search API
can execute.
Defaults to +max(1, (# of data nodes * min(search thread pool size, 10)))+.
@option arguments [Integer] :max_concurrent_shard_requests Maximum number of concurrent shard requests that each sub-search request executes per node. Server default: 5. @option arguments [Integer] :pre_filter_shard_size Defines a threshold that enforces a pre-filter roundtrip to prefilter search shards based on query rewriting if the number of shards the search request expands to exceeds the threshold. This filter roundtrip can limit the number of shards significantly if for instance a shard can not match any documents based on its rewrite method i.e., if date filters are mandatory to match but the shard bounds and the query are disjoint. @option arguments [Boolean] :rest_total_hits_as_int If true, hits.total are returned as an integer in the response. Defaults to false, which returns an object. @option arguments [String] :routing Custom routing value used to route search operations to a specific shard. @option arguments [String] :search_type Indicates whether global term and document frequencies should be used when scoring returned documents. @option arguments [Boolean] :typed_keys Specifies whether aggregation and suggester names should be prefixed by their respective types in the response. @option arguments [Hash] :headers Custom HTTP headers @option arguments [Hash] :body searches
@see www.elastic.co/docs/api/doc/elasticsearch/v9/operation/operation-msearch
Source
# File lib/elasticsearch/api/actions/msearch_template.rb, line 51 def msearch_template(arguments = {}) request_opts = { endpoint: arguments[:endpoint] || 'msearch_template' } defined_params = [:index].each_with_object({}) do |variable, set_variables| set_variables[variable] = arguments[variable] if arguments.key?(variable) end request_opts[:defined_params] = defined_params unless defined_params.empty? raise ArgumentError, "Required argument 'body' missing" unless arguments[:body] arguments = arguments.clone headers = arguments.delete(:headers) || {} body = arguments.delete(:body) _index = arguments.delete(:index) method = Elasticsearch::API::HTTP_POST path = if _index "#{Utils.listify(_index)}/_msearch/template" else '_msearch/template' end params = Utils.process_params(arguments) if body.is_a?(Array) payload = body.map { |d| d.is_a?(String) ? d : Elasticsearch::API.serializer.dump(d) } payload << '' unless payload.empty? payload = payload.join("\n") else payload = body end headers.merge!({ 'Content-Type' => 'application/vnd.elasticsearch+x-ndjson; compatible-with=9' }) Elasticsearch::API::Response.new( perform_request(method, path, params, payload, headers, request_opts) ) end
Run multiple templated searches. Run multiple templated searches with a single request. If you are providing a text file or text input to curl
, use the --data-binary
flag instead of -d
to preserve newlines. For example: + $ cat requests { “index”: “my-index” } { “id”: “my-search-template”, “params”: { “query_string”: “hello world”, “from”: 0, “size”: 10 }} { “index”: “my-other-index” } { “id”: “my-other-search-template”, “params”: { “query_type”: “match_all” }} $ curl -H “Content-Type: application/x-ndjson” -XGET localhost:9200/_msearch/template –data-binary “@requests”; echo +
@option arguments [String, Array] :index A comma-separated list of data streams, indices, and aliases to search.
It supports wildcards (+*+). To search all data streams and indices, omit this parameter or use +*+.
@option arguments [Boolean] :ccs_minimize_roundtrips If true
, network round-trips are minimized for cross-cluster search requests. Server default: true. @option arguments [Integer] :max_concurrent_searches The maximum number of concurrent searches the API
can run. @option arguments [String] :search_type The type of the search operation. @option arguments [Boolean] :rest_total_hits_as_int If true
, the response returns hits.total
as an integer.
If +false+, it returns +hits.total+ as an object.
@option arguments [Boolean] :typed_keys If true
, the response prefixes aggregation and suggester names with their respective types. @option arguments [Hash] :headers Custom HTTP headers @option arguments [Hash] :body search_templates
@see www.elastic.co/docs/api/doc/elasticsearch/v9/operation/operation-msearch-template
Source
# File lib/elasticsearch/api/actions/mtermvectors.rb, line 54 def mtermvectors(arguments = {}) request_opts = { endpoint: arguments[:endpoint] || 'mtermvectors' } defined_params = [:index].each_with_object({}) do |variable, set_variables| set_variables[variable] = arguments[variable] if arguments.key?(variable) end request_opts[:defined_params] = defined_params unless defined_params.empty? arguments = arguments.clone headers = arguments.delete(:headers) || {} body = arguments.delete(:body) _index = arguments.delete(:index) method = if body Elasticsearch::API::HTTP_POST else Elasticsearch::API::HTTP_GET end path = if _index "#{Utils.listify(_index)}/_mtermvectors" else '_mtermvectors' end params = Utils.process_params(arguments) Elasticsearch::API::Response.new( perform_request(method, path, params, body, headers, request_opts) ) end
Get multiple term vectors. Get multiple term vectors with a single request. You can specify existing documents by index and ID or provide artificial documents in the body of the request. You can specify the index in the request body or request URI. The response contains a docs
array with all the fetched termvectors. Each element has the structure provided by the termvectors API
. **Artificial documents** You can also use mtermvectors
to generate term vectors for artificial documents provided in the body of the request. The mapping used is determined by the specified _index
.
@option arguments [String] :index The name of the index that contains the documents. @option arguments [Array<String>] :ids A comma-separated list of documents ids. You must define ids as parameter or set “ids” or “docs” in the request body @option arguments [String, Array<String>] :fields A comma-separated list or wildcard expressions of fields to include in the statistics.
It is used as the default list unless a specific field list is provided in the +completion_fields+ or +fielddata_fields+ parameters.
@option arguments [Boolean] :field_statistics If true
, the response includes the document count, sum of document frequencies, and sum of total term frequencies. Server default: true. @option arguments [Boolean] :offsets If true
, the response includes term offsets. Server default: true. @option arguments [Boolean] :payloads If true
, the response includes term payloads. Server default: true. @option arguments [Boolean] :positions If true
, the response includes term positions. Server default: true. @option arguments [String] :preference The node or shard the operation should be performed on.
It is random by default.
@option arguments [Boolean] :realtime If true, the request is real-time as opposed to near-real-time. Server default: true. @option arguments [String] :routing A custom value used to route operations to a specific shard. @option arguments [Boolean] :term_statistics If true, the response includes term frequency and document frequency. @option arguments [Integer] :version If true
, returns the document version as part of a hit. @option arguments [String] :version_type The version type. @option arguments [Hash] :headers Custom HTTP headers @option arguments [Hash] :body request body
@see www.elastic.co/docs/api/doc/elasticsearch/v9/operation/operation-mtermvectors
Source
# File lib/elasticsearch/api/actions/open_point_in_time.rb, line 69 def open_point_in_time(arguments = {}) request_opts = { endpoint: arguments[:endpoint] || 'open_point_in_time' } defined_params = [:index].each_with_object({}) do |variable, set_variables| set_variables[variable] = arguments[variable] if arguments.key?(variable) end request_opts[:defined_params] = defined_params unless defined_params.empty? raise ArgumentError, "Required argument 'index' missing" unless arguments[:index] arguments = arguments.clone headers = arguments.delete(:headers) || {} body = arguments.delete(:body) _index = arguments.delete(:index) method = Elasticsearch::API::HTTP_POST path = "#{Utils.listify(_index)}/_pit" params = Utils.process_params(arguments) Elasticsearch::API::Response.new( perform_request(method, path, params, body, headers, request_opts) ) end
Open a point in time. A search request by default runs against the most recent visible data of the target indices, which is called point in time. Elasticsearch
pit (point in time) is a lightweight view into the state of the data as it existed when initiated. In some cases, it’s preferred to perform multiple search requests using the same point in time. For example, if refreshes happen between search_after
requests, then the results of those requests might not be consistent as changes happening between searches are only visible to the more recent point in time. A point in time must be opened explicitly before being used in search requests. A subsequent search request with the pit
parameter must not specify index
, routing
, or preference
values as these parameters are copied from the point in time. Just like regular searches, you can use from
and size
to page through point in time search results, up to the first 10,000 hits. If you want to retrieve more hits, use PIT with search_after
. IMPORTANT: The open point in time request and each subsequent search request can return different identifiers; always use the most recently received ID for the next search request. When a PIT that contains shard failures is used in a search request, the missing are always reported in the search response as a NoShardAvailableActionException
exception. To get rid of these exceptions, a new PIT needs to be created so that shards missing from the previous PIT can be handled, assuming they become available in the meantime. **Keeping point in time alive** The keep_alive
parameter, which is passed to a open point in time request and search request, extends the time to live of the corresponding point in time. The value does not need to be long enough to process all data — it just needs to be long enough for the next request. Normally, the background merge process optimizes the index by merging together smaller segments to create new, bigger segments. Once the smaller segments are no longer needed they are deleted. However, open point-in-times prevent the old segments from being deleted since they are still in use. TIP: Keeping older segments alive means that more disk space and file handles are needed. Ensure that you have configured your nodes to have ample free file handles. Additionally, if a segment contains deleted or updated documents then the point in time must keep track of whether each document in the segment was live at the time of the initial search request. Ensure that your nodes have sufficient heap space if you have many open point-in-times on an index that is subject to ongoing deletes or updates. Note that a point-in-time doesn’t prevent its associated indices from being deleted. You can check how many point-in-times (that is, search contexts) are open with the nodes stats API
.
@option arguments [String, Array] :index A comma-separated list of index names to open point in time; use _all
or empty string to perform the operation on all indices (Required) @option arguments [Time] :keep_alive Extend the length of time that the point in time persists. (Required) @option arguments [Boolean] :ignore_unavailable If false
, the request returns an error if it targets a missing or closed index. @option arguments [String] :preference The node or shard the operation should be performed on.
By default, it is random.
@option arguments [String] :routing A custom value that is used to route operations to a specific shard. @option arguments [String, Array<String>] :expand_wildcards The type of index that wildcard patterns can match.
If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams. It supports comma-separated values, such as +open,hidden+. Valid values are: +all+, +open+, +closed+, +hidden+, +none+. Server default: open.
@option arguments [Boolean] :allow_partial_search_results Indicates whether the point in time tolerates unavailable shards or shard failures when initially creating the PIT.
If +false+, creating a point in time request when a shard is missing or unavailable will throw an exception. If +true+, the point in time will contain all the shards that are available at the time of the request.
@option arguments [Integer] :max_concurrent_shard_requests Maximum number of concurrent shard requests that each sub-search request executes per node. Server default: 5. @option arguments [Hash] :headers Custom HTTP headers @option arguments [Hash] :body request body
@see www.elastic.co/docs/api/doc/elasticsearch/v9/operation/operation-open-point-in-time
Source
# File lib/elasticsearch/api/actions/ping.rb, line 31 def ping(arguments = {}) request_opts = { endpoint: arguments[:endpoint] || 'ping' } arguments = arguments.clone headers = arguments.delete(:headers) || {} body = nil method = Elasticsearch::API::HTTP_HEAD path = '' params = {} begin perform_request(method, path, params, body, headers, request_opts).status == 200 rescue Exception => e raise e unless e.class.to_s =~ /NotFound|ConnectionFailed/ || e.message =~ /Not *Found|404|ConnectionFailed/i false end end
Ping the cluster. Get information about whether the cluster is running.
@option arguments [Hash] :headers Custom HTTP headers
@see www.elastic.co/docs/api/doc/elasticsearch/v9/group/endpoint-cluster
Source
# File lib/elasticsearch/api/actions/put_script.rb, line 42 def put_script(arguments = {}) request_opts = { endpoint: arguments[:endpoint] || 'put_script' } defined_params = [:id, :context].each_with_object({}) do |variable, set_variables| set_variables[variable] = arguments[variable] if arguments.key?(variable) end request_opts[:defined_params] = defined_params unless defined_params.empty? raise ArgumentError, "Required argument 'body' missing" unless arguments[:body] raise ArgumentError, "Required argument 'id' missing" unless arguments[:id] arguments = arguments.clone headers = arguments.delete(:headers) || {} body = arguments.delete(:body) _id = arguments.delete(:id) _context = arguments.delete(:context) method = Elasticsearch::API::HTTP_PUT path = if _id && _context "_scripts/#{Utils.listify(_id)}/#{Utils.listify(_context)}" else "_scripts/#{Utils.listify(_id)}" end params = Utils.process_params(arguments) Elasticsearch::API::Response.new( perform_request(method, path, params, body, headers, request_opts) ) end
Create or update a script or search template. Creates or updates a stored script or search template.
@option arguments [String] :id The identifier for the stored script or search template.
It must be unique within the cluster. (*Required*)
@option arguments [String] :context The context in which the script or search template should run.
To prevent errors, the API immediately compiles the script or template in this context.
@option arguments [Time] :master_timeout The period to wait for a connection to the master node.
If no response is received before the timeout expires, the request fails and returns an error. It can also be set to +-1+ to indicate that the request should never timeout. Server default: 30s.
@option arguments [Time] :timeout The period to wait for a response.
If no response is received before the timeout expires, the request fails and returns an error. It can also be set to +-1+ to indicate that the request should never timeout. Server default: 30s.
@option arguments [Hash] :headers Custom HTTP headers @option arguments [Hash] :body request body
@see www.elastic.co/docs/api/doc/elasticsearch/v9/operation/operation-put-script
Source
# File lib/elasticsearch/api/actions/rank_eval.rb, line 39 def rank_eval(arguments = {}) request_opts = { endpoint: arguments[:endpoint] || 'rank_eval' } defined_params = [:index].each_with_object({}) do |variable, set_variables| set_variables[variable] = arguments[variable] if arguments.key?(variable) end request_opts[:defined_params] = defined_params unless defined_params.empty? raise ArgumentError, "Required argument 'body' missing" unless arguments[:body] arguments = arguments.clone headers = arguments.delete(:headers) || {} body = arguments.delete(:body) _index = arguments.delete(:index) method = Elasticsearch::API::HTTP_POST path = if _index "#{Utils.listify(_index)}/_rank_eval" else '_rank_eval' end params = Utils.process_params(arguments) Elasticsearch::API::Response.new( perform_request(method, path, params, body, headers, request_opts) ) end
Evaluate ranked search results. Evaluate the quality of ranked search results over a set of typical search queries.
@option arguments [String, Array] :index A comma-separated list of data streams, indices, and index aliases used to limit the request.
Wildcard (+*+) expressions are supported. To target all data streams and indices in a cluster, omit this parameter or use +_all+ or +*+.
@option arguments [Boolean] :allow_no_indices If false
, the request returns an error if any wildcard expression, index alias, or _all
value targets only missing or closed indices. This behavior applies even if the request targets other open indices. For example, a request targeting +foo*,bar*+ returns an error if an index starts with foo
but no index starts with bar
. Server default: true. @option arguments [String, Array<String>] :expand_wildcards Whether to expand wildcard expression to concrete indices that are open, closed or both. @option arguments [Boolean] :ignore_unavailable If true
, missing or closed indices are not included in the response. @option arguments [String] :search_type Search operation type @option arguments [Hash] :headers Custom HTTP headers @option arguments [Hash] :body request body
@see www.elastic.co/docs/api/doc/elasticsearch/v9/operation/operation-rank-eval
Source
# File lib/elasticsearch/api/actions/reindex.rb, line 180 def reindex(arguments = {}) request_opts = { endpoint: arguments[:endpoint] || 'reindex' } raise ArgumentError, "Required argument 'body' missing" unless arguments[:body] arguments = arguments.clone headers = arguments.delete(:headers) || {} body = arguments.delete(:body) method = Elasticsearch::API::HTTP_POST path = '_reindex' params = Utils.process_params(arguments) Elasticsearch::API::Response.new( perform_request(method, path, params, body, headers, request_opts) ) end
Reindex documents. Copy documents from a source to a destination. You can copy all documents to the destination index or reindex a subset of the documents. The source can be any existing index, alias, or data stream. The destination must differ from the source. For example, you cannot reindex a data stream into itself. IMPORTANT: Reindex requires _source
to be enabled for all documents in the source. The destination should be configured as wanted before calling the reindex API
. Reindex does not copy the settings from the source or its associated template. Mappings, shard counts, and replicas, for example, must be configured ahead of time. If the Elasticsearch
security features are enabled, you must have the following security privileges:
-
The
read
index privilege for the source data stream, index, or alias. -
The
write
index privilege for the destination data stream, index, or index alias. -
To automatically create a data stream or index with a reindex
API
request, you must have theauto_configure
,create_index
, ormanage
index privilege for the destination data stream, index, or alias. -
If reindexing from a remote cluster, the
source.remote.user
must have themonitor
cluster privilege and theread
index privilege for the source data stream, index, or alias.
If reindexing from a remote cluster, you must explicitly allow the remote host in the reindex.remote.whitelist
setting. Automatic data stream creation requires a matching index template with data stream enabled. The dest
element can be configured like the index API
to control optimistic concurrency control. Omitting version_type
or setting it to internal
causes Elasticsearch
to blindly dump documents into the destination, overwriting any that happen to have the same ID. Setting version_type
to external
causes Elasticsearch
to preserve the version
from the source, create any documents that are missing, and update any documents that have an older version in the destination than they do in the source. Setting op_type
to create
causes the reindex API
to create only missing documents in the destination. All existing documents will cause a version conflict. IMPORTANT: Because data streams are append-only, any reindex request to a destination data stream must have an op_type
of create
. A reindex can only add new documents to a destination data stream. It cannot update existing documents in a destination data stream. By default, version conflicts abort the reindex process. To continue reindexing if there are conflicts, set the conflicts
request body property to proceed
. In this case, the response includes a count of the version conflicts that were encountered. Note that the handling of other error types is unaffected by the conflicts
property. Additionally, if you opt to count version conflicts, the operation could attempt to reindex more documents from the source than max_docs
until it has successfully indexed max_docs
documents into the target or it has gone through every document in the source query. NOTE: The reindex API
makes no effort to handle ID collisions. The last document written will “win” but the order isn’t usually predictable so it is not a good idea to rely on this behavior. Instead, make sure that IDs are unique by using a script. **Running reindex asynchronously** If the request contains +wait_for_completion=false+, Elasticsearch
performs some preflight checks, launches the request, and returns a task you can use to cancel or get the status of the task. Elasticsearch
creates a record of this task as a document at +_tasks/<task_id>+. **Reindex from multiple sources** If you have many sources to reindex it is generally better to reindex them one at a time rather than using a glob pattern to pick up multiple sources. That way you can resume the process if there are any errors by removing the partially completed source and starting over. It also makes parallelizing the process fairly simple: split the list of sources to reindex and run each list in parallel. For example, you can use a bash script like this: + for index in i1 i2 i3 i4 i5; do
curl -HContent-Type:application/json -XPOST localhost:9200/_reindex?pretty -d'{ "source": { "index": "'$index'" }, "dest": { "index": "'$index'-reindexed" } }'
done + Throttling Set requests_per_second
to any positive decimal number (1.4
, 6
, 1000
, for example) to throttle the rate at which reindex issues batches of index operations. Requests are throttled by padding each batch with a wait time. To turn off throttling, set requests_per_second
to -1
. The throttling is done by waiting between batches so that the scroll that reindex uses internally can be given a timeout that takes into account the padding. The padding time is the difference between the batch size divided by the requests_per_second
and the time spent writing. By default the batch size is 1000
, so if requests_per_second
is set to 500
: + target_time = 1000 / 500 per second = 2 seconds wait_time = target_time - write_time = 2 seconds - .5 seconds = 1.5 seconds + Since the batch is issued as a single bulk request, large batch sizes cause Elasticsearch
to create many requests and then wait for a while before starting the next set. This is “bursty” instead of “smooth”. Slicing Reindex supports sliced scroll to parallelize the reindexing process. This parallelization can improve efficiency and provide a convenient way to break the request down into smaller parts. NOTE: Reindexing from remote clusters does not support manual or automatic slicing. You can slice a reindex request manually by providing a slice ID and total number of slices to each request. You can also let reindex automatically parallelize by using sliced scroll to slice on _id
. The slices
parameter specifies the number of slices to use. Adding slices
to the reindex request just automates the manual process, creating sub-requests which means it has some quirks:
-
You can see these requests in the tasks
API
. These sub-requests are “child” tasks of the task for the request with slices. -
Fetching the status of the task for the request with
slices
only contains the status of completed slices. -
These sub-requests are individually addressable for things like cancellation and rethrottling.
-
Rethrottling the request with
slices
will rethrottle the unfinished sub-request proportionally. -
Canceling the request with
slices
will cancel each sub-request. -
Due to the nature of
slices
, each sub-request won’t get a perfectly even portion of the documents. All documents will be addressed, but some slices may be larger than others. Expect larger slices to have a more even distribution. -
Parameters like
requests_per_second
andmax_docs
on a request withslices
are distributed proportionally to each sub-request. Combine that with the previous point about distribution being uneven and you should conclude that usingmax_docs
withslices
might not result in exactlymax_docs
documents being reindexed. -
Each sub-request gets a slightly different snapshot of the source, though these are all taken at approximately the same time.
If slicing automatically, setting slices
to auto
will choose a reasonable number for most indices. If slicing manually or otherwise tuning automatic slicing, use the following guidelines. Query performance is most efficient when the number of slices is equal to the number of shards in the index. If that number is large (for example, 500
), choose a lower number as too many slices will hurt performance. Setting slices higher than the number of shards generally does not improve efficiency and adds overhead. Indexing performance scales linearly across available resources with the number of slices. Whether query or indexing performance dominates the runtime depends on the documents being reindexed and cluster resources. **Modify documents during reindexing** Like _update_by_query
, reindex operations support a script that modifies the document. Unlike _update_by_query
, the script is allowed to modify the document’s metadata. Just as in _update_by_query
, you can set ctx.op
to change the operation that is run on the destination. For example, set ctx.op
to noop
if your script decides that the document doesn’t have to be indexed in the destination. This “no operation” will be reported in the noop
counter in the response body. Set ctx.op
to delete
if your script decides that the document must be deleted from the destination. The deletion will be reported in the deleted
counter in the response body. Setting ctx.op
to anything else will return an error, as will setting any other field in ctx
. Think of the possibilities! Just be careful; you are able to change:
-
_id
-
_index
-
_version
-
_routing
Setting _version
to null
or clearing it from the ctx
map is just like not sending the version in an indexing request. It will cause the document to be overwritten in the destination regardless of the version on the target or the version type you use in the reindex API
. **Reindex from remote** Reindex supports reindexing from a remote Elasticsearch
cluster. The host
parameter must contain a scheme, host, port, and optional path. The username
and password
parameters are optional and when they are present the reindex operation will connect to the remote Elasticsearch
node using basic authentication. Be sure to use HTTPS when using basic authentication or the password will be sent in plain text. There are a range of settings available to configure the behavior of the HTTPS connection. When using Elastic Cloud, it is also possible to authenticate against the remote cluster through the use of a valid API
key. Remote hosts must be explicitly allowed with the reindex.remote.whitelist
setting. It can be set to a comma delimited list of allowed remote host and port combinations. Scheme is ignored; only the host and port are used. For example: + reindex.remote.whitelist: [otherhost:9200, another:9200, 127.0.10.*:9200, localhost:*“] + The list of allowed hosts must be configured on any nodes that will coordinate the reindex. This feature should work with remote clusters of any version of Elasticsearch
. This should enable you to upgrade from any version of Elasticsearch
to the current version by reindexing from a cluster of the old version. WARNING: Elasticsearch
does not support forward compatibility across major versions. For example, you cannot reindex from a 7.x cluster into a 6.x cluster. To enable queries sent to older versions of Elasticsearch
, the query
parameter is sent directly to the remote host without validation or modification. NOTE: Reindexing from remote clusters does not support manual or automatic slicing. Reindexing from a remote server uses an on-heap buffer that defaults to a maximum size of 100mb. If the remote index includes very large documents you’ll need to use a smaller batch size. It is also possible to set the socket read timeout on the remote connection with the socket_timeout
field and the connection timeout with the connect_timeout
field. Both default to 30 seconds. **Configuring SSL
parameters** Reindex from remote supports configurable SSL
settings. These must be specified in the elasticsearch.yml
file, with the exception of the secure settings, which you add in the Elasticsearch
keystore. It is not possible to configure SSL
in the body of the reindex request.
@option arguments [Boolean] :refresh If true
, the request refreshes affected shards to make this operation visible to search. @option arguments [Float] :requests_per_second The throttle for this request in sub-requests per second.
By default, there is no throttle. Server default: -1.
@option arguments [Time] :scroll The period of time that a consistent view of the index should be maintained for scrolled search. @option arguments [Integer, String] :slices The number of slices this task should be divided into.
It defaults to one slice, which means the task isn't sliced into subtasks.Reindex supports sliced scroll to parallelize the reindexing process. This parallelization can improve efficiency and provide a convenient way to break the request down into smaller parts.NOTE: Reindexing from remote clusters does not support manual or automatic slicing.If set to +auto+, Elasticsearch chooses the number of slices to use. This setting will use one slice per shard, up to a certain limit. If there are multiple sources, it will choose the number of slices based on the index or backing index with the smallest number of shards. Server default: 1.
@option arguments [Time] :timeout The period each indexing waits for automatic index creation, dynamic mapping updates, and waiting for active shards.
By default, Elasticsearch waits for at least one minute before failing. The actual wait time could be longer, particularly when multiple waits occur. Server default: 1m.
@option arguments [Integer, String] :wait_for_active_shards The number of shard copies that must be active before proceeding with the operation.
Set it to +all+ or any positive integer up to the total number of shards in the index (+number_of_replicas+1+). The default value is one, which means it waits for each primary shard to be active. Server default: 1.
@option arguments [Boolean] :wait_for_completion If true
, the request blocks until the operation is complete. Server default: true. @option arguments [Boolean] :require_alias If true
, the destination must be an index alias. @option arguments [Hash] :headers Custom HTTP headers @option arguments [Hash] :body request body
@see www.elastic.co/docs/api/doc/elasticsearch/v9/operation/operation-reindex
Source
# File lib/elasticsearch/api/actions/reindex_rethrottle.rb, line 41 def reindex_rethrottle(arguments = {}) request_opts = { endpoint: arguments[:endpoint] || 'reindex_rethrottle' } defined_params = [:task_id].each_with_object({}) do |variable, set_variables| set_variables[variable] = arguments[variable] if arguments.key?(variable) end request_opts[:defined_params] = defined_params unless defined_params.empty? raise ArgumentError, "Required argument 'task_id' missing" unless arguments[:task_id] arguments = arguments.clone headers = arguments.delete(:headers) || {} body = nil _task_id = arguments.delete(:task_id) method = Elasticsearch::API::HTTP_POST path = "_reindex/#{Utils.listify(_task_id)}/_rethrottle" params = Utils.process_params(arguments) Elasticsearch::API::Response.new( perform_request(method, path, params, body, headers, request_opts) ) end
Throttle a reindex operation. Change the number of requests per second for a particular reindex operation. For example: + POST _reindex/r1A2WoRbTwKZ516z6NEs5A:36619/_rethrottle?requests_per_second=-1 + Rethrottling that speeds up the query takes effect immediately. Rethrottling that slows down the query will take effect after completing the current batch. This behavior prevents scroll timeouts.
@option arguments [String] :task_id The task identifier, which can be found by using the tasks API
. (Required) @option arguments [Float] :requests_per_second The throttle for this request in sub-requests per second.
It can be either +-1+ to turn off throttling or any decimal number like +1.7+ or +12+ to throttle to that level.
@option arguments [Hash] :headers Custom HTTP headers
@see www.elastic.co/docs/api/doc/elasticsearch/v9/operation/operation-reindex
Source
# File lib/elasticsearch/api/actions/render_search_template.rb, line 34 def render_search_template(arguments = {}) request_opts = { endpoint: arguments[:endpoint] || 'render_search_template' } defined_params = [:id].each_with_object({}) do |variable, set_variables| set_variables[variable] = arguments[variable] if arguments.key?(variable) end request_opts[:defined_params] = defined_params unless defined_params.empty? arguments = arguments.clone headers = arguments.delete(:headers) || {} body = arguments.delete(:body) _id = arguments.delete(:id) method = if body Elasticsearch::API::HTTP_POST else Elasticsearch::API::HTTP_GET end path = if _id "_render/template/#{Utils.listify(_id)}" else '_render/template' end params = {} Elasticsearch::API::Response.new( perform_request(method, path, params, body, headers, request_opts) ) end
Render a search template. Render a search template as a search request body.
@option arguments [String] :id The ID of the search template to render.
If no +source+ is specified, this or the +id+ request body parameter is required.
@option arguments [Hash] :headers Custom HTTP headers @option arguments [Hash] :body request body
@see www.elastic.co/docs/api/doc/elasticsearch/v9/operation/operation-render-search-template
Source
# File lib/elasticsearch/api/actions/scripts_painless_execute.rb, line 40 def scripts_painless_execute(arguments = {}) request_opts = { endpoint: arguments[:endpoint] || 'scripts_painless_execute' } arguments = arguments.clone headers = arguments.delete(:headers) || {} body = arguments.delete(:body) method = if body Elasticsearch::API::HTTP_POST else Elasticsearch::API::HTTP_GET end path = '_scripts/painless/_execute' params = {} Elasticsearch::API::Response.new( perform_request(method, path, params, body, headers, request_opts) ) end
Run a script. Runs a script and returns a result. Use this API
to build and test scripts, such as when defining a script for a runtime field. This API
requires very few dependencies and is especially useful if you don’t have permissions to write documents on a cluster. The API
uses several contexts, which control how scripts are run, what variables are available at runtime, and what the return type is. Each context requires a script, but additional parameters depend on the context you’re using for that script. This functionality is Experimental and may be changed or removed completely in a future release. Elastic will take a best effort approach to fix any issues, but experimental features are not subject to the support SLA of official GA features.
@option arguments [Hash] :headers Custom HTTP headers @option arguments [Hash] :body request body
@see www.elastic.co/docs/reference/scripting-languages/painless/painless-api-examples
Source
# File lib/elasticsearch/api/actions/scroll.rb, line 48 def scroll(arguments = {}) request_opts = { endpoint: arguments[:endpoint] || 'scroll' } arguments = arguments.clone headers = arguments.delete(:headers) || {} body = arguments.delete(:body) _scroll_id = arguments.delete(:scroll_id) method = if body Elasticsearch::API::HTTP_POST else Elasticsearch::API::HTTP_GET end path = '_search/scroll' params = Utils.process_params(arguments) Elasticsearch::API::Response.new( perform_request(method, path, params, body, headers, request_opts) ) end
Run a scrolling search. IMPORTANT: The scroll API
is no longer recommend for deep pagination. If you need to preserve the index state while paging through more than 10,000 hits, use the search_after
parameter with a point in time (PIT). The scroll API
gets large sets of results from a single scrolling search request. To get the necessary scroll ID, submit a search API
request that includes an argument for the scroll
query parameter. The scroll
parameter indicates how long Elasticsearch
should retain the search context for the request. The search response returns a scroll ID in the _scroll_id
response body parameter. You can then use the scroll ID with the scroll API
to retrieve the next batch of results for the request. If the Elasticsearch
security features are enabled, the access to the results of a specific scroll ID is restricted to the user or API
key that submitted the search. You can also use the scroll API
to specify a new scroll parameter that extends or shortens the retention period for the search context. IMPORTANT: Results from a scrolling search reflect the state of the index at the time of the initial search request. Subsequent indexing or document changes only affect later search and scroll requests.
@option arguments [String] :scroll_id The scroll ID @option arguments [Time] :scroll The period to retain the search context for scrolling. Server default: 1d. @option arguments [Boolean] :rest_total_hits_as_int If true, the API
response’s hit.total property is returned as an integer. If false, the API
response’s hit.total property is returned as an object. @option arguments [Hash] :headers Custom HTTP headers @option arguments [Hash] :body request body
*Deprecation notice*: A scroll id can be quite large and should be specified as part of the body Deprecated since version 7.0.0
@see www.elastic.co/docs/api/doc/elasticsearch/v9/operation/operation-scroll
Source
# File lib/elasticsearch/api/actions/search.rb, line 155 def search(arguments = {}) request_opts = { endpoint: arguments[:endpoint] || 'search' } defined_params = [:index].each_with_object({}) do |variable, set_variables| set_variables[variable] = arguments[variable] if arguments.key?(variable) end request_opts[:defined_params] = defined_params unless defined_params.empty? arguments = arguments.clone headers = arguments.delete(:headers) || {} body = arguments.delete(:body) _index = arguments.delete(:index) method = if body Elasticsearch::API::HTTP_POST else Elasticsearch::API::HTTP_GET end path = if _index "#{Utils.listify(_index)}/_search" else '_search' end params = Utils.process_params(arguments) Elasticsearch::API::Response.new( perform_request(method, path, params, body, headers, request_opts) ) end
Run a search. Get search hits that match the query defined in the request. You can provide search queries using the q
query string parameter or the request body. If both are specified, only the query parameter is used. If the Elasticsearch
security features are enabled, you must have the read index privilege for the target data stream, index, or alias. For cross-cluster search, refer to the documentation about configuring CCS privileges. To search a point in time (PIT) for an alias, you must have the read
index privilege for the alias’s data streams or indices. **Search slicing** When paging through a large number of documents, it can be helpful to split the search into multiple slices to consume them independently with the slice
and pit
properties. By default the splitting is done first on the shards, then locally on each shard. The local splitting partitions the shard into contiguous ranges based on Lucene document IDs. For instance if the number of shards is equal to 2 and you request 4 slices, the slices 0 and 2 are assigned to the first shard and the slices 1 and 3 are assigned to the second shard. IMPORTANT: The same point-in-time ID should be used for all slices. If different PIT IDs are used, slices can overlap and miss documents. This situation can occur because the splitting criterion is based on Lucene document IDs, which are not stable across changes to the index.
@option arguments [String, Array] :index A comma-separated list of data streams, indices, and aliases to search.
It supports wildcards (+*+). To search all data streams and indices, omit this parameter or use +*+ or +_all+.
@option arguments [Boolean] :allow_no_indices If false
, the request returns an error if any wildcard expression, index alias, or _all
value targets only missing or closed indices.
This behavior applies even if the request targets other open indices. For example, a request targeting +foo*,bar*+ returns an error if an index starts with +foo+ but no index starts with +bar+. Server default: true.
@option arguments [Boolean] :allow_partial_search_results If true
and there are shard request timeouts or shard failures, the request returns partial results.
If +false+, it returns an error with no partial results.To override the default behavior, you can set the +search.default_allow_partial_results+ cluster setting to +false+. Server default: true.
@option arguments [String] :analyzer The analyzer to use for the query string.
This parameter can be used only when the +q+ query string parameter is specified.
@option arguments [Boolean] :analyze_wildcard If true
, wildcard and prefix queries are analyzed.
This parameter can be used only when the +q+ query string parameter is specified.
@option arguments [Integer] :batched_reduce_size The number of shard results that should be reduced at once on the coordinating node.
If the potential number of shards in the request can be large, this value should be used as a protection mechanism to reduce the memory overhead per search request. Server default: 512.
@option arguments [Boolean] :ccs_minimize_roundtrips If true
, network round-trips between the coordinating node and the remote clusters are minimized when running cross-cluster search (CCS) requests. Server default: true. @option arguments [String] :default_operator The default operator for the query string query: AND
or OR
.
This parameter can be used only when the +q+ query string parameter is specified. Server default: OR.
@option arguments [String] :df The field to use as a default when no field prefix is given in the query string.
This parameter can be used only when the +q+ query string parameter is specified.
@option arguments [String, Array<String>] :docvalue_fields A comma-separated list of fields to return as the docvalue representation of a field for each hit. @option arguments [String, Array<String>] :expand_wildcards The type of index that wildcard patterns can match.
If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams. It supports comma-separated values such as +open,hidden+. Server default: open.
@option arguments [Boolean] :explain If true
, the request returns detailed information about score computation as part of a hit. @option arguments [Boolean] :ignore_throttled If true
, concrete, expanded or aliased indices will be ignored when frozen. Server default: true. @option arguments [Boolean] :ignore_unavailable If false
, the request returns an error if it targets a missing or closed index. @option arguments [Boolean] :include_named_queries_score If true
, the response includes the score contribution from any named queries.This functionality reruns each named query on every hit in a search response.
Typically, this adds a small overhead to a request. However, using computationally expensive named queries on a large number of hits may add significant overhead.
@option arguments [Boolean] :lenient If true
, format-based query failures (such as providing text to a numeric field) in the query string will be ignored.
This parameter can be used only when the +q+ query string parameter is specified.
@option arguments [Integer] :max_concurrent_shard_requests The number of concurrent shard requests per node that the search runs concurrently.
This value should be used to limit the impact of the search on the cluster in order to limit the number of concurrent shard requests. Server default: 5.
@option arguments [String] :preference The nodes and shards used for the search.
By default, Elasticsearch selects from eligible nodes and shards using adaptive replica selection, accounting for allocation awareness. Valid values are: - +_only_local+ to run the search only on shards on the local node. - +_local+ to, if possible, run the search on shards on the local node, or if not, select shards using the default method. - +_only_nodes:<node-id>,<node-id>+ to run the search on only the specified nodes IDs. If suitable shards exist on more than one selected node, use shards on those nodes using the default method. If none of the specified nodes are available, select shards from any available node using the default method. - +_prefer_nodes:<node-id>,<node-id>+ to if possible, run the search on the specified nodes IDs. If not, select shards using the default method. - +_shards:<shard>,<shard>+ to run the search only on the specified shards. You can combine this value with other +preference+ values. However, the +_shards+ value must come first. For example: +_shards:2,3|_local+. - +<custom-string>+ (any string that does not start with +_+) to route searches with the same +<custom-string>+ to the same shards in the same order.
@option arguments [Integer] :pre_filter_shard_size A threshold that enforces a pre-filter roundtrip to prefilter search shards based on query rewriting if the number of shards the search request expands to exceeds the threshold.
This filter roundtrip can limit the number of shards significantly if for instance a shard can not match any documents based on its rewrite method (if date filters are mandatory to match but the shard bounds and the query are disjoint). When unspecified, the pre-filter phase is executed if any of these conditions is met: - The request targets more than 128 shards. - The request targets one or more read-only index. - The primary sort of the query targets an indexed field.
@option arguments [Boolean] :request_cache If true
, the caching of search results is enabled for requests where size
is 0
.
It defaults to index level settings.
@option arguments [String] :routing A custom value that is used to route operations to a specific shard. @option arguments [Time] :scroll The period to retain the search context for scrolling.
By default, this value cannot exceed +1d+ (24 hours). You can change this limit by using the +search.max_keep_alive+ cluster-level setting.
@option arguments [String] :search_type Indicates how distributed term frequencies are calculated for relevance scoring. @option arguments [Array<String>] :stats Specific tag
of the request for logging and statistical purposes. @option arguments [String, Array<String>] :stored_fields A comma-separated list of stored fields to return as part of a hit.
If no fields are specified, no stored fields are included in the response. If this field is specified, the +_source+ parameter defaults to +false+. You can pass +_source: true+ to return both source fields and stored fields in the search response.
@option arguments [String] :suggest_field The field to use for suggestions. @option arguments [String] :suggest_mode The suggest mode.
This parameter can be used only when the +suggest_field+ and +suggest_text+ query string parameters are specified. Server default: missing.
@option arguments [Integer] :suggest_size The number of suggestions to return.
This parameter can be used only when the +suggest_field+ and +suggest_text+ query string parameters are specified.
@option arguments [String] :suggest_text The source text for which the suggestions should be returned.
This parameter can be used only when the +suggest_field+ and +suggest_text+ query string parameters are specified.
@option arguments [Integer] :terminate_after The maximum number of documents to collect for each shard.
If a query reaches this limit, Elasticsearch terminates the query early. Elasticsearch collects documents before sorting.IMPORTANT: Use with caution. Elasticsearch applies this parameter to each shard handling the request. When possible, let Elasticsearch perform early termination automatically. Avoid specifying this parameter for requests that target data streams with backing indices across multiple data tiers. If set to +0+ (default), the query does not terminate early. Server default: 0.
@option arguments [Time] :timeout The period of time to wait for a response from each shard.
If no response is received before the timeout expires, the request fails and returns an error. It defaults to no timeout.
@option arguments [Boolean, Integer] :track_total_hits The number of hits matching the query to count accurately.
If +true+, the exact number of hits is returned at the cost of some performance. If +false+, the response does not include the total number of hits matching the query. Server default: 10000.
@option arguments [Boolean] :track_scores If true
, the request calculates and returns document scores, even if the scores are not used for sorting. @option arguments [Boolean] :typed_keys If true
, aggregation and suggester names are be prefixed by their respective types in the response. @option arguments [Boolean] :rest_total_hits_as_int Indicates whether hits.total
should be rendered as an integer or an object in the rest search response. @option arguments [Boolean] :version If true
, the request returns the document version as part of a hit. @option arguments [Boolean, String, Array<String>] :_source The source fields that are returned for matching documents.
These fields are returned in the +hits._source+ property of the search response. Valid values are: - +true+ to return the entire document source. - +false+ to not return the document source. - +<string>+ to return the source fields that are specified as a comma-separated list that supports wildcard (+*+) patterns. Server default: true.
@option arguments [String, Array<String>] :_source_excludes A comma-separated list of source fields to exclude from the response.
You can also use this parameter to exclude fields from the subset specified in +_source_includes+ query parameter. If the +_source+ parameter is +false+, this parameter is ignored.
@option arguments [String, Array<String>] :_source_includes A comma-separated list of source fields to include in the response.
If this parameter is specified, only these source fields are returned. You can exclude fields from this subset using the +_source_excludes+ query parameter. If the +_source+ parameter is +false+, this parameter is ignored.
@option arguments [Boolean] :seq_no_primary_term If true
, the request returns the sequence number and primary term of the last modification of each hit. @option arguments [String] :q A query in the Lucene query string syntax.
Query parameter searches do not support the full Elasticsearch Query DSL but are handy for testing.IMPORTANT: This parameter overrides the query parameter in the request body. If both parameters are specified, documents matching the query request body parameter are not returned.
@option arguments [Integer] :size The number of hits to return.
By default, you cannot page through more than 10,000 hits using the +from+ and +size+ parameters. To page through more hits, use the +search_after+ parameter. Server default: 10.
@option arguments [Integer] :from The starting document offset, which must be non-negative.
By default, you cannot page through more than 10,000 hits using the +from+ and +size+ parameters. To page through more hits, use the +search_after+ parameter. Server default: 0.
@option arguments [String] :sort A comma-separated list of +<field>:<direction>+ pairs. @option arguments [Boolean] :force_synthetic_source Should this request force synthetic _source?
Use this to test if the mapping supports synthetic _source and to get a sense of the worst case performance. Fetches with this enabled will be slower the enabling synthetic source natively in the index.
@option arguments [Hash] :headers Custom HTTP headers @option arguments [Hash] :body request body
@see www.elastic.co/docs/api/doc/elasticsearch/v9/operation/operation-search
Source
# File lib/elasticsearch/api/actions/search_mvt.rb, line 178 def search_mvt(arguments = {}) request_opts = { endpoint: arguments[:endpoint] || 'search_mvt' } defined_params = [:index, :field, :zoom, :x, :y].each_with_object({}) do |variable, set_variables| set_variables[variable] = arguments[variable] if arguments.key?(variable) end request_opts[:defined_params] = defined_params unless defined_params.empty? raise ArgumentError, "Required argument 'index' missing" unless arguments[:index] raise ArgumentError, "Required argument 'field' missing" unless arguments[:field] raise ArgumentError, "Required argument 'zoom' missing" unless arguments[:zoom] raise ArgumentError, "Required argument 'x' missing" unless arguments[:x] raise ArgumentError, "Required argument 'y' missing" unless arguments[:y] arguments = arguments.clone headers = arguments.delete(:headers) || {} body = arguments.delete(:body) _index = arguments.delete(:index) _field = arguments.delete(:field) _zoom = arguments.delete(:zoom) _x = arguments.delete(:x) _y = arguments.delete(:y) method = Elasticsearch::API::HTTP_POST path = "#{Utils.listify(_index)}/_mvt/#{Utils.listify(_field)}/#{Utils.listify(_zoom)}/#{Utils.listify(_x)}/#{Utils.listify(_y)}" params = Utils.process_params(arguments) Elasticsearch::API::Response.new( perform_request(method, path, params, body, headers, request_opts) ) end
Search a vector tile. Search a vector tile for geospatial values. Before using this API
, you should be familiar with the Mapbox vector tile specification. The API
returns results as a binary mapbox vector tile. Internally, Elasticsearch
translates a vector tile search API
request into a search containing:
-
A
geo_bounding_box
query on the +<field>+. The query uses the +<zoom>/<x>/<y>+ tile as a bounding box. -
A
geotile_grid
orgeohex_grid
aggregation on the +<field>+. Thegrid_agg
parameter determines the aggregation type. The aggregation uses the +<zoom>/<x>/<y>+ tile as a bounding box. -
Optionally, a
geo_bounds
aggregation on the +<field>+. The search only includes this aggregation if theexact_bounds
parameter istrue
. -
If the optional parameter
with_labels
istrue
, the internal search will include a dynamic runtime field that calls thegetLabelPosition
function of the geometry doc value. This enables the generation of new point features containing suggested geometry labels, so that, for example, multi-polygons will have only one label.
For example, Elasticsearch
may translate a vector tile search API
request with a grid_agg
argument of geotile
and an exact_bounds
argument of true
into the following search + GET my-index/_search {
"size": 10000, "query": { "geo_bounding_box": { "my-geo-field": { "top_left": { "lat": -40.979898069620134, "lon": -45 }, "bottom_right": { "lat": -66.51326044311186, "lon": 0 } } } }, "aggregations": { "grid": { "geotile_grid": { "field": "my-geo-field", "precision": 11, "size": 65536, "bounds": { "top_left": { "lat": -40.979898069620134, "lon": -45 }, "bottom_right": { "lat": -66.51326044311186, "lon": 0 } } } }, "bounds": { "geo_bounds": { "field": "my-geo-field", "wrap_longitude": false } } }
} + The API
returns results as a binary Mapbox vector tile. Mapbox vector tiles are encoded as Google Protobufs (PBF). By default, the tile contains three layers:
-
A
hits
layer containing a feature for each +<field>+ value matching thegeo_bounding_box
query. -
An
aggs
layer containing a feature for each cell of thegeotile_grid
orgeohex_grid
. The layer only contains features for cells with matching data. -
A meta layer containing:
-
A feature containing a bounding box. By default, this is the bounding box of the tile.
-
Value ranges for any sub-aggregations on the
geotile_grid
orgeohex_grid
. -
Metadata for the search.
-
The API
only returns features that can display at its zoom level. For example, if a polygon feature has no area at its zoom level, the API
omits it. The API
returns errors as UTF-8 encoded JSON. IMPORTANT: You can specify several options for this API
as either a query parameter or request body parameter. If you specify both parameters, the query parameter takes precedence. **Grid precision for geotile** For a grid_agg
of geotile
, you can use cells in the aggs
layer as tiles for lower zoom levels. grid_precision
represents the additional zoom levels available through these cells. The final precision is computed by as follows: +<zoom> + grid_precision+. For example, if +<zoom>+ is 7 and grid_precision
is 8, then the geotile_grid
aggregation will use a precision of 15. The maximum final precision is 29. The grid_precision
also determines the number of cells for the grid as follows: +(2^grid_precision) x (2^grid_precision)+. For example, a value of 8 divides the tile into a grid of 256 x 256 cells. The aggs
layer only contains features for cells with matching data. **Grid precision for geohex** For a grid_agg
of geohex
, Elasticsearch
uses +<zoom>+ and grid_precision
to calculate a final precision as follows: +<zoom> + grid_precision+. This precision determines the H3 resolution of the hexagonal cells produced by the geohex
aggregation. The following table maps the H3 resolution for each precision. For example, if +<zoom>+ is 3 and grid_precision
is 3, the precision is 6. At a precision of 6, hexagonal cells have an H3 resolution of 2. If +<zoom>+ is 3 and grid_precision
is 4, the precision is 7. At a precision of 7, hexagonal cells have an H3 resolution of 3. | Precision | Unique tile bins | H3 resolution | Unique hex bins | Ratio | | ——— | —————- | ————- | —————-| —– | | 1 | 4 | 0 | 122 | 30.5 | | 2 | 16 | 0 | 122 | 7.625 | | 3 | 64 | 1 | 842 | 13.15625 | | 4 | 256 | 1 | 842 | 3.2890625 | | 5 | 1024 | 2 | 5882 | 5.744140625 | | 6 | 4096 | 2 | 5882 | 1.436035156 | | 7 | 16384 | 3 | 41162 | 2.512329102 | | 8 | 65536 | 3 | 41162 | 0.6280822754 | | 9 | 262144 | 4 | 288122 | 1.099098206 | | 10 | 1048576 | 4 | 288122 | 0.2747745514 | | 11 | 4194304 | 5 | 2016842 | 0.4808526039 | | 12 | 16777216 | 6 | 14117882 | 0.8414913416 | | 13 | 67108864 | 6 | 14117882 | 0.2103728354 | | 14 | 268435456 | 7 | 98825162 | 0.3681524172 | | 15 | 1073741824 | 8 | 691776122 | 0.644266719 | | 16 | 4294967296 | 8 | 691776122 | 0.1610666797 | | 17 | 17179869184 | 9 | 4842432842 | 0.2818666889 | | 18 | 68719476736 | 10 | 33897029882 | 0.4932667053 | | 19 | 274877906944 | 11 | 237279209162 | 0.8632167343 | | 20 | 1099511627776 | 11 | 237279209162 | 0.2158041836 | | 21 | 4398046511104 | 12 | 1660954464122 | 0.3776573213 | | 22 | 17592186044416 | 13 | 11626681248842 | 0.6609003122 | | 23 | 70368744177664 | 13 | 11626681248842 | 0.165225078 | | 24 | 281474976710656 | 14 | 81386768741882 | 0.2891438866 | | 25 | 1125899906842620 | 15 | 569707381193162 | 0.5060018015 | | 26 | 4503599627370500 | 15 | 569707381193162 | 0.1265004504 | | 27 | 18014398509482000 | 15 | 569707381193162 | 0.03162511259 | | 28 | 72057594037927900 | 15 | 569707381193162 | 0.007906278149 | | 29 | 288230376151712000 | 15 | 569707381193162 | 0.001976569537 | Hexagonal cells don’t align perfectly on a vector tile. Some cells may intersect more than one vector tile. To compute the H3 resolution for each precision, Elasticsearch
compares the average density of hexagonal bins at each resolution with the average density of tile bins at each zoom level. Elasticsearch
uses the H3 resolution that is closest to the corresponding geotile density.
@option arguments [String, Array] :index Comma-separated list of data streams, indices, or aliases to search (Required) @option arguments [String] :field Field containing geospatial data to return (Required) @option arguments [Integer] :zoom Zoom level for the vector tile to search (Required) @option arguments [Integer] :x X coordinate for the vector tile to search (Required) @option arguments [Integer] :y Y coordinate for the vector tile to search (Required) @option arguments [Boolean] :exact_bounds If false
, the meta layer’s feature is the bounding box of the tile.
If true, the meta layer's feature is a bounding box resulting from a geo_bounds aggregation. The aggregation runs on <field> values that intersect the <zoom>/<x>/<y> tile with wrap_longitude set to false. The resulting bounding box may be larger than the vector tile.
@option arguments [Integer] :extent The size, in pixels, of a side of the tile. Vector tiles are square with equal sides. Server default: 4096. @option arguments [String] :grid_agg Aggregation used to create a grid for field
. @option arguments [Integer] :grid_precision Additional zoom levels available through the aggs layer. For example, if <zoom> is 7
and grid_precision is 8, you can zoom in up to level 15. Accepts 0-8. If 0, results don't include the aggs layer. Server default: 8.
@option arguments [String] :grid_type Determines the geometry type for features in the aggs layer. In the aggs layer,
each feature represents a geotile_grid cell. If 'grid' each feature is a Polygon of the cells bounding box. If 'point' each feature is a Point that is the centroid of the cell. Server default: grid.
@option arguments [Integer] :size Maximum number of features to return in the hits layer. Accepts 0-10000.
If 0, results don't include the hits layer. Server default: 10000.
@option arguments [Boolean] :with_labels If true
, the hits and aggs layers will contain additional point features representing
suggested label positions for the original features. - +Point+ and +MultiPoint+ features will have one of the points selected. - +Polygon+ and +MultiPolygon+ features will have a single point generated, either the centroid, if it is within the polygon, or another point within the polygon selected from the sorted triangle-tree. - +LineString+ features will likewise provide a roughly central point selected from the triangle-tree. - The aggregation results will provide one central point for each aggregation bucket. All attributes from the original features will also be copied to the new label features. In addition, the new features will be distinguishable using the tag +_mvt_label_position+.
@option arguments [Hash] :headers Custom HTTP headers @option arguments [Hash] :body request body
@see www.elastic.co/docs/api/doc/elasticsearch/v9/operation/operation-search-mvt
Source
# File lib/elasticsearch/api/actions/search_shards.rb, line 52 def search_shards(arguments = {}) request_opts = { endpoint: arguments[:endpoint] || 'search_shards' } defined_params = [:index].each_with_object({}) do |variable, set_variables| set_variables[variable] = arguments[variable] if arguments.key?(variable) end request_opts[:defined_params] = defined_params unless defined_params.empty? arguments = arguments.clone headers = arguments.delete(:headers) || {} body = nil _index = arguments.delete(:index) method = Elasticsearch::API::HTTP_GET path = if _index "#{Utils.listify(_index)}/_search_shards" else '_search_shards' end params = Utils.process_params(arguments) Elasticsearch::API::Response.new( perform_request(method, path, params, body, headers, request_opts) ) end
Get the search shards. Get the indices and shards that a search request would be run against. This information can be useful for working out issues or planning optimizations with routing and shard preferences. When filtered aliases are used, the filter is returned as part of the indices
section. If the Elasticsearch
security features are enabled, you must have the view_index_metadata
or manage
index privilege for the target data stream, index, or alias.
@option arguments [String, Array] :index A comma-separated list of data streams, indices, and aliases to search.
It supports wildcards (+*+). To search all data streams and indices, omit this parameter or use +*+ or +_all+.
@option arguments [Boolean] :allow_no_indices If false
, the request returns an error if any wildcard expression, index alias, or _all
value targets only missing or closed indices.
This behavior applies even if the request targets other open indices. For example, a request targeting +foo*,bar*+ returns an error if an index starts with +foo+ but no index starts with +bar+.
@option arguments [String, Array<String>] :expand_wildcards Type of index that wildcard patterns can match.
If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams. Supports comma-separated values, such as +open,hidden+. Valid values are: +all+, +open+, +closed+, +hidden+, +none+. Server default: open.
@option arguments [Boolean] :ignore_unavailable If false
, the request returns an error if it targets a missing or closed index. @option arguments [Boolean] :local If true
, the request retrieves information from the local node only. @option arguments [Time] :master_timeout The period to wait for a connection to the master node.
If the master node is not available before the timeout expires, the request fails and returns an error. IT can also be set to +-1+ to indicate that the request should never timeout. Server default: 30s.
@option arguments [String] :preference The node or shard the operation should be performed on.
It is random by default.
@option arguments [String] :routing A custom value used to route operations to a specific shard. @option arguments [Hash] :headers Custom HTTP headers
@see www.elastic.co/docs/api/doc/elasticsearch/v9/operation/operation-search-shards
Source
# File lib/elasticsearch/api/actions/search_template.rb, line 54 def search_template(arguments = {}) request_opts = { endpoint: arguments[:endpoint] || 'search_template' } defined_params = [:index].each_with_object({}) do |variable, set_variables| set_variables[variable] = arguments[variable] if arguments.key?(variable) end request_opts[:defined_params] = defined_params unless defined_params.empty? raise ArgumentError, "Required argument 'body' missing" unless arguments[:body] arguments = arguments.clone headers = arguments.delete(:headers) || {} body = arguments.delete(:body) _index = arguments.delete(:index) method = Elasticsearch::API::HTTP_POST path = if _index "#{Utils.listify(_index)}/_search/template" else '_search/template' end params = Utils.process_params(arguments) Elasticsearch::API::Response.new( perform_request(method, path, params, body, headers, request_opts) ) end
Run a search with a search template.
@option arguments [String, Array] :index A comma-separated list of data streams, indices, and aliases to search.
It supports wildcards (+*+).
@option arguments [Boolean] :allow_no_indices If false
, the request returns an error if any wildcard expression, index alias, or _all
value targets only missing or closed indices.
This behavior applies even if the request targets other open indices. For example, a request targeting +foo*,bar*+ returns an error if an index starts with +foo+ but no index starts with +bar+. Server default: true.
@option arguments [Boolean] :ccs_minimize_roundtrips If true
, network round-trips are minimized for cross-cluster search requests. @option arguments [String, Array<String>] :expand_wildcards The type of index that wildcard patterns can match.
If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams. Supports comma-separated values, such as +open,hidden+. Valid values are: +all+, +open+, +closed+, +hidden+, +none+.
@option arguments [Boolean] :explain If true
, the response includes additional details about score computation as part of a hit. @option arguments [Boolean] :ignore_throttled If true
, specified concrete, expanded, or aliased indices are not included in the response when throttled. Server default: true. @option arguments [Boolean] :ignore_unavailable If false
, the request returns an error if it targets a missing or closed index. @option arguments [String] :preference The node or shard the operation should be performed on.
It is random by default.
@option arguments [Boolean] :profile If true
, the query execution is profiled. @option arguments [String] :routing A custom value used to route operations to a specific shard. @option arguments [Time] :scroll Specifies how long a consistent view of the index
should be maintained for scrolled search.
@option arguments [String] :search_type The type of the search operation. @option arguments [Boolean] :rest_total_hits_as_int If true
, hits.total
is rendered as an integer in the response.
If +false+, it is rendered as an object.
@option arguments [Boolean] :typed_keys If true
, the response prefixes aggregation and suggester names with their respective types. @option arguments [Hash] :headers Custom HTTP headers @option arguments [Hash] :body request body
@see www.elastic.co/docs/api/doc/elasticsearch/v9/operation/operation-search-template
Source
# File lib/elasticsearch/api/actions/terms_enum.rb, line 36 def terms_enum(arguments = {}) request_opts = { endpoint: arguments[:endpoint] || 'terms_enum' } defined_params = [:index].each_with_object({}) do |variable, set_variables| set_variables[variable] = arguments[variable] if arguments.key?(variable) end request_opts[:defined_params] = defined_params unless defined_params.empty? raise ArgumentError, "Required argument 'index' missing" unless arguments[:index] arguments = arguments.clone headers = arguments.delete(:headers) || {} body = arguments.delete(:body) _index = arguments.delete(:index) method = if body Elasticsearch::API::HTTP_POST else Elasticsearch::API::HTTP_GET end path = "#{Utils.listify(_index)}/_terms_enum" params = {} Elasticsearch::API::Response.new( perform_request(method, path, params, body, headers, request_opts) ) end
Get terms in an index. Discover terms that match a partial string in an index. This API
is designed for low-latency look-ups used in auto-complete scenarios.
@option arguments [String] :index A comma-separated list of data streams, indices, and index aliases to search.
Wildcard (+*+) expressions are supported. To search all data streams or indices, omit this parameter or use +*+ or +_all+. (*Required*)
@option arguments [Hash] :headers Custom HTTP headers @option arguments [Hash] :body request body
@see www.elastic.co/docs/api/doc/elasticsearch/v9/operation/operation-terms-enum
Source
# File lib/elasticsearch/api/actions/termvectors.rb, line 111 def termvector(arguments = {}) warn '[DEPRECATION] `termvector` is deprecated. Please use the plural version, `termvectors` instead.' termvectors(arguments.merge(endpoint: '_termvector')) end
Deprecated: Use the plural version, {#termvectors}
Source
# File lib/elasticsearch/api/actions/termvectors.rb, line 71 def termvectors(arguments = {}) request_opts = { endpoint: arguments[:endpoint] || 'termvectors' } defined_params = [:index, :id].each_with_object({}) do |variable, set_variables| set_variables[variable] = arguments[variable] if arguments.key?(variable) end request_opts[:defined_params] = defined_params unless defined_params.empty? raise ArgumentError, "Required argument 'index' missing" unless arguments[:index] arguments = arguments.clone headers = arguments.delete(:headers) || {} body = arguments.delete(:body) _index = arguments.delete(:index) _id = arguments.delete(:id) method = if body Elasticsearch::API::HTTP_POST else Elasticsearch::API::HTTP_GET end arguments.delete(:endpoint) path = if _index && _id "#{Utils.listify(_index)}/_termvectors/#{Utils.listify(_id)}" else "#{Utils.listify(_index)}/_termvectors" end params = Utils.process_params(arguments) Elasticsearch::API::Response.new( perform_request(method, path, params, body, headers, request_opts) ) end
Get term vector information. Get information and statistics about terms in the fields of a particular document. You can retrieve term vectors for documents stored in the index or for artificial documents passed in the body of the request. You can specify the fields you are interested in through the fields
parameter or by adding the fields to the request body. For example: + GET /my-index-000001/_termvectors/1?fields=message + Fields can be specified using wildcards, similar to the multi match query. Term vectors are real-time by default, not near real-time. This can be changed by setting realtime
parameter to false
. You can request three types of values: _term information_, _term statistics_, and _field statistics_. By default, all term information and field statistics are returned for all fields but term statistics are excluded. **Term information**
-
term frequency in the field (always returned)
-
term positions (+positions: true+)
-
start and end offsets (+offsets: true+)
-
term payloads (+payloads: true+), as base64 encoded bytes
If the requested information wasn’t stored in the index, it will be computed on the fly if possible. Additionally, term vectors could be computed for documents not even existing in the index, but instead provided by the user.
@option arguments [String] :index The name of the index that contains the document. (Required) @option arguments [String] :id A unique identifier for the document. @option arguments [String, Array<String>] :fields A comma-separated list or wildcard expressions of fields to include in the statistics.
It is used as the default list unless a specific field list is provided in the +completion_fields+ or +fielddata_fields+ parameters.
@option arguments [Boolean] :field_statistics If true
, the response includes:
- The document count (how many documents contain this field). - The sum of document frequencies (the sum of document frequencies for all terms in this field). - The sum of total term frequencies (the sum of total term frequencies of each term in this field). Server default: true.
@option arguments [Boolean] :offsets If true
, the response includes term offsets. Server default: true. @option arguments [Boolean] :payloads If true
, the response includes term payloads. Server default: true. @option arguments [Boolean] :positions If true
, the response includes term positions. Server default: true. @option arguments [String] :preference The node or shard the operation should be performed on.
It is random by default.
@option arguments [Boolean] :realtime If true, the request is real-time as opposed to near-real-time. Server default: true. @option arguments [String] :routing A custom value that is used to route operations to a specific shard. @option arguments [Boolean] :term_statistics If true
, the response includes:
- The total term frequency (how often a term occurs in all documents). - The document frequency (the number of documents containing the current term). By default these values are not returned since term statistics can have a serious performance impact.
@option arguments [Integer] :version If true
, returns the document version as part of a hit. @option arguments [String] :version_type The version type. @option arguments [Hash] :headers Custom HTTP headers @option arguments [Hash] :body request body
@see www.elastic.co/docs/api/doc/elasticsearch/v9/operation/operation-termvectors
Source
# File lib/elasticsearch/api/actions/update.rb, line 66 def update(arguments = {}) request_opts = { endpoint: arguments[:endpoint] || 'update' } defined_params = [:index, :id].each_with_object({}) do |variable, set_variables| set_variables[variable] = arguments[variable] if arguments.key?(variable) end request_opts[:defined_params] = defined_params unless defined_params.empty? raise ArgumentError, "Required argument 'body' missing" unless arguments[:body] raise ArgumentError, "Required argument 'index' missing" unless arguments[:index] raise ArgumentError, "Required argument 'id' missing" unless arguments[:id] arguments = arguments.clone headers = arguments.delete(:headers) || {} body = arguments.delete(:body) _id = arguments.delete(:id) _index = arguments.delete(:index) method = Elasticsearch::API::HTTP_POST path = "#{Utils.listify(_index)}/_update/#{Utils.listify(_id)}" params = Utils.process_params(arguments) if Array(arguments[:ignore]).include?(404) Utils.rescue_from_not_found do Elasticsearch::API::Response.new( perform_request(method, path, params, body, headers, request_opts) ) end else Elasticsearch::API::Response.new( perform_request(method, path, params, body, headers, request_opts) ) end end
Update a document. Update a document by running a script or passing a partial document. If the Elasticsearch
security features are enabled, you must have the index
or write
index privilege for the target index or index alias. The script can update, delete, or skip modifying the document. The API
also supports passing a partial document, which is merged into the existing document. To fully replace an existing document, use the index API
. This operation:
-
Gets the document (collocated with the shard) from the index.
-
Runs the specified script.
-
Indexes the result.
The document must still be reindexed, but using this API
removes some network roundtrips and reduces chances of version conflicts between the GET and the index operation. The _source
field must be enabled to use this API
. In addition to _source
, you can access the following variables through the ctx
map: _index
, _type
, _id
, _version
, _routing
, and _now
(the current timestamp).
@option arguments [String] :id A unique identifier for the document to be updated. (Required) @option arguments [String] :index The name of the target index.
By default, the index is created automatically if it doesn't exist. (*Required*)
@option arguments [Integer] :if_primary_term Only perform the operation if the document has this primary term. @option arguments [Integer] :if_seq_no Only perform the operation if the document has this sequence number. @option arguments [Boolean] :include_source_on_error True or false if to include the document source in the error message in case of parsing errors. Server default: true. @option arguments [String] :lang The script language. Server default: painless. @option arguments [String] :refresh If ‘true’, Elasticsearch
refreshes the affected shards to make this operation visible to search.
If 'wait_for', it waits for a refresh to make this operation visible to search. If 'false', it does nothing with refreshes. Server default: false.
@option arguments [Boolean] :require_alias If true
, the destination must be an index alias. @option arguments [Integer] :retry_on_conflict The number of times the operation should be retried when a conflict occurs. Server default: 0. @option arguments [String] :routing A custom value used to route operations to a specific shard. @option arguments [Time] :timeout The period to wait for the following operations: dynamic mapping updates and waiting for active shards.
Elasticsearch waits for at least the timeout period before failing. The actual wait time could be longer, particularly when multiple waits occur. Server default: 1m.
@option arguments [Integer, String] :wait_for_active_shards The number of copies of each shard that must be active before proceeding with the operation.
Set to 'all' or any positive integer up to the total number of shards in the index (+number_of_replicas++1). The default value of +1+ means it waits for each primary shard to be active. Server default: 1.
@option arguments [Boolean, String, Array<String>] :_source If false
, source retrieval is turned off.
You can also specify a comma-separated list of the fields you want to retrieve. Server default: true.
@option arguments [String, Array<String>] :_source_excludes The source fields you want to exclude. @option arguments [String, Array<String>] :_source_includes The source fields you want to retrieve. @option arguments [Hash] :headers Custom HTTP headers @option arguments [Hash] :body request body
@see www.elastic.co/docs/api/doc/elasticsearch/v9/operation/operation-update
Source
# File lib/elasticsearch/api/actions/update_by_query.rb, line 155 def update_by_query(arguments = {}) request_opts = { endpoint: arguments[:endpoint] || 'update_by_query' } defined_params = [:index].each_with_object({}) do |variable, set_variables| set_variables[variable] = arguments[variable] if arguments.key?(variable) end request_opts[:defined_params] = defined_params unless defined_params.empty? raise ArgumentError, "Required argument 'index' missing" unless arguments[:index] arguments = arguments.clone headers = arguments.delete(:headers) || {} body = arguments.delete(:body) _index = arguments.delete(:index) method = Elasticsearch::API::HTTP_POST path = "#{Utils.listify(_index)}/_update_by_query" params = Utils.process_params(arguments) Elasticsearch::API::Response.new( perform_request(method, path, params, body, headers, request_opts) ) end
Update documents. Updates documents that match the specified query. If no query is specified, performs an update on every document in the data stream or index without modifying the source, which is useful for picking up mapping changes. If the Elasticsearch
security features are enabled, you must have the following index privileges for the target data stream, index, or alias:
-
read
-
index
orwrite
You can specify the query criteria in the request URI or the request body using the same syntax as the search API
. When you submit an update by query request, Elasticsearch
gets a snapshot of the data stream or index when it begins processing the request and updates matching documents using internal versioning. When the versions match, the document is updated and the version number is incremented. If a document changes between the time that the snapshot is taken and the update operation is processed, it results in a version conflict and the operation fails. You can opt to count version conflicts instead of halting and returning by setting conflicts
to proceed
. Note that if you opt to count version conflicts, the operation could attempt to update more documents from the source than max_docs
until it has successfully updated max_docs
documents or it has gone through every document in the source query. NOTE: Documents with a version equal to 0 cannot be updated using update by query because internal versioning does not support 0 as a valid version number. While processing an update by query request, Elasticsearch
performs multiple search requests sequentially to find all of the matching documents. A bulk update request is performed for each batch of matching documents. Any query or update failures cause the update by query request to fail and the failures are shown in the response. Any update requests that completed successfully still stick, they are not rolled back. **Throttling update requests** To control the rate at which update by query issues batches of update operations, you can set requests_per_second
to any positive decimal number. This pads each batch with a wait time to throttle the rate. Set requests_per_second
to -1
to turn off throttling. Throttling uses a wait time between batches so that the internal scroll requests can be given a timeout that takes the request padding into account. The padding time is the difference between the batch size divided by the requests_per_second
and the time spent writing. By default the batch size is 1000, so if requests_per_second
is set to 500
: + target_time = 1000 / 500 per second = 2 seconds wait_time = target_time - write_time = 2 seconds - .5 seconds = 1.5 seconds + Since the batch is issued as a single _bulk request, large batch sizes cause Elasticsearch
to create many requests and wait before starting the next set. This is “bursty” instead of “smooth”. Slicing Update by query supports sliced scroll to parallelize the update process. This can improve efficiency and provide a convenient way to break the request down into smaller parts. Setting slices
to auto
chooses a reasonable number for most data streams and indices. This setting will use one slice per shard, up to a certain limit. If there are multiple source data streams or indices, it will choose the number of slices based on the index or backing index with the smallest number of shards. Adding slices
to _update_by_query
just automates the manual process of creating sub-requests, which means it has some quirks:
-
You can see these requests in the tasks APIs. These sub-requests are “child” tasks of the task for the request with slices.
-
Fetching the status of the task for the request with
slices
only contains the status of completed slices. -
These sub-requests are individually addressable for things like cancellation and rethrottling.
-
Rethrottling the request with
slices
will rethrottle the unfinished sub-request proportionally. -
Canceling the request with slices will cancel each sub-request.
-
Due to the nature of slices each sub-request won’t get a perfectly even portion of the documents. All documents will be addressed, but some slices may be larger than others. Expect larger slices to have a more even distribution.
-
Parameters like
requests_per_second
andmax_docs
on a request with slices are distributed proportionally to each sub-request. Combine that with the point above about distribution being uneven and you should conclude that usingmax_docs
withslices
might not result in exactlymax_docs
documents being updated. -
Each sub-request gets a slightly different snapshot of the source data stream or index though these are all taken at approximately the same time.
If you’re slicing manually or otherwise tuning automatic slicing, keep in mind that:
-
Query performance is most efficient when the number of slices is equal to the number of shards in the index or backing index. If that number is large (for example, 500), choose a lower number as too many slices hurts performance. Setting slices higher than the number of shards generally does not improve efficiency and adds overhead.
-
Update performance scales linearly across available resources with the number of slices.
Whether query or update performance dominates the runtime depends on the documents being reindexed and cluster resources. **Update the document source** Update by query supports scripts to update the document source. As with the update API
, you can set ctx.op
to change the operation that is performed. Set +ctx.op = “noop”+ if your script decides that it doesn’t have to make any changes. The update by query operation skips updating the document and increments the noop
counter. Set +ctx.op = “delete”+ if your script decides that the document should be deleted. The update by query operation deletes the document and increments the deleted
counter. Update by query supports only index
, noop
, and delete
. Setting ctx.op
to anything else is an error. Setting any other field in ctx
is an error. This API
enables you to only modify the source of matching documents; you cannot move them.
@option arguments [String, Array] :index A comma-separated list of data streams, indices, and aliases to search.
It supports wildcards (+*+). To search all data streams or indices, omit this parameter or use +*+ or +_all+. (*Required*)
@option arguments [Boolean] :allow_no_indices If false
, the request returns an error if any wildcard expression, index alias, or _all
value targets only missing or closed indices.
This behavior applies even if the request targets other open indices. For example, a request targeting +foo*,bar*+ returns an error if an index starts with +foo+ but no index starts with +bar+. Server default: true.
@option arguments [String] :analyzer The analyzer to use for the query string.
This parameter can be used only when the +q+ query string parameter is specified.
@option arguments [Boolean] :analyze_wildcard If true
, wildcard and prefix queries are analyzed.
This parameter can be used only when the +q+ query string parameter is specified.
@option arguments [String] :conflicts The preferred behavior when update by query hits version conflicts: abort
or proceed
. Server default: abort. @option arguments [String] :default_operator The default operator for query string query: AND
or OR
.
This parameter can be used only when the +q+ query string parameter is specified. Server default: OR.
@option arguments [String] :df The field to use as default where no field prefix is given in the query string.
This parameter can be used only when the +q+ query string parameter is specified.
@option arguments [String, Array<String>] :expand_wildcards The type of index that wildcard patterns can match.
If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams. It supports comma-separated values, such as +open,hidden+. Valid values are: +all+, +open+, +closed+, +hidden+, +none+.
@option arguments [Integer] :from Skips the specified number of documents. Server default: 0. @option arguments [Boolean] :ignore_unavailable If false
, the request returns an error if it targets a missing or closed index. @option arguments [Boolean] :lenient If true
, format-based query failures (such as providing text to a numeric field) in the query string will be ignored.
This parameter can be used only when the +q+ query string parameter is specified.
@option arguments [Integer] :max_docs The maximum number of documents to process.
It defaults to all documents. When set to a value less then or equal to +scroll_size+ then a scroll will not be used to retrieve the results for the operation.
@option arguments [String] :pipeline The ID of the pipeline to use to preprocess incoming documents.
If the index has a default ingest pipeline specified, then setting the value to +_none+ disables the default ingest pipeline for this request. If a final pipeline is configured it will always run, regardless of the value of this parameter.
@option arguments [String] :preference The node or shard the operation should be performed on.
It is random by default.
@option arguments [String] :q A query in the Lucene query string syntax. @option arguments [Boolean] :refresh If true
, Elasticsearch
refreshes affected shards to make the operation visible to search after the request completes.
This is different than the update API's +refresh+ parameter, which causes just the shard that received the request to be refreshed.
@option arguments [Boolean] :request_cache If true
, the request cache is used for this request.
It defaults to the index-level setting.
@option arguments [Float] :requests_per_second The throttle for this request in sub-requests per second. Server default: -1. @option arguments [String] :routing A custom value used to route operations to a specific shard. @option arguments [Time] :scroll The period to retain the search context for scrolling. Server default: 5m. @option arguments [Integer] :scroll_size The size of the scroll request that powers the operation. Server default: 1000. @option arguments [Time] :search_timeout An explicit timeout for each search request.
By default, there is no timeout.
@option arguments [String] :search_type The type of the search operation. Available options include query_then_fetch
and dfs_query_then_fetch
. @option arguments [Integer, String] :slices The number of slices this task should be divided into. Server default: 1. @option arguments [Array<String>] :sort A comma-separated list of <field>:<direction> pairs. @option arguments [Array<String>] :stats The specific tag
of the request for logging and statistical purposes. @option arguments [Integer] :terminate_after The maximum number of documents to collect for each shard.
If a query reaches this limit, Elasticsearch terminates the query early. Elasticsearch collects documents before sorting.IMPORTANT: Use with caution. Elasticsearch applies this parameter to each shard handling the request. When possible, let Elasticsearch perform early termination automatically. Avoid specifying this parameter for requests that target data streams with backing indices across multiple data tiers.
@option arguments [Time] :timeout The period each update request waits for the following operations: dynamic mapping updates, waiting for active shards.
By default, it is one minute. This guarantees Elasticsearch waits for at least the timeout before failing. The actual wait time could be longer, particularly when multiple waits occur. Server default: 1m.
@option arguments [Boolean] :version If true
, returns the document version as part of a hit. @option arguments [Boolean] :version_type Should the document increment the version number (internal) on hit or not (reindex) @option arguments [Integer, String] :wait_for_active_shards The number of shard copies that must be active before proceeding with the operation.
Set to +all+ or any positive integer up to the total number of shards in the index (+number_of_replicas+1+). The +timeout+ parameter controls how long each write request waits for unavailable shards to become available. Both work exactly the way they work in the bulk API. Server default: 1.
@option arguments [Boolean] :wait_for_completion If true
, the request blocks until the operation is complete.
If +false+, Elasticsearch performs some preflight checks, launches the request, and returns a task ID that you can use to cancel or get the status of the task. Elasticsearch creates a record of this task as a document at +.tasks/task/${taskId}+. Server default: true.
@option arguments [Hash] :headers Custom HTTP headers @option arguments [Hash] :body request body
@see www.elastic.co/docs/api/doc/elasticsearch/v9/operation/operation-update-by-query
Source
# File lib/elasticsearch/api/actions/update_by_query_rethrottle.rb, line 35 def update_by_query_rethrottle(arguments = {}) request_opts = { endpoint: arguments[:endpoint] || 'update_by_query_rethrottle' } defined_params = [:task_id].each_with_object({}) do |variable, set_variables| set_variables[variable] = arguments[variable] if arguments.key?(variable) end request_opts[:defined_params] = defined_params unless defined_params.empty? raise ArgumentError, "Required argument 'task_id' missing" unless arguments[:task_id] arguments = arguments.clone headers = arguments.delete(:headers) || {} body = nil _task_id = arguments.delete(:task_id) method = Elasticsearch::API::HTTP_POST path = "_update_by_query/#{Utils.listify(_task_id)}/_rethrottle" params = Utils.process_params(arguments) Elasticsearch::API::Response.new( perform_request(method, path, params, body, headers, request_opts) ) end
Throttle an update by query operation. Change the number of requests per second for a particular update by query operation. Rethrottling that speeds up the query takes effect immediately but rethrotting that slows down the query takes effect after completing the current batch to prevent scroll timeouts.
@option arguments [String] :task_id The ID for the task. (Required) @option arguments [Float] :requests_per_second The throttle for this request in sub-requests per second.
To turn off throttling, set it to +-1+. Server default: -1.
@option arguments [Hash] :headers Custom HTTP headers
@see www.elastic.co/docs/api/doc/elasticsearch/v9/operation/operation-update-by-query-rethrottle