Elasticsearch 8 for Developers - 2nd Edition: A beginner's guide to indexing, analyzing, searching, and aggregating data (English Edition)

By Anurag Srivastava

Elasticsearch is a powerful tool for handling and managing large amount of data. It is scalable, reliable, and fast, with various features for data analysis and search.

This book is a comprehensive guide to using Elasticsearch to manage data. It starts with an overview of Elasticsearch, detailing its importance in today's world. The book further covers the basics of Elasticsearch, including installation, configuration, and index management. Next, the book covers more advanced topics, such as handling geospatial data and using aggregations to analyze data. It also covers performance optimization and administration. Throughout the book, the author provides practical examples to help you understand and apply the concepts learned.

By the end of this book, you will have a deep understanding of Elasticsearch and use it to manage and extract valuable insights from large amount of data.

• Language: English
• Publisher: BPB Online LLP
• Release date: Oct 30, 2023
• ISBN: 9789355516848

Book preview

C

HAPTER

1

Getting Started with Elasticsearch

Introduction

In this chapter, we will provide an in-depth introduction to Elasticsearch. We will start by discussing the benefits of using Elasticsearch and how it can help businesses achieve their data management goals. From there, we will delve into what Elasticsearch is and how it leverages the powerful search engine Lucene to provide fast and scalable search capabilities. To lay a solid foundation for understanding Elasticsearch, we will cover some basic concepts such as nodes, clusters, documents, indices, and shards. These concepts are essential for understanding how Elasticsearch stores and organizes data for efficient search and retrieval.

We will then explore some of the key use cases for Elasticsearch, including data search, logging and analysis, application and system performance monitoring, and data visualization. These use cases highlight the versatility of Elasticsearch and demonstrate its potential to provide insights and valuable information across a wide range of industries and applications. Additionally, we will discuss the various Elasticsearch clients available for developers, such as Java, PHP, Perl, Python, .NET, and JavaScript. These clients enable developers to leverage Elasticsearch’s search capabilities in their preferred programming language and ecosystem.

Finally, we will discuss how to use Elasticsearch as a primary data source, secondary data source, or as a stand-alone system. We will provide guidance on how to make informed decisions about incorporating Elasticsearch into your data architecture based on your specific business needs and technical requirements.

Structure

In this chapter, we will discuss the following topics:

Introduction to data search

What is Elasticsearch, and why is it important for search and analytics

Overview of Elasticsearch architecture and components

Applications and use cases for Elasticsearch

Different Elasticsearch clients and their usage scenarios

Objectives

This chapter provides an overview of Elasticsearch and its features. It starts by introducing the concept of search and analytics and why they are important in today’s data-driven world. It then goes on to explain how to utilize different Elasticsearch clients effectively.

Introduction to data search

In the modern world, the exponential growth of digitized data from various sources like smart devices, IoT sensors, and online transactions presents a significant challenge. One of the major challenges is converting unstructured data into a structured form to streamline the data storage process. However, the real challenge lies in searching the stored data for relevant information. Traditional data storage systems like RDBMS are not suitable for text search due to their complex SQL query writing process and search inefficiency, even after applying all required indexes. In contrast, Elasticsearch, a search engine built on top of Lucene, offers a sophisticated search mechanism with search relevancy, data aggregation, and many other benefits not available in RDBMS systems. Therefore, understanding the importance of search and how Elasticsearch can help streamline data storage and search is crucial for any organization dealing with large amounts of data.

Search is a critical component of modern-day applications as it enables users to quickly and accurately find the information they need. Whether a blog site, e-commerce platform, or any other application dealing with large volumes of data, a search mechanism is essential to provide users with relevant results. The importance of providing quick and accurate search results cannot be overstated, as users are more likely to abandon an application that does not meet their search expectations. Therefore, optimizing search performance is crucial to ensure a positive user experience and retain user engagement.

Apart from providing relevant and speedy search results, there are other critical aspects of search that need to be considered, such as search relevance, data aggregation, and analysis. These aspects can be effectively addressed by Elasticsearch, which is a powerful and scalable search engine capable of handling a variety of data types and sources. By leveraging Elasticsearch’s capabilities, applications can provide users with fast, accurate, and relevant search results, making it a critical tool for modern-day search-oriented applications.

The importance of search functionality cannot be overstated, as it enables users to quickly and accurately find the information or products they are seeking. In addition to providing a quick response time with relevant results, there are several other aspects of the search that must be considered, such as:

Search suggestion: An effective search system should suggest potential search terms as soon as a user starts typing, allowing for quick and efficient search queries.

Fuzzy data searching: The system should also be able to suggest relevant results even if the user misspells a search term or uses a synonym.

Derivative search: A high-quality search system should recognize derivatives of search terms, such as plural or singular versions, to provide the most comprehensive results.

Data aggregation: The system should support data aggregation to display additional options and filters to users, such as price range, ratings, brands, and other relevant information.

Relevant results: Search results should be displayed in order of relevance, taking into account factors such as search term frequency, recency, and user behavior.

Advanced filters:Users should be able to apply advanced filters to their search results, such as screen resolution, RAM capacity, color, and other relevant criteria.

Quick response time: A search system should provide search results quickly, within a matter of seconds, to ensure a smooth user experience and avoid user frustration.

By considering these aspects, developers can create effective search systems that provide quick and accurate results to users, ultimately leading to increased user engagement and satisfaction.

What is Elasticsearch, and why is it important for search and analytics

Elasticsearch was created by Shay Banon, the founder of Elastic, a company that develops and supports Elasticsearch. Elasticsearch is open-source software that can be run on a single server or distributed across hundreds of servers to handle petabytes of data without any issue. Elasticsearch is a powerful search engine that is used to search for relevant data from a large data store.

In the current information age, the amount of data is growing exponentially due to digitization and the emergence of new data sources like smart devices, IoT sensors, and online transactions. These data can be structured or unstructured, device-specific or time-series data, and come from different sources, which makes it difficult to search through them manually. To overcome these challenges, Elasticsearch provides a distributed, scalable, and document-oriented search engine that is built on top of the Lucene library. Lucene is a high-performance search engine library that provides fast and efficient search results. However, it requires complex Java code to use and is not easily distributable across multiple nodes.

Elasticsearch encapsulates the complexities of Lucene and provides REST APIs that allow users to interact with Elasticsearch in a more user-friendly way. Elasticsearch also provides support for multiple programming languages through language clients, so users can code in their preferred language and still interact with Elasticsearch. Additionally, Elasticsearch can be interacted with using the command-line tool cURL.

In summary, Elasticsearch is a powerful search and analytics engine that provides fast and efficient search capabilities on large volumes of data, making it a vital tool for organizations looking to derive insights and value from their data.

Overview of Elasticsearch architecture and components

Elasticsearch is designed with a distributed architecture that allows it to handle large amounts of data across multiple nodes. It is composed of several components that work together to provide a scalable and highly available search and analytics platform.

Node

In Elasticsearch, a node refers to a discrete running instance of the search engine. Elasticsearch is composed of one or more nodes, which are instances of the Elasticsearch server. For instance, in a cluster of 10 servers running Elasticsearch, each server would be considered a node. In some use cases, a single node cluster of Elasticsearch may suffice for non-production environments. However, as data size increases, the need for additional nodes arises to horizontally scale the cluster, which also provides fault tolerance. Through knowledge of other nodes within the cluster, a node can transfer client requests to the appropriate node. It is worth noting that nodes can take on various roles, including data nodes that store and execute queries, master nodes that manage cluster-wide operations, and coordinating nodes that forward requests to the appropriate nodes. Each node runs independently and communicates with other nodes to form a cluster. Nodes can be added or removed from a cluster dynamically without affecting the overall system. Nodes can be of different types:

Master-eligible node

In Elasticsearch 8, the master-eligible node is responsible for managing the cluster state, including adding or removing nodes, allocating shards to nodes, and maintaining the health of the cluster. It is recommended to have at least three master-eligible nodes in the cluster to ensure high availability and avoid split-brain situations*.

Dedicated master-eligible node

A dedicated master-eligible node is a node in an Elasticsearch cluster that is configured to be eligible for the role of a master node but is not tasked with any other responsibilities, such as storing data or processing search requests. The purpose of a dedicated master-eligible node is to improve the stability and reliability of the cluster by allowing it to elect a dedicated node to perform the tasks of a master node.

To configure a master-eligible node in Elasticsearch 8, you need to set the following options in the elasticsearch.yml configuration file:

  node.roles: [ master ]

The node.roles option should be set to master to indicate that this node is eligible to become the master node.

Voting-only master-eligible node

In Elasticsearch, a voting-only master-eligible node is a type of node that participates in the process of selecting a master node but cannot become a master node itself. When a master node fails or becomes unreachable, the remaining nodes in the cluster must elect a new master node to maintain cluster stability. During this process, nodes that have been configured as master-eligible participate in an election process to select a new master node. To configure a voting-only master-eligible node in Elasticsearch 8, you need to set the following options in the elasticsearch.yml configuration file:

  node.roles: [ data, master, voting_only ]

In above example, we are setting a data node with voting rights for master node. We can also set a dedicated master only voting node using the below option in the elasticsearch.yml configuration file:

  node.roles: [ master, voting_only ]

In the above example, we are setting a master eligible voting only note without any data node responsibilities.

Data node

Data nodes are responsible for storing and managing data, as well as performing CRUD operations, search, and aggregations on data. We can configure a node to be a data node by setting the node.data option to true in the Elasticsearch configuration file. If we want to create a dedicated data node, we can set other types to false in the configuration, as shown in the following code snippet:

  node.roles: [ data ]

In the example above, we are setting the "node.role" option to data and all other options to false, which makes the node a dedicated data node. By adding more data nodes to the cluster, we can horizontally scale the cluster and handle larger amounts of data. Data nodes can also perform shard allocation and rebalancing, which helps to distribute the data evenly across the cluster for better performance and fault tolerance.

Ingest node

The ingest node is a specialized node type in Elasticsearch that allows us to perform pre-processing on documents before they are indexed. It is responsible for processing data as it passes through the Elasticsearch pipeline, such as enriching documents with additional data, manipulating field values, and performing data transformations.

Ingest nodes have their own dedicated pipeline that can be customized with various processors such as grok, dissect, and geoip to extract and transform data. This is particularly useful in cases where we need to extract relevant information from unstructured data, such as log files or social media streams.

To configure a node as an ingest node, we can set the node.roles option to ingest in the Elasticsearch configuration file. Here is an example:

  node.roles: [ ingest ]

By enabling the ingest node, we can perform data pre-processing without having to write custom code or use external tools. This can simplify our data pipeline and make it more efficient, especially in cases where we need to process large amounts of data in real-time.

Machine learning node

Machine learning node is a node that has the ability to run machine learning jobs on data stored in the Elasticsearch cluster. Machine learning nodes have specialized hardware configurations and are optimized for processing large amounts of data in real-time.

To set up a machine learning node, we need to enable the machine learning feature in the Elasticsearch configuration file and assign the node the role of a machine learning node. We can do this by adding the following lines to the elasticsearch.yml configuration file:

  xpack.ml.enabled: true

The xpack.ml.enabled option enables the machine learning feature, while the node.ml option assigns the node the role of a machine learning node.

Once the node is configured as a machine learning node, we can create machine learning jobs using the Elasticsearch Machine Learning API. These jobs can analyze data in real-time and provide insights into patterns, anomalies, and trends.

To create a dedicated machine learning node, edit the Elasticsearch configuration file (elasticsearch.yml) and add the following line:

  node.roles: [ ml, remote_cluster_client]

This line tells Elasticsearch to configure this node as a machine learning node and a remote cluster client node. Remote cluster client setting is required for a ML node because it allows the machine learning node to access data from other clusters that may be necessary for analysis. The Machine Learning node is a paid Elasticsearch feature that allows you to run machine learning models on your Elasticsearch data.

For example, let us assume that you have a machine learning node in one Elasticsearch cluster, but you also have data stored in another cluster that you want to use for machine learning. By configuring the machine learning node as a remote cluster client node, you can access the data stored in the other cluster without needing to move it to the machine learning node’s cluster. This can save time and resources, and it can also help you avoid duplicating data unnecessarily.

Hot data node

Hot data nodes are a specific type of data nodes in Elasticsearch that are optimized for handling high-traffic, high-performance workloads. They are designed to hold the most frequently accessed and queried data, also known as hot data, and are typically deployed with high-performance hardware to ensure fast response times.

Hot data nodes are characterized by their ability to handle a high volume of read and write requests in real-time. They are optimized for efficient indexing and searching of data, which makes them ideal for use cases that require fast and frequent access to data, such as e-commerce, social media, and financial services.

To configure a hot data node, you can specify the following settings in the Elasticsearch configuration file:

  node.roles: [ data-hot ]

Using above setting we can define a hot data node in Elasticsearch.

Warm data node

A warm data node in Elasticsearch is a type of node that is optimized for storing and searching large amounts of less frequently accessed data. This can include older or less frequently accessed logs, historical data, or backups. A warm node typically has lower storage performance and lower memory requirements compared to hot nodes, but it can store a large amount of data at a lower cost.

Warm nodes are also designed to handle read-heavy workloads, and may not be as responsive to write requests as hot nodes. To configure a warm data node in Elasticsearch 8, you can set the following options in the elasticsearch.yml configuration file:

  node.roles: [ data-warm ]

The above setting will ensure that the node is configured as a warm data node and is optimized for storing and accessing less frequently accessed data.

Cold data node

A cold data node is a type of data node in Elasticsearch that is specifically designed for storing rarely accessed or archived data. Cold nodes are typically configured with slower storage and lower compute resources, making them cost-effective for storing large volumes of infrequently accessed data.

To set up a cold data node in Elasticsearch 8, you can do the following changes for node.roles setting in the elasticsearch.yml configuration file:

  node.roles: [ data-cold ]

The above setting will ensure that the node is configured as a cold data node and is optimized for storing and managing cold data, while other nodes in the cluster can handle more active and frequently accessed data.

Cold data nodes are beneficial for long-term data retention, compliance, and regulatory requirements. By separating cold data from hot or warm data, you can optimize resource allocation and performance within your Elasticsearch cluster. Cold data nodes are typically used for data that does not require frequent querying or real-time analysis but still needs to be stored and accessible for compliance or historical purposes.

Frozen data node

In Elasticsearch 8, a frozen data node is a specialized type of data node that is optimized for storing data that is rarely accessed or read-only. Frozen data nodes are designed to provide efficient and cost-effective storage for large volumes of data that are not actively queried or updated. Frozen data nodes are particularly useful for long-term archival and compliance purposes, where data needs to be retained for a specified period but is rarely accessed. By separating frozen data from other types of data, such as hot or warm data, you can optimize resource utilization and improve overall cluster performance. Frozen data nodes allow you to store and manage large amounts of data cost-effectively while still ensuring data availability and compliance with regulatory requirements.

To configure a frozen data node in Elasticsearch, you can do the following changes for node.roles setting in the elasticsearch.yml configuration file:

  node.roles: [ data-frozen ]

The above setting will ensure that the node is configured as a frozen data node and is optimized for storing and managing frozen data, providing optimized storage efficiency for this specific data type.

Cluster

In Elasticsearch, a cluster is a collection of nodes that collaborate to create a cohesive and distributed environment for storing and processing data. Each cluster is identified by a unique name, allowing nodes to join and communicate with the specific cluster they belong to. Nodes within a cluster work together to provide a unified and consistent view of the data stored across the entire cluster. They collaborate to ensure that data is evenly distributed and replicated across multiple nodes, which helps to improve data availability, fault tolerance, and overall system performance.

By distributing data across nodes, a cluster allows for horizontal scalability. As the amount of data increases or the workload grows, additional nodes can be added to the cluster to handle the increased storage and processing requirements. This scalable architecture enables Elasticsearch to handle large datasets and handle high query volumes effectively.

Clusters also play a crucial role in ensuring data reliability and fault tolerance. By replicating data across multiple nodes, the cluster can tolerate failures and prevent data loss. If a node becomes unavailable or fails, the cluster automatically redistributes the data it held to other available nodes, maintaining the desired data redundancy and ensuring that the data remains accessible even in the face of node failures.

In addition to data distribution and fault tolerance, clusters provide a centralized management point for monitoring and administration. Operations such as cluster health monitoring, index management, and data rebalancing can be performed at the cluster level, providing a unified and streamlined approach to managing the entire Elasticsearch deployment.

Index

In Elasticsearch, an index serves as a logical namespace or container that holds a collection of documents. Think of an index as a database in traditional database systems, where data is organized and stored in a structured manner. The purpose of an index is to group together documents that share similar characteristics or belong to the same data category. For example, in an e-commerce application, you might have separate indexes for products, customers, and orders. This allows you to perform efficient searches and retrieve relevant information within specific domains.

Elasticsearch utilizes an inverted index structure to enable fast full-text searches. The inverted index consists of a list of unique terms found across all documents in the index, along with the document IDs pointing to the occurrences of each term. This indexing technique greatly speeds up search operations by precomputing the term-document relationships.

To handle large amounts of data and distribute the workload, an index is divided into one or more shards. Each shard is an independent subset of the index’s data, and it can be stored on a separate node within the Elasticsearch cluster. By splitting the index into shards, Elasticsearch can parallelize search and indexing operations, improving both performance and scalability.

Shards

In Elasticsearch, an index is composed of one or more shards, and each shard is a self-contained unit of the index. By breaking an index into smaller shards, Elasticsearch can distribute the data and operations across multiple nodes, which can improve the performance and scalability of the system. Shards provide a way for Elasticsearch to parallelize search and indexing operations. When a search request is issued, the request is broadcast to all the shards in parallel, and the results are merged and returned to the user. This parallelization allows Elasticsearch to handle large volumes of data and complex search queries.

When creating an index, the number of shards can be specified, and Elasticsearch automatically distributes the shards across the available nodes in the cluster. The number of shards that an index should have depends on various factors, such as the size of the index, the number of documents, and the expected search and indexing performance. Elasticsearch also supports the ability to create replica shards, which are copies of the primary shards. Replica shards provide redundancy and high availability by allowing the system to continue to function even if some nodes fail. The number of replica shards can also be specified when creating an index. The