Learning ELK Stack

By Chhajed Saurabh

If you are a developer or DevOps engineer interested in building a system that provides amazing insights and business metrics out of data sources of various formats and types, using the open source technology stack that ELK provides, then this book is for you. Basic knowledge of Unix or any other programming language will be helpful to make the most out of this book.

• Language: English
• Publisher: Packt Publishing
• Release date: Nov 26, 2015
• ISBN: 9781785886706

Book preview

Table of Contents

Learning ELK Stack

Credits

About the Author

About the Reviewers

www.PacktPub.com

Support files, eBooks, discount offers, and more

Why subscribe?

Free access for Packt account holders

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Downloading the example code

Downloading the color images of this book

Errata

Piracy

Questions

1. Introduction to ELK Stack

The need for log analysis

Issue debugging

Performance analysis

Security analysis

Predictive analysis

Internet of things and logging

Challenges in log analysis

Non-consistent log format

Tomcat logs

Apache access logs – combined log format

IIS logs

Variety of time formats

Decentralized logs

Expert knowledge requirement

The ELK Stack

Elasticsearch

Logstash

Kibana

ELK data pipeline

ELK Stack installation

Installing Elasticsearch

Running Elasticsearch

Elasticsearch configuration

Network Address

Paths

The cluster name

The node name

Elasticsearch plugins

Installing Logstash

Running Logstash

Logstash with file input

Logstash with Elasticsearch output

Configuring Logstash

Installing Logstash forwarder

Logstash plugins

Input plugin

Filters plugin

Output plugin

Installing Kibana

Configuring Kibana

Running Kibana

Kibana interface

Discover

Visualize

Dashboard

Settings

Summary

2. Building Your First Data Pipeline with ELK

Input dataset

Data format for input dataset

Configuring Logstash input

Filtering and processing input

Putting data to Elasticsearch

Visualizing with Kibana

Running Kibana

Kibana visualizations

Building a line chart

Building a bar chart

Building a Metric

Building a data table

Summary

3. Collect, Parse and Transform Data with Logstash

Configuring Logstash

Logstash plugins

Listing all plugins in Logstash

Data types for plugin properties

Array

Boolean

Codec

Hash

String

Comments

Field references

Logstash conditionals

Types of Logstash plugins

Input plugins

file

Configuration options

add_field

codec

delimiter

exclude

path

sincedb_path

sincedb_write_interval

start_position

tags

type

stdin

Configuration options

add_field

codec

tags

type

twitter

Configuration options

add_field

codec

consumer_key

consumer_secret

full_tweet

keywords

oauth_token

oauth_token_secret

tags

type

lumberjack

Configuration options

add_field

codec

host

port

ssl_certificate

ssl_key

ssl_key_passphrase

tags

type

redis

Configuration options

add_field

codec

data_type

host

key

password

port

Output plugins

csv

Configuration options

codec

csv_options

fields

gzip

path

file

Configuration options

email

Configuration options

attachments

body

cc

from

to

htmlbody

replyto

subject

elasticsearch

Configuration options

ganglia

Configuration options

metric

unit

value

jira

Configuration options

kafka

Configuration options

topic_id

lumberjack

Configuration options

hosts

port

ssl_certificate

redis

Configuration options

rabbitmq

stdout

mongodb

Configuration options

collection

database

uri

Filter plugins

csv

Configuration options

date

Configuration options

drop

Configuration options

geoip

Configuration options

source

grok

Custom grok patterns

mutate

Configuration options

sleep

Codec plugins

json

line

multiline

plain

rubydebug

Summary

4. Creating Custom Logstash Plugins

Logstash plugin management

Plugin lifecycle management

Installing a plugin

Updating a plugin

Uninstalling a plugin

Structure of a Logstash plugin

Required dependencies

Class declaration

Configuration name

Configuration options setting

Plugin methods

Input plugin

Filter plugin

Output plugin

Codec plugin

Writing a Logstash filter plugin

Building the plugin

Summary

5. Why Do We Need Elasticsearch in ELK?

Why Elasticsearch?

Elasticsearch basic concepts

Index

Document

Field

Type

Mapping

Shard

Primary shard and replica shard

Cluster

Node

Exploring the Elasticsearch API

Listing all available indices

Listing all nodes in a cluster

Checking the health of the cluster

Health status of the cluster

Creating an index

Retrieving the document

Deleting documents

Deleting an index

Elasticsearch Query DSL

Elasticsearch plugins

Bigdesk plugin

Elastic-Hammer plugin

Head plugin

Summary

6. Finding Insights with Kibana

Kibana 4 features

Search highlights

Elasticsearch aggregations

Scripted fields

Dynamic dashboards

Kibana interface

Discover page

Time filter

Quick time filter

Relative time filter

Absolute time filter

Kibana Auto-refresh setting

Querying and searching data

Freetext search

AND

OR

NOT

Groupings

Wildcard searches

Field searches

Range searches

Special characters escaping

New search

Saving the search

Loading a search

Field searches using field list

Summary

7. Kibana – Visualization and Dashboard

Visualize page

Creating a visualization

Visualization types

Metrics and buckets aggregations

Buckets

Date Histogram

Histogram

Range

Date Range

Terms

Metrics

Count

Average, Sum, Min, and Max

Unique Count

Advanced options

Visualizations

Area chart

Data table

Line chart

Markdown widget

Metric

Pie chart

Tile map

Vertical bar chart

Dashboard page

Building a new dashboard

Saving and loading a dashboard

Sharing a dashboard

Summary

8. Putting It All Together

Input dataset

Configuring Logstash input

Grok pattern for access logs

Visualizing with Kibana

Running Kibana

Searching on the Discover page

Visualizations – charts

Building a Line chart

Building an Area chart

Building a Bar chart

Building a Markdown

Dashboard page

Summary

9. ELK Stack in Production

Prevention of data loss

Data protection

System scalability

Data retention

ELK Stack implementations

ELK Stack at LinkedIn

Problem statement

Criteria for solution

Solution

Kafka at LinkedIn

Operational challenges

Logging using Kafka at LinkedIn

ELK at SCA

How is ELK used in SCA?

How is it helping in analytics?

ELK for monitoring at SCA

ELK at Cliffhanger Solutions

Kibana demo – Packetbeat dashboard

Summary

10. Expanding Horizons with ELK

Elasticsearch plugins and utilities

Curator for index management

Curator commands

Curator installation

Shield for security

Shield installation

Adding users and roles

Using Kibana4 on shield protected Elasticsearch

Marvel to monitor

Marvel installation

Marvel dashboards

ELK roadmap

Elasticsearch roadmap

Logstash roadmap

Event persistence capability

End-to-end message acknowledgement

Logstash monitoring and management API

Kibana roadmap

Summary

Index

Learning ELK Stack

---

Learning ELK Stack

Copyright © 2015 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

First published: November 2015

Production reference: 1231115

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham B3 2PB, UK.

ISBN 978-1-78588-715-4

www.packtpub.com

Credits

Author

Saurabh Chhajed

Reviewers

Isra El Isa

Anthony Lapenna

Blake Praharaj

Commissioning Editor

Veena Pagare

Acquisition Editors

Reshma Raman

Purav Motiwalla

Content Development Editor

Rashmi Suvarna

Technical Editor

Siddhesh Ghadi

Copy Editor

Priyanka Ravi

Project Coordinator

Milton Dsouza

Proofreader

Safis Editing

Indexer

Mariammal Chettiyar

Graphics

Disha Haria

Production Coordinator

Nilesh R. Mohite

Cover Work

Nilesh R. Mohite

About the Author

Saurabh Chhajed is a technologist with vast professional experience in building Enterprise applications that span across product and service industries. He has experience building some of the largest recommender engines using big data analytics and machine learning, and also enjoys acting as an evangelist for big data and NoSQL technologies. With his rich technical experience, Saurabh has helped some of the largest financial and industrial companies in USA build their large product suites and distributed applications from scratch. He shares his personal experiences with technology at http://saurzcode.in.

Saurabh has also reviewed books by Packt Publishing, Apache Camel Essentials and Java EE 7 Development with NetBeans 8, in the past.

I would like to thank my family, Krati, who supported and encouraged me in spite of all the time it took away from them. I would also like to thank all the technical reviewers and content editors without whom this book wouldn't have been possible.

About the Reviewers

Isra El Isa obtained her BSc in computer science from the University of Jordan in January 2014. After graduation, she spent a year working as a software engineer at Seclytics Security Co., Santa Clara, California, where she got to work with various technologies. Isra is currently employed by iHorizons Co., Amman, Jordan, as a software developer.

Anthony Lapenna made a transition to working on the OPS side after having followed a career in software development and is currently a system engineer at WorkIT. He's a huge fan of the automation and DEVOPS culture. He also loves to track the latest technologies and to participate in the open source ecosystem by writing technical articles and sharing his software.

Blake Praharaj is a software engineer who specializes in navigating the hectic start-up environment. He is currently employed at Core Informatics, creating data management solutions for scientists in multiple industries that rely on laboratory testing and effective data interpretation. As with any good developer, he is constantly learning and exploring new technologies!

I would like to thank my significant other for her support and understanding with the time it took to work on this book. I would also like to thank the entire Core Informatics team for their support of the time it took to learn this technology, especially Vico.

www.PacktPub.com

Support files, eBooks, discount offers, and more

For support files and downloads related to your book, please visit www.PacktPub.com.

Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at for more details.

At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks.

https://www2.packtpub.com/books/subscription/packtlib

Do you need instant solutions to your IT questions? PacktLib is Packt's online digital book library. Here, you can search, access, and read Packt's entire library of books.

Why subscribe?

Fully searchable across every book published by Packt

Copy and paste, print, and bookmark content

On demand and accessible via a web browser

Free access for Packt account holders

If you have an account with Packt at www.PacktPub.com, you can use this to access PacktLib today and view 9 entirely free books. Simply use your login credentials for immediate access.

Preface

This book is aimed at introducing the building of your own ELK Stack data pipeline using the open source technologies stack of Elasticsearch, Logstash, and Kibana. This book is also aimed at covering the core concepts of each of the components of the stack and quickly using them to build your own log analytics solutions. The book is divided into ten chapters. The first chapter helps you install all the components of the stack so that you can quickly build your first data pipeline in the second chapter. Chapter 3 to Chapter 7 introduce you to the capabilities of each of the components of the stack in detail. The eighth chapter builds a full data pipeline using ELK. The ninth chapter introduces you to some of the use cases of the ELK Stack in practice. Finally, the tenth chapter helps you know about some of the tools that can work with ELK Stack to enhance its capabilities.

What this book covers

Chapter 1, Introduction to ELK Stack, introduces ELK Stack, and what problems it solves for you. It explains the