Splunk Operational Intelligence Cookbook - Second Edition

By Josh Diakun, Paul R Johnson and Derek Mock

About This Book

• This is the most up-to-date book on Splunk 6.3 and teaches you how to tackle real-world operational intelligence scenarios efficiently
• Get business insights using machine data using this easy-to-follow guide
• Search, monitor, and analyze your operational data skillfully using this recipe-based, practical guide

Who This Book Is For

TThis book is intended for users of all levels who are looking to leverage the Splunk Enterprise platform as a valuable operational intelligence tool. The recipes provided in this book will appeal to individuals from all facets of business, IT, security, production, marketing, and many more!

• Language: English
• Publisher: Packt Publishing
• Release date: Jun 8, 2016
• ISBN: 9781785287497

Book preview

Table of Contents

Splunk Operational Intelligence Cookbook Second Edition

Credits

About the Authors

About the Reviewer

www.PacktPub.com

eBooks, discount offers, and more

Why subscribe?

Instant updates on new Packt books

Preface

What this book covers

What you need for this book

Who this book is for

Sections

Getting ready

How to do it…

How it works…

There's more…

See also

Conventions

Reader feedback

Customer support

Downloading the example code

Downloading the color images of this book

Errata

Piracy

Questions

1. Play Time – Getting Data In

Introduction

Indexing files and directories

Getting ready

How to do it…

How it works…

There's more…

Adding a file or directory data input via the CLI

Adding a file or directory input via inputs.conf

One-time indexing of data files via the Splunk CLI

Indexing the Windows event logs

See also

Getting data through network ports

Getting ready

How to do it…

How it works…

There's more…

Adding a network input via the CLI

Adding a network input via inputs.conf

See also

Using scripted inputs

Getting ready

How to do it…

How it works…

See also

Using modular inputs

Getting ready

How to do it…

How it works…

There's more…

See also

Using the Universal Forwarder to gather data

Getting ready

How to do it…

How it works…

There's more…

Add the receiving indexer via outputs.conf

Loading the sample data for this book

Getting ready

How to do it…

How it works…

See also

Defining field extractions

Getting ready

How to do it…

How it works…

See also

Defining event types and tags

Getting ready

How to do it…

How it works…

There's more…

Adding event types and tags via eventtypes.conf and tags.conf

See also

2. Diving into Data – Search and Report

Introduction

Making raw event data readable

Getting ready

How to do it…

How it works…

There's more…

Tabulating every field

Removing fields, then tabulating everything else

Finding the most accessed web pages

Getting ready

How to do it…

How it works…

There's more…

Searching for the top 10 accessed web pages

Searching for the most accessed pages by user

See also

Finding the most used web browsers

Getting ready

How to do it…

How it works…

There's more…

Searching for the web browser data for the most used OS types

See also

Identifying the top-referring websites

Getting ready

How to do it…

How it works…

There's more…

Searching for the top 10 using stats instead of top

See also

Charting web page response codes

Getting ready

How to do it…

How it works…

There's more…

Totaling success and error web page response codes

See also

Displaying web page response time statistics

Getting ready

How to do it…

How it works…

There's more…

Displaying web page response time by action

See also

Listing the top viewed products

Getting ready

How to do it…

How it works…

There's more…

Searching for the percentage of cart additions from product views

See also

Charting the application's functional performance

Getting ready

How to do it…

How it works…

There's more…

See also

Charting the application's memory usage

Getting ready

How to do it…

How it works…

See also

Counting the total number of database connections

Getting ready

How to do it…

How it works…

See also

3. Dashboards and Visualizations – Making Data Shine

Introduction

Creating an Operational Intelligence dashboard

Getting ready

How to do it…

How it works…

There's more…

Changing dashboard permissions

Using a pie chart to show the most accessed web pages

Getting ready

How to do it…

How it works…

There's more…

Searching for the top 10 accessed web pages

See also

Displaying the unique number of visitors

Getting ready

How to do it…

How it works…

There's more…

Coloring the value based on ranges

Adding trends and sparklines to the values

See also

Using a gauge to display the number of errors

Getting ready

How to do it…

How it works…

There's more…

See also

Charting the number of method requests by type and host

Getting ready

How to do it…

How it works…

See also

Creating a timechart of method requests, views, and response times

Getting ready

How to do it…

How it works…

There's more…

Method requests, views, and response times by host

See also

Using a scatter chart to identify discrete requests by size and response time

Getting ready

How to do it…

How it works…

There's more…

Using time series data points with a scatter chart

See also

Creating an area chart of the application's functional statistics

Getting ready

How to do it…

How it works…

See also

Using a bar chart to show the average amount spent by category

Getting ready

How to do it…

How it works…

See also

Creating a line chart of item views and purchases over time

Getting ready

How to do it…

How it works…

See also

4. Building an Operational Intelligence Application

Introduction

Creating an Operational Intelligence application

Getting ready

How to do it…

How it works…

There's more…

Creating an application from another application

Downloading and installing a Splunk app

See also

Adding dashboards and reports

Getting ready

How to do it…

How it works…

There's more…

Changing permissions of saved reports

See also

Organizing the dashboards more efficiently

Getting ready

How to do it…

How it works…

There's more…

Modifying the Simple XML directly

See also

Dynamically drilling down on activity reports

Getting ready

How to do it…

How it works…

There's more…

Disabling the drilldown feature in tables and charts

See also

Creating a form for searching web activity

Getting ready

How to do it…

How it works…

There's more…

Adding a Submit button to your form

See also

Linking web page activity reports to the form

Getting ready

How to do it…

How it works…

There's more…

Adding an overlay to the Sessions Over Time chart

See also

Displaying a geographical map of visitors

Getting ready

How to do it…

How it works…

There's more…

Adding a map panel using Simple XML

Mapping different distributions by area

See also

Scheduling PDF delivery of a dashboard

Getting ready

How to do it…

How it works…

See also

5. Extending Intelligence – Data Models and Pivoting

Introduction

Creating a data model for web access logs

Getting ready

How to do it…

How it works…

There's more…

Searching data models using the search interface

See also

Creating a data model for application logs

Getting ready

How to do it…

How it works…

See also

Accelerating data models

Getting ready

How to do it…

How it works…

There's more…

Viewing data model and acceleration summary information

Advanced configuration of data model acceleration

See also

Pivoting total sales transactions

Getting ready

How to do it…

How it works…

There's more…

Pivot searching using the pivot command and search interface

See also

Pivoting purchases by geographic location

Getting ready

How to do it…

How it works…

See also

Pivoting slowest responding web pages

Getting ready

How to do it…

How it works…

See also

Pivot charting top error codes

Getting ready

How to do it…

How it works…

See also

6. Diving Deeper – Advanced Searching

Introduction

Calculating the average session time on a website

Getting ready

How to do it…

How it works…

There's more…

Starts with a website visit, ends with a checkout

Defining maximum pause, span, and events in a transaction

See also

Calculating the average execution time for multi-tier web requests

Getting ready

How to do it…

How it works…

There's more…

Calculating the average execution time without using a join

See also

Displaying the maximum concurrent checkouts

Getting ready

How to do it…

How it works…

See also

Analyzing the relationship of web requests

Getting ready

How to do it…

How it works…

There's more…

Analyzing relationships of DB actions to memory utilization

See also

Predicting website traffic volumes

Getting ready

How to do it…

How it works…

There's more…

Predicting the total number of items purchased

Predicting the average response time of function calls

See also

Finding abnormally-sized web requests

Getting ready

How to do it…

How it works…

There's more…

The anomalies command

The anomalousvalues command

The anomalydetection command

The cluster command

See also

Identifying potential session spoofing

Getting ready

How to do it…

How it works…

There's more…

Creating logic for urgency

See also

7. Enriching Data – Lookups and Workflows

Introduction

Looking up product code descriptions

Getting ready

How to do it…

How it works…

There's more…

Manually adding the lookup to Splunk

See also

Flagging suspect IP addresses

Getting ready

How to do it…

How it works…

There's more…

Modifying an existing saved search to populate a lookup table

See also

Creating a session state table

Getting ready

How to do it…

How it works…

There's more…

Use the Splunk KV store to maintain the session state table

See also

Adding hostnames to IP addresses

Getting ready

How to do it…

How it works…

There's more…

Enabling automatic external field lookups

See also

Searching ARIN for a given IP address

Getting ready

How to do it…

How it works…

There's more…

Limiting workflow actions by event types

See also

Triggering a Google search for a given error

Getting ready

How to do it…

How it works…

There's more…

Triggering a Google search from the chart drilldown options

See also

Creating a ticket for application errors

Getting ready

How to do it…

How it works…

There's more…

Adding a workflow action manually in Splunk

See also

Looking up inventory from an external database

Getting ready

How to do it…

How it works…

There's more…

Use DB Connect for direct external DB lookups

See also

8. Being Proactive – Creating Alerts

Introduction

Alerting on abnormal web page response times

Getting ready

How to do it…

How it works…

There's more…

Viewing triggered alerts in Splunk's Alert manager

See also

Alerting on errors during checkout in real time

Getting ready

How to do it…

How it works…

There's more…

Building alerts via a configuration file

Editing alert configuration attributes using Advanced edit

Identify the real-time searches that are running

See also

Alerting on abnormal user behavior

Getting ready

How to do it…

How it works…

There's more…

Alerting on abnormal user purchases without checkouts

See also

Alerting on failure and triggering a scripted response

Getting ready

How to do it…

How it works…

There's more…

See also

Alerting when predicted sales exceed inventory

Getting ready

How to do it…

How it works…

There's more…

Adding an RSS feed notification action to an alert

See also

9. Speeding Up Intelligence – Data Summarization

Introduction

Calculating an hourly count of sessions versus completed transactions

Getting ready

How to do it…

How it works…

There's more…

Generating the summary more frequently

Avoiding summary index overlaps and gaps

See also

Backfilling the number of purchases by city

Getting ready

How to do it…

How it works…

There's more…

Backfilling a summary index from within a search directly

See also

Displaying the maximum number of concurrent sessions over time

Getting ready

How to do it…

How it works…

There's more…

Viewing the status of an accelerated report

See also

10. Above and Beyond – Customization, Web Framework, REST API, HTTP Event Collector, and SDKs

Introduction

Customizing the application navigation

Getting ready

How to do it...

How it works...

There's more…

Adding a force-directed graph of web hits

Getting ready

How to do it...

How it works...

There's more…

Changing the time range on the search manager

See also

Adding a calendar heatmap of product purchases

Getting ready

How to do it...

How it works...

See also

Adding cell highlighting of average product price

Getting ready

How to do it...

How it works...

There's more…

See also

Remotely querying Splunk's REST API for unique page views

Getting ready

How to do it...

How it works...

There's more…

Authenticating with a session token

See also

Creating a Python application to return unique IP addresses

Getting ready

How to do it...

How it works...

There's more...

Paginating the results of your search

See also

Creating a custom search command to format product names

Getting ready

How to do it...

How it works...

See also

Collecting data from remote scanning devices

Getting ready

How to do it...

How it works...

See also

Index

Splunk Operational Intelligence Cookbook Second Edition

---

Splunk Operational Intelligence Cookbook Second Edition

Copyright © 2016 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

First published: October 2014

Second edition: June 2016

Production reference: 1310516

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham B3 2PB, UK.

ISBN 978-1-78528-499-1

www.packtpub.com

Credits

Authors

Josh Diakun

Paul R Johnson

Derek Mock

Reviewer

Jose Hernandez

Commissioning Editor

Veena Pagare

Acquisition Editor

Vinay Argekar

Content Development Editor

Sumeet Sawant

Technical Editor

Mohita Vyas

Copy Editors

Vikrant Phadke

Alpha Singh

Project Coordinator

Shweta H. Birwatkar

Proofreader

Safis Editing

Indexer

Monica Ajmera Mehta

Production Coordinator

Conidon Miranda

Cover Work

Conidon Miranda

About the Authors

Josh Diakun is an IT operations and security specialist with a focus on creating data-driven operational processes. He has over 10 years of experience managing and architecting enterprise-grade IT environments. For the past 7 years, he has been architecting, deploying and developing on Splunk as the core platform for organizations to gain security and operational intelligence. Josh is a founding partner at Discovered Intelligence, a company specializing in data intelligence services and solutions. He is also a co-founder of the Splunk Toronto User Group.

I would first like to thank my co-authors, Derek Mock and Paul Johnson, for their support, endless effort, and those many late nights that led to this book becoming a reality. To my partner, Rachel, an endless thank you for being my biggest supporter and making sure I always remembered to take a break. To my mother, Denyce, and sister, Jessika, thank you for being the two most amazing people in my life and cheering me on as I wrote this book. Finally to my late father, John, who was always an inspiration and brought the best out of me. Without this, I would not be where I am today.

Paul R Johnson has over 10 years of data intelligence experience in the areas of information security, operations, and compliance. He is a partner at Discovered Intelligence, a company specializing in data intelligence services and solutions. Paul previously worked for a Fortune 10 company, leading IT risk intelligence initiatives and managing a global Splunk deployment. Paul co-founded the Splunk Toronto User Group and lives and works in Toronto, Canada.

I would like to thank my fellow authors, Josh Diakun and Derek Mock, for their support and collaborative efforts in writing this book. Thanks guys for giving up nights, days, and weekends to get it completed! I would also like to thank my wife, Stacey, for her continuous support, for keeping me focused, and for her great feedback and patience.

Derek Mock is a software developer and big data architect who specializes in IT operations, information security, and cloud technologies. He has 15 years' experience developing and operating large enterprise-grade deployments and SaaS applications. He is a founding partner at Discovered Intelligence, a company specializing in data intelligence services and solutions. For the past 6 years, he has been leveraging Splunk as the core tool to deliver key operational intelligence. Derek is based in Toronto, Canada, and is a co-founder of the Splunk Toronto User Group.

I could not have asked for better co-authors than Josh Diakun and Paul Johnson, whose tireless efforts over many late nights brought this book into being. I would also like to thank my mentor, Dave Penny, for all his support in my professional life. Finally, thanks to my partner Alison, and my children, Sarah and James, for cheering me on as I wrote and for always making sure I had enough coffee.

About the Reviewer

Jose Hernandez is currently the director of security solutions at Zenedge Inc. with a vast experience in security analytics. He started his professional career at Prolexic Technologies (now Akamai) in DDOS, fighting attacks from Anonymous and LulzSec against Fortune 100 companies. While working at Splunk Inc. as a security architect, he built and released an auto-mitigation framework that has been used to automatically fight attacks in large organizations. In the past, he has helped build security operation centers as well as run a public threat intelligence service. Jose is originally from Miami, Florida, where he completed his master's degree in information security from Nova Southeastern University. He also received two undergraduate bachelor degrees from Florida International University in the field of management of information systems and information technologies. Although security information has been the focus of his career, Jose has found that his true passion is in solving problems and creating solutions. As an example, he built an underwater remote-controlled vehicle called the SensorSub, which was used to test and measure the toxicity in Miami's waterways.

His e-mail is <[e-mail]>, Twitter handle is divious_1, and GitHub profile is divious1.

www.PacktPub.com

eBooks, discount offers, and more

Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at for more details.

At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks.

https://www2.packtpub.com/books/subscription/packtlib

Do you need instant solutions to your IT questions? PacktLib is Packt's online digital book library. Here, you can search, access, and read Packt's entire library of books.

Why subscribe?

Fully searchable across every book published by Packt

Copy and paste, print, and bookmark content

On demand and accessible via a web browser

Instant updates on new Packt books

Get notified! Find out when new books are published by following @PacktEnterprise on Twitter or the Packt Enterprise Facebook page.

Preface

Splunk makes it easy for you to take control of your data, and with Splunk Operational Cookbook, you can be confident that you are taking advantage of the Big Data revolution and driving your business with the cutting edge of operational intelligence and business analytics.

With more than 70 recipes that demonstrate all of Splunk's features, not only will you find quick solutions to common problems, but you'll also learn a wide range of strategies and uncover new ideas that will make you rethink what operational intelligence means to you and your organization.

What this book covers

Chapter 1, Play Time – Getting Data In, introduces you to the many ways in which you can get data into Splunk, whether it is collecting data locally from files and directories, receiving it through TCP/UDP port inputs, directly from a Universal Forwarder, or simply utilizing Scripted and Modular Inputs. Regardless of how Operational Intelligence is approached, the right data at the right time is pivotal to success; this chapter will play a key role in highlighting what data to consider and how to efficiently and effectively get that data into Splunk. It will also introduce the data sets that will be used throughout this book and where to obtain samples that can be used to follow each of the recipes as they are written.

Chapter 2, Diving into Data – Search and Report, introduces you to the first set of recipes in the book. Leveraging the data now available as a result of the previous chapter, the information and recipes will guide you through searching event data using Splunk's SPL (Search Processing Language); applying field extractions; grouping common events based on field values; and then building basic reports using the table, top, chart, and stats commands.

Chapter 3, Dashboards and Visualizations – Make Data Shine, guides you through building visualizations based on reports that can now be created as a result of this chapter. The information and recipes provided in this chapter will empower you to take their data, and reports, and bring it to life through the powerful visualizations provided by Splunk. Visualizations introduced will include single values, charts (bar, pie, line, and area), scatter charts, and gauges.

Chapter 4, Building an Operational Intelligence Application, builds on the understanding of visualizations that you gained as a result of the previous chapter to introduce the concept of dashboards. Dashboards provide a powerful way to bring visualizations together and provide the holistic visibility required to fully capture the operational intelligence that is most important. The information and recipes provided in this chapter will outline the purpose of dashboards, how to properly utilize dashboards, using the dashboard editor to build a dashboard, building a form for searching event data, and much more.

Chapter 5, Extending Intelligence – Data Models and Pivoting, covers one of the newest and most powerful features found in the latest release of Splunk Enterprise, the ability to create data models, and the introduction of the pivot tool. This chapter will take readers deeper into the data by introducing transactions, subsearching, concurrency, associations, and more advanced search commands. Through the information and recipes provided in this chapter readers will harness the ability to coverage data from different sources and understand or build relationships between the events.

Chapter 6, Diving Deeper – Advanced Searching, helps you harness the ability to coverage data from different sources and understand or build relationships between the events. By now you will have an understanding of how to derive operational intelligence from data by using some of Splunk's most common features. This chapter will introduce the concept of lookups and workflow actions for the purpose of augmenting the data being analyzed. The recipes provided will enable readers to apply this functionality to further enhance their understanding of the data being analyzed.

Chapter 7, Enriching Data – Lookups and Workflows, enables you to apply this functionality to further enhance their understanding of the data being analyzed. As illustrated in the preceding chapters, event data, whether from a single tier or multi-tier web application stack, can provide a wealth of operational intelligence and awareness. That intelligence can be further enriched through the use of lookups and workflow actions. This chapter will introduce readers to this concept, the benefits of proactive alerts and provide context of when alerts are best applied. The recipes provided will guide readers through creating alerts based on the knowledge gained from previous chapters.

Chapter 8, Being Proactive – Creating Alerts, guides you through creating alerts based on the knowledge gained from previous chapters. A key asset to complete operational intelligence and awareness is the ability to be proactive through scheduled or real-time alerts. This chapter will introduce readers to the concept of summary indexing for the purposes of accelerating reports and speeding up the time it takes to unlock business insight. The recipes in this chapter will provide readers with a short introduction to common situations where summary indexing can be leveraged to speed up reports or preserve focused statistics over long periods of time.

Chapter 9, Speed Up Intelligence – Data Summarization, provides you with a short introduction to common situations where summary indexing can be leveraged to speed up reports or preserve focused statistics over long periods of time. With big data being just that, big, it can sometimes be very time consuming searching massive sets of data and costly to store the data for long periods of time. This chapter will introduce readers to the concept of building data models and using the pivot tool to quickly design reports based on the data sets used within this book. With such powerful features now available, this chapter will enable readers to apply their extended knowledge of their data to empower non-traditional users to build intelligent operational reports through the use of data models and pivoting.

Chapter 10, Above and Beyond – Customization, Web Framework, REST API, HTTP Event Collector, and SDKs, introduces you to four very powerful features of Splunk, some of which are new to the latest release. These features provide the ability to create a very rich and powerful interactive experience with Splunk. This will open you up to the possibilities beyond core Splunk Enterprise and show you a method in order to create your own operational intelligence application that includes powerful D3 visualizations. It will also provide a recipe for querying Splunk's REST API and a basic Python application leveraging Splunk's SDK to execute a search.

What you need for this book

You'll need Splunk Enterprise 6.4 (or greater).

Who this book is for

This book is intended for users of all levels who are looking to leverage the Splunk Enterprise platform as a valuable operational intelligence tool. The recipes provided in this book will appeal to individuals from all facets of business, IT, security, product, marketing, and many more!

Also, existing users of Splunk who want to upgrade and get up and running with Splunk 6.4 will find this book invaluable.

Sections

In this book, you will find several headings that appear frequently (Getting ready, How to do it, How it works, There's more, and See also).

To give clear instructions on how to complete a recipe, we use these sections as follows:

Getting ready

This section tells you what to expect in the recipe, and describes how to set up any software or any preliminary settings required for the recipe.

How to do it…

This section contains the steps required to follow the recipe.

How it works…

This section usually consists of a detailed explanation of what happened in the previous section.

There's more…

This section consists of additional information about the recipe in order to make the reader more knowledgeable about the recipe.

See also

This section provides helpful links to other useful information for the recipe.

Conventions

In this book, you will find a number of text styles that distinguish between different kinds of information. Here are some examples of these styles and an explanation of their meaning.

Code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles are shown as follows: Alternatively, if we want to search only for Windows logon events, we will search for eventtype=windows_logon.

A block of code is set as follows:

[WinEventLog://Application]

disabled = 0

[WinEventLog://Security]

disabled = 0

[WinEventLog://System]

disabled = 0

When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold:

index=main sourcetype=access_combined | eval firsttime=_time | eval lasttime=_time | fields JSESSIONID firsttime lasttime | inputlookup session_state.csv append=true | stats last(firsttime) as firsttime, first(lasttime) as lasttime by JSESSIONID | outputlookup createinapp=true session_state.csv

Any command-line input or output is written as follows:

[monitor://c:\filelocation\cp01_messages.log] sourcetype = linux_messages

New terms and important words are shown in bold. Words that you see on the screen, for example, in menus or dialog boxes, appear in the text like this: Modular inputs are bundled as Splunk apps and, once installed, contain all the necessary configuration and code to display them in the Data inputs section of Splunk.

Note

Warnings or important notes appear in a box like this.

Tip

Tips and tricks appear like this.

Reader feedback

Feedback from our readers is always welcome. Let us know what you think about this book—what you liked or disliked. Reader feedback is important for us as it helps us develop titles that you will really get the most out of.

To send us general feedback, simply e-mail <[e-mail]>, and mention the book's title in the subject of your message.

If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, see our author guide at www.packtpub.com/authors.

Customer support

Now that you are the proud owner of