Apache Mahout Clustering Designs

By Gupta Ashish

This book is for developers who want to try out clustering on large datasets using Mahout. It will also be useful for those users who don't have a background in Mahout, but have knowledge of basic programming and are familiar with the basics of machine learning and clustering. It will be helpful if you know about clustering techniques for some other tool.

• Language: English
• Publisher: Packt Publishing
• Release date: Oct 8, 2015
• ISBN: 9781783284443

Book preview

Table of Contents

Apache Mahout Clustering Designs

Credits

About the Author

About the Reviewers

www.PacktPub.com

Support files, eBooks, discount offers, and more

Why subscribe?

Free access for Packt account holders

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Downloading the example code

Errata

Piracy

Questions

1. Understanding Clustering

The clustering concept

Application of clustering

Understanding distance measures

Understanding different clustering techniques

Hierarchical methods

The partitioning method

The density-based method

Probabilistic clustering

Algorithm support in Mahout

Clustering algorithms in Mahout

Installing Mahout

Building Mahout code using Maven

Setting up the development environment using Eclipse

Setting up Mahout for Windows users

Preparing data for use with clustering techniques

Summary

2. Understanding K-means Clustering

Learning K-means

Running K-means on Mahout

Dataset selection

Executing K-means

The clusterdump result

Visualizing clusters

Summary

3. Understanding Canopy Clustering

Running Canopy clustering on Mahout

The Canopy generation phase

The Canopy clustering phase

Running Canopy clustering

Using the Canopy output for K-means

Visualizing clusters

Working with CSV files

Summary

4. Understanding the Fuzzy K-means Algorithm Using Mahout

Learning Fuzzy K-means clustering

Running Fuzzy K-means on Mahout

Dataset

Creating a vector for the dataset

Vector reader

Visualizing clusters

Summary

5. Understanding Model-based Clustering

Learning model-based clustering

Understanding Dirichlet clustering

Topic modeling

Running LDA using Mahout

Dataset selection

Steps to execute CVB (LDA)

Summary

6. Understanding Streaming K-means

Learning Streaming K-means

The Streaming step

The BallKMeans step

Using Mahout for streaming K-means

Dataset selection

Converting CSV to a vector file

Running Streaming K-means

Summary

7. Spectral Clustering

Understanding spectral clustering

Affinity (similarity) graph

Getting graph Laplacian from the affinity matrix

Eigenvectors and eigenvalues

The spectral clustering algorithm

Normalized spectral clustering

Mahout implementation of spectral clustering

Summary

8. Improving Cluster Quality

Evaluating clusters

Extrinsic methods

Intrinsic methods

Using DistanceMeasure interface

Summary

9. Creating a Cluster Model for Production

Preparing the dataset

Launching the Mahout job on the cluster

Performance tuning for the job

Summary

Index

Apache Mahout Clustering Designs

---

Apache Mahout Clustering Designs

Copyright © 2015 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

First published: September 2015

Production reference: 1240915

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham B3 2PB, UK.

ISBN 978-1-78328-443-6

www.packtpub.com

Credits

Author

Ashish Gupta

Reviewers

Siva Prakash

Tharindu Rusira

Commissioning Editor

Akram Hussain

Acquisition Editors

Vivek Anantharaman

Divya Poojari

Content Development Editor

Susmita Sabat

Technical Editor

Namrata Patil

Copy Editor

Merilyn Pereira

Project Coordinator

Judie Jose

Proofreader

Safis Editng

Indexer

Rekha Nair

Graphics

Abhinash Sahu

Production Coordinator

Manu Joseph

Cover Work

Manu Joseph

About the Author

Ashish Gupta has been working in the field of software development for the last 10 years. He has worked in companies such as SAP Labs and Caterpillar as a software developer. While working for a start-up predicting potential customers for new fashion apparels using social media, he developed an interest in the field of machine learning. Since then, he has worked on big data technologies and machine learning for different industries, including retail, finance, insurance, and so on. He is passionate about learning new technologies and sharing that knowledge with others. He is the author of the book, Learning Apache Mahout Classification, Packt Publishing. He has organized many boot camps for Apache Mahout and the Hadoop ecosystem.

First of all, I would like to thank the open source communities for their continuous efforts in developing great software. I would also like to thank the reviewers of this book.

Nothing can be accomplished without the support of family, friends, and loved ones; I would like to thank them, especially my wife and son, for their continuous support while writing this book.

About the Reviewers

Siva Prakash has been working in the field of software development for the last 7 years. He is currently working in CISCO, Bangalore. He has extensive development experience in desktop, mobile, and web-based applications in ERP, telecom, and the digital media industry. He is passionate about learning new technologies and sharing knowledge with others. He has worked on big data technologies for the digital media industry. He loves trekking, traveling, music, reading books, and blogging.

He is available on LinkedIn at https://www.linkedin.com/in/techsivam.

Tharindu Rusira is currently working as a graduate research assistant at the School of Computing, University of Utah while pursuing his doctoral studies in computer science, specializing in compiler technology for performance optimization. He is also passionate about machine learning and its applications in a wide spectrum of real-world problems.

Tharindu is available on LinkedIn at https://www.linkedin.com/in/trusira.

www.PacktPub.com

Support files, eBooks, discount offers, and more

For support files and downloads related to your book, please visit www.PacktPub.com.

Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at for more details.

At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks.

https://www2.packtpub.com/books/subscription/packtlib

Do you need instant solutions to your IT questions? PacktLib is Packt's online digital book library. Here, you can search, access, and read Packt's entire library of books.

Why subscribe?

Fully searchable across every book published by Packt

Copy and paste, print, and bookmark content

On demand and accessible via a web browser

Free access for Packt account holders

If you have an account with Packt at www.PacktPub.com, you can use this to access PacktLib today and view 9 entirely free books. Simply use your login credentials for immediate access.

Preface

With the progress in hardware, our storage capacity has increased now, and because of this, there are many organizations that want to store all types of events for analytical purpose. This is giving birth to a new area of machine learning. The field of machine learning is very complex, and writing those algorithms is not a piece of cake. Apache Mahout provides us with readymade algorithms in the area of machine learning and saves us from the complex task