Practical Data Science Cookbook - Second Edition

By Tony Ojeda, Sean Patrick Murphy, Benjamin Bengfort and 2 more

About This Book

• Tackle every step in the data science pipeline and use it to acquire, clean, analyze, and visualize your data
• Get beyond the theory and implement real-world projects in data science using R and Python
• Easy-to-follow recipes will help you understand and implement the numerical computing concepts

Who This Book Is For

If you are an aspiring data scientist who wants to learn data science and numerical programming concepts through hands-on, real-world project examples, this is the book for you. Whether you are brand new to data science or you are a seasoned expert, you will benefit from learning about the structure of real-world data science projects and the programming examples in R and Python.

• Language: English
• Publisher: Packt Publishing
• Release date: Jun 29, 2017
• ISBN: 9781787123267

Tony Ojeda

Tony is the founder of District Data Labs and focuses on applied analytics for business strategy. He has published a book on practical data science, and has experience with hands-on education and data science curricula.

Book preview

Practical Data Science Cookbook

Second Edition

Practical recipes on data pre-processing, analysis and visualization using R and Python

Prabhanjan Tattar

Tony Ojeda

Sean Patrick Murphy

Benjamin Bengfort

Abhijit Dasgupta

BIRMINGHAM - MUMBAI

< html PUBLIC -//W3C//DTD HTML 4.0 Transitional//EN http://www.w3.org/TR/REC-html40/loose.dtd>

Practical Data Science Cookbook

Second Edition

Copyright © 2017 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

First published: September 2014

Second Edition: June 2017

Production reference: 1270617

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham

B3 2PB, UK.

ISBN 978-1-78712-962-7

www.packtpub.com

Credits

About the Authors

Prabhanjan Tattar has 9 years of experience as a statistical analyst. His main thurst has been to explain statistical and machine learning techniques through elegant programming which will clear the nuances of the underlying mathematics. Survival analysis and statistical inference are his main areas of research/interest, and he has published several research papers in peer-reviewed journals and also has authored two books on R: R Statistical Application Development by Example, Packt Publishing, and A Course in Statistics with R, Wiley. He also maintains the R packages gpk, RSADBE, and ACSWR.

I would like to thank the readers for their encouragement and feedback that lead to the improvements in this edition and hope that they find the current edition useful. Thanks are due to Tushar Gupta for introducing me to this project, Cheryl Dsa for bearing with the delays, Karan Thakkar for the eagle-eyed editing, and the entire Packt team for every little support. The authors of the first edition need to be thanked by me as their platform is largely carried forward. On the personal front, I continue to thank my family: Pranathi the kiddo, Chandrika the wifey, Lakshmi the goddess mother, and Narayanachar the beloved father.

Tony Ojeda is an accomplished data scientist and entrepreneur, with expertise in business process optimization and over a decade of experience creating and implementing innovative data products and solutions. He has a master's degree in finance from Florida International University and an MBA with a focus on strategy and entrepreneurship from DePaul University. He is the founder of District Data Labs, is a cofounder of Data Community DC, and is actively involved in promoting data science education through both organizations.

Sean Patrick Murphy spent 15 years as a senior scientist at The Johns Hopkins University, Applied Physics Laboratory, where he focused on machine learning, modeling and simulation, signal processing, and high performance computing in the Cloud. Now, he acts as an advisor and data consultant for companies in San Francisco, New York, and Washington DC. He completed graduation from The Johns Hopkins University and got his MBA from the University of Oxford. He currently co-organizes the Data Innovation DC meetup and co-founded the Data Science MD meetup. He is also a board member and co-founder of Data Community DC.

Benjamin Bengfort is an experienced data scientist and Python developer who has worked in the military, industry, and academia for the past 8 years. He is currently pursuing his PhD in Computer Science at the University of Maryland, College Park, doing research in Metacognition and Natural Language Processing. He holds a Master's degree in Computer Science from North Dakota State University, where he taught undergraduate Computer Science courses. He is also an adjunct faculty member at Georgetown University, where he teaches Data Science and Analytics. Benjamin has been involved in two data science start-ups in the DC region: leveraging large-scale machine learning and Big Data techniques across a variety of applications. He has a deep appreciation for the combination of models and data for entrepreneurial effect, and he is currently building one of these start-ups into a more mature organization.

Abhijit Dasgupta is a data consultant working in the greater DC-Maryland-Virginia area, with several years of experience in biomedical consulting, business analytics, bioinformatics, and bioengineering consulting. He has a PhD in biostatistics from the University of Washington and over 40 collaborative peer-reviewed manuscripts, with strong interests in bridging the statistics/machine-learning divide. He is always on the lookout for interesting and challenging projects, and is an enthusiastic speaker and discussant on new and better ways to look at and analyze data. He is a member of Data Community DC and a founding member and co-organizer of Statistical Programming DC (formerly R Users DC).

About the Reviewer

Alberto Boschetti is a data scientist, with strong expertise in signal processing and statistics. He holds a PhD in telecommunication engineering and currently lives and works in London. In his work projects he daily faces challenges spanning among natural language processing (NLP), machine learning, and distributed processing. He is very passionate about his job and he always tries to be updated on the latest development of data science technologies, attending meetups, conferences and other events. He is the author of Python Data Science Essentials, Regression Analysis with Python, and Large Scale Machine Learning with Python, all published by Packt.

I would like to thank my family, friends, and colleagues. Also, a big thanks to the open source community.

Abhinav Rai has been working as a Data Scientist for nearly a decade, currently working at Microsoft. He has experience working in telecom, retail marketing, and online advertisement. His areas of interest include the evolving techniques of Machine Learning and the associated technologies. He is especially more interested in analyzing large and humongous datasets and likes to generate deep insights in such scenarios. Academically holding a double master's degree in Mathematics from Deendayal Upadhyay Gorakhpur University with an NBHM scholarship and in Computer Science from Indian Statistical Institute, rigor and sophistication is a surety with his analytical deliveries.

www.PacktPub.com

For support files and downloads related to your book, please visit www.PacktPub.com. Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us [e-mail] for more details. At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks.

https://www.packtpub.com/mapt

Get the most in-demand software skills with Mapt. Mapt gives you full access to all Packt books and video courses, as well as industry-leading tools to help you plan your personal development and advance your career.

Why subscribe?

Fully searchable across every book published by Packt

Copy and paste, print, and bookmark content

On demand and accessible via a web browser

Customer Feedback

Thanks for purchasing this Packt book. At Packt, quality is at the heart of our editorial process. To help us improve, please leave us an honest review on this book's Amazon page at https://www.amazon.com/dp/1787129624.

If you'd like to join our team of regular reviewers, you can e-mail us at [e-mail]. We award our regular reviewers with free eBooks and videos in exchange for their valuable feedback. Help us be relentless in improving our products!

Table of Contents

Preface

What this book covers

What you need for this book

Who this book is for

Sections

Getting ready

How to do it…

How it works…

There's more…

See also

Conventions

Reader feedback

Customer support

Downloading the example code

Downloading the color images of this book

Errata

Piracy

Questions

Preparing Your Data Science Environment

Understanding the data science pipeline

How to do it...

How it works...

Installing R on Windows, Mac OS X, and Linux

How to do it...

How it works...

See also

Installing libraries in R and RStudio

Getting ready

How to do it...

How it works...

There's more...

See also

Installing Python on Linux and Mac OS X

Getting ready

How to do it...

How it works...

See also

Installing Python on Windows

How to do it...

How it works...

See also

Installing the Python data stack on Mac OS X and Linux

Getting ready

How to do it...

How it works...

There's more...

See also

Installing extra Python packages

Getting ready

How to do it...

How it works...

There's more...

See also

Installing and using virtualenv

Getting ready

How to do it...

How it works...

There's more...

See also

Driving Visual Analysis with Automobile Data with R

Introduction

Acquiring automobile fuel efficiency data

Getting ready

How to do it...

How it works...

Preparing R for your first project

Getting ready

How to do it...

There's more...

See also

Importing automobile fuel efficiency data into R

Getting ready

How to do it...

How it works...

There's more...

See also

Exploring and describing fuel efficiency data

Getting ready

How to do it...

How it works...

There's more...

Analyzing automobile fuel efficiency over time

Getting ready

How to do it...

How it works...

There's more...

See also

Investigating the makes and models of automobiles

Getting ready

How to do it...

How it works...

There's more...

See also

Creating Application-Oriented Analyses Using Tax Data and Python

Introduction

An introduction to application-oriented approaches

Preparing for the analysis of top incomes

Getting ready

How to do it...

How it works...

Importing and exploring the world's top incomes dataset

Getting ready

How to do it...

How it works...

There's more...

See also

Analyzing and visualizing the top income data of the US

Getting ready

How to do it...

How it works...

Furthering the analysis of the top income groups of the US

Getting ready

How to do it...

How it works...

Reporting with Jinja2

Getting ready

How to do it...

How it works...

There's more...

See also

Repeating the analysis in R

Getting ready

How to do it...

There's more...

Modeling Stock Market Data

Introduction

Requirements

Acquiring stock market data

How to do it...

Summarizing the data

Getting ready

How to do it...

How it works...

There's more...

Cleaning and exploring the data

Getting ready

How to do it...

How it works...

See also

Generating relative valuations

Getting ready

How to do

How it works...

Screening stocks and analyzing historical prices

Getting ready

How to do it...

How it works...

Visually Exploring Employment Data

Introduction

Preparing for analysis

Getting ready

How to do it...

How it works...

See also

Importing employment data into R

Getting ready

How to do it...

How it works...

There's more...

See also

Exploring the employment data

Getting ready

How to do it...

How it works...

See also

Obtaining and merging additional data

Getting ready

How to do it...

How it works...

Adding geographical information

Getting ready

How to do it...

How it works...

See also

Extracting state- and county-level wage and employment information

Getting ready

How to do it...

How it works...

See also

Visualizing geographical distributions of pay

Getting ready

How to do it...

How it works...

See also

Exploring where the jobs are, by industry

How to do it...

How it works...

There's more...

See also

Animating maps for a geospatial time series

Getting ready

How to do it...

How it works...

There is more...

Benchmarking performance for some common tasks

Getting ready

How to do it...

How it works...

There's more...

See also

Driving Visual Analyses with Automobile Data

Introduction

Getting started with IPython

Getting ready

How to do it...

How it works...

See also

Exploring Jupyter Notebook

Getting ready

How to do it...

How it works...

There's more...

See also

Preparing to analyze automobile fuel efficiencies

Getting ready

How to do it...

How it works...

There's more...

See also

Exploring and describing fuel efficiency data with Python

Getting ready

How to do it...

How it works...

There's more...

See also

Analyzing automobile fuel efficiency over time with Python

Getting ready

How to do it...

How it works...

There's more...

See also

Investigating the makes and models of automobiles with Python

Getting ready

How to do it...

How it works...

See also

Working with Social Graphs

Introduction

Understanding graphs and networks

Preparing to work with social networks in Python

Getting ready

How to do it...

How it works...

There's more...

Importing networks

Getting ready

How to do it...

How it works...

Exploring subgraphs within a heroic network

Getting ready

How to do it...

How it works...

There's more...

Finding strong ties

Getting ready

How to do it...

How it works...

There's more...

Finding key players

Getting ready

How to do it...

How it works...

There's more...

The betweenness centrality

The closeness centrality

The eigenvector centrality

Deciding on centrality algorithm

Exploring the characteristics of entire networks

Getting ready

How to do it...

How it works...

Clustering and community detection in social networks

Getting ready

How to do it...

How it works...

There's more...

Visualizing graphs

Getting ready

How to do it...

How it works...

Social networks in R

Getting ready

How to do it...

How it works...

Recommending Movies at Scale (Python)

Introduction

Modeling preference expressions

How to do it...

How it works...

Understanding the data

Getting ready

How to do it...

How it works...

There's more...

Ingesting the movie review data

Getting ready

How to do it...

How it works...

Finding the highest-scoring movies

Getting ready

How to do it...

How it works...

There's more...

See also

Improving the movie-rating system

Getting ready

How to do it...

How it works...

There's more...

See also

Measuring the distance between users in the preference space

Getting ready

How to do it...

How it works...

There's more...

See also

Computing the correlation between users

Getting ready

How to do it...

How it works...

There's more...

Finding the best critic for a user

Getting ready

How to do it...

How it works...

Predicting movie ratings for users

Getting ready

How to do it...

How it works...

Collaboratively filtering item by item

Getting ready

How to do it...

How it works...

Building a non-negative matrix factorization model

How to do it...

How it works...

See also

Loading the entire dataset into the memory

Getting ready

How to do it...

How it works...

There's more...

Dumping the SVD-based model to the disk

How to do it...

How it works...

Training the SVD-based model

How to do it...

How it works...

There's more...

Testing the SVD-based model

How to do it...

How it works...

There's more...

Harvesting and Geolocating Twitter Data (Python)

Introduction

Creating a Twitter application

Getting ready

How to do it...

How it works...

See also

Understanding the Twitter API v1.1

Getting ready

How to do it...

How it works...

There's more...

See also

Determining your Twitter followers and friends

Getting ready

How to do it...

How it works...

There's more...

See also

Pulling Twitter user profiles

Getting ready

How to do it...

How it works...

There's more...

See also

Making requests without running afoul of Twitter's rate limits

Getting ready

How to do it...

How it works...

Storing JSON data to disk

Getting ready

How to do it...

How it works...

Setting up MongoDB for storing Twitter data

Getting ready

How to do it...

How it works...

There's more...

See also

Storing user profiles in MongoDB using PyMongo

Getting ready

How to do it...

How it works...

Exploring the geographic information available in profiles

Getting ready

How to do it...

How it works...

There's more...

See also

Plotting geospatial data in Python

Getting ready

How to do it...

How it works...

There's more...

See also

Forecasting New Zealand Overseas Visitors

Introduction

The ts object

Getting ready

How to do it

How it works...

Visualizing time series data

Getting ready

How to do it...

How it works...

Simple linear regression models

Getting ready

How to do it...

How it works...

See also

ACF and PACF

Getting ready

How to do it...

How it works...

ARIMA models

Getting ready

How to do it...

How it works...

Accuracy measurements

Getting ready

How to do it...

How it works...

Fitting seasonal ARIMA models

Getting ready

How to do it...

How it works...

There's more...

German Credit Data Analysis

Introduction

Simple data transformations

Getting ready

How to do it...

How it works...

There's more...

Visualizing categorical data

Getting ready

How to do it...

How it works...

Discriminant analysis

Getting ready

How to do it...

How it works...

See also

Dividing the data and the ROC

Getting ready

How to do it...

Fitting the logistic regression model

Getting ready

How to do it...

How it works...

See also

Decision trees and rules

Getting ready

How to do it...

How it works...

See also

Decision tree for german data

Getting ready

How to do it ...

How it works...

Preface

Welcome to the second edition of Practical Data Science Cookbook. It was the positive feedback and usefulness that the book has found for its readers that made a second edition possible. When Packt asked me to co-author the second edition, I had a preview of some of its reviews across the web and immediately found the reasons for the popularity of the book and its little weakness. Thus, the current version retains the positives of the acceptance and removes the pain points as much as possible. The two new chapters: Chapter 10, German Credit Data Analysis and Chapter 11, Forecasting New Zealand Overseas Visitors are included to enhance the usefulness of the book.

We live in the age of data. As increasing amounts are generated each year, the need to analyze and create value from this asset is more important than ever. Companies that know what to do with their data and how to do it well will have a competitive advantage over companies that don't. Due to this, there will be an increasing demand for people who possess both the analytical and technical abilities to extract valuable insights from data and the business acumen to create valuable and pragmatic solutions that put these insights to use. This book provides multiple opportunities to learn how to create value from data through a variety of projects that run the spectrum of types of contemporary data science projects. Each chapter stands on its own, with step-by-step instructions that include screenshots, code snippets, and more detailed explanations where necessary and with a focus on process and practical application. The goal of this book is to introduce the data science pipeline, show you how it applies to a variety of different data science projects, and get you comfortable enough to apply it in future to projects of your own. Along the way, you'll learn different analytical and programming lessons, and the fact that you are working through an actual project while learning will help cement these concepts and facilitate your understanding of them.

What this book covers

Chapter 1, Preparing Your Data Science Environment, introduces the data science pipeline and helps you get your data science environment properly set up with instructions for the Mac, Windows, and Linux operating systems. This chapter is a guideline for setting up the environment for R and Python on the preceding platforms.

Chapter 2, Driving Visual Analysis with Automobile Data with R, takes you through the process of analyzing and visualizing automobile data to identify trends and patterns in fuel efficiency over time. The chapter will give you a taste of acquisition, exploration, munging, analysis, and communication. The concepts will be implemented in R.

Chapter 3, Creating Application-Oriented Analyses Using Tax Data and Python, shows you how to use Python to transition your analyses from one-off, custom efforts to reproducible and production-ready code using income distribution data as the base for the project.

Chapter 4, Modeling Stock Market Data, shows you how to build your own stock screener and use moving averages to analyze historical stock prices. You will learn how to acquire, summarize, clean, and generate relative evaluations of data.

Chapter 5, Visually Exploring Employment Data, shows you how to obtain employment and earnings data from the Bureau of Labor Statistics and conduct geospatial analysis at different levels with R. The same will be implemented using Python. The focus of this chapter is on the transformation, manipulation, and visualization of data.

Chapter 6, Driving Visual Analyses with Automobile Data, mirrors the automobile data analyses and visualizations in Chapter 2, Driving Visual Analysis with Automobile Data with R, but does so using the powerful programming language, Python. It focuses on the implementation of the analysis model using Python.

Chapter 7, Working with Social Graphs, shows you how to build, visualize, and analyze a social network that consists of comic book character relationships. You will also see the R and Python implementation.

Chapter 8, Recommending Movies at Scale (Python), walks you through building a movie recommender system with Python. You will also learn the R and Python code to implement a predictive model and the use of collaborative filtering to implement a predictive model.

Chapter 9, Harvesting and Geolocating Twitter Data (Python), shows you how to connect to the Twitter API and plot the geographic information contained in profiles. You will also learn the use of RESTful APIs in TextMining

Chapter 10, Forecasting New Zealand Overseas Visitors, explains how to create time series objects and describes various methods to visualize time series data. You will also learn how to build an appropriate model for the data and identify if the data has any trends and seasonal components.

Chapter 11, German Credit Data Analysis, demonstrates Exploratory Data Analysis (EDA), with a few basic tree methods and random forest. You will learn the method to apply EDA, tree-based methods and random forest on some particular data.

What you need for this book

For this book, you will need a computer with access to the Internet and the ability to install the open source software needed for the projects. The primary software we will be using consists of the R and Python programming languages, with a myriad of freely available packages and libraries. Installation instructions are in the first chapter.

Who this book is for

This book is intended for aspiring data scientists who want to learn data science and numerical programming concepts through hands-on, real-world projects. Whether you are brand new to data science or you are a seasoned expert, you will benefit from learning about the structure of real-world data science projects and the programming examples in R and Python.

Sections

In this book, you will find several headings that appear frequently (Getting ready, How to do it, How it works, There's more, and See also). To give clear instructions on how to complete a recipe, we use these sections as follows.

Getting ready

This section tells you what to expect in the recipe, and describes how to set up any software or any preliminary settings required for the recipe.

How to do it…

This section contains the steps required to follow the recipe.

How it works…

This section usually consists of a detailed explanation of what happened in the previous section.

There's more…

This section consists of additional information about the recipe in order to make the reader more knowledgeable about the recipe.

See also

This section provides helpful links to other useful information for the recipe.

Conventions

In this book, you will find a number of text styles that distinguish between different kinds of information. Here are some examples of these styles and an explanation of their meaning. Code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles are shown as follows: Create a new user for JIRA in the database and grant the user access to the jiradb database we just created using the following command:

A block of code is set as follows:

Any command-line input or output is written as follows:

New terms and important words are shown in bold. Words that you see on the screen, for example, in menus or dialog boxes, appear in the text like this: Select System info from the Administration panel.

Warnings or important notes appear in a box like this.

Tips and tricks appear like this.

Reader feedback

Feedback from our readers is always welcome. Let us know what you think about this book-what you liked or disliked. Reader feedback is important for us as it helps us develop titles that you will really get the most out of. To send us general feedback, simply e-mail [e-mail], and mention the book's title in the subject of your message. If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, see our author guide at www.packtpub.com/authors.

Customer support

Now that you are the proud owner of a Packt book, we have a number of things to help you to get the most from your purchase.

Downloading the example code

You can download the example code files for this book from your account at http://www.packtpub.com. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you. You can download the code files by following these steps:

Log in or register to our website using your e-mail address and password.

Hover the mouse pointer on the SUPPORT tab at the top.

Click on Code Downloads & Errata.

Enter the name of the book in the Search box.

Select the book for which you're looking to download the code files.

Choose from the drop-down menu where you purchased this book from.

Click on Code Download.

You can also download the code files by clicking on the Code Files button on the book's webpage at the Packt Publishing website. This page can be accessed by entering the book's name in the Search box. Please note that you need to be logged in to your Packt account. Once the file is downloaded, please make sure that you unzip or extract the folder using the latest version of:

WinRAR / 7-Zip for Windows

Zipeg / iZip / UnRarX for Mac

7-Zip / PeaZip for Linux

The code bundle for the book is also hosted on GitHub at https://github.com/PacktPublishing/Practical-Data-Science-Cookbook-Second-Edition. We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!

Downloading the color images of this book

We also provide you with a PDF file that has color images of the screenshots/diagrams used in this book. The color images will help you better understand the changes in the output. You can download this file from https://www.packtpub.com/sites/default/files/downloads/PracticalDataScienceCookbookSecondEditon_ColorImages.pdf.

Errata

Although we have taken every care to ensure the accuracy