Applied Reinforcement Learning with Python: With OpenAI Gym, Tensorflow, and Keras

By Taweh Beysolow II

Delve into the world of reinforcement learning algorithms and apply them to different use-cases via Python. This book covers important topics such as policy gradients and Q learning, and utilizes frameworks such as Tensorflow, Keras, and OpenAI Gym.

Applied Reinforcement Learning with Python introduces you to the theory behind reinforcement learning (RL) algorithms and the code that will be used to implement them. You will take a guided tour through features of OpenAI Gym, from utilizing standard libraries to creating your own environments, then discover how to frame reinforcement learning problems so you can research, develop, and deploy RL-based solutions.

What You'll Learn

• Implement reinforcement learning with Python 
• Work with AI frameworks such as OpenAI Gym, Tensorflow, and Keras
• Deploy and train reinforcement learning–based solutions via cloud resources
• Apply practical applications of reinforcement learning

 

Who This Book Is For 

Data scientists, machine learning engineers and software engineers familiar with machine learning and deep learning concepts.

• Language: English
• Publisher: Apress
• Release date: Aug 23, 2019
• ISBN: 9781484251270

Book preview

Taweh Beysolow II

Applied Reinforcement Learning with Python

With OpenAI Gym, Tensorflow, and Keras

../images/480225_1_En_BookFrontmatter_Figa_HTML.png

Taweh Beysolow II

San Francisco, CA, USA

Any source code or other supplementary material referenced by the author in this book is available to readers on GitHub via the book’s product page, located at www.​apress.​com/​978-1-4842-5126-3 . For more detailed information, please visit http://​www.​apress.​com/​source-code .

ISBN 978-1-4842-5126-3e-ISBN 978-1-4842-5127-0

https://doi.org/10.1007/978-1-4842-5127-0

© Taweh Beysolow II 2019

Apress Standard

Trademarked names, logos, and images may appear in this book. Rather than use a trademark symbol with every occurrence of a trademarked name, logo, or image we use the names, logos, and images only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark. The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights.

While the advice and information in this book are believed to be true and accurate at the date of publication, neither the authors nor the editors nor the publisher can accept any legal responsibility for any errors or omissions that may be made. The publisher makes no warranty, express or implied, with respect to the material contained herein.

Distributed to the book trade worldwide by Springer Science+Business Media New York, 233 Spring Street, 6th Floor, New York, NY 10013. Phone 1-800-SPRINGER, fax [phone], e-mail [e-mail], or visit www.springeronline.com. Apress Media, LLC is a California LLC and the sole member (owner) is Springer Science + Business Media Finance Inc (SSBM Finance Inc). SSBM Finance Inc is a Delaware corporation.

This book is dedicated to my friends and family who supported me through the most difficult of times for the past decade. They have enabled me to be the person I am capable of being when operating at my best. Without you, I would not have the ability to continue living as happily as I am.

Introduction

It is a pleasure to return for a third title with Apress! This text will be the most complex of those I have written, but will be a worthwhile addition to every data scientist and engineer’s library. The field of reinforcement learning has undergone significant change in the past couple of years, and it is worthwhile for everyone excited with artificial intelligence to engross themselves in.

As the frontier of artificial intelligence research, this will be an excellent starting point to familiarize yourself with the status of the field as well as the most commonly used techniques. From this point, it is my hope that you will feel empowered to continue on your own research and innovate in your own respective fields.

Acknowledgments

I would like to thank Santanu, Divya, Celestin, and Rita. Without you, this book would not be nearly as much of a success as it will be. Secondarily, I would like to thank my family and friends for their continued encouragement and support. Life would not be worth living without them.

Table of Contents

Chapter 1:​ Introduction to Reinforcement Learning 1

History of Reinforcement Learning 2

MDPs and their Relation to Reinforcement Learning 3

Reinforcement Learning Algorithms and RL Frameworks 7

Q Learning 10

Actor-Critic Models 11

Applications of Reinforcement Learning 12

Classic Control Problems 12

Super Mario Bros.​ 13

Doom 14

Reinforcement-Based Marketing Making 15

Sonic the Hedgehog 16

Conclusion 17

Chapter 2:​ Reinforcement Learning Algorithms 19

OpenAI Gym 19

Policy-Based Learning 20

Policy Gradients Explained Mathematically 22

Gradient Ascent Applied to Policy Optimization 24

Using Vanilla Policy Gradients on the Cart Pole Problem 25

What Are Discounted Rewards and Why Do We Use Them?​ 29

Drawbacks to Policy Gradients 36

Proximal Policy Optimization (PPO) and Actor-Critic Models 37

Implementing PPO and Solving Super Mario Bros.​ 38

Overview of Super Mario Bros.​ 39

Installing Environment Package 40

Structure of the Code in Repository 40

Model Architecture 41

Working with a More Difficult Reinforcement Learning Challenge 47

Dockerizing Reinforcement Learning Experiments 50

Results of the Experiment 52

Conclusion 53

Chapter 3:​ Reinforcement Learning Algorithms:​ Q Learning and Its Variants 55

Q Learning 55

Temporal Difference (TD) Learning 57

Epsilon-Greedy Algorithm 59

Frozen Lake Solved with Q Learning 60

Deep Q Learning 65

Playing Doom with Deep Q Learning 66

Simple Doom Level 71

Training and Performance 73

Limitations of Deep Q Learning 74

Double Q Learning and Double Deep Q Networks 74

Conclusion 75

Chapter 4:​ Market Making via Reinforcement Learning 77

What Is Market Making?​ 77

Trading Gym 81

Why Reinforcement Learning for This Problem?​ 82

Synthesizing Order Book Data with Trading Gym 84

Generating Order Book Data with Trading Gym 85

Experimental Design 87

RL Approach 1:​ Policy Gradients 90

RL Approach 2:​ Deep Q Network 91

Results and Discussion 93

Conclusion 94

Chapter 5:​ Custom OpenAI Reinforcement Learning Environments 95

Overview of Sonic the Hedgehog 95

Downloading the Game 96

Writing the Code for the Environment 98

A3C Actor-Critic 103

Conclusion 111

Appendix A:​ Source Code 113

Market Making Model Utilities 113

Policy Gradient Utilities 115

Models 116

Chapter 1 125

OpenAI Example 125

Chapter 2 125

Cart Pole Example 125

Super Mario Example 130

Chapter 3 134

Frozen Lake Example 134

Doom Example 139

Chapter 4 146

Market Making Example 146

Chapter 5 158

Sonic Example 158

Index 165

About the Author and About the Technical Reviewer

About the Author

Taweh Beysolow II

../images/480225_1_En_BookFrontmatter_Figb_HTML.png

is a data scientist and author currently based in the United States. He has a bachelor of science in economics from St. Johns University and a master of science in applied statistics from Fordham University. After successfully exiting the start-up he co-founded, he now is a Director at Industry Capital, a San Francisco–based private equity firm, where he helps lead the cryptocurrency and blockchain platforms.

About the Technical Reviewer

Santanu Pattanayak

../images/480225_1_En_BookFrontmatter_Figc_HTML.jpg

currently works at GE Digital as a Staff Data Scientist and is author of the deep learning book Pro Deep Learning with TensorFlow (Apress, 2017). He has 8 years of experience in the data analytics/data science field and also has a background in development and database technologies. Prior to joining GE, Santanu worked in companies such as RBS, Capgemini, and IBM. He graduated with a degree in electrical engineering from Jadavpur University, Kolkata, and is an avid math enthusiast. Santanu is currently pursuing a master’s degree in data science from the Indian Institute of Technology (IIT), Hyderabad. He also devotes his time to data science hackathons and Kaggle competitions where he ranks within the top 500 across the globe. Santanu was born and brought up in West Bengal, India, and currently resides in Bangalore, India, with his wife.

© Taweh Beysolow II 2019

T. Beysolow IIApplied Reinforcement Learning with Pythonhttps://doi.org/10.1007/978-1-4842-5127-0_1

1. Introduction to Reinforcement Learning

Taweh Beysolow II¹ 

(1)

San Francisco, CA, USA

To those returning from my previous books, Introduction to Deep Learning Using R¹ and Applied Natural Learning Using Python,² it is a pleasure to have you as readers again. To those who are new, welcome! Over the past year, there have continued to be an increased proliferation and development of Deep Learning packages and techniques that revolutionize various industries. One of the most exciting portions of this field, without a doubt, is Reinforcement Learning (RL). This itself is often what underlies a lot of generalized AI applications, such as software that learns to play video games or play chess. The benefit to reinforcement learning is that the agent can familiarize itself with a large range of tasks assuming that the problems can be modeled to a framework containing actions, an environment, an agent(s). Assuming that, the range of problems can be from solving simple games, to more complex 3D games, to teaching self-driving cars how to pick up and drop off passengers in a variety of different places as well as teaching a robotic arm how to grasp objects and place