Learning Python Application Development

By Ninad Sathaye

About This Book

• Get to grips with Python techniques that address commonly encountered problems in general application development.
• Develop, package, and deploy efficient applications in a fun way.
• All-practical coverage of the major areas of application development, including best practices, exception handling, testing, refactoring, design patterns, performance, and GUI application development.

Who This Book Is For

Do you know the basics of Python and object-oriented programming? Do you want to go the extra mile and learn techniques to make your Python application robust, extensible, and efficient? Then this book is for you.

• Language: English
• Publisher: Packt Publishing
• Release date: Sep 7, 2016
• ISBN: 9781785885709

Book preview

Table of Contents

Learning Python Application Development

Credits

Disclaimers

About the Author

About the Reviewer

www.PacktPub.com

eBooks, discount offers, and more

Why subscribe?

Preface

Important things to note

Very important note for e-book readers

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Downloading the example code

Downloading the color images of this book

Errata

Piracy

Questions

1. Developing Simple Applications

Important housekeeping notes

Installation prerequisites

Installing Python

Option 1 – official distribution

Option 2 – bundled distribution

Python install location

Unix-like operating systems

Windows OS

Verifying Python installation

Installing pip

Installing IPython

Choosing an IDE

The theme of the book

Meet the characters

Simple script – Attack of the Orcs v0.0.1

The game – Attack of the Orcs v0.0.1

Problem statement

Pseudo code – version 0.0.1

Reviewing the code

Running Attack of the Orcs v0.0.1

Using functions – Attack of the Orcs v0.0.5

Revisiting the previous version

Pseudo code with attack feature – Version 0.0.5

Reviewing the code

Running Attack of the Orcs v0.0.5

Using OOP – Attack of the Orcs v1.0.0

Prioritize the feature requests

Problem statement

Redesigning the code

Painting the big picture

Pseudo UML representation

Understanding the pseudo UML diagram

Reviewing the code

Running Attack of the Orcs v1.0.0

Abstract base classes in Python

Exercise

Summary

Very important note for e-book readers

2. Dealing with Exceptions

Revisiting Attack of the Orcs v1.0.0

Debugging the problem

Fixing the bugs…

Exceptions

What is an exception?

Most common exceptions

Exception handling

Raising and re-raising an exception

The else block of try…except

finally...clean it up!

Back to the game – Attack of the Orcs v1.1.0

Preparatory work

Adding the exception handling code

Running Attack of the Orcs v1.1.0

Defining custom exceptions

Preparatory work

Custom exception – The problem

Writing a new exception class

Expanding the exception class

Inheriting from the exception class

Exercise

Summary

3. Modularize, Package, Deploy!

Selecting a versioning convention

Serial increments

Using a date format

Semantic versioning scheme

Modularizing the code

Attack of the Orcs v2.0.0

Creating a package

Importing from the package

Releasing the package on PyPI

Prepare the distribution

Step 1 – Setting up the package directory

Step 2 – Writing the setup.py file

Step 3 – Updating the README and LICENSE.txt files

Step 4 – Updating the MANIFEST.in file

Step 5 – Build a deployment-ready distribution

Uploading the distribution

Step 1 – Creating an account on PyPI test website

Step 2 – Creating a .pypirc file

Step 3 – Register your project

Step 4 – Uploading the package

A single command to do it all

Installing your own distribution

Using a private PyPI repository

Step 1 – Installing pypiserver

Step 2 – Building a new source distribution

Step 3 – Starting a local server

Step 4 – Installing the private distribution

Making an incremental release

Packaging and uploading the new version

Upgrading the installed version

Version controlling the code

Git resources

Installing Git

Configuring your identity

Basic Git terminology

Creating and using a Git repository

Creating a bare remote repository

Clone the repository

Copying the code to the cloned repository

Staging the code and committing

Pushing the changes to the central repository

Using GUI clients for Git

Exercise

Summary

4. Documentation and Best Practices

Documenting the code

Docstrings

Introduction to reStructuredText

Section headings

Paragraphs

Text styles

Code snippets

Mathematical equations

Bullets and numbering

Dosctrings using RST

Docstring formatting styles

Automatically creating docstring stubs

Generating documentation with Sphinx

Step 1 – Installing Sphinx using pip

Step 2 – cd to the source directory

Step 3 – Running sphinx-quickstart

Step 4 – Updating conf.py

Step 5 – Running sphinx-apidoc

Step 6 – Building the documentation

Using sphinx-build

Using Makefile

Python coding standards

Code analysis – How well are we doing?

Code analysis using IDE

Pylint

Pylint in action

PEP8 and AutoPEP8

Exercise

Summary

5. Unit Testing and Refactoring

This is how the chapter is organized

Important housekeeping notes

Why test?

A new feature was requested

You implemented this feature

But something wasn't right...

It required thorough testing

Unit testing

Python unittest framework

Basic terminology

Creating tests with unittest.TestCase

Controlling test execution

Using unittest.TestSuite

Writing unit tests for the application

Setting up a test package

Creating a new class for unit testing

First unit test – Injured unit selection

Running the first unit test

Second unit test – Acquiring the hut

Running only the second test

Creating individual test modules

Batch executing unit tests

Unit tests using mock library

Quick introduction to mock

Let's mock!

Using Mock objects in a unit test

Working with patches

Using patch in a unit test

Third unit test – The play method

Is your code covered?

Resolving import errors, if any

Other unit testing tools

Doctest

Nose

Pytest

Refactoring preamble

Take a detour – Refactor for testability

Refactoring

What is refactoring?

Why refactor?

When to refactor?

How to refactor?

Renaming

Extracting

Moving

Pushing down

Pulling up

Refactoring tools for Python

Unit testing revisited

Refactoring for testability

Fourth unit test – setup_game_scenario

Exercise

Refactoring and redesign exercise

Summary

6. Design Patterns

Introduction to design patterns

Classification of patterns

Behavioral patterns

Creational patterns

Structural patterns

Concurrency patterns

Python language and design patterns

First-class functions

Classes as first-class objects

Closures

Miscellaneous features

Class method

Abstract method

The __getattr__ method

Duck typing

Structure of the rest of the chapter

Fast forward – Attack of the Orcs v6.0.0

Strategy pattern

Strategy scenario – The jump feature

Strategy – The problem

Strategy – Attempted solution

Strategy – Rethinking the design

Strategy solution 1 – Traditional approach

Strategy solution 2 – Pythonic approach

Simple factory

Simple factory scenario – The recruit feature

Simple factory – The problem

Simple factory – Rethinking the design

Simple factory solution 1 – Traditional approach

Simple factory solution 2 – Pythonic approach

Abstract factory pattern

Abstract factory scenario – An accessory store

Abstract factory – The problem

Abstract factory – Rethinking the design

Simplifying the design further

Abstract factory solution – Pythonic approach

Advanced topic – enforcing an interface

Adapter pattern

Adapter scenario – Elf's distant cousin

Adapter – The problem

Adapter – Attempted solution

Adapter solution – Pythonic approach

Adapter – Multiple adapter methods

Summary

7. Performance – Identifying Bottlenecks

Overview of three performance chapters

More focus on the runtime performance

The first performance chapter

The second performance chapter

The third performance chapter

Sneak peek at the upcoming application speedup

Scenario – The Gold Hunt

High-level algorithm

Reviewing the initial code

Running the code

The problem

Identifying the bottlenecks

Measuring the execution time

Measuring the runtime of small code snippets

Code profiling

The cProfile module

The pstats module

The line_profiler package

Memory profiling

The memory_profiler package

Algorithm efficiency and complexity

Algorithm efficiency

Algorithm complexity

Big O notation

Big O complexity classes

O(1) – constant time

O(log n) – logarithmic

O(n) – Linear time

O(n log n) – Log linear

O(n2) – Quadratic

O(n3) – cubic

Upper bound of the complexity

Complexity for common data structures and algorithms

Wrapping up the big O discussion

Summary

8. Improving Performance – Part One

Prerequisite for the chapter

This is how the chapter is organized

Revisiting the Gold Hunt scenario

Selecting a problem size

Profiling the initial code

Optimizing Gold Hunt – Part one

Tweaking the algorithm – The square root

Gold Hunt optimization – Pass one

Skipping the dots

Gold Hunt optimization – Pass two

Using local scope

Gold Hunt optimization – Pass three

Performance improvement goodies

List comprehension

Recording execution time

Dictionary comprehension

Swapping conditional block and for loops

'try' it out in a loop

Choosing the right data structures

The collections module

The deque class

The defaultdict class

Generators and generator expressions

Generator expressions

Comparing the memory efficiency

Generator expressions or list comprehensions?

The itertools module

The itertools.chain iterator

Exercises

Summary

9. Improving Performance – Part Two, NumPy and Parallelization

Prerequisites for this chapter

This is how the chapter is organized

Introduction to NumPy

Installing NumPy

Creating array objects

Simple array operations

Array slicing and indexing

Indexing

Slicing

Broadcasting

Miscellaneous functions

numpy.ndarray.tolist

numpy.reshape

numpy.random

numpy.dstack

numpy.einsum

Computing distance square with einsum

Where to get more information on NumPy?

Optimizing Gold Hunt – Part two

Gold Hunt optimization – pass four

Gold Hunt optimization – pass five

Parallelization with the multiprocessing module

Introduction to parallelization

Shared memory parallelization

Distributed memory parallelization

Global interpreter lock

The multiprocessing module

The Pool class

Parallelizing the Gold Hunt program

Revisiting the gold field

Gold Hunt optimization – Pass six, parallelization

Other methods for parallelization

Further reading

JIT compilers

GPU accelerated computing

Summary

10. Simple GUI Applications

Overview of GUI frameworks

Tkinter

PyQt

PySide

Kivy

wxPython

GUI programming design considerations

Understanding user requirements

Developing a user story

Simplicity and accessibility

Consistency

Predictability and familiarity

Miscellaneous design considerations

Event-driven programming

Event

Event handling

Event loop

GUI programming with Tkinter

Tkinter documentation links

The mainloop() in Tkinter

Simple GUI application – Take 1

Simple GUI application – Take 2

GUI Widgets in Tkinter

Geometry management

Grid geometry manager

Pack geometry manager

Place geometry manager

Events in Tkinter

Event types

Event descriptors

Event object attributes

Event handling in Tkinter

Command callback (Button widget)

The bind() method

The bind_class() method

The bind_all() method

Project-1 – Attack of the Orcs V10.0.0

Background scenario

Problem statement

Writing the code

Overview of the class HutGame

The __init__ method

The occupy_huts method

The create_widgets method

The setup_layout method

The radio_btn_pressed and enter_hut methods

The announce_winner method

Running the application

MVC architecture

Model

View

Controller

Advantages of MVC

Project 2 – Attack of the Orcs v10.1.0

Revisiting the HutGame class

Creating MVC classes

Communication between MVC objects

Controller to Model or View communication

Model to Controller communication

Using method assignment

Using a publish-subscribe pattern

PyPubSub package

View to Controller communication

Communication between View and Model

Reviewing the code

The Controller class

The Model class

The View class

Running the application

Testing GUI applications

Testing considerations

Unit testing and MVC

Manual testing

Automated GUI testing

Exercises

Further reading

Summary

Index

Learning Python Application Development

---

Learning Python Application Development

Copyright © 2016 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

First published: August 2016

Production reference: 1290816

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham B3 2PB, UK.

ISBN 978-1-78588-919-6

www.packtpub.com

Credits

Author

Ninad Sathaye

Reviewer

Will Ware

Commissioning Editor

Priya Singh

Acquisition Editor

Kevin Colaco

Content Development Editor

Deepti Thore

Technical Editors

Kunal Chaudhari

Sunith Shetty

Copy Editor

Zainab Bootwala

Project Coordinator

Shweta H Birwatkar

Proofreader

Safis Editing

Indexer

Tejal Daruwale Soni

Graphics

Abhinash Sahu

Production Coordinator

Melwyn Dsa

Cover Work

Melwyn Dsa

Disclaimers

The names, characters, businesses, places, events, and incidents mentioned in this book are either the products of the author's imagination or used in a fictitious manner. Any resemblance to actual persons, living or dead, or actual events is purely coincidental.

The views and opinions expressed in this book are solely of the author and do not reflect those of author's employer or its clients.

For this book, the cartoons representing imaginary game characters, such as the Dwarf, Knight, Orc, Fairy, Elf, and so on, are created and copyrighted by Packt Publishing.

About the Author

Ninad Sathaye has spent several years of his professional career designing and developing performance-critical engineering applications written in a variety of languages, including Python and C++. He has worked as a software architect in the semiconductor industry, and more recently in the domain of Internet of Things. He holds a master's degree in mechanical engineering.

I would like to thank my wife, Arati, for her creative input on the book's game theme. This book wouldn't have been possible without her continued support. I would also like to express my sincere gratitude to Will Ware for technically reviewing this book. His feedback was valuable and really helped me take this book to the next level! Thank you Deepti, Kunal, Zainab, and the whole Packt Publishing team for your hard work and support. I owe a special thanks to Abhinash Sahu from Packt Publishing for creating the awesome graphics art for all the fictional characters in this book. My sincere thanks to Steve Furkay, Neeshma, and Kevin for their valuable feedback during the initial phase of this book. Finally, I would like to thank my whole family for their encouragement and support!

About the Reviewer

Will Ware is a software engineer in the Boston area. He has worked with embedded systems, mobile phones, and web development. He received degrees in electrical engineering and mathematics from M.I.T. His interests include STEM education and 3D printing.

www.PacktPub.com

eBooks, discount offers, and more

Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at for more details.

At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks.

https://www2.packtpub.com/books/subscription/packtlib

Do you need instant solutions to your IT questions? PacktLib is Packt's online digital book library. Here, you can search, access, and read Packt's entire library of books.

Why subscribe?

Fully searchable across every book published by Packt

Copy and paste, print, and bookmark content

On demand and accessible via a web browser

To my daughter, Anvita

Preface

Python is one of the most widely used dynamic programming languages. It supports a rich set of libraries and frameworks that enable rapid development. Such fast-paced development often comes with its own baggage that can bring down the overall quality, performance, and extensibility of the application. This book will help you push your Python skill level by teaching you how to build and deploy interesting applications.

Starting with a simple program, the book takes you all the way through designing and developing robust and efficient applications. It touches upon several important topics in an accessible and fun way.

A fantasy theme is used as a vehicle to explain various concepts. During the course of this book, you will meet many fictional game characters. While you learn different topics, these imaginary characters will talk to you, ask questions, and request new features.

Each chapter targets a different aspect of application development. A few initial ones focus on software robustness, packaging, and releasing the application code. The next few chapters are about improving the application's lifetime by making the code extensible, reusable, and readable. You will learn about refactoring, unit testing, design patterns, documentation, and best practices.

Techniques for identifying bottlenecks and improving performance are covered in a series of three chapters devoted to performance. The last chapter introduces you to GUI development.

Important things to note

The book uses a fun, text-based game theme as a vehicle to explain various application development aspects. However, the book itself is not about developing game applications!

Every chapter will have its own set of Python source files. Although we will talk through most of the code, you should keep the relevant files at hand. See the Downloading the example code section for more details.

The following is relevant if you are reading the electronic version of this book. Most of the code illustrated in this book is created as images. Try to use 100% zoom for a better reading experience as these code snapshots should appear crisp at zoom level.

The solutions to the exercises (if any) are generally not provided.

This book provides several external links (URLs) for further reading. Over time, some of these links might end up being broken. If that ever happens, try searching the web with the appropriate search terms.

Some experienced readers may find the code explanation a bit verbose. In this case, you can review the code provided in the supporting material for the book.

Very important note for e-book readers

The code illustrations that you see in this book are actually image files or code snapshots.

The rendering quality of these images will vary depending on your PDF reader's page display resolution and the zoom level.

If you have trouble clearly reading this code, you may try the following in your PDF or e-book reader:

Set the zoom level to 100%

Use the page display resolution of 96 pixels/inch or similar

If the problem still persists, you can try with a different resolution.

How do you set this resolution? It will depend on your e-book reader. For example, if you are using Adobe Reader, go to Edit | Preferences and then select Page Display from the left panel. You will see Resolution as an option in the right panel. Select 96 pixels/inch or similar and see if that helps render the images better.

What this book covers

Chapter 1, Developing Simple Applications, starts with installation prerequisites and the theme of the book. The first program is a fantasy text-based game presented as a script. An incremental version of this program with new features is then developed using functions. With more features added, the code becomes difficult to manage. To address this, the game application is redesigned using OOP concepts. This application now becomes the reference version for the next few chapters.

Chapter 2, Dealing with Exceptions, will teach you how to fix the obvious issues that the code written in the previous chapter has. You will learn how to add exception handling code to make the application robust. You will also learn about the try…except…finally clause, raising and re-raising exceptions, creating and using custom exception classes, and so on.

Chapter 3, Modularize, Package, Deploy!, will teach you how to modularize and package the code written in the earlier chapters. After preparing a package, it will show you how to deploy a source distribution, make incremental releases, set up a private Python package repository, and bring the code under version control.

Chapter 4, Documentation and Best Practices, dives into coding standards, which are a set of guidelines that you should follow while developing the code. Complying with these standards can make a significant impact on code readability and the life of the code. In this chapter, you will learn about another important aspect of software development, code documentation, and best practices. It starts with an introduction to the reStructuredText format and uses it to write docstrings. You will create HTML documentation for the code using the Sphinx document generator. The chapter also talks about some important coding standards for writing a Python code and using PyLint to check the code quality.

Chapter 5, Unit Testing and Refactoring, starts with an introduction to the unit testing framework in Python. You will write some unit tests for the game application developed so far. It covers many other topics, such as using Mock library in unit tests and measuring effectiveness of the unit tests with code coverage. The later part of the chapter talks about many code refactoring techniques. This is the last chapter that makes use of the code developed in the earlier chapters. The following chapters will have their own simplified examples tied to the same high-fantasy theme.

Chapter 6, Design Patterns, tells you how, during development, you often encounter a recurring problem. Many times, a general solution or recipe exists, which just works for this problem. This is often referred to as a design pattern. This chapter introduces you to some commonly used design patterns. It covers the strategy, simple and abstract factory, and adapter patterns. For each pattern, a simple game scenario will demonstrate a practical problem. You will learn how the design pattern can help solve this problem. Each of these patterns will be implemented using a Pythonic approach.

Chapter 7, Performance – Identifying Bottlenecks, is the first one in a series of three chapters on performance improvements. You will write a simple program called Gold Hunt that looks harmless until you tweak some parameters. The parameter tweaking reveals performance problems. In this chapter, you will identify the time-consuming blocks of the code. It covers the basic ways to clock the application runtime, profiling the code to identify performance bottlenecks, the basics of memory profiling, and using big-O notation to represent computational complexity.

Chapter 8, Improving Performance – Part One, teaches you how to fix some of the performance bottlenecks identified in the previous chapter. Additionally, you will also learn about several techniques, such as algorithm changes, list comprehension, generator expressions, the right choice of data structures, and so on, to improve the application performance.

Chapter 9, Improving Performance – Part Two, NumPy and Parallelization, is the final chapter on performance improvements, wherein you will drastically improve the performance of the gold hunt application. The chapter will introduce you to the NumPy package. It will also introduce you to parallel processing using Python.

Chapter 10, Simple GUI Applications, is the final chapter and introduces you to simple GUI application development. The chapters so far covered several key aspects of application development using command-line programs. In this chapter, however, you will learn about the Tkinter module, MVC architecture, and develop a GUI version of the first application developed in Chapter 1, Developing Simple Applications.

What you need for this book

The code illustrated in this book is compatible with Python version 3.5. The supporting code bundles also provide files compatible with version 2.7.9; however, throughout the book, Python version 3.5 is assumed. See the Installation prerequisites section of Chapter 1, Developing Simple Applications, for details on the basic packages that need to be installed. Additionally, there are some Python package dependencies that need to be installed. Most of these packages can be installed using pip (Python package manager). These dependencies are mentioned in the chapters that require them.

Who this book is for

Do you know the basics of Python and object-oriented programming?

Do you wish to go the extra mile and learn techniques to make your Python application robust, extensible, and efficient?

This is the book for you if you answered yes to these questions. It is also for those with a different programming background (for instance, C++ or Java) and wish to get to grips with Python application development.

This book is not for you if either of the following statements apply to you:

You are completely new to Python or do not have any background in OOP. The first chapter covers some basics but further understanding will be required.

You are looking for a reference on specific application domains, such as Web, GUI, database, or game applications. Except for GUI, this book does not cover such domain-specific topics. Nonetheless, the techniques you will learn in this book should provide a solid foundation to all such domains.

Conventions

In this book, you will find a number of text styles that distinguish between different kinds of information. Here are some examples of these styles and an explanation of their meaning.

Code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles are shown as follows: There are some changes to the GoldHunt.find_coins method.

A block of code is set as follows:

results = pool.starmap_async(self.find_coins,

                              zip(itertools.repeat(x_list),

                                  itertools.repeat(y_list),

                                  x_centers,

Any command-line input or output is written as follows:

export PATH=$PATH:/usr/bin/

New terms and important words are shown in bold. Words that you see on the screen, for example, in menus or dialog boxes, appear in the text like this: As mentioned earlier, at the time of installation, you should select the Add Python 3.5 to PATH option.

Note

Warnings or important notes appear in a box like this.

Tip

Tips and tricks appear like this.

Reader feedback

Feedback from our readers is always welcome. Let us know what you think about this book—what you liked or disliked. Reader feedback is important for us as it helps us develop titles that you will really get the most out of.

To send us general feedback, simply e-mail <[e-mail]>, and mention the book's title in the subject of your message.

If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, see our author guide at www.packtpub.com/authors.

Customer support

Now that you are the proud owner of a Packt book, we have a number of things to help you to get the most from your purchase.

Downloading the example code

You can download the example code files for this book from your account at http://www.packtpub.com. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you.

You can download the code files by following these steps:

Log in or register to our website using your e-mail address and password.

Hover the mouse pointer on the SUPPORT tab at the top.

Click on Code Downloads & Errata.

Enter the name of the book in the Search box.

Select the book for which you're looking to download the code files.

Choose from the drop-down menu where you purchased this book from.

Click on Code Download.

You can also download the code files by clicking on the Code Files button on the book's webpage at the Packt Publishing website. This page can be accessed by entering the book's name in the Search box. Please note that you need to be logged in to your Packt account.

Once the file is downloaded, please make sure that you unzip or extract the folder using the latest version of:

WinRAR / 7-Zip for Windows

Zipeg / iZip / UnRarX for Mac

7-Zip / PeaZip for Linux

The code bundle for the book is also hosted on GitHub at https://github.com/PacktPublishing/Learning-Python-Application-Development. We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!

Downloading the color images of this book

We also provide you with a PDF file that has color images of the screenshots/diagrams used in this book. The color images will help you better understand the changes in the output. You can download this file from http://www.packtpub.com/sites/default/files/downloads/LearningPythonApplicationDevelopment_ColorImages.pdf.

Errata

Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you find a mistake in one of our books—maybe a mistake in the text or the code—we would be grateful if you could report this to us. By doing so, you can save other readers from frustration and help us improve subsequent versions of this book. If you find any errata, please report them by visiting http://www.packtpub.com/submit-errata, selecting your book, clicking on the Errata Submission Form link, and entering the details of your errata. Once