Good, Better, Best: A Guidebook for Performance Auditing

By Gary Blackmer

This guidebook explains performance audit approaches to produce the maximum value. It describes the development of an audit, from selection of a topic to the public release of the report. Readers will learn about strategies for assembling an audit schedule and spotting the important potential findings, methods to successfully tackle the most difficult topics, practices that save time, and audit approaches to leverage change in government agencies to benefit the public.

• Language: English
• Publisher: Lulu.com
• Release date: Apr 6, 2023
• ISBN: 9781312698482

Book preview

Good, Better, Best

A Guidebook for Performance Auditing

Gary Blackmer

First edition

Copyright © 2023 by Gary Blackmer

All rights reserved. No part of this book may be reproduced in any form or by any electronic or mechanical means, including information storage and retrieval systems, without written permission from the author, except in the case of a reviewer, who may quote brief passages embodied in critical articles or in a review. The information in this book offers suggestions, without warranty.

ISBN 978-1-312-69848-2

Cover photo by Eberhard Grossgasteiger

Good, better, best,

Never let it rest,

Until your good gets better,

And your better best.

-Unknown

Table of Contents

Preface

The audit that taught me auditing

PART I: The Audit Landscape

Mission throughline: Audit to your mission to benefit the public

Continuous effort: Build purposeful audit schedules

Findings throughline: Findings are the DNA for all your audit work

Audit Phase: For best results, begin with scoping

Evidence throughline: Persuasive evidence needs professional judgment

Audit Phase: Make plans centered on findings and evidence

Audit Phase: Follow the fieldwork plan

Writing throughline: Compose a story to engage the public

Audit Phase: Draft a clear, concise, compelling, and corroborated report

Post audit: Relentless auditing

A great audit for Oregon: TANF

PART II: Excursions and experiences

The character of auditors

Serendipity in auditing

Encounters and lessons

A little auditing context and history

Endnotes

Preface

Modern performance auditing in government lacks a full description of how to conduct the work. I don’t claim to be the best auditor, or the most accomplished, but after thirty years of auditing at city, county, and state levels, I have accumulated some worthwhile knowledge and perspective. The methods I describe have produced innovative and impactful reports that benefited the publics of my city, county, and state. Still, I hope there will be other guidebooks in the future as professionals refine performance audit strategies.

My work environment probably differs from many because I was either an elected auditor, or worked for one. That situation offered greater latitude and independence than an appointed auditor answering to a committee or executive would have. I could take on expansive or controversial audit topics, pursue accountability strategies for the long term, and seek every way to meet the responsibilities the voters expected of me.

While your office may not have that latitude, this guidebook can still help you and your audit organization to tackle the most difficult and high-impact topics, and it also describes the framework, strategies, and methods needed to achieve success.

To meet the challenge of this guidebook, audit managers will probably have to break from their old procedures. The audit teams will be sent off the familiar, safe trails to explore areas where there are no maps. They will be watching how frontline workers deliver the agency mission, rather than how back-office employees, guided by written procedures, carry out technical tasks. Their endeavor will not start with a scope and objective, but instead with a gamble: that the team’s searches will come upon findings that have a direct and measurable impact on the public. This guide offers them many strategies for spotting those critical topics. With those findings in mind, audit teams can develop tight plans to gather the necessary evidence and build a persuasive case for big improvements.

Why change your approach to auditing and take these chances? For the same reason your audit reports recommend agencies undergo changes: to produce much greater value for the public. These methods reduce auditors’ frustration and misspent time, and they work for all sizes of organizations.

The ideas in this guide convey some of my experiences and practice as an auditor, and those of others. Descriptions of experiences can teach as much as methods. Throughout my career I was in the privileged position to gain my experience with smart auditors who probed for the management lapses, mistakes, and shortcomings in agencies.

I had the pleasure of working with wise auditors who made me better. I learned from mentors like Jewel Lansing and Dick Tracy, and from peers like Jerry Heer of Milwaukee County, Doug Norman and Ken Gavette in Portland, Suzanne Flynn and Craig Hunt in Multnomah County, Mark Funkhouser in Kansas City, Amanda Noble and Leslie Ward in Atlanta, Corrie Stokes in Austin, Mary Wenger and Scott Learn in Oregon Audits. I was inspired by the enthusiasm and commitment of auditors like Jenny Wong of Berkeley, Kathryn Nichols in Multnomah County, and Amelia Eveland and Jamie Ralls in the state, whose ideas and questions helped my own thinking.

Of course, I relish my memories of the nearly 150 other smart, good-hearted, dedicated auditors that I’ve worked with, and learned from as well. In writing this, I often felt more like a scribe than an author. Occasionally, I use the names of fellow auditors when they were key to a story, but many others contributed in their own way and go unnamed. Any errors in this retelling are mine. The improvements by my editor Eva Fitzsimons clarified my writing and made it more accessible to those who are not auditors already.

I also extend my appreciation to the Government Accountability Office for its Government Auditing Standards. In this guidebook I refer to the 2018 version. I offer some criticisms of the Standards, but they are a remarkably sound and relevant foundation for auditing in any government entity and important for every auditor to know and understand.

And we all must acknowledge the value of public employees, managers, and elected officials who embrace auditing as part of their larger commitment to better serve the public.

You will see a range of illustration styles in this guidebook because they are copied from a thirty-year span of audits I was involved in, from the most primitive computer graphics to the most advanced. Some images are small and difficult to read, due to the size limits of eBooks. When an image might be worthy of closer scrutiny I have included a link to garyblackmer.com for a more legible image. Please note, the ‘back button’ may not return you to the same page on some eReaders, and you may have to reopen the eReader.

In addition, the endnotes of various audits and reports I discuss include a link to garyblackmer.com where they are all available to read.

The earnings from this book will go to the Association of Local Government Auditors, in appreciation of all the efforts these volunteers give to the profession.

Of course, I owe much to my wife, daughter, son, and parents, even as I diverted attention to my work. I depended on their encouragement and support, and I hope that knowing the benefits to the public of my time spent on audits was some compensation for them.

The audit that taught me auditing

This was my second audit. It ended with a few anxious weeks and a rare, tumultuous conclusion. As a new auditor, I was mostly a blank slate, and this experience had a profound and lasting influence on my auditing career. I will tell the story as I remember it some thirty years later, then highlight some of the lessons this audit impressed upon me.

I had just completed my first Portland audit assignment in 1986 when I joined a team with Lloyd and Doug to audit police patrols. It was an area I knew well, since I had spent about six years in the nearby county sheriff’s office, analyzing deputy deployment, among other duties. I knew how to use 911 call data to understand patrol workload, and how to consider the deployment of the Portland officers to handle their calls. And I already knew Doug from that time as well; he had been one of the county auditors who audited the sheriff’s office before he was hired by the city.

I also knew something about the police bureau’s deployment. While at the sheriff’s office I had been in regular contact with my counterparts in the police bureau’s planning and research unit and knew that they had not performed the kind of analysis I had been doing, looking at calls by hour and day of week and comparing them to the number of officers during those same periods. They only had canned monthly reports, programmed in machine language and changeable only at great expense. These reports showed them averages across eight-hour shifts and across days, which smoothed the peaks and valleys and blurred the problems. I had learned how to use a newer statistical reporting tool to conduct more detailed analysis.

I was not aware at the time that the audit had been on the previous year’s schedule. The auditor’s office had delayed it at the request of the police chief, who said it was not good timing. The chief was making the case that the call workload of Portland’s patrol officers exceeded national standards. He submitted a budget request to add another 200 police officers to the 750 he currently had.

We were guided by Dick Tracy (yes, that’s his real name), the wise and experienced director of audit services. He explained that the city commissioners did not understand the basis for the police bureau’s budget request. The commissioners were seeking an independent assessment because it was such a large amount, and because the bureau’s methodology was complex and difficult to question.

Since performance auditing had been introduced just three years earlier, we were still making our way through the bureaus. Not only was this our first look at the police bureau, but it was only my second audit. It would be a learning experience for both them and me.

We initiated the audit and began interviewing personnel, gathering data, reading about best practices, and going on ride-alongs. We watched patrol officers respond to calls and asked them various questions about workload and scheduling. We interviewed shift sergeants and commanders at each of the precincts, as well as dispatchers and central administration managers.

We learned that the justification for an additional two hundred police officers was based upon a computer model called PCAM, recently developed by the RAND Corporation. For its time, it was a sophisticated method for considering deployment options to avoid queuing calls and delayed responses. It also allowed agencies to determine workforce needs in order to meet performance standards.

We developed an audit plan to look at three areas. The first was how the bureau had been directing its lowest priority calls to a telephone report-taking unit, which reduced the workload of district officers. The second was how well the district officers were scheduled to match the 911 call workload. Finally, there was a review of the bureau’s use of PCAM to justify the two hundred requested officers.

The lieutenant who managed the planning and research unit was our designated liaison, and we met with him and the command staff to announce our three areas of focus. At that point, the staff of the unit and other managers were open and cooperative, but the audit would change it all.

Lloyd was our computer whiz, and he also examined the diversion of low-priority calls to the telephone report-taking unit. The unit was intended to take reports on calls where no suspects were known, such as vandalism. He reported the callers were satisfied with a telephone response and the unit was able to divert forty thousand calls annually from patrol.

Although the telephone report-taking unit was supposed to be staffed with four full-time officers and up to a dozen other officers on limited duty because of a work disability, Lloyd found the phone line was only staffed about a quarter of its scheduled hours. As a result, these minor calls required a response from a patrol officer, costing about five times more than a telephone response.

He also found that the types of calls could be expanded if the report-taking unit was sufficiently staffed. Including other types of calls could further reduce patrol workload. For example, burglaries that weren’t discovered for several days could be handled by telephone if losses were less than $200. We also recommended that the bureau ensure adequate staffing and consider using non-sworn personnel, as other police agencies were doing.

Using a sample of four months, I analyzed the rosters for all the precincts for all shifts and days to determine the number of patrol officers on duty. I compared the number of calls in a week (168 hours) to these officers.

The bureau used three 8-hour shifts: 8a.m. to 4p.m., 4p.m. to midnight, and midnight to 8a.m., which seemed to cover all 24 hours in a day. However, the patrol officers only responded to calls for 7 hours in each shift. The first half-hour of each shift was taken up by roll call and traveling to the assigned district. The last half-hour was spent driving back to the precinct, gassing up the car, returning equipment, and turning in reports.

Added together, there were 3 hour-long gaps in each 24-hour period. To accommodate this, the bureau created an overlay by splitting each shift in half, with one group starting an hour earlier. The problem with this scheme is that the number of officers is halved for 6 hours of these splits in each 24 as each of the smaller shifts start and end their day. The changeovers occurred between 6:30am and 8:30am, 2:30 and 4:30p.m., and 10:30p.m. and 12:30a.m. That midnight period was also the peak 911 workload period of the day. The figure below shows the volume of calls and workload expressed in percentages through the average 24-hour day. Here is the dot-matrix graphic from the report.

Chart, diagram, radar chart Description automatically generated

Bureau of Police: Patrol Staffing and Deployment Practices, p.15

When I looked at officer response times to emergency and non-emergency calls it was obvious that arrivals were delayed during those six hours each day. This was reflected in my interviews of 911 dispatchers who complained about lack of cars during shift changes. And patrol officers could rightfully complain about overwhelming workloads during those shift changes, and having to use the rest of their shift to clean up the non-priority calls.

Chart, diagram, radar chart Description automatically generated

Bureau of Police: Patrol Staffing and Deployment Practices, p.18

Contributing to the drawbacks of the shift configuration was officer time off. Sergeants were allowing too many officers to take time off on weekends. The largest number of calls occurred during evenings and weekends but there wasn’t a proportionate increase in the number of officers scheduled.

A picture containing chart Description automatically generated

Bureau of Police: Patrol Staffing and Deployment Practices, p. 24

I had the condition: the patrol force and its overall capacity to handle calls. The criterion was the stated goal of the bureau and the patrol commanders to deploy their resources when needed. I had the effects: the slower response times during shift changes, and the variances in officer workload. And I had two causes: the shift configuration and the mismatch on shifts and days of week.

We recommended the bureau adopt an alternative schedule that better matched the number of officers to the workload. We also recommended that the bureau report its workload and staffing data to supervisors to aid their decision-making, and establish a leave policy that ensured adequate staffing.

Doug was examining the computer program in the meantime and began questioning the basis for calculating the needed staffing. Police Bureau Planning and Research calculated that 47% of an officer’s patrol time was consumed by calls. They then used PCAM to estimate the number of added officers to reduce the time they were busy to 35% for routine patrol work. This left the remainder of officers for directed patrol work that would address crime proactively. PCAM calculated that 111 more patrol officers would be necessary to meet the 35% standard. In addition to the patrol officers, the bureau proposed another 88 detectives and other support officers.

The first part of Doug’s analysis showed patrol officers were only busy 43% of the time. That correction would have reduced the number of needed officers, but beyond that, he began questioning the source of the 35% standard. Planning and Research indicated that it came from the International Association of Chiefs of Police (IACP).

When Doug called, it was clear that the bureau staff had already contacted the IACP about our audit since the researcher said they were already looking for the documentation and would get back to us. Doug examined other research literature for standards but found nothing to support the 35%. He contacted the PCAM developer who cautioned that the computer program could only provide ballpark measures. Doug’s research also discovered that Portland’s workload was in the middle of the range for comparable west coast police agencies that tracked the information.

Doug called the IACP again, several weeks later, and they admitted it wasn’t a hard and fast standard. When Doug explained how it was being used in Portland, the IACP researchers became more evasive but still tried to explain the 35% as professional judgment and rule of thumb. A third call to IACP a week later got an acknowledgment that they could not find any document supporting the 35%.

We recommended that the bureau use multiple factors such as historical staffing and budgets, comparable western cities, and patrol strategies to propose alternative policing levels for city council consideration. We also recommended that the bureau develop more specific plans and procedures for directed patrols.

At this point we were concluding our fieldwork, so we briefed our liaison lieutenant on what we were finding. He agreed that the telephone report-taking unit was not always staffed and that shift changes created significant problems, but said that he would make sure that Planning and Research found the source of the 35% standard. We said that we would schedule a meeting with the chief to provide a similar briefing.

I had also been looking at the deployment of traffic patrols, which worked ten-hour shifts. I was puzzled by the deployment because although the traffic patrols also had the later shift starting as the early shift ended, they had no overlap. I called the captain in charge with a couple questions. First, I asked if the first half hour and the last half hour of every shift was taken up with travel time and housekeeping. The captain confirmed this. I then asked when that occurred. He said, That would be just before five and just after five. I asked if he knew when the largest volume of traffic calls came in. I heard a sigh, almost a groan, before he said, That would be around the five o’clock rush hour. Finally, I had to ask, When you have no traffic cars on the street? which he acknowledged.

We started drafting our report.

My section was long; my rookie approach was to convince them with detail and overwhelm them with arguments. Nonetheless, my excessive writing allowed Dick to cherry-pick the key concepts as he edited down the report.

Our exit conference was scheduled in the chief’s conference room on the top floor of the Justice Center, overlooking the river, the east side of the city, and on a clear day, Mount Hood. The conference table was at least twenty-five feet long with the police chief at the head and every top manager filling most of each side. Dick, Lloyd, Doug, and I were outnumbered about four-to-one.

Dick explained our audit findings, starting with the telephone report-taking unit. We saw some nods and acceptance of the fact that staffing was currently insufficient. Dick’s style was always understated and general in these meetings, noting the information was preliminary until we completed our quality control process.

In the same gentle style, he talked about how the scheduling of patrol officers could better match the workload. When he noted the low coverage during shift changes the room chilled.

How would you suggest we address that concern? the chief asked.

Dick suggested alternative shift configurations such as combinations of four-day ten-hour shifts instead of the current framework. The chief stiffened, and his subordinates watched him closely.

How many shifts are you talking about?

There are a number of alternatives that you could consider. Some involve three twelve-hour shifts; one with four, one with five.

That’s ridiculous, the chief said. Every one of the managers nodded in agreement. That many shifts are impossible to manage.

I couldn’t contain myself and blurted out, Other departments like Akron, Charlottesville, and Long Beach are able to make them work. If I had been thinking, I could have said that the bureau was actually running with six shifts every day. The chief stared at me, then looked at Dick.

What makes you think you can come in here and tell us how to run the police bureau when you hardly know anything about police work? I’ve – we’ve all – spent countless hours behind the wheel of a patrol car, and you?

Well, chief, Dick said. We looked for ways to get more out of the bureau’s resources in this case, which is different than the decisions of individual police officers in patrol cars.

Someone else, in support of the chief, challenged the feasibility of these shifts and declared that they would need money for extra overtime as well as many more supervisors and vehicles. Many snickered, including the captain of one of the precincts who had told us during our interviews that he was preparing a proposal for four-ten shifts. The chain of command killed his idea.

The chief signaled for Dick to continue. Dick introduced our questioning of the police bureau’s use of PCAM-based calculations to justify the budget request for additional officers. At this point, the meeting disintegrated. The chief asked a series of snide questions, warned of dire public safety consequences, and challenged our competencies.

This audit’s timing is purely political, he said. We are just now having our budget considered and it will be voted by city council next month.

This audit was delayed at your request. We did not consider the budget process in starting or ending this audit, Dick responded. The chief was chagrined, knowing that the audit would undermine his efforts to get council backing for additional officers.

We received the brunt of attention for a while. We were the subject in a round robin where each of the commanders had a solo turn raising some farfetched accusation or extreme consequence, followed by a chorus of chuckles, head shaking, and sounds of indignation.

Do you have any idea what this will do to our overtime budget?

How many more of these so-called shifts are you recommending?

"You think a civilian can properly fill out a crime report and the public would be satisfied with that?

None of this is practical. You’re way out of your league with these ideas.

How in the world do you think we could force injured officers to come in and answer phones?

Stick to accounting.

Do you know what will happen when you show the bad guys the timing of our shifts?

Have you looked at officer injuries and stress claims because of overwork?

One manager, livid, said, We tried four tens in the past and they didn’t work.

Dick paused, looked puzzled and spoke, We interviewed a lot of people during this audit, and nobody talked about that. When did the bureau have four-ten shifts?

In 1956, and they didn’t work.

As calm as ever, Dick asked, Do you have any documentation on that?

The manager jumped out of his chair and said loudly, "This meeting – this audit – is ridiculous. I’m going to sell used cars," and left the room.

Dick closed out by saying that we would deliver a draft report that we wanted to sit down and discuss with them. We would then send a revised draft that they could prepare a written response to. We took the elevator down to the street level in silence. I was angry, shot through with adrenaline, and unable to organize my thoughts for a while.

We realized that this conflict was only going to intensify when they saw our draft report. We would double-check our key facts and Dick decided that I should add some example shift configurations used by other police agencies. Lloyd would look into the overtime and fleet impacts. Doug would redouble his efforts to find a source for some standard of call workload, whether it was 35% or not.

Of course, I overdid the alternative shifts. I think Dick was looking for descriptions of the shifts but I took the Portland patrol force and allocated them to show that they could schedule like Akron, or Phoenix, or the Multnomah County Sheriff’s Office. I also came up with the idea of correlating the number of available officers to the call workload by hour for the current and alternative configurations.

Correlation is a measure of how well two things align, in this case, the number of officers on patrol and the volume of 911 calls. When I applied Akron’s configuration, I got a correlation of .82, with 1.0 being a perfect match. The Phoenix approach had a .64 correlation. Multnomah County’s was .65. The correlation for Portland’s configuration was only .25, closer to that of a random allocation, which has a correlation of zero. It took seven pages to