System Administration Ethics: Ten Commandments for Security and Compliance in a Modern Cyber World

By Igor Ljubuncic and Tom Litterer

Successfully navigate through the ever-changing world of technology and ethics and reconcile system administration principles for separation of duty, account segmentation, administrative groups and data protection. As security breaches become more common, businesses need to protect themselves when facing ethical dilemmas in today’s digital landscape. This book serves as a equitable guideline in helping system administrators, engineers – as well as their managers – on coping with the ethical challenges of technology and security in the modern data center by providing real-life stories, scenarios, and use cases from companies both large and small. 
You'll examine the problems and challenges that people working with customer data, security and system administration may face in the cyber world and review the boundaries and tools for remaining ethical in an environment where it is so easy to step over a line - intentionally or accidentally. You'll also see howto correctly deal with multiple ethical situations, problems that arise, and their potential consequences, with examples from both classic and DevOps-based environments.
Using the appropriate rules of engagement, best policies and practices, and proactive “building/strengthening” behaviors, System Administration Ethics provides the necessary tools to securely run an ethically correct environment. 
What You'll Learn

• The concepts of Least Privilege and Need to Know
• Request change approval and conduct change communication
  
• Follow "Break Glass" emergency procedures
• Code with data breaches, hacking and security violations, and proactively embrace and design for failures 
  
• Build and gain trust with employees and build the right ethical culture
  
• Review what managers can do to improve ethics and protect their employees

Who This Book Is For This book’s primary audience includes system administrators and information security specialists engaged with the creation, process and administration of security policies and systems. A secondary audience includes company leaders seeking to improve the security, privacy, and behavioral practices.

• Language: English
• Publisher: Apress
• Release date: Oct 30, 2019
• ISBN: 9781484249888

Igor Ljubuncic

Igor Ljubuncic is a Principal Engineer with Rackspace, a managed cloud company. Previously, Igor has worked as an OS architect within Intel's IT Engineering Computing business group, exploring and developing solutions for a large, global high-performance Linux environment that supports Intel's chip design. Igor has twelve years of experience in the hi-tech industry, first as a physicist and lately in various engineering roles, with a strong focus on data-driven methodologies. To date, Igor has had fifteen patents accepted for filing with the US PTO, emphasizing on data center technologies, scheduling, and Internet of Things. He has authored several open-source projects and technical books, numerous articles accepted for publication in leading technical journals and magazines, and presented at prestigious international conferences. In his free time, Igor writes car reviews, fantasy books and manages his Linux-oriented blog, dedoimedo.com, which garners close to a million views from loyal readers every month.

Book preview

© Igor Ljubuncic, Tom Litterer 2019

I. Ljubuncic, T. LittererSystem Administration Ethicshttps://doi.org/10.1007/978-1-4842-4988-8_1

1. Separate Roles

Igor Ljubuncic¹  and Tom Litterer²

(1)

London, UK

(2)

Portland, OR, USA

Wendell thought back to his interactions with each of the team members. Could something they did have contributed to the breach? Maybe his presence triggered events that led to the breach.

The first days at a new workplace are much like visiting a new city. You’re a tourist, and most likely in a mild state of shock, preferably a pleasant one. You don’t know quite what to do – even if you’ve done it before – and you’re looking for a reputable source of wisdom to help you get around.

For Wendell, the tourist guide was no other than Alex, and he could tell right away that he should tread softly.

You’re lucky, Alex dramatized.

Oh?

Normally, I would wait a few weeks before giving you the access rights, but Mike’s keen on getting you on board as quickly as possible, so we’re doing the expedited version. You did work with privileged accounts before?

Wendell nodded.

"All right. So this is how it works. Your standard user is Wendell, and we’re going to set up wendell_p now. This will be your power user account, and you’re going to use it for privileged access. You don’t use your ordinary account for that, you use this one."

Is this a sudo account? Wendell tried to show some knowledge.

Alex waved impatiently. It’s actually five accounts.

Wendell let out a small laugh. Five? At my last place, we only used one. Well, two. We had the standard account for pretty much everything, and root when we had to do some admin work on the servers.

Alex blinked. He did not like having his show interrupted, it seemed. "Separate accounts are a smart thing. One, they give you more granularity when it comes to work. Two, they minimize risk. Three, you clearly know what each account does and what it’s used for. Now, it’s all tied to your user, but it maps to five user management systems. First, if you need to work on Windows machines, you’re going to have the AD account. Then, there’s also high account. This is a privileged account for Windows administration. Alex paused to look at something on his phone screen. Do you know what privileged access management is?"

Wendell nodded again. You mean PAM tools?

Yes. Good. Because we will need to set you up with SSH access, so you can do remote administration. Now, you SSH with your own account to other systems and then switch to wendell_p there. As a policy, we don’t allow the _p users to SSH to other hosts.

Got it, Wendell said, furiously writing notes down.

Alex swiveled his chair back toward his desk and started typing and running commands, faster than Wendell would have liked, but this was Alex’s thing, and he decided not to interrupt.

Now…the power user account is all good and well, but sometimes, you need proper root. Alex wrote something on a piece of paper. It was an eight-letter string, and Wendell assumed it was a password.

"This is our current root password. Now, normally, you go online to get the password. Let me show you. Here. Go to slash slash accounts; that will redirect to our online password management system. It’s called Sesame, and as you can see, it’s an ugly thing designed in the past millennium. Here, you log in with your power user. And then click Request root password, and it will show the password on the screen for 30 seconds. So if you forget it, or the password rotates, you can always get it this way. Alex raised a finger. Only with a local IP address. Won’t work over the VPN."

Wendell underlined the intranet alias in his notebook. Understood.

Excellent. Now let’s get some coffee, I’m running on fumes. Without waiting for Wendell to actually respond, Alex bounded off for the coffee area, swinging his old, faded-logo mug like a baton. Along the way, he stopped to exchange a few old and not-so-funny jokes with people sitting at other desks. Wendell didn’t really catch the inside meaning of most of it. He believed this was a well-practiced and often repeated ritual.

Once back at Alex’s desk, Wendell remembered the first-day-at-work presentation, a seemingly never-ending stream of slides, slogans, and mini interviews with this or that manager. And what about the VPN access?

Alex snapped his fingers. "That’s only going to work from your work laptop. Now, unlike Sesame, which only works inside the company’s network, you cannot actually use this one from within the intranet. You have to be on the Internet, like at home or in the coffee shop. But let me show you something neat."

Alex unlocked his phone and launched what looked like a console application, black background and a blinking cursor. I’ve set up a special bastion host in the DC that allows me to log in from this device.

Wendell was curious. He also couldn’t recall anything about phones being mentioned anywhere for system access. You SSH to the bastion and then switch to your admin account?

Yup. Saves time. When I’m on call at night, and the ops ring me, it’s so much easier and faster to just log in on the phone right away and see what’s happening. Most of the time, it’s just trivial misconfigurations, and I can quickly fix the issues and close the ticket. But if I need to do lots of work, I’ll then turn the laptop on, VPN in, and do the full hands on.

Wendell pursed his lips. He found the notion practical, but he was wondering about the security model. He decided not to raise the topic, though.

Alex was typing again. Since you’re new, if you’re not sure, it’s best to ask. Now, it’s a well-known fact, if you do system administration work, you’re going to end up making mistakes. Alex stood up, pushing his chair back. Henry!

What’s up?

How many times have you accidentally rebooted a customer server?

Too many, Henry muttered, busy behind his desk.

Alex sat down. So don’t worry about it. But remember. With your power account and root, you can easily cause a severe outage, so double-check your commands; and if you’re running a script on multiple hosts, always get someone else to check it before you hit Enter. Anyone in the team will do.

Wendell nodded. So it doesn’t matter who?

We all touch pretty much every aspect of administration. Doesn’t mean every one of us is an expert on everything, but sometimes, it’s necessary. Just the other day I configured Belinda’s high account for customer data access. She needed to look at some application logs, because her customer was complaining about performance issues.

Wendell kept on writing in his notebook. It was going to be a long day.

Separate Roles

*Don’t use a privileged account for personal work.

*Don’t use a privileged shared account without logging identity.

Why Is Separating Roles Important?

The phrase With great power comes great responsibility is widely attributed to Marvel Comics Amazing Fantasy #15 back in 1962, way before software-based system administration became an item. But the quote, in one form or another, has been ascribed to Winston Churchill, Voltaire, William Lamb, and perhaps even the Bible.¹ It would seem that, throughout the ages, people understood the moral weight of power.

The leap of faith from government and philosophy to system administration feels rather detached from the implication that the quote carries, especially since the core difference between ordinary users and privileged users in a system is merely in the ID number that their accounts carry.

But it is there.

A position of power allows people to utilize that power and make changes that others cannot do, regardless of their desire or skill. In the IT world, privileged system accounts are the most prevalent interface for system administrators for crossing the boundary from ordinary use, restricted to individuals and their own data, to wider activities that can impact other systems – and in turn other people. When you pause and think about it, the only thing that really stops privileged users from causing all-out mayhem is their judgment.

What do you do if you make a wrong judgment?

You cause damage, of course.

Early on in the software world, engineers realized that they could easily fall prey to their own personal weakness, whims, and honest mistakes and that they needed to create external mechanisms between subjective reasoning and hard data.

As a result, various security mechanisms were designed to create forced separation between common use (everyday work that is unique to each individual, group, or project) and privileged administrative work (maintenance of systems and services that enable people to do their everyday work).

The most common example of separating privileged access from the common user account is the root user (account) in UNIX/Linux systems. With this separation, users of the system can utilize applications and services without having the ability to make changes to the underlying operating system or configuration of the server. But for those who have root access, they have the ability to make pretty much any change in the system, including deleting evidence of their own work (purging system logs, disabling logging, changing permissions, etc.). The root users can really do everything, and in general, there’s very little to stop root commands from running.

A major concern of the root account is that there can be only one (Highlander fans, rejoice). In other words, if you or I successfully authenticate as the root user, we are both the root user. There’s no technical differentiation between us, and our commands will be attributed to the same user ID. This makes root an anonymous shared account – the identity of the person logging in is not reliably traceable.

The need to perform administrative tasks – separately from common work – and be able to maintain some level of identification in the system led to the creation of root-like mechanisms (like the popular sudo in Linux or Run As in Windows functionality), which allow users to temporarily switch from their restricted everyday work to privileged activities and have such work properly logged in the system.

It would appear that all worries and problems of system administration should have been solved by the introduction of root-like tools, wouldn’t it?

Well, not quite.

Quite the opposite.

With more people being granted administrative privileges, the IT world created more failure points in its structure. More importantly, none of the tools addresses the fundamental question of ethics. It all comes down to having the right credentials – if you do, you will be allowed access.

Don’t Use a Privileged Account for Personal Work

Credentials give you access – but privileged access doesn’t inherently include ethical judgment. Since each individual’s concept of ethics varies – and it depends on the available data, situation, your mood, your concentration, fatigue, peer pressure, time constraints, and applicable experience – it is important to purposefully minimize the use of privileged access to minimize mistakes. Since it is almost impossible to avoid all mistakes, a healthy and recommended practice is to use privileged access only when absolutely necessary to get the job done, and then to continuously refine processes to minimize the necessity for privileged access use the next time.

Moreover, there are different sets of rules, both technical and business related, for when you use your personal account and when you use a privileged account. For instance, you may decide to delete all the files and folders in your personal account, but doing that with a privileged account would compromise the system and impact all users. In our story, Alex stressed the importance of account separation, and he went through the list of different types of accounts and privileges that existed in their environment, as well as the mechanisms for how to request access to the root account using an intranet portal.

Alex also mentioned the topic of mistakes, and both Henry and he freely admitted to having made them in the course of their work. While one shouldn’t be proud of their errors, it is important to share them, so that people can develop awareness and learn about the situations and conditions when and where these mistakes happened. For a new employee like Wendell, this should help instill a sense of accountability, a healthy dose of risk-taking, and the understanding that honest mistakes can and will happen.

Now, if we look at it from a different angle, we can also see several shortcomings in the encounter between the two colleagues. Alex wanted to set up Wendell as quickly as possible, which indicates a time constraint that does not necessarily allow all relevant criteria to be met.

However, such shortcuts are often dangerous, because they create situations where people with limited knowledge or familiarity with the IT environment are given privileged access. Before granting (and accepting) such access, there should have been some verification of Wendell’s understanding of the appropriate use of privileged access within his new position.

Wendell should be comfortable expressing his concerns, whether it relates to his confidence as a system administrator in a brand-new environment or password sharing. He should feel free to interrupt Alex if he needs clarifications, especially since Alex was working quickly and Wendell had to take notes through the lecture and demonstrations.

After all, the purpose of the meeting between Alex and Wendell is to teach the new employee about the procedures and tools used in the company. The only effective outcome is if Wendell has mastered the domain in an effective way. This could also mean documentation on the account management process, a printed cheat sheet, or reference to online training. Alex could have initiated on-the-job training (OJT) by allowing Wendell to type in the commands – and Wendell should have asked the same.

Don’t Use a Privileged Shared Account Without Logging Identity

There are several reasons why you’d want to log identity when using a privileged shared account. The obvious answer is information security; if and when something goes wrong, you have a forensic trail.

The less obvious answer is that it reduces suspicion for when things go wrong.

Over the years, the IT industry has developed a strong and often necessary focus on the aspects of security, and many if not all privileged activities are instantly associated with this domain.

A privileged shared account is not a great opportunity to do things without consequences; on the contrary, it needs to be treated with maximum responsibility and accountability. Moreover, work with a privileged shared account should be documented. That way, the IT team has a full log of completed work, and it can associate activities with individuals. Not for the purpose of blame, but to achieve clarity and order and minimize mistakes.

In our story, we have only touched on the root account, but the principle applies to all shared identity accounts, like database or application services. We will touch on these in the coming chapters.

By providing root access, Alex set Wendell up to have no options but to violate the shared account portion of the first commandment (Don’t use a privileged shared account without logging identity). Wendell was given access to root but did not receive instructions on the mechanisms to log identity when using it.

Wendell had his reservations, and he should have voiced them to Alex. After all, Wendell did have experience from his previous workplace, and his input could be valuable. If Wendell felt that some of Alex’s suggestions were not ideal, could be improved, or potentially introduce risk into the business, he should feel free to say so.

Collaboration among team members is a great way to build trust – and improve the quality of both the work environment and technical tools used. Wendell could raise his suggestions to Alex. If he did not feel comfortable doing that at the spot, he could compile a list and present it at the next team meeting or send an email to the team, to see what they think about his ideas. In the long run, open discussion almost always leads to a refinement of existing procedures.

Furthermore, the use of a personal bastion for on-call work is another transgression. It goes against the eighth commandment (Communicate Change), as Alex created a setup that was not documented as a formal work process, and also the ninth commandment (Do No Harm), since having a privileged backdoor access into the network from his phone violates work procedures and allows Alex to cause damage in an unprecedented and unexpected way. We will talk more about these commandments in later chapters.

Processes That Support Ethical Behavior

There are several things that system administration teams can use to facilitate ethical behavior, especially when it comes to privileged access.

Knowledge Transfer

One of the most critical steps in the induction of new employees is an efficient onboarding curriculum, which helps people learn about their roles and responsibilities quickly and without