OSINT in the Intelligence Era: Lecture notes

By Gianluigi Me

"OSINT in the Intelligence era. Lecture Notes- VOl.1" is a comprehensive book that takes readers on a captivating journey into the realm of intelligence and Open Source Intelligence (OSINT). With an emphasis on practical knowledge and real-world applications, this book covers a wide range of topics, from the fundamentals of intelligence to the intricacies of OSINT methodologies.
In the early chapters, readers are introduced to the concept of intelligence, its historical evolution, and the various levels it encompasses. The book then delves into the intelligence cycle, warfight domains in a military context, and explores the 5th dimension of cyberspace and network-centric warfare. Additionally, it explores the emerging 6th dimension of cognitive warfare, highlighting its significance in modern intelligence operations.
The book also examines different types of intelligence, including national security intelligence and criminal intelligence. Readers gain insights into the functions of criminal intelligence, the Europol EMPACT policy cycle, and the role of OSINT as a cross-disciplinary intelligence discipline. Furthermore, the book explores the ethical considerations and legal aspects of OSINT activities, with a focus on privacy-respectful OSINT practices.
Addressing the limitations of intelligence, the book sheds light on cognitive biases in both intelligence and OSINT, emphasizing the importance of critical thinking and data quality. It also explores the challenges posed by news bias, the proliferation of fake news, and the rise of deepfakes. Additionally, the book examines the impact of artificial intelligence (AI), natural language processing (NLP), transformers, and ChatGPT technologies on online investigations and intelligence operations.
To equip readers with the necessary technical skills, the book covers the basics of IP networking for OSINT. Topics include network protocols, the OSI blueprint, the TCP/IP protocol suite, internet fundamentals, IP protocols, DNS, hypertexts, and email services. The book also provides guidance on machine preparation, virtual machines, OSINT prebuilt virtual machines, anonymization techniques, and the use of regular expressions for text extraction.
"OSINT in the Intelligence era. Lecture Notes- VOl.1" serves as an invaluable resource for intelligence professionals, cybersecurity practitioners, law enforcement officers, researchers, and anyone interested in understanding the fascinating world of intelligence and harnessing the power of OSINT. With practical insights, ethical considerations, and technical guidance, this book empowers readers to navigate the complex landscape of intelligence with confidence and proficiency.

• Language: English
• Publisher: Edizioni Themis
• Release date: Aug 17, 2023
• ISBN: 9788896069547

Book preview

Title

First edition July 2023

ISBN 9788896069547

©2023 THEMIS S.R.L.

via Veturia, 44 – 00181 Roma [e-mail] THEMIS CRIME

All rights reserved

Reproduction, translation, adaptation, even partial or in extracts, for any use and by any means, including photocopying, microfilm, electronic storage, etc., are strictly prohibited without the prior written authorization of Themis s.r.l. Every abuse will be prosecuted to the full extent of the law.

Preface

The world of intelligence gathering has undergone a massive transformation in the last few decades. With the advent of technology, it has become increasingly easier to access information that was once thought to be elusive. Open-source Intelligence (OSINT) is one such method that has gained immense popularity over the years. While OSINT may seem simple on the surface, it requires a multidisciplinary approach. Firstly, experts in data analysis are needed to sift through the vast amounts of information and identify patterns and trends. Secondly, linguists and cultural experts can provide insights into the nuances of language and behavior displayed online. Thirdly, cybersecurity specialists can ensure the integrity of the data being analyzed. By combining these different areas of expertise, a more comprehensive understanding of OSINT can be achieved. This is crucial for law enforcement agencies, intelligence services, and other organisations that rely on accurate and timely intelligence to make informed decisions. Failure to adopt a multidisciplinary approach to OSINT can result in the oversight or misinterpretation of vital information, potentially leading to dire consequences. Additionally, in order to be successful in the field of OSINT, possessing a diverse range of expertise from various disciplines is equally important. This is because OSINT requires knowledge and skills from various fields, such as computer science, information technology, data analysis, social sciences, and more. A good understanding of how the internet works and how to link data is needed, as well as the comprehension of the psychology of human behavior, which is helpful in analyzing and interpreting open-source data effectively. Finally, having knowledge of legal and ethical considerations related to OSINT is crucial for ensuring that all activities are conducted within legal bounds. Therefore, it is crucial for undergraduate students to develop skills in OSINT. Regardless of the degree they are pursuing: OSINT provides the ability to gather and analyze information from publicly available sources and valuable insights into global events and trends, which can inform decision-making processes in various fields. This book, as a result of many years of international courses and trainings worldwide held by the author aims to provide readers with a methodological approach to OSINT and practical recommendations to use it effectively for intelligence gathering. Whether you are a law enforcement officer, a private investigator, or just someone looking for to learn more about intelligence gathering, this book aims to provide the needed background to rigorously approach OSINT.

Who Is This Book For?

This introductory book on OSINT (Open-source Intelligence) is designed for anyone who wishes to explore the vast world of intelligence gathering and investigation through publicly available information. Whether you are a professional in the fields of cybersecurity, law enforcement, journalism, or private investigation, this book will equip you with essential knowledge and techniques to leverage open-sources effectively.

If you are a beginner with no prior experience in OSINT, fear not! This book provides a solid foundation, guiding you through the fundamental concepts, tools, and methodologies of OSINT. It assumes no prior technical expertise, making it accessible to individuals from various backgrounds.

Are you a digital enthusiast curious about the hidden world of online information? This book will unravel the secrets behind searching, analyzing, and verifying information from social media platforms, websites, forums, and other digital sources. Discover how to harness the power of search engines, social network analysis, and data visualization to extract actionable intelligence.

Furthermore, if you are a concerned individual aiming to protect your online privacy and enhance your digital security, this book offers valuable insights. Learn to safeguard your personal information, identify potential threats, and adopt best practices to navigate the digital landscape safely.

Regardless of your motive for diving into the realm of OSINT, this book serves as your comprehensive guide. By the end, you will possess the skills necessary to conduct effective investigations, uncover hidden connections, and make informed decisions based on the intelligence gathered.

Embark on this enlightening journey and unlock the potential of open-source intelligence with this introductory book, suitable for professionals, enthusiasts, and individuals seeking to enhance their understanding of the ever-evolving information landscape.

To my family and my parents

Contents

Preface

I Introduction

1. Intelligence

1.1 The concept of Intelligence

1.1.1 Levels of Intelligence

1.2 A brief historical evolution of Intelligence

1.3 The Intelligence Cycle

1.4 Warfight domains in military context

1.5 The 5th Dimension: The Cyberspace

1.6 Network Centric Warfare

1.7 The 6th Dimension: Cognitive Warfare

2. Types of intelligence

2.1 National Security Intelligence

2.2 Criminal Intelligence

2.2.1 Functions of criminal intelligence

2.2.2 The Europol EMPACT policy cycle

2.3 Intelligence collection disciplines

2.4 OSINT as a cross Intelligence discipline

2.4.1 OSINT value in the intelligence era

2.5 OSINT support for Cyber threat intelligence

2.6 Ethics and Law in OSINT activities

2.6.1 The privacy respectful OSINT

3. Limitations of Intelligence

3.1 Limitations of intelligence

3.2 Cognitive biases in Intelligence and OSINT

3.3 Data Quality

3.3.1 News bias

3.4 The primary OSINT evil: fake news

3.4.1 The OSINT latter evil: deepfake

3.5 Artificial Intelligence

3.6 AI impact for online investigations and intelligence

3.7 NLP, transformers and ChatGPT support to OSINT

4. The basics of IP networking for OSINT

4.1 Introduction

4.2 Preliminary background

4.3 Network protocols

4.4 The OSI blueprint and Internet

4.5 The Internet

4.6 The TCP/IP protocol suite

4.7 The IP protocol

4.8 The TCP protocol

4.9 The DNS

4.10 Hypertexts and World Wide Web

4.11 The Email service

4.11.1 The SMTP protocol

4.11.2 The POP3/IMAP protocol

5. The must have: machine preparation and text extraction

5.1 OSINT machine preparation

5.2 Virtual machines

5.3 OSINT prebuilt virtual machines

5.4 Using GitHub repositories

5.5 Anonymize yourself

5.6 Regular expressions

5.7 Regex quick reference

5.8 Some OSINT Regexes examples

5.8.1 Extracting mobile phone numbers

5.8.2 Extracting email addresses and URLs

5.8.3 Extracting bitcoin wallets

5.8.4 Entity extraction from tweets

5.8.5 Standardizing extraction groups in text

Bibliography

Index

List of Domains

List of Figures

1.1 The Intelligence cycle

1.2 Search graph starting from search engine

1.3 Warfight domains

1.4 Network-Centric Warfare Domains

2.1 Routine Activity Theory schematization

2.2 EMPACT policy cycle

2.3 The Overlapping Nature of Intelligence Disciplines

2.4 OSINT cycle, as in [36]

2.5 Offensive and Defensive OSINT

2.6 Threat intelligence levels

3.1 Confirmation Bias

3.2 Dunning-Kruger effect

3.3 Data quality 4x4 system (Europol Information management handbook)

3.4 News bias by Adfontesmedia

3.5 Common expressions of social media information

3.6 The fake news cycle carried out by Russia during the war against Ukraine

3.7 Mapping OSINT news on EyesOnRussia.org

3.8 Some of the ways that AI can fit into the intelligence cycle

3.9 Language transformers evolution

3.10 ChatGPT snapshot developing python software

4.1 Analog and Digital communications

4.2 Circuit vs Packet switching

4.3 Network protocol by human metaphor

4.4 Seven-layer architecture of ISO OSI

4.5 OSI communication

4.6 Internet communication

4.7 Comparison of TCP/IP and ISO OSI network models

4.8 TCP IP stack

4.9 IP packet

4.10 IP addresses classes

4.11 TCP packet

4.12 3-way handshake

4.13 DNS domain hierarchy

4.14 DNS domain name example

4.15 HTTP interaction in case of user sending username and password (POST method)

4.16 HTTP request header (GET method)

4.17 HTTP response header

4.18 Number of sent and received e-mails 2017-2025

4.19 How email works

5.1 VirtualBox Linux CSI Desktop

5.2 Visual regex to extract mobile phones number

5.3 Visual regex to extract email

5.4 Visual regex to extract URLs

5.5 Visual regex to extract Bitcoin addresses

5.6 Visual regex to extract Twitter author

5.7 Visual regex to extract tweet mention

5.8 Visual regex to extract hashtag

5.9 Visual regex to extract URL from a tweet

List of Tables

2.1 Intelligence disciplines

2.2 Media usage in an internet minute as of April 2022

2.3 Example of difference between data, information and intelligence

2.4 Example of difference between OSD, OSIF and OSINT

4.1 Port numbers and related services

4.2 DNS records

5.1 Regular Expression Quick Guide

Real education is about getting people involved in thinking for themselves and that is a tricky business to know how to do well, but clearly it requires that whatever it is you’re looking at has to somehow catch people’s interest and make them want to think, and make them want to pursue and explore.

Noam Chomsky, Understanding Power: The Indispensable Chomsky

1. Intelligence

In wartime, truth is so precious that she should always be attended by a bodyguard of lies.

Winston Churchill

1.1 The concept of Intelligence

Intelligence has played a critical role for mankind since the earliest humans began to think and process information. In fact, the information and the derived intelligence directly influence the daily decisions of individuals, businesses, industry, the military, and the government.

In particular, intelligence is a crucial component of the modern military, law enforcement and business because it refers to potential enemies, their intentions, capabilities, and vulnerabilities. For example, military intelligence helps commanders make informed decisions and enables them to plan and execute successful operations.

Conversely, in the context of business, intelligence refers to the gathering and analysis of information about a company’s internal and external environment. The goal of business intelligence is to provide decision-makers with actionable insights that can inform strategic planning, resources allocation, and operational decisions. Many nations have risen and fallen on the power of intelligence and the decisions that have resulted from it. Thus, the ability to know, anticipate, and plan is very powerful.

In fact, decision-makers should be able to anticipate what will happen next, and how different courses of action are likely to play out over time. They should be able to think creatively about potential solutions, and weigh the costs and benefits of each option. Finally, they should be able to communicate their decisions clearly and persuasively, both to their subordinates and to external stakeholders.

Improving decision-making skills is a complex process that requires ongoing training and development. But by focusing on situational awareness, anticipation, creativity, and communication, decision-makers can become more effective leaders who are better equipped to navigate the challenges of an ever-changing world.

In particular, decision-makers typically have the expectation that intelligence will offer them a comprehensive understanding of both quantitative and qualitative factors, providing valuable insights into abstract and intangible aspects. When this happens, intelligence can describe existing situations and identify or confirm capabilities that will shape future conditions.

As a consequence, intelligence is itself a dynamic concept that does not have just one definition or application. As mentioned above, the ultimate purpose of the intelligence product is simple: provide an edge to the decision-maker.

No single definition of intelligence is accepted by all. The term itself is used in a variety of ways, which makes it difficult to come up with a single definition. Moreover, complicating the problem, different agencies have particular missions and operate under different rules.

Definitions carefully formulated by intelligence experts do exist, but all seem deficient in one aspect or another; the concept remains as sprawling and thorny as a briar patch. Each expert tends to view the term through the spectacles of his specialty.

In particular, some definitions consider intelligence solely a product., others recognize that intelligence is also a process, even lacking fundamental aspects, as, e.g. counterintelligence, which is like that entailed in explaining an automobile in terms of its motor without reference to its bumpers or brakes.

For example, the focus of the Central Intelligence Agency (CIA) is international. It has an entirely different set of guidelines than a domestic law enforcement organisation, such as the Federal Bureau of Investigation (FBI). Hence, both define intelligence some-what differently.

In fact, CIA defines¹ the intelligence as

Intelligence is the collecting and processing of that information about foreign countries and their agents which is needed by a government for its foreign policy and for national security, the conduct of non-attributable activities abroad to facilitate the implementation of foreign policy, and the protection of both process and product, as well as persons and organisations concerned with these, against unauthorized disclosure.

while FBI defines²

Simply defined, intelligence is information that has been analyzed and refined so that it is useful to policymakers in making decisions—specifically, decisions about potential threats to our national security.

Apart from the definitions provided by NIST³, for the scope of this book, we adopt the generalization of the definition of Intelligence formulated by the U.S. Director of the National Intelligence (DNI):

Intelligence is information gathered within or outside the country that involves threats to the nation, its people, property, or interests; development, proliferation, or use of weapons of mass destruction; and any other matter bearing on the national or homeland security. Intelligence can provide insights not available elsewhere that warn of potential threats and opportunities, assess probable outcomes of proposed policy options, provide leadership profiles on foreign officials, and inform official travelers of counterintelligence and security threats.

It is the product resulting from the intelligence cycle, a process whereby raw information is acquired, converted into actionable information, and disseminated to the appropriate consumers. Generating reliable and accurate intelligence data are a fundamental requirement; for this reason, the Intelligence cycle, detailed in section 1.3, is a dynamic and sophisticated process through which information is processed in an accurate and usable way, in order to provide secure sources and to make good national decisions. The aim is to develop information into finished intelligence for consumers, including policymakers, law enforcement executives, investigators, and patrol officers. These consumers use this finished intelligence for decision-making and action. Typical intelligence activities are commonly referred to as covert actions and represent fundamental component of national power and decision-making in national security, defense and foreign policy (Pringle and Ransom, 2021) [28].

In order to further clarify when a secret information is accessed/disclosed or, simply, it is discovered, a clear distinction between espionage and intelligence is needed, because they are related concepts, but they are not the same thing.

Espionage refers to the practice of gathering information, often secretly, from a foreign government or organisation with the intent of obtaining strategic, political, or military advantages. It is often associated with covert operations, including the use of spies and other forms of espionage.

On the other hand, intelligence refers to the collection, analysis, and dissemination of information to support decision-making and other activities related to national security, foreign policy, and law enforcement. Intelligence gathering can include not only traditional espionage activities but also more overt methods, such as satellite imagery, intercepted communications, and open source research Champagne, 2006) [5].

Every State, in order to ensure the protection of its citizens, has multiple intelligence agencies, due to several factors, including the complexity of national security threats, the need for specialized expertise, and the historical development of intelligence agencies.

National security threats are often complex and multifaceted, requiring different intelligence capabilities and expertise. As detailed in the following sections, one agency may specialize in human intelligence (HUMINT), while another may focus on signals intelligence (SIGINT) or geospatial intelligence (GEOINT). Each agency brings a unique perspective and expertise to the intelligence landscape, which can help to provide a more comprehensive understanding of national security threats.

In addition, the historical development of intelligence agencies may also contribute to the need for multiple agencies in the same country. Some intelligence agencies may have been established to address specific threats or historical events, and have developed unique capabilities and expertise that are difficult to replicate in other agencies.

Nevertheless, the presence of numerous intelligence agencies can give rise to concerns regarding the coordination and exchange of information, which are vital for the success of intelligence operations. Divergent priorities and interests among these agencies can lead to the creation of information silos and hinder the sharing of crucial intelligence data.

States may bring together intelligence functions of different government services to ensure security around them and forming the so-called intelligence communities (IC), which are networks of government agencies and organisations that are responsible for gathering, analyzing, and disseminating intelligence information to support national security and foreign policy objectives. These communities exist in many countries around the world and are typically led by a central intelligence agency or organisation) [1].

In the United States, the Intelligence Community (IC) is comprised of 18 separate agencies and organisations, including the Central Intelligence Agency (CIA), the National Security Agency (NSA), and the Federal Bureau of Investigation (FBI). These agencies work together to gather and analyze intelligence information and share it with policymakers and decision-makers to support national security objectives.

The European Union (EU) does not have a single intelligence agency or community, as national intelligence agencies in EU member states are responsible for collecting and analyzing intelligence information related to national security and foreign policy objectives. However, there are several institutions and mechanisms in place at the EU level that support intelligence cooperation and information sharing among member states.

Intelligence sharing among EU member states can be traced back to at least the 1970s, although it was