|
Preface |
6 |
|
|
Acknowledgements |
8 |
|
|
Contents |
9 |
|
|
Editors and Contributors |
11 |
|
|
Introduction |
15 |
|
|
1 OSINT as an Integral Part of the National Security Apparatus |
16 |
|
|
Abstract |
16 |
|
|
1.1 Introduction |
16 |
|
|
1.2 OSINT and Counter Terrorism Strategy |
17 |
|
|
1.3 The CENTRIC OSINT Hub |
21 |
|
|
1.4 Concluding Remarks |
22 |
|
|
References |
22 |
|
|
2 Open Source Intelligence and the Protection of National Security |
23 |
|
|
Abstract |
23 |
|
|
2.1 Introduction |
23 |
|
|
2.2 From Threat to Threat |
24 |
|
|
2.3 Online Radicalisation |
26 |
|
|
2.4 Counter Measures |
28 |
|
|
2.5 Conclusions |
30 |
|
|
References |
31 |
|
|
3 Police Use of Open Source Intelligence: The Longer Arm of Law |
32 |
|
|
Abstract |
32 |
|
|
3.1 Introduction |
32 |
|
|
3.2 Understanding Intelligence in Policing |
33 |
|
|
3.3 Intelligence Collection Disciplines |
35 |
|
|
3.4 Characteristics of Open Source Intelligence |
35 |
|
|
3.5 Modelling Open Source Intelligence |
39 |
|
|
3.6 Conclusions |
41 |
|
|
References |
42 |
|
|
4 OSINT as Part of the Strategic National Security Landscape |
43 |
|
|
Abstract |
43 |
|
|
4.1 Introduction |
43 |
|
|
4.2 Understanding the Strategic Landscape into Which OSINT Must Be Applied |
44 |
|
|
4.3 Understanding the Intelligence Cycle in Which OSINT Must Exist and the Wider Intelligence Mix in Which It Must Integrate |
47 |
|
|
4.3.1 Understanding the Application of OSINT in Operational Decision Making |
52 |
|
|
4.3.2 UK Government Intelligence: Its Nature, Collection, Assessment and Use |
53 |
|
|
4.4 How Might an Overarching Information Governance Architecture Support OSINT for Decision Making Within the Wider Intelligence Mix and Cycle? |
58 |
|
|
4.5 Summary |
63 |
|
|
References |
64 |
|
|
5 Taking Stock of Subjective Narratives Surrounding Modern OSINT |
66 |
|
|
Abstract |
66 |
|
|
5.1 Introduction |
66 |
|
|
5.2 Contextual Background |
67 |
|
|
5.3 Lack of Public Clarity |
68 |
|
|
5.4 Opposing Narratives |
69 |
|
|
5.5 Independent Reviews |
71 |
|
|
5.6 Conclusion |
72 |
|
|
References |
73 |
|
|
Methods, Tools and Techiques |
75 |
|
|
6 Acquisition and Preparation of Data for OSINT Investigations |
76 |
|
|
Abstract |
76 |
|
|
6.1 Introduction |
76 |
|
|
6.2 Reasons and Strategies for Data Collection |
78 |
|
|
6.3 Data Types and Sources |
80 |
|
|
6.3.1 Structured and Unstructured Data |
80 |
|
|
6.3.2 Where and How to Obtain Open Source Data |
80 |
|
|
6.3.2.1 Supporting Manual Searches |
81 |
|
|
6.3.2.2 Web Crawling and Spiders |
81 |
|
|
6.3.2.3 Web Metadata |
83 |
|
|
6.3.2.4 APIs |
83 |
|
|
6.3.2.5 Open Data |
84 |
|
|
6.3.2.6 Social Media |
84 |
|
|
6.3.2.7 Traditional Media |
87 |
|
|
6.3.2.8 RSS |
87 |
|
|
6.3.2.9 Grey Literature |
88 |
|
|
6.3.2.10 Paid Data and Consented Data |
88 |
|
|
6.3.2.11 Data on the Deep and Dark Web |
89 |
|
|
6.4 Information Extraction |
90 |
|
|
6.4.1 Natural Language Processing |
90 |
|
|
6.4.1.1 Main Body Extraction |
91 |
|
|
6.4.1.2 Entity Extraction |
93 |
|
|
6.4.2 Modelling |
94 |
|
|
6.4.2.1 Entity Relation Modelling |
94 |
|
|
6.4.3 Feedback Loops |
94 |
|
|
6.4.4 Validation Processes |
95 |
|
|
6.4.5 Disinformation and Malicious Intent |
95 |
|
|
6.4.6 Software Tools for Data Collection and Preparation |
96 |
|
|
6.5 Privacy and Ethical Issues |
97 |
|
|
6.5.1 Privacy by Design |
97 |
|
|
6.5.2 Being Polite Online |
98 |
|
|
6.5.2.1 Monitor Web Crawls and Respecting robots.txt |
98 |
|
|
6.5.2.2 Keeping to API Limits |
98 |
|
|
6.6 Conclusion |
99 |
|
|
References |
99 |
|
|
7 Analysis, Interpretation and Validation of Open Source Data |
101 |
|
|
Abstract |
101 |
|
|
7.1 Introduction |
101 |
|
|
7.2 Types of Data Analysis |
102 |
|
|
7.2.1 Textual Analysis |
102 |
|
|
7.2.1.1 Text Processing |
102 |
|
|
7.2.1.2 Word Sense Disambiguation |
103 |
|
|
7.2.1.3 Sentiment Analysis |
104 |
|
|
7.2.2 Aggregation |
105 |
|
|
7.2.2.1 Document Clustering |
105 |
|
|
7.2.3 Connecting the Dots |
106 |
|
|
7.2.3.1 Network Analysis |
107 |
|
|
7.2.3.2 Co-occurrence Networks |
108 |
|
|
7.3 Location Resolution |
109 |
|
|
7.3.1 Geocoding |
110 |
|
|
7.3.2 Reverse Geocoding |
110 |
|
|
7.4 Validating Open Source Information |
111 |
|
|
7.4.1 Methods for Assigning Priority |
112 |
|
|
7.4.2 Approaches for Recognising Credibility |
113 |
|
|
7.4.3 Methods for Identifying Corroboration |
114 |
|
|
7.5 Conclusion |
114 |
|
|
References |
115 |
|
|
8 OSINT and the Dark Web |
117 |
|
|
Abstract |
117 |
|
|
8.1 Introduction |
117 |
|
|
8.2 Dark Web |
120 |
|
|
8.2.1 Darknets on the Dark Web |
120 |
|
|
8.2.2 Dark Web Size |
124 |
|
|
8.2.3 Dark Web Content |
124 |
|
|
8.3 OSINT on the Dark Web |
126 |
|
|
8.3.1 Landscape of Dark Web Activities of Investigative Interest |
126 |
|
|
8.3.2 Challenges Faced by LEAs on the Dark Web |
128 |
|
|
8.4 OSINT Techniques on the Dark Web |
129 |
|
|
8.4.1 Crawling |
130 |
|
|
8.4.2 Search Engines |
131 |
|
|
8.4.3 Traffic Analysis and de-Anonymization |
132 |
|
|
8.5 Case Study: HME-Related Information on the Dark Web |
133 |
|
|
8.5.1 Methodology |
134 |
|
|
8.5.2 Experimental Evaluation |
135 |
|
|
8.6 Conclusions |
136 |
|
|
References |
137 |
|
|
9 Fusion of OSINT and Non-OSINT Data |
139 |
|
|
Abstract |
139 |
|
|
9.1 Introduction |
139 |
|
|
9.2 OSINT Data |
140 |
|
|
9.2.1 Geographical Data |
140 |
|
|
9.2.2 Statistical Data |
141 |
|
|
9.2.3 Electoral Register |
141 |
|
|
9.2.4 Court Records |
142 |
|
|
9.2.5 Social Media |
142 |
|
|
9.2.6 Blogging Platforms |
142 |
|
|
9.2.7 Search Engines |
143 |
|
|
9.2.8 Internet Archive |
144 |
|
|
9.2.9 Freedom of Information |
144 |
|
|
9.3 Non-OSINT Data |
144 |
|
|
9.3.1 Criminal Records |
145 |
|
|
9.3.2 Financial Records |
146 |
|
|
9.3.3 Telecommunication Records |
147 |
|
|
9.3.4 Medical Records |
148 |
|
|
9.3.5 Imagery, Sensors and Video Data |
149 |
|
|
9.4 Fusion Opportunities |
149 |
|
|
9.4.1 Targeted Search |
150 |
|
|
9.4.2 Validation of Other ‘INTs’ |
150 |
|
|
9.4.3 Filling in the Missing Links |
150 |
|
|
9.4.3.1 Identity Matching |
151 |
|
|
9.4.3.2 Enhanced Social Network Creation |
152 |
|
|
9.4.4 Environmental Scanning |
153 |
|
|
9.4.5 Predictive Policing |
154 |
|
|
9.4.6 Situational Awareness During Major Events |
155 |
|
|
9.4.7 Identification and Tracking of Foreign Fighters |
156 |
|
|
9.4.8 Child Sexual Exploitation |
156 |
|
|
9.5 Conclusions |
157 |
|
|
References |
157 |
|
|
10 Tools for OSINT-Based Investigations |
159 |
|
|
Abstract |
159 |
|
|
10.1 Introduction |
159 |
|
|
10.1.1 Effective Cyber-Risk Management |
160 |
|
|
10.2 Key Assessment Themes |
161 |
|
|
10.2.1 Security |
161 |
|
|
10.2.1.1 Privacy |
161 |
|
|
10.2.1.2 Protecting Against Malware |
162 |
|
|
10.2.1.3 Unnecessary Bundled Software |
162 |
|
|
10.2.1.4 Cloud-Based Services |
162 |
|
|
10.2.2 Reliability |
163 |
|
|
10.2.2.1 Code Quality |
163 |
|
|
10.2.2.2 Open Formats and Standards |
163 |
|
|
10.2.3 Legality |
164 |
|
|
10.2.3.1 Licensing |
164 |
|
|
10.2.3.2 Authorities |
165 |
|
|
10.3 Completing a Tool Review |
165 |
|
|
10.4 Assessment Framework |
166 |
|
|
10.4.1 Document Information |
167 |
|
|
10.4.2 Supplier Assessment |
168 |
|
|
10.4.3 External Assessments |
168 |
|
|
10.4.4 Practitioner’s Assessment |
169 |
|
|
10.5 Conclusion |
170 |
|
|
References |
171 |
|
|
11 Fluidity and Rigour: Addressing the Design Considerations for OSINT Tools and Processes |
172 |
|
|
Abstract |
172 |
|
|
11.1 Introduction |
172 |
|
|
11.2 Intelligence Analysis |
175 |
|
|
11.3 What Do We Design? |
177 |
|
|
11.4 Designing for Fluidity and Rigour |
180 |
|
|
11.4.1 Fluidity as a Design Concept for OSINT Investigations |
182 |
|
|
11.4.2 Rigour as a Design Concept for OSINT Investigations |
184 |
|
|
11.5 Conclusions: Guidance for Designing Analysts’ Tools |
187 |
|
|
Acknowledgments |
188 |
|
|
References |
188 |
|
|
Pratical Application and Cases |
191 |
|
|
12 A New Age of Open Source Investigation: International Examples |
192 |
|
|
Abstract |
192 |
|
|
12.1 Introduction |
192 |
|
|
12.2 Conclusion |
198 |
|
|
References |
199 |
|
|
13 Use Cases and Best Practices for LEAs |
200 |
|
|
Abstract |
200 |
|
|
13.1 Introduction |
200 |
|
|
13.2 OSINT in an Increasingly Digital World |
201 |
|
|
13.3 OSINT Best Practices for LEAs |
203 |
|
|
13.3.1 Absolutes |
203 |
|
|
13.3.2 Exploitables |
203 |
|
|
13.3.3 Information Auditing |
205 |
|
|
13.3.4 Strategic Data Acquisition |
205 |
|
|
13.3.5 OSINT Pitfalls |
206 |
|
|
13.3.5.1 Leakage |
206 |
|
|
13.3.5.2 Anonymization |
206 |
|
|
13.3.5.3 Crowd-Sourcing and Vigilantism |
207 |
|
|
13.3.5.4 Corrupting the Chain of Evidence |
207 |
|
|
13.3.5.5 Source Validation |
208 |
|
|
13.4 LEA Usage of OSINT in Investigations: Case Examples |
208 |
|
|
13.4.1 Exploiting Friendships in an Armed Robbery Case |
208 |
|
|
13.4.2 Locating Wanted People Through Social Media |
209 |
|
|
13.4.3 Locating a Sex Offender |
210 |
|
|
13.4.4 Proactive Investigation Following a Terrorist Attack |
211 |
|
|
13.5 Going Undercover on Social Media |
212 |
|
|
13.6 Conclusions |
212 |
|
|
References |
213 |
|
|
14 OSINT in the Context of Cyber-Security |
215 |
|
|
Abstract |
215 |
|
|
14.1 Introduction |
215 |
|
|
14.2 The Importance of OSINT with a View on Cyber Security |
218 |
|
|
14.3 Cyber Threats: Terminology and Classification |
219 |
|
|
14.4 Cyber-Crime Investigations |
220 |
|
|
14.4.1 Approaches, Methods and Techniques |
220 |
|
|
14.4.2 Detection and Prevention of Cyber Threats |
223 |
|
|
14.5 Conclusions |
229 |
|
|
References |
229 |
|
|
15 Combatting Cybercrime and Sexual Exploitation of Children: An Open Source Toolkit |
234 |
|
|
Abstract |
234 |
|
|
15.1 Introduction |
234 |
|
|
15.2 The Extended Impact of Cybercrime |
235 |
|
|
15.3 Tools for Law Enforcement |
237 |
|
|
15.4 The Role of OSINT |
238 |
|
|
15.5 The UINFC2 Approach |
240 |
|
|
15.5.1 Citizen Reporting Form |
240 |
|
|
15.5.2 LEA/HOTLINE UINFC2 Platform |
242 |
|
|
15.6 Concluding Remarks |
248 |
|
|
Acknowledgments |
249 |
|
|
References |
249 |
|
|
16 Identifying Illegal Cartel Activities from Open Sources |
251 |
|
|
Abstract |
251 |
|
|
16.1 Introduction |
252 |
|
|
16.2 The Principles |
254 |
|
|
16.2.1 The Definition of a Cartel |
254 |
|
|
16.2.2 The Sources of Information |
255 |
|
|
16.2.2.1 Government Procurement Records |
257 |
|
|
16.2.2.2 Company Registry |
257 |
|
|
16.2.2.3 Legal Databases |
257 |
|
|
16.2.2.4 Other Open-Source Intelligence (OSINT) sources |
258 |
|
|
16.2.3 Cartel Patterns |
258 |
|
|
16.2.4 Security Models |
260 |
|
|
16.2.4.1 Negative Security Models and Supervised Learning |
260 |
|
|
16.2.4.2 Positive Security Models and Unsupervised Learning |
261 |
|
|
16.3 Data Acquisition from Open Sources |
261 |
|
|
16.3.1 The Architecture |
261 |
|
|
16.3.2 Entity Extraction |
262 |
|
|
16.3.3 Filtering Out Suspicious Items in the Fusion Centre |
262 |
|
|
16.3.4 Feature Engineering |
264 |
|
|
16.3.5 Fitted Parameters of Economic Models |
265 |
|
|
16.3.6 Network Science and Visualization |
265 |
|
|
16.4 Machine Learning Methodologies |
266 |
|
|
16.4.1 Evaluation of Predictive Methods |
267 |
|
|
16.4.2 Logistic Regression |
268 |
|
|
16.4.3 Decision Trees |
269 |
|
|
16.4.4 Boosting |
269 |
|
|
16.5 Conclusion and Further Work |
270 |
|
|
References |
271 |
|
|
Legal Considerations |
274 |
|
|
17 Legal Considerations for Using Open Source Intelligence in the Context of Cybercrime and Cyberterrorism |
275 |
|
|
Abstract |
275 |
|
|
17.1 Introduction |
275 |
|
|
17.2 Citizens’ Perceptions and Human Rights |
276 |
|
|
17.3 Investigatory Powers |
277 |
|
|
17.3.1 Existing and Proposed Powers |
278 |
|
|
17.3.2 (Un)Lawful Practices |
279 |
|
|
17.4 Data Protection |
280 |
|
|
17.4.1 The Legislation |
280 |
|
|
17.4.2 Further Considerations |
282 |
|
|
17.5 Data Acquisition |
283 |
|
|
17.6 Rules of Evidence |
283 |
|
|
17.6.1 Seizing Digital Evidence |
284 |
|
|
17.7 Unused Material |
284 |
|
|
17.8 Different Jurisdictions |
285 |
|
|
17.9 Overcoming Problems |
286 |
|
|
17.9.1 Europol |
286 |
|
|
17.9.2 Joint Investigation Teams |
286 |
|
|
17.9.3 Eurojust |
287 |
|
|
17.9.4 CEPOL |
287 |
|
|
17.9.5 Interpol |
288 |
|
|
17.10 Summary |
288 |
|
|
17.11 Conclusion |
290 |
|
|
References |
291 |
|
|
18 Following the Breadcrumbs: Using Open Source Intelligence as Evidence in Criminal Proceedings |
293 |
|
|
Abstract |
293 |
|
|
18.1 Introduction |
293 |
|
|
18.2 What Is the Difference Between Intelligence and Evidence? |
294 |
|
|
18.3 Practical Issues |
296 |
|
|
18.4 Legal Framework |
296 |
|
|
18.5 European Convention on Human Rights |
297 |
|
|
18.6 Uses of OSINT as Evidence |
299 |
|
|
18.7 Conclusion |
300 |
|
|
References |
300 |
|