Preface	6
	Acknowledgements	8
	Contents	9
	Editors and Contributors	11
	Introduction	15
	1 OSINT as an Integral Part of the National Security Apparatus	16
	Abstract	16
	1.1 Introduction	16
	1.2 OSINT and Counter Terrorism Strategy	17
	1.3 The CENTRIC OSINT Hub	21
	1.4 Concluding Remarks	22
	References	22
	2 Open Source Intelligence and the Protection of National Security	23
	Abstract	23
	2.1 Introduction	23
	2.2 From Threat to Threat	24
	2.3 Online Radicalisation	26
	2.4 Counter Measures	28
	2.5 Conclusions	30
	References	31
	3 Police Use of Open Source Intelligence: The Longer Arm of Law	32
	Abstract	32
	3.1 Introduction	32
	3.2 Understanding Intelligence in Policing	33
	3.3 Intelligence Collection Disciplines	35
	3.4 Characteristics of Open Source Intelligence	35
	3.5 Modelling Open Source Intelligence	39
	3.6 Conclusions	41
	References	42
	4 OSINT as Part of the Strategic National Security Landscape	43
	Abstract	43
	4.1 Introduction	43
	4.2 Understanding the Strategic Landscape into Which OSINT Must Be Applied	44
	4.3 Understanding the Intelligence Cycle in Which OSINT Must Exist and the Wider Intelligence Mix in Which It Must Integrate	47
	4.3.1 Understanding the Application of OSINT in Operational Decision Making	52
	4.3.2 UK Government Intelligence: Its Nature, Collection, Assessment and Use	53
	4.4 How Might an Overarching Information Governance Architecture Support OSINT for Decision Making Within the Wider Intelligence Mix and Cycle?	58
	4.5 Summary	63
	References	64
	5 Taking Stock of Subjective Narratives Surrounding Modern OSINT	66
	Abstract	66
	5.1 Introduction	66
	5.2 Contextual Background	67
	5.3 Lack of Public Clarity	68
	5.4 Opposing Narratives	69
	5.5 Independent Reviews	71
	5.6 Conclusion	72
	References	73
	Methods, Tools and Techiques	75
	6 Acquisition and Preparation of Data for OSINT Investigations	76
	Abstract	76
	6.1 Introduction	76
	6.2 Reasons and Strategies for Data Collection	78
	6.3 Data Types and Sources	80
	6.3.1 Structured and Unstructured Data	80
	6.3.2 Where and How to Obtain Open Source Data	80
	6.3.2.1 Supporting Manual Searches	81
	6.3.2.2 Web Crawling and Spiders	81
	6.3.2.3 Web Metadata	83
	6.3.2.4 APIs	83
	6.3.2.5 Open Data	84
	6.3.2.6 Social Media	84
	6.3.2.7 Traditional Media	87
	6.3.2.8 RSS	87
	6.3.2.9 Grey Literature	88
	6.3.2.10 Paid Data and Consented Data	88
	6.3.2.11 Data on the Deep and Dark Web	89
	6.4 Information Extraction	90
	6.4.1 Natural Language Processing	90
	6.4.1.1 Main Body Extraction	91
	6.4.1.2 Entity Extraction	93
	6.4.2 Modelling	94
	6.4.2.1 Entity Relation Modelling	94
	6.4.3 Feedback Loops	94
	6.4.4 Validation Processes	95
	6.4.5 Disinformation and Malicious Intent	95
	6.4.6 Software Tools for Data Collection and Preparation	96
	6.5 Privacy and Ethical Issues	97
	6.5.1 Privacy by Design	97
	6.5.2 Being Polite Online	98
	6.5.2.1 Monitor Web Crawls and Respecting robots.txt	98
	6.5.2.2 Keeping to API Limits	98
	6.6 Conclusion	99
	References	99
	7 Analysis, Interpretation and Validation of Open Source Data	101
	Abstract	101
	7.1 Introduction	101
	7.2 Types of Data Analysis	102
	7.2.1 Textual Analysis	102
	7.2.1.1 Text Processing	102
	7.2.1.2 Word Sense Disambiguation	103
	7.2.1.3 Sentiment Analysis	104
	7.2.2 Aggregation	105
	7.2.2.1 Document Clustering	105
	7.2.3 Connecting the Dots	106
	7.2.3.1 Network Analysis	107
	7.2.3.2 Co-occurrence Networks	108
	7.3 Location Resolution	109
	7.3.1 Geocoding	110
	7.3.2 Reverse Geocoding	110
	7.4 Validating Open Source Information	111
	7.4.1 Methods for Assigning Priority	112
	7.4.2 Approaches for Recognising Credibility	113
	7.4.3 Methods for Identifying Corroboration	114
	7.5 Conclusion	114
	References	115
	8 OSINT and the Dark Web	117
	Abstract	117
	8.1 Introduction	117
	8.2 Dark Web	120
	8.2.1 Darknets on the Dark Web	120
	8.2.2 Dark Web Size	124
	8.2.3 Dark Web Content	124
	8.3 OSINT on the Dark Web	126
	8.3.1 Landscape of Dark Web Activities of Investigative Interest	126
	8.3.2 Challenges Faced by LEAs on the Dark Web	128
	8.4 OSINT Techniques on the Dark Web	129
	8.4.1 Crawling	130
	8.4.2 Search Engines	131
	8.4.3 Traffic Analysis and de-Anonymization	132
	8.5 Case Study: HME-Related Information on the Dark Web	133
	8.5.1 Methodology	134
	8.5.2 Experimental Evaluation	135
	8.6 Conclusions	136
	References	137
	9 Fusion of OSINT and Non-OSINT Data	139
	Abstract	139
	9.1 Introduction	139
	9.2 OSINT Data	140
	9.2.1 Geographical Data	140
	9.2.2 Statistical Data	141
	9.2.3 Electoral Register	141
	9.2.4 Court Records	142
	9.2.5 Social Media	142
	9.2.6 Blogging Platforms	142
	9.2.7 Search Engines	143
	9.2.8 Internet Archive	144
	9.2.9 Freedom of Information	144
	9.3 Non-OSINT Data	144
	9.3.1 Criminal Records	145
	9.3.2 Financial Records	146
	9.3.3 Telecommunication Records	147
	9.3.4 Medical Records	148
	9.3.5 Imagery, Sensors and Video Data	149
	9.4 Fusion Opportunities	149
	9.4.1 Targeted Search	150
	9.4.2 Validation of Other ‘INTs’	150
	9.4.3 Filling in the Missing Links	150
	9.4.3.1 Identity Matching	151
	9.4.3.2 Enhanced Social Network Creation	152
	9.4.4 Environmental Scanning	153
	9.4.5 Predictive Policing	154
	9.4.6 Situational Awareness During Major Events	155
	9.4.7 Identification and Tracking of Foreign Fighters	156
	9.4.8 Child Sexual Exploitation	156
	9.5 Conclusions	157
	References	157
	10 Tools for OSINT-Based Investigations	159
	Abstract	159
	10.1 Introduction	159
	10.1.1 Effective Cyber-Risk Management	160
	10.2 Key Assessment Themes	161
	10.2.1 Security	161
	10.2.1.1 Privacy	161
	10.2.1.2 Protecting Against Malware	162
	10.2.1.3 Unnecessary Bundled Software	162
	10.2.1.4 Cloud-Based Services	162
	10.2.2 Reliability	163
	10.2.2.1 Code Quality	163
	10.2.2.2 Open Formats and Standards	163
	10.2.3 Legality	164
	10.2.3.1 Licensing	164
	10.2.3.2 Authorities	165
	10.3 Completing a Tool Review	165
	10.4 Assessment Framework	166
	10.4.1 Document Information	167
	10.4.2 Supplier Assessment	168
	10.4.3 External Assessments	168
	10.4.4 Practitioner’s Assessment	169
	10.5 Conclusion	170
	References	171
	11 Fluidity and Rigour: Addressing the Design Considerations for OSINT Tools and Processes	172
	Abstract	172
	11.1 Introduction	172
	11.2 Intelligence Analysis	175
	11.3 What Do We Design?	177
	11.4 Designing for Fluidity and Rigour	180
	11.4.1 Fluidity as a Design Concept for OSINT Investigations	182
	11.4.2 Rigour as a Design Concept for OSINT Investigations	184
	11.5 Conclusions: Guidance for Designing Analysts’ Tools	187
	Acknowledgments	188
	References	188
	Pratical Application and Cases	191
	12 A New Age of Open Source Investigation: International Examples	192
	Abstract	192
	12.1 Introduction	192
	12.2 Conclusion	198
	References	199
	13 Use Cases and Best Practices for LEAs	200
	Abstract	200
	13.1 Introduction	200
	13.2 OSINT in an Increasingly Digital World	201
	13.3 OSINT Best Practices for LEAs	203
	13.3.1 Absolutes	203
	13.3.2 Exploitables	203
	13.3.3 Information Auditing	205
	13.3.4 Strategic Data Acquisition	205
	13.3.5 OSINT Pitfalls	206
	13.3.5.1 Leakage	206
	13.3.5.2 Anonymization	206
	13.3.5.3 Crowd-Sourcing and Vigilantism	207
	13.3.5.4 Corrupting the Chain of Evidence	207
	13.3.5.5 Source Validation	208
	13.4 LEA Usage of OSINT in Investigations: Case Examples	208
	13.4.1 Exploiting Friendships in an Armed Robbery Case	208
	13.4.2 Locating Wanted People Through Social Media	209
	13.4.3 Locating a Sex Offender	210
	13.4.4 Proactive Investigation Following a Terrorist Attack	211
	13.5 Going Undercover on Social Media	212
	13.6 Conclusions	212
	References	213
	14 OSINT in the Context of Cyber-Security	215
	Abstract	215
	14.1 Introduction	215
	14.2 The Importance of OSINT with a View on Cyber Security	218
	14.3 Cyber Threats: Terminology and Classification	219
	14.4 Cyber-Crime Investigations	220
	14.4.1 Approaches, Methods and Techniques	220
	14.4.2 Detection and Prevention of Cyber Threats	223
	14.5 Conclusions	229
	References	229
	15 Combatting Cybercrime and Sexual Exploitation of Children: An Open Source Toolkit	234
	Abstract	234
	15.1 Introduction	234
	15.2 The Extended Impact of Cybercrime	235
	15.3 Tools for Law Enforcement	237
	15.4 The Role of OSINT	238
	15.5 The UINFC2 Approach	240
	15.5.1 Citizen Reporting Form	240
	15.5.2 LEA/HOTLINE UINFC2 Platform	242
	15.6 Concluding Remarks	248
	Acknowledgments	249
	References	249
	16 Identifying Illegal Cartel Activities from Open Sources	251
	Abstract	251
	16.1 Introduction	252
	16.2 The Principles	254
	16.2.1 The Definition of a Cartel	254
	16.2.2 The Sources of Information	255
	16.2.2.1 Government Procurement Records	257
	16.2.2.2 Company Registry	257
	16.2.2.3 Legal Databases	257
	16.2.2.4 Other Open-Source Intelligence (OSINT) sources	258
	16.2.3 Cartel Patterns	258
	16.2.4 Security Models	260
	16.2.4.1 Negative Security Models and Supervised Learning	260
	16.2.4.2 Positive Security Models and Unsupervised Learning	261
	16.3 Data Acquisition from Open Sources	261
	16.3.1 The Architecture	261
	16.3.2 Entity Extraction	262
	16.3.3 Filtering Out Suspicious Items in the Fusion Centre	262
	16.3.4 Feature Engineering	264
	16.3.5 Fitted Parameters of Economic Models	265
	16.3.6 Network Science and Visualization	265
	16.4 Machine Learning Methodologies	266
	16.4.1 Evaluation of Predictive Methods	267
	16.4.2 Logistic Regression	268
	16.4.3 Decision Trees	269
	16.4.4 Boosting	269
	16.5 Conclusion and Further Work	270
	References	271
	Legal Considerations	274
	17 Legal Considerations for Using Open Source Intelligence in the Context of Cybercrime and Cyberterrorism	275
	Abstract	275
	17.1 Introduction	275
	17.2 Citizens’ Perceptions and Human Rights	276
	17.3 Investigatory Powers	277
	17.3.1 Existing and Proposed Powers	278
	17.3.2 (Un)Lawful Practices	279
	17.4 Data Protection	280
	17.4.1 The Legislation	280
	17.4.2 Further Considerations	282
	17.5 Data Acquisition	283
	17.6 Rules of Evidence	283
	17.6.1 Seizing Digital Evidence	284
	17.7 Unused Material	284
	17.8 Different Jurisdictions	285
	17.9 Overcoming Problems	286
	17.9.1 Europol	286
	17.9.2 Joint Investigation Teams	286
	17.9.3 Eurojust	287
	17.9.4 CEPOL	287
	17.9.5 Interpol	288
	17.10 Summary	288
	17.11 Conclusion	290
	References	291
	18 Following the Breadcrumbs: Using Open Source Intelligence as Evidence in Criminal Proceedings	293
	Abstract	293
	18.1 Introduction	293
	18.2 What Is the Difference Between Intelligence and Evidence?	294
	18.3 Practical Issues	296
	18.4 Legal Framework	296
	18.5 European Convention on Human Rights	297
	18.6 Uses of OSINT as Evidence	299
	18.7 Conclusion	300
	References	300