Open-source Intelligence (OSINT) is an incredibly powerful tool for both the public and private sectors alike. Whether it’s financial institutions conducting sanctions screenings and identifying politically exposed persons or individuals doing due diligence on a property transaction they want to conduct, OSINT has been at the center of these efforts.
Previously we discussed the foundations of OSINT and its benefits, but what are some of the challenges associated with these investigation methods?
As Kent Schneider put it when writing for the Armed Forces Communications and Electronics Association, “The number of countries, governmental agencies and NGOs involved in today’s [OSINT] operations creates a huge issue of coordination; where to consolidate information; how to get it there; and who needs to be involved in dissemination.” This was written back in 2014 and since then the number and type of actors in the OSINT space as well as the amount of data to be searched has increased substantially.
The notion of a “needle in a haystack” is taken to the extreme on the internet. According to statistical analysis, the worldwide surface web hosts approximately 6 billion websites. This by itself seems like a large haystack. However, this figure does not account for the deep web, the unlisted portion of the internet, which is estimated to be 400 to 550 times as large. All of this information is vital for OSINT operations, but being able to efficiently wade through all that information is a daunting task. Developing an intelligence gathering plan, search parameters and other controls makes effective OSINT a challenging feat.
As with any task, it is important to come at it with the correct tools. Once the hurdle of developing a plan has been overcome, a further challenge of OSINT is to find the right tools to leverage. Such tools can be relatively simple, such as theHarvester, which will aggregate search results across many different search engines including Google, Yahoo, Bing as well as lesser known ones. Others like Metagoofil will assist in extracting metadata from various document formats, making otherwise impossible keyword searches a reality. There are also more software solutions that can provide users with everything from powerful network analysis capabilities to source code search engines. However, these can have steep learning curves or hefty price tags – or both.
Finally, the final challenge of OSINT is one related to the searcher’s security. Not everyone on the internet wants to have their information found and ensuring that online attribution is properly managed can protect against the many forms of commonly used malicious retaliation against internet sleuths.
While there are numerous challenges in the investigative process during the digital age, that does not negate the value of the data that is out there. As Sigma continues to improve these skills and dive into the haystack of open-source data, there are growing opportunities to provide offerings that can harness the power of OSINT for our clients while effectively managing the myriad challenges.