What is OSINT (Open Source Intelligence)?
Notice of Non-Affiliation and Disclaimer: Sunny Valley Network is not affiliated with, and does not control or endorse external links in any way. The tools, websites or information provided here are to be used at your own risk. We suggest that those interested in using these resources do their due diligence and assess them carefully and ethically before use.Sunny Valley Networks takes no responsibility for use of these tools for illegitimate purposes
Open source intelligence (OSINT) is likely the most extensively used threat intelligence subcategory, which makes sense. After all, who can say no to something that is basically free?
Regrettably, open source intelligence, like the other key categories — human intelligence, signals intelligence, and geographic intelligence, to name a few — is commonly misunderstood and misapplied.
We'll go through the basics of open source intelligence in this article, along with how it's utilized and the OSINT tools and techniques that may be used to collect and analyze it.
Figure 1. Open source intelligence (OSINT)
The intelligence produced from publicly available information and collected, exploited, and distributed in a timely manner to an appropriate audience for the purpose of answering a specific intelligence demand is known as open-source intelligence (OSINT).
What does OSINT Mean?
Open source intelligence (OSINT) refers to any information about an individual or organization that can be legally collected from free, public sources. In practice, this usually refers to material found on the internet, but theoretically, any public information, whether it's books or reports in a public library, articles in a newspaper, or statements in a press release, falls under the category of OSINT.
OSINT also covers information that can be found through other forms of media. Though we usually conceive of information as being text-based, it also includes graphics, videos, webinars, public speeches, and conferences.
What is OSINT Used For?
OSINT was developed outside of the cybersecurity industry to refer to military and intelligence attempts to collect strategically essential but publicly available information in national security flaws. While post war spy operations centered on various methods of obtaining information (e.g., HUMINT, SIGINT), OSINT was revived in the 1980s. With the advent of the internet, social media, and digital services, OSINT actors now have access to a vast amount of information about an organization's IT infrastructure as well as its workers.
Open source intelligence is used by security professionals to discover possible vulnerabilities in friendly networks so that they can be addressed before they are exploited by threat actors. The following are some of the most commonly discovered flaws:
- Leaks of critical information by accident, such as on social media
- Unsecured internet-connected gadgets or open ports
- Unpatched software, such as websites running obsolete versions of popular CMS programs, is a major security risk.
- Assets that have been leaked or exposed, such as proprietary code on pastebins
The example of an open source intelligence is that it is frequently used in conjunction with other intelligence subtypes. Closed source intelligence, such as internal telemetry, closed dark web communities, and external intelligence-sharing communities, is frequently used to filter and verify open source intelligence. A variety of tools are available to assist analysts in performing these functions.
Who would use OSINT?
OSINT is like a knife: you may use it for evil purposes or you may use it for good purposes. So let's mention at briefly who would use OSINT
The major users of OSINT are government agencies, particularly military departments. Governments require OSINT for a variety of reasons, including national security, counter terrorism, terrorist cyber tracking, understanding domestic and foreign public opinion on various topics, providing policymakers with necessary information to influence their internal and external policies, and obtaining translations of various events from foreign media. Intelligence agencies use OSINT to track events, equipment such as weapons systems, and people. These are the 'targets of interest' (ToIs). Also like government constitutions, companies, cyber security groups, terrorist groups use OSINT as a tool.
OSINT is used by organizations like the United Nations to support peacekeeping activities all over the world. OSINT is used by humanitarian organizations such as the International Red Cross to aid in relief efforts in times of crisis or tragedy. They use OSINT intelligence to protect their supply chain from terrorist groups by analyzing social media and messaging sites in order to predict future terrorist acts.
OSINT is used by the police to safeguard citizens from abuse, sexual violence, identity theft, and other crimes. It is often done by monitoring social media channels for intriguing keywords and images in order to help prevent crimes before they become more serious.
Businesses use OSINT to study new markets, watch their competitors' actions, plan marketing initiatives, and foresee anything that could have an impact on their operations and jeopardize their future growth. Businesses also employ OSINT intelligence for non-financial reasons, such as preventing data leakage, because knowing that a company is disclosing confidential information and network security flaws before the bad guys is invaluable.
Businesses can also use OSINT to develop cyber threat intelligence (CTI) strategies by analyzing OSINT sources from both inside and outside the organization. They combine that data with other data to create an effective cyber-risk management policy that protects their financial interests, reputation, and customer base.
Cybersecurity and Cybercrime Groups
Hackers and penetration testers utilize OSINT to obtain information about a specific target over the internet. It's also regarded as a useful tool for assisting in social engineering attacks. Reconnaissance is the initial step in any penetration testing approach (in other words, with OSINT).
The Privacy Conscious
These are ordinary folks who want to know how strangers can hack into their computers and what information their internet service provider has on them. They may also want to know how exposed they are online in order to plug any security holes and erase any personal information that has been exposed inadvertently. OSINT is a fantastic tool for seeing how your digital identity appears to the rest of the world while maintaining your privacy. Individuals can utilize OSINT to protect themselves from identity theft.
Terrorists use OSINT to plan attacks, gather information about targets before attacking them (for example, by using Google Maps to investigate locations), groom fighters by analyzing social media sites, obtain military information accidentally revealed by governments (for example, how to build bombs), and spread propaganda.
Exploiting open source data has a long history, dating back to the early days of intelligence as a tool for supporting a government's decisions and actions.
In 1939 ,The British government requested that the British Transmission Corporation (BBC) develop a civilian, and then commercial, service to monitor overseas journalism and radio broadcasting. OSINT’s first example starts with the BBC. However, before the Foreign Broadcast Monitoring Service was founded (1946), it was not a systematic approach. FBMS was renamed The Foreign Broadcast Intelligence Service (FBIS) in 1947. Following the 9/11 attacks and the enactment of the Intelligence Reform and Terrorism Prevention Act in 2005, FBIS was renamed the Director of National Intelligence's Open Source Center, along with other research parts (OSC). Since its establishment, OSINT has been responsible for filtering, translating, translating and translating (thus interpreting) news items and information that comes from abroad.
Publicly available information has had a huge impact on every element of modern political, social, and economic life because of Internet technology. However, it is important to remember that the Internet is a method of transporting information and a virtual place, not a source (save for its meta data). OSINT has grown in importance as a source for combining relevant intelligence into valuable products.
Advantages of Using OSINT
The following are some of the benefits of using OSINT tactics and methods:
Regular data collection tools and approaches may prove to be too costly for smaller or independent businesses. OSINT requires little to no financial investment because the information is, by definition, free.
It is Legal
It is perfectly lawful to collect any data you may find because the material acquired is not classified and has been publicly released with the approval of the original source.
Updated On A Regular Basis
Because the resources used in OSINT are public, individuals are likely to upload and update their information on a regular basis.
National Safety and Security
In dealing with national security issues, OSINT has shown to be a very effective instrument.
A Broader Perspective
Using OSINT information, business owners and other corporate decision-makers can acquire a larger view of their investigations, helping them to develop long-term strategic plans to achieve a variety of commercial objectives.
Disadvantages of OSINT
OSINT, on the other hand, has its limitations and some drawbacks.
One of the most serious issues with OSINT is the risk of information overload; separating insight from "noise" can be challenging. Finding and looking for the relevant information can be time-consuming without the use of essential OSINT tools.
Require human analysis
OSINT is also not ready to use; it requires a significant amount of human analysis to identify true, confirmed material from false, misleading, or just incorrect news and information. Validation of OSINT is required.
A deep examination and understanding of the requirements for employing OSINT are required to get the most out of it.
Furthermore, choosing OSINT over traditional intelligence should not be predicated on cost, because OSINT does not preclude traditional intelligence obtained from classified sources. For business intelligence, combining OSINT with traditional intelligence sources is a powerful method.
What is the OSINT Framework?
OSINT Framework is a cybersecurity framework that includes a set of OSINT technologies to make gathering intelligence and data easier.
Security researchers and penetration testers mostly utilize this program for digital footprinting, OSINT research, intelligence collecting, and reconnaissance.
It has a simple web-based interface that allows you to browse several OSINT tools that are categorized.
It also provides a good classification of all existing intel sources, making it a useful tool for determining which infosec areas you're overlooking or what OSINT steps your inquiry should take next.
The OSINT Framework is divided into categories depending on various themes and objectives. When looking at the OSINT tree available through the web interface, this is very clear.
Figure 2. OSINT Framework
When you visit
https://www.osintframework.com, you'll notice the OSINT tree on the left side of your screen.
There are a few highlights you should be aware of; for some of the listed tools, look at the indications on the right side:
( T ) – Indicates a link to a program that must be downloaded and installed locally
( D ) – Google Dork (also known as Google Hacking)
( R ) – Requires registration
( M ) – Indicates a URL that contains the search phrase but must be manually altered.
We can give two examples of OSINT, for educational purposes, to make the subject more clearer. Do not use the tools explained in this article for illegal purposes. Sunny Valley Networks takes no responsibility for use of this tool for illegitimate purposes.
- Locate the Center for Missing Persons Investigations.
Locate Centre for Missing People Investigations is a non-profit organization dedicated to assisting families of missing people whose cases have gone unsolved. Another non-profit organization similar to the first one is Trace Labs which is a non-profit organization whose purpose is to speed up the reuniting of missing individuals' families while also training members in open source intelligence tradecraft (OSINT).
- Technical examination of the target website
OSINT can be used for technical examination of target websites. We can search for vulnerabilities that target these components (especially zero-day vulnerabilities) by knowing the type of programming language, web frameworks, and content management system (CMS) used to create the target website, and then work to exploit any of these vulnerabilities as soon as they are discovered.
How Is Open Source Intelligence Used in Cybersecurity?
OSINT is used to assess, monitor, and track cyber threats that may represent a risk to your company; one of its most useful qualities is that it does not involve any direct interaction with potential threat actors and allows for data collection from afar.
It's critical not to jeopardize your own security while investigating a threat. Using the operational security methodology, or OPSEC, can assist you avoid leaving a trace or disclosing that you're following someone. The OPSEC measures you employ are determined by how you gather data. When conducting OSINT research, it is critical to use some type of OPSEC.
OSINT can be used to identify possible threats and cyber attackers, as well as to determine whether publicly available information about your company puts your company at risk of cyber attacks. You can remove publicly available data that may undermine your organization's security from a public forum once you've been made aware of it. You can also use the data gathered from OSINT research to spot possible dangers and prevent cyber assaults before they happen.
How to use OSINT Framework?
OSINT framework presents you with the information classified into many entity groups.
Figure 3. OSINT Framework
For example, when you want to search for a user name,
- Click on the Username entity and if there is an additional search option for the user (name or person), you can select the relevant one
- Then you can find all the sites where you can access the information in that entity.
- When you click on the site you want to search for a username, the framework directs you to the site.
What are the Best OSINT Tools?
The free OSINT tools listed below are commonly used by penetration testers, social engineers, and security researchers for their various projects.
Shodan is a Google-like search engine. Shodan, on the other hand, searches for internet-connected devices, whereas Google looks for web pages.
Figure 4. Shodan.io
The Shodan search engine allows users to conduct searches based on an IP address, device name, city, and/or a range of other technical criteria. Shodan returns results that are more relevant to security professionals and make more sense. This open-source program aids the security analyst in identifying the target and testing it for various vulnerabilities, passwords, services, ports, and other factors. Users can create free accounts, but they are severely limited - Shodan's free service only allows for 50 search results.
Figure 5. Recon-ng
Recon-ng is a Python tool that is mostly used for information gathering due to its independent modules, keys list, and other modules. This tool comes preloaded with a plethora of modules that make use of online search engines, plugins, and API to aid in the gathering of target information.
3. Google Dorks
Figure 6. Google hacking database (GHDB)
Google dorking, often known as Google hacking, is a hacking technique that makes use of Google Search and other Google tools to find security holes in a website's settings and machine code. Using customized Google search engine operators to detect unique text sequences inside search results is known as "Google hacking."
4. The Harvester
Another OSINT tool for reconnaissance is the harvester. It gathers information from various sources to help us determine the company's perimeter. The Harvester collects emails, subdomains, IP addresses, and URLs. Kali Linux comes with the harvester pre-installed.
Maltego is a free and open-source intelligence and forensics tool. Information may be mined and gathered in a timely manner, and presented in an easy-to-understand way.
In addition to aggregating information from all over the internet, Maltego can locate, aggregate, and visualize information such as the current router configuration on your network or the current whereabouts of your Vice President on his overseas trips.
As a result of Maltego, you can enumerate network and domain information such as:
- Domain Names and Whois Information; DNS Names, Netblocks, IP Addresses, etc.
- Tags and phrases can be searched for in blogs.
- Identify incoming links for websites
- Email addresses associated with a person's name
- Web sites related with a person's name
Maltego additionally allows you to enumerate People information such as:
- Telephone numbers linked to a person's name
- Social groups linked to a person's name
- Companies and organizations linked to a person's name, and so on
Are OSINT tools Legal?
While criminal hackers frequently utilize OSINT tactics as reconnaissance before launching an illegal attack, the tools and techniques are, for the most part, perfectly legal—after all, they're designed to help you zero in on material that's been published or otherwise made public. Even government entities are urged to use OSINT techniques to find security flaws in their own systems.
In response to the growing importance of OSINT research in the business sector, the US Department of Justice recently established guidelines for acquiring online information and purchasing data from illicit sources. The guidelines only apply to cyber threat information collection on Dark Web forums and markets, as well as cybersecurity experts' purchases of data from illegal sources.
- Accessing the Dark Web for passive research, including illicit forums and markets where criminal activity is discussed or carried out, is not prohibited if there is no criminal intent.
- Review the site's terms of service and think about your legal responsibilities.
- Create fictitious identities, but don't suggest that they have any unique significance (such as a government official).
- Don't use someone else's credentials or impersonate them without their permission.
- Don't use a weakness or a "hack" to gain access to the data. Your good intentions may not be enough to protect you from prosecution if your organization does something wrong.
- Don't contact strangers on the Dark Web unless your company has carefully examined whether the risks outweigh the benefits. This puts you at risk of serious harm from evil intent. Don't let yourself become a victim.
- Information that could be used to conduct a crime should not be shared. When engaging with persons on the Dark Web, security professionals should exercise extreme caution.
- Don't buy data that doesn't belong to you or tools that you know are illegal. Do not trade with cybercriminals on the Dark Web unless your company has carefully studied the legal ramifications.
- Keep a record, such as screen shots, to utilize as an audit trail in case your cybersecurity team's involvement on illicit forums is scrutinized.
- Make specific organizational standards for acquiring cybersecurity intelligence. Consider your own business objectives, legal duties, and the terms of service of the websites you're utilizing. It's crucial to measure all of this against public opinion, especially for specific types of organizations.
Read the full DOJ report “Legal Considerations when Gathering Online Cyber Threat Intelligence and Purchasing Data from Illicit Sources“
What are OSINT Techniques?
There are two types of techniques for gathering and analysing open source data: passive collection and active collection.
Threat intelligence platforms (TIPs) are frequently used in passive collection to integrate a multitude of threat feeds into a single, easily accessible area. While this is a huge improvement over manual intelligence gathering, the potential of information overload remains. Advanced threat intelligence solutions overcome this challenge by automating the process of selecting and rejecting warnings depending on an organization's specific needs utilizing artificial intelligence, machine learning, and natural language processing.
Organized threat groups, meanwhile, frequently use botnets to capture important information by employing techniques such as traffic sniffing and keylogging.
Figure 7. Some passive reconnaissance tools
Active collection, on the other hand, is the application of a range of strategies to find specific insights or information. This type of collecting effort is typically done by security specialists for one of two reasons:
- A passively collected warning has identified a possible threat, and more information is needed.
- An intelligence collecting activity, such as a penetration testing exercise, has a very precise goal.
What is OSINT Tools Linux?
OSINT tools, like other open source tools, come in a wide range. Some are well-kept, while others are one-off creations of their creator. It's customary to use several tools depending on the sort of information and data artifacts you're looking for. Using numerous techniques to enrich the retrieved data as much as feasible may be necessary at times. Here are several OSINT tools for Linux that you can utilize.
Figure 8. OSINT tools on Linux
DataSploit is a framework for acquiring intelligence about a target, such as credentials, domain information, and other data. It employs a variety of tactics to gather information on businesses, individuals, phone numbers, and even cryptocoin technology. It has the capability of gathering all raw data and returning it in a variety of formats.
Penetration Testers and Red Teams who want to gather as much information as possible about a target client can use GasMask. The most important phase in discovering preliminary knowledge on the systems, their software, and the individuals involved with the target is acquiring information.
Figure 9. Gasmask tool
Gitem is a GitHub reconnaissance tool that extracts data about organizations. It can be used to track down sensitive data leaks.
Intrigue Core is a framework for analyzing an environment's attack surface. This includes identifying infrastructure and applications, conducting security research, and identifying vulnerabilities.
Intrigue also allows you to augment existing data and do OSINT research (open source intelligence). DNS subdomain brute-forcing, email harvesting, IP geolocation, port scanning, and employing public search engines like Censys, Shodan, and Bing are among the linked scans.
OSINT-SPY is a modular program that can be used to query data on a variety of topics such as an IP address, domain, email address, or even a Bitcoin address. During the reconnaissance phase of a penetration test, this tool can be extremely useful. It can also be used for defense purposes, such as determining what information about your business and its assets is publicly available.
What is OSINT Tools Email?
At some point in most OSINT investigations, an e-mail address is used. Some people begin with nothing more than an e-mail. E-mail addresses can be difficult to find at times, but they can also provide a lot of information about a topic. The rest of this essay will go over a variety of tools and approaches for getting the most out of an email address.
Figure 10. OSINT tools for Emails
The amount of data available for a specific email address varies greatly. This is dependent on a variety of criteria, including the age of the e-mail address, how extensively the owner has publicized it on the internet, and if the provider is a common email provider like Gmail or if the email address is related to the company domain
The following websites may help you to harvest the email addresses.
- Hunter. io
- View DNS Info
Some other prominent OSINT tools for email are the MOSINT and the Proli3r.