How to Search the Deep (and Dark) Web
You may have heard that the “Dark Web” is a place where criminals, terrorists, and hackers conspire to victimize unwitting Internet users. The reality is a bit more nuanced and not so scary. There is a “Deep Web” that you can't access with ordinary search engines, and a “Dark Web” where people lurk anonymously for both good and evil purposes. What's really out there? Read on... |
It's Cold And Lonely In The Deep Dark Web
And unfortunately, there is no dashboard light to guide the way. So what's hiding in the deep and dark webs? Technically defined, the “Deep Web” is simply that vast portion of the Web that search engines don’t (or can't) index. While Google, Bing, and other search engines can provide billions more Web pages than you can live to view, that still leaves over 90% of Internet destinations unsearchable.
One source I found estimated that over 130 TRILLION individual Web pages exist. If you don’t know the URL (web page address), you can’t just “google” it. You have to find the “secret” URL some other way. However, the majority of the Deep Web is unindexed simply because it’s uninteresting. Most of the Internet of Things fits into this category. Who really wants to google a light bulb, doorbell, or toaster? (See my related article Things That Should NOT Be Connected To The Internet.)
There are also password-protected sites that are accessible only to those with memberships or subscriptions to the content stored there. If there's a lock on the door, search engines can't get in to index the pages stored at that location. That would apply to Facebook, Twitter and other social media accounts with billions of pages of user-created content. Many online newspapers, professional journals, and research databases behind "paywalls" also fall into this category.
Many local government websites offer access to public records, such as real estate and legal filings. You can access them, often without a password, but these databases will not be indexed. That's because the search engine "spider" doesn't know what to do when it sees the search input box.
And then there are websites that perform dynamic, real-time queries for things like travel. You can go to Expedia and find out how much it will cost for a ticket from NYC to Miami on July 23rd, but the results won't be indexed by search engines, because they can change from one minute to the next.
There are also websites that have no inbound links (links from pages on other sites) so search engines will never find them. And of course there are legions of websites that just have no useful content. They may be spammy, scammy, ripoffs or duplicates that will never appear in search results, because search engines have gotten wise to many of the tricks that black-hats use to "game" the search results. Back in the mid-1990s, I can recall experimenting with search engine rankings in real time. You could multiply your desired keywords on a page, submit it to the search engine, and see it rise in the search results. It's for the best that tricks like that no longer work.
The Library of Congress Online Catalog is a good example of a Deep Web resource. It's a database containing millions of records of books, periodicals, audio recordings, photographs, and more. None of its records can be retrieved directly through Google. You need visit the LOC's Online Catalog page and enter search terms in the appropriate boxes.
Web archives such as The Wayback Machine store copies of Web sites that have been modified or deleted. Such archived pages are not indexed by search engines, which strive to index the current version. I've often found the Wayback Machine handy when I want to see what a website looked like in the past. (Want to see what Yahoo.com looked like in October of 1996? It's in there.)
To find Deep Web material via Google, et. al, try adding the term, "database" to your search query. "Plane crash database," "drug interaction database," "government grants database," and so on, will often lead to the home page of a database where you can enter search terms specific to that resource.
There are also paid tools such as LexisNexis and Factiva which professional researchers use to find information about legal and business topics. Genealogy researchers can find a wealth of free information online, but often the best sources require payment. Ancestry.com is one such example. It's also becoming more common for online newspapers and magazines to limit free content, and erect paywalls that require a subscription to view more than current headlines.
My article Desperately Seeking Something? Try These Free Research Tools will point you to dozens of specialized search tools, categorized by subject matter.
So the Deep Web isn't scary -- it's just a part of the Internet that can't be (or hasn't been) indexed by search engines, or pages that require human interaction to continue on to the desired content or search query.
The Dark Side of the Internet
But what’s out there in the dark parts of the Web? Yes, there are bad places, people, and activities; they’re part of what’s called the “Dark Web” or “The Darknet” for dramatic effect. These places typically require prior authorization and special software instead of, or in addition to your everyday Web browser, to access. (See discussion of “Tor” below.)
“The Silk Road” was one infamous criminal site where drugs, weapons, data, hacking services and all manner of illicit things were traded until the FBI arrested its owner back in October, 2013. Some referred to Silk Road as the Amazon.com of the underworld, because it made shopping for illegal goods so easy. Ross William Ulbricht, known by his hacker handle "Dread Pirate Roberts," was nailed on charges of narcotics trafficking conspiracy, computer hacking conspiracy and money laundering conspiracy. Court documents allege that over $1.7 million in illegal money changed hands each month on The Silk Road. Other black market websites exist, but Silk Road was the best known.
Private forums exist where cyber-criminals offer services such as hacking, denial of service attacks, ransomware and phishing scams. There are images on dark web pages that you would wish to forget after seeing them. And certainly, terrorists use encrypted messaging channels on the Internet to communicate and collaborate.
Sometimes You Need to Hide
But there are also oases of light in the Dark Web that can’t be called dark by any means. They’re where the struggle for freedom rages. Dissidents, journalists, peace activists, and other good guys often need to hide their activities from oppressive governments and other institutions. Many citizens of totalitarian nations cannot freely access uncensored news or trade opinions and facts about politics or corruption. Some of these people turn to the Dark Web, to hidden forums, sites, and servers of information that protect their secrets and identities.
One of the most popular privacy tools is called Tor. Tor is, essentially, a network of Web proxy servers and browser software designed for them. When using the Tor browser, your identity and location are obscured and your connection to the Tor network is encrypted. Even your ISP doesn’t know where you’re really going because they can’t read the data stream that passes between you and the Tor proxy server. All anyone knows is that you accessed a Tor server.
Your requests for Web content go to a Tor server, which then reaches out to grab the requested content and relay it back to you over that encrypted connection. The destination site sees the Tor server’s location and ID but never yours. Theoretically, there is no way to tell what you accessed via a Tor server.
Along those lines, some users employ a VPN (Virtual Private Network) to protect their privacy and hide their online activities. A VPN is a private network set up on the public Internet, using encryption to ensure that no uninvited parties can eavesdrop on information that flows over the network. My article PRIVACY: Do You Need a VPN? goes into more detail on the pros and cons.
To summarize, the Deep Web is just that portion of the Internet that can't be reached directly by search engines. The Dark Web is actually a mixture of light and dark, good and evil, benefit and harm. It’s symbol might well be the Yin Yang which illustrates how opposite forces can be interconnected and intermingled.
Your thoughts on this topic are welcome. Post your comment or question below…
Most recent comments on "How to Search the Deep (and Dark) Web"
Posted by:
Hugh Gautier
18 Apr 2022
Many years ago while in Military Intelligence in the US Army (1976-1986), I spent many hours in the dark zone and was paid for it. All this after having served in the US Navy (Nov. 1963-1975) aboard the USNS Guantanamo Bay, Cuba where I was assigned to the Fleet Training Group [FTG] Electronics Communications where I had a ball. Most members didn't know what rank I was but those in charge did. I did lots of things to protect the pieces of equipment I was testing these men on. Yes, I was also under-cover as a member of the Naval Criminal Investigation Division.
My rank was CWO-2 but when I went onto these ships I was not wearing the uniform bearing my true rank. The only person who knew what rank I was was the Captain of each ship I was assigned to test. To put it mildly, I had fun doing what I was doing and catching in some cases Officers and enlisted personnel who were willing to go to Ft. Leavenworth Prison if caught, and there was a division officer and most of his men caught doing just that.
Like I said I had fun, they didn't when I'd turn up at the Court Martial, with taped conversations, and pictures. What they never saw was any of the equipment that I'd used to get those conversations and pictures. Those in charge got stiff prison time the longest one was 47 years of hard labor and the shortest was 9 years. They all got dishonorable discharges. A Commander's wife was forced to leave the base, and leave her military dependents ID Card for herself and the children because of his actions. I found that equipment being sold on the Dark Web, so yes, I'm familiar with that area and I don't recommend that anyone who isn't sufficiently prepared venture into that realm because you'll get all sorts of viruses and spyware infecting your system.
If you have it use a throwaway system that has none of your private information on it. I normally put the call-in desk at the local jail as my contact #. It worked. The best advice I can give is to stay away from the "Dark Web" it's the best thing you can do.
Posted by:
JOHN TEMBLADOR
19 Apr 2022
Great article and insights! Thank-you for sharing Bob Rankin!
Posted by:
Thom
22 Apr 2022
It is unfortunately a cesspit for disgusting Anti-Semitism, Islamaphobia, and attacks on the LGBTQIA+ community. When Google became dominant it was a beautiful thing as it pushed that evil fringe into the shadows which is where it belongs as it is hate and NOT true anyway. There is no elite who share ethnicity or religion secretly ruling the world, that is what the whackos believe, we have a fully functioning Democracy where we the people elect our leaders and that is the end of it. This only was threatened by Putin's meddling in recent years but that is why Robert Mueller went to work and rooted it out. The Dark Web is full of nonsense that's how it should be looked at.
Posted by:
Susan
14 Feb 2023
I'm convinced that the Dark Web is not the place for me to explore! I can barely keep ahead of the spammers and phishers as it is, and I don't want to view images that I can't erase from my memory. Life is difficult enough without adding such burdens.
Thanks for the informative article.