Search Tools For the Deep (and Dark) Web

Category: Reference

The “Dark Web” is sometimes portrayed as a place where criminals, terrorists, hackers, and spammers conspire to victimize unwitting Internet users. The reality is a bit more nuanced and not so scary. There is a “Deep Web” that you can't access with ordinary search engines, and a “Dark Web” where people lurk anonymously for both good and evil purposes. What's really out there? Read on...

What's Hiding in the Deep and Dark Webs?

Technically defined, the “Deep Web” is simply that vast portion of the Web that search engines don’t (or can't) index. While Google, Bing, and other search engines can provide billions more Web pages than you can live to view, that still leaves over 90% of Internet destinations unsearchable. (One source I found estimated that over 130 trillion individual Web pages exist.) If you don’t know the URL (web page address), you can’t just “google” it. You have to find the “secret” URL some other way.

The majority of the Deep Web is unindexed simply because it’s uninteresting. Most of the Internet of Things fits into this category. Who really wants to google a light bulb, doorbell, or toaster? (See my related article Things That Should NOT Be Connected To The Internet.)

There are also password-protected sites that are accessible only to those with memberships or subscriptions to the content stored there. If there's a lock on the door, search engines can't get in to index the pages stored at that location. That would apply to Facebook, Twitter and other social media accounts with billions of pages of user-created content. Online newspapers, professional journals, and research databases also fall into this category.

Many local government websites offer access to public records, such as real estate and legal filings. You can access them, often without a password, but these databases will not be indexed. That's because the search engine "spider" doesn't know what to do when it sees the search input box.

Deep Web and Dark Web

And then there are websites that perform dynamic, real-time queries for things like travel. You can go to Expedia and find out how much it will cost for a ticket from NYC to Miami on July 23rd, but the results won't be indexed by search engines, because they can change from one minute to the next.

And of course there are legions of websites that just have no useful content. They may be spammy, scammy, ripoffs or duplicates that will never appear in search results, because search engines have gotten wise to many of the tricks that black-hats use to "game" the search results. There are also websites that have no inbound links (links from pages on other sites) so search engines will never find them.

The Library of Congress Online Catalog is a good example of a Deep Web resource. It's a database containing millions of records of books, periodicals, audio recordings, photographs, and more. None of its records can be retrieved directly through Google. You need visit the LOC's Online Catalog page and enter search terms in the appropriate boxes.

Web archives such as The Wayback Machine store copies of Web sites that have been modified or deleted. Such archived pages are not indexed by search engines, which strive to index the current version. I've often found the Wayback Machine handy when I want to see what a website looked like in the past. (Want to see what Yahoo.com looked like in October of 1996? It's in there.)

To find Deep Web material via Google, et. al, try adding the term, "database" to your search query. "Plane crash database," "drug interaction database," "government grants database," and so on, will often lead to the home page of a database where you can enter search terms specific to that resource.

There are also paid tools such as LexisNexis and Factiva which professional researchers use to find information about legal and business topics. Genealogy researchers can find a wealth of free information online, but often the best sources require payment. Ancestry.com is one such example. It's also becoming more common for online newspapers and magazines to limit free content, and erect paywalls that require a subscription to view more than current headlines.

My article Free Online Research Tools will point you to dozens of specialized search tools, categorized by subject matter.

So the Deep Web isn't scary -- it's just a part of the Internet that can't be (or hasn't been) indexed by search engines, or pages that require human interaction to continue on to the desired content or search query.

The Dark Side of the Internet

But what’s out there in the dark parts of the Web? Yes, there are bad places, people, and activities; they’re part of what’s called the “Dark Web” or “Darknet” for dramatic effect. These places typically require prior authorization and special software instead of, or in addition to your everyday Web browser, to access. (See discussion of “Tor” below.)

“The Silk Road” was one infamous criminal site where drugs, weapons, data, hacking services and all manner of illicit things were traded until the FBI arrested its owner back in October, 2013. Some referred to Silk Road as the Amazon.com of the underworld, because it made shopping for illegal goods so easy. Ross William Ulbricht, known by his hacker handle "Dread Pirate Roberts," was nailed on charges of narcotics trafficking conspiracy, computer hacking conspiracy and money laundering conspiracy. Court documents allege that over $1.7 million in illegal money changed hands each month on The Silk Road. Other black market websites exist, but Silk Road was the best known.

Private forums exist where cyber-criminals offer services such as hacking, denial of service attacks, ransomware and phishing scams. There are images on dark web pages that you would wish to forget after seeing them. And certainly, terrorists use encrypted messaging channels on the Internet to communicate and collaborate.

Sometimes You Need to Hide

But there are also oases of light in the Dark Web that can’t be called dark by any means. They’re where the struggle for freedom rages. Dissidents, journalists, peace activists, and other good guys often need to hide their activities from oppressive governments and other institutions. Many citizens of totalitarian nations cannot freely access uncensored news or trade opinions and facts about politics or corruption. Some of these people turn to the Dark Web, to hidden forums, sites, and servers of information that protect their secrets and identities.

One of the most popular privacy tools is called Tor. Tor is, essentially, a network of Web proxy servers and browser software designed for them. When using the Tor browser, your identity and location are obscured and your connection to the Tor network is encrypted. Even your ISP doesn’t know where you’re really going because they can’t read the data stream that passes between you and the Tor proxy server. All anyone knows is that you accessed a Tor server.

Your requests for Web content go to a Tor server, which then reaches out to grab the requested content and relay it back to you over that encrypted connection. The destination site sees the Tor server’s location and ID but never yours. Theoretically, there is no way to tell what you accessed via a Tor server.

Along those lines, some users employ a VPN (Virtual Private Network) to protect their privacy and hide their online activities. A VPN is a private network set up on the public Internet, using encryption to ensure that no uninvited parties can eavesdrop on information that flows over the network. My article PRIVACY: Do You Need a VPN? goes into more detail on the pros and cons.

To summarize, the Deep Web is just that portion of the Internet that can't be reached directly by search engines. The Dark Web is actually a mixture of light and dark, good and evil, benefit and harm. It’s symbol might well be the Yin Yang which illustrates how opposite forces can be interconnected and intermingled.

Your thoughts on this topic are welcome. Post your comment or question below…

 
Ask Your Computer or Internet Question

 
  (Enter your question in the box above.)

It's Guaranteed to Make You Smarter...

AskBob Updates: Boost your Internet IQ & solve computer problems.
Get your FREE Subscription!


Email:

Check out other articles in this category:



Link to this article from your site or blog. Just copy and paste from this box:

This article was posted by on 27 Dec 2021


For Fun: Buy Bob a Snickers.

Prev Article:
Geekly Update - 22 December 2021

The Top Twenty
Next Article:
AskBob's Best of 2021 - Part One

Most recent comments on "Search Tools For the Deep (and Dark) Web"

Posted by:

Lucy
27 Dec 2021

We happily use a VPN. Not because we want to hide per se, but because we became aware of the vast information that gets sent to websites viewed or CHATed with or when sending them a message. To us a VPN is just another privacy tool.

We also successfully use the Wayback machine after we had requested a hike brochure and were told it was no longer available. It was not to be found on their website either. Fortunately there was enough information still legible on our ratty old copy to enable us to find the original web page using the Wayback machine, and download it for future use.

Both learned of through Ask Bob articles ... Thanks Bob!


Posted by:

Frances
27 Dec 2021

This doesn't matter in the least, but I thought you might find it interesting to know that, in the library world, the Library of Congress is known as LC, not LoC.


Way back when I went to what was then called Library School, I spent the first few weeks puzzling over this LC I kept hearing about. I had never worked in a library but many of my fellow students had and the term often came up in various discussions. Eventually, I figured it out.

As an illustration of how different things are now, we had the catalog as a set of large and heavy bound volumes. And we got an update volume every year. Online catalogs are such a boon for everyone.


Posted by:

Frances
27 Dec 2021

I should add that LC means the Library of Congress Catalog. As in "Have you looked in LC?"


Posted by:

RandiO
27 Dec 2021

SIR Bob Rankin, I bow-down to you for the BEST and the MOST informative article about/on the DarkWeb.
Your first 2 paragraphs started to make my eyes roll, as "we" have been hearing these 2-lines for more than just a few years.
But, OMG, the rest of your topic coverage is near heaven-sent.
Thank you and a Happy New Year.


Posted by:

mike
27 Dec 2021

It is not just "Dissidents, journalists, peace activists, and other good guys" that use the dark web to hide their activity and communication. I suspect the calls for flash mob robbery, spontaneous protests, and group looting is just a few examples of the dark web usage by bad guys. In fact, you may be stretching a point calling dissidents, journalists, and peace activists "good guys".

EDITOR'S NOTE: For the record, when I mentioned dissidents and journalists, I was thinking of people living in totalitarian states where it's hard or dangerous to share information. I don't see how that would stretch any point.


Posted by:

elwill
28 Dec 2021

I write articles for a US financial website - I am based in the UK -but the US site is geo-protected and can't be directly accessed from outside North America. If I want to look at the site where my article appears I use Tor - set up to give the impression I am based in the USA,


Posted by:

Ernest N. Wilcox Jr.
28 Dec 2021

First, I respond to Mike. Bob has already stated that there are bad actors who use the dark web. You are entitled to your opinion, but you would be more convincing if you provide proof of your characterizations. For example, you include calls for "spontaneous protests" with "flash mob robbery" and "group looting". IMHO, this is a misrepresentation. Most protesters are NOT bad actors. You say Bob is stretching a point by calling dissidents, journalists, and peace activists "good guys". I'll address each individually:

Dissident: The dictionary definition is "a person who opposes official policy, especially that of an authoritarian state". If no one opposes "official policy" (authoritarian or otherwise), how will Governments (or Businesses) ever change for the better? Without dissidents, the United States of America would never have come to exist. It is not the dissident that is "bad", but his/her motivation (whether it is to right a wrong, or for personal gain/greed).

Journalists: the dictionary definition is "a person who writes for newspapers, magazines, or news websites or prepares news to be broadcast". Again, the determination of whether a journalist is a "good" or "bad" actor depends on his/her motivation (present a point of view or mislead readers).

Peace Activist: the Collins Dictionary definition is "someone who advocates for peace or an end to conflicts". Yet again, the determination of whether a peace activist is a "good" or "bad" actor depends on his/her motivation.

You can characterize any group of people as bad actors, but the reality is (as Bob so aptly put it) much more nuanced. As with everything, it is not necessarily the group that is either good or bad, but the motivation of each member that must be evaluated.

Too often, we take the easy way out by putting people into groups and branding those groups as "bad". As a hypothetical example, a protest of some five thousand (or five hundred) protesters took to the street over some issue. A (smaller) group of rioters vandalized and robbed stores on the street during the protest. Does this make all five thousand protesters bad? I say not. There is a difference between a rioter and a protester. I protester takes to the street to shout out his/her opinion in hopes that others will join in. A rioter creates violence and chaos to achieve some goal. The trouble with rioters is that their activities usually result in property damage and even death. These outcomes define a riot as a "bad" act, but a riot is not the same thing as a protest.

If you can provide evidentiary proof to support your characterizations, even though I will still stand by my statements here, at least you will have clarified your post so everyone else can better understand your position.


Posted by:

Peg Picone
29 Dec 2021

Thanks for the enlightenment about the dark web...now if I could levitate...


Posted by:

Mike
29 Dec 2021

It is nice to see that Ernest agrees that Bob maybe stretching a point to label groups as good or bad.
But his defense of web notifications of spontaneous protests falls apart when looting and violence seem to find root in such protests. I still maintain the dark web is a vehicle to organize such actions.


Posted by:

Ernest N. Wilcox Jr.
29 Dec 2021

Mike, your first statement in your reply is a mischaracterization. I did not say anything about agreeing that Bob may be stretching a point to label groups as good or bad. You said that. I disagreed.

I also did not defend web notifications of spontaneous protests, I objected to mischaracterizing them. The web notification itself (like a gun) is neither good nor bad, it is simply a tool. How it is used (the intent) determines which it is. I am sure that some malevolent spontaneous protests resulting from a web notification have occurred, but these are usually better described as riots. The mischaracterization comes from the notion that because there are some malevolent spontaneous protests that result from a web notification, all spontaneous protests and web notifications are malevolent. This is simply not true.

What I did say in a nutshell was that you should judge individuals based on their actions, and not on some group-based profile.

At the end of my post, I asked for evidentiary proof, but you replied with what I can only characterize as more of your opinion. You are entitled to your opinion, but please do not state it as if it is a fact. It is NOT! Stating opinions as facts generates fake news. That is a dis-service to everyone who reads what you post.

My bottom line is this. Please judge the individual for his/her acts. Avoid grouping people - that leads to profiling them based on the group you have put them in, and not on who they really are, or by their actions.

If you see a person throw a brick through a store front window during a street protest, that individual is a criminal, and a rioter, but you cannot say that the rioter is part of the street protest, or that the people in the street protest are also criminals, unless you see them throwing bricks through store front windows. Even more importantly, based on the scenario above, no evidence has been presented that the rioter has any connection to the protest. It is much more likely that the rioter is taking advantage of the event rather than participating in it. Taking advantage of some event to perpetrate criminal activity is not uncommon but that does not mean that there is any connection between the criminal, the act, and the event other than that the criminal takes advantage of the event.

These are my opinions. The reader can do with them what they want,

Ernie


Posted by:

Robert T Deloyd
30 Dec 2021

One of your better articles, Bob.
Thanks for enlightening us :)


Posted by:

Claude
02 Jan 2022

Bob: what is your view of WEBROOT as a security tool?


Post your Comments, Questions or Suggestions

*     *     (* = Required field)

    (Your email address will not be published)
(you may use HTML tags for style)

YES... spelling, punctuation, grammar and proper use of UPPER/lower case are important! Comments of a political nature are discouraged. Please limit your remarks to 3-4 paragraphs. If you want to see your comment posted, pay attention to these items.

All comments are reviewed, and may be edited or removed at the discretion of the moderator.

NOTE: Please, post comments on this article ONLY.
If you want to ask a question click here.


Free Tech Support -- Ask Bob Rankin
Subscribe to AskBobRankin Updates: Free Newsletter

Copyright © 2005 - Bob Rankin - All Rights Reserved
Privacy Policy     RSS/XML


Article information: AskBobRankin -- Search Tools For the Deep (and Dark) Web (Posted: 27 Dec 2021)
Source: https://askbobrankin.com/search_tools_for_the_deep_and_dark_web.html
Copyright © 2005 - Bob Rankin - All Rights Reserved